Image Crawler Utils Documentation

A rather customizable image crawler structure, designed to download images with their information using multi-threading method. This GIF depicts a sample run:

Besides, several classes and functions have been implemented to help better build a custom image crawler for yourself.

Please follow the rules of robots.txt, and set a low number of threads with high number of delay time when crawling images. Frequent requests and massive download traffic may result in IP addresses being banned or accounts being suspended.

Installation

It is recommended to install it by

pip install image-crawler-utils

• Requires Python >= 3.9.

Important

nodriver are used to parse information from certain websites. It is suggested to install the latest version of Google Chrome first to ensure the crawler will be correctly running.

Guides

It is recommended to start Image Crawler Utils with the Get Started chapter.

For those using the preset crawling tasks, check out the Notes for Tasks chapter.

For those planning to construct a custom crawler, check out the Build a Custom Crawler chapter.

Quick Start

• Get Started
• CrawlerSettings Class
  • Detailed Information
  • Configs, and how to Save and Load them
• Parser Class (DanbooruKeywordParser)
  • Basic Usage
  • DanbooruKeywordParser Class
  • Standard Keyword String
  • Cookies Class
  • ImageInfo class
• Downloader Class
  • Examples of Downloader

Advanced Usage

• Notes for Tasks
  • Booru Tasks
  • Pixiv Tasks
  • Twitter / X Tasks
• Build a Custom Crawler
  • Construct a Custom Parser
  • Useful Classes and Functions

Modules

• API Reference
  • image_crawler_utils package
  • image_crawler_utils.image_downloader package
  • image_crawler_utils.stations package
  • image_crawler_utils.stations.booru package
  • image_crawler_utils.stations.pixiv package
  • image_crawler_utils.stations.twitter package