Build a Custom Crawler

Image Crawler Utils is designed to keep a balance between running a preset crawling task and building a custom image crawler as simply as possible.

Mostly, you only need to construct a Parser by inheriting the Parser or KeywordParser class. However, you should follow several requirements set by CrawlerSettings, ImageInfo, etc. Read Construct a Custom Parser for more details.

Besides, you may make use of several other classes and functions provided in the submodules of image_crawler_utils. Read Useful Classes and Functions for more details.

• Construct a Custom Parser
  • Inherit a Parser Class
    • Parser Class
    • KeywordParser class
  • Completing the Parser Structure
    • About the Usage of CrawlerSettings Class
    • KeywordParser Tips
• Useful Classes and Functions
  • Logging
  • Progress Bars
    • CustomProgress Class
      • Examples of CustomProgress
    • ProgressGroup Class
      • Examples of ProgressGroup
  • User Agents
  • Other Tools