Spidey v0.2.0 API Reference

Modules

A dead-simple, concurrent web crawler which focuses on ease of use and speed.

PoolManager is in charge of spinning up the pools and queues to crawl new websites. Every time a new website is to be crawled, a pool is created, along with a queue. When the job has been finished, they are terminated.