Spidey v0.2.0 API Reference
Modules
A dead-simple, concurrent web crawler which focuses on ease of use and speed.
PoolManager is in charge of spinning up the pools and queues to crawl new websites. Every time a new website is to be crawled, a pool is created, along with a queue. When the job has been finished, they are terminated.