Spidey v0.2.0 Spidey.Crawler.PoolManager View Source
PoolManager is in charge of spinning up the pools and queues to crawl new websites. Every time a new website is to be crawled, a pool is created, along with a queue. When the job has been finished, they are terminated.
Link to this section Summary
Functions
Returns a specification to start this module under a supervisor.
Starts a pool of crawlers and a queue under the supervisor.
Starts the supervisor.
Terminates both the pool of crawlers and the queue they use.
Link to this section Functions
Returns a specification to start this module under a supervisor.
See Supervisor
.
Starts a pool of crawlers and a queue under the supervisor.
Starts the supervisor.
Terminates both the pool of crawlers and the queue they use.