Spidey v0.2.0 Spidey.Crawler.PoolManager View Source

PoolManager is in charge of spinning up the pools and queues to crawl new websites. Every time a new website is to be crawled, a pool is created, along with a queue. When the job has been finished, they are terminated.

Link to this section Summary

Functions

Returns a specification to start this module under a supervisor.

Starts a pool of crawlers and a queue under the supervisor.

Starts the supervisor.

Terminates both the pool of crawlers and the queue they use.

Link to this section Functions

Returns a specification to start this module under a supervisor.

See Supervisor.

Link to this function

start_child(pool_name, opts \\ [])

View Source

Starts a pool of crawlers and a queue under the supervisor.

Starts the supervisor.

Link to this function

terminate_child(pool_name)

View Source

Terminates both the pool of crawlers and the queue they use.