Crawler v1.1.1 Crawler
A high performance web crawler in Elixir.
Link to this section Summary
Functions
Enqueues a crawl, via Crawler.QueueHandler.enqueue/1
.
Crawls immediately, this is used by Crawler.Dispatcher.Worker.start_link/1
.
Pauses the crawler.
Resumes the crawler after it was paused.
Crawler is an application that gets started automatically with
Stops the crawler.
Link to this section Functions
crawl(url, opts \\ [])
Enqueues a crawl, via Crawler.QueueHandler.enqueue/1
.
This is the default crawl behaviour as the queue determines when an actual
crawl should happen based on the available workers and the rate limit. The
queue kicks off Crawler.Dispatcher.Worker
which in turn calls
Crawler.crawl_now/1
.
crawl_now(opts)
Crawls immediately, this is used by Crawler.Dispatcher.Worker.start_link/1
.
For general purpose use cases, always use Crawler.crawl/2
instead.
pause(opts)
Pauses the crawler.
resume(opts)
Resumes the crawler after it was paused.
start(type, args)
Crawler is an application that gets started automatically with:
- a
Crawler.Store
that initiates aRegistry
for keeping internal data
stop(opts)
Stops the crawler.