Crawler v0.3.1 Crawler
A high performance web crawler in Elixir.
Link to this section Summary
Functions
Enqueues a crawl, via Crawler.QueueHandler.enqueue/1
Crawls immediately, this is used by Crawler.Dispatcher.Worker.start_link/1
Crawler is an application that gets started automatically with
Link to this section Functions
Link to this function
crawl(url, opts \\ [])
Enqueues a crawl, via Crawler.QueueHandler.enqueue/1
.
This is the default crawl behaviour as the queue determines when an actual
crawl should happen based on the available workers and the rate limit. The
queue kicks off Crawler.Dispatcher.Worker
which in term calls
Crawler.crawl_now/1
.
Link to this function
crawl_now(opts)
Crawls immediately, this is used by Crawler.Dispatcher.Worker.start_link/1
.
For general purpose use cases, always use Crawler.crawl/2
instead.
Link to this function
start(type, args)
Crawler is an application that gets started automatically with:
- a
Crawler.Store
that initiates aRegistry
for keeping internal data