Crawler v1.1.1 Crawler.Fetcher
Fetches pages and perform tasks on them.
Link to this section Summary
Functions
Fetches a URL by
Link to this section Functions
Link to this function
fetch(opts)
Fetches a URL by:
- verifying whether the URL needs fetching through
Crawler.Fetcher.Policer.police/1
- recording data for internal use through
Crawler.Fetcher.Recorder.record/1
- fetching the URL
- performing retries upon failed fetches through
Crawler.Fetcher.Retrier.perform/2