Crawler v1.0.0 Crawler.Fetcher.Policer
Checks a series of conditions to determine whether it is okay to continue.
Link to this section Summary
Functions
Checks a series of conditions to determine whether it is okay to continue,
i.e. to allow Crawler.Fetcher.fetch/1
to begin its tasks
Link to this section Functions
Link to this function
police(opts)
Checks a series of conditions to determine whether it is okay to continue,
i.e. to allow Crawler.Fetcher.fetch/1
to begin its tasks.
Examples
iex> Policer.police([
iex> depth: 1,
iex> max_depths: 2,
iex> url: "http://policer/",
iex> url_filter: UrlFilter
iex> ])
{:ok, [depth: 1, max_depths: 2, url: "http://policer/", url_filter: UrlFilter]}
iex> Policer.police([
iex> depth: 2,
iex> max_depths: 2,
iex> html_tag: "a"
iex> ])
{:error, "Fetch failed 'within_fetch_depth?', with opts: [depth: 2, max_depths: 2, html_tag: \"a\"]."}
iex> Policer.police([
iex> depth: 3,
iex> max_depths: 2,
iex> html_tag: "img",
iex> url: "http://policer/hi.jpg",
iex> url_filter: UrlFilter
iex> ])
{:ok, [depth: 3, max_depths: 2, html_tag: "img", url: "http://policer/hi.jpg", url_filter: UrlFilter]}
iex> Policer.police([
iex> depth: 1,
iex> max_depths: 2,
iex> url: "ftp://hello.world"
iex> ])
{:error, "Fetch failed 'acceptable_uri_scheme?', with opts: [depth: 1, max_depths: 2, url: \"ftp://hello.world\"]."}
iex> Crawler.Store.add("http://policer/exist/")
iex> Policer.police([
iex> depth: 1,
iex> max_depths: 2,
iex> url: "http://policer/exist/"
iex> ])
{:error, "Fetch failed 'not_fetched_yet?', with opts: [depth: 1, max_depths: 2, url: \"http://policer/exist/\"]."}