Spidey v0.3.2 Spidey.Crawler View Source

Link to this section Summary

Types

The options passed to Spidey.Crawler.crawl/3. Currently only allows the specification of the filter to apply to crawled URLs, uses Spidey.Filter.DefaultFilter by default.

Functions

Crawls a given url synchronously through a supervision tree under the name of crawler_name. It accepts the filter to apply to crawled urls through the filter option.

Link to this section Types

Specs

crawl_options() :: [{:filter, module()}]

The options passed to Spidey.Crawler.crawl/3. Currently only allows the specification of the filter to apply to crawled URLs, uses Spidey.Filter.DefaultFilter by default.

Link to this section Functions

Link to this function

crawl(seed, crawler_name, opts)

View Source

Specs

crawl(String.t(), atom(), crawl_options()) :: [String.t()]

Crawls a given url synchronously through a supervision tree under the name of crawler_name. It accepts the filter to apply to crawled urls through the filter option.