gollum v0.1.0 Gollum

Robots.txt parser with caching. Modelled after Kryten.

Link to this section Summary

Functions

Returns whether a url is permitted. false will be returned if an error occurs

Link to this section Functions

Link to this function crawlable?(user_agent, url, opts \\ [])
crawlable?(binary, binary, keyword) :: boolean

Returns whether a url is permitted. false will be returned if an error occurs.

Options

  • name - The name of the GenServer. Default value is Gollum.Cache.

  • start_if_needed - Starts the GenServer if it doesn’t exist. Defaults to true.

Any other options passed will be passed to the internal Cache.start_link/1 call.

Examples

iex> Gollum.crawlable?("hello", "http://example.com/hello", fetcher: MockFetcher)
:crawlable
iex> Gollum.crawlable?("hello", "http://example.com/hey")
:uncrawlable