gollum v0.1.0 Gollum
Robots.txt parser with caching. Modelled after Kryten.
Link to this section Summary
Functions
Returns whether a url is permitted. false
will be returned if an error occurs
Link to this section Functions
Link to this function
crawlable?(user_agent, url, opts \\ [])
crawlable?(binary, binary, keyword) :: boolean
Returns whether a url is permitted. false
will be returned if an error occurs.
Options
name
- The name of the GenServer. Default value isGollum.Cache
.start_if_needed
- Starts the GenServer if it doesn’t exist. Defaults totrue
.
Any other options passed will be passed to the internal Cache.start_link/1
call.
Examples
iex> Gollum.crawlable?("hello", "http://example.com/hello", fetcher: MockFetcher)
:crawlable
iex> Gollum.crawlable?("hello", "http://example.com/hey")
:uncrawlable