Gollum.Parser (gollum v0.5.0)
Parses a robots.txt file.
Summary
Functions
Parse the file, passed in as a simple binary.
Functions
Link to this function
parse(string)
Parse the file, passed in as a simple binary.
It follows the spec defined by Google as closely as possible.
Examples
iex> alias Gollum.Parser
iex> Parser.parse("User-agent: Hello\nAllow: /hello\nDisallow: /hey")
%{"hello" => %{allowed: ["/hello"], disallowed: ["/hey"]}}