Gollum.Parser (gollum v0.5.0)

Parses a robots.txt file.

Summary

Functions

Parse the file, passed in as a simple binary.

Functions

@spec parse(binary()) :: map()

Parse the file, passed in as a simple binary.

It follows the spec defined by Google as closely as possible.

Examples

iex> alias Gollum.Parser
iex> Parser.parse("User-agent: Hello\nAllow: /hello\nDisallow: /hey")
%{"hello" => %{allowed: ["/hello"], disallowed: ["/hey"]}}