Copyright © 2019 Antoine Gagné
Authors: Antoine Gagné (gagnantoine@gmail.com).
agent() = binary()
abstract datatype: agent_rules()
code() = 100..599
content() = string() | binary()
sitemap() = binary()
is_allowed/3 | Verifies that the given URL is allowed for the specified agent. |
parse/2 | Parses the content of the robot.txt and returns all the rules indexed by their agents. |
sitemap/1 | Fetches the sitemap of the parsed index. |
is_allowed(Agent::agent(), Url::uri_string:uri_string(), RulesIndex::agent_rules()) -> boolean()
Verifies that the given URL is allowed for the specified agent.
parse(Content::content(), Code::code()) -> {ok, agent_rules()} | {error, term()}
Parses the content of the robot.txt and returns all the rules indexed by their agents.
sitemap(RulesIndex::agent_rules()) -> {ok, sitemap()} | {error, not_found}
Fetches the sitemap of the parsed index.
Generated by EDoc