Module robots

Parse and manipulate robots.txt files according to the specification.

Copyright © 2019 Antoine Gagné

Authors: Antoine Gagné (gagnantoine@gmail.com).

Description

Parse and manipulate robots.txt files according to the specification.

Data Types

agent()

agent() = binary()

agent_rules()

abstract datatype: agent_rules()

code()

code() = 100..599

content()

content() = string() | binary()

sitemap()

sitemap() = binary()

Function Index

is_allowed/3Verifies that the given URL is allowed for the specified agent.
parse/2Parses the content of the robot.txt and returns all the rules indexed by their agents.
sitemap/1Fetches the sitemap of the parsed index.

Function Details

is_allowed/3

is_allowed(Agent::agent(), Url::uri_string:uri_string(), RulesIndex::agent_rules()) -> boolean()

Verifies that the given URL is allowed for the specified agent.

parse/2

parse(Content::content(), Code::code()) -> {ok, agent_rules()} | {error, term()}

Parses the content of the robot.txt and returns all the rules indexed by their agents.

sitemap/1

sitemap(RulesIndex::agent_rules()) -> {ok, sitemap()} | {error, not_found}

Fetches the sitemap of the parsed index.


Generated by EDoc