Modules
Parses robots.txt files and evaluates crawler access rules.
Parsed representation of one robots.txt user-agent group.
Selects robots.txt groups and evaluates rules against escaped targets.
Byte-oriented parser used by RobotsTxt.parse/2.
Byte-oriented robots.txt pattern normalization and matching.
Parsed representation of an allow or disallow rule.