# RobotsTxt v0.1.0 - Table of Contents

> RFC 9309 robots.txt (robotstxt) parser and matcher for the Robots Exclusion Protocol.

## Pages

- [RobotsTxt](readme.md)
- [Changelog](changelog.md)
- [LICENSE](license.md)

## Modules

- [RobotsTxt](RobotsTxt.md): Parses robots.txt files and evaluates crawler access rules.
- [RobotsTxt.Group](RobotsTxt.Group.md): Parsed representation of one robots.txt user-agent group.
- [RobotsTxt.Matcher](RobotsTxt.Matcher.md): Selects robots.txt groups and evaluates rules against escaped targets.
- [RobotsTxt.Parser](RobotsTxt.Parser.md): Byte-oriented parser used by `RobotsTxt.parse/2`.
- [RobotsTxt.Pattern](RobotsTxt.Pattern.md): Byte-oriented robots.txt pattern normalization and matching.
- [RobotsTxt.Rule](RobotsTxt.Rule.md): Parsed representation of an allow or disallow rule.

