Byte-oriented parser used by RobotsTxt.parse/2.
The parser performs a single line-oriented pass. It reproduces compatibility behavior such as partial BOM consumption, CR/LF handling, the 16,663-byte usable line cap, case-insensitive prefix keys, accepted directive typos, and user-agent group boundaries.
Invalid UTF-8 and malformed lines are ordinary input, not parser errors. This
module is an implementation detail; use RobotsTxt.parse/2 in application
code.
Summary
Functions
@spec parse(binary()) :: RobotsTxt.t()
Parses a robots.txt body into a RobotsTxt struct.
This lower-level entry point assumes options were already validated by
RobotsTxt.parse/2.
Example
iex> parsed = RobotsTxt.Parser.parse("User-agent: ExampleBot")
iex> [%RobotsTxt.Group{user_agents: ["ExampleBot"]}] = parsed.groups
iex> parsed.sitemaps
[]