Parsed representation of one robots.txt user-agent group.
A group contains consecutive user-agent declarations and the directives that apply to them. Values remain in file order so matching and inspection can reproduce source precedence.
This module is an implementation detail. Its fields may change before
version 1.0; use functions on RobotsTxt for application behavior.
Summary
Types
@type t() :: %RobotsTxt.Group{ crawl_delays: [binary()], extensions: %{optional(binary()) => [binary()]}, rules: [RobotsTxt.Rule.t()], user_agents: [binary()] }
An inspectable parsed group.
rules contains allow and disallow rules, crawl_delays retains raw values,
and extensions maps lowercased directive names to values in file order.