Selects robots.txt groups and evaluates rules against escaped targets.
The matcher reproduces Google's specific/global precedence and four-slot
accumulation model. This module is an implementation detail; applications
should call RobotsTxt.allowed?/3 or RobotsTxt.matched_rule/3.
Summary
Functions
Returns the rule that decides a target, or :default when no rule decides.
Returns the path, parameters, and query portion of a target.
Returns the groups selected for non-standard metadata accessors.
Functions
@spec matched_rule(RobotsTxt.t(), binary(), binary()) :: {:allow | :disallow, binary(), pos_integer()} | :default
Returns the rule that decides a target, or :default when no rule decides.
Returns the path, parameters, and query portion of a target.
Scheme, authority, and fragment are removed. The result always begins with
/, and malformed or pathless targets fall back to /.
@spec selected_groups(RobotsTxt.t(), binary()) :: [RobotsTxt.Group.t()]
Returns the groups selected for non-standard metadata accessors.
All matching specific groups are returned when any exist; otherwise all matching global groups are returned. File order is preserved.