The curated bot registry behind the Crawlers module's per-group access controls.

Bots are grouped by why they visit, because that is the axis operators actually decide on — allow search engines, block AI training scrapers, decide separately about AI assistants:

Everything defaults to allowed so that enabling the module changes nothing until the operator decides otherwise.

Tokens vs user-agents

Each bot carries two identities, and they are deliberately separate fields:

• :token — the name robots.txt addresses (User-agent: line).
• :ua — a substring found in the bot's actual request User-Agent header, used by the best-effort application-level blocker. nil for robots.txt-only tokens: Google-Extended and Applebot-Extended are opt-out signals read from robots.txt by crawlers that fetch under their main UA (Googlebot / Applebot), so there is no request to block — a UA match on them would either never fire or, worse, catch the search crawler the operator meant to allow.

The list is curated, not exhaustive — a registry of every bot on the internet goes stale weekly. These are the ones operators ask about.