PhoenixKitWeb.Plugs.CrawlerBlocker (phoenix_kit v2.7.0)

Copy Markdown View Source

Best-effort application-level enforcement of the Crawlers module's bot policy: requests whose User-Agent matches a blocked group's bots are answered 403 instead of the page.

What this is, honestly

robots.txt is a request; this plug is for bots that ignore it. It is advisory-grade: anything can send any User-Agent, so a scraper that lies passes through, and real enforcement belongs at the CDN or reverse proxy. The settings page says the same to the operator. It is also deliberately default-off (crawlers_block_at_app setting) — three settings reads per request is a real cost, even ETS-cached, and silent 403s are a support burden the operator should opt into knowingly.

Robots.txt-only tokens (Google-Extended, Applebot-Extended) have no request UA of their own and are excluded from matching by the registry (Bots.ua_fragments/1), so blocking AI training can never 403 Googlebot.

Wiring

Core pipes its own kit routes through this plug. The host's routes are the host's: add it to the host :browser pipeline for full coverage —

pipeline :browser do
  # ...
  plug PhoenixKitWeb.Plugs.CrawlerBlocker
end

All reads are guarded: with no database (installer context, doctor) the plug passes everything through.