PhoenixKit.Modules.Crawlers.RobotsTxt (phoenix_kit v2.6.0)

Renders the complete robots.txt an operator should serve, from the Crawlers module's per-group toggles.

PhoenixKit deliberately does not serve this file — robots.txt is host policy and Plug.Static answers before the router is consulted, so a generated route would be dead code on most installs and a silent override on the rest. What changed with the group toggles is how much there is to say: one Sitemap: line was dictation, a per-group policy is a file. So the settings page renders the whole thing for copy-paste, and this module is the pure builder behind it.

Shape of the output

robots.txt is default-allow, so only blocked groups produce stanzas — one User-agent: line per bot token, sharing a single Disallow: /. An explicit User-agent: * / Disallow: (empty = allow all) stanza leads the file so the operator's intent is visible rather than implied, and the Sitemap: line closes it.

Summary

Functions

build(opts \\ [])

Build the robots.txt contents from the current settings.

build_for(blocked_group_keys, opts \\ [])

Pure variant: build for an explicit list of blocked group keys. build/1 reads the settings and delegates here; tests come in through this door.

Functions

build(opts \\ [])

@spec build(keyword()) :: String.t()

Build the robots.txt contents from the current settings.

Options:

:sitemap_url — absolute URL for the Sitemap: line; omitted when nil.

build_for(blocked_group_keys, opts \\ [])

@spec build_for(
  [PhoenixKit.Modules.Crawlers.Bots.group()],
  keyword()
) :: String.t()

Pure variant: build for an explicit list of blocked group keys. build/1 reads the settings and delegates here; tests come in through this door.