Crawlers module — who may read this site, and what we tell them.
Renamed from PhoenixKit.Modules.SEO (V172 migrates the stored settings and
permission rows): everything here is bot policy — indexing directives,
per-bot-group access, crawler guidance — while actual SEO work (rank
tracking) lives in the external phoenix_kit_seo package. The old name
promised meta tags and Open Graph, which were never here (Open Graph is its
own module).
What it controls
- A global
noindex, nofollowrobots directive — the staging-safety switch, injected into every layout head. - Per-group bot access toggles (
PhoenixKit.Modules.Crawlers.Bots), which drive the generatedrobots.txtcontents (PhoenixKit.Modules.Crawlers.RobotsTxt) and, optionally, best-effort application-level blocking (PhoenixKitWeb.Plugs.CrawlerBlocker). llms.txt— a generated markdown summary served at/llms.txttelling AI assistants what this site is about.- Search-engine verification meta tags (Google Search Console, Bing).
All access toggles default to allowed and enforcement defaults to off, so enabling the module changes nothing until the operator decides.
Summary
Functions
Current Bing (msvalidate.01) verification content (raw setting).
Whether best-effort application-level blocking is on. Default off: robots.txt is the honest mechanism, and UA-matching is advisory-grade enforcement for bots that ignore it.
Keys of the groups currently blocked, in registry order.
Disables the Crawlers module and clears any active directives.
Disables the global noindex, nofollow directive.
Enables the Crawlers module (exposes the settings page).
Enables the global noindex, nofollow directive.
Returns configuration metadata for dashboard cards and settings pages.
Current Google Search Console verification content (raw setting).
Whether a bot group is currently allowed. Unknown groups read as allowed — the registry is the allowlist of what can be blocked, and failing open matches the module's default-allow stance.
The settings key holding a group's allow/block state.
Operator-provided extra markdown appended to the generated llms.txt.
Indicates whether the Crawlers module is available in the admin.
Returns true when the noindex, nofollow directive is active.
Enable or disable application-level blocking of blocked groups' UAs.
Allow or block a bot group.
Store the operator's extra llms.txt markdown.
Updates the directive to the provided boolean value.
Store the verification contents. Values are trimmed; pasting a full
<meta ...> tag is a common slip, so the content="..." value is extracted
from one rather than stored verbatim.
Verification meta tags to inject into layout heads, as {name, content}
tuples. Empty contents are dropped, so unset slots render nothing.
Functions
Current Bing (msvalidate.01) verification content (raw setting).
Whether best-effort application-level blocking is on. Default off: robots.txt is the honest mechanism, and UA-matching is advisory-grade enforcement for bots that ignore it.
@spec blocked_groups() :: [PhoenixKit.Modules.Crawlers.Bots.group()]
Keys of the groups currently blocked, in registry order.
Disables the Crawlers module and clears any active directives.
Disables the global noindex, nofollow directive.
Enables the Crawlers module (exposes the settings page).
Enables the global noindex, nofollow directive.
Returns configuration metadata for dashboard cards and settings pages.
Current Google Search Console verification content (raw setting).
@spec group_allowed?(PhoenixKit.Modules.Crawlers.Bots.group() | String.t()) :: boolean()
Whether a bot group is currently allowed. Unknown groups read as allowed — the registry is the allowlist of what can be blocked, and failing open matches the module's default-allow stance.
@spec group_setting_key(PhoenixKit.Modules.Crawlers.Bots.group() | String.t()) :: String.t()
The settings key holding a group's allow/block state.
Operator-provided extra markdown appended to the generated llms.txt.
Indicates whether the Crawlers module is available in the admin.
Returns true when the noindex, nofollow directive is active.
Enable or disable application-level blocking of blocked groups' UAs.
@spec update_group_allowed( PhoenixKit.Modules.Crawlers.Bots.group() | String.t(), boolean() ) :: {:ok, term()} | {:error, term()}
Allow or block a bot group.
Store the operator's extra llms.txt markdown.
Updates the directive to the provided boolean value.
Store the verification contents. Values are trimmed; pasting a full
<meta ...> tag is a common slip, so the content="..." value is extracted
from one rather than stored verbatim.
Verification meta tags to inject into layout heads, as {name, content}
tuples. Empty contents are dropped, so unset slots render nothing.