Verifies that a request claiming to come from a search engine crawler really does, with forward-confirmed reverse DNS (FCrDNS): the check search engines document themselves, as Google does for Googlebot.
Any automated client that names itself (the :crawler, :ai_crawler
and :link_preview families of Limen.Signal.UserAgent) makes a claim
this signal checks, when the configured crawlers list the domains of the
one it names.
A crawler's address must point back, through its reverse DNS (PTR)
record, to a host under the crawler's published domains (for Googlebot,
googlebot.com or google.com), and that host must resolve back to the
same address. Anyone can put Googlebot in a user agent; only Google
controls those DNS records.
DNS is far too slow for the request path, so this signal only ever reads a
cache. The first request from an unverified crawler address puts it on a
bounded queue and reports :pending; Limen.Signal.Fcrdns.Resolver
resolves queued addresses in the background and caches the result, which
later requests read.
Provides :fcrdns:
:verified- the address belongs to the crawler it claims to be;:failed- it does not: the user agent is spoofed;:pending- verification is queued or a lookup failed transiently;:unverifiable- a crawler Limen has no DNS suffixes for;:not_claimed- the user agent does not claim to be a crawler.
The evidence names the claimed crawler and, once resolved, the host or the
reason verification failed. Crawlers and their suffixes are configured with
the :fcrdns option, see Limen.Config.
Summary
Functions
Verifies that ip belongs to a crawler whose hosts are under suffixes,
with the dns module and at most timeout milliseconds per lookup.
Types
Functions
@spec verify(:inet.ip_address(), [String.t()], module(), pos_integer()) :: {:verified, String.t()} | {:failed, term()} | {:error, :transient}
Verifies that ip belongs to a crawler whose hosts are under suffixes,
with the dns module and at most timeout milliseconds per lookup.
Returns {:verified, host}, {:failed, reason} or {:error, :transient}.