A deliberately small user agent classifier.
It answers the questions bot scoring needs, not the ones analytics does: does the agent claim to be a browser, which engine (and therefore which headers it must send), or which kind of automated client, and which one (for DNS verification, and for rules about a particular one)?
Families
Browsers are :chrome, :edge, :opera, :samsung, :firefox and
:safari. Automated clients that name themselves are:
:crawler- search engines, SEO tools, and anything else calling itself a bot, crawler or spider;:ai_crawler- crawlers and fetchers of AI companies, gathering training data or answering their users' questions, such asGPTBot,ClaudeBot,CCBotorBytespider, after the community list at ai.robots.txt;:link_preview- services fetching a shared link to show a preview, such asfacebookexternalhit,Slackbot,DiscordbotorWhatsApp;:headless- browsers driven by automation;:tool- HTTP libraries and command line tools.
Anything else is :other, and a missing or empty user agent is :none.
Known automated clients also get a name, such as "googlebot" or
"gptbot". Anyone can claim any of these; Limen.Signal.Fcrdns verifies
the claims that can be verified.
The lists are built in and kept current with Limen's releases. The
:user_agents option of Limen.Config extends and overrides them per
instance, by family, with the tokens to look for, matched exactly as they
appear in user agents (case matters):
config :my_app, Limen,
user_agents: [
ai_crawler: ["NewAIBot"],
link_preview: [{"ChatUnfurler/", "chat-unfurler"}],
crawler: ["Amazonbot"],
ignore: ["Mastodon/"]
]A token given as a string is named after itself, in lowercase and without
a trailing /; {token, name} names it explicitly. Listing a built-in
token under another family moves it there, and :ignore removes built-in
tokens altogether.
How it works
Non-browser markers and browser tokens are found with a single pass of a
precompiled Aho-Corasick pattern, which matches many strings at once, and
platforms with a second one: a pass reports matches that do not overlap,
and platform names can overlap markers, as Windows does
WindowsPowerShell/. The leftmost marker wins, so a crawler that embeds a
Chrome token in its user agent is still a crawler. An instance with tokens
of its own looks for them in a third pass.
Summary
Types
An instance's own tokens (see the :user_agents option), as
Limen.Config prepares them.
Functions
Classifies a user agent string with the built-in lists only.
Classifies a user agent string with the built-in lists and an instance's
own tokens (its config.user_agents).
Types
@type custom() :: %{ markers: %{ required(String.t()) => {:marker, family(), String.t() | nil} | :ignore }, pattern: :binary.cp() | nil }
An instance's own tokens (see the :user_agents option), as
Limen.Config prepares them.
@type family() ::
:chrome
| :edge
| :opera
| :samsung
| :firefox
| :safari
| :crawler
| :ai_crawler
| :link_preview
| :headless
| :tool
| :other
| :none
@type t() :: %{ family: family(), name: String.t() | nil, version: non_neg_integer() | nil, engine: :chromium | :gecko | :webkit | nil, platform: :windows | :macos | :linux | :android | :ios | :chromeos | nil, mobile: boolean() }
Functions
Classifies a user agent string with the built-in lists only.
iex> Limen.Signal.UserAgent.parse("curl/8.5.0")
%{family: :tool, name: "curl", version: nil, engine: nil, platform: nil, mobile: false}
iex> ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " <>
...> "(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
iex> Limen.Signal.UserAgent.parse(ua)
%{family: :chrome, name: nil, version: 128, engine: :chromium, platform: :windows, mobile: false}Classifies a user agent string with the built-in lists and an instance's
own tokens (its config.user_agents).
iex> custom = Limen.Config.build(user_agents: [tool: ["MyMonitor/"]]).user_agents
iex> Limen.Signal.UserAgent.parse("MyMonitor/2.0", custom).family
:tool