Spotting automated visitors by what they do, not only by what they call themselves.
The User-Agent check (PhoenixKitWebAnalytics.UserAgent.bot?/1) catches
every bot that announces itself. These signals catch the ones that send a
normal browser's User-Agent:
- Automation flag — the client script reports
navigator.webdriver, which Selenium, Puppeteer and Playwright set. Needs the client script. - Speed — more than 30 page views a minute from one visitor (by
default;
config :phoenix_kit_web_analytics, bot_pageviews_per_minute:); a person can't read pages that fast. - No JavaScript — a LiveView page whose live connection never came:
no exit, click, live navigation or client-script report from that
visitor all day. A browser connects within a second; a scraper that
only fetches HTML never does. Only pages that run
PhoenixKitWebAnalytics.LiveHookcount — elsewhere a missing connection proves nothing.
A visit flagged this way has its events marked is_bot with the reason in
metadata["bot"] ("webdriver", "rate", "no_js"), so every report
drops it like any declared bot, and later hits of the visit inherit the
flag (dropped, while bot traffic isn't kept). A "no_js" verdict can be
wrong, so its later hits are always kept, flagged, and the flag undoes
itself if the visit shows it ran JavaScript after all (a tab left open
without a click until its exit).
On by default; web_analytics_detect_bots switches all three off.
The no-JavaScript judgement waits while LiveView visits are being missed:
behind a reverse proxy, on an endpoint whose connect_info lacks
:x_headers, the hook can't name the visitor and records nothing (see
PhoenixKitWebAnalytics.LiveHook), so every such visit would look like
one that never connected. When the hook skipped at least 5 % of a node's
live visits in the last 24 hours, that node tells the cluster, and no
visit is judged until no node does. The count lives in memory, so a node
that just started knows nothing of the visits before it: the node that
runs the judgement leaves alone every visit that started before it had
been up for 30 minutes.
Summary
Functions
Returns a specification to start this module under a supervisor.
Clears a "no_js" flag from a session that has since shown it ran
JavaScript. Other reasons stay.
Counts one kind of report from visitor_id in the current minute and
returns the count so far (0 when the counters aren't running). The
per-visitor speed limits — page views, recording chunks — share it.
Counts a connected LiveView mount on this node: :tracked when the hook
could name the visitor, :skipped when it couldn't (a private peer and no
:x_headers). at (Unix seconds, now by default) picks the hour.
Counts a page view for visitor_id in the current minute. Returns
:ok, :crossed the moment the visitor goes over the limit, and :over
for every page view after that minute's crossing.
Whether behavioural bot detection is on (web_analytics_detect_bots).
Marks today's events of the visitor behind client as a bot's — the
client script's automation report. Runs off the caller's process.
Marks every event of session_id as a bot's, with reason. Returns the
number of events marked.
Whether an event proves its visitor ran JavaScript: any interaction, or a report from a live connection or the client script.
The sources whose reports prove a visitor ran JavaScript.
Judges the visits that started in the 2 hours before the
last 30 minutes (time enough to connect): a visit
that loaded a LiveView page running the hook, by a visitor with no
JavaScript report all day, is flagged "no_js". Run by every retention
pass. Returns the number of visits flagged.
Live visits this node tracked and skipped in the last 24 hours.
Whether live visits are being skipped often enough to pause the no-JavaScript judgement: at least 5 % of this node's in the last 24 hours, or another node said so in the last 10 minutes.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec clear_no_js(Ecto.UUID.t()) :: non_neg_integer()
Clears a "no_js" flag from a session that has since shown it ran
JavaScript. Other reasons stay.
@spec count(atom(), String.t()) :: non_neg_integer()
Counts one kind of report from visitor_id in the current minute and
returns the count so far (0 when the counters aren't running). The
per-visitor speed limits — page views, recording chunks — share it.
@spec count_live_visit(:tracked | :skipped, integer()) :: :ok
Counts a connected LiveView mount on this node: :tracked when the hook
could name the visitor, :skipped when it couldn't (a private peer and no
:x_headers). at (Unix seconds, now by default) picks the hour.
@spec count_pageview(String.t()) :: :ok | :crossed | :over
Counts a page view for visitor_id in the current minute. Returns
:ok, :crossed the moment the visitor goes over the limit, and :over
for every page view after that minute's crossing.
@spec detect?() :: boolean()
Whether behavioural bot detection is on (web_analytics_detect_bots).
Marks today's events of the visitor behind client as a bot's — the
client script's automation report. Runs off the caller's process.
@spec flag_session(Ecto.UUID.t(), String.t()) :: non_neg_integer()
Marks every event of session_id as a bot's, with reason. Returns the
number of events marked.
Whether an event proves its visitor ran JavaScript: any interaction, or a report from a live connection or the client script.
@spec js_sources() :: [String.t()]
The sources whose reports prove a visitor ran JavaScript.
@spec judge_no_js(DateTime.t(), keyword()) :: non_neg_integer()
Judges the visits that started in the 2 hours before the
last 30 minutes (time enough to connect): a visit
that loaded a LiveView page running the hook, by a visitor with no
JavaScript report all day, is flagged "no_js". Run by every retention
pass. Returns the number of visits flagged.
Stateless on purpose: the window overlaps the previous pass's, and judging
a visit twice gives the same answer (a flagged one is skipped, one with
JavaScript stays clear) — so no watermark is written, which would add an
activity-log entry every hour. Pages through the window in batches
(:batch, 2000 by default), so a busy hour is judged whole.
Judges nothing while the hook is skipping live visits
(skipping_live_visits?/0), and no visit that started before this node's
BotSignals had been up for 30 minutes.
@spec live_visits() :: %{tracked: non_neg_integer(), skipped: non_neg_integer()}
Live visits this node tracked and skipped in the last 24 hours.
@spec skipping_live_visits?() :: boolean()
Whether live visits are being skipped often enough to pause the no-JavaScript judgement: at least 5 % of this node's in the last 24 hours, or another node said so in the last 10 minutes.