Spotting automated visitors by what they do, not only by what they call themselves.
The User-Agent check (PhoenixKitWebAnalytics.UserAgent.bot?/1) catches
every bot that announces itself. These signals catch the ones that send a
normal browser's User-Agent:
- Automation flag — the client script reports
navigator.webdriver, which Selenium, Puppeteer and Playwright set. Needs the client script. - Speed — more than 30 page views a minute from one visitor (by
default;
config :phoenix_kit_web_analytics, bot_pageviews_per_minute:); a person can't read pages that fast. - No JavaScript — a LiveView page whose live connection never came:
no exit, click, live navigation or client-script report from that
visitor all day. A browser connects within a second; a scraper that
only fetches HTML never does. Only pages that run
PhoenixKitWebAnalytics.LiveHookcount — elsewhere a missing connection proves nothing.
A visit flagged this way has its events marked is_bot with the reason in
metadata["bot"] ("webdriver", "rate", "no_js"), so every report
drops it like any declared bot, and later hits of the visit inherit the
flag (dropped, while bot traffic isn't kept). A "no_js" verdict can be
wrong, so its later hits are always kept, flagged, and the flag undoes
itself if the visit shows it ran JavaScript after all (a tab left open
without a click until its exit).
On by default; web_analytics_detect_bots switches all three off.
Summary
Functions
Returns a specification to start this module under a supervisor.
Clears a "no_js" flag from a session that has since shown it ran
JavaScript. Other reasons stay.
Counts one kind of report from visitor_id in the current minute and
returns the count so far (0 when the counters aren't running). The
per-visitor speed limits — page views, recording chunks — share it.
Counts a page view for visitor_id in the current minute. Returns
:ok, :crossed the moment the visitor goes over the limit, and :over
for every page view after that minute's crossing.
Whether behavioural bot detection is on (web_analytics_detect_bots).
Marks today's events of the visitor behind client as a bot's — the
client script's automation report. Runs off the caller's process.
Marks every event of session_id as a bot's, with reason. Returns the
number of events marked.
Whether an event proves its visitor ran JavaScript: any interaction, or a report from a live connection or the client script.
The sources whose reports prove a visitor ran JavaScript.
Judges the visits that started in the 2 hours before the
last 30 minutes (time enough to connect): a visit
that loaded a LiveView page running the hook, by a visitor with no
JavaScript report all day, is flagged "no_js". Run by every retention
pass. Returns the number of visits flagged.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec clear_no_js(Ecto.UUID.t()) :: non_neg_integer()
Clears a "no_js" flag from a session that has since shown it ran
JavaScript. Other reasons stay.
@spec count(atom(), String.t()) :: non_neg_integer()
Counts one kind of report from visitor_id in the current minute and
returns the count so far (0 when the counters aren't running). The
per-visitor speed limits — page views, recording chunks — share it.
@spec count_pageview(String.t()) :: :ok | :crossed | :over
Counts a page view for visitor_id in the current minute. Returns
:ok, :crossed the moment the visitor goes over the limit, and :over
for every page view after that minute's crossing.
@spec detect?() :: boolean()
Whether behavioural bot detection is on (web_analytics_detect_bots).
Marks today's events of the visitor behind client as a bot's — the
client script's automation report. Runs off the caller's process.
@spec flag_session(Ecto.UUID.t(), String.t()) :: non_neg_integer()
Marks every event of session_id as a bot's, with reason. Returns the
number of events marked.
Whether an event proves its visitor ran JavaScript: any interaction, or a report from a live connection or the client script.
@spec js_sources() :: [String.t()]
The sources whose reports prove a visitor ran JavaScript.
@spec judge_no_js(DateTime.t(), keyword()) :: non_neg_integer()
Judges the visits that started in the 2 hours before the
last 30 minutes (time enough to connect): a visit
that loaded a LiveView page running the hook, by a visitor with no
JavaScript report all day, is flagged "no_js". Run by every retention
pass. Returns the number of visits flagged.
Stateless on purpose: the window overlaps the previous pass's, and judging
a visit twice gives the same answer (a flagged one is skipped, one with
JavaScript stays clear) — so no watermark is written, which would add an
activity-log entry every hour. Pages through the window in batches
(:batch, 2000 by default), so a busy hour is judged whole.