Perplexed.Verdict (Perplexed v0.2.0)

Copy Markdown View Source

Judges whether fetched page text is worth handing to a model.

An HTTP 200 alone is not success. A bot challenge, a JavaScript shell that renders nothing without a browser, or a page with almost no readable text all come back as 200s. The fetch ladder uses the verdict to decide which tier to try next: a challenge is worth an impersonated retry, while a JavaScript shell needs a real browser.

Markup markers that only a challenge page carries (such as cf-chl- and challenge-platform) count on any page of at most 4,000 characters. Phrases a real page could also contain ("captcha", "access denied") count only on a page of at most 1,000 characters, so an article that mentions a captcha is not mistaken for a challenge. A page of at most 1,500 characters that says it is behind a sign-in or subscription ("sign in to continue", "subscribe to continue reading"), or a near-empty page holding a password field, is a :login_wall; a prompt to sign in to comment is not. Like a JavaScript shell, a login wall skips the impersonated rung: a browser fingerprint does not sign in.

Summary

Functions

The verdict for a page's readable text, using raw (the undecoded HTML, when there was any) to tell a challenge or JavaScript shell from a page that is merely short.

Types

t()

@type t() :: :ok | :challenge | :login_wall | :js_shell | :thin

Functions

judge(text, raw \\ nil)

@spec judge(String.t(), String.t() | nil) :: t()

The verdict for a page's readable text, using raw (the undecoded HTML, when there was any) to tell a challenge or JavaScript shell from a page that is merely short.