Perplexed.Verdict (Perplexed v0.1.0)

Copy Markdown View Source

Judges whether fetched page text is worth handing to a model.

An HTTP 200 alone is not success. A bot challenge, a JavaScript shell that renders nothing without a browser, or a page with almost no readable text all come back as 200s. The fetch ladder uses the verdict to decide which tier to try next: a challenge is worth an impersonated retry, while a JavaScript shell needs a real browser.

Markup markers that only a challenge page carries (cf-chl-, challenge-platform) count on any page under 4,000 characters. Phrases a real page could also contain ("captcha", "access denied") count only on a page with almost no readable text, so an article that mentions a captcha is not mistaken for a challenge.

Summary

Functions

The verdict for a page's readable text, using raw (the undecoded HTML, when there was any) to tell a challenge or JavaScript shell from a page that is merely short.

Types

t()

@type t() :: :ok | :challenge | :js_shell | :thin

Functions

judge(text, raw \\ nil)

@spec judge(String.t(), String.t() | nil) :: t()

The verdict for a page's readable text, using raw (the undecoded HTML, when there was any) to tell a challenge or JavaScript shell from a page that is merely short.