Judges whether fetched page text is worth handing to a model.
An HTTP 200 alone is not success. A bot challenge, a JavaScript shell that renders nothing without a browser, or a page with almost no readable text all come back as 200s. The fetch ladder uses the verdict to decide which tier to try next: a challenge is worth an impersonated retry, while a JavaScript shell needs a real browser.
Markup markers that only a challenge page carries (cf-chl-,
challenge-platform) count on any page under 4,000 characters. Phrases a
real page could also contain ("captcha", "access denied") count only on a
page with almost no readable text, so an article that mentions a captcha
is not mistaken for a challenge.
Summary
Functions
The verdict for a page's readable text, using raw (the undecoded HTML,
when there was any) to tell a challenge or JavaScript shell from a page
that is merely short.