Judges whether fetched page text is worth handing to a model.
An HTTP 200 alone is not success. A bot challenge, a JavaScript shell that renders nothing without a browser, or a page with almost no readable text all come back as 200s. The fetch ladder uses the verdict to decide which tier to try next: a challenge is worth an impersonated retry, while a JavaScript shell needs a real browser.
Markup markers that only a challenge page carries (such as cf-chl- and
challenge-platform) count on any page of at most 4,000 characters.
Phrases a real page could also contain ("captcha", "access denied") count
only on a page of at most 1,000 characters, so an article that mentions a
captcha is not mistaken for a challenge. A page of at most 1,500 characters that
says it is behind a sign-in or subscription ("sign in to continue",
"subscribe to continue reading"), or a near-empty page holding a
password field, is a :login_wall; a prompt to sign in to comment is not.
Like a JavaScript shell, a login wall skips the impersonated rung: a
browser fingerprint does not sign in.
Summary
Functions
The verdict for a page's readable text, using raw (the undecoded HTML,
when there was any) to tell a challenge or JavaScript shell from a page
that is merely short.