Perplexed.Scrape (Perplexed v0.1.0)

Scrapes a URL to Markdown by climbing a ladder of Perplexed.Scrapers, cheapest first.

The default ladder is [:direct, :impersonated, :firecrawl]: plain HTTP, then a browser TLS fingerprint, both free, then Firecrawl's hosted scraper, which renders JavaScript and gets past most bot challenges. Swap any rung for another scraper, for example [:direct, :impersonated, :browser_run], or for your own module implementing Perplexed.Scraper.

The URL is checked by Perplexed.UrlGuard once, before any rung, so a private address never reaches a hosted scraper either. Each rung's text is judged by Perplexed.Verdict. After a failure the climb skips rungs whose Perplexed.Scraper.retries?/1 says they cannot help with that outcome; impersonation, for one, does nothing for a JavaScript shell. A missing page, a redirect into a private network, a scanned PDF needing OCR, an encrypted PDF, an unsupported file type or an oversized body ends the climb.

A scrape has one deadline (:timeout). Each rung gets only the time left, and once it has passed the next rung is recorded as :deadline and the climb stops. Every rung's cost counts, including a paid rung whose page then failed the readability check.

Summary

Types

attempts()

result()

Functions

batch(urls, opts \\ [])

Scrapes urls concurrently and returns {url, result} pairs in input order.

cost(arg1)

USD cost of a scrape result, whether it succeeded or every rung failed.

default_ladder()

The ladder run/2 uses when none is given: config :perplexed, ladder:, else [:direct, :impersonated, :firecrawl].

run(url, opts \\ [])

Scrapes url, returning the first rung's readable Perplexed.Document.

Types

attempts()

@type attempts() :: [{atom(), atom()}]

result()

@type result() ::
  {:ok, Perplexed.Document.t()}
  | {:error, {:unreadable, %{attempts: attempts(), cost: float()}}}
  | {:error, Perplexed.error()}

Functions

batch(urls, opts \\ [])

@spec batch([String.t()], keyword()) :: [{String.t(), result()}]

Scrapes urls concurrently and returns {url, result} pairs in input order.

Each scrape runs in an unlinked task under Perplexed.TaskSupervisor, with bounded concurrency and a per-URL timeout that kills the task, so a crash or a hang becomes that URL's {:error, {:exit, %{reason: reason}}} instead of taking the caller down.

Options

:max_concurrency - default 6.
:timeout - each URL's scrape deadline (see run/2); the task is killed 1000 ms after it.
:task_supervisor - default Perplexed.TaskSupervisor.

Other options go to run/2.

cost(arg1)

@spec cost(result()) :: float()

USD cost of a scrape result, whether it succeeded or every rung failed.

default_ladder()

@spec default_ladder() :: [Perplexed.Scraper.spec()]

The ladder run/2 uses when none is given: config :perplexed, ladder:, else [:direct, :impersonated, :firecrawl].

run(url, opts \\ [])

@spec run(String.t(), keyword()) :: result()

Scrapes url, returning the first rung's readable Perplexed.Document.

When every rung fails, answers {:error, {:unreadable, %{attempts: attempts, cost: cost}}}, where attempts lists each rung tried with its outcome and cost is what those rungs cost. A blank URL, one the URL guard refuses, or an unknown ladder entry is {:error, {:invalid_request, %{reason: reason, ...}}}.

Options

:ladder - the Perplexed.Scraper.spec/0 entries to try, in order, default default_ladder/0.
:timeout - the whole scrape's deadline in milliseconds, default 60000.
:url_validator - a {Module, :fun, args} URL check for this call, overriding the configured one (see Perplexed.UrlGuard).

Other options go to every scraper; per-scraper options ride on a {Module, opts} ladder entry.