Perplexed.Scraper behaviour (Perplexed v0.2.0)

Copy Markdown View Source

The behaviour of one scrape tier, and the registry of built-in tiers.

A scraper turns one URL into a Perplexed.Document and, when it read HTML, the raw markup (so Perplexed.Verdict can spot challenge pages and JavaScript shells). Failures are plain outcome atoms such as :blocked, :not_found or :timeout.

Built-in scrapers, referenced by name in a Perplexed.Scrape ladder:

NameModule
:directPerplexed.Scraper.Direct
:impersonatedPerplexed.Scraper.Impersonated
:browser_runPerplexed.Scraper.BrowserRun
:firecrawlPerplexed.Scraper.Firecrawl
:deepseekPerplexed.Scraper.DeepSeek

Any module implementing this behaviour can stand in a ladder too, as Module or {Module, opts}.

Summary

Types

result()
rung()
spec()

Callbacks

configured?(keyword)

Whether the scraper can run with opts and the current settings, such as a paid one whose API key is set. Optional; defaults to true. The default ladder leaves out built-in rungs that are not configured.

cost(t, keyword)

USD cost of producing document with the rung's opts. Optional; defaults to 0 (free tiers).

retries?(outcome)

Whether this scraper is worth trying after an earlier tier failed with outcome. Optional; scrapers that leave it out are always tried.

scrape(t, keyword)

Functions

configured?(arg, opts \\ [])

Whether the rung's scraper can run with the call's opts (see configured?/1), such as per-call credentials:.

cost(arg, document)

USD cost of document produced by the rung.

resolve(name)

Resolves a ladder entry to {:ok, {name, module, opts}}, or an :invalid_request error for anything that is not a built-in name or a module implementing scrape/2.

retries?(arg, outcome)

Whether the rung's scraper should be tried after an earlier tier failed with outcome. A scraper without retries?/1 is tried after anything except :unsupported (a file type such as a PDF no earlier tier could convert), which only a scraper that says it reads such files is tried after.

time_left(opts, default)

Milliseconds a scraper may still spend: default, capped by the time left before the scrape's :deadline (a monotonic millisecond time that Perplexed.Scrape.run/2 puts in the options). At least 1.

Types

result()

@type result() ::
  {:ok, Perplexed.Document.t(), raw_html :: String.t() | nil} | {:error, atom()}

rung()

@type rung() :: {name :: atom(), module(), keyword()}

spec()

@type spec() :: atom() | module() | {module(), keyword()}

Callbacks

configured?(keyword)

(optional) 
@callback configured?(keyword()) :: boolean()

Whether the scraper can run with opts and the current settings, such as a paid one whose API key is set. Optional; defaults to true. The default ladder leaves out built-in rungs that are not configured.

cost(t, keyword)

(optional) 
@callback cost(Perplexed.Document.t(), keyword()) :: float()

USD cost of producing document with the rung's opts. Optional; defaults to 0 (free tiers).

retries?(outcome)

(optional) 
@callback retries?(outcome :: atom()) :: boolean()

Whether this scraper is worth trying after an earlier tier failed with outcome. Optional; scrapers that leave it out are always tried.

scrape(t, keyword)

@callback scrape(String.t(), keyword()) :: result()

Functions

configured?(arg, opts \\ [])

@spec configured?(rung(), keyword()) :: boolean()

Whether the rung's scraper can run with the call's opts (see configured?/1), such as per-call credentials:.

cost(arg, document)

@spec cost(rung(), Perplexed.Document.t()) :: float()

USD cost of document produced by the rung.

resolve(name)

@spec resolve(spec()) :: {:ok, rung()} | {:error, {:invalid_request, map()}}

Resolves a ladder entry to {:ok, {name, module, opts}}, or an :invalid_request error for anything that is not a built-in name or a module implementing scrape/2.

retries?(arg, outcome)

@spec retries?(rung(), atom()) :: boolean()

Whether the rung's scraper should be tried after an earlier tier failed with outcome. A scraper without retries?/1 is tried after anything except :unsupported (a file type such as a PDF no earlier tier could convert), which only a scraper that says it reads such files is tried after.

time_left(opts, default)

@spec time_left(keyword(), pos_integer()) :: pos_integer()

Milliseconds a scraper may still spend: default, capped by the time left before the scrape's :deadline (a monotonic millisecond time that Perplexed.Scrape.run/2 puts in the options). At least 1.