CDPEx (CDPEx v0.12.0)

Copy Markdown View Source

OTP-native Chrome DevTools Protocol (CDP) browser automation for Elixir.

CDPEx launches a headless Chrome process and drives it directly over the Chrome DevTools Protocol on a Mint.WebSocket connection — no ChromeDriver and no Node.js. Browsers and their WebSocket connections are supervised processes, so a Chrome crash surfaces to callers as {:error, reason} rather than a hung session.

This module is the high-level facade. See CDPEx.Page for page operations.

Example

{:ok, browser} = CDPEx.launch()
{:ok, page} = CDPEx.new_page(browser)
{:ok, _page} = CDPEx.Page.navigate(page, "https://example.com")
{:ok, html} = CDPEx.Page.html(page)
:ok = CDPEx.stop(browser)

Or, resource-safe, with with_page/3:

CDPEx.with_page([], fn page ->
  {:ok, _} = CDPEx.Page.navigate(page, "https://example.com")
  CDPEx.Page.html(page)
end)

Observability is via :telemetry — see CDPEx.Telemetry for the event taxonomy (launch / navigate spans, page open/close, with_page/3 deadlines, and error events). Silent by default.

Error handling

Every operation returns {:error, reason} on failure; error_reason/0 documents the reason shapes. That includes runtime failures — Chrome that can't be spawned, a browser, page or pool process that is gone or stops mid-call, a call it can't serve in time — which come back as values, never as a raise or an exit in the caller. The guarantee covers return values, not links: a browser from launch/1 or connect/2 stays linked to the process that started it, so if Chrome dies, a starter that doesn't trap exits still receives the exit signal (run it under a supervisor; with_page/3 with launch options contains this for its throwaway browser). Invalid arguments (a bad :deadline or :wait, a wrong type) still raise; that is a bug in the calling code. So does anything with_page/3's own fun raises, throws or exits with. To drive retries without hard-coding that list, classify the reason instead of matching it:

case CDPEx.Page.navigate(page, url) do
  {:ok, page} ->
    {:ok, page}
  {:error, reason} ->
    if CDPEx.transient?(reason), do: retry(), else: {:error, reason}
end

classify_error/1 buckets a reason as :transient (a fresh attempt may succeed), :terminal (it won't), or :unknown (payload-dependent — you decide). It tracks the error surface as the library evolves, so the transient/terminal decision stays in one place rather than being reimplemented (and re-drifting) downstream. Retries stay yours to bound: cap attempts, back off, and on :transient re-establish the resource (a fresh page/browser) rather than reusing a dead handle.

Status

Pages default to one WebSocket each (strong crash isolation); opt into sessionId multiplexing (many pages over the one browser socket) with new_page(browser, transport: :session), trading isolation for fewer sockets. Stealth / anti-fingerprinting presets remain out of scope.

Summary

Types

The reason shapes that appear in {:error, reason} across CDPEx.

Functions

Classifies an error reason as :transient, :terminal, or :unknown.

Closes a page opened with new_page/2.

Connects to an already-running Chrome instead of launching one.

Launches a headless Chrome browser and returns {:ok, pid}, or {:error, reason} (e.g. {:chrome_not_found, path}, {:launch_failed, _}, {:debug_url_not_found, _}).

Opens a new page. See CDPEx.Browser.new_page/2 for options.

Stops a browser started with launch/1 (kills Chrome) or connect/2 (closes the pages it opened, leaves Chrome running). :ok if the browser is already gone.

Convenience over classify_error/1: true only when the error is :transient.

Runs fun with a fresh page, guaranteeing the page (and, when given launch options, the browser) is cleaned up afterwards — even if fun raises.

Types

error_classification()

@type error_classification() :: :transient | :terminal | :unknown

The result of classify_error/1.

Intentionally open: match :transient (or :terminal) explicitly and fall through with a catch-all rather than enumerating all three atoms, so a future bucket can be added without breaking exhaustive matches.

error_reason()

@type error_reason() ::
  CDPEx.Connection.call_error()
  | CDPEx.Chrome.launch_error()
  | {:ws_connect, term()}
  | {:ws_upgrade, term()}
  | :timeout
  | :deadline_exceeded
  | :unknown_page
  | :already_authenticated
  | :already_intercepting
  | {:timeout,
     :await_event | :new_page | :close_page | :authenticate | :subscribe}
  | {:conflict, :authenticated | :intercepting}
  | {:navigate, String.t()}
  | {:no_document_response, String.t()}
  | {:connect_discovery_failed, term()}
  | {:capture_failed, term()}
  | {:idle_wait_failed, term()}
  | {:selector_not_found, String.t()}
  | {:text_not_found, String.t()}
  | {:not_clickable, String.t()}
  | {:unknown_key, String.t()}
  | {:evaluate_exception, term()}
  | {:unserializable_value, String.t()}
  | {:unexpected_evaluate, term()}
  | {:invalid_args, term()}
  | {:invalid_source, term()}
  | {:invalid_error_reason, term()}
  | {:invalid_transport, term()}
  | {:invalid_proxy, term()}
  | {:unsupported_transport, term()}
  | {:unsupported_with_connect, term()}
  | {:invalid_response_body, String.t()}
  | {:invalid_pdf_data, String.t()}
  | {:invalid_screenshot_data, String.t()}
  | {:write_failed, term()}

The reason shapes that appear in {:error, reason} across CDPEx.

Error reasons are part of the public contract — pattern-match the tagged kinds ({:cdp_error, …}, {:timeout, …}, {:ws_closed, …}, …); their payloads (a CDP method, an exit status, a stderr/contents excerpt) are open and may gain detail.

The only bare, context-free reasons are :noproc, the high-level :timeout, :deadline_exceeded (with_page/3's :deadline), :unknown_page, :already_authenticated, and :already_intercepting — self-describing control-flow outcomes with no payload to carry, the way GenServer uses :noproc. Validation failures that do have offending data to surface are tagged instead ({:invalid_response_body, excerpt}, {:invalid_pdf_data, excerpt}, {:invalid_screenshot_data, excerpt}).

To act on a failure without hard-coding this list, use classify_error/1 — it buckets any reason as :transient / :terminal / :unknown and tracks this union, so retry logic isn't reimplemented (and re-drifted) downstream.

Two sub-unions are machine-checked: CDPEx.Connection.call_error/0 and CDPEx.Chrome.launch_error/0 are precisely specced on call/5 / launch/1, so Dialyzer catches a shape change in those at the source. The remaining members — the page-level tagged kinds and bare atoms — are hand-maintained (kinds such as {:cdp_error, method, payload} also wrap arbitrary CDP data), kept honest by a compile-time coverage test that fails if any member here lacks a classify_error/1 test exemplar — so a member can't be added to this type without being classified. That guard is one-directional (type → classified): the reverse — an error a producer returns but never adds here — still relies on review, and such a stray reason would fall through classify_error/1 to :unknown.

Two timeout shapes, by layer: the low-level CDPEx.Connection.call/5, await_event/4 and subscribe/4 and the CDPEx.Browser requests return {:timeout, context} (a CDP method, :await_event, :subscribe, :new_page, :close_page, or :authenticate), while the high-level CDPEx.Page wait_for_* functions and CDPEx.Pool.checkout/2 return a bare :timeout ("the awaited condition didn't happen in time").

A WebSocket frame that fails to decode is not a standalone reason: the connection stops on the decode failure, so callers observe it nested, as {:ws_closed, {:ws_decode, _}}.

Functions

classify_error(arg1)

@spec classify_error(term()) :: error_classification()

Classifies an error reason as :transient, :terminal, or :unknown.

reason is the value from any {:error, reason} this library returns (see error_reason/0). The classification answers one question — might a fresh attempt succeed? — so you drive retries from one place instead of reimplementing the decision (and re-drifting it) in every caller:

case CDPEx.Page.navigate(page, url) do
  {:ok, page} ->
    {:ok, page}
  {:error, reason} ->
    case CDPEx.classify_error(reason) do
      :transient -> retry_with_fresh_page()
      _ -> {:error, reason}
    end
end

The buckets:

  • :transient — environmental or timing failures: the connection dropped or couldn't be established ({:ws_closed, _}, {:ws_connect, _}, {:ws_upgrade, _}, :noproc), a :session page's target closed or detached ({:session_closed, _}, the session-transport counterpart of a dedicated page's :noproc), a wait or call timed out (:timeout, {:timeout, _}, :deadline_exceeded), Chrome died or was slow to start ({:chrome_exited, _, _}, {:debug_url_not_found, _}, {:devtools_file_malformed, _}), an internal capture/idle helper crashed ({:capture_failed, _}, {:idle_wait_failed, _}), or a navigation hit a connection/network-layer net::ERR_* (e.g. {:navigate, "net::ERR_CONNECTION_REFUSED"}).
  • :terminal — deterministic outcomes: a selector or text didn't match, JS threw, a usage/validation error, or a missing Chrome binary. Retrying the same call yields the same error. (:already_authenticated / :already_intercepting are terminal for the ordinary double-call; the narrow post-timeout teardown race authenticate/4 documents — where a retry can still succeed — is signalled by the preceding {:timeout, _}, which is itself :transient.)
  • :unknown — the outcome depends on a payload or timing this function does not crack: an ambiguous navigation net::ERR_* (DNS ERR_NAME_NOT_RESOLVED, ERR_ABORTED, ERR_BLOCKED_BY_* — unlike the connection-layer codes above), the CDP error code ({:cdp_error, _, _}), the file-write posix reason ({:write_failed, _}), why Chrome couldn't be started at all ({:launch_failed, _} — an unwritable profile dir or a refused spawn: :emfile may clear, :eacces won't), whether a {:no_document_response, _} was a same-document hop or a slow miss, or a connect/2 endpoint-discovery failure ({:connect_discovery_failed, _} — a transient network blip and a permanently bad endpoint are indistinguishable here). Also covers any term CDPEx doesn't produce. Decide the retry policy yourself.

Retries are the caller's responsibility: bound the attempts and back off. A :transient result means re-establish the resource — open a fresh page/browser or call CDPEx.Pool.checkout/2 again — not retry the same handle (a dead page keeps returning :noproc, or {:session_closed, _} for a :session page). Some :transient reasons are still a judgment call for your environment — retrying a :timeout / net::ERR_TIMED_OUT multiplies wall-time and browser memory, so a resource-constrained caller may reasonably treat timeouts as terminal. The input is typed term() so the catch-all stays reachable; routing through this instead of matching error_reason/0 directly trades Dialyzer exhaustiveness for a stable, library-maintained dispatch point.

close_page(browser, page)

@spec close_page(pid(), CDPEx.Page.t()) ::
  :ok | {:error, :unknown_page | :noproc | {:timeout, :close_page}}

Closes a page opened with new_page/2.

Returns {:error, :unknown_page} if page was not opened on browser, and {:error, :noproc} if browser is gone (see CDPEx.Browser.close_page/2).

connect(endpoint, opts \\ [])

@spec connect(String.t(), keyword()) :: GenServer.on_start()

Connects to an already-running Chrome instead of launching one.

endpoint is either a ws:///wss:// browser WebSocket URL (used directly) or an http:///https:// base URL (the browser socket is discovered via GET /json/version, with the host/port taken from endpoint). Returns the same handle as launch/1, so new_page/2, with_page/3, close_page/2, and stop/1 all work — but stop/1 only closes the pages cdp_ex opened and drops the socket; it never kills the remote Chrome or touches pre-existing tabs.

http(s) discovery is IP/localhost only

Chrome's DevTools HTTP endpoint rejects /json/version with 403 when the Host header is a non-IP, non-localhost name (DNS-rebinding protection), so the http(s):// discovery form works only for IP / localhost endpoints (a 403 surfaces as {:error, {:connect_discovery_failed, {:http_status, 403}}}). For a named remote/sidecar/cloud host, pass the ws(s):// browser URL directly (or launch that Chrome with a permissive --remote-allow-origins/host).

Pages default to :session transport; transport: :dedicated returns {:error, {:unsupported_transport, :dedicated}} (not yet supported over a connected browser).

Options: :insecure (skip wss:// cert verification, default false), :cacertfile / :cacerts (custom CA for wss://), :discovery_timeout (ceiling in ms for the whole http(s):// /json/version exchange — TCP connect, recv, and body — default 5_000; raise it for a slow remote endpoint), :name (register the process).

launch(opts \\ [])

@spec launch(keyword()) :: GenServer.on_start()

Launches a headless Chrome browser and returns {:ok, pid}, or {:error, reason} (e.g. {:chrome_not_found, path}, {:launch_failed, _}, {:debug_url_not_found, _}).

The browser is linked to the caller, but a failed launch returns its error without exiting the caller, whether or not it traps exits.

Accepts the launch options documented in CDPEx.Chrome (e.g. :headless, :chrome_binary, :extra_args, :window_size, :launch_timeout). On slow cold-start hosts (e.g. headless Chrome in a constrained container) raise :launch_timeout — it is a ceiling, not a fixed wait. For long-lived use, prefer putting CDPEx.Browser under your own supervisor with a :shutdown timeout.

Proxy

Pass :proxy to route the browser through a proxy — a URL or a keyword list:

CDPEx.launch(proxy: "http://user:pass@host:8080")
CDPEx.launch(proxy: [server: "host:8080", username: "u", password: "p"])

It sets Chrome's --proxy-server and, when credentials are given, automatically answers the proxy auth challenge on each page — so you just new_page/2 and CDPEx.Page.navigate/3, no manual CDPEx.Page.authenticate/4. See CDPEx.Proxy for the accepted forms (the keyword form avoids percent-encoding special-character passwords).

A credentialed proxy requires the default :dedicated transport: new_page(transport: :session) on such a browser returns {:error, {:unsupported_transport, :session}}, and an auto-armed page can't also use enable_request_interception/2 (both drive the Fetch domain). A malformed :proxy — or combining it with a full :args override — fails the launch with {:error, {:invalid_proxy, _}} (:proxy appends to :extra_args, which an :args override discards, so the two are mutually exclusive; use one). Don't set --proxy-server in :extra_args yourself when using :proxy.

new_page(browser, opts \\ [])

@spec new_page(pid(), keyword()) :: {:ok, CDPEx.Page.t()} | {:error, term()}

Opens a new page. See CDPEx.Browser.new_page/2 for options.

stop(browser)

@spec stop(pid()) :: :ok

Stops a browser started with launch/1 (kills Chrome) or connect/2 (closes the pages it opened, leaves Chrome running). :ok if the browser is already gone.

transient?(reason)

@spec transient?(term()) :: boolean()

Convenience over classify_error/1: true only when the error is :transient.

Conservative by design — :unknown is not transient, so an unrecognized or payload-dependent error won't be auto-retried. Match classify_error/1 directly when you want to treat :unknown specially, and see its note on bounded, resource-re-establishing retries — this classifies, it does not retry.

with_page(browser_or_opts, fun, opts \\ [])

@spec with_page(pid() | keyword(), (CDPEx.Page.t() -> result), keyword()) ::
  result | {:error, term()}
when result: var

Runs fun with a fresh page, guaranteeing the page (and, when given launch options, the browser) is cleaned up afterwards — even if fun raises.

Pass an existing browser pid to reuse it, or a keyword list of launch options to spin up a throwaway browser for the duration of the call. A [connect: endpoint] list attaches to an already-running Chrome instead of launching one (see connect/2); teardown then closes only the pages it opened and never reaps that Chrome. Returns whatever fun returns, or {:error, reason} if the page/browser could not be created.

Failures in with_page's own work (the launch or connect, opening the page, a browser that is gone or dies under it, the teardown) come back as {:error, reason}, never as a raise or an exit. A raise, throw or exit from fun itself propagates unchanged: that is the caller's code.

With launch options, the throwaway browser is linked but contained: if it fails to start, or crashes during the call (e.g. its connection drops), with_page returns {:error, reason} instead of letting the exit propagate to the caller. To do that it briefly traps exits in the calling process for the duration of the call, launch included. Only the browser's own {:EXIT, _, _} is drained — a foreign process linked to the caller that exits during this window has its exit delivered as a message left in the caller's mailbox, so a caller that links other processes and relies on un-trapped exit propagation should pass a pre-launched browser pid instead. On slow cold-start hosts, raise :launch_timeout (a ceiling, not a fixed wait).

opts go to new_page/2, except :deadline.

Deadline

Each page operation has its own timeout, but they stack: a navigation, a few selector waits and some retries can hold a browser for minutes. Pass deadline: ms (a positive integer) to bound the whole call. The clock starts when with_page is called, so a launch counts against it.

The rule, by the monotonic clock: the call is exceeded iff fun had not returned by the deadline (or never started). Then with_page returns {:error, :deadline_exceeded} however fun came back — a value, an error tuple, or a raise/exit caused by the teardown. A new_page/2 error comes back as itself, late or not (the teardown waits for new_page/2 to return); a page that only arrives after the deadline means fun is not started. If fun returned in time, its result or raise comes back untouched: the deadline stops there, and closing the page or stopping a throwaway browser afterwards runs on their own timeouts. Without a deadline nothing changes.

fun still runs in the calling process (self(), the process dictionary, Logger metadata and links behave as without a deadline). A watchdog process enforces the deadline by tearing down the browser resource underneath it: a throwaway browser (launch options or connect:) is stopped, reaping a launched Chrome; with a browser pid, only the page with_page opened is closed and the browser stays yours (a page fun opens itself is not released). The browser call fun is blocked in then fails fast and fun unwinds. Work in fun that doesn't touch the browser — a sleep, a computation — runs on until its next browser call, so the bound is "the browser is released at the deadline", not "fun is preempted".

Some stages can't be interrupted and are bounded differently:

  • Launch — the browser doesn't exist yet, so :launch_timeout is capped at the deadline when the deadline is smaller or equal (default 15_000), and so is connect-mode's :discovery_timeout (default 5_000, and a nil or :infinity is capped too). A non-integer :launch_timeout is not capped: it raises ArgumentError like any invalid launch option. The WebSocket connect and upgrade after them (normally milliseconds) keep their own timeouts. A start whose capped timeout runs out at or after the deadline returns {:error, :deadline_exceeded}, with its own error in the telemetry event's :launch_error; any other start failure comes back as itself.
  • new_page/2 — the browser serves it as one call (target creation, bootstrap, a proxy's auth arm), so a deadline that fires mid-way takes effect when it returns; fun is then not started. A start that returns after the deadline skips new_page/2 altogether.
  • Closing the page on a browser pid goes through the Browser, so on a shared or pooled browser it queues behind whatever the Browser is serving (another caller's new_page/2). If close_page/2's own 15s timeout runs out first, the page stays open and the call in fun runs to its own timeout.

With a throwaway browser the teardown's {:EXIT, browser, :normal} arrives while fun may still be running (the call traps exits, as for a browser crash); it is drained before with_page returns, but a catch-all receive in fun can take it.

If the caller dies mid-call, a throwaway browser is reaped through its link as before, and on a browser pid the watchdog closes the page — once new_page/2 has returned it; a caller that dies while new_page/2 is in flight leaves that page open, as without a deadline.

An invalid :deadline (not an integer in 1..4_294_967_295) raises ArgumentError. An exceeded deadline emits [:cdp_ex, :with_page, :deadline_exceeded] (see CDPEx.Telemetry).

# against an existing browser
CDPEx.with_page(browser, fn page ->
  {:ok, _} = CDPEx.Page.navigate(page, "https://example.com")
  CDPEx.Page.html(page)
end)

# throwaway browser + page
CDPEx.with_page([headless: true], &CDPEx.Page.html/1)

# against an existing Chrome (discovered via /json/version)
CDPEx.with_page([connect: "http://localhost:9222"], &CDPEx.Page.html/1)

# the whole call, launch included, within 20s
CDPEx.with_page([], &CDPEx.Page.html/1, deadline: 20_000)