HTTP (http_fetch v0.21.19)

Copy Markdown View Source

A browser-like HTTP fetch API for Elixir, built on Erlang's :gen_tcp, :ssl, and the shared QUIC HTTP/3 transport.

This module provides a modern, Promise-based HTTP client interface similar to the browser's fetch() API. It supports asynchronous requests, streaming, request cancellation, and comprehensive telemetry integration.

Features

  • Async by default: All requests use Task.Supervisor with async_nolink/4
  • Automatic streaming: Responses >5MB or with unknown Content-Length automatically stream
  • Request streaming: Pass an HTTP.Stream PID as body with duplex: "half" for HTTP/1.1 or HTTP/2 uploads Supply a Content-Length header for a known-length stream; HTTP/1.1 sends raw body bytes, and both protocols reject a stream whose size differs from the header. HTTP/1.1 streams without a declared length use chunked transfer encoding.
  • Request cancellation: Via HTTP.AbortController for aborting in-flight requests
  • Promise chaining: JavaScript-like promise interface with then/3 support
  • Unix Domain Sockets: Support for HTTP over Unix sockets (Docker daemon, systemd, etc.)
  • Telemetry integration: Comprehensive event emission for monitoring and observability
  • HTTP/3 opt-in: Pass http_version: :http3 for QUIC-backed HTTP/3 requests

Quick Start

# Simple GET request
response =
  HTTP.fetch("https://jsonplaceholder.typicode.com/posts/1")
  |> HTTP.Promise.await()

# Parse JSON response
{:ok, json} = HTTP.Response.json(response)

# POST with JSON body
response =
  HTTP.fetch("https://api.example.com/posts", [
    method: "POST",
    headers: %{"Content-Type" => "application/json"},
    body: JSON.encode!(%{title: "Hello", body: "World"})
  ])
  |> HTTP.Promise.await()

# Unix Domain Socket request (e.g., Docker daemon)
response =
  HTTP.fetch("http://localhost/version",
    unix_socket: "/var/run/docker.sock")
  |> HTTP.Promise.await()

Architecture

The library is structured around these core modules:

Streaming Behavior

Responses are automatically streamed when:

  • Content-Length > 5MB
  • Content-Length header is missing/unknown

Streaming responses expose the stream PID in body; stream is retained as a compatibility alias. Use HTTP.Response.read_all/1 or HTTP.Response.write_to/2 to consume streams.

Telemetry Events

All events use the [:http_fetch, ...] prefix:

  • [:http_fetch, :request, :start] - Request initiated
  • [:http_fetch, :request, :stop] - Request completed
  • [:http_fetch, :request, :exception] - Request failed
  • [:http_fetch, :streaming, :start] - Streaming started
  • [:http_fetch, :streaming, :chunk] - Stream chunk received
  • [:http_fetch, :streaming, :stop] - Streaming completed

See HTTP.Telemetry for detailed event documentation.

Summary

Functions

Performs an HTTP request, similar to global.fetch in web browsers. Uses an internal HTTP/1.1 socket transport asynchronously by default.

Functions

fetch(url, init \\ [])

@spec fetch(String.t() | URI.t(), Keyword.t() | map()) :: HTTP.Promise.t()

Performs an HTTP request, similar to global.fetch in web browsers. Uses an internal HTTP/1.1 socket transport asynchronously by default.

Arguments:

  • url: The URL to fetch (string or URI struct).
  • init: An optional keyword list or map of options for the request.
        Supported options:
          - `:method`: The HTTP method (e.g., "GET", "POST"). Defaults to "GET".
                       Can be a string or an atom (e.g., "GET" or :get).
          - `:proxy`: Explicit HTTP proxy `{:http, host, port, opts}`. Options are
                      proxy-only authorization headers and a bounded timeout.
                      HTTP/1 forwards cleartext requests; HTTPS uses CONNECT
                      before origin TLS and HTTP/1 or HTTP/2 negotiation.
          - `:headers`: A list of request headers as `{name, value}` tuples (e.g., [{"Content-Type", "application/json"}])
                        or a map that will be converted to the tuple format.
          - `:body`: The request body. Buffered bodies may be binary or iodata; streaming
                     bodies may be an `HTTP.Stream` PID when `duplex: "half"` is set.
          - `:request_mode`: `:fetch` (default) or `:proxy`. Proxy mode preserves
                             request bodies for all admitted methods, including GET,
                             DELETE, and HEAD. HEAD response bodies remain empty.
          - `:duplex`: Set to `:half` or `"half"` to enable Fetch-style request body streaming.
                       HTTP/1.1 uses chunked framing; HTTP/2 uses flow-controlled DATA frames.
          - `:content_type`: The Content-Type header value. If not provided for methods with body,
                             defaults to "application/octet-stream" when a body is present.
          - `:redirect`: Redirect mode, one of `:follow`, `:manual`, or `:error`. Defaults to `:follow`.
          - `:signal`: An `HTTP.AbortController` PID. If provided, the request can be aborted
                       via this controller.
          - `:http_version`: `:http1`, `:http2`, `:http3`, `:h2c`, or `:auto`.
          - `:tls_backend`: `:ssl` (default) or `:ex_ssl` for HTTPS requests. It may also be
            configured globally with `config :http_core, :tls_backend, :ex_ssl`; an explicit
            request value takes precedence. HTTP/3 does not support this option.
          - `:timeout`, `:connect_timeout`, `:ssl`, and `:socket_opts`: Elixir-specific transport
            extensions used by the socket or QUIC transport.
          - `:connect_address`: Caller-validated IPv4/IPv6 tuple, preserving the URL authority/TLS hostname. Requires manual/error redirects.
          - `:unix_socket`: Path to a Unix Domain Socket file (e.g., "/var/run/docker.sock").
                            When provided, the request is sent over the Unix socket instead of TCP/IP.

Returns:

  • %HTTP.Promise{}: A Promise struct. The caller should HTTP.Promise.await(promise_struct) to get the final
           `%HTTP.Response{}` or `{:error, reason}`. If the request cannot be initiated
           (e.g., invalid URL, bad arguments), the Promise will contain an error result
           when awaited.

The socket transport defaults redirect to :follow; pass redirect: :manual to return redirect responses, or redirect: :error to fail on redirect. Request timeout handling gives the internal socket owner up to one additional second to report its timeout before the awaiting caller aborts it. For HTTP/1, timeout: :infinity removes the total request deadline, including response body reads. connect_timeout independently bounds connection establishment (including TLS); it accepts :infinity and defaults to at most 30 seconds. The total request deadline still caps a longer connection timeout. HTTP.Promise.await/2 has its own independent wait timeout. Cancellation and caller/reader cleanup remain active for infinite requests. Stream idle and chunk acknowledgement timeouts, explicit socket send timeouts, and proxy timeouts remain separate limits. ExSSL streaming uploads require a finite per-write send timeout; when absent, an infinite request uses HTTP.Config.default_request_timeout/0 for that limit. Managed transport scopes require finite, positive request and connection timeouts.

read_idle_timeout and write_idle_timeout are independent positive millisecond limits for HTTP/1 requests (TCP, OTP TLS and ExSSL). They default to nil; automatic protocol selection, HTTP/2, HTTP/3 and managed transport scopes are unsupported. Read idle measures armed response reads and resets on nonempty transport data, pausing during consumer acknowledgement backpressure. Write idle bounds each transport write of at most 16 KiB and resets before the next write; successful transport acceptance does not establish peer receipt. Waiting for the upload producer does not consume the write budget. Expiry reports :read_idle_timeout or :write_idle_timeout, with existing structured-error wrapping before response delivery and stream errors afterwards. Overall and connection deadlines, socket send limits and cancellation remain independent.

response_stream_idle_timeout sets a per-request response-body inactivity policy for ordinary HTTP/1 (TCP, OTP TLS and ExSSL). It accepts positive milliseconds up to 4_294_967_295 or :infinity; nil keeps the existing global stream timeout. Set it to :infinity to leave network stalls to read_idle_timeout or to allow uncapped response reads. Response convenience readers follow this policy. Consumer chunk acknowledgement budgets, total deadlines, cancellation and owned cleanup remain independent. Automatic/H2/H3 protocols and managed transport scopes reject an explicit policy.

response_stream_ack_timeout sets a separate ordinary HTTP/1 response consumer delivery/acknowledgement deadline in positive milliseconds up to 4_294_967_295 or :infinity; omitted/nil retains the global streaming-operation limit. Both the Stream and socket producer use this policy. Network inactivity, total request deadlines, cancellation and reader cleanup stay independent. Automatic/H2/H3 protocols and managed scopes reject explicit values; upload sources retain their existing policy.

For an ordinary HTTP/1 fixed-length stream upload, a response after the final declared bytes waits for source completion, including enumerable resource finalization, under a finite completion budget capped by the total deadline. Source/finalizer exceptions fail the request even if the peer sent success. Cancellation or expiry may forcibly stop the source; process termination alone does not certify callback completion.

Example Usage:

# GET request and awaiting JSON
promise_json = HTTP.fetch("https://jsonplaceholder.typicode.com/todos/1")
case HTTP.Promise.await(promise_json) do
  %HTTP.Response{} = response ->
    case HTTP.Response.json(response) do
      {:ok, json_body} ->
        IO.puts "GET JSON successful! Title: #{json_body["title"]}"
      {:error, reason} ->
        IO.inspect reason, label: "JSON Parse Error"
    end
  {:error, reason} ->
    IO.inspect reason, label: "GET Error for JSON"
end

# GET request and awaiting text
promise_text = HTTP.fetch("https://jsonplaceholder.typicode.com/posts/1")
case HTTP.Promise.await(promise_text) do
  %HTTP.Response{} = response ->
    text_body = HTTP.Response.text(response)
    IO.puts "GET Text successful! First 50 chars: #{String.slice(text_body, 0, 50)}..."
  {:error, reason} ->
    IO.inspect reason, label: "GET Error for Text"
end

# Simple Promise chaining: fetch -> parse JSON -> print title
HTTP.fetch("https://jsonplaceholder.typicode.com/todos/2")
|> HTTP.Promise.then(fn %HTTP.Response{} = response ->
  HTTP.Response.json(response)
end)
|> HTTP.Promise.then(fn {:ok, json_body} ->
  IO.puts "Chained JSON successful! Title: #{json_body["title"]}"
end)
|> HTTP.Promise.await() # Await the final chained promise to ensure execution

# Promise chaining with error handling: fetch -> parse JSON -> handle success or error
HTTP.fetch("https://jsonplaceholder.typicode.com/nonexistent") # This URL will cause an error
|> HTTP.Promise.then(
  fn %HTTP.Response{} = response ->
    IO.puts "This success branch should not be called for a 404!"
    HTTP.Response.json(response)
  end,
  fn reason ->
    IO.inspect reason, label: "Chained Error Handler Caught"
    {:error, :handled_error} # Return an error tuple to propagate rejection
  end
)
|> HTTP.Promise.await() # Await the final chained promise

# Chaining where a callback returns another promise
HTTP.fetch("https://jsonplaceholder.typicode.com/posts/1")
|> HTTP.Promise.then(fn %HTTP.Response{} = response ->
  # Simulate fetching comments for the post
  post_id = case HTTP.Response.json(response) do
    {:ok, %{"id" => id}} -> id
    _ -> nil
  end
  if post_id do
    IO.puts "Fetched post #{post_id}. Now fetching comments..."
    HTTP.fetch("https://jsonplaceholder.typicode.com/posts/#{post_id}/comments")
  else
    # If post_id is nil, we want to reject this branch of the promise chain
    throw {:error, :post_id_not_found}
  end
end)
|> HTTP.Promise.then(fn comments_response ->
  case HTTP.Response.json(comments_response) do
    {:ok, comments} ->
      IO.puts "Successfully fetched #{length(comments)} comments for the post."
    {:error, reason} ->
      IO.inspect reason, label: "Failed to parse comments JSON"
  end
end)
|> HTTP.Promise.await()


# POST request with JSON body and headers
# If you're using Elixir 1.18+, JSON.encode! is built-in. Otherwise, you'd need a library like Poison.
# promise_post = HTTP.fetch("https://jsonplaceholder.typicode.com/posts",
#        method: "POST",
#        headers: [{"Accept", "application/json"}],
#        content_type: "application/json",
#        body: JSON.encode!(%{title: "foo", body: "bar", userId: 1})
#      )
# case HTTP.Promise.await(promise_post) do
#   %HTTP.Response{status: 201, body: body} ->
#     IO.puts "POST successful! Body: #{body}"
#   {:error, reason} ->
#     IO.inspect reason, label: "POST Error"
# end

# Request with a longer timeout
delayed_promise = HTTP.fetch("https://httpbin.org/delay/5", timeout: 10_000)
case HTTP.Promise.await(delayed_promise) do
  %HTTP.Response{status: status} ->
    IO.puts "Delayed request successful! Status: #{status}"
  {:error, reason} ->
    IO.inspect reason, label: "Delayed Request Result"
end

# Abortable request example
controller = HTTP.AbortController.new() # Create a new controller
IO.puts "Fetching a long request that will be aborted..."
abortable_promise = HTTP.fetch("https://httpbin.org/delay/10", signal: controller, timeout: 20_000)

# Simulate some work, then abort after a short delay
Task.start_link(fn ->
  :timer.sleep(2000) # Wait 2 seconds
  IO.puts "Attempting to abort the request after 2 seconds..."
  HTTP.AbortController.abort(controller)
end)

# Await the result of the abortable promise
case HTTP.Promise.await(abortable_promise) do
  %HTTP.Response{status: status} ->
    IO.puts "Abortable request completed successfully! Status: #{status}"
  {:error, reason} ->
    IO.inspect reason, label: "Abortable Request Result"
    if reason == :aborted or reason == :request_timeout do
      IO.puts "Request was likely aborted. Reason: #{inspect(reason)}"
    end
end

# Unix Domain Socket request to Docker daemon
docker_promise = HTTP.fetch("http://localhost/version", unix_socket: "/var/run/docker.sock")
case HTTP.Promise.await(docker_promise) do
  %HTTP.Response{status: 200} = response ->
    case HTTP.Response.json(response) do
      {:ok, json} ->
        IO.puts "Docker Version: #{json["Version"]}"
        IO.puts "API Version: #{json["ApiVersion"]}"
      {:error, reason} ->
        IO.inspect reason, label: "JSON Parse Error"
    end
  {:error, reason} ->
    IO.inspect reason, label: "Docker Request Error"
end