<p align="center">
  <img src="https://raw.githubusercontent.com/cignosystems/pushx/main/pushx_logo.png" alt="PushX Logo" width="400">
</p>

<p align="center">
  <strong>Push notifications for Elixir that just work: APNS, FCM and Web Push in one call.</strong><br>
  Retries, circuit breaker, dead-token cleanup and telemetry built in. Per-tenant credentials at runtime.<br>
  Nothing to add to your supervision tree.<br>
  <sub>Web Push per <a href="https://www.rfc-editor.org/rfc/rfc8030">RFC 8030</a> (transport), <a href="https://www.rfc-editor.org/rfc/rfc8291">RFC 8291</a> (encryption) and <a href="https://www.rfc-editor.org/rfc/rfc8292">RFC 8292</a> (VAPID) — every browser, no Firebase required.</sub>
</p>

<p align="center">
  <a href="https://github.com/cignosystems/pushx/actions/workflows/ci.yml"><img src="https://github.com/cignosystems/pushx/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://hex.pm/packages/pushx"><img src="https://img.shields.io/hexpm/v/pushx.svg" alt="Hex.pm"></a>
  <a href="https://hexdocs.pm/pushx"><img src="https://img.shields.io/badge/hex-docs-purple.svg" alt="Hex Docs"></a>
  <a href="https://github.com/cignosystems/pushx/blob/main/LICENSE"><img src="https://img.shields.io/hexpm/l/pushx.svg" alt="License"></a>
</p>

> **AI coding assistants:** see [`AGENTS.md`](AGENTS.md) for the mental model,
> idiomatic patterns, and a list of mistakes commonly made when integrating
> PushX. Also rendered on [hexdocs](https://hexdocs.pm/pushx/agents.html).

---

## Table of Contents

- [Features](#features)
- [Quick Start](#quick-start)
- [Usage Guide](#usage-guide)
- [Configuration](#configuration)
- [Dynamic Instances (Runtime Config)](#dynamic-instances-runtime-config)
- [Credential Storage](#credential-storage)
- [Getting Your Credentials](#getting-your-credentials)
- [Telemetry](#telemetry)
- [Circuit Breaker](#circuit-breaker)
- [Health Check](#health-check)
- [Token Cleanup Callback](#token-cleanup-callback)
- [Delivery Semantics](#delivery-semantics)
- [Testing Your App](#testing-your-app)
- [Troubleshooting](#troubleshooting)
- [Contributing](#contributing)
- [License](#license)

---

## Features

- **HTTP/2** connections via Finch (Mint-based) for optimal performance
- **APNS** (Apple Push Notification Service) with JWT authentication
- **FCM** (Firebase Cloud Messaging) with OAuth2 via Goth
- **Web Push** — standards-based for every browser incl. Safari 16+: [RFC 8030](https://www.rfc-editor.org/rfc/rfc8030) transport, [RFC 8291](https://www.rfc-editor.org/rfc/rfc8291) `aes128gcm` encryption (reproduces the RFC's test vectors), [RFC 8292](https://www.rfc-editor.org/rfc/rfc8292) VAPID; FCM webpush and legacy Safari APNS also supported
- **Batch sending** — send to multiple tokens concurrently with configurable parallelism, as a list or a lazy stream
- **FCM topics & conditions** — fan out to `{:topic, "news"}` or `{:condition, "'a' in topics && 'b' in topics"}` with the same call
- **Multi-tenant** — named runtime instances with their own credentials, pools and breakers
- **Token validation** — validate token format before sending to catch errors early
- **Rate limiting** — optional client-side rate limiting to avoid provider throttling
- **Automatic retry** — exponential backoff for rate limits and server errors (opt out per call with `retry: :none`)
- **Circuit breaker & dead-token cleanup** — stop hammering a failing provider; get told which tokens to delete
- **Telemetry** — built-in instrumentation for metrics and monitoring
- **Message builder** — fluent API for constructing notifications
- **Structured responses** — consistent error handling across providers
- **Zero JSON dependency** — uses Elixir 1.18+ built-in JSON module

### Requirements

- Elixir 1.18+ (for the built-in `JSON` module)
- OTP 26+

CI runs the suite on Elixir 1.20/OTP 29, 1.20/28, 1.19/28, 1.19/27 and 1.18/26,
so anything in that range is exercised on every push; the release itself is
built with the latest stable pair.

Tested on Elixir 1.18/1.19 with OTP 26, 27, and 28.

---

## Quick Start

### 1. Install

Add `pushx` to your dependencies in `mix.exs`:

```elixir
def deps do
  [
    {:pushx, "~> 0.15"}
  ]
end
```

### 2. Configure

Add credentials to `config/runtime.exs`:

```elixir
config :pushx,
  # APNS (iOS)
  apns_key_id: System.fetch_env!("APNS_KEY_ID"),
  apns_team_id: System.fetch_env!("APNS_TEAM_ID"),
  apns_private_key: System.fetch_env!("APNS_PRIVATE_KEY"),
  apns_mode: :prod,

  # FCM (Android)
  fcm_project_id: System.fetch_env!("FCM_PROJECT_ID"),
  fcm_credentials: System.fetch_env!("FCM_CREDENTIALS") |> JSON.decode!()
```

PushX starts its own HTTP/2 connection pools and OAuth processes automatically — no additional supervision tree setup needed.

> Need help getting credentials? See [Getting Your Credentials](#getting-your-credentials) below.

### 3. Send a notification

```elixir
# Send to iOS
PushX.push(:apns, device_token, "Hello!", topic: "com.example.app")

# Send to Android
PushX.push(:fcm, device_token, "Hello!")

# With title and body
PushX.push(:apns, token, %{title: "Welcome", body: "Thanks for signing up!"},
  topic: "com.example.app")
```

That's it. PushX handles HTTP/2 connections, JWT/OAuth authentication, and automatic retry.

---

## Usage Guide

### Message Builder

Build rich notifications with the fluent API:

```elixir
message = PushX.message()
  |> PushX.Message.title("Order Update")
  |> PushX.Message.body("Your order #1234 has shipped")
  |> PushX.Message.badge(1)
  |> PushX.Message.sound("default")
  |> PushX.Message.data(%{order_id: "1234", status: "shipped"})

PushX.push(:apns, token, message, topic: "com.example.app")
```

| Function | Description |
|----------|-------------|
| `title(msg, string)` | Set notification title |
| `body(msg, string)` | Set notification body |
| `badge(msg, integer)` | Set app badge count (iOS) |
| `sound(msg, string)` | Set notification sound |
| `data(msg, map)` | Set custom data payload |
| `put_data(msg, key, value)` | Add single data key-value |
| `category(msg, string)` | Set notification category (iOS) |
| `thread_id(msg, string)` | Set thread ID for grouping (iOS) |
| `image(msg, url)` | Set image URL (FCM) |
| `priority(msg, :high \| :normal)` | Set delivery priority |
| `ttl(msg, seconds)` | Set time-to-live |
| `collapse_key(msg, string)` | Set collapse key (FCM) |
| `subtitle(msg, string)` | Subtitle (iOS; delivered to iOS via FCM too) |
| `mutable_content(msg)` | `mutable-content: 1` — lets a Notification Service Extension modify it (rich media) |
| `content_available(msg)` | `content-available: 1` — wake the app to fetch data |
| `interruption_level(msg, level)` | `:passive`, `:active`, `:time_sensitive`, `:critical` (iOS 15+) |
| `relevance_score(msg, 0.0..1.0)` | Ordering in the iOS notification summary |
| `localized_title(msg, key, args)` | Localized title (`title-loc-key`; FCM `title_loc_key`) |
| `localized_body(msg, key, args)` | Localized body (`loc-key`; FCM `body_loc_key`) |
| `localized_subtitle(msg, key, args)` | Localized subtitle (iOS) |

Every field maps to both providers where the concept exists — the iOS-specific ones reach iOS devices addressed through FCM via an automatic `apns` override (`PushX.Message.to_fcm_apns/1`), merged under any explicit `:apns` option you pass.

You can also pass a plain string, a `%{title: ..., body: ...}` map, or a raw APNS/FCM payload map directly to `push/4`.

### Response Handling

Every push returns `{:ok, Response}` or `{:error, Response}`:

```elixir
case PushX.push(:apns, token, message, topic: "com.example.app") do
  {:ok, %PushX.Response{status: :sent, id: apns_id}} ->
    Logger.info("Notification sent with ID: #{apns_id}")

  {:error, %PushX.Response{} = response} ->
    if PushX.Response.should_remove_token?(response) do
      # Token is invalid, expired, or unregistered — delete it
      MyApp.Tokens.delete(token)
    else
      Logger.error("Push failed: #{response.status} - #{response.reason}")
    end
end
```

**Response struct:**

```elixir
%PushX.Response{
  provider: :apns | :fcm,
  status: :sent | :invalid_token | :expired_token | ...,
  id: "message-id" | nil,
  reason: "error reason" | nil,
  raw: raw_response_body,
  retry_after: seconds | nil
}
```

| Status | Description | Action |
|--------|-------------|--------|
| `:sent` | Successfully delivered | None |
| `:invalid_token` | Token is malformed or invalid | Remove token |
| `:expired_token` | Token has expired | Remove token |
| `:unregistered` | Device unregistered | Remove token |
| `:payload_too_large` | Payload exceeds limit (APNS: 4KB, FCM: 4000 bytes) | Reduce payload size |
| `:rate_limited` | Too many requests | Automatic retry with backoff |
| `:server_error` | Provider server error | Automatic retry with backoff |
| `:connection_error` | Network failure | Automatic retry with backoff |
| `:invalid_request` | Missing required option (e.g., no `:topic` for APNS) | Fix request parameters |
| `:auth_error` | JWT/credential failure (e.g., invalid private key, provider rejected the token) | Check credentials |
| `:not_configured` | Provider has no credentials configured (e.g. `push(:fcm, ...)` without FCM config) | Fix deployment config |
| `:circuit_open` | Circuit breaker is open — provider not called | Wait; check `PushX.health_check/0` |
| `:provider_disabled` | Named instance is disabled | `PushX.Instance.enable/1` |
| `:unknown_error` | Unrecognized error | Check `reason` field |

**Helper functions:**

```elixir
PushX.Response.success?(response)              # true if status == :sent
PushX.Response.should_remove_token?(response)  # true for invalid/expired/unregistered
PushX.Response.retryable?(response)            # true for connection_error/rate_limited/server_error
```

### Batch Sending

Send to multiple devices concurrently:

```elixir
results = PushX.push_batch(:apns, tokens, message, topic: "com.example.app")

# Process results
Enum.each(results, fn
  {token, {:ok, response}} -> Logger.info("Sent to #{token}")
  {token, {:error, response}} ->
    if PushX.Response.should_remove_token?(response) do
      MyApp.Tokens.delete(token)
    end
end)
```

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `:concurrency` | `integer()` | `50` | Max concurrent requests |
| `:timeout` | `integer()` | `30_000` | Timeout per request (ms) |
| `:validate_tokens` | `boolean()` | `false` | Filter invalid tokens before sending |

For aggregate counts, use the bang variant:

```elixir
%{success: 95, failure: 5, total: 100} =
  PushX.push_batch!(:fcm, tokens, "Hello!")
```

**Large audiences.** `push_batch/4` holds every result in memory. For tens of thousands of tokens and up, use the lazy variant — it accepts any enumerable (a database stream works; the input is enumerated exactly once), yields `{token, result}` pairs in input order as each one completes, and keeps memory bounded:

```elixir
Repo.transaction(fn ->
  Repo.stream(from t in Token, where: t.provider == :fcm, select: t.value)
  |> then(&PushX.push_batch_stream(:fcm, &1, "Maintenance in 10m", concurrency: 100, retry: :none))
  |> Stream.each(fn
    {token, {:error, resp}} -> if PushX.Response.should_remove_token?(resp), do: Tokens.delete(token)
    _ -> :ok
  end)
  |> Stream.run()
end)
```

Two performance notes: a retrying task holds one of the batch's concurrency slots for its whole backoff (up to a minute on 429s), so pass `retry: :none` for big batches and requeue failures yourself; and if you stick with `push_batch/4`, chunk the input (`Enum.chunk_every(tokens, 10_000)`).

### FCM Topics and Conditions

FCM can fan out server-side. Pass a topic or condition where you would pass a device token — the rest of the API is unchanged, including named instances and `push_data/4`:

```elixir
PushX.push(:fcm, {:topic, "news"}, "Breaking news")
PushX.push(:fcm, {:condition, "'news' in topics && 'sports' in topics"}, message)
PushX.push_data(:my_fcm_tenant, {:topic, "sync"}, %{action: "refresh"})
```

Topic names are the bare name (no `/topics/` prefix), characters `[a-zA-Z0-9-_.~%]`; invalid targets are rejected locally with `:invalid_request`. Topics/conditions never trigger token cleanup, and APNS returns `:invalid_request` for them (topics are an FCM feature).

Manage subscriptions from the server too (Instance ID API; auto-chunked at Google's 1 000 tokens per request; per-token results in input order):

```elixir
{:ok, results} = PushX.subscribe(:fcm, tokens, "news")        # or a named FCM instance
for {token, {:error, "NOT_FOUND"}} <- results, do: MyApp.Tokens.delete(token)
{:ok, _} = PushX.unsubscribe(:fcm, tokens, "news")
```

### Dry Runs and Tracing

- **FCM dry run** — `validate_only: true` asks FCM to validate a message (token registration, payload) without delivering it; a `{:ok, %Response{status: :sent}}` means "would have been accepted", errors are the real ones. Handy for checking stored tokens before a campaign: `PushX.push(:fcm, token, msg, validate_only: true)`.
- **APNS tracing** — pass your own UUID as `apns_id:`; Apple echoes it in `response.id` and shows it in delivery logs and the Push Notifications Console, so a push can be traced end-to-end: `PushX.push(:apns, token, msg, topic: "...", apns_id: Ecto.UUID.generate())`.

### Silent/Background Notification

```elixir
payload = PushX.APNS.silent_notification(%{action: "sync", resource: "messages"})

PushX.APNS.send(token, payload,
  topic: "com.example.app",
  push_type: "background",
  priority: 5
)
```

### Data-Only Message (FCM)

Send data without a visible notification. All values are automatically converted to strings (FCM requirement):

```elixir
# Via unified API (supports named instances)
PushX.push_data(:fcm, token, %{action: "sync", id: 123})
PushX.push_data(:my_fcm, token, %{action: "sync", id: 123})

# Via provider module directly
PushX.FCM.send_data(token, %{action: "sync", id: 123})
```

### Notification with Custom Data (FCM)

Send a visible notification with a custom data payload attached:

```elixir
# Structured payload — notification + data
PushX.push(:fcm, token, %{
  "notification" => %{"title" => "Alert", "body" => "Something happened"},
  "data" => %{"event_id" => "1", "action" => "open_event"}
})

# Works with named instances too
PushX.push(:my_fcm, token, %{
  "notification" => %{"title" => "Alert", "body" => "Something happened"},
  "data" => %{"event_id" => "1"}
})
```

### Web Push

#### Standards-based Web Push (all browsers — recommended)

Chrome, Firefox, Edge, Opera, Samsung Internet and Safari 16+ (macOS Ventura / iOS 16.4+) all speak the same protocol: the browser gives you a **subscription** (endpoint + keys), you encrypt the payload for it (RFC 8291) and authenticate with VAPID (RFC 8292). PushX does all of that; no Firebase, no Apple website-push ID.

```elixir
# 1. once: generate a VAPID key pair
#    $ mix pushx.vapid
config :pushx,
  webpush_vapid_subject: "mailto:ops@example.com",
  webpush_vapid_private_key: System.fetch_env!("WEBPUSH_VAPID_PRIVATE_KEY")
  # webpush_vapid_public_key is optional (derived); it's the front end's applicationServerKey

# 2. the browser subscribes (JS) and POSTs the subscription to you
#    const sub = await registration.pushManager.subscribe({userVisibleOnly: true, applicationServerKey: PUBLIC_KEY})

# 3. send — the subscription object is the target
PushX.push(:webpush, subscription, %{title: "Order shipped", body: "Arrives Tuesday", data: %{url: "/orders/42"}},
  ttl: 3600, urgency: :high, topic: "order-42")
```

`PushX.Message` maps to the Notification API shape (`title`, `body`, `icon`, `tag`, `data`) your service worker shows, and its `ttl`/`priority` become the `TTL`/`Urgency` headers; a map is sent as JSON exactly as given (also through `PushX.push/4` — Notification API options like `icon`/`actions` survive); a string is the title via `PushX.push/4` (`{"title": ..., "body": ""}`, as for APNS/FCM) and verbatim via `PushX.WebPush.send/3`. A `404`/`410` from the push service means the subscription is gone → `:unregistered`, and `:on_invalid_token` fires with the subscription map so you can delete it. Payloads are limited to ~4 KB; `:ttl`, `:urgency` (`:very_low | :low | :normal | :high`) and `:topic` (collapse key) are the RFC 8030 knobs. Multi-tenant: `PushX.Instance.start(name, :webpush, vapid_subject: ..., vapid_private_key: ...)`. See `PushX.WebPush`.

#### FCM Web Push (apps using the Firebase JS SDK)

If your web app already uses Firebase Messaging, its tokens go through FCM like mobile tokens:

```elixir
PushX.push(:fcm, web_token, %{title: "Hello", body: "From web!"})

PushX.FCM.send_web(web_token, "New Message", "Check it out", "https://example.com/messages",
  icon: "https://example.com/icon.png", badge: "https://example.com/badge.png")
```

#### Safari legacy website push (APNS)

Pre-Safari-16 "website push" uses APNS with a `web.` topic (the website push ID) and 64-hex tokens — still supported:

```elixir
payload = PushX.APNS.web_notification("New Article", "Check it out", "https://example.com/article/123")
PushX.APNS.send(safari_token, payload, topic: "web.com.example.website")
```

### Direct Provider Access

The unified `PushX.push/4` normalizes payloads across providers. When you need provider-specific features, use the modules directly:

```elixir
# APNS — full control over headers and payload
PushX.APNS.send(token, payload, topic: "com.app", push_type: "voip")
PushX.APNS.send_once(token, payload, opts)          # no automatic retry
PushX.APNS.send_batch(tokens, payload, opts)
PushX.APNS.notification("Title", "Body", badge)
PushX.APNS.notification_with_data("Title", "Body", %{key: "value"})
PushX.APNS.silent_notification(%{action: "sync"})
PushX.APNS.web_notification("Title", "Body", "https://url")
PushX.APNS.web_notification_with_data("Title", "Body", "https://url", %{key: "val"})

# FCM — full control over android/webpush/data options
PushX.FCM.send(token, payload, data: %{key: "value"})
PushX.FCM.send_once(token, payload, opts)            # no automatic retry
PushX.FCM.send_batch(tokens, payload, opts)
PushX.FCM.send_data(token, %{key: "value"})          # data-only, no visible notification
PushX.FCM.send_web(token, "Title", "Body", "https://link", opts)
PushX.FCM.notification("Title", "Body", image: "https://img")
PushX.FCM.web_notification("Title", "Body", "https://link", opts)
PushX.FCM.web_notification_with_data("Title", "Body", "https://link", %{key: "val"})
```

### Token Validation

Validate tokens before sending to catch format errors early:

```elixir
PushX.valid_token?(:apns, token)    # true/false
PushX.validate_token(:apns, token)  # :ok | {:error, :empty | :invalid_length | :invalid_format}

# In batch — filter out bad tokens automatically
PushX.push_batch(:apns, tokens, message, topic: "...", validate_tokens: true)
```

**APNS tokens:** exactly 64 hexadecimal characters (32 bytes)
**FCM tokens:** 20-500 characters, alphanumeric with hyphens/underscores/colons

---

## Configuration

All configuration goes under `config :pushx`. Here's a complete example with all options:

```elixir
config :pushx,
  # === Credentials ===
  apns_key_id: "ABC123DEFG",
  apns_team_id: "TEAM123456",
  apns_private_key: {:file, "priv/keys/AuthKey.p8"},
  apns_mode: :prod,
  fcm_project_id: "my-project-id",
  fcm_credentials: {:file, "priv/keys/firebase.json"},

  # === HTTP/2 Pool (tune for your traffic level) ===
  finch_pool_count: 2,         # HTTP/2 connections per origin (default: 2; never 1 in prod)
  # finch_pool_size does NOT apply to APNS/FCM (HTTP/2) — it only sizes the HTTP/1 pool (Web Push)
  finch_http2_ping_interval: 60_000,   # HTTP/2 PING keepalive after 60 s idle (default; finch ≥ 0.22)

  # === Timeouts ===
  receive_timeout: 15_000,     # wait for response data (default: 15s)
  pool_timeout: 5_000,         # wait for pool connection (default: 5s)
  connect_timeout: 10_000,     # TCP connect timeout (default: 10s)

  # === Retry ===
  retry_enabled: true,         # default: true
  retry_max_attempts: 3,       # default: 3
  retry_base_delay_ms: 10_000, # default: 10s (Google's recommended minimum)
  retry_max_delay_ms: 60_000,  # default: 60s

  # === Rate Limiting (optional) ===
  rate_limit_enabled: false,   # default: false
  rate_limit_apns: 5000,       # requests per window
  rate_limit_fcm: 5000,        # requests per window
  rate_limit_window_ms: 1000   # 1 second window
```

### Credentials

#### APNS

| Option | Type | Description |
|--------|------|-------------|
| `:apns_key_id` | `String.t()` | 10-character Key ID from Apple |
| `:apns_team_id` | `String.t()` | 10-character Team ID from Apple |
| `:apns_private_key` | `String.t()` \| `{:file, path}` \| `{:system, env_var}` | PEM string, file path, or env var name |
| `:apns_mode` | `:prod` \| `:sandbox` | APNS environment (default: `:prod`) |

#### FCM

| Option | Type | Description |
|--------|------|-------------|
| `:fcm_project_id` | `String.t()` | Firebase project ID |
| `:fcm_credentials` | `map()` \| `{:file, path}` \| `{:json, string}` \| `{:system, env_var}` | Service account as map, file, JSON string, or env var |
| `:fcm_token_fetcher` | `{module, function, args}` | *Optional, advanced.* Bring your own OAuth for the static config: replaces the `PushX.Goth` process PushX would start. Called as `apply(m, f, [goth_name \| args])` (so to reuse your own Goth, wrap it: `def fetch(_), do: Goth.fetch(MyApp.Goth)`), must return `{:ok, %{token: t}}` \| `{:error, reason}`; raises/exits/errors are contained. Named instances use their own `:token_fetcher` config key instead. See `PushX.Config.fcm_token_fetcher/0`. |

#### Web Push (VAPID)

| Option | Type | Description |
|--------|------|-------------|
| `:webpush_vapid_subject` | `String.t()` | `"mailto:..."` or https URL contact for push services |
| `:webpush_vapid_private_key` | `String.t()` | base64url 32-byte key (`mix pushx.vapid`) or EC PEM |
| `:webpush_vapid_public_key` | `String.t()` | Optional (derived); the front end's `applicationServerKey` |

#### Testing

| Option | Type | Description |
|--------|------|-------------|
| `:delivery` | `:live` \| `:test` | `:test` records sends locally instead of contacting the providers — see [Testing Your App](#testing-your-app) and `PushX.Test` |

### Pool Sizing and Keepalive

APNS and FCM are spoken over **HTTP/2**, and in Finch an HTTP/2 "pool" is **one multiplexed connection**. So for APNS/FCM:

- **`finch_pool_count`** = number of HTTP/2 connections per provider origin (default `2`). This is the capacity *and* redundancy knob. Each connection multiplexes up to the server's stream limit (APNS ≈ 1 000 concurrent streams, FCM ≈ 100). Don't run `1` in production: a single socket is a single point of failure, and after a reconnect a retry burst lands entirely on it (`too_many_concurrent_requests`).
- **`finch_pool_size`** is **ignored for HTTP/2** pools. It only sizes the HTTP/1 default pool (Web Push): connections per push-service origin (default 25; the pool's `count` is fixed at 1). If a Web Push batch outruns it, the excess queues for `:pool_timeout` and then fails with a retryable `:connection_error` (`pool_timeout`) — raise `finch_pool_size` or lower batch `:concurrency`. Earlier versions of this README recommended lowering it for low-traffic apps — that had no effect on APNS/FCM.

| Traffic | `finch_pool_count` | Approx. concurrent streams (APNS) |
|---------|--------------------|-----------------------------------|
| Low (< 100/min) | 2 (default) | ~2 000 |
| High (> 1 000/min) | 4–8 | ~4 000–8 000 |

**Idle connections going stale** (Fly.io, AWS NLB, GCP drop idle HTTP/2 sockets; the first send after a quiet period then hits a dead one) is solved by **HTTP/2 PING keepalive**, not by shrinking pools: PushX sends a PING after `finch_http2_ping_interval` ms of idleness (default **60 s**, finch ≥ 0.22), which both keeps the connection warm and detects a dead one before a real send pays for it. Optional proactive recycling: `finch_http2_max_connection_age` (+ `_jitter`) drains and replaces connections on a schedule — useful behind rotating DNS/load balancers. `PushX.reconnect/0` remains for manual recovery, and the retry path still reconnects automatically (coalesced) on the first connection error.

```elixir
config :pushx,
  finch_pool_count: 2,                     # HTTP/2 connections per origin
  finch_http2_ping_interval: 60_000,       # PING after 60 s idle (default)
  finch_http2_max_connection_age: :infinity # or e.g. 30 * 60 * 1000 behind rotating LBs
```

### Retry Behavior

PushX automatically retries transient failures with exponential backoff:

- **Connection errors** — reconnects the HTTP pool, then retries with 1s base delay (1s, 2s, 4s)
- **Server errors (5xx)** — 10s base delay (10s, 20s, 40s) per Google's recommendation
- **Rate limited (429)** — uses `retry-after` header, or 60s default
- **Permanent failures** — never retried (invalid token, payload too large, etc.)

Retries block the calling process (`Process.sleep` backoff) — see "Delivery Semantics". To make exactly one attempt for a specific call, pass `retry: :none` (works on every send function, including `push_batch/4` and named instances) or call the `send_once` variants:

```elixir
PushX.push(:apns, token, "Hi", topic: "com.example.app", retry: :none)
PushX.APNS.send_once(token, payload, topic: "com.example.app")
PushX.FCM.send_once(token, payload)
```

With `retry: :none`, retryable failures come back immediately with `retry_after` set when the provider supplied it, so you can requeue on your own schedule.

### Timeouts

| Option | Default | Description |
|--------|---------|-------------|
| `:receive_timeout` | 15s | How long to wait for response data from APNS/FCM |
| `:pool_timeout` | 5s | How long to wait for a connection from the pool |
| `:connect_timeout` | 10s | TCP connection establishment timeout |

> **Tip:** Increase timeouts if connecting from distant regions (e.g., EU to Apple's US servers).

### Other Options

| Option | Default | Description |
|--------|---------|-------------|
| `:batch_concurrency` | `50` | Default `:concurrency` for `push_batch/4`, `push_batch_stream/4` and `send_batch/3` |
| `:reconnect_cooldown_ms` | `5_000` | Minimum time between *automatic* pool restarts on connection errors, per pool (manual `PushX.reconnect/0` is not gated) |
| `:finch_name` | `PushX.Finch` | Name of the static Finch pool |
| `:on_invalid_token` | — | `{module, function, args}` cleanup callback — see [Token Cleanup Callback](#token-cleanup-callback) |
| `:circuit_breaker_*` | off | See [Circuit Breaker](#circuit-breaker) |
| `:rate_limit_*` | off | See [Rate Limiting](#rate-limiting) |
| `:delivery` | `:live` | `:test` records sends locally — see [Testing Your App](#testing-your-app) |

The complete, authoritative option reference is the `PushX.Config` moduledoc.

You can also override timeouts per-request:

```elixir
PushX.APNS.send(token, payload,
  topic: "com.example.app",
  receive_timeout: 30_000,
  pool_timeout: 10_000
)
```

### Rate Limiting

Optional client-side rate limiting prevents exceeding provider limits. Disabled by default.

```elixir
# Check manually before sending
case PushX.check_rate_limit(:apns) do
  :ok -> # proceed
  {:error, :rate_limited} -> # back off
end
```

When enabled, rate limits are checked automatically before each `send` call.

---

## Dynamic Instances (Runtime Config)

For applications that manage push credentials from a database or admin panel, PushX supports starting, stopping, and reconfiguring provider instances at runtime — no application restart needed.

Each instance gets its own HTTP/2 connection pool, JWT cache (APNS), and OAuth process (FCM). Multiple instances can run concurrently (e.g., APNS sandbox + APNS prod + FCM).

> **Instances live in memory only.** They are not persisted across node restarts and are per-VM (not cluster-wide). Start them on boot from your own source of truth with `PushX.Instance.Loader` — a child you place after your `Repo` that calls a function of yours returning the instances to start — and again when a tenant is provisioned with `PushX.Instance.start/3` (it returns `{:error, :already_started}` for a running name, so re-running is safe):
>
> ```elixir
> children = [MyApp.Repo, {PushX.Instance.Loader, instances: &MyApp.Push.tenant_instances/0}, MyAppWeb.Endpoint]
> ```
>
> One tenant's bad credentials are logged and skipped (pass `on_error: :raise` to fail the boot instead).

### Starting Instances

```elixir
# APNS sandbox (for development/testing)
PushX.Instance.start(:apns_sandbox, :apns,
  key_id: "ABC123",
  team_id: "TEAM456",
  private_key: apns_key_pem,
  mode: :sandbox
)

# APNS production
PushX.Instance.start(:apns_prod, :apns,
  key_id: "ABC123",
  team_id: "TEAM456",
  private_key: apns_key_pem,
  mode: :prod
)

# FCM
PushX.Instance.start(:my_fcm, :fcm,
  project_id: "my-firebase-project",
  credentials: service_account_map
)
```

The names `:apns`, `:fcm` and `:webpush` are reserved for the static config path and cannot be used as instance names.

### Sending via Instances

Pass the instance name instead of `:apns` or `:fcm`:

```elixir
PushX.push(:apns_prod, device_token, "Hello!", topic: "com.example.app")
PushX.push(:my_fcm, device_token, %{title: "Alert", body: "Something happened"})

# Data-only (silent) message via instance
PushX.push_data(:my_fcm, device_token, %{action: "sync", id: 123})
```

Batch sending works the same way:

```elixir
PushX.push_batch(:apns_prod, tokens, message, topic: "com.example.app")
```

### Enable / Disable

Disable an instance to reject new pushes while keeping the connection pool warm:

```elixir
PushX.Instance.disable(:apns_sandbox)
# => PushX.push(:apns_sandbox, ...) returns {:error, %Response{status: :provider_disabled}}

PushX.Instance.enable(:apns_sandbox)
# => pushes work again
```

### Reconfigure

Update config without restarting the application. The old pool is stopped and a new one starts with the merged config:

```elixir
# Switch environment
PushX.Instance.reconfigure(:apns_sandbox, mode: :prod)

# Rotate credentials
PushX.Instance.reconfigure(:apns_prod,
  key_id: "NEW_KEY_ID",
  private_key: new_pem_string
)
```

### List and Status

```elixir
PushX.Instance.list()
#=> [
#=>   %{name: :apns_prod, provider: :apns, enabled: true},
#=>   %{name: :apns_sandbox, provider: :apns, enabled: false},
#=>   %{name: :my_fcm, provider: :fcm, enabled: true}
#=> ]

PushX.Instance.status(:apns_prod)
#=> {:ok, %{provider: :apns, enabled: true}}
```

### Stop

```elixir
PushX.Instance.stop(:apns_sandbox)
```

Cleans up the Finch pool, JWT cache, Goth process (FCM), and ETS entry.

### Example: Database-Backed Admin Panel

```elixir
defmodule MyApp.PushAdmin do
  @doc "Boot all saved instances on application start."
  def boot do
    MyApp.Repo.all(MyApp.PushConfig)
    |> Enum.each(fn config ->
      PushX.Instance.start(
        String.to_atom(config.name),
        String.to_atom(config.provider),
        build_opts(config)
      )
    end)
  end

  @doc "Called from admin panel when config is updated."
  def update(config) do
    name = String.to_atom(config.name)
    PushX.Instance.reconfigure(name, build_opts(config))
  end

  @doc "Called from admin panel toggle."
  def toggle(name, enabled?) do
    if enabled?,
      do: PushX.Instance.enable(name),
      else: PushX.Instance.disable(name)
  end

  defp build_opts(%{provider: "apns"} = c) do
    [
      key_id: c.key_id,
      team_id: c.team_id,
      private_key: c.private_key,
      mode: String.to_atom(c.mode)
    ]
  end

  defp build_opts(%{provider: "fcm"} = c) do
    [
      project_id: c.project_id,
      credentials: JSON.decode!(c.credentials_json)
    ]
  end
end
```

Call `MyApp.PushAdmin.boot()` from your `Application.start/2` after PushX starts.

### Instance Config Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `:key_id` | `String.t()` | required (APNS) | Apple Key ID |
| `:team_id` | `String.t()` | required (APNS) | Apple Team ID |
| `:private_key` | `String.t()` \| `{:file, path}` \| `{:system, env}` | required (APNS) | PEM private key |
| `:mode` | `:prod` \| `:sandbox` | `:prod` | APNS environment |
| `:project_id` | `String.t()` | required (FCM) | Firebase project ID |
| `:credentials` | `map()` \| `String.t()` | required (FCM, unless `:token_fetcher`) | Service account (map or JSON string); validated at start |
| `:token_fetcher` | `{module, function, args}` | — | Bring your own OAuth for this instance (no Goth started; makes `:credentials` optional). The global `:fcm_token_fetcher` never applies to instances. |
| `:pool_size` | `integer()` | `2` | Finch connections per pool |
| `:pool_count` | `integer()` | `1` | Number of Finch pools |
| `:receive_timeout` | `integer()` | `15_000` | Response timeout (ms) |
| `:pool_timeout` | `integer()` | `5_000` | Pool checkout timeout (ms) |
| `:connect_timeout` | `integer()` | `10_000` | TCP connect timeout (ms) |

---

## Credential Storage

### File System (Development)

```elixir
# config/dev.exs
config :pushx,
  apns_private_key: {:file, "priv/keys/AuthKey.p8"},
  fcm_credentials: {:file, "priv/keys/firebase-service-account.json"}
```

Add `/priv/keys/` to `.gitignore`.

### Environment Variables (Production)

```elixir
# config/runtime.exs
config :pushx,
  apns_key_id: System.get_env("APNS_KEY_ID"),
  apns_team_id: System.get_env("APNS_TEAM_ID"),
  apns_private_key: System.get_env("APNS_PRIVATE_KEY"),
  apns_mode: if(System.get_env("APNS_SANDBOX") == "true", do: :sandbox, else: :prod),

  fcm_project_id: System.get_env("FCM_PROJECT_ID"),
  fcm_credentials: System.get_env("FCM_CREDENTIALS") |> JSON.decode!()
```

> **Tip:** For multiline keys (APNS .p8), set the env var directly from the file: `export APNS_PRIVATE_KEY="$(cat AuthKey.p8)"`

### Fly.io Secrets

```bash
fly secrets set APNS_KEY_ID="ABC123DEFG"
fly secrets set APNS_TEAM_ID="TEAM123456"
fly secrets set APNS_PRIVATE_KEY="$(cat AuthKey.p8)"
fly secrets set FCM_PROJECT_ID="my-project-id"
fly secrets set FCM_CREDENTIALS="$(cat firebase-service-account.json)"
```

Then use `System.fetch_env!/1` in `config/runtime.exs`:

```elixir
if config_env() == :prod do
  config :pushx,
    apns_key_id: System.fetch_env!("APNS_KEY_ID"),
    apns_team_id: System.fetch_env!("APNS_TEAM_ID"),
    apns_private_key: System.fetch_env!("APNS_PRIVATE_KEY"),
    apns_mode: :prod,
    fcm_project_id: System.fetch_env!("FCM_PROJECT_ID"),
    fcm_credentials: System.fetch_env!("FCM_CREDENTIALS") |> JSON.decode!()
end
```

### AWS Secrets Manager / Vault

```elixir
if config_env() == :prod do
  {:ok, %{"SecretString" => apns_key}} =
    ExAws.SecretsManager.get_secret_value("pushx/apns-key")
    |> ExAws.request()

  config :pushx,
    apns_private_key: apns_key
end
```

---

## Getting Your Credentials

### Apple APNS Setup

You need: **Key ID**, **Team ID**, and a **Private Key (.p8 file)**.

#### Step 1: Get Your Team ID

1. Go to [Apple Developer Account](https://developer.apple.com/account)
2. Your **Team ID** is shown in the top-right corner (10 characters)

#### Step 2: Create an APNS Key

1. Go to [Certificates, Identifiers & Profiles](https://developer.apple.com/account/resources/authkeys/list)
2. Click **Keys** > **+** (Create a new key)
3. Enter a name (e.g., "Push Notifications Key")
4. Check **Apple Push Notifications service (APNs)**
5. Click **Continue** > **Register**
6. **Download the .p8 file** (you can only download it once!)
7. Note the **Key ID** shown (10 characters)

### Google FCM Setup

You need: **Project ID** and a **Service Account JSON file**.

#### Step 1: Create/Open Firebase Project

1. Go to [Firebase Console](https://console.firebase.google.com/)
2. Create a new project or select an existing one
3. Note your **Project ID** in Project Settings

#### Step 2: Enable Cloud Messaging API

1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Select your Firebase project
3. Go to **APIs & Services** > **Library**
4. Search for "Firebase Cloud Messaging API" and **Enable** it

#### Step 3: Create Service Account Key

1. In Firebase Console, go to **Project Settings** (gear icon)
2. Click **Service accounts** tab
3. Click **Generate new private key**
4. Save the JSON file securely

### Credential Rotation

APNS .p8 keys and FCM service accounts **don't expire**. You only need to rotate them if you revoke a key or want to follow a rotation policy.

#### With restart (simplest)

1. Generate new credentials in Apple/Google console
2. Update your secrets (Fly: `fly secrets set`, AWS: update in Secrets Manager)
3. Redeploy your app
4. Revoke old credentials after all instances are updated

#### Without restart (static config)

The static config path reads credentials from `Application` env on each JWT generation, so you can hot-swap them at runtime:

```elixir
# 1. Update application env with new credentials
Application.put_env(:pushx, :apns_key_id, "NEW_KEY_ID")
Application.put_env(:pushx, :apns_private_key, new_pem_string)

# 2. Clear the cached JWT (otherwise the old token is used for up to 50 min)
:persistent_term.erase(:pushx_apns_jwt_cache)

# 3. Reconnect to discard connections authenticated with the old token
PushX.reconnect()
```

For FCM, Goth manages OAuth2 tokens automatically. To rotate service account credentials without restart, use the dynamic instance API below.

#### Without restart (dynamic instances)

If you use `PushX.Instance`, call `reconfigure/2` — it stops the old pool and starts a fresh one with the new credentials:

```elixir
PushX.Instance.reconfigure(:apns_prod,
  key_id: "NEW_KEY_ID",
  private_key: new_pem_string
)
```

In-flight requests on the old pool get connection errors, which the retry logic handles automatically.

---

## Telemetry

PushX emits telemetry events for monitoring and metrics:

| Event | When | Measurements | Metadata |
|-------|------|--------------|----------|
| `[:pushx, :push, :start]` | Request starts | `system_time` | `provider`, `token` |
| `[:pushx, :push, :stop]` | Request succeeds | `duration` | `provider`, `token`, `status`, `id` |
| `[:pushx, :push, :error]` | Request fails | `duration` | `provider`, `token`, `status`, `reason` |
| `[:pushx, :push, :exception]` | Exception raised | `duration` | `provider`, `token`, `kind`, `reason` |
| `[:pushx, :retry, :attempt]` | Retry attempted | `delay_ms`, `attempt` | `provider`, `status` |

> Tokens are automatically truncated in telemetry metadata for privacy (first 8 + last 4 characters).

**Ready-made metrics.** With the optional `telemetry_metrics` dependency, `PushX.Telemetry.metrics/0` returns a curated, low-cardinality `Telemetry.Metrics` list (sends by provider, errors by status, exceptions, latency distributions with provider-tuned buckets, retry attempts and delays — never tagged by token) that drops straight into LiveDashboard, PromEx, or `TelemetryMetricsPrometheus`:

```elixir
def metrics, do: MyApp.metrics() ++ PushX.Telemetry.metrics()
```

### Example: Attach a Logger

```elixir
# In your Application.start/2
:telemetry.attach_many(
  "pushx-logger",
  [
    [:pushx, :push, :stop],
    [:pushx, :push, :error]
  ],
  fn
    [:pushx, :push, :stop], %{duration: d}, %{provider: p}, _ ->
      ms = System.convert_time_unit(d, :native, :millisecond)
      Logger.info("PushX #{p} sent in #{ms}ms")

    [:pushx, :push, :error], _, %{provider: p, status: s, reason: r}, _ ->
      Logger.warning("PushX #{p} failed: #{s} - #{r}")
  end,
  nil
)
```

### Example: With Telemetry.Metrics

```elixir
defmodule MyApp.Telemetry do
  import Telemetry.Metrics

  def metrics do
    [
      counter("pushx.push.stop.count", tags: [:provider]),
      counter("pushx.push.error.count", tags: [:provider, :status]),
      distribution("pushx.push.stop.duration",
        unit: {:native, :millisecond},
        tags: [:provider]
      )
    ]
  end
end
```

---

## Circuit Breaker

PushX includes an optional circuit breaker that temporarily blocks requests to a provider after consecutive failures. This prevents wasting resources on dead connections. It is keyed per provider (and per named instance): for `:webpush` one key covers every push service, so a run of 5xx from one vendor can open it for all browsers.

```elixir
config :pushx,
  circuit_breaker_enabled: true,
  circuit_breaker_threshold: 5,        # consecutive failures to trip
  circuit_breaker_cooldown_ms: 30_000  # ms before retrying
```

**States:**
- **Closed** — Normal operation, all requests flow through
- **Open** — Provider is failing, requests are immediately rejected with `{:error, %Response{status: :circuit_open}}`
- **Half-open** — After cooldown, one probe request is allowed. Success closes the circuit; failure re-opens it

Only `:connection_error` and `:server_error` responses count as failures. Invalid tokens and rate limits do not trip the circuit.

```elixir
# Check circuit breaker state
PushX.CircuitBreaker.state(:apns)
#=> :closed

# Manual reset
PushX.CircuitBreaker.reset(:apns)
```

---

## Health Check

Check provider configuration and circuit breaker status:

```elixir
PushX.health_check()
#=> %{
#=>   apns: %{configured: true, circuit: :closed},
#=>   fcm: %{configured: true, circuit: :closed},
#=>   instances: %{
#=>     tenant_42_apns: %{provider: :apns, enabled: true, circuit: :closed},
#=>     tenant_7_fcm: %{provider: :fcm, enabled: false, circuit: :open}
#=>   }
#=> }
```

Named instances have their own circuit breakers (keyed by name), so one tenant's outage shows up under `:instances` without affecting the static providers or other tenants.

---

## Token Cleanup Callback

> For Web Push the "token" passed to the callback is the subscription map you sent with — delete it from your store by endpoint.

Automatically clean up invalid tokens from your database when a push fails with `:invalid_token`, `:expired_token`, or `:unregistered`:

```elixir
config :pushx,
  on_invalid_token: {MyApp.Push, :handle_invalid_token, []}
```

The callback receives `(provider, device_token, ...extra_args)` and runs asynchronously:

```elixir
defmodule MyApp.Push do
  def handle_invalid_token(provider, device_token) do
    MyApp.Tokens.delete_by_token(device_token)
    Logger.info("Removed invalid #{provider} token")
  end
end
```

---

## Delivery Semantics

PushX delivers **at least once, not exactly once**. Two mechanisms can
produce duplicate deliveries:

- **Automatic retries.** A send that fails with a connection error may have
  actually reached the provider (e.g. the response was lost after the request
  was processed). The retry then delivers the same notification again.
- **Batch timeouts.** `push_batch/4` kills tasks that exceed `:timeout`. A
  killed task may have an HTTP request already in flight that the provider
  still processes and delivers — even though PushX reports that token as
  `{:error, %Response{status: :connection_error, reason: "timeout"}}`.

Conversely, an `{:ok, ...}` result means the *provider accepted* the message,
not that the device displayed it.

If duplicates matter for your use case:

- **APNS:** set `collapse_id` (or `Message.collapse_key/2`) — Apple replaces
  an undelivered notification carrying the same ID instead of stacking a
  duplicate.
- **FCM:** set `android: %{"collapse_key" => ...}` (or
  `Message.collapse_key/2`) for the same effect on Android.
- **App level:** include your own idempotency key in the payload `data` and
  de-duplicate in the app's notification handler.
- Or skip retries (`retry: :none` per call, or `retry_enabled: false` globally) and handle failures yourself.

---

## Testing Your App

Set test delivery mode and every send is validated exactly as in production (required `:topic`, target format, payload size…) but then **recorded and answered locally** instead of contacting Apple or Google. No credentials needed, no retries, no network:

```elixir
# config/test.exs
config :pushx, delivery: :test
```

Web Push targets are checked cryptographically before they are recorded (a
real P-256 `p256dh` point, 16-byte `auth`, `https` endpoint) — use
`PushX.Test.webpush_subscription/1` for a valid fixture instead of hand-writing
keys.

```elixir
defmodule MyApp.OrdersTest do
  use ExUnit.Case, async: true
  import PushX.Test.Assertions

  test "shipping an order notifies the customer" do
    MyApp.Orders.ship(order)

    push = assert_pushed(%{provider: :apns, target: ^device_token})
    assert push.payload["aps"]["alert"]["title"] == "Order shipped"
    assert push.opts[:topic] == "com.example.app"
    refute_pushed(%{provider: :fcm})
  end

  test "dead tokens get cleaned up" do
    PushX.Test.stub(fn
      %{target: "dead-token"} -> {:error, :unregistered}   # what APNS would say
      _push -> :ok
    end)

    MyApp.Notifier.broadcast("Hi")

    refute MyApp.Tokens.exists?("dead-token")              # your :on_invalid_token ran
  end
end
```

Recorded pushes are scoped to the test process (and processes it spawns, including `push_batch/4` workers), so `async: true` is fine. `PushX.Test.pushes/0` gives you the raw list; `PushX.Test.stub/1` scripts provider responses per test; `PushX.Test.apns_private_key/0` / `fcm_credentials/0` provide throwaway keys for starting named instances in tests. See `PushX.Test` for the details and what test mode does *not* simulate.

---

## Troubleshooting

### `mix pushx.doctor`

Checks the configuration and credentials offline — the same checks the library runs at start/send time, reported all at once, without sending anything. Fails (non-zero exit) on a credential problem, so it can gate a deploy:

```
$ MIX_ENV=prod mix pushx.doctor && mix release
PushX 0.14.0 — configuration check (MIX_ENV=prod)

  APNS  ✔ configured (key ABC123DEFG, team TEAM123456, mode :prod)
        ✔ private key resolves and signs ES256 (P-256)
  FCM   ✔ project my-project
        ✔ service-account credentials resolve and sign RS256

  Delivery: live
  Retries:  enabled, 3 attempts, 10000ms base delay — a batch task may block up to 180s
  Pools:    finch_pool_count 2 HTTP/2 connections per origin; PING after 60000ms idle
  Breaker:  off   Rate limit: off
  Cleanup:  on_invalid_token → MyApp.Tokens.delete_by_token/2

All checks passed.
```

### `too_many_concurrent_requests` Error

```
[warning] [PushX.APNS] HTTP/2 connection saturated (too_many_concurrent_requests): every stream on the pool's connections is in use ...
```

PushX logs this distinctly from a network failure. It means every HTTP/2 stream on the pool's connections is busy — typically either a **retry burst converging on a freshly reconnected connection** (a pool with `finch_pool_count: 1` is especially prone) or genuine overload. It is retried with backoff (jittered, so a burst spreads out).

A canonical real-world trace (pushx 0.12, Fly.io, `finch_pool_count: 1`, 12 minutes idle before a 3-token push) shows the two problems chained — an idle-dead socket, then the retry burst saturating the single fresh connection:

```
[PushX.APNS] Connection error: %Finch.Error{reason: :timeout}
[PushX.APNS] Connection error: %Finch.Error{reason: :connection_process_went_down}
[PushX.Retry] Attempt 1/3 failed for apns (connection_error), retrying in 960ms
[PushX] Reconnected HTTP pools (stale connections discarded)
[PushX.APNS] Connection error: %Finch.HTTPError{reason: :too_many_concurrent_requests, module: Mint.HTTP2}
[PushX.Retry] Attempt 2/3 failed for apns (connection_error), retrying in 2007ms
[Push] Sent to ios x3
```

Delivered, but ~2 s late and two attempts burned. With `finch_pool_count: 2+` and the default PING keepalive, neither line appears: the idle socket is kept alive (or detected dead before the send), and a retry burst has several connections to land on.

**Fix:** raise `finch_pool_count` (HTTP/2 connections per origin; `finch_pool_size` does nothing for HTTP/2) and/or lower batch `:concurrency`; enable rate limiting for sustained volume:
```elixir
config :pushx,
  finch_pool_count: 4,
  rate_limit_enabled: true,
  rate_limit_apns: 2000,
  rate_limit_fcm: 2000
```

### Stale connections after idle periods

On cloud infrastructure (Fly.io, AWS, GCP) idle HTTP/2 connections are silently dropped; the first send after a quiet period then fails with a connection error and is recovered by the retry path (which reconnects the pool) — a second or two late. Prevent it instead of recovering from it: HTTP/2 PING keepalive is on by default (`finch_http2_ping_interval: 60_000`, finch ≥ 0.22 — older finch ignores it with a boot warning); lower it if your platform drops faster, or add `finch_http2_max_connection_age` to recycle connections proactively. Manual: `PushX.reconnect()`.

### `request_timeout` Error

```
[error] [PushX.APNS] Connection error: %Finch.Error{reason: :request_timeout}
```

1. **Increase timeouts** if connecting from distant regions (e.g., EU to US):
   ```elixir
   config :pushx,
     receive_timeout: 30_000,
     connect_timeout: 20_000
   ```

2. PushX automatically retries connection errors with exponential backoff (1s, 2s, 4s)

3. If this follows a `too_many_concurrent_requests` error, see the stale connections fix above

### Debugging Tips

Enable telemetry logging to monitor push performance:

```elixir
:telemetry.attach("pushx-debug", [:pushx, :push, :error], fn _, _, meta, _ ->
  Logger.warning("Push failed: #{meta.provider} - #{meta.status} - #{meta.reason}")
end, nil)
```

---

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

See [CONTRIBUTING.md](https://github.com/cignosystems/pushx/blob/main/CONTRIBUTING.md) for the repo layout, test commands and
the release process, and [ROADMAP.md](https://github.com/cignosystems/pushx/blob/main/ROADMAP.md) for what is planned before
and after 1.0 (Expo design note in `docs/design/`).

## License

MIT License. See [LICENSE](LICENSE) for details.

---

Built with care by [Cigno Systems AB](https://github.com/cignosystems)
