Push notifications for Elixir that just work: APNS, FCM and Web Push in one call.
Retries, circuit breaker, dead-token cleanup and telemetry built in. Per-tenant credentials at runtime.
Nothing to add to your supervision tree.
_{Web Push per RFC 8030 (transport), RFC 8291 (encryption) and RFC 8292 (VAPID) — every browser, no Firebase required.}

AI coding assistants: see AGENTS.md for the mental model, idiomatic patterns, and a list of mistakes commonly made when integrating PushX. Also rendered on hexdocs.

Table of Contents

• Features
• Quick Start
• Usage Guide
• Configuration
• Dynamic Instances (Runtime Config)
• Credential Storage
• Getting Your Credentials
• Telemetry
• Circuit Breaker
• Health Check
• Token Cleanup Callback
• Delivery Semantics
• Testing Your App
• Troubleshooting
• Contributing
• License

Features

• HTTP/2 connections via Finch (Mint-based) for optimal performance
• APNS (Apple Push Notification Service) with JWT authentication
• FCM (Firebase Cloud Messaging) with OAuth2 via Goth
• Web Push — standards-based for every browser incl. Safari 16+: RFC 8030 transport, RFC 8291aes128gcm encryption (reproduces the RFC's test vectors), RFC 8292 VAPID; FCM webpush and legacy Safari APNS also supported
• Batch sending — send to multiple tokens concurrently with configurable parallelism, as a list or a lazy stream
• FCM topics & conditions — fan out to {:topic, "news"} or {:condition, "'a' in topics && 'b' in topics"} with the same call
• Multi-tenant — named runtime instances with their own credentials, pools and breakers
• Token validation — validate token format before sending to catch errors early
• Rate limiting — optional client-side rate limiting to avoid provider throttling
• Automatic retry — exponential backoff for rate limits and server errors (opt out per call with retry: :none)
• Circuit breaker & dead-token cleanup — stop hammering a failing provider; get told which tokens to delete
• Telemetry — built-in instrumentation for metrics and monitoring
• Message builder — fluent API for constructing notifications
• Structured responses — consistent error handling across providers
• Zero JSON dependency — uses Elixir 1.18+ built-in JSON module

Requirements

• Elixir 1.18+ (for the built-in JSON module)
• OTP 26+

CI runs the suite on Elixir 1.20/OTP 29, 1.20/28, 1.19/28, 1.19/27 and 1.18/26, so anything in that range is exercised on every push; the release itself is built with the latest stable pair.

Tested on Elixir 1.18/1.19 with OTP 26, 27, and 28.

Quick Start

1. Install

Add pushx to your dependencies in mix.exs:

def deps do
  [
    {:pushx, "~> 0.15"}
  ]
end

2. Configure

Add credentials to config/runtime.exs:

config :pushx,
  # APNS (iOS)
  apns_key_id: System.fetch_env!("APNS_KEY_ID"),
  apns_team_id: System.fetch_env!("APNS_TEAM_ID"),
  apns_private_key: System.fetch_env!("APNS_PRIVATE_KEY"),
  apns_mode: :prod,

  # FCM (Android)
  fcm_project_id: System.fetch_env!("FCM_PROJECT_ID"),
  fcm_credentials: System.fetch_env!("FCM_CREDENTIALS") |> JSON.decode!()

PushX starts its own HTTP/2 connection pools and OAuth processes automatically — no additional supervision tree setup needed.

Need help getting credentials? See Getting Your Credentials below.

3. Send a notification

# Send to iOS
PushX.push(:apns, device_token, "Hello!", topic: "com.example.app")

# Send to Android
PushX.push(:fcm, device_token, "Hello!")

# With title and body
PushX.push(:apns, token, %{title: "Welcome", body: "Thanks for signing up!"},
  topic: "com.example.app")

That's it. PushX handles HTTP/2 connections, JWT/OAuth authentication, and automatic retry.

Usage Guide

Message Builder

Build rich notifications with the fluent API:

message = PushX.message()
  |> PushX.Message.title("Order Update")
  |> PushX.Message.body("Your order #1234 has shipped")
  |> PushX.Message.badge(1)
  |> PushX.Message.sound("default")
  |> PushX.Message.data(%{order_id: "1234", status: "shipped"})

PushX.push(:apns, token, message, topic: "com.example.app")

Every field maps to both providers where the concept exists — the iOS-specific ones reach iOS devices addressed through FCM via an automatic apns override (PushX.Message.to_fcm_apns/1), merged under any explicit :apns option you pass.

You can also pass a plain string, a %{title: ..., body: ...} map, or a raw APNS/FCM payload map directly to push/4.

Response Handling

Every push returns {:ok, Response} or {:error, Response}:

case PushX.push(:apns, token, message, topic: "com.example.app") do
  {:ok, %PushX.Response{status: :sent, id: apns_id}} ->
    Logger.info("Notification sent with ID: #{apns_id}")

  {:error, %PushX.Response{} = response} ->
    if PushX.Response.should_remove_token?(response) do
      # Token is invalid, expired, or unregistered — delete it
      MyApp.Tokens.delete(token)
    else
      Logger.error("Push failed: #{response.status} - #{response.reason}")
    end
end

Response struct:

%PushX.Response{
  provider: :apns | :fcm,
  status: :sent | :invalid_token | :expired_token | ...,
  id: "message-id" | nil,
  reason: "error reason" | nil,
  raw: raw_response_body,
  retry_after: seconds | nil
}

Helper functions:

PushX.Response.success?(response)              # true if status == :sent
PushX.Response.should_remove_token?(response)  # true for invalid/expired/unregistered
PushX.Response.retryable?(response)            # true for connection_error/rate_limited/server_error

Batch Sending

Send to multiple devices concurrently:

results = PushX.push_batch(:apns, tokens, message, topic: "com.example.app")

# Process results
Enum.each(results, fn
  {token, {:ok, response}} -> Logger.info("Sent to #{token}")
  {token, {:error, response}} ->
    if PushX.Response.should_remove_token?(response) do
      MyApp.Tokens.delete(token)
    end
end)

For aggregate counts, use the bang variant:

%{success: 95, failure: 5, total: 100} =
  PushX.push_batch!(:fcm, tokens, "Hello!")

Large audiences.push_batch/4 holds every result in memory. For tens of thousands of tokens and up, use the lazy variant — it accepts any enumerable (a database stream works; the input is enumerated exactly once), yields {token, result} pairs in input order as each one completes, and keeps memory bounded:

Repo.transaction(fn ->
  Repo.stream(from t in Token, where: t.provider == :fcm, select: t.value)
  |> then(&PushX.push_batch_stream(:fcm, &1, "Maintenance in 10m", concurrency: 100, retry: :none))
  |> Stream.each(fn
    {token, {:error, resp}} -> if PushX.Response.should_remove_token?(resp), do: Tokens.delete(token)
    _ -> :ok
  end)
  |> Stream.run()
end)

Two performance notes: a retrying task holds one of the batch's concurrency slots for its whole backoff (up to a minute on 429s), so pass retry: :none for big batches and requeue failures yourself; and if you stick with push_batch/4, chunk the input (Enum.chunk_every(tokens, 10_000)).

FCM Topics and Conditions

FCM can fan out server-side. Pass a topic or condition where you would pass a device token — the rest of the API is unchanged, including named instances and push_data/4:

PushX.push(:fcm, {:topic, "news"}, "Breaking news")
PushX.push(:fcm, {:condition, "'news' in topics && 'sports' in topics"}, message)
PushX.push_data(:my_fcm_tenant, {:topic, "sync"}, %{action: "refresh"})

Topic names are the bare name (no /topics/ prefix), characters [a-zA-Z0-9-_.~%]; invalid targets are rejected locally with :invalid_request. Topics/conditions never trigger token cleanup, and APNS returns :invalid_request for them (topics are an FCM feature).

Manage subscriptions from the server too (Instance ID API; auto-chunked at Google's 1 000 tokens per request; per-token results in input order):

{:ok, results} = PushX.subscribe(:fcm, tokens, "news")        # or a named FCM instance
for {token, {:error, "NOT_FOUND"}} <- results, do: MyApp.Tokens.delete(token)
{:ok, _} = PushX.unsubscribe(:fcm, tokens, "news")

Dry Runs and Tracing

• FCM dry run — validate_only: true asks FCM to validate a message (token registration, payload) without delivering it; a {:ok, %Response{status: :sent}} means "would have been accepted", errors are the real ones. Handy for checking stored tokens before a campaign: PushX.push(:fcm, token, msg, validate_only: true).
• APNS tracing — pass your own UUID as apns_id:; Apple echoes it in response.id and shows it in delivery logs and the Push Notifications Console, so a push can be traced end-to-end: PushX.push(:apns, token, msg, topic: "...", apns_id: Ecto.UUID.generate()).

Silent/Background Notification

payload = PushX.APNS.silent_notification(%{action: "sync", resource: "messages"})

PushX.APNS.send(token, payload,
  topic: "com.example.app",
  push_type: "background",
  priority: 5
)

Data-Only Message (FCM)

Send data without a visible notification. All values are automatically converted to strings (FCM requirement):

# Via unified API (supports named instances)
PushX.push_data(:fcm, token, %{action: "sync", id: 123})
PushX.push_data(:my_fcm, token, %{action: "sync", id: 123})

# Via provider module directly
PushX.FCM.send_data(token, %{action: "sync", id: 123})

Notification with Custom Data (FCM)

Send a visible notification with a custom data payload attached:

# Structured payload — notification + data
PushX.push(:fcm, token, %{
  "notification" => %{"title" => "Alert", "body" => "Something happened"},
  "data" => %{"event_id" => "1", "action" => "open_event"}
})

# Works with named instances too
PushX.push(:my_fcm, token, %{
  "notification" => %{"title" => "Alert", "body" => "Something happened"},
  "data" => %{"event_id" => "1"}
})

Web Push

Standards-based Web Push (all browsers — recommended)

Chrome, Firefox, Edge, Opera, Samsung Internet and Safari 16+ (macOS Ventura / iOS 16.4+) all speak the same protocol: the browser gives you a subscription (endpoint + keys), you encrypt the payload for it (RFC 8291) and authenticate with VAPID (RFC 8292). PushX does all of that; no Firebase, no Apple website-push ID.

# 1. once: generate a VAPID key pair
#    $ mix pushx.vapid
config :pushx,
  webpush_vapid_subject: "mailto:ops@example.com",
  webpush_vapid_private_key: System.fetch_env!("WEBPUSH_VAPID_PRIVATE_KEY")
  # webpush_vapid_public_key is optional (derived); it's the front end's applicationServerKey

# 2. the browser subscribes (JS) and POSTs the subscription to you
#    const sub = await registration.pushManager.subscribe({userVisibleOnly: true, applicationServerKey: PUBLIC_KEY})

# 3. send — the subscription object is the target
PushX.push(:webpush, subscription, %{title: "Order shipped", body: "Arrives Tuesday", data: %{url: "/orders/42"}},
  ttl: 3600, urgency: :high, topic: "order-42")

PushX.Message maps to the Notification API shape (title, body, icon, tag, data) your service worker shows, and its ttl/priority become the TTL/Urgency headers; a map is sent as JSON exactly as given (also through PushX.push/4 — Notification API options like icon/actions survive); a string is the title via PushX.push/4 ({"title": ..., "body": ""}, as for APNS/FCM) and verbatim via PushX.WebPush.send/3. A 404/410 from the push service means the subscription is gone → :unregistered, and :on_invalid_token fires with the subscription map so you can delete it. Payloads are limited to ~4 KB; :ttl, :urgency (:very_low | :low | :normal | :high) and :topic (collapse key) are the RFC 8030 knobs. Multi-tenant: PushX.Instance.start(name, :webpush, vapid_subject: ..., vapid_private_key: ...). See PushX.WebPush.

FCM Web Push (apps using the Firebase JS SDK)

If your web app already uses Firebase Messaging, its tokens go through FCM like mobile tokens:

PushX.push(:fcm, web_token, %{title: "Hello", body: "From web!"})

PushX.FCM.send_web(web_token, "New Message", "Check it out", "https://example.com/messages",
  icon: "https://example.com/icon.png", badge: "https://example.com/badge.png")

Safari legacy website push (APNS)

Pre-Safari-16 "website push" uses APNS with a web. topic (the website push ID) and 64-hex tokens — still supported:

payload = PushX.APNS.web_notification("New Article", "Check it out", "https://example.com/article/123")
PushX.APNS.send(safari_token, payload, topic: "web.com.example.website")

Direct Provider Access

The unified PushX.push/4 normalizes payloads across providers. When you need provider-specific features, use the modules directly:

# APNS — full control over headers and payload
PushX.APNS.send(token, payload, topic: "com.app", push_type: "voip")
PushX.APNS.send_once(token, payload, opts)          # no automatic retry
PushX.APNS.send_batch(tokens, payload, opts)
PushX.APNS.notification("Title", "Body", badge)
PushX.APNS.notification_with_data("Title", "Body", %{key: "value"})
PushX.APNS.silent_notification(%{action: "sync"})
PushX.APNS.web_notification("Title", "Body", "https://url")
PushX.APNS.web_notification_with_data("Title", "Body", "https://url", %{key: "val"})

# FCM — full control over android/webpush/data options
PushX.FCM.send(token, payload, data: %{key: "value"})
PushX.FCM.send_once(token, payload, opts)            # no automatic retry
PushX.FCM.send_batch(tokens, payload, opts)
PushX.FCM.send_data(token, %{key: "value"})          # data-only, no visible notification
PushX.FCM.send_web(token, "Title", "Body", "https://link", opts)
PushX.FCM.notification("Title", "Body", image: "https://img")
PushX.FCM.web_notification("Title", "Body", "https://link", opts)
PushX.FCM.web_notification_with_data("Title", "Body", "https://link", %{key: "val"})

Token Validation

Validate tokens before sending to catch format errors early:

PushX.valid_token?(:apns, token)    # true/false
PushX.validate_token(:apns, token)  # :ok | {:error, :empty | :invalid_length | :invalid_format}

# In batch — filter out bad tokens automatically
PushX.push_batch(:apns, tokens, message, topic: "...", validate_tokens: true)

APNS tokens: exactly 64 hexadecimal characters (32 bytes) FCM tokens: 20-500 characters, alphanumeric with hyphens/underscores/colons

Configuration

All configuration goes under config :pushx. Here's a complete example with all options:

config :pushx,
  # === Credentials ===
  apns_key_id: "ABC123DEFG",
  apns_team_id: "TEAM123456",
  apns_private_key: {:file, "priv/keys/AuthKey.p8"},
  apns_mode: :prod,
  fcm_project_id: "my-project-id",
  fcm_credentials: {:file, "priv/keys/firebase.json"},

  # === HTTP/2 Pool (tune for your traffic level) ===
  finch_pool_count: 2,         # HTTP/2 connections per origin (default: 2; never 1 in prod)
  # finch_pool_size does NOT apply to APNS/FCM (HTTP/2) — it only sizes the HTTP/1 pool (Web Push)
  finch_http2_ping_interval: 60_000,   # HTTP/2 PING keepalive after 60 s idle (default; finch ≥ 0.22)

  # === Timeouts ===
  receive_timeout: 15_000,     # wait for response data (default: 15s)
  pool_timeout: 5_000,         # wait for pool connection (default: 5s)
  connect_timeout: 10_000,     # TCP connect timeout (default: 10s)

  # === Retry ===
  retry_enabled: true,         # default: true
  retry_max_attempts: 3,       # default: 3
  retry_base_delay_ms: 10_000, # default: 10s (Google's recommended minimum)
  retry_max_delay_ms: 60_000,  # default: 60s

  # === Rate Limiting (optional) ===
  rate_limit_enabled: false,   # default: false
  rate_limit_apns: 5000,       # requests per window
  rate_limit_fcm: 5000,        # requests per window
  rate_limit_window_ms: 1000   # 1 second window

Credentials

APNS

FCM

Web Push (VAPID)

Testing

Pool Sizing and Keepalive

APNS and FCM are spoken over HTTP/2, and in Finch an HTTP/2 "pool" is one multiplexed connection. So for APNS/FCM:

• finch_pool_count = number of HTTP/2 connections per provider origin (default 2). This is the capacity and redundancy knob. Each connection multiplexes up to the server's stream limit (APNS ≈ 1 000 concurrent streams, FCM ≈ 100). Don't run 1 in production: a single socket is a single point of failure, and after a reconnect a retry burst lands entirely on it (too_many_concurrent_requests).
• finch_pool_size is ignored for HTTP/2 pools. It only sizes the HTTP/1 default pool (Web Push): connections per push-service origin (default 25; the pool's count is fixed at 1). If a Web Push batch outruns it, the excess queues for :pool_timeout and then fails with a retryable :connection_error (pool_timeout) — raise finch_pool_size or lower batch :concurrency. Earlier versions of this README recommended lowering it for low-traffic apps — that had no effect on APNS/FCM.

Idle connections going stale (Fly.io, AWS NLB, GCP drop idle HTTP/2 sockets; the first send after a quiet period then hits a dead one) is solved by HTTP/2 PING keepalive, not by shrinking pools: PushX sends a PING after finch_http2_ping_interval ms of idleness (default 60 s, finch ≥ 0.22), which both keeps the connection warm and detects a dead one before a real send pays for it. Optional proactive recycling: finch_http2_max_connection_age (+ _jitter) drains and replaces connections on a schedule — useful behind rotating DNS/load balancers. PushX.reconnect/0 remains for manual recovery, and the retry path still reconnects automatically (coalesced) on the first connection error.

config :pushx,
  finch_pool_count: 2,                     # HTTP/2 connections per origin
  finch_http2_ping_interval: 60_000,       # PING after 60 s idle (default)
  finch_http2_max_connection_age: :infinity # or e.g. 30 * 60 * 1000 behind rotating LBs

Retry Behavior

PushX automatically retries transient failures with exponential backoff:

• Connection errors — reconnects the HTTP pool, then retries with 1s base delay (1s, 2s, 4s)
• Server errors (5xx) — 10s base delay (10s, 20s, 40s) per Google's recommendation
• Rate limited (429) — uses retry-after header, or 60s default
• Permanent failures — never retried (invalid token, payload too large, etc.)

Retries block the calling process (Process.sleep backoff) — see "Delivery Semantics". To make exactly one attempt for a specific call, pass retry: :none (works on every send function, including push_batch/4 and named instances) or call the send_once variants:

PushX.push(:apns, token, "Hi", topic: "com.example.app", retry: :none)
PushX.APNS.send_once(token, payload, topic: "com.example.app")
PushX.FCM.send_once(token, payload)

With retry: :none, retryable failures come back immediately with retry_after set when the provider supplied it, so you can requeue on your own schedule.

Timeouts

Tip: Increase timeouts if connecting from distant regions (e.g., EU to Apple's US servers).

Other Options

The complete, authoritative option reference is the PushX.Config moduledoc.

You can also override timeouts per-request:

PushX.APNS.send(token, payload,
  topic: "com.example.app",
  receive_timeout: 30_000,
  pool_timeout: 10_000
)

Rate Limiting

Optional client-side rate limiting prevents exceeding provider limits. Disabled by default.

# Check manually before sending
case PushX.check_rate_limit(:apns) do
  :ok -> # proceed
  {:error, :rate_limited} -> # back off
end

When enabled, rate limits are checked automatically before each send call.

Dynamic Instances (Runtime Config)

For applications that manage push credentials from a database or admin panel, PushX supports starting, stopping, and reconfiguring provider instances at runtime — no application restart needed.

Each instance gets its own HTTP/2 connection pool, JWT cache (APNS), and OAuth process (FCM). Multiple instances can run concurrently (e.g., APNS sandbox + APNS prod + FCM).

Instances live in memory only. They are not persisted across node restarts and are per-VM (not cluster-wide). Start them on boot from your own source of truth with PushX.Instance.Loader — a child you place after your Repo that calls a function of yours returning the instances to start — and again when a tenant is provisioned with PushX.Instance.start/3 (it returns {:error, :already_started} for a running name, so re-running is safe):

children = [MyApp.Repo, {PushX.Instance.Loader, instances: &MyApp.Push.tenant_instances/0}, MyAppWeb.Endpoint]

One tenant's bad credentials are logged and skipped (pass on_error: :raise to fail the boot instead).

Starting Instances

# APNS sandbox (for development/testing)
PushX.Instance.start(:apns_sandbox, :apns,
  key_id: "ABC123",
  team_id: "TEAM456",
  private_key: apns_key_pem,
  mode: :sandbox
)

# APNS production
PushX.Instance.start(:apns_prod, :apns,
  key_id: "ABC123",
  team_id: "TEAM456",
  private_key: apns_key_pem,
  mode: :prod
)

# FCM
PushX.Instance.start(:my_fcm, :fcm,
  project_id: "my-firebase-project",
  credentials: service_account_map
)

The names :apns, :fcm and :webpush are reserved for the static config path and cannot be used as instance names.

Sending via Instances

Pass the instance name instead of :apns or :fcm:

PushX.push(:apns_prod, device_token, "Hello!", topic: "com.example.app")
PushX.push(:my_fcm, device_token, %{title: "Alert", body: "Something happened"})

# Data-only (silent) message via instance
PushX.push_data(:my_fcm, device_token, %{action: "sync", id: 123})

Batch sending works the same way:

PushX.push_batch(:apns_prod, tokens, message, topic: "com.example.app")

Enable / Disable

Disable an instance to reject new pushes while keeping the connection pool warm:

PushX.Instance.disable(:apns_sandbox)
# => PushX.push(:apns_sandbox, ...) returns {:error, %Response{status: :provider_disabled}}

PushX.Instance.enable(:apns_sandbox)
# => pushes work again

Reconfigure

Update config without restarting the application. The old pool is stopped and a new one starts with the merged config:

# Switch environment
PushX.Instance.reconfigure(:apns_sandbox, mode: :prod)

# Rotate credentials
PushX.Instance.reconfigure(:apns_prod,
  key_id: "NEW_KEY_ID",
  private_key: new_pem_string
)

List and Status

PushX.Instance.list()
#=> [
#=>   %{name: :apns_prod, provider: :apns, enabled: true},
#=>   %{name: :apns_sandbox, provider: :apns, enabled: false},
#=>   %{name: :my_fcm, provider: :fcm, enabled: true}
#=> ]

PushX.Instance.status(:apns_prod)
#=> {:ok, %{provider: :apns, enabled: true}}

Stop

PushX.Instance.stop(:apns_sandbox)

Cleans up the Finch pool, JWT cache, Goth process (FCM), and ETS entry.

Example: Database-Backed Admin Panel

defmodule MyApp.PushAdmin do
  @doc "Boot all saved instances on application start."
  def boot do
    MyApp.Repo.all(MyApp.PushConfig)
    |> Enum.each(fn config ->
      PushX.Instance.start(
        String.to_atom(config.name),
        String.to_atom(config.provider),
        build_opts(config)
      )
    end)
  end

  @doc "Called from admin panel when config is updated."
  def update(config) do
    name = String.to_atom(config.name)
    PushX.Instance.reconfigure(name, build_opts(config))
  end

  @doc "Called from admin panel toggle."
  def toggle(name, enabled?) do
    if enabled?,
      do: PushX.Instance.enable(name),
      else: PushX.Instance.disable(name)
  end

  defp build_opts(%{provider: "apns"} = c) do
    [
      key_id: c.key_id,
      team_id: c.team_id,
      private_key: c.private_key,
      mode: String.to_atom(c.mode)
    ]
  end

  defp build_opts(%{provider: "fcm"} = c) do
    [
      project_id: c.project_id,
      credentials: JSON.decode!(c.credentials_json)
    ]
  end
end

Call MyApp.PushAdmin.boot() from your Application.start/2 after PushX starts.

Instance Config Options

Credential Storage

File System (Development)

# config/dev.exs
config :pushx,
  apns_private_key: {:file, "priv/keys/AuthKey.p8"},
  fcm_credentials: {:file, "priv/keys/firebase-service-account.json"}

Add /priv/keys/ to .gitignore.

Environment Variables (Production)

# config/runtime.exs
config :pushx,
  apns_key_id: System.get_env("APNS_KEY_ID"),
  apns_team_id: System.get_env("APNS_TEAM_ID"),
  apns_private_key: System.get_env("APNS_PRIVATE_KEY"),
  apns_mode: if(System.get_env("APNS_SANDBOX") == "true", do: :sandbox, else: :prod),

  fcm_project_id: System.get_env("FCM_PROJECT_ID"),
  fcm_credentials: System.get_env("FCM_CREDENTIALS") |> JSON.decode!()

Tip: For multiline keys (APNS .p8), set the env var directly from the file: export APNS_PRIVATE_KEY="$(cat AuthKey.p8)"

Fly.io Secrets

fly secrets set APNS_KEY_ID="ABC123DEFG"
fly secrets set APNS_TEAM_ID="TEAM123456"
fly secrets set APNS_PRIVATE_KEY="$(cat AuthKey.p8)"
fly secrets set FCM_PROJECT_ID="my-project-id"
fly secrets set FCM_CREDENTIALS="$(cat firebase-service-account.json)"

Then use System.fetch_env!/1 in config/runtime.exs:

if config_env() == :prod do
  config :pushx,
    apns_key_id: System.fetch_env!("APNS_KEY_ID"),
    apns_team_id: System.fetch_env!("APNS_TEAM_ID"),
    apns_private_key: System.fetch_env!("APNS_PRIVATE_KEY"),
    apns_mode: :prod,
    fcm_project_id: System.fetch_env!("FCM_PROJECT_ID"),
    fcm_credentials: System.fetch_env!("FCM_CREDENTIALS") |> JSON.decode!()
end

AWS Secrets Manager / Vault

if config_env() == :prod do
  {:ok, %{"SecretString" => apns_key}} =
    ExAws.SecretsManager.get_secret_value("pushx/apns-key")
    |> ExAws.request()

  config :pushx,
    apns_private_key: apns_key
end

Getting Your Credentials

Apple APNS Setup

You need: Key ID, Team ID, and a Private Key (.p8 file).

Step 1: Get Your Team ID

1. Go to Apple Developer Account
2. Your Team ID is shown in the top-right corner (10 characters)

Step 2: Create an APNS Key

1. Go to Certificates, Identifiers & Profiles
2. Click Keys > + (Create a new key)
3. Enter a name (e.g., "Push Notifications Key")
4. Check Apple Push Notifications service (APNs)
5. Click Continue > Register
6. Download the .p8 file (you can only download it once!)
7. Note the Key ID shown (10 characters)

Google FCM Setup

You need: Project ID and a Service Account JSON file.

Step 1: Create/Open Firebase Project

1. Go to Firebase Console
2. Create a new project or select an existing one
3. Note your Project ID in Project Settings

Step 2: Enable Cloud Messaging API

1. Go to Google Cloud Console
2. Select your Firebase project
3. Go to APIs & Services > Library
4. Search for "Firebase Cloud Messaging API" and Enable it

Step 3: Create Service Account Key

1. In Firebase Console, go to Project Settings (gear icon)
2. Click Service accounts tab
3. Click Generate new private key
4. Save the JSON file securely

Credential Rotation

APNS .p8 keys and FCM service accounts don't expire. You only need to rotate them if you revoke a key or want to follow a rotation policy.

With restart (simplest)

1. Generate new credentials in Apple/Google console
2. Update your secrets (Fly: fly secrets set, AWS: update in Secrets Manager)
3. Redeploy your app
4. Revoke old credentials after all instances are updated

Without restart (static config)

The static config path reads credentials from Application env on each JWT generation, so you can hot-swap them at runtime:

# 1. Update application env with new credentials
Application.put_env(:pushx, :apns_key_id, "NEW_KEY_ID")
Application.put_env(:pushx, :apns_private_key, new_pem_string)

# 2. Clear the cached JWT (otherwise the old token is used for up to 50 min)
:persistent_term.erase(:pushx_apns_jwt_cache)

# 3. Reconnect to discard connections authenticated with the old token
PushX.reconnect()

For FCM, Goth manages OAuth2 tokens automatically. To rotate service account credentials without restart, use the dynamic instance API below.

Without restart (dynamic instances)

If you use PushX.Instance, call reconfigure/2 — it stops the old pool and starts a fresh one with the new credentials:

PushX.Instance.reconfigure(:apns_prod,
  key_id: "NEW_KEY_ID",
  private_key: new_pem_string
)

In-flight requests on the old pool get connection errors, which the retry logic handles automatically.

Telemetry

PushX emits telemetry events for monitoring and metrics:

Tokens are automatically truncated in telemetry metadata for privacy (first 8 + last 4 characters).

Ready-made metrics. With the optional telemetry_metrics dependency, PushX.Telemetry.metrics/0 returns a curated, low-cardinality Telemetry.Metrics list (sends by provider, errors by status, exceptions, latency distributions with provider-tuned buckets, retry attempts and delays — never tagged by token) that drops straight into LiveDashboard, PromEx, or TelemetryMetricsPrometheus:

def metrics, do: MyApp.metrics() ++ PushX.Telemetry.metrics()

Example: Attach a Logger

# In your Application.start/2
:telemetry.attach_many(
  "pushx-logger",
  [
    [:pushx, :push, :stop],
    [:pushx, :push, :error]
  ],
  fn
    [:pushx, :push, :stop], %{duration: d}, %{provider: p}, _ ->
      ms = System.convert_time_unit(d, :native, :millisecond)
      Logger.info("PushX #{p} sent in #{ms}ms")

    [:pushx, :push, :error], _, %{provider: p, status: s, reason: r}, _ ->
      Logger.warning("PushX #{p} failed: #{s} - #{r}")
  end,
  nil
)

Example: With Telemetry.Metrics

defmodule MyApp.Telemetry do
  import Telemetry.Metrics

  def metrics do
    [
      counter("pushx.push.stop.count", tags: [:provider]),
      counter("pushx.push.error.count", tags: [:provider, :status]),
      distribution("pushx.push.stop.duration",
        unit: {:native, :millisecond},
        tags: [:provider]
      )
    ]
  end
end

Circuit Breaker

PushX includes an optional circuit breaker that temporarily blocks requests to a provider after consecutive failures. This prevents wasting resources on dead connections. It is keyed per provider (and per named instance): for :webpush one key covers every push service, so a run of 5xx from one vendor can open it for all browsers.

config :pushx,
  circuit_breaker_enabled: true,
  circuit_breaker_threshold: 5,        # consecutive failures to trip
  circuit_breaker_cooldown_ms: 30_000  # ms before retrying

States:

• Closed — Normal operation, all requests flow through
• Open — Provider is failing, requests are immediately rejected with {:error, %Response{status: :circuit_open}}
• Half-open — After cooldown, one probe request is allowed. Success closes the circuit; failure re-opens it

Only :connection_error and :server_error responses count as failures. Invalid tokens and rate limits do not trip the circuit.

# Check circuit breaker state
PushX.CircuitBreaker.state(:apns)
#=> :closed

# Manual reset
PushX.CircuitBreaker.reset(:apns)

Health Check

Check provider configuration and circuit breaker status:

PushX.health_check()
#=> %{
#=>   apns: %{configured: true, circuit: :closed},
#=>   fcm: %{configured: true, circuit: :closed},
#=>   instances: %{
#=>     tenant_42_apns: %{provider: :apns, enabled: true, circuit: :closed},
#=>     tenant_7_fcm: %{provider: :fcm, enabled: false, circuit: :open}
#=>   }
#=> }

Named instances have their own circuit breakers (keyed by name), so one tenant's outage shows up under :instances without affecting the static providers or other tenants.

Token Cleanup Callback

For Web Push the "token" passed to the callback is the subscription map you sent with — delete it from your store by endpoint.

Automatically clean up invalid tokens from your database when a push fails with :invalid_token, :expired_token, or :unregistered:

config :pushx,
  on_invalid_token: {MyApp.Push, :handle_invalid_token, []}

The callback receives (provider, device_token, ...extra_args) and runs asynchronously:

defmodule MyApp.Push do
  def handle_invalid_token(provider, device_token) do
    MyApp.Tokens.delete_by_token(device_token)
    Logger.info("Removed invalid #{provider} token")
  end
end

Delivery Semantics

PushX delivers at least once, not exactly once. Two mechanisms can produce duplicate deliveries:

• Automatic retries. A send that fails with a connection error may have actually reached the provider (e.g. the response was lost after the request was processed). The retry then delivers the same notification again.
• Batch timeouts.push_batch/4 kills tasks that exceed :timeout. A killed task may have an HTTP request already in flight that the provider still processes and delivers — even though PushX reports that token as {:error, %Response{status: :connection_error, reason: "timeout"}}.

Conversely, an {:ok, ...} result means the provider accepted the message, not that the device displayed it.

If duplicates matter for your use case:

• APNS: set collapse_id (or Message.collapse_key/2) — Apple replaces an undelivered notification carrying the same ID instead of stacking a duplicate.
• FCM: set android: %{"collapse_key" => ...} (or Message.collapse_key/2) for the same effect on Android.
• App level: include your own idempotency key in the payload data and de-duplicate in the app's notification handler.
• Or skip retries (retry: :none per call, or retry_enabled: false globally) and handle failures yourself.

Testing Your App

Set test delivery mode and every send is validated exactly as in production (required :topic, target format, payload size…) but then recorded and answered locally instead of contacting Apple or Google. No credentials needed, no retries, no network:

# config/test.exs
config :pushx, delivery: :test

Web Push targets are checked cryptographically before they are recorded (a real P-256 p256dh point, 16-byte auth, https endpoint) — use PushX.Test.webpush_subscription/1 for a valid fixture instead of hand-writing keys.

defmodule MyApp.OrdersTest do
  use ExUnit.Case, async: true
  import PushX.Test.Assertions

  test "shipping an order notifies the customer" do
    MyApp.Orders.ship(order)

    push = assert_pushed(%{provider: :apns, target: ^device_token})
    assert push.payload["aps"]["alert"]["title"] == "Order shipped"
    assert push.opts[:topic] == "com.example.app"
    refute_pushed(%{provider: :fcm})
  end

  test "dead tokens get cleaned up" do
    PushX.Test.stub(fn
      %{target: "dead-token"} -> {:error, :unregistered}   # what APNS would say
      _push -> :ok
    end)

    MyApp.Notifier.broadcast("Hi")

    refute MyApp.Tokens.exists?("dead-token")              # your :on_invalid_token ran
  end
end

Recorded pushes are scoped to the test process (and processes it spawns, including push_batch/4 workers), so async: true is fine. PushX.Test.pushes/0 gives you the raw list; PushX.Test.stub/1 scripts provider responses per test; PushX.Test.apns_private_key/0 / fcm_credentials/0 provide throwaway keys for starting named instances in tests. See PushX.Test for the details and what test mode does not simulate.

Troubleshooting

mix pushx.doctor

Checks the configuration and credentials offline — the same checks the library runs at start/send time, reported all at once, without sending anything. Fails (non-zero exit) on a credential problem, so it can gate a deploy:

$ MIX_ENV=prod mix pushx.doctor && mix release
PushX 0.14.0 — configuration check (MIX_ENV=prod)

  APNS  ✔ configured (key ABC123DEFG, team TEAM123456, mode :prod)
        ✔ private key resolves and signs ES256 (P-256)
  FCM   ✔ project my-project
        ✔ service-account credentials resolve and sign RS256

  Delivery: live
  Retries:  enabled, 3 attempts, 10000ms base delay — a batch task may block up to 180s
  Pools:    finch_pool_count 2 HTTP/2 connections per origin; PING after 60000ms idle
  Breaker:  off   Rate limit: off
  Cleanup:  on_invalid_token → MyApp.Tokens.delete_by_token/2

All checks passed.

too_many_concurrent_requests Error

[warning] [PushX.APNS] HTTP/2 connection saturated (too_many_concurrent_requests): every stream on the pool's connections is in use ...

PushX logs this distinctly from a network failure. It means every HTTP/2 stream on the pool's connections is busy — typically either a retry burst converging on a freshly reconnected connection (a pool with finch_pool_count: 1 is especially prone) or genuine overload. It is retried with backoff (jittered, so a burst spreads out).

A canonical real-world trace (pushx 0.12, Fly.io, finch_pool_count: 1, 12 minutes idle before a 3-token push) shows the two problems chained — an idle-dead socket, then the retry burst saturating the single fresh connection:

[PushX.APNS] Connection error: %Finch.Error{reason: :timeout}
[PushX.APNS] Connection error: %Finch.Error{reason: :connection_process_went_down}
[PushX.Retry] Attempt 1/3 failed for apns (connection_error), retrying in 960ms
[PushX] Reconnected HTTP pools (stale connections discarded)
[PushX.APNS] Connection error: %Finch.HTTPError{reason: :too_many_concurrent_requests, module: Mint.HTTP2}
[PushX.Retry] Attempt 2/3 failed for apns (connection_error), retrying in 2007ms
[Push] Sent to ios x3

Delivered, but ~2 s late and two attempts burned. With finch_pool_count: 2+ and the default PING keepalive, neither line appears: the idle socket is kept alive (or detected dead before the send), and a retry burst has several connections to land on.

Fix: raise finch_pool_count (HTTP/2 connections per origin; finch_pool_size does nothing for HTTP/2) and/or lower batch :concurrency; enable rate limiting for sustained volume:

config :pushx,
  finch_pool_count: 4,
  rate_limit_enabled: true,
  rate_limit_apns: 2000,
  rate_limit_fcm: 2000

Stale connections after idle periods

On cloud infrastructure (Fly.io, AWS, GCP) idle HTTP/2 connections are silently dropped; the first send after a quiet period then fails with a connection error and is recovered by the retry path (which reconnects the pool) — a second or two late. Prevent it instead of recovering from it: HTTP/2 PING keepalive is on by default (finch_http2_ping_interval: 60_000, finch ≥ 0.22 — older finch ignores it with a boot warning); lower it if your platform drops faster, or add finch_http2_max_connection_age to recycle connections proactively. Manual: PushX.reconnect().

request_timeout Error

[error] [PushX.APNS] Connection error: %Finch.Error{reason: :request_timeout}

1. Increase timeouts if connecting from distant regions (e.g., EU to US):

   config :pushx,
     receive_timeout: 30_000,
     connect_timeout: 20_000
   

2. PushX automatically retries connection errors with exponential backoff (1s, 2s, 4s)

3. If this follows a too_many_concurrent_requests error, see the stale connections fix above

Debugging Tips

Enable telemetry logging to monitor push performance:

:telemetry.attach("pushx-debug", [:pushx, :push, :error], fn _, _, meta, _ ->
  Logger.warning("Push failed: #{meta.provider} - #{meta.status} - #{meta.reason}")
end, nil)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

See CONTRIBUTING.md for the repo layout, test commands and the release process, and ROADMAP.md for what is planned before and after 1.0 (Expo design note in docs/design/).

License

MIT License. See LICENSE for details.

Built with care by Cigno Systems AB