dream_http_client

Type-safe HTTP client for Gleam with recording + streaming support.

A standalone HTTP/HTTPS client built on Erlang's battle-tested httpc. Supports blocking requests, yielder streaming, and process-based streaming via callbacks. Built with the same quality standards as Dream, but completely independent—use it in any Gleam project.

Contents

• Why dream_http_client?
• Installation
• Quick Start
• Execution Modes
• Recording & Playback
• API Reference
• Examples

Why dream_http_client?

Installation

gleam add dream_http_client

Quick Start

Make a simple HTTP request:

import dream_http_client/client.{
  type HttpResponse, type SendError, host, method, path, port, scheme, send,
}
import gleam/http

pub fn simple_get() -> Result(HttpResponse, SendError) {
  client.new()
  |> method(http.Get)
  |> scheme(http.Http)
  |> host("localhost")
  |> port(9876)
  |> path("/text")
  |> send()
}

Execution Modes

dream_http_client provides three execution modes. Choose based on your use case:

1. Blocking - send()

Best for: JSON APIs, small responses

import dream_http_client/client.{
  HttpResponse, RequestError, ResponseError,
  host, path, port, scheme, send,
}
import gleam/http

let result =
  client.new()
  |> scheme(http.Http)
  |> host("localhost")
  |> port(9876)
  |> path("/text")
  |> send()

case result {
  Ok(HttpResponse(body: body, ..)) -> Ok(body)
  Error(ResponseError(response: response)) -> Error(response.body)
  Error(RequestError(message: msg)) -> Error(msg)
}

2. Yielder Streaming - stream_yielder()

Best for: AI/LLM streaming, file downloads, sequential processing

import dream_http_client/client.{host, path, port, scheme, stream_yielder}
import gleam/bytes_tree
import gleam/http
import gleam/yielder

let total_bytes =
  client.new()
  |> scheme(http.Http)
  |> host("localhost")
  |> port(9876)
  |> path("/stream/fast")
  |> stream_yielder()
  |> yielder.fold(0, fn(total, chunk_result) {
    case chunk_result {
      Ok(chunk) -> total + bytes_tree.byte_size(chunk)
      Error(_) -> total
    }
  })

⚠️ Note: This blocks while waiting for chunks. Not suitable for OTP actors handling concurrent operations.

3. Process-Based Streaming - start_stream()

Best for: Background tasks, concurrent operations, cancellable streams

import dream_http_client/client.{
  await_stream, host, on_stream_chunk, on_stream_end, on_stream_error,
  on_stream_start, path, port, scheme, start_stream,
}
import gleam/bit_array
import gleam/http
import gleam/io

pub fn stream_and_print() -> Result(Nil, String) {
  let stream_result =
    client.new()
    |> scheme(http.Http)
    |> host("localhost")
    |> port(9876)
    |> path("/stream/fast")
    |> on_stream_start(fn(_headers) { io.println("Stream started") })
    |> on_stream_chunk(fn(data) {
      case bit_array.to_string(data) {
        Ok(text) -> io.print(text)
        Error(_) -> io.print("<binary>")
      }
    })
    |> on_stream_end(fn(_headers) { io.println("\nStream completed") })
    |> on_stream_error(fn(reason) {
      io.println_error("Stream error: " <> reason)
    })
    |> start_stream()

  case stream_result {
    Error(reason) -> Error(reason)
    Ok(stream_handle) -> {
      await_stream(stream_handle)
      Ok(Nil)
    }
  }
}

Choosing a Mode

Recording & Playback

Record HTTP requests/responses for testing, debugging, and offline development. All three execution modes fully support both recording and playback:

Streaming recordings capture each chunk along with timing information. The same fixture format is shared between stream_yielder() and start_stream(), so recordings made with one can be played back by either.

Quick Example

import dream_http_client/recorder.{directory, mode, start}
import dream_http_client/client.{host, path, port, recorder as with_recorder, scheme, send}
import gleam/http

// Record real requests
let assert Ok(rec) =
  recorder.new()
  |> directory("mocks/api")
  |> mode("record")
  |> start()

client.new()
  |> scheme(http.Http)
  |> host("localhost")
  |> port(9876)
  |> path("/text")
  |> with_recorder(rec)
  |> send()  // Saved immediately to disk

// Playback later (no network)
let assert Ok(playback) =
  recorder.new()
  |> directory("mocks/api")
  |> mode("playback")
  |> start()

client.new()
  |> scheme(http.Http)
  |> host("localhost")
  |> port(9876)
  |> path("/text")
  |> with_recorder(playback)
  |> send()  // Returns recorded response

Recording Modes

• "record" - Records real requests to disk immediately
• "playback" - Returns recorded responses (no network)
• "passthrough" - No recording/playback

Important: Recordings are saved immediately when captured. recorder.stop() is optional and only performs cleanup. This ensures recordings are never lost even if the process crashes.

Use Cases

Testing:

// test/api_test.gleam
import dream_http_client/recorder.{directory, mode, start}

let assert Ok(rec) =
  recorder.new()
  |> directory("test/fixtures/api")
  |> mode("playback")
  |> start()

// Tests run without external dependencies

Offline Development: Record API responses once, then work offline using recorded responses.

Debugging Production: Record problematic request/response pairs for investigation.

Request Matching

import dream_http_client/matching
import dream_http_client/recorder.{directory, key, mode, start}

// Build a request key function from include/exclude flags
let request_key_fn = matching.request_key(
  method: True,
  url: True,
  headers: False,  // Ignore auth tokens, timestamps
  body: False,     // Ignore request IDs in body
)

let assert Ok(rec) =
  recorder.new()
  |> directory("mocks/api")
  |> mode("playback")
  |> key(request_key_fn)
  |> start()

Scrubbing Secrets (Transformers)

If your requests contain secrets (like Authorization headers) or volatile fields (timestamps, request IDs), you can attach a transformer to normalize requests before the key is computed and before anything is persisted.

import dream_http_client/matching
import dream_http_client/recorder.{
  directory, key, mode, request_transformer, start,
}
import dream_http_client/recording
import gleam/list

let request_key_fn =
  matching.request_key(method: True, url: True, headers: True, body: True)

fn scrub_auth_and_body(
  request: recording.RecordedRequest,
) -> recording.RecordedRequest {
  fn is_not_authorization_header(header: #(String, String)) -> Bool {
    header.0 != "Authorization"
  }

  let recording.RecordedRequest(
    method,
    scheme,
    host,
    port,
    path,
    query,
    headers,
    _body,
  ) = request

  let scrubbed_headers =
    list.filter(headers, is_not_authorization_header)

  recording.RecordedRequest(
    method: method,
    scheme: scheme,
    host: host,
    port: port,
    path: path,
    query: query,
    headers: scrubbed_headers,
    body: "",
  )
}

let assert Ok(rec) =
  recorder.new()
  |> directory("mocks/api")
  |> mode("record")
  |> key(request_key_fn)
  |> request_transformer(scrub_auth_and_body)
  |> start()

// ... requests recorded via this recorder will have secrets scrubbed ...

If you need to scrub responses (cookies, tokens, PII) before fixtures are written to disk, use a response transformer. This runs only in record mode.

import dream_http_client/recorder.{directory, mode, response_transformer, start}
import dream_http_client/recording

fn scrub_response(
  _request: recording.RecordedRequest,
  response: recording.RecordedResponse,
) -> recording.RecordedResponse {
  // Implementation omitted here (see tested snippet)
  response
}

let assert Ok(rec) =
  recorder.new()
  |> directory("mocks/api")
  |> mode("record")
  |> response_transformer(scrub_response)
  |> start()

Ambiguous Matches (Key Collisions)

Playback errors if more than one recording matches the same request key. This is intentional: it forces you to refine your key function (or add a transformer) so each request maps to exactly one recording.

import dream_http_client/matching
import dream_http_client/recorder.{directory, key, mode, start}

let request_key_fn =
  matching.request_key(method: True, url: True, headers: False, body: False)

let assert Ok(playback) =
  recorder.new()
  |> directory("mocks/api")
  |> mode("playback")
  |> key(request_key_fn)
  |> start()

// ... lookup will return Error("Ambiguous recording match ...") if multiple match ...

Recording Storage

Recordings are stored as individual files (one per request) with human-readable filenames:

• Filename format: {method}_{host}_{path}_{key_hash}_{content_hash}.json
• key_hash groups recordings by request key
• content_hash prevents overwrites when multiple recordings share the same key

mocks/api/GET_localhost__text_a3f5b2_19d0a1.json
mocks/api/POST_localhost__text_c7d8e9_4f22bc.json

Benefits:

• O(1) write performance - No read-modify-write cycles
• Atomic fixture writes - Readers never see partial recordings
• Concurrent tests work - No file contention between parallel tests
• Easy inspection - Each recording is a separate, readable file
• Version control friendly - Individual files show clear diffs

API Reference

Builder Pattern

import dream_http_client/client.{
  add_header, body, host, method, path, port, query, scheme, send, timeout,
}
import gleam/http

let json_body = "{\"hello\":\"world\"}"

client.new()
|> method(http.Post)         // HTTP method
|> scheme(http.Http)         // HTTP or HTTPS
|> host("localhost")         // Hostname (required)
|> port(9876)                // Port (optional, defaults 80/443)
|> path("/post")             // Request path
|> query("page=1&limit=10")  // Query string
|> add_header("Content-Type", "application/json")
|> body(json_body)           // Request body
|> timeout(60_000)           // Timeout in ms (default: 30s)
|> send()

Execution

Blocking:

• send(req) -> Result(HttpResponse, SendError) - Returns complete response (status, headers, body)

Yielder Streaming:

• stream_yielder(req) -> Yielder(Result(BytesTree, String)) - Returns yielder producing chunks

Process-Based Streaming:

• start_stream(req) -> Result(StreamHandle, String) - Starts stream, returns handle
• await_stream(handle) -> Nil - Wait for completion (optional)
• cancel_stream_handle(handle) -> Nil - Cancel running stream

Types

Header - HTTP header with name: String and value: String

HttpResponse - Complete HTTP response with status: Int, headers: List(Header), and body: String

SendError - Error from send():

• ResponseError(response: HttpResponse) — HTTP error (4xx/5xx) with full response
• RequestError(message: String) — Transport failure (connection, timeout, DNS)

StreamHandle - Opaque identifier for process-based streams

Error Handling

All modes use Result types for explicit error handling:

import dream_http_client/client.{
  HttpResponse, RequestError, ResponseError,
  host, path, port, scheme, send, timeout,
}
import gleam/http
import gleam/io

let request =
  client.new()
  |> scheme(http.Http)
  |> host("localhost")
  |> port(9876)
  |> path("/text")
  |> timeout(5000)

case send(request) {
  Ok(HttpResponse(body: body, ..)) -> {
    io.println(body)
    Ok(body)
  }
  Error(ResponseError(response: response)) -> {
    io.println_error("HTTP error " <> int.to_string(response.status))
    Error(response.body)
  }
  Error(RequestError(message: msg)) -> {
    io.println_error("Request failed: " <> msg)
    Error(msg)
  }
}

Examples

All examples are tested and verified. See test/snippets/ for complete, runnable code.

Basic requests:

• Blocking request - Simple GET
• POST with JSON - JSON body
• Request builder - Full configuration
• Timeout configuration - Custom timeouts

Streaming:

• Yielder streaming - Sequential processing
• Process-based streaming - Callback-driven streaming
• Stream cancellation - Cancel via StreamHandle

Recording:

• Record and playback - Testing without network
• Playback-only testing - Test fixtures without network
• Recording with stream_yielder() - Record and playback yielder streams
• Recording with start_stream() - Record and playback callback streams
• Custom request keys - Configure request matching
• Request transformers - Scrub secrets before keying/persistence
• Response transformers - Scrub secrets from recorded responses
• Ambiguous match errors - Key collision behavior

Design Principles

This module follows the same quality standards as Dream:

• No nested cases - Clear, flat control flow throughout
• Prefer named functions - Use named functions when it improves readability
• Builder pattern - Consistent, composable request configuration
• Type safety - Result types force error handling at compile time
• OTP-first design - Process-based streaming designed for supervision trees
• Comprehensive testing - Unit tests (no network) + integration tests (real HTTP)
• Battle-tested foundation - Built on Erlang's production-proven httpc

About Dream

This module was originally built for the Dream web toolkit, but it's completely standalone and can be used in any Gleam project. It follows Dream's design principles and will be maintained as part of the Dream ecosystem.

License

MIT — see LICENSE.md