katipo

An HTTP/1.1, HTTP/2 and HTTP/3 client for Erlang, built on libcurl's multi interface and libevent: libcurl's protocol support, many concurrent connections per worker.

Install

{deps, [{katipo, "2.0.1"}]}.

Building needs a C compiler, make, and the libcurl and libevent headers:

apt-get install gcc make libcurl4-openssl-dev libevent-dev   # Debian/Ubuntu
brew install curl libevent pkg-config                        # macOS

On macOS the build prefers Homebrew's curl, which has HTTP/3.

Quick start

{ok, _} = application:ensure_all_started(katipo),
{ok, _} = katipo_pool:start(api, 2, [{pipelining, multiplex}]),

{ok, #{status := 200, body := Body}} = katipo:get(api, <<"https://example.com">>),

{ok, #{status := 200}} =
    katipo:post(api, <<"https://httpbin.org/post">>,
                #{headers => [{<<"content-type">>, <<"application/json">>}],
                  body => <<"{\"name\":\"katipo\"}">>,
                  timeout_ms => 5000}).

Every method has a function (get, post, put, head, options, patch, delete) taking a pool, a URL and an optional options map. katipo:req(Pool, #{url => Url, method => post, ...}) takes the whole request as one map.

Responses

{ok, #{status := pos_integer(), headers := [{binary(), binary()}],
       cookiejar := cookiejar(), body := binary()}}
{error, #{code := atom(), message := binary()}}

code is a curl error name (couldnt_connect, operation_timedout, ...) or one of katipo's: bad_opts, worker_died, overload, admission_timeout, await_timeout.

Async requests

{ok, Ref} = katipo:async_get(api, Url),
{ok, #{status := 200}} = katipo:await(Ref, 5000).

The result arrives as one message, {katipo_response, Ref, Response} or {katipo_error, Ref, Error}, sent to the caller or to the reply_to pid. await/1,2 receives it for you; await_timeout only ends the wait, so a later await can still collect it. cancel(Pool, Ref) aborts the request; a message already sent before the cancel took effect may still be in the mailbox.

An admitted request always gets exactly one message, including worker_died if its worker's C port dies. If no worker can take the request, async_* returns the error itself (worker_died, overload or admission_timeout) and sends nothing. The one gap: a worker killed with exit(Pid, kill) sends nothing, so keep a timeout on the wait.

Streaming

stream => true delivers the body as it arrives instead of buffering it. stream_window => N pauses the transfer after N unacknowledged chunks, which pushes back on the server, until update_flow/3 grants more:

download(Url) ->
    {ok, Ref} = katipo:async_get(api, Url, #{stream => true, stream_window => 8}),
    receive {katipo_headers, Ref, #{status := 200}} -> ok end,
    loop(Ref, []).

loop(Ref, Acc) ->
    receive
        {katipo_chunk, Ref, Bin} ->
            ok = katipo:update_flow(api, Ref, 1),
            loop(Ref, [Acc | Bin]);
        {katipo_done, Ref, _} -> {ok, iolist_to_binary(Acc)};
        {katipo_error, Ref, Error} -> {error, Error}
    end.

A katipo_error can arrive at any point, including mid-body. Grant at least once every N chunks, or the transfer stays paused until timeout_ms fires. Streaming is async-only and await does not apply. With HTTP/2 or HTTP/3 multiplexing, use a recent libcurl: older versions can stall other streams on the connection while one is paused.

Pools

katipo_pool:start(Name, Size, Opts) starts Size workers, each owning one curl multi handle. katipo_pool:stop(Name) stops them.

Option	Type	Default	Note
`pipelining`	`nothing` `http1` `multiplex`	`nothing`	HTTP pipelining CURLMOPT_PIPELINING
`max_total_connections`	`non_neg_integer()`	0 (no limit)	docs
`max_concurrent_streams`	`non_neg_integer()`	100	docs curl >= 7.67.0
`max_in_flight`	`pos_integer()` \| `infinity`	`infinity`	katipo's own: per-worker cap on in-flight requests

With max_in_flight, a request that finds its worker full spills over to another, and fails fast with overload when every worker is full.

Request options

Option	Type	Default	Notes
`headers`	`[{binary(), iodata()}]`	`[]`
`cookiejar`	opaque (returned in response)	`[]`
`body`	`iodata()`	`<<>>`
`connecttimeout_ms`	`pos_integer()`	30000	docs
`followlocation`	`boolean()`	`false`	docs
`ssl_verifyhost`	`boolean()`	`true`	docs
`ssl_verifypeer`	`boolean()`	`true`	docs
`capath`	`binary()`	`undefined`	docs
`cacert`	`binary()`	`undefined`	docs
`ca_cache_timeout`	`integer()`	86400	docs curl >= 7.87.0 (0=disable, -1=forever)
`timeout_ms`	`pos_integer()`	30000	docs
`dns_cache_timeout`	`integer()`	60	docs (0=disable, -1=forever)
`maxredirs`	`non_neg_integer()`	9	docs
`proxy`	`binary()`	`undefined`	docs
`tcp_fastopen`	`boolean()`	`false`	docs curl >= 7.49.0
`pipewait`	`boolean()`	`true`	docs curl >= 7.43.0
`interface`	`binary()`	`undefined`	docs
`unix_socket_path`	`binary()`	`undefined`	docs curl >= 7.40.0
`doh_url`	`binary()`	`undefined`	docs curl >= 7.62.0
`http_version`	`curl_http_version_none` `curl_http_version_1_0` `curl_http_version_1_1` `curl_http_version_2_0` `curl_http_version_2tls` `curl_http_version_2_prior_knowledge` `curl_http_version_3`	`curl_http_version_none`	docs HTTP/3 requires curl >= 7.66.0
`sslversion`	`sslversion_default` `sslversion_tlsv1` `sslversion_tlsv1_0` `sslversion_tlsv1_1` `sslversion_tlsv1_2` `sslversion_tlsv1_3`	`sslversion_default`	docs
`sslcert`	`binary()`	`undefined`	docs
`sslkey`	`binary()`	`undefined`	docs
`sslkey_blob`	`binary()` (DER format)	`undefined`	docs curl >= 7.71.0
`keypasswd`	`binary()`	`undefined`	docs
`http_auth`	`basic` `digest` `ntlm` `negotiate`	`undefined`	docs
`stream`	`boolean()`	`false`	async only; deliver the body incrementally (see Streaming responses)
`stream_window`	`pos_integer()` \| `infinity`	`infinity`	chunk credits before the transfer pauses awaiting `update_flow/3`
`username`	`binary()`	`undefined`	docs
`password`	`binary()`	`undefined`	docs
`userpwd`	`binary()`	`undefined`	docs
`verbose`	`boolean()`	`false`	docs
`reply_to`	`pid()`	`self()`	Async only (`async_*`): process that receives the response message. Accepted but ignored on synchronous requests.

Some options depend on how libcurl was built. Check at runtime with katipo:http3_available/0, doh_url_available/0, tcp_fastopen_available/0, unix_socket_path_available/0 and sslkey_blob_available/0.

Observability

katipo emits OpenTelemetry spans and metrics. Without an SDK they are no-ops. Each request gets a span:

Attribute	Description
`http.request.method`	HTTP method (GET, POST, etc.)
`url.full`	Request URL (query string, fragment and userinfo are stripped for security)
`server.address`	Target host
`http.response.status_code`	Response status code (on success)

Metrics (histograms are in milliseconds and carry http.request.method):

Metric	Type	Description
`http.client.requests`	Counter	Number of HTTP requests (with `result` and `http.response.status_code` attributes)
`http.client.duration`	Histogram	Total request duration (ms)
`http.client.curl_time`	Histogram	Curl total time (ms)
`http.client.namelookup_time`	Histogram	DNS lookup time (ms)
`http.client.connect_time`	Histogram	Connection time (ms)
`http.client.appconnect_time`	Histogram	SSL/TLS handshake time (ms)
`http.client.pretransfer_time`	Histogram	Pre-transfer time (ms)
`http.client.redirect_time`	Histogram	Redirect processing time (ms)
`http.client.starttransfer_time`	Histogram	Time to first byte (ms)

To export them, add the SDK and an exporter:

%% In rebar.config
{deps, [
    {opentelemetry, "1.5.0"},
    {opentelemetry_experimental, "0.5.1"},
    {opentelemetry_exporter, "1.8.0"}
]}.

Configure the exporter in your sys.config:

[
 {opentelemetry, [
   {span_processor, batch},
   {traces_exporter, otlp}
 ]},
 {opentelemetry_experimental, [
   {readers, [
     #{module => otel_metric_reader,
       config => #{exporter => {opentelemetry_exporter, #{}}}}
   ]}
 ]},
 {opentelemetry_exporter, [
   {otlp_endpoint, "http://localhost:4318"}
 ]}
].

Upgrading from 1.x

Metrics moved to OpenTelemetry: mod_metrics, the return_metrics option and the metrics field in responses are gone, and the histograms above carry the same timings. Everything else is additive; see the CHANGELOG.

Development

The tests run against httpbin behind Caddy (HTTPS, HTTP/2 and HTTP/3 on port 8443):

(cd test/http3-httpbin && docker compose up -d)
curl -k https://localhost:8443/get   # wait for a 200; a 502 means httpbin is down
rebar3 ct

The async admission and streaming flow-control protocols are modelled in TLA+ and proved in Lean under formal/, and formal/trace.sh checks the test runs against the Lean model.