Fetch

A small HTTP/1.1 client written in plain Elixir/OTP.

Why it exists

I've always wanted to write my own HTTP client, so here it is. This is purely for fun: I want to walk the whole path myself and understand every step.

URL → DNS → TCP → TLS → HTTP/1.1 request → HTTP/1.1 response → parser → response

It's not trying to replace Req, Finch or Mint. If I have the time, it may grow into a proper client someday. For now it's a hobby project.

The request encoding, the response parser and body framing are written by hand. Sockets and TLS come from OTP (:gen_tcp, :ssl). No runtime dependencies.

Design notes, rules and roadmap live in AGENTS.md.

Installation

def deps do
  [
    {:fetch, "~> 0.1"}
  ]
end

Requires Elixir 1.15+ and OTP 25+.

Basic usage

{:ok, response} = Fetch.get("https://www.erlang.org/")

response.status
#=> 200

Request

Fetch.request(:get, "http://localhost:4000/users?page=2",
  headers: [{"accept", "application/json"}]
)

Fetch.post("http://localhost:4000/users",
  headers: [{"content-type", "application/json"}],
  body: ~s({"name":"Jon Snow"})
)

Helpers: get/2, head/2, post/2, put/2, patch/2, delete/2. request/3 also accepts :options.

Bytes sent for the POST above:

POST /users HTTP/1.1
host: localhost:4000
user-agent: fetch/0.1.0
content-type: application/json
content-length: 19
connection: close

{"name":"Jon Snow"}

host, content-length, transfer-encoding and connection are managed by the client. Header names and values are validated, so a value with \r\n returns {:error, {:request, {:invalid_header_value, name}}} instead of injecting headers.

Response

%Fetch.Response{
  status: 200,
  headers: [{"content-type", "text/html"}, {"set-cookie", "a=1"}, {"set-cookie", "b=2"}],
  body: "<!doctype html>..."
}

Fetch.Response.get_header(response, "Set-Cookie")
#=> ["a=1", "b=2"]

Header names are lowercased; order and duplicates are kept. Any status code, including 4xx and 5xx, is {:ok, response}.

The body is read whole, whether it is framed by content-length, transfer-encoding: chunked or by the server closing the connection.

Errors

{:error, {:url, {:unsupported_scheme, "ftp"}}}
{:error, {:dns, :nxdomain}}
{:error, {:connect, :econnrefused}}
{:error, {:tls, {:tls_alert, {:unknown_ca, _}}}}
{:error, {:recv, :timeout}}
{:error, {:parse, {:invalid_header, "Content-Length : 5"}}}
{:error, {:redirect, :too_many_redirects}}

Stages: :url, :request, :dns, :connect, :tls, :send, :recv, :parse, :redirect. Unknown options and unsupported methods raise ArgumentError.

HTTPS

TLS is OTP :ssl on top of the connected TCP socket, with:

• verify: :verify_peer against the OS trust store (:public_key.cacerts_get/0)
• SNI and hostname verification (wildcard certificates supported)

Extra :ssl options are merged over these defaults, e.g. a private CA:

Fetch.get("https://internal.test/", ssl: [cacertfile: "priv/ca.pem"])

Redirects

301, 302, 303, 307 and 308 are followed by default, at most 10 times:

Fetch.get("http://www.erlang.org/")          # 301 → https://www.erlang.org/
Fetch.get(url, follow_redirects: false)      # returns the 3xx response itself
Fetch.get(url, max_redirects: 3)             # {:error, {:redirect, :too_many_redirects}}

A relative location is resolved against the current URL. When a redirect goes to another origin (scheme, host or port), authorization, proxy-authorization and cookie headers are not sent there.

Keep-alive

Fetch.get/2 and friends use a new connection for every request. Fetch.Conn keeps one connection open and sends requests over it one after another:

{:ok, conn} = Fetch.Conn.new("http://localhost:4000")

{:ok, conn, users} = Fetch.Conn.request(conn, :get, "/users")
{:ok, conn, created} = Fetch.Conn.request(conn, :post, "/users", body: ~s({"name":"Jon Snow"}))

Fetch.Conn.close(conn)

A connection is a value, not a process: every request returns the updated connection, keep using that one. Errors are {:error, conn, {stage, reason}}.

• new/2 does no IO; the first request connects.
• The connection is closed after a response with connection: close, an HTTP/1.0 response, a body delimited by close, or any error. The next request connects again.
• If the server closed the idle connection, the next request notices before sending anything and reconnects.
• If the connection breaks after a request was sent, you get the error. There is no automatic retry: the server may already have handled the request.
• keep_alive: false sends connection: close for the last request.
• No redirects, no pooling, no pipelining. Use a connection from the process that created it.

Timeouts

There is no total request deadline: a server that keeps sending a byte now and then is not stopped by :receive_timeout.

Limitations

• Fetch.get/2 and friends open a new connection per request; reuse needs Fetch.Conn. No pooling, no pipelining, no retries.
• The whole body is kept in memory. No streaming.
• chunked is the only transfer coding. gzip, chunked and friends return {:error, {:parse, {:unsupported_transfer_encoding, value}}}.
• Trailer fields of chunked responses are validated and dropped.
• No compression, cookies, proxies, retries.
• Every redirect opens a new connection. The response does not say which URL it finally came from.
• HTTP/1.1 only.

Roadmap

1. Streaming responses
2. Optional: gzip/deflate, benchmarks

Why not Req/Finch?

For real work, use them. Req is the high-level client, Finch adds pooling, and Mint is the process-less protocol layer they are built on. Fetch exists to learn what those libraries do and why.

Try it

mix run examples/get.exs https://www.erlang.org/
mix run examples/keep_alive.exs https://www.erlang.org/ 10
mix test                      # local servers only
mix test --include external   # also talks to the internet