livery_s3

An S3-compatible object storage client for Erlang, built on the livery HTTP client. Works with AWS S3 and S3-compatible stores (Garage, MinIO, Ceph, Wasabi, …). Signs every request with AWS Signature V4.

Features

Object CRUD: put_object, get_object, head_object, delete_object, server-side copy_object
User + system metadata (x-amz-meta-*, content-type, cache-control, …)
Byte ranges and streaming up/downloads
Conditional requests (if_match/if_none_match/if_modified_since), Content-MD5, and GET/presign response-header overrides
Bucket management: list_buckets, create_bucket, delete_bucket, head_bucket, get_bucket_location, list_objects (V2, with pagination via list_objects_all)
Versioning (where the backend supports it): get_bucket_versioning, put_bucket_versioning, list_object_versions, versioned get/delete
Multipart upload (create/upload_part/complete/abort, upload_part_copy, list_parts, list_multipart_uploads)
Batch delete and presigned URLs (presign)
Resilience via livery_client layers: retries (on by default), circuit breaker, concurrency cap, multi-endpoint balancing
Credential providers: static, env, shared config file, EC2/ECS instance metadata (IMDS), web-identity/STS, and a default chain, with refresh of temporary credentials
Path-style (default) and virtual-hosted addressing; AWS SigV4 signing with session-token support

See docs/features.md for the full reference.

Usage

C = livery_s3:new(#{
    endpoint => <<"https://s3.eu-west-1.amazonaws.com">>,  % or http://127.0.0.1:3900
    region   => <<"eu-west-1">>,
    access_key_id     => <<"AKIA...">>,
    secret_access_key => <<"...">>
    %% addressing => path | virtual   (default path)
    %% session_token => <<"...">>     (temporary credentials)
    %% timeout => 30000               (per-request, ms)
}),

ok = livery_s3:create_bucket(C, <<"photos">>),

{ok, _} = livery_s3:put_object(C, <<"photos">>, <<"cat.jpg">>, Bytes,
                               #{content_type => <<"image/jpeg">>,
                                 metadata => #{<<"album">> => <<"holiday">>}}),

{ok, #{body := Bytes, metadata := #{<<"album">> := <<"holiday">>}}} =
    livery_s3:get_object(C, <<"photos">>, <<"cat.jpg">>),

%% Range request
{ok, #{body := First1k}} =
    livery_s3:get_object(C, <<"photos">>, <<"cat.jpg">>, #{range => {0, 1023}}),

%% Streaming download
{ok, #{body := {stream, Reader}}} =
    livery_s3:get_object(C, <<"photos">>, <<"cat.jpg">>, #{stream => true}),
{ok, All} = livery_client:read_body(Reader),

%% Presigned URL
{ok, Url} = livery_s3:presign(C, get, <<"photos">>, <<"cat.jpg">>, 3600).

Every call returns {ok, _} / ok or {error, Reason}. S3 error bodies surface as {error, {s3, Code, Message, #{status => S, request_id => RId}}}; a missing object/bucket on a HEAD is {error, not_found}.

Resilience

Retries are on by default (transient 5xx + connection errors, idempotent ops, exponential backoff). Circuit breaking, a concurrency cap, and multi-endpoint balancing are opt-in:

C = livery_s3:new(#{
    endpoint => <<"https://s3.eu-west-1.amazonaws.com">>,
    region   => <<"eu-west-1">>,
    access_key_id => <<"AKIA...">>, secret_access_key => <<"...">>,
    retry           => #{max => 5},   % or false to disable
    circuit_breaker => true,          % needs the livery app started
    concurrency     => 50
}).

Streamed uploads and non-idempotent POST operations are never retried. See docs/features.md for ordering and caveats.

Credentials

Static keys, or a provider that sources them without a hardcoded secret:

livery_s3:new(#{endpoint => E, region => R, credentials => env}).        %% AWS_* env vars
livery_s3:new(#{endpoint => E, region => R, credentials => {file, <<"default">>}}).
livery_s3:new(#{endpoint => E, region => R, credentials => imds}).       %% EC2/ECS role (refreshed)
livery_s3:new(#{endpoint => E, region => R, credentials => default}).    %% env -> web-identity -> file -> imds

Refreshing providers (imds, web_identity) cache and rotate temporary credentials and need the livery_s3 application started. See docs/features.md.

Compatibility

addressing => path (the default) keeps the bucket in the URL path, which every S3-compatible store accepts. Use virtual for AWS-native bucket.host addressing. Features the backend does not implement (e.g. versioning on Garage) return a clean {error, {s3, <<"NotImplemented">>, _, _}} rather than crashing.

Testing

Offline unit tests (SigV4 against AWS's published worked examples, URI encoding, XML parsing, and request/response round-trips through a fake adapter):

rebar3 eunit

Integration tests run against a real Garage in Docker:

make test            # garage-up -> rebar3 ct -> garage-down

or manually:

./test/docker/garage-up.sh
rebar3 ct --suite test/livery_s3_garage_SUITE
./test/docker/garage-down.sh

The suite skips itself if no S3 endpoint is reachable. Override the target with LIVERY_S3_ENDPOINT, LIVERY_S3_REGION, LIVERY_S3_ACCESS_KEY, LIVERY_S3_SECRET_KEY, LIVERY_S3_BUCKET.

Full offline gate (compile, xref, dialyzer, lint, fmt, eunit):

make check

Documentation

API docs are generated with ex_doc; docs/features.md lists every capability and the function behind it.

rebar3 ex_doc      # writes HTML to doc/