Fil

CI License Version Hex Docs

Run in Livebook

Fil is a pluggable file storage abstraction for Elixir.

Mix.install([
  {:fil, "~> 0.2"}
])

disk = Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")

disk
|> Fil.write!("hello.txt", "World")
|> Fil.read!()
#=> "World"

With adapter: Fil.Adapter.S3, a bucket and credentials, the same code writes to S3.

To try Fil without a project, run the tour in Livebook.

Table of Contents

Note

Fil is in early development, and the API may still change a lot.

Features and goals

Concepts

Fil uses the function names of Elixir's File module (read, write, stream, stat, ls, cp, rename, rm, rm_rf), but on every adapter they behave like an object store:

The contract in Fil.Adapter lists every difference from File. Every function that can fail returns {:ok, result} or {:error, error}, with an error from the Errors section below.

Disks

A disk says where files are stored: an adapter and its options. It's a plain value, so there's no application config and nothing to add to your supervision tree (Fil starts one process of its own, for the Memory adapter). Every function takes a disk as its first argument and a path relative to the disk's root:

disk = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")

{:ok, _} = Fil.write(disk, "hello.txt", "World")

Fil.read(disk, "hello.txt")
#=> {:ok, "World"}

Refs

A Fil.Ref is a single value for a file on a disk, built with Fil.ref/2. Every function that takes a disk and a path as two arguments also takes a ref as one argument in their place:

hello = Fil.ref(disk, "hello.txt")
#=> #Fil.Ref<local:hello.txt>

{:ok, _} = Fil.write(hello, "World")

Fil.read(hello)
#=> {:ok, "World"}

Actions on files return the ref they acted on, which keeps cross-disk code short:

with {:ok, report} <- Fil.write(s3, "reports/q3.pdf", pdf),
     {:ok, _backup} <- Fil.cp(report, Fil.ref(local, "backups/q3.pdf")) do
  {:ok, report}
end

The bang variants return the ref itself instead of {:ok, ref}, so you can pipe one call into the next:

disk
|> Fil.write!("hello.txt", "World")
|> Fil.cp!("backup/hello.txt")
|> Fil.read!()
#=> "World"

Plugins

Plugins attach to a disk and see every operation on it. Fil ships one that sets the content type from the file extension, so S3 serves reports/q3.pdf as application/pdf:

s3 =
  Fil.disk(adapter: Fil.Adapter.S3, bucket: "my-bucket", region: "eu-central-1")
  |> Fil.Plugin.ContentType.attach()

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)

For a disk built from config, Fil.disk/1 takes plugins as data: plugins: [{Fil.Plugin.ContentType, :call, []}].

Your own plugin is a function. It gets the operation, calls next to run the rest, and returns the result:

local =
  Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")
  |> Fil.attach(:log, fn op, next, _opts ->
    IO.puts("#{op.name} #{op.path}")
    next.(op)
  end)

The plugins guide explains how to write plugins: matching on operations, changing content, handling errors and answering without the adapter.

Errors

Every error is an exception struct that says what you can do about it, and means the same on every disk:

case Fil.read(disk, "report.txt") do
  {:ok, content} -> content
  {:error, %Fil.NotFoundError{}} -> nil
  {:error, %Fil.UnavailableError{}} -> :retry_later
end

The structs contain the operation, the path, the disk and what the storage reported (:reason), so a log line says which file failed and why. Each error has its own page under Errors in the docs, and Errors in Fil.Adapter explains how adapters use them.

Every function that can fail has a bang variant that returns the bare result and raises the same struct instead.

Usage

The installation guide is the full setup for an application: a module for your disks, the config for each environment (local disks in development, memory disks in tests and S3 in production), signed URLs and tests. This section is only a quick tour of the API.

Build one disk per kind of storage:

local = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")

s3 =
  Fil.disk(
    adapter: Fil.Adapter.S3,
    bucket: "my-bucket",
    region: "eu-central-1",
    access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
    secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
  )

Every operation works the same on every disk:

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)
{:ok, pdf} = Fil.read(report)
{:ok, stat} = Fil.stat(report)
{:ok, reports} = Fil.ls(s3, "reports/", recursive: true)
{:ok, backup} = Fil.cp(report, Fil.ref(local, "backups/q3.pdf"))
{:ok, _} = Fil.rm(report)

Large files don't have to fit in memory. stream returns a file's content as a stream of chunks, and write takes a stream as well as a binary, so a file can go from one disk to another piece by piece. {:file, path} streams a local file:

{:ok, video} = Fil.stream(s3, "videos/intro.mp4")
{:ok, _} = Fil.write(local, "cache/intro.mp4", video)

{:ok, _} = Fil.write(s3, "backups/db.dump", {:file, "db.dump"})

With a local file, a File.stream! of bytes or a stream from stream, Fil knows the size, and S3 sends the stream as it's read, in one request. For other streams, pass size: if you know it, or S3 uploads the stream in parts of 8 MiB (the local and memory disks don't need the size).

offset: and length: read part of a file, and the storage reads only that part (S3 with a Range header):

{:ok, header} = Fil.read(s3, "videos/intro.mp4", length: 4096)
{:ok, rest} = Fil.stream(s3, "videos/intro.mp4", offset: 4096)

url returns the public URL of a file, and signed_url an expiring one, so clients can download or upload a file directly instead of going through your application code. S3 serves its URLs itself. For local and in-memory disks, Fil.Plugin.URL builds them and Fil.Plug serves them from your application, with etags, 304s and range requests, so browsers can cache files and seek in videos:

{:ok, logo_url} = Fil.url(s3, "logo.png")
{:ok, url} = Fil.signed_url(report, expires_in: 900)
{:ok, upload_url} = Fil.signed_url(s3, "inbox/new.bin", method: :put)

With checksum:, a write sends a checksum of the content. S3 rejects the upload if what it received doesn't match, and stores the checksum with the object, so later reads can check it:

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf, checksum: :sha256)
{:ok, pdf} = Fil.read(report, verify_checksum: true)
{:ok, %Fil.Stat{checksum: {:sha256, checksum}}} = Fil.stat(report, checksum: :sha256)

To create a file only if it doesn't exist yet, pass if_exists: :error. If the file is already there, nothing is overwritten and the write returns a Fil.AlreadyExistsError. That makes a simple lock: whoever creates the file first runs the job.

case Fil.write(s3, "jobs/today.lock", "started", if_exists: :error) do
  {:ok, _lock} -> :run_the_job
  {:error, %Fil.AlreadyExistsError{}} -> :someone_else_won
end

tmp returns a ref to a file in a temporary directory, which is removed when the calling process exits. It's a local file, so a copy from any disk gives a tool the path it needs:

report = Fil.tmp("report.pdf")
{:ok, _} = Fil.cp(Fil.ref(s3, "reports/q3.pdf"), report)
{text, 0} = System.cmd("pdftotext", [Fil.Tmp.path(report), "-"])

Development

mix test runs the unit tests, which need no network. The integration tests run the conformance suite against RustFS, started with Docker Compose:

docker compose up -d
mix test.integration

FIL_S3_ENDPOINT, FIL_S3_ACCESS_KEY_ID, FIL_S3_SECRET_ACCESS_KEY and FIL_S3_REGION point the integration tests at another S3 endpoint.

Acknowledgments

Fil builds on ideas from these projects:

Thanks to their authors and contributors.