Fil

License Version Hex Docs

Fil is a pluggable file storage abstraction for Elixir.

Table of Contents

Note

Fil is in early development, and the API may still change a lot.

Features and goals

Concepts

Fil uses the function names of Elixir's File module (read, write, stat, ls, cp, rename, rm, rm_rf), but on every adapter they behave like an object store:

The contract in Fil.Adapter lists every difference from File. Every function that can fail returns {:ok, result} or {:error, error}, with an error from the Errors section below.

Disks

A disk says where files are stored: an adapter and its options. It's a plain value, so there's no application config and nothing to add to your supervision tree (Fil starts one process of its own, for the Memory adapter). Every function takes a disk as its first argument and a path relative to the disk's root:

disk = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")



{:ok, _} = Fil.write(disk, "hello.txt", "World")



Fil.read(disk, "hello.txt")

#=> {:ok, "World"}

Refs

A Fil.Ref is a single value for a file on a disk, built with Fil.ref/2. Every function that takes a disk and a path as two arguments also takes a ref as one argument in their place:

hello = Fil.ref(disk, "hello.txt")

#=> #Fil.Ref<local:hello.txt>



{:ok, _} = Fil.write(hello, "World")



Fil.read(hello)

#=> {:ok, "World"}

Actions on files return the ref they acted on, which keeps cross-disk code short:

with {:ok, report} <- Fil.write(s3, "reports/q3.pdf", pdf),

     {:ok, _backup} <- Fil.cp(report, Fil.ref(local, "backups/q3.pdf")) do

  {:ok, report}

end

The bang variants return the ref itself instead of {:ok, ref}, so you can pipe one call into the next:

disk

|> Fil.write!("hello.txt", "World")

|> Fil.cp!("backup/hello.txt")

|> Fil.read!()

#=> "World"

Plugins

Plugins attach to a disk and see every operation on it. Fil ships one that sets the content type from the file extension, so S3 serves reports/q3.pdf as application/pdf:

s3 =

  Fil.disk(adapter: Fil.Adapter.S3, bucket: "my-bucket", region: "eu-central-1")

  |> Fil.Plugin.ContentType.attach()



{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)

For a disk built from config, Fil.disk/1 takes plugins as data: plugins: [{Fil.Plugin.ContentType, :call, []}].

Your own plugin is a function. It gets the operation, calls next to run the rest, and returns the result:

local =

  Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")

  |> Fil.attach(:log, fn op, next, _opts ->

    IO.puts("#{op.name} #{op.path}")

    next.(op)

  end)

The plugins guide explains how to write plugins: matching on operations, changing content, handling errors and answering without the adapter.

Errors

Every error is an exception struct that says what you can do about it, and means the same on every disk:

case Fil.read(disk, "report.txt") do

  {:ok, content} -> content

  {:error, %Fil.NotFoundError{}} -> nil

  {:error, %Fil.UnavailableError{}} -> :retry_later

end

The structs contain the operation, the path, the disk and what the storage reported (:reason), so a log line says which file failed and why. Each error has its own page under Errors in the docs, and Errors in Fil.Adapter explains how adapters use them.

Every function that can fail has a bang variant that returns the bare result and raises the same struct instead.

Usage

The installation guide is the full setup for an application: a module for your disks, the config for each environment (local disks in development, memory disks in tests and S3 in production), signed URLs and tests. This section is only a quick tour of the API.

Add fil to your dependencies:

def deps do

  [

    {:fil, "~> 0.1"}

  ]

end

Build one disk per kind of storage:

local = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")



s3 =

  Fil.disk(

    adapter: Fil.Adapter.S3,

    bucket: "my-bucket",

    region: "eu-central-1",

    access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),

    secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")

  )

Every operation works the same on every disk:

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)

{:ok, pdf} = Fil.read(report)

{:ok, stat} = Fil.stat(report)

{:ok, reports} = Fil.ls(s3, "reports/", recursive: true)

{:ok, backup} = Fil.cp(report, Fil.ref(local, "backups/q3.pdf"))

{:ok, _} = Fil.rm(report)

url returns the public URL of a file, and signed_url an expiring one, so clients can download or upload a file directly instead of going through your application code. S3 serves its URLs itself. For local and in-memory disks, Fil.Plugin.URL builds them and Fil.Plug serves them from your application:

{:ok, logo_url} = Fil.url(s3, "logo.png")

{:ok, url} = Fil.signed_url(report, expires_in: 900)

{:ok, upload_url} = Fil.signed_url(s3, "inbox/new.bin", method: :put)

With checksum:, a write sends a checksum of the content. S3 rejects the upload if what it received doesn't match, and stores the checksum with the object, so later reads can check it:

{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf, checksum: :sha256)

{:ok, pdf} = Fil.read(report, verify_checksum: true)

{:ok, %Fil.Stat{checksum: {:sha256, checksum}}} = Fil.stat(report, checksum: :sha256)

To create a file only if it doesn't exist yet, pass if_exists: :error. If the file is already there, nothing is overwritten and the write returns a Fil.AlreadyExistsError. That makes a simple lock: whoever creates the file first runs the job.

case Fil.write(s3, "jobs/today.lock", "started", if_exists: :error) do

  {:ok, _lock} -> :run_the_job

  {:error, %Fil.AlreadyExistsError{}} -> :someone_else_won

end

Development

mix test runs the unit tests, which need no network. The integration tests run the conformance suite against SeaweedFS, started with Docker Compose:

docker compose up -d

mix test.integration

FIL_S3_ENDPOINT, FIL_S3_ACCESS_KEY_ID, FIL_S3_SECRET_ACCESS_KEY and FIL_S3_REGION point the integration tests at another S3 endpoint.

Acknowledgments

Fil builds on ideas from these projects:

Thanks to their authors and contributors.