ExZarr

Hex.pmHex DocsHex.pm DownloadsLicenseBuild StatusCoverage Status

Elixir implementation of Zarr: compressed, chunked, N-dimensional arrays designed for parallel computing and scientific data storage.

Full Zarr v3 Support: ExZarr implements both Zarr v2 and v3 specifications with production-ready support for v3's unified codec pipeline, improved metadata format, and modern features. Automatic version detection ensures seamless interoperability. See ZARR_V3_STATUS.md for complete v3 support details.

Features

Installation

Add ex_zarr to your list of dependencies in mix.exs:

def deps do
  [
    {:ex_zarr, "~> 1.0"}
  ]
end

Quick Start

Creating an Array

# Create a Zarr v3 array (recommended for new projects)
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  codecs: [
    %{name: "bytes"},
    %{name: "gzip", configuration: %{level: 5}}
  ],
  zarr_version: 3,
  storage: :memory
)

# Or use v2 format for compatibility with older tools
{:ok, array_v2} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  compressor: :zlib,
  zarr_version: 2,
  storage: :memory
)

Saving and Loading Arrays

# Save array to filesystem
:ok = ExZarr.save(array, path: "/tmp/my_array")

# Open existing array
{:ok, array} = ExZarr.open(path: "/tmp/my_array")

# Load entire array into memory
{:ok, data} = ExZarr.load(path: "/tmp/my_array")

Performance

ExZarr v0.8+ includes major performance optimizations:

Benchmark results (400×400 array, 16 chunks):

See Performance Guide for tuning recommendations and Benchmarks for running your own tests.

# Run quick performance check (completes in 6 seconds)
mix run benchmarks/slicing_bench_quick.exs

Zarr Format Support

ExZarr provides production-ready support for both Zarr v2 and v3 specifications. Arrays can be created in either format, and opening arrays automatically detects the version.

Zarr v3 - Fully Supported (Recommended for New Projects)

Zarr v3 is fully implemented with a unified codec pipeline and improved metadata format:

# Create v3 array with unified codec pipeline
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  codecs: [
    %{name: "bytes"},  # Required array-to-bytes codec
    %{name: "gzip", configuration: %{level: 5}}  # Optional compression
  ],
  zarr_version: 3,
  storage: :filesystem,
  path: "/tmp/my_v3_array"
)

Zarr v2 (Default for Compatibility)

Zarr v2 uses separate filters and compressor configuration:

# Create v2 array (explicit version)
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  filters: [{:shuffle, [elementsize: 8]}],
  compressor: :zlib,
  zarr_version: 2,
  storage: :filesystem,
  path: "/tmp/my_v2_array"
)

Automatic Version Detection

When opening arrays, ExZarr automatically detects the format version:

# Opens v2 or v3 transparently
{:ok, array} = ExZarr.open(path: "/tmp/my_array")

# Check which version was detected
array.version  # Returns 2 or 3

Key Differences Between v2 and v3

Feature v2 v3
Metadata file .zarrayzarr.json
Chunk keys Dot-separated (0.1.2) Slash-separated with prefix (c/0/1/2)
Codec organization Separate filters and compressor Unified codecs array
Data types NumPy-style strings (<f8) Simplified names (float64)
Groups Separate .zgroup files Unified zarr.json with node_type
Attributes Separate .zattrs files Embedded in zarr.json

Converting from v2 to v3

v2-style configuration is automatically converted when creating v3 arrays:

# This v2-style configuration
{:ok, array} = ExZarr.create(
  shape: {1000},
  chunks: {100},
  dtype: :int64,
  filters: [{:shuffle, [elementsize: 8]}],
  compressor: :zlib,
  zarr_version: 3  # Request v3 format
)

# Automatically converts to v3 codec pipeline:
# [
#   %{name: "shuffle", configuration: %{elementsize: 8}},
#   %{name: "bytes"},
#   %{name: "gzip", configuration: %{level: 5}}
# ]

For detailed migration guidance, see docs/V2_TO_V3_MIGRATION.md.

Working with Groups

# Create a hierarchical group structure
{:ok, root} = ExZarr.Group.create("/data",
  storage: :filesystem,
  path: "/tmp/zarr_data"
)

# Create arrays within the group
{:ok, measurements} = ExZarr.Group.create_array(root, "measurements",
  shape: {1000},
  chunks: {100},
  dtype: :float64
)

# Create subgroups
{:ok, subgroup} = ExZarr.Group.create_group(root, "experiments")

Interoperability with Python

ExZarr is fully compatible with Python's zarr library. Arrays created by one can be read by the other:

# Run the interoperability demo
elixir examples/python_interop_demo.exs

This demonstrates:

For detailed interoperability information, see INTEROPERABILITY.md which covers:

Custom Codecs Example

See how to create and use custom compression codecs:

# Run the custom codec example
mix run examples/custom_codec_example.exs

This demonstrates:

Custom Storage Backend Example

See the test suite for a complete example of implementing a custom storage backend:

# View the custom storage tests
cat test/ex_zarr_custom_storage_test.exs

The example demonstrates:

Supported Data Types

ExZarr supports the following data types:

All data types use little-endian byte order by default, consistent with the Zarr specification.

Compression Codecs

ExZarr provides the following built-in compression options:

The :zlib codec uses Erlang's built-in :zlib module for maximum reliability and compatibility.

Custom Codecs

ExZarr supports custom codecs through a behavior-based plugin system. You can create your own compression, checksum, or transformation codecs:

defmodule MyCustomCodec do
  @behaviour ExZarr.Codecs.Codec

  @impl true
  def codec_id, do: :my_codec

  @impl true
  def codec_info do
    %{
      name: "My Custom Codec",
      version: "1.0.0",
      type: :compression,  # or :transformation
      description: "My custom compression algorithm"
    }
  end

  @impl true
  def available?, do: true

  @impl true
  def encode(data, opts) when is_binary(data) do
    # Your encoding logic here
    {:ok, compressed_data}
  end

  @impl true
  def decode(data, opts) when is_binary(data) do
    # Your decoding logic here
    {:ok, decompressed_data}
  end

  @impl true
  def validate_config(opts) do
    # Validate options
    :ok
  end
end

# Register your codec
:ok = ExZarr.Codecs.register_codec(MyCustomCodec)

# Use it like any built-in codec
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  compressor: :my_codec
)

For complete examples, see examples/custom_codec_example.exs which includes:

Custom codec features:

Storage Backends

ExZarr includes three built-in storage backends:

Arrays stored on the filesystem use the standard Zarr format:

Using Zip Storage

Zip storage stores the entire array (metadata + all chunks) in a single zip file:

# Create array with zip storage
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  dtype: :float64,
  storage: :zip,
  path: "/tmp/my_array.zip"
)

# Write data
ExZarr.Array.set_slice(array, data, start: {0, 0}, stop: {100, 100})

# Save to zip file
:ok = ExZarr.save(array, path: "/tmp/my_array.zip")

# Open existing zip
{:ok, reopened} = ExZarr.open(path: "/tmp/my_array.zip", storage: :zip)

Custom Storage Backends

ExZarr supports custom storage backends through a behavior-based plugin system, similar to custom codecs. Create backends for S3, databases, cloud storage, or any other storage system:

defmodule MyApp.S3Storage do
  @behaviour ExZarr.Storage.Backend

  @impl true
  def backend_id, do: :s3

  @impl true
  def init(config) do
    # Initialize S3 connection
    bucket = Keyword.fetch!(config, :bucket)
    {:ok, %{bucket: bucket, client: setup_s3_client()}}
  end

  @impl true
  def read_chunk(state, chunk_index) do
    # Read chunk from S3
    key = build_s3_key(chunk_index)
    AWS.S3.get_object(state.client, state.bucket, key)
  end

  @impl true
  def write_chunk(state, chunk_index, data) do
    # Write chunk to S3
    key = build_s3_key(chunk_index)
    AWS.S3.put_object(state.client, state.bucket, key, data)
  end

  # Implement other required callbacks...
end

# Register your backend
:ok = ExZarr.Storage.Registry.register(MyApp.S3Storage)

# Use it like any built-in backend
{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  storage: :s3,
  bucket: "my-zarr-data"
)

Custom storage backend features:

Required callbacks:

Cloud and Database Storage Backends

ExZarr includes several pre-built storage backends for cloud services and databases:

AWS S3 Storage

# Add dependencies
{:ex_aws, "~> 2.5"},
{:ex_aws_s3, "~> 2.5"}

# Register and use
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.S3)

{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  storage: :s3,
  bucket: "my-zarr-bucket",
  prefix: "experiments/array1",
  region: "us-west-2"
)

Azure Blob Storage

# Add dependency
{:azurex, "~> 0.3"}

# Register and use
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.AzureBlob)

{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  storage: :azure_blob,
  account_name: "mystorageaccount",
  account_key: System.get_env("AZURE_STORAGE_KEY"),
  container: "zarr-data",
  prefix: "experiments/array1"
)

Google Cloud Storage

# Add dependencies
{:goth, "~> 1.4"},
{:req, "~> 0.4"}

# Register and use
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.GCS)

{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  storage: :gcs,
  bucket: "my-zarr-bucket",
  prefix: "experiments/array1",
  credentials: "/path/to/service-account.json"
)

Mnesia (Distributed Database)

# No external dependencies - Mnesia is built into Erlang/OTP

# Initialize Mnesia
:mnesia.create_schema([node()])
:mnesia.start()

# Register and use
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.Mnesia)

{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  storage: :mnesia,
  array_id: "experiment_001",
  table_name: :zarr_storage
)

MongoDB GridFS

# Add dependency
{:mongodb_driver, "~> 1.4"}

# Register and use
:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.MongoGridFS)

{:ok, array} = ExZarr.create(
  shape: {1000, 1000},
  chunks: {100, 100},
  storage: :mongo_gridfs,
  url: "mongodb://localhost:27017",
  database: "zarr_db",
  bucket: "arrays",
  array_id: "experiment_001"
)

Mock Storage (Testing)

# No dependencies - built-in for testing

:ok = ExZarr.Storage.Registry.register(ExZarr.Storage.Backend.Mock)

# Test with error simulation
{:ok, array} = ExZarr.create(
  shape: {100},
  chunks: {10},
  storage: :mock,
  pid: self(),
  error_mode: :random,
  delay: 50  # Simulate 50ms latency
)

# Verify operations
assert_received {:mock_storage, :write_chunk, _}

Cloud Storage Features:

Database Storage Features:

Mock Storage Features:

Architecture

ExZarr uses:

Development

# Install dependencies
mix deps.get

# Compile the project
mix compile

# Run tests
mix test

# Run tests with coverage
mix coveralls

# Run specific test suites
mix test test/ex_zarr_property_test.exs              # Property-based tests
mix test test/ex_zarr_python_integration_test.exs    # Python integration tests

# Run static analysis
mix credo

# Run type checking
mix dialyzer

# Generate documentation
mix docs

Quality Checks

Before committing, ensure all quality checks pass:

# Run all tests
mix test

# Check code style
mix credo --strict

# Run type checker
mix dialyzer

# Verify test coverage
mix coveralls

CI/CD

The project uses GitHub Actions for continuous integration. The CI pipeline:

Testing

ExZarr includes comprehensive test coverage:

Key testing areas:

Python Integration Tests

ExZarr includes integration tests that verify compatibility with Python's zarr library:

# Install Python dependencies (one-time setup)
./test/support/setup_python_tests.sh

# Run integration tests
mix test test/ex_zarr_python_integration_test.exs

These tests verify that:

Requirements: Python 3.6+, zarr-python 2.x, numpy

Documentation

Guides

Comprehensive guides for all skill levels:

Examples

Practical examples demonstrating real-world usage:

API Documentation

Full API documentation is available at hexdocs.pm/ex_zarr.

Key modules:

Roadmap

Completed features:

Future improvements planned for ExZarr:

Contributing

Contributions are welcome. Please:

  1. Fork the repository
  2. Create a feature branch
  3. Add tests for new functionality
  4. Ensure all tests pass with mix test
  5. Run code quality checks with mix credo and mix dialyzer
  6. Submit a pull request

License

MIT

Credits

Inspired by zarr-python. Implements both Zarr v2 and v3 specifications for full compatibility with the broader Zarr ecosystem.