ExPmtiles

An Elixir library for working with PMTiles files - a single-file format for storing tiled map data.

Overview

PMTiles is an efficient single-file format for storing tiled map data, designed for cloud storage and CDN delivery. This library provides a complete Elixir implementation for reading and accessing tiles from PMTiles files stored either locally or on Amazon S3.

Features

• Multi-storage support: Read PMTiles files from local storage or Amazon S3
• Efficient caching: Caching system with directory caching
• Concurrent access: Safe concurrent access with request deduplication
• Compression support: Built-in support for gzip compression
• Tile type support: MVT, PNG, JPEG, WebP, and AVIF tile formats
• Performance optimized: Background directory pre-loading and persistent cache storage
• Production ready: Configurable timeouts, connection pooling, and error handling

Installation

Add ex_pmtiles to your list of dependencies in mix.exs:

def deps do
  [
    {:ex_pmtiles, "~> 0.2.1"}
  ]
end

Quick Start

Basic Usage

# Open a local PMTiles file
instance = ExPmtiles.new("path/to/file.pmtiles", :local)

# Open a PMTiles file from S3
instance = ExPmtiles.new("my-bucket", "path/to/file.pmtiles", :s3)

# Get a tile by coordinates
case ExPmtiles.get_zxy(instance, 10, 512, 256) do
  {{offset, length, data}, updated_instance} ->
    # Use the tile data
    data
  {nil, updated_instance} ->
    # Tile not found
    nil
end

Using the Cache

For production applications, use the caching layer for better performance:

# Start the cache for an S3 PMTiles file
{:ok, cache_pid} = ExPmtiles.Cache.start_link(
  name: :cache_one,
  bucket: "maps",
  path: "world.pmtiles",
  max_entries: 100_000,    # Maximum tiles to cache (default: 100,000)
  enable_dets: true        # Enable persistent cache (default: true)
)

# Get tiles with automatic caching
case ExPmtiles.Cache.get_tile(:cache_one, 10, 512, 256) do
  {:ok, tile_data} ->
    # Handle tile data
    tile_data
  {:error, reason} ->
    # Handle error
    nil
end

# Get cache statistics
stats = ExPmtiles.Cache.get_stats(:cache_one)
# Returns: %{hits: 150, misses: 25}

API Reference

Core Functions

ExPmtiles.new/2 and ExPmtiles.new/4

Create a new PMTiles instance:

# Local file
instance = ExPmtiles.new("data/world.pmtiles", :local)

# S3 file
instance = ExPmtiles.new("my-bucket", "maps/world.pmtiles", :s3)

ExPmtiles.get_zxy/4

Get a tile by zoom level and coordinates:

case ExPmtiles.get_zxy(instance, 10, 512, 256) do
  {{offset, length, data}, updated_instance} ->
    # Tile found
    data
  {nil, updated_instance} ->
    # Tile not found
    nil
end

ExPmtiles.zxy_to_tile_id/3 and ExPmtiles.tile_id_to_zxy/1

Convert between coordinates and tile IDs:

# Convert coordinates to tile ID
tile_id = ExPmtiles.zxy_to_tile_id(10, 512, 256)

# Convert tile ID back to coordinates
{z, x, y} = ExPmtiles.tile_id_to_zxy(tile_id)

Cache Functions

ExPmtiles.Cache.start_link/1

Start a cache process:

{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "world.pmtiles",
  max_entries: 100_000,  # Optional: max tiles to cache (default: 100,000)
  enable_dets: true      # Optional: enable persistent cache (default: true)
)

Options:

• :bucket - S3 bucket name (or nil for local files)
• :path - Path to the PMTiles file
• :name - Optional custom name for the cache process (atom)
• :max_entries - Maximum number of tiles to cache (default: 100,000)
• :enable_dets - Enable DETS persistence for directory cache (default: true)

ExPmtiles.Cache.get_tile/4

Get a tile with caching:

case ExPmtiles.Cache.get_tile(pid, 10, 512, 256) do
  {:ok, data} -> data
  {:error, reason} -> nil
end

Configuration

S3 Configuration

For S3 access, ensure you have ExAws configured in your application:

# In config/config.exs
config :ex_aws,
  access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
  secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"},
  region: "us-east-1"

Cache Configuration

Configure cache behavior:

# In config/config.exs
config :ex_pmtiles,
  max_entries: 100_000,  # Maximum tiles to cache
  cache_dir: "priv/pmtiles_cache",  # Directory for persistent cache (when enabled)
  http_pool_size: 100,  # Maximum HTTP connections for S3 (default: 100)
  http_timeout: 15_000  # HTTP timeout in milliseconds (default: 15_000)

DETS Persistence

By default, the cache persists directory structures to disk using DETS (Disk-based Erlang Term Storage). This allows the cache to warm up instantly on restart, avoiding expensive S3 fetches.

Enable/Disable DETS (per-cache instance):

# With DETS persistence (default) - faster restarts
{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "world.pmtiles",
  enable_dets: true  # Default
)

# Without DETS persistence - useful for ephemeral environments
{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "world.pmtiles",
  enable_dets: false
)

Configure DETS storage location:

# In config/config.exs
config :ex_pmtiles,
  cache_dir: "/var/cache/pmtiles"  # Default: System.tmp_dir!() <> "/ex_pmtiles_cache"

When to disable DETS:

• Ephemeral containers (Docker/Kubernetes) with no persistent volumes
• Serverless environments (AWS Lambda, etc.)
• Testing environments where you want a fresh cache each time
• Memory-constrained environments where disk I/O should be minimized

Performance Features

Multi-Level Caching

The library implements a sophisticated caching system:

1. Directory Cache (ETS): Hot cache - deserialized PMTiles directory structures in memory
2. Tile Cache (ETS): Individual tile data with LRU eviction
3. Persistent Cache (DETS): Warm cache - saves directories to disk for faster restarts (optional, enabled by default)

Background Pre-loading

The cache automatically pre-loads directories for zoom levels 0-4 in the background, improving response times for common zoom levels.

Concurrent Request Handling

Multiple processes can safely access the same cache without duplicate requests. The system coordinates requests and broadcasts results to all waiting processes.

Supported Formats

Compression Types

• :none - No compression
• :gzip - Gzip compression
• :unknown - Unknown compression type

Tile Types

• :mvt - Mapbox Vector Tiles
• :png - PNG images
• :jpg - JPEG images
• :webp - WebP images
• :avif - AVIF images

Testing

The library includes test support with mock implementations:

# In test/test_helper.exs
Mox.defmock(ExPmtiles.CacheMock, for: ExPmtiles.Behaviour)

# Mock implementations are automatically configured for test environment

You can also test against a pmtiles file stored in an s3 bucket using export $(cat .env | xargs) && elixir test_s3.exs

This expects you to define environment variables in .env:

AWS_ACCESS_KEY_ID=<>
AWS_REGION=<>
AWS_SECRET_ACCESS_KEY=<>
BUCKET=<>
OBJECT=<>

Performance Testing

The library includes performance tests to ensure O(n) linear complexity for directory deserialization:

# Run performance tests
mix test test/ex_pmtiles/performance_test.exs

# Run detailed benchmarks
mix run bench/deserialize_directory_bench.exs

The performance tests verify that:

• Directory deserialization scales linearly with the number of entries
• Large directories (10,000+ entries) deserialize in under 500ms
• Time per entry remains roughly constant regardless of directory size

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Documentation

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/ex_pmtiles.