ExPmtiles

CILicenseVersionHex Docs

An Elixir library for working with PMTiles files - a single-file format for storing tiled map data.

Overview

PMTiles is an efficient single-file format for storing tiled map data, designed for cloud storage and CDN delivery. This library provides a complete Elixir implementation for reading and accessing tiles from PMTiles files stored either locally or on Amazon S3.

Features

Installation

Add ex_pmtiles to your list of dependencies in mix.exs:

def deps do
  [
    {:ex_pmtiles, "~> 0.3.4"}
  ]
end

Quick Start

Basic Usage

# Open a local PMTiles file
instance = ExPmtiles.new("path/to/file.pmtiles", :local)

# Open a PMTiles file from S3
instance = ExPmtiles.new("my-bucket", "path/to/file.pmtiles", :s3)

# Get a tile by coordinates
case ExPmtiles.get_zxy(instance, 10, 512, 256) do
  {{offset, length, data}, updated_instance} ->
    # Use the tile data
    data
  {nil, updated_instance} ->
    # Tile not found
    nil
end

Using the Cache

For production applications, use the caching layer for better performance:

# Start the cache for an S3 PMTiles file
{:ok, cache_pid} = ExPmtiles.Cache.start_link(
  name: :cache_one,
  bucket: "maps",
  path: "world.pmtiles",
  enable_dir_cache: true,    # Enable directory caching (default: false)
  enable_tile_cache: true    # Enable tile caching (default: false)
)

# Get tiles with automatic caching
case ExPmtiles.Cache.get_tile(:cache_one, 10, 512, 256) do
  {:ok, tile_data} ->
    # Handle tile data
    tile_data
  {:error, reason} ->
    # Handle error
    nil
end

# Get cache statistics
stats = ExPmtiles.Cache.get_stats(:cache_one)
# Returns: %{hits: 150, misses: 25}

API Reference

Core Functions

ExPmtiles.new/2 and ExPmtiles.new/4

Create a new PMTiles instance:

# Local file
instance = ExPmtiles.new("data/world.pmtiles", :local)

# S3 file
instance = ExPmtiles.new("my-bucket", "maps/world.pmtiles", :s3)

ExPmtiles.get_zxy/4

Get a tile by zoom level and coordinates:

case ExPmtiles.get_zxy(instance, 10, 512, 256) do
  {{offset, length, data}, updated_instance} ->
    # Tile found
    data
  {nil, updated_instance} ->
    # Tile not found
    nil
end

ExPmtiles.zxy_to_tile_id/3 and ExPmtiles.tile_id_to_zxy/1

Convert between coordinates and tile IDs:

# Convert coordinates to tile ID
tile_id = ExPmtiles.zxy_to_tile_id(10, 512, 256)

# Convert tile ID back to coordinates
{z, x, y} = ExPmtiles.tile_id_to_zxy(tile_id)

Cache Functions

ExPmtiles.Cache.start_link/1

Start a cache process:

{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "world.pmtiles",
  enable_dir_cache: true,    # Optional: enable directory caching (default: false)
  enable_tile_cache: true    # Optional: enable tile caching (default: false)
  :max_cache_age_ms          # Maximum age of cache before automatic clearing (default: nil, disabled)
)

Options:

ExPmtiles.Cache.get_tile/4

Get a tile with caching:

case ExPmtiles.Cache.get_tile(pid, 10, 512, 256) do
  {:ok, data} -> data
  {:error, reason} -> nil
end

Configuration

S3 Configuration

For S3 access, ensure you have ExAws configured in your application:

# In config/config.exs
config :ex_aws,
  access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
  secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"},
  region: "us-east-1"

Cache Configuration

Configure cache behavior:

# In config/config.exs
config :ex_pmtiles,
  cache_dir: "priv/pmtiles_cache",  # Directory for file-based cache (default: System.tmp_dir!() <> "/ex_pmtiles_cache")
  http_pool_size: 100,               # Maximum HTTP connections for S3 (default: 100)
  http_timeout: 15_000               # HTTP timeout in milliseconds (default: 15_000)

File-Based Cache Persistence

The cache persists directory structures and tiles to disk using individual files. This eliminates size limits and provides better scalability than traditional database approaches. The cache directory structure is:

cache_dir/
├── directories/    # Cached directory structures
│   └── 124044919597_120976.bin
└── tiles/          # Cached tile data
    └── 10512256.bin

Enable/Disable Caching (per-cache instance):

# With both directory and tile caching enabled
{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "world.pmtiles",
  enable_dir_cache: true,    # Enable directory caching
  enable_tile_cache: true    # Enable tile caching
)

# With only directory caching (recommended for most use cases)
{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "world.pmtiles",
  enable_dir_cache: true,    # Enable directory caching
  enable_tile_cache: false   # Disable tile caching (default)
)

# Without any persistent caching - useful for ephemeral environments
{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "world.pmtiles",
  enable_dir_cache: false,   # No directory caching
  enable_tile_cache: false   # No tile caching
)

Configure cache storage location:

# In config/config.exs
config :ex_pmtiles,
  cache_dir: "/var/cache/pmtiles"  # Default: System.tmp_dir!() <> "/ex_pmtiles_cache"

When to disable caching:

Benefits of file-based caching:

Performance Features

Multi-Level Caching

The library implements a sophisticated caching system:

  1. Directory Cache (File-based): Persistent cache - deserialized PMTiles directory structures saved as individual files on disk (optional, enabled by default)
  2. Tile Cache (File-based): Persistent cache - individual tile data saved as files on disk (optional, disabled by default)
  3. Statistics Tracking (ETS): In-memory counters for cache hits and misses

Background Pre-loading

The cache automatically pre-loads frequently accessed directories in the background on startup, improving response times for common zoom levels.

Concurrent Request Handling

The cache uses a simple, efficient architecture where each request process does its own work:

Supported Formats

Compression Types

Tile Types

Testing

The library includes test support with mock implementations:

# In test/test_helper.exs
Mox.defmock(ExPmtiles.CacheMock, for: ExPmtiles.Behaviour)

# Mock implementations are automatically configured for test environment

You can also test against a pmtiles file stored in an s3 bucket using export $(cat .env | xargs) && elixir test_s3.exs

This expects you to define environment variables in .env:

AWS_ACCESS_KEY_ID=<>
AWS_REGION=<>
AWS_SECRET_ACCESS_KEY=<>
BUCKET=<>
OBJECT=<>

Performance Testing

The library includes performance tests to ensure O(n) linear complexity for directory deserialization:

# Run performance tests
mix test test/ex_pmtiles/performance_test.exs

# Run detailed benchmarks
mix run bench/deserialize_directory_bench.exs

The performance tests verify that:

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests for new functionality
  5. Ensure all tests pass
  6. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Documentation

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/ex_pmtiles.