GorillaStream

A high-performance, lossless compression library for time series data in Elixir, implementing Facebook's Gorilla compression algorithm.

Features

• Lossless Compression: Perfect reconstruction of original time series data
• High Performance: 4.3M points/sec average encoding throughput
• Excellent Compression Ratios: 2-42x compression depending on data patterns
• Container Compression: Optional zlib or zstd compression layer for additional size reduction
• VictoriaMetrics Preprocessing: Enabled by default to improve compression for gauges and counters
• Streaming Support: Real-time streaming and chunked processing for large datasets

Installation

Add gorilla_stream to your list of dependencies in mix.exs:

def deps do
  [
    {:gorilla_stream, "~> 1.3"}
  ]
end

For better compression ratios, optionally add zstd support:

def deps do
  [
    {:gorilla_stream, "~> 1.3"},
    {:ezstd, "~> 1.2"}  # Optional - enables zstd compression
  ]
end

Quick Start

# Sample time series data: {timestamp, value} tuples
data = [
  {1609459200, 23.5},
  {1609459260, 23.7},
  {1609459320, 23.4},
  {1609459380, 23.6},
  {1609459440, 23.8}
]

# Compress the data
{:ok, compressed} = GorillaStream.compress(data)

# Decompress back to original
{:ok, decompressed} = GorillaStream.decompress(compressed)

# Verify lossless compression
decompressed == data  # => true

Container Compression

GorillaStream supports optional container compression on top of Gorilla encoding:

# Zlib compression (always available)
{:ok, compressed} = GorillaStream.compress(data, compression: :zlib)
{:ok, decompressed} = GorillaStream.decompress(compressed, compression: :zlib)

# Zstd compression (best ratio, requires ezstd)
{:ok, compressed} = GorillaStream.compress(data, compression: :zstd)
{:ok, decompressed} = GorillaStream.decompress(compressed, compression: :zstd)

# Auto-select best available
{:ok, compressed} = GorillaStream.compress(data, compression: :auto)
{:ok, decompressed} = GorillaStream.decompress(compressed, compression: :auto)

# Check zstd availability at runtime
GorillaStream.zstd_available?()  # => true or false

Streaming and Chunked Processing

Process large datasets efficiently using chunked streams:

alias GorillaStream.Stream, as: GStream

large_dataset
|> GStream.compress_stream(chunk_size: 10_000)
|> Enum.to_list()
# => [{:ok, chunk1, metadata1}, {:ok, chunk2, metadata2}, ...]

See the User Guide for streaming, GenStage, Broadway, and Flow integration examples.

Analysis Tools

GorillaStream includes Mix tasks to help evaluate compression strategies:

# Analyze compression ratios across data patterns
mix gorilla_stream.compression_analysis

# Benchmark VictoriaMetrics preprocessing
mix gorilla_stream.vm_benchmark 10000

When to Use

Ideal for:

• Time series monitoring data (CPU, memory, temperature sensors)
• Financial tick data with gradual price changes
• IoT sensor readings with regular intervals
• System metrics with slowly changing values

Not optimal for:

• Completely random data with no patterns
• Text or binary data (use general-purpose compression)
• Data with frequent large jumps between values

Documentation

• User Guide - Comprehensive usage examples and best practices
• Performance Guide - Optimization strategies and benchmarks
• Troubleshooting - Common issues and solutions
• API Reference

License

MIT - see LICENSE for details.