msgpack_elixir

An implementation of the MessagePack serialization format for Elixir.

It provides functions for encoding and decoding Elixir terms and supports the full MessagePack specification, including the Timestamp and custom Extension types.

Features

• Specification Compliance: Implements the complete MessagePack type system.
• Extensible Struct Support:
  • Natively encodes and decodes DateTime and NaiveDateTime structs via the Timestamp extension type.
  • Allows any custom struct to be encoded via the Msgpack.Encodable protocol.
• Configurable Validation: Provides an option to bypass UTF-8 validation on strings for performance-critical paths.
• Resource Limiting: Includes configurable :max_depth and :max_byte_size limits to mitigate resource exhaustion from malformed or malicious payloads.
• Telemetry Integration: Emits standard :telemetry events for integration with monitoring tools.
• Streaming API: Process large collections or continuous data streams with low memory overhead using Msgpack.encode_stream/2 and Msgpack.decode_stream/2.

Installation

Add msgpack_elixir to your list of dependencies in mix.exs:

def deps do
  [{:msgpack_elixir, "~> 2.0.0"}]
end

Then, run mix deps.get.

Quick Start

# Encode a map. Atom keys are converted to strings by default.
iex> data = %{id: 1, name: "Elixir"}
iex> {:ok, encoded} = Msgpack.encode(data)
<<130, 162, 105, 100, 1, 164, 110, 97, 109, 101, 166, 69, 108, 105, 120, 105, 114>>

# Decode a binary.
iex> Msgpack.decode(encoded)
{:ok, %{"id" => 1, "name" => "Elixir"}}

# Use the exception-raising variants for exceptional failure cases.
iex> Msgpack.decode!(<<0xC1>>)
** (Msgpack.DecodeError) Unknown type prefix: 193. The byte `0xC1` is not a valid MessagePack type marker.

Streaming Large Collections

For datasets that may not fit in memory, you can use the streaming API, which processes one term at a time.

# Create a lazy stream of terms to be encoded.
iex> terms = Stream.cycle([1, "elixir", true])

# The output is a lazy stream of encoded binaries.
iex> encoded_stream = Msgpack.encode_stream(terms)

# The stream is only consumed when you enumerate it.
iex> encoded_stream |> Stream.take(3) |> Enum.to_list()
[
  {:ok, <<1>>},
  {:ok, <<166, 101, 108, 105, 120, 105, 114>>},
  {:ok, <<195>>}
]

Map Encoding

By default, Msgpack.encode/2 serializes Elixir maps in a deterministic manner.

It achieves this by sorting the map keys according to Elixir's standard term ordering before encoding. This ensures that encoding the same map will always produce the exact same binary output, which is critical for tasks like generating signatures or comparing hashes.

iex> map1 = %{a: 1, b: 2}
iex> map2 = %{b: 2, a: 1}

# Both produce the same output because their keys are sorted [:a, :b]
iex> Msgpack.encode!(map1) == Msgpack.encode!(map2)
true

Performance Opt-Out

Sorting keys has a performance cost (O(N log N)).

If you are working in a performance-critical context where byte-for-byte determinism is not required, you can disable it:

Msgpack.encode(map, deterministic: false)

Custom Struct Serialization

You can add custom encoding logic for your own Elixir structs by implementing the Msgpack.Encodable protocol. This allows Msgpack.encode/2 to accept your structs directly, centralizing conversion logic within the protocol implementation.

# 1. Define your application's struct
defmodule Product do
  defstruct [:id, :name]
end

# 2. Implement the `Msgpack.Encodable` protocol for that struct
defimpl Msgpack.Encodable, for: Product do

  # 3. Inside the protocol's `encode/1` function, transform your struct into a basic
  # Elixir term that MessagePack can encode (e.g., a map or a list).
  def encode(%Product{id: id, name: name}) do
    {:ok, %{"id" => id, "name" => name}}
  end
end

iex> product = %Product{id: 1, name: "Elixir"}
iex> {:ok, binary} = Msgpack.encode(product)
<<130, 162, 105, 100, 1, 164, 110, 97, 109, 101, 166, 69, 108, 105, 120, 105, 114>>

iex> Msgpack.decode(binary)
{:ok, %{"id" => 1, "name" => "Elixir"}}

Full Documentation

For detailed information on all features, options, and functions, see the full documentation on HexDocs, which contains a complete API reference for all public modules and functions.

Development

This section explains how to setup the project locally for development.

Dependencies

• Elixir ~> 1.12 (OTP 24+)
  • See Compatibility and deprecations for more information

Get the Source

Clone the project locally:

# via HTTPS
git clone https://github.com/nrednav/msgpack_elixir.git

# via SSH
git clone git@github.com:nrednav/msgpack_elixir.git

Install

Install the project's dependencies:

cd msgpack_elixir/
mix deps.get

Test

Run the test suite:

mix test

Benchmark

Run the benchmarks:

mix run bench/run.exs

Versioning

This project uses Semantic Versioning. For a list of available versions, see the repository tag list.

Issues & Requests

If you encounter a bug or have a feature request, please open an issue on the GitHub repository.

Contributing

Public contributions are welcome! If you would like to contribute, please fork the repository and create a pull request.

License

This project is licensed under the MIT License - see the LICENSE file for details.