Protox

Protox is an Elixir library for working with Google’s Protocol Buffers (proto2 and proto3): encode/decode to/from binary, generate code, or compile schemas at build time.

Protox emphasizes reliability: it uses property testing, mutation testing, maintains near 100% coverage, and passes Google’s conformance suite.

[!NOTE] Using v1? See the v2 migration guide in v1_to_v2_migration.md.

Example

Given the following protobuf definition:

message Msg{
  int32 a = 1;
  map<int32, string> b = 2;
}

Protox will create a regular Elixir Msg struct:

iex> msg = %Msg{a: 42, b: %{1 => "a map entry"}}
iex> {:ok, iodata, iodata_size} = Msg.encode(msg)

iex> binary = # read binary from a socket, a file, etc.
iex> {:ok, msg} = Msg.decode(binary)

Usage

You can use Protox in two ways:

1. pass the protobuf schema (as an inlined schema or as a list of files) to the Protox macro;
2. generate Elixir source code files with the mix task protox.generate.

Prerequisites

• Elixir >= 1.15 and OTP >= 26
• protoc >= 3.0 This dependency is only required at compile-time. It must be available in $PATH.

Installation

Add :protox to your list of dependencies in mix.exs:

def deps do
  [{:protox, "~> 2.0"}]
end

Usage with an inlined schema

The following example generates two modules, Baz and Foo:

defmodule MyModule do
  use Protox, schema: """
  syntax = "proto3";

  message Baz {
  }

  message Foo {
    int32 a = 1;
    map<int32, Baz> b = 2;
  }
  """
end

[!NOTE] The module in which the Protox macro is called is ignored and does not appear in the names of the generated modules. To include the enclosing module’s name, use the namespace option, see here.

Usage with files

Use the :files option to pass a list of files:

defmodule MyModule do
  use Protox, files: [
    "./defs/foo.proto",
    "./defs/bar.proto",
    "./defs/baz/fiz.proto"
  ]
end

Encode

Here’s how to encode a message to binary protobuf:

msg = %Foo{a: 3, b: %{1 => %Baz{}}}
{:ok, iodata, iodata_size} = Protox.encode(msg)
# or using the bang version
{iodata, iodata_size} = Protox.encode!(msg)

You can also call encode/1 and encode!/1 directly on the generated structures:

{:ok, iodata, iodata_size} = Foo.encode(msg)
{iodata, iodata_size} = Foo.encode!(msg)

[!TIP] encode/1 and encode!/1 return iodata for efficiency. Use it directly with file/socket writes, or convert with IO.iodata_to_binary/1 when you need a binary.

Decode

Here’s how to decode a message from binary protobuf:

{:ok, msg} = Protox.decode(<<8, 3, 18, 4, 8, 1, 18, 0>>, Foo)
# or using the bang version
msg = Protox.decode!(<<8, 3, 18, 4, 8, 1, 18, 0>>, Foo)

You can also call decode/1 and decode!/1 directly on the generated structures:

{:ok, msg} = Foo.decode(<<8, 3, 18, 4, 8, 1, 18, 0>>)
msg = Foo.decode!(<<8, 3, 18, 4, 8, 1, 18, 0>>)

Packages

Protox honors the package directive:

package abc.def;
message Baz {}

The example above is translated to Abc.Def.Baz (package abc.def is camelized to Abc.Def).

Namespaces

You can prepend a namespace with a prefix using the :namespace option:

defmodule Bar do
  use Protox, schema: """
    syntax = "proto3";

    package abc;

    message Msg {
        int32 a = 1;
      }
    """,
    namespace: __MODULE__
end

In this example, the module Bar.Abc.Msg is generated:

msg = %Bar.Abc.Msg{a: 42}

Specify include path

One or more include paths (directories in which to search for imports) can be specified using the :paths option:

defmodule Baz do
  use Protox,
    files: [
      "./defs1/prefix/foo.proto",
      "./defs1/prefix/bar.proto",
      "./defs2/prefix/baz/baz.proto"
    ],
    paths: [
      "./defs1",
      "./defs2"
    ]
end

[!NOTE] It corresponds to the -I option of protoc.

Files generation

It’s possible to generate Elixir source code files with the mix task protox.generate:

protox.generate --output-path=/path/to/messages.ex protos/foo.proto protos/bar.proto

The files will be usable in any project as long as Protox is declared in the dependencies as functions from its runtime are used.

[!NOTE] protoc is not needed to compile the generated files.

Options

• --output-path

  The path to the file to be generated or to the destination folder when generating multiple files.

• --include-path

  Specifies the include path. If multiple include paths are needed, add more --include-path options.

• --multiple-files

  Generates one file per Elixir module. It’s useful for definitions with a lot of messages as the compilation will be parallelized. When generating multiple files, the --output-path option must point to a directory.

• --namespace

  Prepends a namespace to all generated modules.

Unknown fields

Unknown fields are fields present on the wire that do not correspond to the protobuf definition. This enables forward-compatibility: older readers keep and re-emit fields added by newer writers.

When unknown fields are encountered at decoding time, they are kept in the decoded message. It’s possible to access them with the unknown_fields/1 function defined with the message.

iex> msg = Msg.decode!(<<8, 42, 42, 4, 121, 97, 121, 101, 136, 241, 4, 83>>)
%Msg{a: 42, b: "", z: -42, __uf__: [{5, 2, <<121, 97, 121, 101>>}]}

iex> Msg.unknown_fields(msg)
[{5, 2, <<121, 97, 121, 101>>}]

Always use unknown_fields/1 since the field name (e.g. __uf__) is generated to avoid collisions with protobuf fields. It returns a list of {tag, wire_type, bytes}. See the protobuf encoding guide for details.

[!NOTE] Unknown fields are retained when re-encoding the message.

Unsupported features

• The Any well-known type is partially supported: you can manually unpack the embedded message after decoding and conversely pack it before encoding;
• Groups (deprecated in protobuf);
• All options other than packed and default are ignored as they concern other languages implementation details.

Implementation choices

• (Protobuf 2) Required fields encoding raises Protox.RequiredFieldsError when a required field is missing.

  defmodule Bar do
    use Protox, schema: """
      syntax = "proto2";
  
      message Required {
        required int32 a = 1;
      }
    """
  end
  
  iex> Protox.encode!(%Required{})
  ** (Protox.RequiredFieldsError) Some required fields are not set: [:a]

• (Protobuf 2) Nested extensions Fields names coming from a nested extension are prefixed with the name of the extender:

  message Extendee {
    extensions 100 to max;
  }
  
  message Extension1 {
    extend Extendee {
      optional Extension1 ext1 = 102;
    }
  }
  
  message Extension2 {
    extend Extendee {
      optional int32 ext2 = 103;
    }
  }
  
  message Extension3 {
    extend Extendee {
      optional int32 identical_name = 105;
    }
  }
  
  message Extension4 {
    extend Extendee {
      repeated int32 identical_name = 106;
    }
  }

  In the above example, the fields of Extendee will be:

    :extension1_ext1
    :extension2_ext2
    :extension3_identical_name
    :extension4_identical_name

  This is to disambiguate cases where fields in extensions have the same name.

• Enum aliases When decoding, the last encountered constant is used. For instance, in the following example, :BAR is always used if the value 1 is read on the wire:

  enum E {
    option allow_alias = true;
    FOO = 0;
    BAZ = 1;
    BAR = 1;
  }

• (Protobuf 2) Unset optional fields are assigned nil. You can use the generated default/1 function to get the default value of a field:

  defmodule Bar do
    use Protox,
    schema: """
      syntax = "proto2";
  
      message Foo {
        optional int32 a = 1 [default = 42];
      }
    """
  end
  
  iex> %Foo{}.a
  nil
  
  iex> Foo.default(:a)
  {:ok, 42}

• (Protobuf 3) Unset fields are assigned to their default values. However, if you use the optional keyword (available in protoc >= 3.15), then unset fields are assigned nil:

  defmodule Bar do
    use Protox,
    schema: """
      syntax = "proto3";
  
      message Foo {
        int32 a = 1;
        optional int32 b = 2;
      }
    """
  end
  
  iex> %Foo{}.a
  0
  
  iex> Foo.default(:a)
  {:ok, 0}
  
  iex> %Foo{}.b
  nil
  
  iex> Foo.default(:b)
  {:error, :no_default_value}

• Messages and enums names are converted using the Macro.camelize/1 function. Thus, in the following example, non_camel_message becomes NonCamelMessage, but the field non_camel_field is left unchanged:

  defmodule Bar do
    use Protox,
    schema: """
      syntax = "proto3";
  
      message non_camel_message {
      }
  
      message CamelMessage {
        int32 non_camel_field = 1;
      }
    """
  end
  
  iex> msg = %NonCamelMessage{}
  %NonCamelMessage{__uf__: []}
  
  iex> msg = %CamelMessage{}
  %CamelMessage{__uf__: [], non_camel_field: 0}

Generated code reference and types mapping

• The detailed reference of the generated code is available in documentation/reference.md.
• Please see documentation/types_mapping.md to see how protobuf types are mapped to Elixir types.

Conformance

The Protox library has been thoroughly tested using the conformance checker provided by Google.

Run the suite with:

mix protox.conformance

[!NOTE] A report will be generated in the directory conformance_report.

Benchmark

See benchmark/launch_benchmark.md for running benchmarks.

Contributing

Please see CONTRIBUTING.md for more information on how to contribute.