Protox

Protox is a native Elixir library to work with Google's Protocol Buffers (version 2 and 3).

Conformance

This library has been tested using the conformance checker provided by Google. More information at https://github.com/EasyMile/protox-conformance.

Prerequisites

Protox uses Google protoc (>= 3.0) to parse .proto files. It must be available in $PATH. This dependency is only required at compile time. You can get it here.

Usage

From Files

defmodule Foo do
  @external_resource "./defs/foo.proto"
  @external_resource "./defs/bar.proto"
  @external_resource "./defs/baz/fiz.proto"

  use Protox, files: [
    "./defs/foo.proto",
    "./defs/bar.proto",
    "./defs/baz/fiz.proto",
  ]
end

From a Textual Description

defmodule Bar do
  use Protox, schema: """
  syntax = "proto3";

  package fiz;

  message Baz {
  }

  message Foo {
    int32 a = 1;
    map<int32, Baz> b = 2;
  }
  """
end

The previous example generates two modules: Fiz.Baz and Fiz.Foo.

Working With Namespaces

It is possible to prepend a namespace to all generated modules:

defmodule Bar do
  use Protox, schema: """
    syntax = "proto3";

    enum Enum {
        FOO = 0;
        BAR = 1;
      }
    """,
    namespace: Namespace
end

In this case, the module Namespace.Enum is generated.

Encode

iex> %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}} |> Protox.Encode.encode()
[[[], "\b", <<3>>], <<18>>, <<4>>, "\b", <<1>>, <<18>>, <<0>>]

Note that Protox.Encode.encode/1 returns an IO list, not a binary. Such IO lists can be used directly with files or sockets write operations. However, you can use :binary.list_to_bin/1 to get a binary:

iex> %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}} |> Protox.Encode.encode() |> :binary.list_to_bin()
<<8, 3, 18, 4, 8, 1, 18, 0>>

Decode

iex> <<8, 3, 18, 4, 8, 1, 18, 0>> |> Fiz.Foo.decode()
{:ok,
 %Fiz.Foo{__uf__: [], a: 3,
  b: %{1 => %Fiz.Baz{__uf__: []}}}}

The __uf__ field is explained in the section Unknown fields.

Unknown Fields

If any unknown field is encountered when decoding, it is kept in the decoded message. It is possible to access them with the function get_unknown_fields/1 defined with the message.

iex> msg = <<8, 42, 42, 4, 121, 97, 121, 101, 136, 241, 4, 83>> |> Msg.decode!()
%Msg{a: 42, b: "", z: -42, __uf__: [{5, 2, <<121, 97, 121, 101>>}]}

iex> msg |> Msg.get_unknown_fields()
[{5, 2, <<121, 97, 121, 101>>}]

You must always use get_unknown_fields/1 as the name of the field (e.g. __uf__) is generated at compile time to avoid collision with the actual fields of the Protobuf message.

This function returns a list of tuples {tag, wire_type, bytes}.

Unsupported Features

Protobuf 3 JSON mapping
groups
rpc

Furthermore, all options other than packed and default are ignored.

Implementation Choices

Required fields (Protobuf 2): an error is raised when decoding a message with a missing required field.
When decoding enum aliases, the last encountered constant is used. For instance, in the following example, :BAR is always used if the value 1 is read on the wire.
```
enum E {
  option allow_alias = true;
  FOO = 0;
  BAZ = 1;
  BAR = 1;
}
```
Unset optionals
- For Protobuf 2, unset optional fields are mapped to nil.
- For Protobuf 3, unset optional fields are mapped to their default values, as mandated by the Protobuf spec.

Types Mapping

The following table shows how Protobuf types are mapped to Elixir ones.

Protobuf	Elixir
int32	integer()
int64	integer()
uint32	integer()
uint64	integer()
sint32	integer()
sint64	integer()
fixed32	integer()
fixed64	integer()
sfixed32	integer()
sfixed64	integer()
float	float(), :infinity, :'-infinity', :nan
double	float(), :infinity, :'-infinity', :nan
bool	boolean()
string	String.t
bytes	binary()
map	%{}
oneof	{:field, value}
enum	atom()
message	struct()

Credits

Both gpb and exprotobuf were very useful in understanding how to implement Protocol Buffers.