# Protox
[](https://github.com/ahamez/protox/actions/workflows/elixir.yml)
[](https://coveralls.io/github/ahamez/protox?branch=master)
[](https://hex.pm/packages/protox)
[](https://hexdocs.pm/protox/)
Protox is an Elixir library for working with [Google's Protocol Buffers](https://developers.google.com/protocol-buffers) (proto2 and proto3): encode/decode to/from binary, generate code, or compile schemas at build time.
Protox emphasizes **reliability**: it uses [property testing](https://hexdocs.pm/stream_data), [mutation testing](https://github.com/devonestes/muzak), maintains [near 100% coverage](https://coveralls.io/github/ahamez/protox?branch=master), and [passes Google’s conformance suite](#conformance).
> [!NOTE]
> Using v1? See the v2 migration guide in [v1_to_v2_migration.md](documentation/v1_to_v2_migration.md).
## Example
Given the following protobuf definition:
```proto
message Msg{
int32 a = 1;
map<int32, string> b = 2;
}
```
Protox will create a regular Elixir `Msg` struct:
```elixir
iex> msg = %Msg{a: 42, b: %{1 => "a map entry"}}
iex> {:ok, iodata, iodata_size} = Msg.encode(msg)
iex> binary = # read binary from a socket, a file, etc.
iex> {:ok, msg} = Msg.decode(binary)
```
## Usage
You can use Protox in two ways:
1. pass the protobuf schema ([as an inlined schema](#usage-with-an-inlined-schema) or as a [list of files](#usage-with-files)) to the `Protox` macro;
2. [generate](#files-generation) Elixir source code files with the mix task `protox.generate`.
## Prerequisites
- Elixir >= 1.15 and OTP >= 26
- [protoc](https://github.com/protocolbuffers/protobuf/releases) >= 3.0 _This dependency is only required at compile-time_. It must be available in `$PATH`.
## Installation
Add `:protox` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[{:protox, "~> 2.0"}]
end
```
## Usage with an inlined schema
The following example generates two modules, `Baz` and `Foo`:
```elixir
defmodule MyModule do
use Protox, schema: """
syntax = "proto3";
message Baz {
}
message Foo {
int32 a = 1;
map<int32, Baz> b = 2;
}
"""
end
```
> [!NOTE]
> The module in which the `Protox` macro is called is ignored and does not appear in the names of the generated modules. To include the enclosing module’s name, use the `namespace` option, see [here](#namespaces).
## Usage with files
Use the `:files` option to pass a list of files:
```elixir
defmodule MyModule do
use Protox, files: [
"./defs/foo.proto",
"./defs/bar.proto",
"./defs/baz/fiz.proto"
]
end
```
## Encode
Here's how to encode a message to binary protobuf:
```elixir
msg = %Foo{a: 3, b: %{1 => %Baz{}}}
{:ok, iodata, iodata_size} = Protox.encode(msg)
# or using the bang version
{iodata, iodata_size} = Protox.encode!(msg)
```
You can also call `encode/1` and `encode!/1` directly on the generated structures:
```elixir
{:ok, iodata, iodata_size} = Foo.encode(msg)
{iodata, iodata_size} = Foo.encode!(msg)
```
> [!TIP]
> `encode/1` and `encode!/1` return [iodata](https://hexdocs.pm/elixir/IO.html#module-use-cases-for-io-data) for efficiency. Use it directly with file/socket writes, or convert with `IO.iodata_to_binary/1` when you need a binary.
## Decode
Here's how to decode a message from binary protobuf:
```elixir
{:ok, msg} = Protox.decode(<<8, 3, 18, 4, 8, 1, 18, 0>>, Foo)
# or using the bang version
msg = Protox.decode!(<<8, 3, 18, 4, 8, 1, 18, 0>>, Foo)
```
You can also call `decode/1` and `decode!/1` directly on the generated structures:
```elixir
{:ok, msg} = Foo.decode(<<8, 3, 18, 4, 8, 1, 18, 0>>)
msg = Foo.decode!(<<8, 3, 18, 4, 8, 1, 18, 0>>)
```
## Packages
Protox honors the [`package`](https://protobuf.dev/programming-guides/proto3/#packages) directive:
```proto
package abc.def;
message Baz {}
```
The example above is translated to `Abc.Def.Baz` (package `abc.def` is camelized to `Abc.Def`).
## Namespaces
You can prepend a namespace with a prefix using the `:namespace` option:
```elixir
defmodule Bar do
use Protox, schema: """
syntax = "proto3";
package abc;
message Msg {
int32 a = 1;
}
""",
namespace: __MODULE__
end
```
In this example, the module `Bar.Abc.Msg` is generated:
```elixir
msg = %Bar.Abc.Msg{a: 42}
```
## Specify include path
One or more include paths (directories in which to search for imports) can be specified using the `:paths` option:
```elixir
defmodule Baz do
use Protox,
files: [
"./defs1/prefix/foo.proto",
"./defs1/prefix/bar.proto",
"./defs2/prefix/baz/baz.proto"
],
paths: [
"./defs1",
"./defs2"
]
end
```
> [!NOTE]
> It corresponds to the `-I` option of protoc.
## Files generation
It's possible to generate Elixir source code files with the mix task `protox.generate`:
```shell
protox.generate --output-path=/path/to/messages.ex protos/foo.proto protos/bar.proto
```
The files will be usable in any project as long as Protox is declared in the dependencies as functions from its runtime are used.
> [!NOTE]
> protoc is not needed to compile the generated files.
### Options
- `--output-path`
The path to the file to be generated or to the destination folder when generating multiple files.
- `--include-path`
Specifies the [include path](#specify-include-path). If multiple include paths are needed, add more `--include-path` options.
- `--multiple-files`
Generates one file per Elixir module. It's useful for definitions with a lot of messages as the compilation will be parallelized.
When generating multiple files, the `--output-path` option must point to a directory.
- `--namespace`
[Prepends a namespace](#prepend-namespaces) to all generated modules.
## Unknown fields
[Unknown fields](https://developers.google.com/protocol-buffers/docs/proto3#unknowns) are fields present on the wire that do not correspond to the protobuf definition. This enables forward-compatibility: older readers keep and re-emit fields added by newer writers.
When unknown fields are encountered at decoding time, they are kept in the decoded message. It's possible to access them with the `unknown_fields/1` function defined with the message.
```elixir
iex> msg = Msg.decode!(<<8, 42, 42, 4, 121, 97, 121, 101, 136, 241, 4, 83>>)
%Msg{a: 42, b: "", z: -42, __uf__: [{5, 2, <<121, 97, 121, 101>>}]}
iex> Msg.unknown_fields(msg)
[{5, 2, <<121, 97, 121, 101>>}]
```
Always use `unknown_fields/1` since the field name (e.g. `__uf__`) is generated to avoid collisions with protobuf fields. It returns a list of `{tag, wire_type, bytes}`. See the [protobuf encoding guide](https://developers.google.com/protocol-buffers/docs/encoding) for details.
> [!NOTE]
> Unknown fields are retained when re-encoding the message.
## Unsupported features
- The [Any](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#any) well-known type is partially supported: you can manually unpack the embedded message after decoding and conversely pack it before encoding;
- Groups ([deprecated in protobuf](https://protobuf.dev/programming-guides/proto2/#groups));
- All [options](https://developers.google.com/protocol-buffers/docs/proto3#options) other than `packed` and `default` are ignored as they concern other languages implementation details.
## Implementation choices
- (**Protobuf 2**) **Required fields** encoding raises `Protox.RequiredFieldsError` when a required field is missing.
```elixir
defmodule Bar do
use Protox, schema: """
syntax = "proto2";
message Required {
required int32 a = 1;
}
"""
end
iex> Protox.encode!(%Required{})
** (Protox.RequiredFieldsError) Some required fields are not set: [:a]
```
- (**Protobuf 2**) **Nested extensions** Fields names coming from a nested extension are prefixed with the name of the extender:
```protobuf
message Extendee {
extensions 100 to max;
}
message Extension1 {
extend Extendee {
optional Extension1 ext1 = 102;
}
}
message Extension2 {
extend Extendee {
optional int32 ext2 = 103;
}
}
message Extension3 {
extend Extendee {
optional int32 identical_name = 105;
}
}
message Extension4 {
extend Extendee {
repeated int32 identical_name = 106;
}
}
```
In the above example, the fields of `Extendee` will be:
```elixir
:extension1_ext1
:extension2_ext2
:extension3_identical_name
:extension4_identical_name
```
This is to disambiguate cases where fields in extensions have the same name.
- **Enum aliases** When decoding, the last encountered constant is used. For instance, in the following example, `:BAR` is always used if the value `1` is read on the wire:
```protobuf
enum E {
option allow_alias = true;
FOO = 0;
BAZ = 1;
BAR = 1;
}
```
- (**Protobuf 2**) **Unset optional fields** are assigned `nil`. You can use the generated `default/1` function to get the default value of a field:
```elixir
defmodule Bar do
use Protox,
schema: """
syntax = "proto2";
message Foo {
optional int32 a = 1 [default = 42];
}
"""
end
iex> %Foo{}.a
nil
iex> Foo.default(:a)
{:ok, 42}
```
- (**Protobuf 3**) **Unset fields** are assigned to their [default values](https://developers.google.com/protocol-buffers/docs/proto3#default). However, if you use the `optional` keyword (available in protoc >= 3.15), then unset fields are assigned `nil`:
```elixir
defmodule Bar do
use Protox,
schema: """
syntax = "proto3";
message Foo {
int32 a = 1;
optional int32 b = 2;
}
"""
end
iex> %Foo{}.a
0
iex> Foo.default(:a)
{:ok, 0}
iex> %Foo{}.b
nil
iex> Foo.default(:b)
{:error, :no_default_value}
```
- **Messages and enums names** are converted using the [`Macro.camelize/1`](https://hexdocs.pm/elixir/Macro.html#camelize/1) function.
Thus, in the following example, `non_camel_message` becomes `NonCamelMessage`, but the field `non_camel_field` is left unchanged:
```elixir
defmodule Bar do
use Protox,
schema: """
syntax = "proto3";
message non_camel_message {
}
message CamelMessage {
int32 non_camel_field = 1;
}
"""
end
iex> msg = %NonCamelMessage{}
%NonCamelMessage{__uf__: []}
iex> msg = %CamelMessage{}
%CamelMessage{__uf__: [], non_camel_field: 0}
```
## Generated code reference and types mapping
- The detailed reference of the generated code is available in [documentation/reference.md](documentation/reference.md).
- Please see [documentation/types_mapping.md](documentation/types_mapping.md) to see how protobuf types are mapped to Elixir types.
## Conformance
The Protox library has been thoroughly tested using the conformance checker [provided by Google](https://github.com/protocolbuffers/protobuf/tree/master/conformance).
Run the suite with:
```shell
mix protox.conformance
```
> [!NOTE]
> A report will be generated in the directory `conformance_report`.
## Benchmark
See [benchmark/launch_benchmark.md](benchmark/launch_benchmark.md) for running benchmarks.
## Contributing
Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for more information on how to contribute.
