View Source Protobuf.JSON (protobuf v0.17.0)

JSON encoding and decoding utilities for Protobuf structs.

It follows Google's specs and reference implementation. Some features such as well-known types are not fully supported yet.

Proto3 is supported as per the specification. Proto2 is supported in practice, but some of its features might not work correctly, such as extensions.

Types

ProtobufJSONSupported
booltrue/falseYes
int32, fixed32, uint32NumberYes
int64, fixed64, uint64StringYes
float, doubleNumberYes
bytesBase64 stringYes
stringStringYes
messageObject ({…})Yes
enumStringYes
map<K,V>Object ({…})Yes
repeated VArray of [v, …]Yes
AnyObject ({…})Yes
TimestampRFC3339 datetimeYes
DurationString (seconds.fraction)Yes
StructObject ({…})Yes
Wrapper typesVarious typesYes
FieldMaskStringYes
ListValueArrayYes
ValueAny JSON valueYes
NullValuenullYes
EmptyObject ({…})Yes

Usage

On Elixir v1.18+, Protobuf.JSON can decode JSON with the built-in JSON parser. On older versions, or when you want Jason-based encoding, make sure you have :jason added to your dependencies:

defp deps do
  [
    {:jason, "~> 1.2"},
    # ...
  ]
end

With encode/1 you can turn any Protobuf message struct into a JSON string:

iex> message = %Car{color: :RED, top_speed: 125.3}
iex> Protobuf.JSON.encode(message)
{:ok, "{\"color\":\"RED\",\"topSpeed\":125.3}"}

And go the other way around with decode/1:

iex> json = ~S|{"color":"RED","topSpeed":125.3}|
iex> Protobuf.JSON.decode(json, Car)
{:ok, %Car{color: :RED, top_speed: 125.3}}

JSON keys are encoded as camelCase strings by default, specified by the json_name field option. So make sure to recompile the .proto files in your project before working with JSON encoding, the compiler will generate all the required json_name options. You can set your own json_name for a particular field too:

message GeoCoordinate {
  double latitude = 1 [ json_name = "lat" ];
  double longitude = 2 [ json_name = "long" ];
}

Known Issues and Limitations

Currently, the protoc compiler won't check for field name collisions. This library won't check that either. Make sure your field names will be unique when serialized to JSON. For instance, this message definition will not encode correctly since it will emit just one of the two fields and the problem might go unnoticed:

message CollidingFields {
  int32 f1 = 1 [json_name = "sameName"];
  float f2 = 2 [json_name = "sameName"];
}

Exact duplicate JSON object keys are rejected when decoding through Elixir's built-in JSON module (Elixir v1.18+). On older runtimes that fall back to Jason, exact duplicate keys cannot be detected because Jason decodes objects directly into maps. This checking behavior is available since v0.17.0.

google.protobuf.Any

The google.protobuf.Any type is supported. It can be used to encode and decode arbitrary messages. When decoding, the "type URL" is used to determine the message type to decode to. The type URL is expected to be in the format of type.googleapis.com/<package>.<message>. For example, the type URL for the google.protobuf.Duration message would be type.googleapis.com/google.protobuf.Duration. To determine the Elixir module from the type URL, the package and message names are split on . and transformed into a module name. In the previous example, we'd end up with Google.Protobuf.Duration. Due to arbitrary atom construction, we're forced to use Module.safe_concat/1 to construct the module name. This means that the module must exist before decoding. If the module doesn't exist, decoding will raise an error.

Summary

Functions

Decodes a JSON iodata into a module Protobuf struct.

Decodes a JSON iodata into a module Protobuf struct.

Generates a JSON representation of the given protobuf struct.

Generates a JSON representation of the given protobuf struct.

Similar to encode/2, but returns iodata

Similar to encode!/2, but returns iodata

Decodes a json_data map into a module Protobuf struct.

Generates a JSON-encodable map for the given Protobuf struct.

Types

Link to this type

decode_opt()

View Source (since 0.17.0)
@type decode_opt() ::
  {:ignore_unknown_fields, boolean()} | {:recursion_limit, pos_integer()}

A decoding option.

@type encode_opt() ::
  {:use_proto_names, boolean()}
  | {:use_enum_numbers, boolean()}
  | {:emit_unpopulated, boolean()}
@type json_data() :: %{optional(binary()) => any()}

Functions

Link to this function

decode(iodata, module, options \\ [])

View Source
@spec decode(iodata(), module(), [decode_opt()]) ::
  {:ok, struct()} | {:error, Protobuf.JSON.DecodeError.t() | Exception.t()}

Decodes a JSON iodata into a module Protobuf struct.

Options

  • :ignore_unknown_fields (boolean): when true, unknown enum string values are treated as if the field was unset (and skipped from repeated fields and map values) instead of raising. Unknown JSON object keys are always ignored regardless of this option. Defaults to false. Available since v0.17.0.

  • :recursion_limit (positive integer): the maximum nesting depth allowed when decoding the dynamically-typed Google.Protobuf.Value, Google.Protobuf.ListValue, and Google.Protobuf.Struct wrappers. Exceeding it raises a Protobuf.JSON.DecodeError. Defaults to 100. Available since v0.17.0.

Examples

Given this Protobuf message:

syntax = "proto3";

message Car {
  enum Color {
    GREEN = 0;
    RED = 1;
  }

  Color color = 1;
  float top_speed = 2;
}

You can build its structs from JSON like this:

iex> Protobuf.JSON.decode("{}", Car)
{:ok, %Car{color: :GREEN, top_speed: 0.0}}

iex> ~S|{"color":"RED"}| |> Protobuf.JSON.decode(Car)
{:ok, %Car{color: :RED, top_speed: 0.0}}

iex> ~S|{"color":"GREEN","topSpeed":80.0}| |> Protobuf.JSON.decode(Car)
{:ok, %Car{color: :GREEN, top_speed: 80.0}}

By default, an unknown enum string value is rejected:

iex> {:error, %Protobuf.JSON.DecodeError{}} =
...>   Protobuf.JSON.decode(~S|{"color":"BLUE"}|, Car)

Pass ignore_unknown_fields: true to drop the unknown value and leave the field at its default instead:

iex> Protobuf.JSON.decode(~S|{"color":"BLUE"}|, Car, ignore_unknown_fields: true)
{:ok, %Car{color: :GREEN, top_speed: 0.0}}
Link to this function

decode!(iodata, module, options \\ [])

View Source
@spec decode!(iodata(), module(), [decode_opt()]) :: struct() | no_return()

Decodes a JSON iodata into a module Protobuf struct.

Similar to decode/3 except it will unwrap the error tuple and raise in case of errors. Accepts the exact same options, documented in decode/3.

Examples

iex> Protobuf.JSON.decode!("{}", Car)
%Car{color: :GREEN, top_speed: 0.0}

iex> ~S|{"color":"RED"}| |> Protobuf.JSON.decode!(Car)
%Car{color: :RED, top_speed: 0.0}

iex> ~S|{"color":"GREEN","topSpeed":80.0}| |> Protobuf.JSON.decode!(Car)
%Car{color: :GREEN, top_speed: 80.0}
Link to this function

encode(struct, opts \\ [])

View Source
@spec encode(
  struct(),
  [encode_opt()]
) :: {:ok, String.t()} | {:error, Protobuf.JSON.EncodeError.t() | Exception.t()}

Generates a JSON representation of the given protobuf struct.

Options

  • :use_proto_names - use original field name instead of the camelCase json_name for JSON keys. Defaults to false.
  • :use_enum_numbers - encode enum field values as numbers instead of their labels. Defaults to false.
  • :emit_unpopulated - emit all fields, even when they are blank, empty, or set to their default value. Defaults to false.

Examples

Suppose that this is you Protobuf message:

syntax = "proto3";

message Car {
  enum Color {
    GREEN = 0;
    RED = 1;
  }

  Color color = 1;
  float top_speed = 2;
}

Encoding is as simple as:

iex> Protobuf.JSON.encode(%Car{color: :RED, top_speed: 125.3})
{:ok, ~S|{"color":"RED","topSpeed":125.3}|}

iex> Protobuf.JSON.encode(%Car{color: :GREEN})
{:ok, "{}"}

iex> Protobuf.JSON.encode(%Car{}, emit_unpopulated: true)
{:ok, ~S|{"color":"GREEN","topSpeed":0.0}|}
Link to this function

encode!(struct, opts \\ [])

View Source
@spec encode!(
  struct(),
  [encode_opt()]
) :: String.t() | no_return()

Generates a JSON representation of the given protobuf struct.

Similar to encode/2 except it will unwrap the error tuple and raise in case of errors.

Examples

iex> Protobuf.JSON.encode!(%Car{top_speed: 80.0})
~S|{"topSpeed":80.0}|
Link to this function

encode_to_iodata(struct, opts \\ [])

View Source
@spec encode_to_iodata(
  struct(),
  [encode_opt()]
) :: {:ok, iodata()} | {:error, Protobuf.JSON.EncodeError.t() | Exception.t()}

Similar to encode/2, but returns iodata

This function should be preferred to encode/2, if the generated JSON will be handed over to one of the IO functions or sent over the socket. The Erlang runtime is able to leverage vectorised writes and avoid allocating a continuous buffer for the whole resulting string, lowering memory use and increasing performance.

Link to this function

encode_to_iodata!(struct, opts \\ [])

View Source
@spec encode_to_iodata!(
  struct(),
  [encode_opt()]
) :: iodata()

Similar to encode!/2, but returns iodata

See encode_to_iodata/2 for more information about when this function should be preferred over encode!/2.

Link to this function

from_decoded(json_data, module, options \\ [])

View Source
@spec from_decoded(json_data(), module(), [decode_opt()]) ::
  {:ok, struct()} | {:error, Protobuf.JSON.DecodeError.t()}

Decodes a json_data map into a module Protobuf struct.

Similar to decode/2 except it takes a JSON map representation of the data. This is especially useful if you want to use custom JSON encoding or a custom JSON library.

Accepts the same options as decode/3.

Field keys in json_data must be strings (either the proto field name or the JSON name). Data produced by Jason.decode(..., keys: :atoms) is not supported: proto map<...> fields are themselves decoded as maps whose keys come from the wire, so atom-keyed input cannot be reliably distinguished from a map<string, _> value and nested maps would not round-trip. Convert keys to strings before calling from_decoded/2 if you need this.

Examples

iex> Protobuf.JSON.from_decoded(%{}, Car)
{:ok, %Car{color: :GREEN, top_speed: 0.0}}

iex> Protobuf.JSON.from_decoded(%{"color" => "RED"}, Car)
{:ok, %Car{color: :RED, top_speed: 0.0}}

iex> Protobuf.JSON.from_decoded(%{"color" => "GREEN","topSpeed" => 80.0}, Car)
{:ok, %Car{color: :GREEN, top_speed: 80.0}}

By default, an unknown enum string value is rejected:

iex> {:error, %Protobuf.JSON.DecodeError{}} =
...>   Protobuf.JSON.from_decoded(%{"color" => "BLUE"}, Car)

Pass ignore_unknown_fields: true to drop it instead:

iex> Protobuf.JSON.from_decoded(%{"color" => "BLUE"}, Car, ignore_unknown_fields: true)
{:ok, %Car{color: :GREEN, top_speed: 0.0}}
Link to this function

to_encodable(struct, opts \\ [])

View Source
@spec to_encodable(
  struct(),
  [encode_opt()]
) :: {:ok, json_data()} | {:error, Protobuf.JSON.EncodeError.t()}

Generates a JSON-encodable map for the given Protobuf struct.

Similar to encode/2 except it will return an intermediate map representation. This is especially useful if you want to use custom JSON encoding or a custom JSON library.

Supports the same options as encode/2.

Examples

iex> Protobuf.JSON.to_encodable(%Car{color: :RED, top_speed: 125.3})
{:ok, %{"color" => :RED, "topSpeed" => 125.3}}

iex> Protobuf.JSON.to_encodable(%Car{color: :GREEN})
{:ok, %{}}

iex> Protobuf.JSON.to_encodable(%Car{}, emit_unpopulated: true)
{:ok, %{"color" => :GREEN, "topSpeed" => 0.0}}