View Source Euneus
A JSON parser and generator in pure Erlang.
Euneus is a rewrite of Thoas.
Like Thoas, both the parser and generator fully conform to RFC 8259 and ECMA 404.
Table of Contents
- Installation
- Basic Usage
- Data Mapping
- Differences to Thoas
- Benchmarks
- Credits
- Why the name Euneus?
- TODO
- Sponsors
- Contributing
- License
Installation
Erlang
% rebar.config
{deps, [euneus]}
Elixir
# mix.exs
def deps do
[{:euneus, "~> 0.3"}]
end
Basic Usage
1> {ok, JSON} = euneus:encode_to_binary(#{name => #{english => <<"Charmander">>, japanese => <<"ヒトカゲ"/utf8>>}, caught_at => erlang:timestamp(), type => [fire], profile => #{height => 0.6, weight => 8}, ability => #{0 => <<"Blaze">>, 1 => undefined}}).
{ok, <<"{\"name\":{\"english\":\"Charmander\",\"japanese\":\"ヒトカゲ\"},\"profile\":{\"height\":0.6,\"weight\":8},\"type\":[\"fire\"],\"caught_at\":\"2023-10-24T05:47:04.939Z\",\"ability\":{\"0\":\"Blaze\",\"1\":null}}">>}
2> euneus:decode(JSON).
{ok,#{<<"ability">> =>
#{<<"0">> => <<"Blaze">>,<<"1">> => undefined},
<<"caught_at">> => {1698,126333,753000},
<<"name">> =>
#{<<"english">> => <<"Charmander">>,
<<"japanese">> =>
<<227,131,146,227,131,136,227,130,171,227,130,178>>},
<<"profile">> => #{<<"height">> => 0.6,<<"weight">> => 8},
<<"type">> => [<<"fire">>]}}
3> euneus:decode(JSON, #{
normalize_key => fun
(<<Char>> = Key, _Opts) when Char >= $0, Char =< $9 ->
binary_to_integer(Key);
(Key, _Opts) ->
binary_to_existing_atom(Key)
end
}).
{ok,#{name =>
#{english => <<"Charmander">>,
japanese =>
<<227,131,146,227,131,136,227,130,171,227,130,178>>},
profile => #{height => 0.6,weight => 8},
type => [<<"fire">>],
caught_at => {1698,126333,753000},
ability => #{0 => <<"Blaze">>,1 => undefined}}}
Data Mapping
Erlang -> | Encode Options -> | JSON -> | Decode Options -> | Erlang |
---|---|---|---|---|
undefined | #{} | null | #{} | undefined |
true | #{} | true | #{} | true |
false | #{} | false | #{} | false |
abc | #{} | "abc" | #{} | <<"abc">> |
"abc" | #{} | [97,98,99] | #{} | "abc" |
<<"abc">> | #{} | "abc" | #{} | <<"abc">> |
{{1970,1,1},{0,0,0}} | #{} | "1970-01-01T00:00:00Z" | #{} | {{1970,1,1},{0,0,0}} |
{0,0,0} | #{} | "1970-01-01T00:00:00.000Z" | #{} | {0,0,0} |
123 | #{} | 123 | #{} | 123 |
123.45600 | #{} | 123.456 | #{} | 123.456 |
[true,0,undefined] | #{} | [true,0,null] | #{null_term => nil} | [true,0,nil] |
#{foo => bar} | #{} | {"foo":"bar"} | #{normalize_key => fun(Key, _Opts) -> binary_to_atom(Key) end} | #{foo => <<"bar">>} |
{myrecord, val} | #{encode_unhandled => fun({myrecord, Val}, Opts) -><br> Encode = maps:get(encode_list, Opts),<br> Encode([myrecord, #{key => Val}], Opts)<br>end}) | ["myrecord", {"key":"val"}] | #{normalize_array => fun([<<"myrecord">>, #{<<"key">> := Val}], _Opts) -><br> {myrecord, binary_to_atom(Val)}<br>end} | {myrecord, val} |
Why not more built-in types?
The goal of Euneus
is to have built-in types that can be encoded and then decoded to the original value. If you have any type that can be encoded and rolled back, feel free to open a new issue to discuss it 😄
Differences to Thoas
Euneus is based on Thoas, so let's discuss the differences.
The main difference between Euneus
to Thoas
is that Euneus gives more control to encoding or decoding data. All encode functions can be overridden and extended and all decoded data can be overridden and transformed.
Encode
Available encode options:
#{
%% null_terms defines what terms will be replaced with the null literal (default: ['undefined']).
null_terms => nonempty_list(),
%% encode_binary allow override the binary() encoding.
encode_binary => function((binary(), Opts :: map()) -> iolist()),
%% encode_atom allow override the atom() encoding.
encode_atom => function((atom(), Opts :: map()) -> iolist()),
%% encode_integer allow override the integer() encoding.
encode_integer => function((integer(), Opts :: map()) -> iolist()),
%% encode_float allow override the float() encoding.
encode_float => function((float(), Opts :: map()) -> iolist()),
%% encode_list allow override the list() encoding.
encode_list => function((list(), Opts :: map()) -> iolist()),
%% encode_map allow override the map() encoding.
encode_map => function((map(), Opts :: map()) -> iolist()),
%% encode_datetime allow override the calendar:datetime() encoding.
encode_datetime => function((calendar:datetime(), Opts :: map()) -> iolist()),
%% encode_timestamp allow override the erlang:timestamp() encoding.
encode_timestamp => function((erlang:timestamp(), Opts :: map()) -> iolist()),
%% encode_unhandled allow encode any custom term (default: raise unsupported_type error).
encode_unhandled => function((term(), Opts :: map()) -> iolist()),
%% escape_binary allow override the binary escaping (default: escape_json/1)
%% There are 4 built-in functions to escape binaries:
%% - escape_json/1;
%% - escape_html/1;
%% - escape_js/1;
%% - escape_unicode/1.
escape_binary => function((binary(), Opts :: map()) -> iolist())
}
For example:
EncodeOpts = #{
encode_binary => fun
(<<"foo">>, Opts) ->
euneus_encoder:escape_binary(<<"bar">>, Opts);
(Bin, Opts) ->
euneus_encoder:escape_binary(Bin, Opts)
end,
encode_unhandled => fun
({_, _, _, _} = Ip, Opts) ->
case inet:ntoa(Ip) of
{error, einval} ->
error(invalid_ip);
IpStr ->
IpBin = list_to_binary(IpStr),
euneus_encoder:escape_binary(IpBin, Opts)
end;
(Term, Opts) ->
euneus_encoder:throw_unsupported_type_error(Term, Opts)
end
},
Data = #{<<"foo">> => bar, ipv4 => {127, 0, 0, 1}, none => undefined},
euneus:encode_to_binary(Data, EncodeOpts).
%% {ok, <<"{\"bar\":\"bar\",\"ipv4\":\"127.0.0.1\",\"none\":null}">>}
Decode
Available decode options:
#{
%% null_term is the null literal override (default: 'undefined').
null_term => term(),
%% normalize_array allow override any array/list().
normalize_array => function((list(), Opts :: map()) -> term()),
%% normalize_object allow override any object/map().
normalize_object => function((map(), Opts :: map()) -> term()),
%% normalize_key allow override the keys from JSON objects.
normalize_key => function((binary(), Opts :: map()) -> term()),
%% normalize_value allow override any other term, like array item or object value.
normalize_value => function((binary(), Opts :: map()) -> term())
}
For example:
DecodeOpts = #{
null_term => nil,
normalize_key => fun
(<<"bar">>, _Opts) ->
foo;
(Key, _Opts) ->
binary_to_atom(Key)
end,
normalize_value => fun
(<<"127.0.0.1">>, _Opts) ->
{127, 0, 0, 1};
(Value, _Opts) ->
Value
end
},
JSON = <<"{\"bar\":\"bar\",\"ipv4\":\"127.0.0.1\",\"none\":null}">>,
euneus:decode(JSON, DecodeOpts).
%% {ok,#{foo"=> <<"bar">>,
%% ipv4 => {127,0,0,1},
%% none => nil}}
Why Euneus over Thoas?
Thoas
is incredible, works performant and perfectly fine, but Euneus
is more flexible, permitting more customizations, and is more performant than Thoas. See the benchmarks.
The motivation for Euneus is this PR.
Benchmarks
All the benchmarks compare Euneus
and Thoas
via Benchee to obtain the results.
Use $ make bench.encode
or $ make bench.decode
to run the benchmarks. Edit the scripts in the ./euneus_bench/script
folder if needed.
Note
Results:
Values in
IPS
(iterations per second), aka how often can the given function be executed within one second (the higher the better - good for graphing), only for run times.
Bold
: best IPS;System info:
- Erlang: 26.1
- Elixir: 1.16.0-dev
- Operating system: Linux
- Available memory: 15.54 GB
- CPU Information: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
- Number of Available Cores: 8
Benchmark setup:
- warmup: 5
- time: 30
Encode
Note
Thoas
does not permit any customization.
File | Euneus | Thoas | Comparison |
---|---|---|---|
blockchain.json | 9.73 K | 7.86 K | 1.24x |
giphy.json | 897.47 | 853.31 | 1.05x |
github.json | 3.14 K | 2.54 K | 1.24x |
govtrack.json | 12.72 | 12.27 | 1.04x |
issue-90.json | 28.92 | 17.50 | 1.65x |
json-generator-pretty.json | 1.16 K | 1.08 K | 1.08x |
json-generator.json | 1.17 K | 1.08 K | 1.08x |
pokedex.json | 1.63 K | 1.73 K | 1.07x |
utf-8-escaped.json | 11.88 K | 10.57 K | 1.12x |
utf-8-unescaped.json | 12.19 K | 10.83 K | 1.13x |
Decode
Note
Thoas
does not permit any customization and does not decodeISO 8601
dates to erlang term, but Euneus decodes out of the box, for example,"1970-01-01T00:00:00Z"
to{{1970,01,01},{0,0,0}}
:: calendar:datetime() and"1970-01-01T00:00:00.000Z"
to{0,0,0}
:: erlang:timestamp().
File | Euneus | Thoas | Comparison |
---|---|---|---|
blockchain.json | 7.18 K | 5.78 K | 1.24x |
giphy.json | 474.91 | 474.75 | 1.00x |
github.json | 2.33 K | 2.02 K | 1.16x |
govtrack.json | 16.35 | 15.65 | 1.04x |
issue-90.json | 25.35 | 17.70 | 1.43x |
json-generator-pretty.json | 617.33 | 542.99 | 1.14x |
json-generator.json | 728.01 | 655.15 | 1.11x |
pokedex.json | 1.37 K | 1.33 K | 1.03x |
utf-8-escaped.json | 1.88 K | 1.66 K | 1.13x |
utf-8-unescaped.json | 10.87 K | 10.47 K | 1.04x |
Credits
Euneus is a rewrite of Thoas, so all credits go to Michał Muskała, Louis Pilfold, also both Jason and Thoas contributors. Thanks for the hard work!
Why the name Euneus?
Euneus is the twin brother of Thoas.
TODO
- [ ] Improve docs
- [X] Specs
- [X] Benchmarks
- [ ] Test suites
Sponsors
If you like this tool, please consider sponsoring me.
I'm thankful for your never-ending support :heart:
I also accept coffees :coffee:
Contributing
Issues
Feel free to submit an issue on Github.
Installation
# Clone this repo
git clone git@github.com:williamthome/euneus.git
# Navigate to the project root
cd euneus
# Compile (ensure you have rebar3 installed)
rebar3 compile
License
Euneus is released under the Apache License 2.0.
Euneus is based off of Thoas, which is also Apache 2.0 licensed.
Some elements have their origins in the Poison library and were initially licensed under CC0-1.0.