Introduction to ExCodecs

# Use this install to work with the source code
# Mix.install(
#   [
#     {:ex_codecs, path: Path.join(__DIR__, "..")}, 
#     {:rustler, "~> 0.36"}
#   ],
#   config:  [rustler_precompiled: [force_build: [ex_codecs: true]]]
# )

Mix.install([
  {:ex_codecs, "~> 0.2.3"}
])

Series

What is a Codec?

A codec (coder-decoder) is an abstraction for transforming data between two representations. In ExCodecs, the two fundamental operations are:

• Encode — transform data into a compact or structured form
• Decode — recover the original data from the encoded form

For compression codecs, encoding means compressing and decoding means decompressing. ExCodecs is one framework with category APIs shaped for their data: binary registry codecs use ExCodecs.encode/3 / decode/3, while point-cloud and Gaussian formats use ExCodecs.Spatial.

Binary registry interface:

{:ok, encoded} = ExCodecs.encode(:some_codec, data)
{:ok, decoded} = ExCodecs.decode(:some_codec, encoded)
decoded == data  # round-trip guarantee

Spatial category interface:

{:ok, encoded} = ExCodecs.Spatial.encode(cloud, format: :ply)
{:ok, decoded} = ExCodecs.Spatial.decode(encoded, format: :ply)

Why Codecs Matter

Compression and encoding are fundamental to production systems:

A codec framework gives you a single consistent API over multiple algorithms, so you can choose the right tool per workload without rewriting integration code.

ExCodecs Philosophy

ExCodecs is a codec framework, not just a compression library:

1. Specialized category APIs - one framework without unsafe argument overloading
2. Shared catalog - query binary and spatial codecs, support, and metadata
3. Extensible - binary codecs use ExCodecs.Codec; domain categories can define suitable contracts
4. NIF-native - Rust NIFs for compression throughput
5. Consistent errors - structured %ExCodecs.Error{} for all failure modes

original = String.duplicate(~S"""
Romeo and Juliet
Excerpt from Act 2, Scene 2

JULIET
O Romeo, Romeo! wherefore art thou Romeo?
Deny thy father and refuse thy name;
Or, if thou wilt not, be but sworn my love,
And I'll no longer be a Capulet.

ROMEO
[Aside] Shall I hear more, or shall I speak at this?

JULIET
'Tis but thy name that is my enemy;
Thou art thyself, though not a Montague.
What's Montague? it is nor hand, nor foot,
Nor arm, nor face, nor any other part
Belonging to a man. O, be some other name!
What's in a name? that which we call a rose
By any other name would smell as sweet;
So Romeo would, were he not Romeo call'd,
Retain that dear perfection which he owes
Without that title. Romeo, doff thy name,
And for that name which is no part of thee
Take all myself.

ROMEO
I take thee at thy word:
Call me but love, and I'll be new baptized;
Henceforth I never will be Romeo.

JULIET
What man art thou that thus bescreen'd in night
So stumblest on my counsel?
""", 100)

"Romeo and Juliet\nExcerpt from Act 2, Scene 2\n\nJULIET\nO Romeo, Romeo! wherefore art thou Romeo?\nDeny thy father and refuse thy name;\nOr, if thou wilt not, be but sworn my love,\nAnd I'll no longer be a Capulet.\n\nROMEO\n[Aside] Shall I hear more, or shall I speak at this?\n\nJULIET\n'Tis but thy name that is my enemy;\nThou art thyself, though not a Montague.\nWhat's Montague? it is nor hand, nor foot,\nNor arm, nor face, nor any other part\nBelonging to a man. O, be some other name!\nWhat's in a name? that which we call a rose\nBy any other name would smell as sweet;\nSo Romeo would, were he not Romeo call'd,\nRetain that dear perfection which he owes\nWithout that title. Romeo, doff thy name,\nAnd for that name which is no part of thee\nTake all myself.\n\nROMEO\nI take thee at thy word:\nCall me but love, and I'll be new baptized;\nHenceforth I never will be Romeo.\n\nJULIET\nWhat man art thou that thus bescreen'd in night\nSo stumblest on my counsel?\nRomeo and Juliet\nExcerpt from Act 2, Scene 2\n\nJULIET\nO Romeo, Romeo! wherefore art thou Romeo?\nDeny thy father and refuse thy name;\nOr, if thou wilt not, be but sworn my love,\nAnd I'll no longer be a Capulet.\n\nROMEO\n[Aside] Shall I hear more, or shall I speak at this?\n\nJULIET\n'Tis but thy name that is my enemy;\nThou art thyself, though not a Montague.\nWhat's Montague? it is nor hand, nor foot,\nNor arm, nor face, nor any other part\nBelonging to a man. O, be some other name!\nWhat's in a name? that which we call a rose\nBy any other name would smell as sweet;\nSo Romeo would, were he not Romeo call'd,\nRetain that dear perfection which he owes\nWithout that title. Romeo, doff thy name,\nAnd for that name which is no part of thee\nTake all myself.\n\nROMEO\nI take thee at thy word:\nCall me but love, and I'll be new baptized;\nHenceforth I never will be Romeo.\n\nJULIET\nWhat man art thou that thus bescreen'd in night\nSo stumblest on my counsel?\nRomeo and Juliet\nExcerpt from Act 2, Scene 2\n\nJULIET\nO Romeo, Romeo! wherefore art thou Romeo?\nDeny thy father and refuse thy name;\nOr, if thou wilt not, be but sworn my love,\nAnd I'll no longer be a Capulet.\n\nROMEO\n[Aside] Shall I hear more, or shall I speak at this?\n\nJULIET\n'Tis but thy name that is my enemy;\nThou art thyself, though not a Montague.\nWhat's Montague? it is nor hand, nor foot,\nNor arm, nor face, nor any other part\nBelonging to a man. O, be some other name!\nWhat's in a name? that which we call a rose\nBy any other name would smell as sweet;\nSo Romeo would, were he not Romeo call'd,\nRetain that dear perfection which he owes\nWithout that title. Romeo, doff thy name,\nAnd for that name which is no part of thee\nTake all myself.\n\nROMEO\nI take thee at thy word:\nCall me but love, and I'll be new baptized;\nHenceforth I never will be Romeo.\n\nJULIET\nWhat man art thou that thus bescreen'd in night\nSo stumblest on my counsel?\nRomeo and Juliet\nExcerpt from Act 2, Scene 2\n\nJULIET\nO Romeo, Romeo! wherefore art thou Romeo?\nDeny thy father and refuse thy name;\nOr, if thou wilt not, be but sworn my love,\nAnd I'll no longer be a Capulet.\n\nROMEO\n[Aside] Shall I hear more, or shall I speak at this?\n\nJULIET\n'Tis but thy name that is my enemy;\nThou art thyself, though not a Montague.\nWhat's Montague? it is nor hand, nor foot,\nNor arm, nor face, nor any other part\nBelonging to a man. O, be some other name!\nWhat's in a name? that which we call a rose\nBy any other name would smell as sweet;\nSo Romeo would, were he not Romeo call'd,\nRetain that dear perfection which he owes\nWithout that title. Romeo, doff thy name,\nAnd for that name which is no part of thee\nTake all myself.\n\nROMEO\nI take thee at thy word:\nCall me but love, and I'll be new baptized;\nHenceforth I never will be Romeo.\n\nJULIET\nWhat man art thou that thus bescreen'd in night\nSo stumblest on my counsel?\nRomeo and Juliet\nExcerpt from Act 2, Scene 2\n\nJULIET\nO Romeo, Romeo! wherefore art thou Romeo?\nDeny thy father and refuse thy name;\nOr, if thou wilt not, be but sworn my love,\nAnd I'll no longer be a Capulet.\n\nROMEO\n[Aside] Shall I hear more, or shall I speak at this?\n\nJULIET\n'Tis but thy name that is my enemy;\nThou art thysel" <> ...

Quick Start

Basic Compression and Decompression

# Simple round-trip with Zstd
# original = String.duplicate("The quick brown fox jumps over the lazy dog", 100)
{:ok, compressed} = ExCodecs.encode(:zstd, original)
{:ok, recovered} = ExCodecs.decode(:zstd, compressed)

IO.puts("Original size:   #{byte_size(original)} bytes")
IO.puts("Compressed size: #{byte_size(compressed)} bytes")
IO.puts("Round-trip OK:   #{recovered == original}")

Original size:   94200 bytes
Compressed size: 562 bytes
Round-trip OK:   true

:ok

Codec Options

Each codec supports its own options:

# Zstd compression levels (1-22, higher = smaller but slower)
{:ok, fast} = ExCodecs.encode(:zstd, original, level: 1)
{:ok, small} = ExCodecs.encode(:zstd, original, level: 22)

IO.puts("Level 1:  #{byte_size(fast)} bytes")
IO.puts("Level 22: #{byte_size(small)} bytes")

# Bzip2 block sizes (1-9, higher = smaller but more memory)
{:ok, bz_small} = ExCodecs.encode(:bzip2, original, block_size: 1)
{:ok, bz_max} = ExCodecs.encode(:bzip2, original, block_size: 9)

IO.puts("Bzip2 block 1: #{byte_size(bz_small)} bytes")
IO.puts("Bzip2 block 9: #{byte_size(bz_max)} bytes")

Level 1:  573 bytes
Level 22: 555 bytes
Bzip2 block 1: 1311 bytes
Bzip2 block 9: 1311 bytes

:ok

Blosc2 for Numerical Data

# Float64 data: byte shuffle groups same-position bytes across elements
# (sign/exponent bytes cluster together), enabling much better compression.
floats = for i <- 1..2048, into: <<>>, do: <<i * 0.125::float-size(64)-little>>

{:ok, plain} = ExCodecs.encode(:blosc2, floats, typesize: 8, shuffle: :none)
{:ok, shuffled} = ExCodecs.encode(:blosc2, floats, typesize: 8, shuffle: :byte)

IO.puts("Original:             #{byte_size(floats)} bytes")
IO.puts("Blosc2 (no shuffle):  #{byte_size(plain)} bytes")
IO.puts("Blosc2 (byte shuffle): #{byte_size(shuffled)} bytes")

Original:             16384 bytes
Blosc2 (no shuffle):  8251 bytes
Blosc2 (byte shuffle): 675 bytes

:ok

Available Codecs

codecs = ExCodecs.available_codecs()
IO.puts("Shared catalog: #{inspect(codecs)}")
IO.puts("Compression:    #{inspect(ExCodecs.available_codecs(:compression))}")
IO.puts("Spatial:        #{inspect(ExCodecs.available_codecs(:spatial))}")

Shared catalog: [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd]
Compression:    [:blosc2, :bzip2, :lz4, :snappy, :zstd]
Spatial:        [:gsplat, :ply, :spatial_binary]

:ok

Codec Details

for codec <- codecs do
  {:ok, info} = ExCodecs.codec_info(codec)
  IO.puts(String.duplicate("-", 50))
  IO.puts("Codec:         #{info.name}")
  IO.puts("Category:      #{info.category}")
  IO.puts("Interface:     #{info.interface}")
  IO.puts("Native?:       #{info.native?}")
  IO.puts("Streaming?:    #{info.streaming?}")
  IO.puts("Configurable?: #{info.configurable?}")
  IO.puts("Version:       #{info.version || "unknown"}")
end

--------------------------------------------------
Codec:         blosc2
Category:      compression
Interface:     binary
Native?:       true
Streaming?:    false
Configurable?: true
Version:       c-blosc2-chunk/pure-rust
--------------------------------------------------
Codec:         bzip2
Category:      compression
Interface:     binary
Native?:       true
Streaming?:    false
Configurable?: true
Version:       bzip2-0.6/libbz2-rs
--------------------------------------------------
Codec:         gsplat
Category:      spatial
Interface:     spatial
Native?:       false
Streaming?:    false
Configurable?: false
Version:       GSPL 1
--------------------------------------------------
Codec:         lz4
Category:      compression
Interface:     binary
Native?:       true
Streaming?:    false
Configurable?: false
Version:       lz4_flex-0.11
--------------------------------------------------
Codec:         ply
Category:      spatial
Interface:     spatial
Native?:       false
Streaming?:    false
Configurable?: true
Version:       PLY 1.0
--------------------------------------------------
Codec:         snappy
Category:      compression
Interface:     binary
Native?:       true
Streaming?:    false
Configurable?: false
Version:       snap-1.1
--------------------------------------------------
Codec:         spatial_binary
Category:      spatial
Interface:     spatial
Native?:       false
Streaming?:    false
Configurable?: false
Version:       EXCP 1
--------------------------------------------------
Codec:         zstd
Category:      compression
Interface:     binary
Native?:       true
Streaming?:    false
Configurable?: true
Version:       structured-zstd-0.0.48

[:ok, :ok, :ok, :ok, :ok, :ok, :ok, :ok]

Codec Feature Summary

Error Handling

# Unsupported codec
{:error, err} = ExCodecs.encode(:nonexistent, "data")
IO.puts("Reason:  #{err.reason}")
IO.puts("Message: #{err.message}")

# Invalid data type
{:error, err} = ExCodecs.encode(:zstd, 12345)
IO.puts("Reason:  #{err.reason}")

Reason:  unsupported_codec
Message: The specified codec is not supported
Reason:  invalid_data

:ok

Navigation

Next: Compression Fundamentals