Introduction to ExCodecs

local_path = Path.join(__DIR__, "../mix.exs")

ex_codecs_dep =
  if File.exists?(local_path) do
    [{:ex_codecs, path: Path.join(__DIR__, "..")}, {:rustler, "~> 0.36"}]
  else
    [{:ex_codecs, "~> 0.2.0"}]
  end

config =
  if File.exists?(local_path) do
    [rustler_precompiled: [force_build: [ex_codecs: true]]]
  else
    []
  end

Mix.install(ex_codecs_dep, config: config)

What is a Codec?

A codec (coder-decoder) is an abstraction for transforming data between two representations. In ExCodecs, the two fundamental operations are:

• Encode — transform data into a compact or structured form
• Decode — recover the original data from the encoded form

For compression codecs, encoding means compressing and decoding means decompressing. ExCodecs is one framework with category APIs shaped for their data: binary registry codecs use ExCodecs.encode/3 / decode/3, while point-cloud and Gaussian formats use ExCodecs.Spatial.

# Binary registry interface:
#   {:ok, encoded} = ExCodecs.encode(:some_codec, data)
#   {:ok, decoded} = ExCodecs.decode(:some_codec, encoded)
#   decoded == data  # round-trip guarantee
#
# Spatial category interface:
#   {:ok, encoded} = ExCodecs.Spatial.encode(cloud, format: :ply)
#   {:ok, decoded} = ExCodecs.Spatial.decode(encoded, format: :ply)

Why Codecs Matter

Compression and encoding are fundamental to production systems:

A codec framework gives you a single consistent API over multiple algorithms, so you can choose the right tool per workload without rewriting integration code.

ExCodecs Philosophy

ExCodecs is a codec framework, not just a compression library:

1. Specialized category APIs — one framework without unsafe argument overloading
2. Shared catalog — query binary and spatial codecs, support, and metadata
3. Extensible — binary codecs use ExCodecs.Codec; domain categories can define suitable contracts
4. NIF-native — Rust-powered NIFs for production throughput
5. Consistent errors — structured %ExCodecs.Error{} for all failure modes

Let's see it in action.

Quick Start

Basic Compression and Decompression

# Simple round-trip with Zstd
original = "The quick brown fox jumps over the lazy dog"
{:ok, compressed} = ExCodecs.encode(:zstd, original)
{:ok, recovered} = ExCodecs.decode(:zstd, compressed)

IO.puts("Original size:   #{byte_size(original)} bytes")
IO.puts("Compressed size: #{byte_size(compressed)} bytes")
IO.puts("Recovered:       #{recovered}")
IO.puts("Round-trip OK:   #{recovered == original}")

Codec Options

Each codec supports its own options:

# Zstd compression levels (1-22, higher = smaller but slower)
{:ok, fast} = ExCodecs.encode(:zstd, original, level: 1)
{:ok, small} = ExCodecs.encode(:zstd, original, level: 22)

IO.puts("Level 1:  #{byte_size(fast)} bytes")
IO.puts("Level 22: #{byte_size(small)} bytes")

# Bzip2 block sizes (1-9, higher = smaller but more memory)
{:ok, bz_small} = ExCodecs.encode(:bzip2, original, block_size: 1)
{:ok, bz_max} = ExCodecs.encode(:bzip2, original, block_size: 9)

IO.puts("Bzip2 block 1: #{byte_size(bz_small)} bytes")
IO.puts("Bzip2 block 9: #{byte_size(bz_max)} bytes")

Blosc2 for Numerical Data

# Blosc2 shines with typed binary data
data = :binary.copy(<<0, 0, 0, 0, 1, 1, 1, 1>>, 512)

{:ok, plain} = ExCodecs.encode(:blosc2, data, typesize: 1, shuffle: :none)
{:ok, shuffled} = ExCodecs.encode(:blosc2, data, typesize: 1, shuffle: :byte)

IO.puts("Original:        #{byte_size(data)} bytes")
IO.puts("Blosc2 (no shuffle): #{byte_size(plain)} bytes")
IO.puts("Blosc2 (byte shuffle): #{byte_size(shuffled)} bytes")

Available Codecs

codecs = ExCodecs.available_codecs()
IO.puts("Shared catalog: #{inspect(codecs)}")
IO.puts("Compression:    #{inspect(ExCodecs.available_codecs(:compression))}")
IO.puts("Spatial:        #{inspect(ExCodecs.available_codecs(:spatial))}")

Codec Details

for codec <- codecs do
  {:ok, info} = ExCodecs.codec_info(codec)
  IO.puts(String.duplicate("-", 50))
  IO.puts("Codec:         #{info.name}")
  IO.puts("Category:      #{info.category}")
  IO.puts("Interface:     #{info.interface}")
  IO.puts("Native?:       #{info.native?}")
  IO.puts("Streaming?:    #{info.streaming?}")
  IO.puts("Configurable?: #{info.configurable?}")
  IO.puts("Version:       #{info.version || "unknown"}")
end

Codec Feature Summary

Error Handling

# Unsupported codec
{:error, err} = ExCodecs.encode(:nonexistent, "data")
IO.puts("Reason:  #{err.reason}")
IO.puts("Message: #{err.message}")

# Invalid data type
{:error, err} = ExCodecs.encode(:zstd, 12345)
IO.puts("Reason:  #{err.reason}")

What's Next?

• Compression Fundamentals — theory, trade-offs, and interactive benchmarks
• Codec Comparison — side-by-side performance analysis
• Building Storage Systems — practical patterns for production use
• Zarr-Style Workloads — scientific dataset compression with Blosc2