Choosing a Compression Codec

This guide helps you select the right compression codec for your use case. It includes a comparison table, decision criteria, and worked examples.

Is the data spatial, or spatially representable?

Compression codecs operate on arbitrary binaries. If records represent coordinates, points, normals, colors, oriented particles, or Gaussian splats, consider converting them to ExCodecs.Spatial types first—even when the source is currently a flat array, CSV rows, maps, or database records.

Use:

• ExCodecs.Spatial.PointCloud for XYZ, XYZRGB(A), normals, and scalar per-point attributes.
• ExCodecs.Spatial.GaussianCloud for position, scale, rotation, opacity, color, and spherical harmonics.
• :ply for interoperable spatial exchange.
• :spatial_binary (EXCP) for compact point clouds.
• :gsplat (GSPL) for compact Gaussian clouds.

Spatial encoding and compression are complementary. Encode the structured data first, then optionally compress the resulting binary:

alias ExCodecs.Spatial.{Point, PointCloud}

cloud =
  rows
  |> Enum.map(fn %{x: x, y: y, z: z} -> Point.new(x, y, z) end)
  |> PointCloud.new()

{:ok, spatial_binary} =
  ExCodecs.Spatial.encode(cloud, format: :spatial_binary)

{:ok, compressed} = ExCodecs.encode(:zstd, spatial_binary, level: 3)

If the numbers have no geometric meaning and only need compact storage, Blosc2 may be a better fit. See Understanding Spatial Codecs and Spatial Wire Formats.

Quick Comparison

* Blosc2 ratio depends heavily on the internal codec (cname) and shuffle settings. With byte shuffle on typed data, ratios can exceed Zstd.

“Streaming” here means an incremental compression API. All current compression codecs operate on complete input buffers.

Decision Framework

Answer the following questions to narrow your choice:

1. What is your data type?

• Coordinates, point samples, particles, normals, or Gaussian splats -- model them with ExCodecs.Spatial, even if they currently arrive as arrays, rows, maps, or CSV. Optionally compress the encoded PLY/EXCP/GSPL binary.
• Text, JSON, logs, general-purpose binary -- Zstd is the best default. Use level 3 for a balance of speed and ratio.
• Typed numerical arrays without spatial semantics (floats, integers, matrices) -- Blosc2 with appropriate typesize and shuffle settings.
• Short-lived messages, RPC payloads -- Snappy or LZ4 for minimal latency.
• Archival storage -- Bzip2 for maximum ratio, or Zstd at level 19-22 for high ratio with better decompression speed.

2. How fast does compression need to be?

• Need >= ~1 GB/s - LZ4 or Snappy.
• Need >= ~300-500 MB/s - Zstd level 1-3, Blosc2 with LZ4 inner codec.
• No constraint - Zstd high levels (9-22) or Bzip2.

3. How fast does decompression need to be?

• Need multi-GB/s - LZ4 or Snappy.
• Need ~1 GB/s class - Zstd any level, Blosc2.
• No constraint - Any codec. Bzip2 decompression is typically much slower.

4. How much memory can you spare?

• Very constrained (embedded, large concurrent load) -- LZ4 or Snappy.
• Moderate -- Zstd at levels 1-14.
• Available -- Zstd at levels 15-22 or Bzip2 at block sizes 7-9.

5. Is data read once or many times?

• Read many times -- Invest in higher compression. The one-time compression cost amortizes over many decompressions. Zstd level 9-14 or Bzip2.
• Read once or rarely -- Use fast compression. LZ4 or Snappy.
• Never decompressed (checksum only) -- Consider whether you need compression at all.

Codec-Specific Selection Guides

When to Use LZ4

{:ok, compressed} = ExCodecs.encode(:lz4, data)

• ExCodecs exposes one fixed, fast LZ4 profile; it does not accept :level.
• Low-latency message queues
• Real-time data pipelines where throughput matters more than size
• Temporary data that will be decompressed quickly
• In-memory caches where the CPU cost of decompression must be negligible
• When you need deterministic compression and decompression speeds

When to Use Snappy

{:ok, compressed} = ExCodecs.encode(:snappy, data)

• RPC frameworks (Snappy is the default in many RPC systems)
• Data that is already partially compressed or has low entropy
• Situations where you want zero configuration
• When every microsecond counts and compression ratio is secondary

When to Use Zstd

{:ok, compressed} = ExCodecs.encode(:zstd, data, level: 3)

• General-purpose compression (Zstd is the best default choice)
• Databases, file storage, and network transmission
• Workloads where decompression speed matters (Zstd decompresses fast regardless of compression level)
• When you need a configurable tradeoff (22 levels from fast to maximum ratio)

ExCodecs does not currently expose Zstd dictionary compression.

When to Use Bzip2

{:ok, compressed} = ExCodecs.encode(:bzip2, data, block_size: 9)

• Archival storage where maximum ratio is the priority
• Offline batch processing where compression time is not constrained
• Data that will be stored for a long time and decompressed rarely
• Interoperability with the .bz2 ecosystem

When to Use Blosc2

{:ok, compressed} = ExCodecs.encode(:blosc2, data, cname: :zstd, clevel: 5, shuffle: :byte, typesize: 8)

• Numerical arrays (float64, int32, etc.)
• Scientific data, time series, matrix storage
• Situations where shuffle filters provide a significant ratio improvement
• When you need fine-grained control over the compression pipeline

Each Blosc2 NIF call is single-threaded (nthreads: 1). Parallelize independent buffers with BEAM processes.

Worked Examples

Example 1: API Response Cache

A web application caches JSON responses. Data is compressed once, read many times.

Choice: Zstd at level 5-9

# Compression (one-time cost)
{:ok, compressed} = ExCodecs.encode(:zstd, json_binary, level: 7)

# Decompression (many reads)
{:ok, original} = ExCodecs.decode(:zstd, compressed)

Rationale: Zstd decompresses quickly regardless of compression level, so invest more in compression to get better ratios for the cache.

Example 2: Real-Time Message Broker

Messages arrive at high volume and must be forwarded with minimal latency.

Choice: LZ4

{:ok, compressed} = ExCodecs.encode(:lz4, message)

Rationale: Latency is the priority. ExCodecs' fixed fast LZ4 profile adds minimal overhead to the pipeline.

Example 3: Scientific Data Archive

A research pipeline archives float64 measurement arrays to cold storage.

Choice: Blosc2 with Zstd inner codec and byte shuffle

{:ok, compressed} = ExCodecs.encode(:blosc2, float_array_binary,
  cname: :zstd,
  clevel: 9,
  shuffle: :byte,
  typesize: 8
)

Rationale: The byte shuffle reorders bytes within each 8-byte float, grouping high-order bytes (often similar) together. On typed arrays this often yields about a 2-4x ratio gain versus compressing the raw array (illustrative; see the Blosc2 guide table).

Example 4: Log File Archival

Monthly log files are compressed and stored in object storage.

Choice: Zstd at level 15-19 or Bzip2 at block size 9

{:ok, compressed} = ExCodecs.encode(:zstd, log_data, level: 17)
# or
{:ok, compressed} = ExCodecs.encode(:bzip2, log_data, block_size: 9)

Rationale: Compression is a one-time batch operation. Maximum ratio reduces storage costs over months. Zstd at high levels offers better decompression speed than Bzip2 if you need occasional access.

Example 5: Short-Lived RPC Payload

An internal service sends compressed protobuf messages over the network.

Choice: Snappy

{:ok, compressed} = ExCodecs.encode(:snappy, protobuf_binary)

Rationale: Protobuf already removes much redundancy. Snappy adds minimal overhead on both compression and decompression, and requires no configuration. The ratio improvement will be modest but the latency impact is negligible.

Summary Table by Use Case