Understanding Blosc2

ExCodecs implements C-Blosc2-compatible chunks only (pure Rust via blosc2-pure-rs). That means:

Standalone Snappy remains ExCodecs.encode(:snappy, data).

Blosc2 is a meta-compressor designed for high-performance compression of binary data, especially numerical arrays. This guide explains how Blosc2 works, its shuffle filters and internal codecs, and how to use it effectively in ExCodecs.

Overview

Blosc2 is not a compression algorithm itself. It is a meta-compressor: a framework that orchestrates other compressors (called "codecs" or "compressors" in Blosc2 terminology) and adds pre-processing and post-processing stages around them.

The key innovation in Blosc2 is combining:

1. Shuffle filters that reorganize bytes to improve compressibility of typed data.
2. Pluggable internal compressors — in ExCodecs: BloscLZ, LZ4, LZ4HC, Zstd, Zlib (not Snappy as cname; use the standalone :snappy codec for Snappy).
3. Block-based chunks — ExCodecs compresses one buffer to one chunk (nthreads: 1 on the NIF; not a multi-chunk super-chunk store).

ExCodecs exposes Blosc2 as a first-class chunk codec with configurable internal compressor, compression level, shuffle mode, and typesize.

How Blosc2 Works

Blosc2 processes data in a pipeline:

Input Binary
    |
    v
[1] Split into internally selected blocks
    |
    v
[2] Apply shuffle filter (none / byte / bit)
    |
    v
[3] Compress each block with internal codec
    |
    v
[4] Assemble one Blosc2 chunk
    |
    v
Output Binary

Decompression reverses the pipeline:

Input Binary (Blosc2 chunk)
    |
    v
[1] Parse chunk header
    |
    v
[2] Decompress each block with internal codec
    |
    v
[3] Apply inverse shuffle filter
    |
    v
[4] Concatenate blocks
    |
    v
Output Binary (original)

Step 1: Block Splitting

Blosc2 chunks split input into internal blocks. In the wider Blosc2 ecosystem, blocks can support parallel or partial processing. ExCodecs currently exposes neither block-size selection nor block-level access; the implementation chooses the block layout and decodes the complete chunk.

Blocks still improve cache utilization inside compression.

Step 2: Shuffle Filters

The shuffle filter is Blosc2's most powerful feature for typed data. It reorders bytes within elements before compression:

Byte Shuffle (shuffle: :byte):

Groups bytes by their position within each element:

Original (4x 32-bit integers):
  [A0 A1 A2 A3] [B0 B1 B2 B3] [C0 C1 C2 C3] [D0 D1 D2 D3]

After byte shuffle:
  [A0 B0 C0 D0] [A1 B1 C1 D1] [A2 B2 C2 D2] [A3 B3 C3 D3]

High-order bytes (A3, B3, C3, D3) often have similar values (e.g., mostly zeros for positive integers). Grouping them together makes the data far more compressible.

Bit Shuffle (shuffle: :bit):

The same principle at the bit level, providing even more correlation for certain data types (e.g., floating-point values where the sign bit and exponent are similar across elements).

No Shuffle (shuffle: :none):

Skip the shuffle step. Use this for data that does not have a regular element structure.

Step 3: Internal Compression

Each (possibly shuffled) block is compressed using one of several codecs:

The choice of internal codec determines the speed/ratio tradeoff within each block. Blosc2 adds the shuffle filtering on top, so the effective ratio can be much higher than using the same codec directly.

Step 4: Chunk Assembly

Blosc2 assembles the compressed blocks into a chunk with metadata including:

• The decompressed size.
• The block size.
• The internal codec, compression level, and shuffle mode.
• The element typesize.

This header allows decompression without any external metadata.

Using Blosc2 in ExCodecs

Basic Usage

# Default: LZ4 inner codec, level 5, byte shuffle, typesize 8
{:ok, compressed} = ExCodecs.encode(:blosc2, data)
{:ok, original} = ExCodecs.decode(:blosc2, compressed)

With Zstd Inner Codec

{:ok, compressed} = ExCodecs.encode(:blosc2, data,
  cname: :zstd,
  clevel: 5,
  shuffle: :byte,
  typesize: 8
)

With No Shuffle (General Binary Data)

{:ok, compressed} = ExCodecs.encode(:blosc2, data,
  cname: :zstd,
  clevel: 3,
  shuffle: :none
)

With Bit Shuffle (Maximum Ratio for Floats)

{:ok, compressed} = ExCodecs.encode(:blosc2, data,
  cname: :zstd,
  clevel: 9,
  shuffle: :bit,
  typesize: 4
)

Single-threaded chunk compression

{:ok, compressed} = ExCodecs.encode(:blosc2, data,
  cname: :lz4,
  clevel: 5,
  shuffle: :byte,
  typesize: 8
)

The current NIF always uses one Blosc2 worker (nthreads: 1) on a BEAM DirtyCpu scheduler. :numthreads is not a public option.

Configuration Options

Choosing cname (Internal Codec)

• :lz4 (default): Best for speed-sensitive applications. The shuffle filter provides most of the ratio improvement.
• :zstd: Best ratio when combined with shuffle. Use :zstd with clevel: 5-9 for maximum compression of numerical data.
• :lz4hc: Higher ratio than :lz4, slower compress.
• :blosclz: Blosc’s own LZ codec (included for C-Blosc2 interop).
• :zlib: Moderate speed, good ratio. Available for compatibility.

cname: :snappy is rejected. Use the standalone registry codec ExCodecs.encode(:snappy, data) instead.

Choosing shuffle

• :byte (default): Best for most typed data. Groups bytes by position within each element. Recommended for float64, int32, and similar numeric types.
• :bit: Finest granularity. Groups bits by position. Can improve ratio further for floating-point data where sign/exponent bits are similar.
• :none: No pre-processing. Use for general binary data without regular element structure, or when the shuffle adds overhead without ratio improvement.

Choosing typesize

The typesize tells Blosc2 how many bytes each element occupies. It is critical for the shuffle filter to work correctly.

Setting typesize: 1 with shuffle: :byte is equivalent to shuffle: :none because there is no intra-element byte grouping.

Setting typesize: 8 with shuffle: :byte on float64 arrays typically provides 2-10x ratio improvement over compressing the raw data directly.

Important: The data length should ideally be a multiple of typesize for the shuffle filter to work correctly. If the data length is not a multiple of typesize, Blosc2 will still work but the last partial element will not benefit from shuffling.

Choosing clevel

With clevel: 0, Blosc2 applies the shuffle filter but does not compress. This is useful when you want the reordering benefit without compression, or as a diagnostic to see how much shuffle alone helps.

Threading

On the BEAM, the NIF runs on a DirtyCpu scheduler. The pure-Rust C-Blosc2-compatible chunk implementation uses nthreads: 1; ExCodecs does not create an extra native thread pool. Run independent codec calls concurrently from separate BEAM processes when workload-level parallelism is needed.

Wire format is a Blosc2 chunk (not super-chunk / B2ND / .b2frame). Fixtures under test/fixtures/blosc2/ are produced with python-blosc2.

The Blosc2 Chunk Format

ExCodecs produces one C-Blosc2-compatible chunk per call. The chunk carries the decompressed size, compressed size, block size, typesize, codec, level, and filter metadata required for full-buffer decompression.

The public API only performs complete chunk decompression. It does not expose partial block reads, super-chunks, B2ND arrays, .b2frame containers, or configurable native decompression threads.

When to Use Blosc2

Use Blosc2 When

• Data is numerical arrays. Float64, int32, and other typed data benefit enormously from shuffle filters.
• You need fine-grained control. Blosc2 offers the most parameters of any codec in ExCodecs.
• Data has regular element structure. If data length is a multiple of a known typesize, Blosc2 can exploit this.
• You need C-Blosc2 chunk interoperability. ExCodecs chunks interoperate with single-buffer C/Python Blosc2 compression APIs.

Consider Alternatives When

• Data is unstructured binary. If there is no regular element structure, the shuffle filter provides no benefit, and Zstd or LZ4 directly may be simpler.
• Data is small. Blosc2's chunk-header overhead makes it less efficient for data under 256 bytes.
• You want simplicity. Blosc2 has many configuration knobs. Zstd with level: 3 is simpler and effective for most data.

Ratio Improvement from Shuffle

The shuffle filter is Blosc2's key advantage for numerical data. Here are typical ratio improvements:

For numerical data, the improvement from shuffle is dramatic and consistent. The more regular the data (arrays of identical types), the greater the benefit.

Comparison with Other Codecs

Best Practices

1. Set typesize correctly. Match it to your element size (4 for float32, 8 for float64). Wrong typesize reduces or eliminates shuffle benefits.

2. Use shuffle: :byte as the default for typed data. It provides the most benefit for the least overhead. Switch to :bit for floating-point data where exponent bits correlate.

3. Use shuffle: :none for unstructured data. Shuffle on unstructured data adds CPU overhead without ratio improvement.

4. Start with cname: :lz4, clevel: 5 for speed, or cname: :zstd, clevel: 5 for ratio. These are the most commonly useful configurations.

5. Parallelize independent calls with BEAM processes. Each NIF call is single-threaded and runs on a DirtyCpu scheduler.

6. Benchmark with your actual data. The effectiveness of shuffle depends heavily on the data. Always measure, do not guess.

7. Ensure data length is a multiple of typesize. If possible, pad your data to a typesize boundary before compressing with Blosc2.