ExZarr.Codecs.ZigCodecs (ExZarr v1.1.0)

View Source

High-performance compression codec implementations using Zig NIFs.

This module provides compression and decompression functions for various codecs by binding to C libraries through Zig's excellent C interop capabilities.

Available Codecs

  • ZLIB: Erlang's built-in :zlib (always available)
  • ZSTD: Zstandard compression via libzstd (requires system library)
  • LZ4: LZ4 compression via liblz4 (requires system library)
  • Snappy: Snappy compression via libsnappy (requires system library)
  • Blosc: Blosc meta-compressor via libblosc (requires system library)
  • Bzip2: Bzip2 compression via libbz2 (requires system library)

System Library Requirements

To use the Zig NIFs, you need to install the compression libraries:

macOS

brew install zstd lz4 snappy c-blosc bzip2

Ubuntu/Debian

apt-get install libzstd-dev liblz4-dev libsnappy-dev libblosc-dev libbz2-dev

Fedora/RHEL

dnf install zstd-devel lz4-devel snappy-devel blosc-devel bzip2-devel

Setup

The Zig NIFs are compiled automatically when you run:

mix zig.get    # Download Zig for zigler
mix compile    # Compile NIFs

Implementation Notes

  • ZLIB uses Erlang's battle-tested :zlib module (no NIF needed)
  • Other codecs use Zig NIFs that call C libraries via Zig's @cImport
  • Memory is managed by the BEAM allocator for safety
  • Errors are converted to Elixir-friendly tuples

See ZIG_NIFS_GUIDE.md for detailed implementation information.

Summary

Functions

blosc_compress(arg1, arg2)

Compresses data using Blosc

blosc_decompress(arg1)

Decompresses Blosc data

bzip2_compress(arg1, arg2)

Compresses data using Bzip2

bzip2_decompress(arg1, arg2)

Decompresses Bzip2 data

crc32c_decode(arg1)

Decodes data with CRC32C checksum validation (bytes-to-bytes codec) Validates and removes the 4-byte checksum from the end of data

crc32c_encode(arg1)

Encodes data with CRC32C checksum appended (bytes-to-bytes codec) Appends 4-byte checksum in little-endian format to the end of data

lz4_compress(arg1)

Compresses data using LZ4

lz4_decompress(arg1, arg2)

Decompresses LZ4 data

snappy_compress(arg1)

Compresses data using Snappy

snappy_decompress(arg1)

Decompresses Snappy data

zlib_compress(data)

Compresses data using ZLIB. Uses Erlang's battle-tested :zlib for maximum compatibility.

zlib_decompress(data)

Decompresses ZLIB data.

zstd_compress(arg1, arg2)

Compresses data using ZSTD

zstd_decompress(arg1)

Decompresses ZSTD data

Functions

blosc_compress(arg1, arg2)

@spec blosc_compress([byte()] | binary(), -2_147_483_648..2_147_483_647) :: term()

Compresses data using Blosc

blosc_decompress(arg1)

@spec blosc_decompress([byte()] | binary()) :: term()

Decompresses Blosc data

bzip2_compress(arg1, arg2)

@spec bzip2_compress([byte()] | binary(), -2_147_483_648..2_147_483_647) :: term()

Compresses data using Bzip2

bzip2_decompress(arg1, arg2)

@spec bzip2_decompress([byte()] | binary(), 0..18_446_744_073_709_551_615) :: term()

Decompresses Bzip2 data

crc32c_decode(arg1)

@spec crc32c_decode([byte()] | binary()) :: term()

Decodes data with CRC32C checksum validation (bytes-to-bytes codec) Validates and removes the 4-byte checksum from the end of data

crc32c_encode(arg1)

@spec crc32c_encode([byte()] | binary()) :: term()

Encodes data with CRC32C checksum appended (bytes-to-bytes codec) Appends 4-byte checksum in little-endian format to the end of data

lz4_compress(arg1)

@spec lz4_compress([byte()] | binary()) :: term()

Compresses data using LZ4

lz4_decompress(arg1, arg2)

@spec lz4_decompress([byte()] | binary(), 0..18_446_744_073_709_551_615) :: term()

Decompresses LZ4 data

snappy_compress(arg1)

@spec snappy_compress([byte()] | binary()) :: term()

Compresses data using Snappy

snappy_decompress(arg1)

@spec snappy_decompress([byte()] | binary()) :: term()

Decompresses Snappy data

zlib_compress(data)

Compresses data using ZLIB. Uses Erlang's battle-tested :zlib for maximum compatibility.

zlib_decompress(data)

Decompresses ZLIB data.

zstd_compress(arg1, arg2)

@spec zstd_compress([byte()] | binary(), -2_147_483_648..2_147_483_647) :: term()

Compresses data using ZSTD

zstd_decompress(arg1)

@spec zstd_decompress([byte()] | binary()) :: term()

Decompresses ZSTD data