Capstan.Zstd (Capstan v1.2.1)

Copy Markdown View Source

Pure-Elixir zstd frame decompressor (RFC 8878) — built to inflate MySQL TRANSACTION_PAYLOAD_EVENT bodies (binlog_transaction_compression=ON, ADR-0011). Decode only; no NIF, no compression (ADR-0008).

The contract is byte-exact or fail closed: every corruption signal the RFC names (bad magic, a set reserved bit, an unknown block type, a distribution overflow, a sequence bitstream not fully consumed, an offset outside the decoded window, a Frame_Content_Size mismatch, a content-checksum mismatch) returns {:error, reason} — never silently mis-decoded bytes. Conformance is proven against real MySQL-produced frames inflated by the reference zstd binary (test/capstan/zstd_test.exs), never self-signed fixtures.

Scope decisions (all fail-closed)

  • Dictionaries are refused (:dictionary_unsupported) — a dictionary ID in the frame header means the frame references out-of-band content capstan does not have; guessing is the silent-corruption class this module exists to prevent.
  • The XXH64 content checksum (RFC §3.1.1), when the frame carries one, is VERIFIED — a present integrity signal is never skipped.
  • Whole-frame in-memory decode: the output accumulator IS the window, so an offset is valid iff it is positive and within the bytes decoded so far and under the declared Window_Size.

Summary

Functions

decompress(bin)

Decompresses a concatenation of zstd frames (magic 0xFD2FB528) and skippable frames (0x184D2A5X), returning the concatenated decompressed content, or {:error, reason} on any corruption signal — never partial or guessed output.

decompress(bin, opts)

Functions

decompress(bin)

Decompresses a concatenation of zstd frames (magic 0xFD2FB528) and skippable frames (0x184D2A5X), returning the concatenated decompressed content, or {:error, reason} on any corruption signal — never partial or guessed output.

The output cap (span review, blocking)

max_output_bytes: bounds the CUMULATIVE decompressed size DURING inflation — a frame that would exceed it fails {:error, :output_too_large} BEFORE the output is materialized. Without this, a valid frame with no content-size TLV and thousands of ≤128 KB RLE blocks inflates a ~MB payload toward tens of GB as a single BEAM binary and OOM-crashes the node: any post-hoc size check fires only after the memory is already spent. Without the option there is no cap (the generic utility form); the binary-log path always passes one.

decompress(bin, opts)

@spec decompress(
  binary(),
  keyword()
) :: {:ok, binary()} | {:error, term()}