Serialization Compatibility Contract (v0.8.0)

This document is the authoritative statement of what ex_data_sketch promises about its binary serialization format across releases. It is intended for downstream users who need to reason about persistence durability, distributed-node compatibility, and long-term storage.

For the byte-level layout itself, see plans/binary_contract.md (v2) and lib/ex_data_sketch/codec.ex (v1). For migration guidance from v0.7.x, see v0.8.0_migration_notes.md.

The promise

For every release in the v0.x series, ex_data_sketch promises:

1. Read compatibility — a v0.N reader can decode any EXSK binary produced by any v0.M release where M <= N.
2. Magic and version stability — the magic bytes "EXSK" and the layout of the version byte are stable across all v0.x releases.
3. Hash algorithm wire-byte stability — the byte values for :phash2 = 0, :xxhash3 = 1, :murmur3 = 2, :custom = 255 are stable across all v0.x releases.
4. Sketch family ID stability — Codec.sketch_id_* constants (1 = HLL, 2 = CMS, ..., 15 = ULL) are stable across all v0.x releases. Future sketch families get new IDs (16+).
5. No silent format changes — bumping the serialization version is announced in the CHANGELOG and documented in the migration notes for that release.
6. Structured failure on incompatible input — readers MUST return {:error, %DeserializationError{}} on any input they cannot parse. They MUST NOT crash the BEAM, return {:ok, _} with corrupted state, or silently produce a sketch from malformed bytes.

The non-promise

For the v0.x series, ex_data_sketch does NOT promise:

1. Write compatibility from N back to M. A v0.N writer is free to produce binaries that v0.M (where M < N) cannot read. v0.8.0 exercised this: it writes EXSK v2 frames that v0.7.x cannot decode.
2. Cross-language interoperability. Only ExDataSketch.Theta has a documented Apache DataSketches interop path (Theta.serialize_datasketches/1, Theta.deserialize_datasketches/2). Other sketch families are ex_data_sketch-native only until v0.10.0's interop track.
3. Stability of internal sketch state binaries. A sketch's state field is internal. Only the framed EXSK output of serialize/1 is stable.
4. Stability across the v0.x to v1.0 boundary. v1.0 is the designated breaking-change opportunity. v0.x readers may not accept v1.x binaries; v1.0 may rename / re-id sketch families that have not yet stabilized.
5. Stability of error messages. DeserializationError.message strings are intended for human consumption. They may evolve in any release.

Format-by-format inventory (current state at v0.8.0)

There is no EXSK v3 today. v3 is reserved for a future frame layout change that cannot be expressed as either a block_version bump or a metadata-block extension.

Versioning axes

EXSK v2 has four orthogonal versioning axes. The promise above applies to each independently.

This layout supports 256 frame versions × 256 metadata block versions × 256 family versions per sketch × up to 64 KiB of forward-compat extension space. There is no realistic scenario in which v0.x exhausts any of these.

Cross-platform stability

For the supported precompiled target matrix (see precompiled_nifs.md):

Cross-OTP stability

:phash2 instability across OTP major versions is a property of the BEAM runtime, not of ex_data_sketch. The library's only mitigation is to NOT default to :phash2 and to mark it stability: :otp_dependent in Hash.algorithm_info/1. Users who persist sketches across an OTP major-version boundary MUST either:

• use :xxhash3 (NIF, fully stable) or :murmur3 (Pure + NIF, fully stable);
• or accept that their :phash2-based sketches are not portable across the boundary.

Cross-language stability

Cross-language interop is OUT OF SCOPE for v0.8.0 except for the preserved ExDataSketch.Theta Apache DataSketches CompactSketch path.

What IS preserved as the foundation for future cross-language work:

• Hash.Murmur3 produces output byte-identical to Apache DataSketches' MurmurHash3_x64_128 high-64-bit convention.
• Hash.Metadata.algorithm_to_byte/1 exposes stable wire bytes that any external implementation can adopt.
• Binary.CRC.crc32c is the standard iSCSI/Btrfs/SCTP/Snappy CRC32C. Any external CRC32C implementation produces the same output.

v0.10.0 will build on these to add full KLL and HLL Apache interoperability.

Forward-compatibility recipes

A future v0.y release wants to add a new field to the metadata block without breaking v0.8.0 readers. Recipe:

1. Write the new field into the metadata block's extension trailer.
2. Increment Hash.Metadata.block_version only if the new field is load-bearing for correctness (rare).
3. Document the new field's wire layout in plans/hash_binary_contract.md.

A v0.8.0 reader, on encountering such a binary:

• Parses the metadata block header (16 bytes) successfully.
• Sees extension_size = N > 0 and consumes N bytes of opaque extension data.
• Round-trips the extension verbatim if the sketch is re-serialized.
• Does NOT interpret the extension bytes — they are forward-compat.

This is the additive-evolution path. The vast majority of future metadata additions should use it.

Breaking-change recipes (escape hatches reserved for v1.0)

If a future change cannot be expressed additively:

For v0.x, only sketch_family_version bumps (which are local to a single sketch and require no global coordination) are realistically in play. The other escape hatches are documented for v1.0+ planning.

Test guarantees

The compatibility contract is locked by tests:

If any of these tests fail in a future release, the compatibility contract has been violated and the release should NOT ship until either the bug is fixed or the violation is documented as an intentional breaking change.

See also

• plans/binary_contract.md — v2 byte-level layout specification.
• plans/hash_binary_contract.md — metadata block byte-level layout.
• plans/corruption_detection.md — CRC32C rationale and error taxonomy.
• v0.8.0_migration_notes.md — v0.7.x to v0.8.0 upgrade guide.
• v0.8.0_architecture.md — layered architecture overview.
• lib/ex_data_sketch/codec.ex — legacy v1 codec (preserved).
• lib/ex_data_sketch/binary.ex — v2 public facade.
• CHANGELOG.md — release-by-release format changes.