This document is the planning stub for the v0.11.0 release. It is intentionally not a commitment -- actual scope will be locked when a v0.11.0 prompt is authored. The purpose here is to:
- Inform users and contributors what the v0.11.0 work surface will likely look like.
- Park v0.10.0 follow-up items so they are not lost.
- Give the next prompt-author a starting point.
For the strategic context see plans/next_steps.md. For the
just-shipped release see CHANGELOG.md [0.10.0].
Release theme (proposed)
v0.11.0 -- Apache HLL Interoperability & New Sketch Families.
Where v0.10.0 built out the operational middle layer (unified sketch contract, storage, windowing, supervised sketches, metrics/dashboard) and closed the first cross-language interop gap (Apache KLL, a pure binary-layout problem since KLL doesn't hash its inputs), v0.11.0 tackles the harder interop case -- HLL -- and starts widening the sketch-family catalog again.
Tracks (proposed)
Track A -- Apache HLL Interoperability
| Deliverable | Notes |
|---|---|
| Hash-function reconciliation | DataSketches HLL is a union of four internal encodings (LIST/SET/HLL_4/HLL_6/HLL_8) keyed to MurmurHash3_x64_128; ExDataSketch.Hash.hash64/1 uses a different function. Unlike KLL, HLL's registers are derived from hashed items, so decoding a foreign binary requires reconciling hash semantics, not just binary layout -- this is the core research/design item. |
HLL.serialize_datasketches/2 / deserialize_datasketches/2 | Replace the not_implemented! stubs, mirroring the Theta/KLL precedent (ExDataSketch.DataSketches.* codec module + Backend.hll_from_components/N callback). |
| Golden fixture corpus | test/fixtures/interop/hll/, generated from the real datasketches Python package (pinned version), following the pattern test/fixtures/interop/kll/ established in v0.10.0. |
guides/apache_interop.md update | Add HLL's row and hash-caveat section (HLL will need the Theta-style caveat, not KLL's hash-free one). |
Track B -- CPC (Compressed Probabilistic Counting)
| Deliverable | Notes |
|---|---|
ExDataSketch.CPC | New cardinality sketch family; smaller serialized footprint than HLL at comparable accuracy. |
| Backend.Pure + Backend.Rust | Full parity from day one, following the established pattern (raw-hashing NIF path per v0.10.0 Phase 6). |
| Apache interop | Evaluate whether CPC's binary format is KLL-shaped (tractable) or HLL-shaped (hash-equality-blocked) before committing to an interop deliverable in the same release. |
Track C -- Tuple Sketch (weighted distinct counting)
| Deliverable | Notes |
|---|---|
ExDataSketch.TupleSketch | Theta-family sketch carrying a summary value per retained entry (e.g. weighted counting, associative aggregation). |
| Summary operations | At minimum sum; consider min/max/last-writer-wins as a configurable summary combiner. |
Carry-forward from v0.10.0 (follow-up issues)
The following v0.10.0 items are candidate v0.11.0 work. None of them is guaranteed scope; the v0.11.0 prompt should choose deliberately.
High-priority carry-forward
| ID | Title | Why v0.11.0? |
|---|---|---|
| 3-R7 | Benchmarks run on a single architecture (Apple M1 Max) only | Twice-deferred (v0.8.0 and v0.9.0 carry-forward). A CI bench matrix on x86_64 and ARM64 Linux was Phase 8's item 5 in v0.10.0 and requires an actual CI infrastructure decision from the maintainer, not just code -- good candidate to finally settle. |
Medium-priority carry-forward
| ID | Title | Notes |
|---|---|---|
| 1-R4 | Deprecate :phash2 | Still data-driven -- needs real adoption telemetry showing :xxhash3/:murmur3 usage before deprecating the fallback. Revisit once v0.10.0's telemetry/dashboard work has been in production for a cycle. |
| -- | FilterChain has no format: :v1 escape hatch | Documented as intentional in its moduledoc (bespoke FCN1 container, not a Codec frame) -- revisit only if a real rolling-upgrade need surfaces for filter chains specifically. |
Explicitly deferred (NOT v0.11.0)
| ID | Title | Target release |
|---|---|---|
| 3-R5 | 6-bit register packing | v1.0 |
| 3-R6 | SIMD intrinsics for HLL | v1.0 |
| 3-R3 | Remove legacy _raw_nif family | v1.0 (binary-stability break) |
| -- | MinHash, Weighted MinHash, VarOpt sampling | v0.12.0 (Similarity & Sampling, per README roadmap) |
Out-of-scope guardrails
The v0.11.0 release should NOT:
- Break the v0.x serialization compatibility contract documented in
serialization_compatibility.md. - Default any opt-out path (e.g., flipping
Backend.default/0silently). Such changes are v1.0 work. - Add Rust dependencies that pull in a C compiler at NIF-build time
(slows down
EX_DATA_SKETCH_BUILD=1users). - Commit to an HLL Apache-interop deliverable before the hash-function reconciliation research in Track A has actually confirmed a tractable approach -- unlike KLL, this is not guaranteed to be a one-phase item.
Suggested v0.11.0 prompt outline
When authoring the v0.11.0 prompt, the structure of the v0.10.0 prompt worked well and should be reused:
- Release theme banner.
- IMPORTANT EXECUTION RULES (architectural, Elixir design philosophy).
- PROJECT GOALS.
- RELEASE SCOPE -- explicit INCLUDES and DOES NOT INCLUDE lists.
- Phase-by-phase breakdown with STOP conditions.
- FINAL RELEASE REQUIREMENTS.
- FINAL OUTPUT REQUIREMENTS.
- Next-release preview (v0.12.0).
Likely phase breakdown:
| Phase | Theme | Primary modules |
|---|---|---|
| 1 | HLL hash-function reconciliation (research + design) | ExDataSketch.DataSketches.*, Backend.hll_from_components |
| 2 | HLL Apache interop implementation + golden fixtures | ExDataSketch.HLL, test/fixtures/interop/hll/ |
| 3 | CPC sketch (Pure + Rust) | ExDataSketch.CPC |
| 4 | Tuple Sketch | ExDataSketch.TupleSketch |
| 5 | Carry-forward (bench matrix, phash2 deprecation review) | per the table above |
| 6 | Carry-forward, hygiene, release | mirrors v0.10.0 Phase 8 |
See also
plans/next_steps.md-- strategic roadmap.apache_interop.md-- what interoperates today (Theta, KLL) and why HLL doesn't yet.serialization_compatibility.md-- the binary compatibility contract v0.11.0 must not break.CHANGELOG.md[0.10.0]-- v0.10.0 changes.