ChDriver.Protocol.Block.Sparse (ch_driver v0.1.0)

Copy Markdown

Sparse serialization

ClickHouse automatically stores a MergeTree column using "sparse" serialization once enough of its values equal the type's default (see ratio_of_defaults_for_sparse_serialization, default 0.9), regardless of whether the user asked for it. On the wire this shows up as the has_custom_serialization byte (see ChDriver.Protocol.NativeBlock's decode_serialization_kind/3) being 1, followed by one more UInt8 "serialization kind" byte (per ClickHouse's ISerialization::Kind enum in ISerialization.h and SerializationInfo::deserializeFromKindsBinary in SerializationInfo.cpp): 0 = DEFAULT (never actually sent, since hasCustomSerialization() is false whenever kind == DEFAULT -- NativeWriter.cpp only writes the kind byte at all when has_custom_serialization is 1) and 1 = SPARSE, the only other kind that exists as of ClickHouse 24.8. Any other byte value is rejected with a clear error rather than guessed at, since it would mean a newer ClickHouse serialization kind this driver doesn't understand.

Sparse's own column-data encoding (matches SerializationSparse.cpp's serializeOffsets/deserializeOffsets/ deserializeBinaryBulkWithMultipleStreams) is two back-to-back substreams -- not a null-map-style parallel array like Nullable:

  • SparseOffsets: a sequence of UInt64 varints, each a "group size" -- the number of consecutive default-valued rows since the last non-default row (or since the start of the block) -- immediately followed by exactly one non-default row (whose actual value lives in the SparseElements substream below, densely packed, one entry per group). The final varint in the sequence has bit 62 (END_OF_GRANULE_FLAG = 1 <<< 62, chosen because ClickHouse's varints only ever carry values < 2^63) set, and its low bits are the count of trailing default rows with no following value -- this is how the reader knows to stop without needing to know the offset-stream's byte length up front. E.g. for 10 rows with non-default values at (0-indexed) rows 2 and 5, the stream is the three varints 2, 2, 4 | END_OF_GRANULE_FLAG (2 defaults, 1 value, 2 defaults, 1 value, 4 trailing defaults, done) -- there is always at least one flagged varint, even for an all-default or zero-row block.
  • SparseElements: exactly as many densely-packed values of the inner type as there were non-default rows above, decoded via that type's own decode_column_data/3 (recursively, same pattern as Nullable/Array in ChDriver.Protocol.Block.Wrappers).

Decoding reconstructs the full num_rows-length column by walking the offsets and interleaving the decoded non-default values at their positions with the inner type's own default value (0/0.0/""/nil/ []/%{}/etc, per default_value/1) everywhere else. Only inner types default_value/1 knows how to produce a default for are supported; anything else surfaces as {:error, {:unsupported_sparse_default, type}} rather than guessing.

Summary

Functions

Sparse's wire format (see this module's moduledoc for the full byte-level explanation; matches SerializationSparse.cpp): a SparseOffsets varint stream giving the position of every non-default row, followed by a SparseElements stream of exactly that many densely-packed values of inner_type, decoded via ChDriver.Protocol.NativeBlock's decode_column_data/3 like any other wrapper type.

Reads SparseOffsets: repeated (group_size, value) pairs -- where group_size is the count of default rows immediately preceding that value's row -- terminated by one final flagged varint (bit 62 set) giving the count of trailing default rows with no following value. Returns the 0-indexed row position of every non-default value, in ascending order (matching the order their values appear in the following SparseElements stream).

Rebuilds the full num_rows-length column from offsets (ascending 0-indexed positions of non-default rows) and values (the correspondingly-ordered decoded non-default values), filling every other position with default.

Functions

decode_sparse(inner_type, num_rows, binary)

Sparse's wire format (see this module's moduledoc for the full byte-level explanation; matches SerializationSparse.cpp): a SparseOffsets varint stream giving the position of every non-default row, followed by a SparseElements stream of exactly that many densely-packed values of inner_type, decoded via ChDriver.Protocol.NativeBlock's decode_column_data/3 like any other wrapper type.

decode_sparse_offsets(binary)

Reads SparseOffsets: repeated (group_size, value) pairs -- where group_size is the count of default rows immediately preceding that value's row -- terminated by one final flagged varint (bit 62 set) giving the count of trailing default rows with no following value. Returns the 0-indexed row position of every non-default value, in ascending order (matching the order their values appear in the following SparseElements stream).

expand_sparse(offsets, values, num_rows, default)

Rebuilds the full num_rows-length column from offsets (ascending 0-indexed positions of non-default rows) and values (the correspondingly-ordered decoded non-default values), filling every other position with default.