ChDriver.Protocol.Messages (ch_driver v0.1.0)

Copy Markdown

Per-message byte layouts for the ClickHouse native protocol.

ChDriver.Protocol owns packet-type dispatch (mapping a leading varint to the right decoder) and revision-gating predicates; this module owns the actual encoding/decoding of every message body -- Hello, Addendum, Query (+ ClientInfo), the empty Data packet, Ping, Exception, Progress, and ProfileInfo.

Summary

Functions

Builds a ClientHello struct advertising this driver's identity, for the given default_database, user, and password.

This driver's advertised protocol revision (@client_revision).

Decodes a ServerHello packet from the front of binary, including its leading packet-type varint (expected to be 0).

Encodes the Addendum sent immediately after receiving ServerHello, when ChDriver.Protocol.addendum_required?/0 is true. This is not a full packet -- just a raw length-prefixed string (the quota key; we always send empty), with no leading packet-type varint. See TCPHandler::receiveAddendum in ClickHouse's Server/TCPHandler.cpp.

Encodes a Cancel packet (Client packet type 3). No body -- just the packet-type varint. Tells the server to stop processing the currently-running query on this connection; the client must keep reading (Data/Progress/ProfileEvents blocks already in flight before the server notices the cancellation may still arrive) until it observes :end_of_stream -- see ChDriver.Connection.cancel_stream/2, the only caller, which does exactly that drain after sending this.

Encodes a ClientHello struct into the plaintext bytes sent to open a connection: packet-type varint (0), then client name/version/revision and the default_database/user/password length-prefixed strings.

Encodes an empty Data packet (Client packet type 2): an empty external table, sent right after a Query packet to signal "no external tables / no input data" (required even for plain SELECTs).

Encodes a Ping packet (Client packet type 4). No body -- just the packet-type varint. The server replies with a bare Pong (Server packet type 4, see ChDriver.Protocol.decode_packet/1) and nothing else.

Encodes a Query packet (Client packet type 1) for query_string.

Functions

client_hello(default_database \\ "default", user \\ "default", password \\ "")

@spec client_hello(binary(), binary(), binary()) :: ChDriver.Protocol.ClientHello.t()

Builds a ClientHello struct advertising this driver's identity, for the given default_database, user, and password.

client_revision()

@spec client_revision() :: non_neg_integer()

This driver's advertised protocol revision (@client_revision).

decode_server_hello(binary)

@spec decode_server_hello(binary()) ::
  {:ok, ChDriver.Protocol.ServerHello.t(), binary()}
  | {:incomplete, binary()}
  | {:error, term()}

Decodes a ServerHello packet from the front of binary, including its leading packet-type varint (expected to be 0).

Returns {:ok, %ServerHello{}, rest}, {:incomplete, binary} if more bytes are needed, or {:error, reason} if the packet type doesn't match Hello (e.g. the server sent an Exception instead, most likely due to bad credentials).

encode_addendum()

@spec encode_addendum() :: iodata()

Encodes the Addendum sent immediately after receiving ServerHello, when ChDriver.Protocol.addendum_required?/0 is true. This is not a full packet -- just a raw length-prefixed string (the quota key; we always send empty), with no leading packet-type varint. See TCPHandler::receiveAddendum in ClickHouse's Server/TCPHandler.cpp.

CRITICAL: ClickHouse requires this immediately after ServerHello, before any Query packet, whenever the client's advertised revision is >= DBMS_MIN_PROTOCOL_VERSION_WITH_ADDENDUM (54458, see ChDriver.Protocol.addendum_required?/0) -- which this driver's fixed revision always is. Omitting this step silently desyncs every byte sent afterwards: the server blocks reading it right after ServerHello, so the next bytes we send -- e.g. the start of a Query packet -- get consumed as this string instead, which surfaces as a bizarre unrelated "Empty query" (or worse) error from the server despite an otherwise-correct Query packet.

encode_cancel()

@spec encode_cancel() :: iodata()

Encodes a Cancel packet (Client packet type 3). No body -- just the packet-type varint. Tells the server to stop processing the currently-running query on this connection; the client must keep reading (Data/Progress/ProfileEvents blocks already in flight before the server notices the cancellation may still arrive) until it observes :end_of_stream -- see ChDriver.Connection.cancel_stream/2, the only caller, which does exactly that drain after sending this.

encode_client_hello(hello)

@spec encode_client_hello(ChDriver.Protocol.ClientHello.t()) :: iodata()

Encodes a ClientHello struct into the plaintext bytes sent to open a connection: packet-type varint (0), then client name/version/revision and the default_database/user/password length-prefixed strings.

encode_empty_data_packet(compression \\ :none)

@spec encode_empty_data_packet(ChDriver.Protocol.Block.Compressed.method()) ::
  iodata()

Encodes an empty Data packet (Client packet type 2): an empty external table, sent right after a Query packet to signal "no external tables / no input data" (required even for plain SELECTs).

The external-table name string is always sent plain -- only the block body (BlockInfo with is_overflows=0/bucket_num=-1, zero columns, zero rows) is ever wrapped. compression (:none (default) or :lz4) must match whatever was negotiated for this query via encode_query/2's :compression opt -- when :lz4, the block body is routed through ChDriver.Protocol.Block.Compressed.encode/2 before being sent, since the server (having been told compression is enabled in the preceding Query packet) expects this block wrapped in a compressed envelope, not sent plain.

encode_ping()

@spec encode_ping() :: iodata()

Encodes a Ping packet (Client packet type 4). No body -- just the packet-type varint. The server replies with a bare Pong (Server packet type 4, see ChDriver.Protocol.decode_packet/1) and nothing else.

encode_query(query_string, opts \\ [])

@spec encode_query(
  binary(),
  keyword()
) :: iodata()

Encodes a Query packet (Client packet type 1) for query_string.

Field order (see Connection::sendQuery / TCPHandler::receiveQuery in ClickHouse's own source, protocol revision 54469):

packet type (varint, 1)
query_id (string, empty lets the server generate one)
ClientInfo (see `encode_client_info/0`)
per-query settings (string "" -- empty settings list terminator)
interserver secret (string "" -- gated on
  DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET, always satisfied here)
query processing stage (varint, 2 = Complete)
compression (varint, 0 = disabled, 1 = enabled -- see below)
query string
query parameters (see `encode_query_parameters/1` -- gated on
  DBMS_MIN_PROTOCOL_VERSION_WITH_PARAMETERS, always satisfied here)

opts accepts:

  • :query_id -- defaults to "".
  • :params -- a list of {name :: binary, raw_text :: binary} pairs binding query_string's {name:Type} placeholders (see ChDriver.Params.text/1 for turning an Elixir value into raw_text and ChDriver.Params.escape_rounds/1 for the matching escape depth -- a bare 2-tuple defaults to the scalar depth of 2). Defaults to [].
  • :compression -- :none (default) or :lz4. This is not a codec choice for this one field alone: per TCPHandler::receiveQuery / TCPHandler::run in ClickHouse's own source (reverse-engineered empirically against a live 24.8 server -- see ChDriver.Connection's moduledoc-adjacent comments for how), setting this varint to 1 (Enable) tells the server that both directions of this query's block traffic are compressed from here on: the server wraps every Data/ProfileEvents block it sends back in a ChDriver.Protocol.Block.Compressed compressed envelope, and it expects the client's own Data packets (e.g. the empty external-table block sent by encode_empty_data_packet/1) to arrive wrapped the same way -- sending a plain block after declaring compression enabled leaves the server blocked forever trying to read a compression envelope header out of un-enveloped bytes. The envelope's method byte is chosen by whichever side is doing the sending independently (LZ4 for both client->server and server->client in this driver); Hello/Addendum and non-block packets (Progress, ProfileInfo, Exception, EndOfStream) are never wrapped, compression enabled or not.