ChDriver. Protocol. Messages
(ch_driver v0.1.0)
Copy Markdown
Per-message byte layouts for the ClickHouse native protocol.
ChDriver.Protocol owns packet-type dispatch (mapping a leading varint to
the right decoder) and revision-gating predicates; this module owns the
actual encoding/decoding of every message body -- Hello, Addendum, Query
(+ ClientInfo), the empty Data packet, Ping, Exception, Progress, and
ProfileInfo.
Summary
Functions
Builds a ClientHello struct advertising this driver's identity, for the
given default_database, user, and password.
This driver's advertised protocol revision (@client_revision).
Decodes a ServerHello packet from the front of binary, including its
leading packet-type varint (expected to be 0).
Encodes the Addendum sent immediately after receiving ServerHello, when
ChDriver.Protocol.addendum_required?/0 is true. This is not a full
packet -- just a raw length-prefixed string (the quota key; we always
send empty), with no leading packet-type varint. See
TCPHandler::receiveAddendum in ClickHouse's Server/TCPHandler.cpp.
Encodes a Cancel packet (Client packet type 3). No body -- just the
packet-type varint. Tells the server to stop processing the
currently-running query on this connection; the client must keep
reading (Data/Progress/ProfileEvents blocks already in flight before
the server notices the cancellation may still arrive) until it
observes :end_of_stream -- see ChDriver.Connection.cancel_stream/2,
the only caller, which does exactly that drain after sending this.
Encodes a ClientHello struct into the plaintext bytes sent to open a
connection: packet-type varint (0), then client name/version/revision
and the default_database/user/password length-prefixed strings.
Encodes an empty Data packet (Client packet type 2): an empty external table, sent right after a Query packet to signal "no external tables / no input data" (required even for plain SELECTs).
Encodes a Ping packet (Client packet type 4). No body -- just the
packet-type varint. The server replies with a bare Pong (Server packet
type 4, see ChDriver.Protocol.decode_packet/1) and nothing else.
Encodes a Query packet (Client packet type 1) for query_string.
Functions
@spec client_hello(binary(), binary(), binary()) :: ChDriver.Protocol.ClientHello.t()
Builds a ClientHello struct advertising this driver's identity, for the
given default_database, user, and password.
@spec client_revision() :: non_neg_integer()
This driver's advertised protocol revision (@client_revision).
@spec decode_server_hello(binary()) :: {:ok, ChDriver.Protocol.ServerHello.t(), binary()} | {:incomplete, binary()} | {:error, term()}
Decodes a ServerHello packet from the front of binary, including its
leading packet-type varint (expected to be 0).
Returns {:ok, %ServerHello{}, rest}, {:incomplete, binary} if more
bytes are needed, or {:error, reason} if the packet type doesn't match
Hello (e.g. the server sent an Exception instead, most likely due to bad
credentials).
@spec encode_addendum() :: iodata()
Encodes the Addendum sent immediately after receiving ServerHello, when
ChDriver.Protocol.addendum_required?/0 is true. This is not a full
packet -- just a raw length-prefixed string (the quota key; we always
send empty), with no leading packet-type varint. See
TCPHandler::receiveAddendum in ClickHouse's Server/TCPHandler.cpp.
CRITICAL: ClickHouse requires this immediately after ServerHello, before
any Query packet, whenever the client's advertised revision is >=
DBMS_MIN_PROTOCOL_VERSION_WITH_ADDENDUM (54458, see
ChDriver.Protocol.addendum_required?/0) -- which this driver's fixed
revision always is. Omitting this step silently desyncs every byte sent
afterwards: the server blocks reading it right after ServerHello, so the
next bytes we send -- e.g. the start of a Query packet -- get consumed
as this string instead, which surfaces as a bizarre unrelated "Empty
query" (or worse) error from the server despite an otherwise-correct
Query packet.
@spec encode_cancel() :: iodata()
Encodes a Cancel packet (Client packet type 3). No body -- just the
packet-type varint. Tells the server to stop processing the
currently-running query on this connection; the client must keep
reading (Data/Progress/ProfileEvents blocks already in flight before
the server notices the cancellation may still arrive) until it
observes :end_of_stream -- see ChDriver.Connection.cancel_stream/2,
the only caller, which does exactly that drain after sending this.
@spec encode_client_hello(ChDriver.Protocol.ClientHello.t()) :: iodata()
Encodes a ClientHello struct into the plaintext bytes sent to open a
connection: packet-type varint (0), then client name/version/revision
and the default_database/user/password length-prefixed strings.
@spec encode_empty_data_packet(ChDriver.Protocol.Block.Compressed.method()) :: iodata()
Encodes an empty Data packet (Client packet type 2): an empty external table, sent right after a Query packet to signal "no external tables / no input data" (required even for plain SELECTs).
The external-table name string is always sent plain -- only the block
body (BlockInfo with is_overflows=0/bucket_num=-1, zero columns, zero
rows) is ever wrapped. compression (:none (default) or :lz4) must
match whatever was negotiated for this query via encode_query/2's
:compression opt -- when :lz4, the block body is routed through
ChDriver.Protocol.Block.Compressed.encode/2 before being sent, since the server (having
been told compression is enabled in the preceding Query packet) expects
this block wrapped in a compressed envelope, not sent plain.
@spec encode_ping() :: iodata()
Encodes a Ping packet (Client packet type 4). No body -- just the
packet-type varint. The server replies with a bare Pong (Server packet
type 4, see ChDriver.Protocol.decode_packet/1) and nothing else.
Encodes a Query packet (Client packet type 1) for query_string.
Field order (see Connection::sendQuery / TCPHandler::receiveQuery in
ClickHouse's own source, protocol revision 54469):
packet type (varint, 1)
query_id (string, empty lets the server generate one)
ClientInfo (see `encode_client_info/0`)
per-query settings (string "" -- empty settings list terminator)
interserver secret (string "" -- gated on
DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET, always satisfied here)
query processing stage (varint, 2 = Complete)
compression (varint, 0 = disabled, 1 = enabled -- see below)
query string
query parameters (see `encode_query_parameters/1` -- gated on
DBMS_MIN_PROTOCOL_VERSION_WITH_PARAMETERS, always satisfied here)opts accepts:
:query_id-- defaults to"".:params-- a list of{name :: binary, raw_text :: binary}pairs bindingquery_string's{name:Type}placeholders (seeChDriver.Params.text/1for turning an Elixir value intoraw_textandChDriver.Params.escape_rounds/1for the matching escape depth -- a bare 2-tuple defaults to the scalar depth of 2). Defaults to[].:compression--:none(default) or:lz4. This is not a codec choice for this one field alone: perTCPHandler::receiveQuery/TCPHandler::runin ClickHouse's own source (reverse-engineered empirically against a live 24.8 server -- seeChDriver.Connection's moduledoc-adjacent comments for how), setting this varint to 1 (Enable) tells the server that both directions of this query's block traffic are compressed from here on: the server wraps every Data/ProfileEvents block it sends back in aChDriver.Protocol.Block.Compressedcompressed envelope, and it expects the client's own Data packets (e.g. the empty external-table block sent byencode_empty_data_packet/1) to arrive wrapped the same way -- sending a plain block after declaring compression enabled leaves the server blocked forever trying to read a compression envelope header out of un-enveloped bytes. The envelope's method byte is chosen by whichever side is doing the sending independently (LZ4 for both client->server and server->client in this driver); Hello/Addendum and non-block packets (Progress, ProfileInfo, Exception, EndOfStream) are never wrapped, compression enabled or not.