Streaming utilities for ClickHouse gRPC operations.
Provides helpers for:
- Building input streams for
ExecuteQueryWithStreamInput - Processing output streams from
ExecuteQueryWithStreamOutput - Bidirectional streaming with
ExecuteQueryWithStreamIO
Summary
Functions
Collects all results from an output stream into a single Result.
Builds a lazy stream of QueryInfo messages for a streaming insert.
Streams rows as maps from an output stream.
Creates a stream that yields parsed Result structs from gRPC results.
Streams rows from an output stream, yielding individual rows.
Transforms an output stream, applying a function to each Result chunk.
Functions
@spec collect_output(Enumerable.t()) :: {:ok, Huginn.Clickhouse.Result.t()} | {:error, term()}
Collects all results from an output stream into a single Result.
@spec input_stream(String.t(), Enumerable.t(), keyword()) :: Enumerable.t(struct())
Builds a lazy stream of QueryInfo messages for a streaming insert.
The first message carries the SQL query plus the first data chunk; every
later message carries only data. The source is enumerated exactly once, so
non-restartable sources (File.stream!/2, a Stream.map/2 with side
effects) are safe.
Options
:chunk_size- Elements per chunk (default: 1000):format- Input data format (default: "TabSeparated")- All options from
Query.build/2
Elements that are already encoded binaries are concatenated verbatim, so the
caller's own framing is preserved. Structured rows are encoded according to
:format.
next_query_info is set by Huginn.Clickhouse.Client.insert_stream/3, which
knows which message is last.
Examples
# Structured rows, encoded by this module
rows = [["a", "1"], ["b", "2"], ["c", "3"]]
input_stream("INSERT INTO t FORMAT TabSeparated", rows)
# Pre-encoded chunks, passed through untouched
File.stream!("data.csv", [], 65_536)
|> then(&input_stream("INSERT INTO t FORMAT CSV", &1, chunk_size: 1))
@spec map_stream(Enumerable.t()) :: Enumerable.t(map())
Streams rows as maps from an output stream.
@spec parse_stream(Enumerable.t()) :: Enumerable.t({:ok, Huginn.Clickhouse.Result.t()} | {:error, term()})
Creates a stream that yields parsed Result structs from gRPC results.
@spec row_stream(Enumerable.t()) :: Enumerable.t(list())
Streams rows from an output stream, yielding individual rows.
Useful for processing large result sets row by row.
@spec transform_output(Enumerable.t(), (Huginn.Clickhouse.Result.t() -> term())) :: Enumerable.t()
Transforms an output stream, applying a function to each Result chunk.