Huginn.Clickhouse.Stream (Huginn v0.5.0)

Copy Markdown View Source

Streaming utilities for ClickHouse gRPC operations.

Provides helpers for:

  • Building input streams for ExecuteQueryWithStreamInput
  • Processing output streams from ExecuteQueryWithStreamOutput
  • Bidirectional streaming with ExecuteQueryWithStreamIO

Summary

Functions

Collects all results from an output stream into a single Result.

Builds a lazy stream of QueryInfo messages for a streaming insert.

Streams rows as maps from an output stream.

Creates a stream that yields parsed Result structs from gRPC results.

Streams rows from an output stream, yielding individual rows.

Transforms an output stream, applying a function to each Result chunk.

Functions

collect_output(stream)

@spec collect_output(Enumerable.t()) ::
  {:ok, Huginn.Clickhouse.Result.t()} | {:error, term()}

Collects all results from an output stream into a single Result.

input_stream(sql, data_enum, opts \\ [])

@spec input_stream(String.t(), Enumerable.t(), keyword()) :: Enumerable.t(struct())

Builds a lazy stream of QueryInfo messages for a streaming insert.

The first message carries the SQL query plus the first data chunk; every later message carries only data. The source is enumerated exactly once, so non-restartable sources (File.stream!/2, a Stream.map/2 with side effects) are safe.

Options

  • :chunk_size - Elements per chunk (default: 1000)
  • :format - Input data format (default: "TabSeparated")
  • All options from Query.build/2

Elements that are already encoded binaries are concatenated verbatim, so the caller's own framing is preserved. Structured rows are encoded according to :format.

next_query_info is set by Huginn.Clickhouse.Client.insert_stream/3, which knows which message is last.

Examples

# Structured rows, encoded by this module
rows = [["a", "1"], ["b", "2"], ["c", "3"]]
input_stream("INSERT INTO t FORMAT TabSeparated", rows)

# Pre-encoded chunks, passed through untouched
File.stream!("data.csv", [], 65_536)
|> then(&input_stream("INSERT INTO t FORMAT CSV", &1, chunk_size: 1))

map_stream(stream)

@spec map_stream(Enumerable.t()) :: Enumerable.t(map())

Streams rows as maps from an output stream.

parse_stream(stream)

@spec parse_stream(Enumerable.t()) ::
  Enumerable.t({:ok, Huginn.Clickhouse.Result.t()} | {:error, term()})

Creates a stream that yields parsed Result structs from gRPC results.

row_stream(stream)

@spec row_stream(Enumerable.t()) :: Enumerable.t(list())

Streams rows from an output stream, yielding individual rows.

Useful for processing large result sets row by row.

transform_output(stream, fun)

@spec transform_output(Enumerable.t(), (Huginn.Clickhouse.Result.t() -> term())) ::
  Enumerable.t()

Transforms an output stream, applying a function to each Result chunk.