Ecto.Adapters.ClickHouse.Migration (clickhouse_adapter_ecto v0.2.0)

Copy Markdown

Validated builders for ClickHouse-specific migration column types that have no direct Ecto.Migration.add/3 equivalent.

Ecto.Migration.add/3 rejects any Ecto.Type/Ecto.ParameterizedType module as a column type, so a type like Ecto.Adapters.ClickHouse.Types.FixedString can never be given to it directly -- only atoms, quoted atoms, composite tuples, and references(...) are accepted. These builders produce the quoted-atom form add/3 does accept, with the parameter validated up front instead of only surfacing as a ClickHouse DDL error at migration time:

add(:code, Ecto.Adapters.ClickHouse.Migration.fixed_string(16))
add(:status, Ecto.Adapters.ClickHouse.Migration.low_cardinality(:string))

FixedString(N) additionally has a full Ecto.ParameterizedType -- Ecto.Adapters.ClickHouse.Types.FixedString -- for the schema side, where add/3's restriction doesn't apply:

field :code, Ecto.Adapters.ClickHouse.Types.FixedString, size: 16

LowCardinality(T) doesn't get one: it's transparent to callers (decoded to the same Elixir value T would decode to on its own), so a plain field :status, :string already works. Map(K, V) gets neither a builder nor a ParameterizedType -- use the quoted atom directly:

add(:m, :"Map(String, UInt32)")

Building a table's options: string

table/2's options: takes a single raw string for everything after the column list -- engine, ORDER BY, PARTITION BY, SETTINGS k = v, .... table_options/1 builds that string from a keyword list instead of requiring migration authors to hand-interpolate and quote it (SETTINGS in particular gets tedious once it accumulates several key/value pairs):

create table(:events, primary_key: false,
  options: Ecto.Adapters.ClickHouse.Migration.table_options(
    engine: "MergeTree",
    partition_by: "toYYYYMM(inserted_at)",
    order_by: "id"
  )
) do
  add :id, :id, primary_key: true
  add :inserted_at, :utc_datetime
end

It also resolves {:system, "ENV_VAR"} settings values from the environment at the time the helper runs (migration-run time, i.e. mix ecto.migrate, not compile time) -- handy for credentials that shouldn't be committed as a literal string in a migration file, like the Kafka-engine example in Ecto.Adapters.ClickHouse.DDL's moduledoc:

execute("""
CREATE TABLE events_queue (id UInt64, payload String)
#{Ecto.Adapters.ClickHouse.Migration.table_options(
  engine: "Kafka",
  settings: [
    kafka_broker_list: {:system, "KAFKA_BROKER_LIST"},
    kafka_topic_list: "events",
    kafka_group_name: "events_consumer",
    kafka_format: "JSONEachRow"
  ]
)}
""")

which renders (given KAFKA_BROKER_LIST=kafka:9092 in the environment the migration runs in) as:

ENGINE = Kafka SETTINGS kafka_broker_list = 'kafka:9092', kafka_topic_list = 'events', kafka_group_name = 'events_consumer', kafka_format = 'JSONEachRow'

Summary

Types

A single :settings value for table_options/1.

Functions

Builds the quoted-atom migration type for FixedString(size).

Builds the quoted-atom migration type for LowCardinality(inner_type), where inner_type is any Ecto type Ecto.Adapters.ClickHouse.DDL's column_type!/1 already knows how to map to a ClickHouse column type (e.g. :string, :integer, :uuid, {:array, :string}).

Builds the options string table/2's options: expects (everything after the column list: ENGINE, PARTITION BY, ORDER BY, SETTINGS), from a keyword list instead of a hand-quoted raw string.

Types

setting_value()

@type setting_value() :: String.t() | number() | boolean() | {:system, String.t()}

A single :settings value for table_options/1.

A plain string is rendered single-quoted (ClickHouse String-typed settings, e.g. kafka_broker_list); a number or boolean is rendered unquoted (ClickHouse numeric/boolean-ish settings, e.g. kafka_num_consumers). {:system, "ENV_VAR"} is resolved from the environment at call time (migration-run time) via System.fetch_env!/1 and then rendered single-quoted, same as a literal string.

Functions

fixed_string(size)

@spec fixed_string(pos_integer()) :: atom()

Builds the quoted-atom migration type for FixedString(size).

add(:code, Ecto.Adapters.ClickHouse.Migration.fixed_string(16))

Raises ArgumentError if size is not a positive integer.

low_cardinality(inner_type)

@spec low_cardinality(term()) :: atom()

Builds the quoted-atom migration type for LowCardinality(inner_type), where inner_type is any Ecto type Ecto.Adapters.ClickHouse.DDL's column_type!/1 already knows how to map to a ClickHouse column type (e.g. :string, :integer, :uuid, {:array, :string}).

add(:status, Ecto.Adapters.ClickHouse.Migration.low_cardinality(:string))

Raises ArgumentError (via column_type!/1) if inner_type isn't a type this adapter's migration DDL knows how to map.

table_options(opts)

@spec table_options(keyword()) :: String.t()

Builds the options string table/2's options: expects (everything after the column list: ENGINE, PARTITION BY, ORDER BY, SETTINGS), from a keyword list instead of a hand-quoted raw string.

Options

  • :engine (required) -- the engine name/clause, e.g. "MergeTree" or a full engine expression like "Kafka".
  • :partition_by -- rendered as PARTITION BY <value>.
  • :order_by -- rendered as ORDER BY <value>.
  • :settings -- a keyword list of key: value pairs, rendered as SETTINGS key1 = 'value1', key2 = value2, .... String values are single-quoted; numbers and booleans are not. A value can also be {:system, "ENV_VAR"} to interpolate an environment variable resolved when the migration runs -- see the moduledoc's Kafka example.

Clause order in the rendered string (ENGINE · PARTITION BY · ORDER BY · SETTINGS) matches ClickHouse's own CREATE TABLE clause order; only clauses that were given are included.

iex> Ecto.Adapters.ClickHouse.Migration.table_options(engine: "MergeTree", order_by: "id")
"ENGINE = MergeTree ORDER BY id"

iex> Ecto.Adapters.ClickHouse.Migration.table_options(
...>   engine: "MergeTree",
...>   partition_by: "toYYYYMM(inserted_at)",
...>   order_by: "id",
...>   settings: [index_granularity: 8192]
...> )
"ENGINE = MergeTree PARTITION BY toYYYYMM(inserted_at) ORDER BY id SETTINGS index_granularity = 8192"

Raises ArgumentError if :engine is missing, if an unrecognized top-level option key is given, if :settings isn't a keyword list, if a setting's value isn't a string/number/boolean/{:system, _}, or if a {:system, "ENV_VAR"} setting's environment variable isn't set.