Pillar (Pillar v0.39.0)

View Source

Elixir client for ClickHouse, a fast open-source Online Analytical Processing (OLAP) database management system.

Overview

Pillar provides a straightforward way to interact with ClickHouse databases from Elixir, supporting both direct connections and connection pools. It handles query building, parameter substitution, and response parsing.

The library offers the following core features:

  • Direct connection to ClickHouse servers
  • Connection pooling for improved performance
  • Synchronous and asynchronous query execution
  • Structured data insertion
  • Parameter substitution
  • Query result parsing
  • Migration support

Direct Usage

For simple usage with a direct connection:

# Create a connection
conn = Pillar.Connection.new("http://user:password@localhost:8123/database")

# Execute a query with parameter substitution
sql = "SELECT count(*) FROM users WHERE lastname = {lastname}"
params = %{lastname: "Smith"}

{:ok, result} = Pillar.query(conn, sql, params)
# => [%{"count(*)" => 347}]

Connection Pool Usage

For production applications, using a connection pool is recommended:

defmodule MyApp.ClickHouse do
  use Pillar,
    connection_strings: [
      "http://user:password@host-1:8123/database",
      "http://user:password@host-2:8123/database"
    ],
    name: __MODULE__,
    pool_size: 15
end

# Start the connection pool
MyApp.ClickHouse.start_link()

# Execute queries using the pool
{:ok, result} = MyApp.ClickHouse.select("SELECT * FROM users WHERE id = {id}", %{id: 123})

Asynchronous Operations

For non-blocking inserts:

# Using direct connection
Pillar.async_insert(conn, "INSERT INTO events (user_id, event) VALUES ({user_id}, {event})", %{
  user_id: 42,
  event: "login"
})

# Using connection pool
MyApp.ClickHouse.async_insert("INSERT INTO events (user_id, event) VALUES ({user_id}, {event})", %{
  user_id: 42,
  event: "login"
})

Summary

Functions

Defines a ClickHouse connection pool module.

Async version of insert/4 that doesn't wait for a response.

Async version of query/4 that doesn't wait for a response.

Executes an INSERT statement against a ClickHouse database.

Inserts data into a specified table, automatically generating the INSERT statement.

Executes an arbitrary SQL query against a ClickHouse database.

Executes a SELECT query and returns the results in a structured format.

Functions

__using__(options)

(macro)

Defines a ClickHouse connection pool module.

This macro sets up a module that manages a pool of ClickHouse connections. The generated module provides functions to execute queries against the connection pool, handling connection acquisition and release automatically.

Options

  • :connection_strings - Required. A list of ClickHouse connection URLs
  • :name - Optional. The name of the pool (defaults to "PillarPool")
  • :pool_size - Optional. The number of connections to maintain (defaults to 10)
  • :pool_timeout - Optional. Timeout for acquiring a connection from the pool (defaults to 5000ms)
  • :timeout - Optional. Default query timeout (defaults to 5000ms)

Generated Functions

The macro generates the following functions in the module:

Example

defmodule MyApp.ClickHouse do
  use Pillar,
    connection_strings: [
      "http://user:password@host-1:8123/database",
      "http://user:password@host-2:8123/database"
    ],
    name: __MODULE__,
    pool_size: 15,
    pool_timeout: 10_000,
    timeout: 30_000
end

async_insert(connection, query, params \\ %{}, options \\ %{})

Async version of insert/4 that doesn't wait for a response.

This function sends an INSERT query to ClickHouse but doesn't wait for a response, making it suitable for fire-and-forget operations where you don't need to confirm the result. Available only when using connection pools through the use Pillar macro.

async_insert_to_table(connection, table_name, record_or_records, options \\ %{})

Async version of insert_to_table/4 that doesn't wait for a response.

This function inserts data into a specified table but doesn't wait for a response, making it suitable for fire-and-forget operations where you don't need to confirm the result. Available only when using connection pools through the use Pillar macro.

async_query(connection, query, params \\ %{}, options \\ %{})

Async version of query/4 that doesn't wait for a response.

This function sends a query to ClickHouse but doesn't wait for a response, making it suitable for fire-and-forget operations where you don't need to confirm the result. Available only when using connection pools through the use Pillar macro.

insert(connection, query, params \\ %{}, options \\ %{})

Executes an INSERT statement against a ClickHouse database.

Parameters

  • connection - A Pillar.Connection struct representing the database connection
  • query - The SQL query string with optional parameter placeholders in curly braces {param}
  • params - A map of parameters to be substituted in the query (default: %{})
  • options - A map of options to customize the request (default: %{})

Options

  • :timeout - Request timeout in milliseconds (default: 5000)
  • Other options are passed to the connection's URL builder

Returns

  • {:ok, result} on success
  • {:error, reason} on failure

Examples

Pillar.insert(conn, "INSERT INTO users (name, email) VALUES ({name}, {email})", %{
  name: "John Doe",
  email: "john@example.com"
})

insert_to_table(connection, table_name, record_or_records, options \\ %{})

Inserts data into a specified table, automatically generating the INSERT statement.

This function allows inserting one record (as a map) or multiple records (as a list of maps) into a ClickHouse table without having to manually construct the SQL INSERT statement.

Parameters

  • connection - A Pillar.Connection struct representing the database connection
  • table_name - The name of the table to insert data into
  • record_or_records - A map or list of maps representing the data to insert
  • options - A map of options to customize the request (default: %{})

Options

  • :timeout - Request timeout in milliseconds (default: 5000)
  • :query_options - Options for the query builder (e.g., %{format: :json})

Returns

  • {:ok, result} on success
  • {:error, reason} on failure

Examples

Single record:

Pillar.insert_to_table(conn, "users", %{
  name: "John Doe",
  email: "john@example.com",
  created_at: DateTime.utc_now()
})

Multiple records:

Pillar.insert_to_table(conn, "users", [
  %{name: "John Doe", email: "john@example.com"},
  %{name: "Jane Smith", email: "jane@example.com"}
])

query(connection, query, params \\ %{}, options \\ %{})

Executes an arbitrary SQL query against a ClickHouse database.

This function can be used for any type of query (SELECT, CREATE, ALTER, etc.) but doesn't format the response as JSON. For SELECT queries with formatted results, consider using select/4 instead.

Parameters

  • connection - A Pillar.Connection struct representing the database connection
  • query - The SQL query string with optional parameter placeholders in curly braces {param}
  • params - A map of parameters to be substituted in the query (default: %{})
  • options - A map of options to customize the request (default: %{})

Options

  • :timeout - Request timeout in milliseconds (default: 5000)
  • Other options are passed to the connection's URL builder

Returns

  • {:ok, result} on success
  • {:error, reason} on failure

Examples

Pillar.query(conn, "CREATE TABLE users (id UInt64, name String, email String) ENGINE = MergeTree() ORDER BY id")

Pillar.query(conn, "ALTER TABLE users ADD COLUMN created_at DateTime")

select(connection, query, params \\ %{}, options \\ %{})

Executes a SELECT query and returns the results in a structured format.

This function appends 'FORMAT JSON' to the query, which makes ClickHouse return the results in JSON format, which are then parsed into Elixir data structures.

Parameters

  • connection - A Pillar.Connection struct representing the database connection
  • query - The SQL query string with optional parameter placeholders in curly braces {param}
  • params - A map of parameters to be substituted in the query (default: %{})
  • options - A map of options to customize the request (default: %{})

Options

  • :timeout - Request timeout in milliseconds (default: 5000)
  • Other options are passed to the connection's URL builder

Returns

  • {:ok, result} on success, where result is a list of maps representing rows
  • {:error, reason} on failure

Examples

{:ok, users} = Pillar.select(conn, 
  "SELECT id, name, email FROM users WHERE signup_date > {date} LIMIT {limit}", 
  %{date: "2023-01-01", limit: 100}
)

# users is a list of maps like:
# [
#   %{"id" => 1, "name" => "John Doe", "email" => "john@example.com"},
#   %{"id" => 2, "name" => "Jane Smith", "email" => "jane@example.com"}
# ]