Pillar (Pillar v0.39.0)
View SourceElixir client for ClickHouse, a fast open-source Online Analytical Processing (OLAP) database management system.
Overview
Pillar provides a straightforward way to interact with ClickHouse databases from Elixir, supporting both direct connections and connection pools. It handles query building, parameter substitution, and response parsing.
The library offers the following core features:
- Direct connection to ClickHouse servers
- Connection pooling for improved performance
- Synchronous and asynchronous query execution
- Structured data insertion
- Parameter substitution
- Query result parsing
- Migration support
Direct Usage
For simple usage with a direct connection:
# Create a connection
conn = Pillar.Connection.new("http://user:password@localhost:8123/database")
# Execute a query with parameter substitution
sql = "SELECT count(*) FROM users WHERE lastname = {lastname}"
params = %{lastname: "Smith"}
{:ok, result} = Pillar.query(conn, sql, params)
# => [%{"count(*)" => 347}]
Connection Pool Usage
For production applications, using a connection pool is recommended:
defmodule MyApp.ClickHouse do
use Pillar,
connection_strings: [
"http://user:password@host-1:8123/database",
"http://user:password@host-2:8123/database"
],
name: __MODULE__,
pool_size: 15
end
# Start the connection pool
MyApp.ClickHouse.start_link()
# Execute queries using the pool
{:ok, result} = MyApp.ClickHouse.select("SELECT * FROM users WHERE id = {id}", %{id: 123})
Asynchronous Operations
For non-blocking inserts:
# Using direct connection
Pillar.async_insert(conn, "INSERT INTO events (user_id, event) VALUES ({user_id}, {event})", %{
user_id: 42,
event: "login"
})
# Using connection pool
MyApp.ClickHouse.async_insert("INSERT INTO events (user_id, event) VALUES ({user_id}, {event})", %{
user_id: 42,
event: "login"
})
Summary
Functions
Defines a ClickHouse connection pool module.
Async version of insert/4
that doesn't wait for a response.
Async version of insert_to_table/4
that doesn't wait for a response.
Async version of query/4
that doesn't wait for a response.
Executes an INSERT statement against a ClickHouse database.
Inserts data into a specified table, automatically generating the INSERT statement.
Executes an arbitrary SQL query against a ClickHouse database.
Executes a SELECT query and returns the results in a structured format.
Functions
Defines a ClickHouse connection pool module.
This macro sets up a module that manages a pool of ClickHouse connections. The generated module provides functions to execute queries against the connection pool, handling connection acquisition and release automatically.
Options
:connection_strings
- Required. A list of ClickHouse connection URLs:name
- Optional. The name of the pool (defaults to "PillarPool"):pool_size
- Optional. The number of connections to maintain (defaults to 10):pool_timeout
- Optional. Timeout for acquiring a connection from the pool (defaults to 5000ms):timeout
- Optional. Default query timeout (defaults to 5000ms)
Generated Functions
The macro generates the following functions in the module:
start_link/1
- Starts the connection poolselect/3
- Executes a SELECT query with JSON formattingquery/3
- Executes an arbitrary SQL queryinsert/3
- Executes an INSERT statementinsert_to_table/3
- Inserts data into a specified tableasync_query/3
- Asynchronously executes a queryasync_insert/3
- Asynchronously executes an INSERT statementasync_insert_to_table/3
- Asynchronously inserts data into a specified table
Example
defmodule MyApp.ClickHouse do
use Pillar,
connection_strings: [
"http://user:password@host-1:8123/database",
"http://user:password@host-2:8123/database"
],
name: __MODULE__,
pool_size: 15,
pool_timeout: 10_000,
timeout: 30_000
end
Async version of insert/4
that doesn't wait for a response.
This function sends an INSERT query to ClickHouse but doesn't wait for a response,
making it suitable for fire-and-forget operations where you don't need to confirm
the result. Available only when using connection pools through the use Pillar
macro.
Async version of insert_to_table/4
that doesn't wait for a response.
This function inserts data into a specified table but doesn't wait for a response,
making it suitable for fire-and-forget operations where you don't need to confirm
the result. Available only when using connection pools through the use Pillar
macro.
Async version of query/4
that doesn't wait for a response.
This function sends a query to ClickHouse but doesn't wait for a response,
making it suitable for fire-and-forget operations where you don't need to confirm
the result. Available only when using connection pools through the use Pillar
macro.
Executes an INSERT statement against a ClickHouse database.
Parameters
connection
- APillar.Connection
struct representing the database connectionquery
- The SQL query string with optional parameter placeholders in curly braces{param}
params
- A map of parameters to be substituted in the query (default:%{}
)options
- A map of options to customize the request (default:%{}
)
Options
:timeout
- Request timeout in milliseconds (default:5000
)- Other options are passed to the connection's URL builder
Returns
{:ok, result}
on success{:error, reason}
on failure
Examples
Pillar.insert(conn, "INSERT INTO users (name, email) VALUES ({name}, {email})", %{
name: "John Doe",
email: "john@example.com"
})
Inserts data into a specified table, automatically generating the INSERT statement.
This function allows inserting one record (as a map) or multiple records (as a list of maps) into a ClickHouse table without having to manually construct the SQL INSERT statement.
Parameters
connection
- APillar.Connection
struct representing the database connectiontable_name
- The name of the table to insert data intorecord_or_records
- A map or list of maps representing the data to insertoptions
- A map of options to customize the request (default:%{}
)
Options
:timeout
- Request timeout in milliseconds (default:5000
):query_options
- Options for the query builder (e.g.,%{format: :json}
)
Returns
{:ok, result}
on success{:error, reason}
on failure
Examples
Single record:
Pillar.insert_to_table(conn, "users", %{
name: "John Doe",
email: "john@example.com",
created_at: DateTime.utc_now()
})
Multiple records:
Pillar.insert_to_table(conn, "users", [
%{name: "John Doe", email: "john@example.com"},
%{name: "Jane Smith", email: "jane@example.com"}
])
Executes an arbitrary SQL query against a ClickHouse database.
This function can be used for any type of query (SELECT, CREATE, ALTER, etc.)
but doesn't format the response as JSON. For SELECT queries with formatted
results, consider using select/4
instead.
Parameters
connection
- APillar.Connection
struct representing the database connectionquery
- The SQL query string with optional parameter placeholders in curly braces{param}
params
- A map of parameters to be substituted in the query (default:%{}
)options
- A map of options to customize the request (default:%{}
)
Options
:timeout
- Request timeout in milliseconds (default:5000
)- Other options are passed to the connection's URL builder
Returns
{:ok, result}
on success{:error, reason}
on failure
Examples
Pillar.query(conn, "CREATE TABLE users (id UInt64, name String, email String) ENGINE = MergeTree() ORDER BY id")
Pillar.query(conn, "ALTER TABLE users ADD COLUMN created_at DateTime")
Executes a SELECT query and returns the results in a structured format.
This function appends 'FORMAT JSON' to the query, which makes ClickHouse return the results in JSON format, which are then parsed into Elixir data structures.
Parameters
connection
- APillar.Connection
struct representing the database connectionquery
- The SQL query string with optional parameter placeholders in curly braces{param}
params
- A map of parameters to be substituted in the query (default:%{}
)options
- A map of options to customize the request (default:%{}
)
Options
:timeout
- Request timeout in milliseconds (default:5000
)- Other options are passed to the connection's URL builder
Returns
{:ok, result}
on success, where result is a list of maps representing rows{:error, reason}
on failure
Examples
{:ok, users} = Pillar.select(conn,
"SELECT id, name, email FROM users WHERE signup_date > {date} LIMIT {limit}",
%{date: "2023-01-01", limit: 100}
)
# users is a list of maps like:
# [
# %{"id" => 1, "name" => "John Doe", "email" => "john@example.com"},
# %{"id" => 2, "name" => "Jane Smith", "email" => "jane@example.com"}
# ]