Connection Pool

View Source

For production environments, using a connection pool is highly recommended. Pillar makes it easy to create and manage a pool of connections to your ClickHouse servers.

Benefits of Connection Pooling

  • Performance: Reusing existing connections reduces overhead
  • Resource Management: Controls the number of concurrent connections
  • Load Balancing: Distributes queries across multiple ClickHouse servers
  • Fault Tolerance: Handles server failures gracefully
  • Supervision: Automatically reconnects failed connections

Setting Up a Connection Pool

To create a connection pool, define a module that uses Pillar:

defmodule MyApp.ClickHouse do
  use Pillar,
    connection_strings: [
      "http://user:password@clickhouse-1:8123/database",
      "http://user:password@clickhouse-2:8123/database"
    ],
    name: __MODULE__,
    pool_size: 15
end

Then start the connection pool, typically in your application's supervision tree:

# In your application.ex
def start(_type, _args) do
  children = [
    # Other children...
    MyApp.ClickHouse
  ]
  
  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

Configuration Options

When setting up a connection pool, you can configure the following options:

OptionDescriptionDefault
connection_stringsList of ClickHouse server URLs(required)
nameName of the poolModule name
pool_sizeNumber of connections to maintain10
pool_timeoutTime to wait for a connection from the pool (ms)5000
timeoutDefault query timeout (ms)5000
max_overflowMaximum overflow connections~30% of pool_size

Example with all options:

defmodule MyApp.ClickHouse do
  use Pillar,
    connection_strings: [
      "http://user:password@clickhouse-1:8123/database",
      "http://user:password@clickhouse-2:8123/database"
    ],
    name: __MODULE__,
    pool_size: 20,
    pool_timeout: 10_000,
    timeout: 30_000
end

Using the Connection Pool

The Pillar-generated module provides functions that mirror the core Pillar API:

# Execute a SELECT query
{:ok, users} = MyApp.ClickHouse.select(
  "SELECT * FROM users WHERE age > {min_age}",
  %{min_age: 21}
)

# Execute an INSERT query
{:ok, _} = MyApp.ClickHouse.insert(
  "INSERT INTO events (user_id, event) VALUES ({user_id}, {event})",
  %{user_id: 123, event: "login"}
)

# Insert data using a map
{:ok, _} = MyApp.ClickHouse.insert_to_table(
  "users",
  %{
    id: 456,
    name: "John Doe",
    email: "john@example.com"
  }
)

Asynchronous Operations

One of the key benefits of using a connection pool is the ability to perform asynchronous operations:

# Async insert
MyApp.ClickHouse.async_insert(
  "INSERT INTO logs (event, timestamp) VALUES ({event}, {timestamp})",
  %{event: "page_view", timestamp: DateTime.utc_now()}
)

# Async table insert
MyApp.ClickHouse.async_insert_to_table(
  "logs",
  %{
    event: "page_view",
    user_id: 123,
    timestamp: DateTime.utc_now()
  }
)

Asynchronous operations:

  • Return immediately without waiting for a response
  • Are ideal for logging, metrics, and other non-critical writes
  • Improve application responsiveness
  • Reduce backpressure

High Availability and Load Balancing

When providing multiple connection strings, Pillar distributes queries across all available servers:

defmodule MyApp.ClickHouse do
  use Pillar,
    connection_strings: [
      "http://user:password@clickhouse-1:8123/database",
      "http://user:password@clickhouse-2:8123/database",
      "http://user:password@clickhouse-3:8123/database"
    ],
    pool_size: 15
end

This setup provides:

  • Load balancing across all servers
  • Automatic failover if a server becomes unavailable
  • Better query throughput

Performance Considerations

  • Pool Size: Should typically match your application's concurrency needs
  • Overflow: Provides flexibility during traffic spikes
  • Timeouts: Balance between responsiveness and resource utilization
  • Server Count: More servers improve throughput but require more connections

For a high-traffic application, consider:

defmodule MyApp.ClickHouse do
  use Pillar,
    connection_strings: [
      "http://user:password@clickhouse-1:8123/database",
      "http://user:password@clickhouse-2:8123/database",
      "http://user:password@clickhouse-3:8123/database"
    ],
    pool_size: 50,  # Larger pool
    pool_timeout: 5_000,  # Short timeout to fail fast
    timeout: 60_000  # Longer query timeout for complex queries
end