Troubleshooting
View SourceThis guide covers common issues you may encounter when using Pillar with ClickHouse, along with solutions and performance optimization tips.
Connection Issues
Failed to Connect to ClickHouse Server
Problem: Cannot establish a connection to the ClickHouse server.
Symptoms:
{:error, %Pillar.HttpClient.TransportError{reason: :econnrefused}}
Solutions:
Check server availability:
curl http://your-clickhouse-host:8123/ping
If this doesn't return "OK", the server might be down or unreachable.
Verify credentials: Ensure your username and password are correct in the connection string.
Check network connectivity:
- Verify your application can reach the ClickHouse host
- Check for firewall rules blocking port 8123
- Ensure the ClickHouse server is configured to accept external connections
Inspect server logs: Look at ClickHouse server logs for any errors related to connections or authentication.
Connection Timeout
Problem: Connection attempts to ClickHouse time out.
Symptoms:
{:error, %Pillar.HttpClient.TransportError{reason: :timeout}}
Solutions:
Increase timeout value:
conn = Pillar.Connection.new( "http://user:password@localhost:8123/database", %{timeout: 30_000} # 30 seconds )
Check server load: High server load can cause timeouts. Check ClickHouse's system metrics:
SELECT * FROM system.metrics
Network latency: Consider using a server in the same region/datacenter as your application to reduce latency.
Query Execution Issues
Invalid Query Syntax
Problem: SQL syntax errors in queries.
Symptoms:
{:error, "Code: 62. DB::Exception: Syntax error: failed at position 10 (...)"
Solutions:
Validate SQL syntax: Test your query directly against ClickHouse using the HTTP interface or clickhouse-client.
Check parameter placeholders: Ensure all parameter placeholders in the query have corresponding values in the params map:
# Correct Pillar.query(conn, "SELECT * FROM users WHERE id = {id}", %{id: 123}) # Incorrect - missing parameter Pillar.query(conn, "SELECT * FROM users WHERE id = {id}", %{})
ClickHouse SQL specifics: Remember that ClickHouse SQL dialect has some differences from standard SQL.
Out of Memory Errors
Problem: Query fails due to insufficient memory.
Symptoms:
{:error, "Code: 241. DB::Exception: Memory limit (for query) exceeded: ..."
Solutions:
Limit query resources:
Pillar.query(conn, "SELECT * FROM large_table", %{}, %{ max_memory_usage: 10000000000, # 10GB max_bytes_before_external_sort: 5000000000 # 5GB })
Add query limits:
Pillar.select(conn, "SELECT * FROM huge_table LIMIT 1000", %{})
Use sampling:
Pillar.select(conn, "SELECT * FROM huge_table SAMPLE 0.1", %{})
Optimize schema: Ensure your tables are using appropriate engines and sorting keys.
Data Insertion Issues
Bulk Insert Failures
Problem: Bulk insert operations fail with data type errors.
Symptoms:
{:error, "Code: 53. DB::Exception: Cannot parse input: expected )..."
Solutions:
Validate data types: Ensure all values match the expected column types in your table.
Check schema compatibility:
# Get table structure {:ok, structure} = Pillar.query(conn, "DESCRIBE TABLE your_table")
Handle NULL values appropriately: ClickHouse has strict NULL handling. Use default values where appropriate.
Break into smaller batches: If inserting large amounts of data, break it into smaller batches:
Enum.chunk_every(records, 1000) |> Enum.each(fn batch -> Pillar.insert_to_table(conn, "your_table", batch) end)
Async Insert Issues
Problem: Async inserts silently fail without error reporting.
Solutions:
Implement logging: While async operations don't return errors, you can log them in your application:
# Start process monitoring ClickHouse logs def monitor_clickhouse_errors do # Poll error log table periodically schedule_check() # ... end
Use buffered inserts with error callbacks:
defmodule MyApp.InsertBuffer do use Pillar.BulkInsertBuffer, pool: MyApp.ClickHouse, table_name: "events", on_errors: &__MODULE__.handle_errors/2 def handle_errors(error, records) do Logger.error("Failed insert: #{inspect(error)}") # Save failed records for retry end end
Pool Connection Issues
Pool Timeout
Problem: Timeout when trying to get a connection from the pool.
Symptoms:
{:error, :timeout}
Solutions:
Increase pool size:
defmodule MyApp.ClickHouse do use Pillar, connection_strings: ["http://..."], pool_size: 30 # Increase from default 10 end
Increase pool timeout:
defmodule MyApp.ClickHouse do use Pillar, connection_strings: ["http://..."], pool_timeout: 10_000 # 10 seconds end
Check for connection leaks: Ensure all operations properly return connections to the pool.
Monitor pool utilization:
# Check current pool status :poolboy.status(MyApp.ClickHouse.name())
Schema and Migration Issues
Migration Failures
Problem: ClickHouse migrations fail to apply.
Solutions:
Check migration syntax: Validate SQL statements directly in ClickHouse.
Use transaction-like behavior: ClickHouse doesn't support transactions, but you can implement rollback functions:
defmodule Pillar.Migrations.CreateComplexSchema do def up do [ "CREATE TABLE table_1 (...)", "CREATE TABLE table_2 (...)" ] end def down do [ "DROP TABLE table_2", "DROP TABLE table_1" ] end end
Inspect migration state: Query the
pillar_migrations
table to see which migrations were applied:{:ok, applied} = Pillar.select(conn, "SELECT * FROM pillar_migrations")
Type Conversion Issues
DateTime Conversion Problems
Problem: DateTime values are not correctly stored or retrieved.
Solutions:
Ensure timezone configuration:
# In config.exs config :elixir, :time_zone_database, Tzdata.TimeZoneDatabase
Use explicit formatting:
# When inserting datetime_str = DateTime.utc_now() |> DateTime.to_iso8601() Pillar.query(conn, "INSERT INTO events VALUES ({timestamp})", %{timestamp: datetime_str})
Customize DateTime conversion:
defimpl Pillar.TypeConvert.ToClickHouse, for: DateTime do def convert(datetime) do Calendar.strftime(datetime, "%Y-%m-%d %H:%M:%S") end end
JSON Data Issues
Problem: Complex JSON structures don't serialize correctly.
Solutions:
Pre-serialize JSON:
json_string = Jason.encode!(complex_map) Pillar.query(conn, "INSERT INTO logs VALUES ({data})", %{data: json_string})
Use ClickHouse JSON functions:
Pillar.select(conn, "SELECT JSONExtractString(data, 'key') FROM logs")
Performance Optimization
Query Performance
Use proper table engines:
- MergeTree family for most analytical workloads
- ReplacingMergeTree for data that needs to be updated
- SummingMergeTree for pre-aggregated data
Optimize ORDER BY clause:
- Include columns frequently used in WHERE clauses
- Put high-cardinality columns first
Use materialized views:
CREATE MATERIALIZED VIEW user_stats ENGINE = SummingMergeTree() ORDER BY (date, user_id) AS SELECT toDate(timestamp) AS date, user_id, count() AS event_count FROM events GROUP BY date, user_id
Add query hints:
Pillar.query(conn, """ SELECT * FROM large_table WHERE date = {date} SETTINGS max_threads = 8, max_memory_usage = 20000000000 """, %{date: Date.utc_today()})
Insertion Performance
Batch inserts appropriately:
- Too small: Network overhead
- Too large: Memory pressure
- Typical sweet spot: 1,000-10,000 records per batch
Use asynchronous inserts for non-critical data:
MyApp.ClickHouse.async_insert_to_table("logs", records)
Optimize for bulk loads:
Pillar.query(conn, """ INSERT INTO table SELECT * FROM input('format CSV') """, %{}, %{input_format_allow_errors_num: 10})
Consider data locality: For distributed ClickHouse clusters, insert to the appropriate shard.
Connection Optimization
Pool sizing formula:
pool_size = min( max_expected_concurrent_queries, (available_memory / avg_query_memory_usage) )
Reuse connections: Avoid creating new connections for every request.
Load balance with multiple servers:
defmodule MyApp.ClickHouse do use Pillar, connection_strings: [ "http://clickhouse-1:8123/db", "http://clickhouse-2:8123/db", "http://clickhouse-3:8123/db" ] end
Use HTTP keep-alive: Pillar's Tesla Mint adapter uses HTTP keep-alive by default, reducing connection overhead.
Debugging Tips
Enable Verbose Logging
# In config/config.exs
config :logger, level: :debug
# In your application
Logger.configure(level: :debug)
Inspect Queries with EXPLAIN
Pillar.query(conn, "EXPLAIN SELECT * FROM large_table WHERE date = {date}", %{date: "2023-01-01"})
Profile Queries
Pillar.query(conn, "EXPLAIN ANALYZE SELECT * FROM large_table WHERE date = {date}", %{date: "2023-01-01"})
Monitor System Tables
# Check currently running queries
Pillar.select(conn, "SELECT * FROM system.processes")
# Check query log for slow queries
Pillar.select(conn, """
SELECT
query_duration_ms,
query
FROM system.query_log
WHERE type = 'QueryFinish'
ORDER BY query_duration_ms DESC
LIMIT 10
""")
Server-Side Logging
Increase ClickHouse server log verbosity if needed:
<!-- In config.xml -->
<logger>
<level>trace</level>
</logger>
Common Error Codes
Code | Description | Common Causes |
---|---|---|
53 | Cannot parse input | Data type mismatch, invalid format |
60 | Invalid query | Syntax error, invalid table name |
62 | Syntax error | Malformed SQL |
81 | Database not found | Wrong database name, incorrect URL |
149 | Tables differ in structure | Schema mismatch in INSERT |
192 | Unknown table | Table doesn't exist |
241 | Memory limit exceeded | Query requires too much memory |
252 | Required server restart | DDL operation needs server restart |