Elixir API Guide

Complete guide to using Concord's Elixir API for read consistency, conditional updates, query language, and value compression.

Read Consistency Levels

Concord supports configurable read consistency levels per operation, allowing you to balance performance and data freshness.

Available Levels

:eventual — Fastest, eventually consistent

Concord.get("user:123", consistency: :eventual)

Reads from any available node. May return slightly stale data. Best for high-throughput reads, dashboards, analytics, and cached data.

:leader — Balanced (default)

Concord.get("user:123", consistency: :leader)
# Or simply:
Concord.get("user:123")

Reads from the leader node. Good balance between performance and freshness. Suitable for most application needs.

:strong — Linearizable

Concord.get("user:123", consistency: :strong)

Reads from leader with heartbeat verification. Most up-to-date. Use for critical financial data, security-sensitive operations, and strict consistency requirements.

Configuration

Set the default in config/config.exs:

config :concord,
  default_read_consistency: :leader  # :eventual, :leader, or :strong

All Read Operations Support Consistency

Concord.get("key", consistency: :eventual)
Concord.get_many(["k1", "k2"], consistency: :strong)
Concord.get_with_ttl("key", consistency: :leader)
Concord.ttl("key", consistency: :eventual)
Concord.get_all(consistency: :strong)
Concord.get_all_with_ttl(consistency: :eventual)
Concord.status(consistency: :leader)

Performance Characteristics

Read Load Balancing

With :eventual consistency, reads are automatically distributed across cluster members:

1..1000 |> Enum.each(fn i ->
  Concord.get("metric:#{i}", consistency: :eventual)
end)

Telemetry Integration

All read operations emit telemetry events with the consistency level:

:telemetry.attach(
  "my-handler",
  [:concord, :api, :get],
  fn _event, %{duration: duration}, %{consistency: consistency}, _config ->
    Logger.info("Read with #{consistency} consistency took #{duration}ns")
  end,
  nil
)

Query Consistency Levels

Each read consistency level maps to a different Ra query primitive, which determines the guarantees and latency characteristics of the read.

Ra Query Mapping

Per-Operation Override

Every read function in the Concord module accepts the :consistency option:

Concord.get("key", consistency: :strong)
Concord.get_many(["k1", "k2"], consistency: :eventual)
Concord.get_with_ttl("key", consistency: :leader)
Concord.ttl("key", consistency: :eventual)
Concord.get_all(consistency: :strong)
Concord.get_all_with_ttl(consistency: :eventual)
Concord.status(consistency: :leader)

Query Module Consistency

Concord.Query functions (keys/1, where/1, count/1, delete_where/1) do not currently accept a :consistency option. They delegate to Concord.get_all/1 and Concord.get_many/2 using whatever default consistency is configured globally. To control consistency for query operations, set the default in your config:

config :concord,
  default_read_consistency: :leader  # :eventual, :leader, or :strong

Index Lookups

Concord.Index.lookup/3 always uses :ra.consistent_query/3 (equivalent to :strong consistency) regardless of the global default. This ensures index lookups return results consistent with the latest writes.

Secondary Indexes

Secondary indexes enable efficient value-based lookups without scanning all keys. Concord maintains indexes automatically as values are inserted, updated, and deleted.

Extractor Specs

Indexes are defined using declarative extractor specs -- plain tuples that describe how to extract the indexed value from stored data. There are four supported spec types:

{:map_get, key} -- Flat map field

Extracts a single top-level key from a map value using Map.get/2:

# Given stored values like %{email: "alice@example.com", name: "Alice"}
Concord.Index.create("users_by_email", {:map_get, :email})

# After inserting data:
Concord.put("user:1", %{email: "alice@example.com", name: "Alice"})
Concord.put("user:2", %{email: "bob@example.com", name: "Bob"})

# Look up by indexed value:
{:ok, ["user:1"]} = Concord.Index.lookup("users_by_email", "alice@example.com")

{:nested, path} -- Nested map path

Extracts a value from a nested map structure using get_in/2:

# Given stored values like %{address: %{city: "Portland", state: "OR"}}
Concord.Index.create("users_by_city", {:nested, [:address, :city]})

Concord.put("user:1", %{name: "Alice", address: %{city: "Portland", state: "OR"}})
Concord.put("user:2", %{name: "Bob", address: %{city: "Seattle", state: "WA"}})

{:ok, ["user:1"]} = Concord.Index.lookup("users_by_city", "Portland")

{:identity} -- Raw value

Indexes the entire stored value as-is. Useful when values are simple scalars (strings, integers):

Concord.Index.create("by_status", {:identity})

Concord.put("order:1", "pending")
Concord.put("order:2", "shipped")
Concord.put("order:3", "pending")

{:ok, ["order:1", "order:3"]} = Concord.Index.lookup("by_status", "pending")

{:element, n} -- Tuple element

Extracts the nth element (zero-indexed) from a tuple value:

# Given stored values like {"electronics", "laptop", 999}
Concord.Index.create("by_category", {:element, 0})

Concord.put("product:1", {"electronics", "laptop", 999})
Concord.put("product:2", {"clothing", "shirt", 29})
Concord.put("product:3", {"electronics", "phone", 799})

{:ok, ["product:1", "product:3"]} = Concord.Index.lookup("by_category", "electronics")

Managing Indexes

# Create an index
:ok = Concord.Index.create("users_by_email", {:map_get, :email})

# Create an index and rebuild it from all existing data
:ok = Concord.Index.create("users_by_email", {:map_get, :email}, reindex: true)

# List all indexes
{:ok, index_names} = Concord.Index.list()

# Look up keys by indexed value
{:ok, keys} = Concord.Index.lookup("users_by_email", "alice@example.com")

# Rebuild an index from scratch
:ok = Concord.Index.reindex("users_by_email")

# Drop an index
:ok = Concord.Index.drop("users_by_email")

Multi-Value Indexing

When the extractor returns a list, each element is indexed separately. This is useful for tagging:

Concord.Index.create("by_tag", {:map_get, :tags})

Concord.put("post:1", %{title: "Elixir Tips", tags: ["elixir", "programming"]})
Concord.put("post:2", %{title: "Raft Consensus", tags: ["distributed", "elixir"]})

{:ok, ["post:1", "post:2"]} = Concord.Index.lookup("by_tag", "elixir")
{:ok, ["post:2"]} = Concord.Index.lookup("by_tag", "distributed")

Warning: No Anonymous Functions

Do not use anonymous functions as extractors. They cause :badfun errors on deserialization across code versions. Always use declarative tuple specs.

# BAD -- will break after code upgrades or on other cluster nodes
Concord.Index.create("by_email", fn user -> user.email end)

# GOOD -- safe for Raft replication and snapshots
Concord.Index.create("by_email", {:map_get, :email})

Anonymous functions are serialized into the Raft log and snapshots. When a node loads a snapshot produced by a different code version, the function reference is invalid and Ra raises a :badfun error. Declarative specs are plain data (tuples of atoms, binaries, and integers) and are always safe to deserialize.

Conditional Updates (Compare-and-Swap)

Atomic conditional operations for CAS, distributed locks, and optimistic concurrency control.

Compare-and-Swap with Expected Value

# Initialize counter
:ok = Concord.put("counter", 0)

# Read current value
{:ok, current} = Concord.get("counter")

# Update only if value hasn't changed
case Concord.put_if("counter", current + 1, expected: current) do
  :ok -> IO.puts("Counter updated to #{current + 1}")
  {:error, :condition_failed} -> IO.puts("Conflict, retrying...")
  {:error, :not_found} -> IO.puts("Key no longer exists")
end

# Conditional delete
:ok = Concord.put("session", "user-123")
:ok = Concord.delete_if("session", expected: "user-123")

Predicate-Based Conditions

# Version-based updates (optimistic locking)
:ok = Concord.put("config", %{version: 1, settings: %{enabled: true}})

new_config = %{version: 2, settings: %{enabled: false}}
:ok = Concord.put_if("config", new_config,
  condition: fn current -> current.version < new_config.version end
)

# Conditional delete based on age
cutoff = ~U[2025-01-01 00:00:00Z]
:ok = Concord.delete_if("temp_file",
  condition: fn file -> DateTime.compare(file.created_at, cutoff) == :lt end
)

Distributed Lock Pattern

defmodule DistributedLock do
  @lock_key "my_critical_resource"
  @lock_ttl 30

  def acquire(owner_id) do
    case Concord.get(@lock_key) do
      {:error, :not_found} ->
        Concord.put(@lock_key, owner_id, ttl: @lock_ttl)
        {:ok, :acquired}
      {:ok, ^owner_id} ->
        {:ok, :already_owned}
      {:ok, _other} ->
        {:error, :locked}
    end
  end

  def release(owner_id) do
    case Concord.delete_if(@lock_key, expected: owner_id) do
      :ok -> {:ok, :released}
      {:error, :condition_failed} -> {:error, :not_owner}
      {:error, :not_found} -> {:error, :not_locked}
    end
  end

  def with_lock(owner_id, fun) do
    case acquire(owner_id) do
      {:ok, _} ->
        try do
          fun.()
        after
          release(owner_id)
        end
      {:error, reason} ->
        {:error, reason}
    end
  end
end

Optimistic Concurrency Control

defmodule BankAccount do
  def transfer(from_account, to_account, amount) do
    {:ok, from_balance} = Concord.get(from_account)
    {:ok, to_balance} = Concord.get(to_account)

    if from_balance >= amount do
      with :ok <- Concord.put_if(from_account, from_balance - amount, expected: from_balance),
           :ok <- Concord.put_if(to_account, to_balance + amount, expected: to_balance) do
        {:ok, :transferred}
      else
        {:error, :condition_failed} ->
          transfer(from_account, to_account, amount)  # Retry
        error -> error
      end
    else
      {:error, :insufficient_funds}
    end
  end
end

API Options

Condition options (required, mutually exclusive):

• :expected — Exact value match (== comparison)
• :condition — Predicate function receiving current value

Additional options (for put_if/3):

• :ttl — TTL in seconds on success
• :timeout — Operation timeout in ms (default: 5000)

Return values:

• :ok — Condition met, operation succeeded
• {:error, :condition_failed} — Value doesn't match
• {:error, :not_found} — Key doesn't exist or expired
• {:error, :missing_condition} — No condition provided
• {:error, :conflicting_conditions} — Both :expected and :condition provided

TTL Interaction

Conditional operations treat expired keys as not found:

:ok = Concord.put("temp", "value", ttl: 1)
Process.sleep(2000)
{:error, :not_found} = Concord.put_if("temp", "new", expected: "value")

Query Language

Pattern matching, range queries, and filtering for efficient data retrieval.

Key Matching

# Prefix matching
{:ok, keys} = Concord.Query.keys(prefix: "user:")

# Suffix matching
{:ok, keys} = Concord.Query.keys(suffix: ":admin")

# Contains substring
{:ok, keys} = Concord.Query.keys(contains: "2024-02")

# Regex pattern
{:ok, keys} = Concord.Query.keys(pattern: ~r/user:\d{3}/)

Range Queries

# Lexicographic range (inclusive)
{:ok, keys} = Concord.Query.keys(range: {"user:100", "user:200"})

# Date range queries
{:ok, keys} = Concord.Query.keys(range: {"order:2024-01-01", "order:2024-12-31"})

Value Filtering

{:ok, pairs} = Concord.Query.where(
  prefix: "product:",
  filter: fn {_k, v} -> v.price > 100 end
)

{:ok, pairs} = Concord.Query.where(
  prefix: "user:",
  filter: fn {_k, v} -> v.age >= 30 and v.role == "admin" end
)

Pagination

{:ok, keys} = Concord.Query.keys(prefix: "user:", limit: 50)
{:ok, keys} = Concord.Query.keys(prefix: "user:", offset: 100, limit: 50)

Count and Delete

{:ok, count} = Concord.Query.count(prefix: "temp:")
{:ok, deleted_count} = Concord.Query.delete_where(prefix: "temp:")
{:ok, count} = Concord.Query.delete_where(range: {"old:2020-01-01", "old:2020-12-31"})

Combined Filters

{:ok, keys} = Concord.Query.keys(
  prefix: "user:",
  pattern: ~r/\d{3}/,
  limit: 10
)

Value Compression

Automatic compression for large values to reduce memory usage.

Configuration

Compression is enabled by default:

config :concord,
  compression: [
    enabled: true,
    algorithm: :zlib,        # :zlib (faster) or :gzip (better ratio)
    threshold_bytes: 1024,   # Compress values > 1KB
    level: 6                 # 0-9 (0=none, 9=max)
  ]

Transparent Operation

# Large value — automatically compressed on put
large_data = String.duplicate("x", 10_000)
Concord.put("large_key", large_data)

# Automatically decompressed on get
{:ok, value} = Concord.get("large_key")
# Returns original uncompressed value

Per-Operation Override

# Force compression regardless of size
Concord.put("small_key", "small value", compress: true)

# Disable compression for this operation
Concord.put("large_key", large_value, compress: false)

Compression Statistics

stats = Concord.Compression.stats(large_data)
# %{
#   original_size: 10_047,
#   compressed_size: 67,
#   compression_ratio: 0.67,
#   savings_bytes: 9_980,
#   savings_percent: 99.33
# }

Performance

Trade-offs: ~5-15% CPU overhead, 60-98% memory reduction, ~0.1-1ms added latency.

API Reference

Core Operations

Concord.put(key, value, opts \\ [])
# Options: :timeout, :token, :ttl, :compress

Concord.get(key, opts \\ [])
# Returns: {:ok, value} | {:error, :not_found} | {:error, reason}
# Options: :timeout, :token, :consistency

Concord.delete(key, opts \\ [])
# Returns: :ok | {:error, reason}

Concord.get_all(opts \\ [])
# Returns: {:ok, map}

Concord.status(opts \\ [])
# Returns: {:ok, %{cluster: ..., storage: ..., node: ...}}

Concord.members()
# Returns: {:ok, [member_ids]}

Batch Operations

Concord.put_many([{key, value} | {key, value, ttl}], opts)
Concord.get_many([keys], opts)
Concord.delete_many([keys], opts)
Concord.touch_many([{key, ttl_seconds}], opts)

Max batch size: 500 items.

TTL Operations

Concord.put(key, value, ttl: seconds)
Concord.touch(key, additional_ttl_seconds, opts)
Concord.ttl(key, opts)
Concord.get_with_ttl(key, opts)
Concord.get_all_with_ttl(opts)

Conditional Operations

Concord.put_if(key, value, expected: current_value)
Concord.put_if(key, value, condition: fn current -> ... end)
Concord.delete_if(key, expected: current_value)
Concord.delete_if(key, condition: fn current -> ... end)

Common Options

• :timeout — Operation timeout in ms (default: 5000)
• :token — Authentication token (required when auth enabled)
• :consistency — Read consistency (:eventual, :leader, :strong)
• :ttl — Time-to-live in seconds
• :compress — Override auto-compression (true/false)

Error Types

:timeout              # Operation timed out
:unauthorized         # Invalid or missing auth token
:cluster_not_ready    # Cluster not initialized
:invalid_key          # Key validation failed
:not_found            # Key doesn't exist
:noproc               # Ra process not running
:condition_failed     # Conditional update failed