Elixir API Guide

Complete guide to using Concord's Elixir API for read consistency, conditional updates, query language, and value compression.

Read Consistency Aliases

Concord accepts :eventual, :leader, and :strong as consistency names for API compatibility. The VSR engine currently routes every name through the same quorum-confirmed read barrier. All three are linearizable; they do not select different replicas, freshness guarantees, or performance profiles.

Configuration

Set the default accepted name in config/config.exs:

config :concord,
  default_read_consistency: :leader  # :eventual, :leader, or :strong

The setting preserves caller compatibility but does not change the current VSR read path.

Per-Operation Alias

Concord.get("key", consistency: :eventual)
Concord.get_many(["k1", "k2"], consistency: :strong)
Concord.get_with_ttl("key", consistency: :leader)
Concord.ttl("key", consistency: :eventual)
Concord.get_all(consistency: :strong)
Concord.get_all_with_ttl(consistency: :eventual)
Concord.status(consistency: :leader)

Telemetry Integration

Read telemetry includes the requested compatibility name:

:telemetry.attach(
  "my-handler",
  [:concord, :api, :get],
  fn _event, %{duration: duration}, %{consistency: consistency}, _config ->
    Logger.info("Read requested #{consistency}; duration=#{duration}ns")
  end,
  nil
)

The :consistency metadata value reports what the caller requested. It does not identify a distinct VSR execution path or guarantee.

Query Module Consistency

Concord.Query functions (keys/1, where/1, count/1, delete_where/1) do not currently accept a :consistency option. They delegate to Concord.get_all/1 and Concord.get_many/2 using the globally configured compatibility name:

config :concord,
  default_read_consistency: :leader  # :eventual, :leader, or :strong

Changing this name does not change the VSR read path or guarantee.

Index Lookups

Concord.Index.lookup/3 uses the replicated VSR query barrier. This ensures index lookups return results consistent with the latest writes.

Secondary Indexes

Secondary indexes enable efficient value-based lookups without scanning all keys. Concord maintains indexes automatically as values are inserted, updated, and deleted.

Extractor Specs

Indexes are defined using declarative extractor specs -- plain tuples that describe how to extract the indexed value from stored data. There are four supported spec types:

{:map_get, key} -- Flat map field

Extracts a single top-level key from a map value using Map.get/2:

# Given stored values like %{email: "alice@example.com", name: "Alice"}
Concord.Index.create("users_by_email", {:map_get, :email})

# After inserting data:
Concord.put("user:1", %{email: "alice@example.com", name: "Alice"})
Concord.put("user:2", %{email: "bob@example.com", name: "Bob"})

# Look up by indexed value:
{:ok, ["user:1"]} = Concord.Index.lookup("users_by_email", "alice@example.com")

{:nested, path} -- Nested map path

Extracts a value from a nested map structure using get_in/2:

# Given stored values like %{address: %{city: "Portland", state: "OR"}}
Concord.Index.create("users_by_city", {:nested, [:address, :city]})

Concord.put("user:1", %{name: "Alice", address: %{city: "Portland", state: "OR"}})
Concord.put("user:2", %{name: "Bob", address: %{city: "Seattle", state: "WA"}})

{:ok, ["user:1"]} = Concord.Index.lookup("users_by_city", "Portland")

{:identity} -- Raw value

Indexes the entire stored value as-is. Useful when values are simple scalars (strings, integers):

Concord.Index.create("by_status", {:identity})

Concord.put("order:1", "pending")
Concord.put("order:2", "shipped")
Concord.put("order:3", "pending")

{:ok, ["order:1", "order:3"]} = Concord.Index.lookup("by_status", "pending")

{:element, n} -- Tuple element

Extracts the nth element (zero-indexed) from a tuple value:

# Given stored values like {"electronics", "laptop", 999}
Concord.Index.create("by_category", {:element, 0})

Concord.put("product:1", {"electronics", "laptop", 999})
Concord.put("product:2", {"clothing", "shirt", 29})
Concord.put("product:3", {"electronics", "phone", 799})

{:ok, ["product:1", "product:3"]} = Concord.Index.lookup("by_category", "electronics")

Managing Indexes

Index names must be non-empty, valid UTF-8 binaries no longer than 255 bytes.

# Create an index
:ok = Concord.Index.create("users_by_email", {:map_get, :email})

# Create an index and rebuild it from all existing data
:ok = Concord.Index.create("users_by_email", {:map_get, :email}, reindex: true)

# List all indexes
{:ok, index_names} = Concord.Index.list()

# Look up keys by indexed value
{:ok, keys} = Concord.Index.lookup("users_by_email", "alice@example.com")

# Rebuild an index from scratch
:ok = Concord.Index.reindex("users_by_email")

# Drop an index
:ok = Concord.Index.drop("users_by_email")

Multi-Value Indexing

When the extractor returns a list, each element is indexed separately. This is useful for tagging:

Concord.Index.create("by_tag", {:map_get, :tags})

Concord.put("post:1", %{title: "Elixir Tips", tags: ["elixir", "programming"]})
Concord.put("post:2", %{title: "VSR Consensus", tags: ["distributed", "elixir"]})

{:ok, ["post:1", "post:2"]} = Concord.Index.lookup("by_tag", "elixir")
{:ok, ["post:2"]} = Concord.Index.lookup("by_tag", "distributed")

Anonymous Functions Are Rejected

Anonymous function extractors return {:error, :invalid_extractor} before a local or replicated command is issued. Always use declarative tuple specs.

# Rejected -- executable code cannot enter the replicated log
{:error, :invalid_extractor} =
  Concord.Index.create("by_email", fn user -> user.email end)

# GOOD -- safe for replication and snapshots
Concord.Index.create("by_email", {:map_get, :email})

Function references are not stable data: their behavior and validity can differ between nodes and code versions. Declarative specs are plain data (tuples of atoms, binaries, and integers) and are safe to replicate and deserialize.

Migrating Version-Zero Index and State Data

Keep command emission at version 0 until every replica can read version 1. Then inspect status and drop each legacy index using the exact returned name:

{:ok, %{storage: storage}} = Concord.status()

Enum.each(storage.legacy_indexes, fn exact_name ->
  :ok = Concord.Index.drop(exact_name)
end)

Do not recreate those indexes yet. Restart every node with command version 1, leaving no old reader in the quorum, and reconcile the legacy state representation once:

{:ok, {:ok, reconciliation}} =
  Concord.Engine.command(:reconcile_legacy_state)

{:ok, %{storage: storage}} = Concord.status()
false = storage.legacy_state_reconciliation_required
:current = storage.legacy_state_representation
0 = storage.legacy_state_conflict_count

Normal version-one commands are rejected until reconciliation completes. Now recreate every dropped index with a valid name and declarative extractor, and backfill it from the reconciled records:

:ok = Concord.Index.create("users_by_email", {:map_get, :email}, reindex: true)

Conditional Updates (Compare-and-Swap)

Atomic conditional operations for CAS, distributed locks, and optimistic concurrency control.

Compare-and-Swap with Expected Value

# Initialize counter
:ok = Concord.put("counter", 0)

# Read current value
{:ok, current} = Concord.get("counter")

# Update only if value hasn't changed
case Concord.put_if("counter", current + 1, expected: current) do
  :ok -> IO.puts("Counter updated to #{current + 1}")
  {:error, :condition_failed} -> IO.puts("Conflict, retrying...")
  {:error, :not_found} -> IO.puts("Key no longer exists")
end

# Conditional delete
:ok = Concord.put("session", "user-123")
:ok = Concord.delete_if("session", expected: "user-123")

Predicate-Based Conditions

# Version-based updates (optimistic locking)
:ok = Concord.put("config", %{version: 1, settings: %{enabled: true}})

new_config = %{version: 2, settings: %{enabled: false}}
:ok = Concord.put_if("config", new_config,
  condition: fn current -> current.version < new_config.version end
)

# Conditional delete based on age
cutoff = ~U[2025-01-01 00:00:00Z]
:ok = Concord.delete_if("temp_file",
  condition: fn file -> DateTime.compare(file.created_at, cutoff) == :lt end
)

Distributed Lock Pattern

defmodule DistributedLock do
  @lock_key "my_critical_resource"
  @lock_ttl 30

  def acquire(owner_id) do
    case Concord.get(@lock_key) do
      {:error, :not_found} ->
        Concord.put(@lock_key, owner_id, ttl: @lock_ttl)
        {:ok, :acquired}
      {:ok, ^owner_id} ->
        {:ok, :already_owned}
      {:ok, _other} ->
        {:error, :locked}
    end
  end

  def release(owner_id) do
    case Concord.delete_if(@lock_key, expected: owner_id) do
      :ok -> {:ok, :released}
      {:error, :condition_failed} -> {:error, :not_owner}
      {:error, :not_found} -> {:error, :not_locked}
    end
  end

  def with_lock(owner_id, fun) do
    case acquire(owner_id) do
      {:ok, _} ->
        try do
          fun.()
        after
          release(owner_id)
        end
      {:error, reason} ->
        {:error, reason}
    end
  end
end

Optimistic Concurrency Control

defmodule BankAccount do
  def transfer(from_account, to_account, amount) do
    {:ok, from_balance} = Concord.get(from_account)
    {:ok, to_balance} = Concord.get(to_account)

    if from_balance >= amount do
      with :ok <- Concord.put_if(from_account, from_balance - amount, expected: from_balance),
           :ok <- Concord.put_if(to_account, to_balance + amount, expected: to_balance) do
        {:ok, :transferred}
      else
        {:error, :condition_failed} ->
          transfer(from_account, to_account, amount)  # Retry
        error -> error
      end
    else
      {:error, :insufficient_funds}
    end
  end
end

API Options

Condition options (required, mutually exclusive):

• :expected — Exact value match (== comparison)
• :condition — Predicate function receiving current value

Additional options (for put_if/3):

• :ttl — TTL in seconds on success
• :timeout — Operation timeout in ms (default: 5000)

Return values:

• :ok — Condition met, operation succeeded
• {:error, :condition_failed} — Value doesn't match
• {:error, :not_found} — Key doesn't exist or expired
• {:error, :missing_condition} — No condition provided
• {:error, :conflicting_conditions} — Both :expected and :condition provided

TTL Interaction

Conditional operations treat expired keys as not found:

:ok = Concord.put("temp", "value", ttl: 1)
Process.sleep(2000)
{:error, :not_found} = Concord.put_if("temp", "new", expected: "value")

Query Language

Pattern matching, range queries, and filtering for efficient data retrieval.

Key Matching

# Prefix matching
{:ok, keys} = Concord.Query.keys(prefix: "user:")

# Suffix matching
{:ok, keys} = Concord.Query.keys(suffix: ":admin")

# Contains substring
{:ok, keys} = Concord.Query.keys(contains: "2024-02")

# Regex pattern
{:ok, keys} = Concord.Query.keys(pattern: ~r/user:\d{3}/)

Range Queries

# Lexicographic range (inclusive)
{:ok, keys} = Concord.Query.keys(range: {"user:100", "user:200"})

# Date range queries
{:ok, keys} = Concord.Query.keys(range: {"order:2024-01-01", "order:2024-12-31"})

Value Filtering

{:ok, pairs} = Concord.Query.where(
  prefix: "product:",
  filter: fn {_k, v} -> v.price > 100 end
)

{:ok, pairs} = Concord.Query.where(
  prefix: "user:",
  filter: fn {_k, v} -> v.age >= 30 and v.role == "admin" end
)

Pagination

{:ok, keys} = Concord.Query.keys(prefix: "user:", limit: 50)
{:ok, keys} = Concord.Query.keys(prefix: "user:", offset: 100, limit: 50)

Count and Delete

{:ok, count} = Concord.Query.count(prefix: "temp:")
{:ok, deleted_count} = Concord.Query.delete_where(prefix: "temp:")
{:ok, count} = Concord.Query.delete_where(range: {"old:2020-01-01", "old:2020-12-31"})

Combined Filters

{:ok, keys} = Concord.Query.keys(
  prefix: "user:",
  pattern: ~r/\d{3}/,
  limit: 10
)

Value Compression

Automatic compression for large values to reduce memory usage.

Configuration

Compression is enabled by default:

config :concord,
  compression: [
    enabled: true,
    algorithm: :zlib,        # :zlib, :gzip, or :none
    threshold_bytes: 1024,   # Compress values > 1KB
    level: 6                 # 0-9 (0=none, 9=max)
  ]

Transparent Operation

# Large value — automatically compressed on put
large_data = String.duplicate("x", 10_000)
Concord.put("large_key", large_data)

# Automatically decompressed on get
{:ok, value} = Concord.get("large_key")
# Returns original uncompressed value

Per-Operation Override

# Force compression regardless of size
Concord.put("small_key", "small value", compress: true)

# Disable compression for this operation
Concord.put("large_key", large_value, compress: false)

Compression does not change admission limits. Both the raw serialized value and its logical representation after expanding compression must fit the 16 MiB v1 value maximum, so forced, automatic, and disabled compression use the same logical policy. A local kv.max_value_bytes setting may lower that maximum but cannot raise it. Transactions have a separate 1,000,000-byte logical spec maximum, and every replicated command (including bulk puts and backup restore) has a 64 MiB logical aggregate maximum.

Raw compressed Erlang terms are a trusted embedded input. Decoding is bounded but can allocate terms and intern atom names; applications accepting untrusted network data should decode a bounded external schema and pass ordinary Elixir values to Concord rather than accepting compression envelopes directly.

Compression Statistics

stats = Concord.Compression.stats(large_data)
# %{
#   original_size: 10_047,
#   compressed_size: 67,
#   compression_ratio: 0.67,
#   savings_bytes: 9_980,
#   savings_percent: 99.33
# }

Performance

Trade-offs: ~5-15% CPU overhead, 60-98% memory reduction, ~0.1-1ms added latency.

API Reference

Core Operations

Concord.put(key, value, opts \\ [])
# Options: :timeout, :token, :ttl, :compress

Concord.get(key, opts \\ [])
# Returns: {:ok, value} | {:error, :not_found} | {:error, reason}
# Options: :timeout, :token, :consistency

Concord.delete(key, opts \\ [])
# Returns: :ok | {:error, reason}

Concord.get_all(opts \\ [])
# Returns: {:ok, map}

Concord.status(opts \\ [])
# Returns: {:ok, %{cluster: ..., storage: ..., node: ...}}

Concord.members()
# Returns: {:ok, [member_ids]}

Batch Operations

Concord.put_many([{key, value} | {key, value, ttl}], opts)
Concord.get_many([keys], opts)
Concord.delete_many([keys], opts)
Concord.touch_many([{key, ttl_seconds}], opts)

Max batch size: 500 items.

TTL Operations

Concord.put(key, value, ttl: seconds)
Concord.touch(key, additional_ttl_seconds, opts)
Concord.ttl(key, opts)
Concord.get_with_ttl(key, opts)
Concord.get_all_with_ttl(opts)

Conditional Operations

Concord.put_if(key, value, expected: current_value)
Concord.put_if(key, value, condition: fn current -> ... end)
Concord.delete_if(key, expected: current_value)
Concord.delete_if(key, condition: fn current -> ... end)

Common Options

• :timeout — Operation timeout in ms (default: 5000)
• :token — Authentication token (required when auth enabled)
• :consistency — Read consistency (:eventual, :leader, :strong)
• :ttl — Time-to-live in seconds
• :compress — Override auto-compression (true/false)

Error Types

:timeout              # Operation timed out
:unauthorized         # Invalid or missing auth token
:cluster_not_ready    # Cluster not initialized
:invalid_key          # Key validation failed
:not_found            # Key doesn't exist
:cluster_not_ready    # VSR cluster is not ready
:condition_failed     # Conditional update failed