Migrating from v0.4 to v0.5

ExDatalog v0.5.0 adds aggregate constraints, BEAM callback predicates, magic-sets program transformation, and a query planner. All v0.4.1 code works unchanged — the new features are opt-in extensions to the existing DSL and builder API.

This guide covers each new feature and the few deprecations.

Backward Compatibility

Every program that compiles and runs under v0.4.1 continues to do so under v0.5.0. No APIs were removed; no default behaviours changed. The only breaking-ish change is that agg(...) in DSL rule bodies now raises with a redirect message instead of a generic error (see below).

Aggregates

v0.5.0 introduces four aggregate constraint operations: count, sum, min, and max. Unlike comparison or arithmetic constraints, aggregates operate over the full set of bindings for a rule, grouped by the non-aggregated variables.

DSL Syntax

Aggregates appear in the rule body, not the head. The result variable is then used in the head:

defmodule HR do
  use ExDatalog.Schema

  relation :employee do
    field :name, :atom
    field :dept, :atom
    field :salary, :integer
  end

  relation :dept_size do
    field :dept, :atom
    field :count, :integer
  end

  fact employee(:alice, :eng, 120)
  fact employee(:bob, :eng, 95)
  fact employee(:carol, :ops, 80)

  rule dept_size(Dept, N) do
    employee(Name, Dept, Salary)
    count(Name, N)
  end
end

{:ok, knowledge} = HR.materialize()
# knowledge contains dept_size(:eng, 2), dept_size(:ops, 1)

Supported aggregate operations in the DSL: count/2, sum/2, min/2, max/2. The first argument is the input variable; the second is the result variable that the engine binds after group-and-reduce.

Builder API

Use Constraint.from_tuple/1 with an aggregate tuple:

alias ExDatalog.{Program, Rule, Atom, Term, Constraint}

program =
  Program.new()
  |> Program.add_relation("employee", [:atom, :atom, :integer])
  |> Program.add_relation("dept_size", [:atom, :integer])
  |> Program.add_fact("employee", [:alice, :eng, 120])
  |> Program.add_fact("employee", [:bob, :eng, 95])

aggregate_constraint = Constraint.from_tuple({:count, Term.var("Name"), Term.var("N")})

program =
  Program.add_rule(program,
    Rule.new(
      Atom.new("dept_size", [Term.var("Dept"), Term.var("N")]),
      [{:positive, Atom.new("employee", [Term.var("Name"), Term.var("Dept"), Term.var("Salary")])}],
      [aggregate_constraint]
    )
  )

Constraint Table

Aggregate constraints use the %Constraint{op: :count | :sum | :min | :max, left: input_term, right: nil, result: {:var, name}} shape. The right field is always nil for aggregates; the grouping key is inferred from the rule's non-aggregated variables.

BEAM Callbacks

BEAM callbacks let rule bodies call deterministic, side-effect-free Elixir functions as predicates. The engine isolates callbacks with a configurable timeout (default 100 ms) and treats exceptions/timeouts as filtered bindings.

predicate/5 Macro

Declare callbacks in the DSL module using predicate/5:

defmodule MyRules do
  use ExDatalog.Schema

  relation :person do
    field :name, :atom
    field :age, :integer
  end

  relation :senior do
    field :name, :atom
  end

  predicate :senior?, MyRules, :senior?, [:integer], :boolean

  fact person(:alice, 70)
  fact person(:bob, 25)

  rule senior(Name) do
    person(Name, Age)
    senior?(Age)
  end

  def senior?(age), do: age >= 65
end

{:ok, knowledge} = MyRules.materialize()

Arguments:

• name — predicate name used in rule bodies (e.g. senior?).
• module / function — the Elixir module and function to call.
• arg_types — declared argument types (informational; length sets the callback arity).
• return_type — :boolean (filter) or :value (binds last argument as result).

Value-Returning Callbacks

For :value predicates, the last argument in the rule-body call is the result variable; remaining arguments are passed to the function:

defmodule ScoreRules do
  use ExDatalog.Schema

  relation :player do
    field :name, :atom
    field :raw, :integer
  end

  relation :ranked do
    field :name, :atom
    field :score, :integer
  end

  predicate :compute_score, ScoreRules, :compute_score, [:integer], :value

  fact player(:alice, 80)
  fact player(:bob, 40)

  rule ranked(Name, Score) do
    player(Name, Raw)
    compute_score(Raw, Score)
  end

  def compute_score(raw), do: div(raw * 3, 2)
end

Builder API: Callback.new/4

alias ExDatalog.{Callback, Term}

# Boolean filter callback
cb = Callback.new(MyMod, :adult?, [Term.var("Age")])
#=> %Callback{module: MyMod, function: :adult?, args: [{:var, "Age"}], result: nil}

# Value-returning callback
cb = Callback.new(MyMod, :score, [Term.var("X")], Term.var("S"))
#=> %Callback{module: MyMod, function: :score, args: [{:var, "X"}], result: {:var, "S"}}

Add a callback to a rule's body as {:callback, %Callback{}}:

rule =
  Rule.new(
    Atom.new("senior", [Term.var("Name")]),
    [
      {:positive, Atom.new("person", [Term.var("Name"), Term.var("Age")])},
      {:callback, Callback.new(MyMod, :senior?, [Term.var("Age")])}
    ]
  )

Safety Contract

Callbacks must be deterministic and side-effect free. The engine enforces only timeout and exception isolation — determinism and purity are caller contracts.

Magic Sets

Magic-sets is a program transformation for demand-driven (goal-directed) evaluation. Instead of computing the full least fixpoint, it rewrites the program so that only facts relevant to a query goal are derived.

materialize/2 Options

Pass strategy: :magic_sets and goal: {relation, pattern} to materialize/2:

{:ok, knowledge} =
  ExDatalog.materialize(program,
    strategy: :magic_sets,
    goal: {"path", [:alice, :_]}
  )

The same options work from the DSL:

{:ok, knowledge} =
  MySchema.materialize(
    strategy: :magic_sets,
    goal: {"path", [:alice, :_]}
  )

When strategy: :magic_sets is specified without a :goal, the engine falls back to full semi-naive evaluation. When the program is outside the supported scope (negation, aggregates), the transformation returns {:fallback, reason} and the engine also falls back silently.

Scope (Experimental)

• Positive recursive programs only.
• A single goal.
• Ground (constant) bound positions.

Programs outside this scope always fall back to full semi-naive evaluation, never producing incorrect results.

Planner

The planner sits between compiled IR and the evaluation engine. It produces an ExDatalog.Planner.Plan describing the chosen strategy, planned strata, joins, and predicates.

plan/2

alias ExDatalog.{Program, Rule, Atom, Term, Compiler, Planner}

{:ok, ir} =
  Program.new()
  |> Program.add_relation("edge", [:atom, :atom])
  |> Program.add_relation("path", [:atom, :atom])
  |> Program.add_rule(
    Rule.new(
      Atom.new("path", [Term.var("X"), Term.var("Y")]),
      [{:positive, Atom.new("edge", [Term.var("X"), Term.var("Y")])}]
    )
  )
  |> Compiler.compile()

{:ok, plan} = Planner.plan(ir)
plan.strategy   #=> :semi_naive
plan.joins      #=> [%Join{relation: "edge", ...}]
plan.predicates #=> []

{:ok, plan} = Planner.plan(ir, strategy: :magic_sets, goal: {"path", [:alice, :_]})
plan.strategy   #=> :magic_sets

explain_plan/1,2

Returns a human-readable description of the plan:

Planner.explain_plan(program)
#=> "Strategy: semi_naive\n  Stratum 0: 1 rule(s), relations: edge, path\nJoins: 1\nPredicates: 0"

Planner.explain_plan(program, strategy: :magic_sets, goal: {"path", [:alice, :_]})
#=> "Strategy: magic_sets\n  Stratum 0: 1 rule(s), relations: magic_path_bf, edge, path\nJoins: 2\nPredicates: 0"

explain_plan/1 validates and compiles the program first; returns an error string if compilation fails.

Telemetry Events

plan/2 emits the following telemetry events:

New Capabilities Fields

ExDatalog.Capabilities has two new boolean fields:

Both participate in merge/2 (AND semantics) and satisfies?/2:

caps = %ExDatalog.Capabilities{aggregate_constraints: true, beam_callbacks: false}
ExDatalog.Capabilities.satisfies?(caps, aggregate_constraints: true)
#=> true
ExDatalog.Capabilities.satisfies?(caps, beam_callbacks: true)
#=> false

Deprecated / Changed

agg(...) raises with a redirect

Using agg(...) in a DSL rule head or body now raises ExDatalog.DSL.CompileError with a redirect message:

rule dept_size(D, agg(:count, E)) do
  employee(E, D)
end
#=> ** (ExDatalog.DSL.CompileError) use count/sum/min/max for aggregates (e.g. `count(X, N)`)

Previously, agg(...) raised a generic "unsupported term" error. The new message directs you to the correct aggregate syntax. Use count/2, sum/2, min/2, or max/2 in the rule body instead.

New Dependencies

• benchee ~> 1.3 — dev-only, not included in production builds. Used for benchmarking the planner and magic-sets transformations.