Dynamic queries

Ecto was designed from the ground up to have an expressive query API that leverages Elixir syntax to write queries that are pre-compiled for performance and safety. When building queries, we may use the keywords syntax

import Ecto.Query

from p in Post,
  where: p.author == "José" and p.category == "Elixir",
  where: p.published_at > ^minimum_date,
  order_by: [desc: p.published_at]

or the pipe-based one

import Ecto.Query

Post
|> where([p], p.author == "José" and p.category == "Elixir")
|> where([p], p.published_at > ^minimum_date)
|> order_by([p], desc: p.published_at)

While many developers prefer the pipe-based syntax, having to repeat the binding p made it quite verbose compared to the keyword one. Furthermore, the compile-time aspect of Ecto queries was at odds with building queries dynamically.

Imagine for example a web application that provides search functionality on top of existing posts. The user should be able to specify multiple criteria, such as the author name, the post category, publishing interval, etc. In this case, Ecto's approach would be to process the parameters into regular data structures and then build the query as late as possible.

Focusing on data structures

Ecto provides a simpler API for both keyword and pipe based queries by making data structures first-class. Let's see an example:

from p in Post,
  where: [author: "José", category: "Elixir"],
  where: p.published_at > ^minimum_date,
  order_by: [desc: :published_at]

and

Post
|> where(author: "José", category: "Elixir")
|> where([p], p.published_at > ^minimum_date)
|> order_by(desc: :published_at)

Notice how we were able to ditch the p selector in most expressions. In Ecto, all constructs, from select and order_by to where and group_by, accept data structures as input. The data structure can be specified at compile-time, as above, and also dynamically at runtime, shown below:

where = [author: "José", category: "Elixir"]
order_by = [desc: :published_at]
Post
|> where(^where)
|> where([p], p.published_at > ^minimum_date)
|> order_by(^order_by)

The advantage of interpolating data structures is that we can decouple the processing of parameters from the query generation. Note however not all expressions can be converted to data structures. Since where converts a key-value to a key == value comparison, order-based comparisons such as p.published_at > ^minimum_date still need to be written as part of the query as before.

The dynamic macro

For cases where we cannot rely on data structures but still desire to build queries dynamically, Ecto includes the Ecto.Query.dynamic/2 macro.

In order to understand how the dynamic macro works let's write a filter/1 function using both data structures and the dynamic macro:

def filter(params) do
  Post
  |> order_by(^filter_order_by(params["order_by"]))
  |> where(^filter_where(params))
  |> where(^filter_published_at(params["published_at"]))
end

def filter_order_by("published_at_desc"), do: [desc: :published_at]
def filter_order_by("published_at"),      do: [asc:  :published_at]
def filter_order_by(_),                   do: []

def filter_where(params) do
  for key <- [:author, :category],
      value = params[Atom.to_string(key)],
      do: {key, value}
end

def filter_published_at(date) when is_binary(date),
  do: dynamic([p], p.published_at > ^date)
def filter_published_at(_date),
  do: true

The dynamic macro allows us to build dynamic expressions that are later interpolated into the query. dynamic expressions can also be interpolated into dynamic expressions, allowing developers to build complex expressions dynamically without hassle.

Because we were able to break our problem into smaller functions that receive regular data structures, we can use all the tools available in Elixir to work with data. For handling the order_by parameter, it may be best to simply pattern match on the order_by parameter. For building the where clause, we can traverse the list of known keys and convert them to the format expected by Ecto. For complex conditions, we use the dynamic macro.

Testing also becomes simpler as we can test each function in isolation, even when using dynamic queries:

test "filter published at based on the given date" do
  assert inspect(filter_published_at("2010-04-17")) ==
         "dynamic([p], p.published_at > ^\"2010-04-17\")"
  assert inspect(filter_published_at(nil)) ==
         "true"
end