View Source Cinema (cinema v0.1.0)
Cinema
Cinema is a simple Elixir framework for managing incremental materialized views entirely in Elixir/Ecto.
Installation
Cinema can be installed by adding cinema
to your list of dependencies in mix.exs
:
def deps do
[
{:cinema, "~> 0.1.0"}
]
end
Cinema has an optional dependency on Oban Pro as an alternate runtime for materializing projection graphs. Oban Pro support is automatically enabled if Hex
detects the oban
repo in your global setup.
Please see the Oban Pro documentation for more information on how to install and configure Oban Pro.
Usage
Cinema introduces two basic concepts:
- Projections: A projection is a behaviour that allows you to declaratively define instructions in Elixir for how to derive rows to write into your materialized views. They statically define all
c:inputs/0
(other projections, if needed),c:output/1
(usually anEcto.Query
or stream passed into subsequent projections), and ac:derivation/2
callback which is run in tandem with all inputs to produce output. - Lenses: A lens is a struct which contains filters and other options which can be used to modify the "scope" of what a projection is required to derive. In simple use cases, you can think of lenses as automatically applying filters such as:
where: x.org_id == ^org_id
to the outputs of all input projections automatically.
When you want to actually incrementally rematerialize a view, you create a Cinema.Lens.t()
(or a simple keyword list for simple filters), and pass that into the Cinema.project/3
function like so:
iex> Cinema.project(MyApp.Projections.AccountsReceivable, [org_id: 123, date: ~D[2022-01-01]])
[
%MyApp.Projections.AccountsReceivable{
org_id: 123,
date: ~D[2022-01-01],
...
},
...
]
Projections generally define their own Ecto.Schema
internally and can also be queried directly -- note that this will not rematerialize any dependencies or rows in the table you're querying:
iex> MyApp.Repo.all(MyApp.Projections.AccountsReceivable)
[
%MyApp.Projections.AccountsReceivable{
org_id: 123,
date: ~D[2022-01-01],
...
},
...
]
Projections can include other projections as inputs, and Cinema will automatically rematerialize those projections as needed. For example, if AccountsReceivable
depends on Invoices
, Cinema will automatically rematerialize Invoices
before rematerializing AccountsReceivable
.
Projection graphs usually begin with "virtual" projections that have no inputs or derivation/2
callback, instead only outputting either an Ecto.Query
or stream which is passed directly through any Cinema.Lens.t()
and into the next projection in the graph.
Cinema does this by building a DAG of all projections and their dependencies. Cinema will likewise try to run any projections in parallel where possible. A minimal example of a projection looks like the following:
defmodule MyApp.Projections.Accounts do
use Cinema.Projection, virtual?: true
@impl Cinema.Projection
def inputs, do: []
@impl Cinema.Projection
def output, do: from(a in "accounts", select: a.id)
end
defmodule MyApp.Projections.AccountsReceivable do
use Cinema.Projection,
conflict_target: [:account_id],
required_fields: [:account_id],
on_conflict: :replace_all,
read_repo: MyApp.Repo.Replica,
write_repo: MyApp.Repo,
timeout: :timer.minutes(5),
alias Cinema.Projection
alias MyApp.Projections.Accounts
@primary_key false
schema "accounts_receivable" do
field(account_id:, :id)
field(total, :integer)
timestamps()
end
@impl Cinema.Projection
def inputs, do: [Accounts]
@impl Cinema.Projection
def output, do: from(a in "accounts_receivable", select: a)
@impl Cinema.Projection
def derivation({Accounts, stream}, lens) do
Projection.dematerialize(lens)
stream
|> Stream.chunk_every(2000)
|> Stream.map(&from x in MyApp.Invoice, where: x.account_id in ^&1, select: %{account_id: x.account_id, total: sum(x.total)})
|> Stream.map(&Projection.materialize/1)
|> Stream.run()
end
end
Configuration
Currently, Cinema lets you configure the following options:
:engine
- The runtime to use for executing projection graphs. Defaults toCinema.Engine.Task
.:async
- Whether to run projections asynchronously. Defaults totrue
.
Additional configuration options can be implemented on a projection-by-projection basis, please see the docs for the Cinema.Projection
behaviour for more information.
License
Cinema is released under the MIT License.
Summary
Functions
Projects a module that implements the Cinema.Projection
behaviour.
Functions
@spec project(Cinema.Projection.t(), Cinema.Projection.Lens.t(), Keyword.t()) :: {:ok, Cinema.Projection.t() | [term()]} | {:error, term()}
Projects a module that implements the Cinema.Projection
behaviour.
Projection is done by reflecting on the given projection and building a dependency graph of all of it's inputs, which is then executed in the correct order.
The output of each dependency projection is streamed into the given projection's Cinema.Projection.derivation/2
callback. This callback is responsible for materializing database table rows which can then be returned as the final
output of the projection.
Optionally takes a Lens.t()
or a Keyword.t()
filter list to apply to the projection.
Additionally, you can pass in a Keyword.t()
list of options to control the behavior of how the projection is
executed.
Note that by default, the given projection's final Cinema.Projection.output/0
will be awaited and
evaluated by the projection's configured read_repo
. If you want to control the execute the projection, pass
async: true
as an option.
Options
:async
- Defaults tofalse
. Iftrue
, the projection will be executed immediately but the given projection's final output will not be awaited and returned. Instead, the projection itself will be returned.You can then await and fetch the final output via
Cinema.fetch/1
.:engine
- The engine to use to execute the projection. Defaults toCinema.Engine.Task
. SeeCinema.Engine
for more information.:timeout
- The timeout to use when executing the projection. Defaults to1 minute
.:allow_empty_filters
- Defaults tofalse
. Iftrue
, skips the warning message that gets logged when an empty filter list is provided.
Options are additionally also passed to the engine that is used to execute the projection, as well as used when building the projection itself.