View Source Cinema.Projection behaviour (cinema v0.1.0)
This module is responsible to defining "projections", which are similar to "views" built into most RDBMS systems.
Key Concepts
A projection is simply a module which tells Cinema
three things:
- What inputs (usually other projections) are required to execute a projection.
- What outputs (usually a
Stream
-compatible term) are produced by executing a projection. - Most importantly, the instructions required to derive data for the projection.
It is important to call out that a Projection
is different from defining a View
in a traditional
RDBMS system. A View
is a stored query that is executed on-demand, while a Projection
is a
series of instructions that can be executed to derive a result, which is optionally stored in a table
for querying.
While View
s can be materialized, at least in PostgreSQL, you're usually forced to refresh the
entire view whenever you want to update any stored data. Projection
s on the other hand are designed
to allow some form of incremental materialization, where only data relevent to the Cinema.Lens
is required to be refreshed.
Currently, Projections need to be manually called to be refreshed; though future work may include automatic triggers for refreshing Projections when their inputs change.
Please see Cinema.project/2
for more details on how projections are executed.
Materialized vs Virtual Projections
Projections currently come in two flavours: virtual and materialized, defaulting to materialized for most cases.
A materialized projection requires users to define an Ecto.Schema
schema and associated database
migration to store the projection's output. These projections can then be indexed and queried like any
other schema.
Virtual projections on the other hand are not stored in the database, and are instead derived on-the-fly by executing the projection's instructions (if any) and immediately returning the Projection's output.
Generally, virtual projections are used for kickstarting larger Cinema
data pipelines from
your application's existing database tables, while materialized projections are used for storing
intermediate results for later querying.
You can control whether a projection is materialized or virtual by setting the :materialized
option
to true
or false
respectively. It is set to true
by default.
Dematerialization
Materialized Projections are automatically dematerialized prior to materialization, which means that the projection's output is deleted before the projection is executed. This is useful for ensuring that the projection's output is always up-to-date, and can be used to ensure that the projection's output is idempotent.
Only data which is in "scope" of a given Projection and Lens is dematerialized, so generally speaking,
assuming (dematerialize(input, lens); materialize(input, lens)) == materialize(input, lens)
, you
don't have to worry about dematerialization causing data loss.
If you have a projection that is not idempotent, you can disable dematerialization by setting the
:dematerialize
option to false
, though you should be aware that you may want to manually manage
the projection's output to ensure that it is up-to-date and any outdated data is cleaned up.
Side Effects
Projections can also, optionally, trigger side-effects by writing to the database or other external
systems. This can be done by executing the desires side-effect causing code within a Projection's
derive/2
callback.
Generally in such projections, it is inadvisable to dematerialize the projection, as the side-effects may not be idempotent, and may rely on the state of the Projection's output as a log of what has already been processed.
If this is the cases, dematerialization can be disabled by setting the :dematerialize
option to false
.
Telemetry
All Projections emit Telemetry events when they are executed, which can be used to monitor said projections.
The Cinema
Telemetry events are as follows: TBA
Summary
Functions
Builds a new instance of the given projection, with the given lens. Does not execute the projection.
See Cinema.Engine
for functions that operate on Projection.t()
s.
Returns the dependency graph needed to execute the given projection.
Returns true
if the given module implements the Elixir.Cinema.Projection
behaviour, otherwise returns false
.
Lists all defined projections defined in the given application or loaded in the VM.
Returns true
if the given module is a virtual projection (does not define a schema), otherwise returns false
.
Types
@type implementation() :: module()
@type input() :: implementation()
@type output() :: Ecto.Queryable.t() | [map()] | term()
Callbacks
@callback derivation( {input(), output()}, Cinema.Projection.Lens.t() ) :: term()
@callback fields() :: [atom()]
@callback inputs() :: [implementation()]
@callback output() :: output()
Functions
@spec build!(implementation(), Cinema.Projection.Lens.t(), Keyword.t()) :: t()
Builds a new instance of the given projection, with the given lens. Does not execute the projection.
See Cinema.Engine
for functions that operate on Projection.t()
s.
Takes an optional lens
argument, which is a Cinema.Lens.t()
struct that can be used to
scope the projection's inputs and outputs.
Also takes an optional opts
argument, which is a Keyword.t()
list of options that can be used to
configure the projection's behavior.
Options
:application
- The application to use when fetching the list of all projections. Defaults tonil
.
@spec dematerialize(Cinema.Projection.Lens.t()) :: term()
Returns the dependency graph needed to execute the given projection.
Returns true
if the given module implements the Elixir.Cinema.Projection
behaviour, otherwise returns false
.
See examples:
iex> Cinema.Projection.implemented?(Enum)
false
iex> defmodule MyApp.SomeProjection do
...> use Cinema.Projection
...>
...> @impl Cinema.Projection
...> def inputs, do: []
...>
...> @impl Cinema.Projection
...> def ouput, do: []
...> end
iex> Cinema.Projection.implemented?(MyApp.SomeProjection)
true
Lists all defined projections defined in the given application or loaded in the VM.
Returns true
if the given module is a virtual projection (does not define a schema), otherwise returns false
.
See examples:
iex> Cinema.Projection.virtual?(Enum)
false
iex> defmodule MyApp.VirtualProjection do
...> use Cinema.Projection
...>
...> @impl Cinema.Projection
...> def inputs, do: []
...>
...> @impl Cinema.Projection
...> def ouput, do: []
...> end
iex> Cinema.Projection.virtual?(MyApp.VirtualProjection)
true