View Source Skitter.Operation behaviour (Skitter v0.6.1)
Operation type definition and utilities.
An operation is a reusable data processing step that can be embedded inside of a workflow. It is defined as the combination of a set of callbacks, which implement the logic of the operation, and some metadata, which define how the operation is embedded inside a workflow and how it is distributed over the cluster at runtime.
A skitter operation is defined as an elixir module which implements the Skitter.Operation
behaviour. This behaviour defines various elixir callbacks which are used to track operation
information such as the defined callbacks. Instead of implementing an operation as an elixir
module, it is recommend to use Skitter.DSL.Operation.defoperation/3
, which automatically
generates the appropriate callbacks.
This module defines the operation type and behaviour along with utilities to handle operations.
callbacks
Callbacks
An operation defines various callbacks: functions which implement the data processing logic of an operation. These callbacks need to have the ability to modify state and emit data when they are called. Callbacks are implemented as elixir functions with a few properties:
- Callbacks accept
state/0
andconfig/0
as their first and second arguments. - Callbacks return a
result/0
struct, which wraps the result of the callback call along with the updated state and emitted data.
Besides this, callbacks track additional information about whether they access or modify state and which data they emit. This information is stored inside the behaviour callbacks defined in this module.
examples
Examples
Since operations need to be defined in a module the example code shown in this module's documentation assumes the following module is defined:
defmodule OperationModule do
@behaviour Skitter.Operation
alias Skitter.Operation.Callback.{Info, Result}
def _sk_operation_info(:strategy), do: Strategy
def _sk_operation_info(:in_ports), do: [:input]
def _sk_operation_info(:out_ports), do: [:output]
def _sk_operation_initial_state, do: 42
def _sk_callbacks, do: MapSet.new(example: 1)
def _sk_callback_info(:example, 1) do
%Info{read?: true, write?: false, emit?: true}
end
def example(state, config, arg) do
result = state * config
%Result{state: state, emit: [arg: arg], result: result}
end
end
Link to this section Summary
Types
Arguments passed to a callback when it is called.
Configuration passed to the callback when it is called.
Output emitted by a callback.
Additional callback information. Can be retrieved with info/2
.
Input/output interface of Skitter operations.
Input/output interface of Skitter operations.
Values returned by a callback when it is called.
State passed to the callback when it is called.
An operation is defined as a module.
Callbacks
Return the callback information of callback name
, arity
.
Return the names and arities of all the callbacks defined in this module.
Returns the meta-information of the operation.
Returns the initial state of the operation.
Functions
Obtain the arity of operation
.
Call callback callback_name
with an empty state and config and arguments.
Call callback callback_name
with an empty state and config and arguments
.
Call callback callback_name
with an empty state, config
and arguments
.
Call callback callback_name
with state
, config
and arguments
.
Call callback_name
defined by operation
if it exists.
Call callback_name
defined by operation
if it exists.
Call callback_name
defined by operation
if it exists.
Call callback_name
defined by operation
if it exists.
Check if operation
defines a callback with name
and arity
.
Obtain the callback information for callback name
, arity
in operation
.
Obtain the set of all callbacks defined in operation
.
Get the index of an in port.
Obtain the list of in ports of operation
.
Convert an index of an in port to a port name.
Convert an index of an out port to a port name.
Create the initial state for operation
.
Check if a given value refers to an operation module.
Get the index of an out port.
Obtain the list of out ports of operation
.
Check if an operation is a sink.
Check if an operation is a source.
Obtain the default strategy of operation
.
Link to this section Types
@type args() :: [any()]
Arguments passed to a callback when it is called.
The arguments are wrapped in a list.
@type config() :: any()
Configuration passed to the callback when it is called.
The configuration represents the immutable state of an operation. It is explicitly separated
from the mutable state/0
to enable strategies to explicitly differentiate between handling
mutable and immutable data.
@type emit() :: [{port_name(), Enumerable.t()}]
Output emitted by a callback.
Emitted data is returned as a keyword list where the output for each out port is specified. When
no data is emitted on a port, the port should be omitted from the list. The data emitted by a
callback for a port should be wrapped in an Enumerable.t/0
. Each element in this enumerable
will be sent to downstream nodes separately.
@type info() :: %Skitter.Operation.Callback.Info{ emit?: boolean(), read?: boolean(), write?: boolean() }
Additional callback information. Can be retrieved with info/2
.
The following information is stored:
:read?
: Boolean which indicates if the callback reads the operation state.:write?
: Boolean which indicates if the callback updates the operation state.:emit
: Boolean which indicates if the callback emits data.
@type port_index() :: non_neg_integer()
Input/output interface of Skitter operations.
The ports of an operation determine its external interface. A port can be referred to by its index in the in or out ports list of an operation.
in_port_to_index/2
and out_port_to_index/2
can be used to convert a port name to a port
index. Names are used in the workflow and operation DSLs, while indices are used inside
strategies.
@type port_name() :: atom()
Input/output interface of Skitter operations.
The ports of an operation determine its external interface. A port can be referred to by its name, which is stored as an atom.
in_port_to_index/2
and out_port_to_index/2
can be used to convert a port name to a port
index. Names are used in the workflow and operation DSLs, while indices are used inside
strategies.
Values returned by a callback when it is called.
The following information is stored:
:result
: The actual result of the callback, i.e. the final value returned in its body.:state
: The (possibly modified) state after calling the callback.:emit
: The list of output emitted by the callback.
@type state() :: any()
State passed to the callback when it is called.
@type t() :: module()
An operation is defined as a module.
This module should implement the Skitter.Operation
behaviour.
Link to this section Callbacks
Return the callback information of callback name
, arity
.
Return the names and arities of all the callbacks defined in this module.
@callback _sk_operation_info(:in_ports) :: [port_name()]
@callback _sk_operation_info(:out_ports) :: [port_name()]
@callback _sk_operation_info(:strategy) :: Skitter.Strategy.t() | nil
Returns the meta-information of the operation.
The following information is stored:
:in_ports
: A list of port names which represents in ports through which the operation receives incoming data.:out_ports
: A list of out ports names which represents the out ports the operation can use to emit data.:strategy
: TheSkitter.Strategy
of the operation.nil
may be provided instead, in which case a strategy must be provided when the operation is embedded in a workflow.
@callback _sk_operation_initial_state() :: any()
Returns the initial state of the operation.
This function returns an initial state for the operation. The state of an operation is operation specific: Skitter places no constraints on this state.
Link to this section Functions
Obtain the arity of operation
.
The arity is defined as the amount of in ports the operation defines.
examples
Examples
iex> arity(OperationModule)
1
Call callback callback_name
with an empty state and config and arguments.
This function calls Skitter.Operation.call/5
with the state created by initial_state/1
.
nil
is used as the value for config
, no arguments are passed.
Call callback callback_name
with an empty state and config and arguments
.
This function calls Skitter.Operation.call/5
with the state created by initial_state/1
.
nil
is used as the value for config
.
Call callback callback_name
with an empty state, config
and arguments
.
This function calls Skitter.Operation.call/5
with the state created by initial_state/1
.
examples
Examples
iex> call(OperationModule, :example, 2, [:foo])
%Skitter.Operation.Callback.Result{state: 42, result: 84, emit: [arg: :foo]}
Call callback callback_name
with state
, config
and arguments
.
examples
Examples
iex> call(OperationModule, :example, 10, 2, [:foo])
%Skitter.Operation.Callback.Result{state: 10, result: 20, emit: [arg: :foo]}
Call callback_name
defined by operation
if it exists.
Like call_if_exists/5
, but state
is replaced by the initial state of the operation, config
is nil
and args
is the empty list.
Call callback_name
defined by operation
if it exists.
Like call_if_exists/5
, but state
is replaced by the initial state of the operation and
config
is nil
.
Call callback_name
defined by operation
if it exists.
Like call_if_exists/5
, but state
is replaced by the initial state of the operation.
Call callback_name
defined by operation
if it exists.
Calls the callback with the given name with state
, config
and args
if
{name, length(args)}
exists. If the callback does not exist, a result with the
initial_state/1
of the operation, an empty emit list and nil
as result is returned.
examples
Examples
iex> call_if_exists(OperationModule, :example, 10, 2, [:foo])
%Skitter.Operation.Callback.Result{state: 10, result: 20, emit: [arg: :foo]}
iex> call_if_exists(OperationModule, :example, 10, 2, [:foo, :bar])
%Skitter.Operation.Callback.Result{state: 42, result: nil, emit: []}
Check if operation
defines a callback with name
and arity
.
examples
Examples
iex> callback_exists?(OperationModule, :example, 1) true
iex> callback_exists?(OperationModule, :example, 2) false
Obtain the callback information for callback name
, arity
in operation
.
examples
Examples
iex> callback_info(OperationModule, :example, 1)
%Info{read?: true, write?: false, emit?: true}
Obtain the set of all callbacks defined in operation
.
examples
Examples
iex> callbacks(OperationModule) MapSet.new([example: 1])
@spec in_port_to_index(t(), port_name()) :: port_index() | nil
Get the index of an in port.
examples
Examples
iex> in_port_to_index(OperationModule, :input)
0
iex> in_port_to_index(OperationModule, :other)
nil
Obtain the list of in ports of operation
.
examples
Examples
iex> in_ports(OperationModule)
[:input]
@spec index_to_in_port(t(), port_index()) :: port_name() | nil
Convert an index of an in port to a port name.
examples
Examples
iex> index_to_in_port(OperationModule, 0)
:input
iex> index_to_in_port(OperationModule, 1)
nil
@spec index_to_out_port(t(), port_index()) :: port_name() | nil
Convert an index of an out port to a port name.
examples
Examples
iex> index_to_out_port(OperationModule, 0)
:output
iex> index_to_out_port(OperationModule, 1)
nil
Create the initial state for operation
.
examples
Examples
iex> initial_state(OperationModule)
42
Check if a given value refers to an operation module.
examples
Examples
iex> operation?(5)
false
iex> operation?(String)
false
iex> operation?(OperationModule)
true
@spec out_port_to_index(t(), port_name()) :: port_index() | nil
Get the index of an out port.
examples
Examples
iex> out_port_to_index(OperationModule, :output)
0
iex> out_port_to_index(OperationModule, :other)
nil
Obtain the list of out ports of operation
.
examples
Examples
iex> out_ports(OperationModule)
[:output]
Check if an operation is a sink.
An operation is a sink if it does not have any out ports.
examples
Examples
iex> sink?(OperationModule)
false
Check if an operation is a source.
An operation is a source if it does not have any in ports.
examples
Examples
iex> source?(OperationModule)
false
@spec strategy(t()) :: Skitter.Strategy.t() | nil
Obtain the default strategy of operation
.
examples
Examples
iex> strategy(OperationModule)
Strategy