View Source Skitter.Operation behaviour (Skitter v0.6.3)

Operation type definition and utilities.

An operation is a reusable data processing step that can be embedded inside of a workflow. It is defined as the combination of a set of callbacks, which implement the logic of the operation, and some metadata, which define how the operation is embedded inside a workflow and how it is distributed over the cluster at runtime.

A skitter operation is defined as an elixir module which implements the Skitter.Operation behaviour. This behaviour defines various elixir callbacks which are used to track operation information such as the defined callbacks. Instead of implementing an operation as an elixir module, it is recommend to use Skitter.DSL.Operation.defoperation/3, which automatically generates the appropriate callbacks.

This module defines the operation type and behaviour along with utilities to handle operations.

callbacks

Callbacks

An operation defines various callbacks: functions which implement the data processing logic of an operation. These callbacks need to have the ability to modify state and emit data when they are called. Callbacks are implemented as elixir functions with a few properties:

  • Callbacks accept state/0 and config/0 as their first and second arguments.
  • Callbacks return a result/0 struct, which wraps the result of the callback call along with the updated state and emitted data.

Besides this, callbacks track additional information about whether they access or modify state and which data they emit. This information is stored inside the behaviour callbacks defined in this module.

examples

Examples

Since operations need to be defined in a module the example code shown in this module's documentation assumes the following module is defined:

defmodule OperationModule do
  @behaviour Skitter.Operation
  alias Skitter.Operation.Callback.{Info, Result}

  def _sk_operation_info(:strategy), do: Strategy
  def _sk_operation_info(:in_ports), do: [:input]
  def _sk_operation_info(:out_ports), do: [:output]

  def _sk_operation_initial_state, do: 42

  def _sk_callbacks, do: MapSet.new(example: 1)

  def _sk_callback_info(:example, 1) do
    %Info{read?: true, write?: false, emit?: true}
  end

  def example(state, config, arg) do
    result = state * config
    %Result{state: state, emit: [arg: arg], result: result}
  end
end

Link to this section Summary

Types

Arguments passed to a callback when it is called.

Configuration passed to the callback when it is called.

Output emitted by a callback.

Additional callback information. Can be retrieved with info/2.

Input/output interface of Skitter operations.

Input/output interface of Skitter operations.

Values returned by a callback when it is called.

State passed to the callback when it is called.

t()

An operation is defined as a module.

Callbacks

Return the callback information of callback name, arity.

Return the names and arities of all the callbacks defined in this module.

Returns the meta-information of the operation.

Returns the initial state of the operation.

Functions

Obtain the arity of operation.

Call callback callback_name with an empty state and config and arguments.

Call callback callback_name with an empty state and config and arguments.

Call callback callback_name with an empty state, config and arguments.

Call callback callback_name with state, config and arguments.

Call callback_name defined by operation if it exists.

Call callback_name defined by operation if it exists.

Call callback_name defined by operation if it exists.

Call callback_name defined by operation if it exists.

Check if operation defines a callback with name and arity.

Obtain the callback information for callback name, arity in operation.

Obtain the set of all callbacks defined in operation.

Get the index of an in port.

Obtain the list of in ports of operation.

Convert an index of an in port to a port name.

Convert an index of an out port to a port name.

Create the initial state for operation.

Check if a given value refers to an operation module.

Get the index of an out port.

Obtain the list of out ports of operation.

Check if an operation is a sink.

Check if an operation is a source.

Obtain the default strategy of operation.

Link to this section Types

@type args() :: [any()]

Arguments passed to a callback when it is called.

The arguments are wrapped in a list.

@type config() :: any()

Configuration passed to the callback when it is called.

The configuration represents the immutable state of an operation. It is explicitly separated from the mutable state/0 to enable strategies to explicitly differentiate between handling mutable and immutable data.

@type emit() :: [{port_name(), Enumerable.t()}]

Output emitted by a callback.

Emitted data is returned as a keyword list where the output for each out port is specified. When no data is emitted on a port, the port should be omitted from the list. The data emitted by a callback for a port should be wrapped in an Enumerable.t/0. Each element in this enumerable will be sent to downstream nodes separately.

@type info() :: %Skitter.Operation.Callback.Info{
  emit?: boolean(),
  read?: boolean(),
  write?: boolean()
}

Additional callback information. Can be retrieved with info/2.

The following information is stored:

  • :read?: Boolean which indicates if the callback reads the operation state.
  • :write?: Boolean which indicates if the callback updates the operation state.
  • :emit: Boolean which indicates if the callback emits data.
@type port_index() :: non_neg_integer()

Input/output interface of Skitter operations.

The ports of an operation determine its external interface. A port can be referred to by its index in the in or out ports list of an operation.

in_port_to_index/2 and out_port_to_index/2 can be used to convert a port name to a port index. Names are used in the workflow and operation DSLs, while indices are used inside strategies.

@type port_name() :: atom()

Input/output interface of Skitter operations.

The ports of an operation determine its external interface. A port can be referred to by its name, which is stored as an atom.

in_port_to_index/2 and out_port_to_index/2 can be used to convert a port name to a port index. Names are used in the workflow and operation DSLs, while indices are used inside strategies.

@type result() :: %Skitter.Operation.Callback.Result{
  emit: emit(),
  result: any(),
  state: state()
}

Values returned by a callback when it is called.

The following information is stored:

  • :result: The actual result of the callback, i.e. the final value returned in its body.
  • :state: The (possibly modified) state after calling the callback.
  • :emit: The list of output emitted by the callback.
@type state() :: any()

State passed to the callback when it is called.

@type t() :: module()

An operation is defined as a module.

This module should implement the Skitter.Operation behaviour.

Link to this section Callbacks

Link to this callback

_sk_callback_info(name, arity)

View Source
@callback _sk_callback_info(name :: atom(), arity()) :: info()

Return the callback information of callback name, arity.

@callback _sk_callbacks() :: MapSet.t({atom(), arity()})

Return the names and arities of all the callbacks defined in this module.

Link to this callback

_sk_operation_info(atom)

View Source
@callback _sk_operation_info(:in_ports) :: [port_name()]
@callback _sk_operation_info(:out_ports) :: [port_name()]
@callback _sk_operation_info(:strategy) :: Skitter.Strategy.t() | nil

Returns the meta-information of the operation.

The following information is stored:

  • :in_ports: A list of port names which represents in ports through which the operation receives incoming data.

  • :out_ports: A list of out ports names which represents the out ports the operation can use to emit data.

  • :strategy: The Skitter.Strategy of the operation. nil may be provided instead, in which case a strategy must be provided when the operation is embedded in a workflow.

Link to this callback

_sk_operation_initial_state()

View Source
@callback _sk_operation_initial_state() :: any()

Returns the initial state of the operation.

This function returns an initial state for the operation. The state of an operation is operation specific: Skitter places no constraints on this state.

Link to this section Functions

@spec arity(t()) :: arity()

Obtain the arity of operation.

The arity is defined as the amount of in ports the operation defines.

examples

Examples

iex> arity(OperationModule)
1
Link to this function

call(operation, callback_name)

View Source
@spec call(t(), atom()) :: result()

Call callback callback_name with an empty state and config and arguments.

This function calls Skitter.Operation.call/5 with the state created by initial_state/1. nil is used as the value for config, no arguments are passed.

Link to this function

call(operation, callback_name, args)

View Source
@spec call(t(), atom(), args()) :: result()

Call callback callback_name with an empty state and config and arguments.

This function calls Skitter.Operation.call/5 with the state created by initial_state/1. nil is used as the value for config.

Link to this function

call(operation, callback_name, config, args)

View Source
@spec call(t(), atom(), config(), args()) :: result()

Call callback callback_name with an empty state, config and arguments.

This function calls Skitter.Operation.call/5 with the state created by initial_state/1.

examples

Examples

iex> call(OperationModule, :example, 2, [:foo])
%Skitter.Operation.Callback.Result{state: 42, result: 84, emit: [arg: :foo]}
Link to this function

call(operation, name, state, config, args)

View Source
@spec call(t(), atom(), state(), config(), args()) :: result()

Call callback callback_name with state, config and arguments.

examples

Examples

iex> call(OperationModule, :example, 10, 2, [:foo])
%Skitter.Operation.Callback.Result{state: 10, result: 20, emit: [arg: :foo]}
Link to this function

call_if_exists(operation, callback_name)

View Source
@spec call_if_exists(t(), atom()) :: result()

Call callback_name defined by operation if it exists.

Like call_if_exists/5, but state is replaced by the initial state of the operation, config is nil and args is the empty list.

Link to this function

call_if_exists(operation, callback_name, args)

View Source
@spec call_if_exists(t(), atom(), args()) :: result()

Call callback_name defined by operation if it exists.

Like call_if_exists/5, but state is replaced by the initial state of the operation and config is nil.

Link to this function

call_if_exists(operation, callback_name, config, args)

View Source
@spec call_if_exists(t(), atom(), config(), args()) :: result()

Call callback_name defined by operation if it exists.

Like call_if_exists/5, but state is replaced by the initial state of the operation.

Link to this function

call_if_exists(operation, callback_name, state, config, args)

View Source
@spec call_if_exists(t(), atom(), state(), config(), args()) :: result()

Call callback_name defined by operation if it exists.

Calls the callback with the given name with state, config and args if {name, length(args)} exists. If the callback does not exist, a result with the initial_state/1 of the operation, an empty emit list and nil as result is returned.

examples

Examples

iex> call_if_exists(OperationModule, :example, 10, 2, [:foo])
%Skitter.Operation.Callback.Result{state: 10, result: 20, emit: [arg: :foo]}
iex> call_if_exists(OperationModule, :example, 10, 2, [:foo, :bar])
%Skitter.Operation.Callback.Result{state: 42, result: nil, emit: []}
Link to this function

callback_exists?(operation, name, arity)

View Source
@spec callback_exists?(t(), atom(), arity()) :: boolean()

Check if operation defines a callback with name and arity.

examples

Examples

iex> callback_exists?(OperationModule, :example, 1) true

iex> callback_exists?(OperationModule, :example, 2) false

Link to this function

callback_info(operation, name, arity)

View Source
@spec callback_info(t(), atom(), arity()) :: info()

Obtain the callback information for callback name, arity in operation.

examples

Examples

iex> callback_info(OperationModule, :example, 1)
%Info{read?: true, write?: false, emit?: true}
@spec callbacks(t()) :: [{atom(), arity()}]

Obtain the set of all callbacks defined in operation.

examples

Examples

iex> callbacks(OperationModule) MapSet.new([example: 1])

Link to this function

in_port_to_index(operation, port)

View Source
@spec in_port_to_index(t(), port_name()) :: port_index() | nil

Get the index of an in port.

examples

Examples

iex> in_port_to_index(OperationModule, :input)
0
iex> in_port_to_index(OperationModule, :other)
nil
@spec in_ports(t()) :: [port_name()]

Obtain the list of in ports of operation.

examples

Examples

iex> in_ports(OperationModule)
[:input]
Link to this function

index_to_in_port(operation, idx)

View Source
@spec index_to_in_port(t(), port_index()) :: port_name() | nil

Convert an index of an in port to a port name.

examples

Examples

iex> index_to_in_port(OperationModule, 0)
:input
iex> index_to_in_port(OperationModule, 1)
nil
Link to this function

index_to_out_port(operation, idx)

View Source
@spec index_to_out_port(t(), port_index()) :: port_name() | nil

Convert an index of an out port to a port name.

examples

Examples

iex> index_to_out_port(OperationModule, 0)
:output
iex> index_to_out_port(OperationModule, 1)
nil
Link to this function

initial_state(operation)

View Source
@spec initial_state(t()) :: state()

Create the initial state for operation.

examples

Examples

iex> initial_state(OperationModule)
42
@spec operation?(any()) :: boolean()

Check if a given value refers to an operation module.

examples

Examples

iex> operation?(5)
false
iex> operation?(String)
false
iex> operation?(OperationModule)
true
Link to this function

out_port_to_index(operation, port)

View Source
@spec out_port_to_index(t(), port_name()) :: port_index() | nil

Get the index of an out port.

examples

Examples

iex> out_port_to_index(OperationModule, :output)
0
iex> out_port_to_index(OperationModule, :other)
nil
@spec out_ports(t()) :: [port_name()]

Obtain the list of out ports of operation.

examples

Examples

iex> out_ports(OperationModule)
[:output]
@spec sink?(t()) :: boolean()

Check if an operation is a sink.

An operation is a sink if it does not have any out ports.

examples

Examples

iex> sink?(OperationModule)
false
@spec source?(t()) :: boolean()

Check if an operation is a source.

An operation is a source if it does not have any in ports.

examples

Examples

iex> source?(OperationModule)
false
@spec strategy(t()) :: Skitter.Strategy.t() | nil

Obtain the default strategy of operation.

examples

Examples

iex> strategy(OperationModule)
Strategy