Sage v0.6.1 Sage View Source

Sage is a dependency-free implementation of Sagas pattern in pure Elixir. It is a go to way when you dealing with distributed transactions, especially with an error recovery/cleanup. Sage does it's best to guarantee that either all of the transactions in a saga are successfully completed or compensating that all of the transactions did run to amend a partial execution.

This is done by defining two way flow with transaction and compensation functions. When one of the transactions fails, Sage will ensure that transaction's and all of it's predecessors compensations are executed. However, it's important to note that Sage can not protect you from a node failure that executes given Sage.

Critical Error Handling

For Transactions

Transactions are wrapped in a try..catch block.

Whenever a critical error occurs (exception is raised, error thrown or exit signal is received) Sage will run all compensations and then reraise the exception with the same stacktrace, so your log would look like it occurred without using a Sage.

For Compensations

By default, compensations are not protected from critical errors and would raise an exception. This is done to keep simplicity and follow "let it fall" pattern of the language, thinking that this kind of errors should be logged and then manually investigated by a developer.

But if that's not enough for you, it is possible to register handler via with_compensation_error_handler/2. When it's registered, compensations are wrapped in a try..catch block and then it's error handler responsibility to take care about further actions. Few solutions you might want to try:

  • Send notification to a Slack channel about need of manual resolution;
  • Retry compensation;
  • Spawn a new supervised process that would retry compensation and return an error in the Sage. (Useful when you have connection issues that would be resolved at some point in future.)

Logging for compensation errors is pretty verbose to drive the attention to the problem from system maintainers.

finally/2 hook

Sage does it's best to make sure final callback is executed even if there is a program bug in the code. This guarantee simplifies integration with a job processing queues, you can read more about it at GenTask Readme.

If an error is raised within finally/2 hook, it's getting logged and ignored. Follow the simple rule - everything that is on your critical path should be a Sage transaction.

Tracing and measuring Sage execution steps

Sage allows you to set a tracer module which is called on each step of the execution flow (before and after transactions and/or compensations). It could be used to report metrics on the execution flow.

If error is raised within tracing function, it's getting logged and ignored.

Link to this section Summary

Types

Options for asynchronous transaction stages.

Compensation callback, can either anonymous function or an {module, function, [arguments]} tuple.

Effects created on Sage execution.

Final hook.

Retry options.

Name of Sage execution stage.

t()

Transaction callback, can either anonymous function or an {module, function, [arguments]} tuple.

Functions

Executes a Sage.

Appends the Sage with a function that will be triggered after Sage execution.

Creates a new Sage.

Appends sage with a transaction that does not have side effect.

Appends sage with a transaction and function to compensate it's effect.

Appends sage with an asynchronous transaction and function to compensate it's effect.

Executes Sage with Ecto.Repo.transaction/2.

Register error handler for compensations.

Registers tracer for a Sage execution.

Link to this section Types

Specs

async_opts() :: [{:timeout, integer() | :infinity}]

Options for asynchronous transaction stages.

Specs

compensation() ::
  (effect_to_compensate :: any(), effects_so_far :: effects(), attrs :: any() ->
     :ok | :abort | {:retry, retry_opts :: retry_opts()} | {:continue, any()})
  | :noop
  | mfa()

Compensation callback, can either anonymous function or an {module, function, [arguments]} tuple.

Receives:

  • effect created by transaction it's responsible for or nil in case effect is not known due to an error;
  • effects created by preceding executed transactions;
  • options passed to execute/2 function.

Returns:

  • :ok if effect is compensated, Sage will continue to compensate other effects;
  • :abort if effect is compensated but should not be created again, Sage will compensate other effects and ignore retries on any stages;
  • {:retry, retry_opts} if effect is compensated but transaction can be retried with options retry_opts;
  • {:continue, effect} if effect is compensated and execution can be retried with other effect to replace the transaction return. This allows to implement circuit breaker.

Circuit Breaker

After receiving a circuit breaker response Sage will continue executing transactions by using returned effect.

Circuit breaking is only allowed if compensation function that returns it is responsible for the failed transaction (they both are parts of for the same execution step). Otherwise curcuit breaker would be ignored and Sage will continue applying backward recovery.

The circuit breaker should use data which is local to the sage execution, preferably from list of options which are set via execute/2 2nd argument. This would guarantee that circuit breaker would not fail when response cache is not available.

Retries

After receiving a {:retry, [retry_limit: limit]} Sage will retry the transaction on a stage where retry was received.

Take into account that by doing retires you can increase execution time and block process that executes the Sage, which can produce timeout, eg. when you trying to respond to an HTTP request.

Compensation guidelines

General rule is that irrespectively to what compensate wants to return, effect must be always compensated. No matter what, side effects must not be created from compensating transaction.

A compensating transaction doesn't necessarily return the data in the system to the state it was in at the start of the original operation. Instead, it compensates for the work performed by the steps that completed successfully before the operation failed.

source: https://docs.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction

You should try to make your compensations idempotent, which makes possible to retry if compensating transaction itself fails. According a modern HTTP semantics, the PUT and DELETE verbs are idempotent. Also, some services support idempotent requests via idempotency keys.

Compensation transactions should not rely on effects created by preceding executed transactions, otherwise it will be more likely that your code is not idempotent and harder to maintain. Use them only as a last resort.

Specs

effects() :: map()

Effects created on Sage execution.

Specs

final_hook() :: (:ok | :error, attrs :: any() -> no_return()) | mfa()

Final hook.

It receives :ok if all transactions are successfully completed or :error otherwise and options passed to the execute/2.

Return is ignored.

Specs

retry_opts() :: [
  retry_limit: pos_integer(),
  base_backoff: pos_integer() | nil,
  max_backoff: pos_integer(),
  enable_jitter: boolean()
]

Retry options.

Retry count for all a sage execution is shared and stored internally, so even trough you can increase retry limit - retry count would be never reset to make sure that execution would not be retried infinitely.

Available retry options:

  • :retry_limit - is the maximum number of possible retry attempts;
  • :base_backoff - is the base backoff for retries in ms, no backoff is applied if this value is nil or not set;
  • :max_backoff - is the maximum backoff value, default: 5_000 ms.;
  • :enable_jitter - whatever jitter is applied to backoff value, default: true;

Sage will log and give up retrying if options are invalid.

Backoff calculation

For exponential backoff this formula is used:

min(max_backoff, (base_backoff * 2) ^ retry_count)

Example:

AttemptBase BackoffMax BackoffSleep time
1103000020
21030000400
310300008000
4103000030000
5103000030000

When jitter is enabled backoff value is randomized:

random(0, min(max_backoff, (base_backoff * 2) ^ retry_count))

Example:

AttemptBase BackoffMax BackoffSleep interval
110300000..20
210300000..400
310300000..8000
410300000..30000
510300000..30000

For more reasoning behind using jitter, check out this blog post.

Specs

stage_name() :: term()

Name of Sage execution stage.

Specs

t() :: %Sage{
  final_hooks: MapSet.t(final_hook()),
  on_compensation_error: :raise | module(),
  stage_names: MapSet.t(),
  stages: [stage()],
  tracers: MapSet.t(module())
}

Specs

transaction() ::
  (effects_so_far :: effects(), attrs :: any() ->
     {:ok | :error | :abort, any()})
  | mfa()

Transaction callback, can either anonymous function or an {module, function, [arguments]} tuple.

Receives effects created by preceding executed transactions and options passed to execute/2 function.

Returns {:ok, effect} if transaction is successfully completed, {:error, reason} if there was an error or {:abort, reason} if there was an unrecoverable error. On receiving {:abort, reason} Sage will compensate all side effects created so far and ignore all retries.

Sage.MalformedTransactionReturnError is raised after compensating all effects if callback returned malformed result.

Transaction guidelines

You should try to make your transactions idempotent, which makes possible to retry if compensating transaction itself fails. According a modern HTTP semantics, the PUT and DELETE verbs are idempotent. Also, some services support idempotent requests via idempotency keys.

Link to this section Functions

Link to this function

execute(sage, opts \\ [])

View Source

Specs

execute(sage :: t(), opts :: any()) ::
  {:ok, result :: any(), effects :: effects()} | {:error, any()}

Executes a Sage.

Optionally, you can pass global options in opts, that will be sent to all transaction, compensation functions and hooks. It is especially useful when you want to have keep sage definitions declarative and execute them with different arguments (eg. you may build your Sage struct in a module attribute, because there is no need to repeat this work for each execution).

If there was an exception, throw or exit in one of transaction functions, Sage will reraise it after compensating all effects.

For handling exceptions in compensation functions see "Critical Error Handling" in module doc.

Raises Sage.EmptyError if Sage does not have any transactions.

Specs

finally(sage :: t(), hook :: final_hook()) :: t()

Appends the Sage with a function that will be triggered after Sage execution.

Registering duplicated final hook is not allowed and would raise an Sage.DuplicateFinalHookError exception.

For hook specification see final_hook/0.

Specs

new() :: t()

Creates a new Sage.

Link to this function

run(sage, name, transaction)

View Source

Specs

run(sage :: t(), name :: stage_name(), transaction :: transaction()) :: t()

Appends sage with a transaction that does not have side effect.

This is an alias for calling run/4 with a :noop instead of compensation callback.

Link to this function

run(sage, name, transaction, compensation)

View Source

Specs

run(
  sage :: t(),
  name :: stage_name(),
  transaction :: transaction(),
  compensation :: compensation()
) :: t()

Appends sage with a transaction and function to compensate it's effect.

Raises Sage.DuplicateStageError exception if stage name is duplicated for a given sage.

Callbacks

Callbacks can be either anonymous function or an {module, function, [arguments]} tuple. For callbacks interface see transaction/0 and compensation/0 type docs.

Noop compensation

If transaction does not produce effect to compensate, pass :noop instead of compensation callback or use run/3.

Link to this function

run_async(sage, name, transaction, compensation, opts \\ [])

View Source

Specs

run_async(
  sage :: t(),
  name :: stage_name(),
  transaction :: transaction(),
  compensation :: compensation(),
  opts :: async_opts()
) :: t()

Appends sage with an asynchronous transaction and function to compensate it's effect.

Asynchronous transactions are awaited before the next synchronous transaction or in the end of sage execution. If there is an error in asynchronous transaction, Sage will await for other transactions to complete or fail and then compensate for all the effect created by them.

Callbacks

Transaction callback for asynchronous stages receives only effects created by preceding synchronous transactions.

For more details see run/4.

Options

  • :timeout - the time in milliseconds to wait for the transaction to finish, :infinity will wait indefinitely (default: 5000);
Link to this function

transaction(sage, repo, opts \\ [], transaction_opts \\ [])

View Source (since 0.3.3)

Specs

transaction(
  sage :: t(),
  repo :: module(),
  opts :: any(),
  transaction_opts :: any()
) :: {:ok, result :: any(), effects :: effects()} | {:error, any()}

Executes Sage with Ecto.Repo.transaction/2.

Transaction is rolled back on error.

Ecto must be included as application dependency if you want to use this function.

Link to this function

with_compensation_error_handler(sage, module)

View Source

Specs

with_compensation_error_handler(sage :: t(), module :: module()) :: t()

Register error handler for compensations.

Adapter must implement Sage.CompensationErrorHandler behaviour.

For more information see "Critical Error Handling" in the module doc.

Link to this function

with_tracer(sage, module)

View Source

Specs

with_tracer(sage :: t(), module :: module()) :: t()

Registers tracer for a Sage execution.

Registering duplicated tracing callback is not allowed and would raise an Sage.DuplicateTracerError exception.

All errors during execution of a tracing callbacks would be logged, but it won't affect Sage execution.

Tracing module must implement Sage.Tracer behaviour. For more information see Sage.Tracer.handle_event/3.