stat_buffer v0.1.2 StatBuffer behaviour View Source

Defines a stat buffer.

A stat buffer is an efficient way to maintain a local incrementable count with a given key that can later be flushed to persistent storage. In fast moving systems, this provides a scalable way keep track of counts without putting heavy loads on a database.

Creating a buffer is as easy as:

defmodule Buffer do
  use StatBuffer
end

Once we have defined our buffer module, all we must do is implement a handle_flush/2 callback that allows us to perform an operation with a provided key and counter. This typically means updating a counter in a database.

defmodule Buffer do
  use StatBuffer

  def handle_flush(key, counter) do
    # write to the database...

    # handle_flush MUST return an :ok atom
    :ok
  end
end

Each flush operation is handled with its own supervised Task process. By default, a failed flush operation will retry about 3 times within 3 seconds.

Usage

Before using our buffer, we must start it.

Buffer.start()

With our buffer started, we can now increment key counters. A key can be any valid term.

Buffer.increment("mykey") # increments by 1

Buffer.increment("mykey", 10) # increments by 10

Buffer.async_increment("mykey") # async increments by 1

Key counts are maintained in an ETS table. All keys are scoped to the given buffer module - so multiple buffers using the same keys will not cause issues.

With the default buffer we setup above, the “mykey” counter will be flushed after 5 seconds. Assuming no new operations occur with our buffer, the process will be placed into hibernation after 10 seconds. All of this is configurable through the options below.

Options

A stat buffer comes with a few configurable options. We can pass any of these options along with the use macro.

use StatBuffer, interval: 60_000, jitter: 20_000
  • :interval - the time in milliseconds between the first increment for a given key and its next flush callback being invoked. Defaults to 5_000.

  • :jitter - a max time in milliseconds that will be added to interval to ensure some randomness in each flush invocation. The time added would be randomly selected between 0 and jitter. Defaults to 0.

  • :timeout - the time in milliseconds between the last operation on a a buffer, and the process being hibernated. Defaults to 10_000.

  • :backoff - the time in milliseconds between a handle_flush/2 callback failing, and the next attempt occuring. Defaults to 1_000.

  • :restart - the :restart option used for the flush task. Please see Task.Supervisor.start_child/2 for more details.

  • :shutdown - the :shutdown option used for the flush task. Please see Task.Supervisor.start_child/2 for more details.

Link to this section Summary

Callbacks

Same as increment/2 except performs the operation asynchronously

The amount of time that will be applied between failed key flush attempts

Returns the current state of a key from the buffer

Asynchronously flushes a given key from the buffer

Callback for flushing a key for the buffer

Increments a given key in the buffer by the provided count

The amount of time between buffer flush operations. If specified in the options, a jitter may be applied

Starts the buffer process

The options that will be used when a supervised Task is started to flush the buffer

The amount of time that a buffer key process will remain without recieving work before shutting down

Link to this section Types

Link to this section Callbacks

Link to this callback async_increment(key, count) View Source
async_increment(key :: any(), count :: integer()) :: :ok

Same as increment/2 except performs the operation asynchronously.

The amount of time that will be applied between failed key flush attempts.

Link to this callback count(key) View Source
count(key :: any()) :: integer() | nil | no_return()

Returns the current state of a key from the buffer.

Link to this callback flush(key) View Source
flush(key :: any()) :: :ok | no_return()

Asynchronously flushes a given key from the buffer.

Link to this callback handle_flush(key, counter) View Source
handle_flush(key :: any(), counter :: integer()) :: :ok

Callback for flushing a key for the buffer.

When a buffer key hits its set time interval, this function will be called and provided with the key as well its current counter.

This function is called within its own Task and is supervised. If the callback does not return :ok - the task will fail and attempt a retry with configurable backoff.

Parameters

  • key: Any valid term.
  • counter: An integer counter.
Link to this callback increment(key, count) View Source
increment(key :: any(), count :: integer()) :: :ok

Increments a given key in the buffer by the provided count.

Each key is scoped to the buffer module. So duplicate keys across different buffer modules will not cause issues.

Parameters

  • key: Any valid term.
  • count: An integer count. Defaults to 1.

The amount of time between buffer flush operations. If specified in the options, a jitter may be applied.

Starts the buffer process.

Link to this callback task_opts() View Source
task_opts() :: list()

The options that will be used when a supervised Task is started to flush the buffer.

The amount of time that a buffer key process will remain without recieving work before shutting down.