Tidefall.Buffer.Partition (Tidefall v1.0.0-rc.0)

Copy Markdown View Source

Buffer partition.

The implementation is based on OpenTelemetry Batch Processor. The use case is very similar. The "OTel batch processor" buffers spans (large/massive amounts of them) and then exports them to an external source after some time (via OTLP). It is designed and implemented for efficiently handling large workloads. The partitioned buffer takes inspiration from the "OTel batch processor" to leverage all these properties.

Double-Buffering Processing Cycle

Phase 1 — Buffering          Phase 2 — Processing
===================          ====================

Client writes                 Timer fires :processing
     |                              |
     v                              v
+-----------+                 Allocate a fresh ETS table
| Table A   |<-- write        and point Tidefall.Metadata
| (current) |                 at it (writes flow there now)
+-----------+                       |
                                    v
                              +-----------+
                              | Table B   |<-- write (new current)
                              +-----------+
                              | Table A   |-- give_away --> Task
                              +-----------+       |
                                                  v
                                            :ets.select (batches)
                                                  |
                                            processor(batch)
                                                  |
                                            batch mode:
                                              :ets.delete(Table A)
                                            :table mode:
                                              processor owns Table A
                                                  |
                                            :processing_completed

ETS access

This module is ETS-agnostic by design. It owns the two backing tables (allocation, swap, give-away on processing, teardown on terminate) but delegates the data-shape concerns — record layout, processor batch shape, conditional-update logic — to the buffer implementation modules (Tidefall.Queue, Tidefall.HashMap, …) through the Tidefall.Buffer behaviour callbacks ets_table_opts/0 and ets_match_spec/0.

Impl modules read the active write table by calling current_table/1 and then issue their own :ets.* operations.

Summary

Functions

Returns the partition's buffer size.

Returns a specification to start this module under a supervisor.

Returns the partition's current write table.

Starts a buffer partition.

Updates the options for the partition.

Functions

buffer_size(partition)

@spec buffer_size(atom()) :: non_neg_integer()

Returns the partition's buffer size.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

current_table(partition)

@spec current_table(partition :: atom()) :: :ets.table()

Returns the partition's current write table.

This is the only sanctioned access to the active table. Buffer implementations call this and then issue their own :ets.* operations against the returned table reference.

Do not cache

The returned reference is single-use. The partition swaps it on every processing tick; holding it across calls (e.g. caching in a struct, sharing across processes, or pipelining many ETS calls after a stale lookup) is a correctness bug. Resolve it fresh inside each operation.

start_link(opts \\ [])

@spec start_link(keyword()) :: GenServer.on_start()

Starts a buffer partition.

update_options(server, opts)

@spec update_options(
  GenServer.server(),
  keyword()
) :: :ok

Updates the options for the partition.