View Source ExBuffer (ExBuffer v0.1.0)
An ExBuffer is a process that maintains a collection of items and flushes them once certain conditions have been met.
ExBuffers can flush based on a timeout, a maximum length (item count), a maximum byte size, or a combination of the three. When multiple conditions are used, the ExBuffer will flush when the first condition is met.
ExBuffers also come with a number of helpful tools for testing and debugging.
Summary
Functions
Returns a specification to start this module under a supervisor.
Lazily chunks an enumerable based on one or more ExBuffer flush conditions.
Dumps the contents of the given ExBuffer
to a list, bypassing a flush
callback and resetting the buffer.
Flushes the given ExBuffer
, regardless of whether or not the flush conditions
have been met.
Inserts the given item into the given ExBuffer
.
Returns the length (item count) of the given ExBuffer
.
Retuns the size (in bytes) of the given ExBuffer
.
Starts an ExBuffer
process linked to the current process.
Types
@type t() :: GenServer.name() | pid()
Functions
Returns a specification to start this module under a supervisor.
See Supervisor
.
@spec chunk_enum!( Enumerable.t(), keyword() ) :: Enumerable.t()
Lazily chunks an enumerable based on one or more ExBuffer flush conditions.
This function currently supports length and size conditions. If multiple conditions are specified, a chunk is emitted once the first condition is met (just like an ExBuffer process).
While this function is useful in it's own right, it's included primarily as another way to synchronously test applications that use ExBuffer.
Options
An enumerable can be chunked with the following options:
:max_length
- The maximum allowed length (item count) of a chunk. (Optionalnon_neg_integer()
, Default =:infinity
):max_size
- The maximum allowed size (in bytes) of a chunk. (Optionalnon_neg_integer()
, Default =:infinity
)
Warning
Not including either of the options above is permitted but will result in a single chunk being emitted. One can achieve a similar result in a more performant way using
Stream.into/2
. In that same vein, including only a:max_length
condition makes this function a less performant version ofStream.chunk_every/2
. This function is optimized for chunking by either size or size and count. Any other chunking strategy can likely be achieved in a more efficient way using other methods.
Example
enum = ["foo", "bar", "baz", "foobar", "barbaz", "foobarbaz"]
enum
|> ExBuffer.chunk_enum!(max_length: 3, max_size: 10)
|> Enum.into([])
#=> [["foo", "bar", "baz"], ["foobar", "barbaz"], ["foobarbaz"]]
Dumps the contents of the given ExBuffer
to a list, bypassing a flush
callback and resetting the buffer.
While this behavior may occasionally be desriable in a production environment, it is intended to be used primarily for testing and debugging.
Example
ExBuffer.insert(:test_buffer, "foo")
ExBuffer.insert(:test_buffer, "bar")
ExBuffer.dump(:test_buffer)
#=> ["foo", "bar"]
ExBuffer.length(:test_buffer)
#=> 0
Flushes the given ExBuffer
, regardless of whether or not the flush conditions
have been met.
While this behavior may occasionally be desriable in a production environment, it is intended to be used primarily for testing and debugging.
Options
An ExBuffer can be flushed with the following options:
:async
- Determines whether or not the flush will be asynchronous. (Optionalboolean()
, Default =true
)
Example
ExBuffer.insert(:test_buffer, "foo")
ExBuffer.insert(:test_buffer, "bar")
# Assuming the flush callback is `IO.inspect/1`
ExBuffer.flush(:test_buffer)
#=> outputs ["foo", "bar"]
ExBuffer.length(:test_buffer)
#=> 0
Inserts the given item into the given ExBuffer
.
Example
ExBuffer.insert(:test_buffer, "foo")
#=> :test_buffer items = ["foo"]
ExBuffer.insert(:test_buffer, "bar")
#=> :test_buffer items = ["foo", "bar"]
@spec length(t()) :: non_neg_integer()
Returns the length (item count) of the given ExBuffer
.
While this behavior may occasionally be desriable in a production environment, it is intended to be used primarily for testing and debugging.
Example
ExBuffer.insert(:test_buffer, "foo")
ExBuffer.insert(:test_buffer, "bar")
ExBuffer.length(:test_buffer)
#=> 2
@spec size(t()) :: non_neg_integer()
Retuns the size (in bytes) of the given ExBuffer
.
Item size is computed using Kernel.byte_size/1
. Because this function requires
a bitstring input, non-bitstring items are first transformed into binary using
:erlang.term_to_binary/1
.
While this behavior may occasionally be desriable in a production environment, it is intended to be used primarily for testing and debugging.
Example
ExBuffer.insert(:test_buffer, "foo")
ExBuffer.insert(:test_buffer, "bar")
ExBuffer.size(:test_buffer)
#=> 6
@spec start_link(keyword()) :: GenServer.on_start()
Starts an ExBuffer
process linked to the current process.
Options
An ExBuffer can be started with the following options:
:callback
- The function that will be invoked to handle a flush. This function should expect a single parameter: a list of items. (Requiredfunction()
):buffer_timeout
- The maximum time (in ms) allowed between flushes of the ExBuffer. Once this amount of time has passed, the ExBuffer will be flushed. (Optionalnon_neg_integer()
, Default =:infinity
):max_length
- The maximum allowed length (item count) of the ExBuffer. Once the limit is hit, the ExBuffer will be flushed. (Optionalnon_neg_integer()
, Default =:infinity
):max_size
- The maximum allowed size (in bytes) of the ExBuffer. Once the limit is hit (or exceeded), the ExBuffer will be flushed. For more information on how size is computed, seeExBuffer.size/1
. (Optionalnon_neg_integer()
, Default =:infinity
)
Additionally, an ExBuffer can also be started with any GenServer
options.