Module kpl_agg

.

Copyright © (C) 2016, AdRoll

Description

Kinesis record aggregator.

Follows the KPL aggregated record format: https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md

This is an Erlang port of the aggregation functionality from: https://pypi.python.org/pypi/aws_kinesis_agg/1.0.0

Creating a new aggregator:

Agg = kpl_agg:new()

Adding user records to an aggregator (the aggregator will emit an aggregated record when it is full):

case kpl_agg:add(Agg, Record) of {undefined, NewAgg} -> ... {FullAggRecord, NewAgg} -> ... end

You can also use kpl:add_all to add multiple records at once. A
Record

is a {PartitionKey, Data} tuple or a {PartitionKey, Data, ExplicitHashKey} tuple.

Getting the current aggregated record (e.g. to get the last aggregated record when you have no more user records to add):

case kpl_agg:finish(Agg) of {undefined, Agg} -> ... {AggRecord, NewAgg} -> ... end

The result currently uses a non-standard magic prefix to prevent the KCL from deaggregating the record automatically. To use compression, instantiate the aggregator using kpl_agg:new(true), which uses another non-standard magic prefix.

Function Index

add/2
add_all/2
count/1
finish/1
new/0
new/1
size_bytes/1

Function Details

add/2

add(State, Record) -> any()

add_all/2

add_all(State, Records) -> any()

count/1

count(State) -> any()

finish/1

finish(State) -> any()

new/0

new() -> any()

new/1

new(ShouldDeflate) -> any()

size_bytes/1

size_bytes(State) -> any()


Generated by EDoc