Streaming compression for very large datasets that don't fit in memory.
This module provides functions to compress/decompress data in chunks, useful for processing massive time series datasets efficiently.
Memory Efficiency
The default chunk size is 5,000 points (~78KB memory) which provides the optimal balance between compression ratio and memory usage. Each chunk is compressed independently, allowing true streaming with constant memory overhead.
Container Compression
Supports optional container compression (zlib/zstd) via the :compression option:
:none- No container compression (default, fastest):zlib- Use zlib compression:zstd- Use zstd compression (requires ezstd package):auto- Use zstd if available, fall back to zlib
Examples
# Basic streaming compression
data_stream
|> GorillaStream.Stream.compress_stream()
|> Enum.each(&store_chunk/1)
# With zstd compression and smaller chunks
data_stream
|> GorillaStream.Stream.compress_stream(chunk_size: 500, compression: :zstd)
|> Enum.each(&store_chunk/1)Summary
Functions
Compresses a stream of data in chunks.
Decompresses a stream of compressed chunks.
Returns the default chunk size used for streaming.
Functions
Compresses a stream of data in chunks.
Options
:chunk_size- Number of points per chunk (default: 5000):compression- Container compression (:none,:zlib,:zstd,:auto):victoria_metrics- Enable VictoriaMetrics preprocessing (default: true):is_counter- Treat data as counter (default: false):scale_decimals- Decimal scaling (:autoor integer)
Examples
iex> large_dataset
...> |> GorillaStream.Stream.compress_stream(chunk_size: 500, compression: :zstd)
...> |> Enum.to_list()
[{:ok, compressed_chunk_1, metadata_1}, {:ok, compressed_chunk_2, metadata_2}, ...]Decompresses a stream of compressed chunks.
Options
:compression- Container compression used (:none,:zlib,:zstd,:auto)
Examples
compressed_chunks
|> GorillaStream.Stream.decompress_stream(compression: :zstd)
|> Stream.flat_map(fn {:ok, points} -> points end)
|> Enum.to_list()Returns the default chunk size used for streaming.