Handles compression/decompression of messages.
To add new compression types:
- Add the appropriate dependency to mix.exs (don't forget to add it to the application list).
- Add the appropriate attribute value and compression_type atom.
- Add new Compression module.
- Add a decompress function in the new module.
- Add a compress function in the new module.
Compression Levels
- Gzip: 1 (fastest) to 9 (best compression), default: 6
- Zstandard: 1 (fastest) to 22 (best compression), default: 3
- Snappy/LZ4: No levels supported
Dependencies
- Snappy:
{:snappyer, "~> 1.2"} - LZ4:
{:lz4b, "~> 0.2.0"} - Zstandard:
{:ezstd, "~> 1.0"}
Zstandard (zstd) Requirements
Zstandard compression (codec 4) requires:
- Produce API version 7 or later (corresponds to KAFKA_2_1_IV0+)
- Kafka broker version 2.1.0 or later
- RecordBatch format (message format v2)
If you use an older Produce API version (v0-v6), you'll receive error code 76 (UNSUPPORTED_COMPRESSION_TYPE). The Confluent Kafka container includes zstd-jni support by default.