Kayrock.Compression (kayrock v1.0.1)

Copy Markdown View Source

Handles compression/decompression of messages.

See https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression

To add new compression types:

  1. Add the appropriate dependency to mix.exs (don't forget to add it to the application list).
  2. Add the appropriate attribute value and compression_type atom.
  3. Add new Compression module.
  4. Add a decompress function in the new module.
  5. Add a compress function in the new module.

Compression Levels

  • Gzip: 1 (fastest) to 9 (best compression), default: 6
  • Zstandard: 1 (fastest) to 22 (best compression), default: 3
  • Snappy/LZ4: No levels supported

Dependencies

  • Snappy: {:snappyer, "~> 1.2"}
  • LZ4: {:lz4b, "~> 0.2.0"}
  • Zstandard: {:ezstd, "~> 1.0"}

Zstandard (zstd) Requirements

Zstandard compression (codec 4) requires:

  • Produce API version 7 or later (corresponds to KAFKA_2_1_IV0+)
  • Kafka broker version 2.1.0 or later
  • RecordBatch format (message format v2)

If you use an older Produce API version (v0-v6), you'll receive error code 76 (UNSUPPORTED_COMPRESSION_TYPE). The Confluent Kafka container includes zstd-jni support by default.

Summary

Types

codec_t()

@type codec_t() :: :gzip | :snappy | :lz4 | :zstd

Functions

available_codecs()

@spec available_codecs() :: [codec_t()]

compress(atom, data)

compress(atom, data, opts)

decompress(int, data)