Understanding Snappy

Snappy (formerly Zippy) is a fast compression codec developed by Google, focused on maximum speed with minimal configuration. This guide covers Snappy's format, performance characteristics, and when to use it.

Overview

Snappy was designed at Google for internal use in their distributed infrastructure (Bigtable, MapReduce, etc.). Its priorities, in order, are:

1. Decompression speed -- as fast as possible.
2. Compression speed -- nearly as fast as decompression.
3. Simplicity -- no configuration knobs, deterministic output.
4. Compression ratio -- secondary to speed.

Snappy achieves compression speeds over 500 MB/s and decompression speeds over 1.5 GB/s, making it suitable for latency-sensitive systems where the compression ratio is not the primary concern.

How Snappy Works

Snappy is an LZ77-family algorithm with aggressive speed optimizations:

Matching Strategy

1. Scan the input linearly.
2. At each position, hash the next 4 bytes and look up a hash table.
3. If a match is found that is at least 4 bytes long (up to 64 KB back), emit a copy command.
4. If no match is found, emit the current byte as a literal.
5. Moves forward by only 1 byte after a literal (no lazy or optimal parsing).

Speed Optimizations

Snappy makes specific design choices that sacrifice ratio for speed:

• No entropy coding. Literals and match lengths are stored in a simple variable-length encoding, not Huffman or arithmetic coding.
• No optimal parsing. The first match found is always used. Snappy does not consider whether skipping a match could lead to a longer one later.
• No match length extension. Snappy does not extend matches beyond the initial find. Some implementations do, but the core format does not require it.
• Minimal branching. The decoder is a simple state machine with few conditional branches, which is CPU-cache and branch-predictor friendly.

Snappy Format

ExCodecs uses the raw Snappy block format (not the separate Snappy framing format). A block is an uncompressed-length prefix followed by tagged elements.

Uncompressed Length Prefix

+-------------------+
| Length (varint)   |  -- Uncompressed length in bytes
+-------------------+

The uncompressed size is encoded as a variable-length integer at the start of the compressed block so the decompressor can size the output buffer.

Elements

Each element starts with a tag byte. The lower two bits select the element type:

Tag byte:
  bits 7-2: length or offset info (depends on type)
  bits 1-0: element type

For literals, short lengths fit in the tag; longer literals use a 1-4 byte length field. Copies encode offset and length as described in Google's Snappy format description. Typical compressor windows are up to 64 KB.

Format Properties

• Self-contained. Each Snappy-compressed block contains all the information needed for decompression. No external dictionary or state is required.
• Deterministic. Identical inputs always produce identical outputs. This is useful for content-addressable storage and caching.
• Single-pass. Both compression and decompression are single-pass with no backtracking.

Performance Characteristics

Speeds and ratios below are illustrative order-of-magnitude figures from typical upstream C Snappy discussions; ExCodecs uses the pure-Rust snap crate. Benchmark your own data.

Compression Speed

Snappy is designed for very high compression throughput. Key factors:

• The hash table is small and fits in L1/L2 cache.
• No expensive operations (no sorting, no entropy coding, no optimal parsing).
• Good SIMD acceleration on x86 and ARM.

Decompression Speed

Decompression is similarly speed-oriented. The decompressor:

• Reads the tag byte to determine the element type.
• For literals: copies bytes verbatim (essentially a memcpy).
• For copies: reads the offset and length, then copies from the already-decompressed output.

Compression Ratio

Snappy's ratio is lower than LZ4, Zstd, and Bzip2. Illustrative ratios:

Snappy's lower ratio is the expected tradeoff for its speed advantage.

Using Snappy in ExCodecs

Snappy has no encode-time configuration knobs (configurable?: false):

# Compress
{:ok, compressed} = ExCodecs.encode(:snappy, data)

# Decompress
{:ok, original} = ExCodecs.decode(:snappy, compressed)

Encode ignores option keywords. Decode accepts :max_output_size (same decompression-bomb guard as other codecs):

{:ok, compressed} = ExCodecs.encode(:snappy, data)
{:ok, original} = ExCodecs.decode(:snappy, compressed, max_output_size: 1_048_576)

{:ok, info} = ExCodecs.codec_info(:snappy)
info.configurable?  # => false
info.streaming?      # => false

Streaming Support

Snappy supports a framing format (the "Snappy framing format" or "snappy-framed") in its full specification, which allows incremental compression and decompression of data streams.

ExCodecs currently provides block-level Snappy compression and decompression. The entire input must be available before compression can begin. For streaming use cases, consider chunking the data into blocks and compressing each block separately.

When to Use Snappy

Use Snappy When

• Latency is the primary concern. Snappy adds the least overhead to your data pipeline.
• Data is short-lived. Messages in RPC frameworks, temporary caches, inter-process communication.
• You want simplicity. No configuration, deterministic output, no versioning concerns.
• Data has low entropy. When the compression ratio matters less than the speed.
• Interoperability with Google systems. Snappy is native to Bigtable, MapReduce, Protobuf, and other Google infrastructure.

Consider Alternatives When

• Storage efficiency matters. Zstd at level 3 achieves 30-50% better ratio with modest compression time penalty and still-fast decompression.
• You need maximum speed with better ratio. LZ4 offers better ratio at comparable speed.
• Data is numerical arrays. Blosc2 with shuffle filters far outperforms Snappy on typed data.
• You need streaming. Snappy's block format requires the entire input. Use Zstd for streaming.

Comparison with LZ4

Snappy and LZ4 serve a similar niche. The main differences:

In practice, LZ4 usually compresses slightly better and decompresses slightly faster. Snappy's advantage is its simpler format and zero configuration.

Best Practices

1. Use Snappy for RPC and messaging. The combination of speed and determinism makes it ideal for network protocols.

2. Do not use Snappy for archival. The compression ratio is too low for storage use cases. Use Zstd instead.

3. Pre-allocate decompression buffers. Snappy encodes the decompressed size in the frame header, allowing you to allocate the exact output buffer.

4. Arrays with shuffle: use Blosc2 with cname: :lz4 (or :zstd), not cname: :snappy — Snappy is not a standard C-Blosc2 inner codec in ExCodecs. For pure Snappy, use this codec: ExCodecs.encode(:snappy, data).

5. Measure on your data. If Snappy provides less than 1.3:1 ratio, compression may not be worthwhile. Consider passing data through uncompressed.