# Understanding Snappy Snappy (formerly Zippy) is a fast compression codec developed by Google, focused on maximum speed with minimal configuration. This guide covers Snappy's format, performance characteristics, and when to use it. ## Overview Snappy was designed at Google for internal use in their distributed infrastructure (Bigtable, MapReduce, etc.). Its priorities, in order, are: 1. **Decompression speed** -- as fast as possible. 2. **Compression speed** -- nearly as fast as decompression. 3. **Simplicity** -- no configuration knobs, deterministic output. 4. **Compression ratio** -- secondary to speed. Snappy achieves compression speeds over 500 MB/s and decompression speeds over 1.5 GB/s, making it suitable for latency-sensitive systems where the compression ratio is not the primary concern. ## How Snappy Works Snappy is an LZ77-family algorithm with aggressive speed optimizations: ### Matching Strategy 1. Scan the input linearly. 2. At each position, hash the next 4 bytes and look up a hash table. 3. If a match is found that is at least 4 bytes long (up to 64 KB back), emit a copy command. 4. If no match is found, emit the current byte as a literal. 5. Moves forward by only 1 byte after a literal (no lazy or optimal parsing). ### Speed Optimizations Snappy makes specific design choices that sacrifice ratio for speed: - **No entropy coding.** Literals and match lengths are stored in a simple variable-length encoding, not Huffman or arithmetic coding. - **No optimal parsing.** The first match found is always used. Snappy does not consider whether skipping a match could lead to a longer one later. - **No match length extension.** Snappy does not extend matches beyond the initial find. Some implementations do, but the core format does not require it. - **Minimal branching.** The decoder is a simple state machine with few conditional branches, which is CPU-cache and branch-predictor friendly. ## Snappy Format ExCodecs uses the **raw Snappy block** format (not the separate Snappy framing format). A block is an uncompressed-length prefix followed by tagged elements. ### Uncompressed Length Prefix ``` +-------------------+ | Length (varint) | -- Uncompressed length in bytes +-------------------+ ``` The uncompressed size is encoded as a variable-length integer at the start of the compressed block so the decompressor can size the output buffer. ### Elements Each element starts with a tag byte. The lower two bits select the element type: | Tag (bits 1-0) | Type | |----------------|------| | `00` | Literal (raw bytes) | | `01` | Copy with 1-byte offset (length 4-11, offset 0-2047) | | `10` | Copy with 2-byte offset (length 1-64, offset 0-65535) | | `11` | Copy with 4-byte offset (legacy; length 1-64) | ``` Tag byte: bits 7-2: length or offset info (depends on type) bits 1-0: element type ``` For literals, short lengths fit in the tag; longer literals use a 1-4 byte length field. Copies encode offset and length as described in Google's Snappy format description. Typical compressor windows are up to 64 KB. ### Format Properties - **Self-contained.** Each Snappy-compressed block contains all the information needed for decompression. No external dictionary or state is required. - **Deterministic.** Identical inputs always produce identical outputs. This is useful for content-addressable storage and caching. - **Single-pass.** Both compression and decompression are single-pass with no backtracking. ## Performance Characteristics Speeds and ratios below are illustrative order-of-magnitude figures from typical upstream C Snappy discussions; ExCodecs uses the pure-Rust `snap` crate. Benchmark your own data. ### Compression Speed Snappy is designed for very high compression throughput. Key factors: - The hash table is small and fits in L1/L2 cache. - No expensive operations (no sorting, no entropy coding, no optimal parsing). - Good SIMD acceleration on x86 and ARM. ### Decompression Speed Decompression is similarly speed-oriented. The decompressor: - Reads the tag byte to determine the element type. - For literals: copies bytes verbatim (essentially a memcpy). - For copies: reads the offset and length, then copies from the already-decompressed output. ### Compression Ratio Snappy's ratio is lower than LZ4, Zstd, and Bzip2. Illustrative ratios: | Data Type | Snappy Ratio | LZ4 Ratio | Zstd Level 3 Ratio | |-------------|--------------|-----------|---------------------| | English text | 1.8:1 | 2.0:1 | 3.0:1 | | JSON | 1.7:1 | 1.9:1 | 2.8:1 | | HTML | 2.0:1 | 2.2:1 | 3.5:1 | | Already compressed | 1.0:1 | 1.0:1 | 1.0:1 | Snappy's lower ratio is the expected tradeoff for its speed advantage. ## Using Snappy in ExCodecs Snappy has no encode-time configuration knobs (`configurable?: false`): ```elixir # Compress {:ok, compressed} = ExCodecs.encode(:snappy, data) # Decompress {:ok, original} = ExCodecs.decode(:snappy, compressed) ``` Encode ignores option keywords. Decode accepts `:max_output_size` (same decompression-bomb guard as other codecs): ```elixir {:ok, compressed} = ExCodecs.encode(:snappy, data) {:ok, original} = ExCodecs.decode(:snappy, compressed, max_output_size: 1_048_576) ``` ```elixir {:ok, info} = ExCodecs.codec_info(:snappy) info.configurable? # => false info.streaming? # => false ``` ## Streaming Support Snappy supports a framing format (the "Snappy framing format" or "snappy-framed") in its full specification, which allows incremental compression and decompression of data streams. ExCodecs currently provides block-level Snappy compression and decompression. The entire input must be available before compression can begin. For streaming use cases, consider chunking the data into blocks and compressing each block separately. ## When to Use Snappy ### Use Snappy When - **Latency is the primary concern.** Snappy adds the least overhead to your data pipeline. - **Data is short-lived.** Messages in RPC frameworks, temporary caches, inter-process communication. - **You want simplicity.** No configuration, deterministic output, no versioning concerns. - **Data has low entropy.** When the compression ratio matters less than the speed. - **Interoperability with Google systems.** Snappy is native to Bigtable, MapReduce, Protobuf, and other Google infrastructure. ### Consider Alternatives When - **Storage efficiency matters.** Zstd at level 3 achieves 30-50% better ratio with modest compression time penalty and still-fast decompression. - **You need maximum speed with better ratio.** LZ4 offers better ratio at comparable speed. - **Data is numerical arrays.** Blosc2 with shuffle filters far outperforms Snappy on typed data. - **You need streaming.** Snappy's block format requires the entire input. Use Zstd for streaming. ## Comparison with LZ4 Snappy and LZ4 serve a similar niche. The main differences: | Property | Snappy | LZ4 | |--------------------|---------------------|-----------------------------| | Compression Speed | Fast | Fast (fixed fast profile) | | Decompression Speed| Fast | Very fast | | Ratio | Slightly lower | Slightly higher | | Configuration | None | None (no levels in ExCodecs) | | Format complexity | Simple | Slightly more complex | | Deterministic | Yes | Yes | | Max offset | 64 KB | 64 KB | In practice, LZ4 usually compresses slightly better and decompresses slightly faster. Snappy's advantage is its simpler format and zero configuration. ## Best Practices 1. **Use Snappy for RPC and messaging.** The combination of speed and determinism makes it ideal for network protocols. 2. **Do not use Snappy for archival.** The compression ratio is too low for storage use cases. Use Zstd instead. 3. **Pre-allocate decompression buffers.** Snappy encodes the decompressed size in the frame header, allowing you to allocate the exact output buffer. 4. **Arrays with shuffle:** use Blosc2 with `cname: :lz4` (or `:zstd`), not `cname: :snappy` — Snappy is not a standard C-Blosc2 inner codec in ExCodecs. For pure Snappy, use this codec: `ExCodecs.encode(:snappy, data)`. 5. **Measure on your data.** If Snappy provides less than 1.3:1 ratio, compression may not be worthwhile. Consider passing data through uncompressed.