# Parity and Performance

`FastestTiktoken` is designed to provide full public-behavior parity with the
official OpenAI [`tiktoken`](https://github.com/openai/tiktoken) `0.13.0`
library while using a faster Rust implementation underneath.

## Official OpenAI tiktoken Parity

The test suite ports the official OpenAI `tiktoken` public behavior tests into
Elixir for the API surfaces this package exposes.

Covered parity areas:

- Encoding and decoding examples for GPT-2/r50k and `cl100k_base`.
- Official model-to-encoding mappings and model-prefix mappings.
- Regex edge cases from upstream `cl100k_base` tests.
- Repeated-token fixtures for GPT-2/r50k behavior.
- Empty input and large repeated input handling.
- Unicode and multilingual roundtrips.
- Special token behavior for ordinary mode, all-special mode, and allowed
  special-token subsets.
- `o200k_harmony` model resolution and special-token behavior for GPT OSS
  models.
- Batch encode/decode helpers.

Run the parity suite with:

```bash
FASTEST_TIKTOKEN_BUILD=1 mix test --only official_openai_parity
```

Or run all tests:

```bash
FASTEST_TIKTOKEN_BUILD=1 mix test
```

OpenAI's `o200k_harmony` encoding uses the same merge table and regex as
`o200k_base`, plus the harmony-specific special-token table from official
`tiktoken` `0.13.0`. The wrapped Rust crate does not expose this encoding
directly, so `FastestTiktoken` layers those official special tokens over the
fast `o200k_base` tokenizer.

## Why This Is Faster Than Older Elixir Wrappers

Compared with other Elixir tokenizer projects, older packages commonly wrap
[`tiktoken-rs`](https://crates.io/crates/tiktoken-rs). `FastestTiktoken` wraps
the newer pure-Rust [`tiktoken`](https://crates.io/crates/tiktoken) crate
instead.

The wrapped crate provides:

- Cached tokenizer instances.
- A zero-allocation count path for ordinary token counting.
- A fast pure-Rust BPE implementation.
- OpenAI-compatible encodings plus additional non-OpenAI tokenizer families
  exposed by the Rust crate.

`FastestTiktoken` keeps the Elixir side thin: validation and official model
mapping happen in Elixir, while tokenization work happens in Dirty CPU NIFs.

## Benchmark Script

The repository includes a simple local comparison script:

```bash
FASTEST_TIKTOKEN_BUILD=1 elixir bench/compare_tiktoken.exs
```

It compares count and encode throughput against the older Hex `:tiktoken`
package. Treat it as a local smoke benchmark rather than a formal benchmark
suite; results depend on CPU, OTP version, target architecture, input text, and
whether precompiled or source-built NIFs are used.