Fuzler

Fuzler is a fast, configurable lexical similarity scorer for Elixir. Its Rust NIF combines Unicode-aware edit distance, token overlap, and bounded partial matching without blocking the BEAM's normal schedulers.

Fuzler.similarity_score("kitten", "sitting")
#=> 0.57

Scores are symmetric, deterministic, rounded to two decimal places, and always between 0.0 and 1.0.

Installation

Add Fuzler to mix.exs:

def deps do
  [
    {:fuzler, "~> 0.1.3"}
  ]
end

Then run mix deps.get.

Fuzler requires Elixir 1.18 or later. Precompiled NIF archives are provided for:

• Linux: aarch64 (GNU and musl), arm-gnueabihf, riscv64gc-gnu, and x86_64 (GNU and musl)
• macOS: Apple Silicon and Intel
• Windows: x86_64 GNU and MSVC

Building a development checkout or an unsupported target requires Rust 1.82 or later. A consuming project that forces a source build must also include Rustler:

def deps do
  [
    {:rustler, "~> 0.36.2"},
    {:fuzler, "~> 0.1.3"}
  ]
end

RUSTLER_PRECOMPILATION_EXAMPLE_BUILD=1 mix compile --force

Basic scoring

iex> Fuzler.similarity_score("Hello, WORLD!", "hello world")
1.0

iex> Fuzler.similarity_score("bella ciao", "ciao bella")
0.7

iex> Fuzler.similarity_score("café", "cafe\u0301")
1.0

iex> Fuzler.similarity_score("needle", "some text with needle inside")
0.6

Useful reference points:

Scores are ranking signals rather than universal thresholds. The reviewed golden corpus currently has perfect precision and recall at 0.5, but applications should calibrate a threshold on their own representative data.

Detailed comparisons

compare/2 exposes the components that produced the final score:

iex> Fuzler.compare("needle", "some text with needle inside")
%Fuzler.Comparison{
  score: 0.6,
  full_score: 0.2,
  partial_score: 0.6,
  token_score: 0.2,
  character_score: 0.21,
  matched_text: "needle",
  algorithm_version: "1"
}

token_score is nil for two single-token inputs. The token and character components describe the full-input comparison; matched_text contains the normalised target window only when partial matching wins.

Batch scoring and ranking

Crossing the NIF boundary once is substantially cheaper than making many individual calls. similarity_scores/2 preserves input order:

iex> Fuzler.similarity_scores("milano", ["Roma", "Milano", "Milano Centrale"])
[0.17, 1.0, 0.75]

top_matches/3 returns ranked values and their zero-based source indexes:

iex> Fuzler.top_matches("milano", ["Roma", "Milano", "Milano Centrale"], 2)
[
  %Fuzler.Match{value: "Milano", score: 1.0, index: 1},
  %Fuzler.Match{value: "Milano Centrale", score: 0.75, index: 2}
]

Equal scores retain input order.

Options

All scoring, comparison, and batch functions accept an options argument. For example:

Fuzler.similarity_score("café", "cafe", strip_diacritics: true)
#=> 1.0

Fuzler.compare(query, target, partial: false, token_weight: 0.5)
Fuzler.similarity_scores(query, targets, max_batch_size: 500)
Fuzler.top_matches(query, targets, 10, normalization: :nfc)

Unknown options and invalid values raise ArgumentError.

Unicode and matching behaviour

By default Fuzler:

• lowercases Unicode text;
• applies NFKC compatibility normalisation;
• preserves accents while making composed and decomposed forms equivalent;
• turns punctuation and other separators into token boundaries;
• collapses repeated whitespace;
• compares non-ASCII text by Unicode grapheme clusters.

Consequently, "foo-bar" equals "foo bar", "café" equals its decomposed Unicode representation, and full-width "ＦＵＺＬＥＲ" equals "fuzler". Set normalization: :nfc to disable compatibility folding, or strip_diacritics: true to make "café" and "cafe" equivalent.

Exact grapheme Levenshtein is bounded to 4,096 graphemes to prevent quadratic work from monopolising a dirty scheduler. Larger Unicode comparisons fall back to the SIMD byte path and remain subject to the configured input limits.

For multi-token text, the full score blends multiset Jaccard overlap with character similarity. The default blend is 70% token overlap and 30% character similarity. Repeated tokens are counted, while word order is ignored by the token component.

When the shorter input has at most 20 tokens, partial matching searches bounded windows in the longer input. Strong contained phrases receive a useful score, but coverage penalties prevent a small phrase from becoming identical to an entire document. Weak windows are discarded rather than accumulated.

algorithm_version/0 returns "1". Applications that persist scores can save this value alongside them when future algorithm revisions need to coexist.

Errors and resource safety

Inputs must be valid UTF-8 binaries. Non-binary arguments raise FunctionClauseError; malformed UTF-8 and exceeded resource limits raise ArgumentError. An unexpected native panic is caught at the NIF boundary and reported as RuntimeError instead of crashing the VM.

Every NIF is scheduled as dirty CPU work. Batch size, per-input bytes, normalised tokens, and total batch bytes are bounded by validated options.

Fuzler measures lexical similarity. It does not provide stemming, synonyms, semantic embeddings, or full-text indexing.

Development and quality

The complete local gate is available through Task:

task test             # ExUnit, doctests, properties, and coverage
task rust-test        # Rust unit and property tests
task benchmark-check  # deterministic performance budgets
task package-smoke    # build and consume the Hex tar in isolation
task check            # complete non-fuzzing quality gate
task fuzz             # continuous libFuzzer session; stop with Ctrl-C

The behavioural suite contains 150 explicit name, city, phrase, structured, typo, partial, and unrelated examples. A separate versioned golden set checks reviewed score intervals, precision/recall, and candidate rankings. StreamData and Proptest add hundreds of generated Unicode cases per run. CI also performs a timed libFuzzer run.

The release workflow builds ten native targets, validates archive integrity and version metadata, and generates checksum-Elixir.Fuzler.Native.exs. The package smoke test extracts the exact Hex tar into an isolated consumer, compiles the native source, loads the NIF, and calls the public API.

See CHANGELOG.md for the full release notes.

License

Fuzler is released under the MIT License.