Obscura.Tiktoken (obscura v0.1.0)

Tiktoken-compatible byte-pair encodings used by Obscura.

This module intentionally implements the subset of Python tiktoken that Obscura needs for privacy-filter tokenization. It does not fetch assets from the network at runtime.

Summary

Types

encoding()

Functions

encoding_for_model(model_name)

encoding_name_for_model(model_name)

get_encoding(name)

Loads a supported encoding from verified local assets.

get_encoding!(name)

list_encoding_names()

Lists the supported encoding names.

Types

encoding()

@type encoding() :: Obscura.Tiktoken.Encoding.t()

Functions

encoding_for_model(model_name)

@spec encoding_for_model(String.t()) :: {:ok, encoding()} | {:error, term()}

encoding_name_for_model(model_name)

@spec encoding_name_for_model(String.t()) :: {:ok, String.t()} | {:error, term()}

get_encoding(name)

@spec get_encoding(String.t() | atom()) :: {:ok, encoding()} | {:error, term()}

Loads a supported encoding from verified local assets.

get_encoding!(name)

@spec get_encoding!(String.t() | atom()) :: encoding()

list_encoding_names()

@spec list_encoding_names() :: [String.t()]

Lists the supported encoding names.