Tokenizer boundary for encoding and decoding text.
Summary
Callbacks
@callback decode([non_neg_integer()], keyword()) :: {:ok, String.t()} | {:error, term()}
@callback encode(String.t(), keyword()) :: {:ok, [non_neg_integer()]} | {:error, term()}
Tokenizer boundary for encoding and decoding text.
@callback decode([non_neg_integer()], keyword()) :: {:ok, String.t()} | {:error, term()}
@callback encode(String.t(), keyword()) :: {:ok, [non_neg_integer()]} | {:error, term()}