View Source Tokenizers.DecodeStream (Tokenizers v0.5.2)

Implements streaming decoding functionality for tokenizers.

Summary

Types

t()

Functions

info(decode_stream)

Returns information about the decode stream state.

new(opts \\ [])

Creates a new decode stream.

step(decode_stream, tokenizer, id)

Steps through the decode stream with the given tokenizer and token ID.

Types

Link to this type

t()

View Source 
@type t() :: %Tokenizers.DecodeStream{resource: reference()}

Functions

Link to this function

info(decode_stream)

View Source

Returns information about the decode stream state.

Link to this function

new(opts \\ [])

View Source 
@spec new(keyword()) :: t()

Creates a new decode stream.

Options

  • :skip_special_tokens - determines whether special tokens should be skipped during decoding. By default, it is set to false.
Link to this function

step(decode_stream, tokenizer, id)

View Source

Steps through the decode stream with the given tokenizer and token ID.

Returns {:ok, String.t()} if there's a decoded string, or {:ok, :out_ofr_range} if the token ID is out of range. Returns {:error, reason} if an error occurs during decoding.