BinClass.Vectorizer (BinClass v0.3.0)

Copy Markdown View Source

Encodes text into fixed-length token ID vectors with explicit padding semantics.

Summary

Functions

build(tokenizer, string, vector_length, pad_token_id)

Encodes one string into exactly vector_length token IDs.

build_with_active_token_count(tokenizer, string, vector_length, pad_token_id)

Encodes one string and also returns its active token count.

Functions

build(tokenizer, string, vector_length, pad_token_id)

Encodes one string into exactly vector_length token IDs.

Tokens beyond the vector length are truncated. Shorter encodings are padded with pad_token_id, which should come from BinClass.Tokenizer.metadata/1.

build_with_active_token_count(tokenizer, string, vector_length, pad_token_id)

Encodes one string and also returns its active token count.

Returns {token_ids, active_token_count}. The token IDs are truncated and padded to vector_length; the count is calculated after truncation and excludes IDs equal to pad_token_id.