Penelope v0.3.0 Penelope.ML.Text.PTBDigitTokenizer

This pipeline component adapts the treebank tokenizer + the digit token preprocessor to the pipeline transformer conventions. It produces a sequence of tokens for each incoming document string.

Link to this section Summary

Functions

transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents

Link to this section Functions

Link to this function transform(model, context, x)
transform(model :: map(), context :: map(), x :: [String.t()]) :: [[String.t()]]

transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents