Penelope v0.4.0 Penelope.ML.Text.PTBDigitTokenizer
This pipeline component adapts the treebank tokenizer + the digit token preprocessor to the pipeline transformer conventions. It produces a sequence of tokens for each incoming document string.
Link to this section Summary
Functions
transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents
Link to this section Functions
Link to this function
transform(model, context, x)
transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents