Penelope v0.3.0 Penelope.ML.Text.PTBTokenizer
This pipeline component adapts the treebank tokenizer to the pipeline transformer conventions. It produces a sequence of tokens for each incoming document string.
Link to this section Summary
Functions
transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents
Link to this section Functions
Link to this function
transform(model, context, x)
transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents