Penelope v0.2.5 Penelope.ML.Text.PTBTokenizer

This pipeline component adapts the treebank tokenizer to the pipeline transformer conventions. It produces a sequence of tokens for each incoming document string.

Link to this section Summary

Functions

transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents

Link to this section Functions

Link to this function transform(model, context, x)
transform(model :: map(), context :: map(), x :: [String.t()]) :: [[String.t()]]

transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents