Penelope v0.5.0 Penelope.ML.Text.PTBTokenizer View Source

This pipeline component adapts the treebank tokenizer to the pipeline transformer conventions. It produces a sequence of tokens for each incoming document string.

Link to this section Summary

Functions

transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents

Link to this section Functions

Link to this function transform(model, context, x) View Source
transform(model :: map(), context :: map(), x :: [String.t()]) :: [
  [String.t()]
]

transforms a list of documents into a list of token lists transforms a list of token lists into a list of documents