Penelope v0.5.0 Penelope.ML.Text.TokenFilter View Source

The the token filter removes tokens from a token list in the pipeline.

Link to this section Summary

Functions

removes stop word tokens from the document stream

Link to this section Functions

Link to this function transform(model, context, x) View Source
transform(
  model :: %{tokens: [String.t()]},
  context :: map(),
  x :: [[String.t()]]
) :: [[String.t()]]

removes stop word tokens from the document stream