Simple Bayes v0.2.0 SimpleBayes.Tokenizer

Summary

Functions

accumulate(map, list, acc_size)

Examples

iex> SimpleBayes.Tokenizer.accumulate(%{}, [:cat, :dog], 1)
%{cat: 1, dog: 1}

iex> SimpleBayes.Tokenizer.accumulate(%{cat: 1, fish: 1}, [:cat, :dog], 2)
%{cat: 3, fish: 1, dog: 2}
filter_out(list, filter_list)

Examples

iex> SimpleBayes.Tokenizer.filter_out(["foo", "bar", "baz"], ["baz"])
["foo", "bar"]

iex> SimpleBayes.Tokenizer.filter_out(["foo", "bar", "baz"], ["baz", "bazz"])
["foo", "bar"]
map_values(list, value)

Examples

iex> SimpleBayes.Tokenizer.map_values([:cat, :dog], 1)
%{cat: 1, dog: 1}
tokenize(string)

Examples

iex> SimpleBayes.Tokenizer.tokenize("foobar")
["foobar"]

iex> SimpleBayes.Tokenizer.tokenize("foo bar")
["foo", "bar"]

iex> SimpleBayes.Tokenizer.tokenize("foo, bar")
["foo", "bar"]

iex> SimpleBayes.Tokenizer.tokenize("foo bar.")
["foo", "bar"]