Simple Bayes v0.3.1 SimpleBayes.Tokenizer
Summary
Functions
Examples
iex> SimpleBayes.Tokenizer.accumulate(%{}, [:cat, :dog], 1)
%{cat: 1, dog: 1}
iex> SimpleBayes.Tokenizer.accumulate(%{cat: 1, fish: 1}, [:cat, :dog], 2)
%{cat: 3, fish: 1, dog: 2}
iex> SimpleBayes.Tokenizer.accumulate(%{cat: 1, fish: 1}, [:cat, :cat, :dog], 1)
%{cat: 3, fish: 1, dog: 1}
Examples
iex> SimpleBayes.Tokenizer.filter_out(["foo", "bar", "baz"], ["baz"])
["foo", "bar"]
iex> SimpleBayes.Tokenizer.filter_out(["foo", "bar", "baz"], ["baz", "bazz"])
["foo", "bar"]
Examples
iex> SimpleBayes.Tokenizer.map_values([:cat, :dog], 1)
%{cat: 1, dog: 1}
iex> SimpleBayes.Tokenizer.map_values([:cat, :cat, :dog], 1)
%{cat: 2, dog: 1}
Examples
iex> SimpleBayes.Tokenizer.tokenize("foobar")
["foobar"]
iex> SimpleBayes.Tokenizer.tokenize("foo bar")
["foo", "bar"]
iex> SimpleBayes.Tokenizer.tokenize("foo, bar")
["foo", "bar"]
iex> SimpleBayes.Tokenizer.tokenize("foo bar.")
["foo", "bar"]
iex> SimpleBayes.Tokenizer.tokenize(~s(fo-o's ba_r"ed.))
~w(fo-o's ba_r"ed)