Gibran
This module includes shortcut functions that bridge Gibran.Tokeniser
and Gibran.Counter
.
This allows the caller to directly work with strings instead of tokenising the
input first before applying any calculations.
Summary
Functions
Takes a string and an atom that refers to any function name in Gibran.Counter
.
It tokenises the string, then applies the given function to the resulting tokens. The function
name must be an atom
. You can pass a list of options to the tokeniser as opts
and a list
of options to the receiving function as fn_opts
Functions
Takes a string and an atom that refers to any function name in Gibran.Counter
.
It tokenises the string, then applies the given function to the resulting tokens. The function
name must be an atom
. You can pass a list of options to the tokeniser as opts
and a list
of options to the receiving function as fn_opts
.
For example, the following two calls are equivalent:
Gibran.from_string("The Prophet", :token_count)
Gibran.Tokeniser.tokenise("The Prophet") |> Gibran.Counter.token_count
Examples
iex> Gibran.from_string("The Prophet", :token_count)
2
iex> Gibran.from_string("The Prophet", :token_count, opts: [exclude: "the"])
1
iex> Gibran.from_string("Eye of The Prophet", :average_chars_per_token, fn_opts: [precision: 1])
3.8
iex> Gibran.from_string("Eye of The Prophet", :average_chars_per_token, opts: [exclude: "of"], fn_opts: [precision: 4])
4.3333
The following functions are available.
["average_chars_per_token/1", "average_chars_per_token/2", "char_count/1", "longest_tokens/1", "most_frequent_tokens/1", "token_count/1", "token_frequency/1", "token_lengths/1", "uniq_token_count/1", "uniq_tokens/1"]
To view all available functions at anytime see Gibran.Counter
or type the following into iex:
Gibran.Counter.__info__(:functions)
See Gibran.Tokeniser.tokenise/2
and Gibran.Counter
for more information.