Gibran

This module includes shortcut functions that bridge Gibran.Tokeniser and Gibran.Counter. This allows the caller to directly work with strings instead of tokenising the input first before applying any calculations.

Summary

Functions

Takes a string and an atom that refers to any function name in Gibran.Counter. It tokenises the string, then applies the given function to the resulting tokens. The function name must be an atom. You can pass a list of options to the tokeniser as opts and a list of options to the receiving function as fn_opts

Functions

from_string(input, func)

Takes a string and an atom that refers to any function name in Gibran.Counter. It tokenises the string, then applies the given function to the resulting tokens. The function name must be an atom. You can pass a list of options to the tokeniser as opts and a list of options to the receiving function as fn_opts.

For example, the following two calls are equivalent:

Gibran.from_string("The Prophet", :token_count)
Gibran.Tokeniser.tokenise("The Prophet") |> Gibran.Counter.token_count

Examples

iex> Gibran.from_string("The Prophet", :token_count)
2
iex> Gibran.from_string("The Prophet", :token_count, opts: [exclude: "the"])
1
iex> Gibran.from_string("Eye of The Prophet", :average_chars_per_token, fn_opts: [precision: 1])
3.8
iex> Gibran.from_string("Eye of The Prophet", :average_chars_per_token, opts: [exclude: "of"], fn_opts: [precision: 4])
4.3333

The following functions are available.

["average_chars_per_token/1", "average_chars_per_token/2", "char_count/1", "longest_tokens/1", "most_frequent_tokens/1", "token_count/1", "token_frequency/1", "token_lengths/1", "uniq_token_count/1", "uniq_tokens/1"]

To view all available functions at anytime see Gibran.Counter or type the following into iex:

Gibran.Counter.__info__(:functions)

See Gibran.Tokeniser.tokenise/2 and Gibran.Counter for more information.

from_string(input, func, list3)