bubble_match v0.2.1 BubbleMatch.Token View Source
A token is a single word or a part of the sentence. A sentence is a sequence of tokens.
Each token contains information and metadata that is used to match sentences on, and to extract information from.
Link to this section Summary
Functions
Test whether a token is an entity of the given kind.
Constructs a token from a Duckling entity definition
Given a single token in Spacy's JSON format, convert it into a token.
Constructs a token from a Spacy entity definition
Test whether a token mathces the given POS (part-of-speech) tag.
Test whether a token's raw value matches the given regular expression.
Test whether a token matches the given (optionally normalized) word.
Link to this section Types
Specs
t() :: %BubbleMatch.Token{ end: term(), index: term(), raw: term(), start: term(), type: term(), value: term() }
Tokens contain the following fields:
raw
- the raw text value of the token, including any surrounding whitespace.value
- the normalized value of the token. In the case of word tokens, this is usually the normalized, lowercased version of the word. In the case of entities, this value holds a map with keyskind
,provider
andvalue
.start
- the start index; where in the original sentence the token starts.end
- the end index; where in the original sentence the token ends.index
- the (zero-based) token index number; 0 if it's the first token, 1 if it's the second, etc.type
- the type of the token; an atom, holding either:entity
,:spacy
,:naive
, depending on the way the token was originally created.
Link to this section Functions
Test whether a token is an entity of the given kind.
Constructs a token from a Duckling entity definition
Specs
Given a single token in Spacy's JSON format, convert it into a token.
Constructs a token from a Spacy entity definition
Test whether a token mathces the given POS (part-of-speech) tag.
Test whether a token's raw value matches the given regular expression.
Test whether a token matches the given (optionally normalized) word.