View Source Needlepoint.Tokenizer behaviour (Needlepoint v0.1.0)

The behaviour for tokenizing.

Tokenization is taking a string and breaking it into pieces (tokens)

Link to this section Summary

Link to this section Callbacks

@callback tokenize(String.t(), Keyword.t()) :: [String.t()]