View Source AI.Tokenizer.Default (fnord v0.6.4)

Implements the default tokenizer behaviour for AI.Tokenizer.