The language model maze pages are written with.
A word-level Markov chain learnt from a corpus: for every pair of consecutive words, the words that followed them and how often. Sentences are drawn one word at a time from the two words before. Where the corpus offers a single continuation, the chain sometimes backs off to the words that followed the last word alone, so sentences recombine instead of repeating the corpus verbatim.
Everything that can be is done when the model is built: words are
HTML-escaped and turned into URL slug fragments once, and every set of
continuations becomes a Limen.Maze.Dice, so drawing a word is a map
lookup and a single random integer. Limen.Maze.Bundled holds the model of
Limen's own corpus, built at compile time; instances with a :corpus of
their own build theirs when they start.
Corpus format
Plain text, one or more sentences per line. Lines starting with # are
comments. Sentences end with ., ! or ?; a line that does not is
treated as a sentence of its own. Write prose that reads like your site:
product descriptions, articles, help pages. Very short sentences (under
three words) are skipped.
Summary
Functions
Builds a model from corpus texts.
Draws a short phrase of min..max words, for titles and link texts.
Renders word ids as a sentence: HTML-escaped words separated by spaces, ending in punctuation.
Renders a phrase: its words without punctuation, the first capitalised.
Draws a sentence, as a list of word ids.
Reads the sentences of a corpus text, as lists of words.
The number of distinct words the model knows.
A URL slug for word ids, such as "regional-freight-guide".
Renders word ids as they are: HTML-escaped words separated by spaces.
Types
@type t() :: %Limen.Maze.Model{ backoff: %{required(non_neg_integer()) => Limen.Maze.Dice.t()}, bare: tuple(), chain: %{ required({non_neg_integer(), non_neg_integer()}) => Limen.Maze.Dice.t() }, leads: tuple(), slugs: tuple(), words: tuple() }
Functions
Builds a model from corpus texts.
Raises ArgumentError when the texts hold no usable sentence.
@spec phrase(t(), Range.t(), :rand.state()) :: {[pos_integer()], :rand.state()}
Draws a short phrase of min..max words, for titles and link texts.
Phrases start after a determiner when the sentence drawn has one, so they tend to be noun phrases, and never end on a small word.
@spec render(t(), [pos_integer()]) :: iolist()
Renders word ids as a sentence: HTML-escaped words separated by spaces, ending in punctuation.
@spec render_phrase(t(), [pos_integer()]) :: String.t()
Renders a phrase: its words without punctuation, the first capitalised.
@spec sentence(t(), :rand.state()) :: {[pos_integer()], :rand.state()}
Draws a sentence, as a list of word ids.
Reads the sentences of a corpus text, as lists of words.
@spec size(t()) :: non_neg_integer()
The number of distinct words the model knows.
@spec slug(t(), [pos_integer()]) :: String.t()
A URL slug for word ids, such as "regional-freight-guide".
@spec words(t(), [pos_integer()]) :: iolist()
Renders word ids as they are: HTML-escaped words separated by spaces.