Pantagruel v0.0.8 Pantagruel.Scan View Source

Preprocessing for Pantagruel. Prepares a raw text file for parsing.

Link to this section Summary

Functions

Magic unicode character inserted into comments to indicate where to split lines

Processes a text file in several passes:

  • Removes all commented areas.
  • Replaces specific character sequences with their unicode equivalents. This allows certain ascii combinations to be treated specially by the parser: arrows are combined into a single character, newline continuations are collapsed and removed, and a small set of identifier-legal characters, if encountered alone, are treated as keywords.
  • Unnecessary spaces and whitespace are removed

Link to this section Functions

Magic unicode character inserted into comments to indicate where to split lines.

Link to this macro is_operator(c) View Source (macro)

Processes a text file in several passes:

  • Removes all commented areas.
  • Replaces specific character sequences with their unicode equivalents. This allows certain ascii combinations to be treated specially by the parser: arrows are combined into a single character, newline continuations are collapsed and removed, and a small set of identifier-legal characters, if encountered alone, are treated as keywords.
  • Unnecessary spaces and whitespace are removed.