Pantagruel v0.0.5 Pantagruel.Scan View Source
Preprocessing for Pantagruel. Prepares a raw text file for parsing.
Link to this section Summary
Functions
Magic unicode character inserted into comments to indicate where to split lines
Processes a text file in several passes:
- Removes all commented areas.
- Replaces specific character sequences with their unicode equivalents. This allows certain ascii combinations to be treated specially by the parser: arrows are combined into a single character, newline continuations are collapsed and removed, and a small set of identifier-legal characters, if encountered alone, are treated as keywords.
- Unnecessary spaces and whitespace are removed
Link to this section Functions
Magic unicode character inserted into comments to indicate where to split lines.
Processes a text file in several passes:
- Removes all commented areas.
- Replaces specific character sequences with their unicode equivalents. This allows certain ascii combinations to be treated specially by the parser: arrows are combined into a single character, newline continuations are collapsed and removed, and a small set of identifier-legal characters, if encountered alone, are treated as keywords.
- Unnecessary spaces and whitespace are removed.