Ichor.Toolkit.Layout (Ichor v0.2.1)

Copy Markdown View Source

The off-side-rule (indentation-sensitive layout) algorithm Python's own tokenizer uses, and Haskell's/F#'s: given each logical line's indentation width, in order, maintain a stack of open indentation levels and emit :indent/:dedent markers as the width rises or falls -- so a parser downstream never has to know about columns at all, just consumes ordinary tokens.

This is deliberately a different strategy from Ichor's own Grammar.IR.Indent/@samecol (column-position checks embedded directly inside grammar rules as zero-width assertions) -- not a replacement for it, and not something Ichor's core needs, since it already made that different design choice. It exists for a grammar author porting a real language whose own reference implementation synthesizes INDENT/DEDENT pseudo-tokens at the lexer level (as Python's, Haskell's, and F#'s all do) and would rather mirror that approach directly than rewrite it as embedded column checks.

Deliberately unopinionated about what counts as a line worth indentation-checking at all (blank lines, comment-only lines, and lines continued inside brackets are all real, genuinely language-specific policy choices) -- a caller decides which lines to feed in and what each one's width is; this only implements the mechanical stack-based algorithm on top of that decision.

Summary

Functions

Every :dedent needed to close all remaining open levels at end of input, leaving only the base level.

Given the next logical line's width and the current stack of open indentation levels ([0] for a fresh start), returns the markers to emit before that line's own content, and the updated stack.

Types

marker()

@type marker() :: :indent | :dedent

width()

@type width() :: non_neg_integer()

Functions

close(stack)

@spec close([width(), ...]) :: [marker()]

Every :dedent needed to close all remaining open levels at end of input, leaving only the base level.

step(width, stack)

@spec step(width(), [width(), ...]) ::
  {:ok, [marker()], [width(), ...]}
  | {:error, {:inconsistent_dedent, width(), [width()]}}

Given the next logical line's width and the current stack of open indentation levels ([0] for a fresh start), returns the markers to emit before that line's own content, and the updated stack.

A width greater than the current top pushes a new level and emits one :indent. An equal width emits nothing. A lesser width pops every level greater than width, emitting one :dedent per pop -- an error if popping never lands exactly on width (dedenting to a level that was never open, e.g. mismatched indentation using tabs vs. spaces).