Makeup.Lexers.DiffLexer (MakeupDiff v0.1.0)

View Source

A Makeup lexer for diffs and patches.

Diffs are a line-oriented format, so — unlike a lexer for a programming language — this lexer classifies whole lines rather than individual syntactic atoms. The following dialects are recognised:

  • Unified diffs (diff -u, git diff, hg diff, svn diff), including the @@ -old,count +new,count @@ section hunk headers and the @@@ ... @@@ headers produced by combined (merge) diffs.
  • Context diffs (diff -c): the *** old / --- new file banner, the *************** hunk separator, the *** 1,5 **** and --- 1,7 ---- range lines, and the ! change marker.
  • Normal diffs (diff with no flags): the 1,5c1,7 / 3d2 / 0a1 command lines and the < / > content markers.
  • Git extended headers: diff --git, index, old mode, new mode, new file mode, deleted file mode, copy from/copy to, rename from/rename to, similarity index, dissimilarity index, Binary files ... differ, and GIT binary patch.
  • Subversion / CVS scaffolding: Index: path, the ====... rule, Property changes on:, RCS file:, retrieving revision.
  • git format-patch mail preambles: the mbox From <sha> Mon Sep 17 line and the usual From: / Date: / Subject: / MIME headers.
  • The \ No newline at end of file marker.

Token mapping

  • :generic_inserted — added lines (+, >)
  • :generic_deleted — removed lines (-, <)
  • :generic_strong — changed lines in context diffs (!)
  • :generic_subheading — hunk headers, range lines, file banners
  • :generic_heading — repository-level metadata (diff --git, index, ...)
  • :name_function — the section hint trailing a unified hunk header
  • :comment_preproc — mail headers of a git format-patch preamble
  • :comment_special — the \ No newline at end of file marker
  • :text — context lines and anything unrecognised

Ambiguity of --- and +++

A line starting with --- is either the old file banner of a unified diff or the removal of a line whose content begins with --. The same holds for +++. The tokenizer therefore emits a provisional token for both and postprocess/2 resolves it from the surrounding lines: a --- immediately followed by a +++ is a banner pair, a --- right after a repository-level heading (or at the very start of the input) is a banner, and everything else is ordinary added/removed content. This is why --- inside a hunk (right after an @@ header, say) is correctly highlighted as a deleted line.

Registering the lexer

The lexer is automatically registered on application start for the language names "diff", "udiff", "patch" and the file extensions ".diff" and ".patch".

Summary

Functions

root(binary, opts \\ [])

Parses the given binary as root.

root_element(binary, opts \\ [])

Parses the given binary as root_element.

Functions

root(binary, opts \\ [])

@spec root(binary(), keyword()) ::
  {:ok, [term()], rest, context, line, byte_offset}
  | {:error, reason, rest, context, line, byte_offset}
when line: {pos_integer(), byte_offset},
     byte_offset: non_neg_integer(),
     rest: binary(),
     reason: String.t(),
     context: map()

Parses the given binary as root.

Returns {:ok, [token], rest, context, position, byte_offset} or {:error, reason, rest, context, line, byte_offset} where position describes the location of the root (start position) as {line, offset_to_start_of_line}.

To column where the error occurred can be inferred from byte_offset - offset_to_start_of_line.

Options

  • :byte_offset - the byte offset for the whole binary, defaults to 0
  • :line - the line and the byte offset into that line, defaults to {1, byte_offset}
  • :context - the initial context value. It will be converted to a map

root_element(binary, opts \\ [])

@spec root_element(binary(), keyword()) ::
  {:ok, [term()], rest, context, line, byte_offset}
  | {:error, reason, rest, context, line, byte_offset}
when line: {pos_integer(), byte_offset},
     byte_offset: non_neg_integer(),
     rest: binary(),
     reason: String.t(),
     context: map()

Parses the given binary as root_element.

Returns {:ok, [token], rest, context, position, byte_offset} or {:error, reason, rest, context, line, byte_offset} where position describes the location of the root_element (start position) as {line, offset_to_start_of_line}.

To column where the error occurred can be inferred from byte_offset - offset_to_start_of_line.

Options

  • :byte_offset - the byte offset for the whole binary, defaults to 0
  • :line - the line and the byte offset into that line, defaults to {1, byte_offset}
  • :context - the initial context value. It will be converted to a map