Sourceror (Sourceror v0.7.1) View Source
Installation
Add :sourceror
as a dependency to your project's mix.exs
:
defp deps do
[
{:sourceror, "~> 0.7.1"}
]
end
A note on compatibility
Sourceror is compatible with Elixir versions down to 1.10 and OTP 21. For Elixir
versions prior to 1.13 it uses a vendored version of the Elixir parser and
formatter modules. This means that for Elixir versions prior to 1.12 it will
successfully parse the new syntax for stepped ranges instead of raising a
SyntaxError
, but everything else should work as expected.
Goals of the library
- Be as close as possible to the standard Elixir AST.
- Make working with comments as simple as possible.
- No runtime dependencies, to simplify integration with other tools.
Background
There have been several attempts at source code manipulation in the Elixir
community. Thanks to its metaprogramming features, Elixir provides builtin tools
that let us get the AST of any Elixir code, but when it comes to turning
the AST back to code as text, we had limited options. Macro.to_string/2
is a
thing, but the produced code is generally ugly, mostly because of the extra
parenthesis or because it turns string interpolations into calls to erlang
modules, to name some examples. This meant that, even if we could use
Macro.to_string/2
to get a string and then give that to the Elixir formatter
Code.format_string!/2
, the output would still be suboptimal, as the formatter
is not designed to change the semantics of the code, only to pretty print it.
For example, call to erlang modules would be kept as is instead of being turned
back to interpolations.
We also had the additional problem of comments being discarded by the tokenizer, and literals not having information like line numbers or delimiter characters. This makes the regular AST too lossy to be useful if what we want is to manipulate the source code, because we need as much information as possible to be able to stay as close to the source as possible. There have been several proposal in the past to bring all this information to the Elixir AST, but they all meant a change that would either break macros due to the addition of new types of AST nodes, or making a compromise in core Elixir itself by storing comments in the nods metadata. This discussion in the Elixir mailing list highlights the various issues faced when deciding if and how the comments would be preserved. Arjan Scherpenisse also did a talk where he discusses about the problems of using the standard Elixir AST to build refactoring tools.
Despite of all these issues, the Elixir formatter is still capable of manipulating the source code to pretty print it. Under the hood it does some neat tricks to have all this information available: on one hand, it tells the tokenizer to extract the comments from the source code and keep it at hand(not in the AST itself, but as a separate data structure), and on the other hand it tells the parser to wrap literals in block nodes so metadata can be preserved. Once it has all it needs, it can start converting the AST and comments into an algebra document, and ultimately convert that to a string. This functionality was private, and if we wanted to do it ourselves we would have to replicate or vendor the Elixir formatter with all its more than 2000 lines of code. This approach was explored by Wojtek Mach in wojtekmach/fix, but it involved vendoring the elixir Formatter code, was tightly coupled to the formatting process, and any change in Elixir would break the code.
Since Elixir 1.13 this functionality from the formatter was finally exposed via
the Code.string_to_quoted_with_comments/2
and Code.quoted_to_algebra/2
functions. The former gives us access to the list of comments in a shape the
Elixir formatter is able to use, and the latter lets us turn any arbitrary
Elixir AST into an algebra document. If we also give it the list of comments,
it will merge them together, allowing us to format AST and preserve the
comments. Now all we need to care about is of manipulating the AST, and let the
formatter do the rest.
Sourceror's AST
Having the AST and comments as separate entities allows Elixir to expose the code formatting utilities without making any changes to it's AST, but also delegates the task of figuring out what's the most appropiate way to work with them to us.
Sourceror's take is to use the node metadata to store the comments. This allows us to work with an AST that is as close to regular elixir AST as possible. It also allows you to move nodes around without worrying about leaving a comment behind and ending up with misplaced comments.
Two metadata fields are added to the regular Elixir AST:
:leading_comments
- holds the comments directly above the node or are in the same line as it. For example:test "parses leading comments" do quoted = """ # Comment for :a :a # Also a comment for :a """ |> Sourceror.parse_string!() assert {:__block__, meta, [:a]} = quoted assert meta[:leading_comments] == [ %{line: 1, previous_eol_count: 1, next_eol_count: 1, text: "# Comment for :a"}, %{line: 2, previous_eol_count: 0, next_eol_count: 1, text: "# Also a comment for :a"}, ] end
:trailing_comments
- holds the comments that are inside of the node, but aren't leading any children, for example:test "parses trailing comments" do quoted = """ def foo() do :ok # A trailing comment end # Not a trailing comment for :foo """ |> Sourceror.parse_string!() assert {:__block__, block_meta, [{:def, meta, _}]} = quoted assert [%{line: 3, text: "# A trailing comment"}] = meta[:trailing_comments] assert [%{line: 4, text: "# Not a trailing comment for :foo"}] = block_meta[:trailing_comments] end
Note that Sourceror considers leading comments to the ones that are found in the
same line as a node, and trailing coments to the ones that are found before the
ending line of a node, based on the end
, closing
or end_of_expression
line. This also makes the Sourceror AST consistent with the way the Elixir
formatter works, making it easier to reason about how a given AST would be
formatted.
License
Copyright (c) 2021 dorgandash@gmail.com
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Link to this section Summary
Functions
Appends comments to the leading or trailing comments of a node.
Compares two positions.
Shifts the line numbers of the node or metadata by the given line_correction
.
Returns the arguments of the node.
Returns the column of a node. If none is found, the default value is returned(defaults to 1).
Returns the line where the given node ends. It recursively checks for end
,
closing
and end_of_expression
line numbers. If none is found, the default
value is returned(defaults to 1).
Returns the end position of the quoted expression. It recursively checks for
end
, closing
and end_of_expression
positions. If none is found, the
default value is returned(defaults to [line: 1, column: 1]
).
Returns the line of a node. If none is found, the default value is returned(defaults to 1).
Returns the metadata of the given node.
Gets the range used by the given quoted expression in the source code.
Returns the start position of a node.
Parses a single expression from the given string. It tries to parse on a per-line basis.
Parses the source code into an extended AST suitable for source manipulation
as described in Code.quoted_to_algebra/2
.
Same as parse_string/1
but raises on error.
Performs a depth-first post-order traversal of a quoted expression.
Performs a depth-first post-order traversal of a quoted expression with an accumulator.
Prepends comments to the leading or trailing comments of a node.
Performs a depth-first pre-order traversal of a quoted expression.
Performs a depth-first pre-order traversal of a quoted expression with an accumulator.
A wrapper around Code.quoted_to_algebra/2
for compatibility with pre 1.13
Elixir versions.
A wrapper around Code.string_to_quoted_with_comments/2
for compatibility
with pre 1.13 Elixir versions.
A wrapper around Code.string_to_quoted_with_comments!/2
for compatibility
with pre 1.13 Elixir versions.
Converts a quoted expression to a string.
Updates the arguments for the given node.
Link to this section Types
Specs
Specs
position() :: keyword()
Specs
Specs
traversal_function() :: (Macro.t(), Sourceror.TraversalState.t() -> {Macro.t(), Sourceror.TraversalState.t()})
Link to this section Functions
Specs
append_comments( quoted :: Macro.t(), comments :: [comment()], position :: :leading | :trailing ) :: Macro.t()
Appends comments to the leading or trailing comments of a node.
Specs
Compares two positions.
Returns :gt
if the first position comes after the second one, and :lt
for
vice versa. If the two positions are equal, :eq
is returned.
nil
values for lines or columns are coalesced to 0
for integer
comparisons.
Specs
correct_lines(Macro.t() | Macro.metadata(), integer(), Macro.metadata()) :: Macro.t() | Macro.metadata()
Shifts the line numbers of the node or metadata by the given line_correction
.
This function will update the :line
, :closing
, :do
, :end
and
:end_of_expression
line numbers of the node metadata if such fields are
present.
Specs
Returns the arguments of the node.
iex> Sourceror.get_args({:foo, [], [{:__block__, [], [:ok]}]})
[{:__block__, [], [:ok]}]
Specs
Returns the column of a node. If none is found, the default value is returned(defaults to 1).
A default of nil
may also be provided if the column number is meant to be
coalesced with a value that is not known upfront.
iex> Sourceror.get_column({:foo, [column: 5], []})
5
iex> Sourceror.get_column({:foo, [], []}, 3)
3
Specs
Returns the line where the given node ends. It recursively checks for end
,
closing
and end_of_expression
line numbers. If none is found, the default
value is returned(defaults to 1).
iex> Sourceror.get_end_line({:foo, [end: [line: 4]], []})
4
iex> Sourceror.get_end_line({:foo, [closing: [line: 2]], []})
2
iex> Sourceror.get_end_line({:foo, [end_of_expression: [line: 5]], []})
5
iex> Sourceror.get_end_line({:foo, [closing: [line: 2], end: [line: 4]], []})
4
iex> """
...> alias Foo.{
...> Bar
...> }
...> """ |> Sourceror.parse_string!() |> Sourceror.get_end_line()
3
Specs
Returns the end position of the quoted expression. It recursively checks for
end
, closing
and end_of_expression
positions. If none is found, the
default value is returned(defaults to [line: 1, column: 1]
).
iex> quoted = ~S"""
...> A.{
...> B
...> }
...> """ |> Sourceror.parse_string!()
iex> Sourceror.get_end_position(quoted)
[line: 3, column: 1]
iex> quoted = ~S"""
...> foo do
...> :ok
...> end
...> """ |> Sourceror.parse_string!()
iex> Sourceror.get_end_position(quoted)
[line: 3, column: 1]
iex> quoted = ~S"""
...> foo(
...> :a,
...> :b
...> )
...> """ |> Sourceror.parse_string!()
iex> Sourceror.get_end_position(quoted)
[line: 4, column: 4]
Specs
Returns the line of a node. If none is found, the default value is returned(defaults to 1).
A default of nil
may also be provided if the line number is meant to be
coalesced with a value that is not known upfront.
iex> Sourceror.get_line({:foo, [line: 5], []})
5
iex> Sourceror.get_line({:foo, [], []}, 3)
3
Specs
get_meta(Macro.t()) :: Macro.metadata()
Returns the metadata of the given node.
iex> Sourceror.get_meta({:foo, [line: 5], []})
[line: 5]
Specs
Gets the range used by the given quoted expression in the source code.
The quoted expression must have at least line and column metadata, otherwise
it is not possible to calculate an accurate range, or to calculate it at all.
This function is most useful when used after Sourceror.parse_string/1
,
before any kind of modification to the AST.
The range is a map with :start
and :end
positions.
iex> quoted = ~S"""
...> def foo do
...> :ok
...> end
...> """ |> Sourceror.parse_string!()
iex> Sourceror.get_range(quoted)
%{start: [line: 1, column: 1], end: [line: 3, column: 4]}
iex> quoted = ~S"""
...> Foo.{
...> Bar
...> }
...> """ |> Sourceror.parse_string!()
iex> Sourceror.get_range(quoted)
%{start: [line: 1, column: 1], end: [line: 3, column: 2]}
Specs
Returns the start position of a node.
iex> quoted = Sourceror.parse_string!(" :foo")
iex> Sourceror.get_start_position(quoted)
[line: 1, column: 2]
iex> quoted = Sourceror.parse_string!("\n\nfoo()")
iex> Sourceror.get_start_position(quoted)
[line: 3, column: 1]
iex> quoted = Sourceror.parse_string!("Foo.{Bar}")
iex> Sourceror.get_start_position(quoted)
[line: 1, column: 1]
iex> quoted = Sourceror.parse_string!("foo[:bar]")
iex> Sourceror.get_start_position(quoted)
[line: 1, column: 1]
iex> quoted = Sourceror.parse_string!("foo(:bar)")
iex> Sourceror.get_start_position(quoted)
[line: 1, column: 1]
Specs
Parses a single expression from the given string. It tries to parse on a per-line basis.
Returns {:ok, quoted, rest}
on success or {:error, source}
on error.
Examples
iex> ~S"""
...> 42
...>
...> :ok
...> """ |> Sourceror.parse_expression()
{:ok, {:__block__, [trailing_comments: [], leading_comments: [],
token: "42", line: 2, column: 1], [42]}, "\n:ok"}
Options
:from_line
- The line at where the parsing should start. Defaults to1
.
Specs
Parses the source code into an extended AST suitable for source manipulation
as described in Code.quoted_to_algebra/2
.
Two additional fields are added to nodes metadata:
:leading_comments
- a list holding the comments found before the node.:trailing_comments
- a list holding the comments found before the end of the node. For example, comments right before theend
keyword.
Comments are the same maps returned by Code.string_to_quoted_with_comments/2
.
Specs
Same as parse_string/1
but raises on error.
Specs
postwalk(Macro.t(), traversal_function()) :: Macro.t()
Performs a depth-first post-order traversal of a quoted expression.
See postwalk/3
for more information.
Specs
postwalk(Macro.t(), term(), traversal_function()) :: {Macro.t(), term()}
Performs a depth-first post-order traversal of a quoted expression with an accumulator.
fun
is a function that will receive the current node as a first argument and
the traversal state as the second one. It must return a {quoted, state}
,
in the same way it would return {quoted, acc}
when using Macro.postwalk/3
.
The state is a map with the following keys:
:acc
- The accumulator. Defaults tonil
if none is given.
Specs
prepend_comments( quoted :: Macro.t(), comments :: [comment()], position :: :leading | :trailing ) :: Macro.t()
Prepends comments to the leading or trailing comments of a node.
Specs
prewalk(Macro.t(), traversal_function()) :: Macro.t()
Performs a depth-first pre-order traversal of a quoted expression.
See prewalk/3
for more information.
Specs
prewalk(Macro.t(), term(), traversal_function()) :: {Macro.t(), term()}
Performs a depth-first pre-order traversal of a quoted expression with an accumulator.
fun
is a function that will receive the current node as a first argument and
the traversal state as the second one. It must return a {quoted, state}
,
in the same way it would return {quoted, acc}
when using Macro.prewalk/3
.
The state is a map with the following keys:
:acc
- The accumulator. Defaults tonil
if none is given.
A wrapper around Code.quoted_to_algebra/2
for compatibility with pre 1.13
Elixir versions.
A wrapper around Code.string_to_quoted_with_comments/2
for compatibility
with pre 1.13 Elixir versions.
A wrapper around Code.string_to_quoted_with_comments!/2
for compatibility
with pre 1.13 Elixir versions.
Specs
Converts a quoted expression to a string.
The comments line number will be ignored and the line number of the associated node will be used when formatting the code.
Options
:line_length
- The max line length for the formatted code.:indent
- how many indentations to insert at the start of each line. Note that this only prepends the indents without checking the indentation of nested blocks. Defaults to0
.:indent_type
- the type of indentation to use. It can be one of:spaces
,:single_space
or:tabs
. Defaults to:spaces
.
Specs
Updates the arguments for the given node.
iex> node = {:foo, [line: 1], [{:__block__, [line: 1], [2]}]}
iex> updater = fn args -> Enum.map(args, &Sourceror.correct_lines(&1, 2)) end
iex> Sourceror.update_args(node, updater)
{:foo, [line: 1], [{:__block__, [line: 3], [2]}]}