1
2
3
defmodule Meeseeks do alias Meeseeks.{Context, Document, Parser, Result, Select, Selector, TupleTree} @moduledoc """ Meeseeks is an Elixir library for extracting data from HTML. ```elixir import Meeseeks.CSS html = Tesla.get("https://news.ycombinator.com/").body for story <- Meeseeks.all(html, css("tr.athing")) do title = Meeseeks.one(story, css(".title a")) %{title: Meeseeks.text(title), url: Meeseeks.attr(title, "href")} end #=> [%{title: "...", url: "..."}, %{title: "...", url: "..."}, ...] ``` ## Dependencies Meeseeks depends on [html5ever](https://github.com/servo/html5ever) via [meeseeks_html5ever](https://github.com/mischov/meeseeks_html5ever). Because html5ever is a Rust library, you will need to have the Rust compiler [installed](https://www.rust-lang.org/en-US/install.html). This dependency is necessary because there are no HTML5 spec compliant parsers written in Elixir/Erlang. ## Getting Started ### Parse Start by parsing a source (HTML string or `Meeseeks.TupleTree`) into a `Meeseeks.Document` so that it can be queried. ```elixir document = Meeseeks.parse("
1
2
3
1
}> import Meeseeks.XPath result = Meeseeks.one(document, xpath("//*[@id='main']//p")) #=> #Meeseeks.Result<{1
}> ``` ### Extract Retrieve information from the result with an extraction function. The `Meeseeks.Result` extraction functions are `attr`, `attrs`, `data`, `dataset`, `html`, `own_text`, `tag`, `text`, `tree`. ```elixir Meeseeks.tag(result) #=> "p" Meeseeks.text(result) #=> "1" Meeseeks.tree(result) #=> {"p", [], ["1"]} ``` ## Custom Selectors Meeseeks is designed to have extremely extensible selectors, and creating a custom selector is as easy as defining a struct that implements the `Meeseeks.Selector` behaviour. ```elixir defmodule CommentContainsSelector do use Meeseeks.Selector alias Meeseeks.Document defstruct value: "" def match(selector, %Document.Comment{} = node, _document, _context) do String.contains?(node.content, selector.value) end def match(_selector, _node, _document, _context) do false end end selector = %CommentContainsSelector{value: "TODO"} Meeseeks.one("", selector) #=> #Meeseeks.Result<{ }> ``` To learn more, check the documentation for `Meeseeks.Selector` and `Meeseeks.Selector.Combinator` """ @type source :: String.t | TupleTree.t @type queryable :: source | Document.t | Result.t @type selectors :: Selector.t | [Selector.t] # Parse @doc """ Parses an HTML string or `Meeseeks.TupleTree` into a `Meeseeks.Document`. ## Examples iex> Meeseeks.parse("Hello, Meeseeks!
1
2
3
1
}> """ @spec all(queryable, selectors) :: [Result.t] def all(queryable, selectors) do all(queryable, selectors, %{}) end @spec all(queryable, selectors, Context.t) :: [Result.t] def all(%Document{} = queryable, selectors, context) do Select.all(queryable, selectors, context) end def all(%Result{} = queryable, selectors, context) do Select.all(queryable, selectors, context) end def all(source, selectors, context) do source |> parse() |> Select.all(selectors, context) end @doc """ Returns a `Result` for the first node in the queryable (depth-first) matching a selector. Optionally accepts a `Meeseeks.Context` map. Parses the source if it is not a `Meeseeks.Document` or `Meeseeks.Result`. If multiple selections are being ran on the same unparsed source, parse first to avoid unnecessary computation. ## Examples iex> import Meeseeks.CSS iex> Meeseeks.one("1
2
3
1
}> """ @spec one(queryable, selectors) :: Result.t def one(queryable, selectors) do one(queryable, selectors, %{}) end @spec one(queryable, selectors, Context.t) :: Result.t def one(%Document{} = queryable, selectors, context) do Select.one(queryable, selectors, context) end def one(%Result{} = queryable, selectors, context) do Select.one(queryable, selectors, context) end def one(source, selectors, context) do source |> parse() |> Select.one(selectors, context) end @doc """ Returns the accumulated result of walking the queryable, accumulating nodes that match a selector. Prefer `all` or `one`- `select` should only be used when a custom accumulator is required. Requires that a `Meeseeks.Accumulator` has been added to the context via `Meeseeks.Context.add_accumulator/2`, and will raise an error if it hasn't. Parses the source if it is not a `Meeseeks.Document` or `Meeseeks.Result`. If multiple selections are being ran on the same unparsed source, parse first to avoid unnecessary computation. ## Examples iex> import Meeseeks.CSS iex> accumulator = %Meeseeks.Accumulator.One{} iex> context = Meeseeks.Context.add_accumulator(%{}, accumulator) iex> Meeseeks.select("1
2
3
1
}> """ @spec select(queryable, selectors, Context.t) :: any def select(%Document{} = queryable, selectors, context) do Select.select(queryable, selectors, context) end def select(%Result{} = queryable, selectors, context) do Select.select(queryable, selectors, context) end def select(source, selectors, context) do source |> parse() |> Select.select(selectors, context) end # Extract @doc """ Returns the value for attribute in result, or nil if there isn't one. ## Examples iex> import Meeseeks.CSS iex> result = Meeseeks.one("