Snap.Bulk (Snap v0.1.1) View Source

Supports streaming bulk operations against a Snap.Cluster.

Link to this section Summary

Link to this section Functions

Link to this function

perform(stream, cluster, index, opts)

View Source

Specs

perform(Enumerable.t(), module(), String.t(), Keyword.t()) ::
  :ok | {:error, [Snap.Exception.t()]}

Performs a bulk operation.

Takes an Enumerable of action structs, where each struct is one of:

actions = [
  %Snap.Bulk.Action.Create{_id: 1, doc: %{foo: "bar"}},
  %Snap.Bulk.Action.Create{_id: 2, doc: %{foo: "bar"}},
  %Snap.Bulk.Action.Create{_id: 3, doc: %{foo: "bar"}}
]

actions
|> Snap.Bulk.perform(Cluster, "index")

It chunks the Enumerable into pages, and pauses between pages for Elasticsearch to catch up. Uses Stream under the hood, so you can lazily feed it a stream of actions, such as out of an Ecto.Repo to bulk load documents from an SQL database.

If no errors occur on any page it returns :ok. If any errors occur, on any page, it returns {:error, [%Snap.Exception{}...]}. It will continue to the end, even if errors occur.

Options:

  • page_size - defines the size of each page, defaulting to 5000 actions.
  • page_wait - defines wait period between pages in ms, defaulting to 15000ms.