View Source OffBroadway.Elasticsearch.SearchAfterStrategy (OffBroadwayElasticsearch v0.1.0)

Implementation of OffBroadway.Elasticsearch.Strategy that's able to fetch more than 10,000 results per request by using Elasticsearch's search_after feature. This strategy is best suited for a single producer and the search query must be sorted in some way. For concurrent searching, see OffBroadway.Elasticsearch.SliceStrategy

Example

defmodule MyBroadway do
  use Broadway

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      ...
      producer: [
        module: {
          OffBroadway.Elasticsearch.Producer,
          [
            host: "http://localhost:9200",
            index: "my-index",
            strategy: :search_after, # <- Select the 'search_after' strategy
            search: search() # <- Provide a search query
          ]
        },
        concurrency: 1 # <- Best
      ],
      ...
    )
  end

  def search do
    %{
      query: %{
        match_all: %{}
      },
      sort: %{ # <- Must be present so Elasticsearch returns the `search_after` keys/values
        created_at: "asc",
        _id: "asc"
      }
    }
  end
end

See Elasticsearch docs: Search After

Summary

Types