Walk Glean's cursor-paginated endpoints.
Glean paginates the same way across /search, /listentities, /listchats
and /getdocumentsbyfacets: the response carries a cursor and, on some
endpoints, hasMoreResults. Sending that cursor back in the next request body
fetches the following page.
The generated operations do one request each, so this module drives the loop.
Examples
config = Gleanex.new(domain: "mycompany", token: token)
config
|> Gleanex.Pagination.stream(&Gleanex.Client.Search.search/2, %{query: "holidays"})
|> Stream.flat_map(& &1.results)
|> Enum.take(100)Pages come back as whole responses, so metadata such as trackingToken is not
thrown away. Use stream_items/5 when only the records matter:
Gleanex.Pagination.stream_items(
config,
&Gleanex.Client.Entities.listentities/2,
%{entityType: "PEOPLE"},
:results
)Errors
A failed page raises the Gleanex.Error it produced. Lazy enumerables have
nowhere to put an error tuple without making every element a tuple, and
silently ending the stream would look exactly like reaching the last page.
Summary
Types
A generated operation function taking a request body and options.
Types
@type operation() :: (map(), keyword() -> {:ok, term()} | {:error, Gleanex.Error.t()})
A generated operation function taking a request body and options.
Functions
@spec stream(Gleanex.Config.t(), operation(), map(), keyword()) :: Enumerable.t()
Stream every page of a cursor-paginated operation.
Stops when the response has no cursor, when hasMoreResults is false, or
when a page repeats the previous cursor, which would otherwise loop forever.
Options
Passed through to the operation, with :config added.
@spec stream_items(Gleanex.Config.t(), operation(), map(), atom(), keyword()) :: Enumerable.t()
Stream the records inside every page.
key is the field holding the records, for example :results for search or
:entities for /listentities. Pages without that field contribute nothing.