Drive Glean's paged bulk indexing protocol.
Every /bulkindex* endpoint works the same way. You upload records in pages,
and each request carries:
uploadId- the same value on every page of one uploadisFirstPage-trueon the first page, which starts a fresh uploadisLastPage-trueon the last page, which commits it
Committing matters: a bulk upload replaces everything uploaded in the previous batch, and Glean only swaps the batch in once it has seen the last page. An upload that stops halfway leaves the previous batch in place.
Examples
Gleanex.Bulk.upload(
config,
&Gleanex.Indexing.Documents.bulkindexdocuments/2,
%{datasource: "mydatasource"},
:documents,
documents
)
Gleanex.Bulk.upload(
config,
&Gleanex.Indexing.Permissions.bulkindexusers/2,
%{datasource: "mydatasource"},
:users,
users,
page_size: 500
)Summary
Types
A generated bulk indexing operation.
Functions
Upload an enumerable in pages.
Types
@type operation() :: (map(), keyword() -> {:ok, term()} | {:error, Gleanex.Error.t()})
A generated bulk indexing operation.
Functions
@spec upload( Gleanex.Config.t(), operation(), map(), atom(), Enumerable.t(), keyword() ) :: {:ok, non_neg_integer()} | {:error, Gleanex.Error.t()}
Upload an enumerable in pages.
Returns {:ok, page_count}, or {:error, %Gleanex.Error{}} for the first page
that fails. Later pages are not attempted after a failure, which leaves the
upload uncommitted and the previous batch intact.
body supplies the fields shared by every page, most importantly
:datasource. key is the field holding the records, for example
:documents or :users.
Options
:page_size- records per request, default100.:upload_id- reuse an ID to resume an interrupted upload. Defaults to a fresh random ID.- anything else is passed to the operation.
An empty enumerable still sends one page, marked as both first and last, which is how an existing batch is emptied.