View Source Traveller (traveller v0.1.3)

Provides a simple way to fetch records from a database table.

Supports cursor and offset based pagination. Allows for multiple cursor fields and multi-directional ordering.

Simplest use-case (assuming a User schema):

Repo
|> Traveller.start_stream(User)
|> Enum.each(fn batch ->
  # do something
end)

Specify a cursor...

Repo
|> Traveller.start_stream(User, cursor: :first_name)
|> Enum.each(fn batch ->
  # do something
end)

...or many

Repo
|> Traveller.start_stream(User, cursor: [desc: :first_name, desc: :last_name, asc: :id])
|> Enum.each(fn batch ->
  # do something
end)

Offset if you prefer:

Repo
|> Traveller.start_stream(User, mode: :offset)
|> Enum.each(fn batch ->
  # do something
end)

Start late and finish early:

Repo
|> Traveller.start_stream(User, start_after: "Albus", stop_before: "Severus", cursor: :first_name)
|> Enum.each(fn batch ->
  # do something
end)

If you want to decorate a specific module:

defmodule YourModule do
  use Traveller,
    repo: Repo,
    schema: Person,
    start_after: "Albus",
    stop_before: "Severus",
    cursor: :first_name
end

Enum.each(YourModule.start_stream(), fn batch ->
  # do something
end)

Link to this section Summary

Functions

Initiates a stream that walks through a database table.

Link to this section Functions

Link to this function

start_stream(repo, schema, opts \\ [])

View Source

Initiates a stream that walks through a database table.

Expects a repo, a schema and 0 or more options.

Assumes the repo passed has already been started.

Options:

  • cursor: only applies when mode = :cursor (default). Defaults to primary key or :id. Can be an atom corresponding to a field in the schema passed; a tuple with an ordering (either :asc or :desc) and a field; or a list of fields or {order, field} tuples. If no order is specified :asc is assumed.

  • chunk_size: determines how many records are returned in each batch.

  • start_after: a value which is used for the initial cursor. Only works when mode is cursor. If none is provided then the highest or lowest value in the table is used, based on whether the ordering is descending or ascending respectively. If a list of cursor fields is passed, then the default applies to the first field only.

  • stop_before: a value which is used to terminate the record fetch early. Currently only supports a single value, and only works when mode is cursor.

  • mode: provide :offset if you want an offset based record fetch.

  • order_by: only applies when mode is :offset. Determines the order that records are returned in. Can be a field or a {direction, field} tuple. If no sort direction is specified, :asc is assumed.

  • offset: only applies when mode is :offset. This is used for the initial offset. The default is 0.

  • next_cursor: default is the values corresponding to the cursor fields from the last record the last record in the result set. You can provide a function which is passed the set of results which can return a value or a list of values used for the next cursor.