View Source Cursornator

Notes: This repo is my customized version of Ecto Cursor based stream for using in my projects.

Cursor-based streaming of Ecto records, that does not require database transaction.

Advantages in comparison to the standard Ecto.Repo.stream/2:

  • streaming can be stopped and continued at any point (by passing option after_cursor: ...),
  • works with tables that have milions of records.

Only limitation is that you have to supply a cursor column or columns (by passing option cursor_field: ..., defaults to :id). Such a column(s):

  • must have unique values,
  • should have a database index. (So that sorting by it, and returning a number of rows larger than x is a performant operation.)

Usage

  1. Add cursornator to your list of dependencies in mix.exs:
def deps do
  [
    {:cursornator, "~> 1.3.0"}
  ]
end
  1. Add use Cursornator to the module that uses Ecto.Repo:
defmodule MyRepo do
  use Ecto.Repo
  use Cursornator
end
  1. Stream the rows using cursor_stream/2:
Post
|> MyRepo.cursor_stream(max_rows: 100)
|> Stream.each(...)
|> Stream.run()
  1. Query if you don't want to stream
{posts, next_cursor} = MyRepo.cursor_query(Post, cursor_field: [published_at: :desc, id: :desc])