View Source EctoBackfiller behaviour (EctoBackfiller v0.2.0)

Orchestrator of a back-pressured backfill strategy for Ecto repos.

Starts a producer process and dynamically start consumers, the amount of consumers is determined by the availability of resources on your infrastructure, such as available database connections or I/O usage.

Define a module to execute the backfill, which must use EctoBackfiller and implement its callbacks.

Lets imagine a silly example to illustrate the use of the library. Suppose you have a User schema described as:

defmodule MyApp.Users.User do
  use Ecto.Schema

  schema do
    field :email_verified_at, :naive_datetime
  end
end

And later on, your business requirements takes you to add email_verified field as a boolean on the schema representing if the user has verified the email. Then you write up the migration and have to update the new column all existing users before execution of the migration.

To do so, you can write a module using EctoBackfiller as:

defmodule MyApp.Backfills.UserEmailVerifiedBackfill do
  use EctoBackfiller, repo: MyApp.Repo

  alias MyApp.Users
  alias MyApp.Users.User

  @impl true
  def query, do: Ecto.Queryable.to_query(User)

  @impl true
  def step, do: 5

  @impl true
  def handle_batch(users) do
    Enum.each(users, fn user ->
      if is_nil(user.email_verified_at) do
        {:ok, user} = Users.update(user, %{email_verified: false})
      else
        {:ok, user} = Users.update(user, %{email_verified: true})
      end
    end)
  end
end

Now you are ready to start executing it and to do so you must start the Supervisor, which will be named as the backfill module's name, or in other words, it is a unique proccess per backfill module.

Inside the application IEx session:

alias MyApp.Backfills.UserEmailVerifiedBackfill

UserEmailVerifiedBackfill.start_link()
:ok

UserEmailVerifiedBackfill.add_consumer()
:ok

UserEmailVerifiedBackfill.start()
:ok

You may add or more consumers on the fly, based on how the application performs based on the step used and the number of consumers subscribed.

Link to this section Summary

Callbacks

Handles the backfill logic given a list of data

Queryable used on Repo.all/2 to fetch chunks of data

Amount of data fetched per step

Link to this section Callbacks

@callback handle_batch([struct()]) :: :ok

Handles the backfill logic given a list of data

@callback query() :: Ecto.Query.t()

Queryable used on Repo.all/2 to fetch chunks of data

@callback step() :: pos_integer()

Amount of data fetched per step