View Source EctoBackfiller behaviour (EctoBackfiller v0.1.0-dev)
Orchestrator of a back-pressured backfill strategy for Ecto
repos.
Starts a producer process and dynamically start consumers, the amount of consumers is determined by the availability of resources on your infrastructure, such as available database connections or I/O usage.
Define a module to execute the backfill, which must use EctoBackfiller
and implement its callbacks.
Lets imagine a silly example to illustrate the use of the library. Suppose you have a User
schema described as:
defmodule MyApp.Users.User do
use Ecto.Schema
schema do
field :email_verified_at, :naive_datetime
end
end
And later on, your business requirements takes you to add email_verified
field as a boolean on
the schema representing if the user has verified the email. Then you write up the migration and
have to update the new column all existing users before execution of the migration.
To do so, you can write a module using EctoBackfiller
as:
defmodule MyApp.Backfills.UserEmailVerifiedBackfill do
use EctoBackfiller, repo: MyApp.Repo
alias MyApp.Users
alias MyApp.Users.User
@impl true
def query, do: Ecto.Queryable.to_query(User)
@impl true
def step, do: 5
@impl true
def handle_batch(users) do
Enum.each(users, fn user ->
if is_nil(user.email_verified_at) do
{:ok, user} = Users.update(user, %{email_verified: false})
else
{:ok, user} = Users.update(user, %{email_verified: true})
end
end)
end
end
Now you are ready to start executing it and to do so you must start the Supervisor, which will be named as the backfill module's name, or in other words, it is a unique proccess per backfill module.
Inside the application IEx session:
alias MyApp.Backfills.UserEmailVerifiedBackfill
UserEmailVerifiedBackfill.start_link()
:ok
UserEmailVerifiedBackfill.add_consumer()
:ok
UserEmailVerifiedBackfill.start()
:ok
You may add or more consumers on the fly, based on how the application performs based on the step used and the number of consumers subscribed.
Link to this section Summary
Callbacks
Handles the backfill logic given a list of data
Queryable used on Repo.all/2
to fetch chunks of data
Amount of data fetched per step
Link to this section Callbacks
@callback handle_batch([struct()]) :: :ok
Handles the backfill logic given a list of data
@callback query() :: Ecto.Query.t()
Queryable used on Repo.all/2
to fetch chunks of data
@callback step() :: pos_integer()
Amount of data fetched per step