Blink.Seeder (blink v0.10.0)

Copy Markdown View Source

The central data structure and operations for the Blink seeding pipeline.

This module provides the Seeder struct and functions for building and inserting seed data into your database.

A Seeder holds:

  • :tables — data that will be inserted into the database.
  • :table_order - the insertion order for tables.
  • :table_opts — per-table options (:batch_size, :concurrency, :reset_sequences) used during the copy operation.
  • :context — auxiliary data used while building the seeder, not inserted.

Summary

Functions

Creates an empty Seeder.

Runs the seeder, inserting all table records into the given repository. Iterates over the tables in order when seeding the database.

Loads context into the seeder by calling the provided builder function.

Loads a table into the seeder by calling the provided builder function.

Types

empty()

@type empty() :: %Blink.Seeder{
  context: %{},
  table_opts: %{},
  table_order: [],
  tables: %{}
}

key()

@type key() :: binary() | atom()

t()

@type t() :: %Blink.Seeder{
  context: map(),
  table_opts: %{optional(key()) => table_opts()},
  table_order: [key()],
  tables: %{optional(key()) => Enumerable.t()}
}

table_opts()

@type table_opts() :: Keyword.t()

Functions

is_key(key)

(macro)

new()

@spec new() :: empty()

Creates an empty Seeder.

Example

iex> Blink.Seeder.new()
%Blink.Seeder{tables: %{}, table_order: [], table_opts: %{}, context: %{}}

run(seeder, repo, opts \\ [])

@spec run(seeder :: t(), repo :: Ecto.Repo.t(), opts :: Keyword.t()) :: :ok

Runs the seeder, inserting all table records into the given repository. Iterates over the tables in order when seeding the database.

The repo parameter must be a module that implements the Ecto.Repo behaviour and is configured with a Postgres adapter (e.g., Ecto.Adapters.Postgres).

Data stored in the Seeder's context is ignored.

Options

:adapter selects the adapter; everything else, including :atomic, is forwarded to the adapter, which owns and validates its option vocabulary — unknown keys and invalid values raise ArgumentError.

  • :atomic - Whether the seed is all-or-nothing (default: true). When true, every table is copied over a single database connection inside one transaction: if any table fails, all tables are rolled back. Set it to false to copy batches over parallel connections for maximum speed, accepting that a failure partway through can leave earlier batches and tables committed.
  • :timeout - The time in milliseconds allowed for each database operation (default: 15,000). Set to :infinity to disable it. See Blink.Adapter.Postgres for the exact semantics in each mode.
  • :truncate - Truncate every declared table before the first copy (default: false), making the seed replace the tables' contents — a re-runnable seed. See "Re-running seeds" below.
  • :adapter - The adapter module to use (default: Blink.Adapter.Postgres).

The following options are specific to Blink.Adapter.Postgres:

  • :batch_size - Number of rows per batch (default: 8,000). Can be overridden per-table via with_table/4.
  • :concurrency - Number of parallel workers: COPY connections when atomic: false (configure the repo's pool_size accordingly), row encoders feeding the single connection when atomic: true. See Blink.Adapter.Postgres for defaults. Can be overridden per-table via with_table/4.
  • :reset_sequences - Advance each table's serial or identity primary key sequence past the highest copied value after its copy (default: false). Primary keys without a sequence are unaffected. Can be overridden per-table via with_table/4. Not safe on tables receiving concurrent inserts; see Blink.Adapter.Postgres.

Atomicity

Seeds are all-or-nothing by default. A failed seed leaves nothing behind, so fixing the data and re-running is always safe. Pass atomic: false to trade that for speed: batches then commit independently over parallel connections, and a failure raises with earlier batches and tables still committed — the failure is never hidden, but you must inspect what was written and clean up before re-running. :atomic and :timeout apply to the whole run and cannot be overridden per-table.

The same distinction applies inside a transaction of your own: an atomic seed enrolls in it, while a non-atomic seed copies over separate connections that cannot see the transaction's uncommitted data and whose commits survive its rollback. Never pass atomic: false to a seed running inside your own transaction.

Re-running seeds

Pass truncate: true to make the seed replace its tables' contents: one TRUNCATE ... RESTART IDENTITY statement over every declared table runs before the first copy, so re-running the seed produces the same end state as running it once against an empty database. The single statement covers all declared tables at once, so foreign keys between them need no special ordering. A foreign key from a table the seeder does not declare makes the truncate fail — declare that table too; Blink never uses CASCADE, which would silently empty tables the seeder never named.

In an atomic seed the truncate joins the transaction: a failed re-seed rolls back to the data you had before it started. With atomic: false the truncate commits on its own before the first batch, so a failure leaves the tables truncated and partially seeded. That partial state needs no hand cleanup, though: the next run's truncate is the cleanup, so fixing the data and re-running still converges — truncate: true gives atomic: false seeds the same fix-and-re-run remedy that atomic seeds get from rollback.

RESTART IDENTITY restarts the sequences behind the truncated tables, so database-assigned ids come out identical on every run; seeds using explicit ids pair truncate: true with reset_sequences: true for the same determinism. The truncate takes an ACCESS EXCLUSIVE lock on every declared table and destroys their contents — like reset_sequences, it is meant for databases the seed owns, never live tables. :truncate is a run-level option and cannot be set per-table.

Telemetry

run/3 is wrapped in a [:blink, :run] telemetry span covering the copy phase; table builders run earlier, at declaration time, under [:blink, :build] spans. See Blink.Telemetry for the event reference and a default logger.

Returns

  • :ok - When all tables have been seeded successfully

Raises an exception when the seeding operation fails.

Examples

# All-or-nothing seeding
run(seeder, MyApp.Repo, atomic: true)

# With custom timeout
run(seeder, MyApp.Repo, timeout: 60_000)

# With custom batch size and concurrency
run(seeder, MyApp.Repo, batch_size: 5_000, concurrency: 4)

with_context(seeder, key, builder)

@spec with_context(
  seeder :: t(),
  key :: key(),
  builder :: (seeder :: t(), key :: key() -> any())
) :: t()

Loads context into the seeder by calling the provided builder function.

This is the low-level form, which takes an explicit builder. If your module calls use Blink, call with_context/2 on that module instead — it dispatches to your context/2 clause for you and reports a missing clause as a Blink.MissingClauseError. See Blink for the full seeder API.

The builder function should take a seeder and key and return the context data.

with_table(seeder, table_name, builder, opts \\ [])

@spec with_table(
  seeder :: t(),
  table_name :: key(),
  builder :: (seeder :: t(), table_name :: key() -> Enumerable.t()),
  opts :: table_opts()
) :: t()

Loads a table into the seeder by calling the provided builder function.

This is the low-level form, which takes an explicit builder. If your module calls use Blink, call with_table/2 or with_table/3 on that module instead — it dispatches to your table/2 clause for you and reports a missing clause as a Blink.MissingClauseError. See Blink for the full seeder API.

The builder function should take a seeder and table name and return an enumerable (list or stream) of maps representing the table data.

Options

Options given here override the ones passed to run/3 for this table only. They are forwarded to the adapter, which owns and validates them — unknown keys and invalid values raise ArgumentError when the seeder runs. The run-level options :adapter, :atomic, :timeout, and :truncate configure the whole run and raise ArgumentError here.

For Blink.Adapter.Postgres the per-table options are:

  • :batch_size - Number of rows per batch. Overrides :batch_size in run/3.
  • :concurrency - Number of parallel workers. Overrides :concurrency in run/3.
  • :reset_sequences - Advance this table's primary key sequence past the highest copied value after its copy. Overrides :reset_sequences in run/3.

Examples

# Without options (uses options from `run/3`)
Seeder.with_table(seeder, "users", &table/2)

# With custom batch size and concurrency
Seeder.with_table(seeder, "users", &table/2, batch_size: 1_000, concurrency: 2)