Blink.Seeder (blink v0.8.0)

Copy Markdown View Source

The central data structure and operations for the Blink seeding pipeline.

This module provides the Seeder struct and functions for building and inserting seed data into your database.

A Seeder holds:

  • :tables — data that will be inserted into the database.
  • :table_order - the insertion order for tables.
  • :table_opts — per-table options (:batch_size, :concurrency) used during the copy operation.
  • :context — auxiliary data used while building the seeder, not inserted.

Summary

Functions

Creates an empty Seeder.

Runs the seeder, inserting all table records into the given repository. Iterates over the tables in order when seeding the database.

Loads context into the seeder by calling the provided builder function.

Loads a table into the seeder by calling the provided builder function.

Types

empty()

@type empty() :: %Blink.Seeder{
  context: %{},
  table_opts: %{},
  table_order: [],
  tables: %{}
}

key()

@type key() :: binary() | atom()

t()

@type t() :: %Blink.Seeder{
  context: map(),
  table_opts: %{optional(key()) => table_opts()},
  table_order: [key()],
  tables: %{optional(key()) => Enumerable.t()}
}

table_opts()

@type table_opts() :: Keyword.t()

Functions

is_key(key)

(macro)

new()

@spec new() :: empty()

Creates an empty Seeder.

Example

iex> Blink.Seeder.new()
%Blink.Seeder{tables: %{}, table_order: [], table_opts: %{}, context: %{}}

run(seeder, repo, opts \\ [])

@spec run(seeder :: t(), repo :: Ecto.Repo.t(), opts :: Keyword.t()) :: :ok

Runs the seeder, inserting all table records into the given repository. Iterates over the tables in order when seeding the database.

The repo parameter must be a module that implements the Ecto.Repo behaviour and is configured with a Postgres adapter (e.g., Ecto.Adapters.Postgres).

Data stored in the Seeder's context is ignored.

Options

:adapter selects the adapter; everything else, including :atomic, is forwarded to the adapter, which owns and validates its option vocabulary — unknown keys and invalid values raise ArgumentError.

  • :atomic - Whether the seed is all-or-nothing (default: true). When true, every table is copied over a single database connection inside one transaction: if any table fails, all tables are rolled back. Set it to false to copy batches over parallel connections for maximum speed, accepting that a failure partway through can leave earlier batches and tables committed.
  • :timeout - The time in milliseconds allowed for each database operation (default: 15,000). Set to :infinity to disable it. See Blink.Adapter.Postgres for the exact semantics in each mode.
  • :adapter - The adapter module to use (default: Blink.Adapter.Postgres).

The following options are specific to Blink.Adapter.Postgres:

  • :batch_size - Number of rows per batch (default: 8,000). Can be overridden per-table via with_table/4.
  • :concurrency - Number of parallel workers: COPY connections when atomic: false (configure the repo's pool_size accordingly), row encoders feeding the single connection when atomic: true. See Blink.Adapter.Postgres for defaults. Can be overridden per-table via with_table/4.

Atomicity

Seeds are all-or-nothing by default. A failed seed leaves nothing behind, so fixing the data and re-running is always safe. Pass atomic: false to trade that for speed: batches then commit independently over parallel connections, and a failure raises with earlier batches and tables still committed — the failure is never hidden, but you must inspect what was written and clean up before re-running. :atomic and :timeout apply to the whole run and cannot be overridden per-table.

The same distinction applies inside a transaction of your own: an atomic seed enrolls in it, while a non-atomic seed copies over separate connections that cannot see the transaction's uncommitted data and whose commits survive its rollback. Never pass atomic: false to a seed running inside your own transaction.

Returns

  • :ok - When all tables have been seeded successfully

Raises an exception when the seeding operation fails.

Examples

# All-or-nothing seeding
run(seeder, MyApp.Repo, atomic: true)

# With custom timeout
run(seeder, MyApp.Repo, timeout: 60_000)

# With custom batch size and concurrency
run(seeder, MyApp.Repo, batch_size: 5_000, concurrency: 4)

with_context(seeder, key, builder)

@spec with_context(
  seeder :: t(),
  key :: key(),
  builder :: (seeder :: t(), key :: key() -> any())
) :: t()

Loads context into the seeder by calling the provided builder function.

This is the low-level form, which takes an explicit builder. If your module calls use Blink, call with_context/2 on that module instead — it dispatches to your context/2 clause for you and reports a missing clause as a Blink.MissingClauseError. See Blink for the full seeder API.

The builder function should take a seeder and key and return the context data.

with_table(seeder, table_name, builder, opts \\ [])

@spec with_table(
  seeder :: t(),
  table_name :: key(),
  builder :: (seeder :: t(), table_name :: key() -> Enumerable.t()),
  opts :: table_opts()
) :: t()

Loads a table into the seeder by calling the provided builder function.

This is the low-level form, which takes an explicit builder. If your module calls use Blink, call with_table/2 or with_table/3 on that module instead — it dispatches to your table/2 clause for you and reports a missing clause as a Blink.MissingClauseError. See Blink for the full seeder API.

The builder function should take a seeder and table name and return an enumerable (list or stream) of maps representing the table data.

Options

Options given here override the ones passed to run/3 for this table only. They are forwarded to the adapter, which owns and validates them — unknown keys and invalid values raise ArgumentError when the seeder runs. The run-level options :adapter, :atomic, and :timeout configure the whole run and raise ArgumentError here.

For Blink.Adapter.Postgres the per-table options are:

  • :batch_size - Number of rows per batch. Overrides :batch_size in run/3.
  • :concurrency - Number of parallel workers. Overrides :concurrency in run/3.

Examples

# Without options (uses options from `run/3`)
Seeder.with_table(seeder, "users", &table/2)

# With custom batch size and concurrency
Seeder.with_table(seeder, "users", &table/2, batch_size: 1_000, concurrency: 2)