Blink.Adapter.Postgres (blink v0.8.0)

Copy Markdown View Source

PostgreSQL adapter for Blink bulk copy operations.

This adapter uses PostgreSQL's COPY FROM STDIN command for efficient bulk insertion of data. It is the default adapter used by Blink.

Usage

This adapter is used automatically by default:

Blink.copy_to_table(rows, "users", MyApp.Repo)

Or explicitly:

Blink.copy_to_table(rows, "users", MyApp.Repo, adapter: Blink.Adapter.Postgres)

Summary

Functions

Copies rows into a database table using PostgreSQL's COPY command.

Functions

call(rows, table_name, repo, opts \\ [])

@spec call(
  rows :: Enumerable.t(),
  table_name :: String.t(),
  repo :: Ecto.Repo.t(),
  opts :: Keyword.t()
) :: :ok

Copies rows into a database table using PostgreSQL's COPY command.

Parameters

  • rows - An enumerable (list or stream) of maps where each map represents a row to insert. All maps must have the same keys, which correspond to the table columns. Using a stream allows for memory-efficient seeding of large datasets. The input is consumed exactly once, so single-use streams are safe.
  • table_name - The name of the table to insert into (string).
  • repo - An Ecto repository module configured with a Postgres adapter.
  • opts - Keyword list of options. Unknown keys and invalid values raise ArgumentError:
    • :atomic - Whether the copy is all-or-nothing (default: true).
      • false - Workers copy batches over up to :concurrency database connections in parallel and each batch commits independently. Fastest, but a failure can leave earlier batches committed. The worker connections do not enroll in a transaction of the caller's own: they cannot see its uncommitted data, and their commits survive its rollback.
      • true - All batches are copied over one connection inside a single transaction while :concurrency workers encode rows in parallel. Any failure rolls back the whole COPY, and the copy enrolls in a surrounding transaction such as the one Blink.Seeder.run/3 opens for atomic seeds. Rows are copied in input order.
    • :concurrency - Number of parallel workers (default: 6 when atomic: false, System.schedulers_online/0 when atomic: true). With atomic: false each worker encodes and copies batches over its own database connection, so configure the repo's pool_size to at least :concurrency. With atomic: true workers only encode; a single connection performs the COPY.
    • :batch_size - Number of rows per batch (default: 8,000). Items are chunked into batches, each written via a separate COPY operation (or a separate write to the single COPY when atomic: true). To disable batching, set this to a value equal to or greater than the total number of rows.
    • :timeout - Time in milliseconds allowed for each database operation (default: 15,000). With atomic: false this bounds each batch's COPY transaction. With atomic: true it is enforced server-side as a statement_timeout on each COPY statement, because a connection checkout deadline cannot bound individual operations inside one transaction. Set to :infinity to disable Blink's timeout (a server-configured statement_timeout still applies).

Returns

  • :ok - When the copy operation succeeds

Raises an exception when the copy operation fails.

Examples

iex> rows = [%{id: 1, name: "Alice"}, %{id: 2, name: "Bob"}]
iex> Blink.Adapter.Postgres.call(rows, "users", MyApp.Repo)
:ok

# Atomic, all-or-nothing copy
iex> Blink.Adapter.Postgres.call(rows, "users", MyApp.Repo, atomic: true)
:ok

# Using a stream for memory-efficient seeding
iex> stream = Stream.map(1..1_000_000, fn i -> %{id: i, name: "User #{i}"} end)
iex> Blink.Adapter.Postgres.call(stream, "users", MyApp.Repo)
:ok

Notes

The function assumes all rows have the same keys. NULL values are represented as \N in the CSV format. Nested maps are automatically JSON-encoded for JSONB columns; values that are already JSON strings are inserted as-is, so passing pre-encoded JSON avoids a redundant Jason.encode!/1 call. Elixir lists are encoded as PostgreSQL array literals for array columns (int[], text[], jsonb[], nested arrays, ...). A JSONB column holding a top-level JSON array should be passed as a pre-encoded JSON string.

Structs are maps, so a struct value (a DateTime, Date, Decimal, ...) is also JSON-encoded. PostgreSQL's date/time parsers accept the quoted result, so calendar structs work in timestamp, date, and time columns; in a text column the stored value keeps the JSON quotes — pass to_string(value) instead.