Bulkinup

View Source

Note

This library is pre-1.0: the API may change between minor versions (see the changelog). The test suite exercises it against a real Postgres database, but review the documented limitations before using it in production.

Bulk inserts and upserts for nested Ecto schemas.

Unlike a plain insert_all/3, this package passes each attrs map through Ecto changesets. This lets it validate your data and write a parent and its children across multiple tables in one call — as a pure insert (Bulkinup.insert/4) or an upsert (Bulkinup.upsert/4).

Supported features:

  • Nested associations: write a parent and its has_many, has_one, and many_to_many associations across multiple tables from a single list of attrs, recursively at any depth (embedded schemas are stored inline on the parent)
  • Validation and data processing (via Ecto changesets)
  • Repo-scoped bulk_insert/3 and bulk_upsert/3 with app-wide defaults, via use Bulkinup
  • Custom values for autogenerated fields (e.g. insert/update timestamps)
  • Recovery of invalid field values via per-schema fallbacks
  • A single transaction wraps the entire call (by default), so any failure rolls back all changes
  • Streaming input: pass any Enumerable (including a lazy Stream) to write arbitrarily large inputs with bounded memory
  • Optional concurrent writes via :max_concurrency, trading the single transaction for throughput

Installation

Add this package to your list of dependencies in mix.exs, then run mix deps.get:

def deps do
  [
    {:bulkinup, "~> 0.6.0"}
  ]
end

Basic example

Given a Person schema with a 1-arity-callable changeset function:

iex> Bulkinup.insert(
...>   YourProject.Repo,
...>   YourProject.Persons.Person,
...>   [%{id: 1, name: "Alice"}, %{id: 2, name: "Bob"}]
...> )
{:ok, %{inserted: 2, skipped: 0}}

iex> Bulkinup.upsert(
...>   YourProject.Repo,
...>   YourProject.Persons.Person,
...>   [%{id: 1, name: "Alicia"}, %{id: 2, name: "Bobby"}]
...> )
{:ok, %{upserted: 2, skipped: 0}}

Rows whose changesets are invalid are skipped rather than written, visibly: the counts in the return value show it, and each call that skips rows emits one :warning log summarizing them.

Learn more

This project made possible by Interline Travel and Tour Inc.

https://www.perx.com/

https://www.touchdown.co.uk/

https://www.touchdownfrance.com/