Pure-ish CSV/plaintext import engine for CRM contact lists (Stage 3 of the
restructuring plan — the user's stated priority: account list import).
Parses rows, applies the restructuring spec's Locked decisions, and writes
only through PhoenixKitCRM.Lists — it never duplicates that context's
insert/counter/broadcast logic.
Locked decisions this engine enforces
- Import always creates a NEW contact — never a merge, never "add to existing contact by matching email". Matching duplicates across contacts is a separate, later, human-reviewed comparison screen (Stage C4c), not something this engine decides on its own.
- Per-row atomicity: contact INSERT + membership INSERT happen in ONE
transaction (
Lists.add_new_contact_to_list/3) — a uniqueness violation rolls both back, so a failed/duplicate row never leaves an orphan contact behind. - Idempotent re-import: importing the identical file twice creates zero
new contacts on the second pass — every row lands in
skippedinstead ofcreated/added(:already_in_listif the existing membership is still active,:unsubscribedif a prior membership was removed but still holds the email slot per the DB's partial unique index — seeListMember's moduledoc). A KNOWN collision is classified straight from a batchedLists.members_by_email/3lookup (one per chunk — seerun_chunk/4) and never reaches the write transaction at all, so re-importing a large duplicate file doesn't pay an INSERT-then-rollback per colliding row; only an email the batched lookup didn't already know about goes through it, where a race against a concurrent import is still caught by the DB's own unique index.
Row pipeline
Each row goes through, in order: normalize (trim + downcase, matching the
email column's citext case-insensitivity) → in-file dedup prefilter
(:duplicate_in_file — a second row with the same email, e.g. a shared
mailbox listed under two names, never reaches the DB) → email format
check (:invalid_email) → the batched members_by_email lookup (a KNOWN
collision short-circuits straight to :already_in_list/:unsubscribed,
no write attempted) → the write transaction for anything left, which
still classifies an idx_crm_list_members_list_email violation (a race
the batched lookup couldn't have seen) by looking up the existing
holder's status.
name is CSV-optional per the spec; a blank name falls back to the email
itself (mirrors Contact.display_name/1's own fallback) so the contact
changeset's validate_required([:name]) never rejects an otherwise-valid
row. locale, if present but not in a recognized format, is silently
dropped rather than rejecting the row over a cosmetic field. company
has no dedicated contact field yet (linking to PhoenixKitCRM.Companies
is out of scope here) — it's stashed in contact.metadata["import_company"]
so the comparison screen (C4c) can still surface it later instead of the
data being silently discarded.
XLSX
Deliberately not supported yet. xlsxir (the library the plan asked to
evaluate) is unmaintained since 2019 — not something to add as a new
dependency. xlsx_reader is an actively-maintained alternative but wasn't
evaluated in depth (its own dependency footprint, streaming behavior, etc.
is a separate task). CSV/TXT cover the stated priority (account import);
XLSX is deferred to a follow-up rather than shipped on an abandoned lib.
Summary
Functions
Puts a report's rows back into file order. Reports threaded through
run_chunk/4 accumulate rows prepended (newest first) — appending per row
would be O(n²) on a file near the upload size limit — so a caller that
renders row detail must finalize once when the run completes. run/3 and
preview_rows/2 already return finalized reports.
Imports from CSV content (already read from an upload or textarea paste).
Header row is required; column matching is case-insensitive and only
looks for email (required), name, company, locale (all optional)
— unrecognized columns are ignored. opts accepts :actor_uuid (passed
through to the activity log on every successful row).
Imports from plaintext/clipboard content — one email address per line.
Blank lines are ignored. opts accepts :actor_uuid.
A fresh {report, seen_emails} accumulator to start chunked processing with run_chunk/4.
Parses CSV content into {line, attrs} rows WITHOUT writing anything —
the same parser import_csv/3 uses internally, exposed for the import
UI's dry-run preview and chunked processing (run_chunk/4) so neither
has to duplicate the parsing. Non-UTF-8 content and malformed CSV (e.g.
an unterminated quote, which NimbleCSV raises on) return [] rather
than crashing the caller.
Same as parse_csv_rows/1, for plaintext/clipboard content.
Classifies every row exactly like import_csv/3/import_text/3 would,
but performs NO writes — :no_email/:invalid_email/:duplicate_in_file
are determined purely from the parsed rows, and an email that would
collide on idx_crm_list_members_list_email is detected via ONE batched
Lists.members_by_email/2 lookup (all distinct emails in the file, one
query) instead of attempting (and rolling back) an insert per row. A
naive per-row Lists.get_member_by_email/2 call here would mean tens of
thousands of sequential round trips for a file near the upload size
limit, blocking the LiveView process on a single, unyielding preview
event — this is a dry-run specifically so it has to stay cheap.
Processes one slice of already-parsed rows (from parse_csv_rows/1 /
parse_text_rows/1), threading the {report, seen_emails} accumulator
from new_accumulator/0 (or a prior run_chunk/4 call) through — so
:duplicate_in_file detection and the running counts stay correct across
chunk boundaries. Lets a caller (the import UI) process a huge file in
chunks with a progress update between each, instead of blocking on the
whole file in one pass. opts accepts :actor_uuid/:source like
import_csv/3. Rows accumulate newest-first (see finalize_report/1) —
call finalize_report/1 on the report once the run completes.
Functions
@spec finalize_report(PhoenixKitCRM.Lists.ImportReport.t()) :: PhoenixKitCRM.Lists.ImportReport.t()
Puts a report's rows back into file order. Reports threaded through
run_chunk/4 accumulate rows prepended (newest first) — appending per row
would be O(n²) on a file near the upload size limit — so a caller that
renders row detail must finalize once when the run completes. run/3 and
preview_rows/2 already return finalized reports.
@spec import_csv(String.t(), PhoenixKitCRM.Schemas.ContactList.t(), keyword()) :: PhoenixKitCRM.Lists.ImportReport.t()
Imports from CSV content (already read from an upload or textarea paste).
Header row is required; column matching is case-insensitive and only
looks for email (required), name, company, locale (all optional)
— unrecognized columns are ignored. opts accepts :actor_uuid (passed
through to the activity log on every successful row).
@spec import_text(String.t(), PhoenixKitCRM.Schemas.ContactList.t(), keyword()) :: PhoenixKitCRM.Lists.ImportReport.t()
Imports from plaintext/clipboard content — one email address per line.
Blank lines are ignored. opts accepts :actor_uuid.
@spec new_accumulator() :: {PhoenixKitCRM.Lists.ImportReport.t(), MapSet.t()}
A fresh {report, seen_emails} accumulator to start chunked processing with run_chunk/4.
@spec parse_csv_rows(String.t()) :: [{pos_integer(), map()}]
Parses CSV content into {line, attrs} rows WITHOUT writing anything —
the same parser import_csv/3 uses internally, exposed for the import
UI's dry-run preview and chunked processing (run_chunk/4) so neither
has to duplicate the parsing. Non-UTF-8 content and malformed CSV (e.g.
an unterminated quote, which NimbleCSV raises on) return [] rather
than crashing the caller.
@spec parse_text_rows(String.t()) :: [{pos_integer(), map()}]
Same as parse_csv_rows/1, for plaintext/clipboard content.
@spec preview_rows([{pos_integer(), map()}], PhoenixKitCRM.Schemas.ContactList.t()) :: PhoenixKitCRM.Lists.ImportReport.t()
Classifies every row exactly like import_csv/3/import_text/3 would,
but performs NO writes — :no_email/:invalid_email/:duplicate_in_file
are determined purely from the parsed rows, and an email that would
collide on idx_crm_list_members_list_email is detected via ONE batched
Lists.members_by_email/2 lookup (all distinct emails in the file, one
query) instead of attempting (and rolling back) an insert per row. A
naive per-row Lists.get_member_by_email/2 call here would mean tens of
thousands of sequential round trips for a file near the upload size
limit, blocking the LiveView process on a single, unyielding preview
event — this is a dry-run specifically so it has to stay cheap.
Returns the same %ImportReport{} shape — created/added mean "would
be created/added" — with the full file's counts, so the caller decides
how many of rows to actually render (e.g. the first 20 for a preview).
Not a 100% guarantee of the real run's outcome: a row that fails
Contacts.create_contact/1's validation for a reason unrelated to email
uniqueness (e.g. an oversized name) only surfaces at the real
import_csv/3/import_text/3 call, since this never attempts the
insert.
@spec run_chunk( [{pos_integer(), map()}], PhoenixKitCRM.Schemas.ContactList.t(), keyword(), {PhoenixKitCRM.Lists.ImportReport.t(), MapSet.t()} ) :: {PhoenixKitCRM.Lists.ImportReport.t(), MapSet.t()}
Processes one slice of already-parsed rows (from parse_csv_rows/1 /
parse_text_rows/1), threading the {report, seen_emails} accumulator
from new_accumulator/0 (or a prior run_chunk/4 call) through — so
:duplicate_in_file detection and the running counts stay correct across
chunk boundaries. Lets a caller (the import UI) process a huge file in
chunks with a progress update between each, instead of blocking on the
whole file in one pass. opts accepts :actor_uuid/:source like
import_csv/3. Rows accumulate newest-first (see finalize_report/1) —
call finalize_report/1 on the report once the run completes.