PhoenixKitCRM.Lists.Import (PhoenixKitCRM v0.3.3)

Copy Markdown View Source

Pure-ish CSV/plaintext import engine for CRM contact lists (Stage 3 of the restructuring plan — the user's stated priority: account list import). Parses rows, applies the restructuring spec's Locked decisions, and writes only through PhoenixKitCRM.Lists — it never duplicates that context's insert/counter/broadcast logic.

Locked decisions this engine enforces

  • Import always creates a NEW contact — never a merge, never "add to existing contact by matching email". Matching duplicates across contacts is a separate, later, human-reviewed comparison screen (Stage C4c), not something this engine decides on its own.
  • Per-row atomicity: contact INSERT + membership INSERT happen in ONE transaction (Lists.add_new_contact_to_list/3) — a uniqueness violation rolls both back, so a failed/duplicate row never leaves an orphan contact behind.
  • Idempotent re-import: importing the identical file twice creates zero new contacts on the second pass — every row lands in skipped instead of created/added (:already_in_list if the existing membership is still active, :unsubscribed if a prior membership was removed but still holds the email slot per the DB's partial unique index — see ListMember's moduledoc). A KNOWN collision is classified straight from a batched Lists.members_by_email/3 lookup (one per chunk — see run_chunk/4) and never reaches the write transaction at all, so re-importing a large duplicate file doesn't pay an INSERT-then-rollback per colliding row; only an email the batched lookup didn't already know about goes through it, where a race against a concurrent import is still caught by the DB's own unique index.

Row pipeline

Each row goes through, in order: normalize (trim + downcase, matching the email column's citext case-insensitivity) → in-file dedup prefilter (:duplicate_in_file — a second row with the same email, e.g. a shared mailbox listed under two names, never reaches the DB) → email format check (:invalid_email) → the batched members_by_email lookup (a KNOWN collision short-circuits straight to :already_in_list/:unsubscribed, no write attempted) → the write transaction for anything left, which still classifies an idx_crm_list_members_list_email violation (a race the batched lookup couldn't have seen) by looking up the existing holder's status.

name is CSV-optional per the spec; a blank name falls back to the email itself (mirrors Contact.display_name/1's own fallback) so the contact changeset's validate_required([:name]) never rejects an otherwise-valid row. locale, if present but not in a recognized format, is silently dropped rather than rejecting the row over a cosmetic field. company has no dedicated contact field yet (linking to PhoenixKitCRM.Companies is out of scope here) — it's stashed in contact.metadata["import_company"] so the comparison screen (C4c) can still surface it later instead of the data being silently discarded.

XLSX

Deliberately not supported yet. xlsxir (the library the plan asked to evaluate) is unmaintained since 2019 — not something to add as a new dependency. xlsx_reader is an actively-maintained alternative but wasn't evaluated in depth (its own dependency footprint, streaming behavior, etc. is a separate task). CSV/TXT cover the stated priority (account import); XLSX is deferred to a follow-up rather than shipped on an abandoned lib.

Summary

Functions

Puts a report's rows back into file order. Reports threaded through run_chunk/4 accumulate rows prepended (newest first) — appending per row would be O(n²) on a file near the upload size limit — so a caller that renders row detail must finalize once when the run completes. run/3 and preview_rows/2 already return finalized reports.

Imports from CSV content (already read from an upload or textarea paste). Header row is required; column matching is case-insensitive and only looks for email (required), name, company, locale (all optional) — unrecognized columns are ignored. opts accepts :actor_uuid (passed through to the activity log on every successful row).

Imports from plaintext/clipboard content — one email address per line. Blank lines are ignored. opts accepts :actor_uuid.

A fresh {report, seen_emails} accumulator to start chunked processing with run_chunk/4.

Parses CSV content into {line, attrs} rows WITHOUT writing anything — the same parser import_csv/3 uses internally, exposed for the import UI's dry-run preview and chunked processing (run_chunk/4) so neither has to duplicate the parsing. Non-UTF-8 content and malformed CSV (e.g. an unterminated quote, which NimbleCSV raises on) return [] rather than crashing the caller.

Same as parse_csv_rows/1, for plaintext/clipboard content.

Classifies every row exactly like import_csv/3/import_text/3 would, but performs NO writes — :no_email/:invalid_email/:duplicate_in_file are determined purely from the parsed rows, and an email that would collide on idx_crm_list_members_list_email is detected via ONE batched Lists.members_by_email/2 lookup (all distinct emails in the file, one query) instead of attempting (and rolling back) an insert per row. A naive per-row Lists.get_member_by_email/2 call here would mean tens of thousands of sequential round trips for a file near the upload size limit, blocking the LiveView process on a single, unyielding preview event — this is a dry-run specifically so it has to stay cheap.

Processes one slice of already-parsed rows (from parse_csv_rows/1 / parse_text_rows/1), threading the {report, seen_emails} accumulator from new_accumulator/0 (or a prior run_chunk/4 call) through — so :duplicate_in_file detection and the running counts stay correct across chunk boundaries. Lets a caller (the import UI) process a huge file in chunks with a progress update between each, instead of blocking on the whole file in one pass. opts accepts :actor_uuid/:source like import_csv/3. Rows accumulate newest-first (see finalize_report/1) — call finalize_report/1 on the report once the run completes.

Functions

finalize_report(report)

Puts a report's rows back into file order. Reports threaded through run_chunk/4 accumulate rows prepended (newest first) — appending per row would be O(n²) on a file near the upload size limit — so a caller that renders row detail must finalize once when the run completes. run/3 and preview_rows/2 already return finalized reports.

import_csv(content, list, opts \\ [])

Imports from CSV content (already read from an upload or textarea paste). Header row is required; column matching is case-insensitive and only looks for email (required), name, company, locale (all optional) — unrecognized columns are ignored. opts accepts :actor_uuid (passed through to the activity log on every successful row).

import_text(content, list, opts \\ [])

Imports from plaintext/clipboard content — one email address per line. Blank lines are ignored. opts accepts :actor_uuid.

new_accumulator()

@spec new_accumulator() :: {PhoenixKitCRM.Lists.ImportReport.t(), MapSet.t()}

A fresh {report, seen_emails} accumulator to start chunked processing with run_chunk/4.

parse_csv_rows(content)

@spec parse_csv_rows(String.t()) :: [{pos_integer(), map()}]

Parses CSV content into {line, attrs} rows WITHOUT writing anything — the same parser import_csv/3 uses internally, exposed for the import UI's dry-run preview and chunked processing (run_chunk/4) so neither has to duplicate the parsing. Non-UTF-8 content and malformed CSV (e.g. an unterminated quote, which NimbleCSV raises on) return [] rather than crashing the caller.

parse_text_rows(content)

@spec parse_text_rows(String.t()) :: [{pos_integer(), map()}]

Same as parse_csv_rows/1, for plaintext/clipboard content.

preview_rows(parsed_rows, list)

Classifies every row exactly like import_csv/3/import_text/3 would, but performs NO writes — :no_email/:invalid_email/:duplicate_in_file are determined purely from the parsed rows, and an email that would collide on idx_crm_list_members_list_email is detected via ONE batched Lists.members_by_email/2 lookup (all distinct emails in the file, one query) instead of attempting (and rolling back) an insert per row. A naive per-row Lists.get_member_by_email/2 call here would mean tens of thousands of sequential round trips for a file near the upload size limit, blocking the LiveView process on a single, unyielding preview event — this is a dry-run specifically so it has to stay cheap.

Returns the same %ImportReport{} shape — created/added mean "would be created/added" — with the full file's counts, so the caller decides how many of rows to actually render (e.g. the first 20 for a preview).

Not a 100% guarantee of the real run's outcome: a row that fails Contacts.create_contact/1's validation for a reason unrelated to email uniqueness (e.g. an oversized name) only surfaces at the real import_csv/3/import_text/3 call, since this never attempts the insert.

run_chunk(rows_chunk, list, opts, acc)

Processes one slice of already-parsed rows (from parse_csv_rows/1 / parse_text_rows/1), threading the {report, seen_emails} accumulator from new_accumulator/0 (or a prior run_chunk/4 call) through — so :duplicate_in_file detection and the running counts stay correct across chunk boundaries. Lets a caller (the import UI) process a huge file in chunks with a progress update between each, instead of blocking on the whole file in one pass. opts accepts :actor_uuid/:source like import_csv/3. Rows accumulate newest-first (see finalize_report/1) — call finalize_report/1 on the report once the run completes.