defmodule Mix.Tasks.PhoenixKit.Doctor do @moduledoc """ Diagnoses PhoenixKit installation, migration, and runtime issues. Runs a comprehensive suite of checks covering database connectivity, pool configuration, PgBouncer detection, migration state, lock conflicts, and application configuration. Prints a clear pass/fail report with actionable remediation steps. ## Usage $ mix phoenix_kit.doctor $ mix phoenix_kit.doctor --prefix=auth $ mix phoenix_kit.doctor --exit-code ## Options * `--prefix` - Database schema prefix. When omitted, resolves from `config :phoenix_kit, :prefix`, then `"public"` — the same resolution `mix phoenix_kit.update` / `--status` use, so a prefixed install is diagnosed against the schema it actually lives in. * `--exit-code` - Exit non-zero when any check FAILED. Without it this task prints "N failures" and still exits 0, so a deploy script that runs it cannot act on the result — the same silent success `mix phoenix_kit.status --exit-code` exists to remove. Warnings never fail the run; they are advisory by construction and several fire on healthy installs. Off by default so deploys that run this purely for its report keep passing. ## Checks Performed 1. **Repo Detection** — Can we find and start the Ecto repo? 2. **DB Connectivity** — Can we execute a simple query? 3. **Pool Configuration** — Pool size, checkout timeout, queue settings 4. **PgBouncer Detection** — Is PgBouncer between app and PostgreSQL? 5. **Migration State** — PhoenixKit version (COMMENT), schema_migrations alignment 6. **Module Schema Versions** — Modules owning their own chain, vs what their code expects 7. **Schema Drift** — Columns a migration should have added but the DB lacks 8. **Pending Migrations** — Migration files not yet recorded in schema_migrations 9. **UUID Column Types** — Detects varchar uuid columns that crash Ecto on startup 10. **UUID Primary Keys** — Detects primary keys that are not the expected uuid type 11. **NULL UUIDs in FK Sources** — Detects NULL uuids that cause infinite backfill loops 12. **Orphaned FK References** — Detects orphaned rows that block FK constraint creation 13. **Lock Conflicts** — Any blocked or long-running queries? 14. **Orphaned Connections** — Idle-in-transaction or stuck connections 15. **Oban Configuration** — Queues and plugins that consume pool connections 16. **PhoenixKit Supervisor** — What's running (update_mode vs full)? 17. **Child Start Order** — Does the Repo start before PhoenixKit/Oban in application.ex? 18. **Update Mode** — Is update_mode active? 19. **daisyUI Version** — Is the host's vendored daisyUI recent enough? 20. **User Dashboard (deprecated)** — Is the host still on the retired dashboard? 21. **Sitemap Discoverability** — Is the sitemap actually reachable? 22. **Demo Auth Pages** — Are the demo auth routes still exposed? 23. **Manifest Repair (dry-run)** — `PhoenixKit.Migrations.Repair.verify/1` runs read-only against the generated `PhoenixKit.Migrations.ExpectedSchema` manifest as an additional, non-fatal check (never `:fail`). Passes and says so if the manifest has been removed or overridden away in this checkout. """ use Mix.Task alias PhoenixKit.Install.ChildOrder alias PhoenixKit.Install.PrefixConfig alias PhoenixKit.Migrations.ExpectedSchema.Resolver alias PhoenixKit.Migrations.Modules, as: MigrationModules alias PhoenixKit.Migrations.Postgres alias PhoenixKit.Migrations.Repair alias PhoenixKit.Migrations.Repair.Report alias PhoenixKit.Modules.Sitemap.RouteResolver @shortdoc "Diagnoses PhoenixKit installation, migration, and runtime issues" @switches [prefix: :string, exit_code: :boolean] @aliases [p: :prefix] # The longest timeout/1 any worker PhoenixKit ships declares # (Storage.Workers.SyncFilesJob). A Lifeline rescue_after at or below this is # unsafe by construction — see check_lifeline_plugin/2. @lifeline_min_rescue_after :timer.minutes(30) @impl Mix.Task def run(argv) do {opts, _argv, _errors} = OptionParser.parse(argv, switches: @switches, aliases: @aliases) # Start app with minimal footprint (same approach as phoenix_kit.update) Mix.Task.run("app.config") # Resolve the prefix AFTER app.config loads config, so a configured # non-public prefix is honored — same resolution the updater/status use # (--prefix flag → config :phoenix_kit, :prefix → "public"). Reading # opts[:prefix] || "public" here queries the version marker at public and # reports a prefixed install as "not installed". prefix = PrefixConfig.resolve_prefix(opts) # Snapshot the host's Oban config BEFORE cap_repo_pool_size/1 zeroes its # queues/plugins (it does that to conserve connections in update_mode) — # otherwise the Oban Configuration check reports "0 queues, 0 plugins". oban_config = Application.get_env(Mix.Project.config()[:app], Oban) cap_repo_pool_size(2) Application.put_env(:phoenix_kit, :update_mode, true) Mix.Task.run("app.start") header("PhoenixKit Doctor") results = [ run_check("Repo Detection", fn -> check_repo_detection() end), run_check("DB Connectivity", fn -> check_db_connectivity() end), run_check("Pool Configuration", fn -> check_pool_config() end), run_check("PgBouncer Detection", fn -> check_pgbouncer() end), run_check("Migration State", fn -> check_migration_state(prefix) end), run_check("Module Schema Versions", fn -> check_module_schema_versions(prefix) end), run_check("Schema Drift", fn -> check_schema_drift(prefix) end), run_check("Pending Migrations", fn -> check_pending_migrations() end), run_check("UUID Column Types", fn -> check_uuid_column_types(prefix) end), run_check("UUID Primary Keys", fn -> check_uuid_primary_keys(prefix) end), run_check("NULL UUIDs in FK Sources", fn -> check_null_uuids(prefix) end), run_check("Orphaned FK References", fn -> check_orphaned_fk_refs(prefix) end), run_check("Lock Conflicts", fn -> check_lock_conflicts() end), run_check("Orphaned Connections", fn -> check_orphaned_connections() end), run_check("Oban Configuration", fn -> check_oban_config(oban_config) end), run_check("PhoenixKit Supervisor", fn -> check_supervisor_state() end), run_check("Child Start Order", fn -> check_child_order() end), run_check("Update Mode", fn -> check_update_mode() end), run_check("daisyUI Version", fn -> check_daisyui() end), run_check("User Dashboard (deprecated)", fn -> check_user_dashboard_deprecation() end), run_check("Sitemap Discoverability", fn -> check_sitemap_serving() end), run_check("Demo Auth Pages", fn -> check_demo_routes() end), run_check("Manifest Repair (dry-run)", fn -> check_manifest_repair(prefix) end) ] IO.puts("") summary(results) maybe_halt(results, opts[:exit_code] || false) end @doc """ The process exit status `--exit-code` should produce: `1` when any check failed, `0` otherwise. Public because `run/1` is not a unit-test seam (it starts the app and needs a real database) — this is the pure decision behind the flag, in the same shape as `Mix.Tasks.PhoenixKit.Status.exit_code/2` and `Mix.Tasks.PhoenixKit.Repair.exit_code/1`. Only `:fail` gates. A `:warn` is advisory by construction — several fire on perfectly healthy installs (a capped pool under `update_mode`, an unreadable `application.ex`) — and gating on them would make the flag unusable, which is how a task ends up back at "reports a problem and exits 0". """ @spec exit_code([{String.t(), {:pass | :warn | :fail, String.t()}}]) :: 0 | 1 def exit_code(results) do if Enum.any?(results, fn {_name, {status, _detail}} -> status == :fail end), do: 1, else: 0 end # Printing "N failures" and exiting 0 makes this task unusable as a deploy # gate — and it now owns a check (Module Schema Versions) whose whole point is # to catch an install nothing else reports on. Opt-in for the same reason # `mix phoenix_kit.status --exit-code` is: an existing pipeline that runs # doctor for its report must not start failing on an upgrade. # # `exit({:shutdown, code})` rather than `Mix.raise/1`, matching # `mix phoenix_kit.repair`: the failures are already printed above in full, # and re-raising would bury them under a second copy. defp maybe_halt(_results, false), do: :ok defp maybe_halt(results, true) do case exit_code(results) do 0 -> :ok 1 -> exit({:shutdown, 1}) end end # ── Check implementations (return {:pass|:warn|:fail, detail}) ────── defp check_repo_detection do app = Mix.Project.config()[:app] repos = Application.get_env(app, :ecto_repos, []) if repos == [] do {:fail, "No :ecto_repos configured for :#{app}"} else repo = hd(repos) info = Enum.join( [ "app: :#{app}", "repo: #{inspect(repo)}", "adapter: #{inspect(repo.__adapter__())}" ], ", " ) {:pass, info} end end defp check_db_connectivity do repo = get_repo!() case repo.query("SELECT 1 AS ok", [], timeout: 5_000) do {:ok, %{rows: [[1]]}} -> {:pass, "Connected"} {:error, %{message: msg}} -> {:fail, "Query failed: #{msg}"} {:error, reason} -> {:fail, "Query failed: #{inspect(reason)}"} end end defp check_pool_config do app = Mix.Project.config()[:app] repo = get_repo!() config = Application.get_env(app, repo, []) pool_size = config[:pool_size] || 10 queue_target = config[:queue_target] || 50 queue_interval = config[:queue_interval] || 1000 info = Enum.join( [ "pool_size: #{pool_size}", "queue_target: #{queue_target}ms", "queue_interval: #{queue_interval}ms" ], ", " ) cond do pool_size > 20 -> {:warn, "pool_size=#{pool_size} is high — may saturate PgBouncer. #{info}"} pool_size < 2 -> {:warn, "pool_size=#{pool_size} is very low. #{info}"} true -> {:pass, info} end end defp check_pgbouncer do app = Mix.Project.config()[:app] repo = get_repo!() config = Application.get_env(app, repo, []) port = cond do config[:port] -> config[:port] config[:url] -> extract_port_from_url(config[:url]) true -> 5432 end hostname = config[:hostname] || extract_host_from_url(config[:url]) || "localhost" if port != 5432 or String.contains?(to_string(hostname), "pgbouncer") do {:warn, "Likely PgBouncer (port=#{port}, host=#{hostname}). " <> "DDL migrations should use @disable_ddl_transaction true"} else {:pass, "Direct PostgreSQL (port=#{port}, host=#{hostname})"} end end defp check_migration_state(prefix) do repo = get_repo!() escaped_prefix = String.replace(prefix, "'", "\\'") # Source 1: COMMENT ON TABLE (set by each V*.up migration) comment_version = get_comment_version(repo, escaped_prefix) # Source 2: migrated_version_runtime (what phoenix_kit.status uses) runtime_version = try do opts = %{prefix: prefix, escaped_prefix: escaped_prefix} Postgres.migrated_version_runtime(opts) rescue _ -> :error end # Source 3: Code's latest version latest_version = Postgres.current_version() lines = [ "COMMENT ON TABLE: V#{comment_version}", "migrated_version_runtime: #{if runtime_version == :error, do: "ERROR", else: "V#{runtime_version}"}", "Code latest: V#{latest_version}" ] info = Enum.join(lines, "\n ") # Detect discrepancies discrepancy = runtime_version != :error and runtime_version != comment_version cond do discrepancy -> {:warn, "DISCREPANCY between version sources!\n #{info}\n " <> "The COMMENT was updated by a migration that didn't commit to schema_migrations " <> "(killed process or missing @disable_ddl_transaction true)."} comment_version == 0 -> {:warn, "PhoenixKit not installed.\n #{info}"} comment_version < latest_version -> {:warn, "Needs migration.\n #{info}"} comment_version == latest_version -> {:pass, info} true -> {:warn, "DB version > code version.\n #{info}"} end end # Columns a given migration version adds. If the version marker claims that # version (or higher) but the column is missing at the prefix, the install # drifted — the marker is ahead of the actual schema (e.g. a version renumber # that crossed an upgrade, or an earlier prefix-confused migration run). A # query that selects the column then crashes at runtime, and re-running the # migrator is a no-op because the marker already covers that version. @expected_columns [ {150, "phoenix_kit_users_tokens", "browser"}, {150, "phoenix_kit_users_tokens", "os"} ] # Core's marker says nothing about a module that owns its own chain, so every # check above can pass while a module's tables sit versions behind the code # querying them. That gap presents as an undefined-column 500 on the module's # admin page — no migration error, and `doctor` previously gave the install a # clean bill of health, which is the worst possible moment to be reassuring. defp check_module_schema_versions(prefix) do modules = MigrationModules.list(prefix: prefix) failed = MigrationModules.failed(modules) pending = MigrationModules.pending(modules) cond do modules == [] -> {:pass, "No installed module owns migrations."} pending != [] -> {:fail, "Behind: #{describe_module_versions(pending)}#{unreadable_suffix(failed)}. " <> "Run mix phoenix_kit.update --yes (mix ecto.migrate alone does not write these)."} failed != [] -> {:warn, "Version unreadable for #{Enum.map_join(failed, ", ", & &1.name)} — " <> "their tables may be behind and nothing can tell. See mix phoenix_kit.status --verbose."} true -> {:pass, "#{length(modules)} module(s), all at the version their code expects."} end end defp describe_module_versions(entries) do Enum.map_join(entries, ", ", fn entry -> "#{entry.name} V#{entry.installed} (code expects V#{entry.target})" end) end # Behind and unreadable are not alternatives — one run can hold both, and the # `cond` above reaches the `failed` branch only when nothing is pending. Left # to it, an install with one module behind and another whose coordinator # raised reported only the first, and the unreadable module vanished from the # report entirely. It is the one an operator cannot discover any other way, # which is why `StatusReport.next_action/3` and the status tree both surface # it first; the severity stays `:fail` because a behind module is actionable. defp unreadable_suffix([]), do: "" defp unreadable_suffix(failed), do: "; version unreadable for #{Enum.map_join(failed, ", ", & &1.name)}" defp check_schema_drift(prefix) do repo = get_repo!() escaped_prefix = String.replace(prefix, "'", "\\'") marker = get_comment_version(repo, escaped_prefix) if marker == 0 do {:pass, "PhoenixKit not installed at prefix #{inspect(prefix)} — nothing to check."} else missing = @expected_columns |> Enum.filter(fn {min_version, _t, _c} -> marker >= min_version end) |> Enum.reject(fn {_v, table, column} -> column_exists?(repo, escaped_prefix, table, column) end) report_schema_drift(missing, marker, prefix) end end defp report_schema_drift([], marker, _prefix), do: {:pass, "Columns expected at V#{marker} are present."} defp report_schema_drift(missing, marker, prefix) do names = Enum.map_join(missing, ", ", fn {v, t, c} -> "#{t}.#{c} (V#{v})" end) lowest = missing |> Enum.map(fn {v, _t, _c} -> v end) |> Enum.min() p = if prefix == "public", do: "public.", else: "#{prefix}." {:fail, "Marker says V#{marker} but these columns are missing: #{names}. The install drifted " <> "(marker ahead of schema). Roll the marker back one version and re-run the migrator — " <> "the column adds are idempotent (add_if_not_exists), so this is safe:\n" <> " COMMENT ON TABLE #{p}phoenix_kit IS '#{lowest - 1}';\n" <> " mix phoenix_kit.update#{prefix_flag(prefix)}"} end defp column_exists?(repo, escaped_prefix, table, column) do query = """ SELECT EXISTS ( SELECT FROM information_schema.columns WHERE table_schema = '#{escaped_prefix}' AND table_name = '#{table}' AND column_name = '#{column}' ) """ case repo.query(query, [], log: false) do {:ok, %{rows: [[true]]}} -> true _ -> false end end defp prefix_flag("public"), do: "" defp prefix_flag(prefix), do: " --prefix=#{prefix}" defp check_pending_migrations do repo = get_repo!() migrations_path = Path.join(["priv", "repo", "migrations"]) migration_files = if File.dir?(migrations_path) do migrations_path |> File.ls!() |> Enum.filter(&String.ends_with?(&1, ".exs")) |> Enum.map(fn f -> case Integer.parse(f) do {version, _rest} -> {version, f} :error -> nil end end) |> Enum.reject(&is_nil/1) |> Enum.sort() else [] end recorded = case repo.query("SELECT version FROM schema_migrations ORDER BY version", []) do {:ok, %{rows: rows}} -> Enum.map(rows, fn [v] -> v end) |> MapSet.new() _ -> MapSet.new() end pending = Enum.reject(migration_files, fn {version, _name} -> MapSet.member?(recorded, version) end) phoenix_kit_pending = Enum.filter(pending, fn {_v, name} -> String.contains?(name, "phoenix_kit") end) # Also check for duplicate PhoenixKit migration files (same version range) pk_files = Enum.filter(migration_files, fn {_v, name} -> String.contains?(name, "phoenix_kit") end) duplicates = find_duplicate_migration_ranges(pk_files) detail_parts = [] detail_parts = if pending != [] do pk_names = Enum.map_join(phoenix_kit_pending, "\n ", fn {_v, n} -> n end) detail_parts ++ [ "#{length(pending)} pending (#{length(phoenix_kit_pending)} PhoenixKit):\n #{pk_names}" ] else detail_parts ++ ["All #{length(migration_files)} files recorded in schema_migrations"] end detail_parts = if duplicates != "" do detail_parts ++ ["DUPLICATE ranges detected:\n #{duplicates}"] else detail_parts end detail = Enum.join(detail_parts, "\n ") cond do duplicates != "" -> {:warn, detail} pending == [] -> {:pass, detail} true -> {:warn, detail} end end defp find_duplicate_migration_ranges(pk_files) do # Extract version ranges from filenames like "phoenix_kit_update_v49_to_v71.exs" ranges = Enum.map(pk_files, fn {_v, name} -> case Regex.run(~r/phoenix_kit_\w+_v(\d+)_to_v(\d+)/, name) do [_, from, to] -> {String.to_integer(from), String.to_integer(to), name} _ -> nil end end) |> Enum.reject(&is_nil/1) # Find overlapping ranges overlaps = for {from1, to1, name1} <- ranges, {from2, to2, name2} <- ranges, name1 < name2, max(from1, from2) < min(to1, to2), do: "#{name1} overlaps #{name2}" Enum.join(overlaps, "\n ") end # A missing primary key is what the varchar column actually COST, and the type # check could not see it: a table can have a perfectly typed uuid column and # still have no key. Reported by a host whose phoenix_kit_email_events had # both problems and whose doctor run named only the first. defp check_uuid_primary_keys(prefix) do repo = get_repo!() query = """ SELECT t.table_name FROM information_schema.tables t WHERE t.table_schema = $1 AND t.table_name LIKE 'phoenix\\_kit\\_%' AND t.table_type = 'BASE TABLE' AND EXISTS ( SELECT 1 FROM information_schema.columns c WHERE c.table_name = t.table_name AND c.table_schema = t.table_schema AND c.column_name = 'uuid' ) AND NOT EXISTS ( SELECT 1 FROM pg_constraint pc JOIN pg_class pcl ON pcl.oid = pc.conrelid JOIN pg_namespace pn ON pn.oid = pcl.relnamespace WHERE pcl.relname = t.table_name AND pn.nspname = t.table_schema AND pc.contype = 'p' ) ORDER BY t.table_name """ case repo.query(query, [prefix], log: false) do {:ok, %{rows: []}} -> {:pass, "Every phoenix_kit table with a uuid column has a primary key"} {:ok, %{rows: rows}} -> tables = Enum.map_join(rows, "\n ", fn [t] -> t end) {:fail, "#{length(rows)} table(s) have a uuid column but NO primary key:\n #{tables}\n " <> "Fix: mix phoenix_kit.repair_uuid (or upgrade — V163 repairs this automatically)"} _ -> {:warn, "Could not check (phoenix_kit tables may not exist yet)"} end end # Pre-migration: check for varchar/text uuid columns that should be native uuid type. # A varchar uuid column on phoenix_kit_settings crashes the Ecto schema loader on startup, # blocking migrations from even running. defp check_uuid_column_types(prefix) do repo = get_repo!() escaped_prefix = String.replace(prefix, "'", "\\'") query = """ SELECT table_name, data_type FROM information_schema.columns WHERE table_name LIKE 'phoenix_kit_%' AND column_name = 'uuid' AND table_schema = '#{escaped_prefix}' AND data_type IN ('character varying', 'text', 'character') ORDER BY table_name """ case repo.query(query, [], log: false) do {:ok, %{rows: []}} -> {:pass, "All uuid columns are native uuid type"} {:ok, %{rows: rows}} -> tables = Enum.map_join(rows, "\n ", fn [table, dtype] -> "#{table} (#{dtype})" end) # The remedy is a task, not a single ALTER. The ALTER alone restores the # TYPE and leaves the column nullable, without a UUIDv7 default and # without a primary key — anyone following it literally ends up in a # state that still fails this doctor. Reported by a host who did. {:fail, "#{length(rows)} table(s) have a varchar uuid column:\n #{tables}\n " <> "These break any Ecto schema that maps to them, and cannot carry a uuid primary key.\n " <> "Fix: mix phoenix_kit.repair_uuid (or upgrade — V163 repairs this automatically)"} _ -> {:warn, "Could not check (phoenix_kit tables may not exist yet)"} end end # Pre-migration: check for NULL uuid values in tables that are FK sources. # NULL source UUIDs cause the V56 batched backfill loop to run forever. defp check_null_uuids(prefix) do repo = get_repo!() escaped_prefix = String.replace(prefix, "'", "\\'") # Key FK source tables whose uuid column must not be NULL source_tables = [ "phoenix_kit_users", "phoenix_kit_user_roles", "phoenix_kit_entities", "phoenix_kit_email_logs", "phoenix_kit_shop_carts", "phoenix_kit_shop_products", "phoenix_kit_shop_categories", "phoenix_kit_shop_shipping_methods", "phoenix_kit_payment_options", "phoenix_kit_billing_profiles", "phoenix_kit_orders", "phoenix_kit_invoices", "phoenix_kit_payment_methods", "phoenix_kit_subscriptions", "phoenix_kit_subscription_types", "phoenix_kit_subscription_plans", "phoenix_kit_referral_codes", "phoenix_kit_ai_endpoints", "phoenix_kit_ai_prompts", "phoenix_kit_sync_connections" ] problems = Enum.reduce(source_tables, [], fn table, acc -> exists_query = """ SELECT EXISTS ( SELECT FROM information_schema.columns WHERE table_name = '#{table}' AND column_name = 'uuid' AND table_schema = '#{escaped_prefix}' ) """ case repo.query(exists_query, [], log: false) do {:ok, %{rows: [[true]]}} -> table_name = prefix_table_name(table, prefix) count_query = "SELECT count(*)::integer FROM #{table_name} WHERE uuid IS NULL" case repo.query(count_query, [], log: false) do {:ok, %{rows: [[count]]}} when count > 0 -> [{table, count} | acc] _ -> acc end _ -> acc end end) if problems == [] do {:pass, "No NULL uuids in FK source tables"} else detail = Enum.map_join(Enum.reverse(problems), "\n ", fn {table, count} -> "#{table}: #{count} rows with NULL uuid" end) {:fail, "NULL uuids found (will cause infinite loop in V56 backfill):\n #{detail}\n " <> "Fix: UPDATE