Objective Create a long-running, repeatable PTC-Lisp conformance review program that finds: - accidental bugs where PTC-Lisp should match Clojure/Java - intentional divergences required by the PTC-Lisp design policy - unsupported Clojure features that should remain out of scope - missing tests/documentation for supported behavior Primary docs to read first: - docs/ptc-lisp-specification.md, especially the overview and design philosophy - docs/clojure-conformance-gaps.md - docs/conformance/index.md - docs/conformance/*-audit.md Repository policy: - This is a 0.x library. Prefer simplification over compatibility shims. - Clojure compatibility is the default. - Sandbox safety and recoverable signal values take precedence. - Java-named dot methods should keep Java semantics. - Generated conformance docs must not be edited by hand. Core Classification Model Every reviewed behavior should end in one of these categories: MATCH PTC-Lisp matches Clojure/Java. BUG PTC-Lisp claims support but behavior differs accidentally. DIV PTC-Lisp differs intentionally because of the language policy. UNSUPPORTED Clojure feature is intentionally outside the PTC-Lisp target surface. PTC_EXTENSION Feature is PTC-specific and should be tested against the PTC spec, not Clojure. UNKNOWN Needs manual decision. For mismatches, use this decision rule: If it is a Clojure-named function and normal finite data input: default to BUG unless the spec clearly justifies DIV. If bad external input can reasonably be recovered from: DIV may be valid if PTC returns nil/false/empty/signal value. If the program itself is invalid: should raise; silent signal values are suspicious. If it is a Java-named method or class/member call: Java semantics should win, including exceptions. If behavior depends on laziness, unbounded execution, host state, I/O, macros, eval, metadata, vars, atoms, refs, agents, or JVM internals: likely UNSUPPORTED or DIV. Phase 1: Inventory And Baseline Read the generated audit metadata, not only the generated docs. Likely files: - priv/function_audit.exs - priv/java_compat_audit.exs - lib/mix/tasks/ptc.gen_docs.ex Tasks: 1. Identify all supported, candidate, and not_relevant entries. 2. Build or script a machine-readable inventory: - namespace - symbol/method - current audit status - compatibility target: Clojure, Java, PTC extension - notes 3. Produce an initial coverage report: - supported functions with conformance tests - supported functions without conformance tests - documented DIV-* entries with regression tests - documented GAP-* entries with regression tests Expected output: conformance_inventory.json or .exs coverage report printed by a Mix task or test helper Phase 2: Conformance Case Format Add a small, explicit case format before importing large test sources. Suggested location: test/support/lisp_conformance_cases/ Suggested shape: %{ id: "core/get-001", namespace: "clojure.core", vars: ["get"], form: ~S[(get {:a 1} :a)], policy: :match } For intentional divergence: %{ id: "core/parse-long-div-001", namespace: "clojure.core", vars: ["parse-long"], form: ~S[(parse-long "abc")], policy: {:diverges, "DIV-xx"}, ptc_expected: nil, reason: "Bad external input returns a signal value instead of raising." } For unsupported behavior: %{ id: "core/lazy-range-unsupported-001", namespace: "clojure.core", vars: ["range"], form: ~S[(take 3 (range))], policy: :unsupported, reason: "Unbounded lazy sequences are outside PTC-Lisp." } Fields to support: id namespace vars form policy source reason ptc_expected clojure_expected tags Tags can include: :smoke :edge :error_semantics :destructuring :truthiness :ordering :numeric :string :collection :java :ptc_extension Phase 3: Oracle Runner Build a conformance runner that executes cases against both PTC-Lisp and Babashka/Clojure. Existing implementation to inspect: lib/ptc_runner/lisp/clojure_validator.ex test/ptc_runner/lisp/sci_conformance_test.exs Runner responsibilities: 1. Run PTC-Lisp form. 2. Run Babashka form. 3. Normalize both outputs. 4. Compare according to policy. 5. Produce readable failures. 6. Include skipped cases with reason. 7. Record unknown mismatches for triage. Important normalization concerns: - map key representation - keyword/string/symbol formatting - sets and ordering - floating point precision - ##NaN, ##Inf, ##-Inf - error vs value - eager vs lazy sequence output - Babashka JSON limitations Do not force all cases through JSON if that loses important Clojure semantics. Use EDN/string rendering where needed. Suggested test module: test/ptc_runner/lisp/conformance_runner_test.exs Suggested helper: test/support/lisp_conformance_runner.ex Phase 4: Seed Manual Cases Start small but high-value. Seed around 80-100 cases: 50 clojure.core 20 clojure.string 10 clojure.set / clojure.walk 10 intentional divergences Prioritize: - and, or, truthiness - if, when, cond, case - let, fn, defn, closure behavior - destructuring - apply - map, filter, reduce - first, rest, next, seq - get, assoc, update, get-in, assoc-in - conj, into, empty - equality and comparison - numeric operations - string split/join/replace/blank?/includes? - set union/intersection/difference - walk/prewalk/postwalk if supported For each supported function, add at least: one smoke case one nil/empty/boundary case where relevant one error-semantics case if risky Phase 5: Boundary/Error Semantics Matrix Create dedicated cases for behavior most likely to reveal policy bugs. For each important function, test: wrong arity wrong type nil input empty collection missing key out-of-range index malformed string NaN / infinity large number large collection Classify using the spec policy: bad external input -> may return signal value bad program -> should raise Java method -> Java behavior sandbox risk -> bounded/eager divergence This phase should produce many DIV-* candidates. Add them to docs/clojure-conformance-gaps.md only after deciding they are intentional. Phase 6: Upstream Test Mining Use other Clojure implementations as test sources, but import conservatively. Potential sources: Clojure official tests Babashka tests SCI tests Joker tests Clojure docstring examples Process: 1. Download or vendor references only if acceptable for licensing. 2. Extract simple assertions first: - (is (= expected form)) - (is (true? form)) - (is (false? form)) - (is (nil? form)) - (is (thrown? ... form)) 3. Filter out unsupported constructs: - macros - metadata - vars - namespaces - dynamic binding - atoms/refs/agents - JVM arrays/classes unless Java audit covers them - lazy infinite sequences - file/network/I/O 4. Convert accepted assertions into the local case format. 5. Mark source and original location. Suggested script: scripts/extract_clojure_conformance_cases.exs Do not try to import arbitrary upstream test code at first. Start with pattern-based extraction. Phase 7: Documentation Example Extraction Extract executable examples from local docs first. Sources: docs/ptc-lisp-specification.md docs/function-reference.md docs/clojure-conformance-gaps.md Then later external Clojure docs/docstrings. Process: 1. Extract fenced clojure blocks. 2. Split into individual forms where possible. 3. Classify: - Clojure-compatible - PTC extension - unsupported - documentation-only 4. Add cases or spec tests. This catches doc drift: examples in docs should either execute or be explicitly illustrative. Phase 8: Differential Fuzzing Add only after the deterministic suite is useful. Generate small supported expressions: literals vectors/maps/sets if let fn map/filter/reduce get/assoc/update string functions numeric ops comparisons Run generated forms against PTC-Lisp and Babashka. Record: seed form PTC result/error Clojure result/error classification Useful constraints: - small depth - finite collections only - no division by zero unless testing numeric edge cases - no infinite/lazy constructs - no unsupported macros - deterministic forms only Initial goal is not a perfect fuzzer. It is a mismatch discovery tool. Phase 9: Documentation And Audit Updates When a mismatch is found: 1. Decide BUG vs DIV vs UNSUPPORTED. 2. If BUG, add or update GAP-* in docs/clojure-conformance-gaps.md. 3. If DIV, add or update DIV-* with: - example Clojure behavior - example PTC behavior - rationale - linked test id 4. If audit metadata is wrong, edit metadata source, not generated docs. 5. Regenerate docs: mix ptc.gen_docs 6. Run relevant tests. Generated docs not to edit manually: docs/conformance/index.md docs/conformance/*-audit.md Phase 10: Namespace Review Order Recommended order: 1. clojure.core high-use subset 2. clojure.string 3. clojure.set 4. clojure.walk 5. Java string/math/number/time methods 6. PTC-specific extensions 7. remaining candidate functions Within clojure.core, review in clusters: syntax/special forms truthiness/control flow function invocation binding/destructuring collections maps sets sequences numbers strings/regex predicates ordering/comparison formatting/printing resource limits Suggested Milestones Milestone 1: Foundation - Add conformance case format. - Add runner. - Add 50 manual cases. - Add basic coverage report. - Verify Babashka availability or clear skip message. Milestone 2: Core Review - Cover high-use clojure.core. - Add mismatch triage workflow. - Document new BUG-* and DIV-*. Milestone 3: Namespace Review - Cover clojure.string, clojure.set, clojure.walk. - Add edge/error matrix. Milestone 4: Importers - Add simple upstream test extractor. - Add local doc example extractor. Milestone 5: Fuzzing - Add optional differential fuzzer. - Keep it separate from normal test suite unless deterministic and fast. Milestone 6: CI/Long-Running Mode - Fast conformance tests run in normal CI. - Large imported/fuzz tests run behind tag, for example: mix test --only conformance_slow Verification Commands Use repo conventions and usage rules. Start with: mix test test/ptc_runner/lisp Check available tasks: mix help For usage rules/docs when touching dependencies or Mix behavior: mix usage_rules.search_docs "mix test" mix usage_rules.docs Enum If Babashka is missing, inspect existing helper: mix ptc.install_babashka Then run targeted tests. Deliverables For The Other Codex Session Ask the other session to produce: 1. A written implementation plan in docs/plans/ptc-lisp-conformance-review.md. 2. Initial conformance case format. 3. Initial runner using Babashka and PTC-Lisp. 4. At least 50 seeded cases. 5. Coverage report for supported audit entries without cases. 6. Documentation update explaining the workflow. 7. Tests proving the runner catches: - a matching case - an intentional divergence - a PTC error - a Clojure error - a mismatch failure Non-Goals For The First Pass Do not start by trying to fully import Clojure’s whole test suite. Avoid: - arbitrary upstream test execution - broad fuzzing before deterministic cases exist - editing generated docs directly - adding compatibility shims for intentional divergences - treating every Clojure difference as a bug - testing PTC extensions against Clojure The first pass should create the machinery and seed enough cases to make the review repeatable. Then the long-running work becomes namespace-by-namespace triage.