ALLM.Pipeline.Registry (allm_pipeline v0.1.0)

Copy Markdown View Source

The host's one declaration of how it wires the framework, and of the domain knowledge the framework must not carry itself.

defmodule MyApp.Pipelines do
  use ALLM.Pipeline.Registry,
    repo: MyApp.Repo,
    store: ALLM.Pipeline.Store.Ecto,
    artifacts: ALLM.Pipeline.Artifacts.Dynamo,
    lock: ALLM.Pipeline.Lock.Noop,
    llm: MyApp.Pipelines.LLM,
    alert_on_empty: ~w(some_scrape),
    lock_keys: [some_refresh: :some]
end

It FEEDS the config keys — it does not become a second way to read them

install/0 writes each declaration into the application environment under the key the framework already reads:

DeclarationKey writtenRead by
repo::repoALLM.Pipeline.Config.repo/0
store:ALLM.Pipeline.Storeimpl:ALLM.Pipeline.Store.impl/0
artifacts:ALLM.Pipeline.Artifactsimpl:ALLM.Pipeline.Artifacts.impl/0
lock:ALLM.Pipeline.Lockimpl:ALLM.Pipeline.Lock.impl/0
llm: (optional)ALLM.Pipeline.LLMimpl:ALLM.Pipeline.LLM.impl/0
alert_on_empty::alert_on_emptyALLM.Pipeline.Config.alert_on_empty/0
lock_keys::lock_keysALLM.Pipeline.Config.lock_keys/0

The pipelines: declaration is the one collection this table does NOT cover: it is not installed into the application environment — it is read directly through __registry__(:pipelines) by host code (a host's pipeline runner) and by the mix allm_pipeline.names codegen task. See "The pipelines: declaration" below.

So no repo/0 (or store/0, artifacts/0, lock/0) accessor is generated on the registry module, deliberately. Batch 1.B decided that ALLM.Pipeline.Config.repo/0 is the package's single, permanent host-repo handle with four consumers, two of which sit outside Store entirely (ALLM.Pipeline.Metrics, ALLM.Pipeline.Lock.Advisory) — see that module's moduledoc. An accessor here would be exactly the second resolution path that decision exists to prevent, and registry_test.exs plus behaviours_test.exs's "exactly one module in the package exposes a repo accessor" pin its absence.

__registry__/1 exists for guards and tooling — it reads the DECLARATION, never the resolved value, and nothing in the framework calls it. It is what lets a host-side test assert "what the framework resolves equals what the registry declared" without hand-mirroring either side.

The pipelines: declaration

pipelines: is the host's per-pipeline metadata table — one entry per cron-dispatchable pipeline, each a map of name / entry / browser / run_names / schedules. It is the single source of truth the extraction plan's §3.8a/§3.8b codegen is built on: a host's pipeline runner derives its dispatch table and name lists from it, and mix allm_pipeline.names emits it as JSON for the shell/SST/stage-scraper consumers.

Unlike the wiring keys above, pipelines: is not installed into the application environment — install/0 ignores it — because it is host domain data read on demand, not framework wiring resolved at runtime. It is read through __registry__(:pipelines) (an empty list when undeclared).

The entry: MFAs name host modules ({CommitteePipeline, :run}), so they are stored as opaque data: package code never references them, only the host's Runner does. That keeps the leaf boundary intact — the package task reads name/browser only, never entry.

The artifacts: tuple form

artifacts: is the one seam that also accepts an {module, keyword} tuple:

artifacts: {ALLM.Pipeline.Artifacts.Tiered,
            small: ALLM.Pipeline.Artifacts.Dynamo,
            large: ALLM.Pipeline.Artifacts.S3}

The module selects the adapter (installed under :impl, exactly as the bare-module form is); the keyword carries that adapter's own wiring — a tiered adapter's small: / large: / threshold:. Those opts install under the adapter's own config key (ALLM.Pipeline.Artifacts.Tiered here), where every adapter already reads its runtime values (Artifacts.Filesystem's :root, Artifacts.S3's :bucket, Artifacts.Dynamo's :dynamo) — NOT beside :impl on the ALLM.Pipeline.Artifacts seam key, which selects the adapter and carries nothing else. Written with put_new semantics like the seam keys, so a config-file override of the adapter's own key wins per environment. store: / lock: / llm: / repo: keep the strict module-only contract — a tuple there raises "must be a module" (pinned by registry_test.exs). small: / large: are module wiring (compile-time appropriate); a threshold: is a policy constant, not an env-specific VALUE like a table name — the adapter defaults it to the DynamoDB item capacity when omitted, so the declaration usually names only small: / large:.

Which module at compile time, which value at runtime

The declaration is evaluated where the host uses it, so module wiring is fixed at compile time. Every value an adapter needs (table names, endpoints, filesystem roots) still resolves at runtime through Application.get_env inside the adapter — a mix release build never evaluates config/runtime.exs, so a registry compiled in Docker cannot know those. Do not fold a config VALUE into a registry declaration (extraction plan §3.3).

When install/0 runs

The write happens at application start, not at compile time: a compile-time Application.put_env mutates only the compiling VM and is not carried into a release's sys.config, so a registry that wrote at compile time would leave the repo unset in production. The host calls install/0 from its Application.start/2, and a host's runner typically also calls Application.ensure_all_started/1 before dispatching, which covers the release bin/<release> eval cron path.

install/0 is idempotent and safe to call more than once.

Precedence: a config file outranks the declaration for the seams

Config files are applied before Application.start/2, so install/0 is the LAST writer of every key in the table above. That would silently overwrite an env-specific adapter override, and two of this package's moduledocs document exactly such an override as the supported route (Artifacts.Filesystem's "run a fresh clone with zero cloud infrastructure", Lock's "restore the advisory lock"). So the seam keys are written with put_new semantics — the declaration supplies the DEFAULT and an explicit

config :allm_pipeline, ALLM.Pipeline.Artifacts,
  impl: ALLM.Pipeline.Artifacts.Filesystem

wins, per environment, exactly as it did before a registry existed. That asymmetry is deliberate: a registry cannot be env-specific (it is one compile-time declaration), while adapter selection legitimately is.

:repo, :alert_on_empty and :lock_keys are written unconditionally. They are not adapter selection, they have no env-specific use, and config/config.exs names the registry as :repo's sole writer — a stale config line silently outranking the declaration is the failure mode there, not a feature. registry_test.exs's "a config-file impl: outranks the declaration" describe pins both halves.

:allm_pipeline is the config namespace

The package reads and writes all of its configuration under its own OTP app name — the conventional Elixir shape. It is hardcoded here exactly as it is in ALLM.Pipeline.Config, Store, Artifacts, Lock, LLM, LLMCallLog and Artifacts.Dynamo; those literals share this one value so the keys this module feeds and the keys the accessors read never fork.

Summary

Types

A validated registry declaration: the four mandatory wiring modules, the optional llm: one (nil when undeclared), the two domain collections (lock_keys normalized as a map), and the per-pipeline metadata list ([] when undeclared).

One per-pipeline metadata entry. browser and schedules are normalized to their defaults (false / []) when a declaration omits them.

One SST cron schedule entry attached to a pipeline.

Functions

Declare a host's framework wiring and domain collections.

Types

declaration()

@type declaration() :: %{
  repo: module(),
  store: module(),
  artifacts: module(),
  artifacts_opts: keyword(),
  lock: module(),
  llm: module() | nil,
  alert_on_empty: [String.t()],
  lock_keys: %{required(atom()) => atom()},
  pipelines: [pipeline_entry()]
}

A validated registry declaration: the four mandatory wiring modules, the optional llm: one (nil when undeclared), the two domain collections (lock_keys normalized as a map), and the per-pipeline metadata list ([] when undeclared).

artifacts_opts is the keyword list an artifacts: {module, keyword} declaration carries ([] for the bare-module form) — a tiered adapter's small: / large: / threshold: wiring. See "The artifacts: tuple form" in the moduledoc.

pipeline_entry()

@type pipeline_entry() :: %{
  name: atom(),
  entry: {module(), atom()},
  browser: boolean(),
  run_names: [String.t()],
  schedules: [schedule_entry()]
}

One per-pipeline metadata entry. browser and schedules are normalized to their defaults (false / []) when a declaration omits them.

schedule_entry()

@type schedule_entry() :: %{id: String.t(), cron: String.t(), description: String.t()}

One SST cron schedule entry attached to a pipeline.

Functions

__using__(opts)

(macro)

Declare a host's framework wiring and domain collections.

Options — :repo, :store, :artifacts and :lock are required modules; :llm (an ALLM.Pipeline.LLM adapter) is an optional module; :alert_on_empty (a list of PipelineRun.name strings) and :lock_keys (a keyword list of pipeline atom → canonical atom) default to empty.

Note :alert_on_empty keys on run-name strings rather than cron atoms: the two namespaces do not line up (one pipeline module emits several run names by mode — extraction plan §3.8a), while :lock_keys keys on the cron atom ALLM.Pipeline.Lock.with_lock/2 is called with.

:pipelines (optional, default []) is the per-pipeline metadata list — see "The pipelines: declaration" in the moduledoc. Each entry is a map declaring at least name: (a unique cron atom), entry: (a {module, function} tuple) and run_names: (a list of PipelineRun.name strings); browser: (default false) and schedules: (default []) are optional. It is read through __registry__(:pipelines) and is not installed into the application environment.