Argus.Pipeline (Panoptes v0.13.0)

Orchestrates parallel fact extraction from BEAM modules.

The pipeline runs in stages:

modules → Disassemble → Emit (Layer 1) + Extractors (Layer 2) → write .facts

Argus.Pipeline.Disassemble resolves module names to .beam paths and reads BEAM files into normalized module data.
Argus.Pipeline.Emit produces base bytecode facts from each module.
User-supplied extractors implementing Argus.Extractor produce domain-specific Layer 2 facts.
run/3 streams each module's facts to .facts files (one per relation, tab-separated) as extraction completes, so the program's fact set never exists in memory at once. extract/2 returns the merged facts in memory.

Summary

Functions

extract(modules, opts \\ [])

Extracts facts from the given modules and returns them as a map without writing to disk.

run(modules, output_dir, opts \\ [])

Extracts facts from the given modules and writes .facts files to output_dir.

write_facts(facts, output_dir)

Writes extracted facts to .facts files in output_dir (one tab-separated file per relation).

Types

extract_opts()

@type extract_opts() :: [
  concurrency: pos_integer(),
  extractors: [module()],
  timeout: timeout(),
  trace_imprecision: boolean(),
  format: :raw | :typed | :interned,
  symbols: Argus.Symbols.t()
]

run_opts()

@type run_opts() :: [
  concurrency: pos_integer(),
  extractors: [module()],
  timeout: timeout(),
  trace_imprecision: boolean(),
  relations: :all | [atom()]
]

Functions

extract(modules, opts \\ [])

@spec extract(modules :: [Argus.Pipeline.Disassemble.module_input()], extract_opts()) ::
  {:ok, Argus.Pipeline.Emit.facts() | Argus.Facts.t()} | {:error, term()}

Extracts facts from the given modules and returns them as a map without writing to disk.

With format: :typed, rows are decoded against the schema via Argus.Facts.decode/1 (field-name-keyed maps, integers, Argus.InstrId structs) instead of the raw string lists that .facts files use. With format: :interned, rows are tuples of Argus.Symbols ids interned in the worker that extracted them against the symbols: table the caller owns (Argus.Facts.interned/0); the caller keeps the table, and reads the rows back through Argus.Facts.materialize/2 or decode/2.

run(modules, output_dir, opts \\ [])

@spec run(
  modules :: [Argus.Pipeline.Disassemble.module_input()],
  output_dir :: Path.t(),
  run_opts()
) :: {:ok, Path.t()} | {:error, term()}

Extracts facts from the given modules and writes .facts files to output_dir.

Modules can be atoms (resolved via :code.which/1), string paths to .beam files, or raw beam data binaries. Returns {:ok, output_dir} or {:error, reason}.

Every schema relation gets a file (Souffle fails on a missing .input file), but relations: limits which ones receive rows: the rest stay empty. Argus.Analysis.extract_facts/3 uses it to leave out the relations that exist only for the in-process control-flow and dataflow passes (Argus.Schema.in_process_only/0), which no Souffle program reads and which are most of the fact volume.

write_facts(facts, output_dir)

@spec write_facts(Argus.Pipeline.Emit.facts(), Path.t()) :: :ok | {:error, term()}

Writes extracted facts to .facts files in output_dir (one tab-separated file per relation).

Empty files are materialized for every schema relation so Souffle never fails on a missing .input file. Callers that merge per-module fact maps themselves (rather than going through run/3) can use this to produce a Souffle-ready facts directory from in-memory facts.

Expects raw-format facts (string rows, as returned by extract/2 with the default format: :raw). The directory must already exist.