Experimental Livebook-friendly DSL for sketching dataflow pipelines.

This Lab DSL compiles to the stable, pipe-first Choreo.Dataflow builders and returns an ordinary %Choreo.Dataflow{}. It uses node constructors for dataflow stages and typed edge vocabulary for normal, error, retry, and dead-letter paths.

Examples

iex> import Choreo.Lab.DSL.Dataflow
...> pipeline = dataflow do
...>   ingest = source("Kafka Ingest", rate: "10k/s")
...>   parser = transform("JSON Parser", latency_ms: 5)
...>   valid = conditional("Valid?")
...>   postgres = sink("Postgres")
...>   dlq = sink("Dead Letter Queue")
...>
...>   ingest ~> parser |> emits("raw events")
...>   parser ~> valid |> emits("parsed events")
...>   valid ~> postgres |> writes("valid records")
...>   dead_letter valid ~> dlq, "invalid records"
...> end
iex> pipeline.graph.nodes[:ingest].node_type
:source
iex> pipeline.edge_meta[{:valid, :dlq}].path_type
:dead_letter

Edge labels can use generic labels, typed data labels, or explicit path types:

source ~> transform |> on("events")
edge source ~> transform, data_type: "events"
emits source ~> transform, "events"
error transform ~> sink, "invalid input"
retry transform ~> buffer, "retry later"
dead_letter transform ~> dlq, "poison message"