What a run reported besides its result: the metrics you asked for, and the ones Spark keeps.

What every *_with_metrics twin hands back beside its result. Two kinds of metric, and they are not the same kind of thing:

• observed — what a Latu.observe/3 in the plan asked for. Yours: the observation name and the metric names came out of your own source, which is why they are atoms.
• metrics — Spark's own per-node SQL metrics, one entry per plan node: rows scanned, time spent, bytes read. The server's, so the names are strings — an atom table must not grow with whatever a future Spark decides to call a metric. Same rule as Latu.collect/2's keys: :strings.

A node's metrics map is keyed by Spark's metric name, and each value carries the number and the kind of number it is ("sum", "timing", "size", "average" and so on — Spark's own metricType, passed through rather than interpreted).

{:ok, rows, info} = Latu.collect_with_metrics(df)

info.observed
#=> %{checks: %{total: 8}}

Enum.find(info.metrics, &(&1.name =~ "Scan"))
#=> %{name: "Scan parquet ", plan_id: 3, parent: 0,
#=>   metrics: %{"number of output rows" => %{value: 8, type: "sum"}}}

metrics is empty unless the server sent any; Spark decides what to report and for which plans, and inventing a zero would be worse than saying nothing. The same is true of a Latu.observe/3 name the server reported nothing for — it is simply absent.

An observation the server could not compute — or one Latu could not decode — arrives as {:error, %Latu.Error{}} under its own name, and the action still succeeds: the rows were produced, and Spark reports the failure inside the metrics message rather than failing the query. PySpark does the same, raising only when Observation.get is called.

info.observed
#=> %{checks: {:error, %Latu.Error{message: "observing checks failed: ..."}}}