Latu.Subquery (latu v0.1.0)

Copy Markdown View Source

An expression that references other DataFrames, and the relations it references.

SubqueryExpression carries only the referenced relation's plan_id; the relation itself has no field to travel in, so it travels here instead — beside the expression — until the verb consuming it hoists it into a WithRelations wrapper. PySpark keeps the same state on its expression objects and collects it in LogicalPlan._collect_references.

Inert data, like Latu.CaseWhen: Latu.Plan's builders take one wherever an expression belongs and pass the references up, so F.abs(scalar(other)) carries as readily as scalar(other) does. Build one with Latu.Plan.subquery/3.

A subquery is the only cross-DataFrame reference Spark resolves. A bare Latu.col/2 pointing outside its own tree is refused whether or not the relation is hoisted — measured, see docs/decisions.md (M9.1).

Summary

Types

t()

expr is the built proto — an Expression, or a SortOrder when the reference sits inside a sort key. refs are the relations to hoist, in the order Spark's client collects them.

Types

t()

@type t() :: %Latu.Subquery{expr: struct(), refs: [struct()]}

expr is the built proto — an Expression, or a SortOrder when the reference sits inside a sort key. refs are the relations to hoist, in the order Spark's client collects them.