Expressions: column references, literals, operators.
import Latu.Column
Latu.filter(df, all([greater(:price, 100), not_equal(:suburb, "Reservoir")]))Names match Explorer.Series wherever the operation is the same one. Latu already depends on
Explorer, so there is no reason to make you carry two vocabularies.
Expressions are values. Extract one into a function, put a list of them through Enum, hand
them around — nothing here is a macro.
asc/1 and friends build a sort key, not an expression: Spark's SortOrder is its own
message. asc puts nulls first and desc puts them last, which is SQL's asymmetry and
PySpark's.
This module is PySpark's Column methods — operators, predicates, casts, sort keys and
over/2. Spark's free functions are Latu.Functions, which is aliased rather than imported.
Where Spark offers both spellings of one idea (Column.isNull and the SQL function isnull),
this module wins and Latu.Functions does not wrap the twin — nine of them; see
docs/deviations.md.
There is no as here: an expression is named where it is projected — select(df, total: F.sum(:price)), Latu.agg/2, Latu.with_columns/2, Latu.observe/3 — so nothing in this
module shares a name with Latu, and importing both is clean. Latu.Plan.as/2 is the bare
alias if a plan needs one.
def expensive, do: greater(:price, 100)
Latu.filter(df, all([expensive() | extra]))Summary
Functions
Spark's +.
Every predicate holds.
Any predicate holds. all/1, with or; none is lit(false).
A sort key, as PySpark's Column.asc.
A sort key, as PySpark's Column.asc_nulls_first.
A sort key, as PySpark's Column.asc_nulls_last.
Between two bounds, inclusive.
Cast to a Spark type, spelled as SQL spells it. See Latu.Plan.cast/2.
A column reference. See Latu.Plan.col/1.
Spark's contains.
A sort key, as PySpark's Column.desc.
A sort key, as PySpark's Column.desc_nulls_first.
A sort key, as PySpark's Column.desc_nulls_last.
Spark's /.
Spark's endsWith.
Spark's ==.
Spark's <=>.
Raw SQL, parsed by the server. See Latu.Plan.expr/1.
Any Spark function, by name.
Spark's >.
Spark's >=.
Spark's ilike.
Case-insensitive like/3.
Spark's isNaN.
Spark's isNotNull.
Spark's isNull.
One of these values, or one of a DataFrame's rows.
Spark's <.
Spark's <=.
Spark's like.
A SQL LIKE pattern with an explicit escape character.
A typed literal. See Latu.Plan.lit/1.
Spark's *.
Negate a predicate. not is an operator — def not(x) is a syntax error.
Not equal.
Evaluate an expression over a window.
Spark's power.
Spark's %.
Spark's rlike.
Every column.
Spark's startsWith.
Spark's -.
cast/2, but null where a cast would fail. See Latu.Plan.try_cast/2.
Functions
@spec add(term(), term()) :: Latu.Plan.expression()
Spark's +.
@spec all([term()]) :: Latu.Plan.expression()
Every predicate holds.
Left-folded, so the tree matches PySpark's a & b & c. One predicate is itself, and none is
lit(true) — the identity, so a filtered list of predicates composes without a branch.
@spec any([term()]) :: Latu.Plan.expression()
Any predicate holds. all/1, with or; none is lit(false).
@spec asc(term()) :: Latu.Plan.sort_order()
A sort key, as PySpark's Column.asc.
@spec asc_nulls_first(term()) :: Latu.Plan.sort_order()
A sort key, as PySpark's Column.asc_nulls_first.
@spec asc_nulls_last(term()) :: Latu.Plan.sort_order()
A sort key, as PySpark's Column.asc_nulls_last.
@spec between(term(), term(), term()) :: Latu.Plan.expression()
Between two bounds, inclusive.
Not a Spark function: PySpark composes it as (c >= lower) and (c <= upper), and so does
this.
@spec cast(term(), String.t()) :: Latu.Plan.expression()
Cast to a Spark type, spelled as SQL spells it. See Latu.Plan.cast/2.
@spec col(String.t() | atom()) :: Latu.Plan.expression()
A column reference. See Latu.Plan.col/1.
@spec contains(term(), term()) :: Latu.Plan.expression()
Spark's contains.
@spec desc(term()) :: Latu.Plan.sort_order()
A sort key, as PySpark's Column.desc.
@spec desc_nulls_first(term()) :: Latu.Plan.sort_order()
A sort key, as PySpark's Column.desc_nulls_first.
@spec desc_nulls_last(term()) :: Latu.Plan.sort_order()
A sort key, as PySpark's Column.desc_nulls_last.
@spec divide(term(), term()) :: Latu.Plan.expression()
Spark's /.
@spec ends_with(term(), term()) :: Latu.Plan.expression()
Spark's endsWith.
@spec equal(term(), term()) :: Latu.Plan.expression()
Spark's ==.
@spec equal_null_safe(term(), term()) :: Latu.Plan.expression()
Spark's <=>.
@spec expr(String.t()) :: Latu.Plan.expression()
Raw SQL, parsed by the server. See Latu.Plan.expr/1.
@spec fun(String.t(), [term()], keyword()) :: Latu.Plan.expression()
Any Spark function, by name.
fun("upper", [:suburb])The escape hatch for a function Latu.Functions has no wrapper for. See Latu.Plan.fun/3;
distinct: true is the one option.
@spec greater(term(), term()) :: Latu.Plan.expression()
Spark's >.
@spec greater_equal(term(), term()) :: Latu.Plan.expression()
Spark's >=.
@spec ilike(term(), term()) :: Latu.Plan.expression()
Spark's ilike.
@spec ilike(term(), term(), term()) :: Latu.Plan.expression()
Case-insensitive like/3.
@spec is_nan(term()) :: Latu.Plan.expression()
Spark's isNaN.
@spec is_not_null(term()) :: Latu.Plan.expression()
Spark's isNotNull.
@spec is_null(term()) :: Latu.Plan.expression()
Spark's isNull.
@spec isin(term(), term()) :: Latu.Plan.expression()
One of these values, or one of a DataFrame's rows.
isin(:suburb, ["Reservoir", "Northcote"])
isin(:id, Latu.select(recent, :id))Spark calls the function in, which Elixir cannot, so this keeps PySpark's name.
A DataFrame builds an IN subquery rather than a function call, as Column.isin does in
PySpark: the frame is hoisted into the plan that uses it and has to have one column — or one
per field, when the left side is a F.struct/1. No same-session check, as in
Latu.DataFrame.scalar/1.
@spec less(term(), term()) :: Latu.Plan.expression()
Spark's <.
@spec less_equal(term(), term()) :: Latu.Plan.expression()
Spark's <=.
@spec like(term(), term()) :: Latu.Plan.expression()
Spark's like.
@spec like(term(), term(), term()) :: Latu.Plan.expression()
A SQL LIKE pattern with an explicit escape character.
like/2 sends two arguments and this sends three — Spark distinguishes them, so this is a
second clause rather than a default. Latu.Functions deliberately does not also wrap like:
the Column spelling owns it. See docs/deviations.md.
@spec lit(term()) :: Latu.Plan.expression()
A typed literal. See Latu.Plan.lit/1.
@spec multiply(term(), term()) :: Latu.Plan.expression()
Spark's *.
@spec not_(term()) :: Latu.Plan.expression()
Negate a predicate. not is an operator — def not(x) is a syntax error.
@spec not_equal(term(), term()) :: Latu.Plan.expression()
Not equal.
Spark has no != function: PySpark negates ==, and the op_compare fixture is why this is
not a row in the table above.
@spec over(Latu.Plan.expression(), Latu.Window.t()) :: Latu.Plan.expression()
Evaluate an expression over a window.
alias Latu.Window, as: W
window = W.partition_by([:suburb]) |> W.order_by([desc(:price)])
Latu.with_columns(df, rank: over(F.rank(), window))A window with no partition_by moves every row into one partition. Spark allows it and it is
occasionally what you want, so this warns rather than refusing — as PySpark does.
W.partition_by([]) says the global window is meant, and is not warned about.
@spec pow(term(), term()) :: Latu.Plan.expression()
Spark's power.
@spec remainder(term(), term()) :: Latu.Plan.expression()
Spark's %.
@spec rlike(term(), term()) :: Latu.Plan.expression()
Spark's rlike.
@spec star() :: Latu.Plan.expression()
Every column.
@spec starts_with(term(), term()) :: Latu.Plan.expression()
Spark's startsWith.
@spec subtract(term(), term()) :: Latu.Plan.expression()
Spark's -.
@spec try_cast(term(), String.t()) :: Latu.Plan.expression()
cast/2, but null where a cast would fail. See Latu.Plan.try_cast/2.