Converting between a Vector column and an ordinary array of numbers.
pyspark.ml.functions' two, and the way to read a Vector column: Latu.collect/2 and
Latu.to_explorer/2 both refuse one, because Spark describes it as a UDT that does not say
its SQL type (docs/deviations.md).
alias Latu.ML.Functions
scored = Latu.ML.transform(model, features)
scored
|> Latu.select([:prediction, features: Functions.vector_to_array(:features)])
|> Latu.collect()Both are lazy: each builds a column expression and reaches no server.
Spark keeps these in an internal function registry rather than the builtin one, so they
are unreachable from SQL — Latu.sql/3 on SELECT vector_to_array(…) answers
UNRESOLVED_ROUTINE on any 4.2.0 server, and that is by design rather than a gap. Over
Connect they resolve, which is what these two build.
Both are UDFs on the server rather than compiled expressions, so they cost a pass over the
rows, and vector_to_array densifies a sparse Vector rather than refusing it.
Summary
Functions
An array of numbers as a dense Vector column.
The dtypes vector_to_array/2 takes, which are the two Spark registers.
A Vector column as an array of numbers.
Types
@type column() :: atom() | String.t() | Latu.Plan.expression()
A column.
An atom or a string names one; anything else is used as the expression it is, so a built
column such as Latu.Column.fun/3's result goes straight through.
Functions
@spec array_to_vector(column()) :: Latu.Plan.expression()
An array of numbers as a dense Vector column.
The input is an array<double>; the answer is a Vector every ML operator accepts, and one
Latu.collect/2 refuses like any other.
Functions.array_to_vector(:embedding)VectorAssembler is the other way to build one, and the better one where the features are
already separate columns — it takes them without a round trip through an array.
@spec dtypes() :: [String.t()]
The dtypes vector_to_array/2 takes, which are the two Spark registers.
@spec vector_to_array(column(), String.t()) :: Latu.Plan.expression()
A Vector column as an array of numbers.
dtype is "float64" — the default — for an array<double>, or "float32" for an
array<float>. Dense and sparse vectors alike; a sparse one comes back dense.
Functions.vector_to_array(:features)
Functions.vector_to_array(:features, "float32")The dtype is checked here rather than on the server, which would answer
INVALID_PARAMETER_VALUE.DTYPE a round trip later.