Spark's Vector and Matrix, both ways.
Spark serialises these as struct literals typed by a JVM class rather than by field names,
so Latu hands them back as Latu.Result.UDT — the class, and the elements in the order that
class defines them, with nothing asserted about how many. Asserting is this module's job, and
the layouts are PySpark's pyspark/ml/connect/serialize.py, element for element.
A dense vector becomes an Nx.Tensor; a sparse one becomes a Latu.ML.SparseVector, because
Nx has no sparse tensor and quietly densifying a vector that is sparse on purpose is not a
decision this package makes. A matrix always comes back row-major, whatever
isTransposed said on the wire — that flag is a storage detail, not a shape.
Summary
Functions
Turn a Latu.Result.UDT into a tensor or a Latu.ML.SparseVector.
See from_udt/1. Raises instead of returning an error.
Turn a tensor or a Latu.ML.SparseVector into the literal Spark expects.
Types
@type decoded() :: Nx.Tensor.t() | Latu.ML.SparseVector.t() | nil
What a Vector or Matrix literal decodes to.
nil where the server sent an empty one. That is not a failure and not a missing value: some
attributes are genuinely empty on some models, and Nx cannot represent a tensor with a
zero-sized dimension.
Functions
@spec from_udt(Latu.Result.UDT.t()) :: {:ok, decoded()} | {:error, Latu.Error.t()}
Turn a Latu.Result.UDT into a tensor or a Latu.ML.SparseVector.
iex> udt = %Latu.Result.UDT{class: "org.apache.spark.ml.linalg.VectorUDT",
...> elements: [1, nil, nil, [0.5, -1.5]]}
iex> {:ok, tensor} = Latu.ML.Linalg.from_udt(udt)
iex> Nx.to_flat_list(tensor)
[0.5, -1.5]
iex> udt = %Latu.Result.UDT{class: "org.apache.spark.ml.linalg.VectorUDT",
...> elements: [0, 4, [1, 3], [2.0, 4.0]]}
iex> Latu.ML.Linalg.from_udt(udt)
{:ok, %Latu.ML.SparseVector{size: 4, indices: [1, 3], values: [2.0, 4.0]}}An empty answer is an answer, and the server does send them — NaiveBayesModel.sigma is a
0x0 Matrix on a multinomial model, because sigma belongs to the gaussian one. Nx has no
empty tensor at all (a dimension must be a positive integer), so this is nil rather than a
shape nothing can hold:
iex> udt = %Latu.Result.UDT{class: "org.apache.spark.ml.linalg.MatrixUDT",
...> elements: [1, 0, 0, nil, nil, [], true]}
iex> Latu.ML.Linalg.from_udt(udt)
{:ok, nil}@spec from_udt!(Latu.Result.UDT.t()) :: decoded()
See from_udt/1. Raises instead of returning an error.
@spec to_literal(Nx.Tensor.t() | Latu.ML.SparseVector.t()) :: Latu.ML.Plan.literal()
Turn a tensor or a Latu.ML.SparseVector into the literal Spark expects.
A rank-1 tensor is a dense vector, a rank-2 tensor a dense matrix sent row-major — which is
what isTransposed: true means on the wire, and why it is set. Element order and the null
placeholders are PySpark's serialize_param, so the oracle can diff this byte for byte.
Raises rather than returning an error: this is a value on its way out, and a caller who hands over a rank-3 tensor has a bug, not a runtime condition.