Deviations from PySpark

Every place latu_ml's public API departs from pyspark.ml, and why. The wire is identical — that is what test/latu/ml/wire_test.exs is for — so everything here is a surface choice.

Latu's own docs/deviations.md covers the DataFrame API this builds on.

Not a deviation, but surprising

A Vector column cannot be collected. On Spark 4.2.0 the server describes features, rawPrediction and probability as UDTs with no sql_type, so Latu's dtype guard refuses them rather than guessing a layout. So is a Vector column you build yourself with Latu.ML.Functions.array_to_vector/1: this is how Connect describes the type, not something a fit does to a column.

Three routes through, and the destination picks one:

• Latu.to_nx/2 (Latu 0.3.0) reads the Arrow bytes, where the type is stated, and gives one {rows, width} f64 tensor. Use it when the destination is a tensor: measured at 100 MB of payload it costs about a twelfth of the BEAM memory of any row-by-row rebuild, and about half the wall time (dev/probe_nx_copies.exs in Latu).
• Latu.ML.Functions.vector_to_array/2 converts server-side, so the column arrives as an ordinary array<double> any reader takes. It is PySpark's own answer and the right one when the destination is an Explorer.DataFrame — the decode to a frame is cheap; it is only carrying on to a tensor row by row that is not.
• select or drop the column, which is often the answer — a prediction column is a double and was never the problem. Latu.to_arrow/2 bypasses the guard too, if you want the bytes.

test/integration/vertical_slice_test.exs pins the refusal, so the day Spark populates sql_type this stops being true and the suite says so.

A helper call sends every argument, every time. MlParams carries only what the caller set and the server fills the rest; the ConnectHelper methods take their arguments positionally, so an unset param has to become something. It becomes the param's own default, or — for PowerIterationClustering.weightCol, the one param Spark gives no default at all — the empty string PySpark's own call site sends. Nothing here is invented: the fallback is read off that call site along with the types.

A loaded model has no training summary. A model's save format does not carry one — MLCache writes a summary to a sibling path when it offloads, and a Read does not go looking — so asking a loaded model for one answers CONNECT_ML.MODEL_SUMMARY_LOST, exactly as an offloaded model's would. The difference is what can be done about it: an offloaded summary is rebuilt from the frame the model was fitted on, and a loaded model has no such frame and never did. So the retry that makes an offload invisible does not fire here, and the server's refusal reaches the caller with Latu.ML.hint/1 naming both ways to arrive at it.

A saved grid's value is a string. estimatorParamMaps stores 0.5 as "0.5" — PySpark writes json.dumps(v) into a field of a document that is itself JSON, and json.loads on the way back. Writing the number instead produces a file PySpark will not load. Latu matches it, because the point of the format is that PySpark reads it.

A fitted search's metadata carries fewer params than an unfitted one's. PySpark's reader calls getParam for every key in paramMap and raises on one the instance lacks; CrossValidatorModel has no collectSubModels where CrossValidator does. So the key is written for a validator and omitted for its model. Nothing on this side would notice the difference — only PySpark's reader does.

A fold is cut with rand(seed) and a range, not random_split. Spark adds one random column above the data and each fold is a slice of it — CrossValidator._kFold and TrainValidationSplit both, despite the name of the latter. Adding the column once is what makes it work: a bare rand() inside a predicate is redrawn per evaluation, so train and validation would overlap and the search would still produce a plausible answer. Latu does the same, names the column <validator uid>_rand as PySpark does, and pins the arrangement in test/latu/ml_test.exs — where the assertion is that both filters read the same projection.

With no seed: the draw differs between builds, so the search is not repeatable. Latu says this for every seeded function; a grid search is where it matters most.

A fold number in a column is trusted. With fold_col:, PySpark asks the server to raise on a number outside 0..num_folds-1 and refuses an empty fold — three extra actions per fold before any fitting. Latu filters and lets the fit fail on its own rows, which costs a worse message for a case the caller created deliberately.

A saved pipeline is readable by PySpark and not by Scala. A pipeline's directory is laid out by the client, and Latu writes PySpark's layout — which means metadata says class: "pyspark.ml.pipeline.Pipeline". Scala's reader does require(className == expectedClassName), so Pipeline.load on the JVM refuses it. This is not a Latu limitation being inherited quietly: PySpark's own saved pipelines cannot be read by Scala either, for the same reason and the same one key.

What is portable: the stages/NN_uid/ directories underneath are written by the server, so each stage is in Scala's own format and loads anywhere. Only the wrapper is Python-flavoured. The rest of the metadata — uid, paramMap.stageUids, the zero-padded stage directory names — is byte-identical between the two, and PySpark 4.2's reader never reads class at all. So the whole gap is one string, and docs/decisions.md has why it is the string it is and what claiming the other one would cost.

A bare model is unaffected: its directory is written entirely by the server, so Latu.ML.save/3 on a model produces a directory any Spark client reads.

A deleted model answers CONNECT_ML.CACHE_INVALID, whose message calls the missing object a "Summary object", blames a 15-minute idle eviction, and suggests model.evaluate(dataset) — all three wrong for a model deleted a moment earlier. Match error_class, never the message. A ref that never existed answers a bare INTERNAL_ERROR with no class at all, so the two cases differ.