Latu ML

Copy Markdown View Source

Latu ML

Spark MLlib from Elixir, over Spark Connect, as a companion package to Latu.

The algorithms run on the cluster, where the data is. The Elixir side holds plans and handles.

MLlib is DataFrame in, DataFrame out. There is no fit(x, y) on tensors: features go into one Vector column, and a fitted model scores a frame.

Install

def deps do
  [
    {:latu, "~> 0.4"},
    {:latu_ml, "~> 0.2"}
  ]
end

Requires Elixir ~> 1.20 and a Spark 4.2.0 Connect server, the same one Latu targets.

A server to talk to

Point at your cluster's sc:// URL, or run one locally:

docker run -p 15002:15002 apache/spark:4.2.0 \
  /opt/spark/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.13:4.2.0

The lines at the top of your file

Latu.ML is called qualified, the way Latu is. There are seven operator groups, PySpark's own grouping, and you name one every time you build something. Alias the ones you need:

alias Latu.ML
alias Latu.ML.{Classification, Clustering, Evaluation, Feature, Regression}

A first model

{:ok, session} = Latu.connect("sc://localhost:15002")

training =
  Latu.sql!(session, """
  SELECT CAST(label AS DOUBLE) AS label, CAST(x1 AS DOUBLE) AS x1, CAST(x2 AS DOUBLE) AS x2
  FROM VALUES (0.0, 0.0, 1.1), (0.0, 0.1, 1.0), (1.0, 2.0, 0.1), (1.0, 2.1, 0.2)
  AS t(label, x1, x2)
  """)

assembler = Feature.vector_assembler(input_cols: [:x1, :x2], output_col: :features)
features = ML.transform(assembler, training)

{:ok, auc} =
  ML.with_model(Classification.logistic_regression(max_iter: 10), features, fn model ->
    ML.evaluate(Evaluation.binary_classification_evaluator(), ML.transform(model, features))
  end)

Three things there are the whole package. ML.transform/2 is a lazy builder. It hands back a Latu.DataFrame and reaches no server. ML.fit/2 is an action, and what it hands back is a reference into a server-side cache rather than a value. ML.with_model/3 deletes that reference in an after, because the BEAM has no finalizer to do it for you.

The quick start is the same walk with the results in it, executed against a live server.

What is in it

Every operator the 4.2.0 server exposes, generated rather than transcribed: 111 rows carrying 991 params, read off PySpark's own classes and the server's own Scala. That is 68 constructors, and 623 accessors across 64 model and summary modules.

The verbs. fit, transform, evaluate, attribute, attribute_frame, summary, save, load, delete, with_model, model_size, cache_info, clean_cache. Actions return {:ok, _} | {:error, %Latu.Error{}} with ! twins; builders return a struct or a Latu.DataFrame.

Attributes off a fitted model, as Nx tensors, Latu.ML.SparseVectors or lazy frames. coefficients, feature_importances, cluster centres, a training summary's ROC curve. What you may ask for is the server's own allowlist, and the generated accessor per class is the discovery surface: h Latu.ML.Regression.LinearRegressionTrainingSummary.r2 prints Spark's own prose.

Pipelines and tuning, client-side on both sides of the wire because Spark exposes no Fit for either: Latu.ML.pipeline/1, Latu.ML.param_grid/2, Latu.ML.cross_validator/1 and Latu.ML.train_validation_split/1, all saveable and loadable in Spark's own on-disk format.

The ConnectHelper route for the things that are neither fitted nor applied: Latu.ML.Stat's three statistical tests, Latu.ML.assign_clusters/2, Latu.ML.find_frequent_sequential_patterns/2, and the model constructors that build a StringIndexerModel from labels or a CountVectorizerModel from a vocabulary.

Two column functions: Latu.ML.Functions.vector_to_array/2 and Latu.ML.Functions.array_to_vector/1. They are the way to read a Vector column, which Latu.collect/2 refuses: Spark describes it as a UDT and declines to say what it serialises as.

Not here. Anything closure-shaped, the RDD-era spark.mllib, OneVsRest, and building a model out of parameters you already hold.

What this is

Remote MLlib, not a local one. Two things are load-bearing:

  1. A model is not a value. It lives on the server, in a per-session cache that offloads to disk under memory pressure and is gone when the session ends. Latu.ML.with_model/3 brackets one; Latu.ML.fit/2 and Latu.ML.delete/1 are the pair for when the handle has to outlive an expression. Nothing here holds a process, so nothing can release it for you.

  2. The surface is generated from two extractors and checked by a probe. Constructors, params, accessors and their docs come from priv/ml_operators.exs and priv/ml_attributes.exs. Those are read off PySpark's classes and the server's MLUtils, not typed by hand. Every constructor's @doc says whether a live server has run it.

Scholar runs classical ML on one node, in memory, and hands you a model you can inspect and ship. This runs MLlib on a cluster and hands you a reference. Reach for Scholar when the training set fits a node; reach for this when it does not, or when the features already live in a lakehouse. With Nx and Scholar is that seam.

Status

0.1.0, and pre-1.0 in the way that matters: a minor version may rename or remove, and CHANGELOG.md carries the migration in one line when it does.

Every runnable operator has been fitted, applied or evaluated against a live 4.2.0 server, with every allowlisted attribute answered.

Where to go next

  • Quick start. Connect, fit, read, score, release. Every line of it is executed.
  • Cookbook. A recipe per model family, and the ones for pipelines, grid search and cleaning up.
  • Coming from PySpark ML. The five differences, and what a model is here.
  • With Nx and Scholar. Getting numbers back onto the BEAM, and where that stops working.
  • Cheatsheet. Every verb on one page.
  • usage-rules.md. The short set of rules that are not guessable from the function names, in the usage_rules convention, so an agent can sync it into its context.
  • docs/deviations.md. Every place the API departs from pyspark.ml, and why.
  • CONTRIBUTING.md. The servers, the two extractors, the oracle and the probe.

Latu's own docs cover the DataFrame API this is built on; its usage rules are worth having in context alongside these.

Development

docker-compose.yml brings up the two Spark servers; CONTRIBUTING.md has the rest.

docker compose up -d --wait
mix check      # format, compile, offline tests
mix check.all  # the above plus integration, against sc://localhost:15003

How this was built

Claude (Anthropic) wrote the overwhelming majority of the code, the tests and the documentation. The maintainers set the scope, made the design calls, ran every gate and reviewed the result. The same division as Latu's, stated for the same reason.

License

Apache License 2.0. See the LICENSE file.