@spec assign_clusters(Latu.ML.Helper.t(), Latu.DataFrame.t()) :: Latu.DataFrame.t()

Assign a cluster to every vertex of an affinity matrix — Power Iteration Clustering.

A lazy builder, not an action: the frame it hands back carries a Fetch on the server's own helper object and nothing has run. What goes in is an edge list — src, dst and weight by default — and what comes back is id and cluster.

PowerIterationClustering is neither fitted nor applied; this is the one thing it does, and that is why it is a verb here rather than a model you keep. Its params ride the call as positional arguments, so every one is sent whether you set it or not.

pic = Latu.ML.Clustering.power_iteration_clustering(k: 2, max_iter: 10)
clusters = Latu.ML.assign_clusters(pic, edges)

@spec attribute(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t()) ::
  {:ok, term()} | {:error, Latu.Error.t()}

Read a model attribute whose answer is a value — a coefficient, an intercept, a count.

An action. A Vector or Matrix comes back as an Nx.Tensor, or a Latu.ML.SparseVector where densifying would be this package's decision rather than yours.

Either spelling. Latu.ML.attributes/1 lists snake_case names and the server's allowlist is camelCase, so :area_under_roc and "areaUnderROC" both work and the table is a list of things you can actually ask for. A name on the allowlist is taken as written, so a wire name never means anything else.

A name neither spelling accounts for goes to the server as written and comes back as CONNECT_ML.ATTRIBUTE_NOT_ALLOWED: a typo is its error and not this package's, and that is also what keeps a class the registry does not know reachable at all. The generated accessor modules — Latu.ML.Classification.LogisticRegressionModel and its 41 siblings — are that allowlist as named functions and are what to reach for first; this is the layer underneath.

@spec attribute(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t(), [term()]) ::
  {:ok, term()} | {:error, Latu.Error.t()}

Read an attribute that takes arguments — predict, fMeasureByLabel, recommendForAllUsers.

An action, and the same Fetch as attribute/2 with the arguments attached to its last method. The types are not yours to state: priv/ml_attributes.exs records them from PySpark's own annotations, so predict knows it wants a Vector and recommendForAllUsers an int, and a wrong count or a wrong kind is refused here rather than on the server.

The generated accessors are these too, and are what to reach for first — Latu.ML.Classification.LogisticRegressionModel.predict/2 is this call with the class checked and the arguments named.

{:ok, label} = Latu.ML.attribute(model, "predict", [Nx.tensor([1.0, 0.0])])
{:ok, f1} = Latu.ML.attribute(summary, "fMeasureByLabel", [1.0])

A DataFrame argument is passed as one and rides the relation arm — Latu.ML.attribute(model, "evaluateEachIteration", [frame, "logLoss"]). Attributes that answer with a frame go to attribute_frame/3 instead. The evaluate attribute — a model's, not the evaluator verb evaluate/2 — answers with another cached object rather than a value, and has no verb of its own: asking for it here reaches the server and comes back as a protocol error naming the arm it answered on.

Needs the model's class, since that is what the types are recorded against. A model built by Latu.ML.Estimator.new/2 without one has nothing to look them up in.

@spec attribute!(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t()) ::
  term()

See attribute/2. Raises instead of returning an error.

@spec attribute!(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t(), [
  term()
]) :: term()

See attribute/3. Raises instead of returning an error.

@spec attribute_frame(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t()) ::
  Latu.DataFrame.t()

Read a model attribute whose answer is a DataFrame — gaussiansDF, itemFactors and their kind.

A lazy builder, not an action: the same Fetch attribute/2 sends, wrapped as a relation rather than a command, so this hands back a Latu.DataFrame and nothing has run.

Which attributes answer this way is not a judgement call — the registry records it from PySpark's own try_remote_attribute_relation decorator, and the generated accessors are split by it. Asking for a literal-valued attribute here builds a relation the server will refuse.

@spec attribute_frame(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t(), [
  term()
]) ::
  Latu.DataFrame.t()

A frame-valued attribute that takes arguments — describeTopics, recommendForAllUsers, approxNearestNeighbors.

attribute_frame/2 and attribute/3 in one: still a lazy builder, still typed from priv/ml_attributes.exs rather than from the caller.

users = Latu.ML.attribute_frame(model, "recommendForAllUsers", [5])
near = Latu.ML.attribute_frame(model, "approxNearestNeighbors", [df, key, 3, "distance"])

@spec attributes(
  module()
  | Latu.ML.Model.t()
  | Latu.ML.Summary.t()
  | String.t()
  | nil
) :: [map()] | nil

What the server will let you Fetch from a class, and how each one answers.

Takes a generated accessor module, a fitted Latu.ML.Model, a Latu.ML.Summary, or a JVM class name. The server's allowlist is inherited, so this is the union up the class's hierarchy rather than what its own entry says.

Where the class is not known — an LDA fit, or a summary whose class the fitted model picks — this answers with what every class it could be allows, since only those are certain to be fetchable. Name the class directly to see its own, wider list.

iex> Latu.ML.attributes(Latu.ML.Clustering.KMeansModel) |> Enum.map(& &1.name)
[:cluster_center_matrix, :has_summary, :num_features, :predict, :summary, :to_string]

@spec best_index(
  Latu.ML.CrossValidatorModel.t()
  | Latu.ML.TrainValidationSplitModel.t()
) ::
  non_neg_integer()

Which param map won, as an index into the validator's param_maps.

Pure. The metric is maximised or minimised according to larger_better?/1 on the evaluator that scored it, and ties go to the first — so the order param_grid/2 built the grid in is what separates two param maps that scored the same.

@spec cache_info(Latu.Session.t()) ::
  {:ok, [cache_entry()]} | {:error, Latu.Error.t()}

Everything the session's ML cache is holding: one entry per model and summary.

An action. Each entry is %{id: ..., class: ..., size: ...} — the reference a %Latu.ML.Model{} carries, the JVM class, and the size in bytes the cache charged it. The server renders these as JSON strings and this decodes them, so a shape the server changes arrives as {:error, %Latu.Error{kind: :protocol}} rather than as a surprise.

size is 0 for a summary, and 0 for a model too when the session has memory control turned off: MLCache only measures what it is going to budget.

{:ok, held} = Latu.ML.cache_info(session)
Enum.map(held, & &1.class)

Takes a session rather than a model — the cache belongs to the session, and the point of asking is usually that you have lost track of what is in it.

@spec cache_info!(Latu.Session.t()) :: [cache_entry()]

See cache_info/1. Raises instead of returning an error.

@spec clean_cache(Latu.Session.t()) ::
  {:ok, non_neg_integer()} | {:error, Latu.Error.t()}

Drop everything the session's ML cache holds, and say how many objects that was.

An action, and a blunt one: MLCache.clear empties the map and the offload directory together, so every model and summary in the session goes at once and every %Latu.ML.Model{} still in hand becomes a reference to nothing — the next fetch on one answers CONNECT_ML.CACHE_INVALID.

delete/1 and with_model/3 are the ordinary way to give a model back. This is for a session you are reusing and want to start clean in, and for the end of a long-lived one.

@spec clean_cache!(Latu.Session.t()) :: non_neg_integer()

See clean_cache/1. Raises instead of returning an error.

@spec cross_validator(keyword()) :: Latu.ML.CrossValidator.t()

A k-fold cross-validation, as data. See Latu.ML.CrossValidator.

A builder, not an action: nothing here reaches a server and no session is needed, because a validator has no wire form at all. fit/2 is what searches.

cv =
  ML.cross_validator(
    estimator: lr,
    param_maps: ML.param_grid(lr, reg_param: [0.1, 0.01]),
    evaluator: Evaluation.binary_classification_evaluator(),
    num_folds: 3,
    seed: 42
  )

estimator:, param_maps: and evaluator: are required. num_folds: defaults to 3, seed: to none — and with no seed the fold cut differs between builds, so pass one for a search you can repeat. fold_col: names a column that already holds a fold number, in which case no draw is added at all. collect_sub_models: true keeps every fold's models instead of releasing them as they are scored.

@spec delete(deletable() | [deletable()]) :: :ok | {:error, Latu.Error.t()}

Release a model, or a list of them, from the server's ML cache.

An action, and the explicit half of the resource rule. The server answers with the references it dropped rather than staying silent, so a failure here is a real one.

Everything goes in one Delete, whatever shape it arrives in. A list is what attribute(model, "trees") needs — the server registers every sub-model it hands back, so a hundred-tree forest is a hundred cache entries and giving them back one at a time is a hundred round trips. A composite is flattened: a PipelineModel owns the models inside it, nested pipelines included, and a search result owns its winner and the fold models where it was asked to keep them. And a mixed list of any of those is fine, because a script that fitted a pipeline and a search writes delete([fitted, best]).

Inside a composite, a stage that holds nothing on the server is skipped rather than refused, so deleting a pipeline of transformers is :ok and sends nothing. At the top level the contract is strict: anything that is not one of the four is refused by name, because a silent :ok for an argument this cannot release is the worse answer.

@spec delete!(deletable() | [deletable()]) :: :ok

See delete/1. Raises instead of returning an error.

@spec error_kind(Latu.Error.t()) :: atom() | nil

Which ML failure this is, as an atom, or nil for anything that is not one.

Match on this rather than on error_class, and never on the message. The class strings carry a CONNECT_ML. prefix that Spark's own documentation leaves off, so a matcher written from the docs compares against a string the server never sends — and the messages are worse: see hint/1.

case Latu.ML.attribute(model, :coefficients) do
  {:ok, value} -> value
  {:error, error} ->
    case Latu.ML.error_kind(error) do
      :cache_invalid -> refit()
      _other -> raise error
    end
end

The five: :cache_invalid (the object is gone), :attribute_not_allowed (the name is not on this class's allowlist), :summary_lost (the cache dropped a summary), :model_too_large and :cache_full (a fit refused against a budget). A reference that never existed at all arrives as a bare INTERNAL_ERROR with no class, and answers nil here.

@spec evaluate(Latu.ML.Evaluator.t(), Latu.DataFrame.t()) ::
  {:ok, float()} | {:error, Latu.Error.t()}

Score a frame with an evaluator: one number comes back.

An action, and the third of Spark's three verbs. The frame is where the session comes from, as it is for fit/2, and it is a frame a model has already scored — an evaluator reads prediction_col and label_col, not features. Which number it is is the evaluator's metric_name, whose default is its own: areaUnderROC for a binary classifier, rmse for a regressor.

scored = ML.transform(model, test)
{:ok, auc} = ML.evaluate(Evaluation.binary_classification_evaluator(), scored)

Nothing is cached: unlike a fit, an evaluate answers with the metric itself.

@spec evaluate!(Latu.ML.Evaluator.t(), Latu.DataFrame.t()) :: float()

See evaluate/2. Raises instead of returning an error.

@spec find_frequent_sequential_patterns(Latu.ML.Helper.t(), Latu.DataFrame.t()) ::
  Latu.DataFrame.t()

Find the frequent sequential patterns in a frame of sequences — PrefixSpan.

assign_clusters/2's twin in shape: a lazy builder over the helper object, for the other operator that is neither fitted nor applied. The input column is sequence, an array<array<T>>; what comes back is sequence and freq.

ps = Latu.ML.FPM.prefix_span(min_support: 0.5, max_pattern_length: 5)
patterns = Latu.ML.find_frequent_sequential_patterns(ps, sequences)

@spec fit(Latu.ML.Estimator.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.Model.t()} | {:error, Latu.Error.t()}

@spec fit(Latu.ML.Pipeline.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.PipelineModel.t()} | {:error, Latu.Error.t()}

@spec fit(Latu.ML.CrossValidator.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.CrossValidatorModel.t()} | {:error, Latu.Error.t()}

@spec fit(Latu.ML.TrainValidationSplit.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.TrainValidationSplitModel.t()} | {:error, Latu.Error.t()}

Fit an estimator, or a whole pipeline, on a frame.

An action: the fit runs on the server and the result is a handle. The frame is where the session comes from, so there is no session argument.

The model carries the estimator's uid and params, because a Fit answers with an object reference and nothing else — no uid, no params, no warning.

A pipeline

A Latu.ML.Pipeline is not a server operator: there is no Fit for one on the wire, so this folds over its stages and does one action per estimator, exactly as PySpark does. What comes back is a Latu.ML.PipelineModel that owns every model it fitted — delete/1 on it releases them all, and with_model/3 brackets the whole thing.

The fold has one surprise worth stating. The frame is transformed by each stage on the way through, but only up to the last estimator: stages after it are carried into the pipeline model untouched, because there is nothing left to learn from them. And if a later stage fails, the models fitted before it are released before the error is returned, so a half-built pipeline leaks nothing.

{:ok, model} = ML.fit(ML.pipeline([assembler, scaler, lr]), training)

A grid search

A Latu.ML.CrossValidator or Latu.ML.TrainValidationSplit has no Fit on the wire either. This runs the search: num_folds * length(param_maps) fits, each scored by the evaluator on its held-out slice, and then one more — the winning param map refit on the whole frame, because the fold models each saw only a fraction of the rows.

Serial, one fit at a time. PySpark runs the grid on a thread pool; whether several processes can drive one session's channel at once is not something this package has measured, so it does not claim it. docs/decisions.md.

A sub-model is released as soon as its metric is read, so a 5-by-6 search holds one cache entry rather than thirty. collect_sub_models: true keeps them and hands them back, and they are then yours — delete/1 on the returned model releases them along with the winner.

cv = ML.cross_validator(estimator: lr, param_maps: grid, evaluator: evaluator)
{:ok, searched} = ML.fit(cv, training)
searched.avg_metrics

@spec fit!(
  Latu.ML.Estimator.t()
  | Latu.ML.Pipeline.t()
  | Latu.ML.CrossValidator.t()
  | Latu.ML.TrainValidationSplit.t(),
  Latu.DataFrame.t()
) :: Latu.ML.Model.t() | Latu.ML.PipelineModel.t() | searched()

See fit/2. Raises instead of returning an error.

@spec helper(Latu.Session.t(), atom(), [term()]) ::
  {:ok, term()} | {:error, Latu.Error.t()}

Call a method on the server's own helper object.

The escape hatch under Latu.ML.Stat, Latu.ML.Feature.load_default_stop_words/2 and the generated Latu.ML.Feature.StringIndexerModel.from_labels/3 and its siblings — the same relationship attribute/3 has to the generated accessors.

ConnectHelper is a singleton the server resolves by name: there is no model behind it, nothing is cached and nothing can be deleted. Its arguments are positional and complete — every declared one is sent, including the uid the three model-building methods take, which the named constructors generate and this does not.

{:ok, locale} = ML.helper(session, :stop_words_remover_get_default_or_us)
{:ok, words} = ML.helper(session, :stop_words_remover_load_default_stop_words, ["german"])

A method whose answer is a DataFrame goes to helper_frame/2 instead; asking for one here raises rather than building a relation the server would refuse. Latu.ML.helpers/0 is the table of what there is.

@spec helper!(Latu.Session.t(), atom(), [term()]) :: term()

See helper/3. Raises instead of returning an error.

@spec helper_frame(atom(), [term()]) :: Latu.DataFrame.t()

Call a helper method whose answer is a DataFrame.

A lazy builder, where helper/3 is an action — the same split attribute_frame/2 and attribute/2 have, and the same Fetch underneath. Five of the eleven answer this way, and every one of them takes the frame it works on as an argument, so the session comes from there and there is none to pass.

pairs = ML.helper_frame(:correlation, [frame, "features", "pearson"])

@spec helpers() :: [map()]

Every method on the server's own helper object, as the registry holds them.

The ConnectHelper route: eleven methods in the same allowlist a model's attributes are in, and attributes of nothing.

Each is a map with :name, the :method that goes on the wire, how it :returns (:value, :frame or :model), its :args named and typed, and the :python call site all of that was read from — a helper has no class and so no signature, so the call site is the only thing that says what it takes.

Ten of the eleven were read that way. handleOverwrite has no such call site — PySpark reaches it through _call_java — so its arguments are pinned in the extractor with the call site they were read off, and its row carries read_by_hand: true rather than passing for something derived. save/3 is what uses it: an operator's own overwrite: rides MlCommand.Write, but a pipeline writes a directory, and clearing one is this method.

iex> Latu.ML.helpers() |> Enum.frequencies_by(& &1.returns) |> Enum.sort()
[frame: 5, model: 3, value: 3]

@spec hint(Latu.Error.t()) :: String.t() | nil

What to do about an ML failure, in this package's words, or nil if it has nothing to add.

Spark's own message reaches the caller untouched — that is Latu's rule, and its text is usually the better one. These five are the exception, because their text is wrong in ways that send a reader the wrong way:

• CACHE_INVALID describes the missing object as a Summary object even for a model, blames a 15-minute idle eviction even for a model deleted a moment earlier, and recommends model.evaluate(dataset). All three measured false.
• every one of them opens "Generic Spark Connect ML error".

So this is a second opinion beside the server's, never a replacement for it.

{:error, error} = Latu.ML.attribute(deleted, :coefficients)
Latu.ML.hint(error)

@spec larger_better?(Latu.ML.Evaluator.t()) :: boolean()

Whether a bigger number from evaluate/2 is a better one, for this evaluator.

Pure — it reaches no server, because there is nothing there to ask. isLargerBetter is not on the server's allowlist, so PySpark's six evaluators each override it client-side to make it work over Connect, and those overrides are what the registry extracts. This resolves one against the evaluator's own metric_name, set or default.

iex> Latu.ML.larger_better?(Latu.ML.Evaluation.regression_evaluator())
false
iex> Latu.ML.larger_better?(Latu.ML.Evaluation.regression_evaluator(metric_name: "r2"))
true

Cross-validation and train/validation split pick their best model by this, which is why it is public: a caller comparing metrics by hand should not have to remember that rmse is minimised and r2 maximised.

@spec load(Latu.Session.t(), loadable(), String.t(), keyword()) ::
  {:ok, persistable()} | {:error, Latu.Error.t()}

Read a model or an operator back from a path.

An action, and the other half of save/3. What is at the path is named rather than discovered: Spark's metadata carries the class, but a Read has to say which class to load before the server will look, so the caller states what they expect and a mismatch is the server's refusal.

Three spellings, one per way a caller knows what they saved:

• an operator name from the registry — :logistic_regression for the estimator, :logistic_regression_model for the model it fits;
• a generated model module — Latu.ML.Classification.LogisticRegressionModel;
• {class, kind} for a JVM class this package does not know, where kind is :estimator, :transformer, :evaluator or :model.

A model comes back holding a fresh cache reference: reading one costs a cache entry exactly as fitting one does, and it is yours to delete/1. It carries no training frame, so summary/1 on it has nothing to rebuild from — and nothing to rebuild either, because a saved model does not carry its summary. Everything else comes back as inert data with nothing cached, carrying the uid and the params the metadata held.

Examples

{:ok, model} = ML.load(session, :logistic_regression_model, "/tmp/lr")
scored = ML.transform(model, features)

{:ok, lr} = ML.load(session, :logistic_regression, "/tmp/lr-unfitted")

@spec load!(Latu.Session.t(), loadable(), String.t(), keyword()) :: persistable()

See load/4. Raises instead of returning an error.

@spec model_size(Latu.ML.Model.t()) ::
  {:ok, non_neg_integer()} | {:error, Latu.Error.t()}

What the server thinks a model weighs, in bytes.

An action, and a server estimate — Model.estimatedSize, which nothing client-side can compute. It is the number the cache's budgets are spent against, so it is what explains a CONNECT_ML.MODEL_SIZE_OVERFLOW_EXCEPTION on a fit that was refused, and what to measure before fitting a hundred of something.

{:ok, bytes} = Latu.ML.model_size(model)

@spec model_size!(Latu.ML.Model.t()) :: non_neg_integer()

See model_size/1. Raises instead of returning an error.

@spec operator(atom()) :: Latu.ML.Operator.t() | nil

One operator by name, or nil.

iex> Latu.ML.operator(:standard_scaler).class
"org.apache.spark.ml.feature.StandardScaler"

@spec operator!(atom()) :: Latu.ML.Operator.t()

See operator/1. Raises on a miss, naming the operators whose spelling is close.

@spec operators(keyword()) :: [Latu.ML.Operator.t()]

Every operator this package knows, as Latu.ML.Operator structs.

The table the constructors are generated from, handed back so you can look at it. Filters are and-ed, and an unknown filter key raises rather than quietly matching nothing.

Options

• :kind — :estimator, :transformer, :evaluator or :model.
• :group — PySpark's own module grouping: :classification, :clustering, :evaluation, :feature, :fpm, :recommendation, :regression.
• :status — :probed, :built or :missing. See Latu.ML.Operator.status/0.

Example

iex> Latu.ML.operators(kind: :evaluator) |> Enum.map(& &1.name)
[:binary_classification_evaluator, :clustering_evaluator,
 :multiclass_classification_evaluator, :multilabel_classification_evaluator,
 :ranking_evaluator, :regression_evaluator]

@spec param_grid([
  {Latu.ML.Estimator.t() | Latu.ML.Transformer.t() | Latu.ML.Evaluator.t(),
   keyword([term()])}
]) :: [param_map()]

See param_grid/2. Takes the pairs together, so the product spans several operators.

@spec param_grid(
  Latu.ML.Estimator.t() | Latu.ML.Transformer.t() | Latu.ML.Evaluator.t(),
  keyword([term()])
) :: [param_map()]

A grid of params to try, as data.

Every combination of the values given, which is what cross-validation and train/validation split search over. A builder in the sense that nothing runs: no session, no server, and the params are checked against the operator's own table now rather than after ten fits.

grid = ML.param_grid(lr, reg_param: [0.1, 0.01], max_iter: [10, 100])
length(grid)
#=> 4

Each entry says which operator's param it sets, by uid. That matters because the thing being tuned is often a pipeline and the param belongs to one stage of it:

lr = Classification.logistic_regression()
pipeline = ML.pipeline([assembler, lr])
grid = ML.param_grid(lr, reg_param: [0.1, 0.01])

ML.cross_validator(estimator: pipeline, param_maps: grid, evaluator: evaluator)

To vary params on two operators at once, pass the pairs together — the product is taken across all of them, not per operator:

ML.param_grid([{assembler, [handle_invalid: ["skip", "keep"]]}, {lr, [max_iter: [10]]}])

Order

The last param given varies fastest, as itertools.product does it in PySpark. This is worth knowing because two param maps that score identically are separated by nothing but their position: the best one is the first maximum.

@spec params(atom()) :: [Latu.ML.Param.t()]

An operator's param table, for a look in iex.

iex> Latu.ML.params(:vector_assembler) |> Enum.map(& &1.name)
[:handle_invalid, :input_cols, :output_col]

@spec pipeline([Latu.ML.Pipeline.stage()]) :: Latu.ML.Pipeline.t()

A pipeline: stages fitted as one, in order.

A builder, not an action — there is no Fit for a pipeline on the wire, so this reaches no server and needs no session. Latu.ML.fit/2 is what folds over the stages, and what it hands back is a Latu.ML.PipelineModel.

A stage is an estimator, a transformer, a model already fitted, or another pipeline. Anything else is refused here rather than partway through a fit that has already cached two models.

ML.pipeline([
  Feature.vector_assembler(input_cols: [:x1, :x2], output_col: :features),
  Feature.standard_scaler(input_col: :features, output_col: :scaled),
  Classification.logistic_regression(features_col: :scaled, max_iter: 10)
])

@spec save(persistable(), String.t(), keyword()) :: :ok | {:error, Latu.Error.t()}

Write a model or an unfitted operator to a path, in Spark's own on-disk format.

An action. The path is the server's, not this machine's: the write happens where the session is, so a bare path is a path on the driver, and anything a second machine has to read wants a URL the cluster's filesystem understands (s3://…, hdfs://…).

What comes out is Spark's own format, so PySpark and Scala read it as readily as load/3 does — and the reverse, which is what the interop check in dev/README.md measures rather than assumes.

Options

• :overwrite — replace what is already at the path. Defaults to false, where the server refuses and names the path.
• :options — the writer's own options, a map of strings, handed to Spark untouched.
• :session — required for an unfitted operator, and refused for a model. An estimator, a transformer and an evaluator are inert data with no session behind them; a model carries its own, and a second one beside it would be a way to write to the wrong server.
• :sub_models — a search only. Whether to write the fold models beside the winner. Defaults to whether the search collected any, which is PySpark's default too, though it reaches it through a lowercased key in the writer's option map rather than an option of its own. Asking for them where the search kept none is refused rather than silently writing nothing.

A search

A Latu.ML.CrossValidator and its model lay out a directory rather than a single operator: metadata with the validator's params and its grid, then estimator/ and evaluator/, and for a fitted one bestModel/ and the metrics. subModels/foldN/M/ where they were kept — a split has no foldN level, because it has one.

:ok = ML.save(searched, "/tmp/cv", overwrite: true, sub_models: true)

Examples

:ok = ML.save(model, "/tmp/lr", overwrite: true)
:ok = ML.save(lr, "/tmp/lr-unfitted", session: session)

@spec save!(persistable(), String.t(), keyword()) :: :ok

See save/3. Raises instead of returning an error.

@spec summary(Latu.ML.Model.t()) :: Latu.ML.Summary.t()

The training summary a fitted model carries.

A lazy builder: Spark has no separate cache key for a summary, so this composes the reference from the model's own and nothing is sent. Its attributes are on the generated summary module for the class — Latu.ML.Classification.BinaryLogisticRegressionSummary and its kind.

Raises for a model class that has no summary, naming the ten that do. Ten of the 43 is not an oversight: a summary is what an estimator recorded while fitting, and most do not record one.

summary = Latu.ML.summary(model)
{:ok, iterations} = Latu.ML.attribute(summary, :totalIterations)

A summary is the one object the server's cache drops rather than offloading, and a fetch that finds it gone answers CONNECT_ML.MODEL_SUMMARY_LOST. attribute/2 recovers from that on its own, with the frame this struct carries; nothing is asked of the caller.

@spec train_validation_split(keyword()) :: Latu.ML.TrainValidationSplit.t()

A single train/validation split, as data. See Latu.ML.TrainValidationSplit.

The same builder as cross_validator/1 with one split instead of k folds, so each param map is scored once rather than averaged. train_ratio: defaults to 0.75.

tvs =
  ML.train_validation_split(
    estimator: lr,
    param_maps: ML.param_grid(lr, reg_param: [0.1, 0.01]),
    evaluator: Evaluation.binary_classification_evaluator()
  )

@spec transform(
  Latu.ML.Transformer.t()
  | Latu.ML.Model.t()
  | Latu.ML.PipelineModel.t()
  | searched(),
  Latu.DataFrame.t()
) :: Latu.DataFrame.t()

Apply a transformer or a fitted model to a frame.

A lazy builder, not an action: the frame it hands back carries the ML relation and nothing has run. Latu.schema/1 will answer from it without executing, and Latu.collect/2 is what finally reaches the server.

@spec with_model(
  Latu.ML.Estimator.t() | Latu.ML.Pipeline.t(),
  Latu.DataFrame.t(),
  (Latu.ML.Model.t()
   | Latu.ML.PipelineModel.t() ->
     result)
) ::
  result | {:error, Latu.Error.t()}
when result: var

Fit, use, release — the bracket, in File.open/3's shape.

The model is deleted in an after, so it goes even if the function raises. A delete that itself fails is logged rather than raised: the useful result is the one in hand, and the session's end bounds the leak anyway.

Returns whatever the function returned, or the fit's error if there was never a model.

{:ok, coefficients} =
  Latu.ML.with_model(lr, features, fn model ->
    Latu.ML.attribute(model, :coefficients)
  end)

@spec with_model!(
  Latu.ML.Estimator.t() | Latu.ML.Pipeline.t(),
  Latu.DataFrame.t(),
  (Latu.ML.Model.t()
   | Latu.ML.PipelineModel.t() ->
     result)
) ::
  result
when result: var

See with_model/3. Raises where the fit would have returned an error.