Regressors, and the models they fit.
Everything here predicts a number. For the linear families the fitted coefficients are the
model, so reading them back as tensors and scoring on the BEAM with Nx is a real option —
see the model modules in this namespace.
Every constructor here is generated from PySpark 4.2.0's own param table, and every
accessor module from the server's own attribute allowlist. Nothing in this file is
hand-written but the words you are reading — see Latu.ML.operators/1 for the table it all
comes from.
Summary
Functions
Accelerated Failure Time (AFT) Model Survival Regression
Decision tree learning algorithm for regression. It supports both continuous and categorical features.
Factorization Machines learning algorithm for regression.
Gradient-Boosted Trees (GBTs) learning algorithm for regression. It supports both continuous and categorical features.
Generalized Linear Regression.
Currently implemented using parallelized pool adjacent violators algorithm. Only univariate (single feature) algorithm supported.
Linear regression.
Random Forest learning algorithm for regression. It supports both continuous and categorical features.
Functions
@spec aft_survival_regression(keyword()) :: Latu.ML.Estimator.t()
Accelerated Failure Time (AFT) Model Survival Regression
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.AFTSurvivalRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:aggregation_depth— suggested depth for treeAggregate (>= 2). Default2.:censor_col— censor column name. The value of this column could be 0 or 1. If the value is 1, it means the event has occurred i.e. uncensored; otherwise censored. Default"censor".:features_col— features column name. Default"features".:fit_intercept— whether to fit an intercept term. Defaulttrue.:label_col— label column name. Default"label".:max_block_size_in_mb— maximum memory in MB for stacking input data into blocks. Data is stacked within partitions. If more than remaining data size in a partition then it is adjusted to the data size. Default 0.0 represents choosing optimal value, depends on specific algorithm. Must be >= 0. Default0.0.:max_iter— max number of iterations (>= 0). Default100.:prediction_col— prediction column name. Default"prediction".:quantile_probabilities— quantile probabilities array. Values of the quantile probabilities array should be in the range (0, 1) and the array should be non-empty. Default[0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.99].:quantiles_col— quantiles column name. This column will output quantiles of corresponding quantileProbabilities if it is set.:tol— the convergence tolerance for iterative algorithms (>= 0). Default1.0e-6.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec decision_tree_regressor(keyword()) :: Latu.ML.Estimator.t()
Decision tree learning algorithm for regression. It supports both continuous and categorical features.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.DecisionTreeRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:cache_node_ids— If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. Defaultfalse.:checkpoint_interval— set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. Default10.:features_col— features column name. Default"features".:impurity— Criterion used for information gain calculation (case-insensitive). Supported options: variance. Default"variance".:label_col— label column name. Default"label".:leaf_col— Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. Default"".:max_bins— Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature. Default32.:max_depth— Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Must be in range [0, 30]. Default5.:max_memory_in_mb— Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. Default256.:min_info_gain— Minimum information gain for a split to be considered at a tree node. Default0.0.:min_instances_per_node— Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. Default1.:min_weight_fraction_per_node— Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in interval [0.0, 0.5). Default0.0.:prediction_col— prediction column name. Default"prediction".:seed— random seed.:variance_col— column name for the biased sample variance of prediction.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec fm_regressor(keyword()) :: Latu.ML.Estimator.t()
Factorization Machines learning algorithm for regression.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.FMRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:factor_size— Dimensionality of the factor vectors, which are used to get pairwise interactions between variables. Default8.:features_col— features column name. Default"features".:fit_intercept— whether to fit an intercept term. Defaulttrue.:fit_linear— whether to fit linear term (aka 1-way term) Defaulttrue.:init_std— standard deviation of initial coefficients. Default0.01.:label_col— label column name. Default"label".:max_iter— max number of iterations (>= 0). Default100.:mini_batch_fraction— fraction of the input data set that should be used for one iteration of gradient descent. Default1.0.:prediction_col— prediction column name. Default"prediction".:reg_param— regularization parameter (>= 0). Default0.0.:seed— random seed.:solver— The solver algorithm for optimization. Supported options: gd, adamW. (Default adamW) Default"adamW".:step_size— Step size to be used for each iteration of optimization (>= 0). Default1.0.:tol— the convergence tolerance for iterative algorithms (>= 0). Default1.0e-6.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec gbt_regressor(keyword()) :: Latu.ML.Estimator.t()
Gradient-Boosted Trees (GBTs) learning algorithm for regression. It supports both continuous and categorical features.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.GBTRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:cache_node_ids— If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. Defaultfalse.:checkpoint_interval— set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. Default10.:feature_subset_strategy— The number of features to consider for splits at each tree node. Supported options: 'auto' (choose automatically for task: If numTrees == 1, set to 'all'. If numTrees > 1 (forest), set to 'sqrt' for classification and to 'onethird' for regression), 'all' (use all features), 'onethird' (use 1/3 of the features), 'sqrt' (use sqrt(number of features)), 'log2' (use log2(number of features)), 'n' (when n is in the range (0, 1.0], use n * number of features. When n is in the range (1, number of features), use n features). default = 'auto'. Default"all".:features_col— features column name. Default"features".:impurity— Criterion used for information gain calculation (case-insensitive). Supported options: variance. Default"variance".:label_col— label column name. Default"label".:leaf_col— Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. Default"".:loss_type— Loss function which GBT tries to minimize (case-insensitive). Supported options: squared, absolute. Default"squared".:max_bins— Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature. Default32.:max_depth— Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Must be in range [0, 30]. Default5.:max_iter— max number of iterations (>= 0). Default20.:max_memory_in_mb— Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. Default256.:min_info_gain— Minimum information gain for a split to be considered at a tree node. Default0.0.:min_instances_per_node— Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. Default1.:min_weight_fraction_per_node— Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in interval [0.0, 0.5). Default0.0.:prediction_col— prediction column name. Default"prediction".:seed— random seed.:step_size— Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of each estimator. Default0.1.:subsampling_rate— Fraction of the training data used for learning each decision tree, in range (0, 1]. Default1.0.:validation_indicator_col— name of the column that indicates whether each row is for training or for validation. False indicates training; true indicates validation.:validation_tol— Threshold for stopping early when fit with validation is used. If the error rate on the validation input changes by less than the validationTol, then learning will stop early (beforemaxIter). This parameter is ignored when fit without validation is used. Default0.01.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec generalized_linear_regression(keyword()) :: Latu.ML.Estimator.t()
Generalized Linear Regression.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.GeneralizedLinearRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:aggregation_depth— suggested depth for treeAggregate (>= 2). Default2.:family— The name of family which is a description of the error distribution to be used in the model. Supported options: gaussian (default), binomial, poisson, gamma and tweedie. Default"gaussian".:features_col— features column name. Default"features".:fit_intercept— whether to fit an intercept term. Defaulttrue.:label_col— label column name. Default"label".:link— The name of link function which provides the relationship between the linear predictor and the mean of the distribution function. Supported options: identity, log, inverse, logit, probit, cloglog and sqrt.:link_power— The index in the power link function. Only applicable to the Tweedie family.:link_prediction_col— link prediction (linear predictor) column name.:max_iter— max number of iterations (>= 0). Default25.:offset_col— The offset column name. If this is not set or empty, we treat all instance offsets as 0.0.:prediction_col— prediction column name. Default"prediction".:reg_param— regularization parameter (>= 0). Default0.0.:solver— The solver algorithm for optimization. Supported options: irls. Default"irls".:tol— the convergence tolerance for iterative algorithms (>= 0). Default1.0e-6.:variance_power— The power in the variance function of the Tweedie distribution which characterizes the relationship between the variance and mean of the distribution. Only applicable for the Tweedie family. Supported values: 0 and [1, Inf). Default0.0.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec isotonic_regression(keyword()) :: Latu.ML.Estimator.t()
Currently implemented using parallelized pool adjacent violators algorithm. Only univariate (single feature) algorithm supported.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.IsotonicRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:feature_index— The index of the feature if featuresCol is a vector column, no effect otherwise. Default0.:features_col— features column name. Default"features".:isotonic— whether the output sequence should be isotonic/increasing (true) orantitonic/decreasing (false). Defaulttrue.:label_col— label column name. Default"label".:prediction_col— prediction column name. Default"prediction".:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec linear_regression(keyword()) :: Latu.ML.Estimator.t()
Linear regression.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.LinearRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:aggregation_depth— suggested depth for treeAggregate (>= 2). Default2.:elastic_net_param— the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty. Default0.0.:epsilon— The shape parameter to control the amount of robustness. Must be > 1.0. Only valid when loss is huber. Default1.35.:features_col— features column name. Default"features".:fit_intercept— whether to fit an intercept term. Defaulttrue.:label_col— label column name. Default"label".:loss— The loss function to be optimized. Supported options: squaredError, huber. Default"squaredError".:max_block_size_in_mb— maximum memory in MB for stacking input data into blocks. Data is stacked within partitions. If more than remaining data size in a partition then it is adjusted to the data size. Default 0.0 represents choosing optimal value, depends on specific algorithm. Must be >= 0. Default0.0.:max_iter— max number of iterations (>= 0). Default100.:prediction_col— prediction column name. Default"prediction".:reg_param— regularization parameter (>= 0). Default0.0.:solver— The solver algorithm for optimization. Supported options: auto, normal, l-bfgs. Default"auto".:standardization— whether to standardize the training features before fitting the model. Defaulttrue.:tol— the convergence tolerance for iterative algorithms (>= 0). Default1.0e-6.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec random_forest_regressor(keyword()) :: Latu.ML.Estimator.t()
Random Forest learning algorithm for regression. It supports both continuous and categorical features.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Regression.RandomForestRegressionModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:bootstrap— Whether bootstrap samples are used when building trees. Defaulttrue.:cache_node_ids— If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. Defaultfalse.:checkpoint_interval— set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. Default10.:feature_subset_strategy— The number of features to consider for splits at each tree node. Supported options: 'auto' (choose automatically for task: If numTrees == 1, set to 'all'. If numTrees > 1 (forest), set to 'sqrt' for classification and to 'onethird' for regression), 'all' (use all features), 'onethird' (use 1/3 of the features), 'sqrt' (use sqrt(number of features)), 'log2' (use log2(number of features)), 'n' (when n is in the range (0, 1.0], use n * number of features. When n is in the range (1, number of features), use n features). default = 'auto'. Default"auto".:features_col— features column name. Default"features".:impurity— Criterion used for information gain calculation (case-insensitive). Supported options: variance. Default"variance".:label_col— label column name. Default"label".:leaf_col— Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. Default"".:max_bins— Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature. Default32.:max_depth— Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Must be in range [0, 30]. Default5.:max_memory_in_mb— Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. Default256.:min_info_gain— Minimum information gain for a split to be considered at a tree node. Default0.0.:min_instances_per_node— Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. Default1.:min_weight_fraction_per_node— Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in interval [0.0, 0.5). Default0.0.:num_trees— Number of trees to train (>= 1). Default20.:prediction_col— prediction column name. Default"prediction".:seed— random seed.:subsampling_rate— Fraction of the training data used for learning each decision tree, in range (0, 1]. Default1.0.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.