Latu.ML.Classification (latu_ml v0.2.0)

Copy Markdown View Source

Classifiers, and the models they fit.

Everything here predicts a label. The estimators take a features Vector column and a label column and answer with a model whose attributes live in this module's namespace — Latu.ML.Classification.LogisticRegressionModel and its kind.

Every constructor here is generated from PySpark 4.2.0's own param table, and every accessor module from the server's own attribute allowlist. Nothing in this file is hand-written but the words you are reading — see Latu.ML.operators/1 for the table it all comes from.

Summary

Functions

Decision tree learning algorithm for classification. It supports both binary and multiclass labels, as well as both continuous and categorical features.

Factorization Machines learning algorithm for classification.

Gradient-Boosted Trees (GBTs) learning algorithm for classification. It supports binary labels, as well as both continuous and categorical features.

This binary classifier optimizes the Hinge Loss using the OWLQN optimizer. Only supports L2 regularization currently.

Logistic regression. This class supports multinomial logistic (softmax) and binomial logistic regression.

Classifier trainer based on the Multilayer Perceptron. Each layer has sigmoid activation function, output layer has softmax. Number of inputs has to be equal to the size of feature vectors. Number of outputs has to be equal to the total number of labels.

Naive Bayes Classifiers. It supports both Multinomial and Bernoulli NB. Multinomial NB can handle finitely supported discrete data. For example, by converting documents into TF-IDF vectors, it can be used for document classification. By making every vector a binary (0/1) data, it can also be used as Bernoulli NB.

Random Forest learning algorithm for classification. It supports both binary and multiclass labels, as well as both continuous and categorical features.

Functions

decision_tree_classifier(opts \\ [])

@spec decision_tree_classifier(keyword()) :: Latu.ML.Estimator.t()

Decision tree learning algorithm for classification. It supports both binary and multiclass labels, as well as both continuous and categorical features.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.DecisionTreeClassificationModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :cache_node_ids — If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. Default false.
  • :checkpoint_interval — set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. Default 10.
  • :features_col — features column name. Default "features".
  • :impurity — Criterion used for information gain calculation (case-insensitive). Supported options: entropy, gini. Default "gini".
  • :label_col — label column name. Default "label".
  • :leaf_col — Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. Default "".
  • :max_bins — Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature. Default 32.
  • :max_depth — Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Must be in range [0, 30]. Default 5.
  • :max_memory_in_mb — Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. Default 256.
  • :min_info_gain — Minimum information gain for a split to be considered at a tree node. Default 0.0.
  • :min_instances_per_node — Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. Default 1.
  • :min_weight_fraction_per_node — Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in interval [0.0, 0.5). Default 0.0.
  • :prediction_col — prediction column name. Default "prediction".
  • :probability_col — Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default "probability".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :seed — random seed.
  • :thresholds — Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.
  • :weight_col — weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

fm_classifier(opts \\ [])

@spec fm_classifier(keyword()) :: Latu.ML.Estimator.t()

Factorization Machines learning algorithm for classification.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.FMClassificationModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :factor_size — Dimensionality of the factor vectors, which are used to get pairwise interactions between variables. Default 8.
  • :features_col — features column name. Default "features".
  • :fit_intercept — whether to fit an intercept term. Default true.
  • :fit_linear — whether to fit linear term (aka 1-way term) Default true.
  • :init_std — standard deviation of initial coefficients. Default 0.01.
  • :label_col — label column name. Default "label".
  • :max_iter — max number of iterations (>= 0). Default 100.
  • :mini_batch_fraction — fraction of the input data set that should be used for one iteration of gradient descent. Default 1.0.
  • :prediction_col — prediction column name. Default "prediction".
  • :probability_col — Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default "probability".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :reg_param — regularization parameter (>= 0). Default 0.0.
  • :seed — random seed.
  • :solver — The solver algorithm for optimization. Supported options: gd, adamW. (Default adamW) Default "adamW".
  • :step_size — Step size to be used for each iteration of optimization (>= 0). Default 1.0.
  • :thresholds — Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.
  • :tol — the convergence tolerance for iterative algorithms (>= 0). Default 1.0e-6.
  • :weight_col — weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

gbt_classifier(opts \\ [])

@spec gbt_classifier(keyword()) :: Latu.ML.Estimator.t()

Gradient-Boosted Trees (GBTs) learning algorithm for classification. It supports binary labels, as well as both continuous and categorical features.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.GBTClassificationModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :cache_node_ids — If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. Default false.
  • :checkpoint_interval — set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. Default 10.
  • :feature_subset_strategy — The number of features to consider for splits at each tree node. Supported options: 'auto' (choose automatically for task: If numTrees == 1, set to 'all'. If numTrees > 1 (forest), set to 'sqrt' for classification and to 'onethird' for regression), 'all' (use all features), 'onethird' (use 1/3 of the features), 'sqrt' (use sqrt(number of features)), 'log2' (use log2(number of features)), 'n' (when n is in the range (0, 1.0], use n * number of features. When n is in the range (1, number of features), use n features). default = 'auto'. Default "all".
  • :features_col — features column name. Default "features".
  • :impurity — Criterion used for information gain calculation (case-insensitive). Supported options: variance. Default "variance".
  • :label_col — label column name. Default "label".
  • :leaf_col — Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. Default "".
  • :loss_type — Loss function which GBT tries to minimize (case-insensitive). Supported options: logistic. Default "logistic".
  • :max_bins — Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature. Default 32.
  • :max_depth — Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Must be in range [0, 30]. Default 5.
  • :max_iter — max number of iterations (>= 0). Default 20.
  • :max_memory_in_mb — Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. Default 256.
  • :min_info_gain — Minimum information gain for a split to be considered at a tree node. Default 0.0.
  • :min_instances_per_node — Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. Default 1.
  • :min_weight_fraction_per_node — Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in interval [0.0, 0.5). Default 0.0.
  • :prediction_col — prediction column name. Default "prediction".
  • :probability_col — Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default "probability".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :seed — random seed.
  • :step_size — Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of each estimator. Default 0.1.
  • :subsampling_rate — Fraction of the training data used for learning each decision tree, in range (0, 1]. Default 1.0.
  • :thresholds — Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.
  • :validation_indicator_col — name of the column that indicates whether each row is for training or for validation. False indicates training; true indicates validation.
  • :validation_tol — Threshold for stopping early when fit with validation is used. If the error rate on the validation input changes by less than the validationTol, then learning will stop early (before maxIter). This parameter is ignored when fit without validation is used. Default 0.01.
  • :weight_col — weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

linear_svc(opts \\ [])

@spec linear_svc(keyword()) :: Latu.ML.Estimator.t()

This binary classifier optimizes the Hinge Loss using the OWLQN optimizer. Only supports L2 regularization currently.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.LinearSVCModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :aggregation_depth — suggested depth for treeAggregate (>= 2). Default 2.
  • :features_col — features column name. Default "features".
  • :fit_intercept — whether to fit an intercept term. Default true.
  • :label_col — label column name. Default "label".
  • :max_block_size_in_mb — maximum memory in MB for stacking input data into blocks. Data is stacked within partitions. If more than remaining data size in a partition then it is adjusted to the data size. Default 0.0 represents choosing optimal value, depends on specific algorithm. Must be >= 0. Default 0.0.
  • :max_iter — max number of iterations (>= 0). Default 100.
  • :prediction_col — prediction column name. Default "prediction".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :reg_param — regularization parameter (>= 0). Default 0.0.
  • :standardization — whether to standardize the training features before fitting the model. Default true.
  • :threshold — The threshold in binary classification applied to the linear model prediction. This threshold can be any real number, where Inf will make all predictions 0.0 and -Inf will make all predictions 1.0. Default 0.0.
  • :tol — the convergence tolerance for iterative algorithms (>= 0). Default 1.0e-6.
  • :weight_col — weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

logistic_regression(opts \\ [])

@spec logistic_regression(keyword()) :: Latu.ML.Estimator.t()

Logistic regression. This class supports multinomial logistic (softmax) and binomial logistic regression.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.LogisticRegressionModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :aggregation_depth — suggested depth for treeAggregate (>= 2). Default 2.
  • :elastic_net_param — the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty. Default 0.0.
  • :family — The name of family which is a description of the label distribution to be used in the model. Supported options: auto, binomial, multinomial. Default "auto".
  • :features_col — features column name. Default "features".
  • :fit_intercept — whether to fit an intercept term. Default true.
  • :label_col — label column name. Default "label".
  • :lower_bounds_on_coefficients — The lower bounds on coefficients if fitting under bound constrained optimization. The bound matrix must be compatible with the shape (1, number of features) for binomial regression, or (number of classes, number of features) for multinomial regression.
  • :lower_bounds_on_intercepts — The lower bounds on intercepts if fitting under bound constrained optimization. The bounds vector size must beequal with 1 for binomial regression, or the number oflasses for multinomial regression.
  • :max_block_size_in_mb — maximum memory in MB for stacking input data into blocks. Data is stacked within partitions. If more than remaining data size in a partition then it is adjusted to the data size. Default 0.0 represents choosing optimal value, depends on specific algorithm. Must be >= 0. Default 0.0.
  • :max_iter — max number of iterations (>= 0). Default 100.
  • :prediction_col — prediction column name. Default "prediction".
  • :probability_col — Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default "probability".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :reg_param — regularization parameter (>= 0). Default 0.0.
  • :standardization — whether to standardize the training features before fitting the model. Default true.
  • :threshold — Threshold in binary classification prediction, in range [0, 1]. If threshold and thresholds are both set, they must match.e.g. if threshold is p, then thresholds must be equal to [1-p, p]. Default 0.5.
  • :thresholds — Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.
  • :tol — the convergence tolerance for iterative algorithms (>= 0). Default 1.0e-6.
  • :upper_bounds_on_coefficients — The upper bounds on coefficients if fitting under bound constrained optimization. The bound matrix must be compatible with the shape (1, number of features) for binomial regression, or (number of classes, number of features) for multinomial regression.
  • :upper_bounds_on_intercepts — The upper bounds on intercepts if fitting under bound constrained optimization. The bound vector size must be equal with 1 for binomial regression, or the number of classes for multinomial regression.
  • :weight_col — weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

multilayer_perceptron_classifier(opts \\ [])

@spec multilayer_perceptron_classifier(keyword()) :: Latu.ML.Estimator.t()

Classifier trainer based on the Multilayer Perceptron. Each layer has sigmoid activation function, output layer has softmax. Number of inputs has to be equal to the size of feature vectors. Number of outputs has to be equal to the total number of labels.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.MultilayerPerceptronClassificationModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :block_size — block size for stacking input data in matrices. Data is stacked within partitions. If block size is more than remaining data in a partition then it is adjusted to the size of this data. Default 128.
  • :features_col — features column name. Default "features".
  • :initial_weights — The initial weights of the model.
  • :label_col — label column name. Default "label".
  • :layers — Sizes of layers from input layer to output layer E.g., Array(780, 100, 10) means 780 inputs, one hidden layer with 100 neurons and output layer of 10 neurons.
  • :max_iter — max number of iterations (>= 0). Default 100.
  • :prediction_col — prediction column name. Default "prediction".
  • :probability_col — Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default "probability".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :seed — random seed.
  • :solver — The solver algorithm for optimization. Supported options: l-bfgs, gd. Default "l-bfgs".
  • :step_size — Step size to be used for each iteration of optimization (>= 0). Default 0.03.
  • :thresholds — Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.
  • :tol — the convergence tolerance for iterative algorithms (>= 0). Default 1.0e-6.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

naive_bayes(opts \\ [])

@spec naive_bayes(keyword()) :: Latu.ML.Estimator.t()

Naive Bayes Classifiers. It supports both Multinomial and Bernoulli NB. Multinomial NB can handle finitely supported discrete data. For example, by converting documents into TF-IDF vectors, it can be used for document classification. By making every vector a binary (0/1) data, it can also be used as Bernoulli NB.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.NaiveBayesModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :features_col — features column name. Default "features".
  • :label_col — label column name. Default "label".
  • :model_type — The model type which is a string (case-sensitive). Supported options: multinomial (default), bernoulli and gaussian. Default "multinomial".
  • :prediction_col — prediction column name. Default "prediction".
  • :probability_col — Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default "probability".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :smoothing — The smoothing parameter, should be >= 0, default is 1.0. Default 1.0.
  • :thresholds — Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.
  • :weight_col — weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

random_forest_classifier(opts \\ [])

@spec random_forest_classifier(keyword()) :: Latu.ML.Estimator.t()

Random Forest learning algorithm for classification. It supports both binary and multiclass labels, as well as both continuous and categorical features.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.Classification.RandomForestClassificationModel.

Status :probeddev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :bootstrap — Whether bootstrap samples are used when building trees. Default true.
  • :cache_node_ids — If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. Default false.
  • :checkpoint_interval — set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. Default 10.
  • :feature_subset_strategy — The number of features to consider for splits at each tree node. Supported options: 'auto' (choose automatically for task: If numTrees == 1, set to 'all'. If numTrees > 1 (forest), set to 'sqrt' for classification and to 'onethird' for regression), 'all' (use all features), 'onethird' (use 1/3 of the features), 'sqrt' (use sqrt(number of features)), 'log2' (use log2(number of features)), 'n' (when n is in the range (0, 1.0], use n * number of features. When n is in the range (1, number of features), use n features). default = 'auto'. Default "auto".
  • :features_col — features column name. Default "features".
  • :impurity — Criterion used for information gain calculation (case-insensitive). Supported options: entropy, gini. Default "gini".
  • :label_col — label column name. Default "label".
  • :leaf_col — Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. Default "".
  • :max_bins — Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature. Default 32.
  • :max_depth — Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. Must be in range [0, 30]. Default 5.
  • :max_memory_in_mb — Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. Default 256.
  • :min_info_gain — Minimum information gain for a split to be considered at a tree node. Default 0.0.
  • :min_instances_per_node — Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1. Default 1.
  • :min_weight_fraction_per_node — Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in interval [0.0, 0.5). Default 0.0.
  • :num_trees — Number of trees to train (>= 1). Default 20.
  • :prediction_col — prediction column name. Default "prediction".
  • :probability_col — Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default "probability".
  • :raw_prediction_col — raw prediction (a.k.a. confidence) column name. Default "rawPrediction".
  • :seed — random seed.
  • :subsampling_rate — Fraction of the training data used for learning each decision tree, in range (0, 1]. Default 1.0.
  • :thresholds — Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold.
  • :weight_col — weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.