Feature transformers, and the models the fitted ones produce.
The half of MLlib that shapes data rather than predicting from it: assembling a features
Vector, scaling it, indexing strings, hashing text. Most are transformers and reach no
server when applied; the ones that must learn something first — a vocabulary, a mean, a set
of quantiles — are estimators, and Latu.ML.fit/2 is what turns those into models.
Every constructor here is generated from PySpark 4.2.0's own param table, and every
accessor module from the server's own attribute allowlist — see Latu.ML.operators/1 for
the table it comes from. The two functions below are the exception, and they are here for
the reason PySpark puts them on StopWordsRemover: they answer questions about that
transformer's defaults, which only a running server can answer.
Summary
Functions
Binarize a column of continuous features given a threshold. Since 3.0.0, Binarize can map
multiple columns at once by setting the inputCols parameter. Note that when both the
inputCol and inputCols parameters are set, an Exception will be thrown. The threshold
parameter is used for single column usage, and thresholds is for multiple columns.
LSH class for Euclidean distance metrics. The input is dense or sparse vectors, each of which represents a point in the Euclidean distance space. The output will be vectors of configurable dimension. Hash values in the same dimension are calculated by the same hash function.
Maps a column of continuous features to a column of feature buckets. Since 3.0.0,
Bucketizer can map multiple columns at once by setting the inputCols parameter. Note
that when both the inputCol and inputCols parameters are set, an Exception will be
thrown. The splits parameter is only used for single column usage, and splitsArray is
for multiple columns.
Chi-Squared feature selection, which selects categorical features to use for predicting a
categorical label. The selector supports different selection methods: numTopFeatures,
percentile, fpr, fdr, fwe.
Extracts a vocabulary from document collections and generates a CountVectorizerModel.
A feature transformer that takes the 1D discrete cosine transform of a real vector. No zero padding is performed on the input vector. It returns a real vector of the same length representing the DCT. The return vector is scaled such that the transform matrix is unitary (aka scaled DCT-II).
The locale a StopWordsRemover uses when you do not name one.
See default_locale/1. Raises instead of returning an error.
Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector. In other words, it scales each column of the dataset by a scalar multiplier.
Feature hashing projects a set of categorical or numerical features into a feature vector of specified dimension (typically substantially smaller than that of the original feature space). This is done using the hashing trick (https://en.wikipedia.org/wiki/Feature_hashing) to map features to indices in the feature vector.
Maps a sequence of terms to their term frequencies using the hashing trick. Currently we use Austin Appleby's MurmurHash 3 algorithm (MurmurHash3_x86_32) to calculate the hash code value for the term object. Since a simple modulo is used to transform the hash function to a column index, it is advisable to use a power of two as the numFeatures parameter; otherwise the features will not be mapped evenly to the columns.
Compute the Inverse Document Frequency (IDF) given a collection of documents.
Imputation estimator for completing missing values, using the mean, median or mode of the columns in which the missing values are located. The input columns should be of numeric type. Currently Imputer does not support categorical features and possibly creates incorrect values for a categorical feature.
A pyspark.ml.base.Transformer that maps a column of indices back to a new column of
corresponding string values. The index-string mapping is either from the ML attributes of
the input column, or from user-supplied labels (which take precedence over ML attributes).
Implements the feature interaction transform. This transformer takes in Double and Vector type columns and outputs a flattened vector of their feature interactions. To handle interaction, we first one-hot encode any nominal features. Then, a vector of the feature cross-products is produced.
Spark's own stop-word list for a language.
See load_default_stop_words/2. Raises instead of returning an error.
Rescale each feature individually to range [-1, 1] by dividing through the largest maximum absolute value in each feature. It does not shift/center the data, and thus does not destroy any sparsity.
LSH class for Jaccard distance. The input can be dense or sparse vectors, but it is more
efficient if it is sparse. For example, Vectors.sparse(10, [(2, 1.0), (3, 1.0), (5, 1.0)])
means there are 10 elements in the space. This set contains elements 2, 3, and 5. Also, any
input vector must have at least 1 non-zero index, and all non-zero values are treated as
binary "1" values.
Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling. The rescaled value for feature E is calculated as,
A feature transformer that converts the input array of strings into an array of n-grams. Null values in the input array are ignored. It returns an array of n-grams where each n-gram is represented by a space-separated string of words. When the input is empty, an empty array is returned. When the input array length is less than n (number of elements per n-gram), no n-grams are returned.
Normalize a vector to have unit norm using the given p-norm.
A one-hot encoder that maps a column of category indices to a column of binary vectors, with
at most a single one-value per row that indicates the input category index. For example with
5 categories, an input value of 2.0 would map to an output vector of [0.0, 0.0, 1.0, 0.0].
The last category is not included by default (configurable via dropLast), because it makes
the vector entries sum up to one, and hence linearly dependent. So an input value of 4.0
maps to [0.0, 0.0, 0.0, 0.0].
PCA trains a model to project vectors to a lower dimensional space of the top k principal
components.
Perform feature expansion in a polynomial space. As said in wikipedia of Polynomial
Expansion, "In mathematics, an expansion
of a product of sums expresses it as a sum of products by using the fact that multiplication
distributes over addition". Take a 2-variable feature vector as an example: (x, y), if we
want to expand it with degree 2, then we get (x, x * x, y, x * y, y * y).
QuantileDiscretizer takes a column with continuous features and outputs a column with
binned categorical features. The number of bins can be set using the numBuckets parameter.
It is possible that the number of buckets used will be less than this value, for example, if
there are too few distinct values of the input to create enough distinct quantiles. Since
3.0.0, QuantileDiscretizer can map multiple columns at once by setting the inputCols
parameter. If both of the inputCol and inputCols parameters are set, an Exception will
be thrown. To specify the number of buckets for each column, the numBucketsArray parameter
can be set, or if the number of buckets should be the same across columns, numBuckets can
be set as a convenience.
Implements the transforms required for fitting a dataset against an R model formula. Currently we support a limited subset of the R operators, including '~', '.', ':', '+', '-', '*', and '^'.
A regex based tokenizer that extracts tokens either by using the provided regex pattern (in Java dialect) to split the text (default) or repeatedly matching the regex (if gaps is false). Optional parameters also allow filtering tokens using a minimal length. It returns an array of strings that can be empty.
RobustScaler removes the median and scales the data according to the quantile range. The quantile range is by default IQR (Interquartile Range, quantile range between the 1st quartile = 25th quantile and the 3rd quartile = 75th quantile) but can be configured. Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Median and quantile range are then stored to be used on later data using the transform method. Note that NaN values are ignored in the computation of medians and ranges.
Implements the transforms which are defined by SQL statement. Currently we only support SQL
syntax like SELECT ... FROM __THIS__ where __THIS__ represents the underlying table of
the input dataset.
Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.
A feature transformer that filters out stop words from input. Since 3.0.0,
StopWordsRemover can filter out multiple columns at once by setting the inputCols
parameter. Note that when both the inputCol and inputCols parameters are set, an
Exception will be thrown.
A label indexer that maps a string column of labels to an ML column of label indices. If the
input column is numeric, we cast it to string and index the string values. The indices are
in [0, numLabels). By default, this is ordered by label frequencies so the most frequent
label gets index 0. The ordering behavior is controlled by setting stringOrderType. Its
default value is 'frequencyDesc'.
Target Encoding maps a column of categorical indices into a numerical feature derived from the target.
A tokenizer that converts the input string to lowercase and then splits it by white spaces.
UnivariateFeatureSelector Feature selector based on univariate statistical tests against
labels. Currently, Spark supports three Univariate Feature Selectors: chi-squared, ANOVA
F-test and F-value. User can choose Univariate Feature Selector by setting featureType and
labelType, and Spark will pick the score function based on the specified featureType and
labelType.
Feature selector that removes all low-variance features. Features with a (sample) variance not greater than the threshold will be removed. The default is to keep all features with non-zero variance, i.e. remove the features that have the same value in all samples.
A feature transformer that merges multiple columns into a vector column.
Class for indexing categorical feature columns in a dataset of Vector.
A feature transformer that adds size information to the metadata of a vector column. VectorAssembler needs size information for its input columns and cannot be used on streaming dataframes without this metadata.
This class takes a feature vector and outputs a new feature vector with a subarray of the original features.
Word2Vec trains a model of Map(String, Vector), i.e. transforms a word into a code for
further natural language processing or machine learning process.
Functions
@spec binarizer(keyword()) :: Latu.ML.Transformer.t()
Binarize a column of continuous features given a threshold. Since 3.0.0, Binarize can map
multiple columns at once by setting the inputCols parameter. Note that when both the
inputCol and inputCols parameters are set, an Exception will be thrown. The threshold
parameter is used for single column usage, and thresholds is for multiple columns.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:input_cols— input column names.:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.:threshold— Param for threshold used to binarize continuous features. The features greater than the threshold will be binarized to 1.0. The features equal to or less than the threshold will be binarized to 0.0. Default0.0.:thresholds— Param for array of threshold used to binarize continuous features. This is for multiple columns input. If transforming multiple columns and thresholds is not set, but threshold is set, then threshold will be applied across all columns.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec bucketed_random_projection_lsh(keyword()) :: Latu.ML.Estimator.t()
LSH class for Euclidean distance metrics. The input is dense or sparse vectors, each of which represents a point in the Euclidean distance space. The output will be vectors of configurable dimension. Hash values in the same dimension are calculated by the same hash function.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.BucketedRandomProjectionLSHModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:bucket_length— the length of each hash bucket, a larger bucket lowers the false negative rate.:input_col— input column name.:num_hash_tables— number of hash tables, where increasing number of hash tables lowers the false negative rate, and decreasing it improves the running performance. Default1.:output_col— output column name. Defaults to this operator's uid plus__output.:seed— random seed.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec bucketizer(keyword()) :: Latu.ML.Transformer.t()
Maps a column of continuous features to a column of feature buckets. Since 3.0.0,
Bucketizer can map multiple columns at once by setting the inputCols parameter. Note
that when both the inputCol and inputCols parameters are set, an Exception will be
thrown. The splits parameter is only used for single column usage, and splitsArray is
for multiple columns.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:handle_invalid— how to handle invalid entries containing NaN values. Values outside the splits will always be treated as errors. Options are 'skip' (filter out rows with invalid values), 'error' (throw an error), or 'keep' (keep invalid values in a special additional bucket). Note that in the multiple column case, the invalid handling is applied to all columns. That said for 'error' it will throw an error if any invalids are found in any column, for 'skip' it will skip rows with any invalids in any columns, etc. Default"error".:input_col— input column name.:input_cols— input column names.:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.:splits— Split points for mapping continuous features into buckets. With n+1 splits, there are n buckets. A bucket defined by splits x,y holds values in the range [x,y) except the last bucket, which also includes y. The splits should be of length >= 3 and strictly increasing. Values at -inf, inf must be explicitly provided to cover all Double values; otherwise, values outside the splits specified will be treated as errors.:splits_array— The array of split points for mapping continuous features into buckets for multiple columns. For each input column, with n+1 splits, there are n buckets. A bucket defined by splits x,y holds values in the range [x,y) except the last bucket, which also includes y. The splits should be of length >= 3 and strictly increasing. Values at -inf, inf must be explicitly provided to cover all Double values; otherwise, values outside the splits specified will be treated as errors.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec chi_sq_selector(keyword()) :: Latu.ML.Estimator.t()
Chi-Squared feature selection, which selects categorical features to use for predicting a
categorical label. The selector supports different selection methods: numTopFeatures,
percentile, fpr, fdr, fwe.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.ChiSqSelectorModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:fdr— The upper bound of the expected false discovery rate. Default0.05.:features_col— features column name. Default"features".:fpr— The highest p-value for features to be kept. Default0.05.:fwe— The upper bound of the expected family-wise error rate. Default0.05.:label_col— label column name. Default"label".:num_top_features— Number of features that selector will select, ordered by ascending p-value. If the number of features is < numTopFeatures, then this will select all features. Default50.:output_col— output column name. Defaults to this operator's uid plus__output.:percentile— Percentile of features that selector will select, ordered by ascending p-value. Default0.1.:selector_type— The selector type. Supported options: numTopFeatures (default), percentile, fpr, fdr, fwe. Default"numTopFeatures".
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec count_vectorizer(keyword()) :: Latu.ML.Estimator.t()
Extracts a vocabulary from document collections and generates a CountVectorizerModel.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.CountVectorizerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:binary— Binary toggle to control the output vector values. If True, all nonzero counts (after minTF filter applied) are set to 1. This is useful for discrete probabilistic models that model binary events rather than integer counts. Default False. Defaultfalse.:input_col— input column name.:max_df— Specifies the maximum number of different documents a term could appear in to be included in the vocabulary. A term that appears more than the threshold will be ignored. If this is an integer >= 1, this specifies the maximum number of documents the term could appear in; if this is a double in [0,1), then this specifies the maximum fraction of documents the term could appear in. Default (2^63) - 1. Default9.223372036854776e18.:min_df— Specifies the minimum number of different documents a term must appear in to be included in the vocabulary. If this is an integer >= 1, this specifies the number of documents the term must appear in; if this is a double in [0,1), then this specifies the fraction of documents. Default 1.0. Default1.0.:min_tf— Filter to ignore rare words in a document. For each document, terms with frequency/count less than the given threshold are ignored. If this is an integer >= 1, then this specifies a count (of times the term must appear in the document); if this is a double in [0,1), then this specifies a fraction (out of the document's token count). Note that the parameter is only used in transform of CountVectorizerModel and does not affect fitting. Default 1.0. Default1.0.:output_col— output column name. Defaults to this operator's uid plus__output.:vocab_size— max size of the vocabulary. Default 1 << 18. Default262144.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec dct(keyword()) :: Latu.ML.Transformer.t()
A feature transformer that takes the 1D discrete cosine transform of a real vector. No zero padding is performed on the input vector. It returns a real vector of the same length representing the DCT. The return vector is scaled such that the transform matrix is unitary (aka scaled DCT-II).
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:inverse— Set transformer to perform inverse DCT, default False. Defaultfalse.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec default_locale(Latu.Session.t()) :: {:ok, String.t()} | {:error, Latu.Error.t()}
The locale a StopWordsRemover uses when you do not name one.
An action, and the reason stop_words_remover/1's locale param has no default this
package can print: it is the JVM's own default locale, or en_US where that one has no
stop-word list. Only the server knows which.
{:ok, locale} = Latu.ML.Feature.default_locale(session)@spec default_locale!(Latu.Session.t()) :: String.t()
See default_locale/1. Raises instead of returning an error.
@spec elementwise_product(keyword()) :: Latu.ML.Transformer.t()
Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector. In other words, it scales each column of the dataset by a scalar multiplier.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:output_col— output column name. Defaults to this operator's uid plus__output.:scaling_vec— Vector for hadamard product.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec feature_hasher(keyword()) :: Latu.ML.Transformer.t()
Feature hashing projects a set of categorical or numerical features into a feature vector of specified dimension (typically substantially smaller than that of the original feature space). This is done using the hashing trick (https://en.wikipedia.org/wiki/Feature_hashing) to map features to indices in the feature vector.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:categorical_cols— numeric columns to treat as categorical.:input_cols— input column names.:num_features— Number of features. Should be greater than 0. Default262144.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec hashing_tf(keyword()) :: Latu.ML.Transformer.t()
Maps a sequence of terms to their term frequencies using the hashing trick. Currently we use Austin Appleby's MurmurHash 3 algorithm (MurmurHash3_x86_32) to calculate the hash code value for the term object. Since a simple modulo is used to transform the hash function to a column index, it is advisable to use a power of two as the numFeatures parameter; otherwise the features will not be mapped evenly to the columns.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:binary— If True, all non zero counts are set to 1. This is useful for discrete probabilistic models that model binary events rather than integer counts. Default False. Defaultfalse.:input_col— input column name.:num_features— Number of features. Should be greater than 0. Default262144.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec idf(keyword()) :: Latu.ML.Estimator.t()
Compute the Inverse Document Frequency (IDF) given a collection of documents.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.IDFModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:min_doc_freq— minimum number of documents in which a term should appear for filtering. Default0.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec imputer(keyword()) :: Latu.ML.Estimator.t()
Imputation estimator for completing missing values, using the mean, median or mode of the columns in which the missing values are located. The input columns should be of numeric type. Currently Imputer does not support categorical features and possibly creates incorrect values for a categorical feature.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.ImputerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:input_cols— input column names.:missing_value— The placeholder for the missing values. All occurrences of missingValue will be imputed. DefaultNaN.:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.:relative_error— the relative target precision for the approximate quantile algorithm. Must be in the range [0, 1]. Default0.001.:strategy— strategy for imputation. If mean, then replace missing values using the mean value of the feature. If median, then replace missing values using the median value of the feature. If mode, then replace missing using the most frequent value of the feature. Default"mean".
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec index_to_string(keyword()) :: Latu.ML.Transformer.t()
A pyspark.ml.base.Transformer that maps a column of indices back to a new column of
corresponding string values. The index-string mapping is either from the ML attributes of
the input column, or from user-supplied labels (which take precedence over ML attributes).
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:labels— Optional array of labels specifying index-string mapping. If not provided or if empty, then metadata from inputCol is used instead.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec interaction(keyword()) :: Latu.ML.Transformer.t()
Implements the feature interaction transform. This transformer takes in Double and Vector type columns and outputs a flattened vector of their feature interactions. To handle interaction, we first one-hot encode any nominal features. Then, a vector of the feature cross-products is produced.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_cols— input column names.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec load_default_stop_words(Latu.Session.t(), String.t()) :: {:ok, [String.t()]} | {:error, Latu.Error.t()}
Spark's own stop-word list for a language.
An action, and the other half of what stop_words_remover/1 cannot default: the list ships
with Spark, and danish, dutch, english, finnish, french, german, hungarian,
italian, norwegian, portuguese, russian, spanish, swedish and turkish are the
languages it has.
{:ok, words} = Latu.ML.Feature.load_default_stop_words(session, "german")
remover = Latu.ML.Feature.stop_words_remover(input_col: :words, stop_words: words)@spec load_default_stop_words!(Latu.Session.t(), String.t()) :: [String.t()]
See load_default_stop_words/2. Raises instead of returning an error.
@spec max_abs_scaler(keyword()) :: Latu.ML.Estimator.t()
Rescale each feature individually to range [-1, 1] by dividing through the largest maximum absolute value in each feature. It does not shift/center the data, and thus does not destroy any sparsity.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.MaxAbsScalerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec min_hash_lsh(keyword()) :: Latu.ML.Estimator.t()
LSH class for Jaccard distance. The input can be dense or sparse vectors, but it is more
efficient if it is sparse. For example, Vectors.sparse(10, [(2, 1.0), (3, 1.0), (5, 1.0)])
means there are 10 elements in the space. This set contains elements 2, 3, and 5. Also, any
input vector must have at least 1 non-zero index, and all non-zero values are treated as
binary "1" values.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.MinHashLSHModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:num_hash_tables— number of hash tables, where increasing number of hash tables lowers the false negative rate, and decreasing it improves the running performance. Default1.:output_col— output column name. Defaults to this operator's uid plus__output.:seed— random seed.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec min_max_scaler(keyword()) :: Latu.ML.Estimator.t()
Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling. The rescaled value for feature E is calculated as,
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.MinMaxScalerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:max— Upper bound of the output feature range. Default1.0.:min— Lower bound of the output feature range. Default0.0.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec ngram(keyword()) :: Latu.ML.Transformer.t()
A feature transformer that converts the input array of strings into an array of n-grams. Null values in the input array are ignored. It returns an array of n-grams where each n-gram is represented by a space-separated string of words. When the input is empty, an empty array is returned. When the input array length is less than n (number of elements per n-gram), no n-grams are returned.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:n— number of elements per n-gram (>=1) Default2.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec normalizer(keyword()) :: Latu.ML.Transformer.t()
Normalize a vector to have unit norm using the given p-norm.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:output_col— output column name. Defaults to this operator's uid plus__output.:p— the p norm value. Default2.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec one_hot_encoder(keyword()) :: Latu.ML.Estimator.t()
A one-hot encoder that maps a column of category indices to a column of binary vectors, with
at most a single one-value per row that indicates the input category index. For example with
5 categories, an input value of 2.0 would map to an output vector of [0.0, 0.0, 1.0, 0.0].
The last category is not included by default (configurable via dropLast), because it makes
the vector entries sum up to one, and hence linearly dependent. So an input value of 4.0
maps to [0.0, 0.0, 0.0, 0.0].
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.OneHotEncoderModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:drop_last— whether to drop the last category. Defaulttrue.:handle_invalid— How to handle invalid data during transform(). Options are 'keep' (invalid data presented as an extra categorical feature) or error (throw an error). Note that this Param is only used during transform; during fitting, invalid data will result in an error. Default"error".:input_col— input column name.:input_cols— input column names.:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec pca(keyword()) :: Latu.ML.Estimator.t()
PCA trains a model to project vectors to a lower dimensional space of the top k principal
components.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.PCAModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:k— the number of principal components.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec polynomial_expansion(keyword()) :: Latu.ML.Transformer.t()
Perform feature expansion in a polynomial space. As said in wikipedia of Polynomial
Expansion, "In mathematics, an expansion
of a product of sums expresses it as a sum of products by using the fact that multiplication
distributes over addition". Take a 2-variable feature vector as an example: (x, y), if we
want to expand it with degree 2, then we get (x, x * x, y, x * y, y * y).
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:degree— the polynomial degree to expand (>= 1) Default2.:input_col— input column name.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec quantile_discretizer(keyword()) :: Latu.ML.Estimator.t()
QuantileDiscretizer takes a column with continuous features and outputs a column with
binned categorical features. The number of bins can be set using the numBuckets parameter.
It is possible that the number of buckets used will be less than this value, for example, if
there are too few distinct values of the input to create enough distinct quantiles. Since
3.0.0, QuantileDiscretizer can map multiple columns at once by setting the inputCols
parameter. If both of the inputCol and inputCols parameters are set, an Exception will
be thrown. To specify the number of buckets for each column, the numBucketsArray parameter
can be set, or if the number of buckets should be the same across columns, numBuckets can
be set as a convenience.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on the model's own module.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:handle_invalid— how to handle invalid entries. Options are skip (filter out rows with invalid values), error (throw an error), or keep (keep invalid values in a special additional bucket). Note that in the multiple columns case, the invalid handling is applied to all columns. That said for 'error' it will throw an error if any invalids are found in any columns, for 'skip' it will skip rows with any invalids in any columns, etc. Default"error".:input_col— input column name.:input_cols— input column names.:num_buckets— Maximum number of buckets (quantiles, or categories) into which data points are grouped. Must be >= 2. Default2.:num_buckets_array— Array of number of buckets (quantiles, or categories) into which data points are grouped. This is for multiple columns input. If transforming multiple columns and numBucketsArray is not set, but numBuckets is set, then numBuckets will be applied across all columns.:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.:relative_error— the relative target precision for the approximate quantile algorithm. Must be in the range [0, 1]. Default0.001.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec r_formula(keyword()) :: Latu.ML.Estimator.t()
Implements the transforms required for fitting a dataset against an R model formula. Currently we support a limited subset of the R operators, including '~', '.', ':', '+', '-', '*', and '^'.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.RFormulaModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:features_col— features column name. Default"features".:force_index_label— Force to index label whether it is numeric or string. Defaultfalse.:formula— R model formula.:handle_invalid— how to handle invalid entries. Options are 'skip' (filter out rows with invalid values), 'error' (throw an error), or 'keep' (put invalid data in a special additional bucket, at index numLabels). Default"error".:label_col— label column name. Default"label".:string_indexer_order_type— How to order categories of a string feature column used by StringIndexer. The last category after ordering is dropped when encoding strings. Supported options: frequencyDesc, frequencyAsc, alphabetDesc, alphabetAsc. The default value is frequencyDesc. When the ordering is set to alphabetDesc, RFormula drops the same category as R when encoding strings. Default"frequencyDesc".
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec regex_tokenizer(keyword()) :: Latu.ML.Transformer.t()
A regex based tokenizer that extracts tokens either by using the provided regex pattern (in Java dialect) to split the text (default) or repeatedly matching the regex (if gaps is false). Optional parameters also allow filtering tokens using a minimal length. It returns an array of strings that can be empty.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:gaps— whether regex splits on gaps (True) or matches tokens (False) Defaulttrue.:input_col— input column name.:min_token_length— minimum token length (>= 0) Default1.:output_col— output column name. Defaults to this operator's uid plus__output.:pattern— regex pattern (Java dialect) used for tokenizing. Default"\\s+".:to_lowercase— whether to convert all characters to lowercase before tokenizing. Defaulttrue.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec robust_scaler(keyword()) :: Latu.ML.Estimator.t()
RobustScaler removes the median and scales the data according to the quantile range. The quantile range is by default IQR (Interquartile Range, quantile range between the 1st quartile = 25th quantile and the 3rd quartile = 75th quantile) but can be configured. Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Median and quantile range are then stored to be used on later data using the transform method. Note that NaN values are ignored in the computation of medians and ranges.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.RobustScalerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:lower— Lower quantile to calculate quantile range. Default0.25.:output_col— output column name. Defaults to this operator's uid plus__output.:relative_error— the relative target precision for the approximate quantile algorithm. Must be in the range [0, 1]. Default0.001.:upper— Upper quantile to calculate quantile range. Default0.75.:with_centering— Whether to center data with median. Defaultfalse.:with_scaling— Whether to scale the data to quantile range. Defaulttrue.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec sql_transformer(keyword()) :: Latu.ML.Transformer.t()
Implements the transforms which are defined by SQL statement. Currently we only support SQL
syntax like SELECT ... FROM __THIS__ where __THIS__ represents the underlying table of
the input dataset.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:statement— SQL statement.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec standard_scaler(keyword()) :: Latu.ML.Estimator.t()
Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.StandardScalerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:output_col— output column name. Defaults to this operator's uid plus__output.:with_mean— Center data with mean. Defaultfalse.:with_std— Scale to unit standard deviation. Defaulttrue.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec stop_words_remover(keyword()) :: Latu.ML.Transformer.t()
A feature transformer that filters out stop words from input. Since 3.0.0,
StopWordsRemover can filter out multiple columns at once by setting the inputCols
parameter. Note that when both the inputCol and inputCols parameters are set, an
Exception will be thrown.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:case_sensitive— whether to do a case sensitive comparison over the stop words. Defaultfalse.:input_col— input column name.:input_cols— input column names.:locale— locale of the input. ignored when case sensitive is true.:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.:stop_words— The words to be filtered out.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec string_indexer(keyword()) :: Latu.ML.Estimator.t()
A label indexer that maps a string column of labels to an ML column of label indices. If the
input column is numeric, we cast it to string and index the string values. The indices are
in [0, numLabels). By default, this is ordered by label frequencies so the most frequent
label gets index 0. The ordering behavior is controlled by setting stringOrderType. Its
default value is 'frequencyDesc'.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.StringIndexerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:handle_invalid— how to handle invalid data (unseen or NULL values) in features and label column of string type. Options are 'skip' (filter out rows with invalid data), error (throw an error), or 'keep' (put invalid data in a special additional bucket, at index numLabels). Default"error".:input_col— input column name.:input_cols— input column names.:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.:string_order_type— How to order labels of string column. The first label after ordering is assigned an index of 0. Supported options: frequencyDesc, frequencyAsc, alphabetDesc, alphabetAsc. Default is frequencyDesc. In case of equal frequency when under frequencyDesc/Asc, the strings are further sorted alphabetically. Default"frequencyDesc".
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec target_encoder(keyword()) :: Latu.ML.Estimator.t()
Target Encoding maps a column of categorical indices into a numerical feature derived from the target.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.TargetEncoderModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:handle_invalid— How to handle invalid data during transform(). Options are 'keep' (invalid data presented as an extra categorical feature) or error (throw an error). Default"error".:input_col— input column name.:input_cols— input column names.:label_col— label column name. Default"label".:output_col— output column name. Defaults to this operator's uid plus__output.:output_cols— output column names.:smoothing— value to smooth in-category averages with overall averages. Default0.0.:target_type— whether the label is 'binary' or 'continuous'. Default"binary".
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec tokenizer(keyword()) :: Latu.ML.Transformer.t()
A tokenizer that converts the input string to lowercase and then splits it by white spaces.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec univariate_feature_selector(keyword()) :: Latu.ML.Estimator.t()
UnivariateFeatureSelector Feature selector based on univariate statistical tests against
labels. Currently, Spark supports three Univariate Feature Selectors: chi-squared, ANOVA
F-test and F-value. User can choose Univariate Feature Selector by setting featureType and
labelType, and Spark will pick the score function based on the specified featureType and
labelType.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.UnivariateFeatureSelectorModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:feature_type— The feature type. Supported options: categorical, continuous.:features_col— features column name. Default"features".:label_col— label column name. Default"label".:label_type— The label type. Supported options: categorical, continuous.:output_col— output column name. Defaults to this operator's uid plus__output.:selection_mode— The selection mode. Supported options: numTopFeatures (default), percentile, fpr, fdr, fwe. Default"numTopFeatures".:selection_threshold— The upper bound of the features that selector will select.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec variance_threshold_selector(keyword()) :: Latu.ML.Estimator.t()
Feature selector that removes all low-variance features. Features with a (sample) variance not greater than the threshold will be removed. The default is to keep all features with non-zero variance, i.e. remove the features that have the same value in all samples.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.VarianceThresholdSelectorModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:features_col— features column name. Default"features".:output_col— output column name. Defaults to this operator's uid plus__output.:variance_threshold— Param for variance threshold. Features with a variance not greater than this threshold will be removed. The default value is 0.0. Default0.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec vector_assembler(keyword()) :: Latu.ML.Transformer.t()
A feature transformer that merges multiple columns into a vector column.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:handle_invalid— How to handle invalid data (NULL and NaN values). Options are 'skip' (filter out rows with invalid data), 'error' (throw an error), or 'keep' (return relevant number of NaN in the output). Column lengths are taken from the size of ML Attribute Group, which can be set usingVectorSizeHintin a pipeline beforeVectorAssembler. Column lengths can also be inferred from first rows of the data since it is safe to do so but only in case of 'error' or 'skip'). Default"error".:input_cols— input column names.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec vector_indexer(keyword()) :: Latu.ML.Estimator.t()
Class for indexing categorical feature columns in a dataset of Vector.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.VectorIndexerModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:handle_invalid— How to handle invalid data (unseen labels or NULL values). Options are 'skip' (filter out rows with invalid data), 'error' (throw an error), or 'keep' (put invalid data in a special additional bucket, at index of the number of categories of the feature). Default"error".:input_col— input column name.:max_categories— Threshold for the number of values a categorical feature can take (>= 2). If a feature is found to have > maxCategories values, then it is declared continuous. Default20.:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec vector_size_hint(keyword()) :: Latu.ML.Transformer.t()
A feature transformer that adds size information to the metadata of a vector column. VectorAssembler needs size information for its input columns and cannot be used on streaming dataframes without this metadata.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:handle_invalid— How to handle invalid vectors in inputCol. Invalid vectors include nulls and vectors with the wrong size. The options areskip(filter out rows with invalid vectors),error(throw an error) andoptimistic(do not check the vector size, and keep all rows).errorby default. Default"error".:input_col— input column name.:size— Size of vectors in column.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec vector_slicer(keyword()) :: Latu.ML.Transformer.t()
This class takes a feature vector and outputs a new feature vector with a subarray of the original features.
Latu.ML.transform/2 applies it. That is a lazy builder, not an action: it hands back a
Latu.DataFrame and reaches no server until you collect one.
Status :probed — dev/probe_ml.exs applied it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:indices— An array of indices to select features from a vector column. There can be no overlap with names. Default[].:input_col— input column name.:names— An array of feature names to select features from a vector column. These names must be specified by ML org.apache.spark.ml.attribute.Attribute. There can be no overlap with indices. Default[].:output_col— output column name. Defaults to this operator's uid plus__output.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec word2vec(keyword()) :: Latu.ML.Estimator.t()
Word2Vec trains a model of Map(String, Vector), i.e. transforms a word into a code for
further natural language processing or machine learning process.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Feature.Word2VecModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:input_col— input column name.:max_iter— max number of iterations (>= 0). Default1.:max_sentence_length— Maximum length (in words) of each sentence in the input data. Any sentence longer than this threshold will be divided into chunks up to the size. Default1000.:min_count— the minimum number of times a token must appear to be included in the word2vec model's vocabulary. Default5.:num_partitions— number of partitions for sentences of words. Default1.:output_col— output column name. Defaults to this operator's uid plus__output.:seed— random seed.:step_size— Step size to be used for each iteration of optimization (>= 0). Default0.025.:vector_size— the dimension of codes after transforming from words. Default100.:window_size— the window size (context words from [-window, window]). Default value is 5. Default5.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.