Clustering estimators, and the models they fit.
Unsupervised: a features column goes in and a cluster assignment comes out, with the
cluster centres reachable as tensors from the model's own module.
Every constructor here is generated from PySpark 4.2.0's own param table, and every
accessor module from the server's own attribute allowlist. Nothing in this file is
hand-written but the words you are reading — see Latu.ML.operators/1 for the table it all
comes from.
Summary
Functions
A bisecting k-means algorithm based on the paper "A comparison of document clustering
techniques" by Steinbach, Karypis, and Kumar, with modification to fit Spark. The algorithm
starts from a single cluster that contains all points. Iteratively it finds divisible
clusters on the bottom level and bisects each of them using k-means, until there are k
leaf clusters in total or no leaf clusters are divisible. The bisecting steps of clusters on
the same level are grouped together to increase parallelism. If bisecting all divisible
clusters on the bottom level would result more than k leaf clusters, larger clusters get
higher priority.
GaussianMixture clustering. This class performs expectation maximization for multivariate Gaussian Mixture Models (GMMs). A GMM represents a composite distribution of independent Gaussian distributions with associated "mixing" weights specifying each's contribution to the composite.
K-means clustering with a k-means++ like initialization mode (the k-means|| algorithm by Bahmani et al).
Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by Lin and Cohen. From the abstract: PIC finds a very low-dimensional embedding of a dataset using truncated power iteration on a normalized pair-wise similarity matrix of the data.
Functions
@spec bisecting_k_means(keyword()) :: Latu.ML.Estimator.t()
A bisecting k-means algorithm based on the paper "A comparison of document clustering
techniques" by Steinbach, Karypis, and Kumar, with modification to fit Spark. The algorithm
starts from a single cluster that contains all points. Iteratively it finds divisible
clusters on the bottom level and bisects each of them using k-means, until there are k
leaf clusters in total or no leaf clusters are divisible. The bisecting steps of clusters on
the same level are grouped together to increase parallelism. If bisecting all divisible
clusters on the bottom level would result more than k leaf clusters, larger clusters get
higher priority.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Clustering.BisectingKMeansModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:distance_measure— the distance measure. Supported options: 'euclidean' and 'cosine'. Default"euclidean".:features_col— features column name. Default"features".:k— The desired number of leaf clusters. Must be > 1. Default4.:max_iter— max number of iterations (>= 0). Default20.:min_divisible_cluster_size— The minimum number of points (if >= 1.0) or the minimum proportion of points (if < 1.0) of a divisible cluster. Default1.0.:prediction_col— prediction column name. Default"prediction".:seed— random seed.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec gaussian_mixture(keyword()) :: Latu.ML.Estimator.t()
GaussianMixture clustering. This class performs expectation maximization for multivariate Gaussian Mixture Models (GMMs). A GMM represents a composite distribution of independent Gaussian distributions with associated "mixing" weights specifying each's contribution to the composite.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Clustering.GaussianMixtureModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:aggregation_depth— suggested depth for treeAggregate (>= 2). Default2.:features_col— features column name. Default"features".:k— Number of independent Gaussians in the mixture model. Must be > 1. Default2.:max_iter— max number of iterations (>= 0). Default100.:prediction_col— prediction column name. Default"prediction".:probability_col— Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. Default"probability".:seed— random seed.:tol— the convergence tolerance for iterative algorithms (>= 0). Default0.01.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec k_means(keyword()) :: Latu.ML.Estimator.t()
K-means clustering with a k-means++ like initialization mode (the k-means|| algorithm by Bahmani et al).
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on Latu.ML.Clustering.KMeansModel.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:distance_measure— the distance measure. Supported options: 'euclidean' and 'cosine'. Default"euclidean".:features_col— features column name. Default"features".:init_mode— The initialization algorithm. This can be either "random" to choose random points as initial cluster centers, or "k-means||" to use a parallel variant of k-means++. Default"k-means||".:init_steps— The number of steps for k-means|| initialization mode. Must be > 0. Default2.:k— The number of clusters to create. Must be > 1. Default2.:max_block_size_in_mb— maximum memory in MB for stacking input data into blocks. Data is stacked within partitions. If more than remaining data size in a partition then it is adjusted to the data size. Default 0.0 represents choosing optimal value, depends on specific algorithm. Must be >= 0. Default0.0.:max_iter— max number of iterations (>= 0). Default20.:prediction_col— prediction column name. Default"prediction".:seed— random seed.:solver— The solver algorithm for optimization. Supported options: auto, row, block. Default"auto".:tol— the convergence tolerance for iterative algorithms (>= 0). Default0.0001.:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec lda(keyword()) :: Latu.ML.Estimator.t()
Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's
ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the
explicit form. Its attributes are on the model's own module.
Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and
every allowlisted attribute it could ask answered.
Params
:checkpoint_interval— set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext. Default10.:doc_concentration— Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").:features_col— features column name. Default"features".:k— The number of topics (clusters) to infer. Must be > 1. Default10.:keep_last_checkpoint— (For EM optimizer) If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Defaulttrue.:learning_decay— Learning rate, set as anexponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. Default0.51.:learning_offset— A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. Default1024.0.:max_iter— max number of iterations (>= 0). Default20.:optimize_doc_concentration— Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Defaulttrue.:optimizer— Optimizer or inference algorithm used to estimate the LDA model. Supported: online, em. Default"online".:seed— random seed.:subsampling_rate— Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1]. Default0.05.:topic_concentration— Concentration parameter (commonly named "beta" or "eta") for the prior placed on topic' distributions over terms.:topic_distribution_col— Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document. Default"topicDistribution".
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.
@spec power_iteration_clustering(keyword()) :: Latu.ML.Helper.t()
Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by Lin and Cohen. From the abstract: PIC finds a very low-dimensional embedding of a dataset using truncated power iteration on a normalized pair-wise similarity matrix of the data.
Latu.ML.assign_clusters/2 is the one thing it does, and it is a lazy builder: it hands
back a Latu.DataFrame and reaches no server until you collect one. There is no fit and
nothing cached — its params ride that call as positional arguments, so every one is
sent whether you set it or not, and the ones you leave alone are sent as the defaults below.
Status :built — generated from PySpark 4.2.0's own param table, and not yet exercised
against a live server by dev/probe_ml.exs.
Params
:dst_col— Name of the input column for destination vertex IDs. Default"dst".:init_mode— The initialization algorithm. This can be either 'random' to use a random vector as vertex properties, or 'degree' to use a normalized sum of similarities with other vertices. Supported options: 'random' and 'degree'. Default"random".:k— The number of clusters to create. Must be > 1. Default2.:max_iter— max number of iterations (>= 0). Default20.:src_col— Name of the input column for source vertex IDs. Default"src".:weight_col— weight column name. If this is not set or empty, we treat all instance weights as 1.0.
Defaults are documented, never sent: a param the caller did not set and a param sent with
its default value are different requests, and only the first is right. A value's kind is
refused here; its range is Spark's own ParamValidators to refuse, with a better
message than this package could write.