Attributes of a fitted DistributedLDAModel.
Every accessor here is one name on the server's allowlist for
org.apache.spark.ml.clustering.DistributedLDAModel — 11 of them — so tab completion is
that allowlist. The server refuses anything else with CONNECT_ML.ATTRIBUTE_NOT_ALLOWED;
these refuse a model of the wrong class before it gets that far, and name the module that
would have taken it.
Allowed by the server but not here: toString is Identifiable's, and tells you less than
the model's own uid. Latu.ML.attribute/2 and Latu.ML.attribute/3 will send any of these
names; it is what comes back that has nowhere to go.
The allowlist is inherited rather than per class: this one is the union of
org.apache.spark.ml.clustering.DistributedLDAModel,
org.apache.spark.ml.clustering.LDAModel, org.apache.spark.ml.util.Identifiable.
Summary
Functions
The JVM class these accessors belong to.
Return the topics described by their top-weighted terms.
Value for LDA.docConcentration estimated from data. If Online LDA was used and
LDA.optimizeDocConcentration was set to false, then this returns the fixed (given) value
for the LDA.docConcentration parameter.
If using checkpointing and LDA.keepLastCheckpoint is set to true, then there may be saved
checkpoint files. This method is provided so that users can manage those files.
Indicates whether this instance is of type DistributedLDAModel.
Calculates a lower bound on the log likelihood of the entire corpus. See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
Calculate an upper bound on perplexity. (Lower is better.) See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)
Convert this distributed model to a local representation. This discards info about the training dataset.
Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, Dirichlet hyperparameters)
Vocabulary size (number of terms or words in the vocabulary)
Functions
@spec class() :: String.t()
The JVM class these accessors belong to.
@spec describe_topics(Latu.ML.Model.t(), integer()) :: Latu.DataFrame.t()
Return the topics described by their top-weighted terms.
A lazy builder: the Fetch rides a relation, so this hands back a Latu.DataFrame and
nothing has run until you collect it.
Arguments
max_terms_per_topic— an integer
@spec estimated_doc_concentration(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Value for LDA.docConcentration estimated from data. If Online LDA was used and
LDA.optimizeDocConcentration was set to false, then this returns the fixed (given) value
for the LDA.docConcentration parameter.
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
@spec get_checkpoint_files(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
If using checkpointing and LDA.keepLastCheckpoint is set to true, then there may be saved
checkpoint files. This method is provided so that users can manage those files.
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
@spec is_distributed(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Indicates whether this instance is of type DistributedLDAModel.
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
@spec log_likelihood(Latu.ML.Model.t(), Latu.DataFrame.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Calculates a lower bound on the log likelihood of the entire corpus. See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
Arguments
dataset— aLatu.DataFrame, sent as a relation rather than a literal
@spec log_perplexity(Latu.ML.Model.t(), Latu.DataFrame.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Calculate an upper bound on perplexity. (Lower is better.) See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
Arguments
dataset— aLatu.DataFrame, sent as a relation rather than a literal
@spec log_prior(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
@spec to_local(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Convert this distributed model to a local representation. This discards info about the training dataset.
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
@spec topics_matrix(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
@spec training_log_likelihood(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, Dirichlet hyperparameters)
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.
@spec vocab_size(Latu.ML.Model.t()) :: {:ok, term()} | {:error, Latu.Error.t()}
Vocabulary size (number of terms or words in the vocabulary)
An action: it reaches the server. A Vector or Matrix comes back as an Nx.Tensor, or a
Latu.ML.SparseVector where densifying would be this package's decision rather than yours.