Quadratic Discriminant Analysis (QDA) for classification.
QDA fits a Gaussian density to each class, letting every class have its own
covariance matrix. Unlike Scholar.DiscriminantAnalysis.Linear, which pools a
single covariance across classes, the per-class covariances leave a quadratic
term in the score, so the decision boundary between two classes is a quadric
rather than a hyperplane. For a sample $x$ the score of class $k$ is
$$ \delta\_{k}(x) = -\frac{1}{2} \log |\Sigma\_{k}| - \frac{1}{2} (x - \mu\_{k})^{T} \Sigma\_{k}^{-1} (x - \mu\_{k}) + \log \pi\_{k} $$
where $\mu\_{k}$ is the class mean, $\Sigma\_{k}$ the class covariance and $\pi\_{k}$ the class prior. The predicted class is the one with the highest score.
Each covariance is stored through its eigendecomposition $\Sigma\_{k} = R\_{k} \Lambda\_{k} R\_{k}^{T}$, which turns both the log-determinant and the quadratic form into elementwise work over the eigenvalues.
Every class covariance is assumed to be invertible, which requires at least as
many samples as features in each class and no feature being a linear
combination of the others. When that does not hold, a covariance is singular
and its smallest eigenvalues come out as zero, which the score then divides by.
Use :reg_param to shrink the eigenvalues towards one in that case.
Note that scikit-learn handles rank deficiency differently: it decomposes the
centered class data and keeps only min(class_size, num_features) components,
so it works in the subspace the class actually spans. This implementation keeps
all num_features directions and relies on :reg_param instead, so results
diverge from scikit-learn's for a class with fewer samples than features.
Forming the covariance squares its condition number, so features on wildly
different scales cost accuracy in the eigendecomposition well before the
matrix is singular. A compiled backend absorbs this, but the pure-Nx
eigendecomposition does not, and features spanning several orders of magnitude
can lose the smallest eigenvalues there entirely. Standardizing the features
beforehand, with Scholar.Preprocessing.StandardScaler, avoids it either way.
Reference:
Summary
Functions
Computes the quadratic discriminant scores of each class for samples x.
Fits a Quadratic Discriminant Analysis model for sample inputs x and target
labels y.
Predicts the class of each sample in x.
Estimates class probabilities for samples x by applying a softmax to the
discriminant scores.
Functions
Computes the quadratic discriminant scores of each class for samples x.
Examples
iex> x = Nx.tensor([[-2.0, -1.0], [-1.0, -1.0], [-1.0, -2.0], [1.0, 1.0], [1.0, 2.0], [2.0, 1.0]])
iex> y = Nx.tensor([0, 0, 0, 1, 1, 1])
iex> model = Scholar.DiscriminantAnalysis.Quadratic.fit(x, y, num_classes: 2)
iex> scores = Scholar.DiscriminantAnalysis.Quadratic.decision_function(model, Nx.tensor([[1.5, 1.5]]))
iex> Nx.shape(scores)
{1, 2}Fits a Quadratic Discriminant Analysis model for sample inputs x and target
labels y.
Options
:num_classes(pos_integer/0) - Required. Number of different classes used in training. Labels must be integers in[0, num_classes).:reg_param- Regularizes the per-class covariance by shrinking its eigenvalues towards one, as(1 - reg_param) * eigenvalues + reg_param. Use a value in[0, 1]when a class has fewer samples than features, or when features are collinear, which would otherwise leave the covariance singular. The default value is0.0.
Return Values
The function returns a struct with the following parameters:
:means- Class-wise mean of the samples, shape{num_classes, num_features}.:priors- Class prior probabilities, shape{num_classes}.:scalings- Eigenvalues of each class covariance, shape{num_classes, num_features}.:rotations- Eigenvectors of each class covariance, shape{num_classes, num_features, num_features}.
Examples
iex> x = Nx.tensor([[-2.0, -1.0], [-1.0, -1.0], [-1.0, -2.0], [1.0, 1.0], [1.0, 2.0], [2.0, 1.0]])
iex> y = Nx.tensor([0, 0, 0, 1, 1, 1])
iex> model = Scholar.DiscriminantAnalysis.Quadratic.fit(x, y, num_classes: 2)
iex> Scholar.DiscriminantAnalysis.Quadratic.predict(model, Nx.tensor([[-1.5, -1.5], [1.5, 1.5]]))
#Nx.Tensor<
s32[2]
[0, 1]
>Predicts the class of each sample in x.
Examples
iex> x = Nx.tensor([[-2.0, -1.0], [-1.0, -1.0], [-1.0, -2.0], [1.0, 1.0], [1.0, 2.0], [2.0, 1.0]])
iex> y = Nx.tensor([0, 0, 0, 1, 1, 1])
iex> model = Scholar.DiscriminantAnalysis.Quadratic.fit(x, y, num_classes: 2)
iex> Scholar.DiscriminantAnalysis.Quadratic.predict(model, Nx.tensor([[-1.5, -1.5], [1.5, 1.5]]))
#Nx.Tensor<
s32[2]
[0, 1]
>Estimates class probabilities for samples x by applying a softmax to the
discriminant scores.
Examples
iex> x = Nx.tensor([[-2.0, -1.0], [-1.0, -1.0], [-1.0, -2.0], [1.0, 1.0], [1.0, 2.0], [2.0, 1.0]])
iex> y = Nx.tensor([0, 0, 0, 1, 1, 1])
iex> model = Scholar.DiscriminantAnalysis.Quadratic.fit(x, y, num_classes: 2)
iex> probs = Scholar.DiscriminantAnalysis.Quadratic.predict_probability(model, Nx.tensor([[1.5, 1.5]]))
iex> Nx.sum(probs) |> Nx.round()
#Nx.Tensor<
f32
1.0
>