View Source Scholar.Metrics.Classification (Scholar v0.2.1)

Classification Metric functions.

Metrics are used to measure the performance and compare the performance of any kind of classifier in easy-to-understand terms.

All of the functions in this module are implemented as numerical functions and can be JIT or AOT compiled with any supported Nx compiler.

Summary

Functions

Computes the accuracy of the given predictions for binary and multi-class classification problems.

Computes area under the curve (AUC) using the trapezoidal rule.

Computes the balanced accuracy score for multi-class classification

Computes the precision of the given predictions with respect to the given targets for binary classification problems.

Computes the recall of the given predictions with respect to the given targets for binary classification problems.

Computes the sensitivity of the given predictions with respect to the given targets for binary classification problems.

Computes the specificity of the given predictions with respect to the given targets for binary classification problems.

Compute the Brier score loss.

Compute Cohen's kappa: a statistic that measures inter-annotator agreement.

Calculates the confusion matrix given rank-1 tensors which represent the expected (y_true) and predicted (y_pred) classes.

It's a helper function for Scholar.Metrics.Classification.roc_curve and Scholar.Metrics.Classification.roc_auc_score functions. You should call it and use as follows

Calculates F1 score given rank-1 tensors which represent the expected (y_true) and predicted (y_pred) classes.

Computes the precision of the given predictions with respect to the given targets for multi-class classification problems.

Compute precision-recall pairs for different probability thresholds.

Computes the recall of the given predictions with respect to the given targets for multi-class classification problems.

Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.

Compute Receiver operating characteristic (ROC).

Computes the sensitivity of the given predictions with respect to the given targets for multi-class classification problems.

Computes the specificity of the given predictions with respect to the given targets for multi-class classification problems.

Top-k Accuracy classification score.

Zero-one classification loss.

Functions

Link to this function

accuracy(y_true, y_pred, opts \\ [])

View Source

Computes the accuracy of the given predictions for binary and multi-class classification problems.

Examples

iex> Scholar.Metrics.Classification.accuracy(Nx.tensor([1, 0, 0]), Nx.tensor([1, 0, 1]))
#Nx.Tensor<
  f32
  0.6666666865348816
>

iex> y_true = Nx.tensor([0, 1, 1, 1, 1, 0, 2, 1, 0, 1], type: :u32)
iex> y_pred = Nx.tensor([0, 2, 1, 1, 2, 2, 2, 0, 0, 1], type: :u32)
iex> Scholar.Metrics.Classification.accuracy(y_true, y_pred)
#Nx.Tensor<
  f32
  0.6000000238418579
>

iex> y_true = Nx.tensor([0, 1, 1, 1, 1, 0, 2, 1, 0, 1], type: :u32)
iex> y_pred = Nx.tensor([0, 2, 1, 1, 2, 2, 2, 0, 0, 1], type: :u32)
iex> Scholar.Metrics.Classification.accuracy(y_true, y_pred, normalize: false)
#Nx.Tensor<
  u64
  6
>

Computes area under the curve (AUC) using the trapezoidal rule.

This is a general function, given points on a curve.

Examples

iex> y = Nx.tensor([0, 0, 1, 1])
iex> pred = Nx.tensor([0.1, 0.4, 0.35, 0.8])
iex> distinct_value_indices = Scholar.Metrics.Classification.distinct_value_indices(pred)
iex> {fpr, tpr, _thresholds} = Scholar.Metrics.Classification.roc_curve(y, pred, distinct_value_indices)
iex> Scholar.Metrics.Classification.auc(fpr, tpr)
#Nx.Tensor<
  f32
  0.75
>
Link to this function

average_precision_score(y_true, probabilities_predicted, distinct_value_indices, weights \\ 1.0)

View Source

Compute average precision (AP) from prediction scores.

AP summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight: $$ AP = sumn (R_n - R{n-1}) P_n $$

where $ P_n $ and $ R_n $ are the precision and recall at the nth threshold.

Examples

iex> y_true = Nx.tensor([0, 0, 1, 1])
iex> scores = Nx.tensor([0.1, 0.4, 0.35, 0.8])
iex> distinct_value_indices = Scholar.Metrics.Classification.distinct_value_indices(scores)
iex> weights = Nx.tensor([1, 1, 2, 2])
iex> ap = Scholar.Metrics.Classification.average_precision_score(y_true, scores, distinct_value_indices, weights)
iex> ap
#Nx.Tensor<
  f32
  0.8999999761581421
>
Link to this function

balanced_accuracy_score(y_true, y_pred, opts \\ [])

View Source

Computes the balanced accuracy score for multi-class classification

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

  • :sample_weights - Sample weights of the observations.

  • :adjusted (boolean/0) - If true, the balanced accuracy is adjusted for chance (depends on the number of classes). The default value is false.

Examples

iex> y_true = Nx.tensor([0, 1, 2, 0, 1, 2], type: {:u, 32})
iex> y_pred = Nx.tensor([0, 2, 1, 0, 0, 1], type: {:u, 32})
iex> Scholar.Metrics.Classification.balanced_accuracy_score(y_true, y_pred, num_classes: 3)
#Nx.Tensor<
  f32
  0.3333333432674408
>
iex> y_true = Nx.tensor([0, 1, 2, 0, 1, 2], type: {:u, 32})
iex> y_pred = Nx.tensor([0, 2, 1, 0, 0, 1], type: {:u, 32})
iex> sample_weights = [1, 1, 1, 2, 2, 2]
iex> Scholar.Metrics.Classification.balanced_accuracy_score(y_true, y_pred, num_classes: 3, sample_weights: sample_weights, adjusted: true)
#Nx.Tensor<
  f32
  0.0
>
Link to this function

binary_precision(y_true, y_pred)

View Source

Computes the precision of the given predictions with respect to the given targets for binary classification problems.

If the sum of true positives and false positives is 0, then the result is 0 to avoid zero division.

Examples

iex> Scholar.Metrics.Classification.binary_precision(Nx.tensor([0, 1, 1, 1]), Nx.tensor([1, 0, 1, 1]))
#Nx.Tensor<
  f32
  0.6666666865348816
>
Link to this function

binary_recall(y_true, y_pred)

View Source

Computes the recall of the given predictions with respect to the given targets for binary classification problems.

If the sum of true positives and false negatives is 0, then the result is 0 to avoid zero division.

Examples

iex> Scholar.Metrics.Classification.binary_recall(Nx.tensor([0, 1, 1, 1]), Nx.tensor([1, 0, 1, 1]))
#Nx.Tensor<
  f32
  0.6666666865348816
>
Link to this function

binary_sensitivity(y_true, y_pred)

View Source

Computes the sensitivity of the given predictions with respect to the given targets for binary classification problems.

Examples

iex> Scholar.Metrics.Classification.binary_sensitivity(Nx.tensor([0, 1, 1, 1]), Nx.tensor([1, 0, 1, 1]))
#Nx.Tensor<
  f32
  0.6666666865348816
>
Link to this function

binary_specificity(y_true, y_pred)

View Source

Computes the specificity of the given predictions with respect to the given targets for binary classification problems.

If the sum of true negatives and false positives is 0, then the result is 0 to avoid zero division.

Examples

iex> Scholar.Metrics.Classification.binary_specificity(Nx.tensor([0, 1, 1, 1]), Nx.tensor([1, 0, 1, 1]))
#Nx.Tensor<
  f32
  0.0
>
Link to this function

brier_score_loss(y_true, y_prob, opts \\ [])

View Source

Compute the Brier score loss.

The smaller the Brier score loss, the better, hence the naming with "loss". The Brier score measures the mean squared difference between the predicted probability and the actual outcome. The Brier score always takes on a value between zero and one, since this is the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 and 1). It can be decomposed as the sum of refinement loss and calibration loss. If predicted probabilities are not in the interval [0, 1], they will be clipped.

The Brier score is appropriate only for binary outcomes.

Options

  • :sample_weights - Sample weights of the observations. The default value is 1.0.

  • :pos_label (integer/0) - Label of the positive class. The default value is 1.

Examples

iex> y_true = Nx.tensor([0, 1, 1, 0])
iex> y_prob = Nx.tensor([0.1, 0.9, 0.8, 0.3])
iex> Scholar.Metrics.Classification.brier_score_loss(y_true, y_prob)
#Nx.Tensor<
  f32
  0.03750000149011612
>
Link to this function

cohen_kappa_score(y1, y2, opts \\ [])

View Source

Compute Cohen's kappa: a statistic that measures inter-annotator agreement.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

  • :weights - Weighting to calculate the score.

  • :weighting_type - Weighting type to calculate the score.

Examples

iex> y1 = Nx.tensor([0, 1, 1, 0, 1, 2])
iex> y2 = Nx.tensor([0, 2, 1, 0, 0, 1])
iex> Scholar.Metrics.Classification.cohen_kappa_score(y1, y2, num_classes: 3)
#Nx.Tensor<
  f32
  0.21739131212234497
>

iex> y1 = Nx.tensor([0, 1, 1, 0, 1, 2])
iex> y2 = Nx.tensor([0, 2, 1, 0, 0, 1])
iex> Scholar.Metrics.Classification.cohen_kappa_score(y1, y2, num_classes: 3, weighting_type: :linear)
#Nx.Tensor<
  f32
  0.3571428060531616
>
Link to this function

confusion_matrix(y_true, y_pred, opts \\ [])

View Source

Calculates the confusion matrix given rank-1 tensors which represent the expected (y_true) and predicted (y_pred) classes.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

Examples

iex> y_true = Nx.tensor([0, 0, 1, 1, 2, 2], type: :u32)
iex> y_pred = Nx.tensor([0, 1, 0, 2, 2, 2], type: :u32)
iex> Scholar.Metrics.Classification.confusion_matrix(y_true, y_pred, num_classes: 3)
#Nx.Tensor<
  u64[3][3]
  [
    [1, 1, 0],
    [1, 0, 1],
    [0, 0, 2]
  ]
>

iex> y_true = Nx.tensor([0, 0, 1, 1, 2, 2], type: {:u, 32})
iex> y_pred = Nx.tensor([0, 1, 0, 2, 2, 2], type: {:u, 32})
iex> sample_weights = [2, 5, 1, 1.5, 2, 8]
iex> Scholar.Metrics.Classification.confusion_matrix(y_true, y_pred, num_classes: 3, sample_weights: sample_weights, normalize: :predicted)
#Nx.Tensor<
  f32[3][3]
  [
    [0.6666666865348816, 1.0, 0.0],
    [0.3333333432674408, 0.0, 0.1304347813129425],
    [0.0, 0.0, 0.8695651888847351]
  ]
>
Link to this function

distinct_value_indices(y_score)

View Source

It's a helper function for Scholar.Metrics.Classification.roc_curve and Scholar.Metrics.Classification.roc_auc_score functions. You should call it and use as follows:

distinct_value_indices = Scholar.Metrics.Classification.distinct_value_indices(scores)
{fpr, tpr, thresholds} = Scholar.Metrics.Classification.roc_curve(y_true, scores, distinct_value_indices, weights)
Link to this function

f1_score(y_true, y_pred, opts \\ [])

View Source

Calculates F1 score given rank-1 tensors which represent the expected (y_true) and predicted (y_pred) classes.

If all examples are true negatives, then the result is 0 to avoid zero division.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

  • :average - This determines the type of averaging performed on the data.

    • :macro - Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.

    • :weighted - Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label).

    • :micro - Calculate metrics globally by counting the total true positives, false negatives and false positives.

    • :none - The f1 scores for each class are returned.

    The default value is :none.

Examples

iex> y_true = Nx.tensor([0, 1, 1, 1, 1, 0, 2, 1, 0, 1], type: :u32)
iex> y_pred = Nx.tensor([0, 2, 1, 1, 2, 2, 2, 0, 0, 1], type: :u32)
iex> Scholar.Metrics.Classification.f1_score(y_true, y_pred, num_classes: 3)
#Nx.Tensor<
  f32[3]
  [0.6666666865348816, 0.6666666865348816, 0.4000000059604645]
>
iex> Scholar.Metrics.Classification.f1_score(y_true, y_pred, num_classes: 3, average: :macro)
#Nx.Tensor<
  f32
  0.5777778029441833
>
iex> Scholar.Metrics.Classification.f1_score(y_true, y_pred, num_classes: 3, average: :weighted)
#Nx.Tensor<
  f32
  0.6399999856948853
>
iex> Scholar.Metrics.Classification.f1_score(y_true, y_pred, num_classes: 3, average: :micro)
#Nx.Tensor<
  f32
  0.6000000238418579
>
iex> Scholar.Metrics.Classification.f1_score(Nx.tensor([1,0,1,0]), Nx.tensor([0, 1, 0, 1]), num_classes: 2, average: :none)
#Nx.Tensor<
  f32[2]
  [0.0, 0.0]
>
Link to this function

precision(y_true, y_pred, opts \\ [])

View Source

Computes the precision of the given predictions with respect to the given targets for multi-class classification problems.

If the sum of true positives and false positives is 0, then the result is 0 to avoid zero division.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

Examples

iex> y_true = Nx.tensor([0, 1, 1, 1, 1, 0, 2, 1, 0, 1], type: :u32)
iex> y_pred = Nx.tensor([0, 2, 1, 1, 2, 2, 2, 0, 0, 1], type: :u32)
iex> Scholar.Metrics.Classification.precision(y_true, y_pred, num_classes: 3)
#Nx.Tensor<
  f32[3]
  [0.6666666865348816, 1.0, 0.25]
>
Link to this function

precision_recall_curve(y_true, probabilities_predicted, distinct_value_indices, weights \\ 1.0)

View Source

Compute precision-recall pairs for different probability thresholds.

Note: this implementation is restricted to the binary classification task.

Examples

iex> y_true = Nx.tensor([0, 0, 1, 1])
iex> scores = Nx.tensor([0.1, 0.4, 0.35, 0.8])
iex> distinct_value_indices = Scholar.Metrics.Classification.distinct_value_indices(scores)
iex> weights = Nx.tensor([1, 1, 2, 2])
iex> {precision, recall, thresholds} = Scholar.Metrics.Classification.precision_recall_curve(y_true, scores, distinct_value_indices, weights)
iex> precision
#Nx.Tensor<
  f32[5]
  [0.6666666865348816, 0.800000011920929, 0.6666666865348816, 1.0, 1.0]
>
iex> recall
#Nx.Tensor<
  f32[5]
  [1.0, 1.0, 0.5, 0.5, 0.0]
>
iex> thresholds
#Nx.Tensor<
  f32[4]
  [0.10000000149011612, 0.3499999940395355, 0.4000000059604645, 0.800000011920929]
>
Link to this function

recall(y_true, y_pred, opts \\ [])

View Source

Computes the recall of the given predictions with respect to the given targets for multi-class classification problems.

If the sum of true positives and false negatives is 0, then the result is 0 to avoid zero division.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

Examples

iex> y_true = Nx.tensor([0, 1, 1, 1, 1, 0, 2, 1, 0, 1], type: :u32)
iex> y_pred = Nx.tensor([0, 2, 1, 1, 2, 2, 2, 0, 0, 1], type: :u32)
iex> Scholar.Metrics.Classification.recall(y_true, y_pred, num_classes: 3)
#Nx.Tensor<
  f32[3]
  [0.6666666865348816, 0.5, 1.0]
>
Link to this function

roc_auc_score(y_true, y_score, distinct_value_indices, weights \\ 1.0)

View Source

Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.

Note: this implementation is restricted to the binary classification task.

Examples

iex> y_true = Nx.tensor([0, 0, 1, 1])
iex> scores = Nx.tensor([0.1, 0.4, 0.35, 0.8])
iex> distinct_value_indices = Scholar.Metrics.Classification.distinct_value_indices(scores)
iex> weights = Nx.tensor([1, 1, 2, 2])
iex> Scholar.Metrics.Classification.roc_auc_score(y_true, scores, distinct_value_indices, weights)
#Nx.Tensor<
  f32
  0.75
>
Link to this function

roc_curve(y_true, y_score, distinct_value_indices, weights \\ 1.0)

View Source

Compute Receiver operating characteristic (ROC).

Note: this implementation is restricted to the binary classification task.

Examples

iex> y_true = Nx.tensor([0, 0, 1, 1])
iex> scores = Nx.tensor([0.1, 0.4, 0.35, 0.8])
iex> distinct_value_indices = Scholar.Metrics.Classification.distinct_value_indices(scores)
iex> weights = Nx.tensor([1, 1, 2, 2])
iex> {fpr, tpr, thresholds} = Scholar.Metrics.Classification.roc_curve(y_true, scores, distinct_value_indices, weights)
iex> fpr
#Nx.Tensor<
  f32[5]
  [0.0, 0.0, 0.5, 0.5, 1.0]
>
iex> tpr
#Nx.Tensor<
  f32[5]
  [0.0, 0.5, 0.5, 1.0, 1.0]
>
iex> thresholds
#Nx.Tensor<
  f32[5]
  [1.7999999523162842, 0.800000011920929, 0.4000000059604645, 0.3499999940395355, 0.10000000149011612]
>
Link to this function

sensitivity(y_true, y_pred, opts \\ [])

View Source

Computes the sensitivity of the given predictions with respect to the given targets for multi-class classification problems.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

Examples

iex> y_true = Nx.tensor([0, 1, 1, 1, 1, 0, 2, 1, 0, 1], type: :u32)
iex> y_pred = Nx.tensor([0, 2, 1, 1, 2, 2, 2, 0, 0, 1], type: :u32)
iex> Scholar.Metrics.Classification.sensitivity(y_true, y_pred, num_classes: 3)
#Nx.Tensor<
  f32[3]
  [0.6666666865348816, 0.5, 1.0]
>
Link to this function

specificity(y_true, y_pred, opts \\ [])

View Source

Computes the specificity of the given predictions with respect to the given targets for multi-class classification problems.

If the sum of true negatives and false positives is 0, then the result is 0 to avoid zero division.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

Examples

iex> y_true = Nx.tensor([0, 1, 1, 1, 1, 0, 2, 1, 0, 1], type: :u32)
iex> y_pred = Nx.tensor([0, 2, 1, 1, 2, 2, 2, 0, 0, 1], type: :u32)
iex> Scholar.Metrics.Classification.specificity(y_true, y_pred, num_classes: 3)
#Nx.Tensor<
  f32[3]
  [0.8571428656578064, 1.0, 0.6666666865348816]
>
Link to this function

top_k_accuracy_score(y_true, y_prob, opts \\ [])

View Source

Top-k Accuracy classification score.

This metric computes the number of times where the correct label is among the top k labels predicted (ranked by predicted scores).

For binary task assumed that y_score have values from 0 to 1.

Options

  • :num_classes (pos_integer/0) - Required. Number of classes contained in the input tensors

  • :k (integer/0) - Number of top elements to look at for computing accuracy. The default value is 5.

  • :normalize (boolean/0) - If true, return the fraction of correctly classified samples. Otherwise, return the number of correctly classified samples. The default value is true.

Examples

iex> y_true = Nx.tensor([0, 1, 2, 2, 0])
iex> y_score = Nx.tensor([[0.5, 0.2, 0.1], [0.3, 0.4, 0.5], [0.4, 0.3, 0.2], [0.1, 0.3, 0.6], [0.9, 0.1, 0.0]])
iex> Scholar.Metrics.Classification.top_k_accuracy_score(y_true, y_score, k: 2, num_classes: 3)
#Nx.Tensor<
  f32
  0.800000011920929
>

iex> y_true = Nx.tensor([0, 1, 2, 2, 0])
iex> y_score = Nx.tensor([[0.5, 0.2, 0.1], [0.3, 0.4, 0.5], [0.4, 0.3, 0.2], [0.1, 0.3, 0.6], [0.9, 0.1, 0.0]])
iex> Scholar.Metrics.Classification.top_k_accuracy_score(y_true, y_score, k: 2, num_classes: 3, normalize: false)
#Nx.Tensor<
  u64
  4
>

iex> y_true = Nx.tensor([0, 1, 0, 1, 0])
iex> y_score = Nx.tensor([0.55, 0.3, 0.1, -0.2, 0.99])
iex> Scholar.Metrics.Classification.top_k_accuracy_score(y_true, y_score, k: 1, num_classes: 2)
#Nx.Tensor<
  f32
  0.20000000298023224
>
Link to this function

zero_one_loss(y_true, y_pred, opts \\ [])

View Source

Zero-one classification loss.

Options

  • :normalize (boolean/0) - If true, return the fraction of incorrectly classified samples. Otherwise, return the number of incorrectly classified samples. The default value is true.

Examples

iex> y_pred = Nx.tensor([1, 2, 3, 4])
iex> y_true = Nx.tensor([2, 2, 3, 4])
iex> Scholar.Metrics.Classification.zero_one_loss(y_true, y_pred)
#Nx.Tensor<
  f32
  0.25
>

iex> y_pred = Nx.tensor([1, 2, 3, 4])
iex> y_true = Nx.tensor([2, 2, 3, 4])
iex> Scholar.Metrics.Classification.zero_one_loss(y_true, y_pred, normalize: false)
#Nx.Tensor<
  u64
  1
>