# When to Accept Automated Predictions and When to Defer to Human Judgment?

Daniel Sikar Corresponding Author. Email<EMAIL_ADDRESS>Artur
Garcez0000-0002-6060-0409  Tillman Weyde0000-0001-8028-9905  Robin Bloomfield
Kaleem Peeroo  Department of Computer Science, School of Science and
Technology, City, University of London

###### Abstract

Ensuring the reliability and safety of automated decision-making is crucial.
It is well-known that data distribution shifts in machine learning can produce
unreliable outcomes. This paper proposes a new approach for measuring the
reliability of predictions under distribution shifts. We analyze how the
outputs of a trained neural network change using clustering to measure
distances between outputs and class centroids. We propose this distance as a
metric to evaluate the confidence of predictions under distribution shifts. We
assign each prediction to a cluster with centroid representing the mean
softmax output for all correct predictions of a given class. We then define a
safety threshold for a class as the smallest distance from an incorrect
prediction to the given class centroid. We evaluate the approach on the MNIST
and CIFAR-10 datasets using a Convolutional Neural Network and a Vision
Transformer, respectively. The results show that our approach is consistent
across these data sets and network models, and indicate that the proposed
metric can offer an efficient way of determining when automated predictions
are acceptable and when they should be deferred to human operators given a
distribution shift.

1654

##  1 Introduction

Ensuring the reliability and safety of automated decision-making systems is
important, particularly in high-impact areas where there exists potential for
significant harm from errors [1]. Machine learning (ML) models, while
powerful, are susceptible to making erroneous predictions when faced with data
that differs from the distribution that they were trained on [22]. This
phenomenon, known as distribution shift, poses a significant challenge in
deploying ML in real-world scenarios [40].

Distribution shift is a pervasive issue in ML, occurring when the distribution
of the data used to train a model differs from the distribution of the data
that the model encounters during deployment [40]. This discrepancy can lead to
a significant degradation in model performance, as it may struggle to
generalize to the new, unseen data [20].

Distribution shifts can manifest in various forms, such as covariate shift,
concept drift, and domain shift [35]. Covariate shift arises when the input
data distribution changes while the conditional distribution of the output
given the input remains the same [45]. Concept drift occurs when the
relationship between the input and output variables changes over time [13].
Domain shift refers to the situation where the model is trained on data from
one domain but applied to data from a different domain [39].

To quantify and address distribution shift, researchers have developed various
metrics and techniques. One common approach is to use statistical divergence
measures, such as Kullback-Leibler (KL) divergence [27] or Maximum Mean
Discrepancy (MMD) [17], to assess differences in training and test data
distributions. These metrics provide a quantitative understanding of the
extent of the distribution shift.

Another approach is to employ domain adaptation techniques, which aim to align
the feature distributions of the source and target domains [48]. This can be
achieved through methods such as importance weighting [46], feature
transformation [38], or adversarial learning [14]. These techniques seek to
mitigate the impact of distribution shift by making the model more robust to
changes in the data distribution.

Recent work has also focused on developing algorithms that can detect and
adapt to distribution shift in real-time [32]. These methods often rely on
monitoring the model’s performance on a stream of data and adjusting the
model’s parameters or architecture when a significant drop in performance is
detected [3]. Such adaptive approaches are particularly relevant in dynamic
environments where the data distribution is likely to change over time.

Therefore, distribution shift is a significant challenge that can lead to poor
model performance if not addressed properly. As ML models are increasingly
deployed in real-world applications, developing robust methods to handle
distribution shift remains an important area of research.

In this paper, we propose a novel approach for quantifying the reliability of
predictions made by neural networks under distribution shift. Our method
leverages clustering techniques to measure the distances between the outputs
of a trained neural network and class centroids in the softmax distance space.
By analyzing these distances, we develop a metric that provides insight into
the confidence that one may attribute to the model’s predictions. We adopt the
most conservative threshold value at which model predictions are expected to
be 100% accurate. The proposed metric offers a computationally efficient,
practical way to determine when to accept an automated prediction and when
human intervention may be necessary.

Furthermore, we explore the relationship between the distance to class
centroids and the model’s predictive accuracy. Our findings confirm as
expected that classes predicted more accurately by the model tend to have
lower softmax distances to their respective centroids. This observation
highlights the potential for using the changes observed in the softmax
distribution as a proxy for our confidence in the model’s predictions. The
objective is to establish a closer link than thus far identified by the
literature between the distance-based metrics proposed for distribution shift
and the approaches that measure the drop in model accuracy. It is expected to
offer a better understanding of the connection between model performance and
reliability, that is, accuracy and how confident we are in the predictions of
neural networks.

The main contributions of this paper are:

  * •

A new lightweight approach for quantifying the reliability of neural networks
under distribution shift using distance metrics and clustering to take full
advantage of the learned softmax distributions.

  * •

A combined analysis of the metric-based and accuracy-based methods to tackle
distribution shift, with experimental results showing consistency of the
proposed approach across CNN and ViT architectures.

###  1.1 Disclaimer

While identifying and mitigating safety risks within the classification model
itself is crucial, it does not guarantee the overall safety, robustness,
reliability, trustworthiness, and confident performance of the entire system
in which the model is deployed.

In critical applications, the classification model is often just one component
of a larger, complex system. The safe and reliable operation of such a system
depends on the proper functioning and interaction of all its components,
including data acquisition, preprocessing, decision-making, and actuation.
Even if the classification model is designed to mitigate safety risks,
failures or vulnerabilities in other components can still compromise the
overall safety and reliability of the system.

Furthermore, the deployment environment and context in which the
classification system operates can introduce additional challenges and risks
that may not be fully captured or addressed during the model development
phase. Factors such as data distribution shifts, adversarial attacks, or
unexpected edge cases can impact the model’s performance and lead to potential
safety issues.

Therefore, it is important to recognize that developing a safe and reliable
classification model is necessary but not sufficient for ensuring the overall
safety and reliability of the deployed system. A holistic approach that
considers the entire system architecture, interfaces, and deployment context
is essential. This includes rigorous testing, validation, and monitoring of
the entire system, as well as establishing robust failsafe mechanisms and
human oversight to handle unexpected situations and maintain safe operation.

In summary, while identifying and mitigating safety risks within the
classification model is important, it is crucial to acknowledge that it does
not guarantee the safe and reliable performance of the entire deployed system.
A comprehensive approach considering all components, their interactions, and
the deployment context is necessary to ensure the safe, robust, reliable,
trustworthy, and confident operation of critical applications.

##  2 Background

Softmax prediction probabilities: The concept of using the entire set of
softmax prediction probabilities, rather than solely relying on the maximum
output, has been extensively studied in the context of enhancing the safety,
robustness, and trustworthiness of machine learning models. By considering the
complete distribution of class predictions provided by the softmax output,
more reliable and informative prediction pipelines may be developed, that go
beyond point estimates [12]. This approach enables the exploration of
uncertainty quantification, anomaly detection, and other techniques that
contribute to building safer, more robust, and trustworthy autonomous systems.

Uncertainty quantification is a crucial aspect of reliable machine learning
systems, as it allows for the estimation of confidence in the model’s
predictions [26]. By leveraging the softmax probabilities, techniques such as
Monte Carlo dropout [12] and ensembling [28] can be employed to estimate the
model’s uncertainty. These methods help identify instances where the model is
less confident, enabling the system to defer to human judgment or take a more
conservative action in safety-critical scenarios [34].

Moreover, the softmax probabilities can be utilized for anomaly detection,
which is essential for identifying out-of-distribution (OOD) samples or novel
classes that the model has not encountered during training [21]. By monitoring
the softmax probabilities, thresholding techniques can be applied to detect
anomalies based on the distribution of the predictions [30]. This enables the
system to flag potentially problematic inputs and take appropriate actions,
such as requesting human intervention or triggering fallback mechanisms.

The use of softmax probabilities also facilitates the development of more
robust models that can handle adversarial examples and other types of input
perturbations [16]. Adversarial attacks aim to fool the model by crafting
input samples that lead to incorrect predictions with high confidence [47]. By
considering the entire softmax distribution, defensive techniques such as
adversarial training [33] and input transformations [19] can be applied to
improve the model’s robustness against these attacks.

Furthermore, the softmax probabilities provide valuable information for
interpretability and explanability of the model’s decisions [41]. By analyzing
the distribution of the predictions, insights can be gained into the model’s
reasoning process and the factors that contribute to its outputs. This
transparency is crucial for building trust in the system and facilitating
human-machine collaboration [7].

The importance of leveraging the entire softmax distribution extends to
various domains, including autonomous vehicles [34], medical diagnosis [29],
and financial risk assessment [11]. In these safety-critical applications, the
consequences of incorrect predictions can be severe, and relying solely on the
maximum softmax output may not provide sufficient safeguards. By considering
the full distribution of predictions, more informed and reliable decisions can
be made, reducing the risk of catastrophic failures.

However, the use of softmax probabilities is not without challenges. The
calibration of the model’s predictions is an important consideration, as
poorly calibrated models may lead to overconfident or underconfident estimates
[18]. Techniques such as temperature scaling [18] and isotonic regression [52]
can be applied to improve the calibration of the softmax probabilities,
ensuring that they accurately reflect the model’s uncertainty.

By considering the complete distribution of class predictions, techniques can
be employed to develop more reliable and informative prediction pipelines.

Clustering: Clustering algorithms are essential for discovering structures and
patterns in data across various domains [23, 51]. K-means, a widely used
algorithm, efficiently assigns data points to the nearest centroid and updates
centroids iteratively [31]. However, it requires specifying the number of
clusters and is sensitive to initial centroid placement [2]. Hierarchical
clustering creates a tree-like structure by merging or dividing clusters [25]
but may not scale well to large datasets [36]. Density-based algorithms, like
DBSCAN, identify clusters as dense regions separated by lower density areas
[10, 43].

Other applications include image segmentation for object detection [44],
anomaly detection for fraud and intrusion detection [5], customer segmentation
for targeted marketing [37], and bioinformatics for gene expression analysis
and disease subtype identification [9, 24]. The choice of algorithm depends on
data characteristics, desired cluster properties, and computational resources
[42].

Given the context and to the best of our knowledge, no prior work exists in
using softmax distance to class centroid to threshold trust in the model
predictive accuracy in classification tasks.

##  3 Clustering and Softmax Distance as a Confidence Metric

We consider a neural network output vector
$\mathbf{p}=(p_{1},p_{2},\dots,p_{K})$ where $\sum p_{i}=1$, representing a
probability distribution obtained by normalizing the logit vector
$\mathbf{z}=(z_{1},z_{2},\dots,z_{K})$ through the softmax function,
$p_{i}=\text{softmax}(z_{i})=e^{z_{i}}/\sum_{j=1}^{K}e^{z_{j}}$. For example,
given $\mathbf{p}=[0.01,0.01,0.01,0.01,0.9,0.01,0.01,0.01,0.01,0.01]$, the
predicted class is ’4’, corresponding to the highest value at index five,
reflecting the confidence of the prediction for each class from ’0’ to ’9’.
The logits, representing log-likelihoods of class memberships, are related to
probabilities by $z_{i}=\log(p_{i}/(1-p_{i}))$ where $z_{i}$ is the logit for
class $i$, and $p_{i}$ is the probability of the input belonging to class
$i$[15, 4].

We store the predictions for MNIST and CIFAR-10 datasets in a matrix
$\mathbf{M}\in\mathbb{R}^{n\times 12}$, where $n$ is the number of
predictions, the first ten columns are the softmax probabilities, column 11 is
the true class and column 12 is the predicted class.

To obtain cluster centroids $\mathbf{C}\in\mathbb{R}^{10\times 10}$ we
calculate the mean of all correct predictions from the training datasets with
Algorithm 1. To calculate the softmax distance threshold we use all incorrect
predictions with Algorithm 2.

Algorithm 1 K-Means Centroid Initialisation from Softmax Outputs

1:$correct\\_preds$: array of shape $(n,12)$, where $n$ is the number of
correct predictions

2:$centroids$: array of shape $(10,10)$, initialised centroids for each digit
class

3:$probs\\_dist\leftarrow corrects\\_preds[:,:10]$ $\triangleright$ Extract
probability distribution for each digit

4:$centroids\leftarrow\text{zeros}((10,10))$ $\triangleright$ Initialise
centroids array

5:for $digit\leftarrow 0$ to $9$ do

6:
$indices\leftarrow\text{where}(\text{argmax}(probs\\_dist,\text{axis}=1)==digit)[0]$
$\triangleright$ Find indices of rows where digit has highest probability

7: $centroid\leftarrow\text{mean}(probs\\_dist[indices],\text{axis}=0)$
$\triangleright$ Compute mean probability distribution for selected rows

8: $centroids[digit]\leftarrow centroid$ $\triangleright$ Assign centroid to
corresponding row in centroids array

9:end for

10:return $centroids$

Convolutional Neural Network: A Convolutional Neural Network (CNN) is used to
classify handwritten digits from the MNIST dataset, consisting of 60,000
training images and 10,000 testing images, each of size 28x28 grayscale
(single channel) pixels, representing digits from 0 to 9.

The CNN architecture, implemented using PyTorch, consists of two convolutional
layers followed by two fully connected layers. The first convolutional layer
has 16 filters with a kernel size of 3x3 and a padding of 1. The second
convolutional layer has 32 filters with the same kernel size and padding. Each
convolutional layer is followed by a ReLU activation function and a max-
pooling layer with a pool size of 2x2. The output of the second convolutional
layer is flattened and passed through two fully connected layers with 128 and
10 neurons, respectively. The final output is passed through a log-softmax
function to obtain the predicted class probabilities.

Algorithm 2 Find Minimum Softmax Distances to Centroids for Incorrectly
Predicted Digits (Threshold)

1:procedure FindMinDistances($data$)

2: $labels\leftarrow[0,1,2,3,4,5,6,7,8,9]$

3: $thresh\leftarrow\text{empty array of shape }(10,2)$

4: for $i\leftarrow 0\text{ to }9$ do

5: $label\leftarrow labels[i]$

6: $min\\_dist\leftarrow\min(data[data[:,1]==label,0])$

7: $thresh[i,0]\leftarrow min\\_dist$

8: $thresh[i,1]\leftarrow label$

9: end for

10: return $thresh$

11:end procedure

The model was trained using the Stochastic Gradient Descent (SGD) optimizer
with a learning rate of 0.01 and a batch size of 64. The learning rate was
determined using a custom learning rate function that decreases the learning
rate over time. The Cross-Entropy Loss function was used as the criterion for
optimization. The model was trained for 10 epochs.

The MNIST dataset was preprocessed using a transformation pipeline that
converted the images to PyTorch tensors and normalized the pixel values to
have a mean of 0.5 and a standard deviation of 0.5. The dataset was then
loaded using PyTorch’s DataLoader, for batch processing and shuffling of the
data.

The total number of parameters for the MNIST classification CNN is 206,922 and
took 5m6s to train on a Dell Precision Tower 5810 with a 6 core Intel Xeon
Processor and 32GB memory running Ubuntu 18.04.

Vision Transformer: The Vision Transformer (ViT) architecture [8] is
implemented using the Hugging Face Transformers [49] library. The model used
in this study, ’google/vit-base-patch16-224-in21k’ [50], is pre-trained on the
ImageNet-21k dataset [6], which contains 14 million labeled images. The pre-
trained model is then fine-tuned on the CIFAR-10 dataset. The ViT model
divides an input image into patches and processes them using a Transformer
encoder. The model used in this study has a patch size of 16x16 and an image
size of 224x224. The ViT model outputs a representation of the image, which is
then passed through a linear layer to obtain the final class probabilities.
The model has 86.4M parameters and is fine-tuned to classify images from the
CIFAR-10 dataset, consisting of 50,000 training images and 10,000 testing
images, each of size 32x32 pixels, representing 10 different classes:
airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

The data preprocessing pipeline involves on-the-fly data augmentation using
the torchvision library’s transforms module. The training data undergoes
random resized cropping, random horizontal flipping, conversion to a tensor,
and normalization. The validation and testing data are resized, center-
cropped, converted to a tensor, and normalized.

The model is trained using the Hugging Face Trainer API. The learning rate is
set to $2\times 10^{-5}$, the per-device train batch size is 10, the per-
device eval batch size is 4, and the number of training epochs is 3. The
weight decay is set to 0.01. The model is trained on Google Colab Pro, T4 GPU
hardware accelerator with high ram.

##  4 Experimental Results and Discussion

In this section, we present and discuss our results for CIFAR-10/ViT and
MNIST/CNN.

| Training Dataset Correct Predictions | Training Dataset Incorrect Predictions
---|---|---
Digit | Mean | Median | Min | Max | $\sigma$ | Count | Mean | Median | Min | Max | $\sigma$ | Count
0 | 0.0166 | 0.0088 | 0.0018 | 0.8433 | 0.0536 | 5890 | 0.9955 | 0.9552 | 0.7037 | 1.3681 | 0.1756 | 33
1 | 0.0154 | 0.0085 | 0.0020 | 0.7051 | 0.0486 | 6704 | 1.0912 | 1.0771 | 0.7247 | 1.3909 | 0.2197 | 38
2 | 0.0370 | 0.0208 | 0.0056 | 0.7453 | 0.0746 | 5859 | 1.0409 | 1.0460 | 0.6962 | 1.3912 | 0.2064 | 99
3 | 0.0352 | 0.0192 | 0.0055 | 0.7509 | 0.0754 | 6019 | 1.0522 | 1.0338 | 0.7139 | 1.3999 | 0.2143 | 112
4 | 0.0295 | 0.0166 | 0.0031 | 0.6933 | 0.0661 | 5755 | 1.0681 | 1.0642 | 0.7139 | 1.4010 | 0.2045 | 87
5 | 0.0298 | 0.0165 | 0.0024 | 0.7634 | 0.0693 | 5339 | 1.0572 | 1.0560 | 0.7066 | 1.3956 | 0.2000 | 82
6 | 0.0154 | 0.0081 | 0.0015 | 0.7981 | 0.0485 | 5884 | 1.0485 | 1.0702 | 0.7068 | 1.4015 | 0.2110 | 34
7 | 0.0302 | 0.0171 | 0.0028 | 0.7365 | 0.0678 | 6199 | 1.0133 | 0.9764 | 0.6960 | 1.3989 | 0.2160 | 66
8 | 0.0619 | 0.0363 | 0.0092 | 0.7788 | 0.0989 | 5581 | 1.0232 | 1.0083 | 0.6852 | 1.3816 | 0.1990 | 270
9 | 0.0399 | 0.0232 | 0.0048 | 0.7616 | 0.0781 | 5844 | 1.0576 | 1.0830 | 0.7179 | 1.3974 | 0.2201 | 105
| Testing Dataset Correct Predictions | Testing Dataset Incorrect Predictions
Digit | Mean | Median | Min | Max | $\sigma$ | Count | Mean | Median | Min | Max | $\sigma$ | Count
0 | 0.0141 | 0.0075 | 0.0021 | 0.7225 | 0.0506 | 977 | 1.0329 | 1.0185 | 0.7923 | 1.2880 | 0.2026 | 3
1 | 0.0126 | 0.0071 | 0.0023 | 0.6737 | 0.0445 | 1129 | 0.9823 | 0.9556 | 0.7278 | 1.2445 | 0.1898 | 6
2 | 0.0354 | 0.0202 | 0.0037 | 0.6697 | 0.0704 | 1015 | 1.0855 | 1.0725 | 0.7010 | 1.3833 | 0.1844 | 17
3 | 0.0331 | 0.0177 | 0.0049 | 0.7515 | 0.0765 | 998 | 0.9823 | 0.9752 | 0.7102 | 1.3230 | 0.2044 | 12
4 | 0.0306 | 0.0169 | 0.0047 | 0.6603 | 0.0678 | 972 | 0.9309 | 0.8725 | 0.7073 | 1.3110 | 0.1923 | 10
5 | 0.0313 | 0.0173 | 0.0051 | 0.6324 | 0.0692 | 883 | 1.1874 | 1.2976 | 0.8477 | 1.3982 | 0.2067 | 9
6 | 0.0172 | 0.0088 | 0.0029 | 0.7135 | 0.0545 | 941 | 1.0737 | 1.0242 | 0.7240 | 1.4009 | 0.2120 | 17
7 | 0.0295 | 0.0165 | 0.0036 | 0.6939 | 0.0678 | 1009 | 0.9665 | 0.8833 | 0.7148 | 1.3628 | 0.2016 | 19
8 | 0.0616 | 0.0372 | 0.0114 | 0.7328 | 0.0918 | 934 | 1.0513 | 1.0399 | 0.7252 | 1.3842 | 0.2022 | 40
9 | 0.0359 | 0.0206 | 0.0042 | 0.6870 | 0.0765 | 980 | 1.0502 | 1.0486 | 0.6968 | 1.3838 | 0.2197 | 29
Table 1: Statistical Summary of MNIST Softmax Output Distances to Centroids
for Different Datasets and Prediction Outcomes. The centroids are obtained
from Softmax outputs for correct class predictions from the training dataset.

The CNN/MNIST classifier took 5m6s to train. The accuracy on the training
dataset is 98.38% and 98.46% on the testing dataset. The ViT/CIFAR-10
classifier took 1h15m to train, registering 99.04% accuracy on the testing
dataset.

Softmax Distance to Class Centroid Statistics: We store the softmax outputs,
true and predicted labels for the MNIST and CIFAR-10 training datasets, we
obtain the mean values for all correctly predicted classes, use the values in
algorithm 1 to initialise the cluster centroids, then using the procedure
described in algorithm 2 and the same hardware used to train the MNIST
classifier CNN, generate K-Means clusters and compute statistics as presented
in Table 1 (for the MNIST results), a summary of the distances between the
softmax outputs of a model and the centroids for each digit class (0-9) in the
MNIST dataset. The centroids are calculated using the softmax outputs for
correct predictions from the training dataset. The table is divided into four
main sections: Training Dataset Correct Predictions, Training Dataset
Incorrect Predictions, Testing Dataset Correct Predictions, and Testing
Dataset Incorrect Predictions. For each digit and section, statistics are
displayed with respect to softmax distance from each class to its centroid:
the mean, median, minimum (highlighted in bold for the testing datasets),
maximum (highlighted in bold for the training datasets), standard deviation
($\sigma$), and count of instances are provided. We note that for the correct
predictions, the mean softmax distance to class centroid for digits zero, one
and six are lowest (network prediction accuracy is high) while for digit 8 the
mean softmax distance to class centroid is a comparatively larger value
(netrowk prediction accuracy is not as high). The significance of the bold
highlight for Max values in the Training section, and Min values in the
Correct Prediction sections, is that the Min values in the Incorrect
Predictions sections, is that the latter is effectively the Threshold we
propose be used as a boundary for accepting the class prediction as accurate.
Anything softmax distances equal or above the threshold will be considered not
safe to use. It becomes evident by comparing both columns, that for the
training dataset, there is a significant overlap between Max and Min columns,
however, the testing dataset presents an interesting property in that there is
less of an overlap. The significance of the overlap is that any correct
predictions within the overlap will be tagged as unsafe, for the scheme we
propose to hold, which means more manual classification and human intervention
on the downside, and on the upside a high likelihood that every prediction
with softmax distance below the threshold is accurate.

In practice a small number of samples, 0.04%, consistent across both CIFAR-10
and MNIST testing dataset prediction, are found in the overlap, as shown later
in Tables 2 and 3. Also shown in Table 1 are the softmax distance standard
deviations and counts for correct and incorrect classifications.

Predictive Accuracy and Softmax Distance to Class Centroid: The confusion
matrices in Figure 1 provide some insights into what are the most likely MNIST
misclassifications. We note that number zero is the digit less likely, and
number eight is the digit most likely to be misclassified. Other digits that
are less likely to be misclassified are digits one and; six in the training
dataset, and five in the testing dataset. Digit eight is likely to be
misclassified as six or nine, while digit three is likely to be misclassified
as two, five or nine.

Figure 1: Please zoom in for detail. Confusion matrices for the MNIST
classification model on the training dataset (left) and testing dataset
(right). The matrices display the true labels on the vertical axis and the
predicted labels on the horizontal axis. The diagonal elements represent
correctly classified instances, while the off-diagonal elements indicate
misclassifications.

The confusion matrices for the CIFAR-10 dataset in Figure 2 illustrate the
performance of a classification model on both the training and testing sets.
For the training set, the model exhibits a strong diagonal, indicative of high
classification accuracy for most classes. Nonetheless, certain classes show
notable confusion, such as ’cat’ with ’dog’ and ’deer’ with ’frog’, and the
confusion is reflected in the class softmax distance to cluster centroid.

Examining the testing dataset, the model achieves good classification success,
with perfect recognition of ’frog’ class. Misclassifications are present but
considerably reduced compared to the training dataset, particularly for pairs
such as ’truck’ and ’automobile’, as well as ’cat’ and ’dog’. The reduced
confusion in the test set is also reflected in the softmax distance to cluster
centroid.

Figure 2: Please zoom in for detail. Confusion matrices for the CIFAR-10
classification model on the training dataset (left) and testing dataset
(right).

Figure 3 suggests how the trained neural network model achieves higher
predictive accuracy for classes that have lower softmax distance to their
respective class centroids, compared to classes for which the model has lower
predictive accuracy.

The scatter plot shows the relationship between classification accuracy and
mean class distance to the centroid for the MNIST dataset. Training data (blue
dots) and testing data (green dots) are each fitted with a linear regression
line, demonstrating a negative correlation where increased mean distance
corresponds to decreased accuracy. The steeper slope of the training data fit
(-0.806) compared to the testing data fit (-0.677) reflects the greater
accuracy and more compact cluster obtained from the correct test predictions
compared to the cluster obtained from correct training predictions.

These results show an inverse relationship: as the mean distance from the data
points in a class to their centroid increases, the propensity for correct
classification diminishes. This could be due to the spread of data points
within a class – greater distances from the centroid reflect a larger variance
within the class, potentially making it more challenging for the classifier to
identify the defining characteristics of each class accurately.

The CIFAR-10 linear fit is steeper as can be observed by the y axis values,
although it can be argued provides a better fit.

Figure 3: Please zoom in for detail. Expected accuracy linear fit based on
prediction softmax distance to class centroid. MNIST fit is on the left,
CIFAR-10 is on the right.

Overlap between maximum and minimum softmax distances to class centroids:
Figure 4 presents box plots showing the distribution of distances to centroids
for correctly and incorrectly classified instances in each class of the
CIFAR-10 training and testing datasets. In the training dataset, the distances
to centroids for correctly classified instances (green boxes) are generally
lower than those for incorrectly classified instances (red boxes), indicating
that correctly classified instances tend to be closer to their respective
class centroids in the feature space. However, there is a small overlap
between the distances of correctly and incorrectly classified instances,
particularly in classes such as cat, dog, and deer. The testing dataset
exhibits a similar trend, with correctly classified instances having lower
distances to centroids compared to incorrectly classified instances. The
overlap between the two groups is less pronounced in the testing dataset, such
as in classes like frog. Note there is no frog boxplot for incorrect
classifications, as reflected in the confusion matrix.

Figure 4: Distribution of Distances to Centroids for Correctly and Incorrectly
Classified Instances in Training and Testing CIFAR-10 data, where training
data is on the left and testing data is displayed on the right. Distances on y
axis are shown on a logarithmic scale. Note, centroids are obtained from
correctly classified training examples, then used for both training and
testing datasets, a cluster is not created from the testing softmax distances.

The side-by-side boxplots convey the distributions of distances to centroids
for both the training and testing sets, categorized by class. These distances
are represented on a logarithmic scale, facilitating the visualization of a
wide range of values. For both sets, there is a clear pattern where the
correctly classified instances tend to have a smaller median distance to the
class centroid compared to the incorrectly classified ones. This observation
is consistent across all classes, suggesting that instances closer to the
centroid of their respective class in the feature space are more likely to be
classified correctly by the model. The notable overlap in the interquartile
ranges, however, indicates that while distance to the centroid is an
informative metric, it is not solely determinative of classification accuracy.

The incorrectly classified instances exhibit greater variance in distances to
the centroid, as evidenced by the longer whiskers and the presence of
outliers. This variability implies that misclassified instances can sometimes
be far off from the core cluster of their true class, potentially falling
closer to the regions dominated by other classes in the feature space. This is
indicative of boundary cases or instances that bear a resemblance to multiple
categories. The consistency of these trends between the training and testing
sets indicates that the distance to the centroid is a robust indicator of
classification difficulty across the model’s application. The presence of some
correctly classified instances at higher distances suggests that other factors
also influence classification accuracy, such as the density of data points in
the surrounding feature space, class overlap, or specific intra-class
variabilities.

Digit | Train Data - Train Centroids | Test Data - Train Centroids
---|---|---
| Count | Total | Overlap | Count | Total | Overlap
0 | 2 | 5890 | 0.03% | 1 | 977 | 0.10%
1 | 0 | 6704 | 0.00% | 0 | 1129 | 0.00%
2 | 1 | 5859 | 0.02% | 0 | 1015 | 0.00%
3 | 3 | 6019 | 0.05% | 1 | 998 | 0.10%
4 | 0 | 5755 | 0.00% | 0 | 972 | 0.00%
5 | 5 | 5339 | 0.09% | 0 | 883 | 0.00%
6 | 2 | 5884 | 0.03% | 1 | 941 | 0.11%
7 | 2 | 6199 | 0.03% | 0 | 1009 | 0.00%
8 | 12 | 5581 | 0.22% | 1 | 934 | 0.11%
9 | 1 | 5844 | 0.02% | 0 | 980 | 0.00%
Totals | 28 | 59074 | 0.05% | 4 | 9838 | 0.04%
Table 2: MNIST counts of correctly classified training and testing image predictions and counts with distance to centroids equal or above error threshold. Class | Train Data - Train Centroids | Test Data - Train Centroids
---|---|---
| Count | Total | Overlap | Count | Total | Overlap
plane | 13 | 4252 | 0.31% | 1 | 990 | 0.10%
auto | 1 | 4326 | 0.02% | 0 | 989 | 0.00%
bird | 9 | 4216 | 0.21% | 0 | 993 | 0.00%
cat | 20 | 4103 | 0.49% | 2 | 973 | 0.21%
deer | 14 | 4255 | 0.33% | 0 | 991 | 0.00%
dog | 11 | 4157 | 0.26% | 1 | 967 | 0.10%
frog | 11 | 4310 | 0.26% | 0 | 1000 | 0.00%
horse | 12 | 4223 | 0.28% | 0 | 991 | 0.00%
ship | 11 | 4258 | 0.26% | 0 | 998 | 0.00%
truck | 10 | 4251 | 0.24% | 0 | 981 | 0.00%
Totals | 112 | 42351 | 0.26% | 4 | 9873 | 0.04%
Table 3: CIFAR10 counts of correctly classified training and testing image
predictions and counts with distance to centroids equal or above error
threshold.

Table 2 presents an analysis of the counts and percentages of correctly
classified images in the training and testing datasets that have softmax
distances to their respective class centroids above a certain threshold. The
threshold is determined by the nearest softmax distance to the centroid for
the incorrectly classified classes.

The table is divided into two main sections: "Train Data - Train Centroids"
and "Test Data - Train Centroids." For each section, the table provides the
count of images that satisfy the threshold condition, the total number of
images in each class, and the percentage of overlap (i.e., the proportion of
images above the threshold relative to the total count).

Looking at the "Train Data - Train Centroids" section, we observe that the
overlap percentages are relatively low, ranging from 0.00% to 0.22%. This
suggests that only a small fraction of the correctly classified training
images have softmax distances to their centroids above the threshold set by
the incorrectly classified images. The highest overlap is observed for digit
class 8, with 12 out of 5,581 images (0.22%) having distances above the
threshold.

Similarly, in the "Test Data - Train Centroids" section, the overlap
percentages are even lower, ranging from 0.00% to 0.11%. This indicates that
the trained model is able to correctly classify the majority of the test
images while maintaining a distance to the centroids below the threshold set
by the incorrectly classified images. The highest overlap in the test data is
observed for digit classes 6 and 8, with 1 out of 941 (0.11%) and 1 out of 934
(0.11%) images, respectively, having distances above the threshold.

The Totals summarise the overall counts and overlap percentages across all
digit classes. For the training data, 28 out of 59,074 images (0.05%) have
distances above the threshold, while for the test data, only 4 out of 9,838
images (0.04%) exceed the threshold.

The low overlap percentages indicate that the majority of the correctly
classified images are well-separated from the incorrectly classified images in
terms of their softmax distances to the centroids. This analysis provides
insights into the softmax distance being a proxy to distinguish between
between correct and incorrect digit classifications based on the proximity of
the images to their respective class centroids in the softmax distance space.
The same analysis applies to Table 3. Most indicative of the consistency of
the suggested approach is than both CIFAR-10 and MNIST present an overlap of
0.04% for correctly classified images at or above threshold. Note we tag the
images in the overlap as incorrectly classified, for the sake of assuring that
all softmax distances to cluster centroid below threshold are correct
classifications.

Accuracy decrease vs threshold decrease: Having observed that the percentage
of correctly classified images that have softmax distances equal or above
threshold is 0.04% for both CIFAR-10 and MNIST testing datasets, we ask how
lowering the threshold affects the accuracy. Figure 5 reveals that for MNIST
(left plot) classes accurately classified by the CNN model, such as zero and
one, even setting the threshold at 10% of the original class thresholds (0.9
on the x axis), still present accuracies greater than 0.97, meaning the class
clusters are tightly placed near the centroids, while for digit 8, the most
misclassified digit, had the threshold been set at 10% of the original value,
the accuracy would expect to drop to 0.84, meaning 16% of the predictions
would be deferred to human judgement, and 84% would be considered 100%
accurate..

Predictions with softmax distance to class centroid below the class threshold
are expected to be 100% accurate, class predictions with distances equal or
greater than threshold are then left to human judgement.

Figure 5: Please zoom in for detail. Expected accuracy decrease as a result of
threshold decrease. MNIST data is on the left, CIFAR-10 is on the right. The x
axis shows the threshold decrement in factors of 0.1, that is, at 0.1 the
threshold is 90% of the original threshold while at 0.9 the threshold is 10%
of the original threshold and consequently neared to the class centroid.
Figure 6: Please zoom in for detail. Average Softmax Probabilities for
Correctly and Incorrectly Classified Digits in the MNIST Testing Dataset.

Figure 6 shows the average softmax probabilities for each digit class (0 to 9)
in the MNIST training dataset, separated into correctly classified instances
(top row) and incorrectly classified instances (bottom row). The probabilities
are displayed on a logarithmic scale. For correctly classified digits, the
highest average probability is observed for the corresponding true digit
class, indicating strong confidence in the correct predictions. In contrast,
for incorrectly classified digits, the average probabilities are more evenly
distributed across different digit classes, suggesting lower confidence and
potential confusion between similar-looking digits. Figure 7 is the
corresponding plot for the MNIST testing dataset.

Figure 7: Please zoom in for detail. Average Softmax Probabilities for
Correctly and Incorrectly Classified Digits in the MNIST Testing Dataset.
Figure 8: Please zoom in for detail. Average Softmax Probabilities for
Correctly and Incorrectly Classified Classes in the CIFAR-10 Training Dataset
Figure 9: Please zoom in for detail. Average Softmax Probabilities for
Correctly and Incorrectly Classified classes in the CIFAR-10 Testing Dataset

For the CIFAR-10 we observe similar results. For correctly classified
instances, the average softmax value corresponding to the true digit class is
significantly higher compared to the softmax values of the other digit
classes. This observation holds true across all digit classes in both the
testing and training datasets. The pronounced difference in magnitudes
indicates that the model assigns a high level of confidence to the correct
predictions, with the predicted probability for the true class being orders of
magnitude larger than the probabilities assigned to the other classes.

In contrast, for incorrectly classified instances, the average softmax values
exhibit a more even distribution across the digit classes. While the softmax
value for the predicted class is still relatively high, the softmax values for
the other classes are notably closer in magnitude, as evident from the shorter
bars on the logarithmic scale. This suggests that when the model makes an
incorrect prediction, it assigns more substantial probabilities to multiple
classes, indicating a higher level of uncertainty or confusion in the
decision-making process. Figures 8 and 9 bar charts show the average softmax
distributions for correct (green/blue) and incorrect (red) class predictions
in both the testing and training CIFAR-10 datasets.

Comparing the testing and training datasets, we observe similar patterns in
the softmax distance distributions for both correct and incorrect predictions.
The consistency of these patterns suggests that our approach is consistent
across datasets and models.

The analysis of softmax distance distributions may be taken as an insight into
the model’s decision-making process. The difference in the softmax values
between correct and incorrect predictions highlights the model’s ability to
discriminate between classes when it makes accurate predictions. On the other
hand, the more evenly distributed softmax values for incorrect predictions
suggest that the model struggles to make clear distinctions between classes in
those cases, assigning considerable probabilities to multiple digits, where
our proposed thresholding method may be an additional tool to help evaluate
predictions in different settings.

##  5 Conclusions and Future Work

We presented a simple approach to thresholding accuracy in an image
classification setting, by training two distinct network architectures, on two
distinct datasets, and showing our results are consistent across both
scenarios. We demonstrated the steps required to effectively create class
clusters by initialising centroids with the mean of correct class predictions
for each distinct class, where the K-Means algorithm quickly converges with
good cluster separation. A threshold was proposed to ensure predictions below
threshold are safe, at a small cost of labelling as incorrect a small number
of correct predictions at or above threshold.

We posit that the optimal threshold is domain-dependent and should be
determined by domain experts who are equipped to balance the efficiency of
automated decision-making against the necessity for time-intensive human
judgment, taking into account the specific stakes involved.

We are applying the concepts discussed in this study to autonomous system
safety, in the context of self-driving cars using the CARLA simulator, and
constituent tasks in the decision-making stack, such as image classification
and segmentation, where our methodology aims to enhance system safety by
identifying scenarios, including OOD scenarios, where human judgment is
preferable to the autonomous system’s assessment.

Finally, we are considering the softmax output as a training dataset in
itself, for a simple multi-layer perceptron binary classifier, to compare and
contrast the softmax distance approach, and a regressor network combined with
our current approach, to provide a quantity e.g. sigmoid function output as a
weight to further study optimal threshold values.

## References

  * Amodei et al. [2016] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané.  Concrete problems in ai safety.  _arXiv preprint arXiv:1606.06565_ , 2016.
  * Arthur and Vassilvitskii [2007] D. Arthur and S. Vassilvitskii.  k-means++: The advantages of careful seeding.  In _Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms_ , pages 1027–1035, 2007.
  * Baena-García et al. [2006] M. Baena-García, J. del Campo-Ávila, R. Fidalgo, A. Bifet, R. Gavaldà, and R. Morales-Bueno.  Early drift detection method.  In _Fourth International Workshop on Knowledge Discovery from Data Streams_ , volume 6, pages 77–86, 2006.
  * Bishop [2006] C. M. Bishop.  Pattern recognition and machine learning.  _Springer_ , 2:645–678, 2006.
  * Chandola et al. [2009] V. Chandola, A. Banerjee, and V. Kumar.  Anomaly detection: A survey.  _ACM computing surveys (CSUR)_ , 41(3):1–58, 2009.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Doshi-Velez and Kim [2017] F. Doshi-Velez and B. Kim.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_ , 2017.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Eisen et al. [1998] M. B. Eisen, P. T. Spellman, P. O. Brown, and D. Botstein.  Cluster analysis and display of genome-wide expression patterns.  _Proceedings of the National Academy of Sciences_ , 95(25):14863–14868, 1998.
  * Ester et al. [1996] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In _Kdd_ , volume 96, pages 226–231, 1996.
  * Feng et al. [2018] G. Feng, J. He, and N. G. Polson.  Deep learning-based quantitative trading strategies for stock markets.  _arXiv preprint arXiv:1805.01039_ , 2018.
  * Gal and Ghahramani [2016] Y. Gal and Z. Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  _International Conference on Machine Learning_ , pages 1050–1059, 2016.
  * Gama et al. [2014] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia.  A survey on concept drift adaptation.  _ACM Computing Surveys (CSUR)_ , 46(4):1–37, 2014.
  * Ganin et al. [2016] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky.  Domain-adversarial training of neural networks.  _The Journal of Machine Learning Research_ , 17(1):2096–2030, 2016.
  * Goodfellow et al. [2016] I. Goodfellow, Y. Bengio, and A. Courville.  _Deep learning_.  MIT press, 2016.
  * Goodfellow et al. [2014] I. J. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  In _International Conference on Learning Representations_ , 2014.
  * Gretton et al. [2012] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773, 2012.
  * Guo et al. [2017] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger.  On calibration of modern neural networks.  In _International Conference on Machine Learning_ , pages 1321–1330. PMLR, 2017.
  * Guo et al. [2018] C. Guo, M. Rana, M. Cisse, and L. Van Der Maaten.  Countering adversarial images using input transformations.  In _International Conference on Learning Representations_ , 2018.
  * Hendrycks and Dietterich [2019] D. Hendrycks and T. Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _arXiv preprint arXiv:1903.12261_ , 2019.
  * Hendrycks and Gimpel [2017] D. Hendrycks and K. Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _International Conference on Learning Representations_ , 2017.
  * Hendrycks et al. [2021] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8340–8349, 2021.
  * Jain [2010] A. K. Jain.  Data clustering: 50 years beyond k-means.  _Pattern recognition letters_ , 31(8):651–666, 2010.
  * Jiang et al. [2004] D. Jiang, C. Tang, and A. Zhang.  Cluster analysis for gene expression data: A survey.  _IEEE Transactions on knowledge and data engineering_ , 16(11):1370–1386, 2004.
  * Johnson [1967] S. C. Johnson.  Hierarchical clustering schemes.  _Psychometrika_ , 32(3):241–254, 1967.
  * Kendall and Gal [2017] A. Kendall and Y. Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Kullback and Leibler [1951] S. Kullback and R. A. Leibler.  On information and sufficiency.  _The Annals of Mathematical Statistics_ , 22(1):79–86, 1951.
  * Lakshminarayanan et al. [2017] B. Lakshminarayanan, A. Pritzel, and C. Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _Advances in Neural Information Processing Systems_ , pages 6402–6413, 2017.
  * Leibig et al. [2017] C. Leibig, V. Allken, M. S. Ayhan, P. Berens, and S. Wahl.  Leveraging uncertainty information from deep neural networks for disease detection.  _Scientific Reports_ , 7(1):1–14, 2017.
  * Liang et al. [2018] S. Liang, Y. Li, and R. Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In _International Conference on Learning Representations_ , 2018.
  * Lloyd [1982] S. Lloyd.  Least squares quantization in pcm.  _IEEE transactions on information theory_ , 28(2):129–137, 1982.
  * Lu et al. [2018] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang.  Learning under concept drift: A review.  _IEEE Transactions on Knowledge and Data Engineering_ , 31(12):2346–2363, 2018.
  * Madry et al. [2017] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations_ , 2017.
  * Michelmore et al. [2018] R. Michelmore, M. Kwiatkowska, and Y. Gal.  Evaluating uncertainty quantification in end-to-end autonomous driving control.  _arXiv preprint arXiv:1811.06817_ , 2018.
  * Moreno-Torres et al. [2012] J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla, and F. Herrera.  A unifying view on dataset shift in classification.  _Pattern Recognition_ , 45(1):521–530, 2012.
  * Müllner [2011] D. Müllner.  Modern hierarchical, agglomerative clustering algorithms.  _arXiv preprint arXiv:1109.2378_ , 2011.
  * Ngai et al. [2009] E. W. Ngai, L. Xiu, and D. C. Chau.  Application of data mining techniques in customer relationship management: A literature review and classification.  _Expert systems with applications_ , 36(2):2592–2602, 2009.
  * Pan and Yang [2009] S. J. Pan and Q. Yang.  A survey on transfer learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 22(10):1345–1359, 2009.
  * Patel et al. [2015] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa.  Visual domain adaptation: A survey of recent advances.  _IEEE Signal Processing Magazine_ , 32(3):53–69, 2015.
  * Quiñonero-Candela et al. [2009] J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence.  Dataset shift in machine learning.  _The MIT Press_ , 2009.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  "why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 1135–1144, 2016.
  * Rodriguez et al. [2019] M. Z. Rodriguez, C. H. Comin, D. Casanova, O. M. Bruno, D. R. Amancio, L. d. F. Costa, and F. A. Rodrigues.  Clustering algorithms: A comparative approach.  _PloS one_ , 14(1):e0210236, 2019.
  * Schubert et al. [2017] E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu.  Dbscan revisited, revisited: why and how you should (still) use dbscan.  _ACM Transactions on Database Systems (TODS)_ , 42(3):1–21, 2017.
  * Shi and Malik [2000] J. Shi and J. Malik.  Normalized cuts and image segmentation.  _IEEE Transactions on pattern analysis and machine intelligence_ , 22(8):888–905, 2000.
  * Shimodaira [2000] H. Shimodaira.  Improving predictive inference under covariate shift by weighting the log-likelihood function.  _Journal of Statistical Planning and Inference_ , 90(2):227–244, 2000.
  * Sugiyama et al. [2007] M. Sugiyama, M. Krauledat, and K.-R. Müller.  Covariate shift adaptation by importance weighted cross validation.  _The Journal of Machine Learning Research_ , 8:985–1005, 2007.
  * Szegedy et al. [2013] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus.  Intriguing properties of neural networks.  In _International Conference on Learning Representations_ , 2013.
  * Wang and Deng [2018] M. Wang and W. Deng.  Deep visual domain adaptation: A survey.  _Neurocomputing_ , 312:135–153, 2018.
  * Wolf et al. [2020] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush.  Huggingface’s transformers: State-of-the-art natural language processing, 2020.
  * Wu et al. [2020] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda.  Visual transformers: Token-based image representation and processing for computer vision, 2020.
  * Xu and Tian [2015] D. Xu and Y. Tian.  A comprehensive survey of clustering algorithms.  _Annals of Data Science_ , 2(2):165–193, 2015.
  * Zadrozny and Elkan [2002] B. Zadrozny and C. Elkan.  Transforming classifier scores into accurate multiclass probability estimates.  In _Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 694–699, 2002.