# A Holistic Assessment of the Reliability of
Machine Learning Systems

thony Corso<EMAIL_ADDRESS>
Aeronautics and Astronautics, Stanford University,
Stanford, CA 94305, USA vid Karamadian<EMAIL_ADDRESS>
Statistics, Stanford University,
Stanford, CA 94305, USA meo Valentin<EMAIL_ADDRESS>
Aeronautics and Astronautics, Stanford University,
Stanford, CA 94305, USA ry Cooper<EMAIL_ADDRESS>
Aeronautics and Astronautics, Stanford University,
Stanford, CA 94305, USA kel J. Kochenderfer<EMAIL_ADDRESS>
Aeronautics and Astronautics, Stanford University,
Stanford, CA 94305, USA

###### Abstract

As machine learning (ML) systems increasingly permeate high-stakes settings
such as healthcare, transportation, military, and national security, concerns
regarding their reliability have emerged. Despite notable progress, the
performance of these systems can significantly diminish due to adversarial
attacks or environmental changes, leading to overconfident predictions,
failures to detect input faults, and an inability to generalize in unexpected
scenarios. This paper proposes a holistic assessment methodology for the
reliability of ML systems. Our framework evaluates five key properties: in-
distribution accuracy, distribution-shift robustness, adversarial robustness,
calibration, and out-of-distribution detection. A reliability score is also
introduced and used to assess the overall system reliability. To provide
insights into the performance of different algorithmic approaches, we identify
and categorize state-of-the-art techniques, then evaluate a selection on real-
world tasks using our proposed reliability metrics and reliability score. Our
analysis of over $500$ models reveals that designing for one metric does not
necessarily constrain others but certain algorithmic techniques can improve
reliability across multiple metrics simultaneously. This study contributes to
a more comprehensive understanding of ML reliability and provides a roadmap
for future research and development.

##  1 Introduction

Rapid progress in the capabilities of machine learning (ML) technology has
prompted its nascent or proposed use in high-stakes settings such as
transportation [240, 278], healthcare [302, 217], military [252], and national
security [268]. Despite this progress, ML components still suffer from various
reliability issues, which compromise the ability for the system to perform its
intended function during deployment. Adversarial or natural environmental
changes can result in significant drops in performance, while poor calibration
of uncertainty estimates can lead to overconfident predictions even in
scenarios where accuracy is compromised. In the extreme case, ML components
fail to detect input faults that should cause them to cease operation. So how
should we evaluate ML models?

Recent guidance and regulation on artificial intelligence (AI) systems [259,
255, 185, 186, 153], which are typically enabled by ML technology, have
outlined several key requirements for ensuring their safety, fairness, and
security. While guidance and regulations vary, we summarize many of the key
points addressed as follows. AI systems must have a well-defined objective and
produce accurate, reliable outputs that generalize well and be robust to new
scenarios. In the event of unexpected scenarios, AI systems must fail
gracefully and minimize harm. To ensure safety, AI systems should undergo
rigorous validation to identify failure modes and corresponding harms and be
regularly audited for performance. To promote fairness, AI systems must be
free from bias in data, algorithms, and human operators. Additionally, they
should be secure, resilient, and protect privacy, through robustness against
adversarial attacks and protection against sensitive data exfiltration. The
design choices and decisions made by AI systems must be transparent and
explainable, promoting trust among users. Lastly, there should be an
accountable party to take responsibility when a bad outcome is realized.

While some of these requirements need to be addressed at an organizational
level, such as appropriate selection of objectives, auditing, transparency,
and accountability, and others are addressed by traditional engineering
safeguards such as data privacy and system-level security, the remaining
requirements boil down to properties of the ML components themselves. Accuracy
is a commonly used training objective and evaluation criterion for ML
components, while generalization and robustness can be assessed through
accuracy on diverse datasets that include distribution shifts, which are
variations of the input data not captured by the training data. Adversarial
robustness, measured by the ease with which performance is degraded by
manipulating its inputs, is an important aspect of security. The ability of an
AI system to fail gracefully depends on the uncertainty calibration and fault-
detection capabilities of its ML components. These properties have been
extensively studied, leading to advancements in improving each property in
isolation, but often without measuring the effect on others. To build reliable
AI systems, we need to evaluate the reliability of ML components holistically,
considering all relevant metrics together. A holistic approach will provide a
clearer understanding of progress toward building AI systems that we can
safely deploy.

To that end, we propose a holistic assessment methodology of the reliability
of machine learning systems. Guided by emerging regulations, we select five
properties that are important for building reliable ML systems:

  1. 1.

In-distribution (ID) accuracy: The accuracy on unseen data that comes from the
same distribution as the training data.

  2. 2.

Distribution-shift (DS) robustness: The ability to generalize to data that
comes from a different distribution than the training data.

  3. 3.

Adversarial robustness: The ability to generalize to data that has been
adversarially manipulated to degrade model performance.

  4. 4.

Calibration: The accuracy of uncertainty estimates of the systems.

  5. 5.

Out-of-distribution (OOD) detection: The ability to flag inputs that are out
of the system’s operational domain.

We identify quantitative metrics for each property and use them to construct a
property score ranging from $0$ to $1$. A final holistic reliability (HR)
score is obtained by averaging the five property scores. We use this
methodology to evaluate the many recent advances in building highly reliable
ML systems. We survey the literature to identify state-of-the-art (SOTA)
approaches to building highly accurate, robust, secure, calibrated, and fault-
tolerant ML components. We group approaches based on where they fall within
the ML system design cycle, which includes training data, representation
learning, training objective, model architecture, model selection, post-hoc
calibration, and run-time monitoring. We then select a representative set of
approaches and use them to train models on three real-world tasks. These
models are combined with other models from prior research trained on the same
tasks, and each is evaluated according to the aforementioned reliability
metrics and compared.

Our analysis of over $500$ models reveals several key insights into the
reliability of ML models. First, we find that these five reliability metrics
are largely independent of each other, with only a few exceptions that appear
to be dataset specific. This finding is consistent with the hypothesis that
deep neural networks are underspecified [179], meaning that designing to one
metric does not constrain others due to the high number of degrees of freedom
of modern models. Second, we find that some algorithmic approaches improve
reliability across multiple metrics simultaneously. For example, we find that
additional training data and model ensembling and fine-tuning from large pre-
trained models all improve the holistic reliability score.

To summarize, in this work we make the following contributions:

  * •

We survey the literature on building reliable ML systems and identify and
categorize SOTA approaches.

  * •

We propose a new methodology for the holistic assessment of the reliability of
ML systems that combines metrics related to in-distribution performance,
distribution-shift robustness, adversarial robustness, calibration, and out-
of-distribution detection.

  * •

We evaluate a representative set of SOTA ML techniques on a variety of real-
world datasets. Through these experiments, we identify several technical
approaches that improve the holistic reliability of ML systems independent of
tasks.

##  2 A Survey of Reliable Machine Learning

We organize approaches for building more reliable ML systems into where they
are applied during the design of a ML component. We summarize our
categorization in fig. 1, and expand in further detail in the following
sections. We focus on systems that are trained offline and then deployed,
rather than systems that adapt online, which we leave for future work. Systems
that adapt online are more difficult to assess because they need to either 1)
be reassessed each time they change, or 2) come with safety or reliability
guarantees of the adaptation process itself.

2.1 Training Data 2.2 Model Architecture 2.3 Pre Training 2.4 Training
Procedure 2.5 Post Training 2.6 Model Deployment • Data Coverage and Bias •
Data Sculpting • Data Augmentation • Network Structure • Maximum Likelihood
vs. Bayesian • Ensembling • Supervised • Weakly Supervised • Contrastive •
Generative • Sub-Group Aware • Self-Training • Auxiliary Losses • Fine-Tuning
• Model Selection • Model Calibration • Out-of-Distribution Detection

Figure 1: A categorization of approaches for reliable ML.

###  2.1 Training Data

The quality, diversity, and representativeness of training data can
significantly impact the capabilities and generalization of ML systems.
Adequate and diverse training data allows models to learn robust patterns,
capture complex relationships, and handle various real-world scenarios.
Insufficient or biased training data can lead to suboptimal performance,
limited generalization, and potential ethical concerns. Therefore, ensuring
the availability of high-quality, diverse, and unbiased training data is
crucial for developing effective and fair ML systems [236].

#### Data Coverage and Bias

Selecting an initial subset of data that is representative of the entire
dataset is a critical step in creating a reliable ML model. The choice of
dataset, which should reflect the model’s intended use case, plays a crucial
role in the performance and robustness of the model. [236] advocate for an
iterative data design process, typically starting with a pilot dataset to
build a prototype model and then collecting more data to correct observed
errors or expand the model’s scope. [174] underscores the importance of
dataset size, finding that larger datasets generally enhance the performance
of large pre-trained models. However, [254] demonstrates that the content of
the pre-training dataset significantly affects the learning of robust models,
reinforcing the necessity for quality and diversity in the data. Despite this,
biases or insufficient data coverage can corrupt initial dataset selection,
potentially leading to issues when the ML model is deployed, as seen in
healthcare applications [258], facial recognition models [170], natural
language processing [168, 171], and in many other applications [248].
Therefore, throughout this process, thorough documentation of data provenance
is crucial, particularly noting biases and the socio-technical context of the
data where applicable [269].

#### Data Sculpting

Data sculpting encompasses the selection, cleaning, annotation, and evaluation
of data to enhance a model’s reliability and generalizability [236]. Despite
the importance of data annotation, it remains a source of significant error in
AI models, with error rates such as 0.15% for MNIST, 6% for ImageNet, and
10.12% for QuickDraw [257]. Mitigation strategies include customized
annotation approaches combining data programming [264] and manual labeling or
the adoption of weak supervision or unsupervised learning. The value of
individual data points can be computed using Shapley values [193], which can
be used to quantify data usefulness and filter out noisy, low-quality data.
Data sculpting also extends to evaluating model effectiveness and
trustworthiness using techniques like data ablation. Data ablation is a
process for detecting random correlations a model might use for predictions by
testing how the model responds to missing key information, such as certain
image details. For instance, researchers might obscure a polar bear in an
image to see if the model predicts the animal’s Arctic nativity based solely
on the snowy background [236].

#### Data Augmentation

Data augmentation is a widely used practice in deep learning that expands a
fixed training set by generating additional samples through various
techniques. It offers several benefits, including the expansion of available
training data, promotion of model invariance, regularization through
consistent predictions for augmented samples, and improved generalization
[191]. Early examples of data augmentation focused on simple geometric
transformations [298]. Model-free methods [296] modify inputs through
transformations [211, 178], combining datapoints [283, 250], or using image-
to-image networks [208]. Model-based methods, such as incorporating synthetic
examples of rare species [163], and policy-based methods such as adversarial
training [241] can also be used. These techniques have demonstrated
performance improvements across various metrics, such as distribution-shift
robustness [267, 209, 282], expected calibration error (ECE) [283], and out-
of-distribution detection [283]. Adversarial training, employing techniques
like Projected Gradient Descent (PGD), has also been successful in enhancing
adversarial robustness against attacks, demonstrating significant improvements
over baseline models [215, 241, 303]. While these techniques can enhance
adversarial robustness, there is currently no known method that fully defends
against adversarial attacks [295].

###  2.2 Model Architecture

Obtaining a high quality dataset is only one part of creating a good ML model.
Another key design decision is the type of model to use. While many types of
models exist, recently deep neural networks have been at the forefront. Deep
neural networks are commonly used for processing text and image data but there
are a large number of design considerations for choosing their architecture.
In this section, we discuss the evolution of convolutional and transformer
architectures, the advantages and challenges of Bayesian neural networks, and
the power of ensembling diverse models.

#### Network Structure

Convolutional neural networks (CNNs) initially emerged as the dominant network
structure for image recognition tasks, following their early success over
dense layers. However, recent advancements have introduced transformer
architectures like the Vision Transformer (ViT) [183, 162], Swin Transformer
[238], and MaxViT [286], which have shown remarkable performance in language
and image processing tasks, particularly with large datasets. Nevertheless,
convolutional architectures such as ConvNeXt [239] and EfficientNet [281] have
re-emerged to compete with transformers on the ImageNet benchmark. When
investigating model architectures, [222] determined that there was a positive,
but weak, correlation between ImageNet accuracy and accuracy on other
unrelated tasks. However, [187] found that while earlier innovations on
ImageNet show improvement on real-world datasets, the correlation does not
exist for architectures developed after VGG [274]. Instead, improvements have
largely come from pre-training on large datasets, where it has been observed
that model size is important for performance on downstream tasks [174].

#### Maximum Likelihood vs. Bayesian

Maximum likelihood estimation (MLE) is a common approach for training machine
learning models, yielding point estimates for model parameters. In contrast,
Bayesian machine learning provides a more natural representation of
uncertainty by estimating a distribution over model parameters, which leads to
a distribution over model outputs. Exact posterior updates for neural network
parameters are intractable so approximations are made. Dropout [189], a
regularization technique that involves randomly zeroing out selected neuron
weights, approximates Bayesian inference and can be applied at test time for
uncertainty quantification. Markov chain Monte Carlo (MCMC) techniques have
also been used but rarely achieve strong predictive performance due to the
difficulty in estimating high-dimensional posterior distributions. [212]
combined dropout with Hamiltonian Monte Carlo to obtain improved MCMC results.
[253] conducted a comprehensive evaluation of approximate Bayesian inference
techniques and identified dropout as a strong contender, though only
relatively small datasets were used in the evaluation.

#### Ensembling

Ensembling involves aggregating the outputs or weights of multiple ML models
into a single output or single model, which can then be used for the task at
hand. Various ensembling techniques have been used to boost model performance
across a number of metrics, including accuracy [290], robustness [291] and
uncertainty quantification, where ensembles are used as an approximation to
Bayesian Neural Networks [227]. In the context of fine-tuning large pre-
trained models, [290] demonstrate that averaging the weights of multiple
models fine-tuned with varying hyperparameter configurations can lead to
improvements in both accuracy and robustness. Their approach with a ViT-G
model led to a new state-of-the-art ImageNet accuracy of 90.94%. Additionally,
with large pre-trained models, [291] highlight the benefits of ensembling the
weights of the zero-shot model and fine-tuned versions of the model. Doing so
leads to large accuracy improvements under distribution shifts while
preserving high accuracy on the distribution targeted by fine-tuning.

When aggregating the outputs of multiple models into an ensemble, it has been
widely shown that good ensembles are comprised of models that are both
accurate and diverse, making errors independent of each other [289, 260, 261,
230], a feature that deep learning systems may often lack [245, 246]. [195]
analyze the latest high-performing models, which are trained with diverse
optimization objectives on different large-scale datasets. Their study and
those of others [156] demonstrate that diverse training methodologies create
models that specialize in different subdomains of the data and produce
increasingly uncorrelated errors. These diverse models ensemble more
efficiently, for example, ensembling a ResNet-50 model (ImageNet accuracy of
76.5%) with its most different counterpart, ALIGN-ZS (ImageNet accuracy of
75.5%) results in a 7% boost in accuracy.

###  2.3 Pre-Training

Pre-training is the process of training an ML component on datasets or tasks
that are not necessarily related to the final downstream task. Often the goal
in pre-training is to learn a representation of the data that eases the
learning of the downstream task. A good representation [166] typically maps
input data into a well-defined, low-dimensional manifold that represents the
causal factors of the data-generating process. A good representation may
improve each of the reliability metrics studied in this work. We overview four
types of pre-training: supervised, weakly-supervised, contrastive, and
generative.

#### Supervised Pre-Training

Supervised pre-training involves training a model on an auxiliary dataset that
is different from the primary task. It was initially used to enhance the
performance of convolutional neural networks on tasks when there was limited
training data [194, 155]. For example, pre-training on ImageNet led to a more
than 40% improvement in mean average precision (mAP) for the PASCAL VOC object
detection task, with larger improvements for larger models [155]. In a
systematic study of pre-training from ImageNet, [222] found that ImageNet
accuracy was a good predictor of downstream task performance both for logistic
regression on the pre-trained featurizer as well as when fine-tuning the
model. They found the largest improvements on datasets that were most similar
to ImageNet, and lower improvements on datasets they deemed “fine-grained”
such as those that make subtle distinctions between a large variety of cars or
aircraft. Many of the convergence and performance benefits of pre-training
were diminished when more data was available for the primary task. Supervised
pre-training was also essential to the success of the Vision Transformer
(ViT). The larger number of parameters and lack of built-in inductive biases
of CNNs [233, 180], mean that ViTs require more data. [183] pre-trained their
ViT architecture on the ImageNet-21k or JFT-300M datasets to get SOTA
performance on ImageNet-1k, with the larger models seeing more improvement
from the larger datasets.

#### Weakly Supervised Pre-Training

Weakly supervised pre-training involves converting unlabeled datasets into
supervised learning tasks, allowing for the learning of effective
representations. Examples of constructing supervised tasks from unsupervised
data include predicting the relative location of two image patches [182],
predicting image colors from grayscale [228], and counting objects [256].
Additionally, [242, 275] demonstrated an effective form of weak supervision by
constructing a dataset of 3.5 billion images from Instagram, where image
hashtags serve as the image labels. Their approach, known as Supervised Weakly
through hashtAGs (SWAG), showcased strong transferability to new datasets and
achieved zero-shot performance comparable to models trained on image-text
pairs [263, 216].

#### Contrastive Pre-Training

Contrastive learning is a self-supervised learning approach that aims to learn
useful representations by maximizing agreement between positive samples and
minimizing agreement between negative samples. Instance contrastive learning
[206, 292, 251, 161] uses different augmented views of the same instance as
positive examples and views of different instances as negative samples [206,
251]. Swapping Assignments between multiple Views (SwAV) [172] assigns each
image view to a cluster, alleviating the memory and compute requirements of
pairwise feature comparison in a batch. By enforcing consistency between codes
from different augmentations of the same image using the cluster assignments,
the model learns discriminative representations without directly comparing the
image features. Contrastive Language-Image Pre-training (CLIP) [263] and A
Large-scale ImaGe and Noisy-text embedding (ALIGN) [216] employ image and text
pairs to train models, with loss functions specifically designed to enhance
matching of correct pairs. The resulting models show substantial improvements
in effective robustness on distribution-shifted variants of ImageNet [263,
216]. [223] finds that contrastive pre-training on datasets similar to the
downstream tasks leads to improved performance in that task and combining
datasets does not provide the benefits of each.

#### Generative Pre-Training

Generative pre-training encompasses unsupervised approaches aimed at
reconstructing synthetically removed parts of the input data. Original methods
employed autoencoder architectures [247] trained with a reconstruction loss or
generative adversarial networks [196] to reconstruct full data examples.
Additional generative pre-training tasks involve partial reconstruction, such
as in-painting masked regions of an image or predicting the next word in a
sentence [169]. Generative pre-training has proven highly effective in natural
language processing, enhancing both accuracy and robustness [169]. In computer
vision, masked image modeling (MIM) has emerged as an effective generative
task [162, 294, 205]. For instance, Masked Autoencoders achieved SOTA
performance on ImageNet-1K and ImageNet-C through MIM [205]. Furthermore,
Contrastive Captioners (CoCa) introduced by [301] combine a contrastive loss
between image and text encoders with a generative loss for captioning, leading
to new state-of-the-art results on ImageNet when fine-tuned [301].

###  2.4 Training Procedure

The traditional ML objective is empirical risk minimization (ERM) [287], which
minimizes a loss over a finite set of samples, typically with some form of
stochastic gradient descent. Additional losses and optimization techniques
have been proposed to improve one or more reliability metrics and are
discussed below.

#### Subgroup-Aware Training

Rather than only minimizing the empirical risk across the dataset, subgroup-
aware training objectives incorporate domain knowledge. To use these
approaches, the data must be categorizable into discrete subgroups. In
classification problems, the label itself can be the group identifier. In
other settings, the data may be augmented with additional metadata to identify
subgroups. Distributionally robust optimization (DRO) [266, 184, 213]
minimizes the worst-case performance in a subgroup, but requires stronger-
than-normal regularization to ensure good generalization [266]. Other
approaches such as invariant risk minimization (IRM) [160] and risk
extrapolation [224] promote risk-invariance across domains in order to produce
more generalizable models. Correlation alignment (CORAL) [279, 280] methods
try to align the second-order statistics of input features (extracted from
deep neural networks in deep CORAL [280]) between domains on labeled or
unlabeled data. Domain-Adversarial Training of Neural Networks (DANN) [190],
promotes the learning of features that are discriminative for the primary task
but are indiscriminative for the domain. Adaptive Feature Norm (AFN) [297],
tries to ensure the magnitude of feature norms is the same across domains and
demonstrates that when this is the case, models have better generalization.

#### Self-Training on Unlabeled Data

Self-training involves iteratively training a model on the labeled data
initially available and then using this model to make predictions on unlabeled
data, which is subsequently incorporated as “pseudo-labeled” data for further
training iterations [300]. This iterative process of self-training with
pseudo-labels helps harness the information present in unlabeled data,
improving model performance and generalization. The PseudoLabels approach
[229] involves labeling unlabeled training data using the predictions of the
current model. Several works have extended this idea by incorporating two
models, a student and a teacher [293, 262, 284]. The teacher labels unlabeled
data for the student, which is trained on input data that is typically noised
through techniques like data augmentation or dropout. The Noisy Student method
[293], utilizing 300M unlabeled examples, achieved state-of-the-art results on
in-distribution and robustness ImageNet benchmarks. FixMatch [276] selectively
uses the pseudo-labels that are predicted with high confidence and introduces
a high level of noise during student training, showcasing remarkable
effectiveness in low-labeled-data scenarios.

#### Auxiliary and Alternative Losses

Auxiliary loss functions play a critical role in improving machine learning
models by incorporating additional objectives into the training process. In
the context of uncertainty quantification, [226] propose a loss function that
directly optimizes model calibration during training and [271] include entropy
regularization, both of which enhance the calibration of uncertainty estimates
without the need for additional calibration steps. For enhancing the
smoothness of the loss landscape, [188] introduce sharpness-aware
minimization, a technique that aims to find minima in the loss landscape with
uniformly low values. Models trained with this loss exhibit improved
generalization performance across various tasks, providing a valuable tool for
reducing overfitting and achieving better model behavior in practice. [220]
propose a method that first predicts a set of human-interpretable concepts as
intermediate representations before performing the final prediction task. By
leveraging such concept-based representations, models can capture meaningful
factors of variation in the data, enabling better interpretability and
performance in complex tasks.

[219] argue that capturing the correct causal model is crucial for strong
generalization properties. They motivate approaches that explicitly
disentangle concepts or employ generative models, enabling better control and
understanding of the underlying data distribution. For example, [288]
introduce a generative classification approach based on a combination of
variational autoencoders (VAEs) and generative adversarial networks (GANs),
which demonstrates superior performance in tasks such as out-of-distribution
detection. These approaches, however, have yet to scale to real-world datasets
and tasks.

#### Fine-Tuning

Recent advancements in fine-tuning have led to improvements in performance on
not only in-distribution accuracy, but also domain-shift robustness and
adversarial robustness. [156] note how in-distibution accuracy versus domain-
shifted accuracy typically follows a single linear trend, and models with a
higher domain-shifted accuracy compared to this baseline exhibit “effective
robustness”. Through their investigations, they demonstrated that zero-shot
pre-trained models, as well as those models that have been minimally fine-
tuned, exhibit effective robustness, but this effective robustness is lost as
the models are fine-tuned to convergence. [291] address this reduction in
effective robustness from fine-tuning by ensembling the weights of the zero-
shot and fine-tuned models; compared to standard fine-tuning, which achieved
accuracies of 86.2% on ImageNet ID and an average of 68.6% across five
distribution shifts, the ensembled model achieved accuracies of 87.1% and
77.4% on the same datasets. Fine-tuning is particularly sensitive to the
optimization algorithm used, and [225] develop new optimization strategies to
improve the accuracy and robustness of fine-tuned models. Adversarial fine-
tuning has been shown to achieve competitive adversarial robustness compared
to adversarial training from scratch, saving substantial computational effort
[215]. However, special care has to be taken to not overfit to the adversarial
examples, and reduce the original performance, so careful learning rate
scheduling was applied.

###  2.5 Post-Training

After training is complete, additional analysis and calibration may be
required before a model can be deployed. We discuss algorithms for selecting
trained models from a set, as well as calibrating models for more accurate
uncertainty estimates.

#### Model Selection

Selecting the most suitable model for deployment can be a challenging task,
even when multiple models demonstrate similar performance on in-distribution
test sets. Variations in other crucial properties, such as robustness, can
exist among models trained using nearly identical procedures [201, 179]. In
their exploration of domain generalization approaches, [201] find that using a
validation set that includes all of the training domains outperforms leave-
one-domain-out cross-validation. Additionally, [179] discover that
underspecification issues result in models exhibiting diverse robustness
properties, which are not reflected solely in the test set performance. These
findings underscore the importance of carefully considering various factors
beyond conventional evaluation metrics when selecting models for deployment.

#### Uncertainty Calibration

Once a model has been trained, its uncertainty estimates can be modified
without affecting the model accuracy. Overconfidence has been observed across
deep learning architectures, leading to the development of techniques that can
correct for it and even provide formal guarantees of the predictive
uncertainty. [202] introduce temperature scaling which is the best of their
compared approaches to uncertainty quantification. For the $K$-class
classification setting with logits $z_{i}$ and temperature $T$, the
probability of the $i$th class is:

$\mathbb{P}(y_{i})=\frac{\exp(z_{i}/T)}{\sum_{k=1}^{K}\exp(z_{k}/T)}$ (1)

The temperature is chosen to minimize the negative log-likelihood of a
validation set. Another calibration procedure is conformal prediction [158],
which allows for the construction of prediction regions with provable
guarantees from heuristic notions of uncertainty. However, without initially
well-calibrated uncertainty estimates, the prediction regions are often too
large to be useful [159].

###  2.6 Model Deployment

Once a ML system has been sufficiently tested and evaluated it may be deployed
in a real-world setting. Deployment, however, is not a on-time event, but
rather a continuous process of monitoring, updating, and improving models
based on new data, feedback, and objectives. Regular assessment of a system’s
performance is essential since performance tends to degrade over time due to
drifts in the data distribution [299].If performance drops below an acceptable
level, the model should be updated with more recent data.

More dramatic shifts in the input distribution due to faults or other
anomalies are detected in a process known as run-time monitoring. If the space
of possible faults is known at design time, a system can be designed to look
out for these faults and respond accordingly. To identify unforseen events,
however, we must rely on approaches for out-of-distribution detection.

#### Out-of-Distribution Detection

Out-of-distribution (OOD) detection (also called anomaly detection or fault
detection) refers to the task of identifying inputs to a ML model that come
from a different distribution than the training data. Algorithms for OOD
detection typically involve computing a score and using it with a predefined
threshold for flagging OOD inputs. The algorithms primarily differ in the way
the score is computed. A baseline approach to OOD detection is to use the max
softmax value of the output as a score indicating the confidence of in-
distribution, with the assumption that in-distribution inputs should be
classified with higher confidence [210]. [207] found that using the max logit
outperformed the max softmax, since information is lost due to the
normalization of the softmax operation. [237] uses an energy score that
combines information from all class labels to better identify OOD inputs.
[235] introduce ODIN (Out-of-Distribution detector for Neural networks) which
computes the max softmax of an adversarially perturbed input. They observe
that OOD samples have a higher sensitivity to such perturbations and therefore
produce higher perturbed softmax scores. Other approaches rely on information
from training such as typical gradients [273].

##  3 Experimental Setup

Given the wide range of approaches to building reliable ML models, we want to
investigate the following two questions:

  1. 1.

How do models trained using different ML approaches perform on a variety of
reliability metrics?

  2. 2.

How do different reliability metrics relate to each other?

To answer these questions, we perform a large scale evaluation of machine
learning models across a variety of reliability metrics. We use a combination
of models trained for prior work and models trained for the present work to
evaluate the relationship between machine learning methodology and reliability
metrics. We also evaluate the relationship between reliability metrics by
measuring the correlation between them. In the following sections, we describe
the datasets, metrics, and models used in our assessment.

###  3.1 Selection of Datasets

A small number of benchmarks such as ImageNet, have become the de-facto
standard in machine learning research, but it is not clear whether progress on
those datasets translates to progress on other tasks [187, 223]. For that
reason, we chose to use other publicly available datasets for evaluation. To
choose the datasets, we used the following criteria:

  1. 1.

The datasets should reflect a real-world task of practical importance.

  2. 2.

The datasets should include data from real-world distribution shifts.

  3. 3.

The datasets should be image-based to make the use of pre-trained models
feasible.

  4. 4.

The task should be binary or multi-class classification for the ease of using
open source adversarial attacks and defining a calibration metric.

While there have been a number of benchmarks dedicated to measuring
distribution-shift generalization [221, 267, 244, 243, 164, 201], we found
that the image classification tasks provided by the WILDS benchmark [221, 267]
satisfied these requirements best due to the real-world nature of the tasks
and the existence of models trained using a variety of approaches from prior
work. We selected the following datasets for our analysis:

  1. 1.

iWildCam: A dataset for species recognition ($182$ species labels) from
camera-trap images. The data is annotated with the camera identification
number that the image was taken from. Each of the train, validation, and test
datasets have a disjoint subset of identification numbers.

  2. 2.

FMoW: A dataset for land-use classification ($62$ land-use labels) from
satellite images. The data is annotated with the year the image was taken and
the geographical region of the image. The training dataset is composed of
images from before 2013, the validation set is from 2015–2016, and the test
set consists of images from 2016 onward.

  3. 3.

Camelyon17: A dataset for the binary classification of metastatic breast
cancer from patches of an image recorded from lymph node tissue under a
microscope. The data is annotated with the hospital that the slide was
prepared and imaged in. The training dataset contains data from three
hospitals while the validation and test sets each have data from a different
hospital.

See the WILDS benchmark for additional information on the datasets [221].

###  3.2 Selection of Evaluation Metrics

Based on the literature review and emerging regulations, we have identified
five important metrics related to the reliability of machine learning systems:
in-distribution performance, distribution-shift robustness, adversarial
robustness, calibration, and out-of-distribution detection. Each of these
metrics can be measured in a variety of ways and in this subsection, we
describe and justify our methods of evaluation. For each metric, we define a
corresponding score $s\in[0,1]$ so that larger values are always preferable.

#### In-Distribution Performance

A primary assumption in statistical machine learning is that data are
independently and identically distributed (iid) [204]. Datasets are typically
split into a train set, used to fit a statistical model, a validation set,
used to select hyperparameters, and a test set, used to evaluate the
performance of the selected model. In classification settings, the primary
metric of interest is accuracy, which is the fraction of correctly predicted
labels. Precision, recall, and F1 score, which are computed on binary
classification tasks and can be aggregated in the multi-class setting, are
also important performance metrics. For regression tasks, the mean absolute
error (MAE) or the mean squared error (MSE) are typical metrics of
performance. To remain general, we indicate the performance of a ML model $m$
with respect to a dataset $\mathcal{D}$ as

$p=\text{performance}(m,\mathcal{D})\text{.}$ (2)

The in-distribution performance $p_{\rm ID}$ is measured on a held-out test
set from the same distribution as the training data. We denote the performance
score

$s_{\rm ID}=p_{\rm ID}$ (3)

where in all of our experiments, $p_{\rm ID}$ is the top-1 classification
accuracy.

#### Distribution-Shift Robustness

Figure 2: Performance of WILDS pre-trained models under various distribution
shifts vs. in-distribution performance. The validation distribution shift and
the test distribution shift come from the WILDS benchmark while C1 measures
the performance drop from synthetic corruptions [209] of strength 1.

When the iid assumption is violated, the performance of modern deep learning
systems can drop significantly—far more than would be normal for a human
[192]. This effect has been demonstrated with synthetic distribution shifts
such as common image corruptions and perturbations [209, 192, 250], as well as
natural distribution shifts [282] arising from re-collecting data [265],
changing data collection criteria [208], perturbations over time [198, 272],
or other real-world changes in the deployment environment [221, 244, 243,
164]. Figure 2 shows the performance of the WILDS pre-trained models subject
to real-world shifts from the WILDS dataset (validation and test datasets,
which each represent a different shift) as well as synthetic shifts from the
standard set of image corruptions with corruption level 1 (C1) [209].

While there exist no guarantees of performance under arbitrary distribution
shifts, it has been shown empirically that there is sometimes strong
correlation between in-distribution generalization and distribution-shift
generalization [265, 198, 249, 156], a phenomenon referred to as accuracy on
the line [249]. We observe these linear trends in fig. 2 for iWildCam and
FMoW, but Camelyon17 is more highly varied, which has also been observed in
prior work [249]. The existence of these linear trends has lead researchers to
measure not just the performance under a distribution shift but the effective
robustness [156, 291, 282] of a model, which is the amount the performance
deviates from the value predicted by a linear model over the in-distribution
generalization. In a similar vein, we separate robustness from performance of
the model by measuring the ratio of performance under the distribution shift
to the in-distribution performance, so a robustness value of $1$ means the
performance does not drop compared to the in-distribution performance.

Figure 3: Comparing distribution-shift accuracy between various distribution
shifts.

For some datasets and machine learning approaches, however, the accuracy on
the line trend is weaker, so it is still necessary to measure the
distribution-shift performance on various distribution shifts. The type and
magnitude of distribution shifts one should evaluate a model on is context-
dependent and should ideally reflect the deployment environment of the model.
However, this is not always possible to know a priori so the question arises:
_Does robustness to one distribution shift imply robustness to others?_ To
investigate this question, we compare the performance of the WILDS models to
three different distribution shifts. Following the conventions of the WILDS
datasets, “val” and “test” are different real-world shifts, while “C1” refers
to a synthetic corruption shift [209] of strength 1. The results shown in fig.
3 indicate that although there may be some correlation, it is weaker than the
correlation to in-distribution generalization, an observation consistent with
prior work [192, 250]. We, therefore, choose to evaluate model robustness on
multiple distribution shifts and average the results. For $N$ distribution-
shifted datasets $\\{\mathcal{D}_{\rm DS,1},\ldots,\mathcal{D}_{\rm DS,N}\\}$,
with model performance $p_{i}$ on the $i$th dataset, the distribution shift
robustness score is

$s_{\rm DS}=\frac{1}{N}\sum_{i}\frac{p_{i}}{p_{\rm ID}}.$ (4)

In our experiments we average across the “val”, “test”, and “C1” distribution
shifts. We choose averaging (instead of worse case) because we want the models
with the highest score to do well on all three distribution shifts.

#### Adversarial Robustness

Despite models being able to generalize to unseen samples from previously seen
distributions, well-chosen yet small input perturbations can catastrophically
change the model outputs [197, 241]. Such “adversarial attacks” can be
constructed in a wide variety of ways, for instance, by using a model’s
gradient information, or by transferring attacks from similar models. In the
presence of an adversary, Adversarial Robustness then denotes the capability
of a model to withstand adversarial attacks (with limited strength) for a
fixed and finite set of data samples. For example, one may measure adversarial
robustness by computing the misclassification rate of a set of samples when
subjected to adversarial attacks with a fixed maximum input perturbation.

In order to evaluate adversarial robustness, we follow the procedure of [177]
and employ an ensemble of parameter-free adversarial attacks, consisting of
two step-size free modifications of Projected Gradient Descent (APGD-CE and
APGD-DLR), as well as Fast Adaptive Boundary Attack [176] and Square Attack
[157]. We note that this is the same approach as used by the popular
“RobustBench” benchmark [175]. For the attack strength, we choose a maximum
perturbation of $\epsilon=3/255$ in the $L_{\infty}$-norm, which is consistent
with common choices for natural images. In order to avoid conflating effects,
we do not consider inputs from distributional shifts, and instead only
evaluate on unseen in-distribution samples.

Then, if $p_{\rm ADV}$ is the performance in the presence of adversarial
examples, we report the adversarial robustness score as the ratio in
performance compared to the in-distribution performance

$s_{\rm ADV}=\frac{p_{\rm ADV}}{p_{\rm ID}}.$ (5)

#### Calibration

While it may not be possible for ML systems to be highly accurate in every
situation, it is essential for these systems to accurately quantify their
level of uncertainty (for a review see [154]). Here we assess uncertainty
quantification using the notion of calibration, where a model is calibrated if
its confidence measure aligns with the error distribution of the outputs. A
model can be both highly accurate and poorly calibrated, or conversely, highly
inaccurate and well-calibrated. Several empirical measures exist for
evaluating model calibration. For example, [202] explore metrics such as the
expected calibration error ($\mathit{ECE}$), which is computed by binning the
model outputs and averaging the absoluted difference in the accuracy of each
bin to the accuracy predicted by an identity function. [202] also introduce
the maximum calibration error, which calculates the largest deviation across
bins. They also discuss the negative log-likelihood, a metric also used by
[227] and [189], as it is applicable to regression tasks in addition to
classification tasks, though $\mathit{ECE}$ has also been extended to the
regression setting [232]. In this work, since we focus on classification tasks
we use $\mathit{ECE}$. To convert $\mathit{ECE}$ to a calibration score, we
normalize it by a maximum error $\mathit{ECE}_{\rm max}$ and subtract the
result from $1$. We use $\mathit{ECE}_{\rm max}=0.5$ as this corresponds with
the pathological case of a binary classifier that is correct 50% of the time
but always predicts with 100% confidence.

Figure 4: Calibration of WILDS pre-trained models under various distribution
shifts vs. in- distribution performance. Figure 5: Comparing distribution-
shift calibration between various distribution shifts.

Ideally, a model should be well-calibrated on both in-distribution data as
well as on distribution-shifted data, even if the model performance decreases.
In fig. 4, we compare the calibration of the WILDS pre-trained models across a
variety of datasets and find that the calibration of a model often drops under
distribution shifts, and in some cases (iWildCam and FMoW) exhibits a
significant correlation with the in-distribution calibration. For Camelyon17,
however no strong correlation seems to exist. We also check whether
calibration on one type of distribution shift implies calibration on others.
Figure 5 shows that similarly to performance, the correlation is weaker
between distribution shifts than it is to the in-distribution calibration. We
therefore choose to compute the calibration score by averaging the calibration
of the model across the in-distribution and distribution-shifted datasets,
giving

$s_{\rm CAL}=1-\frac{1}{(N+1)\mathit{ECE}_{\rm max}}\left[\mathit{ECE}_{\rm
ID}+\sum_{i=1}^{N}\mathit{ECE}_{i}\right]\text{,}$ (6)

where $\mathit{ECE}_{\rm ID}$ is the expected calibration error on the in-
distribution data and $\mathit{ECE}_{i}$ is measured on the $i$th
distribution-shifted dataset. As for DS robustness, we average across the
“val”, “test”, and “C1” distribution shift and use averaging instead of
picking the worst case for the same reasons.

#### Out-of-Distribution Detection

The identification of out-of-distribution (OOD) inputs is a binary
classification task and can therefore be measured according to a variety of
metrics including precision, recall, and F1 score. These metrics, however,
typically require the a priori definition of a threshold parameter, which
would be set using validation data and a context-specific notion of the
desirable precision-recall trade-off. To avoid selecting a threshold, we
follow prior work in using the area under the receiver operator curve
($\mathit{AUROC}$) to assess the ability of a model to identify OOD inputs.

To measure OOD detection we require datasets that are OOD. Ideally, these
datasets should reflect the types of input faults that may occur for the
machine learning model, but this may not be possible to know at the time of
model evaluation. We therefore select $M$ diverse OOD datasets and combine
them into a single OOD dataset

$\mathcal{D}_{\rm OOD}=\mathcal{D}_{\rm OOD}^{(1)}\cup\mathcal{D}_{\rm
OOD}^{(2)}\ldots\cup\mathcal{D}_{\rm OOD}^{(M)}.$ (7)

For each task we evaluate from WILDS, we use the other image-classification
tasks as well as a dataset of Gaussian noise and the RxRx1 dataset [221] as
our OOD data. For example, when performing OOD detection on models trained on
iWildCam, we compose the ID validation sets from Camelyon17, FMoW, RxRx1, and
Gaussian noise to make up a single OOD dataset.

(a) Correlation of $\mathit{AUROC}$ values for each pair of OOD detection
algorithms. Color is Pearson correlation and number is $R^{2}$.

(b) Comparison of the highest and lowest correlations between OOD detection
algorithms.

Figure 6: Comparing various OOD detection algorithms on models trained from
each of the three datasets to check for consistency in detection strategies.

There are many approaches to outputting OOD detection scores from model
outputs. We use the open source
implementation111https://github.com/kkirchheim/pytorch-ood of four different
OOD detection algorithms (Max Logit [207], Max Softmax [210], Energy-based
[237], and ODIN [235]) and compare their $\mathit{AUROC}$. In fig. 6(a), we
plot the Pearson correlation of each approach’s $\mathit{AUROC}$ (combining
all three datasets) and find that although there is some correlation between
approaches, there can be substantial variability (we show scatter plot
comparisons of the highest and lowest correlations in fig. 6(b)). For this
reason, we compute the final OOD detection score as the average of the
$\mathit{AUROC}$ for each of $N_{\rm OOD}$ OOD detection algorithms

$s_{\rm OOD}=\frac{1}{N_{\rm OOD}}\sum_{i=1}^{N_{\rm OOD}}\mathit{AUROC}_{i}.$
(8)

We choose to average this metric under the hypothesis that a model that does
well with multiple OOD detection algorithms has a representation that is well-
suited to OOD detection in general, and should therefore receive a higher
score.

#### Holistic Reliability Score

For many applications it is desirable for a model to score highly on all of
the reliability metrics simultaneously. To measure overall reliability, we
introduce the _holistic reliability_ score as the weighted average across
properties. If we define $\mathbf{w}=[w_{\rm ID},w_{\rm DS},w_{\rm ADV},w_{\rm
CAL},w_{\rm OOD}]$ and $\mathbf{s}=[s_{\rm ID},s_{\rm DS},s_{\rm ADV},s_{\rm
CAL},s_{\rm OOD}]$, then

$s_{\rm HR}=\mathbf{w}^{\top}\mathbf{s}\text{.}$ (9)

The application domain should dictate the relative importance of each metric,
however in all of our analyses, we set the weights equal to each other. We
note that by averaging the scores together, we lose fine-grained detail about
the individual performance characteristics of a model in order to obtain a
scalar value for reliability. This value is therefore not comprehensive, but
is another tool by which models can be evaluated.

###  3.3 Selection of ML Algorithms and Training Approach

Our analysis relies on having many models trained using various ML algorithms
on each of the datasets. Another benefit of using the WILDS benchmarks is that
pre-trained models are publicly available, and we make use of these models.
However, the algorithms used to train these models do not cover the full
spectrum of approaches for reliable ML so we must also train our own models.

#### WILDS Pre-Trained Models

In the original WILDS benchmark work [221], the authors focused on approaches
that promoted distribution-shift robustness by promoting invariance in the
model outputs or optimizing for worst-case performance. The following
approaches were explored: Invariant Risk Minimization (IRM) [160], Correlation
Alignment (CORAL) [280], and Distributionally Robust Optimization (groupDRO)
[266]. In the second version of the benchmark [267], the focus was on the use
of unlabeled training data to improve distribution-shift robustness.
Approaches that promote invariance using unlabeled data were explored such as
Domain-Adversarial Training of Neural Networks (DANN) [190], Adaptive Feature
Norm (AFN) [297], and again CORAL, but with additional domains provided by the
unlabeled data. Approaches that used self-training were also explored such as
PseudoLabels [229], Noisy Student [293], and FixMatch [276]. Additionally, one
contrastive approach, Swapping Assignments between multiple Views (SwAV)
[172], was also used. Baseline models using empirical risk minimization (ERM)
were trained for each version of the benchmark, and on the second version, ERM
with RandAugment [178] was also tried. We use the non-augmented ERM models as
our baselines when evaluating the effect of various reliability-promoting ML
approaches.

For each algorithm, hyperparameters were selected by choosing the best
configuration from 10 random hyperparameter samples. For this set of
hyperparameters, between 3 and 10 models were trained, each with different
random seeds. We ignore the models trained during hyperparameter tuning and
use only the hyperparameter-optimized models in our analysis. All $234$ pre-
trained models we used are publicly
available.222https://worksheets.codalab.org/worksheets/0x52cea64d1d3f4fa89de326b4e31aa50a

#### Selection of Other Reliable ML Algorithms

To decide which additional approaches we explore, we used the following
criteria:

  1. 1.

The approach should show at or near SOTA performance on a large-scale
benchmark (such as ImageNet) for at least one reliability criterion we are
evaluating.

  2. 2.

The approach should be generalizable to all datasets we consider, therefore
approaches that require domain expertise are excluded.

  3. 3.

The approach should be feasible to implement given the time and resources
available. We therefore favor approaches that have open-source availability
and lower implementation overhead.

Using these criteria, we selected the following set of approaches to evaluate
(additional details in section A and code
provided333https://github.com/sisl/holistic-reliability-evaluation):

  * •

Dataset augmentation: We experiment with two augmentation techniques. AugMix
[211] layers a series of simple image augmentation operations in concert with
a consistency loss and has been shown to halve the error rate of corrupted
CIFAR-10. The second technique, RandAugment [178], randomly selects from a
pool of augmentation transformations to apply sequentially to an image and has
been shown to achieve SOTA performance on benchmark datasets like CIFAR-10,
SVHN, and ImageNet.

  * •

Adversarial data augmentation: Following the approach of [241], we use
Projected Gradient Descent (PGD) to adversarially perturb each input during
training. We use $10$ iterations of PGD for each batch causing the training
time to increase by a factor of about 10. Despite it being several years old,
this approach still seems to be among the most effective for adversarial
robustness [295].

  * •

Fine-tuning from pre-trained models: Some large improvements in ID performance
and DS robustness have been observed when fine-tuning pre-trained models [291,
225, 263]. We therefore explore this methodology by using weights that have
been pre-trained using a variety of algorithms and across a variety of
architectures.

  * •

Model ensembling: Ensembling of models has been shown to improve ID
performance, DS robustness, and calibration [227, 290]. We investigate the
composability of models trained as part of the WILDS benchmark. We combine the
outputs of up to $5$ models using scalar weights that are optimized on the ID
validation set.

  * •

Temperature scaling: Temperature scaling is a simple yet effective approach
for the post-hoc calibration of deep neural networks and outperforms many
other approaches. We compare reliability metrics pre- and post-temperature
scaling.

To avoid the costly overhead of tuning hyperparameters for each training
configuration, instead we randomized the hyperparameters for each experiment.
We included all of the trained models in our analysis, even though some would
have sub-optimal hyperparameter configurations. In our analysis, we therefore
focus on the best-performing set of models. In total, we trained an additional
$287$ models to bring the total models analyzed above $500$. We evaluated each
of these models with and without temperature scaling on both validation and
test sets.

###  3.4 Limitations

Prior to discussing the results of our experiments, we want to outline the key
limitations of our experimental approach.

#### Incomplete Assessment of Reliability

Our workflow focuses on only five criteria: in-distribution performance,
distribution-shift robustness, adversarial robustness, calibration, and out-
of-distribution detection. While these metrics cover important aspects of
reliability, they may not capture the full range of challenges and limitations
associated with evaluating machine learning systems. Other relevant factors
such as fairness/bias, interpretability, and privacy are not adequately
addressed by the selected metrics, however could be included if they are
measured quantitatively. Additionally, safety and security are features of the
complete system in the socio-technical context within which it operates [231].
Without that context, it is difficult to draw general conclusions around the
safety of AI systems.

#### Choice of Quantitative Metrics

For each of the reliability criteria we do analyze, we measure a single number
to indicate performance, but often there are multiple relevant metrics. For
example, when measuring performance, we focus on top-1 accuracy but other
metrics such as top-$k$ accuracy, confusion, or subgroup accuracy may all be
relevant depending on the task. Measuring adversarial robustness assumes a
specific set of attack strategies, and uncertainty quantification could be
measured by metrics such as integrity risk [203] rather than calibration. For
OOD detection, we assume a specific set of input faults, which may not reflect
the types of inputs we would wish to flag during model deployment.
Additionally, we selected $\mathit{AUROC}$ to quantify the capability of the
OOD detector, but other metrics such as a false positive rate at a fixed true
positive rate may also be relevant depending on the specific context of the
task.

#### Lack of Target Thresholds

To assess the reliability of ML models it would be helpful to have target
values of our evaluation criteria in mind to quantify how close our models are
to being sufficiently reliable. However, acceptable thresholds for these
parameters are context dependent. For example, in aviation, failure rate
requirements can be as low as ${10}^{-9}$ per hour [181], while in some
medical settings, we tolerate much higher failure rates when the alternative
is also risky. We therefore do not include such thresholds in our analysis.

#### Limited Exploration of Algorithms and Hyperparameters

Due to the high computational cost of training many different ML models across
several datasets, we are unable to explore every algorithm for building
reliable ML models and to exhaustively search the space of hyperparameters
(neither did the researchers who trained the WILDS pre-trained models we
analyze). Therefore, when we observe no impact (or a negative impact) of an
algorithm, it does not constitute proof that the approach is ineffective and
there might be a configuration of hyperparameters that would improve the
impact.

#### Limited Real-World Tasks

While the WILDS benchmark tasks offer a great framework to study real-world
distribution shifts, there are many more real-world domains of high
consequence and the generalizability of our findings should be explored on
each new domain.

While the proposed workflow and evaluation metrics provide a valuable
framework for assessing the reliability of machine learning systems, it is
important to recognize these limitations. Consideration of additional factors
and context-specific requirements are necessary to develop a more
comprehensive and robust evaluation approach.

##  4 Results

In this section, we present the results of our experiments. We start by
exploring the effect of different ML interventions for improving model
reliability and then consider the relationship between reliability metrics.
When possible, we show the results from all three datasets simultaneously.
When assessing individual algorithms, however, we present the results of our
experiments on the iWildCam dataset, then present the results for the
remaining datasets in section B. Unless otherwise mentioned, the conclusions
we draw from the iWildCam results are consistent with the results on the other
datasets.

###  4.1 Data Augmentation

Figure 7: Effect of various data augmentation strategies for iWildCam. Each
circle is a single model. The blue band shows the range of the ERM baseline
models with the dashed line giving the average.

We experimented with two common forms of data augmentation: _RandAug_ [178]
and _Augmix_ [211], as well as _PGD_ -based adversarial training [241]. The
results for iWildCam are shown in fig. 7 and the results for the remaining
datasets are in section B.1. Adversarial training primarily seemed to affect
in-distribution performance (where it reduced slightly) and adversarial
robustness where it tended to increase it significantly, especially on
iWildCam and FMoW, which have a larger number of classes and are therefore
more susceptible to adversarial attack. While adversarial training reduced
calibration and distribution-shift robustness in the worst case for iWildCam,
it did not have a consistent effect for the other datasets, so we draw no
general conclusions. Notably, the highest performing RandAug and Augmix models
improved across all of the metrics (increasing the HR score by about $0.08$),
with the largest gains coming in calibration, which is consistent for the
other datasets.

###  4.2 Model Ensembling

To enhance the reliability of non-ML-based systems, it is common to compose or
ensemble redundant components to reduce errors and the likelihood of faults.
In this experiment, we investigate the effect of ensembling ML-based
components. We randomly choose different numbers of models and combine their
logits with weights optimized for ID performance on a validation set. Each
point in fig. 8 represents the best ensemble (in terms of ID validation
performance) from a selection of $50$ random ensembles of the same size.
Different groups of $50$ were used to get the different points in fig. 8. For
iWildCam, ensembling improves nearly all of the reliability metrics (and
therefore HR score) except for OOD detection, which remains largely
unaffected. We note, however, that uncertainty estimates from ensembles can be
used as a strategy for OOD detection [227], but we do not apply that technique
here. The gains from ensembling appear to diminish as the number of models
increases i.e. using five models is not much better than four in all of the
datasets. This finding is consistent with prior work on ensembles [195].

Figure 8: Ensembles of models for iWildCam.

###  4.3 Fine-Tuning Pre-Trained Models

For these experiments, we assess the effect of fine-tuning models that have
been pre-trained in a variety of ways. The fine-tuning procedure is specified
in more detail in section A.5. We first explore the type of pre-training, then
discuss model architecture, and lastly experiment with the size of pre-trained
transformers.

#### Pre-Training Algorithm

Our goal is to see if the type of pre-training is important for the final
performance of the pre-trained model. We compare four pre-training algorithms:
_Supervised_ is pre-trained using supervised learning on ImageNet-1k, SWAG is
Supervised Weakly from hashtAGs [275], _CLIP_ is Contrastive Language-Image
Pre-training [263], and _MAE_ is for masked-autoencoder [205], a generative
pre-training technique. The results for iWildCam are shown in fig. 9. All of
the pre-training techniques seem to substantially improve DS robustness,
calibration, and OOD detection, leading to higher overall HR scores ($>0.1$
improvement). MAE pre-training seems to most reliably lead to the highest DS
robustness. Given the focus on unsupervised pretraining as a method for
improving robustness [291] it is notable that even supervised pretraining
provided significant reliability benefits across all datasets.

Figure 9: Effect of pre-training algorithm when fine-tuning on iWildCam.

#### Model Architecture

To explore the influence of model architecture when fine-tuning models, we use
a variety of architectures, each pre-trained via supervised learning on
ImageNet-1k. We compare the Vision Transformer (_ViT_) [183],_Swin
Transformer_ [238], _MaxViT_ [286], _ConvNeXt_ [239] and _EfficientNet_ [281]
architectures. In the context of supervised pre-training, the Swin
Transformer, EfficientNet and ConvNeXt architectures had the best ID
performance, while all the transformers and ConVeXt architecture had superior
DS robustness, calibration and OOD detection. On the other datasets, however,
only the ViT and ConvNext architectures showed significant DS robustness
improvements.

Figure 10: Effect of model architecture when fine-tuning on iWildCam.

#### Model Size

Lastly, for a fixed model architecture (ViT) we explore the effect of model
size by comparing the base _ViT-B_ , large _ViT-L_ and huge _ViT-H_ model
versions. We show the result for iWildCam in fig. 11. The larger models show
slight improvements over smaller models in almost all categories. When
considered with the other datasets, the largest transformer seemed to be less
sensitive to hyperparameters, giving consistently good performance.

Figure 11: Effect of transformer size when fine-tuning on iWildCam.

###  4.4 Sub-Group Aware Training

Prior work in improving distribution shift robustness has explored the use of
group invariance in the features output by the model. The WILDS benchmark
trained models with GroupDRO [266], correlation alignment (CORAL) [279, 280]
and invariant risk minimization (_IRM_) [160], which we analyze further by
measuring all their reliability metrics. The results for iWildCam are shown in
fig. 12. While these algorithms do not improve the DS robustness, they do have
a small positive impact on calibration for iWildCam. When considering the
other datasets, however, these approaches have limited impact on any of the
reliability metrics including HR score.

Figure 12: Effect of including sub-group information for iWildCam.

###  4.5 Unlabeled Training Data

For many tasks, unlabeled data is readily available and can be used in a
variety of ways to improve model performance. [267] investigated the use of
unlabeled data for improving robustness to distribution shift. They
investigated techniques that labeled the unlabeled data (_PseudoLabels_ [229],
_Noisy Student_ [293], _FixMatch_ [276]), that used sub-group aware training
on the unlabeled data (Domain-Adversarial Training of Neural Networks (_DANN_)
[190], Adaptive Feature Norm (_AFN_) [297], and Correlation alignment (CORAL)
[279, 280]), and a contrastive learning technique using the unlabeled dataset
(Swapping Assignments between multiple Views (_SwAV_) [172]). We use the
models trained by [267] and see how the unlabeled data affected all
reliability metrics (results for iWildCam shown in fig. 13). For iWildCam we
can see that many of the approaches lead to a small improvement in DS
robustness as well as calibration and leave the other metrics largely
unchanged. There was a larger improvement to DS robustness for the FMoW
dataset and larger improvements on calibration and OOD detection for the
Camelyon17 dataset, leading to a significant improvement in HR score. For each
dataset, the algoriths that performed best for each metric differed.

Figure 13: Effect of using unlabeled training data for iWildCam.

###  4.6 Temperature Scaling

Temperature scaling has been shown to be a simple and effective technique for
post-hoc calibration of deep neural networks. For each model we trained, we
computed the reliability metrics both before and after temperature scaling.
The temperature parameter was optimized by minimizing the negative log-
likelihood of the in-distribution validation set. Since temperature scaling
does not change the model prediction, it has no effect on ID performance nor
DS robustness so we exclude those metrics in the results shown in fig. 14.
First, we find that there is no strong effect on adversarial robustness nor
OOD detection, but calibration is generally improved. For Camelyon17,
temperature scaling had less of an impact than in the other datasets, making
the calibration of the non-temperature-scaled models more important.
Temperature scaling therefore increases HR score by improving calibration
without affecting other metrics.

Figure 14: Effect of temperature scaling on model calibration, adversarial
robustness and OOD detection for all datasets.

###  4.7 Distribution of Scores and Summary of Improvements

In order to summarize the effects of our experiments, we report the
performance of approaches grouped into conceptual categories. Similar to the
breakdown of the previous subsections, we categorize the models into seven
approaches: _Baselines_ , which includes vanilla empirical risk minimization,
_Data Augmentation_ , which includes RandAug and AugMix, _Adversarial
Training_ , which includes just the PGD-based adversarial training,
_Ensembles_ , which includes all the ensembles, _Fine-Tuned_ , which includes
all of the fine-tuned models, _Loss Function_ , which includes the sub-group
aware training approaches, and _Unlabeled Data_ , which includes all of the
approaches that used unlabeled training data.

fig. 15 Provide a comparison of the distribution of scores across groups. We
plot a group-stacked-histogram of each reliability metric for iWildCam for the
seven groups. The results from the other datasets are included in section B.
The aggregate influence of each type of approach can then be compared.
Ensembling has the biggest improvement on ID performance and calibration,
while fine-tuning has the biggest effect on DS robustness, OOD detection and
HR score. Only the adversarial training was able to consistently achieve
higher levels of adversarial robustness.

Figure 15: Distribution of scores for iWildCam for various groups of models.

Figure 16 summarizes the quantitative influence of each group of approaches on
the HR score. We plot the difference between the best HR score in each group
and the best HR score for the baselines across the three datasets (as well as
the the average of this value). We see improvements from all groups of
approaches except those that use auxiliary loss functions. We note that the
lower effect of fine-tuning on the Camelyon17 dataset is primarily due to low
values of adversarial robustness for the fine-tuned models.

Figure 16: HR score improvement for highest HR score model in each group
compared to the highest HR score baseline model.

###  4.8 Metric Comparisons

Figure 17: Correlations between reliability metrics for three datasets. Color
indicates Pearson correlation and value indicates $R^{2}$.

Motivated by prior work that observes correlations between a variety of
metrics such as in-distribution accuracy vs. distribution-shifted accuracy
[249], adversarial accuracy vs. in-distribution accuracy [270, 214, 285], we
explore the possibility of intrinsic correlations between all five reliability
metrics. To remove the correlations induced by using different machine
learning algorithms we first group the approaches as before into seven groups.
Then, for each group, we subtract off the mean for each metric, making it
centered around $0$. We then combine the zero-centered data of each group into
one dataset from which we compute the Pearson correlation and $R^{2}$ values
for each pair of metrics. The results are shown in fig. 17 where the color
corresponds to correlation (and can be negative or positive) while the number
corresponds to $R^{2}$, which is strictly positive. Values of $1$ along the
diagonal are where each metric is compared to itself, and we only show one set
of off-diagonal values as these correlation metrics are symmetric.

For almost all pairs of metrics there does not appear to exist a strong
relationship between them, with most of the $R^{2}$ values being less than
$0.2$. This finding is consistent with the observation that deep neural
networks are underspecified [179] so designing to a particular metric (such as
in-distribution performance) does not constrain other metrics (hence the low
correlation). There are, however, a few exceptions. There is a positive
correlation between DS robustness and calibration for all three datasets. This
effect seems to primarily come from the fine-tuned models and is discussed
further in section B.7. There is a weak negative correlation between ID
performance and adversarial robustness (a relationship previously observed
[270]). There are also dataset-specific relationships. For FMoW, there is a
correlation with OOD detection and several other metrics, while ID performance
and calibration are related for Camelyon17. We note that these correlations
(and lack thereof) may be specific to the range of scores achieved by the
models in this study. Additional relationships may exist for lower or higher
performing models.

##  5 Discussion

In this section, we first distill our investigation into the reliability of
machine learning systems, highlighting three categories of findings: the
pervasive issue of underspecification, the effects of targeted algorithmic
improvements, and the broader reliability enhancements achieved by certain
techniques. We then propose some research questions that emerged out of our
analysis and lastly we conclude.

###  5.1 Summary of Key Findings

Here we summarize the key findings from our holistic assessment of the
reliability of machine learning systems.

#### Underspecification

In accordance with prior work [201, 179], we find that deep neural networks
are underspecified (selecting for one metric does not constrain the others)
across all reliability metrics we measure. We found almost no intrinsic
correlation between any of the reliability metrics when measured on over $500$
models trained in diverse ways across three real-world datasets. This
conclusion has several key implications. Firstly, it underscores the critical
need for comprehensive evaluation methodologies in the development of machine
learning algorithms, of which the HR score is a small step towards. Not only
should individual reliability metrics be assessed and optimized, but
understanding their interdependencies and the potential trade-offs is equally
essential. Secondly, it highlights the importance of model selection prior to
deployment. Any reliability metric that is not measured and selected could
take on a wide range of values, potentially hurting overall system
performance. Lastly, the lack of connections between these metrics means there
can be substantial value in developing techniques that improve one of these
reliability metrics in isolation (like temperature scaling does for
calibration). These bespoke methods could then be combined to construct high-
reliability models in a more modular fashion.

#### Isolated Improvements

Some algorithms we investigated lead to improvements on just one of the
metrics and sometimes at the cost of others. Adversarial training was the only
approach to make substantial and consistent improvements in adversarial
robustness, but in the case of iWildCam it always came at the cost of in-
distribution performance and distribution shift robustness. Temperature
scaling nearly always improved calibration (though it was the least effective
on the binary classification task of Camelyon17), and leaves the other metrics
unchanged. Learning from unlabeled data lead to modest improvements in DS
robustness and sometimes calibration but the improvements varied significantly
across datasets and connection to specific algorithms was tenuous. Sub-group
aware training methods failed to consistently improve DS robustness, the
metric they were designed for, but did seem to have a positive effect on
calibration.

#### Comprehensive Reliability Improvements

Other techniques, however, had substantial improvements to reliability across
almost all metrics. Ensembling models improved all metrics except OOD
detection leading to an average increase of $0.12$ in HR score across
datasets. There was an average increase of about $0.08$ in HR score for the
best models under the two non-adversarial data augmentations where the
improvements came in every metric except in-distribution performance. Fine-
tuning large pre-trained models had large improvements for all three datasets
with an average increase of HR score by about $0.08$.

###  5.2 Future Research Directions

Our study has highlighted several important questions and research gaps that
warrant further exploration.

#### Improved Understanding of Pre-Trained Representations

Pre-trained models offer robust performance and calibration, along with OOD
detection capabilities, but the reasons for their effectiveness remain
unclear. Preliminary work, such as by [173], suggests that features in pre-
trained ViTs contain semantic segmentation information, contrasting with their
supervised counterparts. Additional research is needed to understand these
representations and how they arise.

#### Refining Fine-Tuning Dynamics

To leverage the full potential of fine-tuning, we must explore its
multidimensional design space. This includes understanding which architectures
are best suited for fine-tuning, the ideal pairing of pre-training datasets
with architectures and downstream tasks, and the optimization strategies that
yield the best results. Moreover, exploring whether fine-tuning can enhance
adversarial robustness is of interest, as suggested by [215].

#### Implications of Pre-Trained Models as Backbones

If pre-trained models will form the foundation of highly reliable machine
learning systems, then it will be crucial to ensure their widespread
availability and to safeguard them against bias and security vulnerabilities.
Much work regarding the widespread, fair, and secure deployment of large pre-
trained models will fall into the domain of policy and governance [304, 167,
165]. From a technical standpoint, a variety of reliability metrics and
vulnerabilities should be investigated. Recent work [234] has performed a
holistic evaluation of language models and covers many of the same evaluation
metrics, but focused on natural language tasks. ML systems are also
susceptible to backdoor trojan attacks where models are manipulated by an
adversary to have different behavior when a particular set of circumstances
occurs [199]. Work in this area, such as the TrojAI Software Framework from
[218], allows researchers to rapidly and comprehensively test new trojan
detection methods. Cybersecurity risks are another major security
vulnerability when deploying large pre-trained models; work in this area, for
example, could build upon efforts to leverage ML to detect intruders of these
systems [277]. Beyond these concerns, a challenge lies in devising equitable
mechanisms for shouldering the developmental costs of these foundational
models, thus broadening their benefits.

#### Effective Use of Task-Specific Unlabeled Data

Despite notable reliability improvements achieved by training on extensive
corpora of unlabeled data, the use of task-specific unlabeled data was shown
to be less successful. This discrepancy might stem from the chosen algorithmic
approaches or the relative size disparity between these datasets and those
used in models like CLIP. Further investigation into effective utilization of
task-specific unlabeled data is needed, with a promising starting point in
approaches that achieve SOTA performance on ImageNet without much extra data
such as Data-efficient Image Transformers [284] and Masked Autoencoders [205].

#### Establishing Challenge Problems to Determine Requirements

Comparing machine learning models’ reliability across various metrics is
straightforward; the challenge lies in determining when they are _reliable
enough_. Understanding the context of these systems is vital for gauging the
required reliability properties. A productive endeavor could be to establish
real-world challenge problems (not just benchmarks), where machine learning
models form crucial but not stand-alone components. Researchers would be free
to design systems around these components to help ensure reliability, and from
these designs may emerge target requirements that individual components need
to achieve. Having such requirements could help drive research and inform
regulation.

###  5.3 Conclusion

This study conducted a holistic assessment of machine learning systems,
focusing on their reliability across diverse metrics. Findings confirmed the
phenomenon of underspecification in deep neural networks, emphasizing the
necessity of comprehensive evaluation methodologies and careful model
selection prior to deployment. While some algorithmic approaches showed
improvements on specific metrics, often at the expense of others, techniques
such as ensembling and fine-tuning large pre-trained models yielded
considerable improvements in holistic reliability. As AI continues to be
increasingly deployed in high-stakes applications, we hope that our findings
guide future research to develop more reliable and robust systems.

##  6 Acknowledgements

This work was supported by the Foundational Research Grants program at
Georgetown University’s Center for Security and Emerging Technology. We would
like to thank Helen Toner, Dewey Murdick, Drew Lohn, Andrea Guerrero, Esen
Yel, Robert Moss, Josh Ott, Mansur Arief, and Sydney Katz for their valuable
feedback and insightful discussion.

## References

  * [1] DOD Directive 3000.09  “Autonomy in Weapons Systems”  In _Department of Defense, USA_ , 2023
  * [2] Moloud Abdar et al.  “A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges”  In _Information Fusion_ 76 Elsevier, 2021, pp. 243–297
  * [3] Pulkit Agrawal, Ross Girshick and Jitendra Malik  “Analyzing the Performance of Multilayer Neural Networks for Object Recognition”  In _European Conference on Computer Vision (ECCV)_ , 2014
  * [4] Anders Johan Andreassen, Yasaman Bahri, Behnam Neyshabur and Rebecca Roelofs  “The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning”  In _Transactions on Machine Learning Research_ , 2022
  * [5] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion and Matthias Hein  “Square Attack: A Query-Efficient Black-Box Adversarial Attack via Random Search”  In _European Conference on Computer Vision (ECCV)_ , 2020
  * [6] Anastasios N. Angelopoulos and Stephen Bates  “Conformal prediction: A gentle introduction”  In _Foundations and Trends in Machine Learning_ 16.4 Now Publishers, Inc., 2023, pp. 494–591
  * [7] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan and Jitendra Malik  “Uncertainty Sets for Image Classifiers using Conformal Prediction”  In _International Conference on Learning Representations_ , 2021  URL: https://openreview.net/forum?id=eNdiU_DbM9
  * [8] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani and David Lopez-Paz  “Invariant Risk Minimization”  In _arXiv e-prints_ , 2019  DOI: 10.48550/arXiv.1907.02893
  * [9] Philip Bachman, R Devon Hjelm and William Buchwalter  “Learning Representations by Maximizing Mutual Information Across Views”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019
  * [10] Hangbo Bao, Li Dong, Songhao Piao and Furu Wei  “BEiT: BERT Pre-Training of Image Transformers”  In _International Conference on Learning Representations (ICLR)_ , 2022
  * [11] Sara Beery et al.  “Efficient Pipeline for Automating Species ID in New Camera Trap Projects”  In _Biodiversity Information Science and Standards_ 3 Pensoft Publishers, 2019  DOI: 10.3897/biss.3.37222
  * [12] Sara Beery et al.  “Synthetic Examples Improve Generalization for Rare Classes”  In _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 863–873
  * [13] Emily M Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell  “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”  In _ACM Conference on Fairness, Accountability, and Transparency (FAccT)_ , 2021
  * [14] Yoshua Bengio, Aaron Courville and Pascal Vincent  “Representation Learning: A Review and New Perspectives”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 35.8, 2013, pp. 1798–1828
  * [15] Abeba Birhane et al.  “The values encoded in machine learning research”  In _ACM Conference on Fairness, Accountability, and Transparency (FAccT)_ , 2022
  * [16] Tolga Bolukbasi et al.  “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016
  * [17] Tom Brown et al.  “Language models are few-shot learners”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020
  * [18] Joy Buolamwini and Timnit Gebru  “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification”  In _Conference on Fairness, Accountability and Transparency (FAccT)_ , 2018
  * [19] Aylin Caliskan, Joanna J. Bryson and Arvind Narayanan  “Semantics derived automatically from language corpora contain human-like biases”  In _Science_ 356.6334 American Association for the Advancement of Science, 2017, pp. 183–186
  * [20] Mathilde Caron et al.  “Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020
  * [21] Mathilde Caron et al.  “Emerging Properties in Self-Supervised Vision Transformers”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [22] Mehdi Cherti et al.  “Reproducible Scaling Laws for Contrastive Language-Image Learning”  In _arXiv_ , 2022
  * [23] Francesco Croce and Matthias Hein  “Minimally Distorted Adversarial Examples with a Fast Adaptive Boundary Attack”  In _International Conference on Machine Learning (ICML)_ , 2020
  * [24] Francesco Croce and Matthias Hein  “Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-Free Attacks”  In _International Conference on Machine Learning (ICML)_ , 2020
  * [25] Francesco Croce et al.  “RobustBench: A Standardized Adversarial Robustness Benchmark”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021
  * [26] Ekin D Cubuk, Barret Zoph, Jonathon Shlens and Quoc V Le  “RandAugment: Practical automated data augmentation with a reduced search space”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2020, pp. 702–703
  * [27] Alexander D’Amour et al.  “Underspecification Presents Challenges for Credibility in Modern Machine Learning”  In _Journal of Machine Learning Research_ , 2020
  * [28] Stéphane d’Ascoli et al.  “ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [29] DO-178C  “DO-178C: Software Considerations in Airborne Systems and Equipment Certification”  In _RTCA_ , 2011
  * [30] Carl Doersch, Abhinav Gupta and Alexei A. Efros  “Unsupervised Visual Representation Learning by Context Prediction”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2015
  * [31] Alexey Dosovitskiy et al.  “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”  In _International Conference on Learning Representations (ICLR)_ , 2021
  * [32] John C. Duchi, Peter W. Glynn and Hongseok Namkoong  “Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach”  In _Mathematics of Operations Research_ 46.3 INFORMS, 2021, pp. 946–969
  * [33] European Commission  “Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts”  In _Office for Official Publications of the European Communities Luxembourg_ , 2021
  * [34] Executive Order  “Executive Order 13960: Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government”  In _Executive Office of the President_ , 2020
  * [35] Alex Fang, Simon Kornblith and Ludwig Schmidt  “Does Progress on ImageNet Transfer to Real-World Datasets?”  In _arXiv e-prints_ , 2023, pp. arXiv:2301.04644  DOI: 10.48550/arXiv.2301.04644
  * [36] Pierre Foret, Ariel Kleiner, Hossein Mobahi and Behnam Neyshabur  “Sharpness-Aware Minimization for Efficiently Improving Generalization”  In _International Conference on Learning Representations_ , 2021
  * [37] Yarin Gal and Zoubin Ghahramani  “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning”  In _International Conference on Machine Learning (ICML)_ , 2016
  * [38] Yaroslav Ganin et al.  “Domain-Adversarial Training of Neural Networks”  In _Journal of Machine Learning Research_ 17.1 JMLR. org, 2016, pp. 2096–2030
  * [39] Jonas Geiping et al.  “How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization”  In _arXiv e-prints_ , 2022, pp. arXiv:2210.06441  DOI: 10.48550/arXiv.2210.06441
  * [40] Robert Geirhos et al.  “Generalisation in Humans and Deep Neural Networks”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018
  * [41] Amirata Ghorbani and James Zou  “Data Shapley: Equitable valuation of data for machine learning”  In _International Conference on Machine Learning (ICML)_ , 2019
  * [42] Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik  “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014
  * [43] Raphael Gontijo-Lopes, Yann Dauphin and Ekin Dogus Cubuk  “No One Representation to Rule Them All: Overlapping Features of Training Methods”  In _International Conference on Learning Representations (ICLR)_ , 2022
  * [44] Ian Goodfellow, Jonathon Shlens and Christian Szegedy  “Explaining and Harnessing Adversarial Examples”  In _International Conference on Learning Representations (ICLR)_ , 2015
  * [45] Ian Goodfellow et al.  “Generative adversarial networks”  In _Communications of the ACM_ 63.11 ACM New York, NY, USA, 2020, pp. 139–144
  * [46] Keren Gu et al.  “Using Videos to Evaluate Image Model Robustness”  In _SafeML Workshop at the International Conference on Learning Representations (ICLR)_ , 2019
  * [47] Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg  “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain”  In _arXiv e-prints_ , 2019, pp. arXiv:1708.06733  DOI: 10.48550/arXiv.1708.06733
  * [48] Devin Guillory et al.  “Predicting with Confidence on Unseen Distributions”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 1134–1144
  * [49] Ishaan Gulrajani and David Lopez-Paz  “In Search of Lost Domain Generalization”  In _International Conference on Learning Representations (ICLR)_ , 2021
  * [50] Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q Weinberger  “On Calibration of Modern Neural Networks”  In _International Conference on Machine Learning (ICML)_ , 2017
  * [51] Andrew Hansen  “WAAS Precision Approach Metrics: Accuracy, Integrity, Continuity and Availability”  In _Federal Aviation Administration (FAA)_ , 2004
  * [52] Trevor Hastie, Robert Tibshirani, Jerome H. Friedman and Jerome H. Friedman  “The Elements of Statistical Learning: Data mining, Inference, and Prediction”  Springer, 2009
  * [53] Kaiming He et al.  “Momentum Contrast for Unsupervised Visual Representation Learning”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [54] Kaiming He et al.  “Masked Autoencoders are Scalable Vision Learners”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [55] Dan Hendrycks and Thomas Dietterich  “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”  In _International Conference on Learning Representations (ICLR)_ , 2019
  * [56] Dan Hendrycks and Kevin Gimpel  “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”  In _International Conference on Learning Representations (ICLR)_ , 2017
  * [57] Dan Hendrycks et al.  “AugMix: A Simple Method to Improve Robustness and Uncertainty under Data Shift”  In _International Conference on Learning Representations (ICLR)_ , 2020
  * [58] Dan Hendrycks et al.  “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [59] Dan Hendrycks et al.  “Scaling Out-of-Distribution Detection for Real-World Settings”  In _International Conference on Machine Learning (ICML)_ , 2022
  * [60] Sergio Hernández, Diego Vergara, Matias Valdenegro-Toro and Felipe Jorquera  “Improving Predictive Uncertainty Estimation Using Dropout–Hamiltonian Monte Carlo”  In _Soft Computing_ 24.6 Springer, 2020, pp. 4307–4322
  * [61] Weihua Hu, Gang Niu, Issei Sato and Masashi Sugiyama  “Does Distributionally Robust Supervised Learning Give Robust Classifiers?”  In _International Conference on Machine Learning (ICML)_ , 2018
  * [62] Andrew Ilyas et al.  “Adversarial Examples Are Not Bugs, They Are Features”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019
  * [63] Ahmadreza Jeddi, Mohammad Javad Shafiee and Alexander Wong  “A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning”  In _arXiv e-prints_ , 2020  DOI: 10.48550/arXiv.2012.13628
  * [64] Chao Jia et al.  “Scaling up Visual and Vision-Language Representation Learning with Noisy Text Supervision”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [65] Fei Jiang et al.  “Artificial Intelligence in Healthcare: Past, Present and Future”  In _Stroke and Vascular Neurology_ 2.4 BMJ Specialist Journals, 2017
  * [66] Kiran Karra, Chace Ashcraft and Neil Fendley  “The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models”  In _arXiv e-prints_ , 2020, pp. arXiv:2003.07233  DOI: 10.48550/arXiv.2003.07233
  * [67] Niki Kilbertus, Giambattista Parascandolo and Bernhard Schölkopf  “Generalization in Anti-Causal Learning”  In _NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning_ , 2018
  * [68] Pang Wei Koh et al.  “Concept Bottleneck Models”  In _International Conference on Machine Learning_ , 2020, pp. 5338–5348  PMLR
  * [69] Pang Wei Koh et al.  “WILDS: A Benchmark of in-the-Wild Distribution Shifts”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [70] Simon Kornblith, Jonathon Shlens and Quoc V Le  “Do Better ImageNet Models Transfer Better?”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019
  * [71] Klemen Kotar et al.  “Contrasting Contrastive Self-Supervised Representation Learning Pipelines”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [72] David Krueger et al.  “Out-of-Distribution Generalization via Risk Extrapolation (REx)”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [73] Ananya Kumar, Ruoqi Shen, Sébastien Bubeck and Suriya Gunasekar  “How to Fine-Tune Vision Models with SGD”  In _arXiv e-prints_ , 2022, pp. arXiv:2211.09359  DOI: 10.48550/arXiv.2211.09359
  * [74] Aviral Kumar, Sunita Sarawagi and Ujjwal Jain  “Trainable Calibration Measures for Neural Networks from Kernel Mean Embeddings”  In _International Conference on Machine Learning (ICML)_ , 2018
  * [75] Balaji Lakshminarayanan, Alexander Pritzel and Charles Blundell  “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017
  * [76] Gustav Larsson, Michael Maire and Gregory Shakhnarovich  “Learning Representations for Automatic Colorization”  In _European Conference on Computer Vision (ECCV)_ , 2016
  * [77] Dong-Hyun Lee  “Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks”  In _Workshop on Challenges in Representation Learning, International Conference on Machine Learning (ICML)_ , 2013
  * [78] Stefan Lee et al.  “Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016
  * [79] Nancy G. Leveson  “Engineering a safer world: Systems thinking applied to safety”  The MIT Press, 2016
  * [80] Dan Levi, Liran Gispan, Niv Giladi and Ethan Fetaya  “Evaluating and Calibrating Uncertainty Prediction in Regression Tasks”  In _Sensors_ 22.15 MDPI, 2022, pp. 5540
  * [81] Yingwei Li et al.  “Shape-Texture Debiased Neural Network Training”  In _International Conference on Learning Representations (ICLR)_ , 2021
  * [82] Percy Liang et al.  “Holistic Evaluation of Language Models”  In _arXiv e-prints_ , 2022, pp. arXiv:2211.09110  DOI: 10.48550/arXiv.2211.09110
  * [83] Shiyu Liang, Yixuan Li and R. Srikant  “Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks”  In _International Conference on Learning Representations (ICLR)_ , 2018
  * [84] Weixin Liang et al.  “Advances, Challenges and Opportunities in Creating Data for Trustworthy AI”  In _Nature Machine Intelligence_ 4.8, 2022, pp. 669–677
  * [85] Weitang Liu, Xiaoyun Wang, John Owens and Yixuan Li  “Energy-based Out-of-distribution Detection”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020
  * [86] Ze Liu et al.  “Swin transformer: Hierarchical vision transformer using shifted windows”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [87] Zhuang Liu et al.  “A convnet for the 2020s”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [88] Yifang Ma, Zhenyu Wang, Hong Yang and Lin Yang  “Artificial Intelligence Applications in the Development of Autonomous Vehicles: A Survey”  In _IEEE/CAA Journal of Automatica Sinica_ 7.2, 2020, pp. 315–329
  * [89] Aleksander Madry et al.  “Towards Deep Learning Models Resistant to Adversarial Attacks”  In _International Conference on Learning Representations (ICLR)_ , 2018
  * [90] Dhruv Mahajan et al.  “Exploring the limits of weakly supervised pretraining”  In _European Conference on Computer Vision (ECCV)_ , 2018
  * [91] Andrey Malinin et al.  “Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks”  In _Advances in Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks_ , 2021
  * [92] Andrey Malinin et al.  “Shifts 2.0: Extending The Dataset of Real Distributional Shifts”  In _arXiv e-prints_ , 2022  DOI: 10.48550/arXiv.2206.15407
  * [93] Horia Mania and Suvrit Sra  “Why Do Classifier Accuracies Show Linear Trends Under Distribution Shift?”  In _arXiv e-prints_ , 2020  DOI: 10.48550/arXiv.2012.15483
  * [94] Horia Mania et al.  “Model Similarity Mitigates Test Set Overuse”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019
  * [95] Jonathan Masci, Ueli Meier, Dan Cireşan and Jürgen Schmidhuber  “Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction”  In _International Conference on Artificial Neural Networks (ICANN)_ , 2011
  * [96] Ninareh Mehrabi et al.  “A survey on bias and fairness in machine learning”  In _ACM Computing Surveys (CSUR)_ 54.6 ACM New York, NY, USA, 2021, pp. 1–35
  * [97] John P Miller et al.  “Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [98] Eric Mintun, Alexander Kirillov and Saining Xie  “On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021
  * [99] Ishan Misra and Laurens van der Maaten  “Self-Supervised Learning of Pretext-Invariant Representations”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [100] Forrest E. Morgan et al.  “Military Applications of Artificial Intelligence: Ethical Concerns in an Uncertain World”  RAND Corporation, 2020
  * [101] Jishnu Mukhoti, Pontus Stenetorp and Yarin Gal  “On the Importance of Strong Baselines in Bayesian Deep Learning”  In _Bayesian Deep Learning Workshop at Neural Information Processing Systems (NeurIPS)_ , 2018
  * [102] Thao Nguyen et al.  “Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022
  * [103] NIST  “AI Risk Management Framework: Second Draft”  In _National Institute of Standards and Technology (NIST)_ , 2022
  * [104] Mehdi Noroozi, Hamed Pirsiavash and Paolo Favaro  “Representation Learning by Learning to Count”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2017
  * [105] Curtis G Northcutt, Anish Athalye and Jonas Mueller  “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks”  In _Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track_ , 2021
  * [106] Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan  “Dissecting racial bias in an algorithm used to manage the health of populations”  In _Science_ 366.6464 American Association for the Advancement of Science, 2019, pp. 447–453
  * [107] OECD  “Recommendation of the Council on Artificial Intelligence”  In _Organisation for Economic Co-operation and Development (OECD)_ , 2019
  * [108] David Opitz and Richard Maclin  “Popular Ensemble Methods: An Empirical Study”  In _Journal of Artificial Intelligence Research_ 11, 1999, pp. 169–198
  * [109] Michael P. Perrone and Leon N. Cooper  “When Networks Disagree: Ensemble Methods for Hybrid Neural Networks”  In _How We Learn; How We Remember: Toward An Understanding Of Brain And Neural Systems: Selected Papers of Leon N. Cooper_ World Scientific, 1995, pp. 342–358
  * [110] Hieu Pham, Zihang Dai, Qizhe Xie and Quoc V Le  “Meta Pseudo Labels”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 11557–11568
  * [111] Alec Radford et al.  “Learning Transferable Visual Models from Natural Language Supervision”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [112] Alexander J Ratner et al.  “Data programming: Creating large training sets, quickly”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016
  * [113] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt and Vaishaal Shankar  “Do ImageNet Classifiers Generalize to ImageNet?”  In _International Conference on Machine Learning (ICML)_ , 2019
  * [114] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto and Percy Liang  “Distributionally Robust Neural Networks”  In _International Conference on Learning Representations (ICLR)_ , 2020
  * [115] Shiori Sagawa et al.  “Extending the WILDS Benchmark for Unsupervised Adaptation”  In _International Conference on Learning Representations (ICLR)_ , 2022
  * [116] Kelley M. Sayler and Daniel S. Hoadley  “Artificial Intelligence and National Security”  In _Congressional Research Service_ , 2020
  * [117] Morgan Klaus Scheuerman, Alex Hanna and Emily Denton  “Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development”  In _Proceedings of the ACM on Human-Computer Interaction_ 5.CSCW2 ACM New York, NY, USA, 2021, pp. 1–37
  * [118] Ludwig Schmidt et al.  “Adversarially Robust Generalization Requires More Data”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018
  * [119] Seonguk Seo, Paul Hongsuck Seo and Bohyung Han  “Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic Inferences”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019
  * [120] Vaishaal Shankar et al.  “Do Image Classifiers Generalize Across Time?”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [121] Apoorva Sharma, Navid Azizan and Marco Pavone  “Sketching curvature for efficient out-of-distribution detection for deep neural networks”  In _Conference on Uncertainty in Artificial Intelligence (UAI)_ , 2021
  * [122] Karen Simonyan and Andrew Zisserman  “Very Deep Convolutional Networks for Large-Scale Image Recognition”  In _arXiv e-prints_ , 2014, pp. arXiv:1409.1556  DOI: 10.48550/arXiv.1409.1556
  * [123] Mannat Singh et al.  “Revisiting weakly supervised pre-training of visual perception models”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [124] Kihyuk Sohn et al.  “FixMatch: Simplifying semi-supervised learning with consistency and confidence”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020
  * [125] Robin Sommer and Vern Paxson  “Outside the Closed World: On Using Machine Learning for Network Intrusion Detection”  In _IEEE Symposium on Security and Privacy_ , 2010
  * [126] Banavar Sridhar  “Applications of Machine Learning Techniques to Aviation Operations: Promises and Challenges”  In _International Conference on Artificial Intelligence and Data Analytics for Air Transportation (AIDA-AT)_ , 2020, pp. 1–12
  * [127] Baochen Sun and Kate Saenko  “Deep CORAL: Correlation Alignment for Deep Domain Adaptation”  In _European Conference on Computer Vision (ECCV)_ , 2016
  * [128] Baochen Sun, Jiashi Feng and Kate Saenko  “Return of Frustratingly Easy Domain Adaptation”  In _AAAI Conference on Artificial Intelligence (AAAI)_ , 2016
  * [129] Mingxing Tan and Quoc Le  “EfficientNet: Rethinking model scaling for convolutional neural networks”  In _International Conference on Machine Learning (ICML)_ , 2019
  * [130] Rohan Taori et al.  “Measuring Robustness to Natural Distribution Shifts in Image Classification”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020
  * [131] Sunil Thulasidasan et al.  “On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019
  * [132] Hugo Touvron et al.  “Training Data-efficient Image Transformers & Distillation through Attention”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [133] Dimitris Tsipras et al.  “Robustness May Be at Odds with Accuracy”  In _International Conference on Learning Representations (ICLR)_ , 2019
  * [134] Zhengzhong Tu et al.  “MaxViT: Multi-axis vision transformer”  In _European Conference on Computer Vision (ECCV)_ , 2022
  * [135] Vladimir Vapnik  “Principles of Risk Minimization for Learning Theory”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 4, 1991
  * [136] William Wang et al.  “Safer Classification by Synthesis”  In _arXiv_ , 2017
  * [137] Yeming Wen, Dustin Tran and Jimmy Ba  “BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning”  In _arXiv preprint arXiv:2002.06715_ , 2020
  * [138] Mitchell Wortsman et al.  “Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time”  In _International Conference on Machine Learning (ICML)_ , 2022
  * [139] Mitchell Wortsman et al.  “Robust Fine-Tuning of Zero-Shot Models”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [140] Zhirong Wu, Yuanjun Xiong, Stella X Yu and Dahua Lin  “Unsupervised feature learning via non-parametric instance discrimination”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018
  * [141] Qizhe Xie, Minh-Thang Luong, Eduard Hovy and Quoc V Le  “Self-training with noisy student improves ImageNet classification”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [142] Zhenda Xie et al.  “SimMIM: A Simple Framework for Masked Image Modeling”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [143] Han Xu et al.  “Adversarial Attacks and Defenses in Images, Graphs and Text: A Review”  In _International Journal of Automation and Computing_ 17.2, 2020
  * [144] Mingle Xu, Sook Yoon, Alvaro Fuentes and Dong Sun Park  “A Comprehensive Survey of Image Augmentation Techniques for Deep Learning”  In _Pattern Recognition_ 137, 2023, pp. 109347
  * [145] Ruijia Xu, Guanbin Li, Jihan Yang and Liang Lin  “Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019
  * [146] Larry Yaeger, Richard Lyon and Brandyn Webb  “Effective Training of a Neural Network Character Classifier for Word Recognition”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 1996
  * [147] Huaxiu Yao et al.  “Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time”  In _Shift Happens Workshop at International Conference on Machine Learning (ICML)_ , 2022
  * [148] David Yarowsky  “Unsupervised Word Sense Disambiguation Rivaling Supervised Methods”  In _Annual Meeting of the Association for Computational Linguistics_ , 1995
  * [149] Jiahui Yu et al.  “CoCa: Contrastive Captioners are Image-Text Foundation Models”  In _Transactions on Machine Learning Research_ , 2022
  * [150] Kun-Hsing Yu, Andrew L. Beam and Isaac S. Kohane  “Artificial intelligence in Healthcare”  In _Nature biomedical engineering_ 2.10 Nature Publishing Group UK London, 2018, pp. 719–731
  * [151] Daniel Ziegler et al.  “Adversarial Training for High-Stakes Reliability”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022
  * [152] Remco Zwetsloot, Helen Toner and Jeffrey Ding  “Beyond the AI Arms Race: America, China, and the Dangers of Zero-Sum Thinking”  In _Foreign Affairs_ , 2018

## References

  * [153] DOD Directive 3000.09  “Autonomy in Weapons Systems”  In _Department of Defense, USA_ , 2023
  * [154] Moloud Abdar et al.  “A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges”  In _Information Fusion_ 76 Elsevier, 2021, pp. 243–297
  * [155] Pulkit Agrawal, Ross Girshick and Jitendra Malik  “Analyzing the Performance of Multilayer Neural Networks for Object Recognition”  In _European Conference on Computer Vision (ECCV)_ , 2014
  * [156] Anders Johan Andreassen, Yasaman Bahri, Behnam Neyshabur and Rebecca Roelofs  “The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning”  In _Transactions on Machine Learning Research_ , 2022
  * [157] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion and Matthias Hein  “Square Attack: A Query-Efficient Black-Box Adversarial Attack via Random Search”  In _European Conference on Computer Vision (ECCV)_ , 2020
  * [158] Anastasios N. Angelopoulos and Stephen Bates  “Conformal prediction: A gentle introduction”  In _Foundations and Trends in Machine Learning_ 16.4 Now Publishers, Inc., 2023, pp. 494–591
  * [159] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan and Jitendra Malik  “Uncertainty Sets for Image Classifiers using Conformal Prediction”  In _International Conference on Learning Representations_ , 2021  URL: https://openreview.net/forum?id=eNdiU_DbM9
  * [160] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani and David Lopez-Paz  “Invariant Risk Minimization”  In _arXiv e-prints_ , 2019  DOI: 10.48550/arXiv.1907.02893
  * [161] Philip Bachman, R Devon Hjelm and William Buchwalter  “Learning Representations by Maximizing Mutual Information Across Views”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019
  * [162] Hangbo Bao, Li Dong, Songhao Piao and Furu Wei  “BEiT: BERT Pre-Training of Image Transformers”  In _International Conference on Learning Representations (ICLR)_ , 2022
  * [163] Sara Beery et al.  “Synthetic Examples Improve Generalization for Rare Classes”  In _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 863–873
  * [164] Sara Beery et al.  “Efficient Pipeline for Automating Species ID in New Camera Trap Projects”  In _Biodiversity Information Science and Standards_ 3 Pensoft Publishers, 2019  DOI: 10.3897/biss.3.37222
  * [165] Emily M Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell  “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”  In _ACM Conference on Fairness, Accountability, and Transparency (FAccT)_ , 2021
  * [166] Yoshua Bengio, Aaron Courville and Pascal Vincent  “Representation Learning: A Review and New Perspectives”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 35.8, 2013, pp. 1798–1828
  * [167] Abeba Birhane et al.  “The values encoded in machine learning research”  In _ACM Conference on Fairness, Accountability, and Transparency (FAccT)_ , 2022
  * [168] Tolga Bolukbasi et al.  “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016
  * [169] Tom Brown et al.  “Language models are few-shot learners”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020
  * [170] Joy Buolamwini and Timnit Gebru  “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification”  In _Conference on Fairness, Accountability and Transparency (FAccT)_ , 2018
  * [171] Aylin Caliskan, Joanna J. Bryson and Arvind Narayanan  “Semantics derived automatically from language corpora contain human-like biases”  In _Science_ 356.6334 American Association for the Advancement of Science, 2017, pp. 183–186
  * [172] Mathilde Caron et al.  “Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020
  * [173] Mathilde Caron et al.  “Emerging Properties in Self-Supervised Vision Transformers”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [174] Mehdi Cherti et al.  “Reproducible Scaling Laws for Contrastive Language-Image Learning”  In _arXiv_ , 2022
  * [175] Francesco Croce et al.  “RobustBench: A Standardized Adversarial Robustness Benchmark”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021
  * [176] Francesco Croce and Matthias Hein  “Minimally Distorted Adversarial Examples with a Fast Adaptive Boundary Attack”  In _International Conference on Machine Learning (ICML)_ , 2020
  * [177] Francesco Croce and Matthias Hein  “Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-Free Attacks”  In _International Conference on Machine Learning (ICML)_ , 2020
  * [178] Ekin D Cubuk, Barret Zoph, Jonathon Shlens and Quoc V Le  “RandAugment: Practical automated data augmentation with a reduced search space”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2020, pp. 702–703
  * [179] Alexander D’Amour et al.  “Underspecification Presents Challenges for Credibility in Modern Machine Learning”  In _Journal of Machine Learning Research_ , 2020
  * [180] Stéphane d’Ascoli et al.  “ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [181] DO-178C  “DO-178C: Software Considerations in Airborne Systems and Equipment Certification”  In _RTCA_ , 2011
  * [182] Carl Doersch, Abhinav Gupta and Alexei A. Efros  “Unsupervised Visual Representation Learning by Context Prediction”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2015
  * [183] Alexey Dosovitskiy et al.  “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”  In _International Conference on Learning Representations (ICLR)_ , 2021
  * [184] John C. Duchi, Peter W. Glynn and Hongseok Namkoong  “Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach”  In _Mathematics of Operations Research_ 46.3 INFORMS, 2021, pp. 946–969
  * [185] European Commission  “Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts”  In _Office for Official Publications of the European Communities Luxembourg_ , 2021
  * [186] Executive Order  “Executive Order 13960: Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government”  In _Executive Office of the President_ , 2020
  * [187] Alex Fang, Simon Kornblith and Ludwig Schmidt  “Does Progress on ImageNet Transfer to Real-World Datasets?”  In _arXiv e-prints_ , 2023, pp. arXiv:2301.04644  DOI: 10.48550/arXiv.2301.04644
  * [188] Pierre Foret, Ariel Kleiner, Hossein Mobahi and Behnam Neyshabur  “Sharpness-Aware Minimization for Efficiently Improving Generalization”  In _International Conference on Learning Representations_ , 2021
  * [189] Yarin Gal and Zoubin Ghahramani  “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning”  In _International Conference on Machine Learning (ICML)_ , 2016
  * [190] Yaroslav Ganin et al.  “Domain-Adversarial Training of Neural Networks”  In _Journal of Machine Learning Research_ 17.1 JMLR. org, 2016, pp. 2096–2030
  * [191] Jonas Geiping et al.  “How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization”  In _arXiv e-prints_ , 2022, pp. arXiv:2210.06441  DOI: 10.48550/arXiv.2210.06441
  * [192] Robert Geirhos et al.  “Generalisation in Humans and Deep Neural Networks”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018
  * [193] Amirata Ghorbani and James Zou  “Data Shapley: Equitable valuation of data for machine learning”  In _International Conference on Machine Learning (ICML)_ , 2019
  * [194] Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik  “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014
  * [195] Raphael Gontijo-Lopes, Yann Dauphin and Ekin Dogus Cubuk  “No One Representation to Rule Them All: Overlapping Features of Training Methods”  In _International Conference on Learning Representations (ICLR)_ , 2022
  * [196] Ian Goodfellow et al.  “Generative adversarial networks”  In _Communications of the ACM_ 63.11 ACM New York, NY, USA, 2020, pp. 139–144
  * [197] Ian Goodfellow, Jonathon Shlens and Christian Szegedy  “Explaining and Harnessing Adversarial Examples”  In _International Conference on Learning Representations (ICLR)_ , 2015
  * [198] Keren Gu et al.  “Using Videos to Evaluate Image Model Robustness”  In _SafeML Workshop at the International Conference on Learning Representations (ICLR)_ , 2019
  * [199] Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg  “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain”  In _arXiv e-prints_ , 2019, pp. arXiv:1708.06733  DOI: 10.48550/arXiv.1708.06733
  * [200] Devin Guillory et al.  “Predicting with Confidence on Unseen Distributions”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 1134–1144
  * [201] Ishaan Gulrajani and David Lopez-Paz  “In Search of Lost Domain Generalization”  In _International Conference on Learning Representations (ICLR)_ , 2021
  * [202] Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q Weinberger  “On Calibration of Modern Neural Networks”  In _International Conference on Machine Learning (ICML)_ , 2017
  * [203] Andrew Hansen  “WAAS Precision Approach Metrics: Accuracy, Integrity, Continuity and Availability”  In _Federal Aviation Administration (FAA)_ , 2004
  * [204] Trevor Hastie, Robert Tibshirani, Jerome H. Friedman and Jerome H. Friedman  “The Elements of Statistical Learning: Data mining, Inference, and Prediction”  Springer, 2009
  * [205] Kaiming He et al.  “Masked Autoencoders are Scalable Vision Learners”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [206] Kaiming He et al.  “Momentum Contrast for Unsupervised Visual Representation Learning”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [207] Dan Hendrycks et al.  “Scaling Out-of-Distribution Detection for Real-World Settings”  In _International Conference on Machine Learning (ICML)_ , 2022
  * [208] Dan Hendrycks et al.  “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [209] Dan Hendrycks and Thomas Dietterich  “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”  In _International Conference on Learning Representations (ICLR)_ , 2019
  * [210] Dan Hendrycks and Kevin Gimpel  “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”  In _International Conference on Learning Representations (ICLR)_ , 2017
  * [211] Dan Hendrycks et al.  “AugMix: A Simple Method to Improve Robustness and Uncertainty under Data Shift”  In _International Conference on Learning Representations (ICLR)_ , 2020
  * [212] Sergio Hernández, Diego Vergara, Matias Valdenegro-Toro and Felipe Jorquera  “Improving Predictive Uncertainty Estimation Using Dropout–Hamiltonian Monte Carlo”  In _Soft Computing_ 24.6 Springer, 2020, pp. 4307–4322
  * [213] Weihua Hu, Gang Niu, Issei Sato and Masashi Sugiyama  “Does Distributionally Robust Supervised Learning Give Robust Classifiers?”  In _International Conference on Machine Learning (ICML)_ , 2018
  * [214] Andrew Ilyas et al.  “Adversarial Examples Are Not Bugs, They Are Features”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019
  * [215] Ahmadreza Jeddi, Mohammad Javad Shafiee and Alexander Wong  “A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning”  In _arXiv e-prints_ , 2020  DOI: 10.48550/arXiv.2012.13628
  * [216] Chao Jia et al.  “Scaling up Visual and Vision-Language Representation Learning with Noisy Text Supervision”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [217] Fei Jiang et al.  “Artificial Intelligence in Healthcare: Past, Present and Future”  In _Stroke and Vascular Neurology_ 2.4 BMJ Specialist Journals, 2017
  * [218] Kiran Karra, Chace Ashcraft and Neil Fendley  “The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models”  In _arXiv e-prints_ , 2020, pp. arXiv:2003.07233  DOI: 10.48550/arXiv.2003.07233
  * [219] Niki Kilbertus, Giambattista Parascandolo and Bernhard Schölkopf  “Generalization in Anti-Causal Learning”  In _NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning_ , 2018
  * [220] Pang Wei Koh et al.  “Concept Bottleneck Models”  In _International Conference on Machine Learning_ , 2020, pp. 5338–5348  PMLR
  * [221] Pang Wei Koh et al.  “WILDS: A Benchmark of in-the-Wild Distribution Shifts”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [222] Simon Kornblith, Jonathon Shlens and Quoc V Le  “Do Better ImageNet Models Transfer Better?”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019
  * [223] Klemen Kotar et al.  “Contrasting Contrastive Self-Supervised Representation Learning Pipelines”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [224] David Krueger et al.  “Out-of-Distribution Generalization via Risk Extrapolation (REx)”  In _International Conference on Machine Learning (ICML)_ , 2021
  * [225] Ananya Kumar, Ruoqi Shen, Sébastien Bubeck and Suriya Gunasekar  “How to Fine-Tune Vision Models with SGD”  In _arXiv e-prints_ , 2022, pp. arXiv:2211.09359  DOI: 10.48550/arXiv.2211.09359
  * [226] Aviral Kumar, Sunita Sarawagi and Ujjwal Jain  “Trainable Calibration Measures for Neural Networks from Kernel Mean Embeddings”  In _International Conference on Machine Learning (ICML)_ , 2018
  * [227] Balaji Lakshminarayanan, Alexander Pritzel and Charles Blundell  “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017
  * [228] Gustav Larsson, Michael Maire and Gregory Shakhnarovich  “Learning Representations for Automatic Colorization”  In _European Conference on Computer Vision (ECCV)_ , 2016
  * [229] Dong-Hyun Lee  “Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks”  In _Workshop on Challenges in Representation Learning, International Conference on Machine Learning (ICML)_ , 2013
  * [230] Stefan Lee et al.  “Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016
  * [231] Nancy G. Leveson  “Engineering a safer world: Systems thinking applied to safety”  The MIT Press, 2016
  * [232] Dan Levi, Liran Gispan, Niv Giladi and Ethan Fetaya  “Evaluating and Calibrating Uncertainty Prediction in Regression Tasks”  In _Sensors_ 22.15 MDPI, 2022, pp. 5540
  * [233] Yingwei Li et al.  “Shape-Texture Debiased Neural Network Training”  In _International Conference on Learning Representations (ICLR)_ , 2021
  * [234] Percy Liang et al.  “Holistic Evaluation of Language Models”  In _arXiv e-prints_ , 2022, pp. arXiv:2211.09110  DOI: 10.48550/arXiv.2211.09110