# Towards Robustness of Neural Networks

Steven Basart

(December 2021)

###### Abstract

###### Table of Contents

  1. 0 Introduction
    1. 1 Contributions
  2. 1 Background
    1. 1 Robustness
    2. 2 Adversarial Examples
    3. 3 Meta-Learning
    4. 4 Tasks
      1. 1 Multi-class Classification
      2. 2 Multi-label Classification
      3. 3 Segmentation
  3. 2 Natural Adversarial Examples
    1. 1 Overview
    2. 2 Related Work
    3. 3 The Design and Construction of ImageNet-A and ImageNet-O
      1. 1 Design
      2. 2 Collection
      3. 3 Illustrative Failure Modes
    4. 4 Experiments
    5. 5 Conclusion
  4. 3 Scaling Anomaly Detection to Large Scale Images
    1. 1 Overview
    2. 2 Related Work
    3. 3 Multi-Class Prediction for OOD Detection
    4. 4 The CAOS Benchmark
      1. 1 Experiments
    5. 5 Conclusion
  5. 4 Neural augmentations
    1. 1 Overview
    2. 2 Related Work
    3. 3 New Benchmarks
      1. 1 ImageNet-Renditions (ImageNet-R)
      2. 2 StreetView StoreFronts (SVSF)
    4. 4 DeepAugment
    5. 5 Experiments
      1. 1 Setup
      2. 2 Results
    6. 6 Conclusion
  6. 5 Multi-label Out-Of-Distribution Detection
    1. 1 Overview
    2. 2 Related Work
    3. 3 Methods
    4. 4 Results
    5. 5 Conclusion
  7. 6 Robustness in Few-Shot Learning
    1. 1 Overview
    2. 2 Set-Membership
    3. 3 Methods
    4. 4 Results
    5. 5 Conclusion
  8. 7 Conclusions
  9. Appendix A
    1. 1 ImageNet-A Classes
    2. 2 ImageNet-O Classes
    3. 3 Expanded Results
      1. 1 Full Architecture Results
      2. 2 Calibration
    4. 4 Real Blurry Images and ImageNet-C
    5. 5 Additional Results
    6. 6 Further Dataset Descriptions
    7. 7 DeepAugment Details

###### List of Figures

  1. 1 Examples of two novelty detection methods.
  2. (a) Points are drawn from a 2D Gaussian, and the isolation forest model is labeling the point X as an outlier.
  3. (b) Given differently sampled points leads to different lines which causes some number of points to be labeled as outliers. After sampling sufficient number of points then the best line is determined. Image credit: [Watanabe2013AFR].
  4. 2 An adversarial example showing how a small perturbation can change the label of an image. The adversarial noise added to image is magnified by 1000x to be visible to a human. Image credit: OpenAI
  5. 3 An example of an image with both a cat and a bird which shows multiple labels occurring together.
  6. 4 Highlighting the distinction between semantic segmentation on the left and instance segmentation on the right. Image source: [Arnab2018ConditionalRFNN]
  7. 1 Examples from ImageNet-A and ImageNet-O. The black text is the actual class, and the red text is a ResNet-50 prediction and its confidence. ImageNet-A contains images that classifiers should be able to classify. ImageNet-O contains out-of-distribution anomalies of unforeseen classes which should result in low-confidence predictions. ImageNet-1K models do not train on examples from "Photosphere" nor "Verdigris" classes, so these images are out-of-distribution.
  8. 2  Various ImageNet classifiers of different architectures fail to generalize well to ImageNet-A and ImageNet-O. Higher Accuracy and higher AUPR is better. See 4 for a description of the AUPR out-of-distribution detection measure. These specific model parameters are unseen during adversarial filtration, so our adversarially filtered examples transfer across models.
  9. 3  Previous work on out-of-distribution (OOD) detection uses synthetic anomalies and anomalies from wholly different data generating processes. For instance, previous work uses Bernoulli noise, blobs, the Describable Textures Dataset [cimpoi14describing], and Places365 scenes [zhou2017places] to test ImageNet out-of-distribution detectors. To our knowledge, we propose the first dataset of out-of-distribution examples collected for ImageNet models.
  10. 4  Additional adversarially filtered examples from the ImageNet-A dataset. Examples are adversarially selected to cause classifier accuracy to degrade. The black text is the actual class, and the red text is a ResNet-50 prediction.
  11. 6  Examples from ImageNet-A demonstrating classifier failure modes. Adjacent to each natural image is its heatmap generated by “gradCAM” [Selvaraju2019GradCAMVE]. The heatmap is generated by taking the classification prediction and computing the gradients from the given output prediction back to the image. The gradients are scaled to -1 to 1 range before adding back to the original image. The -1 implies negative influence to the prediction and is depicted as blue, while 1 implies positive influence for the prediction and is the depicted as red. The resulting image is rescaled after adding the image gradient. Classifiers may use erroneous background cues for prediction and the technique gradCAM can be used. Further description of these failure modes is in Section 3.
  12. 7 Some data augmentation methods can slightly improve performance.
  13. 8 Increasing model size and other architecture changes can greatly improve performance. Note Res2Net and ResNet+SE have a ResNet backbone. Normal model sizes are ResNet-50 and ResNeXt-50 ($32\times 4$d), Large model size are ResNet-101 and ResNeXt-101 ($32\times 4$d), and XLarge Model sizes are ResNet-152 and ($32\times 8$d).
  14. 1 We scale up out-of-distribution detection to large-scale multi-class datasets with hundreds of classes, multi-label datasets with complex scenes, and anomaly segmentation in driving environments. In all three settings, we find that an OOD detector based on the maximum logit outperforms previous methods, establishing a strong and versatile baseline for future work on large-scale OOD detection.
  15. 2 Small-scale datasets such as CIFAR-10 have relatively disjoint classes, but larger-scale datasets including ImageNet have several classes with high visual similarity to other classes. The implication is that large-scale classifiers disperse probability mass among several classes. If the prediction confidence is used for out-of-distribution detection, then images which have similarities to other classes will often wrongly be deemed out-of-distribution due to dispersed confidence. The dog is lower resolution for the CIFAR-10 classifier.
  16. 3 A sample of anomalous scenes, model predictions, and anomaly scores. The anomaly scores are thresholded to the top 10% of values for visualization. GT is ground truth, the autoencoder model is based on the spatial autoencoder used in [Baur_2019], MSP is the maximum softmax probability baseline [hendrycks2017baseline], and MaxLogit is the method we propose as a new baseline for large-scale settings.
  17. 1  Images from our three new datasets ImageNet-Renditions (ImageNet-R), DeepFashion Remixed (DFR), and StreetView StoreFronts (SVSF). The SVSF images are recreated from the public Google StreetView, copyright Google 2020. Our datasets test robustness to various naturally occurring distribution shifts including rendition style, camera viewpoint, and geography.
  18. 2  ImageNet-Renditions (ImageNet-R) contains 30,000 images of ImageNet objects with different textures and styles. This figure shows only a portion of ImageNet-R’s numerous rendition styles. The rendition styles (e.g., “Toy”) are for clarity and are _not_ ImageNet-R’s classes; ImageNet-R’s classes are a subset of 200 ImageNet classes. ImageNet-R emphasizes shape over texture.
  19. 3  DeepAugment examples preserve semantics, are data-dependent, and are far more visually diverse than augmentations such as rotations.
  20. 4 Accuracy as a function of corruption severity. Severity “0” denotes clean data. Data augmentation methods such as DeepAugment with AugMix shift the entire Pareto frontier outward.
  21. 5 ImageNet accuracy and ImageNet-C accuracy. Previous architectural advances slowly translate to ImageNet-C performance improvements, but DeepAugment+AugMix on a ResNet-50 yields a $\approx 19\%$ accuracy increase.
  22. 1 Set membership model overview. We concatenate the shot images with the single query images to feed into the network. We use different architectures as our embedding network to learn whether the query image belongs in the set of images. This is an example of a 5 shot (k=5) set membership task in which we ask if the hot dog belongs to the set of images of animals.
  23. 2 Vision Transformer model. The model takes as input non-overlapping patches of the original image whereby each are linearly embedded and position embeddings are added onto each. Finally the embeddings are fed into the Transformer along with a “classification token” to use for classification. The above reflects the original ViT. We modified the architecture from instead of accepting only 1 image it accepts a batch of images as a single query. The change corresponds to roughly increasing the model size by 6 to account for the extra 4 input images and the 1 query image. The MLP head changes to output a single number instead of n class probabilities. Image source: [2021visiontransformer].
  24. 1 A demonstration of color sensitivity. While the leftmost image is classified as “banana” with high confidence, the images with modified color are correctly classified. Not only would we like models to be more accurate, we would like them to be calibrated if they wrong.
  25. 2 The Response Rate Accuracy curve for a ResNeXt-101 (32$\times$4d) with and without Squeeze-and-Excitation (SE). The Response Rate is the percent classified. The accuracy at a $n$% response rate is the accuracy on the $n$% of examples where the classifier is most confident.
  26. 3 Self-attention’s influence on ImageNet-A $\ell_{2}$ calibration and error detection.
  27. 4 Model size’s influence on ImageNet-A $\ell_{2}$ calibration and error detection.
  28. 5  Examples of real-world blurry images from our collected dataset.
  29. (a) Larger models improve robustness on ImageNet-R. The baseline models are ResNet-50, DPN-68, and ResNeXt-50 ($32\times 4$d). The larger models are ResNet-152, DPN-98, and ResNeXt-101 ($32\times 8$d). The baseline ResNeXt has a 7.1% ImageNet error rate, while the large has a 6.2% error rate.
  30. (b)  Accuracy as a function of corruption severity. Severity “0” denotes clean data. DeepAugment with AugMix shifts the entire Pareto frontier outward.abcddas dfsdafsdfasads fsdfasdfadfsafd adsfdfssdfdfs

###### List of Tables

  1. 1 Quantitative comparison of the CAOS benchmark with related datasets. The BDD-Anomaly dataset treats three categories as anomalous and has many unique object instances within those categories. By contrast, Lost and Found has the same objects in multiple images and has only nine unseen objects at test time. StreetHazards leverages a simulated environment to naturally insert hundreds of varied anomalies.
  2. 2 Multi-class out-of-distribution detection results using the maximum softmax probability, maximum logit basline and KL Divergence between predicted and posterior. Results are on ImageNet and Places365. Values are rounded so that $99.995\%$ rounds to $100\%$. Full results on individual $\mathcal{D}_{\text{out}}$ datasets and additional baselines are in the supplementary material.
  3. 3 Results on the Combined Anomalous Object Segmentation benchmark. AUPR is low across the board due to the large class imbalance, but all methods perform substantially better than chance. MaxLogit obtains the best performance. All results are percentages.
  4. 1 ImageNet-200 and ImageNet-R top-1 error rates. ImageNet-200 uses the same 200 classes as ImageNet-R. DeepAugment+AugMix improves over the baseline by over 10 percentage points. ImageNet-21K Pretraining tests _Pretraining_ and CBAM tests _Self-Attention_. Style Transfer, AugMix, and DeepAugment test _Diverse Data Augmentation_ in contrast to simpler noise augmentations such as $\ell_{\infty}$ Adversarial Noise and Speckle Noise. While there remains much room for improvement, results indicate that progress on ImageNet-R is tractable.
  5. 2 SVSF classification error rates. Networks are robust to some natural distribution shifts but are substantially more sensitive the geographic shift. Here _Diverse Data Augmentation_ hardly helps.
  6. 3 A highly simplified account of each hypothesis when tested against different datasets. Evidence for is denoted "$+$", and "$-$" denotes an absence of evidence or evidence against.
  7. 1 Multi-label out-of-distribution detection comparison of the maximum logit, typicality matrix, logit average, Local Outlier Factor, and Isolation Forest anomaly detectors on PASCAL VOC and MS-COCO. The same network architecture is used for all three detectors. All results shown are percentages.
  8. 2 DeepFashion Remixed results. Unlike the previous tables, higher is better since all values are mAP scores for this multi-label classification benchmark. The “OOD” column is the average of the row’s rightmost eight OOD values. All techniques do little to close the IID/OOD generalization gap.
  9. 3 For these results we took a different partitioning of the DeepFashion Remixed dataset namely using all of the training data (excluding the same combinations we test against), and then only splitting up the validation data. All values are mAP scores. ResNet-152 tests the _Larger Models_ hypothesis, ImageNet-21K Pretraining tests _Pretraining_ , CBAM tests _Self-Attention_ , and the other techniques test _Diverse Data Augmentation_. All techniques have limited effects.
  10. 1 Accuracy of different architectures on miniImageNet [Vinyals2016miniimagenet]. These results highlight that no architecture can achieve results than a random baseline.
  11. 2 Accuracy of different architectures on miniImageNet [Vinyals2016miniimagenet]. These results that pretraining has some small effects on performance.
  12. 1 Expanded ImageNet-A and ImageNet-O architecture results.
  13. 2 ImageNet-C vs Real Blurry Images. All values are error rates and percentages. The rank orderings of the models on Real Blurry Images are similar to the rank orderings for “ImageNet-C Blur Mean,” so ImageNet-C’s simulated blurs track real-world blur performance. Hence synthetic image corruptions and real-world image corruptions are not loose and separate.
  14. 3 A highly simplified account of each hypothesis when tested against different datasets. This table includes ImageNet-A results.
  15. 4 ImageNet-200 and ImageNet-Renditions error rates. ImageNet-21K and WSL Pretraining test the _Pretraining_ hypothesis, and here pretraining gives mixed benefits. CBAM and SE test the _Self-Attention_ hypothesis, and these _hurt_ robustness. ResNet-152 and ResNeXt-101 32$\times$8d test the _Larger Models_ hypothesis, and these help. Other methods augment data, and Style Transfer, AugMix, and DeepAugment provide support for the _Diverse Data Augmentation_ hypothesis.
  16. 5 Clean Error, Corruption Error (CE), and mean CE (mCE) values for various models and training methods on ImageNet-C. The mCE value is computed by averaging across all 15 CE values. A CE value greater than 100 (e.g. adversarial training on contrast) denotes worse performance than AlexNet. DeepAugment+AugMix improves robustness by over 23 mCE.
  17. 6 ImageNet-A top-1 accuracy.
  18. 7 Various distribution shifts represented in our three new benchmarks. ImageNet-Renditions is a new test set for ImageNet trained models measuring robustness to various object renditions. DeepFashion Remixed and StreetView StoreFronts each contain a training set and multiple test sets capturing a variety of distribution shifts.
  19. 8 Number of images in each training and test set. ImageNet-R training set refers to the ILSVRC 2012 training set [imagenet]. DeepFashion Remixed test sets are: in-distribution, occlusion - none/slight, occlusion - heavy, size - small, size - large, viewpoint - frontal, viewpoint - not-worn, zoom-in - medium, zoom-in - large. StreetView StoreFronts test sets are: in-distribution, capture year - 2018, capture year - 2017, camera system - new, country - France.

###### Acknowledgements.

I want to thank my parents who have always supported me. I owe this day to
them. I love you mom and pop. I want to thank all of my professors, especially
the professors who went above and beyond as mentors. Thank you Greg for
helping me beyond researching ideas to also thinking about life, and what I
will do beyond schooling. Thank all of my friends and colleagues, Dan,
Mohammadreza, Mantas, Gustav, and Payman all of whom in one way or another
helped me achieve this goal. Also would like to thank my friends Matt and
Kevin. I would also like to thank Keziah for helping me edit the thesis. I
would like to thank Victoria for also helping with edits, and being by my
side. Finally I would like to thank everyone who have helped to make this day
possible.

##  Chapter 0 Introduction

Machine Learning (ML) models are becoming more widespread in their use and
adoption. As their use becomes more prevalent in safety-critical applications
or trust necessary situations, the models must exhibit reliability to ensure
their continued adoption. Some of the current domains that exhibit these
properties include self-driving vehicles and in medical diagnoses; where in
self-driving, both trust and safety are paramount. In the second domain of
medical diagnoses trust is the primary factor such that a model will need to
be able to provide explanations or a confidence such that other professionals
can best decide how to proceed with the outputted information.

Robustness is of especial importance in machine learning where many of the
commonly used models are uncalibrated out of the box. All of the works thus
far that aimed at addressing this short-coming also come at the cost of model
accuracy. While calibration is of importance for building confidence in model
predictions, it only represents one facet of robustness. In this thesis we
will cover several aspects of robustness and begin to show that robustness
covers several related areas. While we explore the related areas, we also aim
to highlight the delineations between them. These delineations help
researchers better focus on specific problem domains, which then allows the
community to make improvements. Beyond covering the delineations, we also
demonstrate that there are still unknown areas of robustness that might not
have proper categories or delineations at the moment.

We define robustness as the preservation of functionality under changing
conditions. Within the ML community robustness has grown to encompass several
different concepts which include the following domains: generalization,
sensitivity, distribution shift, adversarial examples, out-of-distribution
(OOD) detection, calibration, and even interpretability. We shall attempt to
make those distinctions clear. We give a detailed explanations of the domains
in Section 1 and a brief description of the OOD detection task in Section 4.
This task fits into robustness by trying to preserve confidence on in-
distribution (ID) examples in the presence of OOD examples. Therefore, the OOD
detection task is useful for building safe ML systems. For example in robotics
systems it can serve as an indicator for the robot to hand off to an operator
and in medicine, it can signal that the input is malformed and should be
redone or handled by a doctor.

We organize the thesis loosely based around machine learning tasks covering
multi-class classification first, then segmentation, then multi-label, and
finally meta-learning. Segmentation can be viewed as a multi-class
classification problem applied to every individual pixel. Multi-label
classification can be thought of as an extension of multi-class classification
so we cover multi-label after multi-class and segmentation. This extension is
not perfect and we show some limitations that comes from trying to generalize
successes from multi-class to multi-label. We then discuss meta-learning with
respect to robustness which is distinct enough from multi-class and multi-
label to be covered last.

###  1 Contributions

Our contributions to robustness research are as follows:

  * •

Datasets: In this thesis we contribute three distinct datasets to help push
the research area of robustness further. The first dataset explores the
current errors that models make and are termed hard-negative examples or
natural adversarial examples in Chapter 2. The second dataset involves the
creation of a controllable anomalous segmentation dataset created by modifying
the CARLA simulator in Chapter 3. Finally, the third dataset aims to better
explore distribution shift from ImageNet-1K to other representations such as
cartoons, sculptures, or tattoos among others in Chapter 4.

  * •

New Robustness Techniques: We utilize the Kullback–Leibler divergence (KL-
Divergence) to improve OOD detection in the multi-class setting. We further
show that using the maximum logit serves as a better indicator of OOD
detection than the previous baseline. Both of those results are presented in
Chapter 3. We also develop a novel non-linear augmentation technique in
Chapter 4. The novel technique is able to greatly improve robustness in the
multi-class setting and highlights a new unexplored area that can also help
generalization.

  * •

Novel Field Criticisms: By examining different variations of robustness, we
are able to observe where current techniques are still lacking in Chapters 4
and 5. Most notably is the recognition that almost all of the previous
robustness techniques failed to generalize to the multi-label setting.

  * •

Few shot robustness: We explore the relationship of robustness and few shot
learning in Chapter 6. By converting the metric learning task into a set-
membership task we find that while the models are able to learn set-membership
for in-distribution classes, the models are unable to generalize to novel
unseen classes.

##  Chapter 1 Background

###  1 Robustness

We shall cover a few seemingly disparate robustness topics and show how they
relate together. In this thesis, we focus on natural out-of-distribution (OOD)
detection. Since ML systems are deployed in natural settings, natural OOD
detection is of utmost importance and should be solved immediately. In
addition to natural OOD detection, in Chapter 3, we explore anomaly
segmentation in simulated and adversarial examples. We explore these areas to
study whether any of the techniques in simulations or improvements in
adversarial robustness generalize to natural settings. In Section 2 we cover
adversarial examples in greater detail.

(a) Points are drawn from a 2D Gaussian, and the isolation forest model is
labeling the point X as an outlier.

(b) Given differently sampled points leads to different lines which causes
some number of points to be labeled as outliers. After sampling sufficient
number of points then the best line is determined. Image credit:
[Watanabe2013AFR].

Figure 1: Examples of two novelty detection methods.

OOD detection has its roots in outlier and novelty detection. The main
distinction between the more recent OOD detection and outlier detection are
the assumptions on what is considered an outlier. In OOD detection, the task
is set up similarly to a classification problem by having an in-distribution
set, and an unknown and much larger out-of-distribution set. Whereas in
outlier detection the assumption would be that there exists a small subset of
data which are considered outliers, and hence the data provided is considered
“poisoned.” The commonly used approach in statistics is to first define the
model and identify the outliers as points with low probability mass. More
advanced methods exist such as using local outlier factor [Breunig2000LOFID]
or using RANSAC [Fischler1981Ransac] to determine the inliers versus outliers.
See figure 1 for a visual demonstration of the techniques. The former method
works by comparing the distances of points to their neighbors and finding a
threshold for the computed distance. However many of these methods still rely
on determining the model first to fit the data.

In a setting closely related to outlier detection, there is the problem of
training with known label corruptions. Both domains deal with the case of data
poisoning, while this sub-problem exclusively concerns itself with poison
occurring in the labels. Furthermore the problem of label corruption can be
approached from the assumption whether there exists a subset of trusted
labels. [Charikar2017SteinhardtsemiVerified] analyze both conditions to give
theoretical guarantees and an algorithm to use under each setting.
[Patrini2017LabelNoise] address the problem in multi-class classification with
no trusted data where they first estimate the level of corruption, then given
the confusion matrix apply it after the network’s outputs to correct for the
uncertainty. [Ren2018ReweightExamplesLabelCorruption] also approach the
problem under the same assumptions, but they use meta-learning (see Section 3)
to assign and reweight the examples used for training. Follow up work has
shown that MixUp is also an effective way to increase the training given label
corruption [arazo2019unsupervised]. While the problem remains unsolved, some
works highlight that for some models, namely neural networks, label noise
might not be as much of a problem compared to other ML methods as the models
tend to be robust to out of the box [Rolnick2017MassiveLabelNoise].

On the other hand, novelty detection, unlike in outlier detection, operates
under the assumption that the dataset is not polluted with outliers, and
instead tries to detect if a new example is an outlier. Some classical
techniques in novelty detection are that of isolation forest
[Liu2008IsolationF] and one-class SVM [Schlkopf1999OCSVM]. These methods
employ a training and testing phase. Some of the methods can not give
estimations of outliers for the training set by construction. This set-up more
closely reflects the problems in OOD detection. However, many of the novelty
detection methods such as those listed exhibit poor performance
[emmott2013ocsvmrobustnessreview].

In contrast to outlier and novelty detection, OOD detection frames the problem
of detecting anomalies as a learning problem. Novelty detection can be
considered to be within unsupervised classification, while OOD detection
expands on novelty detection by considering supervised and semi-supervised
along with unsupervised classification. [hendrycks2017baseline] uses a trained
neural network’s output as a confidence score for detecting anomalies in the
setting of multi-class classification. As a followup work, ODIN
[Liang2018ODIN] and Mahalanobis detector [Lee2018Mahalanobis] apply small
gradient perturbations derived from label corruptions and then run the
corrupted image through the network again to get the final output score. The
work by [Lee2018Mahalanobis] modifies this approach by training a per-class
dependent model on the perturbations. The main issue with the previous two
approaches in particular is that they fine-tune their corruptions on the
different perturbation types which leads to a form of training on the test
set. Other methods such as the confidence estimator trains a separate
confidence branch to directly predict both the class label and the confidence
of the prediction [devries]. In Chapter 4 we further explore the relationship
between data augmentation and robustness by demonstrating how random
corruptions by a neural network improve OOD detection.

###  2 Adversarial Examples

Adversarial examples are modifications to inputs, such as images, that causes
the label of a function (typically a deep neural network) to change, while a
human would still classify the input as belonging to the same class. It was
first discovered that neural networks are susceptible to these small
perturbations in [Szegedy2014IntriguingPO]. Figure 2 shows how small
adversarial noise can affect the outputs of a deep neural network (DNN). This
work spawned several works in interpretability such as [olah2017DeepDream] and
[Selvaraju_2019GradCam] whereby the gradient perturbations were used to
visualize what the network was attending to. Later on after
[Carlini2017Wagner]’s research findings, the research community began to focus
on the problem treating it as an actual security threat threat instead of as
an anomaly.

Figure 2: An adversarial example showing how a small perturbation can change
the label of an image. The adversarial noise added to image is magnified by
1000x to be visible to a human. Image credit: OpenAI

Let us formally define adversarial examples. Note for the purposes of these
definitions we will restrict the domain of examples or inputs to be that of
images but it can extended to other domains such as speech or language. An
input image is $\mathbf{x}$ and a trained model is $f$ (typically a neural
network). $f(\cdot)$ is the output probability distribution for a set of
classes the model was trained on. We will perturb $\mathbf{x}$ by a small
amount $\epsilon$ which will result in our adversarial example
$\mathbf{x}^{\prime}:=\mathbf{x}+\epsilon$. We say $\mathbf{x}^{\prime}$ is an
adversarial example if $\underset{\mathbf{x}}{\arg\max}\
f(\mathbf{x})\neq\underset{\mathbf{x}^{\prime}}{\arg\max}\
f(\mathbf{x}^{\prime})$, it is also described as a successful attack.
$\epsilon$ is of the same dimensionality as $\mathbf{x}$ but is bounded by
some distance metric. We use several $L_{p}$ distance measures to bound
$\epsilon$ as a proxy for the measure we actually care for, which is human
perceptual distance. More recently, there is work trying to move beyond
$\ell_{p}$ distances such as [Bhattad2020UnrestrictedAEcolorizationAttack]
because bounding $\|\varepsilon\|_{\infty}\leq\delta$ distance is a somewhat
arbitrary limitation on the attacker.

It is common practice in the security research community to detail what the
assumptions are concerning the threat model, which includes what the attacker
has access to, capabilities, and limitations. Most of what has so far been
discussed relates to the scenario where the attacker has full access to the
machine and the model such that they can inject signal noise $\epsilon$ into
the system to be able to change the resulting classification output of $f$.
While this setting has received much attention, it is largely unrealistic as
an actual security threat. More recently, there have been efforts to change
the threat model to consider more real world attacks such as
[Eykholt2018PhysicalAE, Goodfellow2015ExplainingAH,
Yakura_2019AudioAlexAttack] where the attacker has a more limited access to
the model via its external sensors.

As an implicit goal in desiring to make the models more similar to humans, we
treat adversarial examples as inherent flaws in the model. There are some
discussions concerning whether or not adversarial examples are unavoidable
[engstrom2019DiscussionofAdversarialBugs], although this view is currently
held by a minority in the research community. We shall detail a few of the
current hypotheses being investigated for why adversarial examples might
arise. One of the hypotheses by [Goodfellow2015ExplainingAH] purports that the
brittleness is a result of excessive linearity. The high-dimensionality of the
inputs allows an adversary easy access to cross a decision boundary of a
class. The simple example is for a two class problem given a powerful enough
adversary, the inputs can be moved along any direction and with high
probability half of the directions will cross the decision boundary. This
hypothesis has been indirectly challenged by subsequent work
[wong2018adversarialpolytope]. Another hypothesis is that of robust and non-
robust features [Ilyas2019AdversarialAreBugs]. The argument is that the
networks are picking up on highly predictive features that do well on the
training set, but are too brittle and incomprehensible for humans which then
allows for adversarial examples. Some qualitative examples supporting this
claim are that adversarially trained models have features that are more
recognizable to humans. The features themselves are trained on limited
datasets which creates an artificial selection bias and lacks more theoretical
underpinnings. There is no definitive consensus for explanations or solutions
to the problem of adversarial examples at the current time.

While there does not exist a solution to the problem, there are several
defenses that can mitigate the power of adversarial attacks. When a defense is
“broken” it refers to the attacker reducing the accuracy of the model to zero
(or nearly zero). The most prominent defenses for adversarial examples are
some form of training with adversarial examples. The original formulation from
[Madry2018AdversarialTraining] adds an adversarial example during training as
extra training data for the model to learn from. Followup work from
[Zhang2019TRADES] explicitly considers the tradeoff between natural and
adversarial accuracy by incorporating an additional loss term for the
adversarial examples used during training. There are other works that provide
certifications for limited threat models such as $L_{2}$ distance
[Cohen2019SmoothingAdversarialCertification] and other forms of certifications
are being explored. Methods which provide some amount of robustness include
the following: Compression of either inputs, models, or both
[Liu2019FeatureDD, Dziedzic2019BandlimitedTA]; injecting noise into either the
input examples, models, or both [You2019AdversarialNL,
Cohen2019SmoothingAdversarialCertification]; data augmentation, or pre-
training [Hendrycks2019Pretraining]; and ensembling which use multiple models
to give one output using a voting, or some other scheme [Tramr2018EnsembleAT].

Part of the difficulty with comparing defenses is the lack of standardization
to compare the different threat models and environments. Recently, projected
gradient descent attacks with $\ell_{\infty}$ perturbation of size 8/255 has
become the standard because greater perturbations allowed for changing the
label to a human annotator
[Tramr2020FundamentalMnistSensitivityChangingLabel]. Given this limited
adversary/scenario allows for direct comparison between methods. The other
issue is that the threat model or environment is being an unfair or being an
unreasonable limitation to the adversary. Due to this limitation researchers
use adative attacks [Tramr2020OnAdaptiveAttacks] which allow the attacker to
know the defense and modify the attack given this knowledge. Given the
relative infancy of the joint domains of ML and computer security, the field
will eventually settle on realistic threat models and the issue of defense
comparison will become moot.

Although the focus is mostly on neural networks, other classical machine
learning (ML) models are also susceptible to adversarial noise such as k
Nearest Neighbors (k-NN), SVMs, and linear regression. These other machine
learning models have different failure modes for adversarial examples but have
all been susceptible to the attacks. [gilmer19advconsequnce_of_noise_iclr]
suggests that the previous methods fail due to the high dimensionality of the
inputs. Currently neural networks provide the best defenses against
adversarial examples [Goodfellow2016comparemodels]. However, the other ML
models have been less explored both to attack and defend against this threat
model.

It is worth noting the difficulty of transitioning to real-world attacks via
adversarial examples. [Bhattad2020UnrestrictedAEcolorizationAttack]
demonstrates that artificial constraints had to be placed on the model for the
adversary to succeed. This is not to suggest that the signal based attacks are
unimportant, but it should be studied to determine how well adversarial
attacks can survive the transition to complex environments, to 3D, and against
multiple modalities. [carla] have shown how difficult it is to construct
adversarial examples in 3D environments.

###  3 Meta-Learning

Meta-learning is most commonly understood as the task of “learning to learn”,
which refers to the process of improving a learning algorithm over
distributions. This contrasts with the conventional machine learning, which is
the process of learning a model for a single distribution or over many data
instances. Meta-learning is comprised of a two stage process. The first stage
involves a base learner and the second stage (also confusingly called meta-
learning) involves an outer algorithm that optimizes an outer objective. An
example of meta-learning the inner (or lower, base) learning algorithm solves
a task such as image classification [franceschi2018bilevel], defined by a
dataset and objective. Then during meta-learning, an outer (or upper, meta)
algorithm updates the inner one, so that the learned model can perform well in
few-shot learning [hospedales2020metalearning].

Recent work has tried to minimize the distinctions between meta-learning and
supervised classification [chao2020revisiting]. [maurer2005algorithmic]
extended the generalization error bounds from supervised learning to meta-
learning. Formulating the problem of meta-learning more closely with
supervised learning better fits the theme of this thesis trying to improve OOD
detection.

We most prominently explore two techniques in meta-learning. The first
technique is from [finn2017MAML] which uses the optimization algorithm to
perform the meta-learning. The inner learner does standard supervised learning
for the task, e.g. image classification, and the outer optimization algorithm
does an averaged gradient step for several tasks (different image
classification tasks). The second technique imbues the space of learned
features with a metric to be later used for a nearest neighbor classification.
We ignore the other type of meta-learning which uses memory augmented networks
to perform the meta-learning. Those models are currently more akin to a
novelty as opposed to offering a more utilized method such as the others
described before it.

We explore meta-learning in Chapter 6 to explore how the techniques from meta-
learning namely the scenario of learning in the few shot setting can better
assist in generalizing to unseen OOD examples. In both scenarios the amount of
OOD data far exceeds the amount of training data that the model is expected to
generalize from.

###  4 Tasks

In this section, we will cover three computer vision tasks that will be
addressed in this paper namely multi-class classification, multi-label
classification, and segmentation,.

####  1 Multi-class Classification

Multi-class classification is a rich sub-field in machine learning with its
origins in binary classification [Cox1958logisticregression]. From binary
classification there are three approaches that have been taken to handle the
multi-class setting. The first approach is by reduction of the multi-class
problem to binary classification, typically One-vs-Rest where one would use a
classifier to distinguish one class from all of the others. The size of these
approaches scale linearly with the number of classes due to requiring a
different model per class. The second approach handles the problem by
extending the binary classification to multi-class classification directly.
Finally, the last approach handles the problems by creating hierarchies for
classification [Silla2010ASO].

We will be covering the second approach for multi-class classification, the
extension from binary, which include methods such as k-nearest-neighbors
[Altman1992KNN], logistic regression [menard2002appliedLR], neural networks
[Hopfield1988NeuralNA], and random forests [Breiman2004RandomForests]. We will
briefly cover neural networks as they are utilized throughout the paper and
are used in the other tasks as well. For neural networks we mostly utilize
convolutional neural networks (CNN). CNNs is composed of the following
operations: learned filters which perform cross-correlation, convolution, non-
linearity, and pooling which reduces the dimensionality of the inputs and
features. These operations are repeated in succession to form what are
considered layers before finally applying a differentiable loss function so
that the network can learn via stochastic gradient descent. The specific
implementation details of each neural network architecture is left to the
respective chapter.

The problem in multi-class classification is to associate each instance with a
unique label from a set of labels. More formally given a set of training data
$D=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\},\forall i\in\\{1,...,N\\}$ where
$\mathbf{x}\in\mathbb{R}^{d}$, the goal is to learn (or induce) a model
$f(\mathbf{x})=\mathbf{\hat{y}}$ such that $\mathbf{\hat{y}}$ minimizes a loss
function $L:\mathcal{R}\times\mathcal{R}\to\mathcal{R}$. During
classification, the assumption is training and test data points are drawn
i.i.d. (independent and identically distributed) from a full joint
distribution. Although in most cases, the full joint distribution is unknown,
a finite number of examples can be used to measure generalization by splitting
the data into training and test sets. Generative and discriminative techniques
are useful for learning model $f$. Generative techniques learn the full joint
probability distribution while the latter techniques model the class or
decision boundaries. In this thesis, we focus on discriminative methods for
classification.

Some influential datasets in the area of multi-class classification have been
MNIST [mnist], SVHN [svhn], CIFAR [cifar], and ImageNet [imagenet]. MNIST and
SVHN are both datasets consisting of the digits 0-9. MNIST are black and white
digits, and SVHN are numbers occurring on houses which are colored images and
have a variety of backgrounds. On the other hand, CIFAR and ImageNet are
natural images consisting of animals, automotive vehicles, and many other
categories. The former are small images with ten categories and the latter are
large images with one thousand categories. These datasets, among others, have
allowed for great advances in the task of multi-class classification
[Ranzato2006LecunSparseEnergyModel, Beygelzimer2006CoverTreesNN,
Cho2009KernelMethods, AlexNet, Carlini2017Wagner].

####  2 Multi-label Classification

Figure 3: An example of an image with both a cat and a bird which shows
multiple labels occurring together.

Building from multi-class classification, there is multi-label classification
whereby instead of the traditional setting where an instance may be of only
one label, in this setting each instance is associated with a set of labels.
As an example consider the figure 3 where it can be classified as a bird, bald
eagle, cat, porch and forest. This task being a natural extension of multi-
class is therefore a harder task because of determining the presence or
absence of for all classes as opposed to the problem of selecting the most
likely candidate. Multi-label classification technique is useful for (1) text
classification involving lots of documents about several topics, (2) audio
classification, where some songs are a mixture of styles or genres and (3)
image search where different images or videos can capture multiple genres or
styles at the same time [Brhanie2016MultiLabelCM].

We will similarly define the task of multi-label classification. The problem
in multi-label (or sometimes referred to as multi-category or multi-topic)
classification is to associate each instance with a set of labels. This again
follows the same conventions and notations as multi-class classification
whereby training data $D=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\},\forall
i\in\\{1,...,N\\}$ contain examples x that are a $d$-dimensional vector.
However, unlike in multi-class setting y is no longer a one-hot vector and are
instead binary strings without the limitation of only one “$1$” in the string.

Multi-label classification is arguably better for studying robustness. The
main reasons for this is because in multi-class classification the models are
either forced to pick one out of the set of labels or add in a catch-all
“other” class. In the first scenario the model “should” output a uniform
probability distribution over all classes, however, this also seems
unrealistic as there will exist OOD examples which will appear to be similar
to in-distribution classes and so a uniform output appears to be an unfeasible
goal. The second option seems more promising but still has a similar issue.
Classifying images or parts of image as background will lead to the model
“correctly” classifying OOD examples as being background. This scenario again
presents issues that there is no longer any distinction between in and out-of-
distribution examples, which can be a safety hazard in some scenarios. Multi-
label classification does not have these two issues because having a uniform
output of no class is a feasible solution that is also realistic. The
secondary issue is still present unless modifications are made, that of being
able to discern between in- and out-of-distribution examples.

Unfortunately, multi-label classification has not received as much attention
within computer vision as that of multi-class classification. This can be
observed by the historical lag in multi-label datasets from Corel-5k
[corel5k], then PASCAL VOC [Everingham2009ThePV_pascal] and more recently we
have MSCOCO [coco] and Tencent ML-image [tencent-ml-images-2019]. Similarly
multi-label OOD is also a wholly underexplored area. We hope to change this by
releasing some baselines for this task in Chapter 5.

####  3 Segmentation

Figure 4: Highlighting the distinction between semantic segmentation on the
left and instance segmentation on the right. Image source:
[Arnab2018ConditionalRFNN]

Segmentation refers to the task of partitioning the pixels of an image into
regions or segments, where the pixels in each region share similar attributes.
The goal of segmentation is to simplify and/or change the representation of an
image into something meaningful and easier to analyze [shapiro_stockman_2001].
We categorize segmentation methods into two main categories that of semantic
and class agnostic segmentation. Class agnostic segmentation is the task of
segmenting an image into the region boundaries such as foreground and
background. Semantic segmentation on the other hand, deals with the task of
assigning each pixel to an element of a set of classes. As a related task to
semantic segmentation there is instance segmentation where the goal is to
identify all distinct occurrences of the segmented objects in an image.

Class agonostic segmentation involves partitioning an image into coherent
regions. [BSDSmartin2001] show in Berkeley Segmentation Dataset (BSDS) that
human annotations of edges and boundaries are nonrandom and useful for
supervised class agnostic segmentation. However, the BSDS dataset has been
most useful in unsupervised segmentation [arbelaez2010contour,
martin2004learning, achanta2010slic, carreira2010constrained], in such tasks
as foreground/background segmentation [wu2008interactive]. Unsupervised
segmentation typically involve clustering [martin2004learning] and are trained
by learning a distance between pixels within segments. Their outputs have been
used in other downstream tasks due to the reduction in the input space
[mostajabi2015zoomout]. Due to the hardware and algorithmic improvements,
reliance on “super pixels” [achanta2010slic] has waned and the current
approaches label each pixel directly in a given image.

Unlike class agnostic segmentation, semantic segmentation is a supervised task
where the goal is to assign every pixel in an image a label from a given set
of classes. In semantic segmentation multiple instances of the same object
class are not differentiated. While in instance segmentation the goal is to
assign a unique label to every object instance in an image. A limitation of
instance segmentation is with dealing with uncountable classes such as pixels
belonging to the sky or sand. The union of both semantic and instance
segmentation is known as panoptic segmentation [Kirillov2019PanopticS]. The
goal of panoptic segmentation is to assign both a category label to every
pixel and if the category is an instance based category, such as people, then
also assign it a unique id.

Prominent segmentation datasets include BSDS [BSDSmartin2001], PASCAL-VOC
[Everingham2009ThePV_pascal] and MS COCO [coco]. These datasets have allowed
for significant progress in unsupervised and supervised segmentation
[arbelaez2010contour, objectdetectionanalysis, wu2008interactive,
Arnab2018ConditionalRFNN, alex2019pointrend].

##  Chapter 2 Natural Adversarial Examples

###  1 Overview

Figure 1: Examples from ImageNet-A and ImageNet-O. The black text is the
actual class, and the red text is a ResNet-50 prediction and its confidence.
ImageNet-A contains images that classifiers should be able to classify.
ImageNet-O contains out-of-distribution anomalies of unforeseen classes which
should result in low-confidence predictions. ImageNet-1K models do not train
on examples from "Photosphere" nor "Verdigris" classes, so these images are
out-of-distribution.

Research on the ImageNet [imagenet] benchmark has led to numerous advances in
classification [Krizhevsky2017Alexnet], object detection [huang2017speed], and
segmentation [he2017mask]. ImageNet classification improvements are broadly
applicable and highly predictive of improvements on many tasks
[Kornblith2018DoBI]. Improvements on ImageNet classification have been so
great that some call ImageNet classifiers "superhuman" [He2015DelvingDI].
However, performance is decidedly subhuman when the test distribution does not
match the training distribution [hendrycks2019robustness]. The distribution
seen at test-time can include inclement weather conditions and obscured
objects, and it can also include objects that are anomalous.

[Recht2019DoIC] remind us that ImageNet test examples tend to be simple,
clear, close-up images, resulting in the current test set being perhaps too
easy and not representative of harder images encountered in the real world.
[Geirhos2020ShortcutLI] and [Arjovsky2019InvariantRM] argue that image
classification datasets contain “spurious cues” or “shortcuts.” For instance,
models may use an image’s background to predict the foreground object’s class;
a cow tends to co-occur with a green pasture, and even though the background
is inessential to the object’s identity, models may predict “cow” primarily
using the green pasture background cue. When datasets contain spurious cues,
they can lead to performance estimates that are optimistic.

To counteract this, we curate two hard ImageNet test sets of adversarially
filtered examples. By using adversarial filtration, we can test how well
models perform when simple-to-classify examples are removed, including
examples that are solved with simple spurious cues. Some adversarially
filtered examples are depicted in figure 1, which are simple for humans but
hard for models. Previously it has been shown that misclassified images can
transfer between models; however, these demonstrations relied on synthetic
distributions [geirhos, hendrycks2019robustness] and adversarial distortions
[szegedy2013intriguing]. Our examples demonstrate that is is possible to
reliably fool many models with clean natural images highlighting a practical
problem that needs to be addressed as opposed to a more theoretical or
abstract problem.

We demonstrate that clean examples that we collected can reliably degrade and
transfer to other unseen classifiers with our first dataset. Transfer in this
setting means that the average performance of an unseen model is comparable
(meaning not significantly better) to the model we used for filtering. This
phenomenon of transferring is hard to classify. We call this dataset
ImageNet-A, which contains images from a distribution unlike the ImageNet
training distribution. ImageNet-A examples belong to ImageNet classes, but the
examples are harder and transfer to other models. They cause consistent
classification mistakes due to scene complications encountered in the long
tail of scene configurations and by exploiting classifier blind spots (see
Section 3). Since examples transfer reliably, this dataset shows models have
previously unappreciated shared weaknesses.

The second dataset allows us to test model uncertainty estimates when semantic
factors of the data distribution shift. Our second dataset is ImageNet-O,
which contains image concepts from outside ImageNet-1K. These out-of-
distribution images reliably cause models to mistake the examples as high-
confidence in-distribution examples. To our knowledge this is the first
dataset of anomalies or out-of-distribution examples developed to test
ImageNet models. While ImageNet-A enables us to test image classification
performance when the _input data distribution shifts_ , ImageNet-O enables us
to test out-of-distribution detection performance when the _label distribution
shifts_.

Figure 2:  Various ImageNet classifiers of different architectures fail to
generalize well to ImageNet-A and ImageNet-O. Higher Accuracy and higher AUPR
is better. See 4 for a description of the AUPR out-of-distribution detection
measure. These specific model parameters are unseen during adversarial
filtration, so our adversarially filtered examples transfer across models.

We examine methods to improve performance on adversarially filtered examples.
However, this is difficult because figure 2 shows that examples successfully
transfer to unseen or black-box models. To improve robustness, numerous
techniques have been proposed. We find data augmentation techniques such as
adversarial training decrease performance, while others can help by a few
percent. We also find that a $10\times$ increase in training data corresponds
to a less than a 10% increase in accuracy. Finally, we show that improving
model architectures is a promising avenue toward increasing robustness. Even
so, current models have substantial room for improvement.

###  2 Related Work

Adversarial Examples. Real-world images may be chosen adversarially to cause
performance decline. [goodfellowblog] define adversarial examples
[Szegedy2014IntriguingPO] as "inputs to machine learning models that an
attacker has intentionally designed to cause the model to make a mistake".
Most adversarial examples research centers around artificial $\ell_{p}$
adversarial examples, which are examples perturbed by nearly worst-case
distortions that are small in an $\ell_{p}$ sense. Attackers can reliably and
easily create black-box attacks by exploiting these consistent _naturally
occurring_ model errors, and thus carefully applying gradient perturbations to
create an artificial attack is unnecessary. This less restricted threat model
has been discussed but not explored thoroughly before.

Several other forms of adversarial attacks have been considered in the
literature, including elastic deformations [Xiao2018SpatiallyTA], adversarial
coloring [Bhattad2019BigBI, Hosseini2018SemanticAE], synthesis via generative
models [Baluja2017AdversarialTN, Song2018ConstructingUA] and evolutionary
search [Nguyen2015DeepNN], among others. Other work has shown how to print 2D
[Kurakin2017AdversarialEI, brown2017adversarial] or 3D [sharif2016accessorize,
athalye2017synthesizing] objects that fool classifiers. These existing
adversarial attacks are all based on synthesized images or objects, and some
have questioned whether they provide a reliable window into real-world
robustness [Gilmer2018MotivatingTR]. Our examples are closer in spirit to the
hypothetical adversarial photographer discussed in [Brown2018UnrestrictedAE],
and by definition these adversarial photos occur in the real world.

Adversarial Examples. Some types of adversarial attacks eschew $\ell_{p}$ norm
ball constraints completely. For instance, [Baluja2017AdversarialTN]
synthesize adversarial examples with generative adversarial networks, and
[Song2018ConstructingUA] use variational autoencoders as well. Unfortunately,
these examples are often classifier-specific and do not transfer to new
models. Meanwhile, ImageNet-A adversarially filtered examples transfer and
successfully attack current architectures. These adversarially filtered
examples bear semblance to a theorized attack, the attack of the adversarial
photographer [Brown2018UnrestrictedAE]. This attacker is free to take a
photograph of an image with choice over the camera viewpoint, so that the
attack is free of the confines of the $\ell_{p}$ norm ball constraints. This
attack has not been thoroughly studied empirically, but these could be thought
of as a type of natural adversarial example. [Bhattad2019BigBI,
Hosseini2018SemanticAE] attempt to color examples adversarially, and
[Xiao2018SpatiallyTA] create an adversarial elastic deformation. In both cases
the modifications required to fool the network can become quite conspicuous
[kang] and artificial. [Kurakin2017AdversarialEI] show that $\ell_{p}$
adversarial examples can fool machine learning systems if they are carefully
printed and if the perturbation is conspicuous. Meanwhile, adversarially
filtered examples arise in nature and are not necessarily detectable by human
vision alone. [Nguyen2015DeepNN] also violate norm ball constraints and create
adversarial examples through evolutionary algorithms, but these adversarial
examples are far out-of-distribution and are not naturally manifested in the
real world.

Figure 3:  Previous work on out-of-distribution (OOD) detection uses synthetic
anomalies and anomalies from wholly different data generating processes. For
instance, previous work uses Bernoulli noise, blobs, the Describable Textures
Dataset [cimpoi14describing], and Places365 scenes [zhou2017places] to test
ImageNet out-of-distribution detectors. To our knowledge, we propose the first
dataset of out-of-distribution examples collected for ImageNet models.

Out-of-Distribution Detection. Generally, models learn a distribution, such as
the ImageNet-1K distribution, and are tasked with producing quality anomaly
scores that distinguish between usual test set examples and examples from
held-out anomalous distributions. For instance, [hendrycks2017baseline] treat
CIFAR-10 as the in-distribution and treat Gaussian noise and the SUN scene
dataset [Xiao2010SUNDL] as out-of-distribution data. That paper also shows
that the negative of the maximum softmax probability, or the the negative of
the classifier prediction probability, is a high-performing anomaly score that
can separate in- and out-of-distribution examples, so much so that it remains
competitive to this day. Since that time, other works on out-of-distribution
detection continue to use datasets from other research benchmarks as stand-ins
for out-of-distribution datasets. For example, some use the datasets shown in
figure 3 as out-of-distribution datasets [hendrycks2019oe]. However, many of
these anomaly sources are unnatural and deviate in numerous ways from the
distribution of usual examples [Ahmed2019DetectingSA]. In fact, some of the
distributions can be deemed anomalous from local image statistics alone.
[Meinke2019TowardsNN] propose studying adversarial out-of-distribution
detection by detecting adversarially optimized uniform noise. In contrast, we
propose a dataset for more realistic adversarial anomaly detection; our
dataset contains hard anomalies generated by shifting the distribution’s
labels and keeping non-semantic factors similar to the in-distribution.

Spurious Cues and Unintended Shortcuts. Models may learn spurious cues and
obtain high accuracy but for the wrong reasons [Lapuschkin2019UnmaskingCH].
Spurious cues are a studied problem in natural language processing
[Cai2017PayAT, Gururangan2018AnnotationAI]. Many recently introduced datasets
in NLP use adversarial filtration to create "adversarial datasets" by sieving
examples solved with simple spurious cues [Sakaguchi2019WINOGRANDEAA,
Bhagavatula2019AbductiveCR, Zellers2019HellaSwagCA, Dua2019DROPAR]. Like this
recent concurrent research, we also use adversarial filtration
[Sung1995LearningAE], but the technique of adversarial filtration has not been
applied to collecting image datasets before. Additionally, adversarial
filtration in NLP uses filtration to remove only the easiest examples, while
we use filtration to select only the hardest examples. Moreover, our examples
transfer to weaker models, while in NLP the most used adversarial filtration
technique AFLite [Sakaguchi2019WINOGRANDEAA] does not produce examples that
transfer to less performant models [Bisk2019PIQARA]. We show that adversarial
filtration algorithms can find examples that automatically and reliably
transfer to both simpler and stronger models. Since adversarial filtration can
remove examples that are solved by simple spurious cues, models must learn
more robust features for our datasets.

Robustness to Shifted Input Distributions. [Recht2019DoIC] create a new
ImageNet test set resembling the original test set as closely as possible.
They found evidence that matching the difficulty of the original test set
required selecting images deemed the easiest and most obvious by Mechanical
Turkers. ImageNet-A helps measure generalization to harder scenarios.
[Brendel2018ApproximatingCW] show that classifiers that do not know the
spatial ordering of image regions can be competitive on the ImageNet test set,
possibly due to the dataset’s lack of difficulty. Judging classifiers by their
performance on easier examples has potentially masked many of their
shortcomings. For example, [geirhos2019] artificially overwrite each ImageNet
image’s textures and conclude that classifiers learn to rely on textural cues
and under-utilize information about object shape. Recent work shows that
classifiers are highly susceptible to non-adversarial stochastic corruptions
[hendrycks2019robustness]. While they distort images with 75 different
algorithmically generated corruptions, our sources of distribution shift tend
to be more heterogeneous and varied, and our examples are naturally occurring.

Figure 4:  Additional adversarially filtered examples from the ImageNet-A
dataset. Examples are adversarially selected to cause classifier accuracy to
degrade. The black text is the actual class, and the red text is a ResNet-50
prediction.

Figure 5:  Additional adversarially filtered examples from the ImageNet-O
dataset. Examples are adversarially selected to cause out-of-distribution
detection performance to degrade. Examples do not belong to ImageNet classes,
and they are wrongly assigned highly confident predictions. The black text is
the actual class, and the red text is a ResNet-50 prediction and the
prediction confidence.

###  3 The Design and Construction of ImageNet-A and ImageNet-O

####  1 Design

ImageNet-A is a dataset of adversarially filtered examples for ImageNet
classifiers, or real-world examples that fool current classifiers. To find
adversarially filtered examples, we first download numerous images related to
an ImageNet class. Thereafter we delete the images that fixed ResNet-50
[resnet] classifiers correctly predict. We chose ResNet-50 due to its
widespread use. Later we show that examples which fool ResNet-50 transfer
reliably to other unseen models. With the remaining incorrectly classified
images, we manually select a subset of high-quality images.

Next, ImageNet-O is a dataset of adversarially filtered examples for ImageNet
out-of-distribution detectors. To create this dataset, we download
ImageNet-22K and delete examples from ImageNet-1K. With the remaining
ImageNet-22K examples that do not belong to ImageNet-1K classes, we keep
examples that are classified by a ResNet-50 as an ImageNet-1K class with high
confidence. Then we manually select a subset of high-quality images.

Both datasets were manually labelled by graduate students over several months.
This is because a large share of images in the ImageNet test set contain
multiple classes per image [Stock2018ConvNetsAI]. Therefore, producing a high-
quality dataset without multilabel images can be challenging with usual
annotation techniques. By high-quality we refer to both being a singleton and
recognizable instance of the target class. To ensure images do not fall into
more than one of the several hundred classes, we had graduate students
memorize the classes to build a high-quality test set.

ImageNet-A Class Restrictions. We select a 200-class subset of ImageNet-1K’s
1,000 classes so that errors among these 200 classes would be considered
egregious [imagenet]. For instance, wrongly classifying Norwich terriers as
Norfolk terriers does less to demonstrate faults in current classifiers than
mistaking a Persian cat for a candle. We additionally avoid rare classes such
as "snow leopard," classes that have changed much since 2012 such as "iPod,"
coarse classes such as "spiral," classes that are often image backdrops such
as "valley," and finally classes that tend to overlap such as "honeycomb,"
"bee,’" "bee house," and "bee eater"; "eraser," "pencil sharpener" and "pencil
case"; "sink," "medicine cabinet," "pill bottle" and "band-aid"; and so on.
The 200 ImageNet-A classes cover most broad categories spanned by ImageNet-1K;
see the Supplementary Materials 1 for the full class list.

ImageNet-O Class Restrictions. We again select a 200-class subset of
ImageNet-1K’s 1,000 classes. These 200 classes determine the in-distribution
or the distribution that is considered usual. The remaining 800 classes could
be used as data for Outlier Exposure [hendrycks2019oe]. As before, the 200
classes cover most broad categories spanned by ImageNet-1K; see the
Supplementary Materials 2 for the full class list.

####  2 Collection

ImageNet-A Data Aggregation. Curating a large set of adversarially filtered
examples requires combing through an even larger set of images. Fortunately,
the website iNaturalist has millions of user-labeled images of animals, and
Flickr has even more user-tagged images of objects. We download images related
to each of the 200 ImageNet classes by leveraging user-provided labels and
tags. After exporting or scraping data from sites including iNaturalist,
Flickr, and DuckDuckGo, we adversarially select images by removing examples
that fail to fool our ResNet-50 models. Of the remaining images, we select
low-confidence images and then ensure each image is valid through human
review. For this procedure to work, many images are necessary; if we only used
the original ImageNet test set as a source rather than iNaturalist, Flickr,
and DuckDuckGo, some classes would have zero images after the first round of
filtration.

For concreteness, we describe the selection process for the dragonfly class.
We download 81,413 dragonfly images from iNaturalist, and after performing a
basic filter with ResNet-50, we have 8,925 dragonfly images. In the
algorithmically suggested shortlist, 1,452 images remain. From this shortlist,
80 dragonfly images are manually selected, but hundreds more could be chosen.
Hence for just one class we may review over 1,000 images.

We now describe this process in more detail. We use a small ensemble of
ResNet-50s for filtering, one pre-trained on ImageNet-1K then fine-tuned on
the 200 class subset, and one pre-trained on ImageNet-1K where 200 of its
1,000 logits are used in classification. Both classifiers have similar
accuracy on the 200 clean test set classes from ImageNet-1K. The ResNet-50s
perform 10-crop classification of each image, and should any crop be
classified correctly by the ResNet-50s, the image is removed. If either
ResNet-50 assigns greater than 15% confidence to the correct class, the image
is also removed; this is done so that adversarially filtered examples yield
misclassifications with low confidence in the correct class, like in
untargeted adversarial attacks. Now, some classification confusions are
greatly over-represented, such as Persian cat and lynx. We would like
ImageNet-A to have great variability in its types of errors and cause
classifiers to have a dense confusion matrix. Consequently, we perform a
second round of filtering to create a shortlist where each confusion only
appears at most 15 times. Finally, we manually select images from this
shortlist in order to ensure ImageNet-A images are simultaneously valid,
single-class, and high-quality. In all, the ImageNet-A dataset has 7,500
adversarially filtered examples.

ImageNet-O Data Aggregation. Our dataset for adversarial out-of-distribution
detection is created by fooling ResNet-50 out-of-distribution detectors. The
negative of the prediction confidence of a ResNet-50 ImageNet classifier
serves as our anomaly score [hendrycks2017baseline]. Usually in-distribution
examples produce higher confidence predictions than OOD examples, but we
curate OOD examples that have high confidence predictions. To gather candidate
adversarially filtered examples, we use the ImageNet-22K dataset with
ImageNet-1K classes deleted. We choose the ImageNet-22K dataset since it was
collected in the same way as ImageNet-1K. ImageNet-22K allows us to have
coverage of numerous visual concepts and vary the distribution’s semantics
without unnatural or unwanted non-semantic data shift. After excluding
ImageNet-1K images, we process the remaining ImageNet-22K images and keep the
images which cause the ResNet-50 to have high confidence, or a low anomaly
score. We then manually select a high-quality subset of the remaining images
to create ImageNet-O. We suggest only training models with data from the 1,000
ImageNet-1K classes, since the dataset becomes trivial if models train on
ImageNet-22K. To our knowledge, this dataset is the first anomalous dataset
curated for ImageNet models and enables researchers to study adversarial out-
of-distribution detection. The ImageNet-O dataset has 2,000 adversarially
filtered examples since anomalies are rarer; this has the same number of
examples per class as ImageNetV2 [Recht2019DoIC]. Thus we use adversarial
filtration to select examples that are difficult for a fixed ResNet-50, and we
will show these examples straightforwardly transfer to unseen models.
Additional example ImageNet-O images are in 5.

For the collection of ImageNet-O we refrain from testing with classification
models with a background class. While there are many classification tasks
which incorporate a “background” class into the final predictions such as many
segmentation datasets [coco, Everingham2009ThePV_pascal], most of the common
multi-class classification datasets do not include such a category. Most
importantly the ImageNet dataset does not include a background class which is
what our dataset aims to test models from. There still remains the potential
issue that if a model had been trained with a background class then the images
from ImageNet-O would be classified as such. It has been shown that depending
on the score utilized such as from generative models out-of-distribution
examples can actually achieve greater in-distribution (or lower anomaly) score
in [hendrycks2017baseline]. We also show in a different chapter how poorly
background classes perform in distinguishing out from in-distribution see 3.

Figure 6:  Examples from ImageNet-A demonstrating classifier failure modes.
Adjacent to each natural image is its heatmap generated by “gradCAM”
[Selvaraju2019GradCAMVE]. The heatmap is generated by taking the
classification prediction and computing the gradients from the given output
prediction back to the image. The gradients are scaled to -1 to 1 range before
adding back to the original image. The -1 implies negative influence to the
prediction and is depicted as blue, while 1 implies positive influence for the
prediction and is the depicted as red. The resulting image is rescaled after
adding the image gradient. Classifiers may use erroneous background cues for
prediction and the technique gradCAM can be used. Further description of these
failure modes is in Section 3.

####  3 Illustrative Failure Modes

Examples in ImageNet-A uncover numerous failure modes of modern convolutional
neural networks. We describe our findings after having viewed tens of
thousands of candidate adversarially filtered examples. Some of these failure
modes may also explain poor ImageNet-O performance, but for simplicity we
describe our observations with ImageNet-A examples.

Observe figure 6. The first two images suggest models may overgeneralize
visual concepts. It may confuse metal with sundials, or thin radiating lines
with harvestman bugs. We also observed that networks overgeneralize tricycles
to bicycles and circles, digital clocks to keyboards and calculators, and
more. We also observe that models may rely too heavily on color and texture,
as shown with the dragonfly images. Since classifiers are taught to associate
entire images with an object class, frequently appearing background elements
may also become associated with a class, such as wood being associated with
nails. Other examples include classifiers heavily associating hummingbird
feeders with hummingbirds, leaf-covered tree branches being associated with
the white-headed capuchin monkey class, snow being associated with shovels,
and dumpsters with garbage trucks. Additionally figure 6 shows an American
alligator swimming. With different frames, the classifier prediction varies
erratically between classes that are semantically loose and separate. For
other images of the swimming alligator, classifiers predict that the alligator
is a cliff, lynx, and a fox squirrel. Current convolutional networks have
pervasive and diverse failure modes that are tested with ImageNet-A.

###  4 Experiments

We show that adversarially filtered examples collected to fool fixed ResNet-50
models reliably transfer to other models, indicating that current
convolutional neural networks have shared weaknesses and failure modes. In the
following sections, we analyze whether robustness can be improved by using
data augmentation, using more real labeled data, and using different
architectures. For the first two sections, we analyze performance with a fixed
architecture for comparability, and in the final section we observe
performance with different architectures. As a preliminary, we define our
metrics.

Metrics. Our metric for assessing robustness to adversarially filtered
examples for classifiers is the top-1 _accuracy_ on ImageNet-A. For reference,
the top-1 accuracy on the 200 ImageNet-A classes using usual ImageNet images
is usually greater than or equal to $90\%$ for ordinary classifiers.

Our metric for assessing out-of-distribution detection performance of
ImageNet-O examples is the area under the precision-recall curve (_AUPR_).
This metric requires anomaly scores. Our anomaly score is the negative of the
maximum softmax probabilities [hendrycks2017baseline] from a model that can
classify the 200 ImageNet-O classes specified in Section 3. We collect anomaly
scores with the ImageNet validation examples for the said 200 classes. Then,
we collect anomaly scores for the ImageNet-O examples. Higher performing OOD
detectors would assign ImageNet-O examples lower confidences, or higher
anomaly scores. With these anomaly scores, we can compute the area under the
precision-recall curve [auprbaseline]. Random chance levels for the AUPR is
approximately $16.67\%$ with ImageNet-O, and the maximum AUPR is $100\%$.

##### Data Augmentation.

We examine popular data augmentation techniques and note their effect on
robustness. In this section we exclude ImageNet-O results, as the data
augmentation techniques hardly help with out-of-distribution detection as
well. As a baseline, we train a new ResNet-50 from scratch and obtain $2.17\%$
accuracy on ImageNet-A. Now, one purported way to increase robustness is
through adversarial training, which makes models less sensitive to $\ell_{p}$
perturbations. We use the adversarially trained model from [wong2020fast], but
accuracy decreases to $1.68\%$. Next, [geirhos2019] propose making networks
rely less on texture by training classifiers on images where textures are
transferred from art pieces. They accomplish this by applying style transfer
to ImageNet training images to create a stylized dataset, and models train on
these images. While this technique is able to greatly increase robustness on
synthetic corruptions [hendrycks2019robustness], Style Transfer increases
ImageNet-A accuracy by $0.13\%$ over the ResNet-50 baseline. A recent data
augmentation technique, AugMix [hendrycks2019augmix], which takes linear
combinations of different data augmentations increases accuracy to $3.8\%$.
Cutout augmentation [Devries2017ImprovedRO] randomly occludes image regions
and corresponds to $4.4\%$ accuracy. Moment Exchange (MoEx) [Li2020OnFN]
exchanges feature map moments between images, and this increases accuracy to
$5.5\%$. Mixup [zhang2017mixup] trains networks on elementwise convex
combinations of images and their interpolated labels; this technique increases
accuracy to $6.6\%$. CutMix [Yun2019CutMixRS] superimposes images regions
within other images and yields $7.3\%$ accuracy. At best these data
augmentations techniques improve accuracy by approximately $5\%$ over the
baseline. Results are summarized in figure 7.

Figure 7: Some data augmentation methods can slightly improve performance.

##### More Labeled Data.

One possible explanation for consistently low ImageNet-A accuracy is that all
models are trained only with ImageNet-1K, and using additional data may
resolve the problem. To test this hypothesis we pre-train a ResNet-50 on
Places365 [zhou2017places], a large-scale scene recognition dataset. After
fine-tuning the Places365 model on ImageNet-1K, we find that accuracy is
$1.56\%$. Consequently, even though scene recognition models are purported to
have qualitatively distinct features [Zhou2019InterpretingDV], this is not
enough to improve ImageNet-A performance. Likewise, Places365 pre-training
does not improve ImageNet-O detection, as its AUPR is $14.88\%$. Next, we see
whether labeled data from ImageNet-A itself can help. We take baseline
ResNet-50 with $2.17\%$ ImageNet-A accuracy and fine-tune it on $80\%$ of
ImageNet-A. This leads to no clear improvement on the remaining $20\%$ of
ImageNet-A since the top1 and top5 accuracies are below $2\%$ and $5\%$
respectively. Last, we pre-train using an order of magnitude more training
data with ImageNet-21K. This dataset contains approximately 21,000 classes and
approximately 14 million images. To our knowledge this is the largest publicly
available database of labeled natural images. Using a ResNet-50 pretrained on
ImageNet-21K, we fine-tune the model on ImageNet-1K and attain
$11.41\%$accuracy on ImageNet-A, a $9.24\%$ increase. Likewise, the AUPR for
ImageNet-O improves from $16.20\%$ to $21.86\%$, although this improvement is
less significant since ImageNet-O images overlap with ImageNet-21K images.
Overall, an order of magnitude increase in labeled training data can provide
some improvements in accuracy.

Figure 8: Increasing model size and other architecture changes can greatly
improve performance. Note Res2Net and ResNet+SE have a ResNet backbone. Normal
model sizes are ResNet-50 and ResNeXt-50 ($32\times 4$d), Large model size are
ResNet-101 and ResNeXt-101 ($32\times 4$d), and XLarge Model sizes are
ResNet-152 and ($32\times 8$d).

##### Architectural Changes.

We find that model architecture can play a large role in ImageNet-A accuracy
and ImageNet-O detection performance. Simply increasing the width and number
of layers of a network is sufficient to automatically impart more ImageNet-A
accuracy and ImageNet-O OOD detection performance. Increasing network capacity
has been shown to improve performance on $\ell_{p}$ adversarial examples
[kurakin], common corruptions [hendrycks2019robustness], and now also improves
performance for adversarially filtered images. For example, a ResNet-50’s
top-1 accuracy and AUPR is $2.17\%$ and $16.2\%$, respectively, while a
ResNet-152 obtains $6.1\%$ top-1 accuracy and $18.0\%$ AUPR. Another
architecture change that reliably helps is using the grouped convolutions
found in ResNeXts [resnext]. A ResNeXt-50 ($32\times 4$d) obtains a $4.81\%$
top1 ImageNet-A accuracy and a $17.60\%$ ImageNet-O AUPR. Another
architectural change is self attention. Convolutional neural networks with
self-attention [Hu2018GatherExciteE] are designed to better capture long-range
dependencies and interactions across an image. We consider the self-attention
technique called Squeeze-and-Excitation (SE) [Hu2018SqueezeandExcitationN],
which won the final ImageNet competition in 2017. A ResNet-50 with Squeeze-
and-Excitation attains $6.17\%$ accuracy. However, for larger ResNets, self-
attention does little to improve ImageNet-O detection. Finally, we consider
the ResNet-50 architecture with its residual blocks exchanged with recently
introduced Res2Net v1b blocks [Gao2019Res2NetAN]. This change increases
accuracy to $14.59\%$ and the AUPR to $19.5\%$. A ResNet-152 with Res2Net v1b
blocks attains $22.4\%$ accuracy and $23.9\%$ AUPR. Compared to data
augmentation or an order of magnitude more labeled training data, some
architectural changes can provide far more robustness gains. Consequently
future improvements to model architectures is a promising path towards greater
robustness.

###  5 Conclusion

In this chapter, we introduced adversarially filtered examples for image
classifiers and out-of-distribution detectors. Our ImageNet-A dataset degrades
classification accuracy across known classifiers, and it measures robustness
to input data distribution shifts. Likewise, ImageNet-O adversarially filtered
examples reliably degrade ImageNet out-of-distribution detection performance,
and it measures robustness to label distribution shifts. ImageNet-O enables
the measurement of adversarial out-of-distribution detection performance, and
is the first ImageNet out-of-distribution detection dataset. Our adversarial
filtration process removes examples solved by simple spurious cues, so our
datasets enable researchers to observe performance when simple spurious cues
are removed. Our naturally occurring images expose common blindspots of
current convolutional networks, and solving these tasks will require
addressing long-standing but under-explored failure modes of current models
such as over-reliance on texture, over-generalization, and spurious cues. We
found that these failures are slightly less pronounced with different data
augmentation strategies. However, we identified that architectural
improvements can provide large gains in model robustness, and there is much
room for future research. In this work, we introduce two new and difficult
ImageNet test sets to measure model performance under distribution shift—an
important research aim as computer vision systems are deployed in increasingly
precarious real-world environments.

##  Chapter 3 Scaling Anomaly Detection to Large Scale Images

###  1 Overview

Figure 1: We scale up out-of-distribution detection to large-scale multi-class
datasets with hundreds of classes, multi-label datasets with complex scenes,
and anomaly segmentation in driving environments. In all three settings, we
find that an OOD detector based on the maximum logit outperforms previous
methods, establishing a strong and versatile baseline for future work on
large-scale OOD detection.

Detecting out-of-distribution inputs is important in real-world applications
of deep learning. When faced with anomalous inputs flagged as such, systems
may initiate a conservative fallback policy or defer to human judgment. This
is especially important in safety-critical applications of deep learning, such
as self-driving cars or medical applications. Accordingly, research on out-of-
distribution detection has a rich history spanning several decades [OC-SVM,
lof, emmott2015metaanalysis]. Recent work leverages deep neural
representations for out-of-distribution detection in complex domains, such as
image data [hendrycks2017baseline, kimin, hendrycks2019oe]. However, these
works still primarily use small-scale datasets with low-resolution images and
few classes. As the community moves towards more realistic, large-scale
settings, strong baselines and high-quality benchmarks are imperative for
future progress.

In addition to focusing on small-scale datasets, previous formulations of
anomaly detection treat entire images as anomalies. In practice, an image
could be anomalous in localized regions while being in-distribution elsewhere.
Knowing which regions of an image are anomalous could allow for safer handling
of unfamiliar objects in the case of self-driving cars. Creating a benchmark
for this task is difficult, though, as simply cutting and pasting anomalous
objects into images introduces various unnatural giveaway cues such as edge
effects, mismatched orientation, and lighting, all of which trivialize the
task of anomaly segmentation [blum2019fishyscapes].

To overcome these issues, we utilize a simulated driving environment to create
the novel StreetHazards dataset for anomaly segmentation. Using the Unreal
Engine and the open-source CARLA simulation environment [carla], we insert a
diverse array of foreign objects into driving scenes and re-render the scenes
with these novel objects. This enables integration of the foreign objects into
their surrounding context with correct lighting and orientation.

To complement the StreetHazards dataset, we convert the BDD100K semantic
segmentation dataset [bdd100k] into an anomaly segmentation dataset, which we
call BDD-Anomaly. By leveraging the large scale of BDD100K, we reserve
infrequent object classes to be anomalies. We combine this dataset with
StreetHazards to form the Combined Anomalous Object Segmentation (CAOS)
benchmark. The CAOS benchmark improves over previous evaluations for anomaly
segmentation in driving scenes by evaluating detectors on realistic and
diverse anomalies. We evaluate several baselines on the CAOS benchmark and
discuss problems with porting existing approaches from earlier formulations of
out-of-distribution detection.

In more traditional whole-image anomaly detection, large-scale datasets such
as ImageNet [imagenet_cvpr09] and Places365 [zhou2017places] present unique
challenges not seen in small-scale settings, such as a plethora of fine-
grained object classes. We demonstrate that the MSP detector, a state-of-the-
art method for small-scale problems, does not scale well to these challenging
conditions. Moreover, in the common real-world case of multi-label data, the
MSP detector cannot naturally be applied in the first place, as it requires
softmax probabilities.

Through extensive experiments, we identify a detector based on the maximum
logit (MaxLogit) that greatly outperforms strong baselines in large-scale
multi-class, and anomaly segmentation settings. In each of these three
settings, we discuss why MaxLogit provides superior performance, and we show
that these gains are hidden if one looks at small-scale problems alone. The
code for our experiments and the CAOS benchmark datasets are available at
github.com/hendrycks/anomaly-seg.

###  2 Related Work

Anomaly Segmentation. Several prior works explore segmenting anomalous image
regions. One line of work uses the WildDash dataset [zendel2018wilddash],
which contains numerous annotated driving scenes in conditions such as snow,
fog, and rain. The WildDash test set contains fifteen "negative image" from
different domains for which the goal is to mark the entire image as out-of-
distribution. Thus, while the task is segmentation, the anomalies do not exist
as objects within an otherwise in-distribution scene. This setting is similar
to that explored by [hendrycks2017baseline], in which whole images from other
datasets serve as out-of-distribution examples.

To approach anomaly segmentation on WildDash, [kreo2018robust] train on
multiple semantic segmentation domains and treat regions of images from the
WildDash driving dataset as out-of-distribution if they are segmented as
regions from different domains, i.e. indoor classes. [bev2018discriminative]
use ILSVRC 2012 images and train their network to segment the entirety of
these images as out-of-distribution.

In medical anomaly segmentation and product fault detection, anomalies are
regions of otherwise in-distribution images. [Baur_2019] segment anomalous
regions in brain MRIs using pixel-wise reconstruction loss. Similarly,
[haselmann2018anomaly] perform product fault detection using pixel-wise
reconstruction loss and introduce an expansive dataset for segmentation of
product faults. In these relatively simple domains, reconstruction-based
approaches work well. In contrast to medical anomaly segmentation and fault
detection, we consider complex images from street scenes. These images have
high variability in scene layout and lighting, and hence are less amenable to
reconstruction-based techniques.

|  Fishyscapes |  Lost and Found |  BDD-Anomaly (Ours) |  StreetHazards (Ours)
---|---|---|---|---
Train Images |  0 |  1036 |  6280 |  5125
Test Images |  1000 |  1068 |  810 |  1500
Anomaly Types |  12 |  9 |  $3$ |  250
|  |  |  |
Table 1: Quantitative comparison of the CAOS benchmark with related datasets.
The BDD-Anomaly dataset treats three categories as anomalous and has many
unique object instances within those categories. By contrast, Lost and Found
has the same objects in multiple images and has only nine unseen objects at
test time. StreetHazards leverages a simulated environment to naturally insert
hundreds of varied anomalies.

The two works closest to our own are the Lost and Found [Pinggera2016LostAF]
and Fishyscapes [blum2019fishyscapes] datasets. In table 1, we quantitatively
compare the CAOS benchmark to these datasets. The Lost and Found dataset
consists of real images in a driving environment with small road hazards. The
images were collected to mirror the Cityscapes dataset [Cordts2016Cityscapes]
but are only collected from one city and so have less diversity. The dataset
contains 35 unique anomalous objects, and methods are allowed to train on many
of these. For Lost and Found, only nine unique objects are truly unseen at
test time. Crucially, this is a different evaluation setting from our own,
where anomalous objects are not revealed at training time, so their dataset is
not directly comparable. Nevertheless, the BDD-Anomaly dataset fills several
gaps in Lost and Found. First, the images are more diverse, because they are
sourced from a more recent and comprehensive semantic segmentation dataset.
Second, the anomalies are not restricted to small, sparse road hazards.
Concretely, anomalous regions in Lost and Found take up 0.11% of the image on
average, whereas anomalous regions in the BDD-Anomaly dataset are larger and
fill 0.83% of the image on average. Finally, although the BDD-Anomaly dataset
treats three categories as anomalous, compared to Lost and Found it has far
more unique anomalous objects.

The Fishyscapes benchmark for anomaly segmentation consists of cut-and-paste
anomalies from out-of-distribution domains. This is problematic, because the
anomalies stand out as clearly unnatural in context. For instance, the
orientation of anomalous objects is unnatural, and the lighting of the cut-
and-paste patch differs from the lighting in the original image, providing an
unnatural cue to anomaly detectors that would not exist for real anomalies.
Techniques for detecting image manipulation [zhou2018learning] are competent
at detecting artificial image elements of this kind. Our StreetHazards dataset
overcomes these issues by leveraging a simulated driving environment to
naturally insert anomalous 3D models into a scene rather than overlaying 2D
images. These anomalies are integrated into the scene with proper lighting and
orientation, mimicking real-world anomalies and making them significantly more
difficult to detect.

Figure 2: Small-scale datasets such as CIFAR-10 have relatively disjoint
classes, but larger-scale datasets including ImageNet have several classes
with high visual similarity to other classes. The implication is that large-
scale classifiers disperse probability mass among several classes. If the
prediction confidence is used for out-of-distribution detection, then images
which have similarities to other classes will often wrongly be deemed out-of-
distribution due to dispersed confidence. The dog is lower resolution for the
CIFAR-10 classifier.

Multi-Class Out-of-Distribution Detection. A recent line of work leverages
deep neural representations from multi-class classifiers to perform out-of-
distribution (OOD) detection on high-dimensional data, including images, text,
and speech data. [hendrycks2017baseline] formulate the task and propose the
simple baseline of using the maximum softmax probability of the classifier on
an input to gauge whether the input is out-of-distribution. In particular,
they formulate the task as distinguishing between examples from an in-
distribution dataset and various out-of-distribution datasets. Importantly,
entire images are treated as out-of-distribution.

Continuing this line of work, [kimin] propose to improve the neural
representation of the classifier to better separate in- and out-of-
distribution examples. They use generative adversarial networks to produce
near-distribution examples. In training, they encourage the classifier to
output a uniform posterior on these synthetic out-of-distribution examples.
[hendrycks2019oe] observe that outliers are often easy to obtain in large
quantity from diverse, realistic datasets and demonstrate that training out-
of-distribution detectors to detect these outliers generalizes to completely
new, unseen classes of anomalies. Other work investigates improving the
anomaly detectors themselves given a fixed classifier [devries,
Liang2018ODIN]. However, as observed in [hendrycks2019oe], most of these works
tune hyperparameters on a particular type of anomaly that is also seen at test
time, so their evaluation setting is more lenient. We ensure that all
anomalies seen at test time come from entirely unseen categories and are not
tuned on in any way, hence we do not compare to techniques such as
[Liang2018ODIN]. Additionally, in a point of departure from prior work, we
focus primarily on large-scale images and datasets with many classes.

###  3 Multi-Class Prediction for OOD Detection

Problem with existing baselines. In large-scale image classification, a
network is often tasked with predicting an object’s identity from one of
hundreds or thousands of classes, where class distinctions tend to be more
fine and subtle. An increase in similarity and overlap between classes spells
a problem for the multi-class out-of-distribution baseline
[hendrycks2017baseline]. This baseline uses the negative maximum softmax
probability as the anomaly score, or $-\max_{k}p(y=k\mid x)$. Classifiers tend
to have higher confidence on in-distribution examples than out-of-distribution
examples, enabling OOD detection. Assuming single-model evaluation and no
access to other anomalies or test-time adaptation, the maximum softmax
probability (MSP) is the state-of-the-art multi-class out-of-distribution
detection method. However, we show that the MSP is problematic for large-scale
datasets with many classes including ImageNet-1K and Places365
[zhou2017places]. Probability mass can be dispersed among visually similar
classes, as shown in figure 2. Consequently, a classifier may produce a low
confidence prediction for an in-distribution image, not because the image is
unfamiliar or out-of-distribution, but because the object’s exact class is
difficult to determine. To circumvent this problem, we propose using the
negative of the maximum unnormalized logit for an anomaly score, which we call
MaxLogit.

The MaxLogit has several benefits over the previous baseline of MSP.
Empirically we find that MaxLogit outperforms the MSP, although the difference
is marginal in small scale image datasets such as CIFAR-10 but show a larger
improvement on CAOS benchmark and ImageNet see 2. The two main reasons other
than emperical performance to switch from MSP to MaxLogit are that the
MaxLogit can work even if it is not a distributions and MaxLogit is unaffected
by the number of classes. To expand upon each point, for several image tasks
the output comes in the form of multi-label categories which the results are
not from a softmax and not a distribution as the results do not sum to one,
and MaxLogit does not require such a condition to in order to operate. The
other point is a bit more subtle but none-the-less important. While the
softmax operation does not change the maximum score, and does not change the
ordering of classes per item, it does affect the relative ordering across
items. This is because the softmax operation will convert the absolute
differences per item in relative differences which loses information when
comparing across items.

Datasets. To evaluate the MSP baseline out-of-distribution detector and the
MaxLogit detector, we use ImageNet-1K object recognition dataset and Places365
scene recognition dataset as in-distribution datasets
$\mathcal{D}_{\text{in}}$. We use several out-of-distribution test datasets
$\mathcal{D}_{\text{out}}$, all of which are unseen during training. The first
out-of-distribution dataset is _Gaussian_ noise, where each pixel of these
out-of-distribution examples are i.i.d. sampled from $\mathcal{N}(0,0.5)$ and
clipped to be contained within $[-1,1]$. Another type of test-time noise is
_Rademacher_ noise, in which each pixel is i.i.d. sampled from
$2\cdot\text{Bernoulli}(0.5)-1$, i.e. each pixel is $1$ or $-1$ with equal
probability. _Blob_ examples are more structured than noise; they are
algorithmically generated blob images. Meanwhile, _Textures_ is a dataset
consisting in images of describable textures [textures]. When evaluating the
ImageNet-1K detector, we use _LSUN_ images, which is a dataset for scene
recognition [lsun]. Our final $\mathcal{D}_{\text{out}}$ is _Places69_ , a
scene classification dataset that does not share classes with Places365. In
all, we evaluate against out-of-distribution examples spanning synthetic and
realistic images.

| FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$
---|---|---|---
$\mathcal{D}_{\text{in}}$ |  MSP |  MaxLogit |  KL |  MSP |  MaxLogit |  KL |  MSP |  MaxLogit |  KL
ImageNet |  42.42 |  35.77 |  36.22 |  84.60 |  87.20 |  87.29 |  48.26 |  45.68 |  37.32
Places365 |  52.68 |  36.6 |  49.14 |  75.67 |  85.9 |  80.01 |  8.13 |  19.2 |  24.61
|  |  |  |  |  |  |  |  |
Table 2: Multi-class out-of-distribution detection results using the maximum
softmax probability, maximum logit basline and KL Divergence between predicted
and posterior. Results are on ImageNet and Places365. Values are rounded so
that $99.995\%$ rounds to $100\%$. Full results on individual
$\mathcal{D}_{\text{out}}$ datasets and additional baselines are in the
supplementary material.

Results. Results are shown in table 2. Observe that the proposed MaxLogit
method outperforms the maximum softmax probability baseline for all three
metrics on both ImageNet and Places365. These results were computed using a
ResNet-50 trained on either ImageNet-1K or Places365. In the case of
Places365, the AUROC improvement is over 10%. We note that the utility of the
maximum logit could not as easily be appreciated in previous work’s small-
scale settings. For example, using the small-scale CIFAR-10 setup of
[hendrycks2019oe], the MSP attains an average AUROC of 90.08% while the
maximum logit attains 90.22%, a 0.14% difference. However, in a large-scale
setting, the difference can be over 10% on individual
$\mathcal{D}_{\text{out}}$ datasets. We are not claiming that utilizing the
maximum logit is a mathematically innovative formulation, only that it serves
as a consistently powerful baseline for large-scale settings that went
unappreciated in small-scale settings. In consequence, we suggest using the
maximum logit as a new baseline for large-scale multi-class out-of-
distribution detection.

###  4 The CAOS Benchmark

The Combined Anomalous Object Segmentation (CAOS) benchmark is comprised of
two complementary datasets for evaluating anomaly segmentation systems on
diverse, realistic anomalies. First is the StreetHazards dataset, which
leverages simulation to provide a large variety of anomalous objects
realistically inserted into driving scenes. Second is the BDD-Anomaly dataset,
which consists of real images taken from the BDD100K dataset [bdd100k].
StreetHazards contains a highly diverse array of anomalies; BDD-Anomaly
contains anomalies in real-world images. Together, these datasets allow
researchers to judge techniques on their ability to segment diverse anomalies
as well as anomalies in real images. All images have $720\times 1280$
resolution, and we recommend evaluating with the AUROC, AUPR, and FPR$K$
metrics, which we describe in Section 1.

The StreetHazards Dataset. StreetHazards is an anomaly segmentation dataset
that leverages simulation to provide diverse, realistically-inserted anomalous
objects. To create the StreetHazards dataset, we use the Unreal Engine along
with the CARLA simulation environment [carla]. From several months of
development and testing including customization of the Unreal Engine and
CARLA, we can insert foreign entities into a scene while having them be
properly integrated. Unlike previous work, this avoids the issues of
inconsistent chromatic aberration, edge effects, differences in environmental
lighting, and other simple cues that an object is anomalous. Additionally,
using a simulated environment allows us to dynamically insert diverse
anomalous objects in any location and have them render properly with changes
to lighting and weather including time of day, cloudy skies, and rain.

Figure 3: A sample of anomalous scenes, model predictions, and anomaly scores.
The anomaly scores are thresholded to the top 10% of values for visualization.
GT is ground truth, the autoencoder model is based on the spatial autoencoder
used in [Baur_2019], MSP is the maximum softmax probability baseline
[hendrycks2017baseline], and MaxLogit is the method we propose as a new
baseline for large-scale settings.

We use 3 towns from CARLA for training, from which we collect RGB images and
their respective semantic segmentation maps to serve as our training data for
our semantic segmentation model. We generate a validation set from the fourth
town. Finally, we reserve the fifth and sixth town as our test set. We insert
anomalies taken from the Digimation Model Bank Library and semantic ShapeNet
(ShapeNetSem) [shapenet] into the test set in order to evaluate methods for
out-of-distribution detection. In total, we use 250 unique anomaly models of
diverse types. There are 12 classes used for training: background, road,
street lines, traffic signs, sidewalk, pedestrian, vehicle, building, wall,
pole, fence, and vegetation. The thirteenth class is the anomaly class that is
only used at test time. We collect 5,125 image and semantic segmentation
ground truth pairs for training, 1,031 pairs without anomalies for validation,
and 1,500 test pairs with anomalies.

The BDD-Anomaly Dataset. BDD-Anomaly is an anomaly segmentation dataset with
real images in diverse conditions. We source BDD-Anomaly from BDD100K
[bdd100k], a large-scale semantic segmentation dataset with diverse driving
conditions. The original data consists in 7000 images for training and 1000
for validation. There are 18 original classes. We choose motorcycle, train,
and bicycle as the anomalous object classes and remove all images with these
objects from the training and validation sets. This yields 6,280 training
pairs, 910 validation pairs without anomalies, and 810 testing pairs with
anomalous objects.

####  1 Experiments

Metrics. To evaluate out-of-distribution detectors in large-scale settings, we
use three standard metrics of detection performance: area under the ROC curve
(AUROC), false positive rate at 95% recall (FPR95), and area under the
precision-recall curve (AUPR). The AUROC and AUPR are important metrics,
because they give a holistic measure of performance when the cutoff for
detecting anomalies is not a priori obvious or when we want to represent the
performance of a detection method across several different cutoffs.

The AUROC can be thought of as the probability that an anomalous example is
given a higher score than an ordinary example. Thus, a higher score is better,
and an uninformative detector has a AUROC of 50%. AUPR provides a metric more
attuned to class imbalances, which is relevant in anomaly and failure
detection, when the number of anomalies or failures may be relatively small.
Last, the FPR95 metric consists of measuring the false positive rate at 95%.
This is important because it tells us how many false positives (i.e. false
alarms) are necessary for a given method to achieve a desired recall. This
desired recall may be thought of as a safety threshold. Moreover, anomalies
and system failures may require human intervention, so a detector requiring
little human intervention while still detecting most anomalies is of pecuniary
importance.

In anomaly segmentation experiments, each pixel is treated as a prediction,
resulting in many predictions to evaluate. To fit these in memory, we compute
the metrics on each image and average over the images to obtain final values.

Methods. Our first baseline is pixel-wise Maximum Softmax Probability (MSP).
Introduced in [hendrycks2017baseline] for multi-class out-of-distribution
detection, we directly port this baseline to anomaly segmentation.
Alternatively, the background class might serve as an anomaly detector,
because it contains everything not in the other classes. To test this
hypothesis, "Background" uses the posterior probability of the background
class as the anomaly score. The Dropout method leverages MC Dropout [gal] to
obtain an epistemic uncertainty estimate. We follow the implementation by
[Kendall2015BayesianSM].MC Dropout is computed by leaving dropout on during
inference and then one runs several forward passes of the image through the
network. This creates a set of predictions for the given object that we use to
compute the variance over the set of predictions. The variance over the set of
predictions serves as the anomaly score. A higher variance implies higher
uncertainty about which class the object belongs to. We also experiment with
an autoencoder baseline similar to [Baur_2019, haselmann2018anomaly] where
pixel-wise reconstruction loss is used as the anomaly score. By this we mean
that we run the image through the autoencoder and subtract the resulting image
from the input. The absolute difference in magnitude serves as the anomaly
score. This method is called AE. The "Branch" method is a direct port of the
confidence branch detector from [devries] to pixel-wise prediction. This
method trains a separate final output score along with a classification
output. The output score is multiplied to the predictions so that the
predictions are scaled based on the model’s confidence. The training involves
the slight modification to cross entropy $\mathcal{L}=-ylog(p\cdot c)$ where y
is the label, p is the probability, and c is the confidence score. The
confidence score is always between 0-1 by applying the sigmoid function
ensuring that the cross entropy function is still valid. The confidence is
trained via backpropagation similarly to the classification prediction.
Finally, we use the MaxLogit method described in earlier sections.

For all of the baselines except the autoencoder, we train a PSPNet
[Zhao2017PyramidSP] decoder with a ResNet-101 encoder [resnet] for $20$
epochs. The PSPNet follows a similar pattern to Zoomout [mostajabi2015zoomout]
or Hypercolumns [hypercol] whereby the activations of the convolutional layers
are concatenated together before finally undergoing a fully connected layer to
arrive at the appropriate dimensionality. We train both the encoder and
decoder using SGD with momentum of 0.9, a learning rate of $2\times 10^{-2}$,
and learning rate decay of $10^{-4}$. For the autoencoder, we use a 4-layer
U-Net [Ronneberger_2015] with a spatial latent code as in [Baur_2019]. The
U-Net also uses batch norm and is trained for $10$ epochs.

To evaluate the methods, we take all of the scores per pixel that belong to
anomalies and all of the scores for the remaining pixels. Then we sort the
scores, thereby allowing us to compute the false positive rate and true
positive rate at every threshold. We use the thresholds to compute the AUROC
giving us the probability that we correctly select an in-distribution pixel
with the method we’re evaluating. We also predefine a set threshold at 5% and
compute the false positive rate for that threshold. Due to the large number of
scores to be evaluated and sorted, we take the mean over all the images of
each evaluation as a final report.

|  |  |  MSP |  Branch |  Background |  Dropout |  AE |  MaxLogit
---|---|---|---|---|---|---|---|---
StreetHazards |  FPR95 | $\downarrow$ |  33.7 |  68.4 |  69.0 |  79.4 |  91.7 |  26.5
AUROC | $\uparrow$ |  87.7 |  65.7 |  58.6 |  69.9 |  66.1 |  89.3
AUPR | $\uparrow$ |  6.6 |  1.5 |  4.5 |  7.5 |  2.2 |  10.6
BDD-Anomaly |  FPR95 | $\downarrow$ |  24.5 |  25.6 |  40.1 |  16.6 |  74.1 |  14.0
AUROC | $\uparrow$ |  87.7 |  85.6 |  69.7 |  90.8 |  64.0 |  92.6
AUPR | $\uparrow$ |  3.7 |  3.9 |  1.1 |  4.3 |  0.7 |  5.4
|  |  |  |  |  |  |  |
Table 3: Results on the Combined Anomalous Object Segmentation benchmark. AUPR
is low across the board due to the large class imbalance, but all methods
perform substantially better than chance. MaxLogit obtains the best
performance. All results are percentages.

Results and Analysis. MaxLogit outperforms all other methods across the board
by a substantial margin. The intuitive baseline of using the posterior for the
background class to detect anomalies performs poorly, which suggests that the
background class may not align with rare visual features. Even though
reconstruction-based scores succeed in product fault segmentation, we find
that the AE method performs poorly on the CAOS benchmark, which may be due to
the more complex domain. AUPR for all methods is low, indicating that the
large class imbalance presents a serious challenge. However, the substantial
improvements with the MaxLogit method suggest that progress on this task is
possible and there is much room for improvement.

In figure 3, we can qualitatively see that both MaxLogit and MSP have a high
number of false positives, as they assign high anomaly scores to semantic
boundaries, a problem also observed in the recent works of
[blum2019fishyscapes, thesis_canny]. However, the problem is less severe in
MaxLogit. A potential explanation for this could be due to two effects. The
first we mentioned earlier in the benefits of MaxLogit over MSP (see section
3) in that the inter-class variance is better preserved in MaxLogit over MSP.
The second effect builds off of the first, by having a greater range in
MaxLogit as compared to MSP bilinear upsampling from the models final output
to the final output image creates much sharper boundaries. This is because the
interpolation of points that are already close (with MSP) will blur the
boundaries much more and cause more pixels to become classified as in-
distribution by exceeding the threshold.

Autoencoder-based methods are qualitatively different from approaches using
the softmax probabilities, because they model the input itself and can avoid
boundary effects seen in the MaxLogit and MSP rows of figure 3. While
autoencoder methods are successful in medical anomaly segmentation and product
fault detection, we find the AE baseline to be ineffective in the more complex
domain of street scenes. The last row of figure 3 shows pixel-wise
reconstruction loss on example images from StreetHazards. Anomalies are not
distinguished well from in-distribution elements of the scene. New methods
must be developed to mitigate the boundary effects faced by softmax-based
methods while also attaining good detection performance.

###  5 Conclusion

We scaled out-of-distribution detection to more realistic, large-scale
settings by developing a novel benchmark for OOD segmentation. The CAOS
benchmark for anomaly segmentation consists of diverse, naturally-integrated
anomalous objects in driving scenes. Baseline methods on the CAOS benchmark
substantially improve on random guessing but are still lacking, indicating
potential for future work. We also investigated using multi-label classifiers
for out-of-distribution detection and established an experimental setup for
this previously unexplored setting. On large-scale multi-class image datasets,
we identified an issue faced by existing baselines and proposed the maximum
logit detector as a natural solution. Interestingly, this detector also
provides consistent and significant gains in the multi-label and anomaly
segmentation settings, thereby establishing it as a new baseline in place of
the maximum softmax probability baseline on large-scale OOD detection
problems. In all, we hope that our simple baseline and our new OOD
segmentation benchmark will enable further research on out-of-distribution
detection for real-world safety-critical environments.

##  Chapter 4 Neural augmentations

###  1 Overview

While the research community must create robust models that generalize to new
scenarios, the robustness literature [dodge2017study, geirhos2020shortcut]
lacks consensus on evaluation benchmarks and contains many dissonant
hypotheses. While [hendrycks2020pretrained] find that many recent language
models are already robust to many forms of distribution shift,
[yin2019fourier], and [geirhos2019] find that vision models are largely
fragile and argue that data augmentation offers a solution. In contrast,
[taori2020when] provide results suggesting that using pretraining and
improving in-distribution test set accuracy improve natural robustness,
whereas other methods do not.

In this chapter we articulate and systematically study seven robustness
hypotheses. The first four hypotheses concern _methods_ for improving
robustness, while the last three hypotheses concern abstract _properties_
about robustness. These hypotheses are as follows.

  * •

Larger Models: increasing model size improves robustness [Xie2020Intriguing].

  * •

Self-Attention: adding self-attention layers to models improves robustness
[Hendrycks2019NaturalAE].

  * •

Diverse Data Augmentation: robustness can increase through data augmentation
[yin2019fourier].

  * •

Pretraining: pretraining on larger and more diverse datasets improves
robustness [Orhan2019RobustnessPO, Hendrycks2019Pretraining].

  * •

Texture Bias: convolutional networks are biased towards texture, which harms
robustness [geirhos2019].

  * •

Only IID Accuracy Matters: accuracy on independent and identically distributed
test data entirely determines natural robustness
[Taori2020RechtDistributionShiftImagenet].

  * •

Synthetic $\mathrel{{\ooalign{$\not\phantom{=}$\cr$\implies$}}}$ Natural:
_synthetic_ robustness interventions including diverse data augmentations do
not help with robustness on _naturally occurring_ distribution shifts
[taori2020when].

Figure 1:  Images from our three new datasets ImageNet-Renditions
(ImageNet-R), DeepFashion Remixed (DFR), and StreetView StoreFronts (SVSF).
The SVSF images are recreated from the public Google StreetView, copyright
Google 2020. Our datasets test robustness to various naturally occurring
distribution shifts including rendition style, camera viewpoint, and
geography.

It has been difficult to arbitrate these hypotheses because existing
robustness datasets preclude the possibility of controlled experiments by
varying multiple aspects simultaneously. For instance,  Texture Bias was
initially investigated with synthetic distortions [geirhos], which conflicts
with the  Synthetic $\mathrel{{\ooalign{$\not\phantom{=}$\cr$\implies$}}}$
Natural hypothesis. On the other hand, natural distribution shifts often
affect many factors (e.g., time, camera, location, etc.) simultaneously in
unknown ways [Recht2019DoIC, Hendrycks2019NaturalAE]. Existing datasets also
lack diversity such that it is hard to extrapolate which methods will improve
robustness more broadly. To address these issues and test the seven hypotheses
outlined above, we introduce three new robustness benchmarks and a new data
augmentation method.

First we introduce ImageNet-Renditions (ImageNet-R), a 30,000 image test set
containing various renditions (e.g., paintings, embroidery, etc.) of ImageNet
object classes. These renditions are naturally occurring, with textures and
local image statistics unlike those of ImageNet images, allowing us to more
cleanly separate the _Texture Bias_ and _Synthetic
$\mathrel{{\ooalign{$\not\phantom{=}$\cr$\implies$}}}$ Natural_ hypotheses.

Next, we investigate natural shifts in the image capture process with
StreetView StoreFronts (SVSF) and DeepFashion Remixed (DFR). SVSF contains
business storefront images taken from Google Streetview, along with metadata
allowing us to vary location, year, and even the camera type. DFR leverages
the metadata from DeepFashion2 [ge2019deepfashion2] to systematically shift
object occlusion, orientation, zoom, and scale at test time. Both SVSF and DFR
provide distribution shift controls and do not alter texture, which remove
possible confounding variables affecting prior benchmarks. DFR is discussed in
greater detail in 3.

Finally, we contribute DeepAugment to increase robustness to some new types of
distribution shift. This augmentation technique uses image-to-image neural
networks for data augmentation, not data-independent Euclidean augmentations
like image shearing or rotating as in previous work. DeepAugment achieves
state-of-the-art robustness on our newly introduced ImageNet-R benchmark and a
corruption robustness benchmark. DeepAugment can also be combined with other
augmentation methods to outperform a model pretrained on $1000\times$ more
labeled data.

After examining our results on these three datasets and others, we can rule
out several of the above hypotheses while strengthening support for others. As
one example, we find that synthetic data augmentation robustness interventions
improve accuracy on ImageNet-R and real-world image blur distribution shifts,
providing clear counterexamples to  Synthetic
$\mathrel{{\ooalign{$\not\phantom{=}$\cr$\implies$}}}$ Natural while lending
support to the _Diverse Data Augmentation_ and Texture Bias hypotheses. In the
conclusion, we summarize the various strands of evidence for and against each
hypothesis. Across our many experiments, we do not find a general method that
consistently improves robustness, and some hypotheses require additional
qualifications. While robustness is often spoken of and measured as a single
scalar property like accuracy, our investigations suggest that robustness is
not so simple. In light of our results, we hypothesize in the conclusion that
robustness is _multivariate_.

###  2 Related Work

##### Robustness Benchmarks.

Recent works [hendrycks2019robustness, Recht2019DoIC, hendrycks2020pretrained]
have begun to characterize model performance on out-of-distribution (OOD) data
with various new test sets, with dissonant findings. For instance,
[hendrycks2020pretrained] demonstrate that modern language processing models
are moderately robust to numerous naturally occurring distribution shifts, and
that _Only IID Accuracy Matters_ is inaccurate for natural language tasks. For
image recognition, [hendrycks2019robustness] analyze image models and show
that they are sensitive to various simulated image corruptions (e.g., noise,
blur, weather, JPEG compression, etc.) from their “ImageNet-C” benchmark.

[Recht2019DoIC] reproduce the ImageNet [imagenet] validation set for use as a
benchmark of naturally occurring distribution shift in computer vision. Their
evaluations show a 11-14% drop in accuracy from ImageNet to the new validation
set, named ImageNetV2, across a wide range of architectures. [taori2020when]
use ImageNetV2 to measure natural robustness and dismiss _Diverse Data
Augmentation_. [engstrom2020identifying] identify statistical biases in
ImageNetV2’s construction, and they estimate that reweighting ImageNetV2 to
correct for these biases results in a less substantial 3.6% drop.

In contrast to _adversarial_ robustness [szegedy2013intriguing,
goodfellow2014explaining], we focus instead on robustness to unconstrained
out-of-distribution data measured on non-interactive benchmarks.
[carlini2019evaluating] detail the inherent practical difficulty in evaluating
adversarial robustness, and [Gilmer2018MotivatingTR] argue for the inclusion
of unconstrained input modifications in the threat model of attacks against
machine learning systems.

##### Data Augmentation.

[geirhos2019, yin2019fourier, hendrycks2019augmix] demonstrate that data
augmentation can improve robustness on ImageNet-C. The space of augmentations
that help robustness includes various types of noise [madry,
rusak2020increasing, Lopes2019ImprovingRW], highly unnatural image
transformations [geirhos2019, Yun2019CutMixRS, zhang2017mixup], or
compositions of simple image transformations such as Python Imaging Library
operations [Cubuk2018AutoAugmentLA, hendrycks2019augmix]. Some of these
augmentations can improve accuracy on in-distribution examples as well as on
out-of-distribution (OOD) examples.

##### Transfer learning

Pretraining models on larger datasets has been been demonstrated to improve
robustness on ImageNet-C. [Orhan2019RobustnessPO] report that models trained
on the JFT-300m dataset [Sun_2017_ICCV] and the weakly-supervised Instagram
dataset [instagram2018] improve robustness on the ImageNet-C by significant
margins.

###  3 New Benchmarks

In order to evaluate the seven robustness hypotheses, we introduce three new
benchmarks that capture new types of naturally occurring distribution shifts.
ImageNet-Renditions (ImageNet-R) is a newly collected test set intended for
ImageNet classifiers, whereas StreetView StoreFronts (SVSF) and DeepFashion
Remixed (DFR) each contain their own training sets and multiple test sets.
SVSF and DFR split data into a training and test sets based on various image
attributes stored in the metadata. For example, we can select a test set with
images produced by a camera different from the training set camera. We now
describe the structure and collection of each dataset.

####  1 ImageNet-Renditions (ImageNet-R)

Figure 2:  ImageNet-Renditions (ImageNet-R) contains 30,000 images of ImageNet
objects with different textures and styles. This figure shows only a portion
of ImageNet-R’s numerous rendition styles. The rendition styles (e.g., “Toy”)
are for clarity and are _not_ ImageNet-R’s classes; ImageNet-R’s classes are a
subset of 200 ImageNet classes. ImageNet-R emphasizes shape over texture.

While current classifiers can learn some aspects of an object’s shape
[mordvintsev2015inceptionism], they nonetheless rely heavily on natural
textural cues [geirhos2019]. In contrast, human vision can process abstract
visual renditions. For example, humans can recognize visual scenes from line
drawings as quickly and accurately as they can from photographs
[biederman1988surface]. Even some primates species have demonstrated the
ability to recognize shape through line drawings [Itakura1994, Tanaka2006].

To measure generalization to various abstract visual renditions, we create the
ImageNet-Rendition (ImageNet-R) dataset. ImageNet-R contains various artistic
renditions of object classes from the original ImageNet dataset. Note the
original ImageNet dataset discouraged such images since annotators were
instructed to collect "photos only, no painting, no drawings, etc."
[deng2012large]. We do the opposite.

##### Data Collection.

ImageNet-R contains 30,000 image renditions for 200 ImageNet classes. We
collect images primarily from Flickr and use queries such as "art,"
"cartoons," "graffiti," "embroidery," "graphics," "origami," "paintings,"
"patterns," "plastic objects," "plush objects," "sculptures," "line drawings,"
"tattoos," "toys," "video game," and so on. Examples are depicted in figure 2.
Images are filtered by Amazon MTurk workers using a modified collection
interface from ImageNetV2 [Recht2019DoIC]. The resulting images are then
manually filtered by graduate students. ImageNet-R also includes the line
drawings from [wang2019learning], excluding horizontally mirrored duplicate
images, pitch black images, and images from the incorrectly collected "pirate
ship" class.

####  2 StreetView StoreFronts (SVSF)

Computer vision applications often rely on data from complex pipelines that
span different hardware, times, and geographies. Ambient variations in this
pipeline may result in unexpected performance degradation, such as
degradations experienced by health care providers in Thailand deploying
laboratory-tuned diabetic retinopathy classifiers in the field
[beede2020human]. In order to study the effects of shifts in the image capture
process we collect the StreetView StoreFronts (SVSF) dataset, a new image
classification dataset sampled from Google StreetView imagery
[anguelov2010google] focusing on three distribution shift sources: country,
year, and camera.

##### Data Collection.

SVSF consists of cropped images of business store fronts extracted from
StreetView images by an object detection model. Each store front image is
assigned the class label of the associated Google Maps business listing
through a combination of machine learning models and human annotators. We
combine several visually similar business types (e.g. drugstores and
pharmacies) for a total of 20 classes, listed Appendix 6. We are currently
unable to release the SVSF data publicly.

Splitting the data along the three metadata attributes of country, year, and
camera, we create one training set and five test sets. We sample a training
set and an in-distribution test set (200K and 10K images, respectively) from
images taken in US/Mexico/Canada during 2019 using a "new" camera system. We
then sample four OOD test sets (10K images each) which alter one attribute at
a time while keeping the other two attributes consistent with the training
distribution. Our test sets are year: 2017, 2018; country: France; and camera:
“old.”

###  4 DeepAugment

In order to further explore the _Diverse Data Augmentation_ hypothesis, we
introduce a new data augmentation technique we call DeepAugment. DeepAugment
works by passing an image through an image-to-image network (such as an image
autoencoder or a superresolution network), but rather than processing the
image normally, we distort the internal weights and activations. We distort
the image-to-image network’s weights by applying randomly sampled operations
such as zeroing, negating, convolving, transposing, applying activation
functions, and so on. This creates diverse but semantically consistent images
as illustrated in figure 3. We provide the pseudocode in Appendix 7. Whereas
most previous data augmentations techniques use simple augmentation primitives
applied to the raw image itself, we stochastically distort the internal
representations of image-to-image networks to augment images. We did not
experiment with any geometric image-to-image networks such as NeRF