# Spuriosity Rankings: Sorting Data for Spurious Correlation Robustness

Mazda Moayeri, Wenxiao Wang, Sahil Singla, Soheil Feizi
University of Maryland
Department of Computer Science
{mmoayeri, wwx, ssingla<EMAIL_ADDRESS>
###### Abstract

We present a framework for ranking images within their class based on the
strength of spurious cues present. By measuring the gap in accuracy on the
highest and lowest ranked images (we call this spurious gap), we assess
spurious feature reliance for $89$ diverse ImageNet models, finding that even
the best models underperform in images with weak spurious presence. However,
the effect of spurious cues varies far more dramatically across classes,
emphasizing the crucial, often overlooked, class-dependence of the spurious
correlation problem. While most spurious features we observe are clarifying
(i.e. improving test-time accuracy when present, as is typically expected), we
surprisingly find many cases of confusing spurious features, where models
perform better when they are absent. We then close the spurious gap by
training new classification heads on lowly ranked (i.e. without common
spurious cues) images, resulting in improved effective robustness to
distribution shifts (ObjectNet, ImageNet-R, ImageNet-Sketch). We also propose
a second metric to assess feature reliability, finding that spurious features
are generally less reliable than non-spurious (core) ones, though again,
spurious features can be more reliable for certain classes. To enable our
analysis, we annotated $5,000$ feature-class dependencies over all of ImageNet
as core or spurious using minimal human supervision. Finally, we show the
feature discovery and spuriosity ranking framework can be extended to other
datasets like CelebA and WaterBirds in a lightweight fashion with only linear
layer training, leading to discovering a previously unknown racial bias in the
Celeb-A hair classification.

Figure 1: Spuriosity rankings sort images based on the presence of spurious
cues. The rankings reveal hidden spurious features like rippling water, human
hands, and flame in the dark (right; top to bottom) and identify minority
subgroups where spurious cues are absent (left).

##  1 Introduction

The vast size of modern deep networks and their training sets obscures the
data patterns they rely on and the degree to which they do so. Moreover, deep
models are prone to depending on spurious features [41, 36, 1], which are
predictive of a class within the training distribution but irrelevant to the
true labelling function. Spurious feature dependence hinders robust
generalization (e.g. medical systems trained in one hospital may fail when
deployed to others [51, 11]) and can reduce fairness (e.g. facial recognition
and recidivism prediction systems can become biased to race [5, 9]).

Many algorithmic solutions have been proposed to mitigate spurious feature
dependence [38, 20, 23, 2], though they often require some supervision about
whether the spurious feature is present or not (typically in the form of a
binary label). However, re-annotating data is expensive and necessitates
knowledge of some single spurious cues a model may rely on apriori.
Discovering spurious cues is non-trivial, and further, binary labelling of the
presence of an attribute may be misleading, as an attribute can be more
prevalent in one sample than another. Furthermore, the spurious correlation
problem is crucially class-dependent. As shown in Figure 2, the concept pine
trees is spurious to the class ski, but core to the class lakeshore, since
they are a part of the lakeshore. Thus, existing paradigms of adding binary
annotation of a single class-agnostic spurious feature may be incomplete.

Instead, a procedure to reveal when and to what degree spurious features
(specific to each class) are present would provide richer signal to existing
methods for spurious correlation robustness. We propose to rank images within
their classes with respect to spuriosity (i.e. presence of cues spurious to
the class). This ranking provides natural interpretation of the spurious
features by presenting them in context (i.e. in highly ranked images) and
illuminating minority subpopulations where the features are absent (i.e. lowly
ranked images). For example, in figure 1, we can immediately identify spurious
features (i.e. water for an otter) and minority subgroups with spurious
correlations broken (e.g. otters in rare contexts like snow or grass) by
inspecting images with highest and lowest spuriosity ranks respectively.

It has been shown that neural features (i.e. nodes in the penultimate node of
a trained deep network), particularly those that contribute highly to a
specific class logit, encode concepts relevant to the class [47]. Thus, the
activation of a neural feature can serve as an automated proxy for the degree
to which a concept is present. We use robust neural features due to their
enhanced interpretability [42], and after annotating 5000 class-feature
dependencies using minimal human supervision, we discover that at least 357
classes of ImageNet contain spurious cues that models may rely upon. Then, per
class, we rank both training and validation images by their spuriosity using
activations of robust neural features deemed spurious for the class. Indeed,
visual inspection of the highest and lowest ranked images confirms that using
neural features as proxies to concepts that humans deem as spurious is
effective, as images with the highest and lowest spuriosity differ markedly in
the non-core features present (see Figure 1).

With spuriosity rankings, we first assess the reliance of $89$ models with
diverse architectures (attention, conv, and mlp-based) and training methods
(adversarial, vision + language, SSL, etc) on spurious cues using our spurious
gap metric (i.e. drop in accuracy on highest vs. lowest ranked images). We
observe CLIP [34] and adversarially trained [25] models to be the least and
most biased respectively to images with high spuriosity, consistent with prior
work [28, 27] and thus further validating our rankings. All models
underperform on images with low spuriosity, stressing the hidden bias of
models to spurious cues is still pervasive. More striking, though, is the
variance in spurious gap per class, which is $3\times$ as high as the variance
across models.

Figure 2: A feature (e.g. mountain pine trees) can be core or spurious
depending on the class, as shown through our complementary visualizations:
(left) top activating images, (middle) heatmaps showing regions activating the
feature in red, and (right) feature attacks that perturb the image to visually
amplify the feature.

A common belief in the community is that model accuracy drops when spurious
correlations are broken in the test time. In our experiments, we observe this
for many classes, reflecting that test-time spurious cues often help clarify
the class. However, some classes actually have higher test-time accuracy on
images without spurious cues. Here, the spurious features are confusing
because they are also relied upon, more strongly, by other classes. For
example, in predicting the class car wheel, a model may look for the body of a
car. However, if the car body is very prominent, the model will incorrectly
predict a car class as opposed to car wheel. Indeed, models are on average
$54.2\%$ less accurate on car wheel images with high spuriosity. The existence
of both clarifying and confusing spurious features underscores the crucial
class dependence and previously overlooked nuance in the spurious correlation
problem.

(a) Core Feature: antennae, hair

(b) Core Feature: wing

(c) Spurious Feature: purple flower

Figure 3: Examples of visualizations used to annotate important neural
features as core or spurious for small white butterfly class.

We then train linear heads on subsets of data determined by our ranking.
Training on images with low spuriosity closes the spurious gap, removing the
bias towards the spurious feature. Further, this comes at no drop in out-of-
distribution accuracy, suggesting that in-distribution accuracy gains due to
spurious feature reliance offer an inflated sense of model performance.
Conversely, including low spuriosity images may be vital for distributional
robustness.

We further probe the role of core and spurious features for classification in
and out of distribution with a second metric that assesses the reliability of
a neural feature directly. We find that core features are generally more
reliable, but for some classes, surprisingly only spurious features are
reliable, highlighting yet another complexity in which spurious features can
actually help classification, even under distribution shift.

Finally, we extend our robust-neural-feature-based ranking framework to new
domains with only retraining a linear layer (i.e. the simplest transfer
learning setting), expanding its use cases to low data regimes where
adversarial training would be unstable. The lightweight extension of our
framework detects designated spurious attributes for Waterbirds and Celeb-A
classification tasks. Furthermore, we uncover a previously unobserved racial
bias in the Celeb-A hair classification task, demonstrating our framework’s
ability to identify hidden spurious features.

##  2 Defining and Computing Spuriosity

Models often rely on many data patterns to recognize each class. Suppose we
identify a set $\mathcal{S}(i)$ of cues predictive of class label $i$ in a
spurious way. These cues can be abstract (e.g. sand helps predict fiddler
crab), and their presence in an image may not be binary (e.g. there is more
sand at a beach than in a terrarium). We define spuriosity as follows.

###### Definition 1

Spuriosity (conceptual): The average strength of spurious cues
$\mathcal{S}(i)$ for a sample $x$ from class $i$.

The ‘strength’ of a spurious cue is purely qualitative. However, we can
quantitatively proxy this notion via activations of penultimate nodes (called
neural features) in trained classifiers. Neural features have been observed to
respond to stimuli corresponding to concepts related to the classes the model
is trained to predict [32]. Further, since classifiers are composed of linear
layers operating over neural features, it is easy to select features that
contribute the most to a class logit by inspecting linear layer weights,
indicating that the cues they detect pertain to the class.

Specifically, we use robust neural features (i.e. from an adversarially
trained network), because of their enhanced interpretability. Namely, as shown
in [42], one can reliably generate heatmaps highlighting input regions
activating a robust neural feature, and even perturb an input image to
visually amplify the cue that the feature detects. Combined with showing the
highest-ranked images for a cue, annotation of a robust neural feature’s
function as core or spurious becomes feasible. Figure 3 exemplifies these
visualizations for three cues relevant to the small white butterfly class,
including a spurious cue (purple flowers).

We apply the feature discovery framework preliminary introduced in [42] at a
much larger scale to collect class-wise spurious dependencies across ImageNet.
We annotate $5000$ class-feature dependencies (namely the five features per
class contributing most to the corresponding class logit), of which $630$ are
spurious over $357$ classes. We will maintain a web interface for easy viewing
of our analysis, offering a direct visual look into the patterns a neural
network sees and uses across ImageNet. We also verify that heatmaps for
annotated features localize the same cue in images whose activation on the
feature is in the top $20^{th}$ percentile for the class, successfully
validating $95.3\%$ of annotated class-feature pairs (see appendix for
details). Thus, we generate $325,000$ feature segmentations across ImageNet as
a bonus; we characterize model sensitivity to these regions in the appendix.
Crucially, the validation confirms that sorting by feature activation is
effective in gathering instances where a feature is present.

Thus, we can now compute spuriosity empirically, leveraging the annotated
pairs of classes and robust neural features by simply assessing activation on
the features. We use pre-relu activations to better distinguish low-activating
images. Formally, letting $\mathbf{r}_{j}(x)$ denote the activation of image
$x$ on the $j^{th}$ robust neural feature, and $\tilde{\mathcal{S}}(i)$
consisting of the set of robust neural features annotated as spurious for
class $i$, we can re-define spuriosity as follows.

###### Definition 2

Spuriosity (empirical) of $x$ for class $i$, with $\mu_{ij}$ and $\sigma_{ij}$
denoting the mean and standard deviation of activations on feature $j$ over
class $i$, is
$\frac{1}{|\tilde{\mathcal{S}}(i)|}\sum\limits_{j\in\tilde{\mathcal{S}}(i)}\frac{\mathbf{r}_{j}(x)-\mu_{ij}}{\sigma_{ij}}.$

Defined empirically, we can now compute the spuriosity of all images, and rank
them accordingly. Note that the ranking must be within a class, as the
spurious features one considers depends on the class. Thus, our spuriosity
rankings consist of ordering each image within its class (and data split)
based on its spuriosity. As shown in Figure 1, our ranks organize images into
groups that visually differ in spurious features. This suggests that while we
use the neural features of a specific network in computing spuriosity
empirically, the patterns we measure capture semantic concepts that pertain to
the data, and thus affect all models.

##  3 Assessing Spurious Feature Reliance

Figure 4: Model accuracy with spurious cues most/least present (i.e. accuracy
on top-10 highest/lowest spurious ranked validation images respectively).
While zero-shot CLIP models exhibit the most effective robustness and
adversarially trained models exhibit the least, most other models behave
roughly similarly. Figure 5: The effect of spurious features depends strongly
on the feature and class. Spurious features can be clarifying: it is easier to
recognize a bookcase when there are books on it. Spurious features can also be
confusing when they are relied upon (often more strongly) by other classes,
leading to worse performance when the cue is strongly present.

Using spuriosity rankings, we define spurious gap, a metric to assess the
reliance of a model on spurious features. Spurious gap is simply the
difference in accuracy over ImageNet validation images with the ten highest
and ten lowest spuriosity ranks per class. A high spurious gap means that
models perform worse when spurious features are absent, indicating reliance on
those features. We note since only $357$ classes had at least one of the five
most relevant neural features annotated as spurious, spurious gap is evaluated
over this subset of classes. Again, this reflects the data dependency of the
spurious correlation problem: for some classes, all annotated features are
observed to be core (as desired), while for others, they are all spurious.

With the spurious gap metric, we perform a large-scale evaluation of $89$
diverse pretrained models (details in appendix). Figure 4 shows accuracy on
the highest spurious ranked images ($x$-axis) vs. accuracy on the lowest
spurious ranked images ($y$-axis) over all $357$ classes. For all models,
accuracy is higher when spurious features are present, matching the prevailing
thought that the drawback of spurious feature reliance is that models
underperform when spurious correlations are broken. As in [26], we observe a
strong correlation between the two accuracy measures.

Figure 6: Spurious gap varies much more across data classes than models.

However, some models stray from this linear trend, exhibiting ‘effective
robustness’ by performing significantly better on images with spurious cues
absent than the linear trend predicts (e.g. zero-shot CLIP) or the opposite
(e.g. adversarially trained models). Prior work corroborates our findings on
the robustness (and lack thereof) of CLIP and adversarially trained models
[34, 27], thus validating our metric.

We now turn our attention to class-wise spurious gaps (i.e. we average
spurious gaps per class over all models, instead of averaging over all classes
per model). We find that spurious gap varies far more across classes than
across models (Figure 6). Specifically, the sample variance is $3.1\times$
larger for spurious gaps across classes than across models, suggesting that
the spurious correlation problem pertains more to data than to models. We
remark that averaging over classes completely obscures this rich class-wise
signal, as is standard practice; we encourage closer inspection of class-
dependencies in other problems as well.

Digging deeper into the observed class variance, we visualize classes with
extreme spurious gaps in figure 5. Usually, model performance drops when
spurious cues are removed at the test time, as is typically expected,
indicating that the spurious cues help clarify the object class (e.g. it is
easier to recognize a bookcase when books are present). However, surprisingly,
for some classes, models actually perform significantly worse when spurious
cues are present. We attribute this to confusing spurious cues, where the cue
is also relied upon in predicting other classes. For example, it may be useful
to recognize the body of a car when predicting the car wheel class. However, a
car body feature would naturally be relevant in predicting car classes, and if
that feature is strongly present, a model would more often wrongly predict a
car. In summary, the effect of relying on spurious features is nuanced and
strongly dependent on the features, the class, and how other classes also use
the feature. Given the strong class-dependence on how spurious feature affect
model performance, class-agnostic approaches to improving spurious correlation
robustness may be doomed to fail.

##  4 Closing Spurious Gaps

Previously, we focused on spuriosity rankings of validation samples. Now, we
use spuriosity rankings of the training set, so to close spurious gaps and
assess the impact of spurious cues on OOD generalization.

Setup. Given a pretrained model (ResNet-50, DeiT-small), we simply retrain its
last layer (i.e. the linear classification head) from scratch, similar to
[20], on different subsets of the training data that may or may not depend on
spuriosity rankings: bot k, the $k$ samples from each class with the lowest
spuriosity rank; mid k, the $k$ samples from each class with closest ranks to
the median spuriosity rank; top k, the $k$ samples from each class with the
highest spuriosity rank; rnd k, $k$ randomly selected samples from each class.
For evaluation on ImageNet, ImageNet-R, ImageNet-Sketch, and ObjectNet, we
evaluate only accuracy on samples corresponding to classes with at least one
annotated spurious feature.

(a) ResNet-50

(b) DeiT-small

Figure 7: The accuracy of classification heads trained on different subsets of
the training data on validation samples with varying spuriosity ranks. The
dased, horizontal lines correspond to the standard accuracy of different
classification heads. For better readability, curves are smoothed using a
sliding window of size 5 over spuriosity ranks.

Reassessing Standard Accuracy and the Toxic Side of Spuriosity. In Figure 7.
we present the accuracy of the classification heads (trained using 100
training samples per class) on validation samples with varying spuriosity
ranks. Firstly, the spurious cues in the training set actually increase the
standard accuracy as indicated by the dashed, horizontal lines, where mid 100,
top 100, rnd 100 offer higher standard accuracy than bot 100, the
classification heads learned from samples without strong spurious cues.

Does this mean spuriosity is favored? No. As in Figure 7, training on samples
with spurious cues results in degradation of accuracy on samples without
strong spurious cues (i.e. the lowly ranked validation samples). In this
sense, the top-ranked training samples with the most spuriosity can be viewed
as a form of poisoned samples, including them in training is associated with
clear drops of average performance on a particular subset, i.e. the samples
with few spurious cues. Meanwhile, the classification heads trained on only
the lowest ranked training samples offer consistent performance for validation
samples across different spuriosity ranks (i.e. flat curves in Figure 7).
While this comes at a cost of in-distribution accuracy, others may argue
reducing disparity in performance across groups may be preferred to maximizing
overall accuracy; such ideas are prevalent in the fairness community [15].

Closing Spurious Gaps and Improving Effective Robustness. In Figure 8, we
present both spurious gaps and accuracy on validation samples with few
spurious cues (i.e. the 10 lowest ranked validation samples from each class)
for classification heads learned from different subsets of data with varying
sizes. Training linear heads specifically on the lowest-ranked samples
effectively reduces spurious gaps with improved accuracy assuming few spurious
cues.

In Figure 9, we present the evaluation of these classification heads on OOD
benchmarks, including ImageNet-R, ImageNet-Sketch, and ObjectNet. Here we
adapt the notion of effective robustness[44, 26], which, given a pool of base
models, fits a line in the 2d-plane of OOD accuracy v.s. standard accuracy and
measures the robustness of each model with its vertical distance to the fitted
line. Intuitively, it measures how robust a model is compared to the linear
trend of the given pool of base models (as in section 3). We include the
linear fits respectively for ResNet-50 and DeiT-small in Figure 9 and the
average effective robustness for classification heads trained on different
types of subsets in Figure 9. Re-training the linear classification head
specifically on lowly ranked samples offers higher effective robustness than
all other baselines.

(a) ResNet-50

(b) DeiT-small

Figure 8: Spurious gaps and accuracy on validation samples with few spurious
cues (i.e. the 10 lowest ranked validation samples from each class) for
classification heads learned from different subsets of data with different
sizes.

Figure 9:  (left) Average OOD accuracy (over ImageNet-R, ImageNet-Sketch, and
ObjectNet) vs ImageNet accuracy for classification heads learned from
different subsets of data with varying sizes, along with linear fits over all
classification heads; (right) Average effective robustness by classification
head training subset.

##  5 Feature Reliability

Figure 10: Kernel density estimations of the distribution of core and spurious
feature reliability per class, over various datasets. We find that (i) core
and spurious features are generally reliable, though core features are more
reliable, (ii) both features become less reliable under distribution shift
(iii) while rare, there exist classes where only spurious features are
reliable, suggesting that removing spurious features in a data-agnostic
fashion may deteriorate robust classification for certain classes.

Typically, the perceived danger of model reliance on spurious features is that
performance drops when the spurious cues are no longer present; spurious gap
exactly measures the performance drop. However, our analysis revealed a second
danger of relying on spurious features, as they may pertain to multiple
classes, leading to model confusion when the feature is strongly present. In
fact, this vulnerability may also occur with core features that are shared
among many classes. To characterize this issue, we propose another metric
called feature reliability, which compares average activation on some feature
for samples correctly or incorrectly classified to a specific class.

Formally, given a model, we define the reliability of a neural feature $j$ to
some class $i$ as follows. Let $\mathcal{CC}(i)$ and $\mathcal{MC}(i)$ denote
samples correctly and incorrectly predicted to class $i$ respectively, so that
$\mathcal{CC}(i)\cup\mathcal{MC}(i)$ is the set of all samples predicted to
class $i$. W.l.o.g., let $\overline{\mathcal{CC}(i)_{j}}$ denote the average
activation on feature $j$ for samples in $\mathcal{CC}(i)$. We then define the
reliability of feature $j$ for class $i$ as the relative difference between
$\overline{\mathcal{CC}(i)_{j}}$ and $\overline{\mathcal{MC}(i)_{j}}$.

$\textbf{FtrRel}(i,j)=\frac{\overline{\mathcal{CC}(i)_{j}}-\overline{\mathcal{MC}(i)_{j}}}{\max(\overline{\mathcal{CC}(i)_{j}},\overline{\mathcal{MC}(i)_{j}})}$

This metric lies between $1$ and $-1$, with a negative score indicating that
misclassified samples to class $i$ activate feature $j$ more than correctly
classified samples. Thus, $\textbf{FtrRel}(i,j)<0$ indicates feature $j$ is
unreliable for class $i$, as relying upon $j$ to predict $i$ leads to mistakes
(i.e. a sample is wrongly predicted to a class because it activates a relied
upon feature more than usual, increasing the class logit value). We analyze
the reliability of neural features for a model adversarially trained on
ImageNet. All linear head weights for this model on features we analyze are
positive, so higher activation on a feature does indeed translates to a
greater contribution to the class logit. We focus on this model because we
have identified its important neural features per class and annotated their
function as core or spurious. Thus, we can use this model as a case study to
compare the reliability of spurious features to core ones. We note that our
metric is model-agnostic, and while this analysis is specific to a single
model, we argue that the findings may extend to other models, as we observe
diverse models to have strongly correlated feature sensitivities (see
appendix).

Namely, we focus on classes with both core and spurious features annotated. We
compute feature reliability for all annotated features on ImageNet, as well as
multiple OOD variants (ImageNet-R, -Sketch, and ObjectNet). We define
core/spurious feature reliability for a class as the average reliability of
all core/spurious features for that class. Figure 10 shows the joint
distribution of core and spurious feature reliability across classes. First,
we observe that both sets of features are generally reliable, though core
features are more reliable than spurious ones. Also, under distribution shift,
both feature sets become less reliable.

We now focus on classes where only one set of features is unreliable. Here, we
can think of the unreliable set of features as responsible for errors: samples
mistakenly predicted to $i$ do so because they fool one set of features,
activating them even higher than true instances of the class. In the majority
of these cases, core features are reliable while spurious ones are not.
However, surprisingly, there are classes where the opposite is true. Thus, if
one were to remove spurious dependencies entirely, performance would degrade
significantly on this subset of classes. Further, we see the fraction of
classes with only reliable spurious features can be even higher on OOD
datasets ($28.6\%$ for ImageNet-R vs. $17\%$ for ImageNet).

Figure 11: Samples with unreliable spurious but reliable core features for
ObjectNet (left), and vice versa for ImageNet-R (right).

Figure 11 shows examples of both cases: core features for lawn mower are
unreliable in ImageNet-R, where the lawn mowers are plastic toys instead of
real metal ones, but the green grass spurious cue remains reliable, enabling
correct prediction. Conversely, the keyboard feature is unreliable for the
computer mouse class in ObjectNet, leading to a false positive prediction for
a banana placed by a keyboard as a computer mouse. Recall computer mouse had a
highly negative spurious gap, reflecting that its spurious features were
confusing. Indeed, there is a direct connection between feature reliability
and spurious gap in that they both highlight how spurious cues can be
clarifying or confusing, though the former metric focuses on false positive
errors, while the latter emphasizes true negative errors.

(a) Background bias in Waterbird vs. Landbird Classification.
The core features pertain to the bird, but above, the focus is on background.

(b) Racial bias in Celeb-A Blond vs. Brown Hair Color Classification.
The core features pertain to the hair, but above, the focus is on skin.

Figure 12: Our framework can detect spurious features in unseen data. We
detect designated spurious attributes like background correlation in the
Waterbirds benchmark (left), as well as new spurious dependencies such as
racial bias in Celeb-A Hair Classification (right). Presented above are highly
activating images (top row), corresponding heatmaps (middle), and lowly
activating images (bottom) for spurious features observed for the classes
waterbird, landbird, blond hair, brown hair (left to right).

##  6 Beyond ImageNet: Lightweight Feature Discovery in New Domains

While our analysis focuses on ImageNet, we now show our methods extend beyond
it, even with minimal additional training. Our framework for finding relevant
data patterns relies on robust neural features, which have enhanced
interpretability properties. However, adversarial training can be challenging,
particularly when data is limited. We propose to simply fit a linear layer for
the new task atop fixed features from an adversarially robust feature encoder
pretrained on ImageNet; prior work shows transfer learning on robust features
is very effective [40]. We then perform the usual process of identifying
important robust neural features per class and visualizing them via inspecting
the highest-ranked and the lowest-ranked natural images, as well as heatmaps
and/or feature attacks for highly ranked images.

We demonstrate the feature discovery on two spurious correlation benchmarks:
Waterbirds and Celeb-A Hair Color classification. In Figure 12, we show some
of the discovered patterns. For waterbirds, we find neural features that focus
entirely on backgrounds; an important feature for predicting the waterbirds
class is a water background. The lowest-ranked images within a class for these
features often contain the opposite background (land instead of water), thus
revealing the minority subgroup (waterbirds on land). For Celeb-A, we observe
features that focus on faces, making them spurious for the hair classification
task. Further, race appears to be homogeneous amongst the top-ranked images,
and opposite in the bottom-ranked images. This suggests that a model may
spuriously associate brown skin with the brown hair class, likely leading to
failures for blond-haired brown-skinned people.

##  7 Related Work

Spurious correlation benchmarks often only consider a small set of predefined
spurious attributes fixed across classes [50, 39, 24]. Real-world benchmarks
usually are very specific to a narrow task [22], as opposed to our benchmark
which seeks to understand the role of spurious features in the general and
ubiquitous ImageNet dataset [12]. To study the related distributional
robustness problem for ImageNet models, it is standard to assess accuracy on
variants like ImageNet-R, -Sketch, and ObjectNet [16, 45, 3], which present
objects in altered rendition, as sketches, or in household settings (instead
of natural contexts) respectively, as we do in Section 4. Other approaches to
measuring sensitivities require synthetic data [2, 49, 13] or corrupting
spurious regions, which undesirably takes models out of distribution [28, 29].
In contrast, our benchmark operates only on natural images within the
validation set, using spuriosity to obtain subsets with spurious cues strongly
present and absent.

Natural image-based interpretability is typically used to explain inferences,
based on examples from the corpus [7, 10], or on generated examples intended
to visually reflect a traversal between concepts [14], similar to how our
rankings traverse from strong spuriosity to minority examples with spurious
cues absent, though we do not require generative models. Other related
approaches are influence functions [21] and data models [19], though they are
computationally expensive and the former can be fragile [4].

Mitigating spurious correlation dependencies often involves altering the
training objective to learn more invariant features [2, 35] or directly
optimize worst group accuracy [38, 18, 37, 52]. Many methods require at least
some extra supervision with respect to non-class-label attributes [30].
Retraining approaches either focus on correcting errors [23, 53] or using
balanced data (i.e. dominant spurious cues balanced with minority samples)
[20]; our rankings would directly assist in constructing such balanced
subsets.

Neural features have been examined to interpret models [32, 17]. Robust neural
features have also been examined and observed to have enhanced
interpretability [43, 47]. Most notably, [42] introduces a framework based on
robust neural features for discovering core and spurious data patterns with
minimal human supervision. We apply this framework to empirically compute
spuriosity ranks at scale. Key contributions of spuriosity rankings to this
framework include enabling assessment of feature dependence without relying on
corruption (which precludes comparison across models), identifying minority
subpopulations, quantifying feature reliability, and extending the framework
to new domains without requiring adversarial training.

##  8 Acknowledgements

This project was supported in part by NSF CAREER AWARD 1942230, HR001119S0026
(GARD), ONR YIP award N00014-22-1-2271, Army Grant No. W911NF2120076, the NSF
award CCF2212458, and Meta grant 23010098.

## References

  * [1] Michael A. Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh M Nguyen.  Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4840–4849, 2019.
  * [2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization, 2020.
  * [3] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Joshua B. Tenenbaum, and Boris Katz.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  In NeurIPS, 2019.
  * [4] Samyadeep Basu, Philip Pope, and Soheil Feizi.  Influence functions in deep learning are fragile.  ArXiv, abs/2006.14651, 2021.
  * [5] Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In FAT, 2018.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [7] Chaofan Chen, Oscar Li, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin.  This looks like that: deep learning for interpretable image recognition.  ArXiv, abs/1806.10574, 2019.
  * [8] Xinlei Chen*, Saining Xie*, and Kaiming He.  An empirical study of training self-supervised vision transformers.  arXiv preprint arXiv:2104.02057, 2021.
  * [9] Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big data, 5 2:153–163, 2017.
  * [10] Jonathan Crabbe, Zhaozhi Qian, Fergus Imrie, and Mihaela van der Schaar.  Explaining latent representations with a corpus of examples.  In NeurIPS, 2021.
  * [11] Alex J. DeGrave, Joseph D. Janizek, and Su-In Lee.  Ai for radiographic covid-19 detection selects shortcuts over signal.  Nature Machine Intelligence, 3(7):610–619, 2021.
  * [12] J. Deng, Wei Dong, R. Socher, L. Li, K. Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [13] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan I. Moldovan, Sylvan Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic.  On robustness and transferability of convolutional neural networks.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16453–16463, 2021.
  * [14] Asma Ghandeharioun, Been Kim, Chun-Liang Li, Brendan Jou, Brian Eoff, and Rosalind W. Picard.  Dissect: Disentangled simultaneous explanations via concept traversals.  ArXiv, abs/2105.15164, 2022.
  * [15] Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang.  Fairness without demographics in repeated loss minimization.  In ICML, 2018.
  * [16] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320–8329, 2021.
  * [17] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas.  Natural language descriptions of deep visual features.  ArXiv, abs/2201.11114, 2022.
  * [18] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama.  Does distributionally robust supervised learning give robust classifiers?  In ICML, 2018.
  * [19] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.  Datamodels: Predicting predictions from training data.  ArXiv, abs/2202.00622, 2022.
  * [20] P. Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  ArXiv, abs/2204.02937, 2022.
  * [21] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1885–1894. JMLR.org, 2017.
  * [22] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard L. Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang.  Wilds: A benchmark of in-the-wild distribution shifts.  In ICML, 2021.
  * [23] Evan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn.  Just train twice: Improving group robustness without training group information.  In ICML, 2021.
  * [24] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [25] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In ICLR, 2018.
  * [26] John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt.  Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization.  ArXiv, abs/2107.04649, 2021.
  * [27] Mazda Moayeri, Kiarash Banihashem, and Soheil Feizi.  Explicit tradeoffs between adversarial and natural distributional robustness, 2022.
  * [28] Mazda Moayeri, Phillip E. Pope, Yogesh Balaji, and Soheil Feizi.  A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19065–19075, 2022.
  * [29] Mazda Moayeri, Sahil Singla, and Soheil Feizi.  Hard imagenet: Segmentations for objects with strong spurious cues.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [30] Jun Hyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin.  Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation.  ArXiv, abs/2204.02070, 2022.
  * [31] Muzammal Naseer, Kanchana Ranasinghe, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Intriguing properties of vision transformers.  In Neural Information Processing Systems, 2021.
  * [32] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2017.  https://distill.pub/2017/feature-visualization.
  * [33] Yao Qin, Chiyuan Zhang, Ting Chen, Balaji Lakshminarayanan, Alex Beutel, and Xuezhi Wang.  Understanding and improving robustness of vision transformers through patch-based negative augmentation.  ArXiv, abs/2110.07858, 2021.
  * [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [35] Alexander Robey, George J. Pappas, and Hamed Hassani.  Model-based domain generalization.  In NeurIPS, 2021.
  * [36] Amir Rosenfeld, Richard S. Zemel, and John K. Tsotsos.  The elephant in the room.  ArXiv, abs/1808.03305, 2018.
  * [37] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  ArXiv, abs/1911.08731, 2019.
  * [38] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang.  Distributionally robust neural networks.  In International Conference on Learning Representations, 2020.
  * [39] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang.  Distributionally robust neural networks.  In International Conference on Learning Representations, 2020.
  * [40] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry.  Do adversarially robust imagenet models transfer better?  In ArXiv preprint arXiv:2007.08489, 2020.
  * [41] Rakshith Shetty, Bernt Schiele, and Mario Fritz.  Not using the car to see the sidewalk — quantifying and controlling the effects of context in classification and segmentation.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8210–8218, 2019.
  * [42] Sahil Singla and Soheil Feizi.  Salient imagenet: How to discover spurious features in deep learning?  In International Conference on Learning Representations, 2022.
  * [43] Sahil Singla, Besmira Nushi, S. Shah, Ece Kamar, and Eric Horvitz.  Understanding failures of deep networks via robust feature extraction.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12848–12857, 2021.
  * [44] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt.  Measuring robustness to natural distribution shifts in image classification.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18583–18599. Curran Associates, Inc., 2020.
  * [45] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  In Advances in Neural Information Processing Systems, pages 10506–10518, 2019.
  * [46] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [47] Eric Wong, Shibani Santurkar, and Aleksander Madry.  Leveraging sparse linear layers for debuggable deep networks.  ArXiv, abs/2105.04857, 2021.
  * [48] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt.  Robust fine-tuning of zero-shot models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7959–7971, June 2022.
  * [49] Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry.  Noise or signal: The role of image backgrounds in object recognition.  In International Conference on Learning Representations, 2021.
  * [50] Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li.  Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms.  ArXiv, abs/2106.03721, 2021.
  * [51] John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony Beardsworth Costa, Joseph J. Titano, and Eric Karl Oermann.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.  PLoS Medicine, 15, 2018.
  * [52] J. Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra.  Coping with label shift via distributionally robust optimisation.  ArXiv, abs/2010.12230, 2021.
  * [53] Michael Zhang, Nimit Sharad Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher R’e.  Correct-n-contrast: A contrastive approach for improving robustness to spurious correlations.  In ICML, 2022.

Appendix

##  Appendix A Impact and Limitations

We hope our work will allow for better understanding of how spurious feature
reliance affects deep models, leading to improved handling of algorithmic
fairness and model robustness, which are both related to the spurious
correlation problem. We open source all code, and modify a key framework of
our method to vastly increase its usability for people with fewer
computational or data resources. We also create a website for easier viewing
of our analysis, making our entire annotation process completely transparent.
We obtain verbal approval 111We were informed that formal approval is not
needed because we do not obtain information about our crowd worker, and thus
our study does not constitute human-subject research. from our institution’s
IRB for our study, and pay workers above minimum wage. Key limitations are
that our method does not detect all spurious features (only a subset), and
uses a single model to detect and measure them, though we argue that the
features we detect are very relevant and likely relied upon by all models; we
elaborate on this point in Section F.2.

##  Appendix B Characterizing Feature Sensitivities

Using visualizations from our robust neural feature annotation, we obtain soft
segmentations for features in input space that activate a specific neural
feature. Namely, the Neural Activation Map for a neural feature can serve as a
soft segmentation for the feature. Using a Mechanical Turk human study, we
validate that neural activation maps for images in the top $20^{th}$
percentile within its class for activation of the neural feature segment the
same input pattern.

(a) Gray (Color)

(b) Blur (Texture)

(c) Patch Rotate (Shape)

Figure 13: Examples of features that are particularly sensitive to color,
texture, and shape corruptions. Original images (top) and corrupted images
(bottom) shown, along with model predictions. In one case, corrupting the
spurious feature of keyboard on desk for class monitor actually improves
prediction. Figure 14: Sensitivities of each class-feature pair to specific
corruptions for ResNet-50 and DeiT-Small. The two distinct models rely on
features in the same way.

We then use targeted corruptions to gain insight on the information within an
input region that models rely on most. The core premise of our argument is
that degradation in model performance due to corruption of some information
can be a proxy of the importance of the corrupted information. Indeed, similar
arguments were made in [42, 28], where adding Gaussian noise to specific
regions was employed to assess core/spurious or foreground/background
sensitivity respectively. However, in our analysis, we use carefully chosen
corruptions so to remove one aspect of primitive information (i.e. color,
shape, or texture) without disrupting the others. Namely, our corruptions are
graying out image regions, blurring out image regions, or patching an image
region and randomly rotating each patch by a multiple of $90$ degrees. Graying
out an image removes color, blurring out destroys local (i.e. textural)
details, and random patch rotation breaks longer edges, hence disrupting shape
information.

We apply these targeted corruptions to segmented feature regions for all
$5000$ annotated class-feature pairs. Specifically, we focus on the $65$
images per class where a feature is activated most highly (i.e. roughly top
$5^{th}$ percentile), since the feature is most prominent in these images.
Also, for spurious features, we include a filtering step to avoid corrupting
core regions. We do this by removing any overlap of the spurious feature
segmentation with the consolidated core mask [42], which is a pixel-wise
maximum of soft segmentations for any relevant core features. We track model
accuracy on samples with and without each corruption applied, using change in
accuracy as a measure of sensitivity.

Figure 15: Images with highest and lowest spuriosity for classes with the
lowest spurious gaps (aside from the two shown in Figure 5). The spurious
dependencies for these classes are confusing because when the spurious
features are prominent, models are less accurate. These features include
blurry background often with spider in foreground, plates or containers, and
computer equipment for classes spider web, meatloaf, and desk respectively.

Figure 13 shows examples of features that are the most sensitive (i.e. highest
change in accuracy) to each of the three corruption types, visualizing the
corruption and its affect on model prediction. In the left panel, we see that
the color information in the core feature of lizard body is crucial for
successful prediction of the class European Green Lizard (matching intuition).
In the middle panel, we see a case where the spurious feature of keyboard on
desk hurts classification. Namely, blurring out the keyboard and desk leads to
more accurate prediction of the desktop computer. We note that while the shape
and color information is retained, it appears that the local details in the
texture of the keyboard and desk contribute more to confusing the model and
leading to misclassification. Lastly, we revisit the books on a bookcase
spurious feature, observed to be clarifying (i.e. models classify bookcases
much more accurately when the spurious feature is present). Here, we observe
that the shape of the books is crucial, as rotating patches leads to
prediction of tobacco shop.

Figure 16: Images with highest and lowest spuriosity for classes with the
lowest spurious gaps (aside from the two shown in Figure 5). The spurious
dependencies for these classes are clarifying because when the spurious
features are prominent, models are more accurate. These features include white
collar with tie, sand, and aerial viewpoint, often with feet for classes suit,
fiddler crab, and sandal respectively.

We conduct this analysis using two pretrained models of roughly equal size:
ResNet50 and small DeiT. Figure 14 shows the relative accuracy drops incurred
by performing each of the three corruptions for all $5000$ annotated class-
feature pairs on both models. We find that the two models have strongly
correlated sensitivities ($r=0.724$). The least correlated sensitivities
appear to be for the patch-rotate corruption, where the transformer has nearly
zero accuracy drop for many cases where the ResNet experiences greater
accuracy drop. This is most likely due to the unique robustness of vision
transformers to various patch transformations like permutation, rotation, and
ablation [31, 33]. Despite this, the strong correlation between feature
sensitivities for two diverse models suggests that model behavior is
determined much more by the data it operates over than the specific decisions
made during training. In other words, the features that models rely on and the
ways in which they rely on them may be far more a function of data than a
function of the model. If this conjecture holds, it would warrant greater
inspection of the role of data in various open problems in modern deep
learning (robust generalization, efficient learning, etc).

##  Appendix C More Examples of Spuriosity Rankings

(a) Training Split

(b) Validation Split

Figure 17: Images with highest and lowest spuriosity for five randomly
selected classes (top/bottom row in each chunk corresponds to images with
highest/lowest spuriosity respectively). The spurious features depicted are
lake water, branches, water, sky with clouds or lava, and plants. The spurious
features are easier to see in the training split than in the validation split,
because there are roughly $25\times$ more samples to choose from. Nonetheless,
the validation images still seem to be organized so that the top-10 images
contain the spurious feature much more than the bottom-10, indicating that our
spuriosity ranks generalize to the validation split.

To further validate our spuriosity rankings, we now present more examples of
images ranked by spuriosity. While this validation is qualitative, we argue
that it is sufficient since we are attempting to proxy a fundamentally
qualitative notion.

Namely, we show the top ten and bottom ten images by spuriosity ranking.
First, we show more examples of the most confusing spurious dependencies (i.e.
negative spurious gap; figure 15) and the most clarifying spurious
dependencies (i.e. positive spurious gap; figure 16). Then, we also present
images organized by spuriosity ranking for five randomly selected classes.
Here, we show images both ranked in the training and validation split to
demonstrate that even though trends are clearer when ranking training images
(because there are $\sim 25\times$ more of them to choose from), the
underlying semantic concepts by which images are ranked still generalize to
the validation split.

##  Appendix D Full Pretrained Model Evaluation Results

Figure 18: Effective robustness for all $89$ models considered (top), along
with effective robustness averaged over model category (bottom).

We now provide greater detail for the pretrained model evaluation presented in
Section 3. We evaluate $89$ models spanning diverse architectures and training
procedures. Namely, the models we consider fall into the following categories,
with category nicknames in parenthesis: supervised vision transformers (Sup
ViTs), self-supervised vision transformers (SS ViTS), CLIP vision transformers
(CLIP ViTs, both Zero-Shot and Finetuned), CLIP ResNets (CLIP RNs), self- or
semi-supervised ResNets (SS ResNets), supervised ResNets (Sup ResNets), mixer
and ResMLP models (MLP-Based), various other convolutional networks (Other
CNNs), and adversarially trained ResNets (Robust ResNets). The vast of
majority of pretrained weights are obtained from the timm library [46], while
others come directly from their original respective repositories [6, 8, 34,
40]. Figure 18 shows effective robustness for each model, as well as the
average effective robustness per category.

We measure effective robustness by first fitting a line to predict accuracy on
the $10$ validation images per class with the least spuriosity as a function
of accuracy on the $10$ validation images per class with the most spuriosity.
Let $\text{acc}_{\text{top}}$ and $\text{acc}_{\text{bot}}$ denote accuracy on
images with the most and least spuriosity respectively (i.e. top and bottom
with respect to spuriosity rankings). We refer to the line of best fit as
$\beta:\mathbb{R}\rightarrow\mathbb{R}$. Effective robustness for a model with
accuracies on top and bottom spuriosity ranked images
$\text{acc}_{\text{top}},\text{acc}_{\text{bot}}$ respectively is simply
$\text{acc}_{\text{bot}}-\beta(\text{acc}_{\text{top}})$. In other words, we
measure how high above the line of best fit a model’s marker falls in figure
4.

We note that this differs slightly from the originally presented notion of
effective robustness in [26], where a log-linear fit is performed. Moreover,
instead of predicting out-of-distribution accuracy based on in-distribution
accuracy, we compare accuracy on two carefully chosen subsets of in-
distribution data. We argue that we capture subsets where subpopulation shift
has occurred, specifically with respect to common spurious cues for each
class.

Figure 19: Correlation of per-class spurious gaps for all pairs of models.
Numbers shown are Pearson’s $r$ correlation coefficients times $100$. On
average, a pair of diverse models have strongly correlated class-wise spurious
gaps, with $r=0.69$. Notably, zero-shot CLIP models have the worst correlation
with other models, suggesting that their perception is fundamentally
different.

Our findings show that some models are more effectively robust than others.
Namely, zero-shot CLIP models have much higher accuracy on images with lowest
spuriosity than their performance on the images with highest spuriosity would
predict. After finetuning a linear head using ImageNet on top of the fixed
CLIP image encoder, we see effective robustness drops significantly. We note
that both these results were observed using independent distributional
robustness measures in [48]. At the other extreme, adversarially trained
ResNets have the lowest effective robustness, a result carefully studied in
[27].

Aside from these two extremes, most trends are muted. We believe the stronger
signal exists in per-class spurious gaps, as shown in figure 6, suggesting
that while models generally behave similarly on the same inputs, their
behavior (i.e. with respect to spurious feature dependence) varies
dramatically across inputs. Indeed, we find that per-class spurious gaps
correlate strongly between pairs of models in our evaluation suite. Figure 19
visualizes these correlations, with an average correlation of $r=0.69$ being
observed. We thus recommend closer attention to be paid to the individual data
classes and potential spurious features per class a model is to be deployed
over.

##  Appendix E Additional Discovered Features for Waterbirds and Celeb-A

(a) Waterbird core feature. Focus is on elongated and curved necks.

(b) Landbird core feature. Focus is on feathers with bright colors.

Figure 20: Core features discovered for the Waterbirds waterbird vs. landbird
classification task. We identify features that occur in specific
subpopulations of the broader classes. Low activating images reveal
subpopulations where the feature is absent. The rows show (from top to
bottom): highly activating images, corresponding heatmaps and feature attacks,
and lowly activating images.

(a) Focus is on blond curly hair.

(b) Focus is on blond straight hair.

(c) Focus is on brown curly hair.

(d) Focus is on wavy brown hair.

Figure 21: Core features discovered for the Celeb-A blond vs. brown hair
classification task. We identify features corresponding to different types of
hair within the same broader class of hair color. The rows show (from top to
bottom): highly activating images, corresponding heatmaps and feature attacks,
and lowly activating images.

We now present additional features identified via our lightweight extension of
the feature discovery framework of [42]. Specifically, we show core features,
which are relied upon more frequently than spurious features, which we show in
Figure 12. We observe robust neural features to respond to specific patterns
shared among subpopulations within the broader classes. Surprisingly, these
robust neural features were not at all trained on images from the downstream
tasks. Yet, the patterns they respond to are semantically meaningful for the
downstream task. We attribute this to the diversity in ImageNet, which leads
to learning many general visual patterns that are manifested in more specific
contexts for a vast number of downstream tasks.

##  Appendix F Robust Neural Feature Annotation Details

Our work uses the feature discovery framework introduced in [42], also known
as Salient ImageNet. We now i. review the framework, ii. discuss its merits
and potential limitations, and iii. detail our expansion/modifications to it.

###  F.1 Salient ImageNet Framework

The steps of the Salient ImageNet Framework are:

  1. 1.

Adversarially train a model for your classification task.

  2. 2.

Identify important neural features per-class based on feature importance,
which is simply the product of the average feature activation and the weight
of linear head connecting feature to class logit. Feature importance
explicitly computes the average contribution of a feature to a class logit for
samples in the class.

  3. 3.

Select the top-$k$ (i.e. $k=5$) activating instances of a class for a selected
feature and additionally generate heatmaps (via overlaying the Neural
Activation Map on the original image) and feature attacks (via adversarially
attacking the original image using gradients from the adversarially robust
model so to maximize activation on the feature).

  4. 4.

Have humans inspect the $3\times 5=15$ visualizations to determine if the
focus of the neural feature is on the main object (i.e. pertaining to the
class and hence core) or on the background or a separate object (hence making
the feature spurious).

In [42], the annotated robust neural features were primarily used to softly
segment image regions containing core and spurious features at scale. In
addition to the Mechanical Turk study performed to annotate neural features, a
second study was conducted to confirm that neural activation maps for
annotated features softly segment the same input features across the top
$5^{th}$ percentile of samples within the relevant class. With this
confirmation, they computed neural activation maps for all of these images and
used these as feature segmentation maps. To assess reliance on core and
spurious features, they would inspect drop in accuracy due to adding small
amounts of Gaussian noise to these regions.

###  F.2 On Feature Discovery Using Only One Model

The key idea of the Salient ImageNet framework was to use neural features of
an adversarially trained network as automatic ‘concept’ detectors, and further
to use their activation maps as soft segmentations. One may argue then that
the features discovered are biased to the underlying model used. However, we
contend that the primary patterns that models rely on are generally the same.
Note that the existence of patterns predictive of a class is determined by the
data distribution and not the model. While it is true that models may learn to
rely on different patterns and may also vary the degree to which they rely on
the same patterns, we contest that the most predictive patterns will be
detected and relied upon by nearly all models. In our own experiments, we find
that the class-wise spurious gaps are highly correlated for any two pairs of
models, with average correlation of $r^{2}=0.69$ (see figure 19). We also
observe that the accuracy drops experienced due to targeted corruption of
input regions where certain features reside for a ResNet and a vision
transformer are highly correlated too ($r^{2}=0.724$) (see figure 14). These
results suggest that models are sensitive to the same data patterns and even
rely on the different pieces of information (i.e. color, shape, texture)
captured in the data patterns in the same ways.

To be perfectly clear, we do not claim to have captured all spurious (or core)
dependencies, mainly because we only inspect the $5$ most important robust
neural features per class. We acknowledge that another reason for certain
dependencies to be overlooked is because we use a single model to detect and
measure the presence of concepts. However, this only occurs for patterns that
other models are sensitive to but the adversarially trained model is not. The
main patterns that fit this description are adversarial ones, which contain no
semantic meaning and thus arguably should be overlooked for our purposes. In
other work, adversarialy trained models were found to have increased reliance
on spurious features [27]; this result was also corroborated in our spurious
gap analysis. Thus, if anything, it seems like using an adversarially trained
model may overlook core features, favoring spurious ones. Seeing as the goal
of our analysis is to detect and assess spurious feature dependencies, we
believe using robust neural features from an adversarially trained model is
warranted, especially given the enhanced interpretability of this model, which
allows for reliable heatmaps and feature attacks.

###  F.3 Our Contributions

We now detail the novel contributions we make to the Salient ImageNet
framework and analysis from [42]. The most crucial contribution is the notion
that to interpret and improve spurious correlation robustness, one can seek to
simply reorganize the data they already have, as opposed to collecting more
samples or additional instance-wise supervision (i.e. segmentations, manual
annotations beyond class-label) that would require fundamental changes the
training strategy. Using spuriosity rankings, we gain new and surprising
insight into the nuanced ways that models rely on spurious features. Namely,
we find that contrary to common belief, models can perform worse when spurious
features are present than when they are absent (i.e. because these spurious
features are depended on for multiple classes and thus are confusing). Our
analysis on training new heads on curated subsets of data (based on spuriosity
rankings) shed insight on how valuable samples are with respect to accuracy
across distribution shifts, highlighting that spurious feature reliance does
not to add to (and can even hurt) OOD accuracy, while also causing disparate
performance within the training distribution, raising fairness concerns.

Our second analysis based on our new feature reliability metric also yields
curious and complementary findings. We note that this metric can extend to any
model; we choose to use it on the adversarially trained model we annotated
because we can then compare spurious and core neural class-feature
dependencies. Here, we can quantitatively answer questions like ‘are core
features more reliable than spurious ones?’ (yes), ‘are they always more
reliable?’ (no, surprisingly), and ‘how does reliability change over
distribution shift?’ (varies by class).

Finally, we make a crucial adjustment to the feature discovery framework of
[42] that removes its most costly and at times prohibitive step (adversarial
training). Observing that the framework can be applied by simply fitting a
linear layer atop a frozen adversarially robust feature encoder vastly
increases the usability of this framework, cutting the time and data required
to perform feature discovery by huge margins.

Additionally, we believe our rehaul of evaluating spurious feature reliance
corrects crucial shortcomings of the evaluation framework of [42]. First, the
previous framework required taking images out of distribution by corrupting
core and spurious regions. The choice and strength of corruption introduced
hyperparameters that we observed the final metric to be undesirably sensitive
to. Further, because diverse models have varying general robustness to the
corruption applied, the previous evaluation procedure precluded comparisons
across models. In contrast, our method focuses on finding subsets of natural
images with vastly different degrees of spurious cue presence. Since we only
look at accuracy over in-distribution images, comparison across models is
completely appropriate. Moreover, we account for the strong correlation
between accuracy and distributional robustness by using the notion of
effective robustness adapted from [26] to control for the varying accuracies
of the models we consider. Further, our method evaluates models on the same
sets of data, while the previous method did not, instead using images where a
given feature was more present. Finally, we observe that segmentations for
spurious features (used to measure sensitivity to those features via
inspecting drop in accuracy due to noising the region) would often include
core features, thus conflating the sensitivity of core and spurious features
in their metric. Nonetheless, the work of [42] is absolutely instrumental in
enabling our analysis, though our methods and findings are markedly different
and more extensive.

Other contributions that are less novel but still impactful include massively
expanding the scale of the feature annotation analysis and creating a web
interface to view the discovered dependencies. Specifically, we annotate over
$4\times$ more classes, yielding interpretation of feature dependencies for
all $1000$ classes in ImageNet. Given that this benchmark is ubiquitous, we
believe a deep dissection of what one model perceives as important patterns in
ImageNet can be extremely informative in better understanding how machine
perception differs from human perception. We also perform the MTurk study to
demonstrate the generalizability of our feature annotations for the top
$20^{th}$ percentile of images based on feature activation as opposed to the
top $5^{th}$ percentile, leading to reliable feature soft segmentations for
many more images. Lastly, we create a website222We do not include the link to
the website to maintain anonymity. Nonetheless, the website is already
complete., which, while simply a user-interface, will greatly increase ease of
viewing and transparency into our analysis, as we show all the visualizations
used to perform our core/spurious neural feature annotations. We present
screencaps of pages from the website in Figure 22 and Figure 23. As a web-
interface, our work is now accessible to people who do not code or lack the
resources to generate our visualizations on their own devices.

Figure 22: Example of a landing page for one class-feature dependency. We
present all visualizations shown to MTurk workers to annotate the feature as
cor or spurious, as well as the annotations and explanations given by the
MTurk workers. Further, we provide links to pages for other features relevant
to the class, and other classes that also rely on the feature. Figure 23:
Screencap of a main table from our web-interface for accessing all $5000$
class-feature dependencies. One can simply search for a class by its name or
ImageNet class index to see the five features deemed most important for (i.e.
contributing the most to the logit of) the class. Each image then links to an
individual landing page designated for the class-feature pair (see Figure 22
for an example).