# Overcoming Simplicity Bias in Deep Networks using a Feature Sieve

Rishabh Tiwari  Pradeep Shenoy

###### Abstract

Simplicity bias is the concerning tendency of deep networks to over-depend on
simple, weakly predictive features, to the exclusion of stronger, more complex
features. This is exacerbated in real-world applications by limited training
data and spurious feature-label correlations, leading to biased, incorrect
predictions. We propose a direct, interventional method for addressing
simplicity bias in DNNs, which we call the feature sieve. We aim to
automatically identify and suppress easily-computable spurious features in
lower layers of the network, thereby allowing the higher network levels to
extract and utilize richer, more meaningful representations. We provide
concrete evidence of this differential suppression & enhancement of relevant
features on both controlled datasets and real-world images, and report
substantial gains on many real-world debiasing benchmarks (11.4% relative gain
on Imagenet-A; 3.2% on BAR, etc). Crucially, we do not depend on prior
knowledge of spurious attributes or features, and in fact outperform many
baselines that explicitly incorporate such information. We believe that our
feature sieve work opens up exciting new research directions in automated
adversarial feature extraction and representation learning for deep networks.

Machine Learning, ICML

##  1 Introduction

Deep networks are known to be vulnerable to a number of failure modes; in
particular, simplicity bias is the tendency of DNNs to prioritize weak
predictive features over stronger, more difficult-to-extract features (Shah et
al., 2020). This bias has been studied analytically (Pezeshki et al., 2021) as
well as empirically using natural images (texture bias (Geirhos et al., 2018))
and carefully controlled synthetic datasets (Hermann & Lampinen, 2020) that
independently manipulate feature complexity and predictive power. Such
learning biases have significant real-world consequences too, resulting for
instance in biased decision-making in AI-assisted workflows for face
recognition, healthcare, credit rating, etc. Figure 1 illustrates the idea
behind simplicity bias, and some real-world consequences. As a result, much
recent work aims to debias neural network models via a variety of approaches
to achieve more equitable outcomes (Mehrabi et al., 2021; Zafar et al., 2017;
Dwork et al., 2012; Russell et al., 2017).

Previous approaches towards debiasing DNNs include data manipulation via
augmentation & adversarial training (Duboudin et al., 2022; Niu et al., 2022),
data reweighting (Nam et al., 2020), multiple training environments (Arjovsky
et al., 2019; Zhou et al., 2022), robust learning (Pezeshki et al., 2021), and
fairness objectives (Li et al., 2022). Other researchers have proposed
diversity-enhanced ensembles (Kim et al., 2022; Teney et al., 2022; Niu et
al., 2022) and architecture optimization (Bai et al., 2021b).

Figure 1: Simplicity bias and spurious features. a) DNNs focus on color to the
exclusion of shape when both are predictive. b) Image misclassified as
elephant due to overdependence on texture features (adapted from (Geirhos et
al., 2018)). c) Classifiers mislabel blond-haired male faces as female.

Figure 2: SiFeR workflow and results. a) We use an auxiliary network to
alternately identify predictive features and erase them only at lower network
layers. By positioning the auxiliary network at different depths, we control
the complexity of erased features. See Section 3 for details. b) Our approach
successfully suppresses digit and enhances CIFAR decodability at higher layers
for CIFAR_MNIST dataset. c) We show significant gains over other approaches on
many real-world debiasing benchmarks.

We propose a novel, direct approach towards addressing simplicity bias in
neural networks: an adversarial learning challenge that forces the network to
learn sophisticated feature representations. We refer to this learning
challenge as a feature sieve, and enforce it through the use of an auxiliary
network (Figure 2a;b). Our primary intuition is that simple features are
computable early in the neural network, and proliferate throughout the deeper
layers, thereby hindering the learning of complex features. We therefore
propose to use the auxiliary network to alternately predict labels using
available features at some intermediate level (i.e., identify simple
predictive features), and erase those features from the early layers of the
network, using a “forgetting loss” (see Section 3 for details). Critically,
our proposal does not depend on any specific definition or complexity class of
“simple features”, and instead automatically adapts to data characteristics
using generalization error estimates.

We explicate our approach and its inner workings using experiments on
controlled datasets (CMNIST, CIFAR_MINST), and demonstrate its practical value
on real-world debiasing benchmarks including BAR (Nam et al., 2020), CelebA
(Liu et al., 2018), NICO (He et al., 2021), ImageNet-9 (Xiao et al., 2020) and
ImageNet-A (Hendrycks et al., 2021); in nearly all experiments we show
substantial gains over other competitive approaches. Figure 2c provides a
quick visual summary of our findings.

Summing up, we propose SiFeR: Sieving Features for Robust learning, a novel
approach towards mitigating simplicity bias, thereby debiasing neural networks
from spurious correlations in data. Our contributions are listed below.

  * •

We propose and formalize the idea of a feature sieve for mitigating simplicity
bias, and provide an automated learning recipe to control feature complexity
based on validation set.

  * •

We show, using controlled datasets, the effectiveness of our approach in
enhancing the decodability of complex features. We also demonstrate the
customizability of our approach–our work is not restricted only to suppressing
“simple” features, but is more broadly a controllable feature tradeoff tool.

  * •

We show significant gains in debiasing classifiers on real-world datasets:
3.2%, 4%, 11.1% relative gains over baselines on BAR, ImageNet-9, ImageNet-A
(Figure 2c). Crucially, we do not use foreknowledge of biased features / input
dimensions in obtaining these results, unlike many of the baselines we
outperform.

  * •

Finally, we show using feature importance visualizations that SiFeR is able to
correctly identify important visual features of a scene, while suppressing
irrelevant but spuriously-label-correlated background features (Figure 4);
this underscores the relevance of SiFeR to real-world feature understanding.

We hope that our work with SiFeR111Code available at
https://github.com/google-research/google-research/sifer encourages further
work in designing interesting computational barriers for neural networks; by
automating the extraction and combination of diverse features ordered by
complexity and predictive power, we could make significant progress towards
the debiasing of machine learning models.

##  2 Related Work

###  2.1 Simplicity Bias

Shah et al. (2020) showed that neural networks trained with SGD are biased to
learn the simplest predictive features in the data while ignoring others.
Numerous studies have attempted to investigate the correlation and impact of
such shortcuts, yielding a wealth of intriguing findings (Nagarajan et al.,
2020; Hermann & Lampinen, 2020).

###  2.2 Debiasing Spurious Correlations

Unlike our work, the majority of previous work on mitigating simplicity bias
uses explicit biased-attribute labels (Kim et al., 2019; Li & Vasconcelos,
2019; Sagawa et al., 2019; Teney et al., 2020; Krueger et al., 2021; Bai et
al., 2021a) in their debiasing recipes. This reduces their practicality since
both identifying, and manually labeling biased instances and dimensions in
real-life data are significant barriers. Only recently, the focus has shifted
towards debiasing without using explicit attribute labels (Teney et al., 2022;
Kim et al., 2022; Niu et al., 2022; Shrestha et al., 2022; Nam et al., 2020).
Here we discuss different technical approaches used by previous work in both
of the above directions:

Alternate Networks: LfF (Nam et al., 2020) and LWBC (Kim et al., 2022)
initially train a prejudiced network and try to debias the second network by
focusing on samples that go against the bias.
Ensemble: LWBC (Kim et al., 2022) and ESB (Teney et al., 2022) both create a
classifier ensemble; the former enforces debiasing via reweighting of training
instances, and the latter incorporates a diversity constraint in the ensemble.
Architecture Design: NAS-OoD (Bai et al., 2021b) adds an OOD generalization
criterion to network architecture search training to select inherently more
robust network architectures. OccamNet (Shrestha et al., 2022) adds a few
inductive biases in the network–for instance, explaining the dataset with
simple hypotheses and bounded network depth, and applying spatial localization
assumptions about unbiased (visual) features in order to filter spurious
features.
Multiple Environments: IRM (Arjovsky et al., 2019) uses the theory of causal
bayesian networks to find an invariant feature representation using multiple
training environments with different bias correlations. REx (Krueger et al.,
2021) tries to improve on the worst linear combinations of risks from
different training environments. CaaM (Wang et al., 2021) learns causal
attention by partitioning the data on-the-go to break correlation with bias.
Augmentations: DecAug (Bai et al., 2021a) proposed a semantic augmentation and
feature decomposition approach to disentangle context features from category
related features. Niu et al. (2022) adds adversarial augmentations to the
image while training to avoid over-reliance on spurious visual cues. This work
is conceptually closest to our work, in that it builds an ensemble where
previous components compete with a new classifier to encourage it to learn
diverse hypotheses. Our approach directly addresses the competitive
development of features within a network (the “heart” of the simplicity bias
challenge); we also outperform them on the BAR dataset (Nam et al., 2020)
(Table 3), while being more computationally parsimonious.

##  3 SiFeR: a Feature Sieve for Bias Mitigation

###  3.1 Preliminaries & Intuition

We start from the assumption that simple features are (by definition) quickly
learned, available early in the neural network stack (i.e., in layers closer
to the input), and more easily proliferate throughout the subsequent layers
(see e.g., Hermann & Lampinen (2020) for substantial supportive evidence for
these assumptions). Further, the ubiquitous presence of simple features
actively prevents acquisition of more complex hypotheses by subsequent NN
layers, due to the so-called simplicity bias inherent in NN training
methods–see e.g., Shah et al. (2020); Pezeshki et al. (2021) for theoretical
results supporting these claims.

Thus, our primary goal is to identify and actively suppress simple / spurious
predictive features, so as to create room for the learning of complex
predictive features at higher layers of the NN–an approach we refer to as a
“feature sieve”.

We include another key consideration in the design of our approach: do not
leverage any a priori information of simple features, or even the function
class / degree of complexity of simple features. To support this design goal,
we a) build into our design the knobs that control tradeoffs between simpler-
and more complex-to-compute features, and b) focus on reducing generalization
error as the objective in setting these knobs. This allows us to not only
automatically discover useful tradeoffs, but also to ensure that our trained
classifiers are overall more accurate than standard baselines.

As a final remark, we note that the distinctions between simple / complex,
spurious / accurate, early-layer / late-layer, and early-acquisition / late-
acquisition are likely substantially more nuanced than a simple one-to-one
correspondence, even though they are often used interchangeably for ease of
exposition. For instance, depending on the dataset, a “simple” feature may in
fact be the best / most unbiased predictive feature. For this reason, too,
depending upon generalization error for controlling the feature sieve is
strongly preferred to the use of any stronger inductive bias along the
dimensions mentioned above.

###  3.2 The Alternating Identify-and-Erase Workflow

Figure 2(a) provides an overview of SiFeR. Briefly, we use an auxiliary
network, working at an intermediate level of representation in the neural
network, to identify predictive features (simple / spurious) in the
representation, and subsequently to erase them at the lower layers of the
primary network. This is a direct operationalization of our primary goal
stated above.

Identifying simple features: The training of the primary network proceeds in
conventional fashion via forward- and back-propagation (Figure 2(a), left
panel, black & blue arrows respectively), with an additional auxiliary layer
that learns to predict the label from an intermediate representation. Note
that feedback from the auxiliary layer does not back-propogate to the main
network. This is a conscious decision choice to force the auxiliary layer to
learn from already-available features rather than create or reinforce them in
the main network. By controlling the auxiliary network’s capacity and the
layer of the primary network to which it is attached, we can control the
complexity of the predictive features it can identify.

Applying the feature sieve: We aim to erase the identified features in the
early layers of the neural network, by the combination of the following steps:
a) The parameters of auxiliary layer ($\mathcal{A}$) are frozen, and only that
portion of the main network ($\mathcal{M}_{d}$) which is before the auxiliary
layer is kept trainable– this is the region where we wish to “forget” the
simple features, and b) We apply a forgetting loss ($\mathcal{L}_{f}$) at the
output layer of the auxiliary network.

$\displaystyle\hat{\mathbf{y}}_{aux}$ $\displaystyle=$
$\displaystyle\mathcal{A}(\mathcal{M}_{d}(\mathbf{x}))$ (1)
$\displaystyle\mathbf{y}_{ep}$ $\displaystyle=$
$\displaystyle[\frac{1}{n},\frac{1}{n},...]\ \ \textit{(n\ entries)}$ (2)
$\displaystyle\mathcal{L}_{f}$ $\displaystyle=$
$\displaystyle\texttt{CE}(\hat{\mathbf{y}}_{aux},\mathbf{y}_{ep})$ (3)

where $\mathbf{x}$, $\mathbf{y}_{ep}$, $\hat{\mathbf{y}}_{aux}$ and $n$
represent input images, a pseudo-label with uniform probability across
classes, the prediction from auxiliary layer, and number of classes
respectively.

Iterative optimization: A challenge is that this process of identification and
sieving is dynamic in nature; in particular, the two steps may interfere with
each other. In order to handle this challenge, we interleave the two steps
such that each forgetting step happens after regular intervals of some
minibatch iterations ($\mathcal{F}$) which is treated as a hyperparameter
selected using the validation set.

The entire learning recipe is summed up in Algorithm 1

Input : Pretrained Model Weights $\mathbf{W}$;

training data $\mathcal{D}$; training iters $N$

Hparams : Aux Depth $\mathcal{A}_{D}$; Aux Position $\mathcal{A}_{P}$;

main_lr_weight $\alpha_{1}$; aux_lr_weight $\alpha_{2}$;

aux_forget_weight $\alpha_{3}$; forget_after_iters $\mathcal{F}$

Output : robust model weights $\mathbf{W}$

for  _$k=1\ldots N$_ do

$(\mathbf{x},\mathbf{y})\leftarrow\texttt{sample}(\mathcal{D})$

$\hat{\mathbf{y}},\hat{\mathbf{y}}_{aux}\leftarrow\texttt{Forward\\_with\\_aux}(\mathbf{x},\mathcal{A}_{D},\mathcal{A}_{P},\mathbf{W})$

$\mathcal{L}_{1}\leftarrow\Call{CE}{\hat{\mathbf{y}},\mathbf{y}}$

$\mathcal{L}_{2}\leftarrow\Call{CE}{\hat{\mathbf{y}}_{aux},\mathbf{y}}$

$\mathcal{L}_{f}\leftarrow\Call{CE}{\hat{\mathbf{y}}_{aux},\pazocal{U}}$

$\mathcal{L}\leftarrow\alpha_{1}\mathcal{L}_{1}+\alpha_{2}\mathcal{L}_{2}$

if  _k % $\mathcal{F}$ == 0_ then

$\mathcal{L}\leftarrow\mathcal{L}+\alpha_{3}\mathcal{L}_{f}$

end if

$\nabla{\mathbf{W}}\leftarrow\texttt{Backward}(\mathcal{L})$

$\mathbf{W}\leftarrow\texttt{OptimizeStep}(\nabla\mathbf{W})$

end for

Algorithm 1 SiFeR: Mitigating simplicity bias

###  3.3 Controllability of the Feature Sieve

As remarked earlier, we aim to automatically discover notions of and tradeoffs
between so-called simple and complex features, as relevant for the specific
dataset at hand. The feature sieve approach described here allows for many
mechanisms to control this discovery & tradeoff. The primary parameters are
the position & depth of the auxiliary network
($\mathcal{A}_{P},\mathcal{A}_{D}$) which implicitly control the function
complexity of the features available for discovery by the auxiliary network;
and the auxiliary forgetting weight $\alpha_{3}$, which controls the degree to
which the discovered features are suppressed. The interleaving of the feature
identifying & feature sieving steps is controlled by the parameter
$\mathcal{F}$–again, based on the specific dataset and the nature of the
features contained, this controls the dynamics of the training procedure.

Finally, we set these hyperparameters based on the goal of minimizing
validation error–this ensures not only that the parameters are chosen using
unbiased estimates of generalization, but also that at a minimum, we perform
better than the standard training baseline (which, as the trivial solution of
not-forgetting, is included in the search space for the feature sieve).

##  4 Experiment Setup

###  4.1 Datasets for Studying Simplicity Bias

CMNIST: Colored-MNIST is a 2-class synthetic dataset used to study simplicity
bias. We use digits $0$ & $1$ respectively from the MNIST dataset, with an
added color channel (red for $0$ images, green for $1$).

CIFAR-MNIST: Similar to CMNIST, this binary classification dataset has paired-
composite images–Class $0$ pairing MNIST $0$s with CIFAR automobiles, and
Class $1$ pairing MNIST $1$s with CIFAR truck images.

Both datasets contain perfectly predictive simple and complex features; by
training a classifier and manipulating the test set to break one of these
correlations, one can examine which features are being used by the trained
classifier.

###  4.2 Real-World Debiasing Benchmarks

BAR: Biased Activity Recognition (Nam et al., 2020) is a real-world image
benchmark for classifying human actions (images) into 6 classes; each training
image contains spurious correlations with background features (e.g., rocks
with climbing). The test set contains the same set of actions but with
different backgrounds (e.g., ice with climbing). The training data has no
bias-conflicting examples (i.e., examples which violate the spurious
correlation), which makes this a challenging benchmark.

CelebA:  CelebA (Liu et al., 2018) contains human faces, each labelled with 40
attributes. Following Kim et al. (2022), we focus on predicting HairColor, an
attribute heavily correlated with Gender in the dataset. Specifically, most
CelebA images with blond-hair (more than 99% of them in the training set) are
women.

NICO: NICO (He et al., 2021) is a real-world benchmark for out-of-distribution
robustness. Following Bai et al. (2021a), we used its Animal subset containing
10 object classes and 10 context labels. The training set only contains 7
contexts for each object class while the validation and test set contains 3
extra unseen contexts (total 10). Unlike the majority of the baselines, we
don’t use context label attributes in train, validation, or test.

ImageNet-9: ImageNet-9 (Xiao et al., 2020) is a subset of ImageNet (Deng et
al., 2009) containing 9 super classes. It has been established that this
subset has a spurious correlation between object labels and image texture. We
followed the setting used by (Kim et al., 2022) and (Bahng et al., 2020) for
creating train and val split. We report the average accuracy on the validation
split.

ImageNet-A:  ImageNet-A (Hendrycks et al., 2021) contains handpicked real-
world images misclassified by models trained on ImageNet. Since these
misclassifications are due to over-reliance on spurious features like
color&texture, we use this dataset for evaluating models trained on ImageNet-9
as a robustness challenge (i.e., OOD test set).

###  4.3 Training Procedure & Metrics

For all our real-world experiments we consistently used ResNet-18, an
auxiliary layer that uses the same layer structure as of BasicBlock of ResNet
with varying depth, optimized using SGD optimizer with a fixed learning rate
of 0.001. For real-world experiments the model is loaded with ImageNet pre-
trained weights. We repeat the experiments with 5 different random seeds and
report the mean and std deviation of results. Refer Appendix A.1 for more
details.

Choice of Validation Set: For BAR, since there is no validation data provided,
we study it under two settings. In the first, we use 20% of images from the
test set and call it OOD-validation. In the second setting, which is harder
and more realistic, we use 20% images from the train set, calling it In-Domain
(ID) validation. For NICO-Animal, CelebA Hair and ImageNet-9, we use the
already supplied validation data. Table 1 shows the percentage of “bias-
conflicting” examples, i.e., examples that violate the spurious feature
correlation or training domain, for each portion of each dataset. Note that
BAR-ID val setting, NICO and ImageNet-9/ImageNet-A experiments do not have any
bias conflicting examples in the train set, and methods that rely on attribute
labels and/or reweighting of training data will perform poorly on them.

Table 1: Composition of conflicting examples in different datasets. Dataset | % Conflict Examples | hparam goal
---|---|---
| Train | Val | Test |
BAR-ID val | 0 | 0 | 100 | Avg Acc
BAR-OOD val | 0 | 100 | 100 | Avg Acc
CelebA | 0.8 | 0.9 | 0.9 | Unbiased Acc
NICO | 0 | 10 | 10 | Avg Acc
IN-9/IN-A | 0 | 0 | 100 | Avg Acc

Evaluation Metrics: Accuracy means average accuracy on all examples. Unbiased
means accuracy averaged over each label-context group. This metric is more
fair when there is a huge imbalance between the groups. Conflicting means
accuracy only on the bias conflicting examples.

We used Accuracy for BAR, NICO and Imagenet-9 / ImageNet-A and Unbiased for
CelebA-Hair dataset as the performance metric on validation data for
hyperparameter search and early stopping (Table 1).

Feature decodability: To measure the “decodability” of a chosen feature in a
classifier at a given layer, we freeze the classifier and train a linear
decoder on its network representation at the specified layer. The decoder is
trained on validation data, with each instance being assigned as label the
value of its feature. For example, to check decodability of shape in a CMNIST
classifier, input instances are assigned the label of the shape they contain,
while ignoring their color. This decoder’s accuracy is then reported on the
test set.

##  5 Results

Figure 3: Decodability of Simple (MNIST) and Complex (CIFAR) features across layers of ResNet-50 with a) Normal ERM training b) with SiFeR Table 2: Feature Controllability.  DataSet | Target Feature |  SiFeR (Ours) | ERM
---|---|---|---
|  | SR | CR | SR | CR
CMNIST | Complex (Digit) | 99.54$\pm$0.19 | 58.14$\pm$10.69 | 56.96$\pm$6.59 | 92.21$\pm$3.92
| Simple (Color) | 52.44$\pm$1.22 | 99.64$\pm$1.30 | 49.20$\pm$2.60 | 96.27$\pm$0.99
CIFAR_MNIST | Complex (CIFAR) | 62.37$\pm$4.62 | 48.93$\pm$1.92 | 58.14$\pm$1.60 | 100
| Simple (Digit) | 47.17$\pm$0.14 | 99.83$\pm$0.29 | 49.20$\pm$2.60 | 100

###  5.1 Suppressing Simple Features

We first studied the effectiveness of SiFeR on targeted suppression of
specific features. To do this, we experimented with the CIFAR_MNIST dataset,
which consists of composite pairings of MNIST & CIFAR images, each fully
predictive of the assigned label (see Section 4 for more details). DNNs are
known to entirely ignore the CIFAR feature on this training dataset–when the
CIFAR component is randomized at test-time, accuracy is unaffected, but when
the MNIST component is randomized, accuracy drops to chance. We refer to the
MNIST component as the simple feature, and CIFAR as the complex feature.

Figure 3 shows layerwise decodability (Section 4.3) of simple and complex
features, tracked across epochs in the training process. We contrast standard
training (top row) against SiFeR (bottom row). Standard training
overemphasizes the simple feature at higher layers, and ignores the complex
feature. The complex CIFAR feature is in fact decodable to some extent in
earlier layers of the ERM classifier, but is suppressed in later layers, due
to the preponderance of the simple feature. In contrast, the auxiliary
forgetting loss in SiFeR effectively suppresses the simple feature in the
earlier layers, and thereby enhances the decodability of the complex feature
in the higher layers. This shows that removing the availability of spurious
simple features is a direct method of overcoming simplicity bias.

These findings are more remarkable given that no prior knowledge of
“simple”/“complex” features was used in any way whatsoever during training.
SiFeR organically discovers and suppresses the by-design simple feature purely
through the use of the strategically placed auxiliary network, and its
configuration via the training recipe.

###  5.2 Feature Controllability using SiFeR

We described in Section 3.3 the various degrees of freedom in SiFeR for
identifying and suppressing features, and choosing them by the use of a
validation set (i.e., using generalization error). This gives us a simple,
powerful method for certain kinds of domain generalization, by simply using
domain-shift data as the validation set. We demonstrate this capability by
conducting studies on controlled datasets CMNIST & CIFAR_MNIST. In each, the
training data were designed to have pairings of simple and complex features
where both simple and complex features were fully predictive. We then trained
a SiFeR classifier for different choices of validation set, representing which
feature we actually wanted the classifier to focus on. This was achieved by
randomizing the “non-relevant” feature in the validation dataset, and choosing
all our hyperparameters based on that validation dataset. This represents a
real-world scenario where small amounts of vetted data are available for
optimization of a model, but (re)-labeling or manipulating all training data
is infeasible. Table 2 shows the results comparing SiFeR against ERM. SiFeR
shows higher accuracy for chosen features (higher diagonal terms) than for
spurious features (off-diagonal terms), driven by the choice of the validation
data. In contrast, ERM primarily focuses on the simple features, irrespective
of the choice of validation set (higher second column numbers). Thus, our
method is in fact able to focus on the relevant feature–be it simple or
complex–in an easily controllable manner.

Figure 4: Examples of SiFeR’s focus on relevant features while suppressing
irrelevant background information. Top row: Input images; Middle row: GRAD-
CAM-derived feature importance visualizations for the ERM classifier; Bottom
row: feature importance for SiFeR. First 3 columns from BAR (Nam et al.,
2020), and last 3 columns from NICO-Animal (He et al., 2021).

###  5.3 Debiasing Real-World Datasets

Table 3: Classification Accuracy (%) on test set of BAR Dataset. Method | Used OOD Val | Accuracy
---|---|---
ERM | ✓ | 51.85$\pm$ 5.92
BiaSwap (Kim et al., 2021) | ✓ | 52.44
LfF (Nam et al., 2020) | ✓ | 62.98$\pm$ 2.76
PGI (Ahmed et al., 2021) | ✓ | 65.19$\pm$ 1.32
EIIL (Creager et al., 2021) | ✓ | 65.44$\pm$ 1.17
ESB222ESB uses R50 architecture unlike other baseline which uses R18. (Teney et al., 2022) | ✓ | 67.10$\pm$ 0.30
Roadblock (Niu et al., 2022) | ✓ | 69.51$\pm$ 2.43
Debian (Li et al., 2022) | ✓ | 69.88$\pm$ 2.92
SiFeR (Ours) | ✓ | 72.08$\pm$ 0.38
ERM | ✗ | 35.32$\pm$ 0.46
ReBias (Bahng et al., 2020) | ✗ | 37.02$\pm$ 0.26
LfF (Nam et al., 2020) | ✗ | 48.15$\pm$ 0.93
SSL+ERM (Kim et al., 2022) | ✗ | 60.88$\pm$ 0.80
LWBC (Kim et al., 2022) | ✗ | 62.03$\pm$ 0.74
ESB (Teney et al., 2022) | ✗ | 64.40$\pm$ 0.20
SiFeR (Ours) | ✗ | 65.75$\pm$ 1.84
Table 4: Unbiased and Conflicting Accuracy metrics (%) on Test set of CelebA
Hair Dataset

Method Spurious Attribs Unbiased Conflict DRO (Sagawa et al., 2019) ✓
85.43$\pm$ 0.53 83.40$\pm$ 0.67 EnD (Tartaglione et al., 2021) ✓ 91.21$\pm$
0.22 87.45$\pm$ 1.06 CSAD (Zhu et al., 2021) ✓ 89.36 87.53 ERM ✗ 70.25$\pm$
0.35 52.52$\pm$ 0.19 LfF (Nam et al., 2020) ✗ 84.24$\pm$ 0.37 81.24$\pm$ 1.38
SSL+ERM (Kim et al., 2022) ✗ 80.48$\pm$ 0.91 66.79$\pm$ 2.20 LWBC (Kim et al.,
2022) ✗ 88.90$\pm$ 1.55 87.22$\pm$ 1.14 SiFeR (Ours) ✗ 89.00$\pm$ 0.92
88.04$\pm$ 1.25

Table 5: Classification Accuracy (%) on test set of NICO Dataset. Most of the baselines (DecAug, DRO, etc) use spurious attribute labels for training, while we do not. Method | Accuracy
---|---
ERM | 75.87
IRM (Arjovsky et al., 2019) | 59.17
REx (Krueger et al., 2021) | 74.31
JiGen (Carlucci et al., 2019) | 84.95
Mixup (Zhang et al., 2017) | 80.27
Cumix (Mancini et al., 2020) | 76.78
MTL (Blanchard et al., 2021) | 78.89
DANN (Ganin et al., 2016) | 75.59
CORAL (Sun & Saenko, 2016) | 80.27
MMD (Li et al., 2018) | 70.91
DRO (Sagawa et al., 2019) | 77.61
CNBB (He et al., 2021) | 78.16
DecAug (Bai et al., 2021a) | 85.23
SiFeR (Ours) | 86.20$\pm$ 0.85
NAS-OoD333Architecture design optimization based method, hence unfair to compare directly against other methods. (Bai et al., 2021b) | 88.72
Table 6: Classification Accuracy (%) on Validation set of ImageNet-9 and test
set of ImageNet-A.

Method Spurious Attribs ImageNet-9 ImageNet-A Accuracy Accuracy StylisedIN
(Geirhos et al., 2018) ✓ 88.4$\pm$ 0.5 24.6$\pm$ 1.4 LearnedMixin (Clark et
al., 2019) ✓ 64.1$\pm$ 4.0 15.0$\pm$ 1.6 RUBi (Cadene et al., 2019) ✓
90.5$\pm$ 0.3 27.7$\pm$ 2.1 ERM ✗ 90.8$\pm$ 0.6 24.9$\pm$ 1.1 BagNet18
(Brendel & Bethge, 2019) ✗ 67.7$\pm$ 0.3 18.8$\pm$ 1.15 ReBias (Bahng et al.,
2020) ✗ 91.9$\pm$ 1.7 29.6$\pm$ 1.6 LfF (Nam et al., 2020) ✗ 86.00 24.60 CaaM
(Wang et al., 2021) ✗ 95.70 32.80 SSL+ERM (Kim et al., 2022) ✗ 94.18$\pm$ 0.07
34.21$\pm$ 0.49 LWBC (Kim et al., 2022) ✗ 94.03$\pm$ 0.23 35.97$\pm$ 0.49
SiFeR ✗ 97.78$\pm$ 0.12 39.98$\pm$ 0.81

Our method outperforms baselines on four different real world datasets–BAR
(Nam et al., 2020), CelebA Hair (Liu et al., 2018), NICO (He et al., 2021) and
Imagenet-A (Hendrycks et al., 2021), by large margins (upto 11%, see Figure 2c
for a quick summary). Critically, we chose in all our experiments to not use
any knowledge of which attribute labels are considered spurious in each
dataset–this is because in real-world scenarios, it is difficult to know in
advance which attributes may end up containing biased information, or to label
data according to those attributes in order to do targeted debiasing of
models. Nevertheless, we outperform the other baselines, including many that
do use attribute labels as part of their training procedure.

Mitigating Spurious Correlations: Biased Activity Recognition (BAR) and CelebA
Hair Dataset represent background and gender bias in real life. In the BAR
training set, human activity (image categories) is spuriously correlated with
the background in which those activities are performed; in CelebA Hair, hair
color is strongly correlated with gender. Both BAR and CelebA are heavily
biased and contain no or very few conflicting examples–eg. CelebA Hair has
only 1% of men with blond hair in the train set.

For BAR, since no validation set is provided, we show results both using in-
distribution and out-distribution validation sets to compare against both sets
of baselines: those that do and do not require conflicting examples in the
validation set444For instance, methods that work on the principle of
reweighting conflicting samples in the trainset (eg LWBC (Kim et al., 2022))
typically add 1% of conflicting samples from the test set to the training
data. We outperform baselines in both settings by more than 1-2% absolute
accuracy, refer Table 3. Table 4 shows results on CelebA dataset, we get
almost the same unbiased accuracy as LWBC and improve upon conflicting
accuracy.

Domain-shift Generalization: NICO introduces three new contexts in which
object classes appear in the validation and test set, that are absent in the
training set. Table 5 shows that SiFeR beats all baselines on classification
accuracy for the test set; this is despite our not using context information,
unlike a majority of the baselines. Thus, SiFeR is valuable for zero-shot
domain generalization.

Robustness to Texture bias: Table 6 shows results on ImageNet-9, which is
known to be biased towards texture, and ImageNet-A, which consists of natural
images that have bias-conflicting features. This setting is closest to real-
world scenarios for texture bias. We improve the previous best baseline by 3%
absolute on ImageNet-9 validation set and by 4% absolute on ImageNet-A test
set. Thus, SiFeR encourages learning features robust to texture bias,
improving performance on both the in-distribution validation set as well as
bias-conflicting test set. Two critical findings here are a) that SiFeR did
not sacrifice in-distribution accuracy through the process of sieving simple
features, and b) the learned classifier robustly transfers over to a novel
test set, where it provides even larger gains.

Figure 5: Decodability of Spurious (Gender) and Target (Hair Color) features
across layers of ResNet-18 while training on CelebA with a) Normal ERM
training and b) with SIFER.

###  5.4 SiFeR: Debiasing without Extra Information

Table 3 (BAR dataset, bottom half) shows that SiFeR with only in-distribution
validation data (cf. Table 1) outperforms most baselines that leverage an
additional OOD validation set (top half). Further, without using either
attribute knowledge or conflicting examples in the validation set, we show
huge gains over ERM (65.7% accuracy vs 35.3%), demonstrating that SiFeR does
not critically depend on such additional information for debiasing, although
we can certainly leverage such information for additional gains (72.8% vs
65.7% accuracy when using OOD validation data).

###  5.5 Feature Decodability in Real-world Datasets

Figure 5 shows feature decodability (Section 4.3) on the real world dataset
CelebA, where the target label is hair color, and previous work has shown that
gender is a spuriously correlated attribute. Results show that SIFER
suppresses gender decodability, particularly in upper layers, with color
feature achieving stronger decodability (unlike ERM where gender is more
easily decodable than color). This mirrors the results on synthetic datasets
that were presented in Figure 3, and shows that SiFeR can automatically
identify and suppress featural information related to abstract concepts such
as gender, in support of better generalization accuracy.

###  5.6 SiFeR focuses on Relevant Information

We visualize the information in an image that is relevant to a given
classifier (Selvaraju et al., 2017), in order to verify whether our feature
sieving results in semantically relevant modifications to learned classifiers.
Figure 4 shows this evaluation, contrasting ERM classifier’s regions of focus
(middle row) and SiFeR’s regions of focus (bottom row) on a range of input
images (top row, drawn from BAR & NICO). Interestingly, not only does SiFeR
correctly focus on the central object of interest, but also it is able to
effectively suppress the (spuriously label-correlated) background information,
which is highly valued by the ERM classifier. This undercores SiFeR’s ability
to carefully differentiate between relevant and irrelevant features, rather
than some notion of simple vs complex features alone.

##  6 Discussion & Conclusion

We proposed SiFeR–a novel feature sieve approach towards addressing simplicity
bias and spurious correlations in deep neural networks. Our proposal
introduces an auxiliary network attached to the deep network which alternately
identifies and suppresses predictive features. The approach is controllable
through the use of configuration parameters optimized using validation data;
thus, it requires no foreknowledge or hand-coding of the notion of “simple
features”. We demonstrated on controlled datasets the ability of SiFeR to
automatically identify and suppress features; further, we showed that,
strictly speaking, SiFeR rebalances the role of various features in a
controllable manner driven by the needs of generalization. We showed using
extensive experiments on real-world data that our approach provides
significant gains–3-11% relative accuracy improvements on BAR, NICO, and
Imagenet-A. We believe our work is a small, important first step in a fruitful
new direction of research. We hope that follow-up work will build on the
notion of the feature sieve, developing effective computational barriers that
encourage deep networks to discover and utilize richer, more powerful featural
representations. Our current approach strikes a balance between various
competing features, guided by generalization error estimates (validation
error). One could potentially extract even more value if different feature
classes could be isolated into (relatively) independent predictors, then
combined effectively. This is, for instance, the approach taken by Niu et al.
(2022). Thus, a straightforward next step we aim to explore is the study of
ensembling approaches to combine a range of features of varying complexity &
predictive power, and methods for efficiently learning them. We also hope to
develop a systematic theoretical understanding of feature sieve approaches and
their role in supervised learning using DNNs.

## References

  * Ahmed et al. (2021) Ahmed, F., Bengio, Y., van Seijen, H., and Courville, A.  Systematic generalisation with group invariant predictions.  In _International Conference on Learning Representations_ , 2021.
  * Arjovsky et al. (2019) Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Bahng et al. (2020) Bahng, H., Chun, S., Yun, S., Choo, J., and Oh, S. J.  Learning de-biased representations with biased representations.  In _International Conference on Machine Learning_ , pp. 528–539. PMLR, 2020.
  * Bai et al. (2021a) Bai, H., Sun, R., Hong, L., Zhou, F., Ye, N., Ye, H.-J., Chan, S.-H. G., and Li, Z.  Decaug: Out-of-distribution generalization via decomposed feature representation and semantic augmentation.  _AAAI_ , 2021a.
  * Bai et al. (2021b) Bai, H., Zhou, F., Hong, L., Ye, N., Chan, S.-H. G., and Li, Z.  Nas-ood: Neural architecture search for out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8320–8329, 2021b.
  * Blanchard et al. (2021) Blanchard, G., Deshmukh, A. A., Dogan, Ü., Lee, G., and Scott, C.  Domain generalization by marginal transfer learning.  _The Journal of Machine Learning Research_ , 22(1):46–100, 2021.
  * Brendel & Bethge (2019) Brendel, W. and Bethge, M.  Approximating cnns with bag-of-local-features models works surprisingly well on imagenet.  _arXiv preprint arXiv:1904.00760_ , 2019.
  * Cadene et al. (2019) Cadene, R., Dancette, C., Cord, M., Parikh, D., et al.  Rubi: Reducing unimodal biases for visual question answering.  _Advances in neural information processing systems_ , 32, 2019.
  * Carlucci et al. (2019) Carlucci, F. M., D’Innocente, A., Bucci, S., Caputo, B., and Tommasi, T.  Domain generalization by solving jigsaw puzzles.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2229–2238, 2019.
  * Clark et al. (2019) Clark, C., Yatskar, M., and Zettlemoyer, L.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  _arXiv preprint arXiv:1909.03683_ , 2019.
  * Creager et al. (2021) Creager, E., Jacobsen, J.-H., and Zemel, R.  Environment inference for invariant learning.  In _International Conference on Machine Learning_ , pp. 2189–2200. PMLR, 2021.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_ , 2009.
  * Duboudin et al. (2022) Duboudin, T., Dellandréa, E., Abgrall, C., Hénaff, G., and Chen, L.  Look beyond bias with entropic adversarial data augmentation.  In _2022 26th International Conference on Pattern Recognition (ICPR)_ , pp. 2142–2148. IEEE, 2022.
  * Dwork et al. (2012) Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R.  Fairness through awareness.  In _Proceedings of the 3rd innovations in theoretical computer science conference_ , pp. 214–226, 2012.
  * Ganin et al. (2016) Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Geirhos et al. (2018) Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  In _International Conference on Learning Representations_ , 2018.
  * He et al. (2021) He, Y., Shen, Z., and Cui, P.  Towards non-iid image classification: A dataset and baselines.  _Pattern Recognition_ , 110:107383, 2021.
  * Hendrycks et al. (2021) Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D.  Natural adversarial examples.  _CVPR_ , 2021.
  * Hermann & Lampinen (2020) Hermann, K. and Lampinen, A.  What shapes feature representations? exploring datasets, architectures, and training.  _Advances in Neural Information Processing Systems_ , 33:9995–10006, 2020.
  * Kim et al. (2019) Kim, B., Kim, H., Kim, K., Kim, S., and Kim, J.  Learning not to learn: Training deep neural networks with biased data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9012–9020, 2019.
  * Kim et al. (2021) Kim, E., Lee, J., and Choo, J.  Biaswap: Removing dataset bias with bias-tailored swapping augmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 14992–15001, 2021.
  * Kim et al. (2022) Kim, N., Hwang, S., Ahn, S., Park, J., and Kwak, S.  Learning debiased classifier with biased committee.  _arXiv preprint arXiv:2206.10843_ , 2022.
  * Krueger et al. (2021) Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pp. 5815–5826. PMLR, 2021.
  * Li et al. (2018) Li, H., Pan, S. J., Wang, S., and Kot, A. C.  Domain generalization with adversarial feature learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5400–5409, 2018.
  * Li & Vasconcelos (2019) Li, Y. and Vasconcelos, N.  Repair: Removing representation bias by dataset resampling.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9572–9581, 2019.
  * Li et al. (2022) Li, Z., Hoogs, A., and Xu, C.  Discover and mitigate unknown biases with debiasing alternate networks.  In _17th European Conference on Computer Vision_ , pp. 270–288. Springer, 2022.
  * Liu et al. (2018) Liu, Z., Luo, P., Wang, X., and Tang, X.  Large-scale celebfaces attributes (celeba) dataset.  _Retrieved August_ , 15(2018):11, 2018.
  * Mancini et al. (2020) Mancini, M., Akata, Z., Ricci, E., and Caputo, B.  Towards recognizing unseen categories in unseen domains.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16_ , pp. 466–483. Springer, 2020.
  * Mehrabi et al. (2021) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54(6):1–35, 2021\.
  * Nagarajan et al. (2020) Nagarajan, V., Andreassen, A., and Neyshabur, B.  Understanding the failure modes of out-of-distribution generalization.  _arXiv preprint arXiv:2010.15775_ , 2020.
  * Nam et al. (2020) Nam, J., Cha, H., Ahn, S., Lee, J., and Shin, J.  Learning from failure: Training debiased classifier from biased classifier.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Niu et al. (2022) Niu, H., Li, H., Zhao, F., and Li, B.  Roadblocks for temporarily disabling shortcuts and learning new knowledge.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Pezeshki et al. (2021) Pezeshki, M., Kaba, O., Bengio, Y., Courville, A. C., Precup, D., and Lajoie, G.  Gradient starvation: A learning proclivity in neural networks.  _Advances in Neural Information Processing Systems_ , 34:1256–1272, 2021.
  * Russell et al. (2017) Russell, C., Kusner, M. J., Loftus, J., and Silva, R.  When worlds collide: integrating different counterfactual assumptions in fairness.  _Advances in neural information processing systems_ , 30, 2017.
  * Sagawa et al. (2019) Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  _arXiv preprint arXiv:1911.08731_ , 2019.
  * Selvaraju et al. (2017) Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 618–626, 2017.
  * Shah et al. (2020) Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P.  The pitfalls of simplicity bias in neural networks.  _Advances in Neural Information Processing Systems_ , 33:9573–9585, 2020.
  * Shrestha et al. (2022) Shrestha, R., Kafle, K., and Kanan, C.  Occamnets: Mitigating dataset bias by favoring simpler hypotheses.  _arXiv preprint arXiv:2204.02426_ , 2022.
  * Sun & Saenko (2016) Sun, B. and Saenko, K.  Deep coral: Correlation alignment for deep domain adaptation.  In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_ , pp. 443–450. Springer, 2016.
  * Tartaglione et al. (2021) Tartaglione, E., Barbano, C. A., and Grangetto, M.  End: Entangling and disentangling deep representations for bias correction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 13508–13517, 2021.
  * Teney et al. (2020) Teney, D., Abbasnejad, E., and Hengel, A. v. d.  Unshuffling data for improved generalization.  _arXiv preprint arXiv:2002.11894_ , 2020.
  * Teney et al. (2022) Teney, D., Abbasnejad, E., Lucey, S., and van den Hengel, A.  Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16761–16772, 2022.
  * Wang et al. (2021) Wang, T., Zhou, C., Sun, Q., and Zhang, H.  Causal attention for unbiased visual recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3091–3100, 2021.
  * Xiao et al. (2020) Xiao, K., Engstrom, L., Ilyas, A., and Madry, A.  Noise or signal: The role of image backgrounds in object recognition.  _ArXiv preprint arXiv:2006.09994_ , 2020.
  * Zafar et al. (2017) Zafar, M. B., Valera, I., Gomez Rodriguez, M., and Gummadi, K. P.  Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment.  In _Proceedings of the 26th international conference on world wide web_ , pp. 1171–1180, 2017.
  * Zhang et al. (2017) Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Zhou et al. (2022) Zhou, X., Lin, Y., Zhang, W., and Zhang, T.  Sparse invariant risk minimization.  In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 27222–27244. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/zhou22e.html.
  * Zhu et al. (2021) Zhu, W., Zheng, H., Liao, H., Li, W., and Luo, J.  Learning bias-invariant representation by cross-sample mutual information minimization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 15002–15012, 2021.

Appendix

##  Appendix A Additional Details

###  A.1 Training Details

For all experiments we consistently used ResNet-18, an auxiliary layer that
uses the same layer structure as of BasicBlock of ResNet with varying depth.
The ResNet network is composed of 4 layers modules (each itself is made up of
2 BasicBlocks). We apply auxiliary layer only at the end of the of layers
except layer 4, this gives us 3 different choice for aux
position($\mathcal{A}_{P}$) which we treat as hyperparameter. The network is
optimized using SGD optimizer with a fixed learning rate of 0.001. For real-
world experiments the model is loaded with ImageNet pre-trained weights. We
repeat the experiments with 5 different random seeds and report the mean and
std deviation of results. Table 7 shows the hyperparamters search space for
all the hyperparameters that we tune on the basis of validation set. To reduce
hyperparameter search space we fixed the value of $\alpha 1$ to 10. Table 8
shows the hyperparameter values obtained from the hyperparamter tuning.

Table 7: Range for hyperparameters search. Hparam | Range |
---|---|---
$\mathcal{A}_{D}$ | $[1,9]$ |
$\mathcal{A}_{P}$ | $[1,3]$ |
$\alpha 2$ | loguniform$(10^{-1},10^{2})$ |
$\alpha 3$ | loguniform$(10^{-1},10^{2})$ |
$\mathcal{F}$ | $[1,9]*10$ |
Table 8: Hyperparameter values obtained from the tuning. Dataset | $\mathcal{A}_{D}$ | $\mathcal{A}_{P}$ | $\alpha 2$ | $\alpha 3$ | $\mathcal{F}$
---|---|---|---|---|---
BAR - ID val | 4 | 2 | 2 | 4.5 | 70
BAR - OD val | 2 | 2 | 1 | 3 | 30
CelebA | 2 | 2 | 25 | 15 | 50
NICO | 2 | 1 | 1 | 75 | 70
IN-9/IN-A | 4 | 3 | 1 | 4.5 | 70

##  Appendix B Baselines

Here we list and briefly explain all the baselines that we compare against on
real world datasets:
BiaSwap (Kim et al., 2021) proposes a bias-tailored augmentation-based
approach for learning debiased representation without requiring supervision on
the bias type. they divide the data into bias-guiding and bias-conflicting
groups and then swaps the bias in bias guiding group.
LfF (Nam et al., 2020) uses generalized cross-entropy initially trains a
prejudiced net-work and tries to debias the second network by focusing
weighing on samples that go against the bias.
IRM (Arjovsky et al., 2019) uses theory of causal bayesian networks to find an
invariant feature representation using multiple training environments with
different bias correlations.
REx (Krueger et al., 2021) proposed a min-max algorithm to optimize for the
worst linear combination of risks on different environments.
EIIL (Creager et al., 2021) optimizes for bias group assignment to
automatically identify the bias groups to maximize IRM.
PGI (Ahmed et al., 2021) follows EIIL to identify bias groups by training a
small neural network.
Evading Simplicity Bias (ESB) (Teney et al., 2022) creates a ensemble of
diverse classifiers by incorporating a diversity regularizer between the
gradients while training.
Roadblock (Niu et al., 2022) adds adversarial augmentations to the image while
training to avoid over-reliance on spurious visual cues.
Debian (Li et al., 2022) trains two networks in alternate manner namely
discoverer and classifier, the discoverer tries to find multiple unknown
biases of the classifier without any annotations of biases, and the classifier
aims at unlearning the biases identified by the discoverer.
ReBias (Bahng et al., 2020) propose a novel framework to train a de-biased
representation by encouraging it to be different from a set of representations
that are biased by design.
LWBC (Kim et al., 2022) employs a committee of classifiers as an auxiliary
module that identifies bias-conflicting data and assigns large weights to them
when training the main classifier.
Group-DRO (Sagawa et al., 2019) minimizes for worst-case training loss over a
set of pre-defined groups.
EnD (Tartaglione et al., 2021) proposes a regularization technique that uses
the bias attributes to prevent deep models from learning spurious biases by
inserting an information bottleneck.
CSAD (Zhu et al., 2021), given the bias attributes, explicitly extracts target
and bias features disentangled from the latent representation generated by a
feature extractor and then learns to discover and remove the correlation
between the target and bias features.
JiGen (Carlucci et al., 2019) jointly classifies objects and solves
unsupervised jigsaw tasks.
Cumix (Mancini et al., 2020) mixes up data and labels from different domains
to be able to recognize unseen categories in unseen domains.
MTL (Blanchard et al., 2021) argue that problem of Domain Generalization can
be viewed as a kind of supervised learning problem by augmenting the original
feature space with the marginal distribution of feature vectors.
DANN (Ganin et al., 2016) proposes a representation learning approach such
that features are not predictive of the domain from which the model is being
trained on.
CORAL (Sun & Saenko, 2016) proposes an unsupervised domain adaptation method
that aligns the second-order statistics of the source and target distributions
with a linear transformation.
MMD (Li et al., 2018) extend adversarial autoencoders by imposing the Maximum
Mean Discrepancy measure to align the distributions among different domains,
and matching the aligned distribution to an arbitrary prior distribution via
adversarial feature learning.
CNBB (He et al., 2021) is an OoD learning method that based on sample
reweighting inspired by causal inference.
DecAug (Bai et al., 2021a) proposed a semantic augmentation and feature
decomposition approach to distangle context features from category related
features.
NAS-OoD (Bai et al., 2021b) adds an OOD generalization criterion to network
architecture search training to construct inherently more robust network
architectures.
StylisedIN (Geirhos et al., 2018) showed that ImageNet is texture biased and
works on improving shape bias.
LearnedMixin (Clark et al., 2019) trains a robust model as part of an ensemble
with the naive one in order to encourage it to focus on other patterns in the
data that are more likely to generalize.
CaaM (Wang et al., 2021) learns causal attention by partitioning the data on-
the-go to break correlation with bias.


# Land Cover Image Classification

Antonio Rangel
Instituto Tecnológico de Mazatlan
ITM
<EMAIL_ADDRESS>
& Juan R. Terven
Instituto Politecnico Nacional
CICATA-Qro
<EMAIL_ADDRESS>
& Diana M. Cordova-Esparza
Universidad Autónoma de Querétaro
Facultad de Informática
<EMAIL_ADDRESS>
& E.A. Chávez-Urbiola
Instituto Politecnico Nacional
CICATA-Qro
<EMAIL_ADDRESS>

###### Abstract

Land Cover (LC) image classification has become increasingly significant in
understanding environmental changes, urban planning, and disaster management.
However, traditional LC methods are often labor-intensive and prone to human
error. This paper explores state-of-the-art deep learning models for enhanced
accuracy and efficiency in LC analysis. We compare convolutional neural
networks (CNN) against transformer-based methods, showcasing their
applications and advantages in LC studies. We used EuroSAT, a patch-based LC
classification data set based on Sentinel-2 satellite images and achieved
state-of-the-art results using current transformer models.

_Keywords_ remote sensing $\cdot$ land cover $\cdot$ LULC classification

##  1 Introduction

Land Use Land Cover (LULC) is a multidisciplinary field that categorizes and
characterizes the earth’s terrestrial surface. It encompasses various types of
ground, from natural landscapes such as forests, wetlands, and deserts to
human-altered environments such as agricultural fields, urban areas, and
industrial sites. LULC studies provide a snapshot of the earth’s surface at a
given time, offering valuable insights into the spatial distribution and
interaction of various land use types and land cover classes. The dynamic
nature of LULC, driven by both natural processes and human activities,
necessitates continuous monitoring and analysis to capture temporal changes.

The importance of LULC studies extends to numerous fields. In environmental
science, LULC data inform our understanding of biodiversity, ecosystem
services, and the impacts of climate change. In urban planning and
development, it helps to manage land resources, assess environmental impacts,
and guide sustainable practices. LULC helps optimize land use for crop
production in agriculture while minimizing environmental degradation. In
addition, LULC data are integral to policy-making, supporting land
conservation, urban growth, and climate change mitigation decisions.

Changes in Land Use Land Cover (LULC) offer insights into the dynamics of
natural and human-altered landscapes, with applications spanning multiple
disciplines. For example, LULC change data can elucidate the impacts of human
activities on ecosystems, such as tracking deforestation rates to study
biodiversity loss or carbon sequestration changes. LULC changes can also be
used in climate change studies to influence local and global climates and
inform climate modeling and future scenario prediction. LULC change data can
inform the growth patterns of cities, aiding decisions about infrastructure
development, zoning, and resource allocation. Monitoring LULC changes can
optimize land use for crop production, identify shifts in agricultural
practices, and assess environmental impacts. LULC change data also aids in
disaster management, where changes in land cover, like deforestation, can
increase vulnerability to natural disasters such as floods and landslides.
Also, LULC changes inform policy decisions related to land conservation, urban
growth, climate change mitigation, and sustainable development.

Traditionally, remote sensing for LULC mapping involved manual or semi-
automated image classification processes. While efficient in their time, these
techniques were often time-consuming and meticulous tasks that required human
intervention and expert knowledge. The manual interpretation was potentially
prone to errors and discrepancies and lacked the scalability to cover large
geographical areas.

AI, specifically machine learning (ML) algorithms, has significantly improved
the ability to detect remote sensing in LULC studies. These algorithms learn
from vast amounts of data, building models that can forecast future scenarios
or identify patterns with exceptional accuracy. Recent progress in computer
vision, a subfield of AI, has encouraged the development of sophisticated
algorithms to teach computers how to see and analyze large amounts of image
data, including satellite or aerial imagery used in remote sensing.

Among the classic Machine Learning methods for modeling LULC is K-nearest
neighbor from [1] and [2], Random Forests by [1], [3], [4], [5], [6], [7],
[8], and [9], Support Vector Machines (SVM) have also been used in the works
of [1], [2], [3], [4], [7], [8], [10], [11], and [12]. Other approaches such
as Naive Bayes by [4], Decision trees by [1], [2], [4], and [12], Classic
Neural Networks in the works of [1], [4], [7], [8], [10], and [11], Maximum
likelihood classifiers by [10], Bagging in [5], and Boosting in [5].

More recently, Deep Learning-based models have taken the stage and dominated
the remote sensing world with models based on Convolutional Neural Networks
(CNNs) in the works of [13], [14], [15], and [16]. Autoencoders have also been
used to learn a representation of the data in [17], [18], and [19], Stacked
Autoencoders by [20], and [21], 3D Convolutional Autoencoders by [22],
multitask deep learning in [23], Generative models by [24], and Transformers
by [25], [26], [27], [28], [29], and [30].

This paper compares classification methods for land cover. We compare standard
Convolutional Neural Network methods as well as state-of-the-art Transformer-
based methods. We share all the code this project uses at
github.com/jrterven/eurosat_classification.

Figure 1: Sample images from the EuroSAT dataset showing the ten categories.

##  2 The EuroSAT dataset

Land Use Land Cover (LULC) categorizes and characterizes the earth’s
terrestrial surface. _Land use_ describes the human activities directly
related to the land, indicating how people utilize land resources, such as
residential, agricultural, or commercial. _Land cover_ , on the other hand,
refers to the physical material at the surface of the earth, including grass,
asphalt, trees, bare ground, water, etc.

The EuroSAT dataset [31] contains ten classes with 27000 labeled and geo-
referenced images taken from Sentinel-2 satellite images. The images are
$64\times 64$ and cover cities in the European Urban Atlas. The covered cities
are distributed over the 34 European countries: Austria, Belarus, Belgium,
Bulgaria, Cyprus, Czech Republic (Czechia), Denmark, Estonia, Finland, France,
Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania,
Luxembourg, Macedonia, Malta, Republic of Moldova, Netherlands, Norway,
Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland,
Ukraine and United Kingdom. Figure 1 shows one random sample image for each of
the ten classes.

The dataset provides split for training, validation, and testing with 18900,
5400, and 2700 images per split, respectively. Figure 2 shows the distribution
of images per class for the training set.

Figure 2: Images distribution in the Training set.

##  3 Methods

###  3.1 Data Acquisition

We obtained the dataset from the original source ([32]), ensuring authenticity
and data integrity. This dataset provides a collection of ten categories,
mostly balanced (see Figure 2), necessary for training and evaluating our
classification models.

###  3.2 Data Preprocessing and Training

We developed custom code for efficient data loading, preprocessing, and
augmentation. The preprocessing steps included normalization and resizing of
images. We trained ten models, seven convolutional architectures, and three
transformer-based architectures. The convolutional models were AlexNet by
[33], ResNet50 by [34], ResNeXt by [35], DenseNet by [36], MobileNetV3 from
[37], EfficientNetV2 by [38], and ConvNeXt by [39]. The transformed models
were ViT by [40], Swin Transformer by [41], and MaxViT by [42].

We trained all models using an early stop of ten epochs and kept the model
with the best validation accuracy. We used the categorical cross-entropy loss
and the Adam optimizer by [43] with a learning rate of $1\times 10^{-4}$.

####  3.2.1 Training from Scratch.

Initially, we trained each of the ten models from scratch. This approach
involved initializing the weights with the Kaiming Initialization proposed by
[44] and then retraining all layers using the RGB images of the EuroSAT
dataset. The purpose of this step was to evaluate the learning capacity of
each architecture without prior knowledge.

####  3.2.2 Training with Pre-trained Weights.

Subsequently, we employed transfer learning by training the same models using
pre-trained weights. These weights were obtained from models pre-trained on
ImageNet.

###  3.3 Model Evaluation

We evaluated each model on the test set, which was not used during the
training phase. As evaluation metrics, we compute the Top-1 accuracy and the
precision/recall curves.

For our experiments, we used Pytorch and multiple computers: a Colab machine
with NVIDIA V100 GPU, as well as two personal computers, one with TITAN X GPU
and another equipped with RTX 4090 GPU.

##  4 Results

In this section, we present the results of the classification task using ten
influential deep learning architectures trained from scratch and retrained
with pre-trained parameters. Table 1 shows the Top-1 accuracy on the test set.

Table 1: Classification accuracy comparison on ten deep learning models. The Accuracy column shows the results of the models trained from scratch, while the Accuracy pre-trained column shows the results of retraining with pre-trained weights. The first seven are convolutional-based models, while the last three are Transformer-based models. Model name | Accuracy | Accuracy
---|---|---
|  | pre-trained
AlexNet | 0.837 | 0.916
ResNet | 0.835 | 0.947
ResNeXt | 0.843 | 0.981
DenseNet | 0.925 | 0.949
MobileNetV3 | 0.877 | 0.958
EfficientNetV2 | 0.908 | 0.973
ConvNeXt | 0.783 | 0.986
ViT32 | 0.917 | 0.972
SwinB | 0.926 | 0.987
MaxViT | 0.973 | 0.990

As shown in the table, the model with the highest scores is also the latest
architecture that we tried, achieving a state-of-the-art 99% accuracy on this
dataset.

Figures 3(a) and 3(b) show the validation accuracy curves obtained during
training. It should be noted that the models trained with random weights
(Figure 3(a)) took significantly longer to train and reached lower accuracy
than the models trained with pre-trained weights (Figure 3(b)). We used the
[45] platform to log the experiments and generate these graphs.

(a) Models trained from scratch

(b) Pretrained models

Figure 3: Validation accuracy curves for the ten models. (a) Trained from
scratch and (b) using pre-trained weights on ImageNet. Best viewed in color.

(a) Models trained from scratch

(b) Pretrained models

Figure 4: Precision/Recall curves for the ten models. (a) Trained from scratch
and (b) using pre-trained weights on ImageNet. Best viewed in color.

Figures 4(a) and 4(b) show the precision/recall curves for the ten models. The
curves show that the models trained from scratch (Figure 4(a)) present more
variability in their results, with ConvNeXt on the lower end and MaxViT on the
higher end. For the pre-trained models, the curves are closer to each other,
with AlexNet as the lowest-performance model and MaxViT on the higher end.
Curiously, ConvNeXt was the lowest-performance model trained from scratch but
was on par with the best models when trained with pre-trained weights. This
may indicate that the transfer learning used with the pre-trained weights
helps alleviate the relatively small size of this dataset for such a model.

##  5 Conclusion

Land cover image classification is a crucial task in understanding
environmental changes, urban planning, and disaster management. In this paper,
we explored using state-of-the-art deep learning models for enhanced accuracy
and efficiency in LC analysis. We compared convolutional neural networks (CNN)
against transformer-based methods, illuminating their applications and
advantages in LC studies. Our results showed that both CNN and transformer-
based methods achieved high accuracy in LC classification, with transformer-
based models outperforming CNNs in some cases.

Using deep learning models in land cover classification offers a powerful
avenue for research and applications. With the availability of large-scale
remote sensing datasets and the rapid development of deep learning techniques,
we can expect continued advancement in this field. As we move towards more
automated and efficient methods for land cover classification, we can unlock
the full potential of this data in addressing critical environmental and
societal challenges.

##  6 Acknowledgements

We thank the Instituto Politecnico Nacional through the Research and
Postgraduate Secretary (SIP) project number 20232290 and the Consejo Nacional
de Humanidades Ciencias y Tecnologías (CONAHCYT) for its support through the
Sistema Nacional de Investigadoras e Investigadores (SNII).

## References

  * [1] P. Thanh Noi and M. Kappas, “Comparison of random forest, k-nearest neighbor, and support vector machine classifiers for land cover classification using sentinel-2 imagery,” Sensors, vol. 18, no. 1, p. 18, 2017.
  * [2] Y. Qian, W. Zhou, J. Yan, W. Li, and L. Han, “Comparing machine learning classifiers for object-based land cover classification using very high resolution imagery,” Remote Sensing, vol. 7, no. 1, pp. 153–168, 2014.
  * [3] L. Ma, M. Li, X. Ma, L. Cheng, P. Du, and Y. Liu, “A review of supervised object-based land-cover image classification,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 130, pp. 277–293, 2017.
  * [4] F. F. Camargo, E. E. Sano, C. M. Almeida, J. C. Mura, and T. Almeida, “A comparative assessment of machine-learning techniques for land use and land cover classification of the brazilian tropical savanna using alos-2/palsar-2 polarimetric images,” Remote Sensing, vol. 11, no. 13, p. 1600, 2019.
  * [5] B. Ghimire, J. Rogan, V. R. Galiano, P. Panday, and N. Neeti, “An evaluation of bagging, boosting, and random forests for land-cover classification in cape cod, massachusetts, usa,” GIScience & Remote Sensing, vol. 49, no. 5, pp. 623–643, 2012.
  * [6] E. Adam, O. Mutanga, J. Odindi, and E. M. Abdel-Rahman, “Land-use/cover classification in a heterogeneous coastal landscape using rapideye imagery: evaluating the performance of random forest and support vector machines classifiers,” International Journal of Remote Sensing, vol. 35, no. 10, pp. 3440–3458, 2014.
  * [7] T. Shiraishi, T. Motohka, R. B. Thapa, M. Watanabe, and M. Shimada, “Comparative assessment of supervised classifiers for land use–land cover classification in a tropical region using time-series palsar mosaic data,” IEEE Journal of selected topics in applied earth observations and remote sensing, vol. 7, no. 4, pp. 1186–1199, 2014.
  * [8] E. Raczko and B. Zagajewski, “Comparison of support vector machine, random forest and neural network classifiers for tree species classification on airborne hyperspectral apex images,” European Journal of Remote Sensing, vol. 50, no. 1, pp. 144–154, 2017.
  * [9] R.-Y. Lee, D.-Y. Ou, Y.-S. Shiu, and T.-C. Lei, “Comparisons of using random forest and maximum likelihood classifiers with worldview-2 imagery for classifying crop types,” in Proceedings of the 36th Asian Conference Remote Sensing Foster, ACRS, Citeseer, 2015.
  * [10] P. K. Srivastava, D. Han, M. A. Rico-Ramirez, M. Bray, and T. Islam, “Selection of classification techniques for land use/land cover change investigation,” Advances in Space Research, vol. 50, no. 9, pp. 1250–1265, 2012.
  * [11] S. Pal and S. Ziaul, “Detection of land use and land cover change and land surface temperature in english bazar urban centre,” The Egyptian Journal of Remote Sensing and Space Science, vol. 20, no. 1, pp. 125–145, 2017.
  * [12] J. R. Otukei and T. Blaschke, “Land cover change assessment using decision trees, support vector machines and maximum likelihood classification algorithms,” International Journal of Applied Earth Observation and Geoinformation, vol. 12, pp. S27–S31, 2010.
  * [13] D. Marcos, M. Volpi, B. Kellenberger, and D. Tuia, “Land cover mapping at very high resolution with rotation equivariant cnns: Towards small yet accurate models,” ISPRS journal of photogrammetry and remote sensing, vol. 145, pp. 96–107, 2018.
  * [14] B. Huang, B. Zhao, and Y. Song, “Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery,” Remote Sensing of Environment, vol. 214, pp. 73–86, 2018.
  * [15] M. Rezaee, M. Mahdianpari, Y. Zhang, and B. Salehi, “Deep convolutional neural network for complex wetland classification using optical remote sensing imagery,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 11, no. 9, pp. 3030–3039, 2018.
  * [16] Y. Shi, D. Ma, J. Lv, and J. Li, “Actl: Asymmetric convolutional transfer learning for tree species identification based on deep neural network,” IEEE Access, vol. 9, pp. 13643–13654, 2021.
  * [17] W. Zhou, Z. Shao, C. Diao, and Q. Cheng, “High-resolution remote-sensing imagery retrieval using sparse features by auto-encoder,” Remote sensing letters, vol. 6, no. 10, pp. 775–783, 2015.
  * [18] A. Azarang, H. E. Manoochehri, and N. Kehtarnavaz, “Convolutional autoencoder-based multispectral image fusion,” IEEE access, vol. 7, pp. 35673–35683, 2019.
  * [19] M. Hu, C. Wu, L. Zhang, and B. Du, “Hyperspectral anomaly change detection based on autoencoder,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 3750–3762, 2021.
  * [20] C. Zhao, X. Wan, G. Zhao, B. Cui, W. Liu, and B. Qi, “Spectral-spatial classification of hyperspectral imagery based on stacked sparse autoencoder and random forest,” European journal of remote sensing, vol. 50, no. 1, pp. 47–63, 2017.
  * [21] X. Sun, F. Zhou, J. Dong, F. Gao, Q. Mu, and X. Wang, “Encoding spectral and spatial context information for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 12, pp. 2250–2254, 2017.
  * [22] S. Mei, J. Ji, Y. Geng, Z. Zhang, X. Li, and Q. Du, “Unsupervised spatial–spectral feature learning by 3d convolutional autoencoder for hyperspectral classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6808–6820, 2019.
  * [23] S. Liu, Q. Shi, and L. Zhang, “Few-shot hyperspectral image classification with unknown classes using multitask deep learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 6, pp. 5085–5102, 2020.
  * [24] D. Hong, J. Yao, D. Meng, Z. Xu, and J. Chanussot, “Multimodal gans: Toward crossmodal hyperspectral–multispectral image segmentation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 6, pp. 5103–5113, 2020.
  * [25] D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, and J. Chanussot, “Spectralformer: Rethinking hyperspectral image classification with transformers,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2021.
  * [26] Z. Xue, X. Tan, X. Yu, B. Liu, A. Yu, and P. Zhang, “Deep hierarchical vision transformer for hyperspectral and lidar data classification,” IEEE Transactions on Image Processing, vol. 31, pp. 3095–3110, 2022.
  * [27] Y. Qing, W. Liu, L. Feng, and W. Gao, “Improved transformer net for hyperspectral image classification,” Remote Sensing, vol. 13, no. 11, p. 2216, 2021.
  * [28] A. Jamali and M. Mahdianpari, “Swin transformer and deep convolutional neural networks for coastal wetland classification using sentinel-1, sentinel-2, and lidar data,” Remote Sensing, vol. 14, no. 2, p. 359, 2022.
  * [29] H. Dong, L. Zhang, and B. Zou, “Exploring vision transformers for polarimetric sar image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2021.
  * [30] J. Yao, B. Zhang, C. Li, D. Hong, and J. Chanussot, “Extended vision transformer (exvit) for land use and land cover classification: A multimodal deep learning framework,” IEEE Transactions on Geoscience and Remote Sensing, 2023.
  * [31] P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 7, pp. 2217–2226, 2019.
  * [32] Phelber, “Eurosat: Land use and land cover classification with sentinel-2,” 2023.
  * [33] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.
  * [34] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [35] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492–1500, 2017.
  * [36] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.
  * [37] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al., “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314–1324, 2019.
  * [38] M. Tan and Q. Le, “Efficientnetv2: Smaller models and faster training,” in International conference on machine learning, pp. 10096–10106, PMLR, 2021.
  * [39] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976–11986, 2022.
  * [40] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  * [41] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022, 2021.
  * [42] Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li, “Maxvit: Multi-axis vision transformer,” in European conference on computer vision, pp. 459–479, Springer, 2022.
  * [43] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision, pp. 1026–1034, 2015.
  * [45] Weights and Biases, “The ai developer platform,” 2023.  Accessed: 2023-12-01.