11institutetext: 1Nanyang Technological University 2Singapore Management
University 3Panasonic R&D Center Singapore
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Equivariance and Invariance Inductive Bias for Learning from Insufficient
Data

Tan Wang1 Qianru Sun2 Sugiri Pranata3 Karlekar Jayashree3 Hanwang Zhang1

###### Abstract

We are interested in learning robust models from insufficient data, without
the need for any externally pre-trained checkpoints. First, compared to
sufficient data, we show why insufficient data renders the model more easily
biased to the limited training environments that are usually different from
testing. For example, if all the training swan samples are “white”, the model
may wrongly use the “white” environment to represent the intrinsic class swan.
Then, we justify that equivariance inductive bias can retain the class feature
while invariance inductive bias can remove the environmental feature, leaving
the class feature that generalizes to any environmental changes in testing. To
impose them on learning, for equivariance, we demonstrate that any off-the-
shelf contrastive-based self-supervised feature learning method can be
deployed; for invariance, we propose a class-wise invariant risk minimization
(IRM) that efficiently tackles the challenge of missing environmental
annotation in conventional IRM. State-of-the-art experimental results on real-
world benchmarks (VIPriors, ImageNet100 and NICO) validate the great potential
of equivariance and invariance in data-efficient learning. The code is
available at https://github.com/Wangt-CN/EqInv.

###### Keywords:

Inductive Bias, Equivariance, Invariant Risk Minimization

##  1 Introduction

Data is never too big. As illustrated in Fig. 1 (a), if we have sufficiently
large training sample size of swan and dog, e.g., dogs and cats in any
environment such as different colors, shapes, poses, and backgrounds, by using
a conventional softmax cross-entropy based “swan vs. dog” classifier, we can
obtain a “perfect” model that discards the _shared_ environmental features but
retains the _discriminative_ class features. The underlying common sense is
that if the model has seen any “case” in training, the testing data is merely
a seen IID subset of the training data, yielding testing accuracy as good as
training [85].

In this paper, we are interested in learning from insufficient data. Besides
the common motivation that collecting data is expensive, we believe that how
to narrow the performance gap between insufficient and sufficient data is the
key to tackling the non-IID challenge in machine generalization—even if the
training data is sufficient, the testing can still be out of the training
distribution (OOD) [35, 74, 79]. After all, we can always frame up exceptional
testing samples that fail the trained model [40, 31]. Note that different from
few-shot learning which widely adopts pre-training on large-scale training set
[83, 80, 82], our task does not allow using any externally pre-trained
checkpoint and backbone111See https://vipriors.github.io/ for details..

Fig. 1 (b) illustrates why insufficient data hurts generalization. Without
loss of generality, we conduct a thought experiment that we have limited swan
only in “white” color environment while sufficient dog in diverse
environments. So, we can expect that the “dog” feature will still be extracted
to represent dog model, but the “white” feature will be recklessly learned to
represent swan. This is because training swan model by using either “swan” or
“white” feature yields the similar training risk: 1) if the former, the
training loss is minimized as in the perfect case of Fig. 1 (a); 2) if the
latter, the only training error possible would be misclassifying “white dog”
as swan. However, it can be easily corrected in practice, e.g., by
discriminatively training a sample-to-model distance prior that
$\|\mathbf{z}_{\textrm{dog}}\|>\|\mathbf{z}_{\textrm{white}}\|$, where
$\mathbf{z}$ denotes the feature vector222The distance between the “white dog”
sample vector $(\mathbf{z}_{\textrm{white}},\mathbf{z}_{\textrm{dog}})$ and
the swan model vector $(\mathbf{z}_{\textrm{white}},\mathbf{0})$ is:
$\|(\mathbf{z}_{\textrm{white}},\mathbf{z}_{\textrm{dog}})-(\mathbf{z}_{\textrm{white}},\textbf{0})\|=\|(\textbf{0},\mathbf{z}_{\textrm{dog}})\|=\|\mathbf{z}_{\textrm{dog}}\|$;
similarly, we have the distance between “white dog” and dog model as
$\|\mathbf{z}_{\textrm{white}}\|$..

Figure 1: Illustration of how the proposed equivariance and invariance
inductive biases help learning from insufficient data. Cartoon figures such as
denote the class feature. Boxed words such as  denote environmental features.
Grey-boxed figures denote the learned model. For simple illustration, we omit
the environment as background.

Why, under the same training risk, does the swan model prefer “white” but not
“swan” feature? First, feature extraction in deep network follows a bottom-up,
low-level to high-level fashion [53]—“simple” features such as color can be
easily learned at lower layers, while “complex” features such as object parts
will be only emerged in higher layers [78, 95, 91]. Second, the commonly used
cross-entropy loss encourages models to stop learning once “simple” features
suffice to minimize the loss [28, 29]. As a result, “swan” features like
“feather”, “beak”, and “wing” will be lost after training. Such mechanism is
also well-known as the shortcut bias [28] or spurious correlation in causality
literature [71, 87]. We will provide formal justifications in Section 3.1.

By comparing the difference between Fig. 1 (a) and Fig. 1 (b), we can see that
the crux of improving the generalization of insufficient data is to recover
the missing “swan” class feature while removing the “white” environmental
feature. To this end, we propose two inductive biases to guide the learning:
_equivariance_ for class preservation and _invariance_ for environment
removal.

Equivariance. This prior requires that the feature representation of a sample
should be equivariant to its semantic changes, e.g., any change applied to the
sample should be faithfully reflected in the feature change (see Appendix for
the mathematical definition). Therefore, if we impose a contrastive loss for
each sample feature learning, we can encourage that different samples are
mapped into different features (see Section 3.2 for a detailed analysis and
our choice). As illustrated in Fig. 1 (c), equivariance avoids the degenerated
case in Fig 1 (b), where all “white swan” samples collapse to the same “white”
feature. Thus, for a testing “black swan”, the retained “swan” feature can win
back some swan scores despite losing the similarity between “black” and
“white”. It is worth noting that the equivariance prior may theoretically shed
light on the recent findings that self-supervised learning features can
improve model robustness [39, 75, 38, 88]. We will leave it as future work.

Invariance. Although equivariance preserves all the features, due to the
limited environments, the swan model may still be confounded by the “white”
environment, that is, a testing “black swan” may still be misclassified as
dog, e.g., when
$\|(\mathbf{z}_{\textrm{black}}-\mathbf{z}_{\textrm{white}},\mathbf{z}_{\textrm{swan}}-\mathbf{z}_{\textrm{swan}})\|>\|(\mathbf{z}_{\textrm{black}}-\mathbf{0},\mathbf{z}_{\textrm{swan}}-\mathbf{z}_{\textrm{dog}})\|$.
Inspired by invariant risk minimization [4] (IRM) that removes the
environmental bias by imposing the environmental invariance prior (Section
3.3), as shown in Fig. 1 (d), if we split the training data into two
environments: “white swan” vs. “white dog” and “white swan” vs. “black dog”,
we can learn a common classifier (i.e., a feature selector) that focuses on
the “swan” and “dog” features, which are the _only invariance_ across the two
kinds of color environments—one is identical as “white” and the other contains
two colors. Yet, conventional IRM requires environment annotation, which is
however impractical. To this end, in Section 4, we propose class-wise IRM
based on contrastive objective that works efficiently without the need for the
annotation. We term the overall algorithm of using the two inductive biases,
i.e., equivariance and invariance, as EqInv.

We validate the effectiveness of EqInv on three real-world visual
classification benchmarks: 1) VIPriors ImageNet classification [14], where we
evaluate 10/20/50 samples per class; 2) NICO [35], where the training and
testing environmental distributions are severely different; and 3) ImageNet100
[84] which denotes the case of sufficient training data. On all datasets, we
observe significant improvements over baseline learners. Our EqInv achieves a
new single-model state-of-the-art on test split: 52.27% on VIPriors-50 and
64.14% on NICO.

##  2 Related Work

Visual Inductive Bias. For a learning problem with many possible solutions,
inductive bias is a heuristic prior knowledge that regularizes the learning
behavior to find a better solution [61]. It is ubiquitous in any modern deep
learning models: from the shallow MLP [59] to the complex deep ResNet [12, 3]
and Transformers [89, 20]. Inductive biases can be generally grouped into two
camps: 1) Equivariance: the feature representation should faithfully preserve
all the data semantics [21, 22, 56]. 2) Invariance: generalization is about
learning to be invariant to the diverse environments [86, 8]. Common practical
examples are the pooling/striding in CNN [47], dropout [36], denoising
autoencoder [45], batch normalization [25], and data augmentations [13, 16].

Data-Efficient Learning. Most existing works re-use existing datasets [96, 15]
and synthesize artificial training data [52, 24]. We work is more related to
those that overcome the data dependency by adding prior knowledge to deep nets
[10, 30]. Note that data-efficient learning is more general than the popular
setting of few-shot learning [83, 80, 82] which still requires external large
pre-training data as initialization or meta-learning. In this work, we offer a
theoretical analysis for the difference between learning from insufficient and
sufficient data, by posing it in an OOD generalization problem.

OOD Generalization. Conventional machine generalization is based on the
Independent and Identically Distributed (IID) assumption for training and
testing data [85]. However, this assumption is often challenged in
practice—the Out-of-Distribution (OOD) problem degrades the generalization
significantly [37, 93, 74]. Most existing works can be framed into the stable
causal effect pursuit [71, 87, 46] or finding an invariant feature subspace
[66, 90]. Recently, Invariant Risk Minimization (IRM) takes a different
optimization strategy such as convergence speed regularization [4, 51] and
game theory [1]. Our proposed class-wise IRM makes it more practical by
relaxing the restrictions on needing environment annotation.

##  3 Justifications of the Two Inductive Biases

As we discussed in Section 1, given an image $X=x$ with label $Y=y$, our goal
is to extract the intrinsic class feature $\phi(x)$ invariant to the
environmental changes $z\in Z$. Specifically, $Z$ is defined as all the class-
agnostic items in the task of interest. For example, spatial location is the
intrinsic class feature in object detection task, but an environmental feature
in image classification. This goal can be achieved by using the interventional
Empirical Risk Minimization (ERM) [46]. It replaces the observational
distribution $P(Y|X)$ with the interventional distribution $P(Y|do(X))$, which
removes the environmental effects from the prediction of $Y$, making $Y=y$
only affected by $X=x$ [69]. The interventional empirical risk $\mathcal{R}$
with classifier $f$ can be written as (See Appendix for the detailed
derivation):

$\displaystyle\mathcal{R}$ $\displaystyle=\mathbb{E}_{x\sim P(X),y\sim
P(Y|do(X))}\mathcal{L}(f(\phi(x)),y)$ (1)
$\displaystyle=\sum_{x}\sum_{y}\sum_{z}\mathcal{L}(f(\phi(x)),y)P(y|x,z)P(z)P(x),$

where $\mathcal{L}(f(\phi(x)),y)$ is the standard cross-entropy loss. Note
that Eq. (1) is hard to implement since the environment $Z$ is unobserved in
general.

When the training data is sufficient, $X$ can be almost observed in any
environment $Z$, leading to the approximate independence of $Z$ and $X$, i.e.,
$P(Z|X)\approx P(Z)$. Then $\mathcal{R}$ in Eq. (1) approaches to the
conventional ERM $\hat{\mathcal{R}}$:

$\mathcal{R}\approx\hat{\mathcal{R}}=\sum_{x}\sum_{y}\mathcal{L}(f(\phi(x)),y)P(y|x)P(x)=\mathbb{E}_{(x,y)\sim
P(X,Y)}\mathcal{L}(f(\phi(x)),y),$ (2)

Figure 2: The t-SNE [60] data visualization of class “hen” on different-scale
ImageNet dataset using CLIP [72] pretrained feature extractor. Blue dot and
orange triangle represent training and testing samples, respectively. The
testing accuracy is evaluated by ResNet-50 [34] trained from scratch on each
dataset. See Appendix for details.

###  3.1 Model Deficiency in Data Insufficiency

However, when the training data is insufficient, $P(Z|X)$ is no longer
approximate to $P(Z)$ and thus $\hat{\mathcal{R}}\not\approx\mathcal{R}$. For
example,
$P(Z=\raisebox{-2.39996pt}{\includegraphics[height=11.39996pt]{inline_figures/white-
inline.png}}|\raisebox{-2.39996pt}{\includegraphics[height=11.39996pt]{inline_figures/swan-
inline.png}})>P(Z=\raisebox{-2.39996pt}{\includegraphics[height=11.39996pt]{inline_figures/black-
inline.png}}|\raisebox{-2.39996pt}{\includegraphics[height=11.39996pt]{inline_figures/swan-
inline.png}})$. Then, as we discussed in Section 1, some simple environmental
semantics $Z$, e.g.,
$Z=\raisebox{-2.39996pt}{\includegraphics[height=11.39996pt]{inline_figures/white-
inline.png}}$, are more likely dominant in minimizing $\hat{\mathcal{R}}$ due
to $P(y|x)=P(y|x,z)P(z|x)$ in Eq. (2), resulting the learned $\phi$ that
mainly captures the dominant environment but missing the intrinsic class
feature. Empirical results in Fig. 2 also support such analysis. We show the
ImageNet classification results of class hen using various training sizes. We
can observe that with the decreasing of training samples, the accuracy
degrades significantly, from 86.0% to 4.0%. After all, when the training size
is infinite, any testing data is a subset of training.

###  3.2 Inductive Bias I: Equivariant Feature Learning

To win back the missing intrinsic class feature, we impose the contrastive-
based self-supervised learning (SSL) techniques [17, 33, 65], without the need
for any external data, to achieve the equivariance. In this paper, we follow
the definition and implementation in [86] to achieve sample-equivariant by
using contrastive learning, i.e., different samples should be respectively
mapped to different features. Given an image $x$, the data augmentation of $x$
constitute the positive example $x^{+}$, whereas augmentations of other images
constitute $N$ negatives $x^{-}$. The key of contrastive loss is to map
positive samples closer, while pushing apart negative ones in the feature
space:

$\mathbb{E}_{x,x^{+},\\{x_{i}^{-}\\}_{i=1}^{N}}\left[-\log\frac{\mathrm{exp}(\phi(x)^{\mathrm{T}}\phi(x^{+}))}{\mathrm{exp}(\phi(x)^{\mathrm{T}}\phi(x^{+}))+\sum_{i=1}^{N}\mathrm{exp}(\phi(x)^{\mathrm{T}}\phi(x_{i}^{-}))}\right].$
(3)

Note that we are open to any SSL choice, which is investigated in Section 5.

We visualize the features learned by training from scratch and utilizing the
equivariance inductive bias on NICO [35] dataset with both class and context
annotations. In Fig. 3 (a), it is obvious that there is no clear boundary to
distinguish the semantics of class and context in the feature space, while in
Fig. 3 (b), the features are well clustered corresponding to both class and
context.

Figure 3: The t-SNE [60] visualization of learned features w.r.t both class
and context annotations on NICO dataset with (a) training from scratch; (b)
equivariant learning; and (c) equivariant & invariant learning.

###  3.3 Inductive Bias II: Invariant Risk Minimization

Although the equivariance inductive bias preserves all the features, the swan
model may still be confounded by the “white” feature during the downstream
fine-tuning, causing $\hat{\mathcal{R}}\neq\mathcal{R}$. To mitigate such
shortcut bias, a straightforward solution is to use Inverse Probability
Weighting (IPW) [5, 44, 57] (also known as reweighting [6, 64, 55]) to down
weight the overwhelmed “white” feature in swan. However, they must follow the
positivity assumption [42], i.e., all the environmental semantics $Z$ should
exist in each class. However, when the training data is insufficient, such
assumption no longer holds. For example, how do you down weight “white” over
“black” if there is even no “black swan” sample?

Recently, Invariant Risk Minimization (IRM) [4, 51] resolves the non-
positivity issue by imposing the invariance inductive bias to directly remove
the effect of environmental semantics $Z$. Specifically, IRM first splits the
training data into multiple environments $e\in\mathcal{E}$. Then, it
regularizes $\phi$ to be _equally_ optimal in different splits, i.e.,
invariant across environments:

$\sum_{e}\mathcal{L}_{e}(w^{\mathrm{T}}\phi(x),y)+\lambda\|\nabla_{w=1}\mathcal{L}_{e}(w^{\mathrm{T}}\phi(x),y)\|_{2}^{2},$
(4)

where $\lambda$ is trade-off hyper-parameter, $w$ stands for a dummy
classifier [4] to calculate the gradient penalty across splits—though
different environments may induce different losses, the feature $\phi$ must
regularize them optimal at the same time (the lower gradient the better) in
the same way (by using the common dummy classifier). Note that each
environment should contain a unique mode of environmental feature distribution
[4, 23, 2]: suppose that we have $k$ environmental features that are
distributed as $\\{p_{1},p_{2},...,p_{k}\\}$. If $p^{e_{1}}_{i}\neq
p^{e_{2}}_{i}$, $i=1$ to $k$, IRM under the two environments will remove all
the $k$ features—the keeping of any one will be penalized by the second term
of Eq. (4).

Conventional IRM requires the environment annotations, which are generally
impossible in practice. To this end, we propose a novel class-wise IRM to
regularize the invariance within each class, without the need for environment
supervision. We show the qualitative results of imposing such invariance
inductive bias in Fig. 3 (c). Compared to Fig. 3 (b), we can observe that
after applying our proposed class-wise IRM, the equivariance of intrinsic
class features are reserved with well-clustered data points while the context
labels are no longer responsive—the environment features are removed.

##  4 Our EqInv Algorithm

Fig. 4 depicts the computing flow of EqInv . In the following, we elaborate
each of its components.

Input: Insufficient training samples denoted as the pairs $\\{(x,y)\\}$ of an
image $x$ and its label $y$.

Output: Robust classification model $f\cdot\phi$ with intrinsic class feature
$\phi(x)$ and unbiased classifier $f(\phi(x))$.

Step 1: Equivariant Learning via SSL. As introduced in Section 3, a wide range
of SSL pretext tasks are sufficient for encoding the sample-equivariance. For
fair comparison with other methods in VIPriors challenge dataset [14], we use
MoCo-v2 [33, 18], Simsiam [19], and IP-IRM [86] to learn $\phi$ in Fig 4 (a).
We leave the results based on the most recent MAE [32] in Appendix.

Figure 4: The flowchart of our proposed EqInv with 3 steps. Rectangle with
shading denotes the feature and $\mathcal{E}_{j}$ represents the generated
environment of class $j$. $x_{1k}^{-}$ and $x_{2k}^{-}$ in (c) are the $k$-th
negative samples of subset $e_{1}$ and $e_{2}$, respectively. We highlight
that class-wise IRM optimizes the mask layer $m$ (and an extra MLP $g$)
without gradients flowing back to the feature extract $\phi$.

Step 2: Environment Construction based on Adjusted Similarity. Now we are
ready to use IRM to remove the environmental features in $\phi$. Yet,
conventional IRM does not apply as we do not have environment annotations. So,
this step aims to automatically construct environments $\mathcal{E}$. However,
it is extremely challenging to identify the combinatorial number of unique
environmental modes—improper environmental split may contain shared modes,
which cannot be removed. To this end, we propose an efficient _class-wise_
approximation that seeks _two_ environments w.r.t. each class. Our key
motivation is that, for insufficient training data, the environmental variance
within each class is relatively simple and thus we assume that it is single-
modal. Therefore, as shown in Fig. 4 (b), we propose to use each class (we
call anchor class) as an anchor environmental mode to split the samples of the
rest of the classes (we call other classes) into two groups: similar to the
anchor or not. As a result, for $C$ classes, we will have totally $2C$
approximately unique environments. Intuitively, this class-wise strategy can
effectively remove the severely dominant context bias in a class. For example,
if all swan samples are “white”, the “white” feature can still be identified
as a non-discriminative color feature, thanks to the “black” and “white”
samples of dog class.

For each anchor class containing $l$ images, environment Env#1 contains these
$l$ samples as positive and the “similar” samples from other classes as
negative; environment Env#2 contains the same positive samples while the
“dissimilar” samples from other classes as negative. A straightforward way to
define the “similarity” between two samples is to use cosine similarity. We
compute the cosine similarity between the pair images sampled from anchor
class and other classes, respectively. We get the matrix
$\mathbf{S}\in\mathbb{R}^{l\times n}$, where $n$ is the number of images in
other classes. Then, we average this matrix along the axis of anchor class, as
in the pseudocode: $\mathbf{s}^{+}=\textrm{mean}(\mathbf{S},\mathrm{dim=0})$.
After ranking $\mathbf{s}^{+}$, it is easy to get “similar” samples
(corresponding to higher half values in $\mathbf{s}^{+}$) grouped in Env$\\#$1
and “dissimilar” samples (corresponding to lower half values in
$\mathbf{s}^{+}$) grouped in Env$\\#$2\. It is an even split. Fig. 5 (a) shows
the resultant environments for anchor class 0 on the Colored MNIST333It is
modified from MNIST dataset [54] by injecting color bias on each digit
(class). The non-bias ratio is $0.5\%$, e.g., $99.5\%$ samples of 0 are in red
and only $0.5\%$ in uniform colors. [64] using the above straightforward
cosine similarity. We can see that the digit classes distribute differently in
Env$\\#$1 and Env$\\#$2, indicating that the difference of the two
environments also include class information, which will be disastrously
removed after applying IRM.

Figure 5: The obtained environments $\mathcal{E}_{0}$ for an example anchor
class 0 on the Colored MNIST [64], by using (a) the vanilla cosine similarity
and (b) our adjusted similarity. On X-axis, 1-9 are other digit classes, and
c0-c9 denote $10$ colors used to create this color-bias dataset. On Y-axis,
the percentage point denotes the proportion of a digit (or a color) grouped
into a specific environment.

To this end, we propose a similarity adjustment method. It is to adjust every
sample-to-class similarity by subtracting a class-to-class similarity, where
the sample belongs to the class. First, we calculate the class-to-class
similarity $\bar{s}_{i}$ between the $i$-th ($i=1,...,C-1$) other class and
the anchor class: $\bar{s}_{i}=\mathrm{mean}(\mathbf{s}^{+}[a_{i}:b_{i}])$,
where we assume that the image index range of the $i$-th other class is
$[a_{i}:1:b_{i}]$. Such similarity can be viewed as a purer “class effect” to
be removed from the total effect of both class and environment—only
“environment effect” is then left. Therefore, for any sample $x^{j}$ from the
$i$-th other class, its adjusted similarity to the anchor class is:
$s=\mathbf{s}^{+}[j]-\bar{s}_{i}$. Using this similarity, we obtain new
environments and show statistics in Fig. 5 (b). It is impressive that the
biased color of anchor class 0 (i.e., the $0$-th color $c0$ or red) varies
between Env$\\#$1 and Env$\\#$2, but the classes and other colors (red dashed
boxes) distribute almost uniformly in these two environments. It means the
effects of class and environment are disentangled.

Step 3: Class-wise Invariant Risk Minimization. With the automatically
constructed environments, we are ready to remove the environmental feature
from $\phi$. In particular, we propose a class-wise IRM based on the
contrastive objective, which is defined as follows. As shown in Fig. 4 (c),
given a training image $x$ in environment $e$ of class $i$, we use a learnable
vector mask layer $m$ multiplied on $\phi(x)$ to select the invariant feature.
Then, we follow [17] to build a projection layer $g(\cdot)$ to obtain
$\bm{z}=g(m\circ\phi(x))$ for contrastive supervision, where $g$ is a one-
hidden-layer MLP with ReLU activation and $\circ$ denotes element-wise
production. For each anchor class $k$, we define an environment-based
supervised contrastive loss [49]. It is different from the traditional self-
supervised contrastive loss. Specifically, our loss is computed within each
environment $e\in\mathcal{E}_{k}$. We take the representations of anchor class
samples (in $e$) as positives $\bm{z}^{+}$, and the representations of other
class samples (in $e$) as negatives $\bm{z}^{-}$, and we have:

$\ell(e\in\mathcal{E}_{k},w=1)=\sum_{\bm{z}\in
e}\frac{1}{N^{+}}\sum_{\bm{z}^{+}\in
e}\left[-\log\frac{\mathrm{exp}(\bm{z}^{\mathrm{T}}\bm{z}^{+}\cdot
w)}{\mathrm{exp}(\bm{z}^{\mathrm{T}}\bm{z}^{+}\cdot w)+\sum_{\bm{z}^{-}\in
e}\mathrm{exp}(\bm{z}^{\mathrm{T}}\bm{z}^{-}\cdot w)}\right],$ (5)

where $N^{+}$ denotes the number of the positive samples in the current
minibatch and $w=1$ is a “dummy” classifier to calculate the gradient penalty
term [4]. Therefore, the proposed class-wise IRM loss444Please note that in
implementation, we adopt an advanced version [51] of IRM. Please check
appendix for details. is:

$\mathcal{L}_{k}=\sum\nolimits_{e\in\mathcal{E}_{k}}\ell(e,w=1)+\lambda\|\nabla_{w=1}\ell(e,w=1)\|_{2}^{2},$
(6)

where $\lambda$ is the trade-off hyper-parameter. The overall training
objective is the combination of minimizing a conventional cross entropy
$\mathcal{L}_{ce}$ and the class-wise IRM regularization $\mathcal{L}_{k}$:

$\min_{f,g,m,\phi}\mathcal{L}_{ce}(f,m,\phi)+\sum\nolimits_{k=1}^{C}\mathcal{L}_{k}(g,m),$
(7)

where we use $f(m\circ\phi(x))$ for inference. It is worth noting that each
loss trains a different set of parameters—$\phi$ is frozen during the class-
wise IRM penalty update. As the equivariance of $\phi$ is only guaranteed by
SSL pretraining, compared to the expensive SSL equivariance regularization in
training [86], our frozen strategy is more efficient to mitigate the adversary
effect introduced by the invariance bias, which may however discard
equivariant features to achieve invariance. We investigate this phenomenon
empirically in Section 5.4.

##  5 Experiments

###  5.1 Datasets and Settings

VIPriors [14] dataset is proposed in VIPrior challenge [14] for data-efficient
learning. It contains the same 1,000 classes as in ImageNet [26], and also
follows the same splits of train, val and test data. In all splits, each class
contains only $50$ images, so the total number of samples in the dataset is
$150k$. Some related works [9, 58] used the merged set (of train and val) to
train the model. We argue that this to some extent violates the protocol of
data-efficient learning—using insufficient training data. In this work, our
EqInv models as well as comparing models are trained on the standard train set
and evaluated on val and test sets. In addition, we propose two more
challenging settings to evaluate the models: VIPriors-20 and VIPriors-10. The
only difference with VIPriors is they have $20$ and $10$ images per class in
their train sets, respectively. There is no change on val and test sets. We
thus call the original VIPriors-50. NICO [35] is a real-world image dataset
proposed for evaluating OOD methods. The key insight of NICO is to provide
image labels as well as context labels (annotated by human). On this dataset,
it is convenient to “shift” the distribution of the class by “adjusting” the
proportions of specific contexts. In our experiments, we follow the
“adjusting” settings in the related work [87]. Specifically, this is a
challenging OOD setting using the NICO animal set. It mixes three
difficulties: 1) Long-Tailed; 2) Zero-Shot and 3) Orthogonal. ImageNet100 [84]
is a subset of original ImageNet [26] with $100$ classes and 1$k$ images per
class. Different with previous OOD datasets, ImageNet100 is to evaluate the
performances of our EqInv and comparison methods in sufficient training data
settings.

Table 1: Recognition accuracies (%) on the VIPriors-50, -20, -10, NICO and
ImageNet-100 (IN-100) datasets. “Aug.” represents augmentation. Note that due
to the effectiveness of “Random Aug.”, we set it as a default configuration
for the methods trained from SSL. Our results are highlighted.

Model | VIPriors-50 [14] |  |  VIPriors-20 |  |  VIPriors-10 |  |  NICO [35] |  | IN-100 [84] |
---|---|---|---|---|---|---|---|---|---|---
Val | Test |  | Val | Test |  | Val | Test |  | Val | Test |  | Val |
Train from Scratch |  Baseline | 32.30 | 30.60 |  | 13.13 | 12.39 |  | 5.02 | 4.59 |  | 43.08 | 40.77 |  |  | 83.56 |
|  Augmentation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  Stronger Aug. [17] | 36.60 | 34.72 |  | 16.17 | 15.21 |  | 3.49 | 3.26 |  | 42.31 | 43.31 |  |  | 83.72 |
|  Random Aug. [24] | 41.09 | 39.18 |  | 16.71 | 16.03 |  | 3.88 | 4.01 |  | 45.15 | 44.92 |  |  | 85.12 |
|  Mixup [92] | 34.66 | 32.75 |  | 13.35 | 12.69 |  | 2.47 | 2.31 |  | 40.54 | 38.77 |  |  | 84.52 |
|  Label smoothing [62] | 33.77 | 31.87 |  | 12.71 | 12.05 |  | 4.76 | 4.43 |  | 39.77 | 38.15 |  |  | 85.22 |
|  Debias Learning |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  Lff [64] | 35.04 | 33.29 |  | 13.26 | 12.58 |  | 5.20 | 4.79 |  | 41.62 | 42.54 |  |  | 83.74 |
|  Augment Feat. [55] | 35.41 | 33.63 |  | 13.62 | 12.97 |  | 3.43 | 3.12 |  | 42.31 | 43.27 |  |  | 83.88 |
|  CaaM [87] | 36.13 | 34.24 |  | 14.68 | 13.99 |  | 4.88 | 4.63 |  | 46.38 | 46.62 |  |  | 84.56 |
Train from SSL |  MoCo-v2 [18] | 49.47 | 46.98 |  | 30.76 | 28.83 |  | 18.40 | 16.97 |  | 46.45 | 45.70 |  |  | 86.30 |
|  +EqInv (Ours) | 54.21 | 52.09 |  | 38.30 | 36.66 |  | 26.70 | 25.20 |  | 52.55 | 51.51 |  |  | 88.38 |
|  SimSiam [19] | 42.69 | 40.75 |  | 22.09 | 21.15 |  | 6.84 | 6.68 |  | 41.27 | 42.68 |  |  | 85.28 |
|  +EqInv (Ours) | 52.55 | 50.36 |  | 37.29 | 35.65 |  | 24.74 | 23.33 |  | 45.67 | 44.77 |  |  | 86.80 |
|  IP-IRM [86] | 51.45 | 48.90 |  | 38.91 | 36.26 |  | 29.94 | 27.88 |  | 63.60 | 60.26 |  |  | 86.94 |
|  +EqInv (Ours) | 54.58 | 52.27 |  | 41.53 | 39.21 |  | 32.70 | 30.36 |  | 66.07 | 64.14 |  |  | 87.78 |

###  5.2 Implementation Details

We adopted ResNet-50/-18 as model backbones for VIPriors/ImageNet100 and NICO
datasets, respectively. We trained the model with 100 epochs for “training
from scratch” methods. We initialized the learning rate as $0.1$ and decreased
it by $10$ times at the $60$-th and $80$-th epochs. We used SGD optimizer and
the batch size was set as 256. For equivariant learning (i.e., SSL), we
utilized MoCo-v2 [18], SimSiam [19] and IP-IRM [86] to train the model for 800
epochs without using external data, using their default hyper-parameters. We
pretrain the model for 200 epochs on ImageNet100 dataset. Then for downstream
fine-tuning, We used SGD optimizer and set batch size as 128. We set epochs as
50, initialized learning rate as 0.05, and decreased it at the $30$-th and
$40$-th epochs. Please check appendix for more implementation details. Below
we introduce our baselines including augmentation-based methods, debiased
learning methods and domain generalization (DG) methods.

Augmentation-based Methods are quite simple yet effective techniques in the
VIPriors challenges as well as for the task of data-efficient learning. We
chose four top-performing methods in this category to compare with: stronger
augmentation [17], random augmentation[24], mixup [92] and label smoothing
[62].

Debias Learning Methods. Data-efficient learning can be regarded as a task for
OOD. We thus compared our EqInv with three state-of-the-art (SOTA) debiased
learning methods: Lff [64], Augment Feat. [55] and CaaM [87].

Domain Generalization Methods. Domain Generalization (DG) task also tackles
the OOD generalization problem, but requires sufficient domain samples and
full ImageNet pretraining. In this paper, we select three SOTA DG approaches
(SD [70], SelfReg [50] and SagNet [63]) for comparison. These methods do not
require domain labels which share the same setting as ours.

###  5.3 Comparing to SOTAs

Table 1 shows the overall results comparing to baselines on VIPriors-50, -20,
-10, NICO and ImageNet100 datasets. Our EqInv achieves the best performance
across all settings. In addition, we have another four observations. 1)
Incorporating SSL pretraining, vanilla fine-tuning can achieve much higher
accuracy than all the methods of “training from scratch”. This validates the
efficiency of the equivariance inductive bias (learned by SSL) for Etackling
the challenge of lacking training data. 2) When decreasing the training size
of VIPriors from 50 to 10 images per class, the comparison methods of training
from scratch cannot bring performance boosting even hurt the performance. This
is because the extremely insufficent data cannot support to establish an
equivariant representation, not mention to process samples with harder
augmentations. 3) Interestingly, compared to SSL methods, we can see that the
improvement margins by our method are larger in the more challenging
VIPriors-10, e.g., $8.2\%$ on MoCo-v2 and $16.7\%$ on SimSiam. It validates
the invariance inductive bias learned by the class-wise IRM (in our EqInv)
helps to disentangle and alleviate the OOD bias effectively. 4) Results on
ImageNet100 dataset show the consistent improvements of EqInv due to the
additional supervised contrastive loss, indicating the generalizability of our
EqInv in a wide range of cases from insufficient to sufficient data.

Table 2: Test accuracy (%) of DG SOTA methods. V-50/-10 denote
VIPriors-50/-10.

| Methods | V-50 | V-10 |  | Methods | V-50 | V-10
---|---|---|---|---|---|---|---
Train from Scratch | Boardline | 30.60 | 4.59 |  Train from SSL | IP-IRM | 48.90 | 26.88
SD [70] | 33.91 | 4.85 | +SD [70] | 49.91 | 28.01
SelfReg [50] | 23.85 | 3.64 | +SelfReg [50] | 36.48 | 22.75
SagNet [63] | 34.92 | 5.62 | +SagNet [63] | 47.82 | 26.17
|  +EqInv | 52.27 | 30.36

In Table 2, we compare our EqInv with DG methods. We try both “train from
scratch” and “train from SSL” to meet the pretraining requirement of DG. We
can find that our EqInv outperforms DG methods with large margins, showing the
weaknesses of existing OOD methods for handling _insufficient_ data.

In Table 5.4, we compare our EqInv with the solutions from other competition
teams in the challenge with the same comparable setting: no val is used for
training, single model w/o ensemble, similar ResNet50/ResNext50 backbones. We
can observe the best performance is by our method. It is worth noting that the
competitors Zhao et al. also used SSL techniques for pretraining. They took
the knowledge distillation [41, 43] as their downstream learning method. Our
EqInv outperforms their model with a large margin.

###  5.4 Ablation Study

Q1: What are the effects of different components of EqInv?

A1: We traversed different combinations of our proposed three steps to
evaluate their effectiveness. The results are shown in Table 5.4. We can draw
the following observations: 1) By focusing on the first three rows, we can
find that the improvements are relatively marginal without the SSL
equivariance pretraining. This is reasonable as the feature similarity cannot
reflect the semantics change exactly without the equivariance property, thus
affects the environments construction (Step 2) and class-wise IRM (Step 3); 2)
The comparison between row 4 to 6 indicates the significance of our proposed
similarity adjustment (Step 2). It is clear that the vanilla cosine similarity
results in clear performance drops due to the inaccurate environment
construction.

Table 3: Accuracy (%) comparison with other competition teams (single model
w/o ensemble) on the val set of VIPriors-50.

Team | Backbone | Val Acc
---|---|---
Official Baseline | ResNet50 | 33.16
Zhao et al.  [94] | ResNet50 | 44.60
Wang et al.  [14] | ResNet50 | 50.70
Sun et al.  [81] | ResNext50 | 51.82
EqInv (Ours) | ResNet50 | 54.58

Table 4: Evaluation of the effectiveness of our three steps in EqInv on
VIPriors-20.

Components |  Val |  Test
---|---|---
Step 1 |  Step 2 |  Step 3
✗ | ✗ | ✗ | 13.13 | 12.39
✗ | ✗ | ✓ | 13.01 | 12.41
✗ | ✓ | ✓ | 15.69 | 14.17
✓ | ✗ | ✗ | 37.61 | 34.88
✓ | ✗ | ✓ | 38.87 | 36.34
✓ | ✓ | ✓ | 40.15 | 37.78

Q2: What is the optimal $\lambda$ for EqInv? Why does not the class-wise IRM
penalty term update feature backbone $\phi$?

A2: Recall that we highlight such elaborate design in Section 4 Step 3. In
Fig. 6 (a), we evaluate the effect of freezing $\phi$ for Eq. (6) on VIPriors
dataset. First, we can see that setting $\lambda=10$ with freezing $\phi$ can
achieve the best validation and test results. Second, when increasing
$\lambda$ over $10$, we can observe a sharp performance drops for updating
$\phi$, even down to the random guess ($\approx$ 1%). In contrast, the
performances are much more robust with freezing $\phi$ while varying
$\lambda$, indicating the non-sensitivity of our EqInv . This validate the
adversary effect of the equivariance and invariance. Updating $\phi$ with
large $\lambda$ would destroy the previously learnt equivaraince inductive
bias.

Figure 6: (a) Accuracies (%) with different optimization schedules and values
of $\lambda$ on the VIPriors-20 val and test sets. (b) The intra-class feature
variance of training from SSL and our EqInv on VIPriors-10 dataet during
training process. “Feature” and “Masked Feature” represent $\phi(x)$ and
$m\circ\phi(x)$, respectively.

Q3: Does our EqInv achieve invariance with the learned environments
$\mathcal{E}$ and the proposed class-wise IRM (i.e., Step 3)?

A3: In Fig. 6 (b), we calculate the variance of intra-class feature with
training from SSL and our EqInv on VIPriors-10 data. It represents the feature
divergence within the class. We can find that: 1) Compared to our EqInv , the
variance of training from SSL increases dramatically, indicating that
equivariant features are still easily biased to environments without
invariance regularizations. 2) The masked feature $m\circ\phi(x)$ of our EqInv
achieves continuously lower variance than $\phi(x)$, validates the
effectiveness of our learnt mask. See Appendix for more visual attention
visualizations.

Q4: What does the cluster look like for real data with the proposed similarity
adjustment (i.e., Step 2)?

Figure 7: Visualizations of the top-10 images of generated environments for
two classes (i.e., great white shark and ladybug) on VIPriors-10 dataset. We
manually label their main context label (i.e., water and wild).

A4: Recall that we have displayed the cluster results on a toy Colored MNIST
data in Fig. 5 and validated the superiority of our similarity adjustment.
Here we wonder how does it perform on real-world data with much comprehensive
semantics? We visualize the top-10 images of Env#1 and Env#2 for two random
selected classes in Fig. 7. Interestingly, we can find that images of Env#1
mainly share the context (e.g., water) with the anchor class (e.g., Great
White Shark). In contrast, images of Env#2 have totally different context.
More importantly, the classes distribute almost uniformly in both Env#1 and
#2, indicating that our adjusted similarity isolate the effect of the class
feature.

##  6 Conclusion

We pointed out the theoretical reasons why learning from insufficient data is
inherently more challenging than sufficient data—the latter will be inevitably
biased to the limited environmental diversity. To counter such “bad” bias, we
proposed to use two “good” inductive biases: equivariance and invariance,
which are implemented as the proposed EqInv algorithm. In particular, we used
SSL to achieve the equivariant feature learning that wins back the class
feature lost by the “bad” bias, and then proposed a class-wise IRM to remove
the “bad” bias. For future work, we plan to further narrow down the
performance gap by improving the class-muted clustering to construct more
unique environments.

Acknowledgement. The authors would like to thank all reviewers for their
constructive suggestions. This research is partly supported by AISG, A*STAR
under its AME YIRG Grant (Project No.A20E6c0101).

## References

  * [1] Ahuja, K., Shanmugam, K., Varshney, K., Dhurandhar, A.: Invariant risk minimization games. In: ICML. pp. 145–155. PMLR (2020)
  * [2] Ahuja, K., Wang, J., Dhurandhar, A., Shanmugam, K., Varshney, K.R.: Empirical or invariant risk minimization? a sample complexity perspective. arXiv preprint (2020)
  * [3] Allen-Zhu, Z., Li, Y.: What can resnet learn efficiently, going beyond kernels? NeurIPS 32 (2019)
  * [4] Arjovsky, M., Bottou, L., Gulrajani, I., Lopez-Paz, D.: Invariant risk minimization. arXiv preprint arXiv:1907.02893 (2019)
  * [5] Austin, P.C.: An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate behavioral research 46(3), 399–424 (2011)
  * [6] Bahng, H., Chun, S., Yun, S., Choo, J., Oh, S.J.: Learning de-biased representations with biased representations. In: ICML. pp. 528–539. PMLR (2020)
  * [7] Bai, Y., Mei, J., Yuille, A.L., Xie, C.: Are transformers more robust than cnns? NeurIPS 34 (2021)
  * [8] Bardes, A., Ponce, J., LeCun, Y.: Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 (2021)
  * [9] Barz, B., Brigato, L., Iocchi, L., Denzler, J.: A strong baseline for the vipriors data-efficient image classification challenge. arXiv preprint arXiv:2109.13561 (2021)
  * [10] Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al.: Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018)
  * [11] Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., Veit, A.: Understanding robustness of transformers for image classification. In: ICCV. pp. 10231–10241 (2021)
  * [12] Bietti, A., Mairal, J.: On the inductive bias of neural tangent kernels. NeurIPS 32 (2019)
  * [13] Bouchacourt, D., Ibrahim, M., Morcos, A.: Grounding inductive biases in natural images: invariance stems from variations in data. NeurIPS 34 (2021)
  * [14] Bruintjes, R.J., Lengyel, A., Rios, M.B., Kayhan, O.S., van Gemert, J.: Vipriors 1: Visual inductive priors for data-efficient deep learning challenges. arXiv preprint arXiv:2103.03768 (2021)
  * [15] Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: ECCV. pp. 233–248 (2018)
  * [16] Chen, S., Dobriban, E., Lee, J.: A group-theoretic framework for data augmentation. NeurIPS 33, 21321–21333 (2020)
  * [17] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML. pp. 1597–1607. PMLR (2020)
  * [18] Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)
  * [19] Chen, X., He, K.: Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566 (2020)
  * [20] Chrupała, G.: Symbolic inductive bias for visually grounded learning of spoken language. arXiv preprint arXiv:1812.09244 (2018)
  * [21] Cohen, T., Weiler, M., Kicanaoglu, B., Welling, M.: Gauge equivariant convolutional networks and the icosahedral cnn. In: ICML. pp. 1321–1330. PMLR (2019)
  * [22] Cohen, T., Welling, M.: Group equivariant convolutional networks. In: ICML. pp. 2990–2999. PMLR (2016)
  * [23] Creager, E., Jacobsen, J.H., Zemel, R.: Environment inference for invariant learning. In: ICML. pp. 2189–2200. PMLR (2021)
  * [24] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: CVPR Workshops. pp. 702–703 (2020)
  * [25] Daneshmand, H., Joudaki, A., Bach, F.: Batch normalization orthogonalizes representations in deep random networks. NeurIPS 34 (2021)
  * [26] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. Ieee (2009)
  * [27] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint (2020)
  * [28] Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11), 665–673 (2020)
  * [29] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR (2018)
  * [30] Gondal, M.W., Wuthrich, M., Miladinovic, D., Locatello, F., Breidt, M., Volchkov, V., Akpo, J., Bachem, O., Schölkopf, B., Bauer, S.: On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. NeurIPS 32 (2019)
  * [31] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
  * [32] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 (2021)
  * [33] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 (2019)
  * [34] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
  * [35] He, Y., Shen, Z., Cui, P.: Towards non-iid image classification: A dataset and baselines. Pattern Recognition 110, 107383 (2021)
  * [36] Helmbold, D.P., Long, P.M.: On the inductive bias of dropout. The Journal of Machine Learning Research 16(1), 3403–3454 (2015)
  * [37] Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. ICLR (2019)
  * [38] Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., Song, D.: Pretrained transformers improve out-of-distribution robustness. arXiv preprint arXiv:2004.06100 (2020)
  * [39] Hendrycks, D., Mazeika, M., Kadavath, S., Song, D.: Using self-supervised learning can improve model robustness and uncertainty. NeurIPS 32 (2019)
  * [40] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. In: CVPR. pp. 15262–15271 (2021)
  * [41] Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., Choi, J.Y.: A comprehensive overhaul of feature distillation. In: ICCV. pp. 1921–1930 (2019)
  * [42] Hernán, M.A., Robins, J.M.: Causal inference (2010)
  * [43] Hinton, G., Vinyals, O., Dean, J., et al.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2(7) (2015)
  * [44] Imbens, G.W., Rubin, D.B.: Causal inference in statistics, social, and biomedical sciences. Cambridge University Press (2015)
  * [45] Jo, Y., Chun, S.Y., Choi, J.: Rethinking deep image prior for denoising. In: ICCV. pp. 5087–5096 (2021)
  * [46] Jung, Y., Tian, J., Bareinboim, E.: Learning causal effects via weighted empirical risk minimization. NeurIPS 33, 12697–12709 (2020)
  * [47] Kayhan, O.S., Gemert, J.C.v.: On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In: CVPR. pp. 14274–14285 (2020)
  * [48] Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M.: Transformers in vision: A survey. ACM Computing Surveys (CSUR) (2021)
  * [49] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. NeurIPS 33, 18661–18673 (2020)
  * [50] Kim, D., Yoo, Y., Park, S., Kim, J., Lee, J.: Selfreg: Self-supervised contrastive regularization for domain generalization. In: ICCV. pp. 9619–9628 (2021)
  * [51] Krueger, D., Caballero, E., Jacobsen, J.H., Zhang, A., Binas, J., Zhang, D., Priol, R.L., Courville, A.: Out-of-distribution generalization via risk extrapolation (rex). arXiv preprint (2020)
  * [52] Lahiri, A., Kwatra, V., Frueh, C., Lewis, J., Bregler, C.: Lipsync3d: Data-efficient learning of personalized 3d talking faces from video using pose and lighting normalization. In: CVPR. pp. 2755–2764 (2021)
  * [53] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444 (2015)
  * [54] LeCun, Y., Cortes, C., Burges, C.: Mnist handwritten digit database (2010)
  * [55] Lee, J., Kim, E., Lee, J., Lee, J., Choo, J.: Learning debiased representation via disentangled feature augmentation. NeurIPS 34 (2021)
  * [56] Lenssen, J.E., Fey, M., Libuschewski, P.: Group equivariant capsule networks. NeurIPS 31 (2018)
  * [57] Little, R.J., Rubin, D.B.: Statistical analysis with missing data, vol. 793. John Wiley & Sons (2019)
  * [58] Liu, Q., Mohamadabadi, B.B., El-Khamy, M., Lee, J.: Diversification is all you need: Towards data efficient image understanding (2020)
  * [59] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [60] Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)
  * [61] Mitchell, T.M.: The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research … (1980)
  * [62] Müller, R., Kornblith, S., Hinton, G.E.: When does label smoothing help? NeurIPS 32 (2019)
  * [63] Nam, H., Lee, H., Park, J., Yoon, W., Yoo, D.: Reducing domain gap by reducing style bias. In: CVPR. pp. 8690–8699 (2021)
  * [64] Nam, J., Cha, H., Ahn, S., Lee, J., Shin, J.: Learning from failure: De-biasing classifier from biased classifier. NeurIPS 33, 20673–20684 (2020)
  * [65] Van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv e-prints pp. arXiv–1807 (2018)
  * [66] Pan, S.J., Tsang, I.W., Kwok, J.T., Yang, Q.: Domain adaptation via transfer component analysis. IEEE transactions on neural networks 22(2), 199–210 (2010)
  * [67] Parascandolo, G., Kilbertus, N., Rojas-Carulla, M., Schölkopf, B.: Learning independent causal mechanisms. In: ICML. pp. 4036–4044 (2018)
  * [68] Pearl, J.: Causal diagrams for empirical research. Biometrika 82(4), 669–688 (1995)
  * [69] Pearl, J.: Causality. Cambridge university press (2009)
  * [70] Pezeshki, M., Kaba, O., Bengio, Y., Courville, A.C., Precup, D., Lajoie, G.: Gradient starvation: A learning proclivity in neural networks. NeurIPS 34, 1256–1272 (2021)
  * [71] Pfister, N., Bühlmann, P., Peters, J.: Invariant causal prediction for sequential data. Journal of the American Statistical Association 114(527), 1264–1276 (2019)
  * [72] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. Image 2, T2
  * [73] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)
  * [74] Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do imagenet classifiers generalize to imagenet? In: ICML. pp. 5389–5400. PMLR (2019)
  * [75] Saito, K., Kim, D., Sclaroff, S., Saenko, K.: Universal domain adaptation through self supervision. NeurIPS 33, 16282–16292 (2020)
  * [76] Salimans, T., Kingma, D.P.: Weight normalization: A simple reparameterization to accelerate training of deep neural networks. NeurIPS 29 (2016)
  * [77] Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., Mooij, J.: On causal and anticausal learning. In: ICML (2012)
  * [78] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [79] Shen, Z., Liu, J., He, Y., Zhang, X., Xu, R., Yu, H., Cui, P.: Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 (2021)
  * [80] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. NeurIPS 30 (2017)
  * [81] Sun, P., Jin, X., Su, W., He, Y., Xue, H., Lu, Q.: A visual inductive priors framework for data-efficient image classification. In: ECCV. pp. 511–520. Springer (2020)
  * [82] Sun, Q., Liu, Y., Chua, T.S., Schiele, B.: Meta-transfer learning for few-shot learning. In: CVPR. pp. 403–412 (2019)
  * [83] Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: Relation network for few-shot learning. In: CVPR. pp. 1199–1208 (2018)
  * [84] Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. arXiv preprint arXiv:1906.05849 (2019)
  * [85] Vapnik, V.: Principles of risk minimization for learning theory. In: NeurIPS (1992)
  * [86] Wang, T., Yue, Z., Huang, J., Sun, Q., Zhang, H.: Self-supervised learning disentangled group representation as feature. In: Conference and Workshop on Neural Information Processing Systems (NeurIPS) (2021)
  * [87] Wang, T., Zhou, C., Sun, Q., Zhang, H.: Causal attention for unbiased visual recognition. In: ICCV. pp. 3091–3100 (2021)
  * [88] Wen, Z., Li, Y.: Toward understanding the feature learning process of self-supervised contrastive learning. In: ICML. pp. 11112–11122. PMLR (2021)
  * [89] Xu, Y., Zhang, Q., Zhang, J., Tao, D.: Vitae: Vision transformer advanced by exploring intrinsic inductive bias. NeurIPS 34 (2021)
  * [90] You, K., Long, M., Cao, Z., Wang, J., Jordan, M.I.: Universal domain adaptation. In: CVPR. pp. 2720–2729 (2019)
  * [91] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV. pp. 818–833. Springer (2014)
  * [92] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [93] Zhang, X., Zhou, L., Xu, R., Cui, P., Shen, Z., Liu, H.: Towards unsupervised domain generalization. In: CVPR. pp. 4910–4920 (2022)
  * [94] Zhao, B., Wen, X.: Distilling visual priors from self-supervised learning. In: ECCV. pp. 422–429. Springer (2020)
  * [95] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR. pp. 2921–2929 (2016)
  * [96] Zhu, F., Cheng, Z., Zhang, X.y., Liu, C.l.: Class-incremental learning via dual augmentation. NeurIPS 34 (2021)

Appendix

This appendix is organized as follows:

  * $\circ$

Section 0.A presents mathematical definition for the equivariant
representation (Section 1) and derivation of the interventional ERM (Eq. (1)).

  * $\circ$

Section 0.B provides the implementation details of the Fig. 2 and Eq. (6). We
also elaborate the details of the used Colored MNIST dataset (Fig. 5) and the
NICO dataset (Section 5.1).

  * $\circ$

Section 0.C shows the results with MAE pretrained feature (Section 4); the
attention map visualizations (Section 5.4) and the algorithm complexities
(Section 5.4).

##  Appendix 0.A Mathematical Definition & Derivation

###  0.A.1 The Mathematical Definition of Equivariance

Let $\mathcal{U}$ be a set of (unseen) semantics, e.g., attributes such as
“digit” and “color”. There is a set of _independent and causal mechanisms_
[67] $\varphi:\mathcal{U}\to\mathcal{I}$, generating images from semantics,
e.g., writing a digit “0” when thinking of “0” [77]. A visual representation
is the inference process $\phi:\mathcal{I}\to\mathcal{X}$ that maps image
pixels to vector space features, e.g., a neural network. We define semantic
representation as the functional composition
$f:\mathcal{U}\to\mathcal{I}\to\mathcal{X}$. Let $\mathcal{G}$ be the group
acting on $\mathcal{U}$, i.e., $g\cdot u\in\mathcal{U}\times\mathcal{U}$
transforms $u\in\mathcal{U}$, e.g., a “turn green” group element changing the
semantic from “red” to “green”.

Definition 1. (Equivariant Representation) Suppose there is a direct product
decomposition $\mathcal{G}={g}_{1}\times\ldots\times{g}_{m}$ and
$\mathcal{U}=\mathcal{U}_{1}\times\ldots\times\mathcal{U}_{m}$, where
${g}_{i}$ acts on $\mathcal{U}_{i}$ respectively. A feature representation is
equivariant if there exists a group $\mathcal{G}$ acting on $\mathcal{X}$ such
that:

$f(g\cdot u)=g\cdot f(u),~{}~{}\forall g\in\mathcal{G},\forall
u\in\mathcal{U}$ (8)

e.g., the feature of the changed semantic: “red” to “green” in $\mathcal{U}$,
is equivalent to directly change the color vector in $\mathcal{X}$ from “red”
to “green”.

As stated in Section 3.2, we follow the definition and implementation in [86]
to achieve the sample-equvariant by using contrastive loss. Specifically, by
assuming $\mathbf{x}\in\mathcal{X}$ as the feature, we can write the
contrastive loss briefly as
$\ell=-\log\frac{\exp(\mathbf{x}^{T}_{i}\mathbf{x}_{j})}{\sum\nolimits_{\mathbf{x}\in\mathcal{X}}\exp(\mathbf{x}^{T}_{j}\mathbf{x})}$.
Then, if we use all the samples in the denominator of the loss, we can
approximate to $\mathcal{G}$-equivariant features given limited training
samples. This is because the loss minimization guarantees
$\forall(\mathbf{x}_{i},\mathbf{x}_{j})\in\mathcal{X}\times\mathcal{X},i\neq
j\to\mathbf{x}_{i}\neq\mathbf{x}_{j}$. We provide the proof in the following.

Suppose that the training loss $\ell$ is minimized, yet
$\exists\mathbf{x}_{a}=\mathbf{x}_{b}\in\mathcal{X}$ for $a\neq b$. Let
$\mathbf{x}_{i}\in\mathcal{X}$ in the denominator, and we have
$\mathbf{x}_{j}^{T}\mathbf{x}_{i}=\mathrm{cos}(\theta_{i,j})\left\lVert\mathbf{x}_{i}\right\rVert\left\lVert\mathbf{x}_{j}\right\rVert$,
where $\theta_{i,j}$ is the angle between the two vectors. When
$\mathbf{x}_{i}=\mathbf{x}_{j}$, $\mathrm{cos}(\theta_{i,j})=1$. So keeping
$\left\lVert\mathbf{x}_{i}\right\rVert\left\lVert\mathbf{x}_{j}\right\rVert$
constant (i.e., the same regularization penalty such as L2),
$\mathbf{x}_{j}^{T}\mathbf{x}_{i}$ can be further reduced if
$\mathbf{x}_{i}\neq\mathbf{x}_{j}$, which reduces the training loss. This
contradicts with the earlier assumption. Hence by minimizing the training
loss, we can achieve sample-equivariant, i.e., different samples have
different features. Note that this does not necessarily mean group-
equivariant. However, the variation of training samples is all we know about
the group action of $\mathcal{G}$, and we establish that the action of
$\mathcal{G}$ is transitive on $\mathcal{X}$, hence we use the sample-
equivariant features as the approximation of $\mathcal{G}$-equivariant
features.

###  0.A.2 Derivation of Eq. (1)

In this section, we will show the derivation for the backdoor adjustment
formula using the three rules of _do_ -calculus [69], whose detailed proof can
be found in [69, 68]. For a causal directed acyclic graph $\mathcal{G}$, let
$X,Y,Z$ and $W$ be arbitrary disjoint sets of nodes. We use
$\mathcal{G}_{\overline{X}}$ to denote the manipulated graph where all
incoming arrows to node $X$ are deleted. Similarly
$\mathcal{G}_{\underline{X}}$ represents the graph where outgoing arrows from
node $X$ are deleted. We use lower case $x,y,z$ and $w$ for specific values
taken by each set of nodes: $X=x,Y=y,Z=z$ and $W=w$. For any interventional
distribution compatible with $\mathcal{G}$, we have the following three rules:

Rule 1 Insertion/deletion of observations. If $(Y\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z|X,W)_{\mathcal{G}_{\overline{X}}}$:

$P(y|do(x),z,w)=P(y|do(x),w),$ (9)

Rule 2 Action/observation exchange. If $(Y\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z|X,W)_{\mathcal{G}_{\overline{X}\underline{Z}}}$,

$P(y|do(x),do(z),w)=P(y|do(x),z,w),$ (10)

Rule 3 Insertion/deletion of actions. If $(Y\mathchoice{\mathrel{\hbox
to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}Z|X,W)_{\mathcal{G}_{\overline{X}\overline{Z(W)}}}$,

$P(y|do(x),do(z),w)=P(y|do(x),w),$ (11)

where $Z(W)$ is the set of nodes in $Z$ that are not ancestors of any $W$-node
in $\mathcal{G}_{\overline{X}}$.

In our causal graph, the desired interventional distribution $P(Y|do(X))$ can
be derived by:

$\displaystyle P(Y|do(X))$ $\displaystyle=\sum_{z}P(Y|do(X),Z=z)P(Z=z|do(X)s)$
(12) $\displaystyle=\sum_{z}P(Y|do(X),Z=z)P(Z=z)$ (13)
$\displaystyle=\sum_{z}P(Y|X,Z=z)P(Z=z),$ (14)

where Eq. (12) follows the law of total probability; Eq. (13) uses Rule 3
given $S\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}X$ in $\mathcal{G}_{\overline{X}}$; Eq. (14)
uses Rule 2 to change the intervention term to observation as
$(Y\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern
2.0mu{\displaystyle\perp}}}{\mathrel{\hbox
to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptstyle\perp$\hss}\mkern
2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox
to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern
2.0mu{\scriptscriptstyle\perp}}}X|Z)$ in $\mathcal{G}_{\underline{X}}$.
Therefore, by imposing Eq. (14) into Eq. (1) of the main paper, we can have:

$\mathcal{R}=\sum_{x}\sum_{y}\sum_{z}\mathcal{L}(f(\phi(x)),y)P(y|x,z)P(z)P(x).$
(15)

##  Appendix 0.B Implementation Details

Figure 8: We illustrate (a) the example images of the Colored MNIST dataset;
(b) the generated environment $\mathcal{E}_{0}$ with adjusted similarity.

###  0.B.1 Implementation Details of Fig. 2

The t-SNE Visualization. We adopt the t-SNE visualization here to reflect the
true data distribution and expect the feature extraction model to be able to
accurately structure the relationships between images. Recently, Contrastive
Language-Image Pretraining (CLIP) [73] is proposed for solving vision tasks by
exploiting contrastive learning with very large-scale noisy image-text pairs.
Such large data makes it nearly a “Sufficient Data Situation”. And the
conventional ERM algorithm can achieve optimal performance, as we introduced
in Section 3 of the main paper. Indeed, CLIP achieves inspirational
performances on various visual classification tasks. In this paper, we utilize
the CLIP pretrained backbone (“ViT-Base/32”) to extract feature of the
training and testing images of class Hen which is randomly chosen. Then we
draw t-SNE visualization with the open-source
codebase111https://github.com/DmitryUlyanov/Multicore-TSNE.

The Test Accuracy. We trained a ResNet-50 model from scratch on each training
set and evaluated on testing images. We calculated accuracy on class Hen.

###  0.B.2 Implementation Details of Eq. (6)

In Section 4 Step 3, we propose the class-wise IRM to regularize the
invariance within each class as stated in Eq. (6). In practice, we adopt a
more practical version named REx [51] of Eq. (6) which improves the vanilla
IRM in the covariate shift situation. Specifically, [51] discards the dummy
classifier $w$ and changes the penalty term of IRM to the variance of risks as
the regularization:

$\mathcal{L}_{k}=\lambda\mathrm{Var(\\{\ell(1),...,\ell(e)\\})}+\sum\nolimits_{e\in\mathcal{E}_{k}}\ell(e).$
(16)

Same as IRM, REx aims to encourage the invariance across different
environments, but provides a simpler, stabler and more effective
implementation [51].

###  0.B.3 Details of Colored MNIST dataset in Fig. 5

Figure 8 (a) shows the example images of the Colored MNIST [64] dataset. As we
introduced in Section 4 of the main paper, the Colored MNIST dataset injects
color bias on each digit. There are 99.5% bias-aligned samples and only 0.5%
images are non-bias samples. For example, most of 0 are red. Figure 8 (b)
illustrates the generated environment $\mathcal{E}_{0}$ of anchor class 0 with
the adjusted similarity using real images. We can clearly observe that the
biased color c0 (i.e., red) of digit 0 distributes differently in Env#1 and
Env#2, while other semantics keep invariant. This will encourages the bias
color to be removed during the following class-wise IRM process.

Table 5: Recognition accuracies (%) on the VIPriors-10 and NICO datasets with
ViT-Base/16 as the feature backbone and MAE [32] as the SSL method. “Aug.”
represents augmentation. Our results are highlighted.

Model | VIPriors-10 | NICO
---|---|---
Val |  Test |  Val |  Test
Train from Scratch |  |  |  |
Baseline (ViT-Base/16) |  4.74 |  4.50 |  32.23 |  31.46
Random Aug. [24] |  5.40 |  4.92 |  33.54 |  31.92
Train from SSL |  |  |  |
MAE [32] |  16.04 |  14.63 |  54.10 |  52.29
\+ EqInv (Ours) |  16.93 |  15.48 |  56.26 |  52.29
MAE [32] \+ Random Aug. |  16.70 |  15.39 |  56.11 |  52.91
\+ EqInv (Ours) |  17.53 |  16.00 |  57.96 |  54.14

###  0.B.4 Experimental Details

More Implementation Details of the Equivariant Learning. As stated in the main
paper, we utilize different Self-Supervised Learning techniques in Step 1
Equivariant Learning. For implementation, we train for 800 epochs using
ResNet-50/-18 and ViT-Base/16 as the encoder architecture. We just follow the
original methods to use the default parameters and training schedules except
for some slight changes to adapt to the VIPriors and NICO dataset.
Specifically, we set the queue size as 16384 for MoCo-v2 [33, 18]. For the
NICO dataset, we train the MAE for 2000 epochs and adopt the mixup version of
IP-IRM [86] to achieve the reasonable performance. Moreover, please note that
we follow the other team’s solution [94] of VIPriors Challenge to use both
train and val set for the Stage 1 SSL pretraining with no need of the label
for all the comparison methods. Then for the second fine-tuning stage, as
stated in the main paper, we only use the train set images and labels for
training.

More Implementation Details of the Downstream Fine-tuning. In the main paper,
we have introduced most training schedules for ResNet model. Besides the
training schedules introduced in the main paper, we set $\lambda=2,10,100$ for
VIPriors-10, 20, 50 dataset. This parameter choice also makes sense from
intuitive since the demand of the invariance regularization is decreasing with
more training samples. Please note that, for fair comparison with data-
efficient learning methods, we did not apply any strong data augmentation in
our downstream training (after SSL), even though it is common in SSL works.
For MAE with ViT-Base/16, we follow the default end-to-end fine-tuning
schedule: AdamW as the optimizer with base learning rate 5e-4 using the cosine
learning rate decay; the layer-wise learning rate decay is set to 0.65 and
weight decay is set to 0.05; the drop path is set to 0.1 and the warmup epochs
are 5. We decrease the batch size to 256 and not use the advanced augmentation
(i.e., cutmix, mixup, label smoothing) to keep consistent with the ResNet
model. For our proposed class-wise IRM, the hidden size of the MLP $g$ is 512
with batch normalize layer and ReLU activation. The output dimension of $g$ is
128, same with SimCLR [17]. We also utilize the weight normailzation [76] on
the fc layer $f$ for the stable training.

##  Appendix 0.C Additional Results

###  0.C.1 Results with MAE [32] feature

Figure 9: Visual attention visualizations on NICO dataset with our proposed
EqInv and baseline methods. We adopt IP-IRM [86] as the SSL method. The red
box represents the failure case.

Table 5 shows the recognition accuracies on the VIPriors-10 and NICO datasets
with ViT-Base/16 as the feature backbone and MAE [32] as the SSL method.
Similar to the Table 1 in the main paper, we can observe that compared to
training from scratch, both imposing equivariance and invariance inductive
bias can boost the performance. However, we also find that the improvements of
considering invariance inductive bias are not such huge compared to that of
the ResNet structure. The possible reason is the Visual Transformer [27, 48]
structure itself is more robust [11, 7] to the distribution shift than the CNN
model (e.g., ResNet). That is, the self-attention-like architectures of Visual
Transformer have partially achieved the invariance.

###  0.C.2 Algorithm Complexities

Table 6: The model size and computational cost comparison between our proposed
EqInv and baseline models with different feature backbones.

Model |  Params (M) |  Flops (G) |  MACs (G) |  Time (s)
---|---|---|---|---
ResNet-18 | 11.180 | 3.644 | 1.822 | 0.025
\+ EqInv (Ours) | 11.510 | 3.646 | 1.823 | 0.105
ResNet-50 | 25.560 | 8.244 | 4.122 | 0.061
\+ EqInv (Ours) | 26.680 | 8.246 | 4.123 | 0.342
ViT-Base/16 | 86.570 | 33.712 | 16.856 | 0.065
\+ EqInv (Ours) | 87.030 | 33.712 | 16.856 | 0.297

We show the model sizes and the computational costs in Table 6. The “Time (s)”
denotes the forwarding process time with $bs$ images as input. $bs$ is set to
$128$ for ResNet and $64$ for ViT, based on GPU memory consumption. We can see
that compared to baseline models, deploying EqInv does not cause many extra
network parameters and computation costs. This is because in our EqInv, the
invariance inductive bias is implemented by only one learnable mask layer and
one MLP layer, bringing little overhead.

###  0.C.3 Attention Visualizations

Figure 10: Visual attention visualizations on VIPriors-10 dataset with our
proposed EqInv and baseline methods. We adopt IP-IRM [86] as the SSL method.
The red box represents the failure case.

Figure 9 and 10 show the qualitative attention map comparisons between
baseline (i.e., training from scratch), incorporating SSL pretraining and our
proposed EqInv. We utilize CAM [95] for the visualization. We can clearly
observe that:

  * $\circ$

Training from scratch (the second column) produces many inaccurate attentions,
even totally misses the object area (e.g., the last three rows of Fig. 9
left). This indicates the severe environmental bias of the model trained with
the insufficient samples (cf. Section 3.1 of the main paper).

  * $\circ$

Though incorporating SSL pretraining (the third column) greatly alleviates
such problem by imposing the equivariance inductive bias, the model still
attends to partial context area. It means the model may still be confounded by
the environmental feature during the downstream fine-tuning.

  * $\circ$

By further imposing the invariance inductive bias with our proposed EqInv (the
last column), the model can achieve much more accurate and tighter attention
focusing on the object area. We also display the failure cases in red boxes.
We can find that our EqInv cannot accurately attend to multiple objects (e.g.,
three sheeps and two bears in Fig. 9) or small objects (e.g., the flagpole in
Fig. 10). But our EqInv can still achieve relatively better attention maps
compared to comparison methods. We will explore such failure cases in the
future work.