# Supervised Masked Knowledge Distillation for Few-Shot Transformers

Han Lin , Guangxing Han∗†, Jiawei Ma, Shiyuan Huang, Xudong Lin, Shih-Fu Chang
Columbia University
{hl3199, gh2561, jiawei.m, sh3813, xl2798<EMAIL_ADDRESS>Equal
contribution.${}^{\hskip 2.84526pt\dagger}$Corresponding author.

###### Abstract

Vision Transformers (ViTs) emerge to achieve impressive performance on many
data-abundant computer vision tasks by capturing long-range dependencies among
local features. However, under few-shot learning (FSL) settings on small
datasets with only a few labeled data, ViT tends to overfit and suffers from
severe performance degradation due to its absence of CNN-alike inductive bias.
Previous works in FSL avoid such problem either through the help of self-
supervised auxiliary losses, or through the dextile uses of label information
under supervised settings. But the gap between self-supervised and supervised
few-shot Transformers is still unfilled. Inspired by recent advances in self-
supervised knowledge distillation and masked image modeling (MIM), we propose
a novel Supervised Masked Knowledge Distillation model (SMKD) for few-shot
Transformers which incorporates label information into self-distillation
frameworks. Compared with previous self-supervised methods, we allow intra-
class knowledge distillation on both class and patch tokens, and introduce the
challenging task of masked patch tokens reconstruction across intra-class
images. Experimental results on four few-shot classification benchmark
datasets show that our method with simple design outperforms previous methods
by a large margin and achieves a new start-of-the-art. Detailed ablation
studies confirm the effectiveness of each component of our model. Code for
this paper is available here: https://github.com/HL-hanlin/SMKD.

##  1 Introduction

Figure 1: Comparison of the proposed idea and other existing methods for few-
shot Transformers. Our model mitigates the overfitting issue of few-shot
Transformers, by extending the masked knowledge distillation framework into
the supervised setting, and enforcing the alignment of [cls] and corresponding
[patch] tokens for intra-class images.

Vision Transformers (ViTs) [19] have emerge as a competitive alternative to
Convolutional Neural Networks (CNNs) [35] in recent years, and have achieved
impressive performance in many vision tasks including image classification
[19, 45, 67, 74], object detection [5, 90, 13, 30, 32, 31], and object
segmentation [63, 57]. Compared with CNNs, which introduce inductive bias
through convolutional kernels with fixed receptive fields [41], the attention
layers in ViT allow it to model global token relations to capture long-range
token dependencies. However, such flexibility also comes at a cost: ViT is
data-hungry and it needs to learn token dependencies purely from data. This
property often makes it easy to overfit to datasets with small training set
and suffer from severe generalization performance degradation [44, 42].
Therefore, we are motivated to study how to make ViTs generalize well on these
small datasets, especially under the few-shot learning (FSL) setting [23, 46,
70] which aims to recognize unseen new instances at test time just from only a
few (e.g. one or five) labeled samples from each new categories.

Most of the existing methods mitigate the overfitting issue of few-shot
Transformers [17] using various regularizations. For instance, some works
utilize label information in a weaker [36], softer [49] way, or use label
information efficiently through patch-level supervision [17]. However, these
models usually design sophisticated learning targets. On the other hand, self-
distillation techniques [7, 11], and particularly, the recent masked self-
distillation [33, 88, 52], which distills knowledge learned from an
uncorrupted image to the knowledge predicted from a masked image, have lead an
emerging trend in self-supervised Transformers in various fields [18, 76].
Inspired by such success, recent works in FSL attempt to incorporate self-
supervised pretext tasks into the standard supervised learning through
auxiliary losses [44, 53, 50], or to adopt a self-supervised pretraining,
supervised training two-stage framework to train few-shot Transformers [37,
22]. Compared with traditional supervised methods, self-supervision can learn
less biased representations towards base class, which usually leads to better
generalization ability for novel classes [47]. However, the two learning
objectives of self-supervision and supervision are conflicting and it is hard
to balance them during training. Therefore, how to efficiently leverage the
strengths of self-supervised learning to alleviate the overfitting issue of
supervised training remains a challenge.

In this work, we propose a novel supervised masked knowledge distillation
framework (SMKD) for few-shot Transformers, which handles the aforementioned
challenge through a natural extension of the self-supervised masked knowledge
distillation framework into the supervised setting (shown in Fig. 1).
Different from supervised contrastive learning [39] which only utilizes global
image features for training, we leverage multi-scale information from the
images (both global [cls] token and local [patch] tokens) to formulate the
learning objectives, which has been demonstrated to be effective in the recent
self-supervised Transformer methods [88, 33]. For global [cls] tokens, we can
simply maximize the similarity for intra-class images. However, it is non-
trivial and challenging to formulate the learning objectives for local [patch]
tokens because we do not have ground-truth patch-level annotations. To address
this problem, we propose to estimate the similarity between [patch] tokens
across intra-class images using cross-attention, and enforce the alignment of
corresponding [patch] tokens. Particularly, reconstructing masked [patch]
tokens across intra-class images increases the difficulty of model learning,
thus encouraging learning generalizable few-shot Transformer models by jointly
exploiting the holistic knowledge of the image and the similarity of intra-
class images.

As shown in Fig. 2, we compare our model with the existing self-
supervised/supervised learning methods. Our model is a natural extension of
the supervised contrastive learning method [39] and self-supervised knowledge
distillation methods [88, 7]. Thus our model inherits both the advantage of
method [39] for effectively leveraging label information, and the advantages
of methods [88, 7] for not needing large batch size and negative samples.
Meanwhile, the newly-introduced challenging task of masked [patch] tokens
reconstruction across intra-class images makes our method more powerful for
learning generalizable few-shot Transformer models.

Figure 2: Comparison of other self-supervised/supervised frameworks. Our
method (d) is a natural extension of (b) and (c), with the newly-introduced
challenging task of masked [patch] tokens reconstruction across intra-class
images.

Compared with contemporary works on few-shot Transformers [36, 37, 17], our
framework enjoys several good properties from a practical point of view. (1)
Our method does not introduce any additional learnable parameters besides the
ViT backbone and projection head, which makes it easy to be combined with
other methods [36, 37, 83]. (2) Our method is both effective and training-
efficient, with stronger performance and less training time on four few-shot
classification benchmarks, compared with [36, 37]. In a nutshell, our main
contributions can be summarized as follows:

  * •

We propose a new supervised knowledge distillation framework (SMKD) that
incorporates class label information into self-distillation, thus filling the
gap between self-supervised knowledge distillation and traditional supervised
learning.

  * •

Within the proposed framework, we design two supervised-contrastive losses on
both class and patch levels, and introduce the challenging task of masked
patch tokens reconstruction across intra-class images.

  * •

Given its simple design, we test our SMKD on four few-shot datasets, and show
that it achieves a new SOTA on CIFAR-FS and FC100 by a large margin, as well
as competitive performance on _mini_ -ImageNet and _tiered_ -ImageNet using
the simple prototype classification method for few-shot evaluation.

##  2 Related Work

Few-shot Learning. Few-shot learning aims at fast knowledge transfer from seen
base classes to unseen novel classes given only a few labeled samples from
each novel class. The meta-learning paradigm, which simulates few-shot tasks
episodically to mimic the human-learning process in the real world, once
dominated FSL [64, 51, 60, 62, 70, 28, 26, 27, 29]. Surprisingly, recent works
[9, 66, 25, 81] have shown that the state-of-the-art meta-learning methods can
be outperformed by simple baseline methods using just a distance-based
classifier without the complicated design of meta-learning strategies.
Therefore, recent methods [21, 36, 49] in FSL start to focus less on meta-
learning, and more on learning embeddings with good generalization ability.
Our paper follows this trend and proposes a knowledge distillation framework
to learn generalizable embeddings.

Vision Transformers in FSL. Variants of Transformer [69] have achieved great
success in NLP [15, 56], computer vision [19, 67], and multimodal learning
[55]. However, the lack of inductive bias makes Transformer infamous for its
data-hungry property, which is especially significant in few-shot settings
when the amount of data is limited. A line of works focuses on introducing
inductive bias back to the Transformer architecture, including methods using
pyramid structure [72, 71], shifted windows [45], and explicit convolutional
token embeddings [74]. With this said, some recent works [37, 16, 80, 28]
still show that few-shot Transformers have the potential of fast adaptation to
novel classes. Our work also studies few-shot Transformers, and we show that
our method can work well even on the vanilla ViT structure without explicit
incorporation of inductive bias.

Self-Supervision for FSL. Self-supervised learning (SSL) has shown great
potential in FSL due to its good generalization ability to novel classes.
Previous methods incorporate SSL into FSL in various ways. Some works propose
to include self-supervised pretext tasks into the standard supervised learning
through auxiliary losses [44, 53, 50]. For example, [44] proposed a
regularization loss that extracts additional information from images by
predicting the geometric distance between patch tokens. [53] designed their
loss by discouraging spatially disordered attention maps based on the idea
that objects usually occupy connected regions, and [50] derived their
auxiliary loss from self-supervision tasks of rotation and exemplar. Some
other works [37, 22], instead, adopt a two-stage procedure by pretraining a
model via self-supervision before supervised training. [37] takes advantage of
self-supervised training with iBOT [88] as a pretext task, and then uses inner
loop token importance reweighting for supervised fine-tuning. [22] initializes
its model with a pre-trained self-supervised DINO [7], then trained with
supervised cross-entropy loss. Our work follows the second branch of work, but
the difference is obvious from previous works. Instead of designing
complicated training pipeline or leveraging additional learnable modules at
inference time, we use supervised training on self-supervised pre-trained
model, and focus mainly on bridging the gap between self-supervised and
supervised knowledge distillation with minimum extra design.

Another emerging trend in SSL is Masked Image Modeling (MIM) [88, 33, 52, 3],
which aims at recovering the patch-level target (e.g. image pixels, patch
features) of the masked content in a corrupted input image. In iBOT [88], the
class and patch tokens share the same projection head, thus helping the ViT
backbone to learn semantically meaningful patch embeddings, which can be used
as a supplement to image-level global self-supervision.

##  3 Our Approach

Figure 3: Overview of our SMKD framework. Two views are generated from a pair
of images sampled _(with replacement)_ from the same class. The first view,
applied with random blockwise masking, is passed to the student network, while
the second unmasked view is passed to the teacher network. Both networks
consist of a ViT backbone and a projection head. The parameters of the teacher
network are Exponentially Moving Averaged (EMA) updated by the student
network. SMKD distills knowledge between intra-class cross-views on both class
and patch levels. $\mathcal{L}_{[cls]}$ distills knowledge from [cls] tokens,
while $\mathcal{L}_{[patch]}$ distills knowledge from [patch] tokens by
finding dense correspondence of matched token pairs (connected by red dashed
lines) with highest similarities.

We present our Supervised Masked Knowledge Distillation (SMKD) framework in
this section. Fig. 3 shows an overview of our model. Task definition for few-
shot classification is presented in Sec. 3.1. We explain the formulation of
SSL knowledge distillation in Sec. 3.2, and how we extend it into our
supervised knowledge distillation framework in Sec. 3.3. Training pipeline is
given in Sec. 3.4.

###  3.1 Learning Task Definition

In few-shot classification, we are given two datasets $\mathcal{D}_{base}$ and
$\mathcal{D}_{novel}$ with disjoint class labels $\mathcal{C}_{base}$ and
$\mathcal{C}_{novel}$
($\mathcal{C}_{base}\cap\mathcal{C}_{novel}=\varnothing$). The base dataset
$\mathcal{D}_{base}$, which contains abundant labeled samples, is used to
train the feature extractor. And the novel dataset $\mathcal{D}_{novel}$ is
then used to sample episodes for prototype estimation and few-shot evaluation.
In a standard N-way K-shot task, each episode
$\mathcal{D}_{epi}=(\mathcal{D}_{S},\mathcal{D}_{Q})$ covers N classes from
$\mathcal{C}_{novel}$. The support dataset $\mathcal{D}_{S}$, which contains K
samples from each class, is used for class prototype estimation, and the query
dataset $\mathcal{D}_{Q}$ is then used for evaluation. This task aims at
correctly classifying $\mathcal{D}_{Q}$ into N classes from sampled episodes.
And the main focus of this paper is to train a feature extractor with good
generalization ability from $\mathcal{D}_{base}$.

###  3.2 Preliminary: SSL with Knowledge Distillation

Our work is inspired by the self-supervised knowledge distillation frameworks
proposed recently [7]. Specifically, given an input image $\boldsymbol{x}$
uniformly sampled from the training set ${\mathcal{I}}$, random data
augmentations are applied to generate two augmented views $\boldsymbol{x^{1}}$
and $\boldsymbol{x^{2}}$ (_we represent image view indices as superscripts and
patch indices as subscripts_), which are then fed into the teacher and student
networks. The student network, parameterized by $\boldsymbol{\theta_{s}}$,
consists of an encoder with ViT backbone and a projection head with a 3-layer
multi-layer perceptron (MLP) followed by $l_{2}$-normalized bottleneck. The
ViT backbone first generates a [cls] token, which is then entered into the
projection head and outputs a probability distribution
$\boldsymbol{P_{s}}^{\texttt{[cls]}}$ over K dimensions. The teacher network,
parameterized by $\boldsymbol{\theta_{t}}$, is Exponentially Moving Averaged
(EMA) updated by the student network $\boldsymbol{\theta_{s}}$, which distills
its knowledge to the student by minimizing the cross-entropy loss over the
outputs of the categorical distributions from their projection heads:

$\mathcal{L}_{\texttt{[cls]}}=\mathcal{H}(\boldsymbol{P_{t}}^{\texttt{[cls]}}(\boldsymbol{x}^{1}),\boldsymbol{P_{s}}^{\texttt{[cls]}}(\boldsymbol{x}^{2}))$
(1)

where $\mathcal{H}(x,y)=-x\log y$.

Masked Image Modeling (MIM) [3, 65] can be performed via self-distillation as
follows [88]. Given a randomly sampled mask sequence $m\in\\{0,1\\}^{N}$ over
an image with $N$ [patch] tokens
$\boldsymbol{x}=\\{\boldsymbol{x}_{i}\\}_{i=1}^{N}$, $\boldsymbol{x}_{i}$’s
with $m_{i}=1$ are then replaced by a learnable token embedding
$\boldsymbol{e}_{\texttt{[MASK]}}$, which results in a corrupted image
$\widehat{\boldsymbol{x}}=\\{\boldsymbol{\widehat{x}}_{i}\\}_{i=1}^{N}=\\{(1-m_{i})\boldsymbol{x}_{i}+m_{i}\boldsymbol{e}_{\texttt{[MASK]}}\\}_{i=1}^{N}$.
This corrupted image and the original uncorrupted image are fed into the
student and teacher networks respectively. The objective of MIM is to recover
the masked tokens from the corrupted image, which is equivalent to minimizing
the cross-entropy loss between the outputs of the categorical distributions of
the student and teacher networks on masked patches:

$\mathcal{L}_{\text{MIM}}=\sum_{i=1}^{N}m_{i}\cdot\mathcal{H}(\boldsymbol{P_{t}}^{\texttt{[patch]}}(\boldsymbol{x})_{i},\boldsymbol{P_{s}}^{\texttt{[patch]}}(\boldsymbol{\widehat{x}})_{i})\vspace{1mm}$
(2)

###  3.3 Supervised Masked Knowledge Distillation

#### Distill the Class Tokens.

Recent self-distillation frameworks [24, 7, 88] distill knowledge on [cls]
tokens from cross-view images via Eq.(1). To incorporate label information
into such self-supervised frameworks, we further allow knowledge on [cls]
tokens to be distilled from intra-class cross-views. This can be achieved by a
small extension of the way we sample images. Rather than sampling a single
image $\boldsymbol{x}\sim\mathcal{I}$ and generating two views, now we sample
two images $\boldsymbol{x},\boldsymbol{y}\sim\mathcal{I}^{c}$ _(with
replacement)_ and generate one view for each of them111For simplicity of
illustration, we show our method with only one augmented view from each image.
We sample two views from an image in our implementation. Loss is then averaged
over all cross-view pairs.. $\mathcal{I}^{c}\subseteq\mathcal{I}$ here denotes
the set of images with the same class label $c$ in the training set
$\mathcal{I}$. Specifically, we denote $\boldsymbol{{x^{\prime}}}$ and
$\boldsymbol{{y^{\prime}}}$ as the augmented views generated from image
$\boldsymbol{x}$ and $\boldsymbol{y}$ respectively. We apply additional random
blockwise masking on $\boldsymbol{{x^{\prime}}}$ and denote the resulting
corrupted view as $\boldsymbol{\widehat{x}^{\prime}}$. Then the corrupted view
$\boldsymbol{\widehat{x}^{\prime}}$ and uncorrupted view
$\boldsymbol{{y}^{\prime}}$ are sent to the student and teacher networks
seperately. Now our supervised-contrastive loss on [cls] tokens becomes:

$\mathcal{L}_{\texttt{[cls]}}=\mathcal{H}(\boldsymbol{P_{t}}^{\texttt{[cls]}}(\boldsymbol{y^{\prime}}),\boldsymbol{P_{s}}^{\texttt{[cls]}}(\boldsymbol{\widehat{x}}^{\prime}))$
(3)

When $\boldsymbol{x},\boldsymbol{y}$ are sampled as the same image
($\boldsymbol{x}=\boldsymbol{y}$), our loss performs _masked_ self-
distillation across two views of the same image similar to Eq.(1). In the
other situation when $\boldsymbol{x}$ and $\boldsymbol{y}$ represent different
images ($\boldsymbol{x}\neq\boldsymbol{y}$), our loss then minimizes the
cross-entropy loss of projected [cls] tokens among all cross-view pairs
between images $\boldsymbol{x}$ and $\boldsymbol{y}$.

Such design has two major advantages. (1) It can be implemented efficiently.
Instead of intentionally sampling image pairs from the same class, we just
need to look through the images in a mini-batch, find image pairs belonging to
the same class, and then apply our loss in Eq.(3). (2) Unlike previous works
that use either supervised [49, 87] or self-supervised [8, 34] contrastive
losses, our method follows the recent trend in SSL works [82, 10, 24, 7] and
avoids the need for negative examples.

Distill the Patch Tokens. Beyond the knowledge distillation on the global
[cls] tokens, we introduce the challenging task of masked patch tokens
reconstruction across intra-class images, to fully exploit the local details
of the image for training. Our main intuition here is based on the following
hypothesis: for intra-class images, even though their semantic information can
be vastly different at the patch level, there should at least exist some
patches that share similar semantic meanings.

For each patch $k$ from an input view $\boldsymbol{y^{\prime}}$ sent to the
teacher network (with its corresponding token embedding defined as
$\boldsymbol{f}^{\boldsymbol{t}}_{k}$), we need to first find the most similar
patch $k^{+}$ from the masked view $\boldsymbol{\widehat{x}^{\prime}}$ of the
student network (with its corresponding token embedding defined as
$\boldsymbol{f}^{\boldsymbol{s}}_{k^{+}}$), and then perform knowledge
distillation between these two matched tokens. The token embedding
$\boldsymbol{f}^{\boldsymbol{s}}_{k^{+}}$ represents either the ${k^{+}}^{th}$
[patch] token if patch $k^{+}$ is unmasked, or the reconstructed patch token
if it is masked.

As we do not have any patch-level annotations, we use cosine similarity to
find the best-matched patch of $k$ among all the [patch] tokens in the student
network:

$k^{+}=\arg\max_{l\in[N]}\frac{{\boldsymbol{f}^{\boldsymbol{t}}_{k}}^{\top}\boldsymbol{f}^{\boldsymbol{s}}_{l}}{\|{\boldsymbol{f}^{\boldsymbol{t}}_{k}}\|\|\boldsymbol{f}^{\boldsymbol{s}}_{l}\|}$
(4)

Our patch-level knowledge distillation loss now becomes:

$\displaystyle\mathcal{L}_{\texttt{[patch]}}=\sum_{k=1}^{N}{\omega_{k^{+}}}\cdot\mathcal{H}(\boldsymbol{P_{t}}^{\texttt{[patch]}}(\boldsymbol{{y}^{\prime}})_{k},\boldsymbol{P_{s}}^{\texttt{[patch]}}(\boldsymbol{\widehat{x}^{\prime}})_{k^{+}})$
(5)

where $\omega_{k^{+}}$ is a scalar representing the weight we give to each
loss term. We find that taking it as a constant value is effective enough. The
ablation study of more complex designs of $\omega_{k^{+}}$ is given in the
appendix.

Our loss shares some similarities with DenseCL [73]. However, the differences
are also obvious: (1) Our loss serves as an extension of their self-supervised
contrastive loss into a supervised variant. (2) We further incorporate MIM
into our design and allow masked patches to be matched, which makes our task
harder and leads to more semantically meaningful patch embeddings.

###  3.4 Training Pipeline

We train our model in two stages: self-supervised pretraining and supervised
training. In the first stage, we use the recently proposed MIM framework [88]
for self-supervised pretraining. The self-supervised loss is the summation of
$\mathcal{L}_{\texttt{[cls]}}$ and $\mathcal{L}_{\text{MIM}}$ in Eq.(1) and
Eq.(2) without scaling. In the second stage, we continue training the pre-
trained model using our supervised-contrastive losses
$\mathcal{L}_{\texttt{[cls]}}$ and $\mathcal{L}_{\texttt{[patch]}}$ from
Eq.(3) and Eq.(5). We define our training loss as:
$\mathcal{L}=\mathcal{L}_{\texttt{[cls]}}+\lambda\mathcal{L}_{\texttt{[patch]}}$,
where $\lambda$ controls the relative scale of these two components. A
relatively large $\lambda$ will make our model focus more on localization and
less on high-level semantics.

##  4 Experiment

###  4.1 Datasets

Our model is evaluated on four widely used and publicly available few-shot
classification datasets: _mini_ -ImageNet [70], _tiered_ -ImageNet [58],
CIFAR-FS [4], and FC100 [51]. _mini_ -ImageNet and _tiered_ -ImageNet are
derived from ImageNet [14], while CIFAR-FS and FC100 are derived from CIFAR100
[40]. _mini_ -ImageNet contains 100 classes, which are randomly split into 64
base classes for training, 16 classes for few-shot validation, and 20 classes
for few-shot evaluation. There are 600 images in each class. _tiered_
-ImageNet contains 609 classes with 779165 images in total. The class split
for training, few-shot validation and few-shot evaluation are 351, 97, and
160. CIFAR-FS contains 100 classes with class split as 64, 16, and 20. FC100
contains 100 classes with class split as 60, 20, and 20. For both of these two
datasets, each class has 600 images with smaller resolutions (32 $\times$ 32)
compared with ImageNet.

###  4.2 Implementation Details

Self-supervised pretraining. We pre-train our Vision Transformer backbone and
projection head following the same pipeline in iBOT [88]. Most of the hyper-
parameter settings are kept unchanged without tuning. We use a batch size of
640 and a learning rate of 0.0005 decayed with the cosine schedule. _mini_
-ImageNet and _tiered_ -ImageNet are pre-trained for 1200 epochs, and CIFAR-FS
and FC100 are pre-trained for 900 epochs. All models are trained on 8 Nvidia
RTX 3090 GPUs. Detailed training parameter settings are included in the
appendix.

Supervised knowledge distillation After getting the pre-trained model, we
train it with our supervised-contrastive losses. We find that the model with
the best performance can converge within 60 epochs in the validation set. We
use the same batch size and learning rate as in the pretraining stage.
Compared with the first stage, the only extra hyper-parameter is the scaling
parameter $\lambda$. We set it to make the ratio between
$\mathbf{\mathcal{L}_{\texttt{[patch]}}}$ and
$\mathbf{\mathcal{L}_{\texttt{[cls]}}}$ roughly around 2. Ablation over
different $\lambda$ is given in the appendix.

Table 1: Results on mini-ImageNet and tiered-ImageNet. Top three methods are
colored in  red,  blue and  green respectively. A more comprehensive version
of this table is shown in the appendix.

3 Method | Backbone | #Params | miniImageNet,5-way | tieredImageNet,5-way
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
DeepEMD [83] | _ResNet-12_ | 12.4M | ${{65.91\pm 0.82}}$ | ${{82.41\pm 0.56}}$ | ${{71.16\pm 0.87}}$ | ${{86.03\pm 0.58}}$
IE [59] | _ResNet-12_ | 12.4M | ${{67.28\pm 0.80}}$ | ${{84.78\pm 0.52}}$ | ${{72.21\pm 0.90}}$ | ${{87.08\pm 0.58}}$
COSOC [48] | _ResNet-12_ | 12.4M | ${{69.28\pm 0.49}}$ | ${{85.16\pm 0.42}}$ | ${{73.57\pm 0.43}}$ | ${{87.57\pm 0.10}}$
Meta-QDA [85] | _WRN-28-10_ | 36.5M | ${{67.38\pm 0.55}}$ | ${{84.27\pm 0.75}}$ | ${{74.29\pm 0.66}}$ | ${{89.41\pm 0.77}}$
OM [54] | _WRN-28-10_ | 36.5M | ${{66.78\pm 0.30}}$ | ${{85.29\pm 0.41}}$ | ${{71.54\pm 0.29}}$ | ${{87.79\pm 0.46}}$
SUN [17] | _ViT_ | 12.5M | ${{67.80\pm 0.45}}$ | ${{83.25\pm 0.30}}$ | ${{72.99\pm 0.50}}$ | ${{86.74\pm 0.33}}$
FewTURE [37] | _Swin-Tiny_ | 29.0M | ${{72.40\pm 0.78}}$ | ${{86.38\pm 0.49}}$ | ${{76.32\pm 0.87}}$ | ${{89.96\pm 0.55}}$
HCTransformers [36] | 3$\times$_ViT-S_ | 63.0M | ${\color[rgb]{0,0,1}{74.74\pm 0.17}}$ | ${\color[rgb]{0,0,1}{89.19\pm 0.13}}$ | ${\color[rgb]{0,0,1}{79.67\pm 0.20}}$ | ${\color[rgb]{1,0,0}{91.72\pm 0.11}}$
Ours (Prototype) | _ViT-S_ | 21M | ${\color[rgb]{0,1,0}{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$ | ${\color[rgb]{0,1,0}{78.83\pm 0.20}}$ | ${{91.02\pm 0.12}}$
Ours (Classifier) | _ViT-S_ | 21M | ${{74.10\pm 0.17}}$ | ${\color[rgb]{0,1,0}{88.89\pm 0.09}}$ | ${{78.81\pm 0.21}}$ | ${\color[rgb]{0,1,0}{91.21\pm 0.11}}$
Ours + HCT [36] | 3$\times$_ViT-S_ | 63M | ${\color[rgb]{1,0,0}{75.32\pm 0.18}}$ | ${\color[rgb]{1,0,0}{89.57\pm 0.09}}$ | ${\color[rgb]{1,0,0}{79.74\pm 0.20}}$ | ${\color[rgb]{0,0,1}{91.68\pm 0.11}}$
3 |  |  |  |  |  |

Table 2: Results on CIFAR-FS and FC100. A more comprehensive version of this
table is shown in the appendix.

3 Method | Backbone | #Params | CIFAR-FS,5-way | FC100,5-way
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
BML [89] | _ResNet-12_ | 12.4M | ${{73.45\pm 0.47}}$ | ${{88.04\pm 0.33}}$ | ${{45.00\pm 0.41}}$ | ${{63.03\pm 0.41}}$
IE [59] | _ResNet-12_ | 12.4M | ${{77.87\pm 0.85}}$ | ${{89.74\pm 0.57}}$ | ${{47.76\pm 0.77}}$ | ${{65.30\pm 0.76}}$
TPMN [75] | _ResNet-12_ | 12.4M | ${{75.50\pm 0.90}}$ | ${{87.20\pm 0.60}}$ | ${{46.93\pm 0.71}}$ | ${{63.26\pm 0.74}}$
PSST [12] | _WRN-28-10_ | 36.5M | ${{77.02\pm 0.38}}$ | ${{88.45\pm 0.35}}$ | - | -
Meta-QDA [85] | _WRN-28-10_ | 36.5M | ${{75.95\pm 0.59}}$ | ${{88.72\pm 0.79}}$ | - | -
SUN [17] | _ViT_ | 12.5M | ${{78.37\pm 0.46}}$ | ${{88.84\pm 0.32}}$ | - | -
FewTURE [37] | _Swin-Tiny_ | 29.0M | ${{77.76\pm 0.81}}$ | ${{88.90\pm 0.59}}$ | ${{47.68\pm 0.78}}$ | ${{63.81\pm 0.75}}$
HCTransformers [36] | 3$\times$_ViT-S_ | 63.0M | ${\color[rgb]{0,1,0}{78.89\pm 0.18}}$ | ${\color[rgb]{0,1,0}{90.50\pm 0.09}}$ | ${\color[rgb]{0,1,0}{48.27\pm 0.15}}$ | ${\color[rgb]{0,1,0}{66.42\pm 0.16}}$
Ours (Prototype) | _ViT-S_ | 21M | ${\color[rgb]{1,0,0}{80.08\pm 0.18}}$ | ${\color[rgb]{0,0,1}{90.63\pm 0.13}}$ | ${\color[rgb]{1,0,0}{50.38\pm 0.16}}$ | ${\color[rgb]{0,0,1}{68.37\pm 0.16}}$
Ours (Classifier) | _ViT-S_ | 21M | ${\color[rgb]{0,0,1}{79.82\pm 0.18}}$ | ${\color[rgb]{1,0,0}{90.91\pm 0.13}}$ | ${\color[rgb]{0,0,1}{50.28\pm 0.16}}$ | ${\color[rgb]{1,0,0}{68.50\pm 0.16}}$
3 |  |  |  |  |  |

Few-shot Evaluation We use the simple prototype classification method
(_Prototype_) [62, 51] and the linear classifier (_Classifier_) used in S2M2
[50] as our default methods for few-shot evaluation. For each sampled episode
data in an N-way K-shot task, _Prototype_ first estimates each class prototype
using the averaged feature over K support samples. Then a new sampled query
image is classified into one of the N classes which has the highest cosine
similarity between its feature vector and the class prototype. Instead,
_Classifier_ trains a linear classifier from the N$\times$K support samples,
which is then used to classify new query samples. More complex evaluation
methods (e.g. DeepEMD) are also compatible with our framework. The feature we
used for evaluation is the concatenation of the [cls] token with the weighted
average [patch] token (_weighted avg pool_). The weights of _weighted avg
pool_ is the average of the self-attention values of the [cls] token with all
heads of the last attention layer. Ablation over different choices of features
for evaluation is studied in Tab. 7.

###  4.3 Comparison With the State-of-the-arts (SOTAs)

We evaluate our proposed SMKD with the above-mentioned evaluation methods on
four few-shot classification datasets. Our goal in this section is to prove
the effectiveness of our method given its simple training pipeline and
evaluation procedure. This being said, the performance of our method can be
further boosted by adopting strategies from contemporary works (see the last
row "Ours+HCT[36]" in Table 1 and detailed results in Table 3).

Compared with traditional convolutional backbones, Transformer-based models
are still underdeveloped in few-shot classification. Recent methods with
Transformer backbones [37, 36, 17] differ a lot in their training pipelines
and evaluation procedures. SUN [17] first pre-trains a teacher network with
supervised loss, then uses it to generate patch-level supervision for the
student network as a supplement to class-level supervision. HCTransformers
[36] converts class-level prediction into a latent prototype learning problem,
and introduces spectral tokens pooling to merge neighboring tokens with
similar semantic meanings adaptively. FewTURE [37], which adopts a similar
two-stage training pipeline as ours, shows the effectiveness of inner loop
token importance reweighting to avoid supervision collapse.

Despite the success of these works, our SMKD still shows competitive
performance given its simple design. Table 1 contains the results of _mini_
-ImageNet and _tiered_ -ImageNet. With _Prototype_ and _Classifier_ as the
evaluation method, our proposed SMKD outperforms all methods with _ResNet_ and
_WRN_ backbones, and ranked second among methods with Transformer backbones.
By adopting the same strategies (patch size of 8 and spectral tokens pooling)
as in HCTransformers[36], our method achieves a new SOTA on _mini_ -ImageNet.
The effect of each of these two strategies is shown in Table 3. Furthermore,
as displayed in Table 2, our method performs the best on the two small-
resolution datasets (CIFAR-FS and FC100), and outperforms all previous results
by great margins (0.93% for 1-shot and 0.41% for 5-shot on CIFAR-FS, and over
2% for both 1 and 5-shot on FC100). _In conclusion, our method grows more
effective on datasets with smaller resolutions and fewer training images._

Table 3: Results for small patch size and spectral tokens pooling. Results are
evaluated on _mini_ -ImageNet with _Prototype_ and _Classifier_ methods, and
the best is reported.

Method | Patch 8 | Spec. Pool | 1-shot | 5-shot
---|---|---|---|---
HCT [36] | ✓ |  | ${{71.27\pm 0.17}}$ | ${{84.68\pm 0.10}}$
HCT [36] | ✓ | ✓ | ${{74.62\pm 0.20}}$ | ${{89.19\pm 0.13}}$
Ours |  |  | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$
Ours | ✓ |  | ${{74.00\pm 0.17}}$ | ${\mathbf{89.57\pm 0.09}}$
Ours | ✓ | ✓ | $\mathbf{75.32\pm 0.18}$ | $89.23\pm 0.18$

The top competitor to our model is HCTransformer [36]. However, their method
is more complex compared with ours in the following two aspects: (1) Their
method uses a smaller patch size of 8 rather than 16 (which is used in most
other methods as well as ours). As shown in Table 4, this makes their training
procedure 4$\times$ slower than ours in each epoch. (2) Their method contains
three sets of cascaded transformers, each corresponding to a set of teacher-
student network with ViT-S backbone. This triples the total number of
learnable parameters (63M versus 21M in ours) and incurs much longer inference
time.

Table 4: Training clock time comparison on _mini_ -ImageNet. For HCT, we
report the result provided in their paper. All experiments are conducted on 8
Nvidia RTX 3090 GPUs.

| Stage1 | Stage2 | Stage3
---|---|---|---
HCT [36] | 21.1h (400 epoch) | 0.58h (2 epoch) | 0.21h (2 epoch)
| Self-Supervised Pre-train | Supervised Training
Ours | 15.0h (1200 epoch) | 1.08h (60 epoch)

For a fair comparison, we compare few-shot classification results of methods
with the same ViT-S backbone (21M parameters) on _mini_ -ImageNet in Table 5.
The results for HCT[36] and FewTURE[37] are copy-pasted from their paper. Our
model outperforms them by a large margin when the number of trainable
parameters is comparable.

Visualizations of the multi-head self-attention and patch token correspondence
are shown in Fig. 4 and Fig. 5.

Table 5: Comparison results with the same ViT-S backbone on _mini_ -ImageNet.
For HCT, we report their result of the first student transformer (training
time shown in Stage 1 of Table 4).

Method | Backbone | 1-shot | 5-shot
---|---|---|---
HCT [36] | ViT-S | $71.27\pm 0.17$ | $84.68\pm 0.10$
FewTURE [37] | ViT-S | $68.02\pm 0.88$ | $84.51\pm 0.53$
Ours | ViT-S | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$

Figure 4: Visualization of multi-head self-attention maps. The self-attention
of the [cls] tokens with different heads in the last attention layer of ViT
are visualized in different colors. iBOT+CE represents the model first pre-
trained with iBOT, then trained with CE loss. Our SMKD pays more attention to
the foreground objects, especially the most discriminative parts. Figure 5:
Visualization of dense correspondence. We use the patches with the highest
self-attention of the [cls] token on each attention head (6 in total) of the
last layer of ViT-S as queries. Best matched patches with highest similarities
are connected with lines.

###  4.4 Ablation Study

Self-Supervised MIM Pretraining Helps. Our training pipeline contains two
stages. A natural question is whether the first self-supervised pretraining
stage is necessary. We can see from Table 6 that the classification accuracy
of our method drops a lot without pretraining. If we train in a single stage,
with the combination of the self-supervised learning losses and supervised
contrastive learning losses, the model is still hard to converge without a
good initialization (1-shot: 43.71 vs. our 74.28. 5-shot: 60.27 vs. our 88.82
on _mini_ -ImageNet). An explanation is that it will be hard for our patch-
level knowledge distillation to find matched patches if the ViT backbone is
initialized with random weights. What we observe is that our model finds some
shortcut solution that prohibits it from proper training. Additionally, we
also test the traditional CE loss with and without self-supervised
pretraining. The model pre-trained with self-supervision and then trained with
supervised CE loss works surprisingly well and even outperforms most of the
methods in Table 1 on _mini_ -ImageNet. This further supports the usefulness
of our two-stage training procedure. For the masking strategy in the first
pretraining stage, our ablation study over three masking types (blockwise,
random, and no mask) in the appendix shows that blockwise masking balances the
best between 1 and 5-shots classification accuracy.

Table 6: Few-shot evaluation w/wo self-supervised pretraining. For models with
pretraining stage, we pre-train with iBOT for 1200 epochs, then train with CE
or our loss for 60 epochs. For models without pretraining stage, we train with
supervised losses for 60 epochs directly. Results are evaluated by _Prototype_
classification method on _mini_ -ImageNet.

Method | Pre-train | 1-shot | 5-shot
---|---|---|---
CE |  | ${{48.98\pm 0.16}}$ | ${{67.62\pm 0.14}}$
CE | ✓ | ${{71.02\pm 0.18}}$ | ${{87.25\pm 0.10}}$
Ours |  | ${{33.14\pm 0.13}}$ | ${{45.02\pm 0.14}}$
Ours | ✓ | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$

Table 7: Ablation of different losses in the supervised training stage. All
losses are trained for 60 epochs initialized with the pre-trained iBOT model.
The first row represents the pre-trained iBOT without additional supervised
training. ${{\mathcal{L}_{CE}}+{\mathcal{L}_{\texttt{[patch]}}}}$ stands for
the combination of CE loss and our patch-level loss. Results are evaluated by
_Prototype_ method on _mini_ -ImageNet.

Loss | 1-shot | 5-shot
---|---|---
- | ${{60.93\pm 0.17}}$ | ${{80.38\pm 0.12}}$
${{\mathcal{L}_{CE}}}$ | ${{71.02\pm 0.18}}$ | ${{87.25\pm 0.10}}$
${{\mathcal{L}_{\texttt{[cls]}}}}$ | ${{70.21\pm 0.17}}$ | ${{87.03\pm 0.10}}$
${{\mathcal{L}_{\texttt{[patch]}}}}$ | ${{70.84\pm 0.18}}$ | ${{85.90\pm 0.11}}$
${{\mathcal{L}_{CE}}+{\mathcal{L}_{\texttt{[patch]}}}}$ | ${{70.70\pm 0.18}}$ | ${{86.77\pm 0.10}}$
${{\mathcal{L}_{[\texttt{cls}]}}+{\mathcal{L}_{\texttt{[patch]}}}}$ | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$

Class and Patch-Level Distillation Complement Each Other. Our SMKD uses the
combination of class-level and patch-level supervised-contrastive losses as
the final objective. Here we study several different combinations of
supervised losses to confirm our proposed method performs the best. Results in
Table 7 show that combining $\mathcal{L}_{\texttt{[cls]}}$ with
$\mathcal{L}_{\texttt{[patch]}}$ achieves 13.35% improvement over baseline for
1-shot and 8.44% for 5-shot classification, which outperforms each of its
components. Sharing the parameters of projection heads for class and patch
tokens allows the semantics obtained from their distillation to complement
each other, which leads to much better performance. This can also be seen from
the comparison between $\mathcal{L}_{CE}+\mathcal{L}_{\texttt{[patch]}}$ and
$\mathcal{L}_{\texttt{[cls]}}+\mathcal{L}_{\texttt{[patch]}}$.
$\mathcal{L}_{CE}$ alone achieves stronger performance than
$\mathcal{L}_{\texttt{[cls]}}$, but combining it with
$\mathcal{L}_{\texttt{[patch]}}$ does not provide additional benefits with
separate classification heads.

Weighted Average Pooling Boosts Performance. Some recent works [77, 68] find
that using average pooling instead of the [cls] token during evaluation can
encourage token-level tasks (e.g. localization, segmentation). Inspired by
such findings, we evaluate our model with different tokens as well as their
combinations. Results from Table 8 show that the performance of the [cls]
token can be boosted by 3.02% for 1-shot and 0.79% for 5-shot by just
concatenating with the weighted average pooling token. This shows that our
proposed method can learn meaningful representations for both [cls] and
[patch] tokens. Moreover, weighted average pooling and average pooling tokens
share similar information, but the former performs better because it gives
less weight to the background.

Table 8: Ablation of different tokens for few-shot evaluation. "cls" stands
for the [cls] token, "avg pool" stands for using average pooling as token,
"weighted avg pool" stands for weighted average [patch] token with weights
from the self-attention module from the last block of ViT. The last four rows
represent concatenation of these three tokens. All tokens are normalized to
unit length. Results are evaluated by _Prototype_ classification method.

Embedding | 1-shot | 5-shot
---|---|---
\small1⃝ cls | ${{71.26\pm 0.17}}$ | ${{88.03\pm 0.10}}$
\small2⃝ avg pool | ${{71.65\pm 0.19}}$ | ${{86.40\pm 0.11}}$
\small3⃝ weighted avg pool | ${{71.83\pm 0.19}}$ | ${{86.38\pm 0.11}}$
\small1⃝ + \small2⃝ | ${{73.83\pm 0.18}}$ | ${{88.58\pm 0.10}}$
\small1⃝ + \small3⃝ | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$
\small2⃝ + \small3⃝ | ${{71.86\pm 0.19}}$ | ${{86.47\pm 0.11}}$
\small1⃝ + \small2⃝ + \small3⃝ | ${{73.80\pm 0.18}}$ | ${{88.26\pm 0.10}}$

##  5 Conclusion

In this work, we propose a novel supervised knowledge distillation framework
(SMKD) for few-shot Transformers, which extends the self-supervised masked
knowledge distillation framework into the supervised setting. With our design
of supervised-contrastive losses, we incorporate supervision into both class
and patch-level knowledge distillation while still enjoy the benefits of not
needing large batch size and negative samples. Evaluation results together
with ablation studies demonstrate the superiority of our method given its
simple design compared with contemporary works. Our two-stage training is a
special case of curriculum learning from the easy samples to the hard ones. We
can unify the learning objectives of self-supervised learning and supervised
contrastive learning, using a carefully designed curriculum learning strategy
for future works. We hope our work can bridge the gap between self-supervised
and supervised knowledge distillation, and inspire more works on supervised
few-shot learning methods.

##  6 Acknowledgements

This research is sponsored by Air Force Research Laboratory (AFRL) under
agreement number FA8750-19-1-1000. The U.S. Government is authorized to
reproduce and distribute reprints for Government purposes notwithstanding any
copyright notation therein. The views and conclusions contained herein are
those of the authors and should not be interpreted as necessarily representing
the official policies or endorsements, either expressed or implied, of Air
Force Laboratory, DARPA or the U.S. Government.

## References

  * [1] Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné.  Mixture-based feature space learning for few-shot image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9041–9051, 2021.
  * [2] Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee.  Meta-learning with task-adaptive loss function for few-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9465–9474, 2021.
  * [3] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [4] Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi.  Meta-learning with differentiable closed-form solvers.  arXiv preprint arXiv:1805.08136, 2018.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in neural information processing systems, 33:9912–9924, 2020.
  * [7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [9] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  arXiv preprint arXiv:1904.04232, 2019.
  * [10] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [11] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021.
  * [12] Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, and Donglin Wang.  Pareto self-supervised training for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13663–13672, 2021.
  * [13] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen.  Up-detr: Unsupervised pre-training for object detection with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1601–1610, 2021.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [16] Carl Doersch, Ankush Gupta, and Andrew Zisserman.  Crosstransformers: spatially-aware few-shot transfer.  Advances in Neural Information Processing Systems, 33:21981–21993, 2020.
  * [17] Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo.  Self-promoted supervision for few-shot transformer.  In European Conference on Computer Vision (ECCV), 2022.
  * [18] Xiaoyi Dong, Yinglin Zheng, Jianmin Bao, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al.  Maskclip: Masked self-distillation advances contrastive language-image pretraining.  arXiv preprint arXiv:2208.12262, 2022.
  * [19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [20] Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang.  Melr: Meta-learning via modeling episode-level relationships for few-shot learning.  In International Conference on Learning Representations, 2020.
  * [21] Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao.  Rethinking supervised pre-training for better downstream transferring.  arXiv preprint arXiv:2110.06014, 2021.
  * [22] Hanan Gani, Muzammal Naseer, and Mohammad Yaqub.  How to train vision transformer on small-scale datasets?  arXiv preprint arXiv:2210.07240, 2022.
  * [23] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord.  Boosting few-shot visual learning with self-supervision.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 8059–8068, 2019.
  * [24] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [25] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris.  A broader study of cross-domain few-shot learning.  In European conference on computer vision, pages 124–141. Springer, 2020.
  * [26] Guangxing Han, Yicheng He, Shiyuan Huang, Jiawei Ma, and Shih-Fu Chang.  Query adaptive few-shot object detection with heterogeneous graph convolutional networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3263–3272, October 2021.
  * [27] Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, and Shih-Fu Chang.  Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 780–789, 2022.
  * [28] Guangxing Han, Jiawei Ma, Shiyuan Huang, Long Chen, and Shih-Fu Chang.  Few-shot object detection with fully cross-transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5321–5330, June 2022.
  * [29] Guangxing Han, Jiawei Ma, Shiyuan Huang, Long Chen, Rama Chellappa, and Shih-Fu Chang.  Multimodal few-shot object detection with meta-learning based cross-modal prompting.  arXiv preprint arXiv:2204.07841, 2022.
  * [30] Guangxing Han, Xuan Zhang, and Chongrong Li.  Revisiting faster r-cnn: A deeper look at region proposal network.  In International Conference on Neural Information Processing, pages 14–24, 2017.
  * [31] Guangxing Han, Xuan Zhang, and Chongrong Li.  Single shot object detection with top-down refinement.  In 2017 IEEE International Conference on Image Processing (ICIP), pages 3360–3364. IEEE, 2017.
  * [32] Guangxing Han, Xuan Zhang, and Chongrong Li.  Semi-supervised dff: Decoupling detection and feature flow for video object detectors.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1811–1819, 2018.
  * [33] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [34] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [36] Yangji He, Weihan Liang, Dongyang Zhao, Hong-Yu Zhou, Weifeng Ge, Yizhou Yu, and Wenqiang Zhang.  Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9119–9129, 2022.
  * [37] Markus Hiller, Rongkai Ma, Mehrtash Harandi, and Tom Drummond.  Rethinking generalization in few-shot classification.  arXiv preprint arXiv:2206.07267, 2022.
  * [38] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho.  Relational embedding for few-shot classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8822–8833, 2021.
  * [39] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  Advances in Neural Information Processing Systems, 33:18661–18673, 2020.
  * [40] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [41] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [42] Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song.  Vision transformer for small-size datasets.  arXiv preprint arXiv:2112.13492, 2021.
  * [43] Junjie Li, Zilei Wang, and Xiaoming Hu.  Learning intact features by erasing-inpainting for few-shot classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8401–8409, 2021.
  * [44] Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai.  Efficient training of visual transformers with small datasets.  Advances in Neural Information Processing Systems, 34:23818–23830, 2021.
  * [45] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [46] Jiang Lu, Pinghua Gong, Jieping Ye, and Changshui Zhang.  Learning from very few samples: A survey.  arXiv preprint arXiv:2009.02653, 2020.
  * [47] Yuning Lu, Liangjian Wen, Jianzhuang Liu, Yajing Liu, and Xinmei Tian.  Self-supervision can be a good few-shot learner.  arXiv preprint arXiv:2207.09176, 2022.
  * [48] Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian.  Rectifying the shortcut learning of background: Shared object concentration for few-shot image recognition.  arXiv preprint arXiv:2107.07746, 2021.
  * [49] Jiawei Ma, Hanchen Xie, Guangxing Han, Shih-Fu Chang, Aram Galstyan, and Wael Abd-Almageed.  Partner-assisted learning for few-shot image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10573–10582, 2021.
  * [50] Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian.  Charting the right manifold: Manifold mixup for few-shot learning.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2218–2227, 2020.
  * [51] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste.  Tadam: Task dependent adaptive metric for improved few-shot learning.  Advances in neural information processing systems, 31, 2018.
  * [52] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei.  A unified view of masked image modeling.  arXiv preprint arXiv:2210.10615, 2022.
  * [53] Elia Peruzzo, Enver Sangineto, Yahui Liu, Marco De Nadai, Wei Bi, Bruno Lepri, and Nicu Sebe.  Spatial entropy regularization for vision transformers.  arXiv preprint arXiv:2206.04636, 2022.
  * [54] Guodong Qi, Huimin Yu, Zhaohui Lu, and Shuzhao Li.  Transductive few-shot classification on the oblique manifold.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8412–8422, 2021.
  * [55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [56] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [57] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12179–12188, 2021.
  * [58] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel.  Meta-learning for semi-supervised few-shot classification.  arXiv preprint arXiv:1803.00676, 2018.
  * [59] Mamshad Nayeem Rizve, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.  Exploring complementary strengths of invariant and equivariant representations for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10836–10846, June 2021.
  * [60] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  arXiv preprint arXiv:1807.05960, 2018.
  * [61] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4136–4145, 2020.
  * [62] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [63] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7262–7272, 2021.
  * [64] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199–1208, 2018.
  * [65] Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal.  Vimpac: Video pre-training via masked token prediction and contrastive learning.  arXiv preprint arXiv:2106.11250, 2021.
  * [66] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: a good embedding is all you need?  In European Conference on Computer Vision, pages 266–282. Springer, 2020.
  * [67] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  * [68] Hugo Touvron, Matthieu Cord, and Hervé Jégou.  Deit iii: Revenge of the vit.  arXiv preprint arXiv:2204.07118, 2022.
  * [69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [70] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29, 2016.
  * [71] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568–578, 2021.
  * [72] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pvt v2: Improved baselines with pyramid vision transformer.  Computational Visual Media, 8(3):415–424, 2022.
  * [73] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024–3033, 2021.
  * [74] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22–31, 2021.
  * [75] Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu.  Task-aware part mining network for few-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8433–8442, 2021.
  * [76] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik.  Masked visual pre-training for motor control.  arXiv preprint arXiv:2203.06173, 2022.
  * [77] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  arXiv preprint arXiv:2109.14084, 2021.
  * [78] Weijian Xu, Yifan Xu, Huaijin Wang, and Zhuowen Tu.  Attentional constellation nets for few-shot learning.  In International Conference on Learning Representations, 2021.
  * [79] Shuo Yang, Lu Liu, and Min Xu.  Free lunch for few-shot learning: Distribution calibration.  arXiv preprint arXiv:2101.06395, 2021.
  * [80] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha.  Few-shot learning via embedding adaptation with set-to-set functions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8808–8817, 2020.
  * [81] Nikolaos-Antonios Ypsilantis, Noa Garcia, Guangxing Han, Sarah Ibrahimi, Nanne Van Noord, and Giorgos Tolias.  The met dataset: Instance-level recognition for artworks.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [82] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  * [83] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen.  Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [84] Chi Zhang, Henghui Ding, Guosheng Lin, Ruibo Li, Changhu Wang, and Chunhua Shen.  Meta navigator: Search for a good adaptation policy for few-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9435–9444, 2021.
  * [85] Xueting Zhang, Debin Meng, Henry Gouk, and Timothy M Hospedales.  Shallow bayesian meta learning for real-world few-shot recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 651–660, 2021.
  * [86] Jiabao Zhao, Yifan Yang, Xin Lin, Jing Yang, and Liang He.  Looking wider for better adaptive representation in few-shot learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10981–10989, 2021.
  * [87] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin.  What makes instance discrimination good for transfer learning?  arXiv preprint arXiv:2006.06606, 2020.
  * [88] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  arXiv preprint arXiv:2111.07832, 2021.
  * [89] Ziqi Zhou, Xi Qiu, Jiangtao Xie, Jianan Wu, and Chi Zhang.  Binocular mutual learning for improving few-shot classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8402–8411, 2021.
  * [90] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.

##  7 Appendix

###  7.1 Implementation Details for Model Training

#### Self-supervised pretraining.

We pre-train our Vision Transformer backbone and projection head following the
same pipeline in iBOT [88]. Most of the hyper-parameter settings are kept
unchanged without tuning. Vit-Small, which has $\sim$21M parameters is used as
our default architecture. Our default patch size is set as 16. For the student
network, the [cls] token output and [patch] tokens output share the same
projection head. This head-sharing strategy is also applied to the teacher
network. For both networks, we set the output dimension of projection heads as
8192. We linearly warm up the learning rate for 10 epochs to its base value of
5e-4, then use cosine schedule to decay it to 1e-5. Cosine schedule is also
used for weight decay from 0.04 to 0.4. Besides, we use the multi-crop
strategy [6] with 2 global crops (224$\times$224) and 10 local crops
(96$\times$96), with scale range $(0.4,1.0)$ and $(0.05,0.4)$ respectively. We
found that allowing knowledge distillation between global and local crops from
intra-class images harms the performance, which is consistent with [88].
Therefore, local crops here are only used for self-distillation with global
crops from the same image. Furthermore, we apply blockwise masking on global
crops sent into the student network, with a masking ratio uniformly sampled
from $[0,1,0.5]$ with probability 0.5, and 0 with probability 0.5. Ablation of
different masking strategies is given in Sec. 7.3. Our batch size is set as
640 (batch size per GPU equal to 80). _mini_ -ImageNet and _tiered_ -ImageNet
are pre-trained for 1200 epochs, and CIFAR-FS and FC100 are pre-trained for
900 epochs. All models are trained on 8 Nvidia RTX 3090 GPUs.

#### Supervised knowledge distillation.

After finishing the pretraining stage, we train the model with our supervised-
contrastive loss. The best evaluation accuracy on the validation set can
usually be achieved within 60 epochs of training. We use the same set of
hyper-parameters as the first pretraining stage without further tuning.
Ablation of the scaling parameter $\lambda$, which controls the relative size
of $\mathbf{\mathcal{L}_{\texttt{[patch]}}}$ and
$\mathbf{\mathcal{L}_{\texttt{[cls]}}}$ is given in Sec. 7.3.

###  7.2 Few-shot Evaluation Results

We present few-shot evaluation results with more methods on the four benchmark
datasets here in Table 9 and 10. ViT-based methods are better than the
traditional CNN-based methods in general. The ranking of our method remains
unchanged.

Table 9: More comprehensive few-shot evaluation results on mini-ImageNet and
tiered-ImageNet. Top three methods are colored in  red,  blue and  green
respectively.

3 Method | Backbone | #Params | miniImageNet,5-way | tieredImageNet,5-way
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
DeepEMD [83] | _ResNet-12_ | 12.4M | ${{65.91\pm 0.82}}$ | ${{82.41\pm 0.56}}$ | ${{71.16\pm 0.87}}$ | ${{86.03\pm 0.58}}$
IE [59] | _ResNet-12_ | 12.4M | ${{67.28\pm 0.80}}$ | ${{84.78\pm 0.52}}$ | ${{72.21\pm 0.90}}$ | ${{87.08\pm 0.58}}$
BML [89] | _ResNet-12_ | 12.4M | ${{67.04\pm 0.63}}$ | ${{83.63\pm 0.29}}$ | ${{68.99\pm 0.50}}$ | ${{85.49\pm 0.34}}$
PAL [49] | _ResNet-12_ | 12.4M | ${{69.37\pm 0.64}}$ | ${{84.40\pm 0.44}}$ | ${{72.25\pm 0.72}}$ | ${{86.95\pm 0.47}}$
TPMN [75] | _ResNet-12_ | 12.4M | ${{67.64\pm 0.63}}$ | ${{83.44\pm 0.43}}$ | ${{72.24\pm 0.70}}$ | ${{86.55\pm 0.63}}$
MN+MC[84] | _ResNet-12_ | 12.4M | ${{67.14\pm 0.80}}$ | ${{83.82\pm 0.51}}$ | ${{74.58\pm 0.88}}$ | ${{86.73\pm 0.61}}$
DC [79] | _ResNet-12_ | 12.4M | ${{68.57\pm 0.55}}$ | ${{82.88\pm 0.42}}$ | ${{78.19\pm 0.25}}$ | ${{89.90\pm 0.41}}$
MELR [20] | _ResNet-12_ | 12.4M | ${{67.40\pm 0.43}}$ | ${{83.40\pm 0.28}}$ | ${{72.14\pm 0.51}}$ | ${{87.01\pm 0.35}}$
COSOC [48] | _ResNet-12_ | 12.4M | ${{69.28\pm 0.49}}$ | ${{85.16\pm 0.42}}$ | ${{73.57\pm 0.43}}$ | ${{87.57\pm 0.10}}$
CSEI [43] | _ResNet-12_ | 12.4M | ${{68.94\pm 0.28}}$ | ${{85.07\pm 0.50}}$ | ${{73.76\pm 0.32}}$ | ${{87.83\pm 0.59}}$
CNL [86] | _ResNet-12_ | 12.4M | ${{67.96\pm 0.98}}$ | ${{83.36\pm 0.51}}$ | ${{73.42\pm 0.95}}$ | ${{87.72\pm 0.75}}$
FEAT [80] | _WRN-28-10_ | 36.5M | ${{65.10\pm 0.20}}$ | ${{81.11\pm 0.14}}$ | ${{70.41\pm 0.23}}$ | ${{84.38\pm 0.16}}$
Meta-QDA [85] | _WRN-28-10_ | 36.5M | ${{67.38\pm 0.55}}$ | ${{84.27\pm 0.75}}$ | ${{74.29\pm 0.66}}$ | ${{89.41\pm 0.77}}$
OM [54] | _WRN-28-10_ | 36.5M | ${{66.78\pm 0.30}}$ | ${{85.29\pm 0.41}}$ | ${{71.54\pm 0.29}}$ | ${{87.79\pm 0.46}}$
SUN [17] | _ViT_ | 12.5M | ${{67.80\pm 0.45}}$ | ${{83.25\pm 0.30}}$ | ${{72.99\pm 0.50}}$ | ${{86.74\pm 0.33}}$
FewTURE [37] | _ViT-S_ | 21.0M | ${{68.02\pm 0.88}}$ | ${{84.51\pm 0.53}}$ | ${{72.96\pm 0.92}}$ | ${{86.43\pm 0.67}}$
FewTURE [37] | _Swin-Tiny_ | 29.0M | ${{72.40\pm 0.78}}$ | ${{86.38\pm 0.49}}$ | ${{76.32\pm 0.87}}$ | ${{89.96\pm 0.55}}$
HCT (Prototype) [36] | 3$\times$_ViT-S_ | 63.0M | ${\color[rgb]{0,0,1}{74.74\pm 0.17}}$ | ${{85.66\pm 0.10}}$ | ${\color[rgb]{0,0,1}{79.67\pm 0.20}}$ | ${{89.27\pm 0.13}}$
HCT (Classifier) [36] | 3$\times$_ViT-S_ | 63.0M | ${\color[rgb]{0,1,0}{74.62\pm 0.20}}$ | ${\color[rgb]{0,0,1}{89.19\pm 0.13}}$ | ${\color[rgb]{0,1,0}{79.57\pm 0.20}}$ | ${\color[rgb]{1,0,0}{91.72\pm 0.11}}$
Ours (Prototype) | _ViT-S_ | 21.0M | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$ | ${{78.83\pm 0.20}}$ | ${{91.02\pm 0.12}}$
Ours (Classifier) | _ViT-S_ | 21.0M | ${{74.10\pm 0.17}}$ | ${\color[rgb]{0,1,0}{88.89\pm 0.09}}$ | ${{78.81\pm 0.21}}$ | ${\color[rgb]{0,1,0}{91.21\pm 0.11}}$
Ours + HCT [36] | 3$\times$_ViT-S_ | 63.0M | ${\color[rgb]{1,0,0}{75.32\pm 0.18}}$ | ${\color[rgb]{1,0,0}{89.57\pm 0.09}}$ | ${\color[rgb]{1,0,0}{79.74\pm 0.20}}$ | ${\color[rgb]{0,0,1}{91.68\pm 0.11}}$
3 |  |  |  |  |  |

Table 10: More comprehensive few-shot evaluation results on CIFAR-FS and
FC100. Top three methods are colored in  red,  blue and  green respectively.

3 Method | Backbone | #Params | CIFAR-FS,5-way | FC100,5-way
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
DSN-MR [61] | _ResNet-12_ | 12.4M | ${{75.60\pm 0.90}}$ | ${{86.20\pm 0.60}}$ | - | -
BML [89] | _ResNet-12_ | 12.4M | ${{73.45\pm 0.47}}$ | ${{88.04\pm 0.33}}$ | ${{45.00\pm 0.41}}$ | ${{63.03\pm 0.41}}$
IE [59] | _ResNet-12_ | 12.4M | ${{77.87\pm 0.85}}$ | ${{89.74\pm 0.57}}$ | ${{47.76\pm 0.77}}$ | ${{65.30\pm 0.76}}$
PAL [49] | _ResNet-12_ | 12.4M | ${{77.10\pm 0.70}}$ | ${{88.00\pm 0.50}}$ | ${{47.20\pm 0.60}}$ | ${{64.00\pm 0.60}}$
TPMN [75] | _ResNet-12_ | 12.4M | ${{75.50\pm 0.90}}$ | ${{87.20\pm 0.60}}$ | ${{46.93\pm 0.71}}$ | ${{63.26\pm 0.74}}$
MN+MC [84] | _ResNet-12_ | 12.4M | ${{74.63\pm 0.91}}$ | ${{86.45\pm 0.59}}$ | ${{46.40\pm 0.81}}$ | ${{61.33\pm 0.71}}$
RENet [38] | _ResNet-12_ | 12.4M | ${{74.51\pm 0.46}}$ | ${{86.60\pm 0.32}}$ | - | -
ConstellationNet [78] | _ResNet-12_ | 12.4M | ${{75.40\pm 0.20}}$ | ${{86.80\pm 0.20}}$ | ${{43.80\pm 0.20}}$ | ${{59.70\pm 0.20}}$
ALFA+MeTAL [2] | _ResNet-12_ | 12.4M | - | - | ${{44.54\pm 0.50}}$ | ${{58.44\pm 0.42}}$
MixtFSL [1] | _ResNet-12_ | 12.4M | - | - | ${{41.50\pm 0.67}}$ | ${{58.39\pm 0.62}}$
CC+rot [23] | _WRN-28-10_ | 36.5M | ${{73.62\pm 0.31}}$ | ${{86.05\pm 0.22}}$ | - | -
PSST [12] | _WRN-28-10_ | 36.5M | ${{77.02\pm 0.38}}$ | ${{88.45\pm 0.35}}$ | - | -
Meta-QDA [85] | _WRN-28-10_ | 36.5M | ${{75.95\pm 0.59}}$ | ${{88.72\pm 0.79}}$ | - | -
SUN [17] | _ViT_ | 12.5M | ${{78.37\pm 0.46}}$ | ${{88.84\pm 0.32}}$ | - | -
FewTURE [37] | _ViT-S_ | 21.0M | ${{76.10\pm 0.88}}$ | ${{86.14\pm 0.64}}$ | ${{46.20\pm 0.79}}$ | ${{63.14\pm 0.73}}$
FewTURE [37] | _Swin-Tiny_ | 29.0M | ${{77.76\pm 0.81}}$ | ${{88.90\pm 0.59}}$ | ${{47.68\pm 0.78}}$ | ${{63.81\pm 0.75}}$
HCT (Prototype) [36] | 3$\times$_ViT-S_ | 63.0M | ${\color[rgb]{0,1,0}{78.89\pm 0.18}}$ | ${{87.73\pm 0.11}}$ | ${\color[rgb]{0,1,0}{48.27\pm 0.15}}$ | ${{61.49\pm 0.15}}$
HCT (Classifier) [36] | 3$\times$_ViT-S_ | 63.0M | ${{78.88\pm 0.18}}$ | ${\color[rgb]{0,1,0}{90.50\pm 0.09}}$ | ${{48.15\pm 0.16}}$ | ${\color[rgb]{0,1,0}{66.42\pm 0.16}}$
Ours (Prototype) | _ViT-S_ | 21M | ${\color[rgb]{1,0,0}{80.08\pm 0.18}}$ | ${\color[rgb]{0,0,1}{90.63\pm 0.13}}$ | ${\color[rgb]{1,0,0}{50.38\pm 0.16}}$ | ${\color[rgb]{0,0,1}{68.37\pm 0.16}}$
Ours (Classifier) | _ViT-S_ | 21M | ${\color[rgb]{0,0,1}{79.82\pm 0.18}}$ | ${\color[rgb]{1,0,0}{90.91\pm 0.13}}$ | ${\color[rgb]{0,0,1}{50.28\pm 0.16}}$ | ${\color[rgb]{1,0,0}{68.50\pm 0.16}}$
3 |  |  |  |  |  |

###  7.3 Additional Ablation Studies

#### Why $\mathcal{L}_{[cls]}+\mathcal{L}_{\text{MIM}}$ in stage 1?

Our insight is that the [cls] tokens in global loss have better high-level
semantics, but often disregard the rich local structures. While the MIM loss
$\mathcal{L}_{\text{MIM}}$ constructed from [patch] tokens can remedy this
problem, increase task difficulty, and work as strong data augmentations. In
Table 7.3, we can find that using both losses in stage 1 gives the best
results.

Table 11: Ablation of SSL tasks in stage 1 on mini-ImageNet.

3 Stage1 | Stage2: $\mathcal{L}_{\texttt{[cls]}}+\mathcal{L}_{\texttt{[patch]}}$
---|---
$\mathcal{L}_{[cls]}$ | $\mathcal{L}_{\text{MIM}}$ | 1-shot | 5-shot | 1-shot | 5-shot
✓ |  | 58.55 | 78.90 | 72.93 | 88.07
| ✓ | 27.66 | 33.82 | 37.03 | 50.95
✓ | ✓ | 60.93 | 80.38 | 74.28 | 88.82
3 |  |  |  |  |

#### Masking Strategies.

We use blockwise masking as our default in the main text. In Table 12, we test
random mask and no mask while keeping all other hyper-parameters unchanged.
"Block Mask $\xrightarrow[]{}$ No Mask" represents self-supervised pretraining
with blockwise masking, and supervised training with no mask. Using either a
random mask or block mask can boost the classification accuracy in the first
self-supervised pretraining stage, but their advantage over no mask decreases
in the second supervised training stage. We choose blockwise masking as our
default strategy since it balances 1 and 5-shot classification accuracy the
best.

Table 12: Ablation over different masking strategy in self-supervised pretraning stage. 3 Masking Strategy | Self-supervised Pre-train | Supervised Training
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
No Mask | ${{59.15\pm 0.17}}$ | ${{79.23\pm 0.12}}$ | ${{73.94\pm 0.17}}$ | ${{88.93\pm 0.09}}$
Block Mask | ${{60.93\pm 0.17}}$ | ${{80.38\pm 0.12}}$ | ${{74.01\pm 0.17}}$ | ${{88.89\pm 0.09}}$
Random Mask | ${{60.94\pm 0.18}}$ | ${{79.62\pm 0.13}}$ | ${{74.07\pm 0.18}}$ | ${{88.66\pm 0.09}}$
Block Mask $\xrightarrow[]{}$ No Mask | ${{60.93\pm 0.17}}$ | ${{80.38\pm 0.12}}$ | ${{73.44\pm 0.17}}$ | ${{88.87\pm 0.09}}$
3 |  |  |  |

#### Scaling Parameter $\lambda$.

This parameter controls the relative importance of class-level and patch-level
losses in our final loss:
$\mathbf{\mathcal{L}}=\mathbf{\mathcal{L}_{[cls]}}+\lambda\mathbf{\mathcal{L}_{[patch]}}$.
A relatively large value of $\lambda$ will put more focus on localization and
less on high-level semantics. Here in Table 13, we test different $\lambda$
values by keeping the base of $\mathbf{\mathcal{L}_{[cls]}}$ to 1 and scale
$\mathbf{\mathcal{L}_{[patch]}}$. As we can see, the $\lambda$ parameter
influences 1-shot classification accuracy more than 5-shot. We choose
$\lambda=0.25$ as our default (which makes the ratio of
$\mathbf{\mathcal{L}_{[patch]}}/\mathbf{\mathcal{L}_{[cls]}}$ roughly around
2) since it has best 1-shot performance and competitive 5-shot accuracy.

Table 13: The influence of different ratio between
$\mathbf{\mathcal{L}_{[cls]}}$ and $\mathbf{\mathcal{L}_{[patch]}}$.

$\mathbf{\mathcal{L}_{[patch]}}/\mathbf{\mathcal{L}_{[cls]}}$ | 1-shot | 5-shot
---|---|---
$\approx$4 ($\lambda=0.9$) | ${{73.45\pm 0.17}}$ | ${{88.89\pm 0.09}}$
$\approx$2 ($\lambda=0.45$) | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$
$\approx$1 ($\lambda=0.2$) | ${{74.01\pm 0.17}}$ | ${{88.89\pm 0.09}}$
$\approx$0.5 ($\lambda=0.1$) | ${{73.11\pm 0.17}}$ | ${{88.44\pm 0.09}}$

#### Weighting Parameter $\omega_{k^{+}}$ in $\mathbf{\mathcal{L}_{[patch]}}$.

This parameter in Eq.(5) gives weights to each component of our patch-level
contrastive loss $\mathbf{\mathcal{L}_{[patch]}}$. We set $\omega_{k^{+}}=1/N$
in our main text due to its simplicity. In Table 14, we compare it _(Simple
Avg)_ with another variant _(Self-Attention Weighted Avg)_ , which uses the
averaged self-attention weights over attention heads of the [cls] token with
all [patch] tokens in the last attention layer of teacher network to aggregate
pairwise patch matching losses. As found in [7], the self-attention of ViTs is
good at capturing foreground regions. So we use it here as a way to highlight
foreground objects and to attenuate irrelevant background information. Our
default simple average outperforms this variant on both 1 and 5-shot
classification accuracies. One explanation is as follows. If the foreground
objects of two intra-class images differ a lot, then _Self-Attention Weighted
Avg_ tends to minimize $\mathbf{\mathcal{L}_{[patch]}}$ by decreasing the
weights of the losses associated with the patches covering these foreground
objects, which makes our model deviate from optimum.

Table 14: Different weighting schemes of patch-level supervised-contrastive
loss.

Weighting Scheme | 1-shot | 5-shot
---|---|---
Simple Avg | ${{74.28\pm 0.18}}$ | ${{88.82\pm 0.09}}$
Self-Attention Weighted Avg | ${{74.11\pm 0.18}}$ | ${{88.52\pm 0.10}}$

#### Comparison with smaller backbones:

To make ViT-S ($\sim$ 21M parameters) comparable with ResNet-12 ($\sim$ 12M
parameters), we trim by half either its embedding dimension
($d_{\text{embed}}$) or the number of attention heads (#heads). From Table 15,
trimming #heads by half only results in little drop in accuracy, which still
outperforms the best method with ResNet-12 backbones. The training speed also
increases by 10% with fewer #heads. Given this result, our comparison now
becomes complete: our method outperforms both shallow (ResNet-12) and deep
(WRN-28-10) CNN-based backbones, as well as ViTs with the same (see Table 5)
or more (see Table 3 & Table 4) parameters.

Table 15: ViT-S with similar size as ResNet-12 on mini-ImageNet

3 Backbone | $d_{\text{embed}}$ | #heads | #param | Stage1 | Stage2
---|---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
ViT-S | 192 | 6 | 11M | 60.70 | 79.56 | 71.14 | 87.12
ViT-S | 384 | 3 | 11M | 62.12 | 81.27 | 72.70 | 87.90
ViT-S | 384 | 6 | 21M | 60.93 | 80.38 | 74.28 | 88.82
ResNet12 | - | - | 12M | - | - | 69.37 | 85.16
WRN-28-10 | - | - | 36M | - | - | 67.38 | 85.29
3 |  |  |  |  |  |  |

###  7.4 Visualizations

We visualize more self-attention maps and dense correspondence in Fig. 6 and
Fig. 7.

Figure 6: Visualization of multi-head self-attention maps. The self-attention
of the [cls] tokens with different heads in the last attention layer of ViT
are visualized in different colors in the second column. The last six columns
visualize each attention head. Images are from the test set of _mini_
-ImageNet. Figure 7: Visualization of dense correspondence. We use the patches
with the highest self-attention of the [cls] token on each attention head (6
in total) of the last layer of ViT-S as queries. Best-matched patches with the
highest similarities are connected with lines. Images are from the validation
and testing set of _mini_ -ImageNet.


#  : Interactive Toolbox Supporting Precise Data Annotation
for Robust Vision Learning

Chonghan Chen  Haohan Wang  Leyang Hu  Yuhao Zhang  Shuguang Lyu  Jingcheng Wu
Xinnuo Li  Linjing Sun  Eric P. Xing

###### Abstract

We introduce the initial release of our software Robustar, which aims to
improve the robustness of vision classification machine learning models
through a data-driven perspective. Building upon the recent understanding that
the lack of machine learning model’s robustness is the tendency of the model’s
learning of spurious features, we aim to solve this problem from its root at
the data perspective by removing the spurious features from the data before
training. In particular, we introduce a software that helps the users to
better prepare the data for training image classification models by allowing
the users to annotate the spurious features at the pixel level of images. To
facilitate this process, our software also leverages recent advances to help
identify potential images and pixels worthy of attention and to continue the
training with newly annotated data. Our software is hosted at the GitHub
Repository https://github.com/HaohanWang/Robustar.

Machine Learning, ICML

##  1 Introduction

Machine learning has achieved remarkable performances over i.i.d benchmarks in
recent years, which has greatly encouraged the community to test the methods
in other scenarios beyond the i.i.d settings, leading to multiple other topics
such as the study of cross-domain robustness including domain adaptation (Ben-
David et al., 2007, 2010), domain generalization (Muandet et al., 2013), and
the recent advances beyond the settings (Ye et al., 2021; Huang et al., 2022),
as well as the study of robustness against predefined perturbations such as
adversarial robustness (Szegedy et al., 2014; Goodfellow et al., 2015).

While there is a diverse set of topics regarding machine learning robustness
over different directions, recently literature suggests that a central theme
of these challenges is the existence of spurious features that are not
semantics but associated with the labels (Wang et al., 2020). Correspondingly,
a proliferation of machine learning robustness methods are designed by
explicitly or implicitly countering the model’s tendency in learning spurious
features (Wang et al., 2021).

While the central theme of countering the learning of spurious features can
potentially guide the development of many robust machine learning methods
(Wang et al., 2021), we nonetheless consider the repeated case-by-case design
and implementation of machine learning methods over different applications
might be inconvenient at certain scenarios.

Therefore, in this project, we aim to solve this problem in the scope of image
classification with a model-free manner, by focusing on the data perspective
at the image pixel level. We introduce a software that allow the users or
domain experts to annotate the pixels that are associated with the label in
the spurious manner. The annotation will allow the system to continue to train
the model with established robust learning methods of data augmentation
strategies.

In summary, we introduce the software Robustar with the following functions:

  * •

Core Function: to allow users interact with the training samples, annotating
features that are useless used potentially used by the models to help train a
robust model to the request of the domain expert.

  * •

Supporting Functions:

    * –

We use influence function to help identify the samples that need attention.

    * –

We use saliency-map style interpretation to help the user identify the
features need attention.

    * –

We use segmentation models to help the user select certain pixels more
efficiently.

    * –

We use data augmentation and regularizations to help train models invariant to
the identified pixels.

The remainder of this manuscript will be structured as follows. We will first
offer an overview of our system and the main pipeline of the use case in
Section 2. Then in Section 3, we will introduce the core functions presented
in our software. Finally, we will offer some discussions before we conclude in
Section 5.

##  2 System Overview

Figure 1 shows the overview of the software Robustar pipeline, which consists
of five major steps.

  1. 1.

The model can be trained elsewhere and then fed into the software.

  2. 2.

With new test samples, the model can help identify the samples that are
responsible for the prediction through influence function.

  3. 3.

The software offers saliency map to help the user know which part of the
features the model are paying necessary attention.

  4. 4.

The users can use the drawing tools to brush out the spurious pixels.

  5. 5.

New annotation of these images will serve as the role as augmented images for
continued training.

The software can work with models trained elsewhere, as long as these are
standard vision models and have the standard pytorch checkpoints available. If
the influence function results are also calculated, the software can guide the
users to the samples that are believed to be responsible for the incorrect
classification of the images. Further, our software will also guide the user’s
attention of the features by the saliency-style interpretation of the models.
Then the users can user canvas tool to annotate the spurious pixels of the
images. Finally, with the newly annotated spurious features, we can continue
to update the model with data augmentation and consistency regularization to
help us discard these spurious features.

Figure 1: The major working flow of Robustar.

##  3 Major Functions

In this section, we will introduce the functions our software offer following
the natural order of the pipeline of our package will help the domain experts
to annotate the spurious pixels in image classification.

Figure 2: Screenshots of main functions of Robustar: above: the image list to
view training images and their role in the model; below: the canvas to
annotate the details of one image.

### Image Browsing

Our system allows the users to examine every samples in the training set
through the image browsing interface (Figure 2,(upper)).

Through the interface, the users can directly view the each image from either
the training or the testing set. If the user click one image, the sidebar on
the right-hand-side will display multiple relevant statistics of the image of
interest. For example, the prediction summary will indicate what the model
believes the current image to be, offering the users the information of
whether the prediction align with the user’s p perception of the images well.
The remaining statistics at the sidebar will be discussed later.

This interface allows the user to check every image in the training sample and
identify the ones with spurious features for further annotation.

### Automatic Identifying Misleading Samples

Ideally, to achieve a robust learning system, the domain experts will
scrutinize the training samples annotate any spurious pixel features. However,
this examination process might potentially require an unrealistic amount of
working load, despite it potentially significantly amount of robustness gain.
Thus, to suit the need for some users’ concerns in devoting the efforts of
examining every samples, we allow the system to propose the suspiciously
misleading samples (samples with spurious features learned by the model)
first.

The central assumption that enables the automatic proposal of misleading
samples is that the spurious features are not shared between training samples
and testing samples, so that when a test sample is misclassified, it is mostly
due to the fact that the model learns a biased signal that accounts for the
misclassification. This allows us to identify the samples with spurious
features. Influence function (Koh & Liang, 2017) conveniently allows us to
identify the samples that account for the misclassification: for every test
samples, we calculate the most relevant training samples from the training
set, and these samples are proposed to the users to pay particular attention
especially when the test samples are misclassified.

### Annotation of Spurious Pixel Features

The user can choose to enter the annotation page (Figure 2 (bottom right)) by
clicking on any images of interest. In the annotation page, the user can
choose to use pencil to brush out the pixels that are considered spurious.

In addition, to reduce the efforts for user to identify and brushing out, the
system offers two other alternative ways of annotating the images, as shown in
(Figure 2 (bottom left)): the users can choose to directly filter out all the
pixels with certain ranges, we also offer a pretrained segmentation model (He
et al., 2017) that can automatically filter out the background pixels. In our
experiments, we notice that the segmentation model can save significant
efforts in natural images, although not perfect. However, we also notice that,
in medical images such as X-ray, the segmentation model is not always able to
reduce a significant amount of efforts.

### Interpret Model’s Decision

While it will be better to ask the user to identify all the pixels in the
background, it might not be required for all the background pixels to be
annotated for continued training: only the pixels that are learned by the
machine learning model as spurious features need to be annotated and later
countered through the continued training process. Therefore, to further reduce
the users’ efforts in annotation, the system to show where the current model
focuses on the images (Figure 2).

For the visualization of the model’s decision, we simply uses the pioneering
model-interpretation methods, activation maximization (Erhan et al., 2009),
and we notice that the method can fulfill our need well enough, especially
when the model is already trained to be invariant to high-frequency signals of
the images.

### Continued Training with Randomized Pixels

With annotation of spurious features all misleading samples, the users can
directly use our system to update the model to improve its robustness against
its tendency in learning the spurious features through our task center (Figure
3).

Figure 3: Task center and paired training

In particular, to sufficiently force the model to drop the learning of
spurious features, we leverage the recent training paradigm with data
augmentation and consistency regularization (Wang et al., 2022): to train the
model with the original image and an augmented image whose spurious pixel
features are replaced by random noises and a regularization forcing the
model’s logits (pre-softmax embedding) to be the same from both set of images.

We refer to this training paradigm as paired training. Wang et al. (2022)
showed that this generic training paradigm can efficiently improve the
robustness of models empirically better than models specifically designed for
the tested applications, with significantly less computing costs.

This software paper omits the demonstration of the machine learning
performances through these techniques, but one can refer to (Wang et al.,
2022) for the superior performances led by data augmentation and alignment
regularization.

##  4 Usage Instructions

### Preparation

As the goal of this system is to counter the model’s tendency in learning
spurious features, we recommend the users to start with a model with
reasonably small training and testing errors. Users can either upload such a
model to the system or use the functions offered by the system to start from
scratch and train a new model. In either case, the whole training dataset is
expected to be available for the annotation.

To take the most advantage of the system, we recommend the user to use out-of-
domain test samples (i.e., test samples from an independent data collection
instead of from a cross-validated split), because the test data from the same
collection (distribution) with training data tend to share the same spurious
features with training data, thus may be inadequate to help identify the
spurious features. As long as the test samples can represent a wider range of
distributions, we do not need a large number of test samples.

### Setup and Run

With Docker installed, the users can run Robustar with a single-line command,
if the user has pulled the relevant docker images with another command line,
which is only required before the first run.

To run the system, the users need to feed in several path for the Docker to
mount, including the folders of the data (with PyTorch ImageFolder structure),
folder of influence images (precomputed or empty folder path), folder of
checkpoints, and a configuration file with information such as number of
classes and model configuration. Details of these information can be found at
the Robustar GitHub Repository.

Figure 4: GUI console

### GUI Console

For users not comfortable using command line, we also offer a GUI console for
the users to setup the above the configurations and monitor the running of the
system (Figure 4).

##  5 Discussion and Conclusion

We aim to improve machine learning robustness from its root at the data
perspective by allowing the users to train a model with spurious features
annotated. For this purpose, we offer a software for the users to inspect the
training data and annotate the spurious image features. Our software can be
found as https://github.com/HaohanWang/Robustar.

In addition to the facilitate machine learning robustness, we believe our
system can also serve multiple purposes in the machine learning community from
the data perspective. For example, we expect our system to also help the
community to inspect the properties of the data to understand how to annotate
and prepare the data better at the pixel level, and our software has the
potential to generate massive amount of data along the interactive process
between human and the model, leading to another possibility to automatically
understand the properties of the data.

## Acknowledgements

The project team would like to thank Donglin Chen for his contribution at the
early-stage of the development.

## References

  * Ben-David et al. (2007) Ben-David, S., Blitzer, J., Crammer, K., Pereira, F., et al.  Analysis of representations for domain adaptation.  _Advances in neural information processing systems_ , 19:137, 2007.
  * Ben-David et al. (2010) Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W.  A theory of learning from different domains.  _Machine learning_ , 79(1-2):151–175, 2010.
  * Erhan et al. (2009) Erhan, D., Bengio, Y., Courville, A., and Vincent, P.  Visualizing higher-layer features of a deep network.  _University of Montreal_ , 1341(3):1, 2009.
  * Goodfellow et al. (2015) Goodfellow, I. J., Shlens, J., and Szegedy, C.  Explaining and harnessing adversarial examples.  In Bengio, Y. and LeCun, Y. (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., and Girshick, R.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2961–2969, 2017.
  * Huang et al. (2022) Huang, Z., Wang, H., Huang, D., Lee, Y. J., and Xing, E. P.  The two dimensions of worst-case training and the integrated effect for out-of-domain generalization.  _arXiv preprint arXiv:2204.04384_ , 2022.
  * Koh & Liang (2017) Koh, P. W. and Liang, P.  Understanding black-box predictions via influence functions.  In _International conference on machine learning_ , pp. 1885–1894. PMLR, 2017.
  * Muandet et al. (2013) Muandet, K., Balduzzi, D., and Schölkopf, B.  Domain generalization via invariant feature representation.  In _International Conference on Machine Learning_ , pp. 10–18, 2013\.
  * Szegedy et al. (2014) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R.  Intriguing properties of neural networks.  In Bengio, Y. and LeCun, Y. (eds.), _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.
  * Wang et al. (2020) Wang, H., Wu, X., Huang, Z., and Xing, E. P.  High-frequency component helps explain the generalization of convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8684–8694, 2020.
  * Wang et al. (2021) Wang, H., Huang, Z., Zhang, H., and Xing, E.  Toward learning human-aligned cross-domain robust models by countering misaligned features.  _arXiv preprint arXiv:2111.03740_ , 2021.
  * Wang et al. (2022) Wang, H., Huang, Z., Wu, X., and Xing, E. P.  Toward learning robust and invariant representations with alignment regularization and data augmentation.  _arXiv preprint arXiv:2206.01909_ , 2022.
  * Ye et al. (2021) Ye, N., Li, K., Hong, L., Bai, H., Chen, Y., Zhou, F., and Li, Z.  Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms.  _arXiv preprint arXiv:2106.03721_ , 2021.