[subfigure]style=plain,heightadjust=object

# Pixel-Wise Contrastive Distillation

Junqiang Huang$\dagger$ $\ddagger$ Zichao Guo$\dagger$
Shopee
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present the first pixel-level self-supervised distillation framework
specified for dense prediction tasks. Our approach, called Pixel-Wise
Contrastive Distillation (PCD), distills knowledge by attracting the
corresponding pixels from student’s and teacher’s output feature maps. This
pixel-to-pixel distillation demands for maintaining the spatial information of
teacher’s output. We propose a SpatialAdaptor that adapts the well-trained
projection/prediction head of the teacher used to encode vectorized features
to processing 2D feature maps. SpatialAdaptor enables more informative pixel-
level distillation, yielding a better student for dense prediction tasks.
Besides, in light of the inadequate effective receptive fields of small
models, we utilize a plug-in multi-head self-attention module to explicitly
relate the pixels of student’s feature maps. Overall, our PCD outperforms
previous self-supervised distillation methods on various dense prediction
tasks. A backbone of ResNet-18 distilled by PCD achieves $37.4$
AP${}^{\text{bbox}}$ and $34.0$ APmask with Mask R-CNN detector on COCO
dataset, emerging as the first pre-training method surpassing the supervised
pre-trained counterpart.

22footnotetext: Equal Contribution33footnotetext: corresponding author

##  1 Introduction

Self-supervised learning (SSL) has emerged as a promising way for pre-training
due to its remarkable progress for various computer vision tasks [23, 9, 21,
6]. Without the need for annotations, self-supervised pre-trained models
achieve similar or better transfer performance compared to the supervised
counterparts. The advancement of SSL, however, appears to be limited to large
models. Small models like ResNet-18 [25] show inferior results under the
linear probing protocol as reported in [18, 57, 14]. Recently, the performance
lag of small models has been effectively alleviated by self-supervised
distillation [58, 39, 1, 18, 19, 57, 4, 14, 63, 52], where teachers’ (large
pre-trained models) knowledge is transferred [5, 48, 2, 28] to students (small
models) in a self-supervised learning fashion.

The specific practice of self-supervised distillation methods can be
summarized as the following two types: (i) minimizing the feature distance
between students and teachers [1, 18, 4, 19, 57, 52], (ii) classifying the
semantic clusters generated by teachers [58, 39, 14, 63]. These methods enable
competitive performance of small models, especially on classification tasks
(_e.g_., fine-grained and few-shot classification). But the improvement of
dense prediction tasks like object detection and semantic segmentation is not
so bright as that of classification tasks. This imbalanced status seemingly
suggests that the favorable representations learned by teachers can only be
partially transferred to students. A natural question to ask is: what hinders
students from inheriting the knowledge beneficial to dense prediction tasks?
In this study, we seek the answers to this question from the following
aspects.

First, the distillation signals of current self-supervised distillation
methods are mostly at image-level, while the rich pixel-level knowledge is yet
to be utilized. These methods pursue a distilled student capable of extracting
semantically separable features from images. But dense prediction tasks demand
more—the pixels themselves should also be distinguishable. As opposed to large
models, small models are hard to learn pixel-level knowledge from image-level
supervision. Encouraging students to mimic teachers’ “vectorized” behaviors
might fail to transfer the abounding pixel-level knowledge to small models.
Driven by this, we present the first pixel-level self-supervised distillation
framework, Pixel-Wise Contrastive Distillation (PCD), extending the idea of
contrastive learning [22] by combining pixel-level knowledge distillation. For
a given image, our PCD attracts the corresponding pixels from student’s and
teacher’s output feature maps while separating student’s pixels and negative
pixels of a memory queue [23]. In this manner, the student is spurred to mimic
the teacher at pixel-level, resulting in more adequate knowledge transfer.

[] []  []

Figure 1: Architecture of Pixel-Wise Contrastive Distillation Figure 2:
Workflows of Teacher’s Projection Head Figure 3: (a) is the specific
architecture of Pixel-Wise Contrastive Distillation. Before distillation, the
original teacher’s projection head is modified by SpatialAdaptor (represented
by in the figure). Distillation loss is the average contrastive loss computed
over all corresponding pixel pairs of the student and the teacher. (b) depicts
the workflows of teacher’s projection head before and after using
SpatialAdaptor. The pooling layer on the far right is used for demonstrating
the invariability of SpatialAdaptor. Best viewed in color.

Second, it is not straightforward to distill pixel-level knowledge from the
commonly adopted teachers pre-trained by image-level SSL methods [23, 9, 21,
6]. These teachers tend to project images into vectorized features, thus
losing the spatial information indispensable to pixel-to-pixel distillation in
our PCD. An intuitive practice to circumvent this issue would be to drop the
well-trained projection/prediction head (a non-linear MLP attached to the
backbone) and perform distillation before the global pooling layer.
Nevertheless, we experimentally show the ineffectiveness of such simplistic
approach, suggesting the projection/prediction head contains nonnegligible
knowledge for pixel-level distillation. Towards the goal of leveraging this
knowledge, we devise a SpatialAdaptor to adapt the projection/prediction head
used to encode vectorized features to processing 2D feature maps while not
changing the distribution of output features. Though conceptually simple,
SpatialAdaptor is of great significance to pixel-level distillation.

Last, small models are innately weak in capturing spatial information from
regions of large spans due to its limited capacity. We compare the effective
receptive fields (ERF) [37] of a small model (ResNet-18) and a large model
(ResNet-50) to verify this point. To overcome this natural deficiency, we
insert a multi-head self-attention (MHSA) module [49] to the end of the
student model, explicitly relating pixels within the output feature maps.
Adding the MHSA module helps slightly enlarge the ERF of the small model, and
consequently improves transferring results. The MHSA module only introduces a
feasible cost of memory and computation during training and will be deprecated
in fine-tuning. The concrete form of PCD is depicted in Fig. 3.

We evaluate our PCD on several typical dense prediction tasks. Given the
backbone of ResNet-18, PCD surpasses state-of-the-art results among all
downstream tasks. The nontrivial advantages against competitive SSL methods
designed for dense prediction tasks and previous image-level self-supervised
distillation methods exhibit the superiority of pixel-level supervision signal
for self-supervised distillation. PCD achieves $37.4$ AP${}^{\text{bbox}}$ and
$34.0$ AP${}^{\text{mask}}$ on COCO dataset [34] by using Mask R-CNN detector
[24] and ResNet-18-FPN [33] backbone, emerging as the first pre-training
method outperforming the supervised pre-trained counterpart. Under the linear
probing protocol, a ResNet-18 distilled by PCD obtains $65.1\%$ top-1 accuracy
on ImageNet.

Our PCD is robust to the choices of pre-trained teacher models and works well
with various student backbone architectures. Students with larger backbones
can compete with or even exceed the teacher on some dense prediction tasks.
These results reveal a encouraging path for pre-training. We hope our findings
will motivate future research.

##  2 Related Work

Pixel/region-level self-supervised learning aims to learn competitive
representations specialized for dense prediction tasks. Following the
philosophy of contrasting pixel/region-level features from different augmented
views, these methods develop various rules to find the positive pairs.

Intuitive methods [40, 43, 54, 56, 59] record the offsets and the scaling
factors induced by geometric transformations (_e.g_., cropping, resizing, and
flipping) to locate positive pairs of pixels/regions from different augmented
views. In [26, 3], each pixel of the original image is classified into some
appropriate categories by a heuristic method or some unsupervised semantic
segmentation methods. Any two pixels or regions from the same category form a
positive pair. SoCo [51] and ORL [55] utilize the selective search [47] to
identify numerous regions containing a single object and perform region-level
contrastive learning based on these regions. DenseCL [50] and Self-EMD [35]
pair the pixels of feature maps from different views according to some certain
rules, _e.g_., minimizing the cosine distances between pixels or finding the
matching set with minimum earth mover’s distance [29].

Our PCD does not rely on sophisticated rules or preparations to pair
pixels/regions. Instead, we directly contrast the feature maps projected by
the student and the teacher from the same view of image, decoupling the
requirement for delicate augmentation policies from the design of pre-training
framework.

Feature-based knowledge distillation transfers knowledge by matching the
intermediate features between a student and a teacher. Generally, these
features are not comparable because of their different shapes, _i.e_., number
of channels and spatial size. It is a common practice to reshape the features
of the student to have the same shape as the features of the teacher by a
learnable module [44]. Some works [61, 32, 27, 8] transform both student’s and
teacher’s features into tensors of same shape. In PCD, shape aligning
(especially for the number of channels) is completed by student’s non-linear
projection head. It is universally recognized that this projection head also
improves the quality of learned representations in SSL [9, 10].

Self-supervised distillation transfers knowledge in a self-supervised learning
fashion. CompRess [1] and SEED [18] propose to minimize feature similarity
distribution between the student and the teacher. DoGo [4] and DisCo [19] add
a distillation branch for easing the optimization problem of small models
during self-supervised pre-training. Previous works [58, 39] train the student
to classify the images based on the pseudo labels generated by teacher’s
clustering. In BINGO [57], samples are grouped based on their similarity
computed by the teacher, and the student learns to minimize intra-group
distances. Recent work [14] simultaneously trains the teacher from scratch and
perform distillation, where the student is guided by teacher’s on-the-fly
clustering. Our PCD differs from these methods in that the distillation signal
of the former is at pixel-level. Our goal is to pursue a student adept at
dense prediction tasks.

##  3 Method

###  3.1 Pixel-Wise Contrastive Distillation

Unlike general knowledge distillation methods in supervised learning, self-
supervised distillation does not involve with labeled data. The supervision
only comes from teachers, yielding a task-agnostic student that can be fine-
tuned in various downstream tasks.

Image-level self-supervised distillation. Though varying dramatically in the
specific rules of computing distillation loss, self-supervised knowledge
distillation methods share one thing in common—the supervision signals are all
at image-level. Here, we describe a general formulation for these methods. An
input image is fed to student’s and teacher’s backbone, generating feature
maps, $\mathbf{s}\in\mathbb{R}^{C_{s}{\times}H{\times}W}$ and
$\mathbf{t}\in\mathbb{R}^{C_{t}{\times}H{\times}W}$, respectively. $C$ is the
number of channels. $H$ and $W$ are the spatial sizes††It is reasonable to
assume $\mathbf{s}$ and $\mathbf{t}$ have the same spatial size owing to the
popular stage-wise designs of convolutional network architectures.. These
feature maps are then global average/max pooled into vectorized features. The
distillation loss $\mathcal{L}$ with respect to a single input image is
defined by:

$\mathcal{L}=\mathcal{L}\left(\phi\left(\mathbf{s}\right),\phi\left(\mathbf{t}\right)\right),$
(1)

where $\phi\left(\cdot\right)$ is the global pooling layer. Note that
$\mathcal{L}\left(\cdot,\cdot\right)$ is a function composition ††In CompRess
[1], for example, $\mathcal{L}$ is equivalent to first estimating the
similarity distributions of the student and the teacher, then computing the KL
divergence between distributions., but not a simple loss function like $\ell
2$ distance. We let $\phi$ be global average pooling for the ease of analysis,
i.e., $\phi\left(\mathbf{x}\right)=\frac{1}{HW}\sum_{i}\mathbf{x}_{i}$. Here
$i=\left(i_{H},i_{W}\right)$ is a 2-tuple indexing the $(i_{H},i_{W})$-th
pixel of feature maps.

Given the above unified formulation, we consider the derivative with respect
to the $i$-th pixel of student’s feature maps $\mathbf{s}_{i}$. By chain rule,
we have:

$\frac{\partial\mathcal{L}}{\partial\mathbf{s}_{i}}=\frac{\partial\mathcal{L}}{\partial\phi}\frac{\partial\phi}{\partial\mathbf{s}_{i}}=\frac{1}{HW}\frac{\partial\mathcal{L}}{\partial\phi}.$
(2)

It is obvious to see that $\frac{\partial\mathcal{L}}{\partial\mathbf{s}_{i}}$
is a term independent from the position of the pixel. Teacher’s guidance is
not detailed at pixel-level. There probably exists huge disparity between
student’s and teacher’s pixel-level features. Consequently, it is far beyond
reach for students (small models) to inherit competitive pixel-level knowledge
possessed by teachers. We hypothesize that this may be the very reason of
students’ imbalanced performance on classification and dense prediction tasks.

Pixel-Wise Contrastive Distillation. Motivated by the above analysis, we
propose the first pixel-level self-supervised distillation framework, Pixel-
Wise Contrastive Distillation (PCD). Our PCD transfers knowledge by attracting
the positive pairs of pixels from students and teachers and repulsing the
negative pairs.

Different from augmentation-invariant representation learning [46, 23, 38, 9,
56, 50], the positive pairs of PCD are from the corresponding pixels of
student’s and teacher’s output feature maps for the same image, _i.e_.,
$\left(\mathbf{s}_{i},\mathbf{t}_{i}\right)$. Negative samples
$\\{\mathbf{n}^{k}|k=1,...,K\\}$ are stored in a queue in conformity with MoCo
[23]. For an input image, we optimize the average contrastive loss of all
output pixels:

$\mathcal{L}\left(\mathbf{s},\mathbf{t}\right)=\frac{1}{HW}\sum_{i}\ell\left(\mathbf{s}_{i},\mathbf{t}_{i},\\{\mathbf{n}^{k}\\}\right),$
(3)

where $\ell\left(\cdot,\cdot\right)$ stands for the contrastive loss function.
We do not directly contrast $\mathbf{s}_{i}$ and $\mathbf{t}_{i}$ for they may
have different dimensions (_i.e_., the numbers of channels). Following SimCLR
[9], we append a projection head $\varphi\left(\cdot|\theta\right)$ (a non-
linear MLP) to student’s backbone. The details of $\varphi$ will be given in
Sec. 3.2. The projection head $\varphi$ serves as aligning $\mathbf{s}_{i}$
and $\mathbf{t}_{i}$ in terms of dimension. It will be removed once training
is accomplished. We denote the projected output
$\varphi\left(\mathbf{s}_{i}|\theta\right)$ as $\mathbf{s}_{i}^{\ast}$. The
concrete form of $\ell$ is:

$\ell=-\log{\frac{\exp{\left({\mathbf{s}_{i}^{\ast}}^{\mathsf{T}}\mathbf{t}_{i}/\tau\right)}}{\exp{\left({\mathbf{s}_{i}^{\ast}}^{\mathsf{T}}\mathbf{t}_{i}/\tau\right)}+\sum_{k}^{K}\exp{\left({\mathbf{s}_{i}^{\ast}}^{\mathsf{T}}\mathbf{n}^{k}/\tau\right)}}},$
(4)

where $\tau$ is a temperature hyper-parameter. After back-propagation,
teacher’s feature maps, $\mathbf{t}$, will be global pooled, $\ell
2$-normalized, and enqueued as a negative sample used for subsequent
iterations. For notational convenience, we omit the $\ell 2$-normalization
applied to $\mathbf{s}_{i}^{\ast}$ and $\mathbf{t}_{i}$.

It is worth noting that $\mathbf{s}$ and $\mathbf{t}$ do not necessarily have
the same spatial size. In the case of $\mathbf{s}$ and $\mathbf{t}$ having
mismatched spatial size, we perform bilinear interpolation on $\mathbf{t}$ to
match the spatial size of $\mathbf{s}$. Further discussions are in Sec. 4.3.

SpatialAdaptor. For the models pre-trained by image-level SSL methods (_e.g_.,
[23, 9, 21, 6]), their projection/prediction heads (henceforth projection head
for simplicity) are the stacked fully-connected (fc) layers with batch
normalization (BN) layers and ReLU in between. They are used to project the
global pooled features output by backbones. If adopting these models as
teachers (a common practice in previous self-supervised distillation methods),
one has to remove the global pooling layer and the projection heads to be
compatible with our PCD. However, removing the well-trained projection heads
will break the integrity of teachers. This in turn incurs knowledge loss,
resulting in sub-optimal results for transfer learning. We will empirically
verify this in Sec. 4.3.

To meet the demand of utilizing the fruitful knowledge of projection heads, we
propose a SpatialAdaptor to adapting the projection heads to processing 2D
inputs. We next discuss a simple case where the projection head only contains
a fc layer ($f$) and ReLU ($\sigma$), to demonstrate how SpatialAdaptor works.
Previously, the feature maps $\mathbf{t}$ output by teacher’s backbone will be
global pooled and fed to the projection head:

$\displaystyle\sigma\left(f\left(\phi\left(\mathbf{t}\right)\right)\right)$
$\displaystyle=\sigma\left(f\left(\frac{1}{HW}\sum_{i}\mathbf{t}_{i}\right)\right)$
$\displaystyle=\sigma\left(\frac{1}{HW}\sum_{i}f\left(\mathbf{t}_{i}\right)\right).$
(5)

The second equality holds being a consequence of $f$’s linearity. By
interchanging $f$ and $\phi$, $f$ now acts on pixels rather than vectorized
features. It follows that $f$ can be reformulated into a $1{\times}1$
convolution (conv) layer with the stride of $1$. This simple case does not
consider the existence of any BN layer since fc and BN layers together can be
fused into a linear function and represented by $f$.

Furthermore, we present the Channel-Wise ReLU (CW-ReLU) that masks out the
channels whose mean are negative. Let $\sigma^{\ast}$ denote CW-ReLU. By
definition, we have:

$\sigma\left(\phi\left(\mathbf{x}\right)\right)=\phi\left(\sigma^{\ast}\left(\mathbf{x}\right)\right).$
(6)

This means the global pooling layer and the activation function are
“commutative”. From Sec. 3.1 and the commutative property, we can interchange
the global pooling layer and the activation function:

$\sigma\left(f\left(\phi\left(\mathbf{t}\right)\right)\right)=\phi\left(\sigma^{\ast}\left(f\left(\mathbf{t}\right)\right)\right).$
(7)

Note that $\phi$ in the right-hand side of Eq. 7 is only used for illustrating
the invariability of SpatialAdaptor. It will be omitted in actual use for
maintaining spatial information.

We have shown how a projection head composed of a fc layer and ReLU is adapted
by SpatialAdaptor to processing feature maps while not changing the feature
distribution (Fig. 3). As such, the integrity of teachers is maintained, and
our PCD is made compatible with teachers pre-trained by various SSL methods.
Though the case discussed above being so simple, SpatialAdaptor can easily
apply to complex projection heads with more stacked fc layers and ReLU.
SpatialAdaptor helps keep the integrity of teachers and makes our PCD
compatible with teachers pre-trained by various SSL methods.

We are aware that the models pre-trained by pixel-level SSL methods [56, 50]
have projection heads processing 2D inputs. But this does not deprive the
significance of SpatialAdaptor, because we do not want too much constraints on
teacher’s pre-training methods. Such accessibility can also be seen as a
strength of our PCD.

Multi-head self-attention. Apart from the granularity of distillation signals,
the intrinsic properties (_e.g_., limited capacity and smaller receptive
field) of students also play a key role in the performance of dense prediction
tasks. Consider the effective receptive field (ERF) [37] for an example. ERF
measures how much each pixel contributes to the final prediction and has been
proven to be closely related to the performance of abundant computer vision
tasks [7, 13]. Intuitively, models with larger ERF are able to capture
information from a bigger area of image, leading to more robust and reliable
predictions.

According to [37], we draw the ERFs of ResNet-50 and ResNet-18 [25] in Fig. 4.
We can see a clear contrast that ResNet-50 has larger ERF (larger bright
region) than ResNet-18. Therefore, it is unrealistic to expect small models to
perfectly capture rich spatial information like teachers without outside help.

The definition of ERF indicates allowing more pixels to participate in
predictions helps enlarge ERF. From this perspective, we can enhance the
student model by explicitly relating all pixels right before making
predictions. This is made possible by a multi-head self-attention (MHSA)
module [49]. Information from different pixels are aggregated together to make
a more robust prediction. This module only induces a handful of memory and
computation cost for training, and does not influence the fine-tuning phase.
Equipped with the MHSA module, the ERF of ResNet-18 is slightly enlarged
(shown in the third picture of Fig. 4).

Figure 4: Effective receptive field. R50 and R18 are ResNet-50 and ResNet-18
respectively. R18 w/ MHSA stands for ResNet-18 enhanced by a multi-head self-
attention module.

###  3.2 Baseline Settings

In this section, we provide the necessary details for implementing our PCD.

Teacher. In most of our experiments, we adopt the ResNet-50 pre-trained by
MoCo v3††The checkpoint can be found in
https://github.com/facebookresearch/moco-v3/blob/main/CONFIG.md [12] for 1000
epochs as the teacher. We use the momentum updated branch (_i.e_.,
momentum_encoder of the checkpoint) of MoCo v3 model. The projection head has
the following structure: FC–BN–ReLU–FC–BN. The last BN has no affine
parameters. We drop it because its static normalization statistics slow down
the convergence. The global pooling layer of the backbone is removed. Modified
by SpatialAdaptor, the structure of the projection head becomes Conv–BN–CW-
ReLU–Conv. The hidden and output dimension of the projection head is $4096$
and $256$ respectively. The shape of output feature maps is
$256{\times}7{\times}7$ (channels ${\times}$ spatial size).

Student. By default, we use ResNet-18 as student’s backbone. It is followed by
a projection head ($\varphi$ mentioned above). We follow the asymmetric
structure design in [21, 11, 12] and instantiate $\varphi$ with two
consecutive MLPs. The MLPs are structurally similar to teacher’s modified
projection head, except that the activation function is ReLU and that the
input dimension of the first MLP is identical to the output dimension of
student’s backbone. Next, we append a regular MHSA module [49] to the end of
$\varphi$. The MHSA has $8$ heads, each with an embedding dimension of $64$.
In each head, the feature maps are fed to three $1{\times}1$ conv layers to
produce queries, keys, and values, respectively. The attention values of all
heads are concatenated and then projected by a $1{\times}1$ conv layer to
match with input features in dimension. The output of the MHSA module is used
for computing the contrastive distillation loss.

Distilling. We perform self-supervised distillation on the ImageNet [45]
training set. The image augmentation policy is the same as that proposed in
[21], comprised of two distributions of augmentation. It generates a pair of
augmented views for an input image. Our PCD computes distillation loss on each
view and optimizes the mean of two losses. We observe this loss symmetrization
brings about better convergence. We must point out that PCD with asymmetric
loss also provides bright transfer performance (see in Sec. 4.2).

We use the LARS [60] optimizer to train for 100 epochs. The batch size is
1024. The base learning rate ($lr$) is set to be $1.0$ and scaled by the
linear scaling rule [20]: $lr{\times}\text{batch}\\_\text{size}/256$. The
learning rate is linearly increased to $4.0$ for the first 10 epochs of
training (warmup) and then decayed to $0$ based on the cosine schedule. We use
a weight decay of $0.00001$ and a momentum of $0.9$. All biases and the affine
parameters of BN layers are excluded from the weight decay. We set the
temperature of contrastive loss to be $0.2$. The queue storing negative
samples has a capacity of $65536$.

##  4 Experiments

###  4.1 Evaluation Setup

|  ImageNet | VOC 07 | VOC 07+12 | COCO-C4 | COCO-FPN |  CityScapes
---|---|---|---|---|---|---
|  Acc |  AP${}_{\text{50}}$ |  AP |  AP${}_{\text{50}}$ |  AP |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  IoU
Teacher |  74.6 |  77.8 |  48.1 |  83.0 |  56.7 |  37.4 |  32.8 |  40.7 |  36.9 |  73.5
ImageNet Supervised |  69.8 |  70.0 |  38.2 |  76.9 |  47.3 |  30.7 |  28.0 |  36.3 |  33.0 |  70.2
MoCo v2 [10] |  48.7 |  69.7 |  40.2 |  77.5 |  49.7 |  30.7 |  27.9 |  35.0 |  31.8 |  70.4
PixPro [56] |  41.4 |  71.5 |  42.3 |  78.5 |  51.1 |  30.9 |  28.1 |  35.8 |  32.6 |  70.3
CompRess [1] |  63.9 |  71.3 |  41.2 |  78.4 |  50.4 |  31.4 |  28.4 |  35.7 |  32.4 |  70.3
DisCo [19] |  63.5 |  62.5 |  30.7 |  72.6 |  40.1 |  28.2 |  25.8 |  36.0 |  32.8 |  69.3
BINGO [57] |  64.2 |  70.4 |  39.9 |  77.8 |  49.3 |  31.1 |  28.2 |  36.2 |  32.8 |  71.0
PCD, asymmetric |  64.2 |  72.7 |  42.7 |  79.3 |  51.5 |  31.9 |  28.8 |  37.0 |  33.7 |  71.6
PCD |  65.1 |  73.0 |  43.2 |  79.4 |  52.1 |  32.2 |  29.0 |  37.4 |  34.0 |  71.8
Table 1: Comparing different pre-trained models. All pre-trained models adopt
ResNet-18 as backbone. ImageNet supervised pre-trained model is from the model
zoo of PyTorch [41]. Other pre-trained models are from our reproductions built
on their officially released codes. For fair comparisons, we pre-trained these
models for 100 epochs. We use the ResNet-50 pre-trained by MoCo v3 as teacher
for all self-supervised distillation methods. The best results are marked in
bold, and the second best are marked in gray (exclusive of the teacher).

Here, we provide some background for our fine-tuning experiments. We evaluate
our PCD on VOC [17] object detection, COCO [34] object detection and instance
segmentation, and CityScapes [15] semantic segmentation. The codebase for
evaluation is Detectron2 [53]. We strictly follow the fine-tuning settings
proposed in [23, 56] for fair comparisons. We also provide the linear probing
accuracy on ImageNet.

It has been a notorious problem that fine-tuning LARS-trained models with
optimization hyper-parameters best selected for SGD-trained counterparts
yields sub-optimal performance [11, 31]. To address this, recent work [31]
proposes NormRescale to scale the norm of LARS-trained weights by a specific
anchor (_e.g_., the norm of SGD-trained weights or a constant number). It
helps the LARS-trained models fit to the optimization strategy of fine-tuning.
When fine-tuning C4 or FCN backbones pre-trained by PCD, we employ this
technique to multiply the weights by a constant $0.25$. Multiplying a constant
is an efficient choice for not introducing extra training cost.

VOC object detection. We use a C4 backbone with Faster R-CNN [42] detector. We
evaluate the pre-trained models under two fine-tuning settings. The first is
to train on trainval2007 set for 9k iterations, and the second is to train on
trainval07+12 set for 24k iterations. We use the same fine-tuning settings as
per [23]. Fine-tuned models are evaluated on test2007 set. For better
reproducibility, we report the average AP50 and AP over 5 runs.

COCO object detection and instance segmentation. We use two types of backbone,
C4 and FPN, for fine-tuning on COCO dataset. The detector is Mask R-CNN [24].
Pre-trained models are fine-tuned on train2017 set according to the
$1{\times}$ optimization setting (about 12 COCO epochs). We report
AP${}^{\text{bbox}}$ for object detection and AP${}^{\text{mask}}$ for
instance segmentation on val2017 set.

CityScapes semantic segmentation. We implement a FCN-like [36] structure based
on pre-trained backbones. A newly initialized BN layer is added to the end of
pre-trained backbones for helping optimization. Subsequently, we append two
atrous convolutional blocks, each with a $3{\times}3$ conv layer of 256 output
channels, a BN layer, and ReLU. The conv layers have stride 1, dilation 6, and
padding 6. The prediction layer is a $1{\times}1$ conv layer with $19$ output
channels ($19$ classes), whose outputs are then bilinearly interpolated to
match the size of input images. Fine-tuning takes 90k iterations on train_fine
set. More detailed information of fine-tuning can be found in Appendix. We
report the average IoU on val set over 5 runs.

Linear probing in ImageNet. We freeze the backbones pre-trained by PCD and
train a linear classifier on the ImageNet training set. We use nesterov SGD to
train for 100 epochs. The batch size is 1024, and the learning rate is $0.8$
(base lr $=0.2$). We decay the learning rate to $0$ according to the cosine
schedule without restart. The momentum is $0.9$, and the weight decay is $0$.
We use a vanilla image augmentation policy containing random cropping,
resizing to $224{\times}224$, and horizontal flipping. We report the single-
crop classification accuracy on the ImageNet validation set.

###  4.2 Main Results

In Tab. 1, we compare our PCD to the following competitive pre-training
methods, supervised pre-training, SSL methods, and self-supervised
distillation methods. Our PCD shows impressive generalization capacity: it
outperforms all competitors on dense prediction tasks. We achieves $37.4$
AP${}^{\text{bbox}}$ and $34.0$ AP${}^{\text{mask}}$ on COCO using Mask R-CNN
detector and ResNet-18-FPN backbone, emerging as the first pre-training method
surpassing the supervised pre-trained model on this benchmark. Under the
linear probing protocol, our PCD also achieves decent top-1 accuracy
($65.1\%$), which makes PCD a well-rounded self-supervised distillation
method.

We notice that some competitors (_e.g_., supervised learning, MoCo v2 [10],
and CompRess [1]) are pre-trained by asymmetric loss. Here, we provide a
asymmetric variant of PCD to exclude the effect of loss symmetrization. The
change is simple: we adopt the symmetric augmentation policy as per [21], and
sample one augmented view from each input image during training. Indeed, the
symmetric loss endows PCD with better performance, but asymmetric variant
still achieves the second best results on all tasks (marked as gray in Tab.
1). PCD’s nontrivial advantages against other self-supervised distillation
methods have confirmed the fact that pixel-level distillation signals are the
key to transferring knowledge conducive to dense prediction tasks.

###  4.3 Ablation Experiments

We perform extensive ablation experiments to analyze PCD. Unless specified, we
adopt the training settings mentioned in Sec. 3.2.

| VOC 07+12 | COCO-FPN |  CityScapes
---|---|---|---
|  AP${}_{\text{50}}$ |  AP |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  IoU
image-level |  78.7 |  49.8 |  36.5 |  33.3 |  71.1
pixel-level |  79.4 |  52.1 |  37.4 |  34.0 |  71.8
Table 2: Pixel-level _vs_. image-level. We compare PCD to image-level contrastive distillation. The backbone is ResNet-18 and pre-trained for 100 epochs. The best results are marked in bold.  | VOC 07+12 | COCO-FPN |  CityScapes
---|---|---|---
|  AP${}_{\text{50}}$ |  AP |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  IoU
(a) |  76.0 |  48.8 |  36.5 |  33.2 |  70.1
(b) |  77.9 |  50.6 |  36.8 |  33.6 |  70.7
(c) |  77.3 |  50.1 |  36.3 |  33.1 |  69.6
(d) |  77.8 |  50.7 |  36.6 |  33.2 |  70.1
ours |  79.4 |  52.1 |  37.4 |  34.0 |  71.8
Table 3: Ablations on SpatialAdaptor. We compare four variants of PCD (a-d) to
examine the necessity of SpatialAdaptor. The backbone is ResNet-18 and pre-
trained for 100 epochs. The best results are marked in bold.

Pixel-level _vs_. image-level. To directly compare pixel-level and image-level
distillation, we develop an image-level variant of PCD. This variant
vectorizes student’s and teacher’s output feature maps by an extra global
average pooling layer and computes contrastive loss on these vectorized
features. It has highly competitive results (Tab. 2) like those image-level
self-supervised distillation methods in Tab. 1, revealing the effectiveness of
contrastive loss used for self-supervised distillation. But it still lags
behind the original PCD on all downstream tasks. This gap further confirms the
importance of pixel-level distillation signal.

| VOC 07+12 | COCO-FPN |  CityScapes
---|---|---|---
|  AP${}_{\text{50}}$ |  AP |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  IoU
w/o MHSA |  79.2 |  51.8 |  37.1 |  33.7 |  71.6
extra pred. |  79.1 |  52.0 |  37.2 |  33.7 |  71.2
ours |  79.4 |  52.1 |  37.4 |  34.0 |  71.8
Table 4: Ablations on MHSA. We ablate the design of MHSA. “extra pred.” stands for replacing the MHSA module by an extra prediction head. The backbone is ResNet-18 and pre-trained for 100 epochs. The best results are marked in bold.  backbone |  pre-training method | VOC 07 | VOC 07+12 | COCO-C4 | COCO-FPN |  CityScapes
---|---|---|---|---|---|---
|  AP${}_{\text{50}}$ |  AP |  AP${}_{\text{50}}$ |  AP |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  IoU
ResNet-50 |  SwAV |  72.1 |  41.7 |  79.0 |  51.2 |  31.9 |  28.8 |  37.1 |  33.7 |  70.7
ResNet-50 |  BYOL |  71.7 |  42.6 |  78.9 |  51.5 |  32.0 |  29.0 |  37.0 |  33.7 |  71.0
ResNet-50 |  Barlow Twins |  70.7 |  41.6 |  78.4 |  50.7 |  31.3 |  28.4 |  35.9 |  32.7 |  70.9
ViT-Base |  MoCo v3 |  70.6 |  41.7 |  78.1 |  51.1 |  31.3 |  28.5 |  36.6 |  33.6 |  71.4
Table 5: Fine-tuning results of PCD with different teachers. The student backbone is ResNet-18. All teacher models are from the officially released checkpoints. We refer to Appendix for more details about these teachers.  |  pre-training | VOC 07 | VOC 07+12 | COCO-C4 | COCO-FPN |  CityScapes
---|---|---|---|---|---|---
|  method |  AP${}_{\text{50}}$ |  AP |  AP${}_{\text{50}}$ |  AP |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  AP${}^{\text{bbox}}$ |  AP${}^{\text{mask}}$ |  IoU
ResNet-34 |  supervised |  74.8 |  45.0 |  81.0 |  55.0 |  37.7 |  32.9 |  39.4 |  35.6 |  71.9
PCD |  76.3 |  49.1 |  82.0 |  58.2 |  38.2 |  33.7 |  40.9 |  37.0 |  73.2
ResNet-50 |  supervised |  75.2 |  44.5 |  81.5 |  54.1 |  38.2 |  33.5 |  40.2 |  36.3 |  72.3
PCD |  77.0 |  49.0 |  82.8 |  57.7 |  40.1 |  34.9 |  42.4 |  38.1 |  73.3
MobileNet v3 |  supervised |  67.9 |  36.7 |  76.2 |  46.4 |  30.6 |  27.9 |  35.8 |  32.7 |  68.3
PCD |  73.9 |  40.8 |  79.3 |  49.9 |  32.8 |  29.6 |  37.7 |  34.3 |  69.0
Table 6: Fine-tuning results of PCD with different student backbones. The
teacher used for PCD is ResNet-50 pre-trained by MoCo v3. The student
backbones are ResNet-18, ResNet-34, ResNet-50, and MobileNet v3 (Large). We
also fine-tune the supervised pre-trained counterparts for contrast. We refer
to Appendix for more implementation details. The best results for each
backbone are marked as bold.

Ablation on SpatialAdaptor. We examine the necessity of SpatialAdaptor for
learning competitive representations. Without resorting to SpatialAdaptor, we
remove teacher’s projection head (along with the global pooling layer) and
simply use the feature maps output by teacher’s backbone for computing
contrastive loss (variant (a)). The evaluation metrics AP50 and AP on VOC are
rather low (Tab. 3 (a)). This variant overlooks the fact that teacher’s
feature maps have numerous zeros post-processing by ReLU while student’s do
not. Contrasting two features from different distributions naturally leads to
sub-optimal results.

For more reasonable comparisons, we introduce two more variants extended from
variant (a). Variant (b) adds ReLU after the MHSA module. Variant (c) removes
the MHSA module and adds ReLU after student’s projection head. Overall, these
two variants (Tab. 3 (b-c)) are still significantly worse than PCD. And they
are no better than the image-level self-supervised distillation methods in
Tab. 1 and Tab. 2. We argue that preserving the integrity of teachers (with
the help of SpatialAdaptor) is of vital importance to pixel-level
distillation. Otherwise, it will notably lower the quality of learned
representations.

Additionally, we study the impact of invariability of SpatialAdaptor by
replacing CW-ReLU with ReLU (variant (d)). We observe that keeping the
distribution of teacher’s learned features unchanged has massive gains on
dense prediction tasks (Tab. 3 (d)). In sum, SpatialAdaptor is an essential
component of PCD, enabling more informative pixel-wise distillation from
teachers pre-trained by image-level SSL methods.

Ablation on multi-head self-attention. The MHSA module of PCD aims to
explicitly relate pixels. We ablate this design in Tab. 4. Although removing
the MHSA module meets slight performance drop, it can still be regarded as a
competitive framework for self-supervised distillation. Another plausible
explanation for the positive impact of the MHSA module is that it works like a
prediction head to promote the quality of learned representations [21, 11,
12]. We thus supersede the MHSA module by an extra prediction head of roughly
the same parameters like the MHSA module. This design does not bring any
improvement (Tab. 4), suggesting that explicitly relating pixels is useful for
PCD.

Different teachers. Both MoCo v3 and PCD are trained with contrastive loss. To
exclude the positive or negative effect induced by optimizing the same type of
loss, we consider using teachers pre-trained by SwAV [6], BYOL [21], and
Barlow Twins [62]. The fine-tuning results of Tab. 5 show that these teachers
can also inspire favorable representations. It can be concluded that PCD is
robust to the choices of teacher models.

Beyond typical convolutional architectures, we try using ViT [16] as the
teacher to study the effect of cross-architectures distillation. There is an
innate obstacle for ResNet-18 to imitate ViT at pixel-level, since they differ
in the resolutions of output feature maps. Therefore, we downsample††ViT-Base
has $14{\times}14$ output patches for $224{\times}224$ input, and we treat
each patch as a pixel. ViT’s output by a $2{\times}2$ average pooling layer
with a stride of 2. It leads to acceptable results on dense prediction tasks,
whereas unsatisfying linear top-1 accuracy ($57.6\%$) on ImageNet. We believe
cross-architectures distillation (between CNNs and transformers) is a
noteworthy problem for future research.

Different students. We investigate the effectiveness of PCD on different
student backbones: ResNet-34, ResNet-50, and MobileNet v3 (Large) [30].
Compared to supervised pre-training, our PCD consistently outperforms on all
backbones (Tab. 6). A clear trend is that backbones with larger capacity (from
ResNet-18 to ResNet-50) have better transfer performance. A distilled
ResNet-34 or ResNet-50 can rival or even beat the teacher (referred to Tab. 1)
on some downstream tasks, marking the practicability of our PCD.

##  5 Conclusion

Self-supervised pre-training has shown its great potential for many computer
vision tasks. But its remarkable progress seems to be bound together with
large models. Practitioners find it struggling to apply SSL methods on small
models. Considering the necessity of small models for edge devices or resource
constraint regime, it is much essential to tackle this problem. Recently,
self-supervised knowledge distillation methods have been proposed to
effectively boost the accuracy of small models on classification tasks.

We study another equally important yet unexplored problem: how to promote the
transfer performance of small models on dense prediction tasks. From our
extensive experiments, we find it difficult for small models to learn pixel-
level knowledge from image-level pretext tasks, even with distillation
signals. Such observation is in contrast to the case of large models:
ResNet-50 pre-trained by MoCo v2 shows competitiveness on VOC and COCO object
detection. This can be interpreted as the inefficiency of small models to
implicitly learn from image-level pretext tasks. We therefore are inspired to
propose the first pixel-level self-supervised distillation framework, enabling
distilling a more competitive student on dense prediction tasks. We hope our
observations and experiments can motivate future research.

## References

  * [1] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash.  Compress: Self-supervised learning by compressing representations.  Advances in Neural Information Processing Systems, 33:12980–12992, 2020.
  * [2] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  Advances in neural information processing systems, 27, 2014.
  * [3] Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg.  Point-level region contrast for object detection pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16061–16070, 2022.
  * [4] Prashant Bhat, Elahe Arani, and Bahram Zonooz.  Distill on the go: Online knowledge distillation in self-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2678–2687, 2021.
  * [5] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541, 2006.
  * [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems, 33:9912–9924, 2020.
  * [7] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [8] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia.  Distilling knowledge via knowledge review.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008–5017, 2021.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [11] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [12] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021.
  * [13] Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Jiashi Feng.  Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3435–3444, 2019.
  * [14] Hee Min Choi, Hyoa Kang, and Dokwan Oh.  Unsupervised representation transfer for small networks: I believe i can distill on-the-fly.  Advances in Neural Information Processing Systems, 34:24645–24658, 2021.
  * [15] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [17] Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [18] Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu.  Seed: Self-supervised distillation for visual representation.  arXiv preprint arXiv:2101.04731, 2021.
  * [19] Yuting Gao, Jia-Xin Zhuang, Ke Li, Hao Cheng, Xiaowei Guo, Feiyue Huang, Rongrong Ji, and Xing Sun.  Disco: Remedy self-supervised learning on lightweight models with distilled contrastive learning.  arXiv preprint arXiv:2104.09124, 2021.
  * [20] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [22] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [26] Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and João Carreira.  Efficient visual pretraining with contrastive detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10086–10096, 2021.
  * [27] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921–1930, 2019.
  * [28] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [29] Frank L Hitchcock.  The distribution of a product from several sources to numerous localities.  Journal of mathematics and physics, 20(1-4):224–230, 1941.
  * [30] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * [31] Junqiang Huang, Xiangwen Kong, and Xiangyu Zhang.  Revisiting the critical factors of augmentation-invariant representation learning.  In European Conference on Computer Vision, pages 42–58. Springer, 2022.
  * [32] Jangho Kim, SeongUk Park, and Nojun Kwak.  Paraphrasing complex network: Network compression via factor transfer.  Advances in neural information processing systems, 31, 2018.
  * [33] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [34] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [35] Songtao Liu, Zeming Li, and Jian Sun.  Self-emd: Self-supervised object detection without imagenet.  arXiv preprint arXiv:2011.13677, 2020.
  * [36] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [37] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  Advances in neural information processing systems, 29, 2016.
  * [38] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [39] Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash.  Boosting self-supervised learning via knowledge transfer.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9359–9367, 2018.
  * [40] Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville.  Unsupervised learning of dense visual representations.  Advances in Neural Information Processing Systems, 33:4489–4500, 2020.
  * [41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [43] Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim.  Spatially consistent representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1144–1153, 2021.
  * [44] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [46] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In European conference on computer vision, pages 776–794. Springer, 2020.
  * [47] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders.  Selective search for object recognition.  International journal of computer vision, 104(2):154–171, 2013\.
  * [48] Ruth Urner, Shai Shalev-Shwartz, and Shai Ben-David.  Access to unlabeled data can speed up prediction time.  In ICML, 2011.
  * [49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [50] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024–3033, 2021.
  * [51] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin.  Aligning pretraining for detection via object-level contrastive learning.  Advances in Neural Information Processing Systems, 34:22682–22694, 2021.
  * [52] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.  Tinyvit: Fast pretraining distillation for small vision transformers.  arXiv preprint arXiv:2207.10666, 2022.
  * [53] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [54] Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell.  Region similarity representation learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10539–10548, 2021.
  * [55] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy.  Unsupervised object-level representation learning from scene images.  Advances in Neural Information Processing Systems, 34:28864–28876, 2021.
  * [56] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16684–16693, 2021.
  * [57] Haohang Xu, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian.  Bag of instances aggregation boosts self-supervised distillation.  In International Conference on Learning Representations, 2021.
  * [58] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan.  Clusterfit: Improving generalization of visual representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6509–6518, 2020.
  * [59] Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin.  Instance localization for self-supervised detection pretraining.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3987–3996, 2021.
  * [60] Yang You, Igor Gitman, and Boris Ginsburg.  Large batch training of convolutional networks.  arXiv preprint arXiv:1708.03888, 2017.
  * [61] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * [62] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  * [63] Kai Zheng, Yuanjiang Wang, and Ye Yuan.  Boosting contrastive learning with relation knowledge distillation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3508–3516, 2022.

##  Appendix A Details of Fine-Tuning Experiments

###  A.1 Fine-Tuning on CityScapes

The fine-tuning hyper-parameters are listed as follows:

Configuration |  Value
---|---
optimizer |  SGD
base learning rate |  0.01
momentum |  0.9
weight decay |  1e-4
batch size |  16
training steps |  90000
learning rate schedule |  WarmupMultiStepLR
warmup iters |  1000
decay milestones |  63000, 81000
shortest edge of resizing |  $\left[512,768,...,2048\right]$
max input size |  4096
random cropping |  True
random flipping |  True

###  A.2 Fine-Tuning MobileNet v3 (Large)

MobileNet v3 (Large) [30] is not a stage-wise architecture like ResNet series
[25]. We have to manually define the “stem” and “res” stages in MobileNet v3
(Large) to fit in Detectron2. The rules for partitioning the modules of
MobileNet v3 (Large) are: i) modules with the same stride belong to the same
partition; ii) res4 must be of stride 16. We show the partitioning results:

index |  modules |  total stride |  partitions
---|---|---|---
1 |  conv |  2 |  stem
2 |  IBN |  2
3 |  IBN |  4 |  res2
4 |  IBN |  4
5 |  IBN |  8 |  res3
6 |  IBN |  8
7 |  IBN |  8
8 |  IBN |  16 |  res4
9 |  IBN |  16
10 |  IBN |  16
11 |  IBN |  16
12 |  IBN |  16
13 |  IBN |  16
14 |  IBN |  32 |  res5
15 |  IBN |  32
16 |  IBN |  32
17 |  conv |  32

###  A.3 Distilling from Different Teachers

In Tab. 5, we try using other models as teachers. Most of checkpoints are from
the official repository, except for BYOL [21]. The ResNet-50 [25] pre-trained
by BYOL are from the implementation of [31]. We list out the URLs for
downloading these models:

  * •

MoCo v3 (ResNet-50) [12]:
https://dl.fbaipublicfiles.com/moco-v3/r-50-1000ep/r-50-1000ep.pth.tar

  * •

MoCo v3 (ViT-Base) [12]:
https://dl.fbaipublicfiles.com/moco-v3/vit-b-300ep/vit-b-300ep.pth.tar

  * •

SwAV [6]:
https://dl.fbaipublicfiles.com/deepcluster/swav_800ep_pretrain.pth.tar

  * •

BYOL [21]:
https://drive.google.com/file/d/1-5-O49vsro9YW9WTokSc8CoSrmjKfieB/view?usp=sharing

  * •

Barlow Twins [62]:
https://dl.fbaipublicfiles.com/barlowtwins/ljng/checkpoint.pth

Barlow Twins has a projection head with an output dimension of $8192$. We have
to use the variant of PCD with asymmetric loss to distill knowledge from
Barlow Twins due to limited memory.