# Guided Distillation for Semi-Supervised Instance Segmentation

Tariq Berrada1,2 Camille Couprie1 Karteek Alahari2 Jakob Verbeek1
1 Meta AI, 2 Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK

###### Abstract

Although instance segmentation methods have improved considerably, the
dominant paradigm is to rely on fully-annotated training images, which are
tedious to obtain. To alleviate this reliance, and boost results, semi-
supervised approaches leverage unlabeled data as an additional training signal
that limits overfitting to the labeled samples. In this context, we present
novel design choices to significantly improve teacher-student distillation
models. In particular, we (i) improve the distillation approach by introducing
a novel “guided burn-in” stage, and (ii) evaluate different instance
segmentation architectures, as well as backbone networks and pre-training
strategies. Contrary to previous work which uses only supervised data for the
burn-in period of the student model, we also use guidance of the teacher model
to exploit unlabeled data in the burn-in period. Our improved distillation
approach leads to substantial improvements over previous state-of-the-art
results. For example, on the Cityscapes dataset we improve mask-AP from 23.7
to 33.9 when using labels for 10% of images, and on the COCO dataset we
improve mask-AP from 18.3 to 34.1 when using labels for only 1% of the
training data.

Figure 1:  Compared with the state-of-the-art Polite Teacher method [8], we
achieve +15.7 mask-AP when using 1% of labels, for an AP of 34.0, which is
more than what Polite Teacher achieved using 10 times more labels (30.8) on
the COCO dataset.  Figure 2: Results on the COCO dataset when using from 5%
labeled data, with supervised training (top) and our semi-supervised approach
(bottom). Our approach leads to more detected objects segmented with higher
accuracy. Results best viewed by zooming in.

##  1 Introduction

The goal of instance segmentation is to extract a segmentation mask and a
class label for all relevant object instances in an image. This generalizes
both object detection, for which each instance is localized only with a
bounding box, as well as semantic segmentation, in which different instances
of the same class can be lumped together in the same mask. Although recent
segmentation models have brought considerable improvements in performance to
this task, see e.g. [9, 2], these methods require large amounts of images with
pixel-level annotations in order to perform well. Consequently, the difficulty
of manually producing precise instance segmentation maps for training forms a
barrier for applying such models to settings where relevant labeled data sets
are not readily available.

Semi-supervised learning can be used in label-scarce situations by making use
of unlabeled images to boost performance: for the unlabeled images the
labeling process is avoided, while they still provide useful problem specific
input data, see e.g. [13, 10, 20, 23]. For semi-supervised instance
segmentation, the “Noisy Boundaries” method [24] builds upon the detection-
based Mask-RCNN model [9], and leverages pixel-level pseudo labels on
unlabeled images, while also making the training of the mask prediction head
more robust to noise. It achieves comparable performance to Mask-RCNN while
utilizing only 30% of labeled images. While providing an important gain in
label efficiency, this approach relies on a fixed Mask-RCNN teacher model,
which is trained in supervised mode from the available labeled images, which
limits its performance in settings with few labeled images.

The Polite Teacher approach [8] is currently state-of-the-art for semi-
supervised instance segmentation. It performs distillation on a CenterMask2
[12] model with a joint-training framework, where the teacher is updated as an
exponential moving average (EMA) of the student model. Low confidence
thresholding on bounding-box and mask scores is used to filter out noisy
pseudo labels. In our work, we also adopt a student-teacher training with EMA,
with several important differences w.r.t. the Polite Teacher approach: we
introduce a novel burn-in step in the distillation process, and adopt an
improved Mask2Former segmentation architecture [2] in combination with Swin
[16] and DINOv2 [18] feature backbones.

To summarize, our contributions are the following :

  * •

We develop a novel teacher-student distillation strategy to more effectively
leverage unlabeled images.

  * •

We leverage vision transformer architectures for the first time in the context
of semi-supervised instance segmentation.

  * •

We show largely improved performance in terms of mask-AP for both Cityscapes
and COCO datasets across a wide range of labeled data budgets.

##  2 Related work

In this section, we review instance segmentation models and their applications
in the semi-supervised regime.

Segmentation models. Instance segmentation aims at identifying the semantic
class of every pixel in the image, as well as associate each pixel with a
specific instance of the object class. This contrasts with semantic
segmentation which only aims at solving the first part. Instance segmentation
is more challenging because it requires identifying each individual object
instance, associating parts with the correct objects, and accurately
determining the boundaries of individual objects. These difficulties compound
with other challenges, such as variability in scale, color, lighting,
occlusions, and possibly motion blur. Until recently, the state-of-the-art was
held the detection based approaches such as Mask-RCNN [9] and follow-ups such
as ViTDet [14], with a component to detect object bounding-boxes and one for
binary segmentation to isolate the object from background pixels. Further
progress in transformers [3, 2] allowed instance segmentation approaches to
avoid a separate detection step. Additional gains have been brought by
leveraging text-level supervision using CLIP and very large models [7]. The
Segment Anything Model (SAM) also exploits text level supervision among other
kinds, being able to condition its output on a text prompt, bounding box, or
point coordinate. However, the output does not include class prediction and
thus differs from the instance segmentation task considered here. To our
knowledge, our work is the first one to explore the application of ViTs in the
context of semi-supervised instance segmentation. Recently, MaskFormer [3]
observed that mask classification is sufficiently general to solve both
semantic and instance-level segmentation tasks. Their approach converts per-
pixel classification into a mask classification model using a transformer-
based set prediction mechanism. Mask2Former [2] provides further improvements
and presents a universal segmentation model using the same mask classification
mechanism, an efficient masked attention module, multi-scale feature pooling
and other improved techniques for fast and efficient training.

Semi-supervised instance segmentation. Previous work has considered the use of
pseudo labeling to leverage unlabeled data for semi-supervised instance
segmentation [8, 24]. A “teacher” instance segmentation model is used to label
unlabeled images, which are then used by the “student” model in addition to
labeled images for which ground-truth instance segmentations are available. A
key factor for the success of such methods is some level of discrepancy
between the teacher and the student model, which ensures that the teacher
provides stable and strong training signal to the student by means of the
pseudo labels. If both models are identical, the student will not learn
anything new from the teacher. The first condition —stability— can be achieved
by using a fixed teacher, or using a teacher with the same architecture as the
student and updating it as an exponential moving average (EMA) from the
student [8, 21, 24]. The second condition —strong training signal— can be met
by using different data augmentations of the same image for the teacher and
student model: using strong augmentation for the student and weaker
augmentation for the teacher ensures high quality of the pseudo labels of the
teacher and useful training signal for the student being confronted with hard
prediction tasks on strongly augmented training data.

Among the few existing semi-supervised instance segmentation approaches, Noisy
Boundaries [24] performs distillation with a fixed Mask-RCNN teacher model
trained from labeled data only. The student network is learning a noise-
tolerant mask head for boundary prediction by leveraging low-resolution
features. The Polite Teacher [8] approach is closer to our work and also uses
EMA teacher updates. In this work we develop an improved burn-in stage, before
starting the main semi-supervised training loop, in which we pre-train the
student model from both labeled data as well as unlabeled data using pseudo
labels from a fixed teacher model. In contrast, Polite Teacher pre-trains the
student from supervised data only.

##  3 Method

Figure 3: Our semi-supervised training methodology consists of three steps.
(i) The teacher network is trained on labeled data only. (ii) The student is
initialized from scratch, and trained on both labeled data and unlabeled data
using pseudo labels generated by the pre-trained teacher. The teacher remains
fixed during this step. (iii) Copy the checkpoint of the student to the
teacher, and then continue training on both labeled and unlabeled data, while
using EMA updates for the teacher network from the student’s weights.  Figure
4: Workflow of our distillation stage. Student and teacher receive weakly and
strongly augmented versions of the unlabeled image. The student predictions
are filtered and one-hot encoded to obtain the pseudo-labels that serve as
supervision for the student model. We use the same bipartite instance matching
method as Mask2Former [2]. The teacher is then updated using EMA.

Let $D_{s}=\\{(x_{1},y_{1}),...,(x_{N},y_{N})\\}$ be a supervised dataset and
$D_{u}=\\{x^{\prime}_{1},...,x^{\prime}_{M}\\}$ an unlabeled set of images,
where $x$ denotes images, $y$ their segmentations, and $M\gg N.$ The goal is
to make use of $D_{u}$ in order to improve the performance of the model using
the available labels.

In Sec. 3.1 we present model architectures, our improved distillation
approach, and pseudo labeling strategy. In Sec. 3.2 we detail the losses and
EMA teacher updates to train our models.

###  3.1 Architectures, distillation and pseudo-labeling

Model architecture. We follow the Mask2former architecture [2] which achieves
strong results across different segmentation tasks (semantic, instance,
panoptic). For our study, we focus on the instance segmentation variant. The
model consists of a backbone for image-level feature extraction, a pixel
decoder to gradually upsample image features to compute per-pixel embeddings
and a transformer decoder that predicts $N$ mask and class embeddings which
generate $N$, possibly overlapping, binary masks via a dot product between the
image-level features and the class embeddings.

Teacher-student distillation. Our training pipeline can be divided into three
steps as illustrated in Fig. 3 :

  1. 1.

Teacher pre-training: the teacher model, parameterized by $\theta_{t}$, is
trained on annotated data only.

  2. 2.

Improved burn-in: The student model, parameterized by $\theta_{s}$, is
initialized randomly, then trained on both labeled data and unlabeled data
using pseudo labels provided by the pre-trained teacher model. The teacher
model remains fixed during this stage.

  3. 3.

Distillation stage: In this stage, we copy the student weights to the teacher,
and then train the student on both labeled and unlabeled data as before. The
teacher is updated using an EMA of the student’s weights.

Pseudo-label extraction. Mask2Former by default predicts a constant number of
$K\\!=\\!100$ masks, which we need to filter in order to obtain pseudo ground-
truths for the unlabeled images. To do this, we follow a simple thresholding
scheme that takes into account the predicted class probability and the
predicted mask size. A predicted mask is selected as a pseudo label if (i) the
maximum class probability is above the class threshold $p_{c}\geq\alpha_{C}$,
and (ii) the size of the predicted mask is above the size threshold:
$\sum_{p=0}^{H\times W}\sigma(\hat{y}_{(p)})\geq\alpha_{S}$, with $\sigma$ the
sigmoid activation of the binary mask prediction.

###  3.2 Optimisation

In this section we introduce the loss functions used to train the models, and
present our guided distillation approach in detail. An illustration of the
distillation training stage is provided in Fig. 4.

Loss. Following Mask2Former [2], we match predictions from the student and
pseudo ground-truth instances from the teacher by constructing a cost matrix
for bipartite matching similarly to DETR [1]. Once the matching is obtained,
we use a weighted sum of cross entropy and Dice loss for the masks and cross-
entropy loss for the class predictions. In order to minimize memory
requirements, we follow PointRend’s approach and only sample a limited number
of points from the high resolution masks using importance sampling [11].
Furthermore, this loss is computed for every auxiliary output of the model as
in [2].

Let $x_{i}$ be an image and $y_{i}=\\{(y^{k}_{i},c^{k}_{i})\\}_{k=1,\dots,n}$
its associated ground-truth instances defined by the $n$ binary masks
$y_{i}^{k}$ and class indices $c_{i}^{k}$. The model predicts $K$ candidate
instances $(\hat{y}^{k}_{i},\hat{c}^{k}_{i})_{1\leq k\leq K}=S_{m}\circ
G(x_{i};\theta)$, where $G$ is the backbone and $S_{m}$ the segmentation head.
Using bipartite matching, we obtain a matching between the ground truth and
predictions $(k,\tau_{k})_{1\leq k\leq n}$, which is used to calculate the
image-level loss for both masks and predicted classes:

$\displaystyle\mathcal{L}_{s}^{i}$ $\displaystyle\\!=\\!$
$\displaystyle\frac{1}{n}\sum_{k=1}^{n}l_{CE}(\hat{y}^{\tau_{k}}_{i},y^{k}_{i})+\lambda_{D}l_{D}(\hat{y}^{\tau_{k}}_{i},y^{k}_{i})$
(1) $\displaystyle+\lambda_{C}l_{C}(\hat{c}_{i}^{\tau_{k}},c_{i}^{k}),$
$\displaystyle\mathcal{L}_{s}$ $\displaystyle\\!=\\!$
$\displaystyle\frac{1}{|B|}\sum_{i\in B}\mathcal{L}_{s}^{i},$ (2)

where $l_{CE}$ is the cross-entropy loss for the class predictions, $l_{C}$
binary cross-entropy and $l_{D}$ the dice loss function, $\lambda_{C}$ and
$\lambda_{D}$ are scaling parameters, and $B$ is the training batch.

For the unsupervised loss, we follow the same procedure as the supervised
setting. To obtain the pseudo labels, we filter the predictions of the teacher
as described previously, and then define the pseudo labels by one-hot encoding
both the mask and class predictions. Let $x^{\prime}_{j}$ be an unlabeled
image and $y^{\prime}_{j}=(\hat{y}^{\prime k}_{j},\hat{c}^{\prime
k}_{j})_{1\leq k\leq K}=S_{m}\circ G(x^{\prime}_{j};\theta_{t})$ the
predictions of the teacher, then pseudo-labels are defined as:

$\begin{cases}c^{\prime k}_{j}&=\text{argmax}_{c\in C}c^{k}_{j},\\\ y^{\prime
k}_{j}&=\sigma(\hat{y}^{\prime k}_{j})\geq 0.5,\end{cases}$ (3)

Given the pseudo labels, we compute the unsupervised loss term
$\mathcal{L}_{u}$ analogous to how we compute the supervised loss
$\mathcal{L}_{s}$ in Eq. 1 and Eq. 2 above.

Finally, we combine the supervised and unsupervised loss in a weighted sum as:

$\mathcal{L}=\mathcal{L}_{s}+\lambda_{u}\mathcal{L}_{u}.$ (4)

Revisited burn-in stage. To ensure that the quality of the pseudo labels is
high enough, existing distillation-based semi-supervised methods adopt a burn-
in stage before utilizing the teacher in the training pipeline, _e.g_. as in
Polite Teacher [8]. During this stage, the student is trained on labeled data
only before copying its weights to the teacher. However, this is not ideal
since the student model will be more likely to converge towards a local minima
induced by its initial training where only the limited labeled data was taken
into account. Alternatively, we show that it is better to make use of the
teacher model from the beginning of training. To do this, we first train the
teacher model on labeled data only and then use it to provide pseudo labels
for training the student model on both labeled and unlabeled data. During this
initial training stage, which we refer as our revisited burn-in stage, the
teacher model stays constant and is therefore not influenced by the student
predictions. This enables the student model to learn from more data, in
particular in cases where very few labeled images are available in combination
with many unlabeled images, making it less prone to over-fitting on the
limited labeled training data. The length of our burn-in stage is proportional
to the percentage of labeled samples used, with the underlying idea that such
a stage should end when the student starts having better performance than the
fixed teacher. We recap in Tab. 1 the different training strategies from the
literature and highlight the specificity of our approach.

|  | Polite | Guided Distillation | Noisy
---|---|---|---|---
|  | Teacher | (Ours) | Boundaries
Init 0 |  | Random | Pre-train $T$ | Pre-train $T$
Burn-in | Student | on $D_{s}$ | on $D_{s}\\!+\\!D_{u}$ |
Teacher | — | Fixed |
Init 1 |  | $T\leftarrow S$ | $T\leftarrow S$ | —
Training | Student | on $D_{s}\\!+\\!D_{u}$ | on $D_{s}\\!+\\!D_{u}$ | on $D_{s}\\!+\\!D_{u}$
Teacher | EMA | EMA | Fixed

Table 1: Comparing our proposed distillation approach to the ones previously
used in Polite Teacher [8] and Noisy Boundaries [24].

EMA. During the second stage of training, the teacher model is updated with an
exponential moving average of the student’s weights. This approach has been
proven to stabilize training by providing more regular pseudo labels during
the student’s training [21]. Let $\theta_{s}^{n}$ and $\theta_{t}^{n}$ be the
student and teacher’s weights respectively at iteration $n$. The update rule
for the teacher network can be written as:

$\theta_{t}^{n+1}=\alpha\cdot\theta_{t}^{n}+(1-\alpha)\cdot\theta_{s}^{n+1},$
(5)

where $\alpha$ is the decay rate which regulates the contribution of the
student’s weights to the update in every iteration.

##  4 Experiments

Here we present our experimental setup in Sec. 4.1, and present our main
results followed by ablations in Sec. 4.2.

###  4.1 Experimental setup

Datasets and evaluation metric. We use two datasets in our experiments. The
Cityscapes dataset [4] that contains 2,975 training images of size $1024\times
2048$ taken from a car driving in German cities, labeled with 8 semantic
instance categories. Following [24], we train models using various percentages
of the available instance annotations, and evaluate using the 500 validation
images with public annotations. The different data splits are generated by
randomly selecting a random subset from the training images. For the semi-
supervised results we present with $100\%$ annotated images, we use the
additional 20k unlabeled images. The COCO dataset [15] has instance
segmentations for 118,287 training images. We adopt the same evaluation
setting as [8], using 1%, 2%, 5% and 10% of labeled images for semi-supervised
training and using the remaining training images as unlabeled. We use the same
supervised/unsupervised splits as earlier work [8, 17]. Following previous
work [8, 9, 24], we use the standard mask-AP metric to evaluate instance
segmentation quality.

Implementation. We implement our approach in the Detectron2 codebase [25]. By
default, experiments are conducted on one machine with 8 V100 GPUs with 32 GB
RAM each. In some cases less GPUs were needed, _e.g_. only 4 GPUs are needed
to train semi-supervised models with a R50 backbone on Cityscapes. For
experiments using Swin-L, ViT-B or Vit-L backbones on COCO semi-supervised
training we needed 16 GPUs across 2 machines to keep the same batch size. For
our experiments with these large backbones, we enabled automatic mixed
precision in order to reduce the memory footprint.

Hyper-parameters. Our hyper-parameters are set as follows. We set the EMA
decay rate to $\alpha=0.9996$, the unsupervised loss weight to
$\lambda_{u}=2$, the class threshold to $\alpha_{C}=0.7$ and the size
threshold to $\alpha_{S}=5$. For the loss weights, we follow [2] and set
$\lambda_{D}=5$ and $\lambda_{C}=1$. For Cityscapes, we train our models for
90k iterations, except for our semi-supervised model with the additional 20k
images and 100% of labeled images used, in which case we found it beneficial
to double the training length. For COCO, we train our models for $368$k
iterations. For all our models, we use AdamW with a learning rate of
$10^{-4}$, weight decay of $0.05$ and multiplier of $0.1$ for the backbone
updates, a batch size of $16$ and early stopping. The number of iterations
used for the burn-in period is given in Tab. 2.

Labels used | 1% | 2% | 5% | 10% | 20% | 30% | 100%
---|---|---|---|---|---|---|---
Cityscapes | — | — | 15k | 25k | 30k | 35k | 50k
COCO | 20k | 25k | 30k | 60k | — | — | —

Table 2: Number of burn-in iterations used in our approach.

###  4.2 Main results

Semi-supervised instance segmentation results. In Tab. 3(c) we compare our
results on the Cityscapes dataset to these of Noisy Boundaries (NB) [24]. We
provide results using different feature backbones, including the ImageNet-1k
pre-trained ResNet-50 backbone used in NB. We observe large improvements over
the latter using all backbones and percentages of labels used, with best
results obtained using the larger SSL DINOv2 and the supervised Swin-L
backbones. The DINOv2 backbone leads to best results (28.8 _vs_. 25.1 for
Swin) in the most challenging setting where we used only 5% of the labeled
data, while the Swin backbone improves by 0.5 to 1.3 points AP in the other
settings. Notably, compared to NB we improve the mask-AP by more than 10
points from 23.7 to 33.9 when using 10% of labels.

Figure 5: Overview of our results on Cityscapes with 10% labeled data with R50
backbone. Top: Predictions of model trained using $5\%$ of labeled data only.
Bottom : model predictions of model trained using our semi-supervised method.
Without our supervision method, the model tends to merge different instances
from the same semantic category (bounding boxes spanning multiple objects),
the model also misidentifies certain objects or by predicting a the wrong
class or only a certain portion of the instance. Ours method greatly helps in
alleviating all these issues.

Amount of labeled data used | 5% | 10% | 20% | 30% | 100%
---|---|---|---|---|---
Mask-RCNN [9] | 11.3 | 16.4 | 22.6 | 26.6 | 33.7
Mask2former - R50 (IN-1k) | 12.1 | 18.8 | 27.4 | 29.6 | 36.6
Mask2former - ViT-B (DINOv2) | 18.6 | 21.9 | 28.1 | 31.1 | 36.5
Mask2former - ViT-L (DINOv2) | 22.9 | 27.1 | 31.4 | 33.7 | 38.0
Mask2former - Swin-L (IN-21k) | 17.8 | 26.3 | 33.2 | 37.1 | 43.5

(a)  Supervised models on Cityscapes.

Amount of labeled data used | 1% | 2% | 5% | 10% | 100%
---|---|---|---|---|---
Mask-RCNN - R50 (IN-1k) [9] | 3.5 | 9.5 | 17.4 | 21.9 | 37.1
CenterMask2 - R50 (IN-1k) [12] | 10.1 | 13.5 | 18.0 | 22.1 | —
Mask2former - R50 (IN-1k) | 13.5 | 20.0 | 26.0 | 30.5 | 43.5
Mask2former - ViT-B (DINOv2) | 25.9 | 30.0 | 33.6 | 36.3 | 47.0
Mask2former - ViT-L (DINOv2) | 30.2 | 33.1 | 36.6 | 39.1 | 47.9
Mask2former - Swin-L (IN-21k) | 22.3 | 32.1 | 36.9 | 40.4 | 50.0

(b) Supervised models on COCO.

Amount of labeled data used | 5% | 10% | 20% | 30% | 100%
---|---|---|---|---|---
Noisy boundaries [24] | 21.2 | 23.7 | 30.8 | 33.2 | 34.7
Ours - R50 (IN-1k) | 23.0 | 30.8 | 33.1 | 35.6 | 39.6
Ours - ViT-B (DINOv2) | 25.1 | 27.0 | 34.6 | 35.4 | 39.6
Ours - ViT-L (DINOv2) | 28.8 | 33.0 | 36.8 | 39.1 | 42.9
Ours - Swin-L (IN-21k) | 25.1 | 33.9 | 38.1 | 39.6 | 43.8

(c)  Semi-supervised models on Cityscapes.

Amount of labeled data used | 1% | 2% | 5% | 10%
---|---|---|---|---
DD [19] | 3.8 | 11.8 | 20.4 | 24.2
Noisy boundaries [24] | 7.7 | 16.3 | 24.9 | 29.2
Polite teacher [8] | 18.3 | 22.3 | 26.5 | 30.8
Ours - R50 (IN-1k) | 21.5 | 25.3 | 29.9 | 35.0
Ours - ViT-B (DINOv2) | 30.5 | 34.3 | 37.0 | 39.2
Ours - ViT-L (DINOv2) | 34.1 | 37.4 | 40.3 | 42.0
Ours - Swin-L (IN-21k) | 34.0 | 38.2 | 41.6 | 43.1
(d) Semi-supervised models on COCO.

Table 3: Comparison of results on COCO and Cityscapes. Results for Mask-RCNN,
CenterMask2, Polite Teacher and Noisy Boundaries are taken from [8] and [24].
For each backbone, we indicate the pre-training data, DINOv2 is SSL pre-
trained on a dataset of 142M images.

In Tab. 3(d) we report results for the COCO dataset, where we compare to NB
and the Polite Teacher (PT) [8] approaches. We similarly observe substantial
improvements across all backbones and percentages of labeled data used. For
example, improving the 22.3 mask-AP of PT to 38.2 in the case of using 2% of
the labels. In this case the Swin-L backbone yields (near) optimal results.
Our results are better overall, but in particular for small amounts of labeled
data, which makes the performance curve as a function of the amount of labeled
data flatter, see also Fig. 1. For example, between the 1% and 10% label case,
Noisy boundaries improves from 7.7 to 29.2 AP (+21.5), and Polite Teacher
improves from 18.3 to 30.8 (+12.5), where in our case with the Swin-L backbone
the performance increase is 9.1 AP points (34.0 _vs_. 43.1).

On COCO, Wang _et al_. [24] also report a semi-supervised experiment using
100% of labeled data, and adding another 123k unlabeled COCO images. In this
setting they obtain 38.6 mask-AP, which is similar to the 38.2 mask-AP that we
attain using only 2% labeled data on the COCO-train, so using roughly
50$\times$ less labeled images and twice less unlabeled images.

Comparison to supervised baselines. In Tab. 3(a) and Tab. 3(b) we present
results obtained using supervised training only to assess the improvements
brought by our semi-supervised approach w.r.t. this baseline for different
backbone architectures. We also compare to the supervised baselines using the
Mask-RCNN and CenterMask2 segmentation models that underlie Noisy Boundaries
and Polite teacher, respectively.

Comparing the supervised and semi-supervised results for Cityscapes in Tab.
3(a) and Tab. 3(c), and similarly for COCO in Tab. 3(b) and Tab. 3(d), we
observe that our approach shows consistent improvements over the supervised
baseline across all experiments. Using the Swin-L backbone on Cityscapes in
the 5% labeled datacase, we improve the AP by 7.3 points from 17.8 to 25.1 AP,
while for the DINOv2 backbone we improve it by from 22.9 to 28.8 (+5.9 AP).
For COCO at 1% labeled data, our semi-supservised approach improves the
supervised result of 22.3 to 34.0 (+11.7 AP) using Swin-L, and from 30.2 to
34.1 (+3.9 AP) using the DINOv2 backbone.

See Fig. 1 for a graph comparing our semi-supervised results to the supervised
baseline as well as previous methods on COCO. We provide qualitative examples
of instance segmentation using the supervised baseline and our approach on the
COCO dataset in Fig. 2 and for Cityscapes in Fig. 5.

Smaller labeled training sets. We conducted additional experiments on the COCO
dataset using fewer labeled training samples compared to earlier work: from
0.1% to 0.4%. We report results for a supervised training baseline and our
semi-supervised approach in Tab. 4. In this extremely challenging, very
sparsely labeled data regime, our approach shows marked improvements over the
supervised baseline, _e.g_., +3.4 points for 0.1% labeled images using the
DINOv2 backbone. Moreover, we find that in this regime the DINOv2 backbone is
more effective than the Swin-L (IN-21k) backbone, _e.g_., in the 0.4% labeled
data case, corresponding to less than 500 of image annotations: the DINOv2
backbone improves mask-AP by 8 points from 23 to 31, reaching the same
performance as the previous state-of-the-art approach Polite Teacher with 25
times less annotations. This is in line with our observation on the 5% labeled
data case for Cityscapes in Tab. 3(c).

Amount of labeled data used | 0.1% | 0.2% | 0.4%
---|---|---|---
| Supervised models
Mask2former - Swin-L (IN-21k) | 5.3 | 10.2 | 15.9
Mask2former - ViT-L (DINOv2) | 10.2 | 19.4 | 25.7
| Semi-supervised models
Ours - Swin-L (IN-21k) | 5.8 | 16.1 | 23.0
Ours - ViT-L (DINOv2) | 13.6 | 24.9 | 31.0
Table 4: Evaluation of supervised and semi-supervised models on COCO using
extra small labeled training sets.

###  4.3 Interpretation and ablation studies

In the following, unless specified otherwise, the ablations are performed on
Cityscapes with 5% labeled data and a R50 backbone. Results are reported on
the validation set.

Revisited burn-in stage. In order to quantify the performance gains obtained
by our revisited burn-in stage in isolation of other architectural choices, we
compare it to the strategies used by Noisy Boundaries [24] and Polite Teacher
[8]. See Tab. 1 for a comparison. We also consider a fourth variant, which is
identical to our approach but setting the number of burn-in iterations to
zero. Finally, we include the result when using supervised training only as a
baseline result.

The results in Tab. 5 show that our approach brings about $+3.7$ AP in
comparison with the (standard) burn-in strategy used by Polite Teacher, and
$+4.1$ AP over keeping the teacher fixed during training as in Noisy
Boundaries. Dropping the burn-in stage all together gives worst results (16.9
AP), which underlines the importance of this burn-in stage in distillation
approaches.

Burn-in method | mask AP
---|---
Supervised baseline | 12.1
No burn-in | 16.9
Fixed Teacher [24] | 18.9
Standard[8] | 19.3
Ours | 23.0
Table 5: Influence of burn-in stage on performance on Cityscapes with 5% of
labeled data. Mask2Former, RN-50.

Data augmentations. One important component of teacher-student distillation is
the asymmetry between the two models that ensures that the teacher yields
useful pseudo-labels for the student [21]. To fulfill this condition, the
teacher and student receive differently augmented versions of the unlabeled
image. For the student model, the augmentations consist of random sized crops
and horizontal flipping. For the strong augmentations, we add textural-based
augmentations, including color jitter, random gray-scale and blur. We show in
Fig. 6 that this works better than the augmentations used in Polite Teacher
[8] which include random cutout, since small instances are prone to be masked
which results in inconsistencies in the pseudo-labels provided to the student.
Without augmentations, the student model quickly diverges while using cutout
underperforms compared to textural based augmentations only.

Figure 6: Effect of data augmentation on Cityscapes with $5\%$ labeled data.
Ours: Our default setup which includes color jitter, random grayscale and
random blur. PT: Uses C1 plus cutout as in [8]. Same: Teacher and student
receive weakly augmented images. Supervised: Model trained with labeled
samples only.

Weighting of unsupervised loss. We compare models trained with different
values for the weight $\lambda_{u}$ for the unlabeled loss term on Cityscapes
training with 10% of labels. The results in Tab. 6 show that even with a low
$\lambda_{u}$, the results are superior to the supervised baseline
($\lambda_{u}=0$), but the model benefits from putting more emphasis on the
semi-supervised loss with larger $\lambda_{u}$ values. However, if this
emphasis is too strong then the model tends to overemphasize the teacher’s
signal to the detriment of the supervised signal, we find it optimal to use
$\lambda_{u}=2$.

$\lambda_{u}$ | 0 | 0.1 | 0.5 | 2 | 5
---|---|---|---|---|---
AP | 18.8 | 23.6 | 24.4 | 30.8 | 25.5
Table 6: Influence on unlabeled loss weight on validation performance, on
Cityscapes dataset. Mask2Former, RN-50, 10% labels.

Backbone pre-training. Most segmentation models rely on pre-trained backbones
to extract features on which the segmentation head operates. We explore the
effect of different datasets to pre-train the feature backbone. To reduce
computational costs, we perform this exploration using a Swin-B architecture
rather than the heavier Swin-L architecture we used in our main experiments.
In Tab. 7 we report results obtained using ImageNet-1k and ImageNet-21k [5]
pre-training for supervised and unsupervised instance segmentation. We find
that ImageNet-21k pre-training consistently outperforms pre-training on the
smaller ImageNet-1k. The performance gap is higher in the low annotation
regime at $2\%$ labeled where the improvement is of 4 and 4.1 AP points in the
(un)supervised setups. This difference reduces to 1.4 points in the supervised
setting with $100\%$ of labels, indicating a higher importance of the backbone
pre-training when training using fewer segmented training images.

Pre-training | Training | $2\%$ | $100\%$
---|---|---|---
ImageNet-1k | Supervised | 26.3 | 46.8
Imagenet-21k | Supervised | 30.3 | 48.2
ImageNet-1k | Semi-supervised | 32.4 | —
Imagenet-21k | Semi-supervised | 36.5 | —
Table 7: Backbone pre-training effect on COCO using a Swin-B.

SSL backbones. We quantify the performance of self-supervised models trained
on large scale datasets for our task. More specifically, we adapt ViT [6]
models trained using DINOv2 [18] and compare the performance obtained with a
similar model trained using the Deit [22] method on ImageNet-1K. We used
similar ViT models, however the DINOv2 model has a patch token size of 14
while the standard ViT has a patch token size of 16. To adopt ViT models as
backbones for our model, we adapt the feature representation of these models
before the task-specific head. More precisely, we rearrange the patch tokens
to their relative spatial coordinates then use bi-linear interpolation
followed by a $1\\!\times\\!1$ convolution to match the number of features at
different scales of the segmentation head.

Results are reported in Tab. 3(b) and Tab. 3(d). We see that models trained
with DINOv2 backbones show high performance and are on par with Swin
backbones. Such models are also very robust at very low annotation regimes,
yielding SOTA results on COCO with 1–2% of labeled images and also on
Cityscapes on 5–10% labeled setting. However the model does not generalize as
well to larger amounts labeled samples, and the gap in performance between
Swin-L and DINOv2-L gets bigger the more labeled samples are available. This
could be explained by the relatively simple adaptation of the ViT backbone to
the segmentation head. It is interesting to explore transformer architectures
that are better suited for segmentation on large scale datasets in an SSL
setting, but we leave this for future work.

Backbone | Model | 2% | 100%
---|---|---|---
Finetuned | Deit ViT-B | 20.3 | —
DINOv2 ViT-B | 30.0 | 46.9
DINOv2 ViT-L | 33.1 | 46.7
Fixed | DINOv2 ViT-B | 27.7 | 41.2
DINOv2 ViT-L | 29.7 | 41.5
Table 8: Influence of backbone finetuning on training performance using DINOv2
backbone. COCO supervised training.

We also experiment with keeping the DINOv2 backbone fixed, and only training
the segmentation head to evaluate the feature generalisation ability of the
backbone. Finally, to validate whether the robustness at low annotation
regimes is a property of the backbone pre-training or its architecture we
report results of a Vit-B model trained on ImageNet-1k following Deit [22]. As
reported in Tab. 8, we see that the Vit-B Deit pre-trained model is severely
underperforming compared to DINOv2 pre-training, and performs on par with a
R50 backbone despite being much larger. This indicates that the performance
gains are more due to the large scale pre-training than the ViT’s
architecture. On the other hand, fixing the backbone and training the
segmentation head only results in lower AP results by around $3-4$ points at
$2\%$ and $5$ points at $100\%$ supervision. This demonstrates that, while not
ideal, training a segmentation head on top of frozen large scale pre-trained
models such as in DINOv2 can be beneficial when having few annotated samples,
especially when taking into account the memory gains induced by leaving the
backbone fixed.

##  5 Conclusions

In this paper we introduced Guided Distillation, a new distillation recipe for
semi-supervised instance segmentation. This novel approach, combined with
recent advances in segmentation methods and vision feature backbones,
outperforms previous state-of-the-art in instance segmentation by a large
margin on the Cityscapes and COCO benchmarks. We found that the optimal
pretrained feature backbone varied as a function of the amount of labeled
training data: while Swin-L (IN-21k) was optimal for most experiments, we
found the ViT-L (DINOv2) backbones to be more effective in the very low
annotation regime.

In future work, we would like to test our approach for other prediction tasks
such as object detection, to explore even weaker annotation regimes, and
assess the robustness of our approach to domain shifts.

Estimation of carbon footprint. On COCO, it took 25 hours to train our ViT-L
(DINOv2) model using 1% of annotations, and about 2 days to train a Swin-L
(IN-21k) model using 10% of annotations. Trainings are 2.5 times faster on
Cityscapes. Given the same formula used in [18], a Thermal Design Power (TDP)
of V100-32G equal to 250W, a Power Usage Effectiveness (PUE) of 1.1, a carbon
intensity factor of 0.385 kg CO2 per KWh, a time of 2 days $\times$ 24 hours
$\times$ 16 GPUs = 768 GPU hours to train our approach with a SWIN-L, it leads
to 211 kWh, an equivalent CO2 footprint of 211 $\times$ 0.385 = 81.2 kg.

Acknowledgements. Karteek Alahari is supported in part by the ANR grant AVENUE
(ANR-18-CE23-0011).

## References

  * [1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [2] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In CVPR, 2022.
  * [3] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  In NeurIPS, 2021.
  * [4] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The Cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [7] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  EVA: Exploring the limits of masked visual representation learning at scale.  In CVPR, 2023.
  * [8] Dominik Filipiak, Andrzej Zapala, Piotr Tempczyk, Anna Fensel, and Marek Cygan.  Polite teacher: Semi-supervised instance segmentation with mutual learning and pseudo-label thresholding.  arXiv preprint, 2211.03850, 2022.
  * [9] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask R-CNN.  In ICCV, 2017.
  * [10] T. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks.  In ICLR, 2017.
  * [11] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross B. Girshick.  PointRend: Image segmentation as rendering.  In CVPR, 2020.
  * [12] Youngwan Lee and Jongyoul Park.  CenterMask: Real-time anchor-free instance segmentation.  In CVPR, 2020.
  * [13] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler.  Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization.  In CVPR, 2021.
  * [14] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  In ECCV, 2022.
  * [15] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Zitnick.  Microsoft COCO: common objects in context.  In ECCV, 2014.
  * [16] Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le.  Pay attention to MLPs.  arXiv preprint, 2021.
  * [17] Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda.  Unbiased teacher for semi-supervised object detection.  In ICLR, 2021.
  * [18] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * [19] Ilija Radosavovic, Piotr Dollár, Ross B. Girshick, Georgia Gkioxari, and Kaiming He.  Data distillation: Towards omni-supervised learning.  In CVPR, 2018.
  * [20] J. Springenberg.  Unsupervised and semi-supervised learning with categorical generative adversarial networks.  In ICLR, 2016.
  * [21] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In NeurIPS, 2017.
  * [22] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with image transformers.  In ICCV, 2021.
  * [23] J. E. van Engelen and H. H. Hoos.  A survey on semi-supervised learning.  Machine Learning, 109:373–440, 2020.
  * [24] Zhenyu Wang, Yali Li, and Shengjin Wang.  Noisy boundaries: Lemon or lemonade for semi-supervised instance segmentation?  In CVPR, 2022.
  * [25] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.


initiated. Each column corresponds to a released model trained with a
different learning rate and momentum setting. Figure 24: More examples of
CIFAR-10 reconstructions in the default attack setting. Odd columns are
reconstructions and even columns are targets.