# Pseudo-label Correction and Learning For Semi-Supervised Object Detection

Yulin He1 Wei Chen1††footnotemark:  Ke Liang1
Yusong Tan1 Zhengfa Liang2 Yulan Guo1
1 National University of Defense Technology
2 Defense Innovation Institute, Chinese Academy of Military Science
{heyulin, chenwei, liangke, ystan, liangzhengfa10<EMAIL_ADDRESS>Equal contribution.Corresponding author.

###### Abstract

Pseudo-Labeling has emerged as a simple yet effective technique for semi-
supervised object detection (SSOD). However, the inevitable noise problem in
pseudo-labels significantly degrades the performance of SSOD methods. Recent
advances effectively alleviate the classification noise in SSOD, while the
localization noise which is a non-negligible part of SSOD is not well-
addressed. In this paper, we analyse the localization noise from the
generation and learning phases, and propose two strategies, namely pseudo-
label correction and noise-unaware learning. For pseudo-label correction, we
introduce a multi-round refining method and a multi-vote weighting method. The
former iteratively refines the pseudo boxes to improve the stability of
predictions, while the latter smoothly self-corrects pseudo boxes by weighing
the scores of surrounding jittered boxes. For noise-unaware learning, we
introduce a loss weight function that is negatively correlated with the
Intersection over Union (IoU) in the regression task, which pulls the
predicted boxes closer to the object and improves localization accuracy. Our
proposed method, Pseudo-label Correction and Learning (PCL), is extensively
evaluated on the MS COCO and PASCAL VOC benchmarks. On MS COCO, PCL
outperforms the supervised baseline by 12.16, 12.11, and 9.57 mAP and the
recent SOTA (SoftTeacher) by 3.90, 2.54, and 2.43 mAP under 1%, 5%, and 10%
labeling ratios, respectively. On PASCAL VOC, PCL improves the supervised
baseline by 5.64 mAP and the recent SOTA (Unbiased Teacherv2) by 1.04 mAP on
AP50.

Figure 1:  Localization noise in pseudo-labels comes from two phases:
generation and learning. In the generation phase, some candidate pseudo-labels
with inaccurate localization receive high scores (i.e., the box drawn in
blue), leading to the positional mismatch between pseudo-labels and ground-
truth. In the learning phase, the inaccurate pseudo-labels cause a serious
noisy problem in label assignment. Specifically, some candidate proposals are
incorrectly assigned with wrong training targets (i.e., the dashed boxes drawn
in red), which confuses the model in judging which candidate proposal should
give a high score. As these two phases are conducted alternately during
training, the localization noise will be accumulated, making training more
challenging.

##  1 Introduction

Benefiting from large-scale dataset and growing computational power, deep
neural networks have led to great success in computer vision. In the standard
supervised learning protocol, the volume and quality of labels are extremely
crucial to the performance. However, collecting human annotations is labor-
intensive and time-consuming, especially for instance-level annotations such
as object detection. Semi-supervised learning (SSL) is an effective remedy
that utilizes a small portion of labeled data and abundant unlabeled data to
achieve satisfactory performance. SSL has achieved significant progress in
classification tasks in recent years, with performance comparable to fully
supervised learning [21, 29]. However, semi-supervised object detection
(SSOD), which is more challenging than semi-supervised classification, has not
been adequately explored.

Recent state-of-the-art SSOD methods [22, 26, 27, 32] are based on pseudo-
labeling with a mean-teacher structure [24]. The student model is supervised
using the pseudo-labels generated by the teacher model, so the quality of
pseudo-labels directly impacts the performance of SSOD. Nevertheless, the
noise problem of pseudo-labels significantly degrades the performance of SSOD.
This noise consists of classification noise (i.e., misclassification) and
localization noise (i.e., localization inaccuracy). The former one is treated
as a confirmation bias issue in SSL, many off-the-shelf methods [26, 25, 4]
are proposed to significantly reduce the classification errors. However, the
latter one, as a non-negligible component of pseudo-label noise, has not been
well addressed. As shown in Fig. 1, localization noise impacts the training of
SSOD from generation and learning phases. Pseudo-labels with poor localization
quality will lead to incorrect label assignment, which easily makes the model
confused and severely degrades the detection performance. Therefore, (1) how
to guarantee the localization quality of pseudo-labels? and (2) how to learn
to localize objects from noisy pseudo-labels? are the concerns of this paper.

For the first concern, significant efforts [22, 16, 26] have been made to
design appropriate selection strategies to filter out low-quality pseudo-
labels, thereby ensuring the localization quality of pseudo-labels. However,
few have considered to directly revise the pseudo-labels in a self-correcting
manner. We found the inaccuracy of the pseudo-labels comes from two aspects,
i.e., the prediction instability and the unreliability of NMS (Non-Maximum
Suppression). To address the above two issues, a pseudo-label correction
method is proposed to self-correct pseudo-labels with two steps, i.e., multi-
round refining and multi-vote weighting. The primary objective of the multi-
round refining step is to improve the stability of pseudo-labels. To achieve
this, we repeatedly feed the pseudo boxes into the teacher’s RCNN layer for
refinement. As for multi-vote weighting step, we argue that the prediction
boxes are interrelated, in contrast to NMS which considers them independently.
Specifically, we introduce the box jitter function to the pseudo boxes with a
particular variance. Then, we correct the pseudo boxes by assigning weights to
the scores of adjacent jittered boxes. Pseudo-label correction is a heuristic
algorithm that uses the output score as an indicator to search for an
optimized box locally within the feature map space.

For the second concern, the research of it is still in the early stage. Some
existing approaches include Unbiased Teacher [16], which achieves better
performance without learning the pseudo-label localization; Dense Teacher
[31], which utilizes the teacher model’s predictions as soft-labels to guide
the localization learning of the student model; and SCMT[25], which introduces
a localization loss weight function proportional to the IoU, enhancing the
learning of confident pseudo boxes. Despite these efforts, there is no
conclusive answer on how to effectively learn the regression task from pseudo-
labels. In our framework, we compared the aforementioned methods and found
that designing soft weights that are negatively correlated with the IoU value
leads to the best results. These results suggest that pseudo boxes with high
confidence may not be as accurate as human annotations, they can also guide a
relatively accurate optimization direction.

The main contributions of this work are as follows:

  * •

We propose a pseudo-label correction method to improve the stability and
localization accuracy of pseudo-labels, thus providing a more accurate
supervised signal to the student model.

  * •

We provide an in-depth analysis of label and weight designs in the
unsupervised regression loss, and then propose a novel soft-weight function
that can effectively improve the localization accuracy.

  * •

We validate the performance and transferability of our method on three recent
SSOD algorithms and two popular datasets, and the results show consistent and
significant accuracy improvements.

Figure 2: Overview of our proposed method. Firstly, weakly augmented images
are fed into the teacher model to generate pseudo-labels. To ensure the
accuracy of the pseudo-labels, we apply the pseudo-label correction method to
self-correct them. Then, strongly augmented images are input to the student
model to generate proposals, which are further processed by the teacher’s RCNN
layer to obtain soft labels and soft weights, as indicated by the dotted
lines. Finally, we employ the noise-unaware learning strategy to compute
unsupervised loss.

##  2 Related Works

Semi-supervised learning for image classification: The majority of SSL methods
for image classification can be roughly categorized into two types:
consistency regularization and pseudo-labeling. Consistency regularization
methods [1, 20] rely on the assumption that no matter what perturbations are
applied, the model should keep the consistent output for the same unlabeled
data. Data augmentation [1], model perturbation [9], and adversarial training
[18] are the main strategies to introduce perturbations. Additionally, a mean
teacher [24] structure is proposed, which ensembles time-series models through
exponential mean average (EMA) to form a teacher model, and then guides the
student model to maintain consistency during training. Pseudo-labeling methods
[10, 21] employ weak data augmentation to annotate pseudo-labels for unlabeled
data. Due to their general and task-agnostic advantages, they have been widely
used in SSL. However, they tend to underperform due to the noise of pseudo-
labels. [3, 2, 30] explore the consistency between weak and strong data
augmentation through soft pseudo-labels, mitigating the harmful impacts of
noise.

Semi-supervised object detection: Inspired by SSL, pseudo-labeling and
consistency regularization are the mainstream methods of SSOD. In [7, 8], flip
consistency and interpolation consistency are used to perform instance-level
consistency for SSOD. STAC [22] firstly applies pseudo-labels performed on
weak and strong data augmentation into SSOD. Following STAC [22], [16, 32, 26]
further develope pseudo-labeling and achieve a significant performance
improvement. However, the noise of pseudo-labels has always been the crux of
pseudo-labeling methods. Generally, the noise in pseudo-labels consists of two
parts, including classification noise and localization noise.

Classification noise in semi-supervised object detection: Classification noise
has already attracted a lot of attention in the SSOD community. Humble Teacher
[23] exploits KL divergence to narrow classification output distributions of
the teacher and the student model. LabelMatch [4] utilizes soft-labels
generated by the teacher model to guide the learning of the student model.
SoftTeacher [26] adds soft-weights to negative pseudo-labels to alleviate
missing detection problem. SCMT [25] proposes to dynamically re-weight the
unsupervised loss of each proposal from both RPN and RCNN parts. After
extensive research, adopting soft labeling or soft weighting techniques in
SSOD can greatly alleviate the impact of classification noise.

Localization noise in semi-supervised object detection: While, the research of
the localization noise is still in the early stage. Many existing works
guarantee the quality of pseudo-labels by designing pseudo-label selection
strategies. For instance, STAC [22] and SoftTeacher [26] simply set a high
filter threshold. LabelMatch [4] proposes an adaptive thresholding strategy,
which considers the distribution of labeled data and sets different thresholds
for different categories. RPL [13] proposes to add an extra certainty-aware
task to estimate regression quality of pseudo-labels. Unbiased Teacherv2 [17]
proposes to predict uncertainties on the regression branch to select pseudo-
labels for boundary prediction. However, these methods only focus on selecting
pseudo-labels rather than correcting them during generation. In contrast, we
propose to self-correct the pseudo-labels by enhancing their stability and
relatedness. We argue that our proposed pseudo-label correction strategy is
orthogonal to the above works and can work collaboratively with them. Another
branch of research is on learning localization knowledge from noisy pseudo-
labels, with various strategies proposed, such as hard labels [22, 26], soft
labels [31], and soft weights [25, 12]. In our framework, we compared these
three types of techniques, and proposed a novel learning strategy that can
tightly pull the prediction boxes around the objects.

##  3 Methodology

The pipeline of our proposed method is illustrated in Fig. 2. In Sec. 3.1, we
present the preliminary and introduce the baseline of our method. Then, we
propose a pseudo-label correction strategy in Sec. 3.2, which aims to self-
correct pseudo-labels. However, noise still exists after correction and
negatively affects performance. To address this, we propose a noise-unaware
learning strategy in Sec. 3.3 to combat the noise in pseudo-label learning.

###  3.1 Preliminary

Problem Definition: In SSOD, a set of labeled data
$D_{s}=\\{x_{i}^{s},y_{i}^{s}\\}_{i=1}^{N_{s}}$ and a set of unlabeled data
$D_{u}=\\{x_{i}^{u}\\}_{i=1}^{N_{u}}$ are given. $x_{i}$ and $y_{i}$ are the
image and annotation of $i$-th sample, where $y_{i}$ contains the categories
and locations of the objects in $i$-th sample. $N_{s}$ and $N_{u}$ denote the
numbers of $D_{s}$ and $D_{u}$. The goal of SSOD is to leverage massive
unlabeled data to boost the performance of object detection.

SSOD with Mean Teacher Framework: Our SSOD baseline is built upon the mean
teacher framework with pseudo-label learning.

Mean teacher framework consists of a teacher model and a student model, which
are optimized through a mutual learning mechanism. The weights of the teacher
model are updated by the EMA (Exponential Moving Average) weights of the
student model, while the student model exploits gradient backpropagation to
update its weights. The teacher model predicts the pseudo-labels of unlabeled
data, and the student model is trained by both labeled data and unlabeled
data. Therefore, the loss of SSOD can be summarized as a supervised loss
$\mathcal{L}_{s}$ on labeled data and an unsupervised loss $\mathcal{L}_{u}$
on unlabeled data.

$\ \begin{aligned}
\mathcal{L}=\frac{1}{N_{s}}\sum_{i=1}^{N_{s}}\mathcal{L}_{s}(x^{s},y^{s})+\frac{1}{N_{u}}\sum_{i=1}^{N_{u}}\alpha\mathcal{L}_{u}(x^{u},y^{u}),\end{aligned}$
(1)

where $N_{s}$ and $N_{u}$ are the number of labeled data and unlabeled data,
respectively. $y^{u}$ is the pseudo-labels consisting of classification and
regression parts. $\alpha$ controls the contribution of $\mathcal{L}_{u}$.

Figure 3: We randomly sample 10k unlabeled images to conduct experiments based
on the model trained at the 10% labeled data setting. (a) denotes the changes
of multi-round refining, consisting of IoU and score. (b) and (c) show the IoU
and mAP between pseudo-labels (confidence $\geq$ 0.7 ) and the ground-truth.
(d) describes the relations between the IoU with the ground-truth and the
foreground scores. Each scatter represents one pseudo-label. The red circle
indicates the dense place. Algorithm 1 Pseudo-label correction

1:RCNN layer $\mathcal{M}$; Feature map $F$; Initial pseudo boxes ${B^{0}}$;
Number of the refined rounds $n_{r}$; Number and variance of jittered boxes
$n_{j},\sigma$

2:Corrected boxes ${B_{c}}$ and scores ${L_{c}}$

3:for $r=1$ to $n_{r}$ do (multi-round refining)

4: // Refine boxes and scores

5: $B^{r},S^{r}=\mathcal{M}(F,B^{r-1})$

6:end for

7:$B_{c}$ = {}

8:for $b$ in $B^{n_{r}}$ do (multi-vote weighting)

9: // Generate jittered boxes based on $b$

10: $B_{j}=\mathcal{F}_{j}(b,n_{j},\sigma)$

11: // Calculate prediction boxes and scores of $B_{j}$

12: $\hat{B_{j}},S_{j}=\mathcal{M}(F,B_{j})$

13: // Correct pseudo boxes by weighted summation

14:
$b_{c}=\sum_{i=1}^{n_{j}}\hat{B_{j}^{i}}\times\frac{S_{j}^{i}}{\sum_{i=1}^{n_{j}}S_{j}^{i}}$;

15: $B_{c}$ $\leftarrow$ Append($b_{c}$)

16:end for

17:$S_{c}=S^{n_{r}}$;

18:return $B_{c}$, $S_{c}$

###  3.2 Pseudo-label correction

Pseudo-label correction can be regarded as a heuristic algorithm that improves
the localization accuracy of pseudo boxes using the model’s output as a guide.
It involves two main steps: multi-round refining and multi-vote weighting, as
detailed in Algo. 1.

Multi-round refining: Before we present the detail of multi-round refining, we
first introduce two metircs to meansure the differences between multiple
rounds in both classification and localization:

$\displaystyle\mathcal{D}_{cls}^{r}$
$\displaystyle=\frac{\sum_{i=1}^{n_{pse}}|S_{i}^{r}-S_{i}^{r-1}|}{n_{pro}},$
(2) $\displaystyle\mathcal{D}_{loc}^{r}$
$\displaystyle=1-\frac{\sum_{i=1}^{n_{pse}}IoU(B_{i}^{r},B_{i}^{r-1})}{n_{pro}},$
(3)

where $n_{pse}$ is the number of pseudo-labels, $B^{r}$ and $S^{r}$ are the
bounding boxes and scores of $r$-th round refinement. $S^{0}$ and $B^{0}$ are
the scores and boxes of initial pseudo-labels. $\mathcal{D}_{cls}$ and
$\mathcal{D}_{loc}$ are the classification and localization differences
between two rounds, respectively. The smaller the values of
$\mathcal{D}_{cls}$ and $\mathcal{D}_{loc}$, the more stable the prediction.

Specifically, we alternately input prediction boxes and the basic feature map
into the RCNN layer of the teacher model to obtain continuously refined boxes.
As the boxes are refined, the prediction scores change simultaneously. As
shown in Fig 3 (a), after multi-round refining, the output of the teacher
model can be more stable. Furthermore, We observed that this process reaches
saturation after several rounds of refinement, indicating that the model is
confident and requires little correction. The intuitive assumption behind
multi-round refining is that the smaller the output variance between two
rounds, the more confident the model is towards the output, and thus the
pseudo-labels are more accurate.

Multi-vote weighting: NMS is a standard post-process operation used in object
detection. Its “winner-takes-all” strategy speeds up the post-processing and
effectively suppresses overlapping false-positive bounding boxes. However, due
to inaccurate pseudo boxes, some inaccurately localized boxes are assigned
high scores, which makes it unreliable to rely on the score of individual
boxes to determine the output box. Therefore, we propose to exploit the scores
of the surrounding boxes together to decide the position of the output box.

Specifically, we jitter the refined boxes multiple times with a certain
variance, where one refined box corresponds to multiple jittered boxes. The
box jitter function $\mathcal{F}_{j}$ is executed in the following steps:
(1) Sample jitter factors $\xi$ by a standard normal distribution with the
variance of $\sigma_{j}$.

$\displaystyle\xi\sim Sample(N(0,\sigma_{j}),n_{j}),\
\xi\in\mathbb{R}^{n_{j}\times 4}.$ (4)

(2) Get jittered boxes by adding jittered offsets.

$\displaystyle B_{j}=(1+\xi)\times Expand(b,n_{j}),$ (5)

where $Expand$ denotes the function that expands $b$ from the size of
$[{1\times 4}]$ to $[{n_{j}\times 4}]$. Then, The jittered boxes are fed into
the RCNN layer again to obtain their prediction boxes and scores. Finally, we
normalize the prediction scores and use them as weights to sum the prediction
boxes, getting the final self-corrected boxes. This method transforms the
localization of pseudo boxes from being solely determined by the box with the
maximum score to being voted by the surrounding boxes. As shown in Fig 3 (b),
after multi-vote weighting, the localization of output boxes can be more
accurate. The intuitive assumption behind it is that inaccurate boxes will be
corrected by surrounding boxes that are close to objects, as they are likely
to have high weights.

After applying the pseudo-label correction, the quality of pseudo-labels is
improved, which can be demonstrated by the empirical study in Fig. 3 (c).

Table 1:  The $mAP$ performance comparisons of the state-of-the-art SSOD
methods on val2017 data set under COCO-standard.

Method | 1 % labeled data | 5 % labeled data | 10 % labeled data
---|---|---|---
Supervised | 12.20 $\pm$ 0.29 | 21.17 $\pm$ 0.17 | 26.90 $\pm$ 0.08
STAC [22] | 13.97 $\pm$ 0.35 | 24.38 $\pm$ 0.12 | 28.64 $\pm$ 0.21
Humble Teacher [23] | 16.95 $\pm$ 0.35 | 27.70 $\pm$ 0.15 | 31.61 $\pm$ 0.28
ISMT [27] | 18.88 $\pm$ 0.74 | 26.37 $\pm$ 0.24 | 30.53 $\pm$ 0.52
Unbiased Teacher [16] | 20.75 $\pm$ 0.12 | 28.27 $\pm$ 0.11 | 31.50 $\pm$ 0.10
SoftTeacher [26] | 20.46 $\pm$ 0.39 | 30.74 $\pm$ 0.08 | 34.04 $\pm$ 0.14
SCMT [25] | 23.09 $\pm$ 0.16 | 32.14 $\pm$ 0.06 | 35.42 $\pm$ 0.12
RPL [13] | 19.02 $\pm$ 0.25 | 28.40 $\pm$ 0.15 | 32.23 $\pm$ 0.14
ACRST [28] | 26.07 $\pm$ 0.46 | 31.35 $\pm$ 0.13 | 34.92 $\pm$ 0.22
Unbiased Teacher v2 [17] | 25.40 $\pm$ 0.36 | 31.85 $\pm$ 0.09 | 35.08 $\pm$ 0.02
LabelMatch [4] | 25.81 $\pm$ 0.28 | 32.70 $\pm$ 0.18 | 35.49 $\pm$ 0.17
MA-GCP [11] | 21.30 $\pm$ 0.28 | 31.67 $\pm$ 0.16 | 35.02 $\pm$ 0.26
PCL (Ours) | 24.36 $\pm$ 0.12 | 33.28 $\pm$ 0.09 | 36.47 $\pm$ 0.15

Table 2: The $mAP$ performance comparisons of the state-of-the-art SSOD
methods on val2017 data set under COCO-additional.

Methods | $mAP$
---|---
Supervised | 37.63
CSD [7] | 38.82
STAC [22] | 39.21
Instant-Teaching [32] | 40.20
Unbiased Teacher v1 [16] | 41.30
SoftTeacher [26] | 41.40
SCMT [25] | 42.56
PCL (Ours) | 44.33

Table 3: The $AP^{50}$ performance comparisons of the state-of-the-art SSOD
methods under VOC.

Methods | Labeled | Unlabeled | $AP^{50}$
---|---|---|---
Supervised | VOC07 | VOC12 | 76.70
STAC [22] | 77.45
Unbiased Teacher v1 [16] | 77.37
ISMT [27] | 77.23
Instant-Teaching [32] | 79.20
Humble Teacher [23] | 80.94
RPL [13] | 79.00
ACRST [28] | 78.16
Unbiased Teacher v2 [17] | 81.29
PCL (Ours) | 82.33

###  3.3 Noise-unaware learning

After correcting the pseudo-labels, their quality has been improved, but noise
may still be present. Therefore, we propose a noise-unaware strategy to
extract crucial knowledge from the noisy pseudo-labels. Specifically, we first
align the proposals of the student model and the teacher model by an affine
transform function, as in [26, 4, 25]. Then, the proposals of the student are
fed into the teacher model to obtain soft outputs, including boxes and scores.
After that, we can calculate the unsupervised loss between the prediction of
the student model and the output of the teacher model.

For the unsupervised localization task, we use the corrected boxes as the
labels and design a novel re-weighting function. In general, the smaller the
output difference between the student model and the teacher model (i.e., the
larger the IoU), the more credible the sample is, so the loss weights should
be increased. However, we get an opposite conclusion: designing the loss
weight that is negatively correlated with IoU values works better. As shown in
Fig. 3(d), we conduct a empirical study to analysis the relations between
scores and IoU. It shows that most of the pseudo-labels with high confidence
have high IoU values too. Therefore, we argue that the pseudo-labels, although
not as accurate as ground-truth, can also indicate the right direction to
optimize the localization task. When the gap between the predictions and
pseudo-labels is large, they are also likely to be far away from the ground-
truth. Therefore, we design the unsupervised regression loss as follows:

$\displaystyle\mathcal{L}_{u}^{reg}$
$\displaystyle=\sum_{i=1}^{n_{pos}}w_{i}^{reg}L_{1}(B_{i}^{t},B_{i}^{s}),$ (6)
$\displaystyle w_{i}^{reg}$
$\displaystyle=sigmoid(\frac{1}{V_{IoU}^{\lambda}})$

where $n_{pos}$ is the number of positive proposals. $L_{1}$ is the MAE loss
used in FasterRCNN [19]. $B_{i}^{s}$ is the prediction box of the $i$-th
proposal from the student model, and $B_{i}^{t}$ is the corresponding
regression target, i.e., the corrected box. $V_{IoU}$ is the IoU value between
the output boxes of the student and teacher models. $\lambda$ is a hyper-
parameter to control the relative gaps of different IoUs.

For the unsupervised classification task, we adopt the soft-label technique
proposed in LabelMatch [4]. To calculate the unsupervised classification loss,
we utilize a soft cross-entropy loss, which is formulated as follows:

$\displaystyle\mathcal{L}_{u}^{cls}$
$\displaystyle=\sum_{i=1}^{n_{pro}}w_{i}^{cls}\sum_{c=1}^{C}-S_{i,c}^{t}\log
S_{i,c}^{s},$ (7) $\displaystyle w_{i}^{cls}$ $\displaystyle=max(S_{i}^{t})$

where $n_{pro}$ is the number of proposals, and $C$ is the categories of the
dataset. $S_{i,c}^{s}$ and $S_{i,c}^{t}$ are the output scores of the $c$-th
class and $i$-th proposal from the student and the teacher model,
respectively.

Table 4: Ablation study on different components of our method. “PC” and “NL”
indicate pseudo-label correction and noise-unaware learning. “MR” and “MW”
denote multi-round refining and multi-vote weighting. $L_{u}^{cls}$ and
$L_{u}^{reg}$ are the loss design of our method.

PC | NL | $mAP$
---|---|---
MR | MW | $L_{u}^{cls}$ | $L_{u}^{reg}$
|  |  |  | 33.8
|  | ✓ |  | 34.1
|  | ✓ | ✓ | 34.5
✓ |  | ✓ | ✓ | 34.7
✓ | ✓ | ✓ | ✓ | 35.1

Table 5: Different label and weight settings on $L_{u}^{reg}$. IoU indicates
that using IoU value as the loss weight of $L_{u}^{reg}$. Inv IoU means that
using the loss weight in Eq. 6.

Loss label | Loss weight | $mAP$
---|---|---
Soft | Hard | IoU | Inv IoU
✓ |  |  |  | 33.2
| ✓ |  |  | 34.1
| ✓ | ✓ |  | 34.2
| ✓ |  | ✓ | 34.5

##  4 Experiments

###  4.1 Dataset and Evaluation

Our experiments are conducted on MS COCO [15] and PASCAL VOC [5] benchmarks.
MS COCO dataset is split into train2017, unlabeled2017, and val2017. PASCAL
VOC dataset is split into VOC2007-trainval, VOC2012-trainval, and
VOC2007-test. We follow the recent SSOD methods [16, 4, 17] and use three
experimental settings: (1) COCO-strandard: we randomly sample 1%, 5% and 10%
ratios of the train2017 as labeled data, and treat the remainder as unlabeled
data. (2) COCO-additional: we use all the train2017 as labeled data and the
unlabeled2017 as unlabeled data. The val2017 is used to evaluate the
performance on MS COCO benchmark with the metric $mAP$. (3) VOC: we use the
VOC2007-trainval as labeled data and the VOC2012-trainval as unlabeled data.
The VOC2007-test is used to evaluate the performance on PASCAL VOC benchmark
with the metric $AP^{50}$.

###  4.2 Implementation Details

We employ FasterRCNN [19] with ResNet-50 backbone [6] and FPN [14] as the
default detector in our experiments. However, our method can easily be adapted
to other detectors as well. The pseudo-label selection threshold is set to
0.7, which is the same as SCMT [25]. The SSOD model is trained on 4 GPUs with
10 images per GPU (2 labeled and 8 unlabeled images) under COCO-standard and
VOC, and 8 images per GPU (4 labeled and 4 unlabeled images) under COCO-
additional. The initial learning rate is set to 0.01. For COCO-standard, the
training iteration is 180k, and the learning rate is divided by 10 at 120k and
160k iterations. For COCO-additional, the training iteration is 360k, and the
learning rate is divided by 10 at 240k and 320k iterations. For VOC, the
training iteration is 60k, and the learning rate is divided by 10 at 40k
iterations. In ablation studies, we train the model with 90k iterations at 10%
labeled ratio to save computational cost. The data augmentation of this work
is followed as SoftTeacher [26]. The $n_{b}$ and $\sigma$ in box jitter
function are set 10 and 0.06. The number of rounds of multi-round refining
($n_{r}$) is set to 2. More training and implementation details are presented
in the Appendix.

###  4.3 Comparison with State-of-the-Art Methods

The baseline of our method is SoftTeacher [26], where we add the pseudo-label
correction and learning strategies and achieve significant performance
improvements. Under COCO-standard, as shown in Table 1, our method outperforms
SoftTeacher by 3.90, 2.54, and 2.43 mAP, and boosts the accuracy of supervised
learning by 12.16, 12.11, and 9.57 mAP. The most similar method compared with
ours is SCMT [25], which filters pseudo-labels with a low threshold and
utilizes a soft-weight technique to the loss calculation. Our proposed PCL
shows consistent accuracy improvements at all settings of the MS COCO, with AP
values exceeding 1 point. When compared with other recent SOTA methods,
although our method is not the best at the 1% setting, it outperforms the
other methods significantly at 5% and 10% settings. Under COCO-additional, as
shown in Table 2, PCL improves the supervised baseline from 37.63 mAP to 44.33
mAP, and also outperforms SoftTeacher [26] and SCMT [25] by 2.93 mAP and 1.77
mAP, respectively. Under VOC, as illustrated in Table 3, PCL achieves 82.33
mAP on $AP^{50}$, pushing the recent state-of-the-arts by a further 1.04 mAP.

###  4.4 Ablation study

Effect of different components: We conduct ablation experiments at the 10%
labeled data setting to validate the effectiveness of different components in
PCL. As shown in Table 4, after applying the noise-unaware learning strategy,
i.e., $L_{u}^{cls}$ in Eq. 7 and $L_{u}^{reg}$ in Eq. 6, mAP is increased from
33.8 to 34.5 mAP. Adding the pseudo-label correction strategy, i.e., MR
(multi-round refining) and MW (multi-vote weighting), further increases mAP
from 34.5 to 35.1. Furthermore, We conduct a visualisation analysis on pseudo-
label correction, as shown in Fig. 4. It clearly shows that the position of
some inaccurate boxes (drawn in red) has been corrected (drawn in green). This
phenomenon is also consistent with the results of our statistical analysis in
Fig. 3(b)(c), proving the effectiveness of our method.

Effect of unsupervised regression loss: We research the effect of different
label and weight settings in unsupervised regression loss. As shown in Table
5, using the hard label and soft weight that is negatively correlated to IoU
works the best. This result suggests that it is important to enhance the
learning of prediction boxes that are far from the pseudo boxes. In addition,
using the soft label is not ideal, with an AP gap of 0.9 compared to the hard
label. We guess that not all soft labels generated by the teacher model are
accurate, learning them all will make the model confused in training.
Moreover, we conduct a visualisation analyse on the prediction boxes with and
without NMS as shown in Fig. 5. It is clear that our PCL has denser prediction
boxes than SoftTeacher [26]. The design of the loss weight in Eq. 6 pulls the
prediction boxes tightly around the pseudo boxes, avoiding the false-positive
detections shown in the bottom case of Fig. 5.

Figure 4: Visualisation of pseudo-label correction. Red bounding boxes are
original pseudo boxes. Green bounding boxes are corrected pseudo boxes. Figure
5: Visualisation of the prediction boxes with and without NMS. The green boxes
and the grey boxes are prediction boxes after and before NMS, respectively.

###  4.5 Transferability Validation

In this section, we validate the transferability of our method to other SSOD
methods. Specifically, we have re-implemented the Unbiased Teacher [16] within
our framework and applied the same data augmentation as in [26]. By
incorporating the pseudo-label correction (PLC) discussed in Sec 3.2 and the
unsupervised regression loss (URL) defined in Eq. 6, we have observed a
notable improvement of 1.1 mAP in accuracy. As for SoftTeacher [26], we
adopted a low threshold towards pseudo-label selection, similar to SCMT [25],
i.e., 0.7. This modification can significantly alleviate the missing detection
problem in SoftTeacher, improving the accuracy from 31.9 mAP to 33.8 mAP. When
we incorporated PLC and URL into SoftTeacher, we observed an additional
increase in accuracy of 1.0 mAP. In addition, when we applied PLC into PseCo
[12], a more recent SSOD method, there is also a 1.1 mAP improvement. The
consistent improvements in performance demonstrate the general applicability
of our method. Importantly, since PCL is orthogonal to most existing SSOD
methods, it can be easily incorporated to remarkably improve accuracy.

###  4.6 Hyper-parameter Analysis

We have conducted experiments to research the effect of hyper-parameters in
our method. Table 7 shows the effect of different $\lambda$ values in
$L_{u}^{reg}$. When $\lambda=5$, we get the optimal result. Nevertheless, the
other selections of $\lambda$ can also get close accuracy, which shows
$\lambda$ is not parameter-sensitive. Table 8 studies the effect of different
variance values in the box jitter function. When $\sigma=0.06$, The accuracy
is the best. However, when we increase the $\sigma$ to 0.3, the accuracy drops
obviously. This is mainly due to the large box jitter variance causes the
prediction boxes to be severely offset from the objects, thus preventing the
pseudo boxes from being corrected. We set the number of multi-round refining
$n_{r}=2$ by analysing the differences in the output of multiple rounds. As
shown in Fig. 3 (a), it can be clearly observed that after two rounds of
refinement, the differences tend to be small.

Table 6: Transform effect of our method, i.e., pseudo-label correction (PLC)
and unsupervised regression loss (URL).

Method | Strategy | $mAP$
---|---|---
Unbiased Teacher [16] | Baseline | 33.1
PCL & URL | 34.2 (+1.1)
SoftTeacher [26] | Baseline | 33.8
PCL & URL | 34.8 (+1.0)
PseCo [12] | Baseline | 34.2
PCL | 35.3 (+1.1)

Table 7: The effect of $\lambda$ in $L_{u}^{reg}$

$\lambda$ | 1 | 3 | 5 | 7
---|---|---|---|---
AP | 34.4 | 34.3 | 34.5 | 34.3

Table 8: The effect of different variances in Eq.4.

$\sigma$ | 0.03 | 0.06 | 0.1 | 0.15 | 0.30
---|---|---|---|---|---
AP | 34.8 | 35.1 | 34.9 | 34.7 | 33.5

##  5 Conclusion

In this paper, we present a novel approach to address the issue of
localization noise in semi-supervised object detection. Our proposed pseudo-
label correction method is a plug-and-play solution that significantly
improves the stability and localization accuracy of pseudo-labels, leading to
a better overall performance. Moreover, we investigate the challenging problem
of learning the localization knowledge from noisy pseudo-labels, and introduce
a novel regression loss weight design that facilitates precise object
localization. Our approach is extensively evaluated on popular benchmarks
including MS COCO and PASCAL VOC, and the results demonstrate its remarkable
effectiveness.

## References

  * [1] Philip Bachman, Ouais Alsharif, and Doina Precup.  Learning with pseudo-ensembles.  NeurIPS, 27:3365–3373, 2014.
  * [2] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.  In ICLR, 2019.
  * [3] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  NeurIPS, 32, 2019.
  * [4] Binbin Chen, Weijie Chen, Shicai Yang, Yunyi Xuan, Jie Song, Di Xie, Shiliang Pu, Mingli Song, and Yueting Zhuang.  Label matching semi-supervised object detection.  In CVPR, pages 14381–14390, 2022.
  * [5] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [7] Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak.  Consistency-based semi-supervised learning for object detection.  NeurIPS, 32, 2019.
  * [8] Jisoo Jeong, Vikas Verma, Minsung Hyun, Juho Kannala, and Nojun Kwak.  Interpolation-based semi-supervised learning for object detection.  In CVPR, pages 11602–11611, 2021.
  * [9] Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  arXiv preprint arXiv:1610.02242, 2016.
  * [10] Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  ICML, page 896, 2013.
  * [11] Aoxue Li, Peng Yuan, and Zhenguo Li.  Semi-supervised object detection via multi-instance alignment with global class prototypes.  In CVPR, pages 9809–9818, 2022.
  * [12] Gang Li, Xiang Li, Yujie Wang, Shanshan Zhang, Yichao Wu, and Ding Liang.  Pseco: Pseudo labeling and consistency training for semi-supervised object detection.  In ECCV, 2022.
  * [13] Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, and Larry S Davis.  Rethinking pseudo labels for semi-supervised object detection.  In AAAI, pages 1314–1322, 2022.
  * [14] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, pages 740–755. Springer, 2014.
  * [16] Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda.  Unbiased teacher for semi-supervised object detection.  In ICLR, 2021.
  * [17] Yen-Cheng Liu, Chih-Yao Ma, and Zsolt Kira.  Unbiased teacher v2: Semi-supervised object detection for anchor-free and anchor-based detectors.  In CVPR, pages 9819–9828, 2022.
  * [18] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  IEEE TPAMI, 41(8):1979–1993, 2018.
  * [19] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  NeurIPS, 28, 2015.
  * [20] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  NeurIPS, 29:1163–1171, 2016.
  * [21] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  NeurIPS, 33:596–608, 2020.
  * [22] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister.  A simple semi-supervised learning framework for object detection.  arXiv preprint arXiv:2005.04757, 2020.
  * [23] Yihe Tang, Weifeng Chen, Yijun Luo, and Yuting Zhang.  Humble teachers teach better students for semi-supervised object detection.  In CVPR, pages 3132–3141, 2021.
  * [24] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  NeurIPS, 30, 2017.
  * [25] Feng Xiong, Jiayi Tian, Zhihui Hao, Yulin He, and Xiaofeng Ren.  Scmt: Self-correction mean teacher for semi-supervised object detection.  In IJCAI, pages 1488–1494, 2022.
  * [26] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu.  End-to-end semi-supervised object detection with soft teacher.  In CVPR, pages 3060–3069, 2021.
  * [27] Qize Yang, Xihan Wei, Biao Wang, Xian-Sheng Hua, and Lei Zhang.  Interactive self-training with mean teachers for semi-supervised object detection.  In CVPR, pages 5941–5950, 2021.
  * [28] Fangyuan Zhang, Tianxiang Pan, and Bin Wang.  Semi-supervised object detection with adaptive class-rebalancing self-training.  In AAAI, pages 3252–3261, 2022.
  * [29] Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu.  Simmatch: Semi-supervised learning with similarity matching.  In CVPR, pages 14471–14481, 2022.
  * [30] Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu.  Simmatch: Semi-supervised learning with similarity matching.  CVPR, pages 14471–14481, 2022.
  * [31] Hongyu Zhou, Zheng Ge, Songtao Liu, Weixin Mao, Zeming Li, Haiyan Yu, and Jian Sun.  Dense teacher: Dense pseudo-labels for semi-supervised object detection.  In ECCV, pages 35–50. Springer, 2022.
  * [32] Qiang Zhou, Chaohui Yu, Zhibin Wang, Qi Qian, and Hao Li.  Instant-teaching: An end-to-end semi-supervised object detection framework.  In CVPR, pages 4081–4090, 2021.