# 3DIoUMatch: Leveraging IoU Prediction
for Semi-Supervised 3D Object Detection

He Wang1* Yezhen Cong2* Or Litany3 Yue Gao2 Leonidas J. Guibas1
1Stanford University 2Tsinghua University 3NVIDIA

###### Abstract

3D object detection is an important yet demanding task that heavily relies on
difficult to obtain 3D annotations. To reduce the required amount of
supervision, we propose 3DIoUMatch, a novel semi-supervised method for 3D
object detection applicable to both indoor and outdoor scenes. We leverage a
teacher-student mutual learning framework to propagate information from the
labeled to the unlabeled train set in the form of pseudo-labels. However, due
to the high task complexity, we observe that the pseudo-labels suffer from
significant noise and are thus not directly usable. To that end, we introduce
a confidence-based filtering mechanism, inspired by FixMatch. We set
confidence thresholds based upon the predicted objectness and class
probability to filter low-quality pseudo-labels. While effective, we observe
that these two measures do not sufficiently capture localization quality. We
therefore propose to use the estimated 3D IoU as a localization metric and set
category-aware self-adjusted thresholds to filter poorly localized proposals.
We adopt VoteNet as our backbone detector on indoor datasets while we use PV-
RCNN on the autonomous driving dataset, KITTI. Our method consistently
improves state-of-the-art methods on both ScanNet and SUN-RGBD benchmarks by
significant margins under all label ratios (including fully labeled setting).
For example, when training using only 10% labeled data on ScanNet, 3DIoUMatch
achieves 7.7% absolute improvement on<EMAIL_ADDRESS>and 8.5% absolute improvement
on<EMAIL_ADDRESS>upon the prior art. On KITTI, we are the first to demonstrate semi-
supervised 3D object detection and our method surpasses a fully supervised
baseline from 1.8% to 7.6% under different label ratios and categories.

##  1 Introduction

††*: equal contribution††Project page: http://THU17cyz.github.io/3DIoUMatch

Object detection is a key task in 3D scene understanding. It provides a
concise representation of raw sensor measurements in the form of semantically
meaningful 3D bounding boxes. This low-dimensional representation can already
serve numerous applications in autonomous driving and AR/VR, as well as in
robot navigation and manipulation. As a result, in recent years there has been
a surge of interest in developing improved object detection pipelines and
indeed current state-of-the-art methods show impressive performance. Yet, much
of their success is attributed to the availability of large datasets of 3D
scenes that are carefully annotated. While rapid advances in sensor technology
facilitate the collection of 3D scenes at scale, annotating them remains the
main bottleneck. This calls for detection methods that can leverage both
labeled and unlabeled data at train time.

In this work, we aim to address this requirement by proposing a novel semi-
supervised 3D object detection method which we dub 3DIoUMatch. As a generally
applicable method, 3DIoUMatch can be applied to both indoor scene datasets,
i.e. ScanNet[4] and SUN-RGBD[29], and outdoor datasets, i.e. KITTI[7]. We
adopt popular point-based object detectors, VoteNet [19] and PV-RCNN [25], as
our backbone object detection networks for the indoor and outdoor scenes,
correspondingly. To provide supervision to the unlabeled scenes, we leverage a
teacher-student mutual learning framework [30] and use the bounding box
predictions from the teacher network as pseudo-labels to supervise the student
network on unlabeled data. However, unlike most pseudo-label techniques that
were designed for classification, in the highly complex (joint regression and
classification) task of object detection, we observe that the pseudo-labels
suffer from significant noise, and using them directly is suboptimal.

Inspired by FixMatch [27], the state-of-the-art semi-supervised learning (SSL)
method for 2D image classification that proposed confidence-based filtering to
improve pseudo-label quality, we adopt a pseudo-label filtering mechanism for
3D object detection by setting thresholds on predicted class probabilities
(and objectness scores for VoteNet), so as to filter out teacher proposals
with potentially erroneous semantic labels or ones not belong to foreground.
While effective, these criteria alone are not sufficient to capture
localization quality, and the pseudo-labels may still have large errors in the
bounding box parameters. To that end, we further propose to leverage estimated
IoU (intersection over union) as a localization quality measure for pseudo-
label filtering. IoU estimation was first proposed in the context of 2D object
detection as a localization confidence in the pioneering work IoU-Net [13],
where estimated IoU was proven successful in replacement of class confidence
for test-time Non-Maximal Suppression (NMS). To the best of our knowledge,
leveraging IoU estimation for pseudo-label filtering is a novel idea for SSL
on both 2D and 3D object detection. Equipping the detectors with a 3D IoU
estimation module, we are able to filter out poorly localized pseudo-labels
and leverage estimated IoU for both train-time and test-time NMS.

A key challenge when filtering based on IoU estimation is how to properly set
the threshold. Unlike objectness and class confidence for which high threshold
values (e.g. 0.9) work well, 3D IoU is more sensitive to small errors. Setting
the threshold too high would reduce the number of pseudo-labels to very few,
from which little could be learned. To balance between quality and coverage,
we propose a two-stage filtering process: first, using a relatively low IoU
threshold; then, an IoU-guided class-aware Lower-Half Suppression (LHS) that
removes only half of the highly-overlapping boxes with low predicted IoU. Our
proposed LHS thus naturally sets a threshold that is both dynamic and class-
aware. Our experiments show that LHS outperforms IoU-guided NMS, which
suppresses all but the top one during semi-supervised training.

Our method consistently improves upon the previous state-of-the-art method,
SESS [35], on both ScanNet and SUN-RGBD benchmarks by significant margins.
When using only 10% labeled data on ScanNet, 3DIoUMatch outperforms SESS by
7.7 absolute improvement on<EMAIL_ADDRESS>and by 8.5 absolute improvement on
<EMAIL_ADDRESS>When using 5% labeled data on SUN-RGBD, 3DIoUMatch outperforms SESS
by 4.8 absolute improvement on<EMAIL_ADDRESS>and by 8.0 absolute improvement on
<EMAIL_ADDRESS>On KITTI, we are the first to demonstrate semi-supervised 3D object
detection work and surpass fully-supervised baseline by large margins under
all label ratios.

Our main contributions can be summarized as follows:

  1. 1.

We propose a novel semi-supervised method for 3D object detection in point
clouds based on pseudo-label propagation along with a carefully designed
filtering mechanism.

  2. 2.

For the first time, we leverage predicted 3D IoU as a localization confidence
score for pseudo-label filtering, and further propose IoU-guided Lower-Half
Suppression for robust pseudo-label deduplication. This idea is generally
applicable and can be coupled to different 3D detectors on both indoor and
outdoor scenes.

  3. 3.

We achieve markedly improved performance over the previous state-of-the-art
semi-supervised 3D object detection methods on the two major indoor object
detection benchmarks, ScanNet and SUN-RGBD, under low label ratios and fully
labeled setting. As the first semi-supervised 3D object detection work on
KITTI, we also achieve significant improvements compared to fully supervised
method.

##  2 Related Works

Figure 1: 3DIoUMatch pipeline at semi-supervised training stage. We adopt as
our backbone an extended version of VoteNet with an additional 3D IoU
estimation module. For SSL, we utilize a teacher-student mutual learning
framework, composed of a learnable student taking strongly augmented input
data and an EMA teacher taking weakly augmented input samples. On labeled
data, the student network is supervisedly trained. On unlabeled data, the
student network takes pseudo-labels from its EMA teacher. To improve the
quality of pseudo-label, we adopt a confidence-based filtering mechanism that
filters out predictions that fail to pass all thresholds on class probability,
objectness, and 3D IoU. We further use IoU-guided Lower-Half Suppression to
remove the duplicated predictions. Using the filtered pseudo-labels, we
selectively supervise the student predictions that are around the bounding
boxes in the pseudo-labels.

##### Semi-Supervised Learning (SSL)

Many of the recent SSL methods [2, 32, 1] leverage consistency regularization,
first proposed in [24, 14], which enforces the model to predict consistently
across label-preserving data augmentation of different intensity. Borrowing
the concept from Mean Teacher [30], the model with frozen weight can be viewed
as the teacher model, otherwise student model. Some methods [2], following
Mean Teacher, make the teacher model as the EMA of the student model for
further regularization. Pseudo labeling [16] is another popular class of SSL
method which can also be treated as a kind of consistency regularization, as
one output of the unlabeled data is enforced to be consistent with the other
(the pseudo-labels) by being supervised with the other. To improve the quality
of pseudo-labels, FixMatch [27], a state-of-the-art SSL work on image
classification, has shown that the student network can improve significantly
by setting a classification confidence threshold $\tau_{cls}$ and filtering
out low-confidence predictions from the teacher. With the filtered pseudo-
labels, the student model only gets supervised on the unlabeled data whose
pseudo-labels are kept. Another key factor to the success of these methods is
strong data augmentation. It has been shown crucial to many SSL works [24, 14,
32]. Recent works [1, 27] proposed to adopt even more powerful augmentation
such as RandAugment [3] and Cutout [5].

##### Semi-Supervised Object Detection

Since the beginning of the deep learning era, tremendous progress has been
made in 2D object detection, e.g region-based detectors [9, 8, 23] and single-
stage detectors [17, 22, 31]. Similarly in 3D object detection, a number of
deep learning methods have been proposed for different 3D data modalities,
e.g. RGBD-based detectors [20, 18], point-based detectors [34, 26, 15, 19],
voxel-based detectors [36], point-voxel-based detectors [25], etc.

Despite the great progress in both 2D and 3D object detection, most works
focused on a fully-supervised setting. A few works [11, 6] have proposed to
leverage unlabeled data or weakly-annotated data for 2D object detection.
Under a standard SSL setting as we follow, CSD [12] proposed a consistency
regularization method to enforce the consistency between predictions from an
image and its flipped version. STAC [28] adopts a two-stage scheme for
training Faster R-CNN [23]: in the first stage it pre-trains a detector with
labeled data only and then predicts the pseudo labels for the unlabeled data;
in the second stage, STAC leverages asymmetric data augmentation and the
pseudo-label filtering mechanism to remove object proposals with low
confidence. Note that the pseudo-labels are only generated once at the end of
the first stage.

The only prior work on semi-supervised point-based 3D object detection, is
SESS [35]. SESS is built upon VoteNet [19] and adopts a two-stage training
scheme. It leverages a mutual learning framework composed of an EMA teacher
and a student, uses asymmetric data augmentation, and enforces three kinds of
consistency losses between the teacher and student outputs. Although SESS
brings noticeable improvements upon a vanilla VoteNet when using only a small
portion of labeled data, we find their consistency regularization suboptimal,
as it is uniformly enforced on all the student and teacher predictions. In
this work, we instead propose to apply confidence-based filtering to improve
the quality of pseudo-labels from the teacher predictions and we are the first
(in both 2D and 3D object detection) to introduce IoU estimation for
localization filtering.

##### IoU Estimation

IoU estimation was first proposed in a 2D object detection work IoU-Net [13],
which proposed an IoU head that runs in parallel to bounding box refinement
and is differentiable w.r.t. bounding box parameters. IoU-Net adds an IoU
estimation head to several off-the-shelf 2D detectors and uses IoU estimation
instead of classification confidence to guide NMS, which improves the
performance consistently over different backbones. Thanks to its
differentiability, IoU-Net can perform IoU optimization on bounding box
parameters for iterative refinement, which further brings noticeable
performance improvement.

For 3D object detection, STD [33] follows IoU-Net to add a simple IoU
estimation branch parallel with the box estimation branch and to guide NMS
with IoU estimation. PV-RCNN [25] devises a similar 3D IoU estimation module
and use it at IoU-guided NMS stage. These two modules, unfortunately, are not
suitable for IoU optimization as the features fed to the IoU estimation branch
are not differentiable w.r.t. the bounding box size. Since the original
VoteNet is not equipped with an IoU module, we devise a differentiable point-
cloud-based 3D IoU estimation module is simple yet effective that can support
the IoU optimization.

##  3 Method

In this section, we describe our solution in detail. We first formulate our
problem in 3.1 and then summarize the two object detection backbones, PV-RCNN
and VoteNet, in 3.2. We use VoteNet as an example to illustrate our proposed
3DIoUMatch pipeline in 3.3. We further explain how we use the estimated 3D IoU
for pseudo-label filtering and deduplication in 3.4. Finally, we illustrate
how we leverage the pseudo-labels for supervision in 3.5.

###  3.1 Problem Definition

Given a 3D point cloud representation of a scene
$\boldsymbol{x}\in\mathbb{R}^{N\times 3}$ containing a set of objects
$O=\\{o^{(j)}\\}$, we aim at detecting the amodal oriented 3D bounding boxes
of all objects in $O$, along with their semantic class labels. In particular,
we are interested in accomplishing this task under challenging conditions of
limited supervision where we have access to a (small) set of labeled scenes
$\\{\boldsymbol{x}^{l}_{i},\boldsymbol{y}^{l}_{i}\\}_{i=1}^{N_{l}}$ and a set
of unlabeled scenes $\\{\boldsymbol{x}^{u}_{i}\\}_{i=1}^{N_{u}}$, where
$N_{l}$ and $N_{u}$ are the number of labeled and unlabeled scenes,
respectively. For a labeled scene $\boldsymbol{x}$, the label $\boldsymbol{y}$
comprises bounding box parameters $\\{\boldsymbol{b}^{(j)}\\}$ and semantic
class labels $\\{q^{(j)}\\}$ of all ground truth objects $\\{o^{(j)}\\}$.

###  3.2 IoU-aware 3D Object Detection

We experiment our SSL method on two 3D detectors, VoteNet [19] and PV-RCNN
[25]. VoteNet is a single-stage indoor point cloud detector while PV-RCNN is a
two-stage outdoor point cloud detector. They both take point clouds only for
inputs and output a list of bounding boxes after Non-Maximum Suppression (NMS)
for each scene, which contain the prediction of center, size, orientation and
semantic class. However, their architecture are very different partly due to
the great discrepancy between indoor and outdoor scenes.

Indoor scene detector: VoteNet VoteNet [19] is built upon PointNet++ [21]
backbone, and first processes the input point cloud $\\{x_{i}\\}_{i=1}^{N}$ to
generate a sub-sampled set of $M<N$ seed points enriched with high-dimensional
features $\\{[x_{i};f_{i}]\in\mathbb{R}^{3+C}\\}_{i=1}^{M}$. Next, each seed
point votes for the center of the object it belongs to, and the votes are
grouped into $K$ clusters. Finally, each of the K vote clusters is aggregated
to make a prediction of a 3D bounding box parameters $\boldsymbol{b}^{(k)}$, a
corresponding objectness score $s_{k}=\text{P}(\boldsymbol{b}^{(k)}~{}\text{is
an object})$, and a probability distribution $\\{p_{cls}\\}$ over $L$ possible
semantic classes. The bounding box parameters $\boldsymbol{b}$ are its center
location $\mathbf{c}\in\mathbb{R}^{3}$, scale $\mathbf{d}\in\mathbb{R}^{3}$,
and orientation $\theta$ around the upright axis.

At train time, VoteNet jointly minimizes a weighted combination of the
following target losses: vote coordinate regression, objectness score binary
classification, box center regression, bin classification and residual
regression for heading angle, scale regression, and category classification.
At test time, VoteNet applies Non-Maximum Suppression (NMS) based on
objectness score to remove duplicated bounding boxes. Here, we instead rely on
a 3D IoU estimation module designed for VoteNet. For more details, refer to
the supplementary materials.

Outdoor scene detector: PV-RCNN PV-RCNN[25] is a high-performance and
efficient LiDAR point cloud detector that deeply integrates both 3D voxel CNNs
and PointNet++-style set abstraction to learn more discriminative point cloud
features. Specifically, PV-RCNN first passes the 3D scene through a novel
voxel set abstraction module based on sparse 3D CNN to get a set of keypoints
with representative scene features. Then RoI grid pooling is then applied to
the keypoints to abstract proposal-specific features into RoI grid points. The
RoI grid points containing rich context information are finally used to
accurately estimate bounding box parameters.

PV-RCNN itself incorporates an IoU-estimation module which can predict the IoU
of each bounding box and use it to guide the sorting of the boxes.

###  3.3 3DIoUMatch for SSL on 3D object detection

We take VoteNet as our example and our method with PV-RCNN is similar. With
the incorporation of 3D IoU module into VoteNet, we construct an IoU-aware
VoteNet for SSL on 3D object detection. Our proposed solution is comprised of
two training stages: a pre-training stage, where we train our IoU-aware
VoteNet on the labeled data, followed by an SSL stage where the entire data is
utilized by pseudo-labeling the unlabeled scenes.

##### Pre-training.

We start by training our IoU-aware VoteNet in a supervised manner, using the
labeled set
$\\{\boldsymbol{x}^{l}_{i},\boldsymbol{y}^{l}_{i}\\}_{i=1}^{N_{l}}$. The
training loss is a sum over the original VoteNet losses $L_{\text{votenet}}$
and 3D IoU loss $L_{\text{IoU}}$. Once converged, we clone the network to
create a pair of student and teacher networks.

##### Semi-supervised training through a teacher-student framework.

We follow a teacher-student mutual learning framework [30] and train our
networks on both labeled
$\\{\boldsymbol{x}^{l}_{i},\boldsymbol{y}^{l}_{i}\\}_{i=1}^{N_{l}}$ and
unlabeled data $\\{\boldsymbol{x}^{u}_{i}\\}_{i=1}^{N_{u}}$. Each training
batch contains a mixture of $\\{\boldsymbol{x}^{l}_{i}\\}_{i=1}^{B_{l}}$
labeled samples and $\\{\boldsymbol{x}^{u}_{i}\\}_{i=1}^{B_{u}}$ unlabeled
samples.

For labeled samples, we supervise the student network using ground truth
supervisions (as done in the pre-training stage) whereas for unlabeled
samples, the student networks is supervised using pseudo-labels
$\\{\boldsymbol{\tilde{y}}^{u}_{i}\\}_{i=1}^{N_{u}}$ generated from the
teacher network. The final loss is formed as:

$L=L_{l}(\\{\boldsymbol{x}^{l}_{i}\\}_{i=1}^{N_{l}},\\{\boldsymbol{y}^{l}_{i}\\}_{i=1}^{N_{l}})+\lambda_{u}L_{u}(\\{\boldsymbol{x}^{u}_{i}\\}_{i=1}^{N_{u}},\\{\boldsymbol{\tilde{y}}^{u}_{i}\\}_{i=1}^{N_{u}})$

where $\lambda_{u}$ is the unsupervised loss weight.

To succeed in semi-supervised learning, it is crucial for the teacher network
to generate high-quality pseudo-labels and maintain a reliable performance
margin over the student network throughout the training. As commonly used in
SSL literature, e.g. Mean Teacher [30] and SESS [35], we adopt an EMA teacher.
We further leverage asymmetric data augmentation and pseudo-label filtering
(see Sec.3.4).

To be in a position of advantage, the teacher network takes input data with
weak augmentation only while the student network uses stronger data
augmentation. We share the same data augmentation strategy with SESS. The
input point clouds to our teacher network are augmented only by random sub-
sampling while the inputs to the student network further undergo a set of
stochastic transformation $\mathcal{T}$, including random flip, random
rotation around the upright axis, and a random uniform scaling.

###  3.4 Pseudo-Label Filtering and Deduplication

In the teacher-student framework, the performance gap between the teacher and
the student is usually quite marginal given that these two models are just
different by EMA on weight and data augmentation strength. Hence, it is not
always true that the teacher prediction is more accurate than the student’s on
a specific training sample. On unlabeled data, the student model will only
benefit from the pseudo-labels that are more accurate than its predictions.
Therefore we should filter out low-quality predictions from the teacher model
and only supervise the student model with the rest of the teacher model
predictions.

##### Jointly filtering based on class, objectness, localization confidences

For VoteNet, we propose to set an objectness threshold $\tau_{obj}$ and filter
out bounding box predictions with objectness score $s<\tau_{obj}$. We further
propose to set a classification confidence threshold $\tau_{cls}$ for
filtering out predictions that are likely to contain a wrong class label.

Note that none of these two confidence measures capture the accuracy of
bounding box parameter predictions. We propose to predict a 3D IoU for each
predicted bounding box, use the 3D IoU estimation as a localization
confidence, and set a localization threshold $\tau_{\text{IoU}}$ to filter out
poorly localized predictions. Formally, we remove all the predictions that
fail to satisfy all three confidence thresholds, i.e. $s>\tau_{obj}$,
$\max(p_{cls})>\tau_{cls}$, and $v>\tau_{\text{IoU}}$.

##### IoU-guided lower-half suppression for deduplication

After the confidence-based filtering, there is still a lot of duplicated
bounding box predictions that may introduce harmful noise to our pseudo-
labels. NMS is a standard process in object detection for duplicate removal
before evaluation, which takes a set of highly overlapped bounding box
predictions that share the same class prediction, ranks them according to a
confidence score and removes all but the top-1 prediction. STAC [28] applies
class confidence based NMS to teacher predictions during pseudo-label
generation.

The default NMS used in VoteNet is based on objectness confidence. Given that
objectness score doesn’t capture the localization quality, a train-time IoU-
guided NMS will naturally perform better (see Table.2), where we use the
product of predicted IoU and predicted objectness as the ranking metric.
However, using the top one selected by IoU-guided NMS can still be suboptimal,
since the predicted IoU will inevitably carry some errors. We argue that
different from the test time scenario, pseudo-labels do not need to be fully
deduplicated. Imagine this situation: if a bounding box predicted by the
student is $0.2m$ to the left of its corresponding ground truth, it is a
foreground object and will get bounding box supervision in VoteNet. However,
if unfortunately the pseudo-label survives after non-maximal suppression is to
the right of the ground truth more than $0.1m$, this predicted bounding box
may lose supervision and be treated as a background box. This example shows
that strict non-maximal suppression can lead to a smaller number of student
model predictions that can receive supervision. Since we cannot know the best
pseudo label among a bunch of highly-overlapped ones, it’s fine to be less
strict. To this end, we propose a novel Lower-Half Suppression, or in short,
LHS, that only discards half of the proposals with lower predicted IoU. We
argue that since LHS suppresses bounding boxes sharing the same class label,
this suppression can be seen as a second-step class-aware self-adjusted
filtering, which sets dynamic thresholds among the overlapping bounding boxes
to keep the ones with higher confidence and hence find a better balance
between pseudo-label quality and the amount of supervision. We also use the
product of predicted IoU and predicted objectness as the confidence metric.

##### Final-step pseudo-label processing

After the filtering and IoU-guided LHS, we now have high-quality predictions
$\\{\hat{{y}}_{T}^{u}\\}_{k=1}^{K^{\prime}}$ from the teacher network, where
$K^{\prime}$ is the number of bounding boxes remains. Given that the student
model inputs go through a stronger augmentation including an additional
geometric transformation $\mathcal{T}$, in synchronize with the student model
inputs, the bounding box parameters of the pseudo-labels need to go through
the same transformation $\mathcal{T}$, namely
$\tilde{{\boldsymbol{b}}}^{u}=\mathcal{T}(\hat{{\boldsymbol{b}}}_{T}^{u})$. We
further take convert the predicted class probability distribution
$\hat{p}_{T}^{u}$ into semantic class label via
$\tilde{q}^{u}=\max(\hat{p}_{T}^{u})$. Now we obtain the filtered pseudo-
labels
$\tilde{y}=\\{\tilde{\boldsymbol{b}}^{u},\tilde{q}^{u}\\}_{k=1}^{K^{\prime}}$.

|  | 5% | 10% | 20% | 100%
---|---|---|---|---|---
Dataset | Model |  | mAP
---
@0.25
| mAP
---
@0.5
| mAP
---
@0.25
| mAP
---
@0.5
| mAP
---
@0.25
| mAP
---
@0.5
| mAP
---
@0.25
| mAP
---
@0.5
ScanNet | VoteNet | 27.9±0.5 | 10.8±0.6 | 36.9±1.6 | 18.2±1.0 | 46.9±1.9 | 27.5±1.2 | 57.8 | 36.0
SESS reported | $\backslash$ | $\backslash$ | 39.7±0.9 | 18.6 | 47.9±0.4 | 26.9 | 62.1 | 38.8
SESS | 32.0±0.7 | 14.4±0.7 | 39.5±1.8 | 19.8±1.3 | 49.6±1.1 | 29.0±1.0 | 61.3 | 39.0
Ours | 40.0±0.9 | 22.5±0.5 | 47.2±0.4 | 28.3±1.5 | 52.8±1.2 | 35.2±1.1 | 62.9 | 42.1
Abs. improve. | +8.0 | +8.1 | +7.7 | +8.5 | +3.2 | +6.2 | +1.6 | +3.1
SUN-RGBD | VoteNet | 29.9±1.5 | 10.5±0.5 | 38.9±0.8 | 17.2±1.3 | 45.7±0.6 | 22.5±0.8 | 58.0 | 33.4
SESS reported | $\backslash$ | $\backslash$ | 42.9±1.0 | 14.4 | 47.9±0.5 | 20.6 | 61.1 | 37.3
SESS | 34.2±2.0 | 13.1±1.0 | 42.1±1.1 | 20.9±0.3 | 47.1±0.7 | 24.5±1.2 | 60.5 | 38.1
Ours | 39.0±1.9 | 21.1±1.7 | 45.5±1.5 | 28.8±0.7 | 49.7±0.4 | 30.9±0.2 | 61.5 | 41.3
Abs. improve. | +4.8 | +8.0 | +3.4 | +7.9 | +2.6 | +6.4 | +1.0 | +3.2
Table 1: Comparison with VoteNet and SESS on ScanNet val set and SUN RGB-D val
set under different ratios of labeled data. We report the<EMAIL_ADDRESS>and<EMAIL_ADDRESS>as mean±standard deviation across 3 runs under different random data splits.
Due to the randomness of the data splits and our better pre-training protocol,
SESS results provided by us are higher than those reported in the paper on
<EMAIL_ADDRESS>and the<EMAIL_ADDRESS>results differ a little (the only difference is the
pre-trained weights and data splits). The final improvement is the absolute
improvement of our method over SESS results provided by us. Following SESS, we
also report the results with 100% labeled data, where we simply make a copy of
the full dataset as unlabeled data and train our method.

###  3.5 Selective Supervision using Pseudo-Labels

For our generated pseudo-labels, there is no guarantee that the labels can
cover all the ground truth objects from $O$ due to the filtering and
potentially inaccurate teacher predictions. Given the incompleteness of our
filtered pseudo-labels, we are relatively confident about the bounding boxes
in this set but student predictions far away from all of our pseudo-labels are
not necessarily negative. Our experiments show that supervising objectness on
unlabeled data using the pseudo-labels seriously hurts the performance. For
similar reasons, we do not supervise vote loss, which is a unique element in
VoteNet and not shown in other detectors. For more analysis and experimental
proof for this, we refer the readers to the supplementary materials. In this
case, we will only supervise the bounding boxes in the vicinity of the pseudo
bounding boxes and aim to improve their bounding box quality. More
specifically, we stick to the way how VoteNet select foreground objects for
bounding box parameter supervision: we supervise bounding box parameters and
class for a prediction only if the vote that generates this prediction is
within $0.3\text{m}$ of any bounding box in the pseudo-labels. For this set of
pseudo-foreground predictions, we adopt the same way that VoteNet establishes
association and enforce original VoteNet losses except for objectness loss and
vote loss.

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

##### Indoor Datasets: ScanNet and SUNRGB-D

We evaluate our VoteNet-based 3DIoUMatch on two major indoor datasets, ScanNet
[4] and SUN RGB-D [29]. ScanNet is an indoor scene dataset consisting of 1513
reconstructed meshes, among which 1201 are training samples and the rest are
validation samples. SUN RGB-D contains 10335 RGB-D images of indoor scenes
which are split into 5285 training samples and 5050 validation samples. For
both datasets, we follow [19, 35] for pre-processing data and labels to train
our method and we report<EMAIL_ADDRESS>(mean average precision with 3D IoU
threshold 0.25) and<EMAIL_ADDRESS>in the following experiments.

##### Outdoor Dataset: KITTI

As for our PV-RCNN-based 3DIoUMatch, we use KITTI for evaluation. KITTI [7] is
a very popular dataset for autonomous driving which consists of fine
annotations for 3D detection. There are 7481 outdoor scenes for training and
7518 for testing, and the training samples are generally divided into a train
split of 3712 samples and a validation split of 3769 samples. We follow [25]
for data pre-processing and report the mAP with 40 recall positions, with a
rotated IoU threshold 0.7, 0.5, 0.5 for the three classes, car, pedestrian,
and cyclist, respectively.

|  |  |  |  | ScanNet 10% | SUN-RGBD 5%
---|---|---|---|---|---|---
Obj&Cls Filter |  IoU Filter |  Train-time Suppression |  Test-time Suppression |  Test-time IoU opt. |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
|  |  | Obj-NMS |  | 38.4 | 19.8 | 32.9 | 12.5
✓ |  |  | Obj-NMS |  | 44.5 | 24.7 | 36.9 | 17.5
✓ |  | Obj-NMS | Obj-NMS |  | 44.2 | 25.2 | 37.1 | 17.4
✓ | ✓ | IoU-NMS | Obj-NMS |  | 45.9 | 26.8 | 37.4 | 18.7
✓ | ✓ | IoU-LHS | Obj-NMS |  | 46.5 | 26.9 | 37.9 | 18.5
✓ | ✓ | IoU-LHS | IoU-NMS |  | 47.0 | 28.2 | 38.8 | 20.8
✓ | ✓ | IoU-LHS | IoU-NMS | ✓ | 47.2 | 28.3 | 39.0 | 21.1
Table 2: Effects of the different components, including train-time filtering
and deduplication, and test-time improvements.

###  4.2 Experiments on Indoor Scene Datasets

For experiments on indoor datasets, i.e., ScanNet and SUNRGB-D, we use IoU-
aware VoteNet as our backbone detector.

####  4.2.1 Result Comparison

Table 1 shows the results of our method compared to SESS and VoteNet under
different ratios of labeled data on ScanNet and SUN RGB-D, respectively. The
results illustrate that, with our effective train-time filtering and test-time
improvement leveraging IoU estimation, we are able to significantly outperform
current state-of-the-art, SESS, under all labeled ratio settings. With 5%
labeled data, our method outperforms SESS by 8.1 and 8.0 on<EMAIL_ADDRESS>on ScanNet
and SUN RGB-D, respectively. Note that our method gains more improvement on
<EMAIL_ADDRESS>thanks to the high quality of pseudo labels and the IoU guidance for
test-time NMS.

####  4.2.2 Ablation Study

##### Filtering and Deduplication Mechanism.

We study the effect of each component of the filtering and deduplication
mechanism. In Table 2, the second row shows the results of naive pseudo
labeling, which takes all predictions from the teacher model for supervision.
Expectedly the results are not satisfying, only a little higher than VoteNet.
Simply applying the dual filtering of classification and objectness confidence
gives significant improvement, as the filtering picks out the teacher model
proposals that are very likely to be close to true objects and have the
correct class. The conventional objectness-based NMS in VoteNet, however,
fails to improve further, since the remaining proposals already have high
objectness scores and the objectness-based NMS is not capable of picking the
ones with higher localization accuracy.

As shown in the fifth and sixth row, after we introduce IoU during train time,
IoU filtering and train-time IoU-guided NMS contribute to better performance
under both settings. Our proposed IoU-guided LHS improves over IoU guided NMS
on<EMAIL_ADDRESS>since LHS finds a better balance between quality and coverage.
With better filtering and deduplication leveraging IoU estimation during train
time, we gain 2.3 and 1.7 absolute improvement over the without-IoU version on
<EMAIL_ADDRESS>and<EMAIL_ADDRESS>respectively on ScanNet 10%. This verifies that
considering localization confidence is important for getting high-quality
pseudo labels. With test-time improvements, our method gains in total 3.0 and
3.1 absolute improvement respectively.

We set 0.9 for both classification and objectness confidence threshold
following STAC [28] and investigate the effect of different IoU thresholds on
ScanNet 10%, as shown in Figure 2. The performance (with test-time
improvements) is higher than the without-IoU baseline by large margins when
$\tau_{IoU}\leq 0.5$. Note that the performance peaks at $\tau_{IoU}=0.25$ for
<EMAIL_ADDRESS>while peaking at 0.5 for<EMAIL_ADDRESS>simply because<EMAIL_ADDRESS>prefers a
stronger filtering on localization quality. When $\tau_{IoU}>0.5$ , further
increasing $\tau_{IoU}$ may lead to a drastic drop in pseudo-label coverage
and hence is detrimental to the performance.

##### Test-time IoU-guided NMS and IoU optimization.

We then evaluate the improvement brought by using IoU estimations at test
time. The last two rows in Table 2 shows that IoU-guided NMS and IoU
optimization improves the performance further.

####  4.2.3 Result Analysis

In this section, we examine how our 3DIoUMatch works during training on
ScanNet 10%. The upper two curves in Figure 3 show that as the training goes,
the performance on unlabeled data and test data increases conformably, which
indicates the increasing quality of pseudo-labels. We also show how the
coverage of the pseudo-labels changes on the unlabeled data over the training.
Here coverage at a certain threshold simply means the class-agnostic recall,
measuring the percentage of ground truth objects that can find a pseudo-label
with an IoU larger than the threshold. As we can see from the lower two curves
in Figure 3: at the beginning, the coverage of the pseudo-labels is relatively
low due to the strict filtering mechanism; as the semi-supervised learning
goes on, the improving detection performance leads to a higher passing rate of
the filter and hence a higher coverage of the pseudo-labels, which in return
fuels SSL; by the end of training, the coverage at 0.25 and at 0.5 both
increase by about 10%.

Figure 2: 3DIoUMatch results with different IoU thresholds on ScanNet 10%.
Figure 3: The performance improvements and pseudo-label coverage over the
semi-supervised learning stage on ScanNet 10%.

####  4.2.4 Implementation Details

##### Training

For the pre-training stage, we train with a batch size of 8 and follow the
same data augmentation of SESS [35]. We then use those pre-trained weights to
initialize the student and teacher networks. For the SSL stage, we construct
each batch by taking 4 labeled samples and 8 unlabeled samples, with the same
data augmentation. The weights of different loss terms are the same as VoteNet
and we set $\lambda_{u}=2$. The student network is trained for 1000 epochs
(the labeled data is traversed in one epoch), optimized by an ADAM optimizer
with an initial learning rate of 0.002, and the learning rate is decayed by
0.3, 0.3, 0.1, 0.1 at the 400th, 600th, 800th and 900th epoch, respectively.
The number of generated 3D proposals is 128. We use $k=3,D=4$ for the IoU
module. The three thresholds are set to be
$\tau_{obj}=0.9,\tau_{cls}=0.9,\tau_{IoU}=0.25$. For more details, we refer
the readers to the supplementary materials.

##### Inference

We forward the input to the student network to generate proposals. We first
apply IoU optimization to refine box parameters following IoU-Net [13],
followed by an IoU-guided NMS with a 3D IoU threshold of 0.25.

###  4.3 Experiments on KITTI

For experiments on the KITTI dataset, we adopt PV-RCNN[25] as our
backbone111Update to the CVPR camera-ready version: to avoid information
leakage, we no longer apply the ground truth sampling augmentation used by PV-
RCNN to unlabeled data in 1% and 2% labeled data experiments. We therefore
retrain the networks and update the results in Table 3 and 4. See
supplementary materials for more details.. PV-RCNN itself comes with a 3D IoU
module, allowing to use it in our semi-supervised learning pipeline without
modifying its architecture.

####  4.3.1 Results

We evaluate our method on KITTI validation set. Table 3 demonstrates
significant and consistent improvement across all categories with 1%, 2%, and
100% labeled data, compared to supervised training only. Similar to our
experiments on indoor scene datasets, here the 100% labeled data setting means
making a copy of the full dataset as unlabeled data and training the network
using our devised semi-supervised pipeline. With 2% labeled data, our method
outperforms the labeled-data-only baseline by 7.4 and 10.7<EMAIL_ADDRESS>on
pedestrian and cyclist, respectively, which are very challenging for low data
ratios.

####  4.3.2 Ablation Study

We conduct ablation studies on KITTI with 1% labeled data. Table 4 shows our
improvements compared with a pseudo-label baseline and filtering based on
class confidence only. The results validate the effectiveness of our IoU-based
localization confidence filtering.

####  4.3.3 Implementation Differences with VoteNet

In KITTI, we only care about three classes, car, pedestrian, and cyclist,
which differ a lot in the difficulty to detect. Instead of using LHS, we
follow PV-RCNN to set different IoU thresholds for each individual class,
i.e., $\tau_{car}=0.5$, $\tau_{ped}=\tau_{cyc}=0.25$. In contrast to VoteNet,
PV-RCNN is a two-stage detector containing an RPN. After the RPN stage, only a
number of proposals are kept based on classification score, which acts like
objectness score in VoteNet. We further filter with the classification score
obtained from the RPN stage, with a threshold of $\tau_{cls}=0.4$.

Due to the non-differentiability of the IoU module of PV-RCNN, we do not apply
IoU optimization.

We also adopt a two-stage training scheme for our PV-RCNN-based 3DIoUMatch. We
use an unlabeled weight $\lambda_{u}=1$ and only supervise anchor
classification and bounding box regression in PV-RCNN on unlabeled data.
Please refer to the supplementary materials for more details.

| 1% | 2% | 100%
---|---|---|---
| Car | Ped. | Cyc. | Car | Ped. | Cyc. | Car | Ped. | Cyc.
PVR. | 73.5 | 28.7 | 28.4 | 76.6 | 40.8 | 45.5 | 83.0 | 57.9 | 73.1
Ours | 76.0 | 31.7 | 36.4 | 78.7 | 48.2 | 56.2 | 84.8 | 60.2 | 74.9
Improvements | +2.5 | +3.0 | +8.0 | +2.1 | +7.4 | +10.7 | +1.8 | +2.3 | +1.8

Table 3: 3D detection results on KITTI val set with different labeled ratios.
The results are for moderate difficulty level evaluated by the mAP with 40
recall positions, with a rotated IoU threshold 0.7, 0.5, 0.5 for the three
classes, respectively.

| Car | Pedestrian | Cyclist
---|---|---|---
| Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard
PVR. | 87.7 | 73.5 | 67.7 | 32.4 | 28.7 | 26.2 | 48.1 | 28.4 | 27.1
naive psd.-lb. | 88.4 | 75.2 | 69.5 | 32.7 | 29.2 | 26.7 | 51.4 | 30.7 | 28.7
cls. filt. only | 87.9 | 75.5 | 70.5 | 36.6 | 31.0 | 28.3 | 57.3 | 35.3 | 33.0
Ours | 89.0 | 76.0 | 70.8 | 37.0 | 31.7 | 29.1 | 60.4 | 36.4 | 34.3

Table 4: Ablation study on KITTI 1% labeled data. Same evaluation metric as
Table 1.

##  5 Conclusion

In this paper, we propose 3DIoUMatch, a novel semi-supervised 3D object
detection method leveraging IoU estimation. Built upon a teacher-student
mutual learning framework, we leverage asymmetric data augmentation and
pseudo-label filtering and deduplication to facilitate the student learning
from the EMA teacher. With our IoU estimation module, we make filtering and
deduplication aware of localization confidence and apply test-time IoU-guided
NMS and IoU optimization, leading to further improvement. Experiment results
on the ScanNet, SUN-RGBD, and KITTI datasets validate the effectiveness of our
method: we achieve significant gain over the previous state-of-the-art methods
and baselines under all settings. Our idea of leveraging IoU estimation for
semi-supervised learning is generally applicable to different kinds of 3D
object detectors and can be extended to 2D detectors as future works.

Acknowledgement: This research is supported by a grant from the SAIL-Toyota
Center for AI Research, NSF grant CHS-1528025, a Vannevar Bush Faculty
fellowship, a TUM/IAS Hans Fischer Senior Fellowship, and gifts from the
Adobe, Amazon AWS, and Snap corporations.

##  Appendix A 3D IoU Estimation Module for VoteNet

To facilitate the rejection of poorly localized proposals, as well as guiding
deduplication and test-time refinement, we devise a new 3D IoU estimation
module differentiable w.r.t bounding box parameters for point-cloud-based IoU-
unaware detectors like VoteNet.

In detail, for each predicted bounding box $b^{(k)}$, we wish to estimate its
3D IoU $v^{(k)}\in[0,1]$ with respect to its corresponding ground-truth box
$\\{{o^{k}\in\\{o^{(j)}\\}}|k=\text{argmax}_{j}(\text{IoU}(b^{(k)},o^{(j)}))\\}$.
VoteNet does not have intermediate region proposals and only output bounding
box parameters at the end stage. Features used for bounding box parameter
regression are gathered from vote points in a fixed-radius ball vicinity
around each vote cluster, which are unaware of the final bounding box
prediction. So, different from implementation in IoU-Net [13] that parallel
the bounding refinement and IoU estimation, we need to do it serially by
pooling features again specifically for 3D IoU estimation using the final
predicted bounding box.

This feature pooling layer takes a bounding box as input and should generate
continuous features with respect to the change in bounding box parameters.
Existing RoI pooling methods proposed in GSPN [34] and PointRCNN [26] and 3D
IoU module proposed in STD [33] simply set a hard cropping boundary at the
bounding box surface, taking the point features inside the proposal and
ignoring any points outside. These designs have poor differentiability and
cause discontinuities whenever a change in the box parameters modifies the
point population inside the box, thus are not suitable for 3D IoU optimization
(see Table 8).

Here, for the first time, we devise a 3D pooling layer, 3D Grid Pooling, that
is differentiable with respect to the change in all bounding box parameters.
Inspired by RoIAlign [10] in 2D object detection, we propose to construct
virtual grid points spanning the space of the bounding box and their features
are obtained by distance-weighted interpolation from real points not
restricted inside the box.

##### Network architecture for IoU-aware VoteNet

Taking as inputs the seed points $\\{z_{i}\\}_{i=1}^{M}$, predicted bounding
box $b=\\{\mathbf{c},\mathbf{d},\theta\\}$, and a predicted label $l$, our 3D
IoU module estimates the largest 3D IoU between $B$ and all ground truth
bounding boxes. Following IoU-Net [13], the IoU estimation is class-aware.

To build a differentiable 3D IoU module, we first construct a $D\times D\times
D$ grid $\\{g_{m}\in\mathbb{R}^{3}~{}|~{}m\in[0,D^{3}-1]\\}$ that exactly span
over the space of $b$ and evenly divide its width, length, and height. For
each grid point $g_{m}$, we find its $k$ nearest neighbours among all seed
points and interpolate their features to get
$f_{m}=\frac{\Sigma_{i=1}^{k}w_{i}f_{i}}{\Sigma_{i=1}^{k}w_{i}}$, where
$w_{i}=\frac{1}{d(g_{m},g_{i})^{2}}$ and $d$ is the L2 distance. Ideally, if
$k$ is equal to the number of all seed points, then the IoU module is
continuously differentiable. Due to the computational cost, we empirically
find $k=3$ is sufficient for accurate 3D IoU estimation with smooth gradients.
We then concatenate $g_{m}$ and $f_{m}$ for each grid point and form a grid
feature set $\\{[g_{m};f_{m}]\\}$. The feature set will be pushed towards a
PointNet to predict class-aware 3D IoU. A final 3D IoU selection will be
performed using the input class label.

Our IoU-aware VoteNet shares the same structure with VoteNet[19] except for
the IoU estimation module. We provide a more detailed description of the IoU
estimation module here. The IoU estimation module is appended after the
proposal generation module of VoteNet and takes the bounding box proposals as
input. For each bounding box proposal, we create $4\times 4\times 4$ virtual
grid points. We obtain the relative coordinates of the grid points by
subtracting the coordinates of the bounding box center. For every grid point
we find its $k$ nearest neighbours among all seed points and interpolate their
features to get
$f_{m}=\frac{\Sigma_{i=1}^{k}w_{i}f_{i}}{\Sigma_{i=1}^{k}w_{i}}$, where
$w_{i}=\frac{1}{d(g_{m},g_{i})^{2}}$ and $d$ is the L2 distance. The
interpolated features of every grid point is then concatenated with the
relative coordinates and forwarded into an MLP with channel dimensions of
[256+3, 128, 128, 128] to learn a new feature. Then the features of all grid
points go through a global max pooling, after which go through another MLP
with channels [128, 128, 128, C], where $C$ is the number of classes, to make
the IoU prediction class-aware. Finally, we select the per box IoU estimation
by using the class label (during training) or class prediction (during
inference).

##### Training IoU Estimation Module

To train the 3D IoU estimation branch in both stages, we generate on-the-fly
training data via jittering the bounding box predictions, i.e. adding Gaussian
noise to the box center and size. As a way of data augmentation, this
jittering is essential for the generalization of IoU estimation to unlabeled
data. We use an L1 loss to supervise the IoU estimation module.

Figure 4: 3D IoU module takes inputs seed feature points and a bounding box
along with its predicted class label, and estimates the 3D IoU between the box
and its maximum overlapping ground truth. The module constructs a 3D regular
grid with $D^{3}$ virtual grid points spanning over the bounding box. We then
perform a 3D grid feature pooling that applies a distance-weighted
interpolation for feature propagation from the seed points to the grid points.
Then the local coordinates of these grid points along with their features are
pushed through a PointNet to regress class-aware 3D IoU. Finally, we use the
input class label for output selection.

##  Appendix B More Implementation Details for VoteNet-based 3DIoUMatch

##### Training

For the pre-training stage, we find that the network does not converge using
the same protocol as fully-supervised VoteNet. We instead use a new protocol,
where the network is trained for 900 epochs, optimized by an ADAM optimizer
with an initial learning rate of 0.001, and the learning rate is decayed by
0.1, 0.1, 0.1 at the 400th, 600th and 800th, respectively. We observe
convergence using this protocol on all ratios of labeled data.

Inspired by IoU-Net[13], for both stages, we generate on-the-fly training data
via jittering the bounding box predictions for the IoU estimation module.
Specificly, we add $\epsilon_{size}\sim N(0,(0.3\mathbf{d})^{2})$ to each
bounding box size prediction $\mathbf{d}$ and add $\epsilon_{center}\sim
N(0,(0.3\mathbf{d})^{2})$ to each bounding box center prediction $\mathbf{c}$
to obtain $N_{proposal}$ more training samples. The final IoU estimation loss
is the L1 loss averaged over all IoU trainig samples, original predictions or
jitters. The IoU estimation loss weight is 1.

##### Inference

As IoU-Net[13] did not release code, we implemented a simple version of test-
time IoU optimization.

  1. 1.

We obtain the original bounding box proposals.

  2. 2.

We calculate the gradients of the IoU estimation w.r.t. to bounding box size
and center, $grad_{size},grad_{center}$, and update the bounding box size and
center by adding $grad_{size}\ast\lambda,grad_{center}\ast\lambda$ to the box
size and center, respectively, where $\lambda$ is the optimization step size.

  3. 3.

We repeat the second step for $T$ times.

We find setting $T$ to 10 yields noticeable improvement while not slowing
inference speed too much. Choosing $\lambda$ from the range of
$[1e^{-4},5e^{-4}]$ has similar performance.

##  Appendix C More Implementation Details for PV-RCNN-based 3DIoUMatch

We basically follow the training protocols and settings of PV-RCNN [25]. For
pre-training on small amounts of labeled data (1% and 2%), we lengthen the
number of traverses in each epoch to ten times the origin for the model to
converge. For SSL training, we set the batch size to 16 (8 labeled + 8
unlabeled, 8 GPUs) and lengthen the number of traverses in each epoch to five
times the origin.

PV-RCNN uses a data augmentation method called ground truth sampling. This
method first collects all bounding boxes in the whole set of training scenes
as well as the point cloud enclosed in the boxes, and then randomly places a
sampled set of these boxes and points (called the database) in each scene
during training. If a sampled box from the database gets overlapped with an
original box in a training scene, this box is discarded. For semi-supervised
experiments in which we only use a small fraction of labels, we have to
downsize the database to contain boxes from the exact set of used labels only.
For pseudo-label generation, we even should not use this augmentation since we
have no annotation, thus failing to remove sampled boxes that overlap with
ground truth objects.

During training, our pseudo-labels are generated by the teacher model in train
mode. However, with limited labels, we should not leverage ground truth when
selecting RPN proposals to feed to the second stage. Therefore, we adopt the
test-time RPN proposal selection strategy of PV-RCNN when it comes to
generating pseudo-labels under a semi-supervised setting.

Nevertheless, for 100%-label setting, we can leverage all labels, so we are
free to use train mode for pseudo-label generation, and we adopt ground truth
sampling augmentation for the ”unlabeled” data. We also set thresholds to be
$\tau_{car}=0.8,\tau_{ped}=\tau_{cyc}=0.4,\tau_{cls}=0.2$ for 100% data, since
the pre-trained model of 100% data should be more accurate in predicting the
position of bounding boxes. In the second stage of 100%-label experiment, we
train for 40 epochs, without lengthening each epoch.

##  Appendix D Overhead of the IoU module

Our light-weighted IoU estimation module brings moderate overhead to the
network, as shown in Table 5. The memory reported in the second column refers
to the memory consumed by training with batch size 8 on a single GTX 1080Ti
GPU. The last two columns mean the time consumed by a full pass (forwarding
and backwarding) of a batch of 8 on a single GTX 1080Ti GPU, training ScanNet
and SUNRGB-D respectively. Note that regardless of the network design, there
is overhead introduced by calculating the ground truth IoU for supervision.

Method | Mem. (GB) | ScanNet (s) | SUNRGB-D (s)
---|---|---|---
VoteNet | 6.56 | 0.282 | 0.316
Ours | 6.60 | 0.325 | 0.377
Table 5: Memory and time overhead of the IoU module.

cab bed chair sofa table door wind bkshf pic cntr desk curt fridg showr toil
sink bath ofurn VoteNet<EMAIL_ADDRESS>17.9 74.7 74.5 75.3 45.6 18.3 11.7 21.7 0.7
28.4 49.4 21.5 23.2 18.5 79.6 25.7 66.3 11.7 SESS<EMAIL_ADDRESS>20.5 75.1 76.2 76.4
48.1 20.0 14.4 19.4 1.2 30.0 51.8 25.0 30.0 26.4 82.2 29.2 72.3 14.1 Without
IoU<EMAIL_ADDRESS>22.6 79.5 77.8 77.8 49.6 25.4 18.6 27.7 3.3 41.4 56.2 27.4 30.4
53.6 81.3 28.5 74.5 18.8 3DIoUMatch<EMAIL_ADDRESS>26.6 82.6 80.9 83.3 52.1 28.0
19.9 29.4 3.7 45.0 61.9 29.2 34.1 51.2 85.7 32.3 82.8 21.5 VoteNet<EMAIL_ADDRESS>3.2
64.6 43.4 49.3 25.1 2.8 1.1 8.7 0.0 2.4 14.7 3.9 7.6 1.1 46.8 11.9 39.4 1.5
SESS<EMAIL_ADDRESS>3.7 61.2 48.0 44.8 29.5 3.2 2.8 8.4 0.2 7.5 19.2 5.0 12.2 1.8
48.7 15.3 40.8 2.2 Without IoU<EMAIL_ADDRESS>3.9 66.1 52.7 50.7 35.1 7.9 5.0 13.1
0.9 14.5 26.1 10.3 17.5 7.0 63.9 11.7 62.1 4.9 3DIoUMatch<EMAIL_ADDRESS>5.9 72.0
60.5 56.6 39.7 10.3 5.2 18.1 0.7 16.0 35.3 8.3 21.4 6.2 67.5 13.2 67.6 5.2

Table 6: Per class<EMAIL_ADDRESS>and<EMAIL_ADDRESS>on ScanNet val set, with 10% labeled
data.

bathtub bed bookshelf chair desk dresser nightstand sofa table toilet VoteNet
<EMAIL_ADDRESS>67.8 32.2 39.4 58.5 53.5 8.0 1.9 14.7 3.2 20.3 SESS<EMAIL_ADDRESS>70.8
34.7 41.9 60.4 63.0 9.8 3.7 25.2 4.0 28.0 Without IoU<EMAIL_ADDRESS>75.1 33.5 43.0
59.5 76.9 6.8 5.1 33.0 3.5 34.8 3DIoUMatch<EMAIL_ADDRESS>75.4 37.7 45.2 64.2 77.0
6.0 5.7 34.6 4.5 39.4 VoteNet<EMAIL_ADDRESS>31.2 6.2 15.5 29.6 14.6 0.5 0.2 2.0 0.3
5.2 SESS<EMAIL_ADDRESS>36.7 7.2 19.2 31.8 20.4 0.7 0.5 7.0 0.4 7.1 Without IoU
<EMAIL_ADDRESS>41.5 9.7 25.7 34.5 40.8 0.8 0.8 8.3 0.8 11.4 3DIoUMatch<EMAIL_ADDRESS>45.2
14.4 27.8 43.6 47.2 0.8 1.9 15.7 0.6 13.4

Table 7: Per class<EMAIL_ADDRESS>and<EMAIL_ADDRESS>on SUNRGB-D val set, with 5% labeled
data.

##  Appendix E IoU Module Comparison

As mentioned in A, some region-based 3D detectors, e.g. STD [33], crop the
features inside a predicted bounding box and regress the offset. To capture
their core characteristics under IoU optimization, we build a simple IoU
estimation module which only queries points inside the predicted box and
passes the queried feature points through a PointNet to predict the 3D IoU,
namely box query. In principle, the differentiability of this module is the
same as that in STD, which doesn’t release their code and misses the IoU
optimization step in their paper. For a fair comparison, we train another IoU-
aware VoteNet with box query as the IoU estimation module and show the
comparison between it and our proposed method on the full set of ScanNet and
SUN RGB-D. From the results in Table 8, we prove that our method is more
effective on both IoU-guided NMS and IoU optimization than box query.

We provide more explanation on why an IoU estimation network design like that
in STD[33] is less effective in IoU estimation and is not differentiable.
Given a bounding box proposal, STD concatenates the canonized coordinates and
features of the points inside the bounding box to form new features of the
points. Therefore, the new feature of a point $f^{\prime}$ can be denoted as a
function of the point coordinates $\mathbf{p}$, the original point feature
$f$, the bounding box center $\mathbf{c}$ and the bounding box heading angle
$\theta$. Then STD voxelizes the bounding box and sample points in each voxel
to produce the voxel feature $f_{v}$. The process of producing the voxel
feature from points in voxels consist of no other parameters except from the
point features $\\{f^{\prime}_{i}\\}$ and point coordinates $\mathbf{p}_{i}$,
so $f_{v}$ is still a function of
$\\{f_{i}\\},\\{\mathbf{p}_{i}\\},\mathbf{c},\theta$. As all voxel features
are flattened and fed to an MLP, which outputs the final IoU, we can conclude
that the IoU estimation is not differentiable w.r.t. bounding box size.

We also argue that for VoteNet, since the number of seed points with features
are small (1024), box query methods may have difficulty querying points inside
a bounding box, especially if a bounding box is too small. Our method, instead
won’t suffer from this as we are not confined to points inside the bounding
box.

Although STD didn’t release code, we still implemented an IoU estimation
module according to the paper for better comparison. However, some issues need
to be stated. First, since the backbone of STD is very different from VoteNet,
the comparison between IoU estimation module alone is inherently problematic.
Second, STD aims at outdoor object detection, where the task is slightly
different. Third, we adopted most of the parameters of STD in the paper, but
changed number of voxels (to 27) and number of points sampled per voxel (to 6)
due to memory concerns and the small number of seed points in VoteNet. The
results in Table 8 show the better performance of our IoU module. We also
observe serious overfitting using the STD IoU module, suggesting that it may
not be suitable for our problem.

| ScanNet | SUN RGB-D
---|---|---
Method |  | mAP
---
@0.25
| mAP
---
@0.5
| mAP
---
@0.25
| mAP
---
@0.5
Obj-NMS [19] | 57.84 | 35.99 | 58.01 | 33.44
| Box query
---
IoU-NMS only
57.56 | 37.07 | 58.16 | 34.81
| Box query
---
IoU-NMS + Optim.
57.62 | 37.17 | 58.19 | 34.90
| STD
---
IoU-NMS only
57.81 | 36.21 | 58.21 | 34.76
| STD
---
IoU-NMS + Optim.
57.85 | 36.21 | 58.21 | 34.75
| Ours
---
IoU-NMS only
57.92 | 37.01 | 58.82 | 36.22
| Ours
---
IoU-NMS + Optim.
58.46 | 37.43 | 59.11 | 36.71
Table 8: Comparison of our IoU module with box-query on ScanNet 100% and SUN
RGB-D 100% . Figure 5: Qualitative results on ScanNet, with 10% labeled data.
Here green bounding boxes have an IoU $\geq 0.25$ while red bounding boxes are
with an IoU $<0.25$. Figure 6: Qualitative results on SUNRGB-D, with 5%
labeled data.

##  Appendix F Why not Supervise Votes and Objectness in VoteNet?

As we mentioned, we supervise all VoteNet loss terms on unlabeled data except
for vote regression loss and objectness binary classification loss. As we
observe, supervising votes or objectness with pseudo labels leads to degrading
performance. The main reason is that by rigorous filtering and deduplication
we can only be highly confident of a true object being close to a pseudo
bounding box, but we are not sure whether or not there is a true object where
there are no pseudo bounding boxes nearby. If we supervise objectness on
unlabeled data with the pseudo labels the same way as VoteNet, it’s not
difficult to imagine the network would be more and more biased on detecting
objects. In Table 9, our experiments on ScanNet 10% and SUNRGB-D 5% show that
the performance suffers a drop after supervising objectness on unlabeled data.

Vote prediction is an unique component of VoteNet. For a point, the label for
its vote is the center of the object it belongs to. To generate pseudo vote
labels, the straightforward way is to count every point inside a pseudo
bounding box as a vote. However, since this pseudo vote label set is also far
from complete, we face a similar problem supervising with it. In Table 9, our
experiments on ScanNet 10% and SUNRGB-D 5% also show that the performance
drops after supervising vote prediction on unlabeled data.

| ScanNet 10% | SUNRGB-D 5%
---|---|---
Method |  | mAP
---
@0.25
| mAP
---
@0.5
| mAP
---
@0.25
| mAP
---
@0.5
3DIoUMatch | 47.2 | 28.3 | 39.0 | 21.1
| +vote sup.
---
on unlabeled
45.4 | 28.3 | 37.9 | 20.9
| +obj. sup.
---
on unlabeled
40.1 | 26.0 | 38.2 | 20.4
Table 9: Objectness & vote supervision on unlabeled data using pseudo-labels.

##  Appendix G Per-class Evaluation

We report per-class average precision on ScanNet with 10% labeled data and
SUNRGB-D with 5% labeled data, respectively. The bold numbers are the highest
per class. The results in Table 6, 7 show that our method improves the average
precision on nearly all classes over SESS. Our 3DIoUMatch also has better
performance on most classes than the without-IoU version.

##  Appendix H Qualitative Results

We show the qualitative results on ScanNet val set with 10% labeled training
data, Figure 5 and on SUNRGB-D val set with 5% labeled training data, Figure
6. For the results of our method, SESS and VoteNet, green bounding boxes are
the predicted bounding boxes whose IoU $\geq 0.25$, and the red bounding boxes
are those with an IoU $<0.25$. As can be seen in both figures, our method give
more accurate predictions and significantly reduces the number of false
positives.

## References

  * [1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  arXiv preprint arXiv:1911.09785, 2019.
  * [2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In Advances in Neural Information Processing Systems, pages 5049–5059, 2019.
  * [3] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
  * [4] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In CVPR, 2017.
  * [5] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [6] Jiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, and Ram Nevatia.  Note-rcnn: Noise tolerant ensemble rcnn for semi-supervised object detection.  In Proceedings of the IEEE international conference on computer vision, pages 9508–9517, 2019.
  * [7] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [8] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [9] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [11] Judy Hoffman, Sergio Guadarrama, Eric S Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko.  Lsda: Large scale detection through adaptation.  In Advances in Neural Information Processing Systems, pages 3536–3544, 2014.
  * [12] Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak.  Consistency-based semi-supervised learning for object detection.  In Advances in neural information processing systems, pages 10759–10768, 2019.
  * [13] Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang.  Acquisition of localization confidence for accurate object detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 784–799, 2018.
  * [14] Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  arXiv preprint arXiv:1610.02242, 2016.
  * [15] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12697–12705, 2019.
  * [16] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on challenges in representation learning, ICML, volume 3, 2013.
  * [17] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [18] Charles R. Qi, Xinlei Chen, Or Litany, and Leonidas J. Guibas.  Imvotenet: Boosting 3d object detection in point clouds with image votes, 2020.
  * [19] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3d object detection in point clouds.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9277–9286, 2019.
  * [20] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas.  Frustum pointnets for 3d object detection from rgb-d data.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 918–927, 2018.
  * [21] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Advances in neural information processing systems, pages 5099–5108, 2017.
  * [22] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [23] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [24] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  In Advances in neural information processing systems, pages 1163–1171, 2016.
  * [25] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529–10538, 2020.
  * [26] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–779, 2019.
  * [27] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  arXiv preprint arXiv:2001.07685, 2020.
  * [28] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister.  A simple semi-supervised learning framework for object detection.  arXiv preprint arXiv:2005.04757, 2020.
  * [29] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In CVPR, 2015.
  * [30] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Advances in neural information processing systems, pages 1195–1204, 2017.
  * [31] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE international conference on computer vision, pages 9627–9636, 2019.
  * [32] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le.  Unsupervised data augmentation for consistency training.  arXiv preprint arXiv:1904.12848, 2019.
  * [33] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Std: Sparse-to-dense 3d object detector for point cloud.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1951–1960, 2019.
  * [34] Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas J Guibas.  Gspn: Generative shape proposal network for 3d instance segmentation in point cloud.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3947–3956, 2019.
  * [35] Na Zhao, Tat-Seng Chua, and Gim Hee Lee.  Sess: Self-ensembling semi-supervised 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11079–11087, 2020.
  * [36] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4490–4499, 2018.