11institutetext: National Tsing Hua University
11email: {pingchungyu<EMAIL_ADDRESS><EMAIL_ADDRESS>
#  Data Efficient 3D Learner via Knowledge Transferred from 2D Model

Supplementary Material

Ping-Chung Yu  Cheng Sun  Min Sun

###### Abstract

Collecting and labeling the registered 3D point cloud is costly. As a result,
3D resources for training are typically limited in quantity compared to the 2D
images counterpart. In this work, we deal with the data scarcity challenge of
3D tasks by transferring knowledge from strong 2D models via RGB-D images.
Specifically, we utilize a strong and well-trained semantic segmentation model
for 2D images to augment RGB-D images with pseudo-label. The augmented dataset
can then be used to pre-train 3D models. Finally, by simply fine-tuning on a
few labeled 3D instances, our method already outperforms existing state-of-
the-art that is tailored for 3D label efficiency. We also show that the
results of mean-teacher and entropy minimization can be improved by our pre-
training, suggesting that the transferred knowledge is helpful in semi-
supervised setting. We verify the effectiveness of our approach on two popular
3D models and three different tasks. On ScanNet official evaluation, we
establish new state-of-the-art semantic segmentation results on the data-
efficient track. Code: https://github.com/bryanyu1997/Data-
Efficient-3D-Learner.

###### Keywords:

knowledge transfer, 3D semantic segmentation, point cloud recognition, 3D pre-
training, label efficiency

##  1 Introduction

Nowadays, 3D sensors are in demand by applications like AR/VR, 3D
reconstruction, and autonomous driving. To have a high-level scene
understanding (_e.g_., recognition, semantic segmentation) on the captured 3D
data, deep-learning-based models are typically employed for their outstanding
performance. As 3D sensors become easier accessible, the architecture of deep
3D models [29, 30, 39, 42, 36, 8, 14] also progresses steadily for better
result qualities. In this work, we investigate an orthogonal direction to
model architecture design—we present a novel 3D models pre-training strategy
to improve performance in a model agnostic manner.

In 2D vision tasks, model pre-training on ImageNet [10] has become a commonly
applied strategy for achieving better performances across different downstream
tasks. However, there is no standard large-scale dataset like ImageNet to pre-
train 3D models due to the considerable effort to acquire and label a diverse
set of point cloud data compared to the 2D counterpart. As a result, 3D models
are typically trained from scratch, which hinders the performance, especially
under the data scarcity scenario of the registered point cloud.

Figure 1:  Left-panel: collecting labeled 3D data is challenging due to the
need for (a) robust 3D registration and alignment processes, and (b) a time-
consuming human labeling process. Right-panel: a large amount of RGB-D data
serve as the bridge between 2D and 3D knowledge. First, pseudo semantic labels
are generated by applying an off-the-shelf 2D model on RBG images. Next, the
pseudo-labeled 2D data are lifted to 3D using the associated depth map.
Finally, we pre-train a 3D model with the large amount of pseudo-labeled 3D
data.

To avoid the burden of data labeling, self-supervised learning has emerged as
an alternative for pre-training 2D models without labels [6, 3, 19, 7, 16,
18]. To reproduce this trend in 3D data, PointContrast [41] uses contrastive
loss to learn the correspondence between two point clouds with visual overlap,
improving results on the downstream 3D tasks. However, the diversity and scale
of 3D datasets, even releasing the dependency on labeling, are still not
comparable to the 2D datasets. For instance, ScanNet [9], which is used in
PointContrast’s pre-training, has only about a thousand indoor scenes, while
ImageNet has more than a million images covering a thousand different classes.
As a result, the accuracy improvement by self-supervised learning on point
clouds is still limited for 3D tasks.

To address the issue of limited resources of point clouds, we present a novel
3D model pre-training approach via transferring the learned knowledge of a 2D
model via RGB-D datasets (see Fig. 1). The single view depth sensor is much
cheaper than ever and could be widely popular as a built-in function of phones
to capture various scenes. Using RGB-D data as the bridge to transfer
knowledge from strong 2D models to 3D models is thus a valuable direction to
explore. Specifically, we employ a 2D semantic segmentation model, which is
trained on a large and diverse scene parsing dataset, to augment the RGB-D
images with pseudo-labels. We then train a 3D model to take the 3D point cloud
lifted from RGB-D as input and reproduce the pseudo-labels. By doing so, the
3D models can learn from the strong 2D teacher model and also see a large
variety of scenes captured by the RGB-D data.

We demonstrate the effectiveness of our pre-training on semantic segmentation
for scene point cloud of the popular ScanNet dataset. Annotating 3D scene
point clouds is a demanding task, which takes more than 20 minutes to label a
single scene. Some recent approaches emerges to learn from fewer 3D labels to
reduce the labeling cost. However, the lack of large-scale pre-training
hinders their performances. We show that our pre-training with simple fine-
tuning on the scarce label can already outperform existing results tailored
for 3D data efficiency [41, 26, 21]. We also show our pre-training can boost
the performance of the widely used semi-supervised techniques, which suggests
that our pre-training provides an opportunity for future 3D research on both
data efficiency and semi-supervised learning to build upon our results for
better quality. Finally, despite pre-training on a scene level, we also
evaluate our models on object-level tasks and observe improved performances on
3D object classification and shape part segmentation. This suggests that our
pre-training is also well transferred to different tasks and input 3D scales.

We summarize our contributions as follows:

  * •

We introduce a pre-training strategy to transfer knowledge from a strong 2D
scene parsing model via RGB-D images to 3D models.

  * •

We demonstrate the effectiveness of our pre-training under limited data
scenario across two models (_i.e_., O-CNN [37], SparseConv [14]) and three
different tasks (_i.e_., 3D object classification, point-cloud part
segmentation, and indoor point-cloud segmentation).

  * •

By simply finetuning our pre-trained model on a few labels, we establish new
state-of-the-art results on ScanNet [9] official evaluation on data-efficient
setups, verifying that our pre-training results in data-efficient 3D learners.

##  2 Related work

##### Deep 3D models for point cloud understanding.

As the 3D analysis studies flourish, several deep models are proposed to
extract point cloud features for a high-level understanding. Existing
approaches can be classified into point-based and voxel-based methods.
PointNet [29] and PointNet++ [30] are the pioneering point-based methods to
apply multi-layer perceptron layers directly on point clouds. After that,
several convolution-based models [24, 39, 36, 1] are proposed, achieving
better quality. Recently, attention-based models [42] have become a new
effective way for point cloud processing. On the other hand, the voxel-based
method is attractive for its computational-friendly 3D data representations. A
discretized step is typically applied on the point cloud before employing the
voxel-based models [40, 28].

As the cubic memory complexity limits the resolution of dense voxel grids, 3D
sparse CNNs [13, 14, 8] emerges to achieve a feasible space-time complexity
for scene-level point clouds, where the CNNs are working on occupied voxels
only. OctNet [32] and O-CNN [37] further use the octree data structure to
achieve a higher grid resolution efficiently.

As the deep architectures for 3D point clouds progress steadily, all of them
are still trained from scratch due to the lack of a large-scale point cloud
dataset for pre-training. To sidestep the issue of 3D resources, this work
presents a model agnostic pre-training strategy using 2D resources to improve
performances.

##### Data-efficient 3D.

Data-efficient learning restricts the amount and the variety of labeled data
of the target task, which is helpful for tasks on scene-level point clouds.
Registered scene point clouds are hard to acquire and time-consuming to label
(_e.g_., 20 more minutes for a ScanNet scene). Thus, data-efficient solutions
are always welcome. Existing works have explored self-supervised pre-training
and semi-supervised learning using unlabeled scene point clouds to improve
performance. PointContrast [41] sub-samples partial scans and use contrastive
learning to pre-train 3D models to identify point correspondents. CSC [21]
further improves the pre-training by incorporating spatial scene contexts into
the objective. As negative pairs sampling in contrastive learning could be
ambiguous, ViewPointBN [27] proposed to use correlation as the objective
instead. The aforementioned self-supervised pre-training achieves good label
efficiency in a target task agnostic manner. In OTOC [26], pseudo-label-based
semi-supervised learning is employed to simultaneously learn from both labeled
and unlabeled data, achieving superior label efficiency. We use the abundant
2D resources to pre-train instead of 3D, achieving state-of-the-art label
efficiency on scene point clouds.

##### Knowledge transferred from 2D.

Most recently, DepthContrast [45] and Contrastive Pixel-to-point [25] are also
proposed to pre-train 3D models using 2D RGB-D datasets. DepthContrast trains
3D models to discriminate 3D point clouds projected from different RGB-D
images. Contrastive Pixel-to-point treat each 3D point as instance and apply
contrastive loss to learn from pixel features extracted by a trained 2D model.

One challenge of these contrastive-based methods is the ambiguity of the
selected negative pairs. Negative pairs sampling is a crucial problem for
contrastive pre-training to avoid collapsing solutions [7, 11]. The sampling
strategy requires careful designs [19, 6] in the contrastive pre-training,
even for images. As the diversity of the employed indoor pre-training data is
still limited, the model could have difficulty differentiate two room with
similar setup or two 3D points belonging to the same stuff (_e.g_., walls).
Conversely, we directly train the 3D models to reproduce the pseudo labels
generated by a strong and well-trained image scene parser, which is more
straightforward but non-trivial to learn as we have clear and informative
supervision for each 3D point.

2D3DNet [12] trains 3D models with only 2D supervision, which requires the 2D
teacher to be a specialist for the downstream task and the 3D model has to co-
work with the 2D teacher to achieve good results. Our pre-trained 3D model
does not work directly in the downstream 3D task, but it is generally benefit
to different downstream task and our 3D model works independently after pre-
training.

##  3 Approach

Our goal is to pre-train 3D models in an architecture agnostic manner. An
overview of the proposed approach is illustrated in Fig. 2. Below, we first
give a general introduction to 3D models in Section 3.1. In Section 3.2, we
detail the proposed pre-training approach. We also apply our pre-trained model
on some semi-supervised techniques in Section 3.3.

Figure 2:  Overview of our approach. (Below) We use a strong and well-trained
image scene parser to augment single view RGB-D datasets with pseudo-labels,
which is used to pre-train a 3D model in an architecture agnostic manner.
(Top) Our pre-training improves the results of the limited annotation
training.

###  3.1 3D model

Our pre-training approach is model agnostic and does not require specific 3D
architecture designs. Here, we describe a general 3D encoder-decoder to
extract deep features from point clouds. A 3D point cloud
$\bm{x}^{\text{(pts)}}$ represents a 3D scene or object by a set of 3D
coordinates (with additional color or normal features). For voxel-based
models, a pre-process input layer before the first model layer is needed to
discretize the coordinates into a regular 3D grid. The encoder $E$, consisting
of a sequence of 3D convolution layers, batch normalizations, and nonlinear
activation (_e.g_., ReLU), is a bottom-up way to map the point clouds into a
down-sampled latent features $\bm{z}^{\text{(e)}}=E(\bm{x}^{\text{(pts)}})$,
which has the high-level understanding of the input scene or object. The
decoder $D$ then upsamples and incorporates the low-level feature into
$\bm{z}^{\text{(e)}}$ to have a holistic point-wise deep feature
$\bm{z}^{\text{(d)}}=D(\bm{z}^{\text{(e)}})$.

##### Classification.

To classify a whole input point cloud into a pre-defined set of classes, we
discard the decoder $D$ as we do not need point-level features. The high-level
features $\bm{z}^{\text{(e)}}$ are first aggregated into a single latent
vector via global max pooling, which is then followed by a classification head
$\operatorname{ClsHead}$ to map the latent vector into a categorical
distribution:
$\bm{y}^{\text{(cls)}}=\operatorname{ClsHead}(\operatorname{GlobalMaxPool}(\bm{z}^{\text{(e)}}))$,
where the $\operatorname{ClsHead}$ consists of a linear layer and a
$\operatorname{Softmax}$ layer.

##### Point-level semantic segmentation.

The semantic segmentation head simply maps the point features into per-point
class distribution by a linear and a $\operatorname{Softmax}$ layer:
$\bm{y}^{\text{(ss)}}=\operatorname{SegHead}(\bm{z}^{\text{(d)}})$.

###  3.2 Knowledge transferred from 2D

Our approach aims to pre-train a 3D model by transferring the knowledge of a
strong 2D model learned from a large-scale 2D dataset. However, the 2D model
$F^{\text{(2D)}}$ takes images $\bm{x}^{\text{(img)}}$ as input while the 3D
model takes point clouds $\bm{x}^{\text{(pts)}}$. To learn from 2D models, we
use RGB-D dataset
$\left\\{\left(\bm{x}^{\text{(img)}},\bm{d}^{\text{(img)}}\right)\right\\}$ as
the bridge, where $\bm{d}^{\text{(img)}}$ is the depth map of a image. Our
idea is to use a well-trained 2D model to generate pseudo-label for the images
in the RGB-D dataset

$\bm{t}=F^{\text{(2D)}}\left(\bm{x}^{\text{(img)}}\right)~{},$ (1)

and then we lift the image to point cloud using their depth maps

$\bm{x}^{\text{(img2pts)}}=\operatorname{Lift}\left(\bm{x}^{\text{(img)}},\bm{d}^{\text{(img)}}\right)$
(2)

so that we can train the 3D model using the augmented RGB-D dataset
$\left\\{\left(\bm{x}^{\text{(img2pts)}},\bm{t}\right)\right\\}$. Below, we
introduce the explored two different sources of RGB-D data and detail our pre-
training strategy.

##### Lifting perspective images.

With the provided depth maps and the camera intrinsic $K$, we can lift a 2D
image coordinate $\begin{bmatrix}u,v\end{bmatrix}$ to the 3D camera coordinate
$\begin{bmatrix}x,y,z\end{bmatrix}$ by:

$\displaystyle\begin{bmatrix}x\\\ y\\\ z\end{bmatrix}=d\cdot
K^{-1}\begin{bmatrix}u\\\ v\\\ 1\end{bmatrix}~{},$ (3)

where $d$ is the depth value of a pixel and
$\begin{bmatrix}u,v,1\end{bmatrix}^{\intercal}$ is the homogeneous coordinate.

##### Lifting panoramic images.

Panoramic images cover more information in one shot attributed to the
omnidirectional field of view compared to perspective images. Unlike
perspective depth maps where z-values are recorded, panoramic depth maps
directly record the distance between the observed 3D points to camera. We can
lift a panoramic image to 3D by:

$\displaystyle\left\\{\begin{array}[]{llr}x&=d\cdot\cos(v)\cdot\cos(u)\ ;&\\\
y&=d\cdot\cos(v)\cdot\sin(u)\ ;&\\\ z&=-d\cdot\sin(v)~{},&\end{array}\right.$
(7)

where $d$ is the recorded distance of a pixel in the panoramic depth maps, and
$u\in\left[-\pi,\pi\right],v\in\left[-\frac{\pi}{2},\frac{\pi}{2}\right]$ are
the panoramic image coordinate in UV space.

##### Learning from 2D scene parser via soft pseudo-label.

Image scene parsing aims to classify each pixel of images, which provides a
thorough and detailed understanding of the captured scenes. Besides, existing
training corpora for image scene parsing [48, 2] is abundant, with more than
10k images covering a large variety of scenes and classes. Scene parsing
models [46, 5, 49, 44, 22, 31] have also progressed steadily and achieved
strong performances. In this work, we adopt DPT [31] for its outstanding
performance, and we find it generalized well on both perspective and panoramic
images.

We first attach a point semantic segmentation head
$\operatorname{SegHead}^{\text{(pre)}}$ on the 3D encoder decoder to classify
$C$ classes of the trained DPT

$\bm{y}^{\text{(img2pts)}}=\operatorname{SegHead}^{\text{(pre)}}\left(D\left(E\left(\bm{x}^{\text{(img2pts)}}\right)\right)\right)~{}.$
(8)

Instead of the one-hot decision, soft labels are learned as the target during
pre-training. Soft-label is first introduced in a knowledge distillation
method [20], where each pixel or point is labeled by the categorical
distribution over classes. We use cross-entropy loss as the training objective
for the 3D model to learn from the soft-label:

$\mathcal{L}^{\text{(pretrain)}}=\frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{C}\left(-\bm{t}_{i}[c]\log\left(\bm{y}^{\text{(img2pts)}}_{i}[c]\right)\right)~{},$
(9)

where $i$ is the index to the $N$ 3D points, and $t_{i}$ is the soft-label
probability of the pixel corresponding to the $i$-th 3D point.

##### Downstream tasks.

After pre-training, we discard the pre-training head
$\operatorname{SegHead}^{\text{(pre)}}$ and directly used the trained encoder
$E$ and decoder $D$ in the downstream tasks.

###  3.3 Semi-Supervised

Semi-supervised learning is helpful to learn from scarce labels, where the
improvement by our pre-training can be additive to it to achieve an even
better result. We use two simple and commonly used semi-supervised techniques
in this work. First, entropy minimization [15] encourages concentration of the
predicted probability distribution on the unlabeled data:

$\mathcal{L}^{\text{(mini-
entropy)}}=\frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{C}\left(-\bm{y}_{i}[c]\log\left(\bm{y}_{i}[c]\right)\right)~{}.$
(10)

Second, we use mean-teacher [35] to guide our model, where the weights of
teacher model $\theta_{t}{}^{\prime}$ is obtained by exponential moving
averaging (EMA) of the weight of student model $\theta_{t}$ across training
step $t$:

$\theta_{t}{}^{\prime}=\alpha\theta_{t-1}{}^{\prime}+(1-\alpha)\theta_{t}~{},$
(11)

where $\alpha$ is a smoothing hyperparameter. We use mean squared error to
encourage consistency between our predictions $\bm{y}$ and teacher predictions
$\bm{y}^{\prime}$

$\mathcal{L}^{\text{(consistency)}}=\frac{1}{N}\sum_{i=1}^{N}\left\|\bm{y}_{i}-\bm{y}_{i}^{\prime}\right\|^{2}~{}.$
(12)

Different from the 2d teacher model we mention in Section 1, the mean-teacher
here is a typical semi-supervised technique.

##  4 Experiments

In this section, we evaluate our pre-training strategy under the data scarcity
scenario. The implementation details are provided in Section 4.1. To
demonstrate the benefits of our approach, we conduct a series of experiments
on scene understanding task in Section 4.2 and the shape analysis tasks in
Section 4.4. We show the results of our pre-training on different 3D models in
Section 4.3 and present the ablation experiments in Section 4.5.

###  4.1 Implement details

##### Pre-training images dataset.

For perspective pre-training, we utilize SUNRGB-D [34], which consists of more
than 10,000 RGB-D indoor images and the corresponding pixel-wise semantic
labels from 37 categories. In addition to the depth maps, SUNRGB-D also
provides the intrinsic which allow us to lift the 3D scenes. We split the
dataset into 5,285 training sets and 5,050 validation sets in the pre-training
procedure. We only used the provided ground-truth semantic maps in ablation
experiments and used the generated pseudo-label by our approach in all other
experiments. For panoramic pre-training, Matterport3D [4] provides various
indoor scenes captured by panoramic images, each of which covers a
omnidirectional field of view. Matterport3D contains 10,800 panoramic views
with corresponding depth maps from 90 building-scale scenes, including 61
scenes for training, 11 for validation, and 18 for testing following official
data splits.

##### Image scene parser.

We employ DPT [31] as a teacher model in our pre-training. The DPT is first
trained on the ADE20K [48] dataset, which has 20k images with ground-truth
semantic segmentation maps covering 150 different classes. In pre-training,
the DPT is fixed and used to generate soft pseudo-labels to transfer its
knowledge to the 3D models.

##### 3D models.

SparseConv [14] is trained by Adam optimizer [23] for scene semantic
segmentation. The models are trained on NVIDIA GTX 1080Ti GPUs for 200 epochs
with batch size 4 and learning rate 0.001. O-CNN [37] is implemented for shape
analysis and indoor scene semantic segmentation. During lifted scenes pre-
training, the projected-points are formed as $512^{3}$ resolution of leaf
octants as the same as indoor scene semantic segmentation. For part
segmentation, we train networks for 600 epochs with batch size 32, leaning
rate 0.025 and weight decay 0.0001. For scene semantic segmentation, we train
models for 500 epochs with batch size 4 and leaning rate 0.05. Both tasks
adopt the SGD optimizer with a momentum 0.9 and use polynomial schedulers
powered by 0.9. The models are trained on several NVIDIA Tesla V100-SXM2 GPUs.
As the setup for part segmentation, the classification model is trained for
300 epochs with batch size 32 and learning rate 0.05. The O-CNNs are adopted
with $32^{3}$, $64^{3}$ and $512^{3}$ resolution of leaf octants for object
classification, shape part segmentation and scene semantic segmentation,
respectively. The point-wise predictions are interpolated by linear for part
segmentation and nearest for scene semantic segmentation.

##### Data augmentation.

We apply random rotation, random scaling, random elastic distortion, random
color contrast, and random color jittering to the input point cloud as data
augmentation.

###  4.2 Data efficient scene semantic segmentation

We validate the effectiveness of our pre-training on ScanNet Data Efficient
Benchmark. ScanNetV2 [9] consists of various indoor scenes formed as 3D point
clouds and corresponding semantic annotations. The data is taken from 707
distinct spaces and covers 20 semantic classes. We follow the official data
scarcity scenarios: i) Limited Annotations (LA) considers only a few labeled
points in each scene, ii) and Limited Reconstructions(LR) considers the a few
number of labeled scenes. We use O-CNN pre-trained by our approach on
Matterport3D dataset for all the results submitted to official ScanNet.

##### Limited annotations.

Following the official configuration in the _3D Semantic label with Limited
Annotations benchmark_ , there are four different scales, including
$\left\\{20,50,100,200\right\\}$ labeled points per training scene. We show
the quantitative comparison on the testing split in Table 1. We achieve state-
of-the-art results on purely supervised fine-tuning setup and semi-supervised
setup. It is worth noting that our results without using unlabeled data
already outperforms the semi-supervised OTOC [26] on 50, 100, and 200 labeled
points per scene, which further highlights the effectiveness of our pre-
training approach.

Table 1: Quantitative comparisons on official ScanNet Limited Annotations(LA) track. For a fair comparisons, the results are separated based on whether the unlabeled data points are used in training. Method | semi | # of labeled points per scene
---|---|---
20 | 50 | 100 | 200
PointContrast [41] | - | 55.0 | 61.4 | 63.6 | 65.3
CSC [21] | - | 53.1 | 61.2 | 64.4 | 66.5
ViewPointBN [27] | - | 54.8 | 62.3 | 65.0 | 66.9
Ours | - | 57.9 | 65.8 | 71.4 | 71.1
OTOC [26] | ✓ | 59.4 | 64.2 | 67.0 | 69.4
Ours | ✓ | 63.9 | 69.5 | 70.4 | 70.9
Table 2: Quantitative comparisons on offical ScanNet Limited Reconstructions(LR) track.  Method | semi | percentage of labeled scene
---|---|---
1% | 5% | 10% | 20%
PointContrast [41] | - | 25.3 | 43.8 | 55.5 | 60.3
CSC [21] | - | 27.0 | 46.0 | 57.5 | 61.2
ViewPointBN [27] | - | 25.6 | 45.2 | 56.6 | 62.5
Ours | - | 26.6 | 46.7 | 61.2 | 64.0
Ours | ✓ | 26.3 | 50.8 | 60.8 | 66.3

##### Limited reconstruction.

Limited Reconstruction (LR) is constructed by limiting the number of scenes.
The official subset is randomly sampled from 1201 scenes, where
$\left\\{1\%,5\%,10\%,20\%\right\\}$ of training data is subsampled
(corresponding to 12, 60, 120, and 240 scenes). We compare our results on _3D
Semantic label with Limited Reconstructions benchmark_ in Table 2. When only
$1\%$ of the training data is available, all methods achieve similar
performance. When $5\%,10\%,20\%$ of the training data is given, we achieve a
superior mIoU compared to previous methods. Additionally, we adopt semi-
supervised learning and examine the capability under the limited
reconstruction scenario. We first trained our model on the given labeled data,
and then we randomly sample a subset of unlabeled scenes to produce pseudo
labels. The pseudo labels is generated by selecting the top-most 20% confident
predictions of each class. We can then combined the labeled, pseudo-labeled,
and unlabeled data points to train our models.

From the comparisons in Table 1 and Table 2, we find our pre-trained models
with simple supervised fine-tuning already outperform existing methods
tailored for 3D data-efficiency. Combining our pre-training with semi-
supervised sometimes can further improve our results. In cases that semi-
supervised does not improve, we still achieve similar performance. As we only
use the simplest semi-supervised techniques, a more tailored one could be more
helpful. In sum, our pre-training provides a good starting point for future
work to develop 3D data-efficient approach.

###  4.3 Pre-training on different 3D models

To verify the effectiveness of our pre-training strategy on different 3D
models, we implement O-CNN [37] and SparseConv [14] for 3D scene semantic
segmentation. SparseConv uses a sparse voxelized input representation and
keeps the same level of sparsity throughout the model. O-CNN builds up the
octree structure of 3D points representation. As the octree depth increases,
the octree-based CNN layers extract the information with higher resolution.
Both models work on occupied point sets only to ensure computational
efficiency, and the networks are constructed by U-Net [33, 14] architecture to
produce point-wise predictions. The overall results are summarized in Table 3.
On both SparseConv and O-CNN, all different variations of our pre-training
approach improved mIoU significantly over training from scratch.

Table 3: mIoU of different 3D segmentation model and different pre-training strategy. The results are evaluated on ScanNet validation set. The oracle pre-training supervision is the ground-truth semantic segmentation maps provided in the pre-training dataset. Based model | Pre-training strategy |  Oracle pre-training supervision | LA (points)
---|---|---|---
20 | 200
SparseConv | trained from scratch | - | 51.6 | 65.4
perspective pre-training | ✓ | 55.4 | 66.0
perspective pre-training | - | 56.2 | 65.7
panoramic pre-training | - | 58.5 | 67.3
O-CNN | trained from scratch | - | 52.9 | 65.0
perspective pre-training | ✓ | 56.9 | 66.2
perspective pre-training | - | 56.2 | 65.3
panoramic pre-training | - | 55.6 | 67.8

###  4.4 Shape analysis under limited data scenario

Our approach discussed above presents a positive outcome on scene-level
semantic segmentation under the limited data scenario. In this section, we
apply our pre-trained model on object-level 3D tasks such as object
classification and shape part segmentation. With the data scarcity scenario
for shape analysis, the implementation details and results are presented in
the following paragraph. For all the experiments in shape analysis, we use
O-CNN pre-trained by our approach on Matterport3D, which can also show the
generalizibility from scene-level to object-level of our pre-training.

##### Object classification

Object classification is the fundamental task of shape analysis. Given the 3D
shape represented as point clouds, the classification model intends to assign
the category of the input objects. With our approaches, the 3D models are
required to extract features by limited annotations.

_Dataset_. ModelNet40 [40] dataset contains 12,311 shapes from 40 object
categories, and is split into 9,843 objects for training and 2,468 objects for
testing. For the limited data scenario, we randomly sample the subset by ratio
$\left\\{1\%,5\%,20\%\right\\}$ in each category from the training set, and
evaluate on the original testing set. For semi-supervised learning, the
unlabeled batches is sampled randomly from the remained data of the training
set.

_Results_. We finetune the 3D models with panoramic images pre-trained for
object classification. The results are shown in Table 4, which is divided by
whether the semi-supervised training is used. Considering the model trained
without unlabeled data, our pre-training strategy can directly improve the
accuracy compared with the model trained from scratch. In the cases that the
models are trained from scratch, the improvements of using the unlabeled data
are limited. Conversely, our pre-training with only supervised fine-tuning
already outperform the results of semi-supervised with random weight
initialization. Besides, the improvements by semi-supervised learning is much
significant than the improvement from random initialization.

Table 4: Accuracy of 3D object classification in ModelNet40 [40] dataset under limited training data. The training data is random sampled by 1%, 5%, 20% and the results are evaluated on the same testing set.  Pre-training strategy | Semi-supervised | 1% | 5% | 20%
---|---|---|---|---
Trained from scratch | - | 60.2 | 79.6 | 86.5
✓ | 61.1 | 80.4 | 87.6
Our pre-training | - | 65.5 | 80.6 | 87.5
✓ | 69.0 | 82.9 | 89.4
Table 5: mIoU of shape part segmentation in ShapeNet [40] dataset under sampled training data. We use category mIoU across all categories and instance mIoU across all shape instance as the evaluation metrics. The results are evaluated on ShapeNet testing set which split by [43].  Pre-training strategy | Category mIoU | Instance mIoU
---|---|---
1% | 5% | 20% | 1% | 5% | 20%
Trained from scratch | 63.3 | 67.5 | 73.9 | 67.9 | 72.2 | 76.7
Our pretraining | 64.1 | 74.1 | 76.3 | 68.5 | 76.8 | 78.9

##### Shape part segmentation

Shape part segmentation is more complicated than classification for shape
analysis. Part segmentation is expected to generate the dense prediction and
assign the part category to each point in objects.

_Dataset_. Yi  _et al_. [43] annotates a subset of ShapeNet [40] 3D models
with semantic part labels. The annotated subset of ShapeNet contains 2 to 6
parts per category, and 50 distinct parts in total among 16 shape categories.
For training on the limited scenario, we build the limited subset by random
sampling as ratio $\left\\{1\%,5\%,20\%\right\\}$ from each shape category,
and then evaluate on the original validation set. According to the setup of
pre-processing in [37], the input points are condensed by triangle faces and
built in octree structure.

_Results_ We finetune the segmentation models with panoramic images pre-
training in each category separately, and evaluate the performance of part
segmentation by mean IoU across all categories and mIoU across all object
instances. The results are shown in Table 5. As the data split in [47, 17, 38]
are not provided, we only compare the results on our own split. With the
knowledge distilled from off-the-shelf image scene parser, the pre-trained
model performs better category mIoU and instance mIoU across all limited
scales (1%, 5% and 20%).

###  4.5 Ablation study

##### Image modality

To observe the effect of the distillation via different image modalities, we
pre-train O-CNN [37] on perspective images and panoramic images, and the
supervised fine-tune the O-CNN on ScanNet with limited annotations. The
results are demonstrated in Table 3. For all the variations of our pre-
training, we achieve significant better results than training from scratch.
Note that the panoramic images pre-training rises the performance by 2.7% and
2.8% mIoU compared with baseline on 20, 200 labeled points of scene(55.6% vs
52.9% and 67.8% vs 65.0%). The results imply that models can learn non-trivial
and informative representation from the strong and well-trained image scene
parser via RGB-D images. The learned representation boosts the capability of
scene understanding under the limited annotations.

This work can not conclude whether perspective pre-training or panoramic pre-
training is better, as the number of their training data is different. The
panoramic data used in this work leads us to better results, so we use
panoramic pre-training in all the other experiments.

##### Pre-training by 2D annotations and pseudo labels

In our pre-training strategy, we supervise 3D models by the pseudo-labels
introduced in Section 3.2. To examine the difference between the information
distilled from pseudo labels and ground-truth labels, we conduct our pre-
training strategy on perspective images with ground-truth semantic labels and
generated pseudo-labels. The ground-truth semantic labels are provided by
SUNRGB-D [34], while the pseudo labels are predicted by a strong and well-
trained image scene parser. With the 3D model trained on limited annotations
scenario, the results are presented in Table 3. The perspective images attain
higher performance than the model trained from scratch regardless of whether
the ground-truth labels are provided. The results imply that the pseudo labels
can also improve the knowledge distillation during pre-training. Additionally,
with provided annotations of images, our pre-training strategy achieves better
mIoU than the baseline by 4% mIoU on 20 points and 1% mIoU on 200 points.
Overall speaking, the less constrained pseudo-labels pre-training can achieve
comparable results to the ground-truth labels pre-training, which suggests
that ground-truth semantic maps are not necessary in our pre-training
approach.

Table 6: Ablation studies of our pre-training with and without semi-supervised learning. We follow the limited training data from official. The results are presented as mIoU and compared on ScanNet validation set.  Method | semi | # of labeled points per scene
---|---|---
20 | 50 | 100 | 200
Trained from scratch | - | 54.2 | 61.1 | 64.1 | 65.9
Our pre-training | - | 55.9 | 63.4 | 70.2 | 69.4
Trained from scratch | ✓ | 56.6 | 62.5 | 65.3 | 66.0
Our pre-training | ✓ | 61.1 | 66.6 | 69.6 | 69.7
Table 7: mIoU of limited scene variety on ScanNet validation set. Model | Pre-training strategy | LR(%)
---|---|---
1 | 5 | 10 | 20
O-CNN | Trained from scratch | 18.7 | 39.7 | 52.4 | 59.6
Our pre-training | 26.3 | 44.9 | 56.5 | 62.7
Our pre-training + Semi-supervised | 27.5 | 49.4 | 59.2 | 64.8

##### Combining pre-training with semi-supervised learning

We study the effectiveness of our pre-training when combining with semi-
supervised learning to simultaneously learn from labeled and unlabeled data.
We run semi-supervised learning with random initialized model weights and our
pre-trained model weights, and then evaluate the performance on ScanNetv2
validation sets. The approach is examined on both limited annotation(LA) and
limit reconstruction(LR).

For LA, the results are summarized in Table 6. In “trained from scratch”
column, the semi-supervised learning slightly improves mIoU when unlabeled
data is used during training. In contrast, our pre-training strategy increases
the performance significantly when combining with the semi-supervised
learning, particularly in the 20, 50 of labeled points per scene. Using
unlabeled data with our pre-training strategy outperforms the models trained
from scratch by 4.5% and 4.1%.(61.1% vs 56.6% and 66.6% vs 62.5%).

For LR, we follow Section 4.2 for semi-supervised training. As a result, Table
7 shows that our pre-training strategy achieves greater mIoU by 27.5%, 49.4%,
59.2% and 64.8% on $\left\\{1\%,5\%,10\%,20\%\right\\}$. Consequently,
unlabeled data enhances the ability of scene understanding on both limited
annotations and limited reconstruction.

##  5 Conclusion

This work presents a new 3D deep models pre-training strategy. We use RGB-D
images as the bridge to transfer the knowledge from a strong and well-trained
2D scene parsing network to 3D models. Our pre-training strategy is model
agnostic, and we show its effectiveness on two popular 3D models
architectures. On the official ScanNet data-efficient track, we establish new
state-of-the-art results. Besides, we also show that the improvement by our
pre-training is additive to other label-efficient techniques. We hope our pre-
trained weights can serve as a stepping stone for future 3D approaches and
encourage more exploration on how to make good use of 2D resources in 3D
tasks.

####  5.0.1 Acknowledgements

This work is supported in part by Ministry of Science and Technology of Taiwan
(MOST 110-2634-F-002-051). We would like to thank National Center for High-
performance Computing (NCHC) for computational and storage resource.

Data Efficient 3D Learner via Knowledge Transferred from 2D Model

In this supplementary material, we show the per-category breakdown comparisons
in Section A, and then the qualitative results are showed in Section E. In
Section B, we compare with DepthContrast [45] on the efficient ScanNet setup.
In Section C, we provide more implement details. Finally, we discuss the
effectiveness of image scene parser in Section D.

##  A Quantitative results of per-category mIoU

We use mean IoU to validate the effectiveness of our approach on ScanNet [9]
Data Efficient Benchmark in our main paper. Below, we expand the results
presented in the main paper with per-category breakdown.

##### Limited annotations.

Following the official configuration in the _3D Semantic label with Limited
Annotations benchmark_ , we demonstrate the state-of-the-art results of Table
1 in the main paper. Here we show the detailed performance of each category on
20, 50, 100 and 200 labeled points per scene in Table A, Table B, Table C, and
Table D respectively.

Table A: 20-points LA

points | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.550 | 0.735 | 0.676 | 0.601 | 0.475 | 0.794 | 0.288 | 0.621 | 0.378 | 0.430 | 0.940 | 0.303 | 0.089 | 0.379 | 0.580 | 0.531 | 0.689 | 0.422 | 0.852 | 0.758 | 0.468
CSC | 0.531 | 0.659 | 0.638 | 0.578 | 0.417 | 0.775 | 0.254 | 0.537 | 0.396 | 0.439 | 0.939 | 0.284 | 0.083 | 0.414 | 0.599 | 0.488 | 0.698 | 0.444 | 0.785 | 0.747 | 0.440
ViewPointBN | 0.548 | 0.747 | 0.574 | 0.631 | 0.456 | 0.762 | 0.355 | 0.639 | 0.412 | 0.404 | 0.940 | 0.335 | 0.107 | 0.277 | 0.645 | 0.495 | 0.666 | 0.517 | 0.818 | 0.740 | 0.431
OTOC | 0.594 | 0.756 | 0.722 | 0.494 | 0.546 | 0.795 | 0.371 | 0.725 | 0.559 | 0.488 | 0.957 | 0.367 | 0.261 | 0.547 | 0.575 | 0.225 | 0.671 | 0.543 | 0.904 | 0.826 | 0.557
Ours | 0.639 | 0.839 | 0.723 | 0.681 | 0.629 | 0.839 | 0.424 | 0.728 | 0.538 | 0.526 | 0.945 | 0.427 | 0.12 | 0.511 | 0.643 | 0.547 | 0.781 | 0.566 | 0.905 | 0.809 | 0.607

Table B: 50-points LA

points | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.614 | 0.844 | 0.731 | 0.681 | 0.590 | 0.791 | 0.348 | 0.689 | 0.503 | 0.502 | 0.942 | 0.361 | 0.154 | 0.484 | 0.624 | 0.591 | 0.708 | 0.524 | 0.874 | 0.793 | 0.536
CSC | 0.612 | 0.747 | 0.731 | 0.679 | 0.603 | 0.815 | 0.400 | 0.648 | 0.453 | 0.481 | 0.944 | 0.421 | 0.173 | 0.504 | 0.623 | 0.588 | 0.690 | 0.545 | 0.877 | 0.778 | 0.541
ViewPointBN | 0.623 | 0.812 | 0.743 | 0.654 | 0.579 | 0.800 | 0.462 | 0.713 | 0.533 | 0.516 | 0.944 | 0.434 | 0.215 | 0.437 | 0.521 | 0.601 | 0.720 | 0.563 | 0.884 | 0.800 | 0.534
OTOC | 0.642 | 0.725 | 0.735 | 0.717 | 0.635 | 0.829 | 0.457 | 0.639 | 0.421 | 0.552 | 0.967 | 0.460 | 0.240 | 0.558 | 0.788 | 0.621 | 0.720 | 0.477 | 0.915 | 0.842 | 0.539
Ours | 0.695 | 0.897 | 0.784 | 0.728 | 0.697 | 0.846 | 0.441 | 0.77 | 0.615 | 0.585 | 0.951 | 0.504 | 0.232 | 0.672 | 0.76 | 0.655 | 0.772 | 0.599 | 0.877 | 0.834 | 0.678

Table C: 100-points LA

points | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.636 | 0.694 | 0.738 | 0.731 | 0.653 | 0.817 | 0.467 | 0.651 | 0.517 | 0.522 | 0.946 | 0.479 | 0.198 | 0.575 | 0.526 | 0.649 | 0.747 | 0.569 | 0.845 | 0.803 | 0.600
CSC | 0.644 | 0.761 | 0.707 | 0.703 | 0.642 | 0.813 | 0.436 | 0.659 | 0.502 | 0.516 | 0.945 | 0.487 | 0.238 | 0.538 | 0.678 | 0.659 | 0.739 | 0.568 | 0.915 | 0.811 | 0.566
ViewPointBN | 0.650 | 0.778 | 0.731 | 0.688 | 0.617 | 0.812 | 0.446 | 0.739 | 0.618 | 0.540 | 0.945 | 0.415 | 0.204 | 0.623 | 0.676 | 0.594 | 0.744 | 0.576 | 0.868 | 0.811 | 0.582
OTOC | 0.670 | 0.734 | 0.815 | 0.661 | 0.644 | 0.841 | 0.509 | 0.741 | 0.479 | 0.548 | 0.968 | 0.461 | 0.251 | 0.664 | 0.754 | 0.656 | 0.744 | 0.541 | 0.917 | 0.844 | 0.625
Ours | 0.704 | 0.774 | 0.766 | 0.764 | 0.687 | 0.832 | 0.413 | 0.79 | 0.639 | 0.599 | 0.952 | 0.478 | 0.222 | 0.746 | 0.859 | 0.678 | 0.806 | 0.607 | 0.915 | 0.847 | 0.703

Table D: 200-points LA

points | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.653 | 0.717 | 0.775 | 0.754 | 0.626 | 0.804 | 0.391 | 0.689 | 0.485 | 0.572 | 0.945 | 0.448 | 0.232 | 0.603 | 0.813 | 0.591 | 0.775 | 0.537 | 0.885 | 0.816 | 0.608
CSC | 0.665 | 0.857 | 0.756 | 0.763 | 0.647 | 0.852 | 0.432 | 0.684 | 0.543 | 0.514 | 0.948 | 0.469 | 0.179 | 0.599 | 0.702 | 0.620 | 0.789 | 0.614 | 0.911 | 0.815 | 0.607
ViewPointBN | 0.669 | 0.847 | 0.732 | 0.724 | 0.613 | 0.827 | 0.443 | 0.742 | 0.562 | 0.551 | 0.947 | 0.441 | 0.218 | 0.650 | 0.753 | 0.621 | 0.765 | 0.601 | 0.905 | 0.814 | 0.618
OTOC | 0.694 | 0.760 | 0.815 | 0.706 | 0.684 | 0.840 | 0.492 | 0.701 | 0.557 | 0.596 | 0.972 | 0.497 | 0.281 | 0.709 | 0.757 | 0.689 | 0.789 | 0.600 | 0.907 | 0.864 | 0.6
Ours | 0.709 | 0.877 | 0.772 | 0.744 | 0.694 | 0.836 | 0.453 | 0.787 | 0.623 | 0.598 | 0.953 | 0.49 | 0.216 | 0.682 | 0.879 | 0.727 | 0.802 | 0.604 | 0.922 | 0.845 | 0.676

Table E: 1% LR

rate | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.253 | 0.000 | 0.412 | 0.347 | 0.137 | 0.564 | 0.140 | 0.361 | 0.187 | 0.249 | 0.914 | 0.092 | 0.055 | 0.102 | 0.000 | 0.048 | 0.392 | 0.302 | 0.000 | 0.697 | 0.056
CSC | 0.270 | 0.000 | 0.528 | 0.331 | 0.139 | 0.535 | 0.118 | 0.326 | 0.222 | 0.292 | 0.921 | 0.089 | 0.163 | 0.129 | 0.000 | 0.131 | 0.463 | 0.278 | 0.000 | 0.699 | 0.033
ViewPointBN | 0.256 | 0.000 | 0.479 | 0.377 | 0.204 | 0.551 | 0.205 | 0.219 | 0.235 | 0.224 | 0.903 | 0.092 | 0.088 | 0.122 | 0.000 | 0.003 | 0.354 | 0.354 | 0.000 | 0.676 | 0.034
Ours | 0.263 | 0 | 0.547 | 0.235 | 0.184 | 0.566 | 0.165 | 0.249 | 0.196 | 0.309 | 0.938 | 0.07 | 0.186 | 0.069 | 0 | 0 | 0.368 | 0.356 | 0 | 0.698 | 0.118

Table F: 5% LR

rate | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.438 | 0.517 | 0.659 | 0.251 | 0.332 | 0.783 | 0.244 | 0.408 | 0.411 | 0.409 | 0.935 | 0.206 | 0.119 | 0.200 | 0.048 | 0.355 | 0.682 | 0.414 | 0.647 | 0.743 | 0.391
CSC | 0.460 | 0.472 | 0.731 | 0.465 | 0.398 | 0.817 | 0.292 | 0.442 | 0.311 | 0.387 | 0.939 | 0.218 | 0.181 | 0.302 | 0.076 | 0.449 | 0.743 | 0.430 | 0.444 | 0.737 | 0.368
ViewPointBN | 0.452 | 0.587 | 0.569 | 0.172 | 0.391 | 0.769 | 0.290 | 0.512 | 0.501 | 0.373 | 0.935 | 0.251 | 0.173 | 0.201 | 0.003 | 0.352 | 0.619 | 0.454 | 0.783 | 0.719 | 0.390
Ours | 0.508 | 0.824 | 0.53 | 0.314 | 0.479 | 0.746 | 0.334 | 0.49 | 0.508 | 0.477 | 0.95 | 0.269 | 0.221 | 0.324 | 0.029 | 0.421 | 0.626 | 0.49 | 0.727 | 0.782 | 0.62

Table G: 10% LR

rate | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.555 | 0.711 | 0.668 | 0.622 | 0.425 | 0.830 | 0.433 | 0.552 | 0.273 | 0.440 | 0.938 | 0.287 | 0.096 | 0.470 | 0.576 | 0.612 | 0.687 | 0.438 | 0.781 | 0.785 | 0.474
CSC | 0.575 | 0.671 | 0.740 | 0.727 | 0.445 | 0.847 | 0.380 | 0.602 | 0.512 | 0.447 | 0.942 | 0.291 | 0.184 | 0.353 | 0.468 | 0.508 | 0.745 | 0.602 | 0.855 | 0.765 | 0.420
ViewPointBN | 0.566 | 0.780 | 0.659 | 0.677 | 0.484 | 0.799 | 0.419 | 0.636 | 0.480 | 0.432 | 0.940 | 0.238 | 0.124 | 0.396 | 0.609 | 0.432 | 0.735 | 0.527 | 0.787 | 0.752 | 0.423
Ours | 0.608 | 0.853 | 0.689 | 0.593 | 0.483 | 0.83 | 0.466 | 0.652 | 0.528 | 0.482 | 0.954 | 0.288 | 0.25 | 0.448 | 0.595 | 0.532 | 0.748 | 0.503 | 0.822 | 0.806 | 0.647

Table H: 20% LR

rate | avg. | bathtub | bed | book. | cabinet | chair | counter | curtain | desk | door | floor | otherf. | picture | frige | shower | sink | sofa | table | toilet | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointContrast | 0.603 | 0.740 | 0.700 | 0.700 | 0.546 | 0.843 | 0.419 | 0.592 | 0.462 | 0.513 | 0.946 | 0.374 | 0.104 | 0.530 | 0.687 | 0.571 | 0.694 | 0.519 | 0.850 | 0.781 | 0.484
CSC | 0.612 | 0.739 | 0.794 | 0.687 | 0.564 | 0.850 | 0.347 | 0.590 | 0.587 | 0.521 | 0.945 | 0.358 | 0.140 | 0.522 | 0.496 | 0.627 | 0.725 | 0.598 | 0.850 | 0.792 | 0.508
ViewPointBN | 0.625 | 0.873 | 0.727 | 0.709 | 0.535 | 0.820 | 0.402 | 0.643 | 0.540 | 0.501 | 0.946 | 0.352 | 0.181 | 0.535 | 0.594 | 0.596 | 0.685 | 0.543 | 0.927 | 0.792 | 0.592
Ours | 0.663 | 0.851 | 0.77 | 0.76 | 0.615 | 0.83 | 0.439 | 0.67 | 0.546 | 0.587 | 0.955 | 0.406 | 0.177 | 0.627 | 0.758 | 0.606 | 0.74 | 0.549 | 0.888 | 0.84 | 0.652

##### Limited reconstruction

We expand the Table 2 in our main paper and show the per-category results on
_3D Semantic label with Limited Reconstructions benchmark_ in Table E, Table
F, Table G and Table H on 1%, 5%, 10% and 20% percentage of labeled scene.
When only 1% of the training data is available, all methods achieve similar
performance. When 5%,10%,20% of the training data is given, we outperform the
previous state-of-the-art.

##  B Comparison with DepthContrast [45]

We use the released code and the pre-trained weight by DepthContrast to train
on the efficient ScanNet setup. We note that DepthContrast uses Minkowski
while we use O-CNN as the base model, so we also run the Minkowski without
pre-training for a fair comparison. We present the results of the LR setup in
Table I.

Table I: Comparison our pre-training with DepthContrast [45]. The results are presented as mIoU and compared on ScanNet validation set. base model | pre-training | LR 1% | LR 5% | LR 10% | LR 20%
---|---|---|---|---|---
Minkowski | none | 24.8 | 40.5 | 53.5 | 59.8
Minkowski | [45] | 27.7 | 43.0 | 56.8 | 61.8
O-CNN | none | 18.7 | 39.7 | 52.4 | 59.6
O-CNN | ours | 26.3 | 44.9 | 56.5 | 62.7

Without any pre-training, Minkowski outperforms O-CNN. However, O-CNN with our
pre-training can overtake Minkowski with DepthContrast pre-training in LR 5%
and LR 20%, and achieve similar mIoU in LR 10%. The results show that the
proposed pre-training can boost better than the previous work in the LR setup.

##  C Additional implement details

In semi-supervised learning, the loss of unlabeled data are weighted sum by
different coefficients across downstream tasks. For object classification, the
weights of mini-entropy loss and consistency loss referenced in the main paper
are 0.01 and 10. For scene semantic segmentation, the weights of mini-entropy
loss and consistency loss are 0.25 and 10. Among Mean-Teacher training, the
weights of the teacher model were updated each training step by EMA with
smoothing hyperparameter $\alpha$ = 0.999, and the coefficient of consistency
cost will ramp up during the first 30 epochs by a sigmoid-shaped function
$e^{-5(1-x)^{2}}$, where $x\in\left[0,1\right]$.

##  D Statistic of the pre-training pseudo-labels

We adopt DPT [31], which has been trained on the ADE20k [48] dataset, to
predict the categorical probability distribution over 150 classes for all
pixels. We sum the predicted probability of pixels in all images in
Matterport3D [4] dataset for each class. The histogram of the summed pseudo-
label probability are presented in Fig. A. The horizontal axis represents the
150 different classes in ADE20k and the vertical axis shows the log of summed
probabilities, then the yellow bars are the overlap classes for ADE20k and
ScanNet. As shown in Fig. A, the pseudo-labels for pre-training provide
extensive knowledge beyond the target ScanNet dataset and is non-trivial to
learn.

Figure A: The histogram for the summed probability of the pre-training pseudo-
label. The x-axis are the 150 different classes of a image scene parse, which
is DPT in this work, well-trained on the ADE20k dataset.

##  E Qualitative results of data efficient on Scannet

Under limited annotations scenario, Figs. C and D are the results of O-CNN
[37] models supervised by limited ScanNet annotation. Then Fig. E shows the
results of O-CNN models supervised under limited ScanNet reconstruction. We
show the visual comparisons between the results of training from scratch and
training with our pre-training. We use red frames to highlight the difference,
where our pre-training lead to more complete shapes and consistent prediction.
The visual results echo our quantitative improvement, which shows that our
pre-trained models improve the generalizability for scene understanding by the
2D transferred knowledge.

Figure B: Corresponding colors to the categories in ScanNet. Figure C:
Qualitative results for ScanNet LA

| 20 | 50 | 100 | 200
---|---|---|---|---
|  |  |  |
Input RGB | Trained from scratch
|  |  |  |
Ground-truth | Pre-training

Figure D: Qualitative results for ScanNet LA

| 20 | 50 | 100 | 200
---|---|---|---|---
|  |  |  |
Input RGB | Trained from scratch
|  |  |  |
Ground-truth | Pre-training
|  |  |  |
Input RGB | Trained from scratch
|  |  |  |
Ground-truth | Pre-training
|  |  |  |
Input RGB | Trained from scratch
|  |  |  |
Ground-truth | Pre-training

Figure E: Qualitative results for ScanNet LR

| 1% | 5% | 10% | 20%
---|---|---|---|---
|  |  |  |
Input RGB | Trained from scratch
|  |  |  |
Ground-truth | Pre-training
|  |  |  |
Input RGB | Trained from scratch
|  |  |  |
Ground-truth | Pre-training
|  |  |  |
Input RGB | Trained from scratch
|  |  |  |
Ground-truth | Pre-training

## References

  * [1] Boulch, A.: Convpoint: Continuous convolutions for point cloud processing. Comput. Graph. pp. 24–34 (2020)
  * [2] Caesar, H., Uijlings, J.R.R., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: CVPR. pp. 1209–1218 (2018)
  * [3] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: NeurIPS (2020)
  * [4] Chang, A.X., Dai, A., Funkhouser, T.A., Halber, M., Nießner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3d: Learning from RGB-D data in indoor environments. In: 3DV. pp. 667–676 (2017)
  * [5] Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. pp. 834–848 (2018)
  * [6] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. In: ICML. pp. 1597–1607 (2020)
  * [7] Chen, X., He, K.: Exploring simple siamese representation learning. In: CVPR. pp. 15750–15758 (2021)
  * [8] Choy, C.B., Gwak, J., Savarese, S.: 4d spatio-temporal convnets: Minkowski convolutional neural networks. In: CVPR. pp. 3075–3084 (2019)
  * [9] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T.A., Nießner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: CVPR. pp. 2432–2443 (2017)
  * [10] Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255 (2009)
  * [11] et al., Z.: Barlow twins: Self-supervised learning via redundancy reduction. In: ICML (2021)
  * [12] Genova, K., Yin, X., Kundu, A., Pantofaru, C., Cole, F., Sud, A., Brewington, B., Shucker, B., Funkhouser, T.A.: Learning 3d semantic segmentation with only 2d image supervision. In: 3DV (2021)
  * [13] Graham, B.: Sparse 3d convolutional neural networks. In: Xie, X., Jones, M.W., Tam, G.K.L. (eds.) BMVC. pp. 150.1–150.9 (2015)
  * [14] Graham, B., Engelcke, M., van der Maaten, L.: 3d semantic segmentation with submanifold sparse convolutional networks. In: CVPR. pp. 9224–9232 (2018)
  * [15] Grandvalet, Y., Bengio, Y.: Semi-supervised learning by entropy minimization. In: NeurIPS. pp. 281–296 (2005)
  * [16] Grill, J., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.Á., Guo, Z., Azar, M.G., Piot, B., Kavukcuoglu, K., Munos, R., Valko, M.: Bootstrap your own latent - A new approach to self-supervised learning. In: NeurIPS (2020)
  * [17] Hassani, K., Haley, M.: Unsupervised multi-task feature learning on point clouds. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. pp. 8159–8170. IEEE (2019)
  * [18] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.B.: Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 (2021)
  * [19] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.B.: Momentum contrast for unsupervised visual representation learning. In: CVPR. pp. 9726–9735 (2020)
  * [20] Hinton, G.E., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [21] Hou, J., Graham, B., Nießner, M., Xie, S.: Exploring data-efficient 3d scene understanding with contrastive scene contexts. In: CVPR. pp. 15587–15597 (2021)
  * [22] Hsiao, C., Sun, C., Chen, H., Sun, M.: Specialize and fuse: Pyramidal output representation for semantic segmentation. In: ICCV (2021)
  * [23] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)
  * [24] Li, Y., Bu, R., Sun, M., Wu, W., Di, X., Chen, B.: Pointcnn: Convolution on x-transformed points. In: Bengio, S., Wallach, H.M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) NeurIPS. pp. 828–838 (2018)
  * [25] Liu, Y.C., Huang, Y.K., Chiang, H.Y., Su, H.T., Liu, Z.Y., Chen, C.T., Tseng, C.Y., Hsu, W.H.: Learning from 2d: Contrastive pixel-to-point knowledge transfer for 3d pretraining. arXiv preprint arXiv:2104.04687 (2021)
  * [26] Liu, Z., Qi, X., Fu, C.: One thing one click: A self-training approach for weakly supervised 3d semantic segmentation. In: CVPR. pp. 1726–1736 (2021)
  * [27] Luo, L., Tian, B., Zhao, H., Zhou, G.: Pointly-supervised 3d scene parsing with viewpoint bottleneck. arXiv preprint arXiv:2109.08553 (2021)
  * [28] Maturana, D., Scherer, S.A.: Voxnet: A 3d convolutional neural network for real-time object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2015, Hamburg, Germany, September 28 - October 2, 2015. pp. 922–928 (2015)
  * [29] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: CVPR. pp. 77–85 (2017)
  * [30] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: NeurIPS. pp. 5099–5108 (2017)
  * [31] Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: ICCV. pp. 12179–12188 (2021)
  * [32] Riegler, G., Ulusoy, A.O., Geiger, A.: Octnet: Learning deep 3d representations at high resolutions. In: CVPR. pp. 6620–6629 (2017)
  * [33] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241 (2015)
  * [34] Song, S., Lichtenberg, S.P., Xiao, J.: SUN RGB-D: A RGB-D scene understanding benchmark suite. In: CVPR. pp. 567–576 (2015)
  * [35] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: ICLR (2017)
  * [36] Thomas, H., Qi, C.R., Deschaud, J., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: Flexible and deformable convolution for point clouds. In: ICCV. pp. 6410–6419 (2019)
  * [37] Wang, P., Liu, Y., Guo, Y., Sun, C., Tong, X.: O-CNN: octree-based convolutional neural networks for 3d shape analysis. ACM Trans. Graph. pp. 72:1–72:11 (2017)
  * [38] Wang, P., Yang, Y., Zou, Q., Wu, Z., Liu, Y., Tong, X.: Unsupervised 3d learning for shape analysis via multiresolution instance discrimination. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. pp. 2773–2781. AAAI Press (2021)
  * [39] Wu, W., Qi, Z., Li, F.: Pointconv: Deep convolutional networks on 3d point clouds. In: CVPR. pp. 9621–9630 (2019)
  * [40] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: CVPR. pp. 1912–1920 (2015)
  * [41] Xie, S., Gu, J., Guo, D., Qi, C.R., Guibas, L.J., Litany, O.: Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In: ECCV. pp. 574–591 (2020)
  * [42] Yan, X., Zheng, C., Li, Z., Wang, S., Cui, S.: Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In: CVPR. pp. 5588–5597 (2020)
  * [43] Yi, L., Kim, V.G., Ceylan, D., Shen, I., Yan, M., Su, H., Lu, C., Huang, Q., Sheffer, A., Guibas, L.J.: A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph. pp. 210:1–210:12 (2016)
  * [44] Yuan, Y., Chen, X., Wang, J.: Object-contextual representations for semantic segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) ECCV. pp. 173–190 (2020)
  * [45] Zhang, Z., Girdhar, R., Joulin, A., Misra, I.: Self-supervised pretraining of 3d features on any point-cloud. In: ICCV (2021)
  * [46] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR. pp. 6230–6239 (2017)
  * [47] Zhao, Y., Birdal, T., Deng, H., Tombari, F.: 3d point capsule networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 1009–1018. Computer Vision Foundation / IEEE (2019)
  * [48] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ADE20K dataset. In: CVPR. pp. 5122–5130 (2017)
  * [49] Zhu, Z., Xu, M., Bai, S., Huang, T., Bai, X.: Asymmetric non-local neural networks for semantic segmentation. In: ICCV. pp. 593–602 (2019)