Lanxiao<EMAIL_ADDRESS>Michael<EMAIL_ADDRESS>Institute of Industrial Information Technology (IIIT)
Karlsruhe Institute of Technology
Karlsruhe, Germany Lanxiao Li, Michael Heizmann

# Randomized 3D Scene Generation for Generalizable Self-supervised Pre-
training

###### Abstract

Capturing and labeling real-world 3D data is laborious and time-consuming,
which makes it costly to train strong 3D models. To address this issue,
previous works generate randomized 3D scenes and pre-train models on generated
data. Although the pre-trained models gain promising performance boosts,
previous works have two major shortcomings. First, they focus on only one
downstream task (_i.e_ , object detection). Second, a fair comparison of
generated data is still lacking. In this work, we systematically compare data
generation methods using a unified setup. To clarify the generalization of the
pre-trained models, we evaluate their performance in multiple tasks (_e.g_ ,
object detection and semantic segmentation) and with different pre-training
methods (_e.g_ , masked autoencoder and contrastive learning). Moreover, we
propose a new method to generate 3D scenes with spherical harmonics. It
surpasses the previous formula-driven method with a clear margin and achieves
on-par results with methods using real-world scans and CAD models.

##  1 Introduction

Deep neural networks are data-hungry, whereas capturing and labeling data
requires much time and human effort. This problem is especially concerning in
3D computer vision, as 3D data and labels are more scarce and expensive. To
train strong 3D neural networks with lower costs, many works apply synthetic
data in pre-training and fine-tune the models on real-world data. One possible
approach for generating synthetic data is simulation [Deschaud et
al.(2021)Deschaud, Duque, Richa, Velasco-Forero, Marcotegui, and Goulette,
Griffiths and Boehm(2019), Hurl et al.(2019)Hurl, Czarnecki, and Waslander, Wu
et al.(2018)Wu, Wan, Yue, and Keutzer]. Although realistic scenes can be
simulated, developing the simulation environment, crafting the source
materials, and designing scene layouts still require a lot of effort.
Recently, randomized 3D scene generation [Rao et al.(2021)Rao, Liu, Wei, Lu,
Hsieh, and Zhou, Yamada et al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata]
has shown promising results. This approach generates 3D scenes by randomly
placing ‘objects’, which can be either CAD models [Rao et al.(2021)Rao, Liu,
Wei, Lu, Hsieh, and Zhou] or formula-driven shapes [Yamada et al.(2022)Yamada,
Kataoka, Chiba, Domae, and Ogata], based on pre-defined rules. Randomized 3D
scene generation requires neither real-world data nor manual annotation. Also,
it saves effort in designing the scene layouts. However, previous works [Rao
et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou, Yamada et al.(2022)Yamada,
Kataoka, Chiba, Domae, and Ogata] have two major issues. First, their pre-
training methods are task-specific. Specifically, they are only designed for
and evaluated on object detection, which limits their application to other
tasks. Moreover, since they use different pretext tasks for pre-training, the
contribution of the generated data is not clear. A fair comparison of
different data-generating methods is still missing.

For a better understanding and application of randomized 3D scene generation
in 3D computer vision, it’s important to compare previous methods under a fair
setup. Also, to pre-train models which can generalize on different downstream
tasks, it’s necessary to apply a general pre-training approach instead of
task-specific ones. In this work, we choose masked autoencoder (MAE) [He et
al.(2022)He, Chen, Xie, Li, Dollár, and Girshick] and contrastive learning [He
et al.(2020)He, Fan, Wu, Xie, and Girshick, Chen et al.(2020)Chen, Fan,
Girshick, and He] to pre-train models, as they generalize well on multiple
tasks and show an impressive performance in previous research. Our
experimental results show that randomly generated 3D scenes bring on-par
improvement as real-world data. Also, the pre-trained models generalize well
on different downstream tasks, _e.g_ , object detection and semantic
segmentation.

Figure 1: Concept of randomized 3D scene generation. Key components: an object
set and generation rules. RNG: random number generator.

Also, using formula-driven shapes, _e.g_ , fractal point clouds [Yamada et
al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata], instead of CAD models for
scene generation, is more promising since it doesn’t rely on additional data.
However, we empirically find that fractal point clouds show sub-optimal
results in pre-training. Since real-world 3D data are usually captured with
depth sensors (_e.g_ , laser scanners or RGB-D cameras), which cannot measure
the internal structure of objects, real-world data only contain sample points
on object surfaces. Our intuition is that the objects used for scene
generation should contain sufficient surfaces so that the pre-trained models
can be transferred to downstream tasks with real-world data. Fractal point
clouds, on the contrary, don’t contain continuous surfaces (see Fig. 2). We
hypothesize that this domain gap makes the pre-training less effective. To
solve this problem, we propose to use spherical harmonics, which are formula-
driven with natural surfaces, as objects for randomized 3D scene generation.
Experimental results show that spherical harmonics outperform fractal point
clouds with a clear margin and are competitive compared to CAD models and even
real-world data in pre-training.

The contribution of this work is many-fold: (1) our work, for the first time,
explores generalizable pre-training using randomized scene generation and
systematically compares the generated data in multiple tasks. (2) our
experiments provide a deeper understanding of randomized 3D scene generation,
_e.g_ , the impact of object sets and view angles. (3) we propose to generate
scenes with formula-driven spherical harmonics and achieve competitive results
in 3D object detection and semantic segmentation without using real-world data
or CAD models in the pre-training stage.

##  2 Related Works

Self-supervised Pre-training for 3D Vision. A lot of methods utilize the
invariance of 3D features to establish correspondence between different views
and apply contrastive learning [Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and
Litany, Zhang et al.(2021)Zhang, Girdhar, Joulin, and Misra, Hou et
al.(2021)Hou, Graham, Nießner, and Xie, Xu et al.(2022)Xu, Yang, Zhai, Wu,
Yue, Zhan, Vajda, Keutzer, and Tomizuka, Li and Heizmann(2022)]. Also, some
works transfer knowledge from other domains, _e.g_ , color images [Qian et
al.(2022)Qian, Zhang, Hamdi, and Ghanem] and language [Zhang et
al.(2022b)Zhang, Guo, Zhang, Li, Miao, Cui, Qiao, Gao, and Li]. Moreover, some
works [Yu et al.(2022)Yu, Tang, Rao, Huang, Zhou, and Lu, Fu et al.(2022)Fu,
Gao, Liu, Zhang, Qiao, and Wang, Wang et al.(2021)Wang, Liu, Yue, Lasenby, and
Kusner] reconstruct partially visual point clouds in the pretext task. Recent
works [Pang et al.(2022)Pang, Wang, Tay, Liu, Tian, and Yuan, Zhang et
al.(2022a)Zhang, Guo, Gao, Fang, Zhao, Wang, Qiao, and Li, Liu et
al.(2022)Liu, Cai, and Lee, Li and Heizmann(2023)] apply the successful masked
autoencoder [He et al.(2022)He, Chen, Xie, Li, Dollár, and Girshick] on point
clouds.

Simulation-based Scene Generation. Deschaud  _et al_ [Deschaud et
al.(2021)Deschaud, Duque, Richa, Velasco-Forero, Marcotegui, and Goulette]
employ CARLA simulator [Dosovitskiy et al.(2017)Dosovitskiy, Ros, Codevilla,
Lopez, and Koltun] and generate synthetic LiDAR point clouds for 3D mapping
tasks. Griffiths and Boehm [Griffiths and Boehm(2019)] present a large-scale
point cloud of the urban environment using BlenSor [Gschwandtner et
al.(2011)Gschwandtner, Kwitt, Uhl, and Pree]. Xiao  _et al_ [Xiao et
al.(2022)Xiao, Huang, Guan, Zhan, and Lu] collect point cloud data using
Unreal Engine 4 (UE4) and investigate the transfer learning from synthetic to
real-world data in semantic segmentation. Wu  _et al_ [Wu et al.(2018)Wu, Wan,
Yue, and Keutzer] and Hurl  _et al_ [Hurl et al.(2019)Hurl, Czarnecki, and
Waslander] synthesize training data using Grand Theft Auto V, a popular video
game.

Randomized Scene Generation. Rao  _et al_ [Rao et al.(2021)Rao, Liu, Wei, Lu,
Hsieh, and Zhou] generate scenes by randomly placing CAD models in randomly
created rooms. They create pairs of scenes with the same objects but different
layouts and perform contrastive learning using object-level correspondence.
Yamada  _et al_ [Yamada et al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata]
employ fractal point clouds as objects instead of CAD models. They generate
bounding box labels for each object and pre-train object detectors in a fully
supervised manner. Both methods are only evaluated in object detection tasks.

##  3 Method

###  3.1 Concept of Randomized 3D Scene Generation

As shown in Fig. 1, randomized 3D scene generation requires an object set and
pre-defined rules. A room with a random size is first created to generate a
new scene. A 3D object is randomly picked from the set, undergoes data
augmentation (_e.g_ , rotation and scaling), and is randomly placed in the
room. The process is repeated until the room contains enough objects. The
rules define _e.g_ , the distribution of room size, choices and parameters for
data augmentation, distribution of objects in the room, and the number of
objects in each scene. With an object set and rules are given, a vast amount
scenes can be easily generated.

|  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Figure 2: Examples of objects used for scene generation. Left: CAD models from
ModelNet40 [Wu et al.(2015)Wu, Song, Khosla, Yu, Zhang, Tang, and Xiao].
Middle: fractal point clouds [Yamada et al.(2022)Yamada, Kataoka, Chiba,
Domae, and Ogata]. Right: spherical harmonics. Some objects are shown as
meshes, although only sampled points are used for pre-training.

Previous works use similar rules while having different choices on object
sets. Rao  _et al_ [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou] use
CAD models, which is straightforward since openly accessible datasets of CAD
models are commonly used in 3D computer vision [Wu et al.(2015)Wu, Song,
Khosla, Yu, Zhang, Tang, and Xiao, Chang et al.(2015)Chang, Funkhouser,
Guibas, Hanrahan, Huang, Li, Savarese, Savva, Song, Su, et al.]. This choice
has the drawback that creating or gathering CAD models is still laborious and
time-consuming. On the contrary, Yamada  _et al_ [Yamada et al.(2022)Yamada,
Kataoka, Chiba, Domae, and Ogata] propose generating the object sets in a
randomized manner. Specifically, they create fractal point clouds by randomly
sampling affine transformations and iteratively applying them to 3D points.
However, we hypothesize this method is sub-optional as the fractal point
clouds don’t have surfaces (see Fig. 2) and lead to a domain gap.

###  3.2 Objects and Scenes

Spherical Harmonics. In this work, we propose to generate object sets using
spherical harmonics, which can be represented in a spherical coordinate system
$(r,\theta,\phi)$:

$\displaystyle
r=\sin(m_{1}\phi)^{p_{1}}+\cos(m_{2}\phi)^{p_{2}}+\sin(m_{3}\theta)^{p_{3}}+\cos(m_{4}\theta)^{p_{4}}\,,$
(1)

where $m_{i}\in\mathbb{R}$ and $p_{i}\in\mathbb{Z^{*}}$ with
$i\in\\{1,2,3,4\\}$. With fixed coefficients $m_{i}$ and $p_{i}$, Eq. 1
describes a closed surface in 3D space. Diverse 3D objects can be generated
when the coefficients are randomly set. Some spherical harmonics are
visualized in Fig. 2. The spherical harmonics are originally employed to solve
partial differential equations [Müller(2006)]. In computer vision, they are
also applied to describe surfaces and shapes [Kazhdan et al.(2003)Kazhdan,
Funkhouser, and Rusinkiewicz, Saupe and Vranić(2001)]. The form in Eq. 1 is
motivated by the original definition but doesn’t strictly follow it. In this
work, the mathematical and physical meanings of spherical harmonics are not
considered, and we solely utilize them to parameterize the object set. The
idea is also inspired by a web page written by Paul
Bourke111http://paulbourke.net/geometry/sphericalh/. Last access on 2023.4.3..
Notice that the generated spherical harmonics can be easily represented as
(rectangular) meshes, when coordinates $\phi$ and $\theta$ are sampled with
constant intervals. The mesh representation greatly simplifies further
processing, _e.g_ , point sampling and ray-casting (explained later in this
subsection).

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)
Figure 3: Examples of generated 3D scenes in point cloud representation. From
left to right: using CAD models from ShapeNetCore [Chang et al.(2015)Chang,
Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva, Song, Su, et al.],
CAD models from ModelNet40 [Wu et al.(2015)Wu, Song, Khosla, Yu, Zhang, Tang,
and Xiao], fractal point clouds, and spherical harmonics, respectively. The
colors here are solely for visualization purposes. The generated synthetic
point clouds don’t contain color channels.

From Objects to Scenes. We assume all scenes are static, following previous
works. Also, we only consider downstream tasks in indoor scenes, while
extending the methods to outdoor scenarios is straightforward. We adopt the
generation rules of Rao  _et al_ [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh,
and Zhou] and represent generated 3D scenes as point clouds. For each randomly
picked object, we normalize it into a unit sphere, randomly sample 3000
points, and apply random data augmentation. Each generated scene contains 12
to 16 objects. Since random scaling is used as data augmentation, the point
density on each object might differ. To create a point cloud with consistent
density, we further apply grid sampling with the voxel size of
$0.04\text{\,}\mathrm{m}$. The generated scenes are visualized in Fig. 3. Our
data generation process is simpler than previous works [Rao et al.(2021)Rao,
Liu, Wei, Lu, Hsieh, and Zhou, Yamada et al.(2022)Yamada, Kataoka, Chiba,
Domae, and Ogata] since we only generate single point clouds independently
(_cf_ Sec. 2).

Single-View Data. Previous works represent the generated scenes as multi-view
point clouds. In this work, a point cloud is referred to as multi-view when it
cannot be projected to one surface without information loss. A typical example
of multi-view point clouds is a 3D scan reconstructed from data sequences. On
the contrary, many point clouds in practice are single-view and represented as
depth maps (_e.g_ , from depth cameras) or range images (_e.g_ , from
rotational laser scanners). Besides comparing different methods for randomized
scene generation, it’s also meaningful to compare the synthetic data with
real-world ones. A lot of works use single-view real-world data for self-
supervised pre-training [Zhang et al.(2021)Zhang, Girdhar, Joulin, and Misra,
Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany, Li and Heizmann(2022),
Hou et al.(2021)Hou, Graham, Nießner, and Xie, Xu et al.(2022)Xu, Yang, Zhai,
Wu, Yue, Zhan, Vajda, Keutzer, and Tomizuka], since they are the direct output
from sensors and easy to obtain.

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)
Figure 4: Examples of generated depth maps via ray-casting. (a): with objects
from ShapeNetCore [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan,
Huang, Li, Savarese, Savva, Song, Su, et al.]. (b): with objects from
ModelNet40 [Wu et al.(2015)Wu, Song, Khosla, Yu, Zhang, Tang, and Xiao]. (c)
and (d): with spherical harmonics. According to the rules in [Rao et
al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou], objects might be stacked on
others.

To clarify the impact of single-view and multi-view representations in pre-
training and for a fairer comparison with the real-world data, we also
generate single-view point clouds in this work. We create scenes using meshes
as objects instead of sampled point clouds but still following the same rules
in [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou]. Then, we set a
virtual camera with a random pose and capture a depth map using ray-casting.
The camera pose is checked so that each depth map contains sufficient objects
(set to 7 in this work). In pre-training, the depth maps are converted into
point clouds on the flight. Some generated depth maps are visualized in Fig.
4. We don’t generate depth maps with fractal point clouds since ray-casting is
non-trivial in this case.

###  3.3 Self-supervised Pre-training

In this work, we explore the generalizable pre-training using generated
synthetic data. Instead of using a pre-training strategy specific to one
downstream task, we apply masked autoencoder (MAE) [He et al.(2022)He, Chen,
Xie, Li, Dollár, and Girshick] and MoCoV2 [Chen et al.(2020)Chen, Fan,
Girshick, and He], a successful method for contrastive learning, to evaluate
the generated data. In this subsection, we briefly introduce the concept of
the two pre-training methods.

MAE [He et al.(2022)He, Chen, Xie, Li, Dollár, and Girshick] is employed to
pre-train vision transformers [Dosovitskiy et al.(2020)Dosovitskiy, Beyer,
Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold,
Gelly, et al., Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin]. An input point cloud is divided into small
patches, and a large proportion is masked. A transformer encoder extracts
patch-wise features from the remaining patches, and a transformer decoder is
trained to reconstruct the masked patches. After the pre-training, the decoder
is abandoned, and the encoder can be transferred into downstream tasks. In
this work, we adopt the method of Li and Heizmann [Li and Heizmann(2023)] for
MAE. Also, to increase the diversity of training data, we randomly crop a
region from a generated scene in each iteration instead of using the entire
scene.

We pre-train a point cloud encoder using scene-level contrastive learning
following [Zhang et al.(2021)Zhang, Girdhar, Joulin, and Misra]. Specifically,
we randomly crop two overlapping regions from each generated scene to create a
positive pair. Point clouds from different scenes are considered negative
samples. Given a point cloud, the encoder is trained to distinguish its
positive sample from many negative samples. We apply MoCoV2 [Chen et
al.(2020)Chen, Fan, Girshick, and He] by using an exponentially moving
averaged momentum encoder and a memory bank to save global features from
negative samples.

##  4 Experiments

###  4.1 Setup

Real-World Baseline. To create a baseline using real-world data, we pre-train
our models on ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser,
and Nießner], a large-scale indoor dataset of 2M RGB-D frames captured in 1500
rooms. We sample 78K frames (approximately one in every 25 frames or one per
second) from the raw data in the training set, following previous works [Xie
et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany, Li and Heizmann(2022), Hou
et al.(2021)Hou, Graham, Nießner, and Xie].

Scene Generation. To evaluate the impact of different object sets, we generate
data with four object sets, summarized in Tab. 1. For CAD models, we use
ShapeNetCore [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang,
Li, Savarese, Savva, Song, Su, et al.] and ModelNet40 [Wu et al.(2015)Wu,
Song, Khosla, Yu, Zhang, Tang, and Xiao]. The former consists of $\sim$50K
models of ordinary everyday objects (_e.g_ , chairs and shelves), while the
latter contains $\sim$10K. We follow their official train/val split and use
only objects from training sets. In our default setup, we create 10K formula-
driven 3D objects to align the object number with ModelNet40. For fractal
point clouds, we follow the configuration of Yamada  _et al_ [Yamada et
al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata]. We generate 78K scenes
with each object set in our default setup, following the real-world baseline.
We name the generated datasets RM-ShapeNet, RM-ModelNet, RM-Fractal, and RM-
Harmonics according to object sets. The abbreviation RM stands for RandomRooms
[Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou], as we adopt its
generation rules.

Dataset | Obj. Set | Obj. Num.
---|---|---
RM-ShapeNet |  ShapeNetCore [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva, Song, Su, et al.] |  $\sim$50K
RM-ModelNet |  ModelNet40 [Wu et al.(2015)Wu, Song, Khosla, Yu, Zhang, Tang, and Xiao] |  $\sim$10K
RM-Fractal |  Fractal Points [Yamada et al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata] | 10K
RM-Harmonics | Spherical Harmonics | 10K
Table 1: Summary of generated datasets.

Masked Autoencoder. We use the same architecture as [Li and Heizmann(2023)]. A
3-layer transformer encoder [Vaswani et al.(2017)Vaswani, Shazeer, Parmar,
Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] extracts features from input
patches. A decoder with 2 layers is attached to the encoder for pre-training
with MAE. We follow the configuration in [Li and Heizmann(2023)] and train all
models for 120 epochs with a batch size of 64 and an AdamW optimizer. We
evaluate the pre-trained point cloud transformer for 3D object detection on
ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner]
benchmark. We use 3DETR [Misra et al.(2021)Misra, Girdhar, and Joulin] as the
detection head and fine-tune the model for 1080 epochs with batch size 8,
following the original setup. Mean average precision with a 3D IoU threshold
of 25% and 50% (_i.e_ , AP25 and AP50) over 18 classes are evaluation
matrices. We also fine-tune the pre-trained transformer for semantic
segmentation on S3DIS [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese]
dataset. We use the same segmentation head and protocol as [Li and
Heizmann(2023)] and train for 300 epochs with batch size 12. We report mean
accuracy (mAcc) and mean IoU (mIoU) over 13 classes in Area 5.

Contrastive Learning. We train a PointNet++ [Qi et al.(2017)Qi, Yi, Su, and
Guibas] for contrastive learning, following previous works [Zhang et
al.(2021)Zhang, Girdhar, Joulin, and Misra, Li and Heizmann(2022), Rao et
al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou, Yamada et al.(2022)Yamada,
Kataoka, Chiba, Domae, and Ogata]. The network architecture is adopted from
[Qi et al.(2019)Qi, Litany, He, and Guibas], which consists of 4 Set
Abstraction modules [Qi et al.(2017)Qi, Yi, Su, and Guibas] and 2 Feature
Propagation modules [Qi et al.(2017)Qi, Yi, Su, and Guibas]. We use MoCoV2 to
pre-train the model. Specifically, we apply a global max-pooling and 3 fully
connected layers to encode the features from PointNet++. Momentum of the
momentum encoder is set to 0.999, and the size of the memory bank is 24576. We
train for 120 epochs using an SGD optimizer, an initial learning rate of 0.01,
a batch size of 12, and a cosine annealing schedule. After pre-training, we
initialize a VoteNet [Qi et al.(2019)Qi, Litany, He, and Guibas] with the pre-
trained model and fine-tune it for 3D object detection on ScanNet [Dai et
al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner] and SUN RGB-D
[Song et al.(2015)Song, Lichtenberg, and Xiao] benchmark. The SUN RGB-D
dataset consists of $\sim$10K point clouds, with half for training and half
for validation. AP25 and AP50 are calculated over 10 representative classes.
We apply the original configuration of VoteNet for fine-tuning. Each model is
trained for 180 epochs, a batch size of 8, and a learning rate of 0.001,
decayed with a factor of 0.1 after 120, 140, and 160 epochs, respectively.

More technical details are provided in our supplementary materials.

###  4.2 Experimental Results

Mask Autoencoder. We pre-train an MAE with generated data and report the fine-
tuning performance in Tab. 3. Compared to results without pre-training (row
1), all pre-trained models gain a significant improvement in object detection
and semantic segmentation.

|  | Det. | Seg.
---|---|---|---
Dataset | View | AP25 | AP50 | mAcc | mIoU
Scratch [Li and Heizmann(2023)] | - | 61.6 | 38.8 | 66.4 | 60.0
ScanNet [Li and Heizmann(2023)] | single | 64.1 | 43.0 | 74.7 | 67.6
RM-ShapeNet | multi | 63.8 | 42.6 | 74.6 | 67.8
RM-ModelNet | multi | 63.5 | 42.0 | 73.6 | 67.0
RM-Fractal | multi | 62.8 | 40.4 | 70.2 | 64.1
RM-Harmonics | multi | 63.8 | 43.2 | 74.1 | 67.1
RM-ShapeNet | single | 63.8 | 42.9 | 73.8 | 67.2
RM-ModelNet | single | 62.5 | 41.6 | 73.5 | 66.3
RM-Harmonics | single | 63.5 | 42.1 | 73.5 | 67.1
Table 2: Downstream task performance of an MAE pre-trained on different
datasets. Detection results are on the ScanNet detection benchmark.
Segmentation results are on S3DIS dataset. The first and second columns refer
to the pre-training data. All metrics in percentage.

|  | SUN RGB-D | ScanNet
---|---|---|---
Dataset | View | AP25 | AP50 | AP25 | AP50
Scratch [Qi et al.(2019)Qi, Litany, He, and Guibas] | - | 58.4 | 33.3 | 60.0 | 37.6
ScanNet | single | 60.8 | 35.7 | 62.2 | 39.0
RM-ShapeNet | multi | 59.8 | 34.1 | 62.7 | 37.9
RM-ModelNet | multi | 59.3 | 33.7 | 62.3 | 37.3
RM-Fractal | multi | 59.1 | 33.4 | 61.3 | 38.6
RM-Harmonics | multi | 59.6 | 35.7 | 62.6 | 39.7
RM-ShapeNet | single | 58.9 | 33.0 | 64.1 | 40.0
RM-ModelNet | single | 57.9 | 32.4 | 63.4 | 39.8
RM-Harmonics | single | 58.6 | 33.1 | 63.5 | 39.8
Rao  _et al_ [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou] | multi | 59.2 | - | 61.3 | -
Yamada  _et al_ [Yamada et al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata] | multi | 59.4 | 33.9 | 61.9 | 38.3
Table 3: Fine-tuning results of contrastive learning using generated data.
Evaluation for 3D object detection on SUN RGB-D and ScanNet benchmark. All
metrics in percentage.

Among multi-view variants of randomly generated data (row 3 to 6), using
ShapeNet as object set (RM-ShapeNet) delivers the best performance since
ShapeNet is significantly larger than other object sets (see Tab. 1).
Meanwhile, RM-Fractal brings the least improvement. It verifies our intuition
that fractal point clouds as objects are non-optimal in data generation due to
lack of surface. Interestingly, with the same object number, RM-Harmonics
outperform RM-ModelNet. The result is promising since it implies that randomly
generated objects can replace CAD models for scene generation. However, most
CAD models in ModelNet40 are ordinary objects in daily life, _e.g_ , chairs
and tables, while both datasets in downstream tasks are captured in indoor
environments, _e.g_ , living rooms and offices. One should expect ModelNet40
to perform better than spherical harmonics because of the similarity between
pre-training and fine-tuning data. We believe this contradiction is caused by
the diversity of the object set. Although ModelNet40 contains 40 classes, the
appearance of the objects in the same class is similar. Also, the class
distribution in ModelNet40 is long-tailed, which might also have a negative
impact [Tian et al.(2021)Tian, Henaff, and van den Oord]. On the other hand,
spherical harmonics are generated by uniformly sampling the coefficients,
which results in a more diverse and balanced object set. Furthermore, we
compare the results of generated synthetic data with the real-world ScanNet
dataset (row 2). With the same number of point clouds, The performance of RM-
ShapeNet and RM-Harmonics is close to real-world data, demonstrating the
effectiveness of randomized scene generation in self-supervised training.

With generated single-view data (_i.e_ , depth maps), a similar trend can also
be observed: RM-ShapeNet performs the best, while RM-Harmonics is more
effective than RM-ModelNet. Data generation with fractal points is not
performed since ray-casting relies on mesh data. However, compared to their
corresponding multi-view variants, all single-view datasets perform worse with
MAE.

Contrastive Learning. For contrastive learning, we pre-train a PointNet++
using MoCoV2 and fine-tune a VoteNet using it as the backbone. Object
detection results on ScanNet and SUN RGB-D are reported in Tab. 3. Generally,
pre-trained models achieve better detection quality than the baseline without
pre-training on both benchmarks. For the multi-view datasets (row 3 to 6), RM-
ShapeNet and RM-Harmonics show on-par performance. RM-ModelNet performs worse
than these two datasets, while RM-Fractal brings the smallest improvement in
pre-training. Compared to real-world data (ScanNet), randomly generated data
shows significantly worse results on SUN RGB-D dataset, while the gap
disappears on the ScanNet detection benchmark.

The relative performance of single-view variants of generated data is similar
to the multi-view ones, _e.g_ , RM-ShapeNet $>$ RM-Harmonic $>$ RM-ModelNet.
However, the detection results on SUN RGB-D dataset are worse than with multi-
view data. On the contrary, single-view datasets show significantly better
results on the ScanNet benchmark. This observation is counter-intuitive.
Notice that SUN RGB-D consists of single-view point clouds, whereas the
ScanNet benchmark applies point clouds reconstructed from multi-views. One
should expect that models pre-trained using single-view data perform worse on
the ScanNet benchmark due to the domain gap. The reason for the observation is
not fully understood. However, it must be correlated with the pre-training
schema since it’s not observed with MAE.

We also compare our results with two previous works on randomized 3D scene
generation, _e.g_ , RandomRooms [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and
Zhou] and PC-FractalDB [Yamada et al.(2022)Yamada, Kataoka, Chiba, Domae, and
Ogata], each of which applies a training strategy specialized for object
detection. Both methods use VoteNet as the detector and apply a PointNet++ as
the backbone. Our model pre-trained with RM-Fractal achieves similar
performance as the method of Yamada  _et al_ [Yamada et al.(2022)Yamada,
Kataoka, Chiba, Domae, and Ogata], which also generates scenes using fractal
point clouds. Also, our RM-Harmonics (multi-view) outperforms both previous
works by a clear margin. It shows that the choice of object sets has a larger
impact than pre-training methods. Also, it implies that a task-specific design
is unnecessary in pre-training and general purpose methods (_e.g_ , MAE and
MoCoV2) are sufficient.

Main Results and Discussions. The experimental results in Tab. 3 and 3 can be
summarized as follows: (1) Randomized scene generation is effective for pre-
training 3D models with self-supervision. This approach generalizes well
across different pre-training methods and downstream tasks. Also, its
performance is comparable with real-world data. (2) Formula-driven spherical
harmonics perform significantly better than fractal point clouds and achieve
on-par results as hand-crafted CAD models. (3) The diversity of objects
greatly impacts the effect of pre-training. An object set with a larger
diversity is generally beneficial. (4) Single-view and multi-view variants of
generated data bring different results. However, the impact is inconsistent
across different pre-training methods and fine-tuning datasets.

The observation that randomly generated data have a similar effect as real-
world ones in self-supervised pre-training is interesting since they don’t
look photo-realistic (see Fig. 3), especially when formula-driven objects are
applied. We believe it’s because neural networks perform different tasks in
self-supervised pre-training and fine-tuning (_i.e_ , pre-training and
downstream tasks are decoupled). Therefore, the performance in fine-tuning is
not sensitive to the domain gap between pre-training and fine-tuning data.
However, a too large gap can still significantly harm the performance, _e.g_ ,
fractal point clouds without surfaces perform worse than spherical harmonics
in our experiments.

Comparison with other Pre-training Methods. We also compare the fine-tuning
performance of our models pre-trained on RM-Harmonics with other pre-training
methods in Tab. 5. All methods indicated with ‘real’ are pre-trained on real-
world data from ScanNet, while DepthContrast† in Tab. 5 also uses data from
the larger Redwood indoor RGB-D scan dataset [Park et al.(2017)Park, Zhou, and
Koltun]. All methods in the upper half of Tab. 5 fine-tune a VoteNet with a
pre-trained PointNet++ as the backbone, while methods in the lower half use a
3DETR with a pre-trained transformer. Compared with methods using real-world
data for pre-training, our methods achieve competitive performance. However,
our pre-training data are randomly generated and require neither real-world 3D
scans nor hand-crafted CAD models.

Methods | Data | Model | AP25 | AP50
---|---|---|---|---
VoteNet [Qi et al.(2019)Qi, Litany, He, and Guibas] | - | P | 60.0 | 37.6
Hou  _et al_ [Hou et al.(2021)Hou, Graham, Nießner, and Xie] | real | P | - | 39.3
DepthContrast [Zhang et al.(2021)Zhang, Girdhar, Joulin, and Misra] | real | P | 61.3 | -
DepthContrast† [Zhang et al.(2021)Zhang, Girdhar, Joulin, and Misra] | real | P | 64.0 | 42.9
DPCo [Li and Heizmann(2022)] | real | P | 64.2 | 41.5
Rao  _et al_ [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou] | syn. | P | 61.3 | -
Yamada  _et al_ [Yamada et al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata] | syn. | P | 61.9 | 38.3
Ours (multi-view) | syn. | P | 62.6 | 39.7
Ours (single-view) | syn. | P | 63.5 | 39.8
3DETR [Misra et al.(2021)Misra, Girdhar, and Joulin] | - | T | 62.1 | 37.9
MaskPoint (L3) [Liu et al.(2022)Liu, Cai, and Lee] | real | T | 63.4 | 40.6
MaskPoint (L12) [Liu et al.(2022)Liu, Cai, and Lee] | real | T | 64.2 | 42.1
Li & Heizmann [Li and Heizmann(2023)] | real | T | 64.1 | 43.0
Ours (multi-view) | syn. | T | 63.8 | 43.2
Ours (single-view) | syn. | T | 63.5 | 42.1
Table 4: Object detection results on ScanNet detection benchmark. Data: type
of pre-training data (real-world, synthetic or none). Model: the architecture
of backbones. P: PointNet++, T: transformer.

Objects | Scenes | AP25 | AP50
---|---|---|---
- | - | 61.6 | 38.8
$\times$1 |  $\times$0.2 | 62.9 | 41.1
$\times$1 |  $\times$0.5 | 63.5 | 42.7
$\times$1 |  $\times$1 | 63.8 | 43.2
$\times$1 |  $\times$2 | 63.8 | 43.4
$\times$1 |  $\times$5 | 63.4 | 42.9
$\times$0.2 |  $\times$1 | 62.4 | 41.8
$\times$0.5 |  $\times$1 | 63.9 | 42.8
$\times$1 |  $\times$1 | 63.8 | 43.2
$\times$2 |  $\times$1 | 63.2 | 41.7
$\times$5 |  $\times$1 | 62.5 | 40.6
$\times 2$ | $\times 2$ | 63.2 | 42.5
$\times 5$ | $\times 5$ | 63.0 | 42.2
Table 5: Impact of object and scene numbers. Multi-view point clouds with
spherical harmonics. Evaluated for object detection on ScanNet. $\times 1$:
10K objects or 78K scenes, respectively. $\times a$: scaling with factor $a$.

Object and Scene Numbers. In this work, we propose generating scenes using
spherical harmonics as objects. In this experiment, we investigate the impact
of object and scene numbers. We first fix the number of used spherical
harmonics in scene generation (_i.e_ , 10K) and scale the scene number. As
shown in the upper half of Tab. 5, using more scenes in pre-training is
generally beneficial. But there is no significant improvement when more than
78K scenes ($\times 1$) are applied.

Furthermore, we fix the scene number to our default value and scale the object
number. With a small object number ($\times 0.2$), the generated scenes have a
smaller diversity, which harms the effect of pre-training. However, a large
object number ($\times 5$) also produces significantly worse results. Also,
scaling up the scene number and object number simultaneously (the last two
rows) doesn’t bring improvement, compared to the default setup. We hypothesize
it’s due to the limited diversity of spherical harmonics described by Eq. 1.
Since the equation has only 8 free coefficients, with a too large object
number, the minimal ‘distance’ between generated objects becomes
smaller222Consider the situation where $n$ points in a continuous interval [0,
1] have to be independently and uniformly sampled. The minimal distance
between two points decreases with increasing $n$.. As a result, some generated
scenes contain very similar objects, which has a negative impact on pre-
training. Based on the discussion, searching for new formulas with more
degrees of freedom is beneficial. We leave the exploration to future works.

##  5 Conclusion

This work demonstrates the power of randomized 3D scene generation in self-
supervised pre-training. The generated data perform well with different pre-
training methods, and the pre-trained models achieve impressive results in
multiple downstream tasks. Also, we take a closer look at the impact of object
sets and propose to replace CAD models with formula-driven spherical
harmonics. The method is promising since it reduces pre-training costs without
harming performance. We hope our work can inspire more research on randomized
data generation for self-supervised pre-training.

## References

  * [Armeni et al.(2017)Armeni, Sax, Zamir, and Savarese] Iro Armeni, Sasha Sax, Amir Roshan Zamir, and Silvio Savarese.  Joint 2d-3d-semantic data for indoor scene understanding.  _CoRR_ , abs/1702.01105, 2017.
  * [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva, Song, Su, et al.] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  ShapeNet: An information-rich 3D model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * [Chen et al.(2020)Chen, Fan, Girshick, and He] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020.
  * [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-annotated 3D reconstructions of indoor scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5828–5839, 2017.
  * [Deschaud et al.(2021)Deschaud, Duque, Richa, Velasco-Forero, Marcotegui, and Goulette] Jean-Emmanuel Deschaud, David Duque, Jean Pierre Richa, Santiago Velasco-Forero, Beatriz Marcotegui, and François Goulette.  Paris-carla-3d: A real and synthetic outdoor point cloud dataset for challenging tasks in 3D mapping.  _Remote Sensing_ , 13(22):4713, 2021.
  * [Dosovitskiy et al.(2017)Dosovitskiy, Ros, Codevilla, Lopez, and Koltun] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  CARLA: An open urban driving simulator.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , pages 1–16, 2017.
  * [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * [Fu et al.(2022)Fu, Gao, Liu, Zhang, Qiao, and Wang] Kexue Fu, Peng Gao, ShaoLei Liu, Renrui Zhang, Yu Qiao, and Manning Wang.  POS-BERT: Point cloud one-stage BERT pre-training.  _arXiv preprint arXiv:2204.00989_ , 2022.
  * [Griffiths and Boehm(2019)] David Griffiths and Jan Boehm.  SynthCity: A large scale synthetic point cloud.  _arXiv preprint arXiv:1907.04758_ , 2019.
  * [Gschwandtner et al.(2011)Gschwandtner, Kwitt, Uhl, and Pree] Michael Gschwandtner, Roland Kwitt, Andreas Uhl, and Wolfgang Pree.  BlenSor: Blender sensor simulation toolbox.  In _Advances in Visual Computing: 7th International Symposium, ISVC 2011, Las Vegas, NV, USA, September 26-28, 2011. Proceedings, Part II 7_ , pages 199–208. Springer, 2011.
  * [He et al.(2020)He, Fan, Wu, Xie, and Girshick] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * [He et al.(2022)He, Chen, Xie, Li, Dollár, and Girshick] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * [Hou et al.(2021)Hou, Graham, Nießner, and Xie] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie.  Exploring data-efficient 3D scene understanding with contrastive scene contexts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15587–15597, 2021.
  * [Hurl et al.(2019)Hurl, Czarnecki, and Waslander] Braden Hurl, Krzysztof Czarnecki, and Steven Waslander.  Precise synthetic image and lidar (presil) dataset for autonomous vehicle perception.  In _2019 IEEE Intelligent Vehicles Symposium (IV)_ , pages 2522–2529. IEEE, 2019.
  * [Kazhdan et al.(2003)Kazhdan, Funkhouser, and Rusinkiewicz] Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz.  Rotation invariant spherical harmonic representation of 3D shape descriptors.  In _Symposium on geometry processing_ , volume 6, pages 156–164, 2003\.
  * [Li and Heizmann(2022)] Lanxiao Li and Michael Heizmann.  A closer look at invariances in self-supervised pre-training for 3D vision.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX_ , pages 656–673. Springer, 2022.
  * [Li and Heizmann(2023)] Lanxiao Li and Michael Heizmann.  Applying plain transformers to real-world point clouds.  _arXiv preprint arXiv:2303.00086_ , 2023.
  * [Liu et al.(2022)Liu, Cai, and Lee] Haotian Liu, Mu Cai, and Yong Jae Lee.  Masked discrimination for self-supervised learning on point clouds.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2022.
  * [Misra et al.(2021)Misra, Girdhar, and Joulin] Ishan Misra, Rohit Girdhar, and Armand Joulin.  An end-to-end transformer model for 3D object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2906–2917, 2021.
  * [Müller(2006)] Claus Müller.  _Spherical harmonics_ , volume 17.  Springer, 2006.
  * [Pang et al.(2022)Pang, Wang, Tay, Liu, Tian, and Yuan] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  _ECCV_ , 2022.
  * [Park et al.(2017)Park, Zhou, and Koltun] Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Colored point cloud registration revisited.  In _Proceedings of the IEEE international conference on computer vision_ , pages 143–152, 2017.
  * [Qi et al.(2017)Qi, Yi, Su, and Guibas] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  PointNet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ , 30, 2017.
  * [Qi et al.(2019)Qi, Litany, He, and Guibas] Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas.  Deep Hough voting for 3D object detection in point clouds.  In _proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9277–9286, 2019.
  * [Qian et al.(2022)Qian, Zhang, Hamdi, and Ghanem] Guocheng Qian, Xingdi Zhang, Abdullah Hamdi, and Bernard Ghanem.  Pix4Point: Image pretrained transformers for 3D point cloud understanding.  _arXiv preprint arXiv:2208.12259_ , 2022.
  * [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou] Yongming Rao, Benlin Liu, Yi Wei, Jiwen Lu, Cho-Jui Hsieh, and Jie Zhou.  Randomrooms: unsupervised pre-training from synthetic shapes and randomized layouts for 3D object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3283–3292, 2021.
  * [Saupe and Vranić(2001)] Dietmar Saupe and Dejan V Vranić.  3D model retrieval with spherical harmonics and moments.  In _Pattern Recognition: 23rd DAGM Symposium Munich, Germany, September 12–14, 2001 Proceedings 23_ , pages 392–397. Springer, 2001.
  * [Song et al.(2015)Song, Lichtenberg, and Xiao] Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao.  SUN RGB-D: A RGB-D scene understanding benchmark suite.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 567–576, 2015.
  * [Tian et al.(2021)Tian, Henaff, and van den Oord] Yonglong Tian, Olivier J Henaff, and Aäron van den Oord.  Divide and contrast: Self-supervised learning from uncurated data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10063–10074, 2021.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * [Wang et al.(2021)Wang, Liu, Yue, Lasenby, and Kusner] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J. Kusner.  Unsupervised point cloud pre-training via occlusion completion.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9782–9792, 2021.
  * [Wu et al.(2018)Wu, Wan, Yue, and Keutzer] Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer.  Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3D lidar point cloud.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 1887–1893. IEEE, 2018.
  * [Wu et al.(2015)Wu, Song, Khosla, Yu, Zhang, Tang, and Xiao] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3D ShapeNets: A deep representation for volumetric shapes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1912–1920, 2015.
  * [Xiao et al.(2022)Xiao, Huang, Guan, Zhan, and Lu] Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu.  Transfer learning from synthetic to real lidar point cloud for semantic segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 2795–2803, 2022.
  * [Xie et al.(2020)Xie, Gu, Guo, Qi, Guibas, and Litany] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  PointContrast: Unsupervised pre-training for 3D point cloud understanding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 574–591. Springer, 2020.
  * [Xu et al.(2022)Xu, Yang, Zhai, Wu, Yue, Zhan, Vajda, Keutzer, and Tomizuka] Chenfeng Xu, Shijia Yang, Bohan Zhai, Bichen Wu, Xiangyu Yue, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka.  Image2point: 3D point-cloud understanding with pretrained 2d convnets.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2022.
  * [Yamada et al.(2022)Yamada, Kataoka, Chiba, Domae, and Ogata] Ryosuke Yamada, Hirokatsu Kataoka, Naoya Chiba, Yukiyasu Domae, and Tetsuya Ogata.  Point cloud pre-training with natural 3D structures.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21283–21293, 2022.
  * [Yu et al.(2022)Yu, Tang, Rao, Huang, Zhou, and Lu] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-BERT: Pre-training 3D point cloud transformers with masked point modeling.  In _CVPR_ , pages 19313–19322, 2022.
  * [Zhang et al.(2022a)Zhang, Guo, Gao, Fang, Zhao, Wang, Qiao, and Li] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li.  Point-M2AE: Multi-scale masked autoencoders for hierarchical point cloud pre-training.  _arXiv preprint arXiv:2205.14401_ , 2022a.
  * [Zhang et al.(2022b)Zhang, Guo, Zhang, Li, Miao, Cui, Qiao, Gao, and Li] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  PointCLIP: Point cloud understanding by CLIP.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8552–8562, 2022b.
  * [Zhang et al.(2021)Zhang, Girdhar, Joulin, and Misra] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra.  Self-supervised pretraining of 3D features on any point-cloud.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10252–10263, 2021.

## Appendix

##  Appendix A Generating Spherical Harmonics

To generate sufficient objects for creating randomized scenes, we randomly set
the coefficients in spherical harmonics. In this work, we constrain $m_{i}$ in
the range $[-5,~{}5]$ and $p_{i}$ in $[0,~{}4]$, since too large coefficients
lead to high-frequency structures. Due to the sampling theorem, these details
are lost when we sample points from generated spherical harmonics and are thus
invisible in the pre-training.

|  |  |
---|---|---|---
$p_{1}=0$ | $p_{1}=1$ | $p_{1}=2$ | $p_{1}=3$
|  |  |
$p_{1}=4$ | $p_{1}=5$ | $p_{1}=6$ | $p_{1}=7$
Figure 5: Impact of coefficient $p_{1}$ in spherical harmonics. For all objects: $[m_{1},m_{2},m_{3},m_{4}]=[2,1,2,2]$ and $[p_{2},p_{3},p_{4}]=[2,1,2]$.  |  |  |
---|---|---|---
$m_{1}=0$ | $m_{1}=1$ | $m_{1}=2$ | $m_{1}=3$
|  |  |
$m_{1}=4$ | $m_{1}=5$ | $m_{1}=6$ | $m_{1}=7$
Figure 6: Impact of coefficient $m_{1}$ in spherical harmonics. For all
objects: $[m_{2},m_{3},m_{4}]=[1,2,2]$ and
$[p_{1},p_{2},p_{3},p_{4}]=[2,2,1,2]$.

The impact of the coefficients is illustrated in Fig. 5, where all
coefficients are fixed except $p_{1}$. With increasing $p_{1}$, the ‘fins’ of
the corresponding spherical harmonics become finer. A similar effect can be
seen in Fig. 6, where the influence of $m_{1}$ is visualized. With large
coefficients, the overall appearance of spherical harmonics is close to each
other. Thus, we constrain the spherical harmonics in the low-frequency range
to generate diverse scenes.

##  Appendix B Pseudo Color in Pre-training

Dataset | Colors | mAcc | mIoU
---|---|---|---
From Scratch | - | 66.4 | 60.0
RM-Harmonics | const. | 69.8 | 62.8
RM-Harmonics | const. + DA | 74.1 | 67.1
ScanNet | const. | 69.8 | 63.6
ScanNet | const. + DA | 73.1 | 67.0
ScanNet | real | 74.7 | 67.6
Table 6: Impact of colors in pre-training using MAE. Fine-tuned for semantic
segmentation on S3DIS dataset and evaluated in Area 5. The first two columns
refer to the pre-training data. Const.: constant RGB values for all points.
DA: data augmentation on color channels.

Since simulation and rendering are not involved, the scenes generated in this
work don’t contain color information. However, color channels are crucial in
some downstream tasks, _e.g_ , semantic segmentation. To transfer models
trained on colorless point clouds to such tasks, we apply pseudo colors and
data augmentation to color channels in pre-training. Specifically, we set all
color channels to a constant value (_i.e_ , 0.5) and apply random color jitter
to each point. Also, we simultaneously set color values of all points to 0
with the probability 0.5 (_i.e_ , color drop out).

Experiments are conducted to analyze the effect of this approach. We first
pre-train an MAE on the RM-Harmonics dataset. As shown in Tab. 6, when no data
augmentation (_i.e_ , jitter and drop out) is applied to color values, the
constant pseudo color leads to significantly worse results with RM-Harmonics.
Also, we pre-train the model on ScanNet and compare the performance of pseudo
colors and real-world colors. Without data augmentation on pseudo colors, the
performance in segmentation is much lower than using real-world colors.
However, applying data augmentation significantly shrinks the gap.
Interestingly, when both use pseudo colors, RM-Harmonics slightly outperforms
ScanNet in pre-training. Results in Tab. 6 show that pseudo colors can be
applied to pre-train models requiring colors on colorless point clouds. Also,
data augmentation is essential for improving performance. We believe models
learn color invariance via data augmentation and focus on the geometric
information in the pre-training. In contrast, the color-dependent relationship
is learned in fine-tuning where real-world colors are available.

##  Appendix C Data Generation

We generate scenes based on the rules of Rao  _et al_ [Rao et al.(2021)Rao,
Liu, Wei, Lu, Hsieh, and Zhou]. Modifications are made to the original
configurations. Our procedure can be summarized as follows:

  1. 1.

Randomly pick 12 to 16 objects from the object set.

  2. 2.

Independently apply data augmentation to each object (more details in Sec. D)

  3. 3.

Sort the objects in descending order based on their projection areas on the
horizontal plane (X-Y plane).

  4. 4.

Randomly set the area (X-Y plane) of a rectangular room depending on the area
sum of objects so that the objects can fit into the generated scene.

  5. 5.

Randomly generate walls and floors according to the area.

  6. 6.

Place each object by randomly choosing a position on the X-Y plane. Objects
can be stacked on previous ones as long as the current height of the position
is below $2\text{\,}\mathrm{m}$. Otherwise, choose a new position.

  7. 7.

For multi-view point clouds, voxelize the entire scene with voxel size
$0.04\text{\,}\mathrm{m}$. We further randomly sample 40000 points and save
them for pre-training.

  8. 8.

For single-view point clouds, randomly place a virtual camera in the scene and
capture a depth map via ray-casting. Check the result so that each depth map
contains at least 7 objects. All depth maps have the resolution
640$\times$480, following the real-world dataset ScanNet [Dai et al.(2017)Dai,
Chang, Savva, Halber, Funkhouser, and Nießner]. The camera intrinsic and
camera pose are also saved.

Since Rao  _et al_ [Rao et al.(2021)Rao, Liu, Wei, Lu, Hsieh, and Zhou] didn’t
open source their code, our work is based on our implementation. Our code will
be made publicly available.

##  Appendix D Pre-processing and Data Augmentation

In Data Generation. Data augmentation is applied on each object during scene
generation. Specifically, we first normalize an object into a unit sphere and
randomly scale it with a factor uniformly distributed in [0.7, 1.5]. Then we
flip the object left-right with a probability of 0.5 and rotate the object
around the vertical axis (Z-axis) with an arbitrary angle. For spherical
harmonics, we further swap Z-axis and Y-axis with the probability 0.5 since
spherical harmonics show approximate rotational symmetry around Z-axis (see
Tab. 5 and 6). We do so to prevent models from learning the bias that all
objects are approximately symmetric around the vertical axis. At last, we
randomly sample 3000 points from each object for generating multi-view point
clouds. To generate single-view data via ray-casting, we use mesh
representation without sampling.

For Pre-training. The primary pre-processing involved in our pipelines is
cropping a region from an entire scene. However, the generated data must be
differently pre-processed depending on pre-training methods and inputs:

  1. 1.

For MAE with multi-view point clouds: we randomly choose a point in a complete
point cloud and crop the closest 20000 points (_i.e_ , 50 %).

  2. 2.

For MAE with single-view point clouds: we crop a rectangular region with a
random ratio in [0.6, 0.8] from a depth map and transform the crop into a
point cloud.

  3. 3.

For contrastive learning: similar approaches are applied as for MAE.
Specifically, we crop spherical regions in multi-view point clouds and crop
depth maps for single-view point clouds. However, in each scene, we make two
overlapping crops instead of one to create a positive pair for contrastive
learning.

Subsequently, we apply standard data augmentation to each crop, including
random translation, rotation around Z-axis, scaling, point jitter, and left-
right flipping.

|
---|---
|
|
|
Figure 7: More examples of generated scenes. From the first to the fourth row:
RM-ShapeNet, RM-ModelNet, RM-Fractal, and RM-Harmonics, respectively.

##  Appendix E More Visualizations

We visualize more generated scenes in Fig. 7.