# Diffusion Probabilistic Models for Scene-Scale 3D Categorical Data

Jumin Lee Woobin Im Sebin Lee Sung-Eui Yoon

Korea Advanced Institute of Science and Technology (KAIST)
<EMAIL_ADDRESS>
###### Abstract

In this paper, we learn a diffusion model to generate 3D data on a scene-
scale. Specifically, our model crafts a 3D scene consisting of multiple
objects, while recent diffusion research has focused on a single object. To
realize our goal, we represent a scene with discrete class labels, i.e.,
categorical distribution, to assign multiple objects into semantic categories.
Thus, we extend discrete diffusion models to learn scene-scale categorical
distributions. In addition, we validate that a latent diffusion model can
reduce computation costs for training and deploying. To the best of our
knowledge, our work is the first to apply discrete and latent diffusion for 3D
categorical data on a scene-scale. We further propose to perform semantic
scene completion (SSC) by learning a conditional distribution using our
diffusion model, where the condition is a partial observation in a sparse
point cloud. In experiments, we empirically show that our diffusion models not
only generate reasonable scenes, but also perform the scene completion task
better than a discriminative model. Our code and models are available at
https://github.com/zoomin-lee/scene-scale-diffusion.

##  1 Introduction

(a) Object-scale generation

(b) Scene-scale generation (ours)

Figure 1: Comparison of object-scale and scene scale generation (ours). Our
result includes multiple objects in a generated scene, while the object-scale
generation crafts one object at a time. (a) is obtained by Point-E [1].

Learning to generate 3D data has received much attention thanks to its high
performance and promising downstream tasks. For instance, a 3D generative
model with a diffusion probabilistic model [2] has shown its effectiveness in
3D completion [2] and text-to-3D generation [3, 1].

While recent models have focused on 3D object generation, we aim beyond a
single object by generating a 3D scene with multiple objects. In Fig. 1(b), we
show a sample scene from our generative model, where we observe the plausible
placement of the objects, as well as their correct shapes. Compared to the
existing object-scale model [1] (Fig. 1(a)), our scene-scale model can be used
in a broader application, such as semantic scene completion (Sec. 4.3), where
we complete a scene given a sparse LiDAR point cloud.

We base our scene-scale 3D generation method on a diffusion model, which has
shown remarkable performance in modeling complex real-world data, such as
realistic 2D images [4, 5, 6] and 3D objects [2, 3, 1]. We develop and
evaluate diffusion models learning a scene-scale 3D categorical distribution.

First, we utilize categorical data for a voxel entity since we have multiple
objects in contrast to the existing work [2, 3, 1], so each category tells
each voxel belongs to which category. Thus, we extend discrete diffusion
models for 2D categorical data [7, 8] into 3D categorical data (Sec. 3.1).
Second, we validate the latent diffusion model for the 3D scene-scale
generation, which can reduce training and testing computational cost (Sec.
3.2). Third, we propose to perform semantic scene completion (SSC) by learning
a conditional distribution using our generative models, where the condition is
a partial observation of the scene (Sec. 3.1). That is, we demonstrate that
our model can complete a reasonable scene in a realistic scenario with a
sparse and partial observation.

Lastly, we show the effectiveness of our method in terms of the unconditional
and conditional (SSC) generation tasks on the CarlaSC dataset [9] (Sec. 4).
Especially, we show that our generative model can outperform a discriminative
model in the SSC task.

##  2 Related Work

###  2.1 Semantic Scene Completion

Leveraging 3D data for semantic segmentation has been studied from different
perspectives. Vision sensors (e.g., RGB-D camera and LiDAR) provide depth
information from a single viewpoint, giving more information about the world.
One of the early approaches is using an RGB-D (i.e., color and depth) image
with a 2D segmentation map [10]. In addition, using data in a 3D coordinate
system has been extensively studied. 3D semantic segmentation is the extension
of 2D segmentation, where a classifier is applied to point clouds or voxel
data in 3D coordinates [11, 12].

One of the recent advances in 3D semantic segmentation is semantic scene
completion (SSC), where a partially observable space – observed via RGB-D
image or point clouds – should be densely filled with class labels [13, 14,
15, 16]. In SSC, a model gets the point cloud obtained in one viewpoint; thus,
it contains multiple partial objects (e.g., one side of a car). Then, the
model not only reconstructs the unobserved shape of the car but also labels it
as a car. Here, the prediction about the occupancy and the semantic labels can
mutually benefit [17].

Due to the partial observation, filling in occluded and sparse areas is the
biggest hurdle. Thus, a generative model is effective for 3D scene completion
as 2D completion tasks [18, 19]. Chen et al. [20] demonstrate that generative
adversarial networks (GANs) can be used to improve the plausibility of a
completion result. However, a diffusion-based generative model has yet to be
explored in terms of a 3D semantic segmentation map. We speculate that using a
diffusion model has good prospects, thanks to the larger size of the latent
and the capability to deal with high-dimensional data.

In this work, we explore a diffusion model in the context of 3D semantic scene
completion. Diffusion models have been rapidly growing and they perform
remarkably well on real-world 2D images [21]. Thus, we would like to delve
into the diffusion to generate 3D semantic segmentation maps; thus, we hope to
provide the research community a useful road map towards generating the 3D
semantic scene maps.

###  2.2 Diffusion Models

Recent advances in diffusion models have shown that a deep model can learn
more diverse data distribution by a diffusion process [5]. A diffusion process
is introduced to adopt a simple distribution (e.g., Gaussian) to learn a
complex distribution [4]. Especially, diffusion models show impressive results
for image generation [6] and conditional generation [22, 23] on high
resolution compared to GANs. GANs are known to suffer from the mode collapse
problem and struggle to capture complex scenes with multiple objects [24]. On
the other hand, diffusion models have a capacity to escape mode collapse [6]
and generate complex scenes [23, 25] since likelihood-based methods achieve
better coverage of full data distribution.

Diffusion models have been studied to a large extent in high-dimensional
continuous data. However, they often lack the capacity to deal with discrete
data (e.g., text and segmentation maps) since the discreteness of data is not
fully covered by continuous representations. To tackle such discreteness,
discrete diffusion models have been studied for various applications, such as
text generation [8, 7] and low-dimensional segmentation maps generation [7].

Since both continuous and discrete diffusion models estimate the density of
image pixels, a higher image resolution means higher computation. To address
this issue, latent diffusion models [26, 23] operate a diffusion process on
the latent space of a lower dimension. To work on the compressed latent space,
Vector-Quantized Variational Auto-Encoder (VQ-VAE) [27] is employed. Latent
diffusion models consist of two stages: VQ-VAE and diffusion. VQ-VAE trains an
encoder to compress the image into a latent space. Equipped with VQ-VAE,
autoregressive models [28, 29] have shown impressive performance. Recent
advances in latent diffusion models further improve the generative performance
by ameliorating the unidirectional bias and accumulated prediction error in
existing models [23, 26].

Our work introduces an extension of discrete diffusion models for high-
resolution 3D categorical voxel data. Specifically, we show the effectiveness
of a diffusion model in terms of unconditional and conditional generation
tasks, where the condition is a partial observation of a scene (i.e., SSC).
Further, we propose a latent diffusion models for 3D categorical data to
reduce the computation load caused by high-resolution segmentation maps.

###  2.3 Diffusion Models for 3D Data

Diffusion models have been used for 3D data. Until recently, research has been
mainly conducted for 3D point clouds with xyz-coordinates. PVD [2] applies
continuous diffusion on point-voxel representations for object shape
generation and completion without additional shape encoders. LION [3] uses
latent diffusion for object shape completion (i.e., conditional generation)
with additional shape encoders.

In this paper, we aim to learn 3D categorical data (i.e., 3D semantic
segmentation maps) with a diffusion model. The study of object generation has
shown promising results, but as far as we know, our work is the first to
generate a 3D scene with multiple objects using a diffusion model. Concretely,
our work explores discrete and latent diffusion models to learn a distribution
of volumetric semantic scene segmentation maps. We develop the models in an
unconditional and conditional generation; the latter can be used directly for
the SSC task.

##  3 Method

(a) Discrete Diffusion Models

(b) Latent Diffusion Models

Figure 2: Overview of (a) Discrete Diffusion Models and (b) Latent Diffusion
Models. Discrete diffusion models conduct diffusion process on voxel space,
whereas latent diffusion models operate diffusion process on latent space.

Our goal is to learn a data distribution $p(\mathbf{x})$ using diffusion
models, where each data $\mathbf{x}\sim p(\mathbf{x})$ represents a 3D
segmentation map described with the one-hot representation. 3D segmentation
maps are samples from the data distribution $p(\mathbf{x}$), which is the
categorical distribution $\text{Cat}(k_{0},k_{1},\cdots,k_{M})$ with $M+1$
probabilities of the free label $k_{0}$ and $M$ main categories. The discrete
diffusion models could learn data distribution by recovering the noised data,
which is destroyed through the successive transition of the label [8].

Our method aims to learn a distribution of voxelized 3D segmentation maps with
discrete diffusion (Sec. 3.1). Specifically, it includes unconditional and
conditional generation, where the latter corresponds to the SSC task. In
addition, we explore a latent diffusion model for 3D segmentation maps (Sec.
3.2).

###  3.1 Discrete Diffusion Models

Fig. 2(a) summarizes the overall process of discrete diffusion, consisting of
a forward process and a reverse process; the former gradually adds noise to
the data and the latter learns to denoise the noised data.

In the forward process in the discrete diffusion, an original segmentation map
$\mathbf{x}_{0}$ is gradually corrupted into a $t$-step noised segmentation
map $\mathbf{x}_{t}$ with $1\leq t\leq T$. Each forward step can be defined by
a Markov uniform transition matrix $Q_{t}$ [8] as
$\mathbf{x}_{t}=\mathbf{x}_{t-1}Q_{t}$. Based on the Markov property, we can
derive the $t$-step noised segmentation map $\mathbf{x}_{t}$ straight from the
original segmentation map $\mathbf{x}_{0}$,
$q(\mathbf{x}_{t}|\mathbf{x}_{0})$, with a cumulative transition matrix
$\bar{Q}_{t}=Q_{1}Q_{2}\cdots Q_{t}$:

$q(\mathbf{x}_{t}|\mathbf{x}_{0})=\text{Cat}(\mathbf{x}_{t};p=\mathbf{x}_{0}\bar{Q}_{t}).$
(1)

In the reverse process parametrized by $\theta$, a learnable model is used to
reverse a noised segmentation map by
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$. Specifically, we use a
reparametrization trick [5] to make the model predict a denoised map
$\tilde{\mathbf{x}}_{0}$ and subsequently get the reverse process
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$:

$\displaystyle p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$
$\displaystyle=q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\tilde{\mathbf{x}}_{0})p_{\theta}(\tilde{\mathbf{x}}_{0}|\mathbf{x}_{t}),$
(2) $\displaystyle q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\tilde{\mathbf{x}}_{0})$
$\displaystyle=\frac{q(\mathbf{x}_{t}|\mathbf{x}_{t-1},\tilde{\mathbf{x}}_{0})q(\mathbf{x}_{t-1}|\tilde{\mathbf{x}}_{0})}{q(\mathbf{x}_{t}|\tilde{\mathbf{x}}_{0})}.$
(3)

We optimize a joint loss that consists of the KL divergence of the forward
process $q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})$ from the reverse
process $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$; of the original
segmentation map $q(\mathbf{x}_{0})$ from the reconstructed one
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ for an auxiliary loss:

$\mathcal{L}=D_{KL}(\,q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})\,\|\,p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})\,)\\\
+w_{0}D_{KL}(\,q(\mathbf{x}_{0})\,\|\,p_{\theta}(\tilde{\mathbf{x}}_{0}|\mathbf{x}_{t})\,),$
(4)

where $w_{0}$ is an auxiliary loss weight.

Unlike existing discrete diffusion models [7, 8], our goal is to learn the
distribution of 3D data. Thus, to better handle 3D data, we use a point cloud
segmentation network [30] with modifications for discrete data and time
embedding.

#### Conditional generation.

We propose discrete diffusion for Semantic Scene Completion (SSC) with
conditional generation. SSC jointly estimates a scene’s complete geometry and
semantics, given a sparse occupancy map $\mathbf{s}$. Thus, it introduces a
condition into Eq. 2, resulting in:

$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{s})=q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\tilde{\mathbf{x}}_{0})p_{\theta}(\tilde{\mathbf{x}}_{0}|\mathbf{x}_{t},\mathbf{s}),$
(5)

where $\mathbf{s}$ is a sparse occupancy map. We give the condition by
concatenating a sparse occupancy map $\mathbf{s}$ with a corrupted input
$\mathbf{x}_{t}$.

###  3.2 Latent Diffusion Models

Fig. 2(b) provides an overview of latent diffusion on 3D segmentation maps.
Latent diffusion models project the 3D segmentation maps into a smaller latent
space and operate a diffusion process on the latent space instead of the high-
dimensional input space. A latent diffusion takes advantage of a lower
training computational cost and a faster inference by processing diffusion on
a lower dimensional space.

To encode a 3D segmentation map into a latent representation, we use Vector
Quantized Variational AutoEncoder (VQ-VAE) [27]. VQ-VAE extends the VAE by
adding a discrete learnable codebook
$E=\\{\mathbf{e}_{n}\\}^{N}_{n=1}\in\mathbb{R}^{N\times d}$, where $N$ is the
size of the codebook and $d$ is the dimension of the codes. The encoder
$\mathcal{E}$ encodes 3D segmentation maps $\mathbf{x}$ into a latent
$\mathbf{z}=\mathcal{E}(\mathbf{x})$, and the quantizer $VQ(\cdot)$ maps the
latent $\mathbf{z}$ into a quantized latent $\mathbf{z}_{q}$, which is the
closest codebook entry $\mathbf{e}_{n}$. Note that the latent
$\mathbf{z}\in\mathbb{R}^{h\times w\times z\times d}$ has a smaller spatial
resolution than the segmentation map $\mathbf{x}$. Then the decoder
$\mathcal{D}$ reconstructs the 3D segmentation maps from the quantized latent,
$\tilde{\mathbf{x}}=\mathcal{D}(VQ(\mathcal{E}(\mathbf{x})))$. The encoder
$\mathcal{E}$, the decoder $\mathcal{D}$, and the codebook $E$ can be trained
end-to-end using the following loss function:

$\mathcal{L}_{VQVAE}=-\sum_{k}w_{k}\mathbf{x}_{k}\log(\tilde{\mathbf{x}}_{k})+\|sg(\mathbf{z})-\mathbf{z}_{q}\|_{2}^{2}\\\
+\|\mathbf{z}-sg(\mathbf{z}_{q})\|_{2}^{2},$ (6)

where $w_{k}$ is a class weight and $sg(\cdot)$ is the stop-gradient
operation. Training the latent diffusion model is similar to that of discrete
diffusion. Discrete diffusion models diffuse between labels, but latent
diffusion models diffuse between codebook indexes using Markov Uniform
transition matrix $Q_{t}$ [8].

##  4 Experiments

In this section, we empirically study the effectiveness of the diffusion
models on 3D voxel segmentation maps. We divide the following sub-sections
into the learning of the unconditional data distribution $p(\mathbf{x})$ (Sec.
4.2) and the conditional data distribution $p(\mathbf{x}|\mathbf{s})$ given a
sparse occupancy map $\mathbf{s}$ (Sec. 4.3); note that the latter corresponds
to semantic scene completion (SSC).

###  4.1 Implementation Details

Dataset. Following prior work [9], we employ the CarlaSC dataset – a synthetic
outdoor driving dataset – for training and evaluation. The dataset consists of
24 scenes in 8 dynamic maps under low, medium, and high traffic conditions.
The splits of the dataset contain 18 training, 3 validation, and 3 test
scenes, which are annotated with 10 semantic classes and a free label. Each
scene with a resolution of $128\times 128\times 8$ covers a range of
$25.6\text{\,}\mathrm{m}$ ahead and behind the car, $25.6\text{\,}\mathrm{m}$
to each side, and $3\text{\,}\mathrm{m}$ in height.

Metrics. Since SSC requires predicting the semantic label of a voxel and an
occupancy state together, we use mIoU and IoU as SSC and VQ-VAE metrics. The
mIoU measures the intersection over union averaged over all classes, and the
IoU evaluates scene completion quality, regardless of the predicted semantic
labels.

Experimental settings. Experiments are deployed on two NVIDIA GTX 3090 GPUs
with a batch size of 8 for diffusion models and 4 for VQ-VAE. Our models
follow the same training strategy as multinomial diffusion [7]. We set the
hyper-parameters of the diffusion models with the number of time steps $T=100$
timesteps. And for VQ-VAE, we set the codebook
$E=\\{\mathbf{e}_{n}\\}^{N}_{n=1}\in\mathbb{R}^{N\times d}$ where the codebook
size $N=1100$, dimension of codes $d=11$ and
$\mathbf{e}_{n}\in\mathbb{R}^{32\times 32\times 2\times d}$. For diffusion
architecture, we slightly modify the encoder–decoder structure in Cylinder3D
[30] for time embedding and discreteness of the data. And for VQ-VAE
architecture, we also use encoder–decoder structure in Cylinder3D [30], but
with the vector quantizer module.

###  4.2 3D Segmentation Maps Generation

We use the discrete and the latent diffusion models for 3D segmentation map
generation. Fig. 3 shows the qualitative results of the generation. As seen in
the figure, both the discrete and latent models learn the categorical
distribution as they produce a variety of reasonable scenes. Note that our
models are learned on a large-scale data distribution like the 3D scene with
multiple objects; this is worth noting since recent 3D diffusion models for
point clouds have been performed on an object scale [2, 31, 32, 3].

Model | Resolution |  | Training
---
(time/epoch)
| Sampling
---
(time/img)
D-Diffusion |  128$\times$128$\times$8 | 19m 48s | 0.883s
L-Diffusion |  32$\times$32$\times$2 | 7m 37s | 0.499s
16$\times$16$\times$2 | 4m 41s | 0.230s
8$\times$8$\times$2 | 4m 40s | 0.202s
Table 1: Computation time comparison between discrete diffusion models and
latent diffusion models for 3D segmentation maps generation. ‘D-Diffusion’ and
‘L-Diffusion’ denote discrete diffusion models and latent diffusion models,
respectively. ‘Resolution’ means the resolution of the space in which
diffusion process operates. A latent diffusion models process diffusion on a
lower dimensional latent space, as a result, it shows advantage of a faster
training and sampling time.

In Tab. 1, we compare training and sampling time models for different
resolutions on which each diffusion model operates. Compared to the discrete
diffusion, the latent diffusion tends to show shorter training and inference
time. This is because the latent diffusion models compress the data into a
smaller latent so that the time decreases as the compression rate increases.
In particular, compared to discrete diffusion, which performs a diffusion
process in voxel space, $32\times 32\times 32$ latent diffusion has 2.6 times
faster training time for one epoch and 1.8 times faster sampling time for
generating one image.

| Codebook size
---
($N$)
| Resolution
---
($h\times w\times z$)
IoU | mIoU
220 |  8$\times$8$\times$2 | 72.5 | 27.3
16$\times$16$\times$2 | 78.7 | 36.9
32$\times$32$\times$2 | 84.6 | 56.5
550 |  8$\times$8$\times$2 | 67.7 | 25.7
16$\times$16$\times$2 | 79.4 | 39.7
32$\times$32$\times$2 | 85.8 | 58.4
1,100 |  8$\times$8$\times$2 | 70.3 | 25.7
16$\times$16$\times$2 | 79.3 | 35.0
32$\times$32$\times$2 | 89.1 | 65.1
2,200 |  8$\times$8$\times$2 | 70.2 | 26.5
16$\times$16$\times$2 | 77.7 | 37.9
32$\times$32$\times$2 | 89.2 | 64.2
Table 2: Ablation study on VQ-VAE hyper-parameters. We compare different sizes
of codebook $N$ and resolutions of the latent space $h{\times}w{\times}z$.

#### Ablation study on VQ-VAE.

Latent diffusion models consist of two stages. The VQ-VAE compresses 3D
segmentation maps to latent space, and then discrete diffusion models apply on
the codebook index of latent. Therefore, the performance of VQ-VAE may set the
upper bound for the final generation quality. So we conduct an ablation study
about VQ-VAE while adjusting the resolution of the latent space $h\times
w\times z$ and the codebook capacities $N$ while keeping the code dimension
$d$ fixed. Concretely, we compress the 3D segmentation maps from $128\times
128\times 8$ to $32\times 32\times 2$, $16\times 16\times 2$, and $8\times
8\times 2$ with four different codebook size $N\in\\{220,550,1100,2200\\}$.

The quantitative comparison is shown in Tab. 2. The bigger the codebook size
is, the higher the performance is, but it saturates around 1,100. That is
because most of the codes are not updated, and the update of the codebook can
lapse into a local optimum [33].

The resolution of latent space has a significant impact on performance. As the
resolution of the latent space becomes smaller, it cannot contain all the
information of the 3D segmentation map. Setting the resolution to $32\times
32\times 2$ with a $1{,}100$ codebook size strike a good balance between
efficiency and fidelity.

Methods | IoU | mIoU
---|---|---
LMSCNet SS [16] | 85.98 | 42.53
SSCNet Full [17] | 80.69 | 41.91
MotionSC (T=1) [9] | 86.46 | 46.31
Our network w/o Diffusion | 80.70 | 39.94
Discrete Diffusion (Ours) | 80.61 | 45.83
Table 3: Semantic Scene Completion results on test set of CarlaSC

Figure 3:  Samples from our unconditional diffusion models. The first column
shows samples from training datasets. From the second column, we show samples
from our discrete diffusion and latent diffusion models. We can observe our
diffusion models learn the 3D categorical distribution well, so that it is
capable to generate a variety of plausible maps. Color assignment for each
class is available in Tab. 4.

| Class IoU |
---|---|---
| mIoU | Free |  [HTML]FFC800Building |  [HTML]FF7832Barrier |  [HTML]375A50 Other |  [HTML]FF1E1EPedestrian |  [HTML]FFF096Pole |  [HTML]FF00FFRoad |  [HTML]AF004B Ground |  [HTML]4B004B Sidewalk |  [HTML]00AF00Vegetation |  [HTML]6496F5Vehicles | IoU
w/o Diffusion | 39.94 | 96.40 | 27.72 | 3.15 | 8.77 | 22.15 | 37.14 | 89.02 | 18.22 | 59.25 | 29.74 | 47.72 | 80.70
Discrete Diffusion (Ours) | 45.83 | 96.00 | 31.75 | 3.42 | 25.43 | 46.22 | 43.32 | 84.57 | 13.01 | 67.50 | 37.45 | 55.46 | 80.61

Table 4: Semantic scene completion results on test set of CarlaSC. The
discriminative learning result with the diffusion model architecture is
denoted as ‘w/o Diffusion’. Values with a difference equal to or greater than
0.5%p are bold.

Figure 4: Qualitative comparison of a deterministic model (w/o diffusion) and
ours (discrete diffusion) on the test split of CarlaSC. The first row shows
the sparse inputs for the scene completion task, and the last row shows the
corresponding ground-truth. Compared to the deterministic model, our
probabilistic model produces more plausible shape and class inference, as
highlighted by the red circles. Note that the both models (w/o diffusion and
discrete diffusion) use the same network architecture. Color assignment for
each class is available in Tab. 4.

###  4.3 Semantic Scene Completion

We use a discrete diffusion model for conditional 3D segmentation map
generation (i.e., SSC). As a baseline model against the diffusion model, we
train a network with an identical architecture by discriminative learning
without a diffusion process. We optimize the baseline with a loss term
$\mathcal{L}=-\sum_{k}w_{k}\mathbf{x}_{k}\log(\tilde{\mathbf{x}}_{k})$, where
$w_{k}$ is a weight for each semantic class. We visualize results from the
baseline and our discrete diffusion model in Fig. 4. Despite the complexities
of the networks being identical, our discrete diffusion model improves mIoU
(i.e., class-wise IoU) up to 5.89%p than the baseline model as shown in Tab.
4. Especially, our method achieves outstanding results in small objects and
fewer frequency categories like ‘pedestrian’, ‘pole’, ‘vehicles,’ and ‘other’.
The qualitative results in Fig. 4 better demonstrate the improvement.

In Tab. 3, we compare our model with existing SSC models whose network
architectures and training strategies are specifically built for the SSC task.
Nonetheless, our diffusion model outperforms LMSCNet [16] and SSCNet [17], in
spite of the simpler architecture and training strategies. Although MotionSC
[9] shows a slightly better result, we speculate that the diffusion
probabilistic model can be improved by extensive future research dedicated to
this field.

##  5 Conclusion

In this work, we demonstrate the extension of the diffusion model to scene-
scale 3D categorical data beyond generating a single object. We empirically
show that our models have impressive generative power to craft various scenes
through a discrete and latent diffusion process. Additionally, our method
provides an alternative view for the SSC task, showing superior performance
compared to a discriminative counterpart. We believe that our work can be a
useful road map for generating 3D data with a diffusion model.

## References

  * [1] A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, and M. Chen, “Point-e: A system for generating 3d point clouds from complex prompts,” 2022. [Online]. Available: https://arxiv.org/abs/2212.08751
  * [2] L. Zhou, Y. Du, and J. Wu, “3d shape generation and completion through point-voxel diffusion,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5826–5835.
  * [3] X. Zeng, A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, and K. Kreis, “Lion: Latent point diffusion models for 3d shape generation,” _arXiv preprint arXiv:2210.06978_ , 2022.
  * [4] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 2256–2265.
  * [5] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6840–6851, 2020.
  * [6] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 8780–8794, 2021.
  * [7] E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling, “Argmax flows and multinomial diffusion: Learning categorical distributions,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 12 454–12 465, 2021.
  * [8] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg, “Structured denoising diffusion models in discrete state-spaces,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 981–17 993, 2021.
  * [9] J. Wilson, J. Song, Y. Fu, A. Zhang, A. Capodieci, P. Jayakumar, K. Barton, and M. Ghaffari, “Motionsc: Data set and network for real-time semantic mapping in dynamic environments,” _arXiv preprint arXiv:2203.07060_ , 2022.
  * [10] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [11] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [12] G. Riegler, A. Osman Ulusoy, and A. Geiger, “Octnet: Learning deep 3d representations at high resolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3577–3586.
  * [13] R. Cheng, C. Agia, Y. Ren, X. Li, and L. Bingbing, “S3cnet: A sparse semantic scene completion network for lidar point clouds,” in _Conference on Robot Learning_. PMLR, 2021, pp. 2148–2161.
  * [14] X. Yan, J. Gao, J. Li, R. Zhang, Z. Li, R. Huang, and S. Cui, “Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 4, 2021, pp. 3101–3109.
  * [15] C. B. Rist, D. Emmerichs, M. Enzweiler, and D. M. Gavrila, “Semantic scene completion using local deep implicit functions on lidar data,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 10, pp. 7205–7218, 2021.
  * [16] L. Roldao, R. de Charette, and A. Verroust-Blondet, “Lmscnet: Lightweight multiscale 3d semantic completion,” in _2020 International Conference on 3D Vision (3DV)_. IEEE, 2020, pp. 111–119.
  * [17] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser, “Semantic scene completion from a single depth image,” _Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [18] C. H. Jo, W. B. Im, and S.-E. Yoon, “In-n-out: Towards good initialization for inpainting and outpainting,” in _The 32nd British Machine Vision Conference, BMVC 2021_. British Machine Vision Association (BMVA), 2021.
  * [19] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 461–11 471.
  * [20] Y.-T. Chen, M. Garbade, and J. Gall, “3d semantic scene completion from a single depth image using adversarial training,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1835–1839.
  * [21] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [22] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in _ACM SIGGRAPH 2022 Conference Proceedings_ , 2022, pp. 1–10.
  * [23] S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, “Vector quantized diffusion model for text-to-image synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 696–10 706.
  * [24] S.-H. Shim, S. Hyun, D. Bae, and J.-P. Heo, “Local attention pyramid for scene image generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7774–7782.
  * [25] W.-C. Fan, Y.-C. Chen, D. Chen, Y. Cheng, L. Yuan, and Y.-C. F. Wang, “Frido: Feature pyramid diffusion for complex scene image synthesis,” _arXiv preprint arXiv:2208.13753_ , 2022.
  * [26] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 684–10 695.
  * [27] A. Van Den Oord, O. Vinyals _et al._ , “Neural discrete representation learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [28] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 12 873–12 883.
  * [29] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8821–8831.
  * [30] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin, “Cylindrical and asymmetrical 3d convolution networks for lidar segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 9939–9948.
  * [31] M. Xu, L. Yu, Y. Song, C. Shi, S. Ermon, and J. Tang, “Geodiff: A geometric diffusion model for molecular conformation generation,” _arXiv preprint arXiv:2203.02923_ , 2022.
  * [32] S. Luo and W. Hu, “Diffusion probabilistic models for 3d point cloud generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2837–2845.
  * [33] M. Hu, Y. Wang, T.-J. Cham, J. Yang, and P. N. Suganthan, “Global context with discrete diffusion in vector quantised modelling for image generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 502–11 511.