# ChangeSim: Towards End-to-End Online Scene Change Detection in Industrial
Indoor Environments

Jin-Man Park1, Jae-Hyuk Jang1, Sahng-Min Yoo, Sun-Kyung Lee, Ue-Hwan Kim,
and Jong-Hwan Kim The authors are with the School of Electrical Engineering,
KAIST, Daejeon, 34141, Republic of Korea. e-mail: {jmpark, jhjang, smyoo,
sklee, uhkim<EMAIL_ADDRESS>1These authors contributed equally to
this work.

###### Abstract

We present a challenging dataset, ChangeSim, aimed at online scene change
detection (SCD) and more. The data is collected in photo-realistic simulation
environments with the presence of environmental non-targeted variations, such
as air turbidity and light condition changes, as well as targeted object
changes in industrial indoor environments. By collecting data in simulations,
multi-modal sensor data and precise ground truth labels are obtainable such as
the RGB image, depth image, semantic segmentation, change segmentation, camera
poses, and 3D reconstructions. While the previous online SCD datasets evaluate
models given well-aligned image pairs, ChangeSim also provides raw unpaired
sequences that present an opportunity to develop an online SCD model in an
end-to-end manner, considering both pairing and detection. Experiments show
that even the latest pair-based SCD models suffer from the bottleneck of the
pairing process, and it gets worse when the environment contains the non-
targeted variations. Our dataset is available at
https://sammica.github.io/ChangeSim/.

##  I Introduction

Scene change detection (SCD), the task of localizing changes given two scenes,
past and present, is one of the most important tasks for patrol robots. SCD
divides into two categories: offline and online. On the one hand, offline SCD
[1, 2, 3, 4] performs two global mappings at two different time-steps;
compares the two reconstructions; and detects changes between the two scenes.
Since offline SCD can detect changes only after the second mapping is
completed, immediate detection is impossible. On the other hand, online SCD
performs frame-level detection on-the-fly during the second mapping.
Therefore, it enables immediate response, which is a crucial prerequisite for
surveillance. In this work, we focus on online SCD.

222 Minor lighting changes with little visual variation included. * Not yet
publicly released. Dataset Images Change annotation Size #change class #sem.
class 3D recon. Environment Target change Non-target change Outdoor VL-CMU-CD
[5] P every frame 1362 images 1 9 - R/street-view structural change weather,
light PCD [6] P every frame 200 images 1 - - R/street-view structural change
weather, light CD2014 [7] P every frame 70,000 images 1 - - R/street-view,
cctv dynamic object - PSCD* [8] P every frame 500 images 2 10 - R/street-view
structural change weather, light CARLA-OBJCD* [9] P every frame 60,000 images
1 10 - S/street-view new/missing object - GSV-OBJCD* [9] P every frame 500
images 1 10 - R/urban area new/missing object weather, light Indoor 3RScan [1]
U final map only 1482 scans of 478 scenes 1 7 ✓ R/household dynamic object
-222 InteriorNet [10] U final map only Millions scans / unknown - 23 ✓
S/household dynamic object light (color, intensity) Langer et al. [2] U final
map only 31 scans of 5 scenes 1 8 ✓ R/household new object -222 Fehr et al.
[3] U final map only 23 scans of 3 scenes 1 - ✓ R/household dynamic object
-222 Ambrus et al. [4] U final map only 88 scans of 1 scenes 1 - ✓ R/office
dynamic object -222 ChangeSim (ours) P&U every frame 80 scans of 10 scenes,
Approx. 130,000 images 4 24 ✓ S/warehouse new/missing/ rotated/replaced object
dusty air, low-illumination

TABLE I: Comparison of previous datasets for scene change detection. We
broadly categorize these datasets as being either in outdoor or indoor. The
Images column indicates whether or not images are provided in a paired way
(i.e., reference image (at time $t_{0}$) and query image (at time $t_{1}$), P:
Paired, U: Unpaired). The Change annotation column indicates the type of
change label (every frame: ground-truths are provided for each frame of a
sequence, final map only: ground-truths are provided for the 3D reconstruction
only). The Environment column indicates its domain (i.e., R: real, S:
synthetic). Note that ChangeSim is the first dataset providing both raw
unpaired images and frame-by-frame change annotations, which are the
prerequisite for online indoor SCD.

Prior to change detection, online SCD goes through a pairing process in which
a reference frame whose camera view matches that of the query (current) frame
gets extracted from the past scene. At the core of the pairing process lies
localization as camera poses play a key role in measuring the similarity
between two frames. In the case of online outdoor SCDs such as SCDs for
street-view [5] and remote sensing [11], the availability of the global
positioning system (GPS) allows robust and accurate pose estimation regardless
of how much the environment has visually changed. Therefore, most outdoor
online SCD datasets assume perfect localization and provide paired image sets
as input rather than pairs of raw image sequences [5, 6, 7, 8, 9].

Figure 1: The proposed ChangeSim dataset, aimed at online scene change
detection in industrial indoor environment (warehouse). While the previous
online SCD datasets provided only paired images (b) and ground-truth labels
(c), the proposed dataset also provides raw unpaired sequences (a) for the
development of end-to-end online SCD.

On the other hand, localization becomes a challenging task for online indoor
SCD since a localization algorithm can only utilize visual (+ inertial)
features but not GPS signals. In addition, indoor environments frequently
contain non-targeted visual changes such as air turbidity or light intensity.
These variations not only degrade change detection performance for well-
matched pairs, but also cause a bottleneck by deteriorating the performance of
visual localization, the preceding stage of actual change detection. Due to
these challenges, online indoor SCD has not displayed much progress over the
past decades despite its substantial significance. Furthermore, the lack of a
corresponding dataset prevents the employment of recent advances in deep
learning. Though a set of indoor SCD datasets was reported in the literature
[1, 2, 3, 4, 10], they merely assume the offline setting—which is not
realistic in practical deployment.

We claim the following properties as an effective online indoor SCD dataset:

  * •

Unpaired images: The dataset should provide raw unpaired image sequences
rather than paired image sets, since localization takes a key role in online
indoor SCD.

  * •

Non-targeted variation: The dataset should include environmental non-targeted
variations (e.g., light condition) because they frequently occur in real-world
scenarios.

  * •

Diversity: The dataset should contain diverse types of environments and change
classes.

The claimed properties of online indoor SCD datasets enhance the robustness
and practicality of online indoor SCD models.

In this work, we introduce ChangeSim, a photo-realistic dataset towards end-
to-end online indoor SCD, as shown in Figure 1. To the best of our knowledge,
we are the first to explicitly define the task of online indoor SCD, collect a
dataset—satisfying the claimed properties—for development and evaluation of
online indoor SCD; and propose baseline models.

First of all, the proposed ChangeSim provides raw unpaired sequences captured
at time $t_{0}$ and $t_{1}$ as input (for end-to-end SCD), requiring models to
perform the pairing process and change detection. At the same time, ChangeSim
includes paired image sets so that conventional pair-based SCD models can also
benefit from ChangeSim. Next, ChangeSim provides controllable non-target
variations, such as air turbidity and light condition. We show in the
experiment that state-of-the-art SCD models struggle in our dataset mainly due
to the bottleneck of visual localization, especially when the non-target
variations are applied in the environment. Finally, ChangeSim provides rich
multi-modal data and annotations, as described in Table I (last row),
including depth, semantic segmentation label, multi-class change segmentation
label, 7-D pose, and 3D reconstructions.

Figure 2: An overview of the ChangeSim environments. Figure 3: ChangSim
dataset statistics. (a) ChangeSim dataset semantic label statistics. (left)
Average probability of occurrence per image. (right) Average area per image.
(b) ChangeSim dataset change label statistics. (left) Average probability of
appearance per image. (right) Average area per image. (c) Comparison of change
detection datasets on blobs per image. For fair comparison, the frequency
values of Panoramic image datasets (TSUNAMI, GSV) are divided by four,
ensuring the same field of view (90∘). (d) Histogram of blob size. (e)
Histogram of average depth per blob.

In summary, the main contributions of this paper are (1) a large SCD dataset
with raw multi-modal data and rich annotations in diverse challenging virtual
indoor environments, (2) a data collection and annotation pipeline for multi-
label SCD, and (3) verification of the proposed ChangeSim by evaluating a
representative SLAM algorithm, semantic segmentation models, and SCD models,
and (4) unveiling insights on future directions of the SCD models.

##  II Dataset Features

We adopt a leading game engine, Unreal Engine 4 (UE4), to implement the
virtual environment for change detection, where UE4 delivers photo-realistic
environments, high-quality textures, dynamic shadows, and a variety of
rendering options. These features not only minimize the gap between real-world
and simulation environments, but also drastically lower the cost of high-
quality ground-truth labels, which enables the creation of large, richly-
annotated datasets. In particular, we have found that most traditional change
detection datasets suffer from their sizes (i.e., lack of labeled changed
objects and images) restraining learning-based methods. We expect that the
following dataset features made by photo-realistic simulation achieve better
coverage of environments and scenarios.

###  II-A Diverse and Realistic Indoor Industrial Domain

We select warehouse environments as the domain of ChangeSim. A warehouse is
appropriate as the domain of change detection benchmark because it is 1) a
large-scale indoor environment compared to the household environment, 2) an
industrial environment that needs periodic surveillance, and 3) a challenging
environment for object change detection due to the densely arranged objects.
ChangeSim includes ten real-world-like warehouse environments, ranging from a
small personal warehouse to a large logistics warehouse, as shown in Figure 2.
ChangeSim contains 24 semantic categories, including domain-specific
industrial objects (e.g., pallet, forklift, pallet_jack, rack, etc.). Figure
3(a) displays the list of semantic categories, and its average occurrence (%)
and area (%) per image. It reveals the long tail-shaped, which is a natural
feature of the real-world.

###  II-B Raw Data for the Whole Pipeline for Online Change Detection

ChangeSim provides two unpaired raw sequence, reference sequence,
$S_{0}=[f_{0}^{1},...,f_{0}^{i},...,f_{0}^{n}]$ and query sequence,
$S_{1}=[f_{1}^{1},...,f_{1}^{j},...,f_{1}^{m}]$, captured at time $t_{0}$ and
$t_{1}$, respectively, where $n$ and $m$ are the number of frames in $S_{0}$
and $S_{1}$, respectively. Given a current frame, $f_{1}^{j}$, an online SCD
model, D, detects changes as follows:

$c=\textbf{{D}}(f_{1}^{j},S_{0},[f_{1}^{0},...,f_{1}^{j-1}]),$ (1)

where $c\in\mathbb{R}^{w\times h\times l}$ is a multi-class change
segmentation. Here, $w$, $h$, and $l$ indicate width, height, and the number
of change classes, respectively. Assuming that D infer $c$ in a two-stage
process, D can be separated into two sub-models, matching frame extractor, E,
and pair-based scene change detector, I, as follows:

$\begin{split}f_{0}^{i}&=\textbf{{E}}(f_{1}^{j},S_{0},[f_{1}^{0},...,f_{1}^{j-1}]),\\\
c&=\textbf{{I}}(f_{0}^{i},f_{1}^{j}),\end{split}$ (2)

where $f_{0}^{i}$ is the extracted frame from $S_{0}$, where the camera view
of $f_{1}^{j}$ best matches that of $f_{0}^{i}$ among frames in $S_{0}$. While
the traditional SCD datasets only provided the pairs, $f_{1}^{j}$ and
$f_{0}^{i}$, focusing only on training and evaluation of I, we generalize
online SCD by providing the raw sequences, $S_{0}$ and $S_{1}$, so that both E
and I are considered. Furthermore, an end-to-end SCD model, D can be developed
leveraging our dataset.

In ChangeSim, there are 80 sequences in total (20 sequences for $S_{0}$, 60
sequences for $S_{1}$), and each sequence contains 500-6,500 frames. Each
frame in $S_{0}$ has synchronized RGB, depth, semantic segmentation image, and
camera pose. A 3D reconstruction is also provided for each reference sequence,
$S_{0}$. The organization of each frame in $S_{1}$ is the same as that in
$S_{0}$, except that a change segmentation label and a matching frame,
$f_{0}^{i}$, are included.

Dataset | Langer et al. [2] | Fehr et al. [3] | Ours
---|---|---|---
# of changed obj. | 8.4 / 260 | 78.3 / 235 |  114.6 / 1146
|  |  |
TABLE II: Average number of changed objects per environment (left), total
number of changed objects in dataset (right).

###  II-C Rich Changes with Novel Multi-class Change Labels

####  II-C1 Multi-class Change Labels

While most SCD datasets provide binary change segmentation labels for the
purpose of binary SCD, ChangeSim provides multi-class change segmentation
labels for four change types: missing, new, replaced, and rotated, as shown in
Figure 4. In the binary SCD, detecting pixel-level change is sufficient, but
for the multi-class SCD, the ability to comprehensively consider the location,
semantics, and shape of an object is required.

####  II-C2 Rich Changes

We analyze our dataset at the levels of image and map to show that our dataset
contains rich changes. At the image-level, as shown in Figure 3(b-e), about
84% of all images contain changes, and each image contains an average of 8%
changed pixels (b). Also, change blobs with fewer than 5,000 pixels are the
most common (d), and the change blobs between 5 and 10 meters in depth are the
most frequent (e). Besides, our dataset provides approximately 10 change blobs
per image, which is the largest compared to the online SCD datasets providing
image-level change labels (c). At the map-level, as summarized in TABLE II, we
compare indoor/offline SCD datasets to our dataset according to an average
number of changed objects per environment (left), and a total number of
objects in the dataset (right). It shows that ChangeSim provides the most
number of changed objects compared to the existing indoor SCD datasets.

(a)

(b)

(c)

(d)

(e)

Figure 4: Examples of changes in ChangeSim. Four categories of novel change
types are defined, such as new (rows 1-2), missing (rows 3-4), replaced (rows
5-6), and rotated (rows 7-8), respectively.  Figure 5: Controllable
environmental variations.

###  II-D Controllable Environmental Variations

Figure 6: An overview of the data collection pipeline, including data
annotation. In (c-d), new, missing, replaced, and rotated objects are marked
as brown, navy, lime, and green, respectively. Best viewed when magnified.

For each reference sequence, $S_{0}$, ChangeSim provides three versions of the
query sequence, $S_{1}$,: normal, dusty-air, and low-illumination, allowing to
directly measure the impact of environmental variations on an online SCD
model. Figure 5 shows examples of each scenario. The normal situation is when
no other change has occurred other than the change of the object. The other
two situations are devised assuming extreme cases that can occur in industrial
environments. In dusty-air, the indoor air becomes dusty and cloudy, making it
challenging to identify distant objects. Also, light is scattered as it passes
through the dusty air. Low-illumination is a situation in which most of the
indoor lighting has been removed. Instead, the camera is equipped with several
flashlights. The field of view is limited to the area illuminated by the
flashlights. Furthermore, the color tone becomes darker, and the texture of
objects changes according to the lighting intensity.

##  III Method

To collect the raw data reflecting real-world characteristics as well as
detailed annotations with minimum human efforts, we design a four-stage
pipeline as shown in Figure 6. The included stages are semantic labeling,
reference sequence capturing (at time $t_{0}$), change labeling, and query
sequence capturing (at time $t_{1}$). The semantic labeling process assigns
class labels to objects in the map based on their names defined by the map
creators (i.e., professional designers). Then, in the capturing process of a
reference sequence, a quadrotor-type drone empowered by AirSim [12] moves
along a hand-drawn trajectory to acquire an RGBD sequence. After that, we
create a modified environment where some of the objects change their state,
and the environment can be controlled to have optional environmental
variations such as dusty-air, low illumination, and minor variation of
trajectories. Finally, in the query sequence capturing process, the drone
traverses the changed environment and acquires the RGBD sequence and ground-
truth labels.

###  III-A Semantic Labeling

Each object in the environment has a unique name defined by professional
designers, usually in the form of ModelingName_Number (e.g., bottle_7), where
ModelingName generally includes semantics and Number is a randomly assigned
integer to distinguish the same objects. Using the form, we define various
candidate names for each semantic category (e.g., candidate names for
canister: can, bottle, container, etc.), and if a given object contains a
candidate name, we label it with the corresponding category. Figure 6(a) shows
the environment from the top-view where the semantic labeling is completed.

###  III-B Reference Sequence Capturing

####  III-B1 Trajectory Drawing

Auto-generated trajectories have the advantage of being able to include
various movements, so they are mainly used for synthetic SLAM datasets.
However, auto-generated trajectories are often unsuitable for surveillance
tasks, such as when the trajectory is unnatural or when it gets too close to
an obstacle. As ChangeSim aims at surveillance-oriented change detection, we
adopt hand-drawing trajectories to minimize this side-effect. We design the
trajectories mainly in the form of traversing one lap through the corridors,
keeping the distance to nearby obstacles greater than a pre-defined threshold
to obtain a view suitable for change detection.

####  III-B2 RGBD-SLAM based 3D Mapping

The multirotor traverses the environment and collects RGBD, 7-D pose, semantic
labels for each frame, and 3D reconstruction for each sequence along the hand-
drawn trajectories (see Figure 6(b)). One of the most latest RGBD-SLAM models,
RTABMAP [13], is employed to collect pose and 3D reconstruction. The data is
collected at a velocity of up to 3 m/s, an acquisition rate of 1-3 fps, and a
clock-speed of 0.03. Each sequence took about 30 minutes to 1 hour.

###  III-C Change Labeling

We limit the candidate semantic classes to be changed to 16 classes out of 24
in total, mainly things (i.e., countable objects), because stuff objects
(i.e., uncountable objects, such as floor, ceiling, etc.) have little
probability of changing within a short time in the real world. Figure 6(c)
shows the result of applying the object changes with the change labels.

Missing: One of the advantages of virtual environments, including the Unreal
Engine, is that the types of rendering are selectable. We disable RGB
rendering and Depth rendering for the selected object. Through this, the
object becomes transparent, that is, it does not appear in the RGB and Depth
images, but is still displayed in the segmentation image, resulting in missing
type of change label. New: We randomly select 3D modelings and place them on
the floor, shelves, or other objects in the environment. Replaced: After
deleting the selected object, another 3D modeling is placed in the same place.
The replaced one may or may not belong to the same semantic class as the
deleted object. Rotated: The selected object is rotated randomly in the x, y,
and z axes.

###  III-D Query Sequence Capturing

####  III-D1 Data acquisition in a changed environment

As shown in Figure 6(d), the multirotor traverses the changed environment and
collects RGB, Depth, estimated semantic & change labels, and 7-D poses. The
poses are obtained employing RTABMAP’s localization mode, where the SLAM
algorithm stops mapping and does only localization (i.e., estimate 7-D poses),
given a 3D reconstruction and its corresponding RGBD sequence. Note that the
multirotor is localized in the 3D map reconstructed from the traverse at time
$t_{0}$, as shown in (d). At this time, the multirotor follows the noise-added
version of the trajectory used in the previous mapping (b), and Gaussian noise
$N(0,0.3^{2})$ is applied to each coordinate of waypoints in the trajectory.

We capture three scenarios for each sequence: normal, dusty-air, low-
illumination. In the case of Dusty-air, the weather control option provided by
AirSim is used (Dust 0.5, Foggy 0.5). For the low-illumination, almost all
lights in the environment are turned off, or their brightness is minimized.
Instead, two forward-facing lights are installed at the front of the
multirotor, ensuring a minimum amount of sight.

####  III-D2 Matching frame extraction

For the current frame, a frame with the closest L1 distance from the reference
sequence is selected using 7-D poses, and then the two frames are paired.
While this method is the most intuitive and straightforward, it highly depends
on the visual SLAM algorithm’s localization performance and is therefore
vulnerable to visual changes. As the pairing becomes inaccurate, the
overlapping area within a pair also reduces, which leads to deterioration in
change detection.

##  IV Baseline and analysis

In order to verify the usefulness of ChangeSim, we propose a straightforward
baseline model of a two-step strategy for online SCD and evaluate each module
of this baseline: 1) matching frame extraction and 2) pair-based scene change
detection. In matching frame extraction, a pairing process is performed using
the estimated pose obtained through the SLAM algorithm. In pair-based scene
change detection, changes are detected using the latest learning-based SCD
models, CSCDNet [8] and ChangeNet [14].

In particular, we measure the impact of environmental variation for each
module and show that the existing models are vulnerable to environmental
variation. Also, in 2), the effect of multi-class on SCD is measured. While
existing models manage to identify changes, they show weakness in classifying
the type of change, implying that they perform as a pixel-level change
detector function rather than perceiving the essence of changes. As an example
of using semantic labels, we show the effect of simple fine-tuning-based
transfer learning on the online SCD. We hope that various approaches to online
SCD will be developed and evaluated without restraint via ChangeSim, towards
end-to-end, multi-class, and visually robust online SCD.

###  IV-A Train-test Split

To establish concrete benchmarks, we split our data into training and test
sets according to the provided maps. We selected 6 maps (12 sequences, 2
sequences for each map) as the training set (rows 1-3, Figure 2) and the
remaining 4 maps (8 sequences, 2 sequences for each map) as the test set (rows
4-5).

###  IV-B Matching Frame Extraction

Given the current frame, visual localization is performed estimating the 7-D
pose through RTABMAP, and the frame with the least L1 distance is extracted
from the reference sequence, $S_{0}$. Table III shows visual localization
performance measured by absolute trajectory error (ATE), according to three
environmental variations for six randomly selected sequences. Assuming that
the degree of environmental variation becomes more severe in the order of
normal$<$dusty-air$<$low-illumination, the visual localization tend to become
more and more inaccurate as the environmental variation becomes more severe.
Figure 7(a) visualizes the estimated trajectory for each of the three
environmental variation scenarios for the two reference sequences, and
examples of paired images are shown in Figure 7(b). The results confirm that
the performance degradation of visual localization is directly linked to low
pairing quality, which severely interferes with the following pair-based SCD.

Figure 7: Qualitative results matching frame extraction for the two reference
sequences (blue). (a) The trajectories from the query sequences drawn as
orange, green and red, respectively. (b) Pairing results are shown.

ATE (m) Seq. 1 Seq. 2 Seq. 3 Seq. 4 Seq. 5 Seq. 6 Normal 0.46 0.83 1.11 1.24
1.33 1.52 Dusty-air 4.43 0.99 1.29 1.34 1.31 1.69 Low-illumination 2.66 1.11
1.32 3.37 4.04 1.93

TABLE III: Impact of environmental variation on visual localization. Absolute
trajectory errors (ATEs) were shown for the six randomly chosen query
sequences. (a) Figure 8: Qualitative comparison of the two representative
pair-based SCD models, ChangeNet [14] and CSCDNet [8] for the ChangeSim
dataset. The RGB images at $t_{0}$ (column 1) are obtained by the matching
frame extraction process. (B) and (M) indicate binary SCD and multi-class SCD,
respectively.

###  IV-C Pair-based Scene Change Detection

Given a pair of the reference and query images, SCD is performed to infer
changes for the present view. We benchmarked the following representative
pair-based SCD models: ChangeNet [14] which consists of siamese networks [15]
and fully convolutional networks (FCNs), and CSCDNet [8] which consists of
siamese networks and correlation networks [16].

For the implementation details, we used the Adam optimizer with the learning
rate of $1\times 10^{-4}$. The ImageNet-pretrained ResNet backbone was used,
unless otherwise specified. We used the batch size of $16$ and minimized
Cross-entropy loss for 20 epochs. Horizontal flips and color jittering were
applied during training. The training time took approximately 24 hours with a
single RTX2080Ti. The inputs to the models are a pair of RGB images, where the
images are resized to 240$\times$320, and 256$\times$256 for the ChangeNet and
CSCDNet, respectively, to meet their input-size requirements. In particular,
the pairs are obtained by the matching frame extraction; thus, both train/test
sets contain mismatched pairs and well-matched pairs. We report macro-F1
(i.e., average of per-class F1) and mean intersection over union (mIoU) for
the following SCD results. The results are from multi-class SCDs unless
otherwise specified.

Binary Multi-class C S mIoU M N Re Ro S mIoU ChangeNet 17.6 73.3 45.4 6.9 9.1
6.6 11.6 80.6 23.0 CSCDNet 22.9 87.3 55.1 6.0 12.4 7.9 17.5 90.2 26.8

TABLE IV: Impact of multi-class learning. IoUs of Binary and Multi-class SCD
are reported (C: change, S: static, M: missing, N: new, Re: replaced, Ro:
rotated).

Normal Dusty-air Low-illumination mIoU macro-F1 mIoU macro-F1 mIoU macro-F1
ChangeNet 23.0 29.8 21.3 26.0 20.9 24.6 CSCDNet 26.8 30.6 24.4 27.2 22.6 24.5

TABLE V: Impact of environmental variations. mIoUs and macro-F1s are reported
for the three cases of environmental variations.

ChangeSim Pascal VOC Cityscapes mIoU fwIoU P. Acc. ($\%$) mIoU mIoU DeepLab-V2
34.65 58.69 73.48 79.7 70.4 DeepLab-V3 39.80 62.95 76.68 85.7 81.3

TABLE VI: Semantic segmentation results for ChangeSim. For an intuitive
comparison, the mIoU performances for PASCAL VOC [17] and CityScapes [18] are
also reported.

Scratch ImageNet ChangeSim mIoU macro-F1 mIoU macro-F1 mIoU macro-F1 ChangeNet
18.8 22.0 21.1 25.0 23.0 29.8 CSCDNet 22.1 25.3 26.1 30.2 26.8 30.6

TABLE VII: Impact of pre-training. SCD results are shown for the three pre-
trained backbone ResNet.

####  IV-C1 Impact of Multi-class Learning

In Table IV, the effect of multi-class learning on SCD is reported. There is a
significant performance drop when performing multi-class SCD, implying that
the models’ abilities to distinguish characteristics of each change category
(missing, new, replaced, or rotated) are poor while they are good at detecting
pixel-level changes. This is because multi-class SCD requires a comprehensive
understanding of changes in semantics, position, size, and pose of an object
to distinguish change categories.

####  IV-C2 Impact of Environmental Variations

Table V shows the impact of environmental variations between the two
sequences, $S_{0}$ and $S_{1}$, on the model performance. Note that
environmental variations are not included in the training set. As expected, it
is confirmed that model performances decrease as the environmental variations
become more severe. In particular, the performance significantly decreases in
low-illumination, which seems to be due to the incorrect pairing process. That
is, two frames of a pair barely overlap their camera view.

####  IV-C3 Impact of Pre-training

We performed a simple fine-tuning-based transfer-learning experiment to
confirm the effectiveness of ChangeSim’s semantic labels for SCD. As an
upstream task, we trained representative semantic segmentation models,
DeepLab-V2 [19] and DeepLab-V3 [20], using semantic segmentation labels. The
original image size of 640$\times$480 was used, and the cross-entropy loss was
minimized using SGD, with the batch size of 3. Each model was trained up to 20
epochs without any augmentation. Table VI shows the evaluation results in the
test set. The performance difference of mIoU between ChangeSim and other
datasets, PASCAL VOC and Cityscapes, is analyzed to be due to a large number
of classes and the appearance characteristics of industrial indoor environment
objects, in particular, the presence of sharp and complex objects such as
wires and racks.

For the downstream task, we took the trained backbone ResNet50 of DeepLab-V3
as the backbone of ChangeNet and CSCDNet, respectively. The remaining
implementation details were set the same as in other experiments. As shown in
Table VII, the SCD performance when using the backbone trained from ChangeSim
was higher than that trained from ImageNet, implying that semantic-awareness
helps SCD.

###  IV-D Qualitative Evaluation

####  IV-D1 Scene Change Detection

Figure 8 visualizes the SCD results of two pair-based SCD models, CSCDNet and
ChangeNet. Binary SCD and multi-class SCD are denoted by B and M,
respectively. Consistent with the results shown in Table IV, CSCDNet caught
the changes better than ChangeNet. In the multi-class SCD setting, the
silhouette of changed objects is better captured compared to the binary SCD.
However, the changes are often misclassified.

####  IV-D2 Semantic Segmentation

We also show the qualitative results for the segmentation networks, as shown
in Figure 9. The results show that the segmentation network trained on the
train split of ChangeSim works well on both a) the test split of ChangeSim and
b) real-word samples, proving that ChangeSim successfully represents the real-
world characteristics.

##  V Conclusion

We have presented ChangeSim, a new dataset for online indoor scene change
detection (SCD), satisfying the following properties for the effective
development and evaluation of online SCDs: unpaired image sets; non-targeted
variation; and diversity. We have verified the applicability of ChangeSim
through a two-stage online SCD model, whose stages are named as 1) matching
frame extraction; and 2) pair-based SCD. In particular, we have found that
existing pair-based SCD models suffer from the bottleneck of 1), mainly due to
the vulnerability of visual localization. This result implies that an end-to-
end and visually robust method should be developed for the application of
online indoor SCD models in practice. We hope that ChangeSim will complement
other datasets and benchmarks, and stimulate the development of the online
indoor SCD, leveraging the claimed properties.

Figure 9: Qualitative test results of the representative semantic segmentation
networks.

## Acknowledgment

This work was supported by Institute for Information & communications
Technology Promotion (IITP) grant funded by the Korea government (MSIT)
(No.2020-0-00440, Development of artificial intelligence technology that
continuously improves itself as the situation changes in the real world).

## References

  * [1] J. Wald, A. Avetisyan, N. Navab, F. Tombari, and M. Nießner, “Rio: 3d object instance re-localization in changing indoor environments,” in _Proc. IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 7658–7667.
  * [2] E. Langer, T. Patten, and M. Vincze, “Robust and efficient object change detection by combining global semantic information and local geometric verification,” in _Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020.
  * [3] M. Fehr, F. Furrer, I. Dryanovski, J. Sturm, I. Gilitschenski, R. Siegwart, and C. Cadena, “Tsdf-based change detection for consistent long-term dense reconstruction and dynamic object discovery,” in _Proc. IEEE International Conference on Robotics and automation (ICRA)_ , 2017, pp. 5237–5244.
  * [4] R. Ambrus, J. Folkesson, and P. Jensfelt, “Unsupervised object segmentation through change detection in a long term autonomy scenario,” in _Proc. IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)_ , 2016, pp. 1181–1187.
  * [5] P. F. Alcantarilla, S. Stent, G. Ros, R. Arroyo, and R. Gherardi, “Street-view change detection with deconvolutional networks,” _Autonomous Robots_ , vol. 42, no. 7, pp. 1301–1322, 2018.
  * [6] K. Sakurada and T. Okatani, “Change detection from a street image pair using cnn features and superpixel segmentation.” in _Proc. British Machine Vision Conference (BMVC)_ , vol. 61, 2015, pp. 1–12.
  * [7] Y. Wang, P.-M. Jodoin, F. Porikli, J. Konrad, Y. Benezeth, and P. Ishwar, “Cdnet 2014: An expanded change detection benchmark dataset,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2014, pp. 387–394.
  * [8] K. Sakurada, M. Shibuya, and W. Wang, “Weakly supervised silhouette-based semantic scene change detection,” in _Proc. IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 6861–6867.
  * [9] R. Hamaguchi, S. Iwase, R. Yokota, Y. Matsuo, and K. Sakurada, “Epipolar-guided deep object matching for scene change detection,” _arXiv preprint arXiv:2007.15540_ , 2020.
  * [10] W. Li, S. Saeedi, J. McCormac, R. Clark, D. Tzoumanikas, Q. Ye, Y. Huang, R. Tang, and S. Leutenegger, “Interiornet: Mega-scale multi-sensor photo-realistic indoor scenes dataset,” in _Proc. British Machine Vision Conference (BMVC)_ , 2018.
  * [11] S. Ji, S. Wei, and M. Lu, “Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 1, pp. 574–586, 2018.
  * [12] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in _Proc. Field and Service Robotics (FSR)_. Springer, 2018, pp. 621–635.
  * [13] M. Labbé and F. Michaud, “Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation,” _Journal of Field Robotics_ , vol. 36, no. 2, pp. 416–446, 2019.
  * [14] A. Varghese, J. Gubbi, A. Ramaswamy, and P. Balamuralidhar, “Changenet: A deep learning architecture for visual change detection,” in _Proc. European Conference on Computer Vision (ECCV) Workshops_ , 2018.
  * [15] J. Mueller and A. Thyagarajan, “Siamese recurrent architectures for learning sentence similarity,” in _Proc. AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016.
  * [16] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in _Proc. IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 2758–2766.
  * [17] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International Journal of Computer Vision_ , vol. 111, no. 1, pp. 98–136, 2015.
  * [18] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 3213–3223.
  * [19] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [20] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proc. European Conference on Computer Vision (ECCV)_ , 2018, pp. 801–818.


# VDG: Vision-Only Dynamic Gaussian for Driving Simulation

Hao Li 1,2,∗, Jingfeng Li1,∗, Dingwen Zhang1,†, Chenming Wu2, Jieqi Shi3, Chen
Zhao2,
Haocheng Feng2, Errui Ding2, Jingdong Wang2 , Junwei Han1 *This work was not
supported by any organization1The School of Automation, Northwestern
Polytechnical University<EMAIL_ADDRESS>of Computer
Vision Technology (VIS), Baidu Inc<EMAIL_ADDRESS>Aerial
Robotics Group, Hong Kong University of Science and Technology, HKSAR, China.
<EMAIL_ADDRESS>
###### Abstract

Dynamic Gaussian splatting has led to impressive scene reconstruction and
image synthesis advances in novel views. Existing methods, however, heavily
rely on pre-computed poses and Gaussian initialization by Structure from
Motion (SfM) algorithms or expensive sensors. For the first time, this paper
addresses this issue by integrating self-supervised VO into our pose-free
dynamic Gaussian method (VDG) to boost pose and depth initialization and
static-dynamic decomposition. Moreover, VDG can work with RGB image input only
and construct dynamic scenes at a faster speed and larger scenes compared with
the pose-free dynamic view-synthesis method. We demonstrate the robustness of
our approach via extensive quantitative and qualitative experiments. Our
results show favorable performance over the state-of-the-art dynamic view
synthesis methods. Additional video and source code will be posted on our
project page at https://3d-aigc.github.io/VDG.

Figure 1: Our proposed VDG is crafted to effectively and uniformly reconstruct
large, dynamic urban scenes as well as predicted poses with only image input.
Here, we showcase our reconstruction results and pose evaluation on KITTI [1]
and Waymo [2] datasets, and further compared with the latest pose-free
methods. The reconstructed visualizations reveal that our method enables us to
model static and dynamic objects without pose priors. Moreover, our method
achieves much more accurate pose prediction than other pose-free methods.

11footnotetext: BRAIN Lab, NWPU, China<EMAIL_ADDRESS>{li0jingfeng, zhangdingwen2006yyy<EMAIL_ADDRESS>Department of Computer Vision Technology (VIS), Baidu Inc., China.
{wuchenming, zhaochen02, fenghaocheng, dingerrui,
<EMAIL_ADDRESS>Aerial Robotics Group, HKUST, Hong
Kong, China<EMAIL_ADDRESS>Equal
Contribution22footnotetext: Corresponding Author

##  I INTRODUCTION

Autonomous driving (AD) holds tremendous promise and could potentially
revolutionize the future of transportation. Ensuring the safety of autonomous
driving systems has become a primary focus of extensive development efforts.
Driving simulation has gained significant prominence as a reliable, secure,
and efficient alternative for training and evaluating AD software and
algorithms [3, 4, 5, 6]. Simulation provides a robust platform that enables
comprehensive testing and assessment in a controlled virtual environment,
which is also a safe and controlled space for training, allowing for extensive
testing of software and algorithms under various scenarios. A significant
challenge in scalable driving simulation is to model and represent large-scale
urban scenes [1, 7, 8].

Recently invented Neural Radiance Fields (NeRF) [9] and 3D Gaussian Splatting
(3D-GS) [10] bridge the gap between computer vision and computer graphics in
the tasks of image-based novel view synthesis and 3D reconstruction. By
incorporating these synthetic views, it becomes feasible to expose the
algorithms to a broader range of viewing conditions, thereby enhancing their
robustness and generalization capabilities. This approach has the potential to
significantly improve the performance of perception algorithms in real-world
scenarios, contributing to the advancement of autonomous driving technology.
Approaches like MARS [6], PNF [11], and Driving-Gaussian [12] decompose scenes
into objects and backgrounds by integrating detection or segmentation
auxiliary tasks. However, obtaining accurate object-level supervision in the
real world is challenging, and manual annotations are always expensive. Other
approaches (PVG [13], EmerNeRF [14], etc.) decompose dynamic and static scenes
in a self-supervision manner, while they still require additional LiDAR point
clouds and accurate pose priors as input.

Besides, an intuitive approach involves pre-processing the scene using
Structure from Motion (SfM) such as COLMAP [15]. Although it can get poses and
coarse point clouds for Gaussian initialization, it still suffers from long-
time consumption. To address this issue, several NeRF-based [16, 17, 18, 19,
20] and 3DGS-based [21, 22, 23] methods have been developed. By introducing a
state-of-the-art depth estimation network [24] or RGB-D cameras and learnable
poses, these methods can train NeRF / Gaussian parameters and recover camera
poses without requiring camera poses. However, these methods are only confined
to static scenes on a small scale and fail to handle forward-moving driving
scenarios. RoDynRF [25] optimizes NeRF, and camera poses in dynamic scenes.
However, it requires additional computation for forward/backward flow, leading
to a significant workload, even with just a dozen images, thus making it
difficult to apply in scalable driving simulations.

Our work aims to tackle the aforementioned issues by proposing the Vision-only
Dynamic Gaussian (VDG), which can construct entire dynamic driving scenes
represented by 3DGS [26] with only vision input (i.e., pose-free). In
particular, our method introduces a self-supervised visual odometry approach
that yields precise camera pose estimation. Leveraging its self-supervised
nature, our method additionally provides dense depth estimation, which is on
par with the scale of pose estimation. This dense depth estimation plays a
crucial role in Gaussian initialization, enhancing the effectiveness of our
approach. Furthermore, we propose a scene decomposition step in our framework
that separates scenes into static and dynamic components. To achieve improved
results, we parameterize and update the poses accordingly. Since our framework
employs a self-decomposing approach, decomposing scenes without ground-truth
poses becomes challenging. To address this, we leverage motion masks generated
by the visual odometry and introduce a motion-mask supervised mechanism. This
mechanism boosts the network’s ability to identify dynamic objects within the
scenes. We summarize the contributions of this paper as follows.

  * •

We propose a novel method, VDG, that introduces self-supervised visual
odometry for camera pose and dense depth estimation, enabling training dynamic
Gaussian scenes for driving simulation with vision-only inputs without pose
priors.

  * •

We further incorporate a motion-mask supervised mechanism to decompose scenes
into static and dynamic components, improving the network’s ability to
identify dynamic objects.

  * •

We conduct comprehensive experiments on urban scene datasets (i.e. Waymo and
KITTI), demonstrating that our VDG method surpasses state-of-the-art baselines
and delivers superior quantitative and qualitative results.

##  II Related Work

###  II-A Novel View Synthesis for Driving Simulation

Recently, there have been significant advancements in scene reconstruction and
novel view synthesis, thanks to advanced implicit neural techniques [9] and
explicit representation methods [10]. These techniques have played a pivotal
role in various applications, including autonomous driving [2], auto
navigation [27], high-fidelity data generation [28], and digital maps [29].
Simultaneously, it is important to acknowledge that our world exhibits
inherent dynamic and complex characteristics in both spatial and temporal
dimensions, particularly in urban scenes. NeRF-based techniques, such as [30,
31, 32, 14, 33], have extended NeRF’s static scene assumption to dynamic
environments. However, these methods face various challenges, such as handling
large-scale scenes, long training times, and the absence of geometric priors
regarding temporal influences on scenes.

More recently, 3D Gaussian Splatting (3D-GS) [26] has introduced innovative
explicit 3D scene representation variants allowing fast training and real-time
rendering. Building upon these variants, Driving-Gaussian [12] and PVG [13]
facilitate the decomposition of static and dynamic Gaussians by incorporating
LiDAR priors. Additionally, Street-Gaussians [34] introduces 3D object
detection to identify and decompose dynamic objects within scenes. However,
the above methods require additional inputs, such as precise camera parameters
and expensive LiDAR hardware. In contrast, our method achieves comparable
performance in reconstructing urban scenes using only collected images as
input.

###  II-B Pose-free Novel View Synthesis

Accurate camera poses are essential for both NeRF-based and 3DGS-based methods
in novel view synthesis and image reconstruction. The initial attempt towards
pose-free novel view synthesis was iNeRF [18], which estimated poses by
matching keypoints using a pre-trained neural radiance field model. Another
method, Nope-NeRF [16], improved pose estimation by incorporating monocular
depth estimation to constrain relative poses between images. However, these
NeRF-based methods struggle with large-scale scenes due to inaccurate poses.
To address this limitation, $\mathrm{F}^{2}$-NeRF and LocalRF split the scene
into multiple blocks, utilizing several NeRF models to reconstruct the entire
scene. Nonetheless, these methods still suffer from slow training speed and
lack geometric priors.

More recently, various works [35, 10] have leveraged Structure-from-Motion
(SfM) [15] to solve camera poses and initialize Gaussian points.
Unfortunately, this preprocessing step incurs additional computational burden.
Therefore, several pose-free 3D-GS methods, such as SplaTAM [22], CF-3DGS
[21], GS-SLAM [36], and others, have introduced RGBD or off-the-shelf depth
estimation networks for Gaussian initialization and parameterized poses for
pose optimization. However, these strategies face challenges in reconstructing
large-scale urban scenes, particularly when dynamic objects disrupt the pose
regression. Furthermore, handling large scenes and free camera trajectories
remains challenging for learnable poses without scene decomposition. In
contrast to previous methods, our VDG approach enables the reconstruction of
entire urban scenes and conducts novel view synthesis using only input images.

##  III Preliminary

3D Gaussian Splatting (3D-GS) [26] represents a 3D scene as a collection of 3D
Gaussian primitives
$\mathbb{G}=\\{(\mu_{k},\Sigma_{k},\alpha_{k},S_{k})\\}^{K}$. Each Gaussian
primitive consists of a mean $\mu_{k}$, a covariance $\Sigma_{k}$, an opacity
$\alpha_{k}$, and spherical harmonics coefficients $S_{k}$. These primitives
parameterize the 3D radiance field of the scene through a tiled-based
rasterization process; 3DGS facilitates real-time alpha blending of numerous
Gaussians to render novel-view images.

The 3D primitives are projected to 2D screen space and rasterized using
$\alpha$-blending. The $\alpha$-blending weights are given as: $\alpha=oG$
with the projected Gaussian contribution on pixel $(x,y)$ given as:

$G(x,y)=e^{-\frac{1}{2}([x,y]^{T}-\mu^{\prime})^{T}\Sigma^{\prime-1}([x,y]^{T}-\mu^{\prime})}$
(1)

where $\mu^{\prime}$ and $\Sigma^{\prime}$ are the projected 2D mean and
covariance matrix, respectively. The combined effect of converting SHs to per-
view color values and $\alpha$-blending them recreates the appearance of the
captured scene.

##  IV Proposed Method

Figure 2: The proposed VDG. (a) VDG Initialization: uses the off-the-shelf VO
network $\mathcal{P}(\cdot)$, $\mathcal{M}(\cdot)$, and $\mathcal{D}(\cdot)$
to estimate the global poses $T_{t}$, motion masks $M_{t}$, and depth maps
$D_{t}$ (see Sec. IV-A1). Given poses $T_{t}$ and corresponding depth maps
$D_{t}$, we project the depth maps into 3D space to initialize the Gaussian
points
$G^{k}_{t}=\\{\tilde{\mu}^{k}_{t},\Sigma^{k},\widetilde{\alpha}^{k}_{t},S^{k}\\}$.
Note that the velocity $v$ of each Gaussian is set to 0 (see Sec. IV-A2). (b)
VDG Training Procedure: Given initialized Gaussians $G^{k}_{t}$, we train our
VDG using RGB and depth supervision (see Sec. IV-A3). Moreover, we apply
motion mask supervision to decompose static and dynamic scenes (Sec. IV-B). In
the end, we adopt a training strategy to refine vo-given poses $T_{t}$ (Sec.
IV-C).

Given an arbitrary sequence of unposed images alongside their corresponding
timestamps $\left\\{(\mathit{I}_{i},t_{i})\mid i=1,\cdots,N\right\\}$ in an
unbounded scene(i.e. Autonomous Driving), VDG aims to achieve precise 3D
reconstruction and synthesize novel view at any desired timestamp $t$ and
camera pose $[\mathrm{R},\mathrm{t}]$. To accomplish the above objective, we
develop VDG that leverages self-supervised VO for accurate pose and monocular
depth estimation for our Gaussian initialization (Sec. IV-A). Besides, motion
supervision is proposed to decompose dynamic and static scenes for better
reconstruction (Sec. IV-B). In the end, the training strategy and optimization
for large-scale scenarios are introduced in sec. IV-C.

###  IV-A Overall Framework

####  IV-A1 Off-the-Shelf VO Inference

Given the advantage that self-supervised VO uses a photometric mechanism to
generate depth and pose, the predicted depth and pose are aligned in the same
scale, which is necessary for Gaussian initialization. Therefore, with two
adjacent images $\mathit{I}_{t},\mathit{I}_{t-1}$, the self-supervised VO [37]
are used to estimate their relative pose $T_{t\rightarrow t-1}$, motion mask
$M_{t\rightarrow t-1}$ and $I_{t}$’s depth map $D_{t}$ via ego-motion pose
network $\mathcal{P}$, depth network $\mathcal{D}$ and motion mask
$\mathcal{M}$:

$\displaystyle T_{t\rightarrow t-1}$
$\displaystyle=\mathcal{P}(I_{t},I_{t-1}),\quad$ $\displaystyle
T_{t\rightarrow t-1}$ $\displaystyle\in\mathbb{R}^{3\times 4}$ (2)
$\displaystyle M_{t\rightarrow t-1}$
$\displaystyle=\mathcal{M}(I_{t},I_{t-1}),\quad$ $\displaystyle
M_{t\rightarrow t-1}$ $\displaystyle\in\mathbb{R}^{H\times W}$ $\displaystyle
D_{t}$ $\displaystyle=\mathcal{D}(I_{t}),\quad$ $\displaystyle D_{t}$
$\displaystyle\in\mathbb{R}^{H\times W}$

With relative pose $T_{t\rightarrow t-1}$ from the previous time, one might
easily infer the absolute pose $[\mathrm{R}_{t},\mathrm{t}_{t}]$.

####  IV-A2 Gaussian Initialization on Single View

After that, for each time step $t$, we use the predicted depth map $D_{t}$ and
the corresponding absolute pose $[\mathrm{R}_{t},\mathrm{t}_{t}]$ to
initialize 3DGS $\mathbb{G}_{t}$ with points lifted from monocular depth,
leveraging camera intrinsic and orthogonal projection, instead of the widely
used SfM points.

####  IV-A3 Network Training

With the initialized Gaussian points $\mathbb{G}_{t}$, pose
$[\mathrm{R}_{t},\mathrm{t}_{t}]$, and timestamp $t$, we then train our
dynamic Gaussian and further optimize our pose given the fact that the process
is differentiable. We adopt a motion supervision mechanism during the training
to decompose static and dynamic Gaussians better using motion mask
$M_{t\rightarrow t-1}$. Ultimately, a training strategy is further proposed to
maintain our geometry representation during pose optimization.

###  IV-B Static-Dynamic Scenes Model and Decomposition

Traditional 3D-Gaussian cannot model time-varying scenes like urban scenes,
which possess many dynamic objects Therefore, following the idea of PVG [13],
our proposed VDG decomposes static background and dynamic objects in a self-
supervision manner. It modifies the traditional Gaussian’s mean $\mu_{k}$ and
opacity $\alpha_{k}$ to be time-dependent functions centered around the life
peak $\tau$, denoted as $\tilde{\mu}(t)$ and $\tilde{\alpha}(t)$. Formally,
each Gaussian $G_{k}(t)$ can be expressed as:

$\displaystyle G^{k}_{t}$
$\displaystyle=\\{\tilde{\mu}^{k}_{t},\Sigma^{k},\widetilde{\alpha}^{k}_{t},S^{k}\\},$
(3) $\displaystyle\widetilde{\mu}^{k}_{t}$
$\displaystyle=\mu^{k}+\frac{l}{2\pi}\cdot\sin\left(\frac{2\pi(t-\tau)}{l}\right)\cdot
v^{k},$ $\displaystyle\widetilde{\alpha}^{k}_{t}$
$\displaystyle=\alpha^{k}\cdot e^{-\frac{1}{2}(t-\tau)^{2}\beta^{-2}},$

where hyper-parameter $l$ represents the cycle length, serving as the scene
prior, learnable parameters
$v_{k}=\left.\frac{\mathrm{d}\tilde{\mu}(t)}{\mathrm{d}t}\right|_{t=\tau}\in\mathbb{R}^{3}$
denotes the instant velocity at time $\tau$.

However, unlike pose-given methods like PVG, decomposing dynamic objects and
estimating our poses simultaneously can be challenging. It is caused by two
factors: (1) changes in observation can be attributed to either camera motion
or the Gaussians’ ability to model view-dependent appearances, which disturbs
the parameterized pose optimization with photometric supervision; (2) The
estimated poses provided by VO are still rough. To this end, we need a more
explicit way to guide our VDG in decomposing static and dynamic scenes.
Intuitively, we propose motion mask supervision $\mathcal{L}_{motion}$ based
on the motion masks $M\in\\{0,1\\}$ produced by VO. Given the rendered
velocity map $V$, instead of directly using L2 loss, we convert the smooth
velocity $v_{k}\in[0,1]$ from Eq. 3 to a discrete velocity
$\hat{v}_{k}\in\\{0,1\\}$ before the L2 loss. The formation is shown below:

$\displaystyle\hat{V}=V+\not\nabla\left[\left(V>v_{thr}\right)-V\right]$ (4)
$\displaystyle\mathcal{L}_{motion}=||\hat{V}-M||^{2}_{2},$

where $v_{thr}$ is the hyper-parameters that threshold the static Gaussians
and dynamic Gaussians, and $\not\nabla[\cdot]$ is the stop-gradient operator.
It is essential to make diversity between the static and dynamic Gaussians
while keeping the proper distribution of dynamic Gaussians on different $v$.
Notably, as shown in Fig. 3, directly adopting $\mathcal{L}_{motion}$ could
lead to a negative impact on the final render quality. Because the network
tends to overfit the dynamic supervision by changing the parameters of the
Gaussian point (e.g. location, opacity, and colors).

Figure 3: Illustration of Frozen Gaussian Parameters.

###  IV-C Training Strategy

In the early stage of the training, we use $[\mathrm{R}_{t},\mathrm{t}_{t}]$
as a pose before training our Gaussian model. However, imperfect poses will
limit the final render quality of our Gaussian model. Therefore, at the end of
the training phase, we adopt pose refinement by translating poses
$[\mathrm{R}_{t},\mathrm{t}_{t}]$ into learnable parameters
$[\tilde{\mathrm{R}}_{t},\tilde{\mathrm{t}}_{t}]$ to achieve more precise
scene reconstruction, as shown in Fig. 1. Moreover, to eliminate the negative
impact of moving objects for pose optimization, we only use static parts (e.g.
Gaussians with low velocity $\hat{V}$) to optimize our refined poses.

Figure 4: Qualitative comparison on KITTI dataset regarding pose accuracy and
rendered quality. Our method outperforms other baselines, even in cases where
pose estimation is relatively poor.

TABLE I: Quantitative performance of novel view synthesis on the Waymo Open Dataset [2] and KITTI benchmark [1]. ’-’ means SplaTAM cannot rendering original resolution image on a single NVIDIA V100 GPU. Methods |  Waymo Open Dataset [2] |  KITTI Dataset [1]
---|---|---
FPS | Image Reconstruction | Novel View Synthesis | FPS | Image Reconstruction | Novel View Synthesis
PSNR $\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
Pose-free ✘ | StreetSurf | 0.097 | 27.00 | 0.850 | 0.361 | 21.83 | 0.776 | 0.416 | 0.370 | 24.14 | 0.819 | 0.257 | 22.48 | 0.768 | 0.304
SUDS | 0.008 | 28.83 | 0.805 | 0.289 | 21.83 | 0.656 | 0.405 | 0.040 | 28.31 | 0.876 | 0.185 | 22.77 | 0.797 | 0.171
EmerNeRF | 0.053 | 28.11 | 0.786 | 0.373 | 25.92 | 0.763 | 0.384 | 0.280 | 26.95 | 0.828 | 0.218 | 25.24 | 0.801 | 0.237
3D-GS | 63.00 | 27.99 | 0.866 | 0.293 | 25.08 | 0.822 | 0.319 | 125.0 | 21.02 | 0.811 | 0.202 | 19.54 | 0.776 | 0.224
PVG | 50.00 | 32.46 | 0.910 | 0.229 | 28.11 | 0.849 | 0.279 | 59.00 | 31.54 | 0.927 | 0.083 | 26.63 | 0.885 | 0.127
Pose-free ✔ | Nope-NeRF | 0.516 | 25.26 | 0.756 | 0.536 | 21.75 | 0.738 | 0.547 | 0.518 | 23.54 | 0.756 |  0.325 | 22.54 | 0.747 | 0.327
LocalRF | 0.046 | 25.20 | 0.800 |  0.409 | 23.77 | 0.775 | 0.425 | 0.143 | 22.56 | 0.861 |  0.136 | 23.48 | 0.864 | 0.135
SplaTAM | - | - | - | - | - | - | - | 360.0 | 17.51 | 0.716 | 0.291 | 17.14 | 0.691 | 0.313
RoDynRF | 0.010 | 16.03 | 0.617 | 0.615 | 14.94 | 0.595 | 0.627 | 0.040 | 26.04 | 0.817 | 0.239 | 17.47 | 0.544 | 0.371
Ours | 44.54 | 31.65 | 0.900 |  0.250 | 26.68 | 0.817 | 0.324 | 60.86 | 31.61 | 0.933 | 0.074 | 25.29 | 0.851 | 0.152

Figure 5: Qualitative comparison of our approach and other baselines,
including Pose-free methods, GT pose needed methods and GT images on Waymo
Open Dataset[2]. We show two case views of synthesis results under different
scenes.

##  V EXPERIMENTS

###  V-A Experimental Setup

Datasets. We conduct detailed experiments on two datasets, KITTI benchmark[1]
and Waymo Open Dataset[2], with large amounts of moving objects, especially
fast ego-camera motion, and complex light conditions. show the quantitative
results in Tab<EMAIL_ADDRESS>We selected 4 dynamic scenes in Waymo and 3 dynamic scenes
in KITTI to evaluate pose accuracy and novel view synthesis. We adopt a novel
view for each scene by selecting every 4th frame in sequences. And we use
original resolution for training and rendering.

Baselines. Our approach primarily surpasses pose-free methods and is
comparable in performance to other methods with accurate poses. (a) pose-free
methods: NoPe-NeRF[16], LocalRF[38], RoDynRF[25], SplaTAM[22]. (b) other
methods with GT poses: EmerNeRF[14], StreetSurf[39], SUDS[40], 3D-GS[26],
PVG[13]. Furthermore, we evaluate image reconstruction, novel view synthesis,
and estimated pose accuracy tasks. Due to the length of AR dataset sequences
and the forward motion of cameras in autonomous driving scenes, COLMAP often
struggles to localize.

Figure 6: Qualitative comparison of our approach and other baselines,
including Pose-free methods, GT pose needed methods and GT images on KITTI
benchmark[1]. We show two case views of synthesis results under different
scenes.

Metrics. We follow the evaluation protocol like NoPe-NeRF. We evaluate the
model using two primary tasks: novel view synthesis and camera pose
estimation. For novel view synthesis, we use Peak Signal-to-Noise Ratio
(PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual
Image Patch Similarity (LPIPS). We evaluate camera pose accuracy using
standard evaluation metrics, including the absolute trajectory error (ATE) and
relative pose error (RPE). ATE measures the absolute difference between
predicted camera positions and ground truth positions. RPE measures relative
rotation and translation errors between consecutive images. The estimated
trajectory is aligned with ground truth using Sim(3) with 7 DoF.
Implementation Details. We initialize Gaussian points exclusively using the
point cloud derived from the estimated monocular depth map projection. The
color of the point cloud is acquired by mapping the points onto the
corresponding image plane and retrieving the corresponding pixel values. Our
train process involves 30k/40k iterations with the Adam optimizer[41] that
requires around 2 hours. All experiments are conducted on a single NVIDIA V100
GPU.

###  V-B Evaluation on Dynamic View Synthesis

Figure 7: Qualitative comparisons of other baselines and our method in terms
of RGB, depth, binary mask, and static RGB.

TABLE II: Pose accuracy on the Waymo Open [2] and KITTI [1] datasets. Note that the unit of $RPE_{r}$ is in degrees, ATE is in the ground truth scale and $RPE_{t}$ is scaled by 100. | scenes |  | Ours |  | Nope-NeRF |  | RoDynRF |  | LocalRF
---|---|---|---|---|---|---|---|---|---
|  | $RPE_{t}\downarrow$ | $RPE_{r}\downarrow$ | $ATE\downarrow$ |  | $RPE_{t}\downarrow$ | $RPE_{r}\downarrow$ | $ATE\downarrow$ |  | $RPE_{t}\downarrow$ | $RPE_{r}\downarrow$ | $ATE\downarrow$ |  | $RPE_{t}\downarrow$ | $RPE_{r}\downarrow$ | $ATE\downarrow$
Waymo | 0017085 |  | 2.619 | 0.049 | 0.112 |  | 271.658 | 1.921 | 8.366 |  | 235.341 | 1.799 | 2.400 |  | 205.941 | 1.745 | 0.898
0145050 |  | 4.563 | 0.059 | 0.175 |  | 208.550 | 1.760 | 8.885 |  | 217.774 | 0.240 | 2.580 |  | 185.714 | 0.253 | 0.567
0147030 |  | 2.040 | 0.031 | 0.069 |  | 206.084 | 0.222 | 7.041 |  | 200.030 | 0.134 | 2.465 |  | 169.140 | 0.132 | 0.539
0158150 |  | 1.829 | 0.039 | 0.080 |  | 178.742 | 0.536 | 3.417 |  | 254.535 | 1.826 | 5.380 |  | 139.159 | 0.620 | 0.450
Avg. |  | 2.763 | 0.045 | 0.109 |  | 216.259 | 1.110 | 6.941 |  | 235.999 | 1.030 | 4.698 |  | 174.989 | 0.688 | 0.614
KITTI | 0001 |  | 16.723 | 0.152 | 0.074 |  | 9.501 | 0.078 | 0.457 |  | 10.338 | 0.066 | 0.469 |  | 5.590 | 0.047 | 0.201
0002 |  | 4.536 | 0.059 | 0.175 |  | 3.251 | 0.038 | 0.194 |  | 3.335 | 0.017 | 0.198 |  | 1.683 | 0.019 | 0.059
0006 |  | 0.127 | 0.018 | 0.006 |  | 0.162 | 0.078 | 0.007 |  | 0.133 | 0.026 | 0.007 |  | 0.121 | 0.018 | 0.006
Avg. |  | 7.262 | 0.069 | 0.058 |  | 4.305 | 0.065 | 0.219 |  | 4.602 | 0.036 | 0.225 |  | 2.465 | 0.028 | 0.089

In this section, we primarily analyze the dynamic and static components
decoupling effects of our method and the images synthesized through our
approach are significantly clearer than those produced by other pose-free
methods. As illustrated in Fig. 7, The first two columns show the comparative
results of geometric in novel view, and the last three columns demonstrate the
ability of our method to distinguish between dynamic Gaussian points and
static Gaussian points.

Unlike other w/ GT pose methods in which the camera poses of testing views are
given, pose-free methods freeze the pre-trained NeRF/Gaussian Splatting model
that trained on the training views and optimize the testing view’s camera
poses via training loss. TableLABEL:view_synthesis_table presents the
comparison results of our approach with other baselines[16, 25, 38] on the
KITTI and Waymo dataset. Our method significantly outperforms other baselines
in all metrics, including rendering speed and rendering quality. Notably,
NeRF-based methods take significantly longer training time than our approach,
and their rendering results are much worse than ours. Our method achieves
comparable performance with well-known methods, even 3DGS and SUDS, which rely
on LiDAR point cloud. In this subsection, we show some visualization results
with baselines on KITTI and Waymo Dynamic view synthesis in Fig. 6 and Fig. 5.

Figure 8: Qualitative comparsion for camera pose estimation, including camera
position(bottom) and camera orientation(top).

###  V-C Evaluation on Camera Poses Estimation

We follow the post-process of learned camera poses like Nope-NeRF and compare
learned poses with ground truth poses of training views. We compare our method
with 3 pose-free baselines in two datasets that provide ground truth poses for
reference, including Nope-NeRF[16], RoDynRF[25], and LocalRF[38]. Nope-NeRF
and RoDynRF jointly optimize the radiance field and camera pose to achieve
satisfactory results in small-scale scenes, especially RoDynRF, trained in 12
images. LocalRF represents a more robust pose estimation using a progressive
scheme rather than only a single radiance field. The quantitative camera pose
evaluation results are summarized in Tab. II. Our approach surpasses existing
NeRF-based pose-free approaches in inference speed and effectiveness.
Estimating accurate camera poses in unbounded urban street scenes relies
solely on loss functions, which is challenging. So, we demonstrate the
effectiveness and robustness of our method on KITTI and Waymo in Fig. 8, which
present more complex and large camera motions. We also show the quantitative
results of estimated camera poses in Tab. II. We find that the estimated poses
of our approach in Waymo are more accurate than others. Even though our
estimated camera poses are slightly worse, we still achieve relatively good
results due to our robust Gaussian model, depicted in Fig. 4.

Figure 9: Visual ablation results on Waymo Open Dataset. The results indicate
that utilizing $\mathcal{L}_{motion}$ improves the quality of decoupling
dynamic and static elements.

###  V-D Ablation Study

We conduct an ablation study to indicate the impact of our primary component
on the Waymo Open Dataset. Visualization results of the influence of
$\mathcal{L}_{motion}$ motion mask loss is shown in Fig. 9. As Tab. III shows,
we further investigate the impact of the weight of $\mathcal{L}_{motion}$ on
the results. In Fig. 9, the first column illustrates the effectiveness of
using an estimated monocular depth map to supervise our VDG Gaussians set
geometric representation; The second column shows the visualization results of
dynamic-static decomposition, and the third column demonstrates that we can
distinguish dynamic Gaussians points and static Gaussians points.

| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
0.50 * $\mathcal{L}_{motion}$ | 26.67 | 0.848 | 0.279
0.05 * $\mathcal{L}_{motion}$ | 25.79 | 0.816 | 0.308
0.10 * $\mathcal{L}_{motion}$ | 26.923 | 0.840 | 0.278

TABLE III: Ablation studies of the weight of  $\mathcal{L}_{motion}$ on the
Waymo Open Dataset.

##  VI Conclusion

Unlike previous methods that need LiDAR or RGB-D sensors, we propose a novel
approach for dynamic driving simulation from multi-camera monocular videos.
Our approach utilizes the projection of the estimated monocular depth map
rather than random initialization. By integrating self-supervised visual
odometry, monocular depth estimation, and $\mathcal{L}_{motion}$ motion mask
loss, our model significantly outperforms the state-of-the-art methods on the
Waymo Open Dataset and KITTI benchmark in rendering quality and dynamic-static
decomposition. With the proposed model designs, we demonstrate that our model
cannot reconstruct precise geometry.

## References

  * [1] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _CVPR_.
  * [2] “Waymo open dataset: An autonomous driving dataset,” 2019.
  * [3] W. Li, C. Pan, R. Zhang, J. Ren, Y. Ma, J. Fang, F. Yan, Q. Geng, X. Huang, H. Gong, _et al._ , “Aads: Augmented autonomous driving simulation using data-driven algorithms,” _Science robotics_ , 2019.
  * [4] A. Amini, I. Gilitschenski, J. Phillips, J. Moseyko, R. Banerjee, S. Karaman, and D. Rus, “Learning robust control policies for end-to-end autonomous driving from data-driven simulation,” _IEEE Robotics and Automation Letters_ , 2020.
  * [5] A. Amini, T.-H. Wang, I. Gilitschenski, W. Schwarting, Z. Liu, S. Han, S. Karaman, and D. Rus, “Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles,” in _ICRA_ , 2022.
  * [6] Z. Wu, T. Liu, L. Luo, Z. Zhong, J. Chen, H. Xiao, C. Hou, H. Lou, Y. Chen, R. Yang, Y. Huang, X. Ye, Z. Yan, Y. Shi, Y. Liao, and H. Zhao, “Mars: An instance-aware, modular and realistic simulator for autonomous driving,” _CICAI_ , 2023.
  * [7] Z. Li, C. Wu, L. Zhang, and J. Zhu, “Dgnr: Density-guided neural point rendering of large driving scenes,” _IEEE Transactions on Automation Science and Engineering_ , 2024.
  * [8] Z. Li, Y. Zhang, C. Wu, J. Zhu, and L. Zhang, “Ho-gaussian: Hybrid optimization of 3d gaussian splatting for urban scenes,” _arXiv preprint arXiv:2403.20032_ , 2024.
  * [9] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _Communications of the ACM_ , 2021.
  * [10] S.-F. Chng, S. Ramasinghe, J. Sherrah, and S. Lucey, “Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation,” in _ECCV_ , 2022.
  * [11] A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. J. Guibas, A. Tagliasacchi, F. Dellaert, and T. Funkhouser, “Panoptic neural fields: A semantic object-aware neural scene representation,” in _CVPR_ , 2022.
  * [12] X. Zhou, Z. Lin, X. Shan, Y. Wang, D. Sun, and M.-H. Yang, “Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes,” in _CVPR_ , 2024.
  * [13] Y. Chen, C. Gu, J. Jiang, X. Zhu, and L. Zhang, “Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering,” _arXiv preprint arXiv:2311.18561_ , 2023.
  * [14] J. Yang, B. Ivanovic, O. Litany, X. Weng, S. W. Kim, B. Li, T. Che, D. Xu, S. Fidler, M. Pavone, _et al._ , “Emernerf: Emergent spatial-temporal scene decomposition via self-supervision,” in _ICLR_ , 2024.
  * [15] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in _CVPR_ , 2016.
  * [16] W. Bian, Z. Wang, K. Li, J.-W. Bian, and V. A. Prisacariu, “Nope-nerf: Optimising neural radiance field with no pose prior,” in _CVPR_ , 2023.
  * [17] C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey, “Barf: Bundle-adjusting neural radiance fields,” in _ICCV_ , 2021.
  * [18] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin, “inerf: Inverting neural radiance fields for pose estimation,” in _IROS_ , 2021.
  * [19] Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu, “Nerf–: Neural radiance fields without known camera parameters,” _arXiv preprint arXiv:2102.07064_ , 2021.
  * [20] Y. Fu, I. Misra, and X. Wang, “Mononerf: Learning generalizable nerfs from monocular videos without camera poses,” in _NeurIPS_ , 2023.
  * [21] Y. Fu, S. Liu, A. Kulkarni, J. Kautz, A. A. Efros, and X. Wang, “Colmap-free 3d gaussian splatting,” in _CVPR_ , 2024.
  * [22] N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten, “Splatam: Splat, track & map 3d gaussians for dense rgb-d slam,” in _CVPR_ , 2024.
  * [23] J. Hu, X. Chen, B. Feng, G. Li, L. Yang, H. Bao, G. Zhang, and Z. Cui, “Cg-slam: Efficient dense rgb-d slam in a consistent uncertainty-aware 3d gaussian field,” _arXiv preprint arXiv:2403.16095_ , 2024.
  * [24] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, “Depth-supervised nerf: Fewer views and faster training for free,” in _CVPR_ , 2022.
  * [25] Y.-L. Liu, C. Gao, A. Meuleman, H.-Y. Tseng, A. Saraf, C. Kim, Y.-Y. Chuang, J. Kopf, and J.-B. Huang, “Robust dynamic radiance fields,” in _CVPR_ , 2023.
  * [26] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” _ACM Transactions on Graphics_ , July 2023.
  * [27] O. Kwon, J. Park, and S. Oh, “Renderable neural radiance map for visual navigation,” in _CVPR (CVPR)_ , June 2023, pp. 9099–9108.
  * [28] R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P. P. Srinivasan, D. Verbin, J. T. Barron, B. Poole, and A. Holynski, “Reconfusion: 3d reconstruction with diffusion priors,” _arXiv_ , 2023.
  * [29] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _CVPR_ , 2020.
  * [30] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, “D-nerf: Neural radiance fields for dynamic scenes,” in _CVPR_ , 2021.
  * [31] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz, “Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields,” _arXiv preprint arXiv:2106.13228_ , 2021.
  * [32] T. Wu, F. Zhong, A. Tagliasacchi, F. Cole, and C. Oztireli, “D^ 2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video,” _NeurIPS_ , 2022.
  * [33] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” in _CVPR_ , 2021, pp. 7210–7219.
  * [34] Y. Yan, H. Lin, C. Zhou, W. Wang, H. Sun, K. Zhan, X. Lang, X. Zhou, and S. Peng, “Street gaussians for modeling dynamic urban scenes,” _arXiv preprint arXiv:2401.01339_ , 2024.
  * [35] Z. Yu, A. Chen, B. Huang, T. Sattler, and A. Geiger, “Mip-splatting: Alias-free 3d gaussian splatting,” in _CVPR_ , 2024.
  * [36] C. Yan, D. Qu, D. Wang, D. Xu, Z. Wang, B. Zhao, and X. Li, “Gs-slam: Dense visual slam with 3d gaussian splatting,” in _CVPR_ , 2024.
  * [37] Y. Sun and B. Hariharan, “Dynamo-depth: Fixing unsupervised depth estimation for dynamical scenes,” _NeurIPS_ , vol. 36, 2024.
  * [38] A. Meuleman, Y.-L. Liu, C. Gao, J.-B. Huang, C. Kim, M. H. Kim, and J. Kopf, “Progressively optimized local radiance fields for robust view synthesis,” in _CVPR_ , 2023, pp. 16 539–16 548.
  * [39] J. Guo, N. Deng, X. Li, Y. Bai, B. Shi, C. Wang, C. Ding, D. Wang, and Y. Li, “Streetsurf: Extending multi-view implicit surface reconstruction to street views,” _arXiv preprint arXiv:2306.04988_ , 2023.
  * [40] H. Turki, J. Y. Zhang, F. Ferroni, and D. Ramanan, “Suds: Scalable urban dynamic scenes,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 12 375–12 385.
  * [41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.