# ALSTER: A Local Spatio-Temporal Expert for
Online 3D Semantic Reconstruction

Silvan Weder1 Francis Engelmann1,4 Johannes L. Schönberger2 Akihito Seki3
Marc Pollefeys1,2 Martin R. Oswald1,5
1 ETH Zurich 2 Microsoft 3 Toshiba 4 Google 5 University of Amsterdam

###### Abstract

We propose an online 3D semantic segmentation method that incrementally
reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline
methods, ours is directly applicable to scenarios with real-time constraints,
such as robotics or mixed reality. To overcome the inherent challenges of
online methods, we make two main contributions. First, to effectively extract
information from the input RGB-D video stream, we jointly estimate geometry
and semantic labels per frame in 3D. A key focus of our approach is to reason
about semantic entities both in the 2D input and the local 3D domain to
leverage differences in spatial context and network architectures. Our method
predicts 2D features using an off-the-shelf segmentation network. The
extracted 2D features are refined by a lightweight 3D network to enable
reasoning about the local 3D structure. Second, to efficiently deal with an
infinite stream of input RGB-D frames, a subsequent network serves as a
temporal expert predicting the incremental scene updates by leveraging 2D, 3D,
and past information in a learned manner. These updates are then integrated
into a global scene representation. Using these main contributions, our method
can enable scenarios with real-time constraints and can scale to arbitrary
scene sizes by processing and updating the scene only in a local region
defined by the new measurement. Our experiments demonstrate improved results
compared to existing online methods that purely operate in local regions and
show that complementary sources of information can boost the performance. We
provide a thorough ablation study on the benefits of different architectural
as well as algorithmic design decisions. Our method yields competitive results
on the popular ScanNet benchmark and SceneNN dataset.

##  1 Introduction

Figure 1: We propose an online semantic 3D reconstruction pipeline, which
fuses RGB-D observations into a globally consistent semantic map. The key
component is a local spatio-temporal expert network that fuses new
observations into a learned scene representation. This temporal expert learns
to select information from 2D, 3D, and previous steps using an attention
mechanism.

To interact with the world, humans not only require low-level spatial
awareness of their surroundings, but also on real-time higher-level semantic
understanding. Building such spatial awareness through 3D reconstruction has
been a long-standing topic in computer vision. In this work, we address the
task of online 3D semantic reconstruction. Specifically, given an incoming
stream of RGB-D frames, the goal is to reconstruct a semantically enriched 3D
scene representation that is continuously updated.

The world around us is complex and 3D scenes can be understood at different
levels – from object-level understanding [37], over humans [40] and full rooms
[45, 1] to large-scale outdoor spaces [18], both in carefully selected
semantic classes [27, 19] or more abstract concepts and affordances [29, 39].
While high-level understanding can be sufficient for planning and navigation,
detailed scene interactions require more fine-grained understanding with
accurate semantic boundaries. As such, fine-grained semantic understanding is
at the heart of algorithms enabling real-world interactions.

Numerous works in 3D scene understanding focus on offline reconstructed point-
clouds [31, 19], meshes [36], or voxel-grids [27, 4]. While there has been
impressive progress with these works in recent years, a large majority of
these have one major shortcoming that we aim to address. All these works
require an a priori reconstruction of the scene and use this global
information for the understanding task. Therefore, they are considered offline
methods. However, autonomous agents (as well as humans) typically build a
“mental” map of the environment in an incremental manner, that is,
continuously update it over time as new information is collected. Thus, scene
understanding must inherently be an iterative process, as an autonomous agent
cannot assume all information known a priori. In this work, we investigate
this particular problem, where we incrementally build a semantic map given a
stream of posed RGB-D data that allows for online processing of the incoming
data streams and can be integrated in real-time systems. This online
processing is essential for enabling real-world applications, such as robotics
and mixed reality, where an updated semantic map is required to solve complex
tasks.

Only few approaches tackle the problem specified above. The seminal works of
Vineet  _et al_. [41] and SemanticFusion [22] map 2D semantic predictions into
3D. One step further, PanopticFusion [26] predicts a semantic instance map in
2D that is mapped and aggregated in 3D. While these methods reason in 2D as
well as 3D, the 3D reasoning is a CRF-based regularizer that is limited
compared to modern neural networks. Further, the CRF requires global
information of the entire scene that limits the scalability of the methods to
small scenes. Similarly, INS-Conv [21] estimates semantic instance maps using
3D processing with a large UNet that requires global processing to avoid
drifting errors. In contrast, other works [46, 16] perform 3D reasoning using
point- or supervoxel convolutions in a local frame. However, they only store
explicit labels that only encode per-point information, while our work uses
learned features encoding low-, mid- and high-level information.

Our work is based on the observation that 2D and 3D information is
complementary for the task of scene understanding. Some elements are better to
be understood in 2D depending on context and geometry whereas others are
easier to be segmented in 3D given their spatial structure. To this end, we
present a novel attention-based aggregation mechanism that fuses 2D, 3D, as
well as existing features into the scene. Our method only operates in a local
region defined by the new measurement and integrates the updates into the
learned global scene representation. Through this design, our method is
independent of the scene size and can scale to large-scale scenes.

In an extensive experimental evaluation, we show that our method is
competitive with existing approaches to online semantic 3D reconstruction
while not requiring passes over the entire reconstruction as opposed to some
other methods [21]. This is particularly important for online processing on
mobile devices and agents that are constrained in the amount of compute and
memory available. We evaluate our method on ScanNet as well as SceneNN and
present in-depth ablation studies to motivate our design choices. We will
release the source code on acceptance of this paper to foster further research
in this direction. In summary, the key contributions in this work are:

  * •

We show that 2D and 3D information are complementary for the task of online 3D
semantic reconstruction and improve the overall result.

  * •

We propose a novel local fusion approach that leverages an attention mechanism
to combine existing features with new 2D and 3D information in an online
fashion. We evaluate our pipeline design on the well-known ScanNet [6]
benchmark and show competitive results compared to existing online local
reconstruction methods.

##  2 Related Work

Offline _vs_. online processing. Most existing 3D segmentation methods follow
on offline approach: the 3D geometry of the scene and its corresponding
features (color, normals, _etc_.) are known a-priori and then processed by the
segmentation method. We first review prior work on _offline_ semantic
segmentation and then look at existing _online_ methods in the context of
incrementally building semantic 3D maps.

3D Semantic Segmentation is the problem of assigning a class label to each
point, voxel, or vertex of the 3D scene. It is central to many applications
and pipelines that require some form of understanding. In recent years, many
different methods tackled this problem. Semantic Stixels [35] predict 2D
semantic labeling and stereo depth maps that are aggregated in a 3D stixel
representation. While this representation can be sufficient for outdoor
applications, it lacks representation power for indoor applications. Kundu
_et al_. [19] address the problem of lack of context in the 2D views by
rendering views from an already reconstructed mesh to have a larger field-of-
view that improves the performance of 2D semantic segmentation. The
predictions are afterwards aggregated again on the 3D mesh. As this approach
is dependent on an already reconstructed mesh, it is not suitable for an
online fusion approach. Atlas [25] jointly reconstructs a semantic and
geometric map from visual inputs by learning multi-view fusion. As this
approach needs to aggregate dense viewing frustums to solve the multi-view
stereo problem, it is not suitable for fast online updates. SemanticNeRF [48]
proposes the application of recently proposed neural radiance fields [24] to
the problem of 3D semantic segmentation. While this approach shows impressive
results, it is also not applicable to fast and accurate online updates of a
semantic map. Mix3D [27] boosts the performance of 3D segmentation methods by
proposing a novel data augmentation technique that combines different scenes.
While this augmentation works for global methods, it cannot be applied to
online fusion systems since we jointly learn the fusion across time and
segmentation of the scene. BPNet [13] couples 2D and 3D predictions of scene
labels by proposing a bi-directional projection module. This boosts the
performance on 3D semantic segmentation but is dependent on global processing
and a-prior scene reconstructions. VMNet [14] combines Euclidean and geodesic
information to address the short-comings of voxel-only approaches. Yet, it
also requires global processing to unfold its full potential. OccuSeg [10]
enhances supervoxel-based geometric segmentation with learned features and
refines them using graph-based clustering but requires a global receptive
field, which makes them unnecessarily expensive for online processing.

Online 3D Semantic Segmentation. In contrast to the previously mentioned
approaches, online methods iteratively process the scene making them better
suitable to real-time applications, where agents are interacting with their
environment such as robotics or mixed reality in unknown environments. There
is a long line of work aiming at the real-time reconstruction of geometry and
appearance [5, 28, 43, 44]. These works have been extended to scene
understanding to enable agents with understanding capabilities. The approaches
[41] and [22] proposed to fuse 2D semantic predictions into a global semantic
map that is refined using a conditional random field (CRF). This idea has been
extended by several works. SceneCode [47] stores a per-keyframe latent code
encoding the semantic information of the scene that is optimized at test time.
Meanwhile, MaskFusion [34] and Fusion++ [23] focus on 3D object segmentation
while ignoring their semantic class. ProgressiveFusion [30] improves
efficiency by clustering voxels into supervoxels and apply CRF on that level.
SemanticReconstruction [17] follows a similar approach as [22], but shows that
their scene representation can be used for downstream tasks such as scene
completion and manipulation. PanopticFusion [26] estimates 3D semantic
instance maps by predicting 2D semantic and instance segmentation using off-
the-shelf networks, aggregates them in 3D, and also regularizes them using a
CRF. While these works leverage 2D processing in combination with
optimization-based 3D regularization, they all resort to traditional voxel
fusion and do not utilize trainable 3D neural networks. This shortcoming has
been addressed in SVCNN [16], which clusters voxels that store explicit
semantic information into supervoxels, and then processes them using a special
convolutional operator designed for supervoxels. However, [16] still resorts
to an explicit fusion of 2D semantic information into voxels. An alternative
is FusionAware [46] that represents scenes using efficient point cloud
representations and uses point-convolutions to aggregate new information. More
recently, Liu  _et al_. [21] presented an online method predicting semantic
instance maps using only 3D processing. Nevertheless, these two works
disregard useful 2D information. In our work, we address these limitations by
a) combining 2D and 3D information in a temporal expert network leveraging
both sources of information, and b) applying a powerful yet lightweight 3D
network on the current viewing frustum.

##  3 Method

This section presents our method for online 3D semantic reconstruction.
Firstly, we give an overview of our model (Fig. 2) and the 3D scene
representation. Then, we describe the spatial-temporal expert that enables
efficient local updates of the scene representation. Lastly, we discuss
training protocols, loss functions and sequential optimization.

###  3.1 Semantic 3D Reconstruction Pipeline

Figure 2: Pipeline Overview: Our pipeline consists of three main stages. The
2D encoder extracts information from incoming RGB-D imagery. This information
is enhanced with 3D information using a light-weight 3D encoder. The
information from these two sources is combined with existing information in
the learned scene representation using the temporal expert.

Overview. Our proposed model consists of three major components (Fig. 2) and a
learned 3D scene representation. The first stage is a 2D encoder
$F^{\text{2D}}$ that extracts 2D feature maps from an incoming stream of RGB-D
images. The second stage is a 3D encoder $F^{\text{3D}}$ that incorporates 3D
geometry into each feature map after lifting it to 3D using the given camera
parameters and depth maps. The third stage is a new _temporal expert_ network
$F^{\Delta T}$ that consolidates 3D scene representations using complementary
information from 2D and 3D as well as the so-far reconstructed 3D scene.

Scene Representation. The backbone of every online reconstruction method is a
suitable scene representation. Typically, the primary choice are voxels,
points, meshes or implicit (neural) representation. Meshes and implicit
representations are difficult to update with new observations, while points
lack information about geometric connectivity. This is crucial for scene
understanding where decisions about segmentation boundaries are oftentimes
guided by geometric boundaries. Therefore, we represent scenes using a hybrid
representation $S$ combining learned and explicit features that are stored in
a sparse voxel grid. Sparse voxel grids allow for efficient processing using
neural networks. In particular, each voxel stores a learned feature $F$ of
dimension $D_{F}$ = $40$ encoding the aggregated information about the scene
content. This learned scene representation allows to store high-, mid-, and
low-level information useful for the semantic segmentation task. This
mitigates the need for expensive re-processing in deep neural networks at
every time step to fuse existing and new information. The voxels also store
the number of per-voxel observations, which is relevant for the subsequent
fusion step.

2D Encoder. The aim of the first stage is to extract semantic features from
incoming 2D RGB-D images using a 2D convolutional network $F^{\text{2D}}$. The
2D network $F^{\text{2D}}$ with trainable parameters $\theta^{\text{2D}}$
takes RGB-D frames $(I_{t},D_{t})$ as input and predicts semantic features
$\tilde{f}^{\text{2D}}_{t}$ per frame:

$\tilde{f}^{\text{2D}}_{t}=F^{\text{2D}}\left(\left[I_{t},D_{t},N_{t}\right];\theta^{\text{2D}}\right)$
(1)

The normal map $N_{t}$ is estimated from the depth map $D_{t}$ and serves as
additional input. The network consists of DeepLabV3+ [2] and uses, similar to
[19], an Xception65 [3] encoder that is adjusted to handle RGB-D-N input data
(color, depth, normals) since geometric information improves semantic
segmentation [9, 42, 11]. The semantic features $\tilde{f}^{\text{2D}}_{t}$
are directly obtained from the DeepLab decoder. However, the original
dimension of the features $\tilde{f}^{\text{2D}}_{t}$ is
$D_{\tilde{f}^{\text{2D}}}$ = $256$ which is too memory intensive for online
processing of large indoor scenes. Instead, we project the feature maps to
${D}^{\text{2D}}$ = $40$. This compression allows online processing for the
remaining of the pipeline while still retaining all relevant information
needed for 3D semantic understanding. The 2D network $F^{\text{2D}}$ is pre-
trained on ImageNet [8] and fine-tuned on 2D training data from ScanNet [6].
During the training of the full pipeline, the 2D encoder is partially fine-
tuned using an auxiliary semantic segmentation head enforcing consistent
performance across frames and for regularization of the joint feature space.

3D Encoder. We additionally process the incoming information in 3D as geometry
is complementary to 2D appearance. This processing is particularly motivated
by the possible reasoning about hidden geometric object boundaries occluded in
the current 2D frame. To this end, we lift the obtained 2D feature map
${f}^{\text{2D}}_{t}$ to 3D point clouds by projecting the depth map $D_{t}$
using the known, gravity-aligned camera orientation $R\in SO(3)$ and
intrinsics. Due to noisy depth estimates, we additionally filter out points
that are more than $3$ m away from the camera. The resulting _local_ 3D
feature volume is refined using a light-weight U-Net [33] $F^{\text{3D}}$
yielding a 3D feature map $f^{\text{3D}}_{t}$ with the same feature dimension
$D_{F}=40$ as the 2D feature map $f^{\text{2D}}_{t}$.

Spatio-Temporal Expert. In the previous stages, 2D features are extracted and
enhanced with 3D information. In the next step, this information is integrated
into the existing global scene representation $S_{t-1}$. To this end, we
propose the spatio-temporal expert network $F^{\Delta T}$ with weights
$\theta^{\Delta T}$. The task of the spatio-temporal expert network $F^{\Delta
T}$ is to update the features stored in the learned scene representation given
the new information from $f^{\text{2D}}_{t}$, $f^{\text{3D}}_{t}$, and the
existing information $f^{\text{global}}_{t-1}$. The feature volume
$f^{\text{global}}_{t-1}$ is a crop of the relevant local sub-volume from the
global scene representation $S_{t-1}$ using the known camera pose
$\left[R|t\right]\in SE(3)$. The overall mapping is computed as:

$f^{\text{global}}_{t}=F^{\Delta
T}\left(\left[f^{\text{global}}_{t-1},f^{\text{3D}}_{t},f^{\text{2D}}_{t}\right];\theta^{\Delta
T}\right)$ (2)

The resulting local volume $f^{\text{global}}_{t}$ is then written back, using
the inverse camera pose, to obtain the new global scene representation
$S_{t}$. By providing access to both the 2D and 3D features in parallel, the
expert network can learn where it is beneficial to rely more on 2D appearance
features or where it is advantageous to trust the 3D geometry-based features,
see Fig. 4 for an illustration. The 3D features reveal geometrical details
while the 2D features provide textural information in flat areas with little
geometric information.

$F^{\Delta T}$ is implemented as a Transformer consisting of cross-attention
and feed-forward layers (see Fig. 3). The task of the cross-attention layer is
to extract relevant information from the three sources of information
($f^{\text{2D}}_{t}$, $f^{\text{3D}}_{t}$, $f^{\text{global}}_{t-1}$) using
$f^{\text{global}}_{t-1}$ as query features. The attention is defined as
$f=w_{f^{\text{global}}_{t-1}}v_{f^{\text{global}}_{t-1}}+w_{f^{\text{2D}}_{t}}v_{f^{\text{2D}}_{t}}+w_{f^{\text{3D}}_{t}}v_{f^{\text{3D}}_{t}}$
and the weights
$(w_{f^{\text{global}}_{t-1}},w_{f^{\text{2D}}_{t}},w_{f^{\text{3D}}_{t}})$
are defined as:

$w_{f^{k}}=Q\left(f^{\text{global}}_{t-1}\right)^{T}K\left(f^{k}\right)\enspace,$
(3)

where
$f^{k}\in\\{f^{\text{global}}_{t-1},f^{\text{3D}}_{t},f^{\text{2D}}_{t}\\}$.
The values $v_{f^{k}}$ are obtained using a linear projection layer
$v_{f^{k}}=V\left(f^{k}\right)$.

The features extracted from the cross-attention layer are first normalized
using layer norm and then refined using a standard feed-forward layer.
Furthermore, both layers are augmented with a skip connection guaranteeing
healthy gradients during training. The refined features that consist of
information extracted from the three sources $f^{\text{2D}}_{t}$,
$f^{\text{3D}}_{t}$, and $f^{\text{global}}_{t-1}$ are written back into the
global scene representation. The temporal expert is also supervised by a
point-wise segmentation loss, ensuring optimal segmentation given the
currently available 3D scene information.

Figure 3: Temporal expert network. The temporal expert network takes the three features ($f^{\text{global}}_{t-1}$, 2D, and 3D) as input, and iteratively refines the old feature vector to obtain the update feature that can be stored in the scene representation. The old feature is used as the query in the attention mechanism. 2D Labels | 2D Attention | 3D Label | 3D Attention
---|---|---|---
|  |  |
|  |  |
Figure 4: Temporal expert attention maps. We visualize the attention maps for
the 2D and 3D input features in the expert network. The expert attends to the
3D features to refine the segmentation of fine details (legs of the chair,
lamps), while it attends to the 2D features to predict the label for large
areas (table, walls, _etc_.).

Loss Function. The pipeline is trained using focal loss [20] at several stages
in the pipeline. These losses are applied after the 2D encoder
$F^{\text{2D}}$, the 3D encoder $F^{\text{3D}}$, and the temporal expert
network $F^{\Delta T}$. These auxiliary supervision signals are required to
constrain the feature space that encodes the information throughout the entire
pipeline. Further, these auxiliary losses ensure that each stage solves the
task of semantic segmentation as good as possible for themselves providing the
temporal expert network with valuable information. The overall loss is the sum
of these losses:

$\mathcal{L}=\lambda_{\text{2D}}\mathcal{L}_{\text{2D}}+\lambda_{\text{3D}}\mathcal{L}_{\text{3D}}+\lambda_{\text{Expert}}\mathcal{L}_{\text{Expert}}$
(4)

where each term
$\mathcal{L}_{\text{2D}},\mathcal{L}_{\text{3D}},\mathcal{L}_{\text{Expert}}$
is a focal loss, defined as:

$\mathcal{L}=(1-\hat{y})^{\gamma}\text{CE}(\hat{y},y)$ (5)

As the loss is applied on a per-voxel level, the corresponding ground-truth
labels first need to be mapped from the ground truth polygon mesh to a
voxelized representation. To this end, we first voxelize all scenes using the
target resolution and assign the label of the closest vertex of the mesh.
Closest points are efficiently found using KD-tree-based nearest neighbor
search.

Sequential Training. The temporal expert network needs to learn how to fuse
new information into the existing scene representation based on sequential
data. A key challenge is catastrophic forgetting, where the network forgets
what it has learned during the beginning of a sequence and only focuses on the
last few frames along a camera trajectory. To overcome this challenge, it is
critically important to randomly select camera views along each video
trajectory, _i.e_., a permutation of the original frame order. Similarly, to
avoid that the model only sees fully reconstructed scenes after some initial
training time, we randomly reset the reconstructed scenes so that the model
always sees scenes at varying levels of reconstruction.

##  4 Experiments

| Method | Processing | mIoU$\uparrow$ |  Bathtub |  Bed |  Bookshelf |  Cabinet |  Chair |  Counter |  Curtain |  Desk |  Door |  Floor |  Other
Furniture |  Picture |  Fridge |  Shower
Curtain |  Sink |  Sofa |  Table |  Toilet |  Wall |  Window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Offline | Mix3D [27] | Global | 78.1 | 96.4 | 85.5 | 84.3 | 78.1 | 85.8 | 57.5 | 83.1 | 68.5 | 71.4 | 97.9 | 59.4 | 31.0 | 80.1 | 89.2 | 84.1 | 81.9 | 72.3 | 94.0 | 88.7 | 72.5
VirtualMVFusion [19] | Global | 74.6 | 77.1 | 81.0 | 84.8 | 70.2 | 86.5 | 39.7 | 89.9 | 69.9 | 66.4 | 94.8 | 58.8 | 33.0 | 74.6 | 85.1 | 76.4 | 79.6 | 70.4 | 93.5 | 86.6 | 72.8
Minkowski [4] | Global | 73.6 | 85.9 | 81.8 | 83.2 | 70.9 | 84.0 | 52.1 | 85.3 | 66.0 | 64.3 | 95.1 | 54.4 | 28.6 | 73.1 | 89.3 | 67.5 | 77.2 | 68.3 | 87.4 | 85.2 | 72.7
Online | PanopticFusion [26] | Global | 52.9 | 49.1 | 68.8 | 60.4 | 38.6 | 63.2 | 22.5 | 70.5 | 43.4 | 29.3 | 81.5 | 34.8 | 24.1 | 49.9 | 66.9 | 50.7 | 64.9 | 44.2 | 79.6 | 60.2 | 56.1
INS-Conv [21] | Global | 71.7 | 75.1 | 75.9 | 81.2 | 70.4 | 86.8 | 53.7 | 84.2 | 60.9 | 60.8 | 95.3 | 53.4 | 29.3 | 61.6 | 86.4 | 71.9 | 79.3 | 64.0 | 93.3 | 84.5 | 66.3
FusionAware [46] | Local | 63.0 | 60.4 | 74.1 | 76.6 | 59.0 | 74.7 | 50.1 | 73.4 | 50.3 | 52.7 | 91.9 | 45.4 | 32.3 | 55.0 | 42.0 | 67.8 | 68.8 | 54.4 | 89.6 | 79.5 | 62.7
SVCNN [16] | Local | 63.5 | 65.6 | 71.1 | 71.9 | 61.3 | 75.7 | 44.4 | 76.5 | 53.4 | 56.6 | 92.8 | 47.8 | 27.2 | 63.6 | 53.1 | 66.4 | 64.5 | 50.8 | 86.4 | 79.2 | 61.1
ALSTER (Ours) | Local | 66.8 | 82.2 | 77.1 | 49.6 | 65.1 | 83.3 | 54.1 | 76.1 | 55.5 | 61.1 | 96.6 | 48.9 | 37.0 | 38.8 | 58.0 | 77.6 | 75.1 | 57.0 | 95.6 | 81.7 | 64.6

Table 1:  3D Semantic Segmentation on ScanNet [6] Test. Offline baselines
predict semantic labels using a-priori 3D scene reconstructions and global
passes over the entire scene. Online but global baselines do online reasoning
but require global passes over the full scene. Local methods are online and
reason only on local information within the viewing frustum and on currently
updated points. Among local methods, our proposed approach improves over
existing baselines by at least +$3.3$ mIoU.

###  4.1 Implementation Details

We implement the proposed pipeline in PyTorch. We use the MinkowskiEngine [4]
for the sparse 3D convolutions in the 3D encoder, and Pytorch3D [32] for the
geometric projections. The entire pipeline is trained with the Adam optimizer
and a OneCycle [38] learning rate scheduler. Due to memory constraints, the
batch size is $4$ but we obtain an effective batch size of 8 by aggregating
the gradients across two batches. We set the maximum learning rate to $0.001$
for the 3D and temporal expert networks, while the maximum learning rate for
the pre-trained 2D encoder is set to $1e-05$. We equally weight the different
terms in the loss function setting $\lambda_{\text{3D}}$ =
$\lambda_{\text{2D}}$ = $\lambda_{\text{Expert.}}$ = $1$. Further, we set the
parameter of the focal loss $\gamma$ = $1.$ We use five layers in the temporal
expert transformer with a hidden dimension of $D_{\text{hidden}}$ = $128$ in
the feed-forward layers. The voxel grid resolution for the entire pipeline is
set to $4$ cm.

2D Labels |  |  |  |
---|---|---|---|---
3D Labels |  |  |  |
Expert Labels |  |  |  |
Groundtruth |  |  |  |
Figure 5:  Qualitative results of our proposed method and comparison between
the different stages. The expert successfully selects the correct information
from the two encoders. The 2D encoder is focused on object-level decisions
(_e.g_., table in column 1, bench in column 2) while the 3D information is
used for fine details (_e.g_. lamp in column 3, desk in column 5).

###  4.2 Online Methods in Comparison

FusionAware [46]. Unlike our voxel-based representation, FusionAware is a
_point_ -based online 3D semantic segmentation method. The method aggregates
measurements in 3D space using point convolutions and computes intra- and
inter-frame features.

SVCNN [16]. Similar to ours, Supervoxel Convolution (SVCNN) is another
candidate from the space of voxel-based approaches. SVCNN uses dedicated
convolutional operators that operate directly on supervoxels and aggregate
multi-view features during online reconstruction.

###  4.3 Datasets and Metrics

ScanNet [6] consists of 1513 scans from 707 unique indoor scenes containing
2.5M RGB-D frames. All scenes provide dense 3D semantic annotations mapped to
the NYU40 class labels. Each scene is recorded up to three times using an iPad
equipped with an Occipital depth sensor. The camera poses and dense
reconstruction of the scenes are obtained using BundleFusion [7]. The 3D
labels are projected into all 2D frames to provide the 2D labels.

SceneNN [15]. SceneNN consists of 76 scenes with semantic annotations and
corresponding posed RGB-D data. We follow [16, 21] and demonstrate the
generalization capabilities of our method by training on ScanNet and
evaluating on the 76 SceneNN scenes.

Method | mIoU$\uparrow$ |  Wall |  Floor |  Cabinet |  Bed |  Chair |  Sofa |  Table |  Door |  Window |  Bookshelf |  Picture |  Counter |  Desk |  Curtain |  Fridge |  Shower Curt. |  Toilet |  Sink |  Bathtub |  Other Furn.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Ours - 2D | 68.2 | 83.4 | 96.3 | 60.1 | 75.0 | 85.8 | 77.0 | 68.7 | 65.7 | 60.3 | 66.7 | 28.2 | 64.8 | 56.1 | 68.3 | 61.8 | 58.8 | 86.6 | 62.7 | 86.0 | 52.1
Ours - 3D | 69.0 | 85.0 | 96.8 | 59.2 | 73.9 | 87.3 | 75.8 | 69.7 | 69.5 | 61.3 | 62.9 | 33.1 | 65.7 | 56.1 | 68.4 | 61.2 | 60.2 | 89.0 | 64.1 | 87.0 | 52.8
Ours - Temp | 70.6 | 85.6 | 96.9 | 59.6 | 77.1 | 87.3 | 75.0 | 71.9 | 68.4 | 65.8 | 65.9 | 36.6 | 67.2 | 61.1 | 69.4 | 62.6 | 65.0 | 89.8 | 65.2 | 87.1 | 53.8
$D_{F}$ = 64 | 67.9 | 84.4 | 96.2 | 59.4 | 76.9 | 85.6 | 73.0 | 69.5 | 66.3 | 63.1 | 36.4 | 24.8 | 65.6 | 61.6 | 71.3 | 61.5 | 67.1 | 89.1 | 64.7 | 89.4 | 52.9
$D_{F}$ = 40 | 70.6 | 85.6 | 96.9 | 59.6 | 77.1 | 87.3 | 75.0 | 71.9 | 68.4 | 65.8 | 65.9 | 36.6 | 67.2 | 61.1 | 69.4 | 62.6 | 65.0 | 89.8 | 65.2 | 87.1 | 53.8
$D_{F}$ = 32 | 68.2 | 83.0 | 94.9 | 60.8 | 78.5 | 86.3 | 76.0 | 69.8 | 61.6 | 58.7 | 63.6 | 24.6 | 64.5 | 61.1 | 70.0 | 60.9 | 50.4 | 90.6 | 67.5 | 87.5 | 53.0
$D_{F}$ = 16 | 68.1 | 83.2 | 95.2 | 58.5 | 77.9 | 84.9 | 75.8 | 69.4 | 61.5 | 63.8 | 52.9 | 33.0 | 60.1 | 58.2 | 64.9 | 61.5 | 64.7 | 90.8 | 63.9 | 85.9 | 55.0
Xception | 70.6 | 85.6 | 96.9 | 59.6 | 77.1 | 87.3 | 75.0 | 71.9 | 68.4 | 65.8 | 65.9 | 36.6 | 67.2 | 61.1 | 69.4 | 62.6 | 65.0 | 89.8 | 65.2 | 87.1 | 53.8
MobileNet | 66.0 | 81.2 | 95.9 | 57.8 | 73.9 | 83.1 | 70.8 | 68.0 | 60.3 | 50.2 | 63.9 | 37.3 | 63.1 | 61.9 | 63.4 | 56.3 | 50.1 | 88.6 | 58.5 | 86.2 | 50.0

Table 2: Ablating different aspects of our pipeline on ScanNet [6] validation
set. We show that the expert selects valuable information from the two
encoders and the existing scene representation by evaluating the individual
pipeline outputs. The expert network consistently improves upon the two other
stages in terms of IoU. We also evaluate the impact of the stored feature
dimension $D_{F}$ on the overall result. While the smaller feature sizes
suffer from compression due to limited capacity the larger features
($D_{F}=64$) suffer from slight overfitting. Finally, we compare the Xception
2D encoder to the lighter MobileNet ($\times 20$ fewer parameters).
Unsurprisingly, the smaller encoder leads to a slight deterioration of
performance, but the results indicate potential for runtime-accuracy tradeoffs
in time-critical applications.

3D Semantic Segmentation
---
|  |  | ScanNet | SceneNN
| Processing | Res. [cm] | val. mIoU | wIoU | mAcc
SemanticFusion [22] | Global | N/A | 42.3 | 47.1 | 58.5
PanopticFusion [26] | Global | 2.4 | 53.1 | – | –
InsConv [21] | Global | 2 | 72.4 | – | 79.5
SemanticReconstruction [17] | Local | N/A | 44.0 | – | –
ProgressiveFusion [30] | Local | 0.8 | 55.0 | 52.2 | 61.6
FusionAware [46] | Local | N/A | 67.2 | 63.9 | 71.7
SVCNN [16] | Local | N/A | 68.3 | 69.0 | 76.9
ALSTER (Ours) | Local | 4 | 70.6 | 67.8 | 76.9

Table 3: 3D Semantic Segmentation on ScanNet and SceneNN. Scores are mean
intersection over union (mIoU) on ScanNet [6] validation, the mean accuracy
(mAcc) and weighted IoU (wIoU) on SceneNN [30]. All other scores are as
reported in [16] and [21].

Metrics. We follow the standard metrics of the ScanNet and SceneNN datasets.
In particular, we compute the mean and per-class intersection over union (IoU)
on ScanNet, as well as the mean accuracy (mAcc) and weighted intersection over
union (wIoU) on SceneNN.

###  4.4 3D Semantic Segmentation

Table 1 reports 3D semantic segmentation scores of our and recent methods on
ScanNet [6]. We compare online and offline methods, as well as local and
global methods. While offline methods rely on a pre-computed 3D scene
reconstruction in the form of a point cloud or polygon mesh, online methods
are able to reconstruct the 3D scene on the fly as new frames become
available. This functionality is attractive for online applications in
robotics or AR/VR devices, however they cannot rely on the full scene context
which makes semantic reasoning harder and semantic scores are generally higher
for offline methods [27, 19, 4]. In the group of online methods, our approach
improves over the existing local methods like SVCNN [16] and FusionAware [46]
by at least +$3.3$ mIoU. Local methods operate on a local window defined by
the newly incoming frames and are therefore memory and computationally
efficient, both attractive qualities for real-time processing. Global methods
require global passes over the reconstructed scenes either by CRF
regularization or neural network processing. This step takes increasingly more
time as the reconstructed scene becomes larger in size. These methods are
therefore less applicable for real-time applications, since no upper bound on
the processing time can be guaranteed.

In Table 3, we compare our method to existing baselines on the ScanNet
validation set as well as SceneNN. For ScanNet, we report the mIoU over all
benchmark classes. For SceneNN, we compute the weighted intersection over
union (wIoU) and the mean accuracy (mAcc) for all annotated NYU40 classes. In
addition to the quantitative metrics, we also report the voxel resolution for
all methods where it is available. A smaller voxel size generally results in
better scores since finer details can be represented, however this comes at
increased memory costs. Our proposed method performs best among all local
methods on both ScanNet and SceneNN (with a close second on the wIoU metric).
When also compared to global methods, INS-Conv [21] performs only marginally
better on ScanNet, even when using a voxel resolution that is twice as high,
highlighting the memory efficiency of our method.

###  4.5 Ablation Studies

Does the temporal expert network improve upon the individual 2D and 3D
networks? In Table 2, we report the numbers for the different stages in our
pipeline. For each stage, we compute the per-class intersection over union on
the ScanNet validation set. The per-stage labels are obtained from the
auxiliary heads used during training to constrain the joint feature space and
aggregated using a simple voting mechanism. These labels are then mapped to
the ScanNet ground-truth and evaluated using their evaluation pipeline. The
numbers show that the expert is consistently better than the individual
branches (Ours - 2D and Ours - 3D). Further, the fact that sometimes the 2D
labeling is better than the 3D labels and the significant margin between 3D
and expert indicate that 1) bypassing the 2D information around the 3D encoder
is useful and 2) our attention-based fusion mechanism allows better reasoning
over time than simple voting. We also show the differences between the
different stages in Figure 5, where one can see the benefits of selecting
information from the two different encoders.

What does the temporal expert network attend to? In Figure 4, we visualize the
attention maps for different frames during the fusion process together with
the corresponding predicted labels. We qualitatively show that they learn to
leverage the two different encoder according to their individual strengths.
The temporal expert network attends to the 3D network for fine-details usually
refining edges and geometric details (_e.g_., legs of a chair, edge of a
table) while it attends to the 2D feature for information about large regions.
Further, we observe that the fusion with the old scene representation happens
in later layers while earlier layers combine the 2D and 3D information.

What is the impact of the feature dimension $D_{F}$? The dimension $D_{F}$ of
the features stored in the learned scene representation is a key
hyperparameter of the pipeline. Thus, we evaluate its impact on the overall
performance in Table 2. One can see that for $D_{F}$ below 40 (the default
value), the performance is slightly deteriorated due to the required
compression of the semantic information. For $D_{F}=64$ the main reason for
the slight performance drop is the increased overfitting to the training data
due to the increased capacity of the features.

How fast is our method? An average step through our entire (non-optimized)
pipeline takes $116.1$ ms ($8.6$ FPS) on an NVIDIA RTX 2080 and $3.6$ GHz
Intel CPU i9-9900K. To identify the main bottleneck, we analyse the runtime of
the individual components in Fig. 6. One pass through the 2D network DeepLabV3
[2] plus lifting takes on average $83.5$ ms. One pass through our light-weight
3D UNet takes $28.6$ ms on average, and the temporal expert network operates
at $4$ ms per frame. These numbers reveal the 2D DeepLabV3 as the main
bottleneck. Thus, we also report the performance of MobileNet [12] in Tab. 2
instead of the standard Xception encoder. This reduces the 2D processing time
to $45.2$ ms and boosts the overall runtime to $12.9$ FPS.

Figure 6: Runtime analysis for different 2D backbones. A smaller 2D encoder
trades runtime for accuracy (_cf_. Tab. 2).

How many parameters does our pipeline have? Our pipeline consists in $51\cdot
10^{6}$ parameters in total. The largest share is due to the the DeepLabV3
(Xception) model, which consists of $41\cdot 10^{6}$ parameters. The 3D U-Net
consists of $10\cdot 10^{6}$ parameters. Compared to both encoders, the expert
model takes a relatively small share of $92\cdot 10^{3}$ parameters. This
further justifies our architecture design, with a marginal increase in model
size and runtime, we obtain a notable boost in performance (+$1.6$ mIoU, see
Tab. 2)

##  5 Conclusion

We presented a novel pipeline for online joint geometric and semantic 3D
reconstruction. The pipeline consists of three components and a learned scene
representation that represents the scene as sparse voxel grid. In order to
leverage the complementary nature of 2D and 3D information, the first two
stages encode 2D RGB-D data and enhance the encoded features with 3D spatial
information. At the heart of your pipeline sits a temporal expert fusion
network, that sequentially updates the learned scene representation. This
network attends to the 2D, 3D, and existing features to extract relevant
information for the updates. We experimentally show that this design improves
the performance on semantic segmentation upon the two individual branches.

Acknowledgments. This research is partially supported by Toshiba, an ETH AI
Center PostDoc Fellowship, and an ETH Career Seed Award funded through the ETH
Zurich Foundation.

## References

  * Chen et al. [2023] Jiacheng Chen, Ruizhi Deng, and Yasutaka Furukawa.  Polydiffuse: Polygonal shape reconstruction via guided set diffusion models.  _arXiv preprint arXiv:2306.01461_ , 2023.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking Atrous Convolution for Semantic Image Segmentation.  _arXiv preprint arXiv:1706.05587_ , 2017.
  * Chollet [2017] François Chollet.  Xception: Deep Learning with Depthwise Separable Convolutions.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Choy et al. [2019] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Curless and Levoy [1996] Brian Curless and Marc Levoy.  A Volumetric Method for Building Complex Models from Range Images.  In _Conference on Computer Graphics and Interactive Techniques (SIGGRAPH)_ , 1996.
  * Dai et al. [2017a] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017a.
  * Dai et al. [2017b] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt.  Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration.  _ACM Trans. Graph._ , 36(3):24:1–24:18, 2017b.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * Gupta et al. [2014] Saurabh Gupta, Ross B. Girshick, Pablo Andrés Arbeláez, and Jitendra Malik.  Learning rich features from RGB-D images for object detection and segmentation.  In _Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII_ , pages 345–360. Springer, 2014.
  * Han et al. [2020] Lei Han, Tian Zheng, Lan Xu, and Lu Fang.  OccuSeg: Occupancy-Aware 3D Instance Segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Hazirbas et al. [2016] Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers.  Fusenet: Incorporating depth into semantic segmentation via fusion-based CNN architecture.  In _Computer Vision - ACCV 2016 - 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part I_ , pages 213–228. Springer, 2016.
  * Howard et al. [2017] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _CoRR_ , abs/1704.04861, 2017.
  * Hu et al. [2021a] Wenbo Hu, Hengshuang Zhao, Li Jiang, Jiaya Jia, and Tien-Tsin Wong.  Bidirectional Projection Network for Cross Dimension Scene Understanding.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021a.
  * Hu et al. [2021b] Zeyu Hu, Xuyang Bai, Jiaxiang Shang, Runze Zhang, Jiayu Dong, Xin Wang, Guangyuan Sun, Hongbo Fu, and Chiew-Lan Tai.  Vmnet: Voxel-mesh network for geodesic-aware 3d semantic segmentation.  In _International Conference on Computer Vision (ICCV)_ , 2021b.
  * Hua et al. [2016] Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung.  SceneNN: A Scene Meshes Dataset with Annotations.  In _International Conference on 3D Vision (3DV)_ , 2016.
  * Huang et al. [2021] Shi-Sheng Huang, Ze-Yu Ma, Tai-Jiang Mu, Hongbo Fu, and Shi-Min Hu.  Supervoxel Convolution for Online 3D Semantic Segmentation.  _ACM Transactions on Graphics (TOG)_ , 2021.
  * Jeon et al. [2018] Junho Jeon, Jinwoong Jung, Jungeon Kim, and Seungyong Lee.  Semantic Reconstruction: Reconstruction of Semantically Segmented 3D Meshes via Volumetric Semantic Fusion.  In _Computer Graphics Forum_ , 2018.
  * Kreuzberg et al. [2022] Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, and Bastian Leibe.  4D-STOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation.  In _European Conference on Computer Vision (ECCV) Workshops_ , 2022.
  * Kundu et al. [2020] Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, and Caroline Pantofaru.  Virtual Multi-view Fusion for 3D Semantic Segmentation.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Lin et al. [2017] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , pages 2999–3007. IEEE Computer Society, 2017.
  * Liu et al. [2022] Leyao Liu, Tian Zheng, Yun-Jou Lin, Kai Ni, and Lu Fang.  INS-Conv: Incremental Sparse Convolution for Online 3D Segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * McCormac et al. [2017] John McCormac, Ankur Handa, Andrew J. Davison, and Stefan Leutenegger.  SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks.  In _International Conference on Robotics and Automation (ICRA)_ , 2017.
  * McCormac et al. [2018] John McCormac, Ronald Clark, Michael Bloesch, Andrew J. Davison, and Stefan Leutenegger.  Fusion++: Volumetric Object-Level SLAM.  In _International Conference on 3D Vision (3DV)_ , 2018.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Murez et al. [2020] Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich.  Atlas: End-to-end 3d scene reconstruction from posed images.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Narita et al. [2019] Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji.  PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2019.
  * Nekrasov et al. [2021] Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, and Francis Engelmann.  Mix3D: Out-of-Context Data Augmentation for 3D Scenes.  In _International Conference on 3D Vision (3DV)_ , 2021.
  * Newcombe et al. [2011] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew W. Fitzgibbon.  KinectFusion: Real-Time Dense Surface Mapping and Tracking.  In _IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_ , 2011.
  * Peng et al. [2023] Songyou Peng, Kyle Genova, Chiyu ”Max” Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser.  Openscene: 3d scene understanding with open vocabularies.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Pham et al. [2019] Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung.  Real-Time Progressive 3D Semantic Segmentation for Indoor Scenes.  In _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2019.
  * Qi et al. [2017] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Ravi et al. [2020] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  _arXiv:2007.08501_ , 2020.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , 2015.
  * Rünz et al. [2018] Martin Rünz, Maud Buffier, and Lourdes Agapito.  MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects.  In _IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_ , 2018.
  * Schneider et al. [2016] Lukas Schneider, Marius Cordts, Timo Rehfeld, David Pfeiffer, Markus Enzweiler, Uwe Franke, Marc Pollefeys, and Stefan Roth.  Semantic Stixels: Depth is not enough.  In _IEEE Intelligent Vehicles Symposium (IV)_ , 2016.
  * Schult* et al. [2020] Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, and Bastian Leibe.  DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Schult et al. [2023] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe.  Mask3D for 3D Semantic Instance Segmentation.  In _International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Smith and Topin [2017] Leslie N. Smith and Nicholay Topin.  Super-convergence: Very fast training of residual networks using large learning rates.  _CoRR_ , abs/1708.07120, 2017.
  * Takmaz et al. [2023a] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann.  OpenMask3D: Open-Vocabulary 3D Instance Segmentation.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023a.
  * Takmaz et al. [2023b] Ayça Takmaz, Jonas Schult, Irem Kaftan, Mertcan Akçay, Bastian Leibe, Robert Sumner, Francis Engelmann, and Siyu Tang.  3D Segmentation of Humans in Point Clouds with Synthetic Data.  In _International Conference on Computer Vision (ICCV)_ , 2023b.
  * Vineet et al. [2015] Vibhav Vineet, Ondrej Miksik, Morten Lidegaard, Matthias Nießner, Stuart Golodetz, Victor A Prisacariu, Olaf Kähler, David W Murray, Shahram Izadi, Patrick Pérez, et al.  Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction.  In _2015 IEEE international conference on robotics and automation (ICRA)_ , pages 75–82. IEEE, 2015.
  * Wang et al. [2016] Jinghua Wang, Zhenhua Wang, Dacheng Tao, Simon See, and Gang Wang.  Learning common and specific features for RGB-D semantic segmentation with deconvolutional networks.  In _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V_ , pages 664–679. Springer, 2016.
  * Weder et al. [2020] Silvan Weder, Johannes L. Schönberger, Marc Pollefeys, and Martin R. Oswald.  RoutedFusion: Learning Real-Time Depth Map Fusion.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Weder et al. [2021] Silvan Weder, Johannes L. Schönberger, Marc Pollefeys, and Martin R. Oswald.  NeuralFusion: Online Depth Fusion in Latent Space.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Yue et al. [2023] Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann.  Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Zhang et al. [2020] Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, and Kai Xu.  Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Zhi et al. [2019] Shuaifeng Zhi, Michael Bloesch, Stefan Leutenegger, and Andrew J. Davison.  Scenecode: Monocular dense semantic reconstruction using learned encoded scene representations.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Zhi et al. [2021] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place Scene Labelling and Understanding with Implicit Scene Representation.  In _International Conference on Computer Vision (ICCV)_ , 2021.


* [92] Y. Huang, S. Gupta, Z. Zhong, K. Li, and D. Chen, “Privacy implications of retrieval-based language models,” _arXiv preprint arXiv:2305.14888_ , 2023.
  * [93] S. Arora, P. Lewis, A. Fan, J. Kahn, and C. Ré, “Knowledge retrieval over public and private data,” 2023.
  * [94] F.-E. Yang, C.-Y. Wang, and Y.-C. F. Wang, “Efficient model personalization in federated learning via client-specific prompt generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 19 159–19 168.
  * [95] S. Su, M. Yang, B. Li, and X. Xue, “Federated adaptive prompt tuning for multi-domain collaborative learning,” 2024.
  * [96] T. Guo, S. Guo, J. Wang, X. Tang, and W. Xu, “Promptfl: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model,” _IEEE Transactions on Mobile Computing_ , 2023.
  * [97] S. Ishihara, “Training data extraction from pre-trained language models: A survey,” _arXiv preprint arXiv:2305.16157_ , 2023.
  * [98] X. Zhou, J. Lu, T. Gui, R. Ma, Z. Fei, Y. Wang, Y. Ding, Y. Cheung, Q. Zhang, and X. Huang, “Textfusion: Privacy-preserving pre-trained model inference via token fusion,” in _Proceedings of the 2022 conference on empirical methods in natural language processing_ , Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for computational linguistics, Dec 2022, pp. 8360–8371. [Online]. Available: https://aclanthology.org/2022.emnlp-main.572
  * [99] X. Wu, F. Li, A. Kumar, K. Chaudhuri, S. Jha, and J. Naughton, “Bolt-on differential privacy for scalable stochastic gradient descent-based analytics,” in _Proceedings of the 2017 ACM International Conference on Management of Data_ , 2017, pp. 1307–1322.
  * [100] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [101] B. Weggenmann and F. Kerschbaum, “Differential privacy for directional data,” in _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , 2021, pp. 1205–1222.
  * [102] S. Hegselmann, A. Buendia, H. Lang, M. Agrawal, X. Jiang, and D. Sontag, “Tabllm: Few-shot classification of tabular data with large language models,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2023, pp. 5549–5581.
  * [103] S. L. Warner, “Randomized response: A survey technique for eliminating evasive answer bias,” _Journal of the American Statistical Association_ , vol. 60, no. 309, pp. 63–69, 1965.
  * [104] A. Sordoni, X. Yuan, M.-A. Côté, M. Pereira, A. Trischler, Z. Xiao, A. Hosseini, F. Niedtner, and N. L. Roux, “Deep language networks: Joint prompt training of stacked llms using variational inference,” _arXiv preprint arXiv:2306.12509_ , 2023.
  * [105] B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” _arXiv preprint arXiv:2104.08691_ , 2021.
  * [106] N. Shazeer and M. Stern, “Adafactor: Adaptive learning rates with sublinear memory cost,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4596–4604.
  * [107] C. Dwork and J. Lei, “Differential privacy and robust statistics,” in _Proceedings of the forty-first annual ACM symposium on Theory of computing_ , 2009, pp. 371–380.
  * [108] J. Gillenwater, M. Joseph, A. Munoz, and M. R. Diaz, “A joint exponential mechanism for differentially private top-$k$,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 7570–7582.
  * [109] U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis, “Generalization through memorization: Nearest neighbor language models,” _arXiv preprint arXiv:1911.00172_ , 2019.
  * [110] G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave, “Few-shot learning with retrieval augmented language models,” _arXiv preprint arXiv:2208.03299_ , 2022.
  * [111] S. Min, W. Shi, M. Lewis, X. Chen, W.-t. Yih, H. Hajishirzi, and L. Zettlemoyer, “Nonparametric masked language modeling,” _arXiv preprint arXiv:2212.01349_ , 2022.
  * [112] D. Biesner, R. Ramamurthy, R. Stenzel, M. Lübbering, L. Hillebrand, A. Ladi, M. Pielka, R. Loitz, C. Bauckhage, and R. Sifa, “Anonymization of german financial documents using neural network-based language models with contextual word representations,” _International journal of data science and analytics_ , pp. 1–11, 2022.
  * [113] Y. Nakamura, S. Hanaoka, Y. Nomura, N. Hayashi, O. Abe, S. Yada, S. Wakamiya, and E. Aramaki, “Kart: Privacy leakage framework of language models pre-trained with clinical records,” _arXiv preprint arXiv:2101.00036_ , 2020.
  * [114] N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin, “Analyzing leakage of personally identifiable information in language models,” _arXiv preprint arXiv:2302.00539_ , 2023.
  * [115] W. Xiong, X. L. Li, S. Iyer, J. Du, P. Lewis, W. Y. Wang, Y. Mehdad, W.-t. Yih, S. Riedel, D. Kiela _et al._ , “Answering complex open-domain questions with multi-hop dense retrieval,” _arXiv preprint arXiv:2009.12756_ , 2020.
  * [116] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in _European Conference on Computer Vision_. Springer, 2022, pp. 709–727.
  * [117] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” _International Journal of Computer Vision_ , vol. 130, no. 9, pp. 2337–2348, 2022.
  * [118] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [119] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in _Proceedings of the 38th international conference on machine learning_ , ser. Proceedings of machine learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8748–8763. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html
  * [120] H.-Y. Chen and W.-L. Chao, “On bridging generic and personalized federated learning for image classification,” _arXiv preprint arXiv:2107.00778_ , 2021.
  * [121] A. Shamsian, A. Navon, E. Fetaya, and G. Chechik, “Personalized federated learning using hypernetworks,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 9489–9502.
  * [122] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [123] A. Panda, T. Wu, J. T. Wang, and P. Mittal, “Differentially private in-context learning,” _arXiv preprint arXiv:2305.01639_ , 2023.
  * [124] D. Ofer, “Dbpedia classes,” accessed on: March 30, 2024. [Online]. Available: https://www.kaggle.com/datasets/danofer/dbpedia-classes?select=DBP_wiki_data.csv
  * [125] X. Li and D. Roth, “Learning question classifiers,” in _COLING 2002: The 19th international Ccnference on computational linguistics_ , 2002. [Online]. Available: https://www.aclweb.org/anthology/C02-1150
  * [126] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 2013, pp. 1631–1642.
  * [127] J. Wiebe, T. Wilson, and C. Cardie, “Annotating expressions of opinions and emotions in language,” _Language resources and evaluation_ , vol. 39, pp. 165–210, 2005.
  * [128] A. Howard, Devrishi, P. Culliton, and Y. Guo, “Natural language processing with disaster tweets,” 2019. [Online]. Available: https://kaggle.com/competitions/nlp-getting-started
  * [129] Huggingface, “Datasets: super_glue cb,” accessed on: March 30, 2024. [Online]. Available: https://huggingface.co/datasets/super_glue/viewer/cb
  * [130] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” _arXiv preprint arXiv:1804.07461_ , 2018.
  * [131] B. Bose, “Bbc news classification,” 2019. [Online]. Available: https://kaggle.com/competitions/learn-ai-bbc
  * [132] J. Liu, S. Cyphers, P. Pasupat, I. McGraw, and J. Glass, “A conversational movie search system based on conditional random fields,” in _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012.
  * [133] D. Kershaw and R. Koeling, “Elsevier oa cc-by corpus,” _arXiv preprint arXiv:2008.00774_ , 2020.
  * [134] Cornell University, “arxiv dataset,” Access on: February 20, 2024. [Online]. Available: https://www.kaggle.com/datasets/Cornell-University/arxiv
  * [135] W. Christopher, S. Stephanie, M. Julie, and M. Kazuaki, “Ace 2005 multilingual training corpus,” 2006. [Online]. Available: https://doi.org/10.35111/mwxc-vh88
  * [136] B. Klimt and Y. Yang, “The enron corpus: A new dataset for email classification research,” in _European conference on machine learning_. Springer, 2004, pp. 217–226.
  * [137] S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” _arXiv preprint arXiv:1609.07843_ , 2016.
  * [138] A. Fan, D. Grangier, and M. Auli, “Controllable abstractive summarization,” _arXiv preprint arXiv:1711.05217_ , 2017.
  * [139] R. Tito, K. Nguyen, M. Tobaben, R. Kerkouche, M. A. Souibgui, K. Jung, L. Kang, E. Valveny, A. Honkela, M. Fritz, and D. Karatzas, “Privacy-aware document visual question answering,” _arXiv preprint arXiv:2312.10108_ , 2023.
  * [140] B. Gliwa, I. Mochol, M. Biesek, and A. Wawer, “Samsum corpus: A human-annotated dialogue dataset for abstractive summarization,” _arXiv preprint arXiv:1911.12237_ , 2019.
  * [141] Skillsmuggler, “Amazon-ratings (beauty products),” accessed on: March 30, 2024. [Online]. Available: https://www.kaggle.com/datasets/skillsmuggler/amazon-ratings
  * [142] B. Becker and R. Kohavi, “Adult,” UCI Machine Learning Repository, 1996, DOI: https://doi.org/10.24432/C5XW20.
  * [143] P. Rathi, “Banking dataset - marketing targets,” accessed on: March 30, 2024. [Online]. Available: https://www.kaggle.com/datasets/prakharrathi25/banking-dataset-marketing-targets
  * [144] M. M. Marchetti, “Predicting blood donations,” accessed on: March 30, 2024. [Online]. Available: https://www.kaggle.com/code/mmmarchetti/predicting-blood-donations
  * [145] O. R. Opeyemi, “California_housing_dataset,” accessed on: March 30, 2024. [Online]. Available: https://www.kaggle.com/code/olanrewajurasheed/california-housing-dataset
  * [146] A. Kadra, M. Lindauer, F. Hutter, and J. Grabocka, “Well-tuned simple nets excel on tabular datasets,” _Advances in neural information processing systems_ , vol. 34, pp. 23 928–23 941, 2021.
  * [147] UCI, “Pima indians diabetes database,” accessed on: March 30, 2024. [Online]. Available: https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database
  * [148] Fedesoriano, “Heart failure prediction dataset,” accessed on: March 30, 2024. [Online]. Available: https://www.kaggle.com/datasets/fedesoriano/heart-failure-prediction
  * [149] J. N. van Rijn and J. K. Vis, “Endgame analysis of dou shou qi,” _ICGA journal_ , vol. 37, no. 2, pp. 120–124, 2014.
  * [150] G. Griffin, A. Holub, and P. Perona, “Caltech-256 object category dataset,” 2007.
  * [151] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, “Adapting visual category models to new domains,” in _Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11_. Springer, 2010, pp. 213–226.
  * [152] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 1406–1415.
  * [153] Z. Chen, M. Zhu, C. Yang, and Y. Yuan, “Personalized retrogress-resilient framework for real-world medical federated learning,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24_. Springer, 2021, pp. 347–356.
  * [154] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [155] J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni, “Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification,” _Scientific Data_ , vol. 10, no. 1, p. 41, 2023.
  * [156] P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” _Scientific data_ , vol. 5, no. 1, pp. 1–9, 2018.
  * [157] N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler _et al._ , “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic),” in _2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018)_. IEEE, 2018, pp. 168–172.
  * [158] Anonos, “Anonos prompt protector,” accessed on: March 27, 2024. [Online]. Available: https://www.anonos.com/solutions/prompt-protector-ai-privacy
  * [159] Prompt, “The singular platform for genai security,” accessed on: March 27, 2024. [Online]. Available: https://www.prompt.security/
  * [160] Whylabs, “Ensure safe and responsible usage of large language models,” accessed on: March 27, 2024. [Online]. Available: https://whylabs.ai/llm-security
  * [161] CalypsoAI, “Harness ai safely,” accessed on: March 27, 2024. [Online]. Available: https://calypsoai.com/
  * [162] Lakera, “Protect your ai against safety and security threats, instantly,” accessed on: March 27, 2024. [Online]. Available: https://www.lakera.ai/
  * [163] LLM Guard, “Llm guard - the security toolkit for llm interactions,” accessed on: March 27, 2024. [Online]. Available: https://llm-guard.com/
  * [164] Guardrails, “Guardrails ai,” accessed on: March 27, 2024. [Online]. Available: https://github.com/guardrails-ai/guardrails
  * [165] B. D. Rouhani, M. S. Riazi, and F. Koushanfar, “Deepsecure: Scalable provably-secure deep learning,” in _Proceedings of the 55th annual design automation conference_ , 2018, pp. 1–6.
  * [166] B. Pecher, I. Srba, and M. Bielikova, “Fine-tuning, prompting, in-context learning and instruction-tuning: How many labelled samples do we need?” _arXiv preprint arXiv:2402.12819_ , 2024.
  * [167] P. Sun, “Fine-tuning vs prompting, can language models understand human values?” _arXiv preprint arXiv:2403.09720_ , 2024.
  * [168] M. Mosbach, T. Pimentel, S. Ravfogel, D. Klakow, and Y. Elazar, “Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation,” _arXiv preprint arXiv:2305.16938_ , 2023.
  * [169] J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [170] I. S. Rubinstein and W. Hartzog, “Anonymization and risk,” _Wash. L. Rev._ , vol. 91, p. 703, 2016.