(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Nanyang Technological University, † S-Lab 22institutetext:
University of Oxford $~{}~{}~{}^{3}$ Monash University

# ClusteringSDF: Self-Organized Neural
Implicit Surfaces for 3D Decomposition

Tianhao Wu† 11 Chuanxia Zheng 22 Tat-Jen Cham 11 Qianyi Wu 33

###### Abstract

3D decomposition/segmentation still remains a challenge as _large-scale 3D
annotated data is not readily available_. Contemporary approaches typically
leverage 2D machine-generated segments, integrating them for 3D consistency.
While the majority of these methods are based on NeRFs, they face a potential
weakness that the instance/semantic embedding features derive from independent
MLPs, thus preventing the segmentation network from learning the geometric
details of the objects directly through radiance and density. In this paper,
we propose ClusteringSDF, a novel approach to achieve both segmentation and
reconstruction in 3D via the neural implicit surface representation,
specifically Signal Distance Function (SDF), where the segmentation rendering
is directly integrated with the volume rendering of neural implicit surfaces.
Although based on ObjectSDF++, ClusteringSDF _no longer requires the ground-
truth segments for supervision_ while maintaining the capability of
reconstructing individual object surfaces, but purely with the noisy and
inconsistent labels from pre-trained models. As the core of ClusteringSDF, we
introduce a high-efficient clustering mechanism for lifting the 2D labels to
3D and the experimental results on the challenging scenes from ScanNet and
Replica datasets show that ClusteringSDF can achieve competitive performance
compared against the state-of-the-art with significantly reduced training
time.

###### Keywords:

3D segmentation neural implicit surface representation clustering

##  1 Introduction

Figure 1: Taking RGB images from a scene and the corresponding 2D labels from
a pre-trained segmentation model, our ClusteringSDF is able to reconstruct the
surface of the scene while fusing these inconsistent segments to be coherent
and more accurate in 3D. Furthermore, it learns an object-compositional neural
implicit representation and can reconstruct the surfaces of individual objects
purely from these noisy labels.

The vision community has rapidly improved 3D reconstruction and novel view
synthesis (NVS) results with various neural implicit representations, such as
the Occupancy networks [21], DeepSDF [26], Scene Representation Network [30],
Neural Radiance Fields (NeRF) [22], and Light Field Network (LFN) [29].
Despite their remarkable performance, these systems ignore one critical fact:
that the world is composed of individual 3D objects. It is significant for
multiple downstream applications, such as Augmented and Virtual Reality (AR &
VR) [6, 16], autonomous driving [17] and embodied AI [18, 2], to _decompose
and be able to interact with these 3D assets within a complex scene_.

At the same time, segmentation has undergone improvements [45, 3, 12, 40] on
2D images in recent years. However, recognizing and decomposing multiple 3D
assets in a complex 3D scene is still a major challenge. This is largely due
to the complexity of _processing and fusing 2D segmentation maps from multiple
views into a consistent 3D representation_ , especially when working with _3D
instances_. The accuracy of instance segmentation results, although
potentially high, pertains exclusively to the current single view. However,
labels assigned to each unique object in the scene _do not remain consistent
across views_ [34].

In this paper, we aim to tackle the problem of _object-centered_ 3D
reconstruction, wherein the goal is to _simultaneously_ enable 3D scene
reconstruction, as well as _3D assets decomposition or segmentation_. Unlike
previous approaches [44, 37, 38] that rely on ground-truth annotations, our
method leverages unaligned 2D segmentation results obtained from pre-trained
models, fusing them to be consistent in the 3D representation. Our _key
motivation_ is that, annotating precisely consistent 3D labels is challenging,
while the robust 2D segmentation models are widely available in the current
vision zoo. Although recent research [15, 20, 10, 35, 28, 1] have already
focused on this area and have yielded good results in distilling 2D labels or
features in 3D space, they are mostly built upon a holistic NeRF [22] for all
instances within a scene. A potential weakness lies in the disconnect between
segmentation results and 3D reconstruction results, because the embedding
features for semantic/instance rendering are derived from independent MLPs
that do _not_ necessarily correlate with the underlying RGB or depth. This
separation makes it more difficult for the segmentation module to learn
geometric details of the objects through radiance and density.

In contrast, neural implicit surface representations [42, 25, 41, 36, 37] that
express 3D scenes as signed distance functions (SDFs) have the potential
capability to resolve this segmentation challenge. This is achieved by
decomposing each instance as an independent _object-level_ surface, and then
further fusing all these 3D assets into a whole 3D scene (Fig. 2). However,
these SDF-based methods require precisely consistent labels for supervision,
and currently, there is no proper solution to fuse the 2D information that is
inconsistent across different views.

To address these aforementioned challenges, we propose a new neural implicit
surface representation method, named _ClusteringSDF_ , which is built upon
ObjectSDF++ [38], to lift the inconsistent 2D segmentation into consistent 3D
assets. While ObjectSDF++ [38] volume-renders object opacity masks and
directly matches them to ground-truth labels, we replace this component
entirely with our novel clustering method, so as to achieve segment
assignments purely with the machine-generated 2D labels. The crucial idea
behind our ClusteringSDF is to merge the predicted individual SDF channels
from the implicit network, and treat them as SDF probability distributions or
SDF features. Instead of simply assigning each pixel to a specific
predetermined SDF channel, the objective of ClusteringSDF is to _bring SDF
distributions belonging to the same object closer, while concurrently
separating those from different objects._ The determination of whether pixels
belong to the same object depends on their labels in the current view.
Compared against NeRF-based segmentation methods that predict radiance field
densities, ClusteringSDF renders neural implicit surfaces, where the gradients
are concentrated around surface regions, leading to faster training and
sharper segmented results.

In summary, our main contributions are as follows:

  * •

A novel _self-organized_ approach for fusing 2D machine-generated instance/
semantic labels in 3D via neural implicit surface representation while
simultaneously enabling holistic 3D reconstruction;

  * •

A novel clustering mechanism for aligning these multi-view inconsistent
labels, resulting in coherent 3D segment representations. Instead of computing
similarity between each pair of pixels as contrastive loss, our method
achieves high-efficient clustering by constraining the clustering centers
within a simplex through normalization;

  * •

Without ground-truth labels as supervision, our ClusteringSDF maintains the
object-compositional representations of the scene and its objects.

##  2 Related Work

###  2.1 NeRFs for Semantic/Panoptic 3D Scene Modeling

As one of the major approaches to achieve high-quality 3D reconstruction in
recent years, the NeRF [22] learning-by-volume-rendering paradigm has led to
considerable progress in the task of 3D scene understanding [44, 9, 15, 35,
13, 32, 10]. Previous approaches have widely explored semantic 3D modeling and
scene decomposition, from unsupervised foreground-background scene
decomposition [39], to leveraging weak signals like text or object motion [23,
33, 7], and further to enhancing NeRF models with semantic labeling
capabilities using annotations from 2D datasets [27]. SemanticNeRF [44]
augmented the original NeRF to enable scene-specific semantic representation,
by appending a segmentation renderer before injecting viewing directions into
the MLP.

Later works started focusing on the challenge of panoptic segmentation in 3D.
DM-NeRF [35] learns unique codes of each identical object in 3D space from
only 2D supervision, while Instance-NeRF [19], by pre-training a NeRF and
extracting its radiance and density field, utilizes 3D supervision to match
inconsistent 2D panoptic segmentation maps. Similarly, Panoptic-NeRF [9]
leverages coarse 3D bounding primitives for panoramic and semantic
segmentation of open scenes. To decompose the scene into dynamic “things” and
background “stuff” given a sequence of images, Panoptic Neural Fields (PNF)
[15] presents each individual object in the scene with a unique MLP and a
dynamic track.

More recently, Panoptic Lifting [28] and Contrastive Lift [1] focused on
directly lifting the 2D semantic/instance segment information without pre-
training a 3D model or requiring any 3D bounding boxes. Despite the impressive
performance of these works, there remain some issues that have not been
resolved. While NeRF-based methods excel in rendering photorealistic 3D
scenes, the semantic/instance embedding is not integrated directly with the
learning of radiance (color) and density across the scene as they are derived
from independent MLPs. Such deficiency makes it difficult for the model to
learn the relationship between segmentation and 3D reconstruction, usually
taking more training iterations to achieve a nuanced understanding of the
scene. On the other hand, the 2D machine-generated labels are inconsistent,
with some over-segmented or under-segmented cases in the same scene. When the
geometry information of the segmentation provided by these labels is
inaccurate, such independence prevents the MLPs for segmentation from
capturing the full geometric details of the objects through radiance and
density, leading to inaccurate results.

###  2.2 Object-Compositional Neural Implicit Surface Representation

Several research works [38, 37, 42, 41, 36] have revealed the powerful surface
reconstruction ability of neural implicit surface representations. By
implicitly representing 3D scenes as SDFs, the gradients are concentrated
around the surface regions, leading to a direct connection with the geometric
structure. On the contrary, for methods like Contrastive Lift [1] and Panoptic
Lifting [28], semantic rendering does not directly affect RGB or depth
rendering, implying weak geometric constraints to supervise the correctness of
segmentation.

Previous ObjectSDF [37] and ObjectSDF++ [38] introduce mechanisms for
representing individual objects within complex indoor scenes. Compared to
NeRF-based methods, the segmentation result has a direct impact on the final
geometry rendering, including RGB, depth, and normals. An excessive segmented
region would require the model to render unrelated geometry at the
corresponding location in space to satisfy such structure, while an incomplete
segmentation would result in parts being missed from the geometric rendering,
both of which will be penalized by the supervision. However, they necessitate
the input of ground-truth instance labels for accurate surface reconstruction,
making it difficult to apply them to in-the-wild scene segmentation. In this
paper, we propose a new framework, based on ObjectSDF++ [38], that
incorporates a clustering alignment technique designed to fuse the
inconsistent 2D instance/semantic labels.

###  2.3 Clustering Operators for Segmentation

Due to the fact that the labels of instances are almost entirely different
across different camera viewpoints, relying directly on the values of the
labels for alignment is inappropriate. Previous research [43, 5, 8, 14, 24]
have explored clustering mechanisms for consistent 3D segmentation.
Contrastive Lift [43] attempts to make multi-view labels compatible using a
contrastive loss function, together with a concentration loss between pixel-
level embedding features. However, the computation of pixel-level contrastive
loss can be time-consuming. To address this issue, we designed a more
efficient way for clustering in this paper while achieving competitive
segmentation results against the state-of-the-art models.

##  3 Methodology

Figure 2: Overview of ClusteringSDF. It is designed to fuse inconsistent 2D
segments to 3D while reconstructing the object geometry by predicting their
SDFs. To achieve this, we sample rays from single 2D segment maps and split
them into $N$ groups corresponding to the $N$ distinct 2D labels
$\\{C_{1},...,C_{N}\\}$. The proposed $\mathcal{L}_{\text{diff}}$ then
leverages normalized SDF distributions encompassing $c$ channels for
individual objects (presented by different colors) and keeps the clustering
centers to be apart, with $\mathcal{L}_{\text{onehot}}$ further encouraging
the predicted clusters to be in the one-hot format.

Given a set of $K$ posed RGB images $\mathcal{I}=\\{I_{1},...,I_{K}\\}$ and
the associated segmentation mask set $\mathcal{S}=\\{S_{1},...,S_{K}\\}$
_derived from a pre-trained 2D segmenter_ , our main goal is to learn a model
that not only reconstructs the whole 3D scene but also _decomposes all 3D
assets within it_. Different from conventional methods that require precisely
annotated semantic labels, here we consider using _labels from pre-trained 2D
segmentation models_. However, there are two main challenges if we directly
use these machine-generated labels: i) For any object in the scene, its
corresponding segmented regions in different camera views are generally
assigned different instance labels, even though it is the same object. ii)
Furthermore, the estimated semantic labels can be noisy and inconsistent.

To leverage these off-the-shelf labels from pre-trained 2D models, our
ClusteringSDF, illustrated in Fig. 2, learns to align inconsistent 2D
segmentation via a neural implicit surface representation framework. We first
introduce an efficient clustering mechanism that overrides the need for
precise object labels in Sec. 3.2. To refine the segmentation in 3D, we
further design one-hot and regularization terms to constrain the predicted
value of each point in a single SDF channel, and a foreground-background loss
to differentiate between foreground instances and background stuff in Sec.
3.3. Moreover, in Secs. 3.4 and 3.5, we introduce a semantic loss for matching
the predetermined semantic labels, and a multi-view segmentation module to
address the challenge of distinguishing between two or more objects that have
never appeared together in any input image.

###  3.1 Preliminaries

####  3.1.1 Neural Implicit Surface.

In order to learn the SDF from multi-view images, a current approach [36, 41]
is to replace the NeRF volume density output $\sigma(p)$ by a learnable
transformation function of SDF value $d_{\Omega}(p)$, which is defined as the
signal distance from point $\mathbf{p}$ to the boundary surface. Such an
approach assumes that the objects have solid surfaces, and is supervised by
reconstruction loss together with an Ekional regularization loss.

####  3.1.2 Occlusion-Aware Object Opacity Rendering.

To further decompose the holistic scene representation into _individual 3D
assets_ , the previous ObjectSDF [37] directly predicts an SDF for each
object, with minimum values across all such SDFs yielding the complete SDF for
the scene. More recently, ObjectSDF++ [38] has replaced the semantic-based
cross-entropy loss by using an occlusion-aware object opacity, which can be
approximately expressed as:

$\hat{O}_{\mathcal{O}_{i}}(\mathbf{r})=\int_{v_{n}}^{v_{f}}T_{\Omega}(v)\sigma_{\mathcal{O}_{i}}(\mathbf{r}(v))\,dv,\quad
i=1,...,M.$ (1)

Considering a ray $\mathbf{r}(v)$, the rendering opacity $\hat{O}$ for an
object $\mathcal{O}_{i}$ can be expressed as an integral of its density
$\sigma_{\mathcal{O}_{i}}$ and the scene transmittance $T_{\Omega}(v)$ along
this ray from near bound ${v_{n}}$ to far bound ${v_{f}}$. Here $M$ is the
number of objects in the scene. The surface representations for each object
are then learned with supervision from ground truth instance segmentation
masks:

$\mathcal{L}_{O}=\mathbb{E}_{\mathbf{r}\in\mathcal{R}}[\frac{1}{M}\sum_{i=1...M}\lVert\hat{O}_{\mathcal{O}_{i}}(\mathbf{r})-O_{\mathcal{O}_{i}}(\mathbf{r})\rVert]$
(2)

Here $\mathcal{R}$ is the set of rays sampled in a minibatch and
$O_{\mathcal{O}_{i}}$ is the GT mask.

###  3.2 Clustering Loss

Let $\hat{O}$ denote the set of predicted opacity values in a minibatch, $y$
be the set of corresponding machine-generated 2D instance/semantic values in
this batch and $L$ be the set of unique labels in $y$. For each label
$l\in{L}$, we define mask $m_{l}$ to identify the elements in $y$ belonging to
the label $l$. Given a set of rays $\mathcal{R}$ sampled under the same view
in a minibatch, which contains $N$ unique labels $l_{1},l_{2},...,l_{N}$, we
split them into $N$ groups, denoted as $m_{l_{1}},m_{l_{2}},...,m_{l_{N}}$.
Instead of simply treating the predicted opacities as multiple individual SDF
channels, we adopt a perspective that, after Softmax normalization across
different channels, _they can instead be considered as probability
distributions that assign each ray to different instance categories with
different probabilities_. Based on this idea, rather than optimising Eq. 2, we
define a novel clustering loss:

$\mathcal{L}_{\text{clustering}}=\lambda_{1}\mathcal{L}_{\text{diff}}+\lambda_{2}\mathcal{L}_{\text{onehot}}+\lambda_{3}\mathcal{L}_{\text{reg}}$
(3)

It consists of three main components: a differentiation loss, a one-hot loss
and a regularization loss, which will be discussed in detail in this section.

The differentiation loss is formally defined as:

$\mathcal{L}_{\text{diff}}=-\frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j=i+1}^{N}\lVert\mu(\hat{O}^{s}_{m_{l_{i}}})-\mu(\hat{O}^{s}_{m_{l_{j}}})\rVert_{2},$
(4)

where $\hat{O}^{s}=\texttt{Softmax}(\hat{O})$, $\hat{O}_{m_{l*}}$ denotes the
prediction set corresponding to set $m_{l*}$, and
$\mu(\hat{O}^{s}_{m_{l_{*}}})$ is the clustering center of each category. For
the opacity predicted for each ray, which contains $c$ individual SDF
channels, we consider the corresponding ($c$-1)-dim simplex in a high-
dimension space. The vectors, _i.e_. the probabilities, can be seen as the
points in this space that are constrained to lie within or on the surface of
this simplex due to the softmax operation. As illustrated in Fig. 2, _the
distance between the vertices is greater than the distance between any other
pair of points inside a simplex_. Hence our $\mathcal{L}_{\text{diff}}$, which
aims to maximize the L2 distances between each pair of clustering centers
($\mu(\hat{O}^{s}_{m_{l_{i}}})$ and $\mu(\hat{O}^{s}_{m_{l_{j}}})$), will keep
them apart while encouraging them to occupy different vertices of this
simplex. Therefore, it creates a gradient pushing the clustering centers to
have only one channel to be 1 and the rest of the channels to be 0’s.

For each pixel, our objective is to assign it to one of the SDF channels.
However, $\mathcal{L}_{\text{diff}}$ only takes into account the average
predicted probabilities of each cluster, rather than the predictions for
individual rays. Based on this observation, we design the following one-hot
loss $\mathcal{L}_{\text{onehot}}$ for further encouraging the model to
produce approximate one-hot predictions for each pixel:

$\mathcal{L}_{\text{onehot}}=\frac{1}{|\mathcal{P}_{fg}|}\sum_{p_{i}\in{\mathcal{P}_{fg}}}\lVert\hat{O}_{p_{i}}-\text{onehot}(\hat{O}^{s}_{p_{i}})\rVert,$
(5)

where $\mathcal{P}_{fg}$ are the pixels recognized as foreground objects in 2D
segments. The onehot operation sets the largest-valued channel in
$\hat{O}^{s}_{p_{i}}$ to 1, and other channels to 0’s. However, our
experiments show that the $\mathcal{L}_{\text{onehot}}$ causes the model to
converge too fast and fall into local optima, leading to some cluster centers
that should be different being pushed to the same vertices. To allow the model
to differentiate between different clusters at a more steady rate, we added an
additional $\mathcal{L}_{reg}$ that encourages higher entropy by minimizing
the _variance of softmax values_ in each cluster.

$\mathcal{L}_{\text{reg}}=\frac{1}{N}\sum_{i=1}^{N}\text{Var}(\hat{O}^{s}_{m_{l_{i}}}),\quad
l_{i}\neq 0.$ (6)

Despite the fact that the objectives of $\mathcal{L}_{\text{onehot}}$ and
$\mathcal{L}_{\text{reg}}$ contradict each other, by adjusting the weights, we
are able to slow down the entropy reduction of the model, providing
$\mathcal{L}_{\text{diff}}$ with enough time to separate the clustering
centers.

Compared with the contrastive loss in [1], $\mathcal{L}_{\text{clustering}}$
does _not_ focus on each positive/negative pixel pair, thus reducing the
complexity from $O(N*|\mathcal{R}|^{2})$ to $O(N^{2}*|\mathcal{R}|)$, where
the number of unique labels $N$ is considerably smaller than the number of
rays $|\mathcal{R}|$. The Softmax operation here plays a crucial role in
transforming the predicted opacity into a probability distribution over the
SDF channels. Besides magnifying differences between channels and preventing
the exploding or vanishing gradients, it inherently favors an unimodal
distribution, where one class is significantly more probable than others,
encouraging the model to be more decisive.

###  3.3 Foreground-background Loss

For 2D segmentation results, some objects are partially visible in some views,
in which case they may not be segmented, while background stuff may be
incorrectly segmented as foreground objects. To better distinguish between
background stuff and foreground objects, we can regulate the value of channel
#0 of the predicted opacity, which is the default channel to represent the
background. When the value of channel #0 approaches 1, it indicates a higher
probability that the pixel belongs to the background; conversely, a lower
value suggests a greater likelihood of being foreground. We design the
following loss function to aid the model in distinguishing between foreground
and background more rapidly:

$\displaystyle\mathcal{L}_{\text{fg-
bg}}=\lambda_{4}\mathcal{L}_{\text{bg}}+\lambda_{5}\mathcal{L}_{\text{fg}}=\lambda_{4}\frac{1}{|\mathcal{P}_{bg}|}\sum_{p_{i}\in{\mathcal{P}_{bg}}}|\hat{O}_{p_{i}0}-1|+\lambda_{5}\frac{1}{|\mathcal{P}_{fg}|}\sum_{p_{i}\in{\mathcal{P}_{fg}}}|\hat{O}_{p_{i}0}|$
(7)

By adjusting the values of $\lambda_{4}$ and $\lambda_{5}$, we can control how
well the model differentiates between foreground objects and background stuff.

###  3.4 Semantic Segmentation

To handle semantic segmentation, $\mathcal{L}_{\text{clustering}}$ alone is
not sufficient, as it only requires that pixels of the same category be
assigned to the same label, not that they correctly match the predefined
semantic labels. Therefore, we use an additional semantic loss to match
cluster centers with one-hot vectors of the semantic categories, as derived
from a pre-trained 2D generator.

$\mathcal{L}_{\text{sem}}=\frac{1}{N}\sum_{i=1}^{N}\mu(\mathcal{C}_{m_{l_{i}}})\lVert\mu(\hat{O}^{s}_{m_{l_{i}}})-\text{onehot}(y_{i})\rVert_{2},\quad
l_{i}\neq 0$ (8)

Here $\mathcal{C}$ is the confidence map obtained from test-time augmentation,
like as done in [28]. By combining $\mathcal{L}_{\text{sem}}$ with our
$\mathcal{L}_{\text{clustering}}$, the predicted semantic labels are
efficiently aligned with the predetermined labels.

###  3.5 Instance Segmentation

Existing methods, including Contrastive Lift [1] and ObjectSDF++ [38],
calculate the loss within the same view for instance segmentation. However,
this leads to a potential problem: if two objects never appear in the same
input image, they might end up being assigned to the same label. To address
this issue, we modify the sampling strategy from one single camera view to
multiple views. While 2D instance segmentation cannot support this multi-view
supervision as the objects are generally labelled inconsistently between
views, we instead utilize 2D semantic segmentation to alleviate this problem,
which provides both coherency and disambiguation across views.

Specifically, to avoid excessive increases in computation, we randomly select
a small number of views, from which we sample a limited number of rays within
a minibatch, while maintaining the previous single-view sampling setting. Then
we introduce a cross-view loss function to encourage the model to assign
different instance labels to objects belonging to distinct semantic classes:

$\mathcal{L}_{\text{cross\\_view}}=\exp(-\sum_{v\in\mathcal{V}}\sum_{\begin{subarray}{c}l_{1}\in{L_{c}}\\\
l_{2}\in{L_{v}}\end{subarray}}\lVert\mu(\hat{O}^{s}_{m_{l_{1}}})-\mu(\hat{O}^{s}_{m_{l_{2}}})\rVert_{2}),\quad
l_{1}\neq{l_{2}}$ (9)

Here $\mathcal{V}$ is the set of random sampled extra views, while ${L_{c}}$
and ${L_{v}}$ are the sets of unique _semantic_ labels under the current view
and the extra view respectively.

###  3.6 Model Training

Except for the 2D segmentation fusion part, we continue to use the loss
function of ObjectSDF++ [38] for color/normal/depth map
$\mathcal{L}_{\text{rec}}$ and SDF reconstruction $\mathcal{L}_{\text{SDF}}$.
Therefore, the overall training loss for semantic segmentation:

$\mathcal{L}_{\text{semantic}}=\mathcal{L}_{\text{rec}}+\mathcal{L}_{\text{SDF}}+\mathcal{L}_{\text{clustering}}+\mathcal{L}_{\text{sem}}+\mathcal{L}_{\text{fg-
bg}}$ (10)

and for instance segmentation:

$\mathcal{L}_{\text{instance}}=\mathcal{L}_{\text{rec}}+\mathcal{L}_{\text{SDF}}+\mathcal{L}_{\text{clustering}}+\mathcal{L}_{\text{cross\\_view}}+\mathcal{L}_{\text{fg-
bg}}$ (11)

###  3.7 Label-comprehensive Sampling and Key-frame Selection

The baseline ObjectSDF++ [38] randomly samples a number of rays from the same
camera view within a minibatch. However, in our ClusteringSDF, random sampling
does not guarantee the representation of every instance label, and the absence
of one or more labels in the sampling process may lead to their omission in
the model’s training phase. This oversight results in slower convergence rates
or an inability to correctly distinguish all labels in the final segmentation
result. In order to make it easier for the model to capture different labels,
we adjusted the sampling strategy to ensure that the 1024 rays contain at
least one of each of the different label values for the current view.
Furthermore, frames with more unique label values will be recognized as
keyframes during the data loading phase and will be given extra attention by
the model, with higher training frequency.

##  4 Evaluation

###  4.1 Experiment settings

####  4.1.1 Baselines.

We performed a thorough comparison of ClusteringSDF to the state-of-the-art
segmentation methods, including SemanticNeRF [44]${}_{\text{NeurIPS'21}}$,
Panoptic Neural Fields [15]${}_{\text{CVPR'22}}$, Mask2Former
[3]${}_{\text{CVPR'22}}$, DM-NeRF [35]${}_{\text{ICLR'23}}$, Panoptic Lifting
[28]${}_{\text{CVPR'23}}$, and Contrastive Lift [1]${}_{\text{NeurIPS'23}}$.

####  4.1.2 Dataset.

We assessed all models on ScanNet [4] and Replica [31] datasets, following
prior Panoptic Lifting [28] and Contrastive Lift [1]. The segmentation labels
for supervision were derived from Mask2Former [3], while for quantitative
metrics we compared the predicted results with the ground truth labels.

####  4.1.3 Metrics.

Following prior methods [28, 1], we report the standard metrics, including
scene-level Panoptic Quality ($\text{PQ}^{\text{scene}}$) [28] and mean
intersection over union (mIoU). For the ablation study, in order to make
comparisons at a more granular level, we add two more metrics –
$\text{RQ}^{\text{scene}}$ [28], and our self-designed edge accuracy (EA).
Based on the standard Panoptic Quality (PQ) and Recognition Quality (RQ) [11],
$\text{PQ}^{\text{scene}}$ measures the scene-level consistency of all
instance/stuff IDs across multiple views, while $\text{RQ}^{\text{scene}}$
measures the scene-level ID recognition accuracy. To reflect the geometric
accuracy, EA calculates the average Dice value between the edge of predicted
and GT labels, which are paired by matching the densest labels in the
predicted result with each GT label.

####  4.1.4 Implementation Details.

Our ClusteringSDF is small enough to be trained on a single 12 GB GTX 2080Ti
GPU. In particular, we train the model for 200 epochs, around 120k iterations
on all scenes. For each scene we train the model from scratch with Eq. 10 and
Eq. 11 separately.

###  4.2 Main Results

Figure 3: Examples of semantic and instance segmentation on novel view. Our
ClusteringSDF achieves very competitive results against previous 3D
segmentation methods. Note that the instance and semantic results of our model
are slightly different as we train two individual networks respectively.
Limited by the space, multi-view consistent segmentation results are provided
in the supplementary materials. (Zoom in to see the details.)  Figure 4:
Examples of our segmentation results in 3D. Detail comparisons to existing
methods on 3D consistency are provided in supplementary materials.  Table 1:
Results on ScanNet and Replica datasets. The performance of all prior work is
sourced from [28, 1]. For each dataset, we report the
$\text{PQ}^{\text{scene}}$ and mIoU metrics.

Method |  ScanNet [4] |  |  Replica [31]
---|---|---|---
$\text{PQ}^{\text{scene}}$ $\uparrow$ | mIoU $\uparrow$ |  |  $\text{PQ}^{\text{scene}}$ $\uparrow$ | mIoU $\uparrow$
Mask2Former [3] | - | 46.7 |  | - | 52.4
SemanticNeRF [44] | - | 59.2 |  | - | 58.5
DM-NeRF [35] | 41.7 | 49.5 |  | 44.1 | 56.0
PNF [15] | 48.3 | 53.9 |  | 41.1 | 51.5
PNF + GT BBoxes | 54.3 | 58.7 |  | 52.5 | 54.8
Panoptic Lifting [28] | 58.9 | 65.2 |  | 57.9 | 67.2
Contrastive Lift [1] | 62.3 | 65.2 |  | 59.1 | 67.0
ClusteringSDF (Ours) | 61.0 | 66.2 |  | 59.3 | 68.9

We compare ClusteringSDF to the state-of-the-art methods in 3D segmentation on
ScanNet [4] and Replica [31] in Tab. 1. Here, we report both the
$\text{PQ}^{\text{scene}}$ and mIoU metrics. As shown in the table, the
proposed ClusteringSDF outperforms the baseline models on most of the metrics,
except for the $\text{PQ}^{\text{scene}}$ on ScanNet [4] against the state-of-
the-art Contrastive Lift [1]. We believe it is mainly due to the relatively
low quality RGB input of ScanNet [4], making it harder for our model to
reconstruct the surface of the scene. Especially on mIoU, which reflects the
quality of the predicted semantic maps, our model outperforms the state-of-
the-art methods by $+1.0$ and $+1.9$ points respectively. These results
demonstrate the effectiveness of our designed method.

The qualitative comparisons are visualized in Fig. 3. ClusteringSDF achieves
good results _without manually labeled dense semantic annotations_. It is
worth noting that our ClusteringSDF can segment tiny objects better than the
baseline models (_e.g_. the pillow on the sofa). Furthermore, to demonstrate
that our segmentation results are consistent in 3D, we render the
corresponding results on the mesh, seen as Fig. 4. These results provide
qualitative evidence that our model is effective for 3D segmentation and
decomposition.

Table 2:  Ablation study on Replica dataset. To test the validity of each
component of ClusteringSDF, we remove each loss and test the performance. It
should be noted that we use the same semantic segment results to highlight the
difference in instance segmentation, which is more important for individual
object representation.

Component | Supervision |  |  $\text{PQ}^{\text{scene}}$ $\uparrow$ |  |  $\text{RQ}^{\text{scene}}$ $\uparrow$ |  | EA $\uparrow$
---|---|---|---|---|---|---|---
ObjectSDF++ | ground truth |  | 64.2 |  | 88.3 |  | 28.9
ObjectSDF++ w/ $\mathcal{L}_{\text{clustering}}$ | ground truth |  | 69.1 |  | 95.4 |  | 58.9
ObjectSDF++ | M2F [3] |  | 43.8 |  | 55.3 |  | 12.9
ClusteringSDF w/o $\mathcal{L}_{\text{onehot}}$ | M2F [3] |  | 56.3 |  | 61.3 |  | 23.2
ClusteringSDF w/o $\mathcal{L}_{\text{reg}}$ | M2F [3] |  | 52.6 |  | 57.8 |  | 15.7
ClusteringSDF w/o $\mathcal{L}_{\text{fg-bg}}$ | M2F [3] |  | 57.7 |  | 65.8 |  | 21.3
ClusteringSDF | M2F [3] |  | 59.2 |  | 69.1 |  | 24.8

###  4.3 Ablation study

We ran a number of ablations on Replica [31] to analyze the effectiveness of
each core component in ClusteringSDF. The results are shown in Tab. 2 and Fig.
5. Note that for semantic segmentation, $\mathcal{L}_{\text{sem}}$ (Eq. 8)
plays an irreplaceable role for matching the labels, without which the
semantic segmentation cannot be proceeded. Therefore, we mainly explore the
effect of our other proposed loss functions by probing their performance on
instance segmentation, which directly impact the subsequent object surface
reconstruction.

####  4.3.1 Clustering Loss.

To investigate the efficiency of the primary component for instance
segmentation – $\mathcal{L}_{\text{clustering}}$ (Eq. 3), we integrate it into
ObjectSDF++ [38] to assess whether it leads to further improvements when the
GT instance labels are available. The results are shown in the first two rows
in Tab. 1, where the overall performance demonstrates enhancement with the
addition of $\mathcal{L}_{\text{clustering}}$. Notably, the edge accuracy
exhibits a substantial improvement of 30 points. We attribute this to
ObjectSDF++ being inferior at reconstructing planar objects, tending to
integrate these objects into the background or into the surface of other
objects. Conversely, $\mathcal{L}_{\text{clustering}}$ substantially
encourages the model to discriminate between different objects, thus enhancing
the performance. This improvement is further validated by the qualitative
results depicted in Fig. 5(a).

(a)

(b)

(c)

Figure 5: Qualitative results for ablation study.

####  4.3.2 One-hot Loss, Reg Loss and Fg-bg Loss.

The loss term $\mathcal{L}_{\text{diff}}$ (Eq. 4) is also _indispensable for
segmentation_ , omitting which the model will no longer work. Therefore, we
demonstrate the effect of our other designed loss terms
$\mathcal{L}_{\text{onehot}}$, $\mathcal{L}_{\text{reg}}$ and
$\mathcal{L}_{\text{fg-bg}}$ on different settings. The quantitative results
in Tab. 2 show that disabling each of these terms will lead to a substantial
drop in the final performance. We also provide rendering examples for more
intuitive comparisons in Fig. 5(b). It shows that i) removing
$\mathcal{L}_{\text{onehot}}$ significantly affects the smoothness, resulting
in the same object being assigned with multiple labels; ii) the performance
will be noticeably impaired without $\mathcal{L}_{\text{reg}}$ to constrain
the speed of entropy reduction; iii) $\mathcal{L}_{\text{fg-bg}}$ encourages
the model to distinguish between the foreground _objects_ and the background
_stuff_ , without which they can be incorrectly segmented (_e.g_. the table in
the center should not be segmented).

####  4.3.3 Cross-View Loss.

Qualitative examples in Fig. 5(c) shows that when different instances never
appear together in one image, they can be potentially assigned with same
labels (the left bed and the chair, the right bed and the footrest), while
with our designed $\mathcal{L}_{\text{cross\\_view}}$ they are correctly
assigned with diverse labels.

###  4.4 Object Surface Reconstruction

As ClusteringSDF is built upon object-compositional neural implicit surface
representation, while our $\mathcal{L}_{\text{onehot}}$ encourages the model
to assign each pixel to distinct SDF channels, it maintains the capability of
reconstructing the surface of individual objects in the scene. To validate
this, in Fig. 6, we show examples of surface reconstruction on both ScanNet
and Replica scenes. Compared against the reconstruction results of ObjectSDF++
_trained with ground-truth labels_ , our ClusteringSDF can achieve nearly
comparable reconstruction quality _purely with 2D machine-generated labels_.

Figure 6: Examples of object surface reconstruction. Note that as ObjectSDF++
uses ground-truth labels for supervision, it imposes more detailed
segmentation of some objects, resulting in some of the results appearing to be
inferior to our approach.

###  4.5 Training time comparison

Here we report the total iteration number and time needed to complete the
training for the state-of-the-art [1, 28, 38] and ClusteringSDF in Tab. 3. The
results indicate that ClusteringSDF can converge significantly more rapidly
than the baseline methods while still achieving competitive or even better
results.

Table 3: Training time comparison.

Method | iteration | training |  |  | Method | iteration | training
---|---|---|---|---|---|---|---
Panoptic Lifting [28] | 400K | >40h |  |  | ObjectSDF++ [38] | 200K | <12h
Contrastive Lift [1] | 400K | >30h |  |  | ClusteringSDF (Ours) | 120K | <8h

##  5 Limitations

While ClusteringSDF shows good ability in fusing machine-generated 2D
segmentation labels to 3D and reconstructing the object surfaces, there still
remains some limitations. ClusteringSDF is not able to handle semantic and
instance segmentation simultaneously and requires separate training for now,
thus leading to slight inconsistencies between the semantic and instance
segmentation results. To differentiate instances that had never appeared
together in any input image, using semantic labels is still insufficient, as
these instances may belong to the same semantic category. In fact, the
semantic labels here can easily be replaced with other supervisions as long as
they are consistent across views, such as sparse manual labeling, which can be
a practical direction for future development.

##  6 Conclusion

We propose ClusteringSDF, a novel approach for lifting 2D machine-generated
segmentation labels to 3D via the object-compositional neural implicit surface
representation while being able to simultaneously reconstruct the surfaces for
the objects in the scene. Experimental results have shown that our model
achieves state-of-the-art performance for rendering consistent 3D segmentation
labels. Through our designed high-efficient clustering mechanism, our
ClusteringSDF costs less than a quarter of the training time on each scene
compared with the latest NeRF-based 3D segmentation methods. Furthermore, we
show the capability of our approach in reconstructing surfaces for individual
objects in the scene entirely from these inconsistent 2D labels, which we
believe will provide inspiration for the subsequent research.

## References

  * [1] Bhalgat, Y., Laina, I., Henriques, J.F., Zisserman, A., Vedaldi, A.: Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion. arXiv preprint arXiv:2306.04633 (2023)
  * [2] Byravan, A., Humplik, J., Hasenclever, L., Brussee, A., Nori, F., Haarnoja, T., Moran, B., Bohez, S., Sadeghi, F., Vujatovic, B., et al.: Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 9362–9369. IEEE (2023)
  * [3] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)
  * [4] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5828–5839 (2017)
  * [5] De Brabandere, B., Neven, D., Van Gool, L.: Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551 (2017)
  * [6] Deng, N., He, Z., Ye, J., Duinkharjav, B., Chakravarthula, P., Yang, X., Sun, Q.: Fov-nerf: Foveated neural radiance fields for virtual reality. IEEE Transactions on Visualization and Computer Graphics 28(11), 3854–3864 (2022)
  * [7] Fan, Z., Wang, P., Jiang, Y., Gong, X., Xu, D., Wang, Z.: Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776 (2022)
  * [8] Fathi, A., Wojna, Z., Rathod, V., Wang, P., Song, H.O., Guadarrama, S., Murphy, K.P.: Semantic instance segmentation via deep metric learning. arXiv preprint arXiv:1703.10277 (2017)
  * [9] Fu, X., Zhang, S., Chen, T., Lu, Y., Zhu, L., Zhou, X., Geiger, A., Liao, Y.: Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In: 2022 International Conference on 3D Vision (3DV). pp. 1–11. IEEE (2022)
  * [10] Kerr, J., Kim, C.M., Goldberg, K., Kanazawa, A., Tancik, M.: Lerf: Language embedded radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19729–19739 (2023)
  * [11] Kirillov, A., He, K., Girshick, R., Rother, C., Dollár, P.: Panoptic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9404–9413 (2019)
  * [12] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [13] Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems 35, 23311–23330 (2022)
  * [14] Kong, S., Fowlkes, C.C.: Recurrent pixel embedding for instance grouping. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 9018–9028 (2018)
  * [15] Kundu, A., Genova, K., Yin, X., Fathi, A., Pantofaru, C., Guibas, L.J., Tagliasacchi, A., Dellaert, F., Funkhouser, T.: Panoptic neural fields: A semantic object-aware neural scene representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12871–12881 (2022)
  * [16] Li, C., Li, S., Zhao, Y., Zhu, W., Lin, Y.: Rt-nerf: Real-time on-device neural radiance fields towards immersive ar/vr rendering. In: Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design. pp. 1–9 (2022)
  * [17] Li, Z., Li, L., Zhu, J.: Read: Large-scale neural scene rendering for autonomous driving. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1522–1529 (2023)
  * [18] Liang, S., Huang, C., Tian, Y., Kumar, A., Xu, C.: Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. Advances in Neural Information Processing Systems 36 (2024)
  * [19] Liu, Y., Hu, B., Huang, J., Tai, Y.W., Tang, C.K.: Instance neural radiance field. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 787–796 (2023)
  * [20] Mascaro, R., Teixeira, L., Chli, M.: Diffuser: Multi-view 2d-to-3d label diffusion for semantic scene segmentation. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 13589–13595. IEEE (2021)
  * [21] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4460–4470 (2019)
  * [22] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [23] Mirzaei, A., Kant, Y., Kelly, J., Gilitschenski, I.: Laterf: Label and text driven object radiance fields. In: European Conference on Computer Vision. pp. 20–36. Springer (2022)
  * [24] Novotny, D., Albanie, S., Larlus, D., Vedaldi, A.: Semi-convolutional operators for instance segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 86–102 (2018)
  * [25] Oechsle, M., Peng, S., Geiger, A.: Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5589–5599 (2021)
  * [26] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: Learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR). pp. 165–174 (2019)
  * [27] Sharma, P., Tewari, A., Du, Y., Zakharov, S., Ambrus, R.A., Gaidon, A., Freeman, W.T., Durand, F., Tenenbaum, J.B., Sitzmann, V.: Neural groundplans: Persistent neural scene representations from a single image. In: International Conference on Learning Representations (2023)
  * [28] Siddiqui, Y., Porzi, L., Bulò, S.R., Müller, N., Nießner, M., Dai, A., Kontschieder, P.: Panoptic lifting for 3d scene understanding with neural fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9043–9052 (2023)
  * [29] Sitzmann, V., Rezchikov, S., Freeman, B., Tenenbaum, J., Durand, F.: Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems (NeurIPS) 34, 19313–19325 (2021)
  * [30] Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems 32 (2019)
  * [31] Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J.J., Mur-Artal, R., Ren, C., Verma, S., et al.: The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 (2019)
  * [32] Tschernezki, V., Laina, I., Larlus, D., Vedaldi, A.: Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In: 2022 International Conference on 3D Vision (3DV). pp. 443–453. IEEE (2022)
  * [33] Tschernezki, V., Larlus, D., Vedaldi, A.: Neuraldiff: Segmenting 3d objects that move in egocentric videos. In: 2021 International Conference on 3D Vision (3DV). pp. 910–919. IEEE (2021)
  * [34] Ulku, I., Akagündüz, E.: A survey on deep learning-based architectures for semantic segmentation on 2d images. Applied Artificial Intelligence 36(1), 2032924 (2022)
  * [35] Wang, B., Chen, L., Yang, B.: Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images. arXiv preprint arXiv:2208.07227 (2022)
  * [36] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 (2021)
  * [37] Wu, Q., Liu, X., Chen, Y., Li, K., Zheng, C., Cai, J., Zheng, J.: Object-compositional neural implicit surfaces. In: European Conference on Computer Vision. pp. 197–213. Springer (2022)
  * [38] Wu, Q., Wang, K., Li, K., Zheng, J., Cai, J.: Objectsdf++: Improved object-compositional neural implicit surfaces. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21764–21774 (2023)
  * [39] Xie, C., Park, K., Martin-Brualla, R., Brown, M.: Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling. In: 2021 International Conference on 3D Vision (3DV). pp. 962–971. IEEE (2021)
  * [40] Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., De Mello, S.: Open-vocabulary panoptic segmentation with text-to-image diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2955–2966 (2023)
  * [41] Yariv, L., Gu, J., Kasten, Y., Lipman, Y.: Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems 34, 4805–4815 (2021)
  * [42] Yariv, L., Kasten, Y., Moran, D., Galun, M., Atzmon, M., Ronen, B., Lipman, Y.: Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems 33, 2492–2502 (2020)
  * [43] Zhao, X., Vemulapalli, R., Mansfield, P.A., Gong, B., Green, B., Shapira, L., Wu, Y.: Contrastive learning for label efficient semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10623–10633 (2021)
  * [44] Zhi, S., Laidlow, T., Leutenegger, S., Davison, A.J.: In-place scene labelling and understanding with implicit scene representation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15838–15847 (2021)
  * [45] Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., Misra, I.: Detecting twenty-thousand classes using image-level supervision. In: European Conference on Computer Vision. pp. 350–368. Springer (2022)

##  Appendix 0.A Appendix

In Sec. 0.A.1, we provide more implementation details. In Sec. 0.A.2, we
provide a detail proofs of our clustering loss. In Sec. 0.A.3, we provide more
visually qualitative results.

###  0.A.1 Implement Details

####  0.A.1.1 Training Parameters

As mentioned in the main paper, the attainment of
$\mathcal{L}_{\text{clustering}}$ requires a meticulous adjustment of the
weights among various components to show the model’s entropy diminution.
Furthermore, within $\mathcal{L}_{\text{fg-bg}}$, it is imperative to control
the model’s proficiency in distinguishing between foreground objects and
background stuff. To this end, we set the hyperparameters $\lambda_{1}$,
$\lambda_{2}$, $\lambda_{3}$ in clustering loss as 20, 0.5, 10000, and
$\lambda_{4}$, $\lambda_{5}$ in foreground-background loss as 0.2, 0.1 during
training.

####  0.A.1.2 Key-frame Selection

To enhance the stability of the training process, it is imperative to
encourage the model to focus more on the input images with more distinct
labels, rather than dealing with single or even no segment labels. To achieve
this, when loading the entire training segmentation dataset, we will sort the
label counts of each 2D segmentation map, _i.e_. the number of different
labels in each of the input segmentation maps. Then we choose the top 10%
input maps with the most labels as key frames and those with no labels as
empty frames. During training, when empty frames are loaded, they will be
randomly replaced with key frames.

###  0.A.2 Proofs of Clustering Loss

To further clarify how our novel-designed clustering loss works as we expect,
we provide detailed proofs to illustrate the theory behind our design.

####  0.A.2.1 Maximum Distance in a Probability Simplex

Considering $p,q\in{\mathbb{R}^{N}},~{}s.t.~{}\forall
i\in{[1,N]},~{}p_{i},q_{i}\geq{0}$, and
$\sum_{i=1}^{N}p_{i}=1,~{}\sum_{i=1}^{N}q_{i}=1$, the $\text{L}_{2}$ distance
between $p$ and $q$ is:

$\lVert p-q\rVert_{2}^{2}=(p-q)^{\top}(p-q)=\lVert p\rVert_{2}^{2}+\lVert
q\rVert_{2}^{2}-2p^{\top}{q},$ (12)

Since $\forall i\in{[1,N]},~{}p_{i},q_{i}\geq{0}$, there are:

$(\sum_{i=1}^{N}p_{i})^{2}\geq{\sum_{i=1}^{N}p_{i}^{2}}\Rightarrow\lVert
p\rVert_{2}^{2}\leq{1},~{}(\sum_{i=1}^{N}q_{i})^{2}\geq{\sum_{i=1}^{N}q_{i}^{2}}\Rightarrow\lVert
q\rVert_{2}^{2}\leq{1}$ (13)

where the equality holds only if there is one and only one $p_{i}\in{p}$ and
$q_{j}\in{q}$ is non-zero. As
$\sum_{i=1}^{N}p_{i}=1,~{}\sum_{i=1}^{N}q_{i}=1$, the non-zero $p_{i}$ and
$q_{j}$ must be 1, _i.e_. both $p$ and $q$ are in one-hot format. Furthermore,
since $\forall_{i}{p_{i}\geq{0},q_{i}\geq{0}}$, we have $p^{\top}{q}\geq{0}$,
with the equality holds only if $p$ and $q$ are orthogonal.

Therefore, $\lVert p-q\rVert_{2}^{2}\leq 2$ and the bound is reached only when
$p$ and $q$ are one-hot and orthogonal at the same time, _i.e_. they occupy
two different vertices of the simplex, which is the main objective of
$\mathcal{L}_{\text{diff}}$.

####  0.A.2.2 Minimum Variance in a Probability Simplex

In order to slow down the entropy reduction and provide more time to separate
the clusters, we designed $\mathcal{L}_{\text{reg}}$, which minimizes the
variance of softmax values in each cluster. Here we demonstrate how it serves
such an objective. Considering $p$ described same as in Sec. 0.A.2.1:

$\mu(p)=\frac{1}{N}\sum_{i=1}^{N}(p_{i})=\frac{1}{N}$ (14)
$\text{Var}(p)=\frac{1}{N}\sum_{i=1}^{N}(p_{i}-\mu(p))^{2}=\frac{1}{N}\lVert
p-\frac{1}{N}\mathbf{1}\rVert_{2}^{2}$ (15)

$\text{Var}(p)$ is minimized when $p=\frac{1}{N}\mathbf{1}$, where
$\frac{1}{N}\mathbf{1}$ is a uniform distribution of the same size as $p$ and
all elements are $\frac{1}{N}$. Therefore, $\mathcal{L}_{\text{reg}}$
encourages an increase in the entropy of the predicted result $p$.

###  0.A.3 Additional Results and Examples

####  0.A.3.1 Results of ObjectSDF++ with Machine-Generated Labels

This is an extension of Tab. 2, which reports only the quantitative results of
training ObjectSDF++ [38] using the instance labels generated with Mask2Former
[3]. Herein, we present the corresponding qualitative results in Fig. A. As we
can see, the original ObjectSDF++ cannot handle these view-inconsistent
labels, resulting in many objects not being able to be segmented, or almost
all objects being labeled the same way.

Figure A: Qualitative results of ObjectSDF++ with machine-generated instance
labels.

####  0.A.3.2 Qualitative Results without Semantic-Matching Loss

In Sec. 4, we highlighted the indispensable role of $\mathcal{L}_{\text{sem}}$
in matching the labels, without which the semantic segmentation cannot
proceed. To demonstrate this, in Fig. B we provide the results where the
$\mathcal{L}_{\text{sem}}$ is omitted. These results demonstrate that while
the model retains the capacity to differentiate between objects of various
categories, it _fails to align the predicted labels with the predefined
semantic labels accurately_.

Figure B: Qualitative results without Semantic-Matching Loss.

####  0.A.3.3 Qualitative Results without Differentiation Loss

We also design another variant (w/o the loss term $\mathcal{L}_{\text{diff}}$)
of the model, because we also claimed it indispensable in Sec. 4. Here, we
provide the corresponding results in Fig. C to prove this statement. The
qualitative results show that without $\mathcal{L}_{\text{diff}}$, the model
will no longer work and cannot segment any objects.

Figure C: Qualitative results of without $\mathcal{L}_{\text{diff}}$.

####  0.A.3.4 Comparison of Training Results at Early Stage

To further demonstrate the efficiency of our method, we compare the
segmentation results with the current state-of-the-art model, Contrastive Lift
[1] in the early stage of training. The results are shown as Fig. D. Here we
train our ClusteringSDF and Contrastive Lift for less than 10% of the complete
training time, 3 hours for Contrastive Lift and 30 minutes for our model.
Qualitative results show that our model can achieve sharp semantic/instance
segmentation results at a very early stage. In contrast, for Contrastive Lift
[1], 10% of the training has apparently not yet enabled a relatively stable
segmentation, especially for instance segmentation, which is significantly
worse than our model.

Figure D: Segmentation results at early training stage against Contrastive
Lift [1]. Columns 2-4: semantic segmentation. Columns 5-7: instance
segmentation. Please zoom in to see the details.

####  0.A.3.5 Multi-View Comparisons

In Fig. E we provide the segmentation comparisons from multiple camera views
against Contrastive Lift [1], which can demonstrate that our segmentation is
consistent across multiple views. On the other hand, it can also be observed
that the segmentation results of ClusteringSDF are closer to the actual
geometry of the objects (the areas marked by the red boxes), which is
compatible with the objective of ClusteringSDF.

Figure E: Multi-View Comparisons against Contrastive Lift [1].  Columns 2-4:
semantic segmentation. Columns 5-7: instance segmentation. Please zoom in to
see the details.