# Neural Mesh Fusion:
Unsupervised 3D Planar Surface Understanding

###### Abstract

This paper presents Neural Mesh Fusion (NMF), an efficient approach for joint
optimization of polygon mesh from multi-view image observations and
unsupervised 3D planar-surface parsing of the scene. In contrast to implicit
neural representations, NMF directly learns to deform surface triangle mesh
and generate an embedding for unsupervised 3D planar segmentation through
gradient-based optimization directly on the surface mesh. The conducted
experiments show that NMF obtains competitive results compared to state-of-
the-art multi-view planar reconstruction, while not requiring any ground-truth
3D or planar supervision. Moreover, NMF is significantly more computationally
efficient compared to implicit neural rendering-based scene reconstruction
approaches.

Index Terms—  Multi-view Planar Reconstruction, Neural Radiance Fields,
Contrastive Learning, Triangle Mesh

##  1 Introduction

Parsing 3D planar surfaces in indoor scenes from a posed monocular video
enables various applications such as augmented and virtual reality, robot
navigation, and 3D interior modeling. Since most surfaces in man-made
environments are locally planar, approximating scene geometry with a set of
planar primitives provides a compact and efficient representation for
interacting with the physical space.

Recent deep learning-based solutions formulate 3D planar surface understanding
as a supervised learning problem, and require 2D plane annotations [1, 2, 3,
4, 5] or 3D annotations [6]. Utilizing large-scale plane annotations
simplifies the learning problem. However, high-quality plane annotations are
costly to obtain and such trained models do not perform well when test data is
very different from training data. As such, it is important to develop an
unsupervised planar understanding method, which can be optimized on various
unseen scenes with different imaging sensors.

Recent advances in differentiable rendering have made it possible to
reconstruct 3D geometry by using only posed multi-view images, without
requiring 3D ground-truth geometry. While Neural Radiance Fields (NeRF) and
subsequent works [7, 8, 9] achieves impressive novel view synthesis quality,
it is non-trivial to parse the planar surfaces which are represented
implicitly [10]. For instance, volume-based methods [11, 12, 13] require
complicated steps including ray marching and density field prediction for
implicit surface modeling, Marching Cubes [14] to extract the surfaces, and
Sequential RANSAC [15] to detect the planes, which are computationally
expensive and require careful tuning of multiple hyper-parameters (e.g., in
RANSAC).

Fig. 1: Overview of our proposed Neural Mesh Fusion pipeline. The predicted
pseudo-depth and normal maps, obtained by the pre-trained networks, are used
for 2D plane segmentation. Sampling pixels in planar regions results in a
triangular mesh fragment that is transferred into 3D space. The collection of
posed mesh fragments are fused together, guided by the explicit neural
rendering process.

In contrast, explicit neural representation provides several advantages for 3D
scene geometry reconstruction, which allows one to directly optimize the
surface mesh and impose regularization (e.g., planar constraints) on the
learned geometry. Only a few recent works have adopted explicit mesh
representation for object- or scene-level reconstruction from multi-view
images. For instance, [16, 17, 18, 19] employ a 3D grid representation.
However, they are primarily developed for object reconstruction and novel view
synthesis, and are challenging to scale up for large scenes due to memory and
computation costs of a dense 3D grid.

Compared to previous methods that reconstruct 3D by deforming volumetric
meshes, our proposed NMF learns the scene geometry by combining of local
surface meshes via neural rendering. More specifically, NMF initializes
multiple mesh fragments from keyframes in video and approximates 3D vertex
positions with estimated depths. Through neural rendering, NMF optimizes the
3D positions of all vertices and uses an MLP network to predict consistent
radiance and plane-instance fields across observed images. The learned plane-
instance field acts as a discriminative latent embedding for triangle
surfaces. Unsupervised clustering of this feature space then identifies
distinct planar surfaces.

Our main contributions are summarized as follows:

  * •

We introduce Neural Mesh Fusion (NMF), a novel framework for unsupervised 3D
planar surface understanding. NMF is among the first mesh reconstruction
methods to employ explicit neural rendering with triangular surface meshes.

  * •

NMF achieves 3D plane instance segmentation by combining contrastive learning
and neural rendering. As such, only posed RGB images are needed, eliminating
the need for 3D or planar ground truth.

  * •

NMF shows superior 3D planar reconstruction performance, as compared to models
trained with 3D ground truth on ScanNet [20] data. It seamlessly handles new
test scenes, while pre-trained models suffer significant performance
degradation due to domain disparities.

##  2 Related work

Single-view planar reconstruction: Several works have looked into planar
reconstruction from a single RGB image via deep learning. Some works [21, 1]
propose CNN-based models, trained to predict both segmentation and 3D plane
parameters. These methods require a prescribed maximum number of planes in an
image, which limits model applicability. Some other methods [2, 3, 22] can
handle any number of planes. Recently, PlaneTR [4] leverages transformers to
consider context information and geometric cues in a sequence-to-sequence way.

Multi-view planar reconstruction: Multi-view reconstruction utilizes multiple
images, which contain richer geometric information. Several works share a
common two-stage approach: local plane detection and plane parameter
estimation [23, 24]. More recently, PlanarRecon [6] proposes to detect planes
from video fragments and combine them to create a comprehensive global planar
reconstruction, which is supervised by 3D ground-truth planes in training. In
contrast, this paper presents a multi-view 3D planar surface reconstruction
method without requiring 2D or 3D plane annotations.

##  3 Neural Mesh Fusion

NMF learns to blend 3D planar mesh fragments extracted from keyframes via
efficient neural rendering and simultaneously predicts 3D plane instances in
an unsupervised way (see Fig. 1). In Section 3.1, we first describe how NMF
constructs 3D planar mesh fragments from posed 2D images. Section 3.2 explains
an efficient explicit rendering method to fuse the local mesh fragments by
optimizing their vertex positions. Section 3.3 presents the unsupervised 3D
plane instance segmentation. Fig. 1 shows the overall NMF pipeline.

###  3.1 Constructing Initial 3D Scene Surface Mesh

Keyframe selection. In this step, we select a minimal set of images to
construct the initial 3D mesh while covering the entire scene. Given the
camera poses and the intrinsic parameters, we first compute the pairwise 3D
Intersection-over-Union (IoU) of all camera frustums. Since the actual depth
map of each frame is unknown, the near and far planes of the frustums are
considered fixed for all the frames (e.g., between 1 and 3 meters from the
camera). The obtained IoU matrix is used as a similarity matrix across camera
views, which indicates the amount of 3D view overlap between every two frames
in the video. By applying spectral clustering, which decomposes the similarity
matrix into K (e.g. equal to 20) clusters, we obtain groups of images which
have high similarity in terms of their 3D field of view (FoV). In this way,
the images in each group provide diverse views of objects in their 3D view
frustums (see Fig. 2). At the beginning of training, one image per cluster is
selected randomly and added to the stack of keyframes. Later in the rendering
process, other images may be added to the stack of keyframes, depending on the
detected void regions in the rendered image; more details in Section 3.2. The
keyframes are used to construct the initial local 3D mesh fragments.

(a) (b)

Fig. 2: Camera views clustering and keyframes selection; (a) different
clusters are shown with different colors, (b) examples of some images belong
to three different clusters. Each cluster includes diverse 3D views of the
objects in their FoV.

2D planar segmentation. Given a keyframe $I_{j}$, we first segment it into
planar regions. Since ground-truth planes are not available, we leverage
predicted monocular depth and normal maps, which can be obtained via off-the-
shelf predictors. Specifically, a pretrained Omnidata model [25] is used to
predict the pseudo-depth map ($\overline{d}$) and normal map
($\overline{\mathbf{n}}$). Note that the predicted pseudo-depth has unknown
scale and offset w.r.t. the real depth values. Using the pinhole camera model
and camera intrinsic parameters, e.g., focal lengths $(f_{x},f_{y})$ and the
principal point $(u_{0},v_{0})$, a pixel in the 2D image
$\mathbf{p}=(u,v)^{\top}$ can be mapped to the corresponding 3D point
$\mathbf{P}=(X,Y,Z)^{\top}$ using the pseudo-depth value. Assuming every pixel
$\mathbf{p}$ belongs to one planar surface in the scene, the point
$\mathbf{p}$ and the plane satisfy the point-normal equation:
$\overline{\mathbf{n}}\cdot\mathbf{P}+d_{p}=0$, where
$\overline{\mathbf{n}}=(n_{1},n_{2},n_{3})^{\top}$ is the predicted normal
vector. So, the _planar distance_ (PD) of every pixel is given as follows:

$d_{p}=\overline{d}_{(u,v)}\cdot\bigl{(}\frac{n_{1}}{f_{x}}(u_{0}-u)+\frac{n_{2}}{f_{y}}(v_{0}-v)-n_{3}\bigr{)}.$
(1)

The $\overline{\mathbf{n}}$ and $d_{p}$ are view dependent. Using the camera
pose, they can be transferred into the 3D world coordinate system. For each 2D
pixel in an image, we use its corresponding normal vector, PD value, and the
2D positional encoding to form a feature vector. These vectors are then
clustered using mean-shift clustering [3], which generates pixel-wise 2D plane
segmentation (see Fig. 3-middle).

Fig. 3: Mesh initialization; (left) input image; (middle) sampled pixels and
triangular mesh of 2D planar segments; (right) initial mesh fragment shown
from a different camera view.

Constructing triangular mesh fragment. To construct a mesh fragment from a
keyframe, a set of sparse pixels are sampled from each segmented region of the
image, using _farthest first traversal_ algorithm. The number of sampled
points from each segment is proportional to the segment area. A maximum of
total $N$ (e.g. equal to 500) points are sampled as the vertices of a 2D
triangular mesh that is obtained by performing the Delaunay triangulation. The
edges (i.e., faces $\mathcal{F}$) of the mesh which cross two distinct
segments are suppressed to ensure that the obtained mesh does not include
edges across two planes with probably different planar depths. Given the
predicted depth map $\overline{d}$ of the image, the 2D vertices are lifted to
3D, creating a 3D mesh fragment
$\mathcal{M}^{j}=\\{(\mathcal{V}_{ij},\mathcal{F}_{ij})\\}$, where
$\mathcal{V}_{i}j$ and $\mathcal{F}_{ij}$ represent the vertices and faces.
Additionally, each 3D vertex $\mathcal{V}_{ij}$ inherits a ray direction
$\mathbf{v}_{ij}$ (passes through the center of keyframe camera) and a cluster
index of its corresponding plane segment, which are later utilized for
rendering. Fig. 3 shows an example of the constructed mesh fragment from an
image.

Since each fragment $\mathcal{M}^{j}$ is constructed based on the predicted
monocular depth map, the positions of its vertices have an unknown scale and
shift w.r.t. their actual positions in 3D. As such, two learnable parameters,
$\pi_{j}$ and $\beta_{j}$, are introduced to correct the scale and shift,
respectively. They are optimized to adjust the position of the $i$th vertex of
the $j$th mesh fragment as follows:

$\mathcal{V}_{ij}^{\tiny{\,\text{updated}}}=\bigl{(}(\pi_{j}-1)\cdot
t_{ij}^{\,\text{init}}+\beta_{j}\bigr{)}\cdot\mathbf{v}_{ij}^{\,\text{init}}+\mathcal{V}_{ij}^{\tiny{\,\text{init}}},\vspace{-5pt}$
(2)

where $\mathbf{v}_{ij}\in\mathbb{R}^{3}$ denotes the ray direction and
$t_{ij}$ denotes the distance along the ray from the camera origin
($\mathbf{o}_{j}\in\mathbb{R}^{3}$) of the $j$th keyframe. So,
$\mathcal{V}_{ij}^{\tiny\text{ init}}=\mathbf{o}_{j}+t_{ij}^{\text{
init}}\cdot\mathbf{v}_{ij}^{\text{init}}$ is the initial position of $i$th
vertex in $j$th fragment. Eq. 2 gives the updated position of each vertex
which has been constrained by a 3D displacement along the ray direction.

###  3.2 Triangle Mesh Rendering

We optimize the position of vertices and their learnable descriptors in the
set of local meshes $\mathcal{M}$ by performing differentiable rendering in a
discrete field. Suppose that a camera ray $\mathbf{r}$ emanates from the
camera origin, passes through a pixel, and hits a set of triangular faces in
$\mathcal{M}$. Suppose that multiple faces
${\mathcal{F}_{1},...,\mathcal{F}_{L}}$ are hit by the ray, where $L$ is the
total number of intersection points; in practice, the value of $L$ is small
and likely equals to one. The intersection points
$\\{\mathbf{P}^{h}\\}\in\mathbb{R}^{3}$ of ray-triangular faces (and
consequently their depths) are computed in closed-from using the fast Möller-
Trumbore algorithm [26]. Given the intersected points, a modular MLP with a
composition of functions $g\circ f(x)$ is used to estimate the transparency
$\alpha$ and radiance $\mathbf{c}$ along the ray at the point
$\mathbf{P}^{h}$. The radiance field can be estimated by the prediction head
$g$ as follows:

$\bigl{(}\mathbf{c},\alpha\bigl{)}=g\circ
f\bigl{(}\gamma(\mathbf{P}^{h})\bigr{)},\vspace{-5pt}$ (3)

where $\gamma$ denotes a fixed positional encoding using trigonometric
functions [7]. Since the network evaluates only a few intersection points per
ray, our method is highly efficient in scene-level geometry learning as
compared to ray-marching in implicit neural rendering methods (e.g., [7, 13,
11]) or evaluating a dense tetrahedral volumetric mesh in existing explicit
neural rendering methods (e.g., [16, 19]).

Following common volume rendering procedure, the intersected points
$\\{\mathbf{P}^{h}\\}$ of each ray are sorted based on their depths from the
camera origin in an increasing order. We use standard alpha-composition to
blend the point-set colors:

$\hat{\mathbf{c}}(\mathbf{r})=\sum_{i=1}^{L}\alpha_{i}\cdot\mathbf{c}_{i}^{h}\cdot\prod\limits_{j=1}^{i-1}(1-\alpha_{j}),\vspace{-5pt}$
(4)

where $L=|\\{\mathbf{P}^{h}\\}|$ is the number of hit points along the ray.

Since $\mathcal{M}$ gives an explicit representation of scene surfaces,
rendering depth $\hat{d}(\mathbf{r})$ and normal
$\hat{\mathbf{n}}(\mathbf{r})$ for each ray $\mathbf{r}$ are performed by
alpha-composition of depths and normals of intersected triangles, similar to
the case of color in Eq. (4).

Fig. 4: Two examples of contrastive sampling. (left) two training images with
some markers, which indicate a sampled pixel (green stripe triangle) and its
positive pair (green stripe circle) and some negative pairs (red stripe
circle). (middle) the computed planar distances (PD map), (right) normal maps
in 3D world coordinates.

(a) (b) (c) (d) (e)

Fig. 5: Visualization: (a) an input camera image, and its corresponding (b)
rendered plane instance embedding, (c) radiance, and (d-e) depth and normal
geometrical fields. As shown in (b), the spherical embedding vectors
distinguish plane instances in the scene.

The set $\mathcal{M}$ consists of several overlapping fragments. Therefore,
some parts of surfaces in the scene may be duplicated and represented by more
than one surface mesh. To stitch and deform the neighboring fragments
coherently, a _divergence_ loss is introduced to pull the overlapped surfaces
of mesh fragments toward each other as follows:

$\displaystyle\mathcal{L}_{\text{div}}=$
$\displaystyle\sum_{\mathbf{r}}\sum_{i=1}^{L}\left\|\mathbf{P}_{i}^{h}-\mathbf{P}_{M}^{r}\right\|_{2}^{2},$
(5)
$\displaystyle\text{where}\>\mathbf{P}_{M}^{r}=\sum_{i=1}^{L}\alpha_{i}\cdot\mathbf{P}_{i}^{h}\cdot\prod\limits_{j=1}^{i-1}(1-\alpha_{j}),\vspace{-10pt}$
(6)

where $\mathbf{P}_{M}^{r}$ is a _soft_ mean of intersected points along the
ray.

It’s important to note that if a ray fails to intersect any triangle in the
scene (i.e., L = 0), it implies that the set $\mathcal{M}$ does not adequately
represent certain portions of the scene that are visible in the current frame.
If the proportion of undetected rays exceeds a predetermined threshold (e.g.,
5%), the current frame is appended to the stack of key frames, and its
corresponding mesh is incorporated into the set $\mathcal{M}$.

###  3.3 3D Plane Instance Segmentation

To segment the mesh vertices into individual planes, a prediction head $h$ in
the modular MLP with a composition of functions $h\circ f(x)$ maps the 3D
intersection points associated with ray $\mathbf{r}$ into a latent vector
$\mathbf{s}\in\mathbb{R}^{3}$ as follows:

$\mathbf{s}(\mathbf{r})\\!=\\!\sum_{i=1}^{L}\alpha_{i}\cdot\mathbf{s}_{i}^{h}\cdot\prod\limits_{j=1}^{i-1}(1-\alpha_{j}),\,\text{where
}\mathbf{s}^{h}_{i}\\!=\\!h\circ
f\bigl{(}\gamma(\mathbf{P}^{h}_{i})\bigr{)}\vspace{-5pt}$ (7)

which forms a spherical embedding after normalization
$\left\|\mathbf{s}\right\|_{2}=1$.

|
---|---
Fig. 6: Spherical embedding representing 3D plane instances is generated by
minimizing the contrastive loss over positive and negative camera rays in the
scene.

Contrastive learning. To form a discriminative spherical embedding, a
contrastive learning loss is introduced into the overall optimization, which
is computed over pairs of latent vectors $\mathbf{s}(\mathbf{r})$. This
requires sampling positive ($y_{ij}=1$) and negative ($y_{ij}=0$) pairs of
rays within an image and across images (see Fig. 4). For a pair of rays,
sampled within an image, their corresponding PDs (computed by Eq. (1)) and
normal vectors ($\overline{\mathbf{n}}$) at the location where the rays
intersect with the 2D image plane are used. If those values differ by less
than a threshold (e.g. 30 cm for PD and 30 degrees of normal vector), the two
rays are identified as positive pairs, otherwise are negative. Since planar
distances are derived from the scale- and offset-ambiguous pseudo depths, they
are not comparable across images. Hence, we only sample negative pairs across
images by comparing their normal vectors in the world coordinates. Minimizing
the contrastive loss in Eq. (8) obtains a learned spherical embedding that can
distinguish 3D plane instances.

$\begin{split}&\mathcal{L}_{cont}(\mathbf{s}(\mathbf{r}_{i}),\mathbf{s}(\mathbf{r}_{j}))\\!=\\!\\\
&\begin{cases}\vspace{2pt}y_{ij}\cdot(1\\!-\\!\omega)\\!+\\!(1\\!-\\!y_{ij})\cdot\text{max}(\epsilon,1\\!-\\!\omega),\>\text{if
}(\mathbf{r}_{i},\mathbf{r}_{j})\\!\in\\!\mathcal{R},\\\
(1\\!-\\!y_{ij})\cdot\text{max}(\epsilon,1\\!-\\!\omega),\>\text{otherwise,}\end{cases}\end{split}\vspace{-5pt}$
(8)

where
$\omega=\langle\mathbf{s}(\mathbf{r}_{i}),\mathbf{s}(\mathbf{r}_{j})\rangle$
is the inner product of two latent vectors and $\epsilon$ is a hyperparameter
defining a lower-bound distance between samples of different classes. The
condition $(\mathbf{r}_{i},\mathbf{r}_{j})\in\mathcal{R}$ indicates that the
pairs $\mathbf{r}_{i}$ and $\mathbf{r}_{j}$ are sampled from the same ray
vector field $\mathcal{R}$ of the same image. An example of learned embedding
is shown in Fig. 5.

Finally, after the optimization procedure and forming a discriminative
embedding, plane instances are identified by applying mean-shift clustering on
the spherical embedding of the mesh faces; see an example in Fig. 6.

Overall optimization. The loss for learning scene geometry includes multi-view
color, depth, and normal consistencies:

$\displaystyle\mathcal{L}_{\text{rgb}}=$
$\displaystyle\sum_{\\{\mathbf{r}\\}}\left\|\hat{\mathbf{c}}(\mathbf{r})-\mathbf{c}(\mathbf{r})\right\|_{1},$
(9) $\displaystyle\mathcal{L}_{\text{depth}}=$
$\displaystyle\sum_{\\{\mathbf{r}\\}}\left\|\hat{d}(\mathbf{r})-a\cdot\overline{d}(\mathbf{r})+b\right\|_{2}^{2},$
(10) $\displaystyle\mathcal{L}_{\text{normal}}=$
$\displaystyle\sum_{\\{\mathbf{r}\\}}\Bigl{(}1-\bigl{\langle}\hat{\mathbf{n}}(\mathbf{r})\cdot\overline{\mathbf{n}}(\mathbf{r})\bigr{\rangle}\Bigr{)},\vspace{-5pt}$
(11)

and the divergence loss in Eq.( 5). Similar to [13], for depth consistency
loss, two unknown parameters $(a,b)$ are estimated using least squares for
each image. For parsing the mesh into plane instances, the contrastive loss in
Eq. (8) is added to the optimization. So, the overall loss has the following
form:

$\mathcal{L}_{\text{total}}=w_{1}\cdot\mathcal{L}_{\text{rgb}}+w_{2}\cdot\mathcal{L}_{\text{depth}}+w_{3}\cdot\mathcal{L}_{\text{normal}}+w_{4}\cdot\mathcal{L}_{\text{div}}+w_{5}\cdot\mathcal{L}_{\text{cont}},\vspace{-3pt}$
(12)

where scalars $w_{i}$ are weights of the loss terms, which are adjusted
empirically and are fixed across all experiments.

##  4 Experiments

We conduct experiments to evaluate the 3D planar surface instance segmentation
of NMF, as well as compare with existing state-of-the-art methods. We further
perform ablation study to analyze various design choices in the proposed
pipeline.

Method | VOI $\downarrow$ | RI $\uparrow$ | SC$\uparrow$ | Supervision | Inference
---|---|---|---|---|---
RGB | 3D | 3D labels | time
NeuralRecon [27]\+ RANSAC | 5.540 | 0.696 | 0.139 | ✓ | ✓ | - | 2 minutes
PlanarRecon[6] | 3.458 | 0.861 | 0.359 | ✓ | ✓ | ✓ | real-time
MonoSDF [13]\+ RANSAC | 3.641 | 0.861 | 0.399 | ✓ | - | - | 15 hours
NMF (ours) | 3.253 | 0.880 | 0.381 | ✓ | - | - | 40 minutes
Table 1: 3D plane instance segmentation results on ScanNet.

Experiment F-score $\uparrow$ VOI$\downarrow$ RI $\uparrow$ SC$\uparrow$ NMF
(proposed) 0.441 3.253 0.880 0.381 Num. vertices 80k 0.443 3.208 0.881 0.383
40k 0.431 3.298 0.878 0.379 20k 0.395 3.514 0.878 0.366 MLP 8-layer 0.391
3.399 0.870 0.379 2-layer 0.425 3.464 0.883 0.365

Table 2: Ablation study on ScanNet.

Data: We evaluate our method on ScanNetv2 [20] dataset. ScanNetv2 contains RGB
videos taken by a mobile device from indoor scenes with the camera pose
information associated with each frame. We run our experiments on 8 scenes. 4
of them are the same as in [28].

Baseline: Since there are only a few prior works that focus on learning-based
multi-view 3D planar reconstruction, we compare our method to two types of
approaches: (1) PlanarRecon [6] trained with 3D geometry and 3D plane
supervisions, (2) dense 3D reconstruction methods like NeuralRecon[27] with 3D
geometry supervision and MonoSDF [13] with 2D image supervision, followed by
the Sequential RANSAC to extract planes [15].

Metrics: Similar to prior works [6, 2, 4], we evaluate the performance of
plane instance segmentation by measuring the rand index (RI), variation of
information (VOI), and segmentation covering (SC).

(a) (b) (c)

Fig. 7: Examples of 3D planar reconstruction on ScanNet data. (a) Ground-truth
textured meshes (with holes on floor which are due to unseen regions in
video), (b) our proposed NMF, (c) PlanarRecon [6] supervised by 3D GT planes.
PlanarRecon inaccurately places some surfaces and/or incorrectly split one
surface into multiple planes. It also ignores smaller objects. In contrast,
NMF provides more accurate, comprehensive, and detailed 3D planar
reconstruction.

###  4.1 Results

Evaluation results of 3D plane instance segmentation on ScanNet data are shown
in Table 1. The proposed NMF (using a maximum of 60k vertices and a 4-layer
MLP network) is compared with other methods. In terms of VOI and RI metrics,
NMF outperforms other approaches including PlanarRecon which is trained with
3D plane labels. While MonoSDF+RANSAC shows slightly higher SC score than our
proposed NMF, it requires about 15 hours to run MonoSDF, while NMF only needs
40 minutes. Fig. 7 shows qualitative results on ScanNet data.

The proposed NMF method, requiring neither 3D plane annotations nor 3D
geometry meshes, surpasses the performance of existing supervised learning
methods. To assess the generalizability of our method compared to the state-
of-the-art supervised approach, planarRecon, we conducted further qualitative
tests on the Replica dataset. As illustrated in Fig. 8, planarRecon struggles
to effectively segment the scene into a set of valid planar surfaces, even
after we adjusted image color statistics to match those of the ScanNet
dataset. In contrast, our method, when applied with the same NMF
configurations used on ScanNet data, demonstrates good performance.

Ablation study. We perform ablation study on design choices of the mesh
granularity and complexity of the MLP network. The conducted experiments on
three different numbers of mesh vertices are reported in Table 2. The results
show a subtle reduction in the plane instance segmentation performance when a
smaller number of vertices is used. However, reducing the number of vertices
from 40k to 20k noticeably affects the reconstruction quality in term of
F-score. Additionally, using a lighter MLP with two hidden layers or a deeper
one with eight hidden layers does not significantly affect the plane instance
segmentation performance.

(a) (b) (c) (d)

Fig. 8: Qualitative results on a Replica scene; (a) Ground truth textured
mesh, (b) ground truth plane instances, (c) overlaid PlanarRecon prediction,
(d) proposed NMF.

##  5 Discussions and conclusion

In this paper, we presented Neural Mesh Fusion (NMF), a novel unsupervised
method for jointly 3D planar surface reconstruction and parsing using
monocular posed images. The introduced scene-level surface reconstruction was
performed by optimizing the position of several mesh fragments, each estimated
from partially observed 3D scene via camera images. NMF jointly estimates the
radiance and plane instance embedding for the discrete field of intersections
between camera ray and the triangular meshes. For segmenting the plane
instances without accessing the ground truth annotation, we integrated a
contrastive learning in the process of image rendering to obtain an embedding
which represents the distinct plane instances. The conducted experiments have
showed that the proposed NMF outperforms state-of-the-art supervised learning
methods in 3D plane instance segmentation.

Limitations. NMF is a per-scene optimization method. While it avoids the
generalization issues that typically plague supervised training methods, its
inference time can reach several minutes. Since inference time scales with the
number of vertices, reconstructing the scene with fewer vertices would speed
up the optimization process. Although the introduced progressive mesh
construction attempts to mitigate this issue, developing an adaptive
variational sampling technique based on the visual content of images might be
a more promising direction for further reducing inference time.

## References

  * [1] Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, and Yasutaka Furukawa,  “PlaneNet: Piece-wise planar reconstruction from a single RGB image,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [2] Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz,  “PlaneRCNN: 3D plane detection and reconstruction from a single image,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [3] Zehao Yu, Jia Zheng, Dongze Lian, Zihan Zhou, and Shenghua Gao,  “Single-image piece-wise planar 3d reconstruction via associative embedding,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [4] Bin Tan, Nan Xue, Song Bai, Tianfu Wu, and Gui-Song Xia,  “PlaneTR: Structure-guided transformers for 3d plane recovery,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [5] Samir Agarwala, Linyi Jin, Chris Rockwell, and David F Fouhey,  “PlaneFormers: From sparse view planes to 3D reconstruction,”  in Proceedings of the European Conference on Computer Vision, 2022\.
  * [6] Yiming Xie, Matheus Gadelha, Fengting Yang, Xiaowei Zhou, and Huaizu Jiang,  “PlanarRecon: Real-time 3D plane detection and reconstruction from posed monocular videos,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [7] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng,  “NeRF: Representing scenes as neural radiance fields for view synthesis,”  Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  * [8] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan,  “Ref-NeRF: Structured view-dependent appearance for neural radiance fields,”  Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022.
  * [9] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan,  “Depth-supervised NeRF: Fewer views and faster training for free,”  in Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2022.
  * [10] Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman,  “Controlling neural level sets,”  Advances in Neural Information Processing Systems, 2019.
  * [11] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang,  “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,”  arXiv preprint arXiv:2106.10689, 2021.
  * [12] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman,  “Volume rendering of neural implicit surfaces,”  Advances in Neural Information Processing Systems, vol. 34, pp. 4805–4815, 2021.
  * [13] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger,  “MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction,”  Advances in neural information processing systems, 2022.
  * [14] William E Lorensen and Harvey E Cline,  “Marching cubes: A high resolution 3D surface construction algorithm,”  ACM SIGGRAPH Computer Graphics, vol. 21, no. 4, pp. 163–169, 1987\.
  * [15] Martin A Fischler and Robert C Bolles,  “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,”  Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
  * [16] Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler,  “Learning deformable tetrahedral meshes for 3D reconstruction,”  Advances In Neural Information Processing Systems, vol. 33, pp. 9936–9947, 2020.
  * [17] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler,  “Deep marching tetrahedra: a hybrid representation for high-resolution 3D shape synthesis,”  Advances in Neural Information Processing Systems, 2021.
  * [18] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler,  “Extracting triangular 3D models, materials, and lighting from images,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [19] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi,  “MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [20] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner,  “Scannet: Richly-annotated 3D reconstructions of indoor scenes,”  in Proc. of the IEEE conf. on computer vision and pattern recognition, 2017.
  * [21] Fengting Yang and Zihan Zhou,  “Recovering 3D planes from a single image via convolutional neural networks,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [22] Yiming Qian and Yasutaka Furukawa,  “Learning pairwise inter-plane relations for piecewise planar reconstruction,”  in Proceedings of the European Conference on Computer Vision, 2020\.
  * [23] Linyi Jin, Shengyi Qian, Andrew Owens, and David F Fouhey,  “Planar surface reconstruction from sparse views,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [24] Jiachen Liu, Pan Ji, Nitin Bansal, Changjiang Cai, Qingan Yan, Xiaolei Huang, and Yi Xu,  “PlaneMVS: 3D plane reconstruction from multi-view stereo,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [25] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir,  “Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3D scans,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [26] Tomas Möller and Ben Trumbore,  “Fast, minimum storage ray/triangle intersection,”  in ACM SIGGRAPH 2005 Courses. 2005.
  * [27] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao,  “NeuralRecon: Real-time coherent 3D reconstruction from monocular video,”  in Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2021, pp. 15598–15607.
  * [28] Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou,  “Neural 3D scene reconstruction with the manhattan-world assumption,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.