(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Technical University of Munich, Germany22institutetext:
Tianjin University, China 33institutetext: National University of Singapore,
Singapore 44institutetext: Google, Zurich, Switzerland

# GeoGaussian: Geometry-aware Gaussian Splatting for Scene Rendering

Yanyan Li 11 Chenyu Lyu 22 Yan Di 11 Guangyao Zhai 11 Gim Hee Lee 33 Federico
Tombari 1144

###### Abstract

During the Gaussian Splatting optimization process, the scene’s geometry can
gradually deteriorate if its structure is not deliberately preserved,
especially in non-textured regions such as walls, ceilings, and furniture
surfaces. This degradation significantly affects the rendering quality of
novel views that deviate significantly from the viewpoints in the training
data. To mitigate this issue, we propose a novel approach called GeoGaussian.
Based on the smoothly connected areas observed from point clouds, this method
introduces a novel pipeline to initialize thin Gaussians aligned with the
surfaces, where the characteristic can be transferred to new generations
through a carefully designed densification strategy. Finally, the pipeline
ensures that the scene’s geometry and texture are maintained through
constrained optimization processes with explicit geometry constraints.
Benefiting from the proposed architecture, the generative ability of 3D
Gaussians is enhanced, especially in structured regions. Our proposed pipeline
achieves state-of-the-art performance in novel view synthesis and geometric
reconstruction, as evaluated qualitatively and quantitatively on public
datasets.
Keywords: Gaussian Splatting $\cdot$ Geometry-aware Densification $\cdot$
Geometric Consistency
Website: https://yanyan-li.github.io/project/gs/geogaussian

GeoGaussian (ours)Reference3DGS [18]GeoGaussian (ours)Reference3DGS
[18]GeoGaussian (ours)Reference3DGS [18]GeoGaussian (ours)Reference3DGS
[18]GeoGaussian (ours)Reference3DGS [18]GeoGaussian (ours)Reference3DGS
[18]GeoGaussian (ours)Reference3DGS [18] Figure 1:  Comparisons of novel view
rendering and 3D Gaussian model on the Replica Datasets. As highlighted in the
second row, the proposed method shows a very clear boundary between two low-
textured walls, but 3DGS has blurring issues since the geometry of its 3D
Gaussian model is not accurate in this area.

##  1 Introduction

Thanks to the impressive rendering quality of Neural Radiance Fields (NeRF)
[25], the area of photo-realistic novel view synthesis (NVS) has become a
popular research topic in the communities of computer vision, graphics, and
robotics. While NeRFs offer high-quality rendering, 3D Gaussian Splatting
([18, 8]) shows promising performance in terms of training speed and rendering
quality. Using Gaussian parametrization, each Gaussian point contains
position, orientation, and spherical harmonics parameters. An additional
rasterization step re-projects these 3D Gaussians back to training images to
capture scene geometry and appearance by using alpha-blending. However, in the
Gaussian Splatting optimization process, the geometry of 3D Gaussian models
lacks sufficient constraints especially in low-textured regions, leading to
significant degradation in rendering performance for novel views that deviate
substantially from the training data.

NVS methods typically represent 3D scenes implicitly [28, 42] or explicitly
[4, 16] based on multiple 2D views and corresponding camera poses. Previous
solutions for NVS tasks have primarily relied on 3D surface reconstruction
using technologies such as Structure-from-Motion [11], multi-view stereo [34],
TSDF [14], Marching Cubes [22], Poisson [16], and texturing [38]. While these
methods enable the rendering of new views in texture and depth based on 3D
surface models, achieving photo-realistic rendering quality remains
challenging. Since the impressive achievements of convolutional neural
networks are made on many tasks, such as point detection [6] and scene
completion [40], deep learning-based mesh representations [9] have been
proposed to enhance view synthesis, while the challenges for these mesh-based
methods are to capture accurate geometry and appearance in complex scenarios.
Compared to learning-based mesh methods, NeRF [25] proposes a continuous
volumetric function representation using a multi-layer perceptron (MLP), which
produces high-quality renderings with impressive details. To address the heavy
computational burden and intensive memory consumption, recent improvements
have been made using sparse volumes [21], hash tables [27], and hierarchical
sampling methods [2, 39]. Despite these improvements significantly reducing
training time, improving rendering efficiency is still a pressing requirement
for applications such as SLAM. Recently, 3D Gaussian Splatting (3DGS) [18] has
gained significant attention in the community, which shows that the rendering
speed in high-quality NVS tasks can be made up to 150 FPS [8]. In the Gaussian
Splatting optimization process, approaches often prioritize image clarity over
geometric fidelity. While some viewpoints may render well, issues arising from
confusing geometry can impact rendering quality at certain viewpoints. To
extract mesh surface from 3D Gaussians, SuGaR [12] serves as a refinement
module to jointly optimize mesh and Gaussians. However, the method requires
more GPU resources for training compared with 3DGS.

In this paper, we propose a geometry-aware Gaussian Splatting method
emphasizing rendering fidelity and geometry structure simultaneously.
Initially, normal vectors are extracted from input point clouds, and then
smoothly connected areas are detected based on normals. For general regions,
we follow the traditional initialization process [18] that represents every
point as a sphere. But for smoothly connected areas, each point is
parameterized as a thin ellipsoid with explicit geometric information. To be
specific, the third value of the scale vector $\mathbf{S}$ is fixed to make
the ellipsoid thin. Additionally, the third column of rotation matrix
$\mathbf{R}$ decoupled from the covariance matrix $\mathbf{\Sigma}$ is
initialized by the normal vector. Based on the design, we encourage these thin
ellipsoids to lie on the surface of smooth regions. Leveraging the thin
ellipsoid representation, our densification strategy containing split and
clone steps operates within a well-constrained process. On one hand, a large
Gaussian point can be split into two smaller co-planar Gaussians lying on the
plane established by the position and normal vectors of the large Gaussian,
whereas traditional approaches [8, 18, 23] randomly position new Gaussians
within the ellipsoid. On the other hand, our Gaussian clone approach ensures
that new generations have to lie on the same plane with the original since the
new Gaussian is also required to align with the surface of the smooth region.
The step is supported by accumulating the gradient’s descent direction of the
origin’s position $\bm{\mu}$, and then the component of the direction that is
perpendicular to the normal vector of origin is decoupled to guide the
position for the cloned Gaussian. Through the learning rate and direction, the
Gaussian map undergoes densification for continuous training. In the
optimization module, we propose a new geometrically consistent constraint for
thin ellipsoids lying on the smooth areas by encouraging the nearest neighbors
to be co-planar, which are jointly optimized with the widely used photometric
residuals [18] in an iterative process. The contributions of the paper are
summarized as:

  * •

A parameterization with explicit geometry meaning for thin 3D Gaussians is
employed in our carefully designed initialization and densification strategies
to establish reasonable 3D Gaussian models.

  * •

A geometrically consistent constraint is proposed to encourage thin Gaussians
to align with the smooth surfaces.

  * •

Evaluations on the public datasets demonstrate that the proposed method
improves rendering quality compared to state-of-the-art Gaussian Splatting
methods.

##  2 Related Works

###  2.1 Map Reconstruction and Texturing

Traditional approaches to view synthesis typically involve 3D model
reconstruction and texture mapping modules. 3D surface reconstruction can be
achieved using algorithm modules such as structure-from-motion [41],
simultaneous localization and mapping [19], and depth map fusion [14, 3].
After obtaining surface meshes based on methods like Marching Cubes [22] and
Poisson surface reconstruction [16], another crucial step is to refine the
texture of triangle faces based on input visual images [20] or through a form
of blending [1] between these images. Learning-based solutions for 3D
reconstruction [24, 10] and texturing [7, 29] tasks have also been proposed,
representing high-frequency texture and employing techniques for natural
blending.

###  2.2 Neural Rendering and Radiance Fields

Instead of explicitly representing scenes, NeRF [25] records environments
using MLP neural networks, opening new possibilities for high-quality view
synthesis based on volumetric ray tracing. However, limited by inference speed
and convergence difficulties, explicit structures such as voxels [21] or deep
features are incorporated into ray-based representations to accelerate
rendering. Relu fields [15] investigate grid-based representations to maintain
the high-fidelity rendering performances of MLPs while speeding up the
reconstruction and reference processes through fixed non-linearity on
interpolated grid parameters. Instant-NGP [27] proposes a multi-resolution
hash table of trainable feature vectors to improve neural networks for the
same speeding-up goal of neural graphics primitives. Despite the significant
improvements achieved by these methods [21, 31, 5], current approaches may not
fully satisfy the requirements of real-time rendering applications.

smooth area$y$$z$$x$3D Position: $\bm{\mu}$Scales:
$\mathbf{S}=[s_{1}\;s_{2}\;0.001]^{T}$Rotation:
$\mathbf{R}=[\mathbf{R}_{1}\;\mathbf{R}_{2}\;\mathbf{n}]$Spherical Harmonics:
$\mathbf{C}$Opacity: $\alpha$Parametrization of Thin GaussiansClone on Tangent
spacenormal accumulated direction modified
direction $\pi$Split on Tangent spaceDensification$Y$$Z$$X$ nearest
neighbors Co-planar ConstraintSmooth Constraintsmooth area$y$$z$$x$3D
Position: $\bm{\mu}$Scales: $\mathbf{S}=[s_{1}\;s_{2}\;0.001]^{T}$Rotation:
$\mathbf{R}=[\mathbf{R}_{1}\;\mathbf{R}_{2}\;\mathbf{n}]$Spherical Harmonics:
$\mathbf{C}$Opacity: $\alpha$Parametrization of Thin GaussiansClone on Tangent
spacenormal accumulated direction modified
direction $\pi$Split on Tangent spaceDensification$Y$$Z$$X$ nearest
neighbors Co-planar ConstraintSmooth Constraint

Figure 2: Geometry-aware strategies of GeoGaussian. In smoothly connected
areas, the parameterization of thin Gaussians contains clear geometry meanings
in the mean vector and covariance matrix. Furthermore, the densification
operation for these thin Gaussians encourages the new generations to lie in
the tangent space established by the position and normal vectors of the
original Gaussian. Finally, these thin Gaussians, measured by a training view,
are used to establish smooth constraints with photometric constraints in the
optimization process.

###  2.3 Gaussian Splatting in Rendering and Reconstruction

Recently, 3DGS [18], combining point-based splatting and blending techniques
for rendering, has achieved real-time speed with photo-realistic rendering
quality. To make the representation more compact, LightGS [8] proposes a
strategy to remove unimportant Gaussians based on each 3D Gaussian’s volume
and opacity. Building on the results of 3DGS, SuGaR [12] extracts mesh faces
from unorganized 3D Gaussian maps, serving as an optional refinement module to
jointly optimize surface and Gaussians. However, this method requires
significantly more GPU resources compared to 3DGS. In contrast to the
architecture of Colmap+GS solutions, 3D Gaussians working as the only
representation are used in incremental tracking and mapping systems [44, 23,
17]. Gaussian-SLAM [44] selects every 5th frame as a keyframe for training 3D
Gaussians based on RGB-D images. Compared with 3DGS and related refinements,
our method focuses on modeling geometry and achieving photo-realistic
rendering performance through redesigned parameterization and densification
processes, as well as geometry-aware constraints.

##  3 Methodology

###  3.1 Gaussian Splatting

In the widely used Gaussian Splatting representation, the scene is built by
several anisotropic Gaussians, and each Gaussian point is characterized by
attributes including the mean $\bm{\mu}_{w}^{i}$ and covariance
$\Sigma_{w}^{i}$ in the world coordinate, where the mean vector and covariance
matrix also represent the position and shape of the $i^{th}$ Gaussian
ellipsoid $\mathcal{G}^{i}$. Additionally, the covariance matrix can be
represented by scaling vector
$\mathbf{S}_{w}^{i}=[s_{1}^{i}\;s_{2}^{i}\;s_{3}^{i}],\mathbf{S}_{w}^{i}\in\mathbb{R}^{3}$
and rotation matrix
$\mathbf{R}_{w}^{i}=[\mathbf{r}_{1}^{i}\;\mathbf{r}_{2}^{i}\;\mathbf{r}_{3}^{i}],\mathbf{R}_{w}^{i}\in
SO(3)$ via the following formulation,

$\Sigma_{w}^{i}=\mathbf{R}_{w}^{i}\mathbf{S}_{w}^{i}{\mathbf{S}_{w}^{i}}^{T}{\mathbf{R}_{w}^{i}}^{T}$
(1)

here $SO$(3) is the special orthogonal group.

Besides, position and shape parameters, spherical harmonics coefficients
$\mathbf{C}_{w}^{i}\in\mathbb{R}^{(m+1)^{2}\times 3}$ where $m$ is the degrees
of the freedom and opacity $\alpha^{i}_{w}\in\mathbb{R}$ have important roles
in color definition. The color of the target pixel can be synthesized by
splatting and blending these $N$ organized Gaussian points that have overlaps
with the pixel. First, the splatting operation is to form 2D Gaussians
($\mathcal{N}(\mu_{I},\Sigma_{I})$) on image planes from 3D Gaussians
($\mathcal{N}(\mu_{w},\Sigma_{w})$) in the world coordinate based on camera
poses,

$\mu_{I}=\Pi(\mathbf{T}_{cw}\mu_{w}),\;\Sigma_{I}=\mathbf{J}\mathbf{W}_{cw}\Sigma_{w}\mathbf{W}_{cw}^{T}\mathbf{J}^{T}$
(2)

here
$\mathbf{T}_{cw}=\left[\begin{array}[]{cc}\mathbf{W}_{cw}&\mathbf{t}_{cw}\\\
\mathbf{0}&1\end{array}\right],\mathbf{T}_{cw}\in SE$(3) is the camera pose
from the world to the camera coordinate, $\mathbf{W}_{cw}$ and
$\mathbf{t}_{cw}$ are rotational and translational components, respectively.
$\mathbf{J}$ is the Jacobian matrix of the projective transformation. Then,
the blending operation is given as follows

$\mathcal{C}_{\mathbf{p}}=\sum_{i\in
N}c^{i}\alpha_{w}^{i}\prod_{j=1}^{i-1}(1-\alpha_{w}^{i}).$ (3)

here $c^{i}$ and $\alpha^{i}_{w}$ represent the color and opacity of the
point. Similar to factor graph optimization problems in SLAM methods, The 3D
Gaussian Splatting and blending operations are also differentiable. Based on
optimization solvers, Gaussian parameters can be trained gradually and
supervised via photometric residuals.

###  3.2 Gaussian Initialization and Densification

#### Initialization.

Surface normals are first extracted from point clouds using a method [26],
wherein unreliable normals are detected based on two rules: 1) points with
long distances from others, and 2) points whose neighbors within a distance
threshold have different normals with large angles with the point. This leads
to the creation of two groups of points, $\mathcal{S}{co}$ and
$\mathcal{S}{ind}$, where $\mathcal{S}{co}$ contains points located on smooth
surfaces, while $\mathcal{S}{ind}$ comprises more individual points.

These sparse points are then fed to the Gaussian Initialization module. For
Gaussians initialized by $\mathcal{S}_{ind}$, we adopt an approach similar to
that used in 3DGS, using the distance between the point and its nearest
neighbor to set the scale. For Gaussians related to $\mathcal{S}{co}$, we
encode more geometric information in the parameters as shown in Figure 2.
Specifically, the scale vector has only 2 DoFs, as the value in the z-axis,
$\mathbf{S}_{3}$, is fixed to a small value to create a thin ellipsoid that
will be aligned with surfaces. Additionally, we assume that the third column,
$\mathbf{R}_{3}$, of the rotation matrix decoupled from the covariance matrix
$\Sigma$ represents the normal, while the other two columns are set as a group
of normalized basis vectors perpendicular to $\mathbf{R}_{3}$.

#### Densification in the tangential space.

For general Gaussians initialized by points from $\mathcal{S}_{ind}$, they are
densified in the traditional manner, which involves splitting a large
ellipsoid into two smaller ones located randomly inside the ellipsoid.
However, the approach differs for thin Gaussians, denoted as
$\mathcal{G}_{\theta}$. To ensure that the new clones and splats align with
the surface, the strategy is designed as follows: First, the gradient
$(\delta\bm{\mu}^{i})$ of the position is calculated over 10 iterations. If
the accumulated gradient exceeds a threshold, the position $\bm{\mu}^{i+1}$ of
the new Gaussian is obtained. However, instead of being set along the
direction of the gradient, the new position of the clone Gaussian is
determined as follows:

$\bm{\mu}^{i+1}=\bm{\mu}^{i}+\delta\bm{\mu}^{i}-\mathbf{R}_{3}^{T}\delta\bm{\mu}^{i}$
(4)

here, $\bm{\mu}^{i}$ is the position of the original Gaussian
$\mathcal{G}_{\theta}$. Additionally, parameters such as $\mathbf{R}_{3}$ are
also cloned to the new Gaussian.

Furthermore, the split process also follows the rule of being co-planar. Due
to our thin Gaussian representation, the new position $\bm{\mu}^{i+1}$ lies on
the plane established by the normal and position vectors of
$\mathcal{G}_{\theta}$, satisfying the equation
$[\bm{\mu}^{i+1}|{\bm{\mu}^{i+1}}^{T}\mathbf{n}^{i+1}={\bm{\mu}^{i}}^{T}\mathbf{n}^{i},\mathbf{S}^{i}\in\mathbf{S}_{\theta}]$.
Based on this method, the Gaussian clone occurs in the tangential space near
the original Gaussian point, and the split operation also ensures that the new
Gaussians lie in the tangential space.

###  3.3  View-dependent Optimization

The iterative optimization process is necessary to ensure that these Gaussians
have the ability to render photo-realistic images at given viewpoints.
Following the strategy of 3DGS [18], the goal of our designed loss functions
is to create correct geometry and adjust incorrectly positioned Gaussians. The
photometric residual between the rendered pixel $c_{i}$ and the ground truth
pixel $\bar{c}_{i}$ is represented as:

$\begin{split}E_{pho}^{i}&=||c_{i}-\bar{c}_{i}||\\\
&=||f(\mathcal{G},\mathbf{T}_{cw})-\bar{c}_{i}||\end{split}$ (5)

where $f(\mathcal{G},\mathbf{T}_{cw})$ rasterizes and blends the relevant
Gaussians to produce the color.

After selecting view-dependent thin Gaussians $\mathcal{G}_{\theta}$, which
are encouraged to preserve smooth connections with their nearest neighbors, we
propose a smoothness loss function to further improve the geometry of the
model. First, only Gaussians detected by the current view are considered in
this function. We then utilize the K-NN algorithm to detect the eight nearest
neighbors around $\mathcal{G}_{\theta}$, which are passed through a filter to
remove outliers that have large angle differences by comparing their normal
vectors with that of $\mathcal{G}_{\theta}$. Therefore, the proposed smooth
constraint for optimizing position and normal vectors can be represented as

$E_{geo}=\left\\{\begin{array}[]{l}||\mathbf{R}_{3}^{T}\bm{\mu}-\frac{1}{m}\sum_{j\in
m}{\mathbf{R}_{3}^{j}}^{T}\bm{\mu}^{j}||\\\
||\mathbf{R}_{3}^{j}-\mathbf{R}_{3}||_{2}\end{array}\right.$ (6)

here $\mathbf{R}_{3}^{T}\bm{\mu}$ shows the distance between the origin of the
world coordinate and the plane established by normal $\mathbf{R}_{3}$ and
position $\bm{\mu}$ vectors of $\mathcal{G}_{\theta}$, while
$\frac{1}{m}\sum_{j\in m}{\mathbf{R}_{3}^{j}}^{T}\bm{\mu}^{j}$ shows the
average point-plane distance of the neighborhood.

By jointly optimizing the normal alignment and depth consistency of the
Gaussian point tangent space in the neighborhood, we can obtain Gaussian
points with a smooth distribution and alignment on the scene surface.

Finally, the loss function $L$ combines both photometric $E_{pho}$ and
geometric $E_{geo}$ residuals using the following architecture,

$L=\lambda_{1}\sum_{\forall k\in
H\times{W}}E_{pho}^{k}+\lambda_{2}\sum_{\mathcal{G}_{\theta}^{i}}E_{geo}^{i}$
(7)

here $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters, $H$ and $W$
represent the height and width of the image respectively, and
$\mathcal{G}_{\theta}^{i}$ is detected by the image.

##  4 Experiments

###  4.1 Implementation Details

In this section, we detail the experimental settings. The proposed GeoGaussian
approach is trained and evaluated on a desktop PC equipped with an Intel Core
i9 12900K 3.50GHz processor and a single GeForce RTX 3090 GPU. Throughout all
experiments, we maintain a consistent learning rate of 0.0002 for Gaussian
optimization. We also use a photometric loss weighting of $0.8$ and a
geometric loss weighting of $0.3$.

In the evaluation phase, we employ a standard procedure. Specifically, we
train our models on public datasets, similar to 3DGS [18], for 30,000
iterations. During the first 2,000 iterations, only photometric constraints
are used in the optimization process. Densification operations cease after
10,000 iterations. Additionally, Gaussians with small opacity values (under
0.05) are removed from the map. The list of nearest neighbors is updated every
100 iterations before reaching 20,000 iterations. After 20,000 iterations, the
update frequency is set to 1,000 iterations, as no more Gaussians are added to
the map at that time.

(a)

(b)

(c)

(d)

Figure 3: Comparisons of novel view rendering on public datasets. At some
challenging viewpoints having bigger differences in translation and
orientation motions compared with training views, the 3DGS and LightGS have
issues with photorealistic rendering. (d) shows the two training views around
the novel one.

###  4.2 Datasets and Metrics

#### Datasets.

Three public datasets, including Replica [36], TUM RGB-D [37], and ICL-NUIM
[13], are utilized in the evaluation process. The Replica dataset comprises
$8$ sequences featuring living room and office scenarios. Similarly, ICL-NUIM
is also a synthetic dataset that offers environments comparable to those in
Replica. We also select $4$ sequences from the TUM RGB-D dataset, which were
captured in real-world settings unlike Replica and ICL-NUIM. These TUM RGB-D
sequences consist of distortion-free images, making them suitable as
references for evaluation.

#### Metrics.

Following the popular evaluation protocol used in methods [18, 36], standard
photometric rendering quality metrics are employed in the experiment section
to evaluate the quality of novel view rendering. These metrics include Peak
Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and
Learned Perceptual Image Patch Similarity (LPIPS). The first metric, PSNR,
evaluates on a color-wise basis, while SSIM measures the similarity between
two images, considering changes in structural information, luminance, and
contrast that can occur with various types of distortion. LPIPS, instead of
comparing two images directly, compares features of two images extracted by a
pre-trained neural network, such as VGG-Net [35].

Method | Metric | R0 | R1 | R2 | Off0 | Off1 | Off2 | Off3 | Off4 | Avg.
---|---|---|---|---|---|---|---|---|---|---
| Vox-
---
Fusion [43]
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 22.39
---
0.683
0.303
| 22.36
---
0.751
0.269
| 23.92
---
0.798
0.234
| 27.79
---
0.857
0.241
| 29.83
---
0.876
0.184
| 20.33
---
0.794
0.243
| 23.47
---
0.803
0.213
| 25.21
---
0.847
0.199
| 24.41
---
0.801
0.236
| Point-
---
SLAM [32]
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 32.40
---
0.974
0.113
| 34.08
---
0.977
0.116
| 35.50
---
0.982
0.111
| 38.26
---
0.983
0.100
| 39.16
---
0.986
0.118
| 33.99
---
0.960
0.156
| 33.48
---
0.960
0.132
| 33.49
---
0.979
0.142
| 35.17
---
0.975
0.124
| Gaussian-
---
Splatting
SLAM [23]
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 34.83
---
0.954
0.068
| 36.43
---
0.959
0.076
| 37.49
---
0.965
0.07
| 39.95
---
0.971
0.072
| 42.09
---
0.977
0.055
| 36.24
---
0.964
0.078
| 36.70
---
0.963
0.065
| 36.07
---
0.957
0.099
| 37.50
---
0.960
0.070
| GeoGaussian
---
(ours)
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 35.20
---
0.952
0.029
| 38.24
---
0.979
0.021
| 39.14
---
0.970
0.024
| 42.74
---
0.981
0.016
| 42.20
---
0.970
0.040
| 37.31
---
0.970
0.029
| 36.66
---
0.964
0.029
| 38.74
---
0.967
0.031
| 38.78
---
0.969
0.027

Table 1: Comparison of different solutions for novel view rendering on the
Replica dataset, GeoGaussian is fed by point clouds, initial camera poses, and
monocular images, while other methods work in real-time with RGB-D streams.
The numbers for Gaussian-Splatting SLAM, Vox-Fusion, and Point-SLAM are taken
from [23].

###  4.3 Scene Rendering Solutions

Different solutions are available for NVS tasks, classified as end-to-end and
hybrid methods. Gaussian Splatting SLAM [23] represents end-to-end methods
that use Gaussians as the map for incremental localization, reconstruction,
and rendering tasks. On the other hand, Vox-Fusion [43] integrates neural
implicit representations into traditional volumetric methods, which estimates
6-DoF camera poses following a traditional pose estimation method without
using neural embeddings saved in voxels. Similarly, GeoGaussian starts from
point clouds and camera poses, which can be generated by SLAM [30] and
Structure-from-Motion [33] methods, and uses differentiable 3D Gaussian
representations for rendering instead of neural implicit ones. But the big
difference is that methods like Vox-Fusion work online, Gaussian Splatting
optimization functions more like a global refinement module, requiring more
time and unable to perform real-time optimization.

Putting aside the differences between these methods for the time being, as the
latest method focusing on the rendering task, we first compare rendering
results against Gaussian-Splatting SLAM, Point-SLAM, and Vox-Fusion on the
same dataset, Replica. As listed in Table 1, the proposed method using offline
Gaussian Splatting achieves the best novel view rendering results, especially
in PSNR and LPIPS metrics. Point-SLAM is also very robust, especially in SSIM,
but the LPIPS metric is worse than Gaussian-Splatting SLAM. Therefore, the
Gaussian Splatting algorithm shows impressive performance in photo-realistic
rendering problems compared with neural representation methods used in Vox-
Fusion and Point-SLAM. Continuing, we will delve into more experiments
comparing our method with state-of-the-art Gaussian Splatting methods in the
following sections.

###  4.4 Gaussian Splatting in NVS

In this section, the proposed method is benchmarked against Gaussian Splatting
methods which are fed by the same inputs, such as sparse point clouds, initial
camera poses, and monocular images, based on the preprocessing method [30].
3DGS [18] is the most important baseline in this areas, while LightGS [8]is a
refinement approach based on 3DGS, involving the pruning of insignificant
Gaussian points and further fine-tuning.

For the experiments, we feed these three methods with the same inputs to
validate their rendering performance, as listed in Table 5. Additionally, we
evaluate these methods in sparse views, which is a significant metric for
evaluating the generality of the trained Gaussian models. Evaluating on a
dataset that is too similar to the training dataset can make it difficult to
assess the model’s generality. To address this, we first select the evaluation
dataset at a frequency of one frame every five frames. The remaining data is
considered as the training dataset, referred to as R1 (100%) as an example in
the second row of Table 5 and Figure 4. Similarly, we use only 10% of the
training dataset for training, referred to as R1 (10%). It is important to
note that the removal process is uniformly implemented across all sequences.

Method | 3DGS | LightGS | GeoGaussian
---|---|---|---
Data | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100%
R1 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 30.49
---
0.932
0.051
| 33.98
---
0.951
0.036
| 37.45
---
0.964
0.029
| 37.60
---
0.965
0.028
| 30.54
---
0.932
0.051
| 34.06
---
0.951
0.035
| 37.72
---
0.965
0.028
| 38.44
---
0.967
0.025
| 31.65
---
0.937
0.041
| 35.17
---
0.957
0.027
| 38.00
---
0.968
0.022
| 38.24
---
0.979
0.021
R2 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 31.53
---
0.935
0.050
| 35.82
---
0.959
0.031
| 38.53
---
0.968
0.028
| 38.70
---
0.968
0.029
| 31.54
---
0.935
0.049
| 35.93
---
0.959
0.031
| 38.78
---
0.968
0.027
| 39.07
---
0.968
0.028
| 32.13
---
0.943
0.041
| 36.81
---
0.963
0.025
| 38.84
---
0.969
0.024
| 39.14
---
0.970
0.024
OFF3 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 30.90
---
0.928
0.052
| 33.86
---
0.946
0.040
| 36.26
---
0.958
0.037
| 36.56
---
0.959
0.036
| 30.93
---
0.928
0.052
| 33.90
---
0.947
0.040
| 36.38
---
0.958
0.036
| 36.63
---
0.958
0.037
| 31.62
---
0.938
0.040
| 33.91
---
0.953
0.032
| 36.42
---
0.963
0.029
| 36.66
---
0.964
0.029
OFF4 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 29.55
---
0.920
0.070
| 32.98
---
0.941
0.049
| 37.70
---
0.962
0.037
| 38.48
---
0.964
0.035
| 29.51
---
0.920
0.070
| 32.97
---
0.941
0.049
| 37.95
---
0.962
0.036
| 38.59
---
0.964
0.036
| 31.90
---
0.936
0.050
| 34.61
---
0.953
0.036
| 38.30
---
0.966
0.030
| 38.74
---
0.967
0.031
Avg. |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 30.62
---
0.929
0.056
| 34.16
---
0.949
0.039
| 37.49
---
0.964
0.032
| 37.84
---
0.964
0.032
| 30.63
---
0.929
0.056
| 34.22
---
0.950
0.039
| 37.71
---
0.964
0.032
| 38.18
---
0.964
0.032
| 31.83
---
0.939
0.043
| 35.13
---
0.957
0.030
| 38.18
---
0.967
0.026
| 38.20
---
0.970
0.026

Table 2: Comparison of rendering on the Replica dataset. The position and
orientation of viewpoints used in training and evaluation are illustrated in
Appendix.

R1 (10%)R1R2 (10%)R2OFF3 (10%)OFF3OFF4
(10%)OFF4$0$$0.2$$0.4$$0.6$$0.8$$1$Number of Gaussian Points (M)3DGS
[18]LightGS [8]GeoGaussian

Figure 4: Statistics of the number of Gaussians in sequences of Replica.

#### Evaluation on Replica.

The proposed method demonstrates superior rendering performance compared to
Gaussian Splatting methods in the R1, R2, and OFF3 sequences, as shown in
Table 1. Particularly in the R1 and R2 sequences, our method exhibits
comprehensive improvements across all three metrics. These environments
consist mostly of less-textured regions, such as single-color walls, where
3DGS [18] and its refinement method LightGS [8] lack sufficient photometric
constraints for their optimization processes. This is evident in the first two
rows of Figure 9, where the ceiling appears blurry in these two methods but is
well-preserved in our method.

Furthermore, the distributions of 3D Gaussians illustrate that the geometry in
our method is more reasonable compared to the results from 3DGS, with flatter
walls and clearer junction lines highlighted in Figure 9. When the training
data becomes sparser, as shown in the OFF3 sequence, our 3D Gaussian model
shows more generality and robustness in rendering performance. Specifically,
in OFF3 (100%), the PSNR distance between 3DGS and ours is $0.1$, while the
distance becomes $0.7$ in OFF3(10%) . Similar improvements are witnessed in
R1, where the distance between these two methods in R1 (100%) is around $0.6$,
and this difference continues to increase in R1 (16.6%) to $1.2$. Since our
evaluation dataset includes rich scenarios, these improvements demonstrate
that the proposed method has advantages in novel view rendering tasks by using
geometry-aware strategies.

In Figure 4, we present the number of Gaussians used for each sequence. All
methods were optimized for 30,000 iterations. LightGS received an additional
5,000 iterations based on the 25,000 models of 3DGS, following LightGS’s
official settings. Generally, 3DGS requires more Gaussians in average compared
to the other two methods. For instance, in OFF4, 3DGS employs about one-fourth
more Gaussians than the other methods, yet the rendering performance remains
similar across all three methods.

#### Evaluation on TUM-RGBD.

In Table 3, sequences from TUM-RGBD are used to evaluate Gaussian Splatting
approaches. In sequences like f3/cabinet and f3/strtex-far, our method
outperforms 3DGS and LightGS significantly. For example, our method achieves a
PSNR of $28.64$, while 3DGS obtains $27.85$ in f3/strtex-far. As shown in the
last row of Figure 9, the 3D Gaussian points in our method are well-organized
in structure, whereas in 3DGS, the geometry is not preserved, appearing as a
big blue ellipsoid passing through the wall. In the cabinet sequence,
characterized by a lack of textured regions, our proposed method demonstrates
impressive improvements in both sparse (10%) and full (100%) training data
scenarios.

However, in the lag-cabinet sequences where the camera moves around a cabinet
placed in the middle of the scene, both 3DGS and LightGS perform better than
GeoGaussian.

Method | Metric |  | f3/100%
---
cabinet
| f3/10%
---
cabinet
| f3/100%
---
strtex-far
| f3/10%
---
strtex-far
Avg.
| 3DGS [18]
---
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 27.24
---
0.907
0.125
| 22.43
---
0.850
0.186
| 27.85
---
0.896
0.073
| 21.41
---
0.737
0.192
| 24.73
---
0.848
0.144
| LightGS [8]
---
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 27.35
---
0.908
0.124
| 22.36
---
0.849
0.190
| 27.90
---
0.896
0.073
| 21.39
---
0.734
0.194
| 24.75
---
0.847
0.145
| GeoGaussian
---
(ours)
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 28.17
---
0.916
0.106
| 25.59
---
0.887
0.138
| 28.64
---
0.907
0.064
| 22.84
---
0.779
0.137
| 26.31
---
0.872
0.111

Table 3: Rendering performance comparison on the TUM RGB-D datasets.

###  4.5 Ablation Studies

#### Sparse Views for Training.

In the section, we first list more settings for sparse view rendering in Table
4, and more results will be provided in Appendix.

Method | Metric |  | 10%
---
(train/test)
| 12.5%
---
(train/test)
| 16.6%
---
(train/test)
| 25%
---
(train/test)
| 50%
---
(train/test)
| 3DGS [18]
---
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 30.49
---
0.932
0.051
| 33.15
---
0.944
0.039
| 33.98
---
0.951
0.035
| 36.58
---
0.961
0.028
| 37.45
---
0.964
0.028
| GeoGaussian
---
(ours)
| PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 31.65
---
0.942
0.041
| 33.95
---
0.949
0.032
| 35.17
---
0.957
0.027
| 36.83
---
0.964
0.023
| 38.00
---
0.968
0.022

Table 4: Sparse view rendering on the R1 sequence of Replica dataset. The
position and orientation of viewpoints used in training and evaluation are
illustrated in Appendix.

#### Training and Evaluation

Figure 5 illustrates the PSNR results collected during training and evaluation
iterations. In Figure 5(a), our method achieves significantly better results
in the first 15K iterations than 3DGS. This improvement can be attributed to
the accuracy provided by our proposed initialization and densification
modules, which assist in better convergence. The similar phenomenon can be
seen in Figure 5(b).

$2$$7$$15$$30$$25$$27$$29$$31$$33$$35$$Iterations(K)$$PSNR$GeoGaussian-
TrainingGeoGaussian-Evaluation3DGS-Training3DGS-Evaluation

(a)

$1$$2$$3$$4$$5$$6$$7$$20$$30$$40$$50$$Iterations(K)$$PSNR$GeoGaussian-100%GeoGaussian-10%3DGS-100%3DGS-10%

(b)

Figure 5: Rendering performance in training and evaluation. Each curve in (b)
is obtained during the evaluation process.

##  5 Conclusion

In this paper, we introduce a novel approach called GeoGaussian, which
emphasizes the importance of preserving accurate geometry in Gaussian models
to enhance their representation in 3D space. Our method first gives Gaussian
parameters a clearer geometric meaning. The third parameter of the scale
vector is used to control thickness, while the third column of the rotation
matrix specifies the normal direction of the thin ellipsoid, where the initial
normal direction is extracted from the initial point clouds. Additionally, we
propose a carefully designed densification approach to effectively organize
the newly generated ellipsoids. In the optimization module, we encourage
ellipsoids in the neighborhood to lie in a co-planar area, further enhancing
the representation quality. Experimental results on public datasets
demonstrate that our method achieves superior performance in terms of geometry
accuracy and photo-realistic novel view rendering compared to state-of-the-art
approaches.

In the future, we plan to explore more comprehensive solutions for improving
the geometry of Gaussian models. This will involve incorporating depth,
normals, and camera poses into the 3D Gaussian Splatting optimization process,
reducing the reliance on point cloud normal vectors.

##  Appendix 0.A Novel View Rendering

In the evaluation section, we present a comprehensive set of experiments to
evaluate the performance of our method. Here, we provide additional
quantitative and qualitative results to further demonstrate the effectiveness
of our approach.

###  0.A.1 Replica Sequences

In Table 2 (see Section 4.1), we compare novel view rendering performance
using the R1, R2, OFF3, and OFF4 sequences. Additionally, we conduct
experiments on four additional sequences (R0, OFF0, OFF1, and OFF2) in this
section. Following the same setup as described in the section of Gaussian
Splatting in NVS (see Section 4.4), we evaluate the rendering performance of
3DGS, LightGS, and GeoGaussian using three metrics, as shown in Table 5.

methods | 3DGS | LightGS | GeoGaussian
---|---|---|---
data | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100%
R0 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 27.24
---
0.875
0.070
| 31.14
---
0.928
0.040
| 34.54
---
0.949
0.031
| 35.10
---
0.952
0.032
| 27.23
---
0.875
0.070
| 31.21
---
0.929
0.040
| 34.68
---
0.950
0.031
| 35.28
---
0.953
0.031
| 28.84
---
0.899
0.057
| 31.77
---
0.931
0.036
| 34.66
---
0.949
0.028
| 35.23
---
0.952
0.029
OFF0 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 31.61
---
0.927
0.062
| 35.99
---
0.959
0.034
| 41.70
---
0.979
0.018
| 42.67
---
0.981
0.017
| 31.63
---
0.927
0.062
| 36.00
---
0.959
0.034
| 41.82
---
0.979
0.018
| 43.08
---
0.981
0.016
| 32.42
---
0.924
0.053
| 36.34
---
0.960
0.030
| 42.11
---
0.980
0.016
| 42.74
---
0.981
0.017
OFF1 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 34.81
---
0.943
0.060
| 38.39
---
0.956
0.046
| 42.27
---
0.972
0.036
| 42.88
---
0.974
0.035
| 34.83
---
0.943
0.060
| 38.43
---
0.956
0.045
| 42.43
---
0.972
0.036
| 43.01
---
0.974
0.035
| 34.86
---
0.939
0.061
| 38.35
---
0.952
0.045
| 41.89
---
0.968
0.039
| 42.17
---
0.970
0.039
OFF2 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 28.27
---
0.911
0.074
| 32.91
---
0.948
0.043
| 36.72
---
0.964
0.033
| 37.34
---
0.966
0.033
| 28.25
---
0.911
0.073
| 32.85
---
0.948
0.045
| 36.95
---
0.965
0.032
| 37.49
---
0.967
0.033
| 29.52
---
0.925
0.054
| 33.97
---
0.955
0.032
| 36.90
---
0.968
0.028
| 37.32
---
0.970
0.029
Avg. |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 30.48
---
0.914
0.067
| 34.61
---
0.948
0.041
| 38.81
---
0.966
0.030
| 39.50
---
0.968
0.029
| 30.49
---
0.914
0.066
| 34.87
---
0.948
0.041
| 38.97
---
0.967
0.029
| 39.72
---
0.969
0.029
| 31.41
---
0.922
0.056
| 35.11
---
0.950
0.036
| 38.89
---
0.966
0.028
| 39.37
---
0.968
0.029

Table 5: Comparison of rendering on the Replica dataset.

GeoGaussian 3DGS

(a) R0 (100%)

(b) OFF0(100%)

(c) OFF1(100%)

(d) OFF2(100%)

Figure 6: Gaussian models generated by GeoGaussian (ours) and 3DGS on the
Replica sequences.

The 3D Gaussian models obtained by GeoGaussian and 3DGS are represented in
Figure 6. Benefiting from the proposed densification method and geometric
constraints, our method, GeoGaussian, preserves the reasonable geometry of
environments. For example, the walls in Figure 6(b) and 6(c) are very thin,
but the geometry is very noisy in the corresponding models of 3DGS.

#### Viewpoints in training and evaluation.

In Table 2, we also evaluate the relationship between rendering performance
and the sparsity of the training views. Therefore, we visualize the position
and orientation of viewpoints used in training and evaluation in Figure 7.

(a) OFF3 (10%)

(b) OFF3 (16.6%)

(c) OFF3 (50%)

(d) OFF3 (100%)

Figure 7: Viewpoints are used during training and evaluation, where the red
camera is used for evaluation.

As shown in Figure 7, when the views used in training become sparse from
OFF3(100%) to OFF3(10%), the scenarios seen from the training frames are
difficult to cover from evaluation viewpoints. Therefore, the structure of
models plays an important role in maintaining rendering performance as
illustrated in Table 2 (see Section 4.4).

###  0.A.2 ICL-NUIM Sequences

$2$$7$$15$$30$$18$$23$$35$$47$$59$$Iterations(K)$$PSNR$GeoGaussian-
TrainingGeoGaussian-Evaluation3DGS-Training3DGS-Evaluation

(a)

$2$$7$$15$$30$$18$$23$$35$$47$$59$$71$$Iterations(K)$$PSNR$GeoGaussian-
TrainingGeoGaussian-Evaluation3DGS-Training3DGS-Evaluation

(b)

Figure 8: Comparison of Rendering performance in training and evaluation
datasets.

methods | 3DGS | LightGS | GeoGaussian
---|---|---|---
data | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100%
ICL-R1 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 40.09
---
0.971
0.022
| 40.39
---
0.973
0.023
| 40.59
---
0.973
0.024
| 40.79
---
0.973
0.025
| 40.25
---
0.972
0.021
| 40.53
---
0.974
0.022
| 41.03
---
0.974
0.023
| 41.26
---
0.974
0.023
| 40.58
---
0.974
0.019
| 41.21
---
0.976
0.019
| 41.51
---
0.977
0.019
| 41.43
---
0.976
0.019
ICL-R2 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 23.31
---
0.777
0.251
| 30.15
---
0.906
0.080
| 37.33
---
0.965
0.023
| 39.10
---
0.974
0.017
| 23.28
---
0.775
0.252
| 30.18
---
0.906
0.080
| 37.51
---
0.965
0.022
| 39.23
---
0.974
0.017
| 25.23
---
0.841
0.142
| 31.26
---
0.925
0.059
| 37.70
---
0.968
0.023
| 39.46
---
0.975
0.018
ICL-O2 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 26.24
---
0.844
0.145
| 29.82
---
0.896
0.078
| 35.54
---
0.943
0.036
| 37.88
---
0.962
0.024
| 26.22
---
0.843
0.146
| 29.77
---
0.896
0.078
| 35.58
---
0.943
0.035
| 37.99
---
0.962
0.023
| 28.35
---
0.874
0.100
| 32.34
---
0.917
0.055
| 36.76
---
0.952
0.026
| 38.54
---
0.967
0.017
ICL-O3 |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 19.28
---
0.718
0.219
| 22.86
---
0.848
0.143
| 32.20
---
0.950
0.044
| 36.04
---
0.975
0.017
| 19.19
---
0.716
0.219
| 22.82
---
0.847
0.143
| 32.21
---
0.949
0.044
| 36.06
---
0.975
0.016
| 21.42
---
0.769
0.145
| 27.06
---
0.907
0.082
| 33.52
---
0.966
0.024
| 36.19
---
0.977
0.015
Avg. |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 27.23
---
0.827
0.159
| 30.80
---
0.905
0.081
| 36.41
---
0.967
0.031
| 38.45
---
0.971
0.020
| 27.23
---
0.826
0.159
| 30.82
---
0.905
0.087
| 36.58
---
0.957
0.031
| 38.63
---
0.971
0.019
| 28.89
---
0.864
0.101
| 32.96
---
0.931
0.053
| 37.37
---
0.965
0.024
| 38.90
---
0.973
0.017

Table 6: Comparison of rendering on the ICL-NUIM dataset.

As we mentioned in the Experiment section (see Section 4.2), the four
sequences are used to evaluate these Gaussian Splatting approaches. As listed
in Table 6, the proposed method shows robust rendering performance in
different training settings compared with state-of-the-art Gaussian Splatting
methods. When the number of training frames is reduced, our method shows more
robust performance since our models have better geometry to alleviate
overfitting problems in rendering tasks. For example, our model trained on
ICL-O3 (10%) achieves a PSNR of $21.42$, while models of 3DGS and LightGS
obtain $19.28$ and $19.19$, respectively.

(a)

(b)

(c)

(d)

Figure 9: Comparisons of novel view rendering on the ICL-NUIM datasets. In
these scenarios, 3DGS and LightGS struggle with photorealistic rendering. As
depicted in (d), the two training views surrounding the novel view.

As we mentioned in the Experiment section (see Section 4.2), the four
sequences are used to evaluate these Gaussian Splatting approaches. As listed
in Table 6, the proposed method shows robust rendering performance in
different training settings compared with state-of-the-art Gaussian Splatting
methods. When the number of training frames is reduced, our method shows more
robust performance since our models have better geometry to alleviate
overfitting problems in rendering tasks. For example, our model trained on
ICL-O3 (10%) achieves a PSNR of $21.42$, while models of 3DGS and LightGS
obtain $19.28$ and $19.19$, respectively.

In an extreme case, when we observe the training process of these methods on
ICL-O3 (10%), as shown in Figure 8, it is evident that Gaussian Splatting
methods tend to overfit the training views due to the limited number of
training views. However, our method maintains a certain level of
generalization ability based on geometry constraints. For example, as the
model improves its rendering results on the training dataset, the performance
on the test dataset does not decrease significantly, as shown in Figure 8(a).

###  0.A.3 TUM RGB-D Sequences

methods | 3DGS | LightGS | GeoGaussian
---|---|---|---
data | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100% | 10% | 16.6% | 50% | 100%
lag-cabinet |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 20.05
---
0.772
0.226
| 23.27
---
0.831
0.143
| 24.63
---
0.864
0.123
| 24.87
---
0.866
0.127
| 20.09
---
0.773
0.225
| 23.32
---
0.832
0.142
| 24.74
---
0.865
0.123
| 24.89
---
0.867
0.126
| 20.85
---
0.804
0.178
| 22.22
---
0.833
0.158
| 24.15
---
0.863
0.130
| 24.79
---
0.868
0.125
long-office |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 16.67
---
0.598
0.336
| 18.88
---
0.686
0.246
| 23.54
---
0.812
0.148
| 24.33
---
0.832
0.139
| 16.69
---
0.600
0.335
| 18.93
---
0.689
0.245
| 23.66
---
0.813
0.146
| 24.43
---
0.835
0.138
| 18.89
---
0.643
0.279
| 20.55
---
0.736
0.196
| 24.56
---
0.833
0.128
| 25.12
---
0.843
0.125
Avg. |  | PSNR$\uparrow$
---
SSIM$\uparrow$
LPIPS$\downarrow$
| 18.31
---
0.685
0.281
| 21.07
---
0.758
0.194
| 24.08
---
0.838
0.135
| 24.60
---
0.849
0.133
| 18.39
---
0.686
0.280
| 21.12
---
0.760
0.193
| 24.20
---
0.839
0.134
| 24.66
---
0.851
0.132
| 19.87
---
0.723
0.228
| 21.38
---
0.748
0.177
| 24.35
---
0.848
0.129
| 24.95
---
0.850
0.125

Table 7: Comparison of rendering on the non-structured sequences on the TUM
RGB-D dataset.

(a) Real scene

(b) 3DGS

(c) GeoGaussian(ours)

Figure 10: Scene of the lag-cabinet sequence and render quality.

Structured environments from the TUM RGB-D dataset are utilized to evaluate
rendering performance in Section 4.4. For a comprehensive evaluation of our
method in general scenarios, we test it on the lag-cabinet and long-office
sequences, which are non-structured Scenes, as shown in Table 7.

As mentioned in Section 4.4, the proposed method is not as effective as 3DGS
and LightGS in the lag-cabinet sequence. This is because the far walls in this
sequence cannot be accurately captured based on point clouds, making it
challenging to obtain good normal vectors from the point clouds.

##  Appendix 0.B Model Reconstruction

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Ground truth mesh Red points from 3DGSGreen points from GeoGaussian

(i)

Figure 11: Reconstruction error visualization.

In this section, we present more quantitative results in the comparison of 3D
Gaussian models. Based on the ground truth mesh models provided by the Replica
dataset, we align these mesh models with point clouds from Gaussian models,
where we randomly sample three points in each Gaussian ellipsoid.

methods | R0 | R1 | R2 | OFF0 | OFF1 | OFF2 | OFF3 | OFF4 | Avg.
---|---|---|---|---|---|---|---|---|---
3DGS |  | mean
---
std
| 0.026
---
0.066
| 0.025
---
0.081
| 0.042
---
0.146
| 0.017
---
0.050
| 0.019
---
0.055
| 0.039
---
0.201
| 0.032
---
0.066
| 0.032
---
0.112
| 0.029
---
0.097
| GeoGaussian
---
(ours)
| mean
---
std
| 0.018
---
0.032
| 0.014
---
0.016
| 0.015
---
0.028
| 0.020
---
0.042
| 0.029
---
0.067
| 0.013
---
0.024
| 0.018
---
0.020
| 0.014
---
0.023
| 0.018
---
0.031

Table 8: Comparison of rendering on the Replica dataset.

As illustrated in Table 8, the proposed method achieves better performance in
mean and standard errors compared to 3DGS. Specifically, in the R2 sequence,
the standard error of our method is $0.028$, while the corresponding value for
3DGS is $0.146$, which is $5$ times worse than ours. Additionally, in OFF2,
the mean error of 3DGS is $3$ times greater than ours.

As shown in Figure 11, the point clouds from 3D Gaussians are aligned with
mesh models, where the green and red points represent GeoGaussian and 3DGS,
respectively. It is evident that points from our method align well with the
model surface, indicating that our Gaussians are distributed according to the
structure and texture of the training views. In contrast, the points from 3DGS
are distributed more uniformly around the training perspective. This indicates
that 3DGS is more inclined to perform densification along the angle of the
training line of sight, while our method is more focused on training a highly
versatile model rather than overfitting to the training perspectives.

##  Appendix 0.C Initialization and Densification

$0$$1$$2$$3$$4$$5$$6$$7$$10$$20$$30$$40$$50$$Iterations(K)$$PSNR$GeoGaussian-100%GeoGaussian-10%3DGS-100%3DGS-10%

(a)

$0$$1$$2$$3$$4$$5$$6$$7$$10$$20$$30$$40$$50$$Iterations(K)$$PSNR$GeoGaussian-100%GeoGaussian-10%3DGS-100%3DGS-10%

(b)

Figure 12: Rendering performance in the first 10,000 iterations.

In Figure 12, the rendering performance of 3DGS and GeoGaussian approaches in
the first 10,000 iterations is presented. Our method stops increasing
Gaussians after 10,000 iterations, while 3DGS continues to increase Gaussians
until 15,000 iterations. Therefore, the densification progress mainly occurs
in the first 10,000 iterations, where Gaussians are initialized and
dramatically densified under the supervision of photometric and geometric
constraints.

As shown in Figure 12, the PSNR performance increases rapidly in the first
10,000 iterations for both 3DGS and GeoGaussian methods. But, our method is
easy to train, especially in sparse view rendering tasks like ICL-R2(10%) and
ICL-O2(10%).

## References

  * [1] Allene, C., Pons, J.P., Keriven, R.: Seamless image-based texture atlases using multi-band blending. In: 2008 19th international conference on pattern recognition. pp. 1–4. IEEE (2008)
  * [2] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 (2023)
  * [3] Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG) 36(4), 1 (2017)
  * [4] Debevec, P.E., Taylor, C.J., Malik, J.: Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 465–474 (2023)
  * [5] Deng, B., Barron, J.T., Srinivasan, P.P.: Jaxnerf: an efficient jax implementation of nerf. URL http://github. com/googleresearch/google-research/tree/master/jaxnerf (2020)
  * [6] DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: Self-supervised interest point detection and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 224–236 (2018)
  * [7] Dong, X., Dong, J., Sun, G., Duan, Y., Qi, L., Yu, H.: Learning-based texture synthesis and automatic inpainting using support vector machines. IEEE Transactions on Industrial Electronics 66(6), 4777–4787 (2018)
  * [8] Fan, Z., Wang, K., Wen, K., Zhu, Z., Xu, D., Wang, Z.: Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv preprint arXiv:2311.17245 (2023)
  * [9] Feng, Y., Feng, Y., You, H., Zhao, X., Gao, Y.: Meshnet: Mesh neural network for 3d shape representation. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 8279–8286 (2019)
  * [10] Gao, J., Chen, W., Xiang, T., Jacobson, A., McGuire, M., Fidler, S.: Learning deformable tetrahedral meshes for 3d reconstruction. Advances In Neural Information Processing Systems 33, 9936–9947 (2020)
  * [11] Goesele, M., Snavely, N., Curless, B., Hoppe, H., Seitz, S.M.: Multi-view stereo for community photo collections. In: 2007 IEEE 11th International Conference on Computer Vision. pp. 1–8. IEEE (2007)
  * [12] Guédon, A., Lepetit, V.: Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 (2023)
  * [13] Handa, A., Whelan, T., McDonald, J., Davison, A.J.: A benchmark for rgb-d visual odometry, 3d reconstruction and slam. In: 2014 IEEE international conference on Robotics and automation (ICRA). pp. 1524–1531. IEEE (2014)
  * [14] Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al.: Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on User interface software and technology. pp. 559–568 (2011)
  * [15] Karnewar, A., Ritschel, T., Wang, O., Mitra, N.: Relu fields: The little non-linearity that could. In: ACM SIGGRAPH 2022 Conference Proceedings. pp. 1–9 (2022)
  * [16] Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the fourth Eurographics symposium on Geometry processing. vol. 7, p. 0 (2006)
  * [17] Keetha, N., Karhade, J., Jatavallabhula, K.M., Yang, G., Scherer, S., Ramanan, D., Luiten, J.: Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:2312.02126 (2023)
  * [18] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (July 2023), https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [19] Kerl, C., Sturm, J., Cremers, D.: Dense visual slam for rgb-d cameras. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 2100–2106. IEEE (2013)
  * [20] Lempitsky, V., Ivanov, D.: Seamless mosaicing of image-based texture maps. In: 2007 IEEE conference on computer vision and pattern recognition. pp. 1–6. IEEE (2007)
  * [21] Liu, L., Gu, J., Zaw Lin, K., Chua, T.S., Theobalt, C.: Neural sparse voxel fields. Advances in Neural Information Processing Systems 33, 15651–15663 (2020)
  * [22] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Seminal graphics: pioneering efforts that shaped the field, pp. 347–353 (1998)
  * [23] Matsuki, H., Murai, R., Kelly, P.H.J., Davison, A.J.: Gaussian Splatting SLAM (2024)
  * [24] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4460–4470 (2019)
  * [25] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [26] Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1) 2(331-340), 2 (2009)
  * [27] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)
  * [28] Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3504–3515 (2020)
  * [29] Oechsle, M., Mescheder, L., Niemeyer, M., Strauss, T., Geiger, A.: Texture fields: Learning texture representations in function space. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4531–4540 (2019)
  * [30] R. Yunus, Y.L., Tombari, F.: Manhattanslam: Robust planar tracking and mapping leveraging mixture of manhattan frames. In: 2021 IEEE international conference on Robotics and automation (ICRA) (2021)
  * [31] Rebain, D., Jiang, W., Yazdani, S., Li, K., Yi, K.M., Tagliasacchi, A.: Derf: Decomposed radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14153–14161 (2021)
  * [32] Sandström, E., Li, Y., Van Gool, L., Oswald, M.R.: Point-slam: Dense neural point cloud-based slam. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 18433–18444 (October 2023)
  * [33] Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [34] Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06). vol. 1, pp. 519–528. IEEE (2006)
  * [35] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [36] Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J.J., Mur-Artal, R., Ren, C., Verma, S., et al.: The replica dataset: A digital replica of indoor spaces.(2019). arXiv preprint arXiv:1906.05797 2(8), 9 (2019)
  * [37] Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. In: 2012 IEEE/RSJ international conference on intelligent robots and systems. pp. 573–580. IEEE (2012)
  * [38] Waechter, M., Moehrle, N., Goesele, M.: Let there be color! large-scale texturing of 3d reconstructions. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 836–850. Springer (2014)
  * [39] Wang, P., Liu, Y., Chen, Z., Liu, L., Liu, Z., Komura, T., Theobalt, C., Wang, W.: F2-nerf: Fast neural radiance field training with free camera trajectories. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4150–4159 (2023)
  * [40] Wang, Y., Tan, D.J., Navab, N., Tombari, F.: Forknet: Multi-branch volumetric semantic completion from a single depth image. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8608–8617 (2019)
  * [41] Westoby, M.J., Brasington, J., Glasser, N.F., Hambrey, M.J., Reynolds, J.M.: âstructure-from-motionâphotogrammetry: A low-cost, effective tool for geoscience applications. Geomorphology 179, 300–314 (2012)
  * [42] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in neural information processing systems 32 (2019)
  * [43] Yang, X., Li, H., Zhai, H., Ming, Y., Liu, Y., Zhang, G.: Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In: 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). pp. 499–507. IEEE Computer Society, Los Alamitos, CA, USA (oct 2022)
  * [44] Yugay, V., Li, Y., Gevers, T., Oswald, M.R.: Gaussian-slam: Photo-realistic dense slam with gaussian splatting. arXiv preprint arXiv:2312.10070 (2023)