# 2D Gaussian Splatting for Geometrically Accurate Radiance Fields

Binbin Huang  ShanghaiTech UniversityChina ,  Zehao Yu  University of
Tübingen, Tübingen AI CenterGermany ,  Anpei Chen  University of Tübingen,
Tübingen AI CenterGermany ,  Andreas Geiger  University of Tübingen, Tübingen
AI CenterGermany and  Shenghua Gao  ShanghaiTech UniversityChina

###### Abstract.

3D Gaussian Splatting (3DGS) has recently revolutionized radiance field
reconstruction, achieving high quality novel view synthesis and fast rendering
speed. However, 3DGS fails to accurately represent surfaces due to the multi-
view inconsistent nature of 3D Gaussians. We present 2D Gaussian Splatting
(2DGS), a novel approach to model and reconstruct geometrically accurate
radiance fields from multi-view images. Our key idea is to collapse the 3D
volume into a set of 2D oriented planar Gaussian disks. Unlike 3D Gaussians,
2D Gaussians provide view-consistent geometry while modeling surfaces
intrinsically. To accurately recover thin surfaces and achieve stable
optimization, we introduce a perspective-accurate 2D splatting process
utilizing ray-splat intersection and rasterization. Additionally, we
incorporate depth distortion and normal consistency terms to further enhance
the quality of the reconstructions. We demonstrate that our differentiable
renderer allows for noise-free and detailed geometry reconstruction while
maintaining competitive appearance quality, fast training speed, and real-time
rendering. Our code will be made publicly available.

novel view synthesis, radiance fields, surface splatting, surface
reconstruction

††journal: TOG††ccs: Computing methodologies Reconstruction††ccs: Computing
methodologies Rendering††ccs: Computing methodologies Machine learning
approaches

https://surfsplatting.github.io

Figure 1. Our method, 2DGS, (a) optimizes a set of 2D oriented disks to
represent and reconstruct a complex real-world scene from multi-view RGB
images. These optimized 2D disks are tightly aligned to the surfaces. (b) With
2D Gaussian splatting, we allow real-time rendering of high quality novel view
images with view consistent normals and depth maps. (c) Finally, our method
provides detailed and noise-free triangle mesh reconstruction from the
optimized 2D disks.

##  1\. Introduction

Photorealistic novel view synthesis (NVS) and accurate geometry reconstruction
stand as pivotal long-term objectives in computer graphics and vision.
Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) has emerged as an
appealing alternative to implicit (Mildenhall et al., 2020; Barron et al.,
2022a) and feature grid-based representations (Barron et al., 2023; Müller et
al., 2022) in NVS, due to its real-time photorealistic NVS results at high
resolutions. Rapidly evolving, 3DGS has been quickly extended with respect to
multiple domains, including anti-aliasing rendering (Yu et al., 2024),
material modeling (Shi et al., 2023; Jiang et al., 2023), dynamic scene
reconstruction (Yan et al., 2023), and animatable avatar creation (Zielonka et
al., 2023; Qian et al., 2023). Nevertheless, it falls short in capturing
intricate geometry since the volumetric 3D Gaussian, which models the complete
angular radiance, conflicts with the thin nature of surfaces.

On the other hand, earlier works (Pfister et al., 2000; Zwicker et al., 2001a)
have shown surfels (surface elements) to be an effective representation for
complex geometry. Surfels approximate the object surface locally with shape
and shade attributes, and can be derived from known geometry. They are widely
used in SLAM (Whelan et al., 2016) and other robotics tasks (Schöps et al.,
2019) as an efficient geometry representation. Subsequent advancements (Yifan
et al., 2019) have incorporated surfels into a differentiable framework.
However, these methods typically require ground truth (GT) geometry, depth
sensor data, or operate under constrained scenarios with known lighting.

Inspired by these works, we propose 2D Gaussian Splatting for 3D scene
reconstruction and novel view synthesis that combines the benefits of both
worlds, while overcoming their limitations. Unlike 3DGS, our approach
represents a 3D scene with 2D Gaussian primitives, each defining an oriented
elliptical disk. The significant advantage of 2D Gaussian over its 3D
counterpart lies in the accurate geometry representation during rendering.
Specifically, 3DGS evaluates a Gaussian’s value at the intersection between a
pixel ray and a 3D Gaussian (Keselman and Hebert, 2022, 2023), which leads to
inconsistency depth when rendered from different viewpoints. In contrast, our
method utilizes explicit ray-splat intersection, resulting in a perspective
accuracy splatting, as illustrated in Figure 2, which in turn significantly
improves reconstruction quality. Furthermore, the inherent surface normals in
2D Gaussian primitives enable direct surface regularization through normal
constraints. In contrast with surfels-based models (Pfister et al., 2000;
Zwicker et al., 2001a; Yifan et al., 2019), our 2D Gaussians can be recovered
from unknown geometry with gradient-based optimization.

While our 2D Gaussian approach excels in geometric modeling, optimizing solely
with photometric losses can lead to noisy reconstructions, due to the
inherently unconstrained nature of 3D reconstruction tasks, as noted in
(Barron et al., 2022b; Zhang et al., 2020; Yu et al., 2022b). To enhance
reconstructions and achieve smoother surfaces, we introduce two regularization
terms: depth distortion and normal consistency. The depth distortion term
concentrates 2D primitives distributed within a tight range along the ray,
addressing the rendering process’s limitation where the distance between
Gaussians is ignored. The normal consistency term minimizes discrepancies
between the rendered normal map and the gradient of the rendered depth,
ensuring alignment between the geometries defined by depth and normals.
Employing these regularizations in combination with our 2D Gaussian model
enables us to extract highly accurate surface meshes, as demonstrated in
Figure 1.

In summary, we make the following contributions:

  * •

We present a highly efficient differentiable 2D Gaussian renderer, enabling
perspective-accurate splatting by leveraging 2D surface modeling, ray-splat
intersection, and volumetric integration.

  * •

We introduce two regularization losses for improved and noise-free surface
reconstruction.

  * •

Our approach achieves state-of-the-art geometry reconstruction and NVS results
compared to other explicit representations.

Figure 2. Comparison of 3DGS and 2DGS. 3DGS utilizes different intersection
planes for value evaluation when viewing from different viewpoints, resulting
in inconsistency. Our 2DGS provides multi-view consistent value evaluations.

##  2\. Related work

###  2.1. Novel view synthesis

Significant advancements have been achieved in NVS, particularly since the
introduction of Neural Radiance Fields (NeRF) (Mildenhall et al., 2021). NeRF
employs a multi-layer perceptron (MLP) to represent geometry and view-
dependent appearance, optimized via volume rendering to deliver exceptional
rendering quality. Post-NeRF developments have further enhanced its
capabilities. For instance, Mip-NeRF (Barron et al., 2021) and subsequent
works (Barron et al., 2022a, 2023; Hu et al., 2023) tackle NeRF’s aliasing
issues. Additionally, the rendering efficiency of NeRF has seen substantial
improvements through techniques such as distillation (Reiser et al., 2021; Yu
et al., 2021) and baking (Reiser et al., 2023; Hedman et al., 2021; Yariv et
al., 2023; Chen et al., 2023a). Moreover, the training and representational
power of NeRF have been enhanced using feature-grid based scene
representations (Chen et al., 2022; Müller et al., 2022; Liu et al., 2020; Sun
et al., 2022a; Chen et al., 2023c; Fridovich-Keil et al., 2022).

Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) has emerged,
demonstrating impressive real-time NVS results. This method has been quickly
extended to multiple domains (Yu et al., 2024; Yan et al., 2023; Zielonka et
al., 2023; Xie et al., 2023). In this work, we propose to “flatten” 3D
Gaussians to 2D Gaussian primitives to better align their shape with the
object surface. Combined with two novel regularization losses, our approach
reconstructs surfaces more accurately than 3DGS while preserving its high-
quality and real-time rendering capabilities.

###  2.2. 3D reconstruction

3D Reconstruction from multi-view images has been a long-standing goal in
computer vision. Multi-view stereo based methods (Schönberger et al., 2016;
Yao et al., 2018; Yu and Gao, 2020) rely on a modular pipeline that involves
feature matching, depth prediction, and fusion. In contrast, recent neural
approaches (Niemeyer et al., 2020; Yariv et al., 2020) represent surface
implicilty via an MLP (Park et al., 2019; Mescheder et al., 2019) , extracting
surfaces post-training via the Marching Cube algorithm. Further advancements
(Oechsle et al., 2021; Wang et al., 2021; Yariv et al., 2021) integrated
implicit surfaces with volume rendering, achieving detailed surface
reconstructions from RGB images. These methods have been extended to large-
scale reconstructions via additional regularization (Yu et al., 2022b; Li et
al., 2023; Yu et al., 2022a), and efficient reconstruction for objects (Wang
et al., 2023). Despite these impressive developments, efficient large-scale
scene reconstruction remains a challenge. For instance, Neuralangelo (Li et
al., 2023) requires 128 GPU hours for reconstructing a single scene from the
Tanks and Temples Dataset (Knapitsch et al., 2017). In this work, we introduce
2D Gaussian splatting, a method that significantly accelerates the
reconstruction process. It achieves similar or slightly better results
compared to previous implicit neural surface representations, while being an
order of magnitude faster.

###  2.3. Differentiable Point-based Graphics

Differentiable point-based rendering (Insafutdinov and Dosovitskiy, 2018;
Yifan et al., 2019; Aliev et al., 2020; Wiles et al., 2020; Rückert et al.,
2022)has been explored extensively due to its efficiency and flexibility in
representing intricate structures. Notably, NPBG (Aliev et al., 2020)
rasterizes point cloud features onto an image plane, subsequently utilizing a
convolutional neural network for RGB image prediction. DSS (Yifan et al.,
2019) focuses on optimizing oriented point clouds from multi-view images under
known lighting conditions. Pulsar (Lassner and Zollhofer, 2021) introduces a
tile-based acceleration structure for more efficient rasterization. More
recently, 3DGS (Kerbl et al., 2023) optimizes anisotropic 3D Gaussian
primitives, demonstrating real-time photorealistic NVS results. Despite these
advances, using point-based representations from unconstrained multi-view
images remains challenging. In this paper, we demonstrate detailed surface
reconstruction using 2D Gaussian primitives. We also highlight the critical
role of additional regularization losses in optimization, showcasing their
significant impact on the quality of the reconstruction.

###  2.4. Concurrent work

Since 3DGS (Kerbl et al., 2023) was introduced, it has been rapidly adapted
across multiple domains. We now review the closest work in inverse rendering.
These work (Liang et al., 2023; Gao et al., 2023; Jiang et al., 2023; Shi et
al., 2023) extend 3DGS by modeling normals as additional attributes of 3D
Gaussian primitives. Our approach, in contrast, inherently defines normals by
representing the tangent space of the 3D surface using 2D Gaussian primitives,
aligning them more closely with the underlying geometry. Additionally, the
aforementioned works predominantly focus on estimating the material properties
of the scene and evaluating their results for relighting tasks. Notably, none
of these works specifically target surface reconstruction, the primary focus
of our work.

We also highlight the distinctions between our method and concurrent works
SuGaR (Guédon and Lepetit, 2023) and NeuSG (Chen et al., 2023b). Unlike SuGaR,
which approximates 2D Gaussians with 3D Gaussians, our method directly employs
2D Gaussians, simplifying the process and enhancing the resulting geometry
without additional mesh refinement. NeuSG optimizes 3D Gaussian primitives and
an implicit SDF network jointly and extracts the surface from the SDF network,
while our approach leverages 2D Gaussian primitives for surface approximation,
offering a faster and conceptually simpler solution.

##  3\. 3D Gaussian Splatting

Kerbl et al. (Kerbl et al., 2023) propose to represent 3D scenes with 3D
Gaussian primitives and render images using differentiable volume splatting.
Specifically, 3DGS explicitly parameterizes Gaussian primitives via 3D
covariance matrix $\boldsymbol{\Sigma}$ and their location $\mathbf{p}_{k}$:

(1)
$\mathcal{G}(\mathbf{p})=\exp({-\frac{1}{2}(\mathbf{p}-\mathbf{p}_{k})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{p}-\mathbf{p}_{k})})$

where the covariance matrix
$\boldsymbol{\Sigma}=\mathbf{R}\mathbf{S}\mathbf{S}^{\top}\mathbf{R}^{\top}$
is factorized into a scaling matrix $\mathbf{S}$ and a rotation matrix
$\mathbf{R}$. To render an image, the 3D Gaussian is transformed into the
camera coordinates with world-to-camera transform matrix $\mathbf{W}$ and
projected to image plane via a local affine transformation $\mathbf{J}$
(Zwicker et al., 2001a):

(2)
$\boldsymbol{\Sigma}^{{}^{\prime}}=\mathbf{J}\mathbf{W}\boldsymbol{\Sigma}\mathbf{W}^{\top}\mathbf{J}^{\top}$

By skipping the third row and column of $\boldsymbol{\Sigma}^{{}^{\prime}}$,
we obtain a 2D Gaussian $\mathcal{G}^{2D}$ with covariance matrix
$\boldsymbol{\Sigma}^{2D}$. Next, 3DGS (Kerbl et al., 2023) employs volumetric
alpha blending to integrate alpha-weighted appearance from front to back:

(3)
$\mathbf{c}(\mathbf{x})=\sum^{K}_{k=1}\mathbf{c}_{k}\,\alpha_{k}\,\mathcal{G}^{2D}_{k}(\mathbf{x})\prod_{j=1}^{k-1}(1-\alpha_{j}\,\mathcal{G}^{2D}_{j}(\mathbf{x}))$

where $k$ is the index of the Gaussian primitives, $\alpha_{k}$ denotes the
alpha values and $\mathbf{c}_{k}$ is the view-dependent appearance. The
attributes of 3D Gaussian primitives are optimized using a photometric loss.

Challenges in Surface Reconstruction: Reconstructing surfaces using 3D
Gaussian modeling and splatting faces several challenges. First, the
volumetric radiance representation of 3D Gaussians conflicts with the thin
nature of surfaces. Second, 3DGS does not natively model surface normals,
essential for high-quality surface reconstruction. Third, the rasterization
process in 3DGS lacks multi-view consistency, leading to varied 2D
intersection planes for different viewpoints (Keselman and Hebert, 2023), as
illustrated in Figure 2 (a). Additionally, using an affine matrix for
transforming a 3D Gaussian into ray space only yields accurate projections
near the center, compromising on perspective accuracy around surrounding
regions (Zwicker et al., 2004). Therefore, it often results in noisy
reconstructions, as shown in Figure 5.

##  4\. 2D Gaussian Splatting

To accurately reconstruct geometry while maintaining high-quality novel view
synthesis, we present differentiable 2D Gaussian splatting (2DGS).

###  4.1. Modeling

Unlike 3DGS (Kerbl et al., 2023), which models the entire angular radiance in
a blob, we simplify the 3-dimensional modeling by adopting “flat” 2D Gaussians
embedded in 3D space. With 2D Gaussian modeling, the primitive distributes
densities within a planar disk, defining the normal as the direction of
steepest change of density. This feature enables better alignment with thin
surfaces. While previous methods (Kopanas et al., 2021; Yifan et al., 2019)
also utilize 2D Gaussians for geometry reconstruction, they require a dense
point cloud or ground-truth normals as input. By contrast, we simultaneously
reconstruct the appearance and geometry given only a sparse calibration point
cloud and photometric supervision.

As illustrated in Figure 3, our 2D splat is characterized by its central point
$\mathbf{p}_{k}$, two principal tangential vectors $\mathbf{t}_{u}$ and
$\mathbf{t}_{v}$, and a scaling vector $\mathbf{S}=(s_{u},s_{v})$ that
controls the variances of the 2D Gaussian. Notice that the primitive normal is
defined by two orthogonal tangential vectors
$\mathbf{t}_{w}=\mathbf{t}_{u}\times\mathbf{t}_{v}$. We can arrange the
orientation into a $3\times 3$ rotation matrix
$\mathbf{R}=[\mathbf{t}_{u},\mathbf{t}_{v},\mathbf{t}_{w}]$ and the scaling
factors into a $3\times 3$ diagonal matrix $\mathbf{S}$ whose last entry is
zero.

A 2D Gaussian is therefore defined in a local tangent plane in world space,
which is parameterized:

(4) $\displaystyle
P(u,v)=\mathbf{p}_{k}+s_{u}\mathbf{t}_{u}u+s_{v}\mathbf{t}_{v}v=\mathbf{H}(u,v,1,1)^{\top}$
(5)
$\displaystyle\text{where}\,\mathbf{H}=\begin{bmatrix}s_{u}\mathbf{t}_{u}&s_{v}\mathbf{t}_{v}&\boldsymbol{0}&\mathbf{p}_{k}\\\
0&0&0&1\\\ \end{bmatrix}=\begin{bmatrix}\mathbf{R}\mathbf{S}&\mathbf{p}_{k}\\\
\boldsymbol{0}&1\\\ \end{bmatrix}$

where $\mathbf{H}\in{4\times 4}$ is a homogeneous transformation matrix
representing the geometry of the 2D Gaussian. For the point $\mathbf{u}=(u,v)$
in $uv$ space, its 2D Gaussian value can then be evaluated by standard
Gaussian

(6) $\mathcal{G}(\mathbf{u})=\exp\left(-\frac{u^{2}+v^{2}}{2}\right)$

The center $\mathbf{p}_{k}$, scaling $(s_{u},s_{v})$, and the rotation
$(\mathbf{t}_{u},\mathbf{t}_{v})$ are learnable parameters. Following 3DGS
(Kerbl et al., 2023), each 2D Gaussian primitive has opacity $\alpha$ and
view-dependent appearance $c$ parameterized with spherical harmonics.

Figure 3. Illustration of 2D Gaussian Splatting. 2D Gaussian Splats are
elliptical disks characterized by a center point $\mathbf{p}_{k}$, tangential
vectors $\mathbf{t}_{u}$ and $\mathbf{t}_{v}$, and two scaling factors
($s_{u}$ and $s_{v}$) control the variance. Their elliptical projections are
sampled through the ray-splat intersection ( Section 4.2) and accumulated via
alpha-blending in image space. 2DGS reconstructs surface attributes such as
colors, depths, and normals through gradient descent.

###  4.2. Splatting

One common strategy for rendering 2D Gaussians is to project the 2D Gaussian
primitives onto the image space using the affine approximation for the
perspective projection (Zwicker et al., 2001a). However, as noted in (Zwicker
et al., 2004), this projection is only accurate at the center of the Gaussian
and has increasing approximation error with increased distance to the center.
To address this issue, Zwicker et al. proposed a formulation based on
homogeneous coordinates. Specifically, projecting the 2D splat onto an image
plane can be described by a general 2D-to-2D mapping in homogeneous
coordinates. Let $\mathbf{W}\in{4\times 4}$ be the transformation matrix from
world space to screen space. The screen space points are hence obtained by

(7)
$\displaystyle\mathbf{x}=(xz,yz,z,z)^{\top}=\mathbf{W}P(u,v)=\mathbf{W}\mathbf{H}(u,v,1,1)^{\top}$

where $\mathbf{x}$ represents a homogeneous ray emitted from the camera and
passing through pixel $(x,y)$ and intersecting the splat at depth $z$. To
rasterize a 2D Gaussian, Zwicker et al. proposed to project its conic into the
screen space with an implicit method using
$\mathbf{M}=(\mathbf{W}\mathbf{H})^{-1}$. However, the inverse transformation
introduces numerical instability especially when the splat degenerates into a
line segment (i.e., if it is viewed from the side). To address this issue,
previous surface splatting rendering methods discard such ill-conditioned
transformations using a pre-defined threshold. However, such a scheme poses
challenges within a differentiable rendering framework, as thresholding can
lead to unstable optimization. To address this problem, we utilize an explicit
ray-splat intersection inspired by (Sigg et al., 2006).

Ray-splat Intersection: We efficiently locate the ray-splat intersections by
finding the intersection of three non-parallel planes, a method initially
designed for specialized hardware (Weyrich et al., 2007). Given an image
coordinate $\mathbf{x}=(x,y)$, we parameterize the ray of a pixel as the
intersection of two orthogonal planes: the x-plane and the y-plane.
Specifically, the x-plane is defined by a normal vector $(-1,0,0)$ and an
offset $x$. Therefore, the x-plane can be represented as a 4D homogeneous
plane $\mathbf{h}_{x}=(-1,0,0,x)$. Similarly, the y-plane is
$\mathbf{h}_{y}=(0,-1,0,y)$. Thus, the ray $\mathbf{x}=(x,y)$ is determined by
the intersection of the $x$-plane and the $y$-planes.

Next, we transform both planes into the local coordinates of the 2D Gaussian
primitives, the $uv$-coordinate system. Note that transforming points on a
plane using a transformation matrix $\mathbf{M}$ is equivalent to transforming
the homogeneous plane parameters using the inverse transpose
$\mathbf{M}^{-\top}$ (Vince, 2008). Therefore, applying
$\mathbf{M}=(\mathbf{W}\mathbf{H})^{-1}$ is equivalent to
$(\mathbf{W}\mathbf{H})^{\top}$, eliminating explicit matrix inversion and
yielding:

(8)
$\displaystyle\mathbf{h}_{u}=(\mathbf{W}\mathbf{H})^{\top}{\mathbf{h}_{x}}\quad\mathbf{h}_{v}=(\mathbf{W}\mathbf{H})^{\top}{\mathbf{h}_{y}}$

As introduced in Section 4.1, points on the 2D Gaussian plane are represented
as $(u,v,1,1)$. At the same time, the intersection point should fall on the
transformed $x$-plane and $y$-plane. Thus,

(9)
$\displaystyle\mathbf{h}_{u}\cdot(u,v,1,1)^{\top}=\mathbf{h}_{v}\cdot(u,v,1,1)^{\top}=0$

This leads to an efficient solution for the intersection point
$\mathbf{u}(\mathbf{x})$:

(10) $\displaystyle
u(\mathbf{x})=\frac{\mathbf{h}_{u}^{2}\mathbf{h}_{v}^{4}-\mathbf{h}_{u}^{4}\mathbf{h}_{v}^{2}}{\mathbf{h}_{u}^{1}\mathbf{h}_{v}^{2}-\mathbf{h}_{u}^{2}\mathbf{h}_{v}^{1}}\qquad
v(\mathbf{x})=\frac{\mathbf{h}_{u}^{4}\mathbf{h}_{v}^{1}-\mathbf{h}_{u}^{1}\mathbf{h}_{v}^{4}}{\mathbf{h}_{u}^{1}\mathbf{h}_{v}^{2}-\mathbf{h}_{u}^{2}\mathbf{h}_{v}^{1}}$

where $\mathbf{h}_{u}^{i},\mathbf{h}_{v}^{i}$ are the $i$-th parameters of the
4D homogeneous plane parameters. We obtain the depth $z$ of the intersected
points via Eq. 7 and evaluate the Gaussian value with Eq 6.

Degenerate Solutions: When a 2D Gaussian is observed from a slanted viewpoint,
it degenerates to a line in screen space. Therefore, it might be missed during
rasterization. To deal with these cases and stabilize optimization, we employ
the object-space low-pass filter introduced in (Botsch et al., 2005):

(11)
$\hat{\mathcal{G}}(\mathbf{x})=\max\left\\{\mathcal{G}(\mathbf{u}(\mathbf{x})),\mathcal{G}(\frac{\mathbf{x}-\mathbf{c}}{\sigma})\right\\}$

where $\mathbf{u}(\mathbf{x})$ is given by (10) and $\mathbf{c}$ is the
projection of center $\mathbf{p}_{k}$. Intuitively,
$\hat{\mathcal{G}}(\mathbf{x})$ is lower-bounded by a fixed screen-space
Gaussian low-pass filter with center $\mathbf{c}_{k}$ and radius $\sigma$. In
our experiments, we set $\sigma=\sqrt{2}/2$ to ensure sufficient pixels are
used during rendering.

Rasterization: We follow a similar rasterization process as in 3DGS (Kerbl et
al., 2023). First, a screen space bounding box is computed for each Gaussian
primitive. Then, 2D Gaussians are sorted based on the depth of their center
and organized into tiles based on their bounding boxes. Finally, volumetric
alpha blending is used to integrate alpha-weighted appearance from front to
back:

(12)
$\mathbf{c}(\mathbf{x})=\sum_{i=1}\mathbf{c}_{i}\,\alpha_{i}\,\hat{\mathcal{G}}_{i}(\mathbf{u}(\mathbf{x}))\prod_{j=1}^{i-1}(1-\alpha_{j}\,\hat{\mathcal{G}}_{j}(\mathbf{u}(\mathbf{x})))$

The iterative process is terminated when the accumulated opacity reaches
saturation.

Figure 4. Visual comparisons (test-set view) between our method, 3DGS, and
SuGaR using scenes from an unbounded real-world dataset - kitchen, bicycle,
stump, and treehill. The first column displays the ground truth novel view,
while the second and the third columns showcase our rendering results,
demonstrating high-fidelity novel view and accurate geometry. Columns four and
five present surface reconstruction results from our method and 3DGS,
utilizing TSDF fusion with a depth range of 6 meters. Background regions
(outside this range) are colored light gray. The sixth column displays results
from SuGaR. In contrast to both baselines, our method excels in capturing
sharp edges and intricate geometry.

##  5\. Training

Our 2D Gaussian method, while effective in geometric modeling, can result in
noisy reconstructions when optimized only with photometric losses, a challenge
inherent to 3D reconstruction tasks (Barron et al., 2022b; Zhang et al., 2020;
Yu et al., 2022b). To mitigate this issue and improve the geometry
reconstruction, we introduce two regularization terms: depth distortion and
normal consistency.

Depth Distortion: Different from NeRF, 3DGS’s volume rendering doesn’t
consider the distance between intersected Gaussian primitives. Therefore,
spread out Gaussians might result in a similar color and depth rendering. This
is different from surface rendering, where rays intersect the first visible
surface exactly once. To mitigate this issue, we take inspiration from Mip-
NeRF360 (Barron et al., 2022a) and propose a depth distortion loss to
concentrate the weight distribution along the rays by minimizing the distance
between the ray-splat intersections:

(13) $\mathcal{L}_{d}=\sum_{i,j}\omega_{i}\omega_{j}|z_{i}-z_{j}|$

where
$\omega_{i}=\alpha_{i}\,\hat{\mathcal{G}}_{i}(\mathbf{u}(\mathbf{x}))\prod_{j=1}^{i-1}(1-\alpha_{j}\,\hat{\mathcal{G}}_{j}(\mathbf{u}(\mathbf{x})))$
is the blending weight of the $i-$th intersection and $z_{i}$ is the depth of
the intersection points. Unlike the distortion loss in Mip-NeRF360, where
$z_{i}$ is the distance between sampled points and is not optimized, our
approach directly encourages the concentration of the splats by adjusting the
intersection depth $z_{i}$. Note that we implement this regularization term
efficiently with CUDA in a manner similar to (Sun et al., 2022b).

Table 1. Quantitative comparison on the DTU Dataset (Jensen et al., 2014). Our
2DGS achieves the highest reconstruction accuracy among other methods and
provides $100\times$ speed up compared to the SDF based baselines.

| 24 | 37 | 40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 |  | Mean | Time
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
implicit | NeRF (Mildenhall et al., 2021) |  | 1.90 | 1.60 | 1.85 | 0.58 | 2.28 | 1.27 | 1.47 | 1.67 | 2.05 | 1.07 | 0.88 | 2.53 | 1.06 | 1.15 | 0.96 |  | 1.49 | ¿ 12h
VolSDF (Yariv et al., 2021) |  | 1.14 | 1.26 | 0.81 | 0.49 | 1.25 | 0.70 | 0.72 | 1.29 | 1.18 | 0.70 | 0.66 | 1.08 | 0.42 | 0.61 | 0.55 |  | 0.86 | ¿12h
NeuS (Wang et al., 2021) |  | 1.00 | 1.37 | 0.93 | 0.43 | 1.10 | 0.65 | 0.57 | 1.48 | 1.09 | 0.83 | 0.52 | 1.20 | 0.35 | 0.49 | 0.54 |  | 0.84 | ¿12h
explicit | 3DGS (Kerbl et al., 2023) |  | 2.14 | 1.53 | 2.08 | 1.68 | 3.49 | 2.21 | 1.43 | 2.07 | 2.22 | 1.75 | 1.79 | 2.55 | 1.53 | 1.52 | 1.50 |  | 1.96 | 11.2 m
SuGaR (Guédon and Lepetit, 2023) |  | 1.47 | 1.33 | 1.13 | 0.61 | 2.25 | 1.71 | 1.15 | 1.63 | 1.62 | 1.07 | 0.79 | 2.45 | 0.98 | 0.88 | 0.79 |  | 1.33 |  $\sim$ 1h
2DGS-15k (Ours) |  | 0.48 | 0.92 | 0.42 | 0.40 | 1.04 | 0.83 | 0.83 | 1.36 | 1.27 | 0.76 | 0.72 | 1.63 | 0.40 | 0.76 | 0.60 |  | 0.83 | 5.5 m
| 2DGS-30k (Ours) |  | 0.48 | 0.91 | 0.39 | 0.39 | 1.01 | 0.83 | 0.81 | 1.36 | 1.27 | 0.76 | 0.70 | 1.40 | 0.40 | 0.76 | 0.52 |  | 0.80 | 18.8 m

Normal Consistency: As our representation is based on 2D Gaussian surface
elements, we must ensure that all 2D splats are locally aligned with the
actual surfaces. In the context of volume rendering where multiple semi-
transparent surfels may exist along the ray, we consider the actual surface at
the median point of intersection, where the accumulated opacity reaches 0.5.
We then align the splats’ normal with the gradients of the depth maps as
follows:

(14) $\mathcal{L}_{n}=\sum_{i}\omega_{i}(1-\mathbf{n}_{i}^{\top}\mathbf{N})$

where $i$ indexes over intersected splats along the ray, $\omega$ denotes the
blending weight of the intersection point, $\mathbf{n}_{i}$ represents the
normal of the splat that is oriented towards the camera, and $\mathbf{N}$ is
the normal estimated by the nearby depth point $\mathbf{p}$. Specifically,
$\mathbf{N}$ is computed with finite differences as follows:

(15)
$\mathbf{N}(x,y)=\frac{\nabla_{x}\mathbf{p}\times\nabla_{y}\mathbf{p}}{|\nabla_{x}\mathbf{p}\times\nabla_{y}\mathbf{p}|}$

By aligning the splat normal with the estimated surface normal, we ensure that
2D splats locally approximate the actual object surface.

Final Loss: Finally, we optimize our model from an initial sparse point cloud
using a set of posed images. We minimize the following loss function:

(16) $\mathcal{L}=\mathcal{L}_{c}+\alpha\mathcal{L}_{d}+\beta\mathcal{L}_{n}$

where $\mathcal{L}c$ is an RGB reconstruction loss combining $\mathcal{L}_{1}$
with the D-SSIM term from (Kerbl et al., 2023), while $\mathcal{L}_{d}$ and
$\mathcal{L}_{n}$ are regularization terms. We set $\alpha=1000$ for bounded
scenes, $\alpha=100$ for unbounded scenes and $\beta=0.05$ for all scenes.

##  6\. Experiments

We now present an extensive evaluation of our 2D Gaussian Splatting
reconstruction method, including appearance and geometry comparison with
previous state-of-the-art implicit and explicit approaches. We then analyze
the contribution of the proposed components.

###  6.1. Implementation

We implement our 2D Gaussian Splatting with custom CUDA kernels, building upon
the framework of 3DGS (Kerbl et al., 2023). We extend the renderer to output
depth distortion maps, depth maps and normal maps for regularizations. During
training, we increase the number of 2D Gaussian primitives following the
adaptive control strategy in 3DGS. Since our method does not directly rely on
the gradient of the projected 2D center, we hence project the gradient of 3D
center $\mathbf{p}_{k}$ onto the screen space as an approximation. Similarly,
we employ a gradient threshold of $0.0002$ and remove splats with opacity
lower than $0.05$ every $3000$ step. We conduct all the experiments on a
single GTX RTX3090 GPU.

Mesh Extraction: To extract meshes from reconstructed 2D splats, we render
depth maps of the training views using the median depth value of the splats
projected to the pixels and utilize truncated signed distance fusion (TSDF) to
fuse the reconstruction depth maps, using Open3D (Zhou et al., 2018). We set
the voxel size to $0.004$ and the truncated threshold to 0.02 during TSDF
fusion. We also extend the original 3DGS to render depth and employ the same
technique for surface reconstruction for a fair comparison.

Table 2. Quantitative results on the Tanks and Temples Dataset (Knapitsch et
al., 2017). We report the F1 score and training time.

| NeuS | Geo-Neus | Neurlangelo | SuGaR | 3DGS | Ours
---|---|---|---|---|---|---
Barn | 0.29 | 0.33 | 0.70 | 0.14 | 0.13 | 0.36
Caterpillar | 0.29 | 0.26 | 0.36 | 0.16 | 0.08 | 0.23
Courthouse | 0.17 | 0.12 | 0.28 | 0.08 | 0.09 | 0.13
Ignatius | 0.83 | 0.72 | 0.89 | 0.33 | 0.04 | 0.44
Meetingroom | 0.24 | 0.20 | 0.32 | 0.15 | 0.01 | 0.16
Truck | 0.45 | 0.45 | 0.48 | 0.26 | 0.19 | 0.26
Mean | 0.38 | 0.35 | 0.50 | 0.19 | 0.09 | 0.30
Time | ¿24h | ¿24h | ¿24h | ¿1h | 14.3 m | 34.2 m

###  6.2. Comparison

Dataset: We evaluate the performance of our method on various datasets,
including DTU (Jensen et al., 2014), Tanks and Temples (Knapitsch et al.,
2017), and Mip-NeRF360 (Barron et al., 2022a). The DTU dataset comprises 15
scenes, each with 49 or 69 images of resolution $1600\times 1200$. We use
Colmap (Schönberger and Frahm, 2016) to generate a sparse point cloud for each
scene and down-sample the images into a resolution of $800\times 600$ for
efficiency. We use the same training process for 3DGS (Kerbl et al., 2023) and
SuGar (Guédon and Lepetit, 2023) for a fair comparison.

Geometry Reconstruction: In Table 1 and Table 3, we compare our geometry
reconstruction to SOTA implicit (i.e., NeRF (Mildenhall et al., 2020), VolSDF
(Yariv et al., 2021), and NeuS (Wang et al., 2021)), explicit (i.e., 3DGS
(Kerbl et al., 2023) and concurrent work SuGaR (Guédon and Lepetit, 2023)) on
Chamfer distance and training time using the DTU dataset. Our method
outperforms all compared methods in terms of Chamfer distance. Moreover, as
shown in Table 2, 2DGS achieves competitive results with SDF models (i.e.,
NeuS (Wang et al., 2021) and Geo-Neus (Fu et al., 2022)) on the TnT dataset,
and significantly better reconstruction than explicit reconstruction methods
(i.e., 3DGS and SuGaR). Notably, our model demonstrates exceptional
efficiency, offering a reconstruction speed that is approximately 100 times
faster compared to implicit reconstruction methods and more than 3 times
faster than the concurrent work SuGaR. Our approach can also achieve
qualitatively better reconstructions with more appearance and geometry details
and fewer outliers, as shown in Figure 5. We include the geometry
reconstruction results in supplementary Figure 10. Moreover, SDF-based
reconstruction methods require predefining the spherical size for
initialization, which plays a critical role in the success of SDF
reconstruction. Bu contrast, our method leverages radiance field based
geometry modeling and is less sensitive to initialization.

Table 3. Performance comparison between 2DGS (ours), 3DGS and SuGaR on the DTU
dataset (Jensen et al., 2014). We report the averaged chamfer distance, PSNR,
reconstruction time, and model size.

| CD $\downarrow$ | PSNR $\uparrow$ | Time $\downarrow$ | MB (Storage) $\downarrow$
---|---|---|---|---
3DGS (Kerbl et al., 2023) | 1.96 | 35.76 | 11.2 m | 113
SuGaR (Guédon and Lepetit, 2023) | 1.33 | 34.57 |  $\sim$1 h | 1247
2DGS-15k (Ours) | 0.83 | 33.42 | 5.5 m | 52
2DGS-30k (Ours) | 0.80 | 34.52 | 18.8 m | 52

Figure 5. Qualitative comparison on the DTU benchmark (Jensen et al., 2014).
Our 2DGS produces detailed and noise-free surfaces.

Appearance Reconstruction: Our method represents 3D scenes as radiance fields,
providing high-quality novel view synthesis. In this section, we compare our
novel view renderings using the Mip-NeRF360 dataset against baseline
approaches, as shown in Table 4 and Figure 4. Note that, since the ground
truth geometry is not available in the Mip-NeRF360 dataset and we hence focus
on quantitative comparison. Remarkably, our method consistently achieves
competitive NVS results across state-of-the-art techniques while providing
geometrically accurate surface reconstruction.

Table 4. Quantitative results on Mip-NeRF 360 (Barron et al., 2022a) dataset.
All scores of the baseline methods are directly taken from their papers
whenever available. We report the performance of 3DGS, SuGaR and ours using
$30k$ iterations.

| Outdoor Scene | Indoor scene
---|---|---
| PSNR $\uparrow$ | SSIM $\uparrow$ | LIPPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LIPPS $\downarrow$
NeRF | 21.46 | 0.458 | 0.515 | 26.84 | 0.790 | 0.370
Deep Blending | 21.54 | 0.524 | 0.364 | 26.40 | 0.844 | 0.261
Instant NGP | 22.90 | 0.566 | 0.371 | 29.15 | 0.880 | 0.216
MERF | 23.19 | 0.616 | 0.343 | 27.80 | 0.855 | 0.271
MipNeRF360 | 24.47 | 0.691 | 0.283 | 31.72 | 0.917 | 0.180
BakedSDF | 22.47 | 0.585 | 0.349 | 27.06 | 0.836 | 0.258
Mobile-NeRF | 21.95 | 0.470 | 0.470 | - | - | -
3DGS | 24.24 | 0.705 | 0.283 | 30.99 | 0.926 | 0.199
SuGaR | 22.76 | 0.631 | 0.349 | 29.44 | 0.911 | 0.216
2DGS (Ours) | 24.33 | 0.709 | 0.284 | 30.39 | 0.924 | 0.182

###  6.3. Ablations

In this section, we isolate the design choices and measure their effect on
reconstruction quality, including regularization terms and mesh extraction. We
conduct experiments on the DTU dataset (Jensen et al., 2014) with 15$k$
iterations and report the reconstruction accuracy, completeness and average
reconstruction quality. The quantitative effect of the choices is reported in
Table 5.

Regularization: We first examine the effects of the proposed normal
consistency and depth distortion regularization terms. Our model (Table 5 E)
provides the best performance when applying both regularization terms. We
observe that disabling the normal consistency (Table 5 A) can lead to
incorrect orientation, as shown in Figure 6 A. Additionally, the absence of
depth distortion (Table 5 B) results in a noisy surface, as shown in Figure 6
B.

Table 5. Quantitative studies for the regularization terms and mesh extraction
methods on the DTU dataset.

| Accuracy $\downarrow$ | Completion $\downarrow$ | Averagy $\downarrow$
---|---|---|---
A. w/o normal consistency | 1.35 | 1.13 | 1.24
B. w/o depth distortion | 0.89 | 0.87 | 0.88
C. w / expected depth | 0.88 | 1.01 | 0.94
D. w / Poisson | 1.25 | 0.89 | 1.07
E. Full Model | 0.79 | 0.86 | 0.83

Figure 6. Qualitative studies for the regularization effects. From left to
right – input image, surface normals without normal consistency, without depth
distortion, and our full model. Disabling the normal consistency loss leads to
noisy surface orientations; conversely, omitting depth distortion
regularization results in blurred surface normals. The complete model,
employing both regularizations, successfully captures sharp and flat features.

Mesh Extraction: We now analyze our choice for mesh extraction. Our full model
(Table 5 E) utilizes TSDF fusion for mesh extraction with median depth. One
alternative option is to use the expected depth instead of the median depth.
However, it yields worse reconstructions as it is more sensitive to outliers,
as shown in Table 5 C. Further, our approach surpasses screened Poisson
surface reconstruction (SPSR)(Table 5 D) (Kazhdan and Hoppe, 2013) using 2D
Gaussians’ center and normal as inputs, due to SPSR’s inability to incorporate
the opacity and the size of 2D Gaussian primitives.

##  7\. Conclusion

We presented 2D Gaussian splatting, a novel approach for geometrically
accurate radiance field reconstruction. We utilized 2D Gaussian primitives for
3D scene representation, facilitating accurate and view consistent geometry
modeling and rendering. We proposed two regularization techniques to further
enhance the reconstructed geometry. Extensive experiments on several
challenging datasets verify the effectiveness and efficiency of our method.

Limitations: While our method successfully delivers accurate appearance and
geometry reconstruction for a wide range of objects and scenes, we also
discuss its limitations: First, we assume surfaces with full opacity and
extract meshes from multi-view depth maps. This can pose challenges in
accurately handling semi-transparent surfaces, such as glass, due to their
complex light transmission properties. Secondly, our current densification
strategy favors texture-rich over geometry-rich areas, occasionally leading to
less accurate representations of fine geometric structures. A more effective
densification strategy could mitigate this issue. Finally, our regularization
often involves a trade-off between image quality and geometry, and can
potentially lead to oversmoothing in certain regions.

Acknowledgement: BH and SG are supported by NSFC #62172279, #61932020, Program
of Shanghai Academic Research Leader. ZY, AC and AG are supported by the ERC
Starting Grant LEGO-3D (850533) and DFG EXC number 2064/1 - project number
390727645.

## References

  * (1)
  * Aliev et al. (2020) Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. 2020.  Neural point-based graphics. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_. Springer, 696–712.
  * Barron et al. (2021) Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. 2021.  Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.  _ICCV_ (2021).
  * Barron et al. (2022a) Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022a.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5470–5479.
  * Barron et al. (2022b) Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022b.  Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields.  _CVPR_ (2022).
  * Barron et al. (2023) Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2023.  Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields.  _ICCV_ (2023).
  * Botsch et al. (2005) Mario Botsch, Alexander Hornung, Matthias Zwicker, and Leif Kobbelt. 2005.  High-quality surface splatting on today’s GPUs. In _Proceedings Eurographics/IEEE VGTC Symposium Point-Based Graphics, 2005._ IEEE, 17–141.
  * Chen et al. (2022) Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022.  TensoRF: Tensorial Radiance Fields. In _European Conference on Computer Vision (ECCV)_.
  * Chen et al. (2023b) Hanlin Chen, Chen Li, and Gim Hee Lee. 2023b.  NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance.  _arXiv preprint arXiv:2312.00846_ (2023).
  * Chen et al. (2023a) Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. 2023a.  Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16569–16578.
  * Chen et al. (2023c) Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. 2023c.  NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4182–4194.
  * Fridovich-Keil et al. (2022) Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022.  Plenoxels: Radiance Fields without Neural Networks. In _CVPR_.
  * Fu et al. (2022) Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. 2022.  Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction.  _Advances in Neural Information Processing Systems (NeurIPS)_ (2022).
  * Gao et al. (2023) Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. 2023.  Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing.  _arXiv:2311.16043_ (2023).
  * Guédon and Lepetit (2023) Antoine Guédon and Vincent Lepetit. 2023.  SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering.  _arXiv preprint arXiv:2311.12775_ (2023).
  * Hedman et al. (2021) Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. 2021.  Baking neural radiance fields for real-time view synthesis. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5875–5884.
  * Hu et al. (2023) Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. 2023.  Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fields. In _ICCV_.
  * Insafutdinov and Dosovitskiy (2018) Eldar Insafutdinov and Alexey Dosovitskiy. 2018.  Unsupervised learning of shape and pose with differentiable point clouds.  _Advances in neural information processing systems_ 31 (2018).
  * Jensen et al. (2014) Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. 2014.  Large scale multi-view stereopsis evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 406–413.
  * Jiang et al. (2023) Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. 2023.  GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces.  _arXiv preprint arXiv:2311.17977_ (2023).
  * Kazhdan and Hoppe (2013) Michael Kazhdan and Hugues Hoppe. 2013.  Screened poisson surface reconstruction.  _ACM Transactions on Graphics (ToG)_ 32, 3 (2013), 1–13.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 4 (July 2023).  https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * Keselman and Hebert (2022) Leonid Keselman and Martial Hebert. 2022.  Approximate differentiable rendering with algebraic surfaces. In _European Conference on Computer Vision_. Springer, 596–614.
  * Keselman and Hebert (2023) Leonid Keselman and Martial Hebert. 2023.  Flexible techniques for differentiable rendering with 3d gaussians.  _arXiv preprint arXiv:2308.14737_ (2023).
  * Knapitsch et al. (2017) Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017.  Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction.  _ACM Transactions on Graphics_ 36, 4 (2017).
  * Kopanas et al. (2021) Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. 2021.  Point-Based Neural Rendering with Per-View Optimization. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 29–43.
  * Lassner and Zollhofer (2021) Christoph Lassner and Michael Zollhofer. 2021.  Pulsar: Efficient sphere-based neural rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1440–1449.
  * Li et al. (2023) Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. 2023.  Neuralangelo: High-Fidelity Neural Surface Reconstruction. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Liang et al. (2023) Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. 2023.  GS-IR: 3D Gaussian Splatting for Inverse Rendering.  _arXiv preprint arXiv:2311.16473_ (2023).
  * Liu et al. (2020) Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020.  Neural Sparse Voxel Fields.  _NeurIPS_ (2020).
  * Luiten et al. (2024) Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2024.  Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis. In _3DV_.
  * Mescheder et al. (2019) Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019.  Occupancy Networks: Learning 3D Reconstruction in Function Space. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _ECCV_.
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ 65, 1 (2021), 99–106.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.  _ACM Trans. Graph._ 41, 4, Article 102 (July 2022), 15 pages.
  * Niemeyer et al. (2020) Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020.  Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Oechsle et al. (2021) Michael Oechsle, Songyou Peng, and Andreas Geiger. 2021.  UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In _International Conference on Computer Vision (ICCV)_.
  * Park et al. (2019) Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Pfister et al. (2000) Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, and Markus Gross. 2000.  Surfels: Surface elements as rendering primitives. In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_. 335–342.
  * Qian et al. (2023) Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. 2023.  GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians.  _arXiv preprint arXiv:2312.02069_ (2023).
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021.  KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. In _International Conference on Computer Vision (ICCV)_.
  * Reiser et al. (2023) Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. 2023.  Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–12.
  * Rückert et al. (2022) Darius Rückert, Linus Franke, and Marc Stamminger. 2022.  Adop: Approximate differentiable one-pixel point rendering.  _ACM Transactions on Graphics (ToG)_ 41, 4 (2022), 1–14.
  * Schönberger and Frahm (2016) Johannes Lutz Schönberger and Jan-Michael Frahm. 2016.  Structure-from-Motion Revisited. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Schönberger et al. (2016) Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016.  Pixelwise View Selection for Unstructured Multi-View Stereo. In _European Conference on Computer Vision (ECCV)_.
  * Schöps et al. (2019) Thomas Schöps, Torsten Sattler, and Marc Pollefeys. 2019.  Surfelmeshing: Online surfel-based mesh reconstruction.  _IEEE transactions on pattern analysis and machine intelligence_ 42, 10 (2019), 2494–2507.
  * Shi et al. (2023) Yahao Shi, Yanmin Wu, Chenming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang, Jian Zhang, Bin Zhou, Errui Ding, and Jingdong Wang. 2023.  GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorization.  _Arxiv_ (2023).  arXiv:2312.05133
  * Sigg et al. (2006) Christian Sigg, Tim Weyrich, Mario Botsch, and Markus H Gross. 2006.  GPU-based ray-casting of quadratic surfaces.. In _PBG@ SIGGRAPH_. 59–65.
  * Sun et al. (2022a) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022a.  Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In _CVPR_.
  * Sun et al. (2022b) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022b.  Improved Direct Voxel Grid Optimization for Radiance Fields Reconstruction.  _arxiv cs.GR 2206.05085_ (2022).
  * Vince (2008) John Vince. 2008.  _Geometric algebra for computer graphics_.  Springer Science & Business Media.
  * Wang et al. (2021) Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  _Advances in Neural Information Processing Systems_ 34 (2021), 27171–27183.
  * Wang et al. (2023) Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. 2023.  NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Weyrich et al. (2007) Tim Weyrich, Simon Heinzle, Timo Aila, Daniel B Fasnacht, Stephan Oetiker, Mario Botsch, Cyril Flaig, Simon Mall, Kaspar Rohrer, Norbert Felber, et al. 2007\.  A hardware architecture for surface splatting.  _ACM Transactions on Graphics (TOG)_ 26, 3 (2007), 90–es.
  * Whelan et al. (2016) Thomas Whelan, Renato F Salas-Moreno, Ben Glocker, Andrew J Davison, and Stefan Leutenegger. 2016.  ElasticFusion: Real-time dense SLAM and light source estimation.  _The International Journal of Robotics Research_ 35, 14 (2016), 1697–1716.
  * Wiles et al. (2020) Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. 2020.  SynSin: End-to-end View Synthesis from a Single Image. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Xie et al. (2023) Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. 2023.  PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics.  _arXiv preprint arXiv:2311.12198_ (2023).
  * Yan et al. (2023) Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. 2023.  Street Gaussians for Modeling Dynamic Urban Scenes.  (2023).
  * Yao et al. (2018) Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018.  MVSNet: Depth Inference for Unstructured Multi-view Stereo.  _European Conference on Computer Vision (ECCV)_ (2018).
  * Yariv et al. (2021) Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ 34 (2021), 4805–4815.
  * Yariv et al. (2023) Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. 2023.  BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis.  _arXiv_ (2023).
  * Yariv et al. (2020) Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. 2020.  Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Yifan et al. (2019) Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019.  Differentiable surface splatting for point-based geometry processing.  _ACM Transactions on Graphics (TOG)_ 38, 6 (2019), 1–14.
  * Yu et al. (2021) Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021.  PlenOctrees for Real-time Rendering of Neural Radiance Fields. In _ICCV_.
  * Yu et al. (2022a) Zehao Yu, Anpei Chen, Bozidar Antic, Songyou Peng, Apratim Bhattacharyya, Michael Niemeyer, Siyu Tang, Torsten Sattler, and Andreas Geiger. 2022a.  SDFStudio: A Unified Framework for Surface Reconstruction.  https://github.com/autonomousvision/sdfstudio
  * Yu et al. (2024) Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. 2024.  Mip-Splatting: Alias-free 3D Gaussian Splatting.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ (2024).
  * Yu and Gao (2020) Zehao Yu and Shenghua Gao. 2020.  Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yu et al. (2022b) Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. 2022b.  MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction.  _Advances in Neural Information Processing Systems (NeurIPS)_ (2022).
  * Zhang et al. (2020) Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. 2020.  NeRF++: Analyzing and Improving Neural Radiance Fields.  _arXiv:2010.07492_ (2020).
  * Zhou et al. (2018) Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. 2018.  Open3D: A Modern Library for 3D Data Processing.  _arXiv:1801.09847_ (2018).
  * Zielonka et al. (2023) Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. 2023.  Drivable 3D Gaussian Avatars.  (2023).  arXiv:2311.08581 [cs.CV]
  * Zwicker et al. (2001a) Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2001a.  EWA volume splatting. In _Proceedings Visualization, 2001. VIS’01._ IEEE, 29–538.
  * Zwicker et al. (2001b) Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2001b.  Surface splatting. In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_. 371–378.
  * Zwicker et al. (2004) Matthias Zwicker, Jussi Rasanen, Mario Botsch, Carsten Dachsbacher, and Mark Pauly. 2004.  Perspective accurate splatting. In _Proceedings-Graphics Interface_. 247–254.

Figure 7. Affine-approximation adopted in (Zwicker et al., 2001b)(Kerbl et
al., 2023) causes perspective distortion and inaccurate depth.

(a) Ground-truth

(b) MipNeRF360 (Barron et al., 2022b), SSIM=0.813

Figure 8. We visualize the depth maps generated by MipNeRF360 (Barron et al.,
2022b), 3DGS (Kerbl et al., 2023), and our method. The depth maps for 3DGS (d)
and 2DGS (f) are rendered using Eq. 18 and visualized following MipNeRF360. To
highlight the surface smoothness, we further visualize the normal estimated
from depth points using Eq. 15 for both 3DGS (c) and ours (e). While
MipNeRF360 is capable of producing plausibly smooth depth maps, its sampling
process may result in the loss of detailed structures. Both 3DGS and 2DGS
excel at modeling thin structures; however, as illustrated in (c) and (e), the
depth map of 3DGS exhibits significant noise. In contrast, our approach
generates sampled depth points with normals consistent with the rendered
normal map (refer to Figure 1b), thereby enhancing depth fusion during the
meshing process.

(c) 3DGS, normals from depth points (d) 3DGS (Kerbl et al., 2023), SSIM=0.834
(e) Our model (2DGS), normals from depth points (f) Our model (2DGS),
SSIM=0.845

##  Appendix A Details of Depth Distortion

While Barron et al. (Barron et al., 2022b) calculates the distortion loss with
samples on the ray, we operate Gaussian primitives, where the intersected
depth may not be ordered. To this end, we adopt an $\mathcal{L}_{2}$ loss and
transform the intersected depth $z$ to NDC space to down-weight distant
Gaussian primitives, $m=\text{NDC}(z)$, with near and far plane empirically
set to 0.2 and 1000. We implemented our depth distortion loss based on (Sun et
al., 2022b), also powered by tile-based rendering. Here we show that the
nested algorithm can be implemented in a single forward pass:

(17)
$\small\begin{split}\mathcal{L}&=\sum_{i=0}^{N-1}\sum_{j=0}^{i-1}\omega_{i}\omega_{j}(m_{i}-m_{j})^{2}\\\
&=\sum_{i=0}^{N-1}\omega_{i}\left(m_{i}^{2}\sum_{j=0}^{i-1}\omega_{j}+\sum_{j=0}^{i-1}\omega_{j}m_{j}^{2}-2m_{i}\sum_{j=0}^{i-1}\omega_{j}m_{j}\right)\\\
&=\sum_{i=0}^{N-1}\omega_{i}\left(m_{i}^{2}A_{i-1}+D^{2}_{i-1}-2m_{i}D_{i-1}\right),\end{split}$

where $A_{i}=\sum_{j=0}^{i}\omega_{j}$, $D_{i}=\sum_{j=0}^{i}\omega_{j}m_{j}$
and $D^{2}_{i}=\sum_{j=0}^{i}\omega_{j}m_{j}^{2}$.

Specifically, we let
$e_{i}=\left(m_{i}^{2}A_{i-1}+D^{2}_{i-1}-2m_{i}D_{i-1}\right)$ so that the
distortion loss can be “rendered” as
$\mathcal{L}_{i}=\sum_{j=0}^{i}\omega_{j}e_{j}$. Here, $\mathcal{L}_{i}$
measures the depth distortion up to the $i$-th Gaussian. During marching
Gaussian front-to-back, we simultaneously accumulate $A_{i}$, $D_{i}$ and
$D^{2}_{i}$, preparing for the next distortion computation
$\mathcal{L}_{i+1}$. Similarly, the gradient of the depth distortion can be
back-propagated to the primitives back-to-front. Different from implicit
methods where $m$ are the pre-defined sampled depth and non-differentiable, we
additionally back-propagate the gradient through the intersection $m$,
encouraging the Gaussians to move tightly together directly.

##  Appendix B Depth calculations

Mean depth:There are two optional depth computations used for our meshing
process. The mean (expected) depth is calculated by weighting the intersected
depth:

(18) $\displaystyle
z_{\text{mean}}=\sum_{i}\omega_{i}z_{i}/({\sum_{i}\omega_{i}+\epsilon})$

where $\omega_{i}=T_{i}\alpha_{i}\hat{\mathcal{G}}_{i}(\mathbf{u}(\mathbf{x})$
is the weight contribution of the $i$-th Gaussian and
$T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j}\,\hat{\mathcal{G}}_{j}(\mathbf{u}(\mathbf{x})))$
measures its visibility. It is important to normalize the depth with the
accumulated alpha $A=\sum_{i}\omega_{i}$ to ensure that a 2D Gaussian can be
rendered as a planar 2D disk in the depth visualization.

Median depth:We compute the median depth as the largest “visible” depth,
considering $T_{i}=0.5$ as the pivot for surface and free space:

(19) $z_{\text{median}}=\max\\{z_{i}|T_{i}>0.5\\}.$

We find our median depth computation is more robust to (Luiten et al., 2024).
When a ray’s accumulated alpha does not reach 0.5, while Luiten et al. sets a
default value of 15, our computation selects the last Gaussian, which is more
accurate and suitable for training.

##  Appendix C Additional Results

Our 2D Gaussian Splatting method achieves comparable performance even without
the need for regularizations, as Table 7 shows. Additionally, we have provided
a breakdown of various per-scene metrics for the MipNeRF360 dataset (Barron et
al., 2022b) in Tables 6. Figure 8 presents a comparison of our rendered depth
maps with those from 3DGS and MipNeRF360.

Table 6. PSNR$\uparrow$, SSIM$\uparrow$, LIPPS$\downarrow$ scores for
MipNeRF360 dataset.

| bicycle | flowers | garden | stump | treehill | room | counter | kitchen | bonsai | mean
---|---|---|---|---|---|---|---|---|---|---
3DGS | 24.71 | 21.09 | 26.63 | 26.45 | 22.33 | 31.50 | 29.07 | 31.13 | 32.26 | 27.24
SuGaR | 23.12 | 19.25 | 25.43 | 24.69 | 21.33 | 30.12 | 27.57 | 29.48 | 30.59 | 25.73
Ours | 24.82 | 20.99 | 26.91 | 26.41 | 22.52 | 30.86 | 28.45 | 30.62 | 31.64 | 27.03
3DGS | 0.729 | 0.571 | 0.834 | 0.762 | 0.627 | 0.922 | 0.913 | 0.926 | 0.943 | 0.803
SuGaR | 0.639 | 0.486 | 0.776 | 0.686 | 0.566 | 0.910 | 0.892 | 0.908 | 0.932 | 0.755
Ours | 0.731 | 0.573 | 0.845 | 0.764 | 0.630 | 0.918 | 0.908 | 0.927 | 0.940 | 0.804
3DGS | 0.265 | 0.377 | 0.147 | 0.266 | 0.362 | 0.231 | 0.212 | 0.138 | 0.214 | 0.246
SuGaR | 0.344 | 0.416 | 0.220 | 0.335 | 0.429 | 0.245 | 0.232 | 0.164 | 0.221 | 0.290
Ours | 0.271 | 0.378 | 0.138 | 0.263 | 0.369 | 0.214 | 0.197 | 0.125 | 0.194 | 0.239

Table 7. PSNR scores for Synthetic NeRF dataset (Mildenhall et al., 2021).

| Mic | Chair | Ship | Materials | Lego | Drums | Ficus | Hotdog | Avg.
---|---|---|---|---|---|---|---|---|---
Plenoxels | 33.26 | 33.98 | 29.62 | 29.14 | 34.10 | 25.35 | 31.83 | 36.81 | 31.76
INGP-Base | 36.22 | 35.00 | 31.10 | 29.78 | 36.39 | 26.02 | 33.51 | 37.40 | 33.18
Mip-NeRF | 36.51 | 35.14 | 30.41 | 30.71 | 35.70 | 25.48 | 33.29 | 37.48 | 33.09
Point-NeRF | 35.95 | 35.40 | 30.97 | 29.61 | 35.04 | 26.06 | 36.13 | 37.30 | 33.30
3DGS | 35.36 | 35.83 | 30.80 | 30.00 | 35.78 | 26.15 | 34.87 | 37.72 | 33.32
2DGS (Ours) | 35.09 | 35.05 | 30.60 | 29.74 | 35.10 | 26.05 | 35.57 | 37.36 | 33.07

Figure 9. Comparison of surface reconstruction using our 2DGS and 3DGS (Kerbl
et al., 2023). Meshes are extracted by applying TSDF to the depth maps. Figure
10. Qualitative studies for the Tanks and Temples dataset (Knapitsch et al.,
2017). Figure 11. Appearance rendering results from reconstructed 2D Gaussian
disks, including DTU, TnT, and Mip-NeRF360 datasets. Figure 12. Illustration
of limitations: Our 2DGS struggles with the accurate reconstruction of semi-
transparent surfaces, for example, the glass shown in example (A). Moreover,
our method tends to create holes in areas with high light intensity, as shown
in (B).


# Machine Translation with Cross-lingual Word Embeddings

Marco Berlot
<EMAIL_ADDRESS>
Evan Kaplan
<EMAIL_ADDRESS>

###### Abstract

Learning word embeddings using distributional information is a task that has
been studied by many researchers, and a lot of studies are reported in the
literature. On the contrary, less studies were done for the case of multiple
languages. The idea is to focus on a single representation for a pair of
languages such that semantically similar words are closer to one another in
the induced representation irrespective of the language. In this way, when
data are missing for a particular language, classifiers from another language
can be used.

##  1 Introduction

Methods for machine translations have been studied for years, and at the same
time algorithms to generate word embeddings are becoming more and more
accurate. Still, there is a lot of research aiming at unifying word embeddings
across multiple languages. In this experience we try a technique for machine
translation that relates word embeddings between two different languages.
Based on the literature we found that it is possible to infer missing
dictionary entries using distributed representations of words and phrases. One
way of doing it is to create a linear mapping between the two vector spaces of
two different languages. In order to achieve this, we first built two
dictionaries of the two different languages. Next, we learned a function that
projects the first vector space to the second one. In this way, we are able to
translate every word belonging to the first language into the second one. Once
we obtain the translated word embedding, we output the most similar word
vector as the translation. The word embeddings were learnt using the Skip Gram
method proposed by (Mikolov et al., 2013a). An example of how the method would
work is reported in figure 1 and figure 2. After creating the word embeddings
from the two dictionaries, we plotted the numbers in the two graphs using PCA.
Figure 3 reports the results after creating a linear mapping between the
embeddings from the two languages. You can see how similar words are closer
together.

Figure 1: PCA visualization of numbers in English Figure 2: PCA visualization
of numbers in French Figure 3: PCA visualization of numbers in French and
English after the linear mapping

##  2 Related Work

In recent years, various models for learning cross-lingual representations
have been proposed. Two main broad categories with some related papers are
identified here:

  * •

Monolingual mapping: In this approach, models are trained using word
embeddings from a monolingual corpora. Then, an objective function is used to
minimize a linear mapping that enable them to map unknown words from the
source language to the target language. 111
Linear projection (Mikolov et al., 2013),Projection via CCA (Faruqui and Dyer,
2014).

  * •

Pseudo-cross-lingual:  In this case we create a pseudo-cross-lingual corpus by
mixing contexts of different languages. We then train an off-the-shelf word
embedding model on the created corpus. Ideally the cross-lingual contexts
allow the learned representations to capture cross-lingual relations.
222Mapping of translations to same representation (Xiao and Guo, 2014),Random
translation replacement (Gouws and Sogaard, 2015)

##  3 Dataset

In the literature, two main types of datasets are used for machine
translation: Word-aligned data and Sentence-aligned data. The first one is
basically a dictionary between the two languages, where there is a direct
relation between same words in different languages. The second one has the
relation between corresponding sentences in the two languages. We decided to
start with the sentence aligned corpus, since it was more interesting to infer
dependency from contexts among words. For our experiment we decided to use the
Europarl dataset, using the data from the WMT11 .The Europarl parallel corpus
is extracted from the proceedings of the European Parliament. It includes
versions in 21 European languages: Romanic (French, Italian, Spanish,
Portuguese, Romanian), Germanic (English, Dutch, German, Danish, Swedish),
Slavik (Bulgarian, Czech, Polish, Slovak, Slovene), Finni-Ugric (Finnish,
Hungarian, Estonian), Baltic (Latvian, Lithuanian), and Greek. For this
experience, we used the English-French parallel corpus, which contains
2,007,723 sentences and the English-Italian corpus, that contains 1,909,115
sentences.

##  4 Linear Mapping

This represents the linear function that maps word embeddings from one
language to another. Given a word x in one language, and the respective word
in the other language z, the following equation reports the objective function
we want to minimize:

$\sum\limits_{i=1}^{n}|Wx_{i}-z_{i}|^{2}$ (1)

Were W represents the final matrix that will contain the values for the
mapping.

##  5 Normalization

Since the word embeddings in a single language are based on the cosine
similarity, we realized that through the objective function reported in
equation 1 we were losing the property of this similarity. As a result we
wanted the dot products to be preserved after the mapping. In order to do that
we normalized the vectors $x_{i}$ and $z_{i}$.The new objective function looks
like:

$\sum\limits_{i=1}^{n}|W\frac{x_{i}}{|x_{i}|}-\frac{z_{i}}{|z_{i}|}|^{2}$ (2)

In order to preserve the dot products after the linear mapping we also had to
constrain W to be an orthogonal matrix. In order to orthogonalize the matrix,
it’s required to solve the following optimization problem:

$\min|W-W^{\prime}|\textrm{ s.t. }W^{\prime}{{W^{\prime}}^{T}}=I$ (3)

One can show that this problem can be solved by taking the singular value
decomposition (SVD) of W and replacing the singular values to ones. This
approximation would only work when the dimensions of the source vector and the
target vector are the same, which is the set up we are working with. So
finally, the method that we tried for this experience uses normalized vectors,
and is reduced by using the cosine similarity function.

##  6 Objective Function

Taking equation 2 we expanded it getting the following equation:

$\sum\limits_{i=1}^{n}|\frac{W{x_{i}}}{|Wx_{i}|}|^{2}+|\frac{z_{i}}{|z_{i}|}|^{2}-2\frac{Wx_{i}}{|Wx_{i}|}^{T}\frac{z_{i}}{|z_{i}|}$
(4)

Which is really easy to show that through some simplifications will result
into:

$\textrm{argmax}\sum\limits_{i=1}^{n}\cos({Wx_{i},z_{i}})$ (5)

Where cos represents the cosine similarity between the two embeddings.

##  7 Setup Description

For this experience we tried monolingual mapping using the Europarl Dataset
and the μtopia parallel corpus 333Microblogs as Parallel Corpora, Wang Ling,
Guang Xiang, Chris Dyer, Alan Black and Isabel Trancoso, ACL 2013. Concerning
the preprocessing, we tokenized the text into single words, and every number
was substituted with a NUM symbol. In addition, all the special characters
were removed. To obtain the dictionaries, we used the words from the English
corpora and translated them into the target languages using the Google
translate API. In this way we built two dictionaries with corresponding words
in the two languages, extracted from the same parallel corpora.

##  8 Methods

###  8.1 Skipgram

It was recently shown that the distributed representations of words capture
surprisingly many linguistic regularities, and that there are many types of
similarities among words that can be expressed as linear translations (Mikolov
et al., 2013c). In the Skip-gram model, the training objective is to learn
word vector representations that are good at predicting its context in the
same sentence (Mikolov et al., 2013a). The objective function that skip gram
tries to minimize is the following:

$\sum\limits_{i=1}^{N}\sum\limits_{y=-k}^{+k}\log P(w_{i+j}|w_{i})$

Where N represents the total number of words in a sentence, P the probability
of a word at the position i+j to belong to the sentence, with respect to the
word i. Ideally, by using this approach we will be able to provide non-trivial
translations that will be related to the context of a word.

###  8.2 Minimizing the loss function

In order to minimize the loss function we decided to setup a neural network
and reduce the objective by using Stochastic Gradient Descent. Figure 4
represents the architecture of our base model. We repeated the same process
both for objective functions expressed in equation 1 and 5. We started by
getting an English word from the English corpus and we got the corresponding
target word by using Google Translate. If this word was contained in the
target corpus, this pair of words was used to train the model; otherwise, it
was ignored. We then get the two embeddings and get the result by multiplying
the first one by the matrix W and then substracting the second embedding. For
this experiment, we used embeddings with a dimension of 100. For this reason W
is a matrix of size 100*100.

Figure 4: Model architecture used to reduce the objective function

##  9 Results

###  9.1 Accuracy

The process for checking the accuracy of the model consisted of first taking a
random subset of the corpus that was not used for training. This subset
consists of slightly more common words because the more common words should
have more accurate word embeddings. Infrequent words (like websites and serial
numbers) would not necessarily have accurate word embeddings that were
generated through the word2vec model. The English word embedding was
multiplied by the transformation matrix to create the predicted translated
word embedding. The cosine similarity was then generated with each translated
word embedding stored in the corpus. The 20 words with the highest cosine
similarity were then outputted. Then the English word was translated through
Google Translate and compared to the 20 outputted translated words. It was
considered a match if the translated word was found in this list of 20 similar
words. The reason why the translated word was not just compared to the most
similar embedding was because each word could have multiple semantic meanings
or have different synonyms in the other language. After running this accuracy
formulation, results for different languages and objective functions are
reported in Table 1. In the table you can see the different results we
obtained across different languages. You can notice that we increased the
number of languages from the first review of our paper. In addition, the table
shows the comparison between our baseline method (Least Squares) and our final
one (Normalized vectors, with cosine similarity), which shows a slight
improvement with respect to the first one.

Language | Least Squares | Normalized Vectors
---|---|---
French | 47.1% | 49.3%
Italian | 36.4% | 39.2%
Spanish | 45% | 47.24%
Chinese | 1.1% | 1.4%
Table 1: Comparison of results using the baseline method (Least Squares) and
the Normalized Vectors

##  10 Error Analysis

###  10.1 Chinese Translations

It’s easy to notice how the performance with the Chinese translations was much
lower with respect to the other languages. One of the reason is that even the
tokenization of the chinese language is not trivial. While using a standard
library, we noticed that some words were tokenized ambiguously. Another
problem was getting the specific translation from Google translate. A lot of
words might have very similar meanings, and Google Translate is not as
accurate as it is with the other languages we have worked with. One reason is
that Chinese is structured in a very different way compared to the other
western languages, which share common roots. For example, a simple word like
”Yes”, does not have a direct translation in Chinese.

###  10.2 Google Translate

We noticed that the model was performing worse than expected. For this reason
we studied what was the main source of error. It was interesting that the
model was able to always predict very close French words in French that had
the same meaning as the English ones. The problem was that they did not
completely match the ones obtained from Google translate. In fact, it resulted
that often, the Google translated word was not the most accurate one. Table 2
reports two of these examples. The first one is represented by the word help.
All the three translations are pretty accurate, especially the first one,
which is the right translation for the verb ”to help”. The problem is the
translation provided by Google, which literally means ”Help Me”. The second
example reports the word ”fire”. In French there are different words to
express the concept of a fire, the concept of an apartment ”on fire” or the
verb ”fire a gun”. All the translations provided by the model represent the
different meanings that the English word has.

This problem was alleviated by translating the 20 outputted French words back
into English and comparing those with the original English word. By doing this
process, the accuracy of the system increased to 47%. We expected the results
of the cosine similarity objective function to produce a higher accuracy, but
in reality, we a achieved slightly worse results.

English Word | Model Translations | Google
---|---|---
help | aider | Aidez-moi
| motiver |
| encourager |
fire | incendie | feu
| pistolet |
| fusil |

##  11 The Code

The code can be found at
https://github.com/MarcoBerlot/Languages_for_Machine_Translation. The
Predictive model file contains all the implementations, from the feature
engineering to the training of the model.

##  12 References

Linear projection (Mikolov et al., 2013)

Lexicon Projection via CCA (Faruqui and Dyer, 2014)

Normalisation and orthogonal transformation (Xing et al., 2015)
Alignment-based projection (Guo et al., 2015)


$\mathcal{B}_{{\mathcal{Q}_{v}}}^{y}$, we compare the probabilities of the
events $S_{1}=G_{v^{\prime}}$ and of $S_{2}=G_{v^{\prime}}$ in a valuation
$v^{\prime}$ sampled from
$\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}$.

For a valuation $v^{\prime}$ let $\mathcal{F}_{v^{\prime}}$ be the family of
sets of size $k$ who are not contained in any of the sets from
$\mathcal{B}_{v^{\prime}}$. Recall that all sets of size $k$ that are in
$\mathcal{F}_{v^{\prime}}$ have the same ex-ante probability of being
$G_{v^{\prime}}$.

Consider some set $S$ of size $k$. The probability that $S=G_{v^{\prime}}$
when the valuation $v^{\prime}$ is sampled from
$\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}$
is $0$ whenever $\mathcal{B}_{{\mathcal{Q}_{v}}}^{y}$ contains a superset of
$S$. Else, none of the supersets of $S$ is in
$\mathcal{B}_{{\mathcal{Q}_{v}}}^{y}$, and by Lemma 5.6, if the number of its
supersets that are in $\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}$ is $\ell$ then
$\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[S=G_{v^{\prime}}]=\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[S\in\mathcal{F}_{v^{\prime}}]\cdot\frac{1}{|\mathcal{F}_{v^{\prime}}|}=\left(1-\frac{1}{m^{2}}\right)^{m-k-\ell}\cdot\frac{1}{|\mathcal{F}_{v^{\prime}}|}$.

For two sets $S_{1}$ and $S_{2}$ that are not contained in any set from
$\mathcal{B}_{{\mathcal{Q}_{v}}}^{y}$, let $\ell_{i}$ be the number of sets in
$\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}$ that contain $S_{i}$. We have that:

$\frac{\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[S_{1}=G_{v^{\prime}}]}{\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[S_{2}=G_{v^{\prime}}]}=\frac{\left(1-\frac{1}{m^{2}}\right)^{m-k-\ell_{1}}}{\left(1-\frac{1}{m^{2}}\right)^{m-k-\ell_{2}}}=\frac{1}{\left(1-\frac{1}{m^{2}}\right)^{(\ell_{1}-\ell_{2})}}\leq
2$

where the last inequality uses the fact that since $0\leq\ell_{1},\ell_{2}\leq
m-k\leq m$, we have that $\ell_{1}-\ell_{2}<m$ and thus the ratio is at most
$\frac{1}{\left(1-\frac{1}{m^{2}}\right)^{m}}$, which approaches to $1$ as $m$
goes to infinity.

Assume by contradiction that for some $S$ we have that
$\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[S=G_{v^{\prime}}]>\frac{2}{\binom{m}{k}-m\cdot
t}$, as the ratio of probabilities for any two sets $S_{1},S_{2}$ to be the
$k$-optimal set is bounded by $2$, the minimal probability for each of the
sets of size $k$ (of positive probability) to be the $k$-optimal set is at
least $\frac{1}{\binom{m}{k}-m\cdot t}$. As there are at least
$\binom{m}{k}-m\cdot t$ sets with positive probability, summing the
probabilities over all of them exceeds $1$, a contradiction.

We have shown that when conditioning on $\mathcal{B}_{{\mathcal{Q}_{v}}}^{y}$
and $\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}$ the probability of a set being the
good set is low. To fully condition on ${\mathcal{Q}_{v}}$ we also need to
condition on $\mathcal{K}_{{\mathcal{Q}_{v}}}$. We show that even after
conditioning on $\mathcal{K}_{{\mathcal{Q}_{v}}}$ the probability for finding
$G_{v}$ is still small.

###### Claim C.5

For $|\mathcal{K}_{{\mathcal{Q}_{v}}}|<1.9^{m}$ and a large enough $m$, if for
any set $S^{\prime}$,
$\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[S^{\prime}=G_{v^{\prime}}]<\frac{1}{1.95^{m}}$,
then for any set $S$, either $S\in{\mathcal{Q}_{v}}$ or
$\Pr_{v^{\prime}\sim\mathcal{D}_{{\mathcal{Q}_{v}}}}[S=G_{v^{\prime}}]<\frac{2}{1.95^{m}}$.
Moreover,
$\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[S\in\mathcal{K}_{{\mathcal{Q}_{v}}}]\leq\frac{2|\mathcal{K}_{{\mathcal{Q}_{v}}}|}{1.95^{m}}$.

Proof:  We prove the claim by induction on
$r=|\mathcal{K}_{{\mathcal{Q}_{v}}}|$. For $r=0$ the claim trivially holds.
Assume the claim holds for any $r^{\prime}<r$, we prove that it holds for $r$.

For a set $S\in\mathcal{K}_{{\mathcal{Q}_{v}}}$, let
$\mathcal{Q}^{S}={\mathcal{Q}_{v}}\setminus\left\\{S\right\\}$. By the
induction hypothesis, the probability that a set $S\in{\mathcal{Q}_{v}}$ is
$G_{v^{\prime}}$ for $v^{\prime}$ sampled from $\mathcal{D}$ conditioned on
the values of sets in $\mathcal{Q}^{S}$, is at most $\frac{2}{1.95^{m}}$.

By the induction hypothesis, since $|\mathcal{Q}^{S}|=r-1$, with probability
at least $1-\frac{2(r-1)}{1.95^{m}}$, none of the sets in $\mathcal{Q}^{S}$ is
$G_{v}$. Together, with probability at least $1-\frac{2r}{1.95^{m}}$, none of
the sets in ${\mathcal{Q}_{v}}$ is $G_{v}$.

Fix any set $T\notin\mathcal{K}_{{\mathcal{Q}_{v}}}$ of size $k$. Let $C$ be
the event that all sets in $\mathcal{K}_{{\mathcal{Q}_{v}}}$ are not
$k$-optimal (which also determines their exact values). By Bayes’ theorem,
conditioned on $C$ we have that

$\Pr_{v^{\prime}\sim\mathcal{D}_{{\mathcal{Q}_{v}}}}[T=G_{v^{\prime}}|C]=\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[T=G_{v^{\prime}}|C]=\frac{\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[T=G_{v^{\prime}}]}{\Pr_{v^{\prime}\sim\mathcal{D}^{\mathcal{B}_{{\mathcal{Q}_{v}}}^{y},\mathcal{B}_{{\mathcal{Q}_{v}}}^{n}}}[C]}<\frac{\frac{1}{1.95^{m}}}{1-\frac{2r}{1.95^{m}}}<\frac{2}{1.95^{m}}$

where the last inequality holds for $r<1.9^{m}$ and a large enough $m$.

For $t<1.9^{m}$, $k=\frac{m}{2}$ and a large enough $m$, by Claim C.4, we have
that
$\Pr_{v^{\prime}\sim\mathcal{D}_{{\mathcal{Q}_{v}}}}[S=G_{v^{\prime}}]<\frac{2}{\binom{m}{k}-mt}<\frac{2}{1.95^{m}}$
and the conditions of Claim C.5 hold.

Considering the set $S$ returned by the algorithm as a $t+1$’th query, the
probability that either $G_{v}\in{\mathcal{Q}_{v}}$ or $S=G_{v}$ is at most
$\frac{2(t+1)}{1.95^{m}}$ with probability over $\mathcal{D}$. For large
enough $m$, we have that $\frac{2(t+1)}{1.95^{m}}<\frac{t}{1.9^{m}}$.