# Sp2360: Sparse-view 360∘ Scene Reconstruction
using Cascaded 2D Diffusion Priors

Soumava Paul1, 2, Christopher Wewer1, Bernt Schiele1, Jan Eric Lenssen1
1 Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
2Saarland University, Saarland Informatics Campus, Germany
<EMAIL_ADDRESS>{cwewer, schiele<EMAIL_ADDRESS>
###### Abstract

We aim to tackle sparse-view reconstruction of a $360^{\circ}$ 3D scene using
priors from latent diffusion models (LDM). The sparse-view setting is ill-
posed and underconstrained, especially for scenes where the camera rotates
$360$ degrees around a point, as no visual information is available beyond
some frontal views focused on the central object(s) of interest. In this work,
we show that pretrained 2D diffusion models can strongly improve the
reconstruction of a scene with low-cost fine-tuning. Specifically, we present
_SparseSplat360 (Sp 2360)_, a method that employs a cascade of in-painting and
artifact removal models to fill in missing details and clean novel views. Due
to superior training and rendering speeds, we use an explicit scene
representation in the form of 3D Gaussians over NeRF-based implicit
representations. We propose an iterative update strategy to fuse generated
pseudo novel views with existing 3D Gaussians fitted to the initial sparse
inputs. As a result, we obtain a multi-view consistent scene representation
with details coherent with the observed inputs. Our evaluation on the
challenging Mip-NeRF360 dataset shows that our proposed 2D to 3D distillation
algorithm considerably improves the performance of a regularized version of
3DGS adapted to a sparse-view setting and outperforms existing sparse-view
reconstruction methods in $360^{\circ}$ scene reconstruction. Qualitatively,
our method generates entire $360^{\circ}$ scenes from as few as $9$ input
views, with a high degree of foreground and background detail.

##  1 Introduction

Obtaining high-quality 3D reconstructions or novel views from a set of images
has been a long-standing goal in computer vision and has received increased
interest recently. Recent 3D reconstruction methods, such as those based on
Neural Radiance Fields (NeRF) [40], Signed Distance Functions (SDFs) [66], or
the explicit 3D Gaussian Splatting (3DGS) [31], are now able to produce
photorealistic novel views of $360^{\circ}$ scenes. However, in doing so, they
rely on hundreds of input images that densely capture the underlying scene.
This requirement is both time-consuming and often an unrealistic assumption
for complex, large-scale scenes. Ideally, one would like a 3D reconstruction
pipeline to offer generalization to unobserved parts of the scene and be able
to successfully reconstruct areas that are only observed a few times. In this
work, we present a method to efficiently obtain high-quality 3D Gaussian
representations from just a few views, moving towards this goal.

Standard 3DGS, much like NeRF, is crippled in a sparse observation setting. In
the absence of sufficient observations and global geometric cues, 3DGS
invariably overfits to training views. This leads to severe artifacts and
background collapse already in nearby novel views due to the inherent depth
ambiguities associated with inferring 3D structure from few-view 2D images.
There exists a long line of work to improve performance of both NeRF and 3DGS
in sparse novel view synthesis [14, 50, 33, 73, 80, 11, 64, 28, 44, 72]. While
these works can reduce artifacts in sparsely observed regions, they are not
able to fill in missing details due to the use of simple regularizations or
weak priors.

In the setting of large-scale $360^{\circ}$ scenes, the problem is even more
ill-posed and under-constrained. Much stronger priors are needed, such as
those from large pre-trained 2D diffusion models [43, 47, 53, 51], capturing
knowledge about typical structures in the 3D world. Recent approaches [36, 54,
71] add view-conditioning to these image generators by fine-tuning them on a
large mixture of real-world and synthetic multi-view datasets. Leveraging
these strong priors for optimization of radiance fields yields realistic
reconstructions in unobserved areas of challenging $360^{\circ}$ scenes. In
this work, we propose to forego augmenting a 2D diffusion model with
additional channels for pose or context to make it 3D-aware. Instead, we
perform low-cost fine-tuning of pretrained models to adapt them to specific
sub-tasks of few-view reconstruction. This weakens the assumption of large-
scale 3D training data, which is expensive to obtain.

We present SparseSplat360 (Sp2360), an efficient method that addresses the
given task of sparse 3D reconstruction by iteratively adding synthesized views
to the training set of the 3D representation. The generation of new training
views is divided and conquered as the 2D sub-tasks of (1) _in-painting missing
areas_ and (2) _artifact elimination_. The in-painting model for the first
stage is pre-trained on large 2D datasets and efficiently fine-tuned on the
sparse views of the given scene. The artifact elimination network for the
second stage is an image-to-image diffusion model, fine-tuned to specialize on
removing typical artifacts appearing in sparse 3DGS. Thus, both stages utilize
2D diffusion models that are fine-tuned on small amounts of data. In each
iteration, the models are conditioned on rendered novel views of the already
existing scene, thus using 2D images as input and output, avoiding the
requirement of training on large datasets of 3D scenes. In contrast to
previous works, our method leverages stronger priors than simple regularizers
and does not rely on million-scale multi-view data or huge compute resources
to train a 3D-aware diffusion model.

In summary, our contributions are:

  * •

We present a novel systematic approach to perform sparse 3D reconstruction of
$360^{\circ}$ scenes by autoregressively adding generated novel views to the
training set.

  * •

We introduce a two-step approach for generating novel training views by
performing _in-painting_ and _artifact removal_ with 2D diffusion models,
which avoids fine-tuning on large-scale 3D data.

  * •

We introduce a sparse 3DGS baseline that improves reconstruction from sparse
observation by applying regularization techniques without the need for pre-
trained models.

  * •

We show that Sp2360 outperforms recent works in reconstructing large 3D scenes
from as few as $9$ input views.

##  2 Related Work

###  2.1 Sparse-View Radiance Fields

Following the breakthroughs of NeRF [40] and 3D Gaussian Splatting [31] for
inverse rendering of radiance fields, there have been many approaches to
weaken the requirement of dense scene captures to sparse input views only.
These methods can be categorized into regularization techniques and
generalizable reconstruction priors.

#### Regularization Techniques

Fitting a 3D representation from sparse observations only is an ill-posed
problem and very prone to local minima. In the case of radiance fields, this
is typically visible as ’floaters’ during rendering of novel views. A
classical technique for training with limited data is regularization. Many
existing methods leverage depth from Structure-from-Motion [14, 50], monocular
estimation [33, 73, 80, 11], or RGB-D sensors [64]. DietNeRF [28] proposes a
semantic consistency loss based on CLIP [46] features. FreeNeRF [74]
regularizes frequency range of NeRF inputs by increasing the frequencies of
positional-encoding features in a coarse-to-fine manner. Moving closer to
generative priors, RegNeRF [44] and DiffusioNeRF [72] maximize likelihoods of
rendered patches under a trained normalizing flow or diffusion model,
respectively.

#### Generalizable Reconstruction

In the case of very few or even a single view only, regularization techniques
are usually not strong enough to account for the ambiguity in reconstruction.
Therefore, another line of research focuses on training priors for novel view
synthesis across many scenes. pixelNeRF [75] extracts pixel-aligned CNN
features from input images at projected sample points during volume rendering
as conditioning for a shared NeRF MLP. Similarly, many approaches [63, 7, 22,
35] define different NeRF conditionings on 2D or fused 3D features. Following
the trend of leveraging explicit data structures for accelerating NeRFs,
further priors have been learned on triplanes [27], voxel grids [19], and
neural points [68]. Building upon the success of 3D Gaussian Splatting [31]
and its broad applications to, e.g., surface [18, 26] or non-rigid
reconstruction [37, 70, 12], recent methods like pixelSplat [6] and MVSplat
[10] achieve state-of-the-art performance in stereo view interpolation while
enabling real-time rendering. However, all of these works train regression
models that infer blurry novel views in case of high uncertainty. Bridging the
gap of generalizable and generative priors, GeNVS [5] and latentSplat [69]
render view-conditioned feature fields followed by a 2D generative decoding to
obtain a novel view.

###  2.2 Generative Priors

In case of ambiguous novel views, the expectation over all possible
reconstructions might itself not be a reasonable prediction. Therefore,
regression-based approaches fail. Generative methods, on the other hand, try
to sample from this possibly multi-modal distribution.

#### Diffusion Models

In recent years, diffusion models [24, 15] emerged as the state-of-the-art for
image synthesis. They are characterized by a pre-defined forward noising
process that gradually destroys data by adding random (typically Gaussian)
noise. The objective is to learn a reverse denoising process with a neural
network that, after training, can sample from the data distribution given pure
noise. Important improvements include refined sampling procedures [60, 30] and
the more efficient application in a spatially compressed latent space compared
to the high-resolution pixel space [51]. Their stable optimization, in
contrast to GANs, enabled today’s text-to-image generators [43, 47, 53, 51]
trained on billions of images [58].

#### 2D Diffusion for 3D

While diffusion models have been applied directly on 3D representations like
triplanes [59, 8], voxel grids [41], or (neural) point clouds [77, 38, 57], 3D
data is scarce. Given the success of large-scale diffusion models for image
synthesis, there is a great research interest in leveraging them as priors for
3D reconstruction and generation. DreamFusion [45] and follow-ups [65, 34, 9,
13, 61] employ score distillation sampling (SDS) to iteratively maximize the
likelihood of radiance field renderings under a conditional 2D diffusion
prior. For sparse-view reconstruction, existing approaches incorporate view-
conditioning via epipolar feature transform [79], cross-attention to encoded
relative poses [36, 54], or pixelNeRF [75] feature renderings [71]. However,
this fine-tuning is expensive and requires large-scale multi-view data, which
we circumvent with Sp2360.

##  3 Method

In this section, we describe our method in detail. The section begins with a
general overview of Sp2360 in Sec. 3.1, outlining the autoregressive algorithm
for training view generation. In the following sections, the individual parts
of the system are introduced: the in-painting module in Sec. 3.2, the artifact
removal procedure in Sec. 3.3, and the sparse 3DGS baseline method in Sec.
3.4.

###  3.1 Sp2360 for Sparse-View 3D Reconstruction

Given a sparse set of input images
$\mathcal{I}=\\{\mathbf{I}_{1},\mathbf{I}_{2},...,\mathbf{I}_{M}\\}$ with
camera poses $\\{\pi_{1},\pi_{2},...,\pi_{M}\\}$, and a sparse point cloud
$\mathbf{P}\in\mathbb{R}^{S\times 3}$, estimated by Structure-from-Motion
(SfM) [55, 56], our goal is to obtain a 3D Gaussian representation of the
scene, which enables the rendering of novel views from camera perspectives
that are largely different from given views in $\mathcal{I}$. The tackled
scenario of sparse-view inputs is extremely challenging, as the given
optimization problem is heavily under-constrained and leads to severe
artifacts if done naively.

Figure 1: Overview of Sp2360. We render 3D Gaussians fitted to our sparse set
of $M$ views from a novel viewpoint. The image has missing regions and
Gaussian artifacts, which are fixed by a combination of in-painting and
denoising diffusion models. This then acts as pseudo ground truth to spawn and
update 3D Gaussians and satisfy the new view constraints. This process is
repeated for several novel views spanning the $360^{\circ}$ scene until the
representation becomes multi-view consistent.  Algorithm 1 Sp2360 Algorithm

1:Sparse input image set $\mathcal{I}$, camera poses
$\\{\pi_{1},\pi_{2},...,\pi_{M}\\}$, sparse point cloud
$\mathbf{P}\in\mathbb{R}^{S\times 3}$

2:Set of 3D Gaussians $\mathcal{G}$

3:$\hat{\mathcal{I}}\leftarrow\mathcal{I}$

4:$\mathcal{G}\leftarrow$ Optimize _Sparse 3DGS_ for $k$ iterations

5:for $N$ iterations do

6: $\pi\leftarrow$ Sample novel camera pose.

7: $\mathbf{I}\leftarrow R_{\pi}(\mathcal{G})$ \- Render from camera $\pi$

8: $\mathbf{I}\leftarrow$ In-paint$(\mathbf{I})$

9: $\mathbf{I}\leftarrow$ ArtifactRemoval$(\mathbf{I})$

10: $\hat{\mathbf{I}}\leftarrow\ \hat{\mathbf{I}}\cup\\{\mathbf{I}\\}$

11: $\mathcal{G}\leftarrow$ Optimize _Sparse 3DGS_ for $k$ iterations

12:end for

An overview of Sp2360 is given in Fig. 1 and Alg. 1. The approach begins by
optimizing a set of 3D Gaussians [31] to reconstruct the initial sparse set of
input images $\hat{\mathcal{I}}=\mathcal{I}$. For this, we introduce a _Sparse
3DGS_ baseline (c.f. Sec. 3.4), which combines best practices from previous
works on NeRFs. The obtained representation serves as initial prior for
$360^{\circ}$ reconstruction. Next, we autoregressively add new generated
views to our training set: (1) we sample novel cameras and render novel views
with artifacts and missing areas, make them look plausible by (2) performing
in-painting (c.f. 3.2) and then (3) artifact removal (c.f. 3.3), before (4)
adding them to set $\hat{\mathcal{I}}$ and continuing optimizing our 3D
representation for $k$ iterations.

For training 3DGS in an iterative fashion, as outlined above, special
precaution has to be taken to prevent overfitting on the initial views. An
optimal schedule involves finding the number of iterations per cycle $k$ and
the hyperparameters for the original 3DGS [31]. We provide a detailed
evaluation in App. A.4. In the following, we will detail the in-painting and
artifact removal stages of our pipeline.

###  3.2 In-Painting Novel Views

When using only a sparse observation set, many areas remain unobserved,
leading to areas of no Gaussians in the 3D representation and zero opacity
regions in some novel views (c.f. Fig.1). Regularization techniques cannot
help with inferring details. Instead, we incorporate a generative in-painting
diffusion model to fill such regions indicated by a binary mask $\phi$, which
is obtained by rendering opacity from our current Gaussian representation
$\mathcal{G}$ and binarizing it with a threshold $\tau$.

We fine-tune Stable Diffusion $2$ [51] to perform in-painting on our novel
view renderings. The current training images $\hat{\mathcal{I}}$ are used as
training data to adapt it to the current scene. For the fine-tuning technique,
we get inspired by recent work [62] that uses LoRA [25] adapters for the UNet
$\boldsymbol{\epsilon_{\theta}}$ and text encoder $c_{\theta}(y)$. Given
images $\hat{\mathcal{I}}=\\{\mathbf{I}_{1},...,\mathbf{I}_{M}\\}$, we create
artificial masks $\\{\phi_{1},...,\phi_{M}\\}$ by creating random rectangular
masks over each image and taking either their union or the complement of the
union. Then, the adapter weights are fine-tuned using the following objective:

$\displaystyle\mathcal{L}=\mathbb{E}_{i\sim\mathcal{U}(M),y,\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{1}),t}\Big{[}\|\epsilon_{t}-\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,\phi_{i},c_{\theta}(y))\|_{2}^{2}\Big{]}\textnormal{,}$
(1)

where $\mathbf{z}_{t}$ is the diffused latent encoding of image
$\mathbf{I}_{i}$ at step $t$. Here, $y$ is a simple text prompt - “A photo of
[V]”, where [V] is a rare token like in DreamBooth [52] whose embedding is
optimized for in-painting. For views $\mathbf{I}\notin\mathcal{I}_{i}$ (not
from the original set), Eq 1 is only evaluated for regions that have a
rendered opacity $>\tau$. This in-painting objective for fine-tuning enables
our model to in-paint missing regions in a novel view rendering with details
faithful to the observed $M$ views. At inference,
$\boldsymbol{\epsilon_{\theta}}$ predicts the noise in $\mathbf{z}_{t}$ as:

$\displaystyle\hat{\epsilon_{t}}=\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,\phi_{i},c_{\theta}(y)),$
(2)

which is used to progressively obtain less noisy latents in $s$ DDIM [60]
sampling steps starting from $t$. After passing the denoised latent
$\hat{z}_{0}$ through a VAE decoder $\mathcal{D}$. we obtain the in-painted
image $x_{0}$.

###  3.3 Removing Sparse-View Artifacts

The in-painting technique can fill in plausible details in low-opacity areas
of novel view renders. However, it cannot deal with the dominance of blur,
floaters, and color artifacts, which are not detected by the in-painting mask
$\phi$. As such, we resort to a diffusion-based approach to learn how typical
artifacts in 3D Gaussian representations from sparse views look like and how
to remove them.

We fine-tune an image-conditioned diffusion model [4] to edit images based on
short, user-friendly edit instructions. Here, the UNet
$\boldsymbol{\epsilon_{\theta}}$ is trained to predict noise in $z_{t}$
conditioned on a given image $c$, in addition to the usual text conditioning
$c_{\theta}(y)$. To enable this, additional input channels are added to the
first convolutional layer of $\boldsymbol{\epsilon_{\theta}}$ so that $z_{t}$
and $\mathcal{E}(c)$ can be concatenated. Weights of these channels are
initialized to zero, whereas rest of the model is initialized from the pre-
trained Stable Diffusion v1.5 checkpoint. For architectural design choices, we
resort to those made by Instruct-Pix2Pix [4].

#### Dataset Creation

For training, we rely on a set
$\mathcal{X}=\\{(\mathbf{x}^{i},\mathbf{c}^{i},y^{i})_{i=1}^{N}\\}$ of data
triplets, each containing a clean image $\mathbf{x}^{i}$, an image with
artifacts $\mathbf{c}^{i}$, and the corresponding edit instruction for fine-
tuning $y^{i}$, to “teach” a diffusion model how to detect Gaussian artifacts
and generate a clean version of the conditioning image. For this, we build an
_artifact simulation engine_ comprising a 3DGS model fitted to dense views,
_Sparse 3DGS_ fitted to few views, and camera interpolation and perturbation
modules to use supervision of the dense model at viewpoints beyond ground
truth camera poses. The fine-tuning setup is illustrated in Fig. 2. For a
given scene, we fit sparse models for $M\in\\{3,6,9,18\\}$ number of views.
For larger $M$, we observe that there are very few artifacts beyond standard
Gaussian blur. To have diversity in edit instructions, we start with a base
instruction - “Denoise the noisy image and remove all floaters and Gaussian
artifacts.” and ask GPT-4 to generate 50 synonymous instructions. During
training, each clean, artifact image pair is randomly combined with one of
these 51 instructions.

Figure 2: Artifact removal fine-tuning. Pairs of clean images and images with
artifacts are obtained from 3DGS fitted to sparse and dense observations,
respectively, across $36$ scenes. These are combined with one of 51 synonymous
prompts generated by GPT-4 [1] from a base instruction. SD v1.5 [51] is then
fine-tuned with a dataset of $10.5K$ samples for the Gaussian artifact removal
task.

#### Training the Artifact Removal Module

Using our synthetically curated dataset $\mathcal{X}$, we fine-tune SD v1.5 as
follows:

$\displaystyle\mathcal{L}=\mathbb{E}_{i\sim\mathcal{U}(N),\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t}\Big{[}\|\epsilon_{t}-\boldsymbol{\epsilon_{\theta}}(\mathbf{z}^{i}_{t};t,\mathcal{E}(\mathbf{c}^{i}),c_{\theta}(y^{i}))\|_{2}^{2}\Big{]}$
(3)

where $\mathbf{z}_{t}^{i}$ is the encoded image $\mathbf{x}^{i}$ diffused with
sampled noise $\epsilon$ at time step $t$. Thus, the model is fine-tuned to
generate clean images $\mathbf{x}^{i}$, conditioned on artifact images
$\mathbf{c}^{i}$ and text prompts $y^{i}$.

#### Generating Clean Renders

Given an in-painted rendering $\mathbf{x}_{0}$ and the base prompt
$y=$"Denoise the noisy image and remove all floaters and Gaussian artifacts"
at inference time, the fine-tuned artifact removal UNet
$\boldsymbol{\epsilon_{\theta}}$ predicts the noise in latent $\mathbf{z}_{t}$
according to $t\sim\mathcal{U}[t_{min},t_{max}]$ as:

$\displaystyle\hat{\epsilon}_{t}=$
$\displaystyle\,\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,\varnothing,\varnothing)$
(4)
$\displaystyle+s_{I}\cdot(\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,\mathcal{E}(\mathbf{x}_{0}),\varnothing)-\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,\varnothing,\varnothing))$
$\displaystyle+s_{T}\cdot(\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,\varnothing,c_{\theta}(y))-\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,\mathcal{E}(\mathbf{x}_{0}),\varnothing))$

where $s_{I}$ and $s_{T}$ are the image and prompt guidance scales, dictating
how strongly the final multistep reconstruction agrees with the in-painted
render $\mathbf{x}_{0}$ and the edit prompt $y$, respectively. After $s$ DDIM
[60] sampling steps, we obtain our final image by decoding the denoised
latent.

###  3.4 Sparse 3DGS

The sparse 3DGS baseline serves as our starting point that already improves
reconstruction quality over standard 3DGS. It is inspired by several recent
works on sparse neural fields [14, 44] and 3DGS [73, 80, 33]. The baseline
unifies depth regularization from monocular depth estimators and depth priors
from pseudo views, while using specific hyperparameter settings, such as
densification thresholds and opacity reset configurations for Gaussian
splatting. It is outlined in detail in App. A.2 and evaluated individually in
the ablation studies in Sec. 4.4.

##  4 Experiments

This section compares _Sp 2360_ with state-of-the-art sparse-view
reconstruction techniques. We also provide detailed ablation studies
motivating specific design choices across different components of our
approach. The setup for _Iterative 3DGS_ (c.f. A.4) translates seamlessly to
the iterative distillation procedure (c.f. A.5) with diffusion priors, and
hence, we refrain from further analysis here.

###  4.1 Experimental Setup

#### Evaluation Dataset

We evaluate _Sp 2360_ on the 9 scenes of the MipNeRF360 dataset [2],
comprising 5 outdoor and 4 indoor scenes. Each scene has a central object or
area of complex geometry with an equally intricate background. This makes it
the most challenging $360^{\circ}$ dataset compared to CO3D [49], RealEstate
10K [78], DTU [29], etc. We retain the train/test split of the MipNeRF360
dataset, where every $8th$ image is kept aside for evaluation. To create
$M$-view subsets, we sample from the train set of each scene using a geodesic
distance-based heuristic to encourage maximum possible scene coverage (see
supplement for details).

#### Fine-tuning dataset

We fine-tune the in-painting module only on the $M$ input views. For the
artifact removal module, we train 3DGS on sparse and dense subsets of
$360^{\circ}$ scenes from MipNeRF360 [2], Tanks and Temples [32] and Deep
Blending [21] across a total of 36 scenes to obtain $\sim 10.5K$ data
triplets. We train 9 separate artifact removal modules holding out the
MipNeRF360 scene we want to reconstruct. On a single A100 GPU, fine-tuning the
in-painting and artifact removal modules takes roughly $2$h and $1$h,
respectively.

#### Baselines

We compare our approach against $7$ baselines. FreeNeRF [74], RegNeRF [44],
DiffusioNeRF [72], and DNGaussian [33] are few-view regularization methods
based on NeRFs or 3D Gaussians. ZeroNVS [54] is a recent generative approach
for reconstructing a complete 3D scene from a single image. We use the
ZeroNVS∗ baseline introduced in ReconFusion [71], designed to adapt ZeroNVS to
multi-view inputs. Conditioning the diffusion model on the input view closest
to the sampled random view enables scene reconstruction for a general $M$-view
setting. We also compare against 3DGS, the reconstruction pipeline for Sp2360,
and against Sparse 3DGS, our self-created baseline. We cannot compare with
ReconFusion [71] as their code is not publicly available, and their method
builds on top of closed-source diffusion models. We expect to perform slightly
worse due to weaker priors, but our approach is more lightweight and data-
efficient.

#### Metrics

Due to the generative nature of our approach, we employ FID [23] and KID [3]
to measure similarity of distribution of reconstructed novel views and ground
truth images. We also compute two perceptual metrics - LPIPS [76] and DISTS
[16] \- to measure similarity in image structure and texture in the feature
space. Despite their known drawbacks as evaluators of generative techniques
[5, 54], we additionally provide PSNR and SSIM scores for completeness. Both
favor pixel-aligned blurry estimates over high-frequency details, making them
ill-suited to our setting.

###  4.2 Implementation Details

We implement our entire framework in Pytorch 1.12.1 and run all experiments on
single $A100$ or $A40$ GPUs. We work with image resolutions in the 400-600
pixel range as this is closest to the output resolution of $512$ for both
diffusion models. We set $\lambda_{1}=0.2$ (same as 3DGS) and vary
$\lambda_{depth},\lambda_{pseudo}$ in $\\{0.0,0.05,0.1\\}$ for _Sparse 3DGS_.
For in-painting, we set $\tau=0.8$ and fine-tune the in-painting module at
$512\times 512$ resolution for 3000-5000 steps with LoRA modules of rank in
$\\{8,16,32\\}$ (depending on $M$). We use a batch size of $16$ and learning
rates of 2e-4 for the UNet and 4e-5 for the text encoder. The artifact removal
module is trained at $256\times 256$ resolution for $2500$ iterations with
batch size $16$ and learning rate 1e-4. To enable classifier-free guidance for
both models, we randomly dropout conditioning inputs (text, mask, image, etc.)
with probability of $0.1$ during training. The classifier-free guidance scales
are set to $s_{I}=2.5$ and $s_{T}=7.0$. We use $t_{max}=0.99$ for both in-
painting and artifact removal and linearly decrease $t_{min}$ for the in-
painting module from $0.98$ to $0.90$, and from $0.98$ to $0.70$ for the
artifact removal module. We sample both the in-painted and clean renders for
$s=20$ DDIM sampling steps. We also linearly decay the weight of
$\mathcal{L}_{sample}$ (c.f. A.5) from $1$ to $0.1$ over $30k$ iterations.

###  4.3 Comparison Results

We evaluate all baselines on our proposed splits for each scene. We report
averaged quantitative results in Tables 1 and 2 and compare novel view
rendering quality in Fig 3. We outperform all baselines for $9$-view
reconstruction across all metrics and are second only to DiffusioNeRF on FID,
KID, and DISTS for $3$ and $6$-view reconstruction. Unlike the baselines, our
approach consistently improves with increasing $M$ across all metrics.

|  3DGS |  RegNeRF |  DNGaussian |  DiffusioNeRF |  ZeroNVS* |  Ours |  Ground Truth
---|---|---|---|---|---|---|---
Bicycle (9) |  |  |  |  |  |  |
Treehill (9) |  |  |  |  |  |  |
Garden (9) |  |  |  |  |  |  |
Stump (9) |  |  |  |  |  |  |
Kitchen (9) |  |  |  |  |  |  |
Flowers (9) |  |  |  |  |  |  |
Bonsai (6) |  |  |  |  |  |  |
Figure 3: Qualitative comparison of _Sp 2360_ with few-view methods. Our
approach consistently fairs better in recovering image structure from foggy
geometry, where baselines typically struggle with “floaters” and color
artifacts. We encourage the reader to refer to our supplemental 360° video,
where the benefits of our method can be observed along a smooth trajectory.
Table 1: Quantitative comparison with state-of-the-art sparse-view
reconstruction techniques on classical metrics. Despite being a generative
solution, we outperform all baselines across all view splits on both pixel-
aligned and perceptual metrics.

|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Method | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view
3DGS | 10.288 | 11.628 | 12.658 | 0.102 | 0.141 | 0.190 | 0.709 | 0.661 | 0.605
FreeNeRF | 9.948 | 9.599 | 10.641 | 0.124 | 0.129 | 0.145 | 0.682 | 0.679 | 0.668
RegNeRF | 11.030 | 10.764 | 11.020 | 0.117 | 0.134 | 0.145 | 0.663 | 0.660 | 0.661
DNGaussian | 9.867 | 10.671 | 11.275 | 0.124 | 0.137 | 0.179 | 0.754 | 0.730 | 0.729
DiffusioNeRF | 10.749 | 11.728 | 11.430 | 0.093 | 0.116 | 0.112 | 0.709 | 0.678 | 0.654
ZeroNVS* | 10.406 | 9.990 | 9.719 | 0.079 | 0.079 | 0.082 | 0.709 | 0.711 | 0.700
Ours | 12.927 | 13.701 | 14.121 | 0.211 | 0.231 | 0.261 | 0.647 | 0.622 | 0.591

Table 2: Quantitative comparison with few-view reconstruction techniques on
metrics suited for generative reconstruction. We are second only to
DiffusioNeRF for $3$ and $6$-view reconstruction but achieve higher scores for
$9$ views.

|  FID $\downarrow$ |  KID $\downarrow$ |  DISTS $\downarrow$
---|---|---|---
Method | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view
3DGS | 392.620 | 343.336 | 292.324 | 0.313 | 0.268 | 0.229 | 0.476 | 0.429 | 0.321
FreeNeRF | 347.625 | 343.833 | 342.917 | 0.254 | 0.249 | 0.258 | 0.392 | 0.388 | 0.388
RegNeRF | 362.856 | 347.045 | 349.043 | 0.291 | 0.266 | 0.247 | 0.399 | 0.403 | 0.404
DNGaussian | 431.687 | 420.110 | 414.307 | 0.311 | 0.285 | 0.281 | 0.581 | 0.571 | 0.544
DiffusioNeRF | 273.096 | 225.661 | 290.184 | 0.158 | 0.104 | 0.183 | 0.362 | 0.319 | 0.378
ZeroNVS* | 351.090 | 335.155 | 337.457 | 0.283 | 0.282 | 0.290 | 0.437 | 0.429 | 0.428
Ours | 318.470 | 283.504 | 230.565 | 0.273 | 0.229 | 0.162 | 0.384 | 0.357 | 0.315

###  4.4 Ablation Studies

|  Sparse 3DGS |  w/o Artifact R. |  w/o In-painting |  w/o Schedule |  3DGS Loss |  Ours |  Ground Truth
---|---|---|---|---|---|---|---
Render 1 |  |  |  |  |  |  |
Render 2 |  |  |  |  |  |  |
Figure 4: Ablation Study on $9$-view reconstruction of _garden_ scene. Our
fine-tuned artifact removal module and iterative schedule contribute the most
toward quality of the final reconstruction. 3D Gaussians from _Sparse 3DGS_
act as suitable geometric prior in the absence of explicit view conditioning.

In Tab LABEL:tab:ablation and Fig 4, we ablate the relative importance of each
component towards $360^{\circ}$ reconstruction. We pick the garden scene and
its $9$ view split for this study. We first show benefits of the
regularization heuristics in _Sparse 3DGS_ over native 3DGS. We attempt
reconstruction using either the in-painting or artifact removal module and
show that their combination works best for both novel view synthesis and final
reconstruction. Interestingly, leaving out the artifact removal module results
in the best PSNR and SSIM scores across all variants, emphasizing that
classical metrics reward blurry reconstructions, whereas FID and KID favor
sharp, realistic details in novel views. We remove the iterative update
formulation and add in-painted clean renders for all novel views
simultaneously. This variant performs worse than _Sparse 3DGS_ , highlighting
the need for autoregressive scene generation in our setting. Additionally, we
try to supervise the renderings at novel views using the original 3DGS
objective. However, this performs slightly worse than Eq 12. Note that an SDS-
based formulation [54] is not applicable here due to the two-step nature of
generative view synthesis.

Table 3: Ablation study on the $9$ view split of _garden_ scene. Our combination of diffusion priors complements each other effectively. Without an iterative schedule to fuse novel views, our method fairs worse than a regularized baseline. Using the 3DGS objective for novel view renders leads to slightly worse performance on FID and KID. Method | FID $\downarrow$ | KID $\downarrow$ | DISTS $\downarrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---|---|---|---
3DGS | 223.594 | 0.146 | 0.247 | 0.502 | 14.470 | 0.287
Sparse 3DGS | 200.703 | 0.127 | 0.247 | 0.522 | 16.589 | 0.367
Sp2360 w/o Artifact R. | 209.477 | 0.120 | 0.253 | 0.528 | 16.850 | 0.380
Sp2360 w/o In-painting | 151.098 | 0.071 | 0.230 | 0.524 | 15.121 | 0.299
Sp2360 w/o Schedule | 214.674 | 0.111 | 0.277 | 0.576 | 14.132 | 0.292
Sp2360 w/ $\mathcal{L}_{D-SSIM}$ | 133.875 | 0.051 | 0.225 | 0.502 | 15.677 | 0.326
Sp2360 | 124.768 | 0.048 | 0.224 | 0.504 | 15.759 | 0.326

###  4.5 Scaling to More Views

In Fig 5, we analyze the relevance of diffusion priors with increasing $M$. We
evaluate Sp2360 and 3DGS on view splits of increasing size with
$M\in\\{3,6,9,18,27,54,81\\}$. For $M\leq 27$, our method consistently
improves 3DGS’ generalization at novel views. We observe that as ambiguities
resolve with increasing scene coverage, diffusion-based regularization becomes
less important, and for $M\geq 54$, our method performs either equally or
slightly worse across the 6 performance measures.

|  |  |  |  |
---|---|---|---|---|---
Figure 5: Scalability of Sp2360 with input views. Our combination of fine-
tuned diffusion priors improves performance of 3DGS up to $27$ input views of
the _bicycle_ scene, alleviating the need for dense captures.

###  4.6 Efficiency

Our approach focuses on limited use of multi-view data, minimal training
times, and fast inference. We fit 3D Gaussians to our initial $M$ inputs
inside $1$ hr and fine-tune the in-painting module in $\sim 2$ hrs. Sp2360
trains inside $30$ mins on a single A100 GPU, significantly faster than all
baselines with NeRF backbones. FreeNeRF takes $\sim 1$ day while RegNeRF takes
$>2$ days to train on a single A40 GPU. Both ZeroNVS* and DiffusioNeRF require
$\sim 3$ hrs to distill 2D diffusion priors into Instant-NGP [42], whereas
ReconFusion trains in 1 hour on 8 A100 GPUs 111Source: ReconFusion authors.
All 3 approaches use custom diffusion models that take several days to train
on large-scale 3D datasets. On the contrary, we only need _10.5K_ samples to
obtain a generalized artifact removal module that efficiently eliminates
Gaussian artifacts and recovers image details in the foreground and
background. We currently use a leave-one-out mechanism for the $9$ scenes in
MipNeRF360 to train the artifact removal module for a particular MipNeRF360
scene. However, we only do this to have enough diverse data pairs across the
$3$ datasets. This requirement can be easily alleviated using multi-view
training data from other sources. Given the availability of a generalized
artifact removal module, our entire pipeline can reconstruct a 360 scene in
under $3.5$ hrs on a single A100 GPU. Courtesy of our 3D representation, we
retain the real-time rendering capabilities of 3DGS.

##  5 Limitations & Conclusion

#### Limitations

Despite being a low-cost and efficient approach for reconstructing complex
$360^{\circ}$ scenes from a few views, Sp2360 has certain limitations. Our
approach is limited by the sparse geometry prior from an SfM point cloud,
estimated from few views. For example, the point cloud for 9 views of the
_bicycle_ scene only has $628$ points. This prevents our method from achieving
even higher fidelity and restricting artifacts in distant, ambiguous novel
views. DUSt3R [67] is a recent stereo reconstruction pipeline that can
potentially provide a stronger geometry prior.

#### Conclusion

We present Sp2360, a low-cost, data-efficient approach to reconstruct complex
$360^{\circ}$ scenes from a few input images. We proposed a system that
combines diffusion priors specialized for in-painting and Gaussian artifact
removal to generate artificial novel views, which are iteratively added to our
training image set. In our experiments we show that our approach approves over
previous methods on the challenging MipNeRF360 dataset and illustrate the
contributions of individual components in our ablation studies. For future
work, we see the potential of our system to be improved with additional
geometry cues from 3D vision foundation models.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  In arXiv, 2023.
  * [2] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In CVPR, 2022.
  * [3] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying MMD GANs.  In ICLR, 2018.
  * [4] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  Instructpix2pix: Learning to follow image editing instructions.  In CVPR, 2023.
  * [5] Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.  GeNVS: Generative novel view synthesis with 3D-aware diffusion models.  In ICCV, 2023.
  * [6] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann.  pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.  In CVPR, 2024.
  * [7] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In ICCV, 2021.
  * [8] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su.  Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction.  In ICCV, 2023.
  * [9] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  In ICCV, 2023.
  * [10] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai.  Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.  2024\.
  * [11] Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee.  Depth-regularized optimization for 3d gaussian splatting in few-shot images.  In arXiv, 2023.
  * [12] Devikalyan Das, Christopher Wewer, Raza Yunus, Eddy Ilg, and Jan Eric Lenssen.  Neural parametric gaussians for monocular non-rigid object reconstruction.  In CVPR, 2024.
  * [13] Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al.  Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors.  In CVPR, 2023.
  * [14] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised NeRF: Fewer views and faster training for free.  In CVPR, 2022.
  * [15] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In NeurIPS, 2021.
  * [16] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli.  Image quality assessment: Unifying structure and texture similarity.  In IEEE TPAMI, 2020.
  * [17] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
  * [18] Antoine Guédon and Vincent Lepetit.  Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering.  In CVPR, 2024.
  * [19] Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M. Susskind, and Qi Shan.  Fast and explicit neural view synthesis.  In WACV, 2022.
  * [20] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa.  Instruct-nerf2nerf: Editing 3d scenes with instructions.  In ICCV, 2023.
  * [21] Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow.  Deep blending for free-viewpoint image-based rendering.  In ACM TOG, 2018.
  * [22] Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny.  Unsupervised learning of 3d object categories from videos in the wild.  In CVPR, 2021.
  * [23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, 2017.
  * [24] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [25] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In ICLR, 2022.
  * [26] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao.  2d gaussian splatting for geometrically accurate radiance fields.  In SIGGRAPH, 2024.
  * [27] Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Vitor Guizilini, Thomas Kollar, Adrien Gaidon, Zsolt Kira, and Rares Ambrus.  Neo 360: Neural fields for sparse view synthesis of outdoor scenes.  In ICCV, 2023.
  * [28] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In ICCV, 2021.
  * [29] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In CVPR, 2014.
  * [30] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  In NeurIPS, 2022.
  * [31] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  In ACM TOG, 2023.
  * [32] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Tanks and temples: benchmarking large-scale scene reconstruction.  In ACM TOG, 2017.
  * [33] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu.  Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization.  In CVPR, 2024.
  * [34] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In CVPR, 2023.
  * [35] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.  Vision transformer for nerf-based view synthesis from a single input image.  In WACV, 2023.
  * [36] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In ICCV, 2023.
  * [37] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan.  Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.  In 3DV, 2024.
  * [38] Luke Melas-Kyriazi, Christian Rupprecht, and Andrea Vedaldi.  Pc2: Projection-conditioned point cloud diffusion for single-image 3d reconstruction.  In CVPR, 2023.
  * [39] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations, 2022.
  * [40] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [41] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner.  Diffrf: Rendering-guided 3d radiance field diffusion.  In CVPR, 2023.
  * [42] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  In ACM TOG, 2022.
  * [43] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  In ICML, 2022.
  * [44] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In CVPR, 2022.
  * [45] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In ICLR, 2023.
  * [46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [47] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  In arXiv, 2022.
  * [48] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  In IEEE TPAMI, 2020.
  * [49] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In ICCV, 2021.
  * [50] Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner.  Dense depth priors for neural radiance fields from sparse input views.  In CVPR, 2022.
  * [51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [52] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In CVPR, 2023.
  * [53] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In NeurIPS, 2022.
  * [54] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu.  ZeroNVS: Zero-shot 360-degree view synthesis from a single real image.  In CVPR, 2024.
  * [55] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [56] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In ECCV, 2016.
  * [57] Philipp Schröppel, Christopher Wewer, Jan Eric Lenssen, Eddy Ilg, and Thomas Brox.  Neural point cloud diffusion for disentangled 3d shape and appearance generation.  In CVPR, 2024.
  * [58] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In NeurIPS, 2022.
  * [59] J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein.  3d neural field generation using triplane diffusion.  In CVPR, 2023.
  * [60] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [61] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng.  Dreamgaussian: Generative gaussian splatting for efficient 3d content creation.  In ICLR, 2024.
  * [62] Luming Tang, Nataniel Ruiz, Chu Qinghao, Yuanzhen Li, Aleksander Holynski, David E Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, and Michael Rubinstein.  Realfill: Reference-driven generation for authentic image completion.  In arXiv, 2023.
  * [63] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d scene representation and rendering.  In ICCV, 2021.
  * [64] Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu.  Sparsenerf: Distilling depth ranking for few-shot novel view synthesis.  In ICCV, 2023.
  * [65] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In CVPR, 2023.
  * [66] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In NeurIPS, 2021.
  * [67] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud.  Dust3r: Geometric 3d vision made easy.  In CVPR, 2024.
  * [68] Christopher Wewer, Eddy Ilg, Bernt Schiele, and Jan Eric Lenssen.  SimNP: Learning self-similarity priors between neural points.  In ICCV, 2023.
  * [69] Christopher Wewer, Kevin Raj, Eddy Ilg, Bernt Schiele, and Jan Eric Lenssen.  latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction.  In arXiv, 2024.
  * [70] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang.  4d gaussian splatting for real-time dynamic scene rendering.  2024\.
  * [71] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski.  Reconfusion: 3d reconstruction with diffusion priors.  In CVPR, 2024.
  * [72] Jamie Wynn and Daniyar Turmukhambetov.  DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models.  In CVPR, 2023.
  * [73] Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi.  Sparsegs: Real-time 360° sparse view synthesis using gaussian splatting.  In arXiv, 2023.
  * [74] Jiawei Yang, Marco Pavone, and Yue Wang.  Freenerf: Improving few-shot neural rendering with free frequency regularization.  In CVPR, 2023.
  * [75] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelNeRF: Neural radiance fields from one or few images.  In CVPR, 2021.
  * [76] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [77] Linqi Zhou, Yilun Du, and Jiajun Wu.  3d shape generation and completion through point-voxel diffusion.  In ICCV, 2021.
  * [78] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  In SIGGRAPH, 2018.
  * [79] Zhizhuo Zhou and Shubham Tulsiani.  Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction.  In CVPR, 2023.
  * [80] Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang.  Fsgs: Real-time few-shot view synthesis using gaussian splatting.  In arXiv, 2023.

##  Appendix A Appendix / Supplemental Material

In this section, we provide further details on certain heuristics and design
choices of our pipeline not covered in the main paper. We begin with our
strategy for creating $M$-view subsets of a scene (c.f A.1). We then provide
details on the regularization introduced in our improved baseline to adapt
3DGS to sparse-view reconstruction (c.f A.2). Next, we compare our Gaussian
artifact removal module with existing image-to-image diffusion models and show
the importance of finetuned priors for adapting to this task (c.f A.3). We
perform a toy experiment to determine the best choice of parameters for our
iterative schedule (c.f. A.4). We end with some minor details on the
distillation procedure to lift 2D diffusion priors to 3D Gaussians (c.f. A.5).
Please also note our supplemental video, showing 360° reconstruction results.

###  A.1 Creating M-view Sparse Sets

3DGS relies on COLMAP for a sparse point cloud initialization, and COLMAP
requires sufficient feature correspondence across the input images to produce
a 3D consistent sparse scene geometry. To produce a subset where the views are
far apart to have sufficient scene coverage and at the same time have enough
feature correspondence among the images to produce a 3D consistent (albeit
sparse) point cloud, we devise a well-defined strategy such that view subsets
across different scenes follow a certain heuristic and are not just randomly
picked.

We are given a set $\mathcal{X}=\\{\mathbf{I}_{i},\pi_{i}\\}_{i=1}^{N}$ of
images and corresponding camera poses of a scene where
$\mathbf{I}_{i}\in\mathbb{R}^{h\times w\times 3}$ and
$\pi_{i}=[\mathrm{K}^{i}|\mathrm{R}^{i}|\mathrm{t}^{i}]\in SE(3)$. Here
$K^{i}$ is the camera intrinsics matrix, $\mathrm{R}^{i}\in\mathbb{R}^{3\times
3}$ is the rotation matrix, and $t^{i}\in\mathbb{R}^{3\times 1}$ is the
translation vector. From this dense set, we aim to create $M$-view sparse sets
$(M<N)$ $\\{\mathbf{I}_{k},\mathrm{\pi}_{k}\\}_{k=1}^{M}$ with a corresponding
sparse point cloud $P_{M}\in\mathbb{R}^{S\times 3}$, obtained via COLMAP. To
maximize the scene coverage of the $M$-view subset, we use the SE(3) (Special
Euclidean group in three dimensions) representation to define a geodesic
distance between a pair of camera viewpoints. SE(3) represents all rigid
transformations (translations and rotations) that can be applied to a three-
dimensional space while preserving distances and angles. Mathematically, SE(3)
is defined as:

$\displaystyle SE(3)=\\{(\mathrm{R},t)|\mathrm{R}\in
SO(3),t\in\mathbb{R}^{3}\\}$ (5)

Here, $SO(3)$ is the special orthogonal group in three dimensions,
representing all possible 3D rotations without reflection or scaling. To
quantify “closeness” between 2 SE(3) elements $\mathrm{\pi}_{1}$ and
$\mathrm{\pi}_{2}$, we use the geodesic distance that measures the shortest
path between 2 viewpoints on a curved surface - in this case, the minimum
distance along a curve in SE(3) that connects $\mathrm{\pi}_{1}$ and
$\mathrm{\pi}_{2}$. One common approach of measuring the geodesic distance
involves separating the rotational and translational components and combining
individual distances through a weighted average. If $\mathrm{R_{1}}$,
$\mathrm{T_{1}}$ and $\mathrm{R_{2}}$, $\mathrm{T_{2}}$ are the rotation and
translation matrices corresponding to $\mathrm{\pi}_{1}$ and
$\mathrm{\pi}_{2}$ respectively, the geodesic distance can be defined as:

$\displaystyle\mathcal{D}_{geodesic}=\lVert\textnormal{Rodrigues}(\mathrm{R}_{rel})\rVert+w_{T}\lVert\mathrm{T_{1}}-\mathrm{T_{2}}\rVert$
(6)

where the Rodrigues formula determines the relative angle of rotation $\theta$
between 2 matrices $\mathrm{R_{1}}$ and $\mathrm{R_{2}}$ by measuring the
angle of the difference rotation matrix
$\mathrm{R}_{rel}=\mathrm{R_{1}}\mathrm{R_{2}}^{T}$ as follows:

$\displaystyle\textnormal{Rodrigues}(\mathrm{R}_{rel})=\mathcal{\theta}=\cos^{-1}\left(\frac{\text{trace}(R_{1}R_{2}^{T})-1}{2}\right)$
(7)

$w_{T}$ is a weighting factor. We set $w_{T}=0.1$ for all our experiments as
we usually find that rotational distance has more impact on scene coverage
given an $M$-view constraint.

Suppose the size of the current view stack is $n_{s}<M$, and now we want to
add a new viewpoint $\mathrm{\pi}_{k}$ from a stack of $(N-n_{s})$ available
viewpoints. Now, we can either pick the closest, the $2^{nd}$ closest, or the
$n^{th}$ closest viewpoint with respect to current stack
$\\{\mathrm{\pi}_{j}\\}_{j=1}^{n_{s}}$. We calculate a matrix of geodesic
distances of dimension $n_{s}\times(N-n_{s})$ between
$\\{\mathrm{\pi}_{j}\\}_{j=1}^{n_{s}}$ and
$\\{\mathrm{\pi}_{j}\\}_{j=n_{s}+1}^{N}$ and pick the $n^{th}$ closest
viewpoint from $\\{\mathrm{\pi}_{j}\\}_{j=n_{s}+1}^{N}$. This process
continues until we have $M$ total viewpoints in our stack. We observe
experimentally that, as we go on increasing $n$,
$\underset{i,j}{\max}\,\mathcal{D}^{\mathrm{\pi}}_{i,j}(n)$, i.e., the maximum
geodesic distance across all pairs of viewpoints in
$\\{\mathrm{\pi}_{j}\\}_{j=1}^{M}$, also shows an increasing trend (non-
monotonous), while the size of $P_{M}$ shows a decreasing trend (also non-
monotonous). After a certain value of $n$, COLMAP can no longer find enough
feature correspondences among image pairs to register all images to a single
point cloud $P_{M}$. This is where we terminate and pick the $M$-view subset
corresponding to $\underset{n}{\max}\,\mathcal{D}^{\mathrm{\pi}}(n)$.

(a) $M=9$ in the $treehill$ scene

(b) $M=18$ in the $stump$ scene

Figure 6: Plot of maximum geodesic distance across training viewpoints and SfM
point cloud size vs. $n$ \- the index of the $n^{th}$ closest viewpoint from
the available training stack. The locations of the maxima in the blue plots
are used to create our view subset of size $M$.

Figure 7: We find that the maximum geodesic distance between all possible
pairs of cameras in a view subset directly correlates with how much of a
$360^{\circ}$ scene we can observe. Here we show two farthest views (in terms
of $\mathcal{D}_{geodesic}$) of the _treehill_ scene for $M=9$.

An example plot is shown in Fig 6 where we observe the change in
$\mathcal{D}^{\mathrm{\pi}}(n)$ and the point cloud size $P_{M}$ as we
steadily increase $n$ by 1. For $M=9$ in the _treehill_ scene (left), we pick
the subset corresponding to $n=17$, the global maxima for
$\mathcal{D}^{\mathrm{\pi}}(n)$. Similarly, for $M=18$ in the _stump_ scene
(right), we pick the subset corresponding to $n=31$. Visually, in Fig 7, we
observe that this heuristic gives us considerable scene coverage.

###  A.2 Sparse 3DGS

Inspired by recent works [73, 80], we introduce 4 different regularization
techniques to prevent 3DGS from overfitting to the observed few views.

#### Tunable Hyperparameters

Similar to [11], we observe that the periodic resetting of Gaussian opacities,
designed to restrict over-reconstruction in a dense setting, harms
generalization to novel views in a sparse setting. Either several Gaussians
are removed through pruning or get trapped in local minima of the optimization
process, not receiving enough updates to match the given scene. Visually, we
observe much lower artifacts in novel views for extremely sparse settings
($M\leq 18$) if opacities $\alpha$ are optimized with no resets during
training. The densification threshold for positional gradients - $\tau_{pos}$
is another hyperparameter that controls the overall number of Gaussians.
Experimentally, we observe varying $\tau_{pos}$ between 2-5 times the default
value for dense setting in 3DGS leads to smoother geometry and prevents
overfitting in a sparse setting for $M\leq 54$. We show visual evidence in Fig
8 and 9 below.

(a) No opacity ($\alpha$) reset for $30k$ iterations.

(b) opacity ($\alpha$) reset every $3k$ iterations.

Figure 8: Periodic reset of the opacity ($\alpha$) of Gaussians has a
detrimental effect on novel view synthesis in a sparse reconstruction setting.
For example, scene details are more clearly observed in the left image, where
there is no $\alpha$ reset for $30k$ iterations. ($M=9$)

(a) $\tau_{pos}=0.0002$

(b) $\tau_{pos}=0.001$

Figure 9: Effect of densification threshold ($\tau_{pos}$) for $M=9$ in the
$bicycle$ scene. A higher value of $\tau_{pos}$ in a view-constrained scenario
limits artifacts and favors smoother geometry.

#### Depth Regularization

Inspired by depth regularization techniques in NeRF [14, 44] and 3DGS
literature [33], we introduce dense depth priors from pretrained monocular
depth estimators (MDEs) [48]. Due to the severe sparsity of the initial point
cloud $P_{M}$, we observe that this regularization is essential to make
Gaussians grow towards a reasonable approximation of the true scene geometry.
If $\mathcal{D}_{est}$ is the estimated monocular depth from a training view,
and $\mathcal{D}_{ras}$ is the rendered depth, we employ the Pearson
correlation coefficient (PCC) as a loss function to align the two depth map
distributions [73, 80]. Specifically, we add a third loss term to the 3DGS
optimization as:

$\displaystyle\mathcal{L}_{depth}=1-PCC(\hat{\mathcal{D}}_{ras},\hat{\mathcal{D}}_{est})=1-\frac{Cov(\hat{\mathcal{D}}_{ras},\hat{\mathcal{D}}_{est})}{\sqrt{Var(\hat{\mathcal{D}}_{ras})Var(\hat{\mathcal{D}}_{est})}}$
(8)

where $\hat{\mathcal{D}}_{ras}$, $\hat{\mathcal{D}}_{est}$ are obtained after
normalizing $\mathcal{D}_{ras}$ and $\mathcal{D}_{est}$ to the range $[0,1]$.

#### Pseudo Views

As a final regularization, we generate pseudo views using camera interpolation
and employ depth regularization as above to improve generalization in
unobserved areas. We synthesize novel views on a path by interpolating between
a viewpoint $\pi_{1}$ and its closest view $\pi_{2}$ in SE3 space.
Specifically, we sample $\mathbf{t}$ on a B-spline fitted between the
translation matrices and estimate an averaged orientation $\mathbf{R}$ through
slerp interpolation on the rotation matrices. This yields an augmented
viewpoint $\hat{\pi}$ with shared camera intrinsics and camera extrinsics
$(\mathbf{R}|\mathbf{t})$. Without a ground truth image for $\hat{\pi}$, we
only enable depth regularization on the rendered RGB image. Overall, the
optimization objective becomes:

$\displaystyle\mathcal{L}=(1-\lambda_{1})\mathcal{L}_{1}+\lambda_{1}\mathcal{L}_{D-SSIM}+\lambda_{depth}\mathcal{L}_{depth}+\lambda_{pseudo}\mathcal{L}_{pseudo}$
(9)

We switch on $\mathcal{L}_{pseudo}$ after 2000 iterations when the
optimization has stabilized and Gaussians have learned at least a coarse scene
geometry. We call this improved baseline _Sparse 3DGS_. Visually, we observe
in Fig 10 that depth regularization through both ground truth and synthesized
views helps 3DGS learn plausible geometry of the scene - like the structure of
the bicycle and bench while smoothening out a lot of the grainy noise in the
background.

(a) $\lambda_{depth}=0.05,\lambda_{pseudo}=0.05$

(b) $\lambda_{depth}=\lambda_{pseudo}=0.0$

Figure 10: Effect of $\mathcal{L}_{depth}$ and $\mathcal{L}_{pseudo}$ for
$M=9$ in $bicycle$ scene. Enabling a depth-based regularization loss during
training helps resolve depth ambiguities and learn true scene geometry at
nearby novel viewpoints.

###  A.3 Baselines for Artifact Removal Module

In Fig 11, we compare the quality of diffusion samples from our proposed
Artifact Removal Module (Sec 3.3) and two possible baseline solutions based on
image-to-image diffusion. For this, we pick the $9$-view split of the
_bicycle_ scene and run our two-step view synthesis pipeline with the
baselines. All $3$ methods take inpainted renders obtained from the Inpainting
Module as input (Sec 3.2).

|  Render |  In-painted render |  SD Img2Img |  Instruct Pix2Pix |  Ours |  Ground Truth
---|---|---|---|---|---|---
View 1 |  |  |  |  |  |
View 2 |  |  |  |  |  |
View 3 |  |  |  |  |  |
View 4 |  |  |  |  |  |
Figure 11: Qualitative comparison of _Artifact Removal Module_ with image-to-
image diffusion baselines. The baselines either produce unrealistic images
with background collapse or fail to restore areas in the render corrupted by
artifacts. Our finetuned prior, on the contrary, identifies Gaussian artifacts
and removes them more efficiently to recover both foreground and background
details in the scene.

#### Stable Diffusion Image-to-Image

Image-to-image diffusion is similar to text-to-image synthesis, but in
addition to a text prompt, the final generation is also conditioned on an
initial image. We use Stable Diffusion Image-to-Image [39] to try and remove
Gaussian artifacts from inpainted renders. If $\mathbf{x}_{0}$ is the
inpainted render, it is first encoded as
$\mathbf{z}_{0}=\mathcal{E}(\mathbf{x}_{0})$, and then noised corresponding to
a certain timestep $t$ to obtain $\mathbf{z}_{t}$. The UNet
$\boldsymbol{\epsilon_{\theta}}$ predicts the noise in $\mathbf{z}_{t}$ as:

$\displaystyle\hat{\epsilon_{t}}=\boldsymbol{\epsilon_{\theta}}(\mathbf{z}_{t};t,c_{\theta}(y)),$
(10)

Here $y=$“A photo of [V]”, where $[V]$ is a special token by textual inversion
[17] from the $M$ reference images, to capture visual and semantic details of
the scene. We set $t_{max}=0.3$ to ensure the output does not deviate too much
from $\mathbf{x}_{0}$ and keeps at least geometry intact. However, the model
has trouble differentiating Gaussian artifacts from image structures and hence
cannot recover any scene details lost to artifacts. Instead, it creates images
with oversmoothed geometry that look unrealistic and lack any high-frequency
details of the original scene.

#### Instruct Pix2Pix

We try to obtain clean renders with the pretrained Instruct Pix2Pix (IP2P) [4]
checkpoint using the same base instruction as our finetuned artifact removal
module. However, IP2P was originally designed to perform image edits like
replacing objects, changing the style or setting of a given image, etc, and
not an image restoration task we aim to solve. As such, both the conditioning
artifact image and the edit instruction are out-of-distribution for IP2P, and
hence it fails to identify or remove any of the Gaussian artifacts.

Our finetuned artifact removal module, on the other hand, efficiently deals
with blur, floaters, and color artifacts in $\mathbf{x}_{0}$ and recovers
image structures lost to Gaussian artifacts. Artifacts induced at novel views
by a 3D Gaussian representation are not observed in images used to train
generic image-to-image diffusion models, necessitating the creation of our
synthetically curated fine-tuning dataset.

###  A.4 Iterative 3DGS

Our iterative schedule fuses clean, in-painted novel view renders with the 3D
Gaussians fitted by _Sparse 3DGS_. This part of our method is inspired, in
part, by Instruct NeRF2NeRF [20]. However, unlike their setting, we do not
have ground truths for novel views to condition our diffusion models, making
it more challenging. We propose to find the best combination of parameters for
our algorithm in a simplified setting.

We first fit 3D Gaussians to the $9$ view split of the _bicycle_ scene and
then autoregressively sample closest novel views to fuse corresponding ground
truth image with existing scene Gaussians. We argue that this variant should
closely match the performance of a _full model_ fitted to dense views. Both
models see the same training views but with different initializations - SfM
point cloud of the entire scene for the _full model_ vs 3D Gaussians fitted to
$M$ views for _Iterative 3DGS_. The _full model_ sees all training views
simultaneously, while the iterative version does so incrementally, adjusting
the number and position of Gaussians with information from each novel view.

We fuse the heldout $N-M$ novel views over 30k iterations, sampling $m$ views
at a time where $m\in\\{1,2,3\\}$. After each incremental update of the
training stack, the Gaussians are optimized for $n_{k}$ iterations where
$\\{n_{k}\\}_{k=1}^{\lceil\frac{N-M}{m}\rceil}$ is sampled from a schedule,
which is a function of the total iterations (30k) and the number of novel
views $N-M$. Specifically, $n_{k}$’s satisfy the following constraint:

$\displaystyle\sum_{k=1}^{\lceil\frac{N-M}{m}\rceil}n_{k}$
$\displaystyle=30000$ (11) $\displaystyle n_{k+1}$ $\displaystyle=f(k)\cdot
n_{k}\hskip 20.00003ptk\in\\{1,2,\ldots,\lceil\frac{N-M}{m}\rceil-1\\}$

In the simplest case, when we have a constant number of iterations per train
stack update, $f(k)=1$ and
$n_{k}=\left\lfloor\frac{30000}{\frac{N-M}{m}}\right\rfloor$. For linear and
quadratic schedules, $f(k)=a\cdot k$ or $f(k)=a^{2}\cdot k$, where $a$ is
calculated from the summation constraint (Eq 11). A generic solver for Eq 11
typically finds the largest $a$ for which $\sum_{k}n_{k}\leq 30000$ is
satisfied, and the residual iterations are added to one of the $n_{k}$’s,
usually $n_{\lceil\frac{N-M}{m}\rceil}$.

Table 4: Ablation Study on design choices of Iterative 3DGS for $M=9$ in $bicycle$ scene. We fuse ground truth images for heldout novel views and attempt to match the performance of a _full model_. Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
$m=1$ \+ constant | 21.13 | 0.51 | 0.39
$m=2$ \+ constant | 21.87 | 0.52 | 0.36
$m=3$ \+ constant | 21.29 | 0.53 | 0.39
$m=2$ \+ cosine | 21.01 | 0.52 | 0.37
$m=2$ \+ linear | 22.26 | 0.57 | 0.34
$m=2$ \+ quadratic | 22.43 | 0.58 | 0.33
$m=2$ \+ quadratic + no $\alpha$-reset | 20.37 | 0.48 | 0.39
$m=2$ \+ quadratic + $\eta=0.95$ | 22.72 | 0.60 | 0.32
$m=2$ \+ quadratic + $\eta=0.97$ ($\mathcal{O}$) | 22.95 | 0.63 | 0.29
_Full Model_ | 24.34 | 0.73 | 0.22

#### Findings

We summarize the findings of our toy experiment in Tab 4. We experiment with
constant, linear, quadratic, and cosine schedules and find that a quadratic
schedule gets us closest to the performance of the full model. We find $m=2$
gives better performance compared to adding just 1 novel viewpoint at a time.
We hypothesize that adding 2 viewpoints at a time provides an implicit depth
prior that better guides incremental optimization of the Gaussians. Adding
$m>2$ viewpoints does not provide any further improvement. Each Gaussian has a
3D scaling vector $s$ as one of its attributes that determines the scaling of
the ellipsoid in 3 axial directions in world space. Our experiments show that
shrinking $s$ by a scale adjustment factor $\eta<1$, once every set of
viewpoint(s) is added, impacts generalization to novel views. Intuitively,
some of the Gaussians, fitted to sparse views, would have grown towards large
volumes to create over-smoothed geometry in empty space. So, shrinking the
extent of the ellipsoids, as further observations are introduced, gives more
incentive to the _adaptive control module_ to clone Gaussians in under-
reconstructed regions and reconstruct small-scale geometry better.

Lastly, we find that regularizations introduced in _Sparse 3DGS_ are no longer
needed during the iterative process due to the growing set of observations and
diminishing ambiguities. As such, setting $\tau_{pos}=0.0002$, resetting
opacity($\alpha$) every $3k$ iterations, and removing $\mathcal{L}_{depth}$,
$\mathcal{L}_{pseudo}$ gets us closest to the performance of the _full model_.
We call the best-performing variant our oracle $\mathcal{O}$, which gives us
the experimental maximum for performance on the held-out test set.
$\mathcal{O}$ also has similar storage requirements as the _full model_
($3.82M$ v $3.80M$ Gaussians), meaning the _iterative 3DGS_ does not lead to a
drop in FPS at inference time.

###  A.5 Distilling 2D priors to 3D

Our diffusion priors infer plausible detail in unobserved regions and the
iterative update algorithm incrementally grows and updates scene Gaussians by
fusing information at novel viewpoints. We initialize the scene with 3D
Gaussians fitted by _Sparse 3DGS_ to $M$ input views, autoregressively sample
closest novel viewpoints (based on SE3 distance), obtain pseudo ground truths
for novel views using our combination of diffusion priors, add them to the
training stack and optimize for certain iterations (determined by the schedule
in Sec A.4). At every iteration, we sample either an observed or unobserved
viewpoint from the current training stack. We optimize Gaussian attributes
using the 3DGS objective for known poses and the SparseFusion [79] objective
for novel views:

$\displaystyle\mathcal{L}_{sample}(\psi)$
$\displaystyle=\mathbb{E}_{\pi,t}\Big{[}w(t)(\|I_{\pi}-\hat{I}_{\pi}\|_{1}+\mathcal{L}_{p}(I_{\pi},\hat{I}_{\pi}))\Big{]}$
(12)

where $\mathcal{L}_{p}$ is the perceptual loss [76], $w(t)$ a noise-dependent
weighting function, $I_{\pi}$ is the 3DGS render at novel viewpoint $\pi$, and
$\hat{I}_{\pi}$ is the inpainted, clean render version of $I_{\pi}$ obtained
with our cascaded diffusion priors.


#  BERTić \- The Transformer Language Model for
Bosnian, Croatian, Montenegrin and Serbian

Nikola Ljubešić
Jožef Stefan Institute
Jamova cesta 39
Ljubljana, Slovenia
<EMAIL_ADDRESS>
Davor Lauc
Faculty of Humanities and Social Sciences
Ivana Lučića 3
Zagreb, Croatia
<EMAIL_ADDRESS>

###### Abstract

In this paper we describe a transformer model pre-trained on 8 billion tokens
of crawled text from the Croatian, Bosnian, Serbian and Montenegrin web
domains. We evaluate the transformer model on the tasks of part-of-speech
tagging, named-entity-recognition, geo-location prediction and commonsense
causal reasoning, showing improvements on all tasks over state-of-the-art
models. For commonsense reasoning evaluation we introduce COPA-HR - a
translation of the Choice of Plausible Alternatives (COPA) dataset into
Croatian. The BERTić model is made available for free usage and further task-
specific fine-tuning through HuggingFace.

##  1 Introduction

In recent years, pre-trained transformer models have taken the NLP world by
storm Devlin et al. (2018); Liu et al. (2019); Brown et al. (2020), yielding
new state-of-the-art results in various tasks and settings. While such models,
requiring significant computing power and data quantity, started to emerge for
non-English languages Martin et al. (2019); de Vries et al. (2019), as well as
in multilingual flavours Devlin et al. (2018); Conneau et al. (2019), there is
a significant number of languages for which better models can be obtained with
the available pre-training techniques.

This paper describes such an effort - training a transformer language model on
more than 8 billion tokens of text written in the Bosnian, Croatian,
Montenegrin or Serbian language, all these languages being very closely
related, mutually intelligible, and classified under the same HBS (Serbo-
Croatian) macro-language by the ISO-693-3
standard.111https://iso639-3.sil.org/code_tables/macrolanguage_mappings/data

The name of the model – BERTić – points at two facts: (1) the language model
was trained in Zagreb, Croatia, in whose vernacular diminutives ending in _ić_
are frequently used (_fotić_ eng. photo camera, _smajlić_ eng. smiley,
_hengić_ eng. hanging together), and (2) in all the countries / languages of
this model the patronymic surnames end to a great part with the suffix _ić_ ,
with likely diminutive etymology.

The paper is structured as follows: in the following section we describe the
data the model is based on, in the third section we give a short description
of the modelling performed, and in the fourth section we present a detailed
evaluation of the model.

##  2 Data

As our data basis we use already existing datasets, namely (1) the hrWaC
corpus of the Croatian top-level domain, crawled in 2011 Ljubešić and Erjavec
(2011) and again in 2014 Ljubešić and Klubička (2014), (2) the srWaC corpus of
the Serbian top-level domain, crawled in 2014 Ljubešić and Klubička (2014),
(3) the bsWaC corpus of the Bosnian top-level domain, crawled in 2014 Ljubešić
and Klubička (2014), (4) the cnrWaC corpus of the Montenegrin top-level
domain, crawled in 2019, and (5) the Riznica corpus consisting of Croatian
literary works and newspapers Ćavar and Rončević (2012).

Given that most of the crawls contain data only up to year 2014, we performed
new crawls of the Bosnian, Croatian and Serbian top-level domains. We brand
these corpora as CLASSLA web corpora given that CLASSLA is the CLARIN
knowledge centre for South Slavic
languages222https://www.clarin.si/info/k-centre/ under which we perform most
of the described activities. We deduplicate the CLASSLA corpora by removing
identical sentences that were already present in the WaC corpora. The amount
of data removed through this deduplication is minor, in all cases in single
digit percentages.

We further exploit the recently published cc100 corpora Conneau et al. (2019)
that are based on the CommonCrawl data collection. We perform the same level
of deduplication as with the CLASSLA corpora, with every sentence already
present in the WaC or CLASSLA corpus being removed from the cc100 corpus. This
round of deduplication removed around 15% of the CommonCrawl data.

The resulting sizes of the datasets used for training the BERTić model are
presented in Table 1. The overall text collection consists of 8,387,681,518
words.

dataset | language | # of words
---|---|---
hrWaC | Croatian | 1,250,923,836
CLASSLA-hr | Croatian | 1,341,494,461
cc100-hr | Croatian | 2,880,490,449
Riznica | Croatian | 87,724,737
srWaC | Serbian | 493,202,149
CLASSLA-sr | Serbian | 752,916,260
cc100-sr | Serbian | 711,014,370
bsWaC | Bosnian | 256,388,597
CLASSLA-bs | Bosnian | 534,074,921
cnrWaC | Montenegrin | 79,451,738
Table 1: Datasets used for training the BERTić model with their size (in
number of words) after deduplication.

##  3 Model training

For training this model we selected the Electra approach to training
transformer models Clark et al. (2020). These models are based on training a
smaller generator model and the main, larger, discriminator model whose task
is to discriminate whether a specific word is the original word from the text,
or a word generated by the generator model. The authors claim that the Electra
approach is computationally more efficient than the BERT models Devlin et al.
(2018) based on masked language modelling.

As in BERT and similar transformers models, we constructed a WordPiece
vocabulary with a vocabulary size of 32 thousand tokens. A WordPiece model was
trained using the HuggingFace tokenizers
library333https://huggingface.co/transformers/main_classes/tokenizer.html on
the random sample of 10 million paragraphs from the whole dataset. Text pre-
processing and cleaning differ from the original BERT only in preserving all
Unicode characters, while in the original pre-processing diacritics are
removed.

Training of the model was performed to the most part with the hyperparameters
set for base-sized models (110 million parameters in 12 transformer layers) as
defined in the Electra paper Clark et al. (2020). Training batch size was kept
at 1024, the maximum size for the 8 TPUv3 units on which the training was
performed. The training was run for 2 million steps (roughly 50 epochs).

##  4 Evaluation

In this section we present an exhaustive evaluation of the newly trained
BERTić model on two token classification tasks – morphosyntactic tagging and
named entity recognition, and two sequence classification tasks – geolocation
prediction and commonsense causative reasoning.

The reference points in each task are the state-of-the art transformer models
covering the macro-language - multilingual BERT Devlin et al. (2018) and
CroSloEngual BERT Ulčar and Robnik-Šikonja (2020). While multilingual BERT
(mBERT onwards) was trained on Wikipedia corpora, CroSloEngual BERT (cseBERT
onwards) was trained on a similar amount of Croatian data used to train
BERTić, but without the data from the remaining languages.

dataset | language | variety | CLASSLA | mBERT | cseBERT | BERTić
---|---|---|---|---|---|---
hr500k | Croatian | standard | 93.87 | 94.60 | 95.74 | ***95.81
reldi-hr | Croatian | non-standard | - | 88.87 | 91.63 | ***92.28
SETimes.SR | Serbian | standard | 95.00 | 95.50 | 96.41 | 96.31
reldi-sr | Serbian | non-standard | - | 91.26 | 93.54 | ***93.90
Table 2: Average microF1 results on the morphosyntactic annotation task over five training iterations. The highest score per dataset is marked with bold. The statistical significance is tested with the two-sided t-test over the five runs between the two strongest results. Level of significance is labeled with asteriks signs (*** p$<=$0.001). dataset | language | variety | CLASSLA | mBERT | cseBERT | BERTić
---|---|---|---|---|---|---
hr500k | Croatian | standard | 80.13 | 85.67 | 88.98 | ****89.21
ReLDI-hr | Croatian | non-standard | - | 76.06 | 81.38 | ****83.05
SETimes.SR | Serbian | standard | 84.64 | 92.41 | 92.28 | 92.02
ReLDI-sr | Serbian | non-standard | - | 81.29 | 82.76 | ***87.92
Table 3: Average F1 results on the named entity recognition task over five
training iterations. The highest score per dataset is marked with bold. The
statistical significance is tested with the two-sided t-test over the five
runs between the two strongest results. Level of significance is labeled with
asteriks signs (*** p$<=$0.001, **** p$<=$0.0001).

###  4.1 Morphosyntactic tagging

On the task of morphosyntactic tagging (assigning each word one among multiple
hundreds of detailed morphosyntactic classes, e.g. Ncmsay referring to a
common masculine noun, in accusative case, singular number, animate) we
compare the three transformer models, mBERT, cseBERT and BERTić. We
additionally report results, when available, for the current production tagger
for the two languages - the CLASSLA tool Ljubešić and Dobrovoljc (2019), based
on Stanford’s Stanza, exploiting static embedding and BiLSTM technology Qi et
al. (2020).

We perform evaluation of the models on this task on four datasets: the
Croatian standard language dataset hr500k Ljubešić et al. (2018), the Croatian
non-standard language dataset ReLDI-hr Ljubešić et al. (2019a), the Serbian
standard language dataset SETimes.SR Batanović et al. (2018) and the Serbian
non-standard Twitter language dataset ReLDI-sr Ljubešić et al. (2019b).

For each dataset and model we perform hyperparameter optimization via Bayesian
search on the wandb.ai platform Biewald (2020), allowing for 30 iterations. We
optimize the initial learning rate (we search between the values of 9e-6 and
1e-4) and the epoch number (we search between the values of 3 and 15).

We report average microF1 results of five runs per dataset and model in Table
2. The highest score per dataset is marked with bold. The statistical
significance is tested with the two-sided t-test over the five runs between
the two highest average results. We can observe that the BERTić model
outperforms all the remaining models, cseBERT coming second, on three out of
four datasets. Only on the Serbian standard dataset the difference between
these two models is insignificant. We argue that this is due to the simplicity
of the dataset - it consists of texts from one newspaper only, therefore
containing text with little variation even between the training and the
testing data.

###  4.2 Named entity recognition

On the task of named entity recognition we compare the same models on the same
datasets as was the case in the previous Section 4.1. We also perform an
identical hyperparameter optimisation and experimentation and report the
results in Table 3. The results show again that the two best performing models
are cseBERT and BERTić with the latter performing better on three out of four
datasets, again, with no significant difference on the standard Serbian task
for the same reasons as with the previous task.

###  4.3 Social media geolocation

In this subsection we compare the three transformer models on the Social Media
Geolocation (SMG2020) shared task, which part of the VarDial 2020 Evaluation
Campaign Gaman et al. (2020). The task consists of predicting the exact
latitude and longitude of a geo-encoded tweet published in Croatia, Bosnia,
Montenegro or Serbia. The shared task winner in 2020 was using the cseBERT
model Scherrer and Ljubešić (2020) in its approach.

We evaluate the model on the two evaluation metrics of the shared task -
median and mean of the distance between gold and predicted geolocations. Given
the large size of the training dataset (320,042 instances), we do not perform
any additional hyperparameter tuning beyond the one performed during the
participation in the shared task and apply the same methodology: we fine-tune
the transformer model with batch size of 64 for 40 epochs and retain the model
with minimum median distance on development data. The results in Table 4 show
that the BERTić model improved the results of the shared task winner – the
cseBERT model.

| median | mean
---|---|---
centroid | 107.10 | 145.72
mBERT | 42.25 | 82.05
cseBERT | 40.76 | 81.88
BERTić | 37.96 | 79.30
Table 4:  Median distance and mean distance between gold and predicted
geolocation (lower is better) on the task of social media geolocation
prediction. The best results are marked in bold. No statistical testing was
performed due to a large size of the test dataset (39,723 instances).

###  4.4 Commonsense causal reasoning

The final evaluation round of the new BERTić model is performed on the task of
commonsense causal reasoning on a translation of the COPA dataset Roemmele et
al. (2011) into Croatian, the COPA-HR dataset. The translation is performed by
following the methodology laid out while preparing the XCOPA dataset Ponti et
al. (2020), a translation of the COPA dataset into 11 typologically balanced
languages.

The dataset consists of 400 training, 100 development and 500 examples. Each
instance in the dataset consists of a premise (_The man broke his toe_), a
question (_What was the cause?_),444Roughly half of the instances contain the
other question: _What was the effect?_ , and two alternatives, one of them to
be chosen by the system as being more plausible (_He got a hole in his sock_ ,
or _He dropped a hammer on his foot_).

While translating the dataset, the translator was also given the task of
selecting the more plausible alternative given their translation. The observed
agreement between the annotations in the English dataset and the annotations
of the Croatian translator was perfect on the training set and the development
set, while on the test set one out of 500 choices differed. The problematic
example proved to be a rather unclear case – the premise being _I paused to
stop talking._ , with the question _What was the cause?_ , and the
alternatives _I lost my voice._ and _I ran out of breath._.555The Croatian
translator chose the second alternative, while in the original dataset the
first alternative is chosen. The dataset is available from the CLARIN.SI
repository Ljubešić (2021).666http://hdl.handle.net/11356/1404

The approach taken to benchmarking the three transformer models is that of
sentence pair classification, each original instance becoming two sentence
pair instances (each sentence pair containing the premise and one
alternative), with different models being trained for _cause_ and _effect_
questions. During evaluation, separate predictions are made on each of the
alternatives, the per-class predictions being fed to a softmax function, and
the higher positive-class alternative being chosen as the correct one.

The standard evaluation metric for this dataset is accuracy. Given the
balanced nature of the test set, the random baseline is 50%. For
hyperparameter optimization the same approach was taken as with the token
classification tasks.

The results presented in Table 5 show that both language-specific transformer
models outperform mBERT significantly, with BERTić obtaining a significant
lead over cseBERT.

| accuracy
---|---
random | 50.00
mBERT | 54.12
cseBERT | 61.80
BERTić | **65.76
Table 5: Average accuracy results on the commonsense causal reasoning task
over five training iterations. The highest score per dataset is marked with
bold. The statistical significance is tested with the two-sided t-test over
the five runs between the two strongest results (** p$<=$0.01).

##  5 Conclusion

In this paper we have presented a newly published Electra transformer language
model, BERTić, trained on more than 8 billion tokens of previously and newly
collected web text written in Bosnian, Croatian, Montenegrin or Serbian. We
have applied a very thorough evaluation of the model, comparing it primarily
to other state-of-the-art transformer models that support the languages in
question. We have obtained significant improvements on all four tasks, with no
difference obtained only on one single-source-dataset with little text
variation and high training and testing data similarity.

The main conclusions we can draw from our results are the following. (1)
Although cseBERT and BERTić use a different approach to building transformer
language models, our assumption is that the performance difference between
these two lies primarily in the larger amount of data presented to the BERTić
model. (2) The improvements on the four tasks with the BERTić model seem to be
smaller on the morphosyntactic tagging task than the remaining three tasks
that require more world and commonsense reasoning knowledge. (3) Except for
the named entity recognition task on the Serbian non-standard dataset, we fail
to observe greater improvements on Serbian tasks than on Croatian ones between
cseBERT and BERTić, regardless the fact that the former has seen none and the
latter has seen huge quantities of Serbian text, showing the irrelevance of
minor language differences for performance of large transformer models. (4)
While BiLSTM models are still close-to-competitive on the morphosyntactic
tagging task, they cannot hold up on the named entity recognition task as it
requires more common knowledge. Such knowledge transformer models manage to
absorb to a much higher level than pre-trained static embeddings used by
BiLSTMs.

The BERTić model is available from the HuggingFace repository at
https://huggingface.co/classla/bcms-bertic.

## Acknowledgments

This work has been supported by the Slovenian Research Agency and the Flemish
Research Foundation through the bilateral research project ARRS N6-0099 and
FWO G070619N “The linguistic landscape of hate speech on social media”, the
Slovenian Research Agency research core funding No. P6-0411 “Language
resources and technologies for Slovene language”, and the European Union’s
Rights, Equality and Citizenship Programme (2014-2020) project IMSyPP (grant
no. 875263). We would like to thank the anonymous reviewers and Ivo-Pavao
Jazbec for their useful feedback.

## References

  * Batanović et al. (2018) Vuk Batanović, Nikola Ljubešić, Tanja Samardžić, and Tomaž Erjavec. 2018.  Training corpus SETimes.SR 1.0.  Slovenian language resource repository CLARIN.SI.
  * Biewald (2020) Lukas Biewald. 2020.  Experiment tracking with weights and biases.  Software available from wandb.com.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Ćavar and Rončević (2012) Damir Ćavar and Dunja Brozović Rončević. 2012.  Riznica: the Croatian language corpus.  _Prace filologiczne_ , 63:51–65.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gaman et al. (2020) Mihaela Gaman, Dirk Hovy, Radu Tudor Ionescu, Heidi Jauhiainen, Tommi Jauhiainen, Krister Lindén, Nikola Ljubešić, Niko Partanen, Christoph Purschke, Yves Scherrer, et al. 2020.  A report on the VarDial evaluation campaign 2020.  In _Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects_. International Committee on Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ljubešić (2021) Nikola Ljubešić. 2021.  Choice of plausible alternatives dataset in Croatian COPA-HR.  Slovenian language resource repository CLARIN.SI.
  * Ljubešić et al. (2018) Nikola Ljubešić, Željko Agić, Filip Klubička, Vuk Batanović, and Tomaž Erjavec. 2018.  Training corpus hr500k 1.0.  Slovenian language resource repository CLARIN.SI.
  * Ljubešić and Dobrovoljc (2019) Nikola Ljubešić and Kaja Dobrovoljc. 2019.  What does neural bring? analysing improvements in morphosyntactic annotation and lemmatisation of Slovenian, Croatian and Serbian.  In _Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing_ , pages 29–34, Florence, Italy. Association for Computational Linguistics.
  * Ljubešić and Erjavec (2011) Nikola Ljubešić and Tomaž Erjavec. 2011.  hrWaC and slWaC: Compiling web corpora for Croatian and Slovene.  In _International Conference on Text, Speech and Dialogue_ , pages 395–402. Springer.
  * Ljubešić et al. (2019a) Nikola Ljubešić, Tomaž Erjavec, Vuk Batanović, Maja Miličević, and Tanja Samardžić. 2019a.  Croatian twitter training corpus ReLDI-NormTagNER-hr 2.1.  Slovenian language resource repository CLARIN.SI.
  * Ljubešić et al. (2019b) Nikola Ljubešić, Tomaž Erjavec, Vuk Batanović, Maja Miličević, and Tanja Samardžić. 2019b.  Serbian twitter training corpus ReLDI-NormTagNER-sr 2.1.  Slovenian language resource repository CLARIN.SI.
  * Ljubešić and Klubička (2014) Nikola Ljubešić and Filip Klubička. 2014.  $\\{$bs, hr, sr$\\}$ wac-web corpora of Bosnian, Croatian and Serbian.  In _Proceedings of the 9th Web as Corpus Workshop (WaC-9)_ , pages 29–35.
  * Martin et al. (2019) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019.  Camembert: a tasty french language model.  _arXiv preprint arXiv:1911.03894_.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online. Association for Computational Linguistics.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020\.  Stanza: A Python natural language processing toolkit for many human languages.  _arXiv preprint arXiv:2003.07082_.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning.  In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_ , pages 90–95.
  * Scherrer and Ljubešić (2020) Yves Scherrer and Nikola Ljubešić. 2020.  HeLju@ VarDial 2020: Social media variety geolocation with BERT models.  In _Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 202–211.
  * Ulčar and Robnik-Šikonja (2020) Matej Ulčar and Marko Robnik-Šikonja. 2020.  FinEst BERT and CroSloEngual BERT.  In _International Conference on Text, Speech, and Dialogue_ , pages 104–111. Springer.
  * de Vries et al. (2019) Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019.  Bertje: A dutch BERT model.  _arXiv preprint arXiv:1912.09582_.