# CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental
Learning

Qingsong Yan 1, Qiang Wang 2,, Kaiyong Zhao 3, Jie Chen 4,
Bo Li 5, Xiaowen Chu 6,5,11footnotemark: 1, Fei Deng 1,7 Corresponding author

###### Abstract

Neural Radiance Fields (NeRF) have demonstrated impressive performance in
novel view synthesis. However, NeRF and most of its variants still rely on
traditional complex pipelines to provide extrinsic and intrinsic camera
parameters, such as COLMAP. Recent works, like NeRFmm, BARF, and L2G-NeRF,
directly treat camera parameters as learnable and estimate them through
differential volume rendering. However, these methods work for forward-looking
scenes with slight motions and fail to tackle the rotation scenario in
practice. To overcome this limitation, we propose a novel camera parameter
free neural radiance field (CF-NeRF), which incrementally reconstructs 3D
representations and recovers the camera parameters inspired by incremental
structure from motion (SfM). Given a sequence of images, CF-NeRF estimates the
camera parameters of images one by one and reconstructs the scene through
initialization, implicit localization, and implicit optimization. To evaluate
our method, we use a challenging real-world dataset NeRFBuster which provides
12 scenes under complex trajectories. Results demonstrate that CF-NeRF is
robust to camera rotation and achieves state-of-the-art results without
providing prior information and constraints.

## Introduction

3D reconstruction is a hot topic in computer vision that aims to recover 3D
geometry from RGB images. However, traditional methods contain lots of complex
procedures, such as feature extraction and matching (Lowe 2004; Yi et al.
2016), sparse reconstruction (Agarwal et al. 2011; Wu 2013; Schonberger and
Frahm 2016; Moulon et al. 2017), and dense reconstruction (Yao et al. 2018;
Mi, Di, and Xu 2022; Yan et al. 2023). Consequently, traditional methods are
not a differential end-to-end reconstruction pipeline and require high-quality
results from each sub-module to achieve accurate results. When the quality of
results is poor, it is challenging to identify which module is causing the
problem.

(a) NeRFmm

(b) SiRENmm

(c) BARF

(d) GARF

(e) L2G-NeRF

(f) CF-NeRF

Figure 1:  Rendering results on the NeRFBuster We select an image sequence
from NeRFBuster (Warburg et al. 2023) and use novel views synthesis to compare
the quality of camera parameters from NeRFmm (Wang et al. 2021b), SiRENmm (Guo
and Sherwood 2021), BARF (Lin et al. 2021), GARF (Chng et al. 2022), L2G-NeRF
(Chen et al. 2023) and our method CF-NeRF.

Recently, Neural Radiance Fields (NeRF) (Mildenhall et al. 2020; Müller et al.
2022; Yu et al. 2021a; Barron et al. 2021) has demonstrated a novel way to
render highly realistic novel views with impressive quality. Without
recovering 3D geometry, NeRF relies on multi-layer perception (MLP) to predict
color and sigma for each point in the scene and samples several points along a
ray to render a pixel through differential volume rendering. Unlike
traditional 3D reconstruction, NeRF simplifies the reconstruction into one
step and implicitly represents the 3D scene. Benefiting from the excellent
ability of NeRF, it has been further extended to dynamic scenes (Pumarola et
al. 2021), large-scale (Turki, Ramanan, and Satyanarayanan 2022), and even
surface (Wang et al. 2021a) and material reconstruction (Boss et al. 2021a).

Despite the remarkable performance of NeRF and its variants in novel view
synthesis, they still require camera parameters before training. The most
common processing pipeline is first recovering camera parameters using
traditional complex methods like COLMAP (Schonberger and Frahm 2016) or
OpenMVG (Moulon et al. 2017), and then training the NeRF through differential
volume rendering. In other words, the differentiability of the whole
reconstruction pipeline is destroyed and divided into two separate parts,
resulting in the NeRF not being end-to-end and the reconstruction quality
being unidirectionally dependent on traditional methods.

To unify camera parameter estimation and reconstruction, researchers have
tried to recover or optimize camera parameters along with NeRF (Wang et al.
2021b; Lin et al. 2021; Chng et al. 2022; Kuang et al. 2022; Tancik et al.
2023). The straightforward idea is to treat camera parameters as learnable.
NeRFmm (Wang et al. 2021b) proves that it is possible to estimate camera
parameters by NeRF. BARF (Lin et al. 2021) recovers extrinsic camera
parameters and the NeRF model by dynamically adjusting weights of different
frequencies of positional encoding. GARF (Chng et al. 2022) replaces ReLU with
Gaussian activations to obtain high-accuracy results. NeROIC (Kuang et al.
2022) and NeRFStudio (Tancik et al. 2023) optimize camera parameters obtained
from COLMAP and the NeRF simultaneously. However, these methods are only
suitable for forward-looking scenes or scenes with initial camera parameters
and cannot be directly used in the real world with complex movement.

This paper proposes a new end-to-end approach called camera parameter free
NeRF (CF-NeRF) to address the limitations of existing NeRF-based methods in
estimating camera parameters. Figure 7 compares rendered novel views by camera
parameters estimated by NeRFmm (Wang et al. 2021b), SiRENmm (Guo and Sherwood
2021), BARF (Lin et al. 2021), GARF (Chng et al. 2022), L2G-NeRF (Chen et al.
2023), and our method CF-NeRF, where CF-NeRF is the only method that
successfully reconstructs the 3D scene with rotation. Unlike other methods
that simultaneously estimate all camera parameters, CF-NeRF inherits ideas
from incremental structure from motion (SfM) and recovers camera parameters
one by one. CF-NeRF contains three major components: initialization, implicit
localization, and implicit optimization. CF-NeRF first uses initialization to
recover the camera parameters and NeRF by a few images and then gradually
estimates the camera parameters of other images through two steps: the
implicit localization provides an initial camera parameter for the newly added
image, and the implicit optimization optimizes the camera parameter of all
images to reduce drift. Our contributions are as follows:

  1. 1.

We propose a novel end-to-end method, CF-NeRF, that does not need prior
information or constraints to recover the intrinsic and extrinsic camera
parameters and the NeRF simultaneously.

  2. 2.

We design an incremental training pipeline for the CF-NeRF, inspired by the
incremental SfM, to avoid trapping to local minimal and is suitable for
complex trajectories.

  3. 3.

Experiments of our method achieve state-of-the-art results on the NeRFBuster
dataset (Warburg et al. 2023) captured in the real world, proving that the CF-
NeRF can estimate accurate camera parameters with the specifically designed
training procedure.

## Related Work

In this section, we first introduce the development of NeRF-related methods
with known camera parameters. Then, we describe several camera parameter
estimation methods using traditional SfM&SLAM (simultaneous localization and
mapping) and the NeRF.

### NeRF

NeRF (Mildenhall et al. 2020) uses the MLP to represent the 3D scene
implicitly and can be trained through differential volume rendering from a set
of images with known camera parameters. However, NeRF suffers from efficiency
and needs around 1-2 days to train a scene and several minutes to render a
novel view at the testing. Instant-NGP (Müller et al. 2022) builds a multi-
resolution hash table to store space-aware feature vectors and reduces the
complexity of the MLP network. Meanwhile, (Sun, Sun, and Chen 2022; Fridovich-
Keil et al. 2022; Zhang et al. 2022a) try to use the coarse-to-fine strategy
and (Yu et al. 2021a; Chen et al. 2022; Garbin et al. 2021; Wadhwani and
Kojima 2022) update the network structure to speed up training or testing.
Besides, NeRF faces another problem that it cannot work for large-scale,
unbounded 3D scenes. NeRF++ (Zhang et al. 2020) and MipNeRF360 (Barron et al.
2021, 2022) utilize different sampling strategies for foreground and
background to model unbounded 3D scenes by a finite volume. MegaNeRF (Turki,
Ramanan, and Satyanarayanan 2022) and BlockNeRF (Tancik et al. 2022) split a
large scene into multiple small regions and assign a network for each part.
Moreover, (Martin-Brualla et al. 2021; Pumarola et al. 2021; Park et al. 2021;
Attal et al. 2021) extend NeRF to dynamic scenes and (Jain, Tancik, and Abbeel
2021; Yu et al. 2021b; Niemeyer et al. 2022; Kim, Seo, and Han 2022) introduce
context or geometry information into NeRF to suit scenes with sparse views. In
addition to the advances in novel view synthesis, NeRF has made significant
progress in geometric reconstruction (Yariv et al. 2021; Wang, Skorokhodov,
and Wonka 2022; Darmon et al. 2022; Long et al. 2023, 2022; Fu et al. 2022).
UniSURF (Oechsle, Peng, and Geiger 2021) and NeUS (Wang et al. 2021a) estimate
the zero-level set of an implicit signed distance function instead of the
space density. Furthermore, some work (Zhang et al. 2021; Verbin et al. 2022;
Boss et al. 2021a; Kuang et al. 2022; Boss et al. 2021b, 2022) even combines
BRDF and NeRF to decompose a scene into shape, reflectance, and illumination.
However, all of these methods split the reconstruction into two steps and
require traditional methods to provide camera parameters, which significantly
limits the application of NeRF.

Figure 2:  The pipeline of CF-NeRF CF-NeRF can estimate the weight $\theta$ of
NeRF $\mathcal{F}$ and the camera parameter $\delta$. After initializing
through a few selected images, CF-NeRF recovers $\delta$ of the image one by
one through implicit localization that only optimizes the newly added image
and implicit optimization that refines $\theta$ and $\delta$. Implicit
optimization can be divided into partial and global optimization depending on
the number of images used. We visualize $\delta$ reconstructed by CF-NeRF and
sparse points from COLMAP (Schonberger and Frahm 2016) to show that CF-NeRF
can reconstruct rotation in image sequences.

### Camera Parameter Estimation

Traditional SfM (Agarwal et al. 2011; Wu 2013; Chatterjee and Govindu 2013;
Moulon, Monasse, and Marlet 2013; Schonberger and Frahm 2016; Moulon et al.
2017) and SLAM (Mur-Artal, Montiel, and Tardos 2015; Engel, Schöps, and
Cremers 2014; Sumikura, Shibuya, and Sakurada 2019; Engel, Koltun, and Cremers
2017) can estimate camera parameters for given images. However, these methods
divide the reconstruction pipeline into several different modules that need
hand-crafted features (Lowe 2004; Bay, Tuytelaars, and Van Gool 2006) or
learning-based methods (Yi et al. 2016; DeTone, Malisiewicz, and Rabinovich
2018) or the optical flow (Lucas and Kanade 1981; Teed and Deng 2020) to
establish image correspondences, and then reconstruct a sparse scene and
camera parameters through several separate non-differentiable steps.

In light of these limitations, it is worth exploring to estimate camera
parameters during the training process of NeRF. The most direct attempt to
utilize NeRF is the visual localization, where iNeRF (Yen-Chen et al. 2021),
NeDDF (Ueda et al. 2022), and PNeRFP (Lin et al. 2023) try to estimate the
extrinsic camera parameter of a new image by a pre-trained NeRF model. Then,
NeRFmm (Wang et al. 2021b) and SiRENmm (Guo and Sherwood 2021) take the NeRF
and camera parameters as learnable and prove that it is possible to train the
NeRF model from scratch without camera parameters, but they only work for
forward-looking scenes. To further enhance accuracy in forward-looking or
rotation scenes with initial camera parameters, BARF (Lin et al. 2021)
dynamically adjusts the weight of the positional encoding, GARF (Chng et al.
2022) replaces the ReLU activate function with the Gaussian activation
function, and L2G-NeRF (Chen et al. 2023) introduces a local-to-global
registration. Interestingly, GNeRF (Meng et al. 2021) and VMRF (Zhang et al.
2022b) assume there is a prior known distribution of camera parameters to
decrease the freedom of camera parameters during training the NeRF model.
Meanwhile, other researchers try to add different external restrictions to
guide the camera parameter estimation. SCNeRF (Jeong et al. 2021) and
Level-$S^{2}$fM (Xiao et al. 2023) rely on feature matches to guide camera
parameters estimation. NoPe-NeRF (Bian et al. 2023), iMap (Sucar et al. 2021),
NeRF-SLAM (Rosinol, Leonard, and Carlone 2022), Nice-SLAM (Zhu et al. 2022),
and Nicer-SLAM (Zhu et al. 2023) integrate depth maps from active sensors or
CNN networks to tune the NeRF. Additionally, LocalLR (Meuleman et al. 2023)
combines depth maps and optical flow to train NeRF.

Regrettably, images acquired from real-world scenarios often exhibit a
multitude of challenges. These challenges include rotations and the absence of
prior information of camera parameters. Furthermore, the introduction of
external constraints can augment the intricacy and unpredictability of the
reconstruction process. To solve these problems, we propose CF-NeRF inspired
by the traditional incremental SfM, which does not require any prior
information or external constraints while reconstructing the 3D scene and
camera parameters end-to-end from image sequences, demonstrating the powerful
reconstruction capability of the NeRF after using a specific training
strategy.

## Method

In this section, we provide an overview of the proposed method. Firstly, we
introduce the preliminary background of the NeRF and the traditional
incremental SfM. Then, we explain the details of CF-NeRF that can recover
camera parameters from image sequences.

### Preliminary Background

Figure 3:  Estimated Parameters CF-NeRF estimates the weight $\theta$ of NeRF
model and the camera parameter $\delta$, which include the camera rotation
$\delta_{R}$, the camera translation $\delta_{T}$, and camera intrinsic
parameter $\delta_{K}$.

#### NeRF

NeRF can generate realistic images from a set of images
$I=(I_{1},I_{2},...,I_{N})$ from $N$ different places without explicitly
reconstructing. However, NeRF needs associated camera parameters $\delta$,
including camera rotation
$\delta_{R}=(\delta_{R_{1}},\delta_{R_{2}},...,\delta_{R_{N}})$, camera
translation $\delta_{T}=(\delta_{T_{1}},\delta_{T_{2}},...,\delta_{T_{N}})$,
and intrinsic camera parameter $\delta_{K}$. Given a NeRF model $\mathcal{F}$
and corresponding weight $\theta$ , it can estimate color $c$ and density
$\sigma$ through a implicit function
$c(x,\vec{d}),\sigma(x)=\mathcal{F}_{\theta}(x,\vec{d})$ with a point $x$ and
a view direction $\vec{d}$. To render a pixel $p$, NeRF needs to sample
several points $x_{p}(t)=o+\vec{d}t$ along a ray shooting from the view
position $o$ and generate the color $c_{p}$ by the volume rendering function
$\mathcal{R}$ as Eq. 1 shows, where
$\mathcal{T}(t)=exp(-\int_{t_{n}}^{t}\sigma(x_{p}(s))ds)$ indicates the
accumulated transmittance along the ray. $t_{n}$ and $t_{f}$ are the near and
far bounds of the ray.

$\displaystyle
c_{p}=\mathcal{R}(p|\theta)=\int_{t_{n}}^{t_{f}}\mathcal{T}(t)\sigma(x_{p}(t))c(x_{p}(t),\vec{d})dt$
(1)

Benefiting from the differential property of the volume rendering, NeRF can be
trained end to end by minimizing the difference between $c_{p}$ and observed
color $I(p)$ as Eq. 2 shows, where $\mathcal{L}$ is the loss function. To be
noted, NeRF only estimates $\theta$ and borrows $\delta$ from traditional SfM
methods. However, NeRFmm (Wang et al. 2021b) prove that it is possible to
estimate $\theta$ and $\delta$ simultaneously under the forward-looking
situation.

$\displaystyle{\underset{\theta}{{\arg\min}\,}\\{\sum_{I_{i}\in I}\sum_{p\in
I_{i}}{\mathcal{L}(\mathcal{R}(p|\theta),I_{i}(p))}\\}}$ (2)

#### Incremental SfM

Given a set of images, the incremental SfM can recover $\delta$ one by one in
a linear time (Wu 2013) and contains four steps (Schonberger and Frahm 2016):

Initialization The selection of an initial two-view is essential because a
suitable initial two-view improves the robustness and quality of the
reconstruction. With a given two-view and its matched features, incremental
SfM computes the relative pose by multi-view geometry (MVG) and triangulates
3D points to initial the scene.

Image Registration After initialization, incremental SfM adds images to the
scene in order. Given a new image, incremental SfM builds the 2D-3D
relationship by matching its features with images in the scene and recovers
the camera parameter by Perspective-n-Point (PnP).

Triangulation As a newly added image observes additional information that can
extend the scale of the scene, incremental SfM triangulates more 3D points
based on the new image and matched features.

Bundle Adjustment Adding new images and 3D points without refinement leads to
drift. Therefore, it is essential to apply bundle adjustment (BA) by
minimizing the re-projection error. In terms of efficiency, incremental SfM
proposes partial BA that refines only a subset of images, and global BA that
optimizes all images.

### CF-NeRF

Fusing the differentiability of NeRF and the reconstruction strategy of SfM,
we propose CF-NeRF, which is capable of estimating the camera parameter under
complex movement from sequential images. CF-NeRF consists of three modules:
initialization, implicit localization, and implicit optimization, as Figure 2
shows. To convenient later introduction, we define the set of images we have
completed estimating the camera parameter as $E$, which starts from
$\varnothing$.

#### Parameter

CF-NeRF estimates the camera parameter $\delta$ and the weight $\theta$ of
NeRF, as Figure 3 shows. The $\delta$ includes $\delta_{R}$, $\delta_{T}$, and
$\delta_{K}$. During the differential volume rendering, we calculate the ray
$\vec{r}_{p}(t)=\delta_{T_{i}}+\delta_{R_{i}}\delta_{K_{i}}^{-1}\tilde{p}t$ of
pixel $p$ in image $I_{i}\in I$, where $\tilde{p}$ is the homogeneous
expression of $p$. Following NeRFmm (Wang et al. 2021b), we use the axis-angle
to represent $\delta_{R}$ and assume all images have the same camera intrinsic
parameter without distortion so that $\delta_{K}$ only contains the focal
length. We initialize $\delta_{R}$ and $\delta_{T}$ to zero, and set
$\delta_{K}$ to $53^{\circ}$ by a common field of view. The $\theta$ is the
weight of $\mathcal{F}$ that implicitly represents the scene. The activation
function determines how to initialize $\theta$. NeRF using ReLU are
initialized according to NeRF (Mildenhall et al. 2020), while NeRF using sine
are initialized according to SIREN (Sitzmann et al. 2020).

(a) GT

(b) NeRFmm

(c) SiRENmm

(d) BARF

(e) GARF

(f) L2G-NeRF

(g) CF-NeRF

Figure 4:  Qualitative results on NeRFBuster. We select two sequences from
NeRFBuster (Warburg et al. 2023) and render novel views to compare the quality
of camera parameters. Our method CF-NeRF generate high-quality images, while
results of NeRFmm (Wang et al. 2021b), SiRENmm (Guo and Sherwood 2021), BARF
(Lin et al. 2021), GARF (Chng et al. 2022) and L2G-NeRF (Chen et al. 2023)
contain lots of noise.

#### Initialization

Similar to incremental SfM, CF-NeRF requires initialize $\theta$,
$\delta_{R_{1}}$, $\delta_{T_{1}}$, and $\delta_{K}$ before adding images to
$E$. We select the first $N_{init}$ images $I_{init}$ from $I$ to optimise
these parameters by Eq. 3 with $\xi_{init}$ iterations. Since the rotation
between adjacent images is not large and NeRF is hard to estimate rotation
(Lin et al. 2021; Chng et al. 2022; Chen et al. 2023), we do not estimate the
rotation in the initialization to reduce the freedom. After initialization, we
add $I_{1}$ to $E$ and keep $\theta$, $\delta_{R_{1}}$, $\delta_{T_{1}}$, and
$\delta_{K}$ but discard other camera parameters. Note that, unlike the
initialization in the previous section, the initialization here is data-
specific, similar to the warm-up procedure.

$\displaystyle{\underset{\theta,\delta_{T},\delta_{K}}{{\arg\min}\,}\\{\sum_{I_{i}\in
I_{init}}\sum_{p\in
I_{i}}{\mathcal{L}(\mathcal{R}(p|\theta,\delta_{T_{i}},\delta_{K}),I_{i}(p))\\}}}$
(3)

#### Implicit Localization

After initialization, CF-NeRF estimates the camera parameter of the remaining
images one by one and determines $\delta_{R_{n}}$ and $\delta_{T_{n}}$ for
each new image $I_{n}$ by localization. Specifically, we first initialize
$\delta_{R_{n}}$ and $\delta_{T_{n}}$ by $\delta_{R_{n-1}}$ and
$\delta_{T_{n-1}}$, and then optimize them by minimizing Eq. 4 with fixed
$\theta$ through $\xi_{loc}$ iterations. The localization is similar to iNeRF
(Yen-Chen et al. 2021), but CF-NeRF does not have a pre-trained $\mathcal{F}$.

$\displaystyle{\underset{\delta_{R_{n}},\delta_{T_{n}}}{{\arg\min}\,}\\{\sum_{p\in
I_{n}}{\mathcal{L}(\mathcal{(}p|\delta_{R_{n}},\delta_{T_{n}}}),I_{n}(p))}\\}$
(4)

#### Implicit Optimization

Although implicit localization can roughly determine $\delta_{R_{n}}$ and
$\delta_{T_{n}}$, it faces two problems: the observation from $I_{n}$ is not
added to NeRF, and the localization does not take the multi-view consistency
into account to reduce drift. Incremental SfM solves these problems using two
separate steps: triangulation and BA, while CF-NeRF benefits from the volume
rendering and deals with these problems together. However, it is time-
consuming to optimize all images in $E$ every time a new image is added.
Therefore, CF-NeRF splits optimization into implicit partial optimization and
implicit global optimization.

Each time localizing a new image $I_{n}$, CF-NeRF performs implicit partial
optimization. We select $I_{n}$ and previous $N_{part}-1$ images to construct
the partial image set $I_{part}$, then optimizes them with $\xi_{part}$
iterations, as Eq.5 shows.

$\displaystyle{\underset{\theta,\delta_{R},\delta_{T}}{{\arg\min}\,}\\{\sum_{I_{i}\in
I_{part}}\sum_{p\in
I_{i}}{\mathcal{L}(\mathcal{R}(p|\theta,\delta_{R_{i}},\delta_{T_{i}}),I_{i}(p))\\}}}$
(5)

When the number of images in $E$ can be evenly divided by $N_{glob}$, CF-NeRF
employs implicit global optimization for $\theta$ and all images in $E$ to
enhance the overall accuracy and reduce drifts with $\xi_{glob}$ iterations,
as Eq. 6 shows.

$\displaystyle{\underset{\theta,\delta_{R},\delta_{T},\delta_{K}}{{\arg\min}\,}\\{\sum_{I_{i}\in
I_{E}}\sum_{p\in
I_{i}}{\mathcal{L}(\mathcal{R}(p|\theta,\delta_{R_{i}}\delta_{T_{i}},\delta_{K}),I_{i}(p))\\}}}$
(6)

Table 1:  We quantitatively evaluated on 12 scenes from the NeRFBuster dataset
(Warburg et al. 2023), which is captured in the real world with complex
trajectories, including rotations and forward-looking. CF-NeRF achieves state-
of-the-art in $\Delta R$, $\Delta T$, PSNR, and LPIPS compared to NeRFmm (Wang
et al. 2021b), SiRENmm (Guo and Sherwood 2021), BARF (Lin et al. 2021), GARF
(Chng et al. 2022), L2G-NeRF (Chen et al. 2023).

|  | NeRFBuster
---|---|---
|  | aloe | art | car | century | flowers | garbage | picnic | pikachu | pipe | plant | roses | table
$\Delta R\downarrow$ | NeRFmm | 159.9731 | 177.5907 | 129.5799 | 119.6261 | 106.9195 | 150.8233 | 154.7781 | 113.7001 | 164.8207 | 165.0298 | 102.2748 | 115.2991
SiRENmm | 155.1513 | 177.3642 | 127.2673 | 89.0172 | 103.8741 | 82.9375 | 44.3671 | 25.3603 | 159.7565 | 114.0761 | 132.5378 | 93.2612
BARF | 158.6689 | 59.1868 | 133.4527 | 101.6013 | 88.6842 | 88.7832 | 69.7201 | 41.0302 | 64.5250 | 143.1975 | 133.7586 | 111.2884
GARF | 125.9797 | 171.9169 | 153.5585 | 105.1868 | 106.0595 | 84.4992 | 49.8503 | 32.7285 | 126.9602 | 156.6058 | 118.9748 | 164.6562
L2G-NeRF | 124.2368 | 24.3753 | 55.7291 | 131.9683 | 96.3498 | 110.4782 | 146.3119 | 116.8911 | 70.9131 | 70.7562 | 95.9982 | 116.4972
CF-NeRF | 12.1226 | 19.2496 | 17.5570 | 9.6811 | 8.2556 | 9.7658 | 12.6501 | 11.3067 | 19.9926 | 4.8968 | 5.1229 | 4.5837
$\Delta T\downarrow$ | NeRFmm | 11.5935 | 15.0762 | 23.9514 | 24.5934 | 12.8753 | 16.3842 | 12.9675 | 25.6841 | 19.3563 | 23.6613 | 8.0367 | 13.3849)
SiRENmm | 11.4912 | 14.8720 | 27.7235 | 28.8582 | 15.4841 | 13.3099 | 8.3607 | 15.8052 | 20.4572 | 31.2943 | 9.0498 | 13.8350
BARF | 9.6196 | 17.3299 | 36.0351 | 26.0549 | 15.3166 | 17.1936 | 9.6679 | 18.7184 | 18.8936 | 31.8629 | 8.5266 | 17.5453
GARF | 13.9100 | 17.1527 | 26.5184 | 26.0096 | 14.1459 | 16.1054 | 11.7021 | 17.4975 | 18.8366 | 32.3170 | 11.2654 | 15.0367
L2G-NeRF | 14.4012 | 13.4240 | 20.5634 | 23.2650 | 7.4559 | 17.7167 | 12.9408 | 32.4048 | 11.4012 | 18.5012 | 10.7110 | 12.8061
CF-NeRF | 3.3788 | 2.2821 | 6.5452 | 2.7383 | 2.7026 | 4.0535 | 1.2833 | 4.0586 | 9.4491 | 3.4346 | 1.1945 | 1.2127
PSNR$\uparrow$ | NeRFmm | 20.6912 | 16.8220 | 17.7504 | 16.0326 | 18.4377 | 17.7229 | 18.4300 | 25.3819 | 20.0978 | 21.1558 | 17.7735 | 14.1651
SiRENmm | 22.7462 | 20.3890 | 22.0268 | 18.0252 | 19.4640 | 17.2283 | 21.5628 | 27.8706 | 20.9538 | 23.4980 | 16.7480 | 18.8135
BARF | 22.4366 | 21.1947 | 16.7665 | 15.3436 | 17.8350 | 15.9065 | 19.1846 | 23.0386 | 19.9728 | 25.5135 | 13.6741 | 13.8227
GARF | 19.0241 | 19.3556 | 15.4460 | 14.4117 | 16.2955 | 15.3383 | 15.4035 | 20.9663 | 18.5371 | 20.5600 | 13.1274 | 12.6677
L2G-NeRF | 21.3398 | 19.8099 | 17.3255 | 16.6476 | 18.0016 | 13.6077 | 18.5268 | 22.4939 | 18.1787 | 19.0160 | 17.2614 | 15.5658
CF-NeRF | 26.9367 | 26.5293 | 22.4654 | 21.7072 | 21.6950 | 22.4736 | 22.5475 | 32.3661 | 22.2719 | 25.7312 | 24.3918 | 26.8491
LPIPS$\downarrow$ | NeRFmm | 0.5560 | 0.4954 | 0.5991 | 0.5793 | 0.5778 | 0.5661 | 0.6113 | 0.3683 | 0.5614 | 0.4927 | 0.5371 | 0.6073
SiRENmm | 0.4508 | 0.4034 | 0.4450 | 0.4785 | 0.5048 | 0.5193 | 0.5227 | 0.2883 | 0.5170 | 0.3256 | 0.5333 | 0.4659
BARF | 0.3328 | 0.3511 | 0.5361 | 0.5394 | 0.5552 | 0.5480 | 0.5358 | 0.3440 | 0.5198 | 0.3217 | 0.6138 | 0.5913
GARF | 0.5257 | 0.4055 | 0.5984 | 0.5845 | 0.6158 | 0.5931 | 0.6086 | 0.3987 | 0.5688 | 0.4345 | 0.6189 | 0.6356
L2G-NeRF | 0.4620 | 0.4186 | 0.5409 | 0.5116 | 0.5466 | 0.6016 | 0.5530 | 0.4051 | 0.4741 | 0.3840 | 0.4788 | 0.5309
CF-NeRF | 0.1939 | 0.2316 | 0.3983 | 0.3627 | 0.3983 | 0.3859 | 0.4686 | 0.1679 | 0.4453 | 0.2594 | 0.2831 | 0.3011

#### Coarse-to-Fine

CF-NeRF uses a coarse-to-fine strategy to improve robustness. CF-NeRF first
constructs a Gaussian pyramid with depth $d_{G}$, then recovers all parameters
at a low-resolution image through the incremental pipeline. Finally, CF-NeRF
directly performs implicit global optimization with a higher resolution in
each scale of the Gaussian pyramid with $\xi_{G}$ iterations.

#### Loss Function

To improve robustness, we employ the Smooth-L1 loss function, as Eq. 7 shows,
where $gt$ represents the ground truth, $pr$ is the estimated value, and
$\beta$ is the set to 1.0 by default.

$\mathcal{L}(pr,gt)=\begin{cases}0.5*(gt-pr)^{2}/\beta&if|gt-pr|<\beta\\\ |gt-
pr|-0.5*\beta&otherwise\\\ \end{cases}$ (7)

## Experiments

### Dataset

Instead of using the forward-looking dataset LLFF (Mildenhall et al. 2020) as
usually conducted in the previous studies, we use a real-world dataset from
NeRFBuster (Warburg et al. 2023), mainly rotating around an object, to
evaluate our method. We sample around 50 frames for each scene and resize all
images to $480\times$270\. We use COLMAP (Schonberger and Frahm 2016) to
estimate ground truth (GT) camera parameters for all scenes. More details of
the dataset can be found in the supplementary material.

### Implementation

CF-NeRF is implemented using PyTorch. Similar to NeRFmm (Wang et al. 2021b),
CF-NeRF does not have hierarchical sampling and uses the coarse network, which
has eight layers and the dimension of the hidden layers is set to 128. And we
use the sine activation function instead of the ReLU, as SiRENmm (Guo and
Sherwood 2021) is more robust than NeRFmm. We utilize the Adam optimizer
(Kingma and Ba 2014) to optimize all learnable parameters. Specifically, we
set the learning rate of $\theta$ to 0.001, which undergoes a decay of 0.9954
every 200 epochs. Similarly, the learning rate of $\delta$ is set to 0.001 and
undergoes a decay of 0.9000 every 2000 epochs. Here we describe how to set the
hyper-parameters in CF-NeRF. We set $N_{init}$ and $N_{part}$ to 3 to meet the
minimum requirements that can filter outliers based on MVG. To balance drift
and efficiency, we set $N_{glob}$ to 5. Considering the input image
resolution, we set $d_{G}$ to 3 to reconstruct all parameters by coarse-to-
fine strategy. The most important parameter in CF-NeRF is iteration, which is
the epoch number for each image. During initialization, we set $\xi_{init}$ to
3000 to guarantee that $\theta$ and $\delta$ can be correctly initialized with
fewer images. Subsequently, during the incremental training, we maintain a
consistent value of $\xi$, setting
$\xi=\xi_{loc}=\xi_{part}=\xi_{glob}=\xi_{G}$ to 900, thus reconstructing the
scene from images one by one. Throughout all our experiments, we use the
NVIDIA RTX3090.

Table 2:  Ablation experiments. We compare the accuracy of camera parameter
estimation of CF-NeRF under different hyper-parameter settings, including the
iteration $\xi$, the global optimization frequency $N_{glob}$ and the coarse-
to-fine strategy, where $C$ means the coarse stage and $F$ means the fine
stage.

|  | NeRFBuster
---|---|---
|  | aloe | art | car | century | flowers | garbage | picnic | pikachu | pipe | plant | roses | table
$\Delta R\downarrow$ | $F,\xi=600,N_{glob}=10$ | 17.8029 | 24.3389 | 17.1692 | 11.5924 | 11.6163 | 9.1240 | 14.6452 | 13.0037 | 19.0749 | 5.3354 | 5.9091 | 6.4731
$F,\xi=900,N_{glob}=10$ | 14.8730 | 22.8142 | 17.8879 | 11.1201 | 10.4707 | 8.6973 | 11.4209 | 12.0625 | 18.4305 | 4.7303 | 5.8538 | 6.8481
$C,\xi=900,N_{glob}=5$ | 12.4862 | 19.1647 | 17.4755 | 9.7177 | 8.4555 | 9.6460 | 12.3162 | 10.9802 | 19.9855 | 5.1579 | 5.5133 | 5.2821
$F,\xi=900,N_{glob}=5$ | 12.1226 | 19.2496 | 17.5570 | 9.6811 | 8.2556 | 9.7658 | 12.6501 | 11.3067 | 19.9926 | 4.8968 | 5.1229 | 4.5837
$\Delta T\downarrow$ | $F,\xi=600,N_{glob}=10$ | 4.5457 | 5.9307 | 7.5697 | 2.9652 | 3.6234 | 4.5340 | 2.6677 | 5.3105 | 9.3544 | 4.4384 | 1.3324 | 1.9535
$F,\xi=900,N_{glob}=10$ | 3.9111 | 6.1190 | 7.3752 | 3.6834 | 3.3956 | 4.3080 | 3.4918 | 3.2682 | 8.4666 | 3.6109 | 1.3013 | 2.3973
$C,\xi=900,N_{glob}=5$ | 3.4681 | 2.2770 | 6.6250 | 2.8224 | 2.7405 | 4.1085 | 1.2886 | 4.2462 | 9.6998 | 3.5309 | 1.2182 | 1.2232
$F,\xi=900,N_{glob}=5$ | 3.3788 | 2.2821 | 6.5452 | 2.7383 | 2.7026 | 4.0535 | 1.2833 | 4.0586 | 9.4491 | 3.4346 | 1.1945 | 1.2127

### Evaluation

To demonstrate the performance of the proposed method, we conduct a
comprehensive comparison between CF-NeRF and several state-of-the-art models,
including NeRFmm (Wang et al. 2021b) SiRENmm (Guo and Sherwood 2021), BARF
(Lin et al. 2021), GARF (Chng et al. 2022), and L2G-NeRF (Chen et al. 2023).
We use all images for camera parameter estimation without employing a
train/test split. To evaluate the quality of the camera parameters, we
calculate the average translation error $\Delta T$ and the average rotation
error $\Delta R$ by aligning the estimated camera parameters $\delta_{R}$ and
$\delta_{T}$ with COLMAP using a similarity transformation Sim(3) (Lin et al.
2021). It is worth noting that $\delta_{T}$ represents a relative translation
error rather than an absolute measurement, as COLMAP can not reconstruct an
absolute scale of the scene. We further evaluate the estimated camera
parameters through a novel view synthesis by PSNR and LPIPS. To ensure a fair
comparison and avoid the influence of varying network backbones across
different methods, we uniformly use the NerfAcc (Li, Tancik, and Kanazawa
2022), where we select one image for testing in every eight images and the
remaining is for training.

### Results

We performed qualitative and quantitative evaluations of these methods on 12
scenes of the NeRFBuster (Warburg et al. 2023) dataset. Notably, BARF (Lin et
al. 2021), GARF (Chng et al. 2022), and L2G-NeRF (Chen et al. 2023) require
manual setting the focal length. In contrast, NeRFmm (Wang et al. 2021b),
SiRENmm (Guo and Sherwood 2021), and CF-NeRF have the ability to estimate the
focal length.

Table 3 shows the results of qualitative experiments. Our method obtains the
highest accuracy camera parameters, while all other methods fail outright. It
is important to understand that $\Delta R$ and $\Delta T$ are calculated by
aligning the camera positions with Sim(3) and that a slight difference in
camera position can lead to huge errors. The rotation error $\Delta R$ of our
method CF-NeRF is roughly around $10^{\circ}$, while the other methods are
around $100^{\circ}$. Moreover, the translation error $\delta_{T}$ of CF-NeRF
is approximately about 4, while all other methods are around 15. Although
NeRFmm, SiRENmm, BARF, GARF, and L2G-NeRF claim high accuracy on forward-
looking scenes from scratch, they are unsuitable for scenes with rotation and
are prone to be trapped in a local minimum. In contrast, CF-NeRF recovers the
camera parameters sequentially and can effectively handle image sequences with
complex trajectories. Furthermore, SiRENmm outperforms NeRFmm in camera
parameter estimation, which is why CF-NeRF uses the sine activate function.

Table 3 also shows the quality of the novel view synthesis, which serves as an
additional evaluation criterion for the quality of camera parameters. CF-NeRF
achieves state-of-the-art results on PSNR and LPIPS, whereas other methods
struggle to generate high-quality results across multiple novel views.
Interestingly, the reconstruction results of other methods appear reasonable
compared to their poor camera parameters, mainly due to the high over-fitting
ability of NeRF and the fact that partial camera parameters are correctly
reconstructed. We further visualize the rendering results of three scenes from
different methods in Figure 7 and Figure 4. CF-NeRF can generate high-quality
results, while other methods have lots of noise in their results due to their
inability to provide accurate camera parameters for all images. More
visualization results are in supplementary.

### Ablation Experiments

To understand the influence of hyper-parameters in CF-NeRF, we conduct several
ablation experiments on the iteration $\xi$, the global optimization frequency
$N_{glob}$, and the coarse-to-fine strategy. These experiments are performed
on the NeRFBuster dataset (Warburg et al. 2023), and results are presented in
Table 2.

#### The iteration $\xi$

The iteration $\xi$ is the most important hyper-parameter in our method,
determining how many times to optimize the camera parameter for each image. We
compare two configurations: $F,\xi=600,N_{glob}=10$ and
$F,\xi=900,N_{glob}=10$. Table 2 reveals that increasing $\xi$ improves the
final results for almost all scenes. This observation aligns with NeRF
(Mildenhall et al. 2020) and iNeRF (Yen-Chen et al. 2021), where NeRF requires
a large number of iterations to converge, and iNeRF enhances the quality of
camera parameters through more iterations.

#### The global optimization frequency $N_{glob}$

To mitigate drift while maintaining efficiency, CF-NeRF employs the implicit
global optimization when every $N_{glob}$ image is added $E$. We conduct two
experiments $F,\xi=900,N_{glob}=10$ and $F,\xi=900,N_{glob}=5$ to find out the
influence of $N_{glob}$. As highlighted in Table 2, reducing $N_{glob}$ yields
improved final results, which can be attributed to the fact that global
optimization ensures global consistency to avoid NeRF trap into a local
minimum.

#### The coarse-to-fine stragety

CF-NeRF adopts a coarse-to-fine strategy to avoid directly estimating camera
parameters on high-resolution images, where the fine stage refines initial
results from the coarse stage. We conduct two experiments
$C,\xi=900,N_{glob}=5$ and $F,\xi=900,N_{glob}=5$ , where $C$ means the coarse
stage and $F$ means the fine stage. Results in Table 2 demonstrate that the
fine stage outperforms the coarse stage across almost all scenes. Adopting the
coarse-to-fine strategy facilitates the training process of CF-NeRF, as the
pixel gradient is smoother at the coarse stage and has less RGB information to
learn.

### Limitation

Although CF-NeRF achieves state-of-the-art results in camera parameter
estimation, surpassing other NeRF-based methods, there are still some gaps
between CF-NeRF and COLMAP (Schonberger and Frahm 2016), and the accuracy can
be further improved through the adjustment of the sample space (Wang et al.
2023) or the utilization of a more robust function (Sabour et al. 2023).

## Conclusion

This paper presents CF-NeRF, a novel end-to-end method that does not require
prior camera parameters to deal with image sequences with complex
trajectories. Following the pipeline of incremental SfM, CF-NeRF contains
three major sub-modules: initialization, implicit localization, and implicit
optimization. Experiments on the NeRFBuster dataset demonstrate that CF-NeRF
achieves state-of-the-art results, while NeRFmm, SiRENmm, BARF, GARF, and
L2G-NeRF only work for forward-looking scenes and get trapped in the local
minimum on the NeRFBuster dataset. More importantly, CF-NeRF highlights the
unlimited potential of NeRF and differential volume rendering, showing that
NeRF has impressive reconstruction capabilities and can also be used to
estimate camera parameters in complex trajectories.

#### Acknowledgments

The research was supported in part by a RGC RIF grant under the contract
R6021-20, RGC CRF grants under the contracts C7004-22G and C1029-22G, and RGC
GRF grants under the contracts 16209120, 16200221, and 16207922. This research
was also supported by the National Natural Science Foundation of China (No.
62302126), and the Shenzhen Science and Technology Program (No.
RCBS20221008093125065, No. JCYJ20220818102414030).

Supplementary material

## How to use COLMAP

COLMAP (Schonberger and Frahm 2016) is a widely used 3D reconstruction
software that can produce stable reconstruction results, providing the
intrinsic and extrinsic camera parameters and the sparse 3D point clouds. In
this paper, we use COLMAP to process all datasets and use these results as the
ground truth (GT) to evaluate camera parameters estimated by different
methods. However, we make some modifications to the default settings of COLMAP
to align with the unique characteristics of the datasets.

  1. 1.

We assume no distortions in images and use the pinhole camera model that only
considers the focal length.

  2. 2.

We upsample the image twice during feature extraction to ensure that COLMAP
can extract enough feature points with traditional feature extraction
algorithms.

  3. 3.

We use the sequential matching method to match images, which is much faster
than the exhaustive matching method and more suitable for the incremental
structure from motion (SfM) for an image sequence.

##  Details of the NeRFBuster dataset

In all experiments, we use the challenging NeRFBuster dataset (Warburg et al.
2023), which is captured in the real world. This dataset contains 12 scenes,
including aloe, art, car, century, flowers, garbage, picnic, pikachu, pipe,
plant, flowers, and table, as Figure 5 shows. Furthermore, we present camera
parameters and sparse point clouds results obtained from COLMAP in Figure 6.
Unlike the LLFF dataset (Mildenhall et al. 2020), which is limited to forward-
looking motion, the NeRFBuster dataset is more challenging for camera
parameters estimation, because it is captured under complex movement and
contains a variety of objects.

Figure 5:  12 scenes from NeRFBuster We show one image from each scene in the
NeRFBuster dataset (Warburg et al. 2023), which contains various objects from
indoor or outdoor environments.  Figure 6:  Sparse reconstruction results of
the NeRFBuster The NeRFBuster dataset (Warburg et al. 2023) is captured under
complex trajectories and mainly rotates around objects.

## How to use NeRFacc

Instead of only evaluating the camera parameter by comparing the camera
translation and the camera rotation, we also indirectly evaluate the camera
parameter by novel view synthesis. Because we use all images during camera
parameter estimation and different methods use different backbones, we
uniformly use NeRFAcc (Li, Tancik, and Kanazawa 2022) to measure Peak Signal-
to-Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS).
For each scene in NeRFBuster, we follow these steps:

  1. 1.

We align the estimated camera parameters to the GT camera parameters provided
by COLMAP through a similarity transformation Sim(3) (Lin et al. 2021).

  2. 2.

We split the images in each scene into a training/split set, with one-eighth
of the images as test images and the rest as training images.

  3. 3.

We use the bounding box of sparse points provided by the COLMAP and utilize
NeRFAcc to reconstruct the scene with the aligned camera parameters
implicitly.

  4. 4.

We train the model for 20000 epochs on the train split and evaluate PSNR and
LPIPS on the test split.

During the training process, we employ the default settings from NeRFAcc and
utilize the MipNeRF (Barron et al. 2021) as the backbone, which is motivated
by the abundance of open-space scenes present in the NeRFBuster dataset.

## How to align camera parameters

We follow BARF (Lin et al. 2021) to align two camera parameters through
Sim(3), where only the camera translation is used. However, we find this
method is unreliable when all cameras lie in a line or the camera translation
contains noise. To overcome this problem and improve robustness, we add a
virtual point $(0,0,1)$ in the camera coordinate of each image and use the
camera parameter to transform it to the world coordinate, thus using the
camera rotation during the alignment process.

## Visualization Results

We qualitatively compare the camera parameters of our method CF-NeRF with
state-of-the-art methods NeRFmm (Wang et al. 2021b), SiRENmm (Guo and Sherwood
2021), BARF (Lin et al. 2021), GARF (Chng et al. 2022), and L2G-NeRF (Chen et
al. 2023) on the NeRFBuster dataset (Warburg et al. 2023). Figure 7 shows the
camera parameter estimation results of the different methods as well as the
sparse points of COLMAP, and it is not difficult to notice that only our CF-
NeRF generates correct results, while the other methods generate entirely
wrong results. For a full comparison, we also provide a video in the
supplementary, using NeRFAcc to render the video according to the camera
parameters of the different methods.

Table 3:  Quantitative evaluation results. We evaluate our method on LLFF and
Real360 from NeRF. BARF (Lin et al. 2021) needs a pre-calculated focal length
from COLMAP. NeRFmm (Wang et al. 2021b) and CF-NeRF (ours) do not need prior
camera parameters.

scene | rotation$\downarrow$ | translation$\downarrow$ | PSNR$\uparrow$
---|---|---|---
| NF$-$ | ours | BARF | NF$-$ | ours | BARF | NF$-$ | ours | BARF
fern | 1.2724 | 0.8669 | 0.2244 | 0.2469 | 0.2218 | 0.0811 | 23.71 | 26.38 | 27.19
flower | 4.5220 | 10.7385 | 1.1812 | 0.6570 | 0.5587 | 0.0482 | 23.04 | 26.80 | 29.06
fortress | 174.296 | 3.0570 | 0.2691 | 6.0263 | 1.3652 | 0.0972 | 9.62 | 27.19 | 26.92
horns | 4.3958 | 3.3078 | 0.2477 | 0.7677 | 1.1096 | 0.0644 | 18.74 | 26.71 | 29.29
leaves | 0.4504 | 11.2940 | 1.7657 | 0.2216 | 0.8991 | 0.1823 | 18.94 | 20.88 | 20.12
orchids | 4.7278 | 0.1951 | 0.2309 | 0.2728 | 0.0703 | 0.0641 | 14.79 | 22.26 | 22.21
room | 3.1178 | 3.7610 | 0.2552 | 0.3987 | 2.2731 | 0.1712 | 24.20 | 28.76 | 32.63
trex | 5.0095 | 0.6298 | 1.0239 | 0.3608 | 0.4072 | 0.3525 | 23.61 | 28.37 | 28.86
vasedeck | 148.9928 | 6.0560 | 165.0392 | 18.6165 | 0.3143 | 12.2520 | 16.64 | 23.63 | 14.74
pinecone | 122.9956 | 0.4887 | 42.6464 | 29.5298 | 0.1387 | 8.2233 | 10.66 | 23.80 | 10.88

## Experiments on LLFF

Throughout our paper, we use the NeRFBuster, since it is a real-world dataset
captured by mobile phones and contains quite complex trajectories, whereas the
commonly used LLFF contains only forward views with simple trajectories.
However, to fully compare wtih other methods, we conduct experiments on LLFF
as well. As shown in Table 3, our method achieves competitive results to
NeRFmm and BARF (with pre-calculated focal length) in the eight forward-
looking scenes of LLFF. However, for the two scenes with rotation in NeRF360,
our method achieves much better results than NeRFmm and BARF.

(a) GT

(b) NeRFmm

(c) SiRENmm

(d) BARF

(e) GARF

(f) L2G-NeRF

(g) CF-NeRF

Figure 7:  Comparison of camera parameters. We compare camera parameters of
our method CF-NeRF with NeRFmm (Wang et al. 2021b), SiNeRF (Guo and Sherwood
2021), BARF (Lin et al. 2021), GARF (Chng et al. 2022), and L2G-NeRF (Chen et
al. 2023) along with sparse points from COLMAP.

## References

  * Agarwal et al. (2011) Agarwal, S.; Furukawa, Y.; Snavely, N.; Simon, I.; Curless, B.; Seitz, S. M.; and Szeliski, R. 2011.  Building rome in a day.  _Communications of the ACM_ , 54(10): 105–112.
  * Attal et al. (2021) Attal, B.; Laidlaw, E.; Gokaslan, A.; Kim, C.; Richardt, C.; Tompkin, J.; and O’Toole, M. 2021.  Törf: Time-of-flight radiance fields for dynamic scene view synthesis.  _Advances in neural information processing systems_ , 34: 26289–26301.
  * Barron et al. (2021) Barron, J. T.; Mildenhall, B.; Tancik, M.; Hedman, P.; Martin-Brualla, R.; and Srinivasan, P. P. 2021.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5855–5864.
  * Barron et al. (2022) Barron, J. T.; Mildenhall, B.; Verbin, D.; Srinivasan, P. P.; and Hedman, P. 2022.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5470–5479.
  * Bay, Tuytelaars, and Van Gool (2006) Bay, H.; Tuytelaars, T.; and Van Gool, L. 2006.  Surf: Speeded up robust features.  _Lecture notes in computer science_ , 3951: 404–417.
  * Bian et al. (2023) Bian, W.; Wang, Z.; Li, K.; Bian, J.-W.; and Prisacariu, V. A. 2023.  Nope-nerf: Optimising neural radiance field with no pose prior.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4160–4169.
  * Boss et al. (2021a) Boss, M.; Braun, R.; Jampani, V.; Barron, J. T.; Liu, C.; and Lensch, H. 2021a.  Nerd: Neural reflectance decomposition from image collections.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 12684–12694.
  * Boss et al. (2022) Boss, M.; Engelhardt, A.; Kar, A.; Li, Y.; Sun, D.; Barron, J. T.; Lensch, H.; and Jampani, V. 2022.  SAMURAI: Shape and material from unconstrained real-world arbitrary image collections.  _arXiv preprint arXiv:2205.15768_.
  * Boss et al. (2021b) Boss, M.; Jampani, V.; Braun, R.; Liu, C.; Barron, J.; and Lensch, H. 2021b.  Neural-pil: Neural pre-integrated lighting for reflectance decomposition.  _Advances in Neural Information Processing Systems_ , 34: 10691–10704.
  * Chatterjee and Govindu (2013) Chatterjee, A.; and Govindu, V. M. 2013.  Efficient and robust large-scale rotation averaging.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 521–528.
  * Chen et al. (2022) Chen, A.; Xu, Z.; Geiger, A.; Yu, J.; and Su, H. 2022.  Tensorf: Tensorial radiance fields.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , 333–350. Springer.
  * Chen et al. (2023) Chen, Y.; Chen, X.; Wang, X.; Zhang, Q.; Guo, Y.; Shan, Y.; and Wang, F. 2023.  Local-to-global registration for bundle-adjusting neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8264–8273.
  * Chng et al. (2022) Chng, S.-F.; Ramasinghe, S.; Sherrah, J.; and Lucey, S. 2022.  Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation.  In _European Conference on Computer Vision_ , 264–280. Springer.
  * Darmon et al. (2022) Darmon, F.; Bascle, B.; Devaux, J.-C.; Monasse, P.; and Aubry, M. 2022.  Improving neural implicit surfaces geometry with patch warping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6260–6269.
  * DeTone, Malisiewicz, and Rabinovich (2018) DeTone, D.; Malisiewicz, T.; and Rabinovich, A. 2018.  Superpoint: Self-supervised interest point detection and description.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 224–236.
  * Engel, Koltun, and Cremers (2017) Engel, J.; Koltun, V.; and Cremers, D. 2017.  Direct sparse odometry.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(3): 611–625.
  * Engel, Schöps, and Cremers (2014) Engel, J.; Schöps, T.; and Cremers, D. 2014.  LSD-SLAM: Large-scale direct monocular SLAM.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II 13_ , 834–849. Springer.
  * Fridovich-Keil et al. (2022) Fridovich-Keil, S.; Yu, A.; Tancik, M.; Chen, Q.; Recht, B.; and Kanazawa, A. 2022.  Plenoxels: Radiance fields without neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5501–5510.
  * Fu et al. (2022) Fu, Q.; Xu, Q.; Ong, Y. S.; and Tao, W. 2022.  Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction.  _Advances in Neural Information Processing Systems_ , 35: 3403–3416.
  * Garbin et al. (2021) Garbin, S. J.; Kowalski, M.; Johnson, M.; Shotton, J.; and Valentin, J. 2021.  Fastnerf: High-fidelity neural rendering at 200fps.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 14346–14355.
  * Guo and Sherwood (2021) Guo, J.; and Sherwood, A. 2021.  imporved-nerfmm.  https://github.com/ventusff/improved-nerfmm.
  * Jain, Tancik, and Abbeel (2021) Jain, A.; Tancik, M.; and Abbeel, P. 2021.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5885–5894.
  * Jeong et al. (2021) Jeong, Y.; Ahn, S.; Choy, C.; Anandkumar, A.; Cho, M.; and Park, J. 2021.  Self-calibrating neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5846–5854.
  * Kim, Seo, and Han (2022) Kim, M.; Seo, S.; and Han, B. 2022.  Infonerf: Ray entropy minimization for few-shot neural volume rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12912–12921.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kuang et al. (2022) Kuang, Z.; Olszewski, K.; Chai, M.; Huang, Z.; Achlioptas, P.; and Tulyakov, S. 2022.  NeROIC: neural rendering of objects from online image collections.  _ACM Transactions on Graphics (TOG)_ , 41(4): 1–12.
  * Li, Tancik, and Kanazawa (2022) Li, R.; Tancik, M.; and Kanazawa, A. 2022.  NerfAcc: A General NeRF Acceleration Toolbox.  _arXiv preprint arXiv:2210.04847_.
  * Lin et al. (2021) Lin, C.-H.; Ma, W.-C.; Torralba, A.; and Lucey, S. 2021.  Barf: Bundle-adjusting neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5741–5751.
  * Lin et al. (2023) Lin, Y.; Müller, T.; Tremblay, J.; Wen, B.; Tyree, S.; Evans, A.; Vela, P. A.; and Birchfield, S. 2023.  Parallel inversion of neural radiance fields for robust pose estimation.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 9377–9384. IEEE.
  * Long et al. (2023) Long, X.; Lin, C.; Liu, L.; Liu, Y.; Wang, P.; Theobalt, C.; Komura, T.; and Wang, W. 2023.  Neuraludf: Learning unsigned distance fields for multi-view reconstruction of surfaces with arbitrary topologies.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 20834–20843.
  * Long et al. (2022) Long, X.; Lin, C.; Wang, P.; Komura, T.; and Wang, W. 2022.  Sparseneus: Fast generalizable neural surface reconstruction from sparse views.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , 210–227. Springer.
  * Lowe (2004) Lowe, D. G. 2004.  Distinctive image features from scale-invariant keypoints.  _International journal of computer vision_ , 60: 91–110.
  * Lucas and Kanade (1981) Lucas, B. D.; and Kanade, T. 1981.  An iterative image registration technique with an application to stereo vision.  In _IJCAI’81: 7th international joint conference on Artificial intelligence_ , volume 2, 674–679.
  * Martin-Brualla et al. (2021) Martin-Brualla, R.; Radwan, N.; Sajjadi, M. S.; Barron, J. T.; Dosovitskiy, A.; and Duckworth, D. 2021.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7210–7219.
  * Meng et al. (2021) Meng, Q.; Chen, A.; Luo, H.; Wu, M.; Su, H.; Xu, L.; He, X.; and Yu, J. 2021.  Gnerf: Gan-based neural radiance field without posed camera.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6351–6361.
  * Meuleman et al. (2023) Meuleman, A.; Liu, Y.-L.; Gao, C.; Huang, J.-B.; Kim, C.; Kim, M. H.; and Kopf, J. 2023.  Progressively optimized local radiance fields for robust view synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16539–16548.
  * Mi, Di, and Xu (2022) Mi, Z.; Di, C.; and Xu, D. 2022.  Generalized binary search network for highly-efficient multi-view stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12991–13000.
  * Mildenhall et al. (2020) Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; and Ng, R. 2020.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I_ , 405–421.
  * Moulon, Monasse, and Marlet (2013) Moulon, P.; Monasse, P.; and Marlet, R. 2013.  Global fusion of relative motions for robust, accurate and scalable structure from motion.  In _Proceedings of the IEEE international conference on computer vision_ , 3248–3255.
  * Moulon et al. (2017) Moulon, P.; Monasse, P.; Perrot, R.; and Marlet, R. 2017.  Openmvg: Open multiple view geometry.  In _Reproducible Research in Pattern Recognition: First International Workshop, RRPR 2016, Cancún, Mexico, December 4, 2016, Revised Selected Papers 1_ , 60–74. Springer.
  * Müller et al. (2022) Müller, T.; Evans, A.; Schied, C.; and Keller, A. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4): 1–15.
  * Mur-Artal, Montiel, and Tardos (2015) Mur-Artal, R.; Montiel, J. M. M.; and Tardos, J. D. 2015.  ORB-SLAM: a versatile and accurate monocular SLAM system.  _IEEE transactions on robotics_ , 31(5): 1147–1163.
  * Niemeyer et al. (2022) Niemeyer, M.; Barron, J. T.; Mildenhall, B.; Sajjadi, M. S.; Geiger, A.; and Radwan, N. 2022.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5480–5490.
  * Oechsle, Peng, and Geiger (2021) Oechsle, M.; Peng, S.; and Geiger, A. 2021.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5589–5599.
  * Park et al. (2021) Park, K.; Sinha, U.; Hedman, P.; Barron, J. T.; Bouaziz, S.; Goldman, D. B.; Martin-Brualla, R.; and Seitz, S. M. 2021.  HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields.  _ACM Transactions on Graphics (TOG)_ , 40(6): 1–12.
  * Pumarola et al. (2021) Pumarola, A.; Corona, E.; Pons-Moll, G.; and Moreno-Noguer, F. 2021.  D-nerf: Neural radiance fields for dynamic scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10318–10327.
  * Rosinol, Leonard, and Carlone (2022) Rosinol, A.; Leonard, J. J.; and Carlone, L. 2022.  NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields.  _arXiv preprint arXiv:2210.13641_.
  * Sabour et al. (2023) Sabour, S.; Vora, S.; Duckworth, D.; Krasin, I.; Fleet, D. J.; and Tagliasacchi, A. 2023.  RobustNeRF: Ignoring Distractors with Robust Losses.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 20626–20636.
  * Schonberger and Frahm (2016) Schonberger, J. L.; and Frahm, J.-M. 2016.  Structure-from-motion revisited.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4104–4113.
  * Sitzmann et al. (2020) Sitzmann, V.; Martel, J.; Bergman, A.; Lindell, D.; and Wetzstein, G. 2020.  Implicit neural representations with periodic activation functions.  _Advances in Neural Information Processing Systems_ , 33: 7462–7473.
  * Sucar et al. (2021) Sucar, E.; Liu, S.; Ortiz, J.; and Davison, A. J. 2021.  iMAP: Implicit mapping and positioning in real-time.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6229–6238.
  * Sumikura, Shibuya, and Sakurada (2019) Sumikura, S.; Shibuya, M.; and Sakurada, K. 2019.  OpenVSLAM: A versatile visual SLAM framework.  In _Proceedings of the 27th ACM International Conference on Multimedia_ , 2292–2295.
  * Sun, Sun, and Chen (2022) Sun, C.; Sun, M.; and Chen, H.-T. 2022.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5459–5469.
  * Tancik et al. (2022) Tancik, M.; Casser, V.; Yan, X.; Pradhan, S.; Mildenhall, B.; Srinivasan, P. P.; Barron, J. T.; and Kretzschmar, H. 2022.  Block-nerf: Scalable large scene neural view synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8248–8258.
  * Tancik et al. (2023) Tancik, M.; Weber, E.; Ng, E.; Li, R.; Yi, B.; Wang, T.; Kristoffersen, A.; Austin, J.; Salahi, K.; Ahuja, A.; et al. 2023.  Nerfstudio: A modular framework for neural radiance field development.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 1–12.
  * Teed and Deng (2020) Teed, Z.; and Deng, J. 2020.  Raft: Recurrent all-pairs field transforms for optical flow.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , 402–419. Springer.
  * Turki, Ramanan, and Satyanarayanan (2022) Turki, H.; Ramanan, D.; and Satyanarayanan, M. 2022.  Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12922–12931.
  * Ueda et al. (2022) Ueda, I.; Fukuhara, Y.; Kataoka, H.; Aizawa, H.; Shishido, H.; and Kitahara, I. 2022.  Neural Density-Distance Fields.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , 53–68. Springer.
  * Verbin et al. (2022) Verbin, D.; Hedman, P.; Mildenhall, B.; Zickler, T.; Barron, J. T.; and Srinivasan, P. P. 2022.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 5481–5490. IEEE.
  * Wadhwani and Kojima (2022) Wadhwani, K.; and Kojima, T. 2022.  SqueezeNeRF: Further factorized FastNeRF for memory-efficient inference.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2717–2725.
  * Wang et al. (2021a) Wang, P.; Liu, L.; Liu, Y.; Theobalt, C.; Komura, T.; and Wang, W. 2021a.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_.
  * Wang et al. (2023) Wang, P.; Liu, Y.; Chen, Z.; Liu, L.; Liu, Z.; Komura, T.; Theobalt, C.; and Wang, W. 2023.  F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4150–4159.
  * Wang, Skorokhodov, and Wonka (2022) Wang, Y.; Skorokhodov, I.; and Wonka, P. 2022.  HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details.  In _Advances in Neural Information Processing Systems_.
  * Wang et al. (2021b) Wang, Z.; Wu, S.; Xie, W.; Chen, M.; and Prisacariu, V. A. 2021b.  NeRF–: Neural radiance fields without known camera parameters.  _arXiv preprint arXiv:2102.07064_.
  * Warburg et al. (2023) Warburg, F.; Weber, E.; Tancik, M.; Holynski, A.; and Kanazawa, A. 2023.  Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFs.  _arXiv preprint arXiv:2304.10532_.
  * Wu (2013) Wu, C. 2013.  Towards linear-time incremental structure from motion.  In _2013 International Conference on 3D Vision-3DV 2013_ , 127–134. IEEE.
  * Xiao et al. (2023) Xiao, Y.; Xue, N.; Wu, T.; and Xia, G.-S. 2023.  Level-$S^{2}$fM: Structure From Motion on Neural Level Set of Implicit Surfaces.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 17205–17214.
  * Yan et al. (2023) Yan, Q.; Wang, Q.; Zhao, K.; Li, B.; Chu, X.; and Deng, F. 2023.  Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 3091–3099.
  * Yao et al. (2018) Yao, Y.; Luo, Z.; Li, S.; Fang, T.; and Quan, L. 2018.  Mvsnet: Depth inference for unstructured multi-view stereo.  In _Proceedings of the European conference on computer vision (ECCV)_ , 767–783.
  * Yariv et al. (2021) Yariv, L.; Gu, J.; Kasten, Y.; and Lipman, Y. 2021.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ , 34: 4805–4815.
  * Yen-Chen et al. (2021) Yen-Chen, L.; Florence, P.; Barron, J. T.; Rodriguez, A.; Isola, P.; and Lin, T.-Y. 2021.  inerf: Inverting neural radiance fields for pose estimation.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 1323–1330. IEEE.
  * Yi et al. (2016) Yi, K. M.; Trulls, E.; Lepetit, V.; and Fua, P. 2016.  Lift: Learned invariant feature transform.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14_ , 467–483. Springer.
  * Yu et al. (2021a) Yu, A.; Li, R.; Tancik, M.; Li, H.; Ng, R.; and Kanazawa, A. 2021a.  Plenoctrees for real-time rendering of neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5752–5761.
  * Yu et al. (2021b) Yu, A.; Ye, V.; Tancik, M.; and Kanazawa, A. 2021b.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4578–4587.
  * Zhang et al. (2022a) Zhang, J.; Huang, J.; Cai, B.; Fu, H.; Gong, M.; Wang, C.; Wang, J.; Luo, H.; Jia, R.; Zhao, B.; et al. 2022a.  Digging into Radiance Grid for Real-Time View Synthesis with Detail Preservation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV_ , 724–740. Springer.
  * Zhang et al. (2022b) Zhang, J.; Zhan, F.; Wu, R.; Yu, Y.; Zhang, W.; Song, B.; Zhang, X.; and Lu, S. 2022b.  Vmrf: View matching neural radiance fields.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , 6579–6587.
  * Zhang et al. (2020) Zhang, K.; Riegler, G.; Snavely, N.; and Koltun, V. 2020.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_.
  * Zhang et al. (2021) Zhang, X.; Srinivasan, P. P.; Deng, B.; Debevec, P.; Freeman, W. T.; and Barron, J. T. 2021.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  _ACM Transactions on Graphics (TOG)_ , 40(6): 1–18.
  * Zhu et al. (2023) Zhu, Z.; Peng, S.; Larsson, V.; Cui, Z.; Oswald, M. R.; Geiger, A.; and Pollefeys, M. 2023.  Nicer-slam: Neural implicit scene encoding for rgb slam.  _arXiv preprint arXiv:2302.03594_.
  * Zhu et al. (2022) Zhu, Z.; Peng, S.; Larsson, V.; Xu, W.; Bao, H.; Cui, Z.; Oswald, M. R.; and Pollefeys, M. 2022.  Nice-slam: Neural implicit scalable encoding for slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12786–12796.