# Improving Robustness for Joint Optimization of Camera Poses
and Decomposed Low-Rank Tensorial Radiance Fields

Bo-Yu Cheng, Wei-Chen Chiu, Yu-Lun Liu

###### Abstract

In this paper, we propose an algorithm that allows joint refinement of camera
pose and scene geometry represented by decomposed low-rank tensor, using only
2D images as supervision. First, we conduct a pilot study based on a 1D signal
and relate our findings to 3D scenarios, where the naive joint pose
optimization on voxel-based NeRFs can easily lead to sub-optimal solutions.
Moreover, based on the analysis of the frequency spectrum, we propose to apply
convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-
fine training schedule that enables joint camera pose optimization. Leveraging
the decomposition property in decomposed low-rank tensor, our method achieves
an equivalent effect to brute-force 3D convolution with only incurring little
computational overhead. To further improve the robustness and stability of
joint optimization, we also propose techniques of smoothed 2D supervision,
randomly scaled kernel parameters, and edge-guided loss mask. Extensive
quantitative and qualitative evaluations demonstrate that our proposed
framework achieves superior performance in novel view synthesis as well as
rapid convergence for optimization. The source code is available at
https://github.com/Nemo1999/Joint-TensoRF.

##  1 Introduction

In recent years, neural rendering has become a widely-used method for high-
quality novel view synthesis. NeRF as a pioneer work (Mildenhall et al. 2020)
represents a 3D radiance field as an implicit continuous function built upon
multilayer perceptrons (MLPs) which is trained with differentiable volume
rendering. While achieving excellent synthesis quality, NeRF suffers from
training/inference inefficiency due to dense evaluation of the computationally
expensive MLPs.

To this end, voxel-based methods built upon the explicit scene representation
of 3D voxel grid (Sun, Sun, and Chen 2022; Fridovich-Keil et al. 2022; Liu et
al. 2020) are proposed to achieve faster training and provide better rendering
quality than the original MLP-based NeRF, hence becoming the more preferred
choices for downstream applications.

Nevertheless, maintaining a dense 3D voxel grid is in turn memory intensive,
thus still restricting wider applications of voxel-based methods. Fortunately,
TensoRF (Chen et al. 2022) proposes to tackle such memory-intensive issue of
the voxel grid via replacing the dense 3D grid with _decomposed low-rank
tensor_. TensoRF achieves a high data compression ratio and low computational
cost at the same time while also achieving state-of-the-art performance.
Providing a win-win situation on memory usage and computational efficiency,
the decomposed low-rank tensor architecture has been widely adopted in many
recent works (Xu et al. 2023; Fridovich-Keil et al. 2023; Goel et al. 2022;
Han and Xiang 2023; Shao et al. 2023; Tang et al. 2022; Meuleman et al. 2023).

Figure 1: Robust joint pose refinement on decomposed tensor. Our method
enables joint optimization of camera poses and decomposed voxel representation
by applying efficient _separable component-wise convolution_ of Gaussian
filters on 3D tensor volume and 2D supervision images.

On the other hand, the effectiveness of NeRF (and most of the aforementioned
works) hinges on precise camera poses of input images, which are often
calculated using Structure-from-Motion (SfM) algorithms like COLMAP
(Schönberger and Frahm 2016). While some works (Wang et al. 2021; Lin et al.
2021; Chng et al. 2022) aim to bypass the slow and occasionally inaccurate
COLMAP process by optimizing camera pose and scene representation jointly on
the original MLP-based NeRF, their success is often tied to the spectral bias
(Yüce et al. 2022) of the MLP architecture which ensures the smoothness of 3D
radiance field early in training. Voxel-based methods, however, lack such
properties and can overemphasize sharp edges, making naive joint optimization
problematic as getting trapped in local optima (Fig. 2 (a)).

In this work, we present simple yet effective methods for refining the camera
pose and the 3D scene using decomposed low-rank tensors (cf. Fig. 1). We
identify that controlling the frequency spectrum is vital for pose alignment,
while directly realizing such control in a dense 3D grid could be
nontrivial/challenging as well as computationally demanding. To this end, we
introduce an efficient 3D filtering method using _component-wise separable
convolution_ for enabling the spectral control, which is more efficient than
the traditionally well-known trick of separable convolution kernel as we
additionally utilize the separability of the input signal. To further ensure
stability in the optimization process, we propose several techniques,
including _smoothed 2D supervision_ , _randomly scaled kernel paramter_ , and
the _edge-guided loss mask_. These techniques are experimentally proven
crucial for successful pose refinement in our ablation studies. In results,
our proposed method requires only 50k training iterations, where all the
previous methods typically needs 200k iterations (e.g. the overall training
time is reduced to 25%, compared to previous MLP-based methods). The main
reason behind this advantage is not only based on property of voxel-based
architecture, but also relies on our carefully designed efficient spectral
filtering algorithm that requires only single reusable voxel grid (please
refer to Sec. 4.3). Moreover, our method performs favorably against state-of-
the-art methods on novel view synthesis. Our contributions are three-fold:

  * •

With 1D pilot study, we provide insights into the impact of spectral property
of 3D scene on the convergence of joint optimization beyond the coarse-to-fine
heuristic discussed in prior research, and propose a learning strategy built
upon specially designed efficient component-wise convolution algorithm.

  * •

To enhance the robustness of our joint optimization, we introduce techniques
of smoothed 2D supervision, scaled kernel parameters, and the edge-guided loss
mask.

  * •

Training time drops by 25% versus existing MLP-based methods, with requiring
only 50k iterations against 200k of previous methods. Results show state-of-
the-art performance in novel view synthesis with unknown pose.

##  2 Related Work

Accelerating Neural Rendering. As the seminal work of neural rendering, NeRF
adopts MLPs to construct the implicit representation of the 3D scene,
providing high-quality view synthesis but having a time-consuming training
process due to the computational demands of MLPs. For addressing such issue,
different variants of NeRF are proposed to use custom spatial data structures
where the scene information is distributed only locally thus aiding faster
training and rendering, in which those spatial data structures include _point
cloud_ (Xu et al. 2022; Hu et al. 2023), _space partitioning tree_ (Wang et
al. 2022; Yu et al. 2021), _triangular mesh_ (Chen et al. 2023b; Kulhanek and
Sattler 2023), and _voxel gird_ (Sun, Sun, and Chen 2022; Fridovich-Keil et
al. 2022; Liu et al. 2020; Hedman et al. 2021). Among these variants, the
voxel grid has become more popular due to its easy implementation and quality
reconstruction. However, as scene dimensions grow, the memory usage of the
voxel grid becomes inefficient. To address this, (Müller et al. 2022)
recommends compressing the grid via hash encoding, while (Chen et al. 2022;
Fridovich-Keil et al. 2023) suggest adopting tensor decomposition for 3D
feature compression, in which our method is mainly based on (Chen et al. 2022)
but can be adaptable to other tensor decomposition-based voxel structures like
K-Planes (Fridovich-Keil et al. 2023).

Joint Pose Estimation on MLP-based NeRFs. (Wang et al. 2021) is one of the
first NeRF-based attempts to tackle the joint problem of estimating camera
poses and learning 3D scene representation by directly adjusting camera pose
using gradient propagation on neural radiance fields. The robustness of such
joint optimization is further enhanced by (Lin et al. 2021; Chng et al. 2022),
where they propose various methods to smooth the pose gradient derived from
the underlying MLP. (Chen et al. 2023a) further increases the noise tolerance
by a specially designed local-global joint alignment approach. Our method also
tackles joint problems but is specifically designed for the voxel-based NeRF
built upon the decomposed low-rank tensor architecture.

Pose Estimation on Decomposed Low-rank Tensors. There do exist works that
optimize camera pose on decomposed low-rank tensor (Liu et al. 2023; Meuleman
et al. 2023) but require rich additional geometry clues (e.g., depth map and
optical flow). To our best knowledge, we are the first attempt to jointly
optimize the camera pose and the _decomposed low-rank tensor_ using only 2D
image supervision.

Figure 2: Comparison of naive joint pose optimization and our proposed method
on voxel-based NeRFs. (a) Naively applying joint optimization on voxel-based
NeRFs leads to dramatic failure as premature high-frequency signals in the
voxel volume would curse the camera poses to stuck in local minima. (b) We
propose a computationally effective manner to directly control the spectrum of
the radiance field by performing _separable component-wise convolution_ of
Gaussian filters on the decomposed tensor. The proposed training scheme allows
the joint optimization to converge successfully to a better solution.

Pose Estimation on Multi-Resolution Hash Encoding. Aside from decomposed low-
rank tensor, _multi-resolution hash encoding_ is another compressed voxel-
based architecture proposed by (Müller et al. 2022). Along with such a choice
of architecture, recently (Heo et al. 2023) has proposed to address the joint
optimization of camera pose and multi-resolution hash encoding. They suggest a
new interpolation scheme that provides smooth gradients hence preventing
gradient fluctuation in the hash volume, along with a curriculum learning
scheme that controls the learning rate of the hash table at each resolution
level. Although achieving impressive results on joint optimization, the
effectiveness of their method is limited to multi-resolution hash encoding and
is not applicable to _decomposed low-rank tensor_ , While our proposed
separable component-wise 3D convolution (and randomly scaled kernel) is
specifically designed for _decomposed low-rank tensor_ and not directly
applicable to _multi-resolution hash encoding_ , in which these two
representations have their respective pros and cons.

##  3 Our Proposed Method

###  3.1 Joint Refinement of 3D Scenes and Poses

Volume Rendering for Radiance Field Reconstruction. Based on the setting of
neural volume rendering in NeRF, the radiance fields respective for geometry
and appearance for a 3D scene are represented via two functions (implemented
by MLPs): $F_{\sigma}:\mathbb{R}^{3}\to\mathbb{R}^{1}$ and
$F_{c}:\mathbb{R}^{6}\to\mathbb{R}^{3}$, where $F_{\sigma}$ returns the volume
density of an input 3D coordinate, while $F_{c}$ outputs the color at an input
3D coordinate given a 3D viewing direction. For rendering a pixel on 2D
coordinate $u$ with its homogeneous form $\bar{u}=[u;1]^{\top}$, we first
sample a sequence of $N$ 3D-coordinates $\\{s_{n}\\}_{n=1\cdots N}$ along the
camera ray defined by the camera center $\vec{c}\in\mathbb{R}^{3}$ and the ray
direction $\vec{d_{u}}=K^{-1}\bar{u}$,

$\begin{split}\medmath{\left\\{s_{n}\right\\}_{n=1\cdots
N}=s(\vec{c},\vec{d_{u}})=\\{\vec{c}+t_{n}\cdot\vec{d_{u}}\\}_{n=1\cdots
N}},\end{split}$ (1)

where $K$ is the intrinsic camera matrix and $\\{t_{n}\\}_{n=1\cdots N}$ are
$N$ samples equidistantly distributed along the depth axis in between the near
and far planes of the view frustum. The resultant color of the pixel is
obtained by integrating through the density field $F_{\sigma}$ and color field
$F_{c}$ using the volume rendering equation (Kajiya and Von Herzen 1984;
Mildenhall et al. 2020), where we denote the _discretized volume rendering
intregral_ by a function $\mathbf{V}$:

$\begin{split}\medmath{\mathbf{V}(F_{\sigma},F_{c},s(\vec{c},\vec{d_{u})})=\sum_{s_{n}\in
s(\vec{c},\vec{d})}{T_{n}\cdot\alpha_{n}\cdot\mathbf{C}_{n}}},\end{split}$ (2)

where $T_{n}=exp(-\sum_{j=1}^{n}{\delta_{j}F_{\sigma}(s_{j})})$ represents
accumulated transmittance prior to $s_{n}$,
$\alpha_{n}=1-exp(-\delta_{n}F_{\sigma}(s_{n}))$ represents the opacity of
sample $s_{n}$, and $\mathbf{C}_{n}=F_{c}(s_{n},\ \vec{d_{u}}\ )$ represents
the color of sample $s_{n}$, and $\delta_{j}=\lVert s_{j}-s_{j-1}\rVert$ is
the euclidean distance between two adjacent samples.

In the typical setting of NeRF, given a set of $L$ 2D-images
$\mathbf{I}=\\{I_{1},\cdots,I_{L}\\}$ with their corresponding camera poses
$\mathbf{P}=\\{P_{1},\cdots,P_{L}\\}$ $\in\mathfrak{se}(3)$ Lie algebra
(parametrizing rigid 3D transformation as $\mathfrak{se}(3)$ is a very common
technique in robotics, here we follow the usage of (Lin et al. 2021)) as
input, we aim to reconstruct the 3D scene represented by $F_{\sigma}^{*}$ and
$F_{c}^{*}$, via minimizing the loss $\mathcal{L}_{\text{rec}}$ of 2D
photometric reconstruction with the gradient-based optimization algorithm, in
which

$\begin{split}\medmath{\mathcal{L}_{\text{rec}}(F_{\sigma},F_{c})={\sum_{i=1}^{L}\sum_{u\in\mathbf{U}}{\lVert\mathbf{V}(F_{\sigma},F_{c},\mathcal{W}_{\text{3d}}(P_{i},s(\vec{0},\vec{d_{u}})))-I_{iu}\rVert}}},\end{split}$
(3)

where $\mathbf{U}$ is the set of all possible 2D coordinates in the input
images, $I_{iu}\in\mathbb{R}^{3}$ is the RGB color of pixel location $u$ on
training image $I_{i}$, warping function
$\mathcal{W}_{\text{3d}}(P,\\_):\mathbb{R}^{3}\to\mathbb{R}^{3}$ performs
rigid 3D transformation parameterized by $P\in\mathfrak{se}(3)$ Lie algerbra,
and $\mathcal{W}_{\text{3d}}(P,s(\vec{0},\vec{d_{u}}))$ maps each sample 3D
coordinate in canonical ray ($\vec{c}=\vec{0}$) into a 3D sample coordinate of
camera ray with pose $P$. Note that this is an ill-posed reconstruction
problem that suffers from shape-radiance ambiguity (Zhang et al. 2020).

3D Joint Optimization. When it comes to jointly estimating camera poses (where
the camera poses $\mathbf{P}$ are also unknown) and learning scene
representation (Lin et al. 2021; Chng et al. 2022; Chen et al. 2023a; Heo et
al. 2023), the problem is even more ill-defined with the objective now being
extended from Eq. 3 and defined as:

$\begin{split}\medmath{\mathcal{L}_{\text{joint}}(F_{\sigma},F_{c},\mathbf{P})=\sum_{i=1}^{L}\sum_{u\in\mathbf{U}}{\lVert\mathbf{V}(F_{\sigma},F_{c},\mathcal{W}_{\text{3d}}(P_{i},s(\vec{0},\vec{d_{u}})))-I_{iu}\rVert}}.\end{split}$
(4)

Such joint optimization is highly influenced by the structural bias of the
underlying representation of $\\{F_{\sigma},F_{c}\\}$, which we will conduct a
pilot study with a simpler 1D case in Sec. 3.2.

###  3.2 Gaussian Filter on 1D Signal Alignment

Here we aim to analyze the effect of the signal spectrum (spectrum of $F_{c}$,
$F_{\sigma}$, and $\mathbf{I}$ in Eq. 4) on the joint optimization process. We
begin by reducing 3D joint optimization of camera pose and scene
reconstruction into a simpler 1D counterpart of signal alignment.

1D Signal Alignment. Let us consider a target ground truth 1D signal $f_{GT}$
(assuming the signal to be continuous, bounded, and have finite support),
which we aim to reconstruct and align with. We are given randomly translated
versions $f_{1}$, $f_{2}$ of the ground truth signal $f_{GT}$, where
$f_{1}=\mathcal{W}_{\text{1d}}(f_{GT},p_{1}),f_{2}=\mathcal{W}_{\text{1d}}(f_{GT},p_{2})$
with having $\mathcal{W}_{\text{1d}}$ a signal translation operation defined
as $\mathcal{W}_{\text{1d}}(f,p)(x)=f(x-p)$, and $p_{1},p_{2}$ are the
translation values.

Although the reconstruction is trivial in such a 1D setting, in order to mimic
the case of 3D joint optimization, we attempt to estimate a signal $g$ as well
as the translation values $q_{1}$ and $q_{2}$ via adopting the iterative
gradient-based optimization on the reconstruction loss.

$\begin{split}\medmath{\mathcal{L}_{\text{1d}}(g,q_{1},q_{2})}&\medmath{=\sum_{i\in[1,2]}\int\lVert\mathcal{W}_{\text{1d}}(g,q_{i})(x)-f_{i}(x)\rVert^{2}dx}\\\
&\medmath{=\sum_{i\in[1,2]}\int\lVert
g(x)-f_{GT}(x-p_{i}+q_{i})\rVert^{2}dx}.\end{split}$ (5)

Note that Eq. 5 and Eq. 4 are analogous in terms of their
structure/formulation, where the difference only lies in the dimensionality.
And $\mathcal{L}_{\text{1d}}$ achieves the optimum whenever
$q_{1}-q_{2}=p_{1}-p_{2}$ and $g$ =
$\mathcal{W}_{\text{1d}}(f_{GT},p_{1}-q_{1})$. Please check Figure 3(a) for a
simple visual representation of Equation 5, where $f_{1}$ and $f_{2}$ are
connected to $g$ by the reconstruction loss $\mathcal{L}_{\text{1d}}$ (i.e
blue arrows), whose gradients are used to update $g$ and the translation
values $\\{q_{1},q_{1}\\}$.

Connection between 1D Signal Alignment and 3D Joint Optimization. The
formulation of 1D signal alignment effectively simulates the “local
phenomenon” of joint camera pose alignment and 3D scene reconstruction on a 2D
cross-section: As shown in Figure 3(c), where we consider two neighboring
camera poses as well as a cross-section in the 3D space passing through both
camera planes and intersecting with each camera plane on a projected straight
line, the RGB color values on such projected lines correspond to the 1D
shifted ground truth signals $f_{1}$, $f_{2}$ in Equation 5, and the value of
the radiance field on the cross-section corresponds to reconstructed signal
$g$ in Equation 5. Similar to the loss $\mathcal{L}_{\text{1d}}$ in Equation
5, the projected lines on the camera planes and the corresponding cross-
section in the 3D radiance field are connected by the volume rendering
function $V$ and reconstruction loss $\mathcal{L}_{\text{joint}}$ in Equation
4. As a result, the complete 3D joint optimization can be intuitively viewed
as simultaneously performing many 1D signal analyses on the superposition of
all possible combinations of camera poses and cross-sections.

Figure 3:  Spectrum analysis and effect of Gaussian filtering on 1D signal
alignment. (a) 1D signal alignment comparison: noisy signals can get trapped
in local optima without Gaussian filtering. (b)(_Top_) Visualization of
$H(u,k)$ in Eq. 7, which shows alternating signs as $k$ departs from $0$,
causing misdirection in gradient-based optimization if there has too much
high-frequency energy in the signal. (b)(_Bottom_) Visualization of
$\tilde{H}(u,k)$ in Eq. 8, which is the modulated version of $H(u,k)$ with the
help of Gaussian filter $\mathcal{N}$. (c) 1D alignment relates to 3D joint
optimization in Eq. 4, where effective pose refinement stems from the 1D
alignment in specific cross-sections, with the red lines in 3D scene
correlating to horizontal shifts (blue arrows) and rotations (green arrows).

Spectrum Analysis and Effect of Gaussian Filtering on 1D Signal Alignment.
First we transform the problem into a simpler form with a assumption that is
reflected by the fast convergent property of voxel grids (cf. our supplement
for detailed derivation of the theorem):

###### Theorem 1

If we assume rapid convergence of signal $g$ (which means $g$ achieves local
optima $g^{*}$ w.r.t current $q_{1},q_{2}$ whenever we update $q_{1},q_{2}$.),
we find that the problem in Eq.5 is equivalent to pure alignment between two
ground-truth signals, that is

$\displaystyle\medmath{\mathcal{L}_{\text{1d}}(g,q_{1},q_{2})}$
$\displaystyle=\medmath{\mathcal{L}_{\text{1d}}(g^{*},q_{1},q_{2})}$ (6)
$\displaystyle=\medmath{\mathcal{L}_{\text{1d}}(u)=\int\lVert
f_{GT}(x)-f_{GT}(x+u)\rVert^{2}dx},$

where $u=(p_{1}-p_{2})-(q_{1}-q_{2})$ is the shift between two ground truth
signals, which has an initial value of $p_{1}-p_{2}$

We aim for $u$ to reach $0$ with gradient-based optimization.

Next, by analyzing the relationship between $f_{GT}$ and the optimization
gradient ${d\over du}\mathcal{L}_{\text{1d}}$ in terms of their spectral
properties, we get the following result (cf. our supplement for detailed
derivation of the theorem):

###### Theorem 2

$\displaystyle\medmath{{d\over du}\mathcal{L}_{\text{1d}}}$
$\displaystyle\medmath{=\int\lVert\ \mathfrak{F}[{f_{GT}]}\ \rVert^{2}\cdot
H(u,k)\ dk},$ (7)

where $H(u,k)=4\pi k\ sin(2\pi ku)$, $\mathfrak{F}[f_{GT}]$ is Fourier
transform of $f_{GT}$, and $k$ is the wavenumber in frequency domain.

Particularly, we are interested in the sign of ${d\over
du}\mathcal{L}_{\text{1d}}$ which determines the direction of our iterative
optimization. We plot the value of $H(u,k)$ in Fig. 3(b)(_Top_), where we can
observe that the sign of $H$ is well-behaved when the magnitude of $k$ is
small (here well-behaving means the direction of the gradient is able to let
$u$ descend to $0$, i.e., being positive when $u>0$ and negative when $u<0$).
However, when $k$ increases, the sign of $H$ quickly begins to alternate, and
the magnitude increases, which causes the gradient to be large and noisy.
Hence high-frequency signals with a spreading spectrum can easily lead the
optimization process to get stuck in the local optima.

To this end, we demonstrate that applying a Gaussian filter on the signal
$f_{GT}$ effectively mitigates the sign-alternating issue of the original $H$
function. Specifically, we show that filtering the input signal is equivalent
to modulating $H$ by a Gaussian window (cf. our supplement for derivation):

###### Theorem 3

Let $\tilde{\mathcal{L}}_{\text{1d}}$ denotes $\mathcal{L}_{\text{1d}}$
calculated with Gaussian convoluted signal $\mathcal{N}\ast f_{GT}$, and
$\mathfrak{F}[\mathcal{N}]$ denotes the Fourier transform of the Gaussian
kernel $\mathcal{N}$, then we have

$\displaystyle\medmath{{d\over du}\tilde{\mathcal{L}}_{\text{1d}}}$
$\displaystyle\medmath{=\int\lVert\ \mathfrak{F}[f_{GT}]\
\rVert^{2}\cdot\tilde{H}(u,k)\ dk},$ (8)

where $\tilde{H}(u,k)=\rVert\ \mathfrak{F}[\mathcal{N}]\ \lVert^{2}\cdot
H(u,k).$

In Fig. 3(b)(_Bottom_), we plot the modulated $\tilde{H}(u,k)$, with observing
that the misbehave region is suppressed (note that we set the variance of
$\mathcal{N}$ to $4$ here). The gradient descent will likely converge to $u=0$
once the initial magnitude of $u$ is less than $6.0$. The region where
${d\over du}\tilde{\mathcal{L}}_{\text{1d}}$ does well-behave is _quasi-
convex_ and is guaranteed to converge to global optima given suitable learning
rate that prevents us from getting stuck on saddle points. Our analysis agrees
with the motivation behind the coarse-to-fine training schedule of (Lin et al.
2021) and (Heo et al. 2023). Specifically, observing that the well-behaved
region in $H(u,k)$ grows wider as $u$ approaches $0$ (cf. Fig. 3(b)(_Top_)),
which means that we can loosen the filtering strength of Gaussian kernel as
$u$ approaches $0$, leading to larger and more accurate gradient.

###  3.3 2D Planar Image Alignment

In addition to the 3D joint optimization problem, previous works (Lin et al.
2021; Chng et al. 2022) also consider a 2D image patches alignment task as a
simpler example of joint optimization, in which there are $L_{\text{2d}}$
overlapping image patches
$\mathbf{I_{\text{2d}}}=\\{I_{1},\cdots,I_{L_{\text{2d}}}\\}$ cropped from a
single ground truth image $I_{gt}$ before being transformed by 2D homography.
The homography transforms are parameterized by
$\mathbf{P_{\text{2d}}}=\\{P_{1},\cdots,P_{L_{\text{2d}}}\\}\in\mathfrak{sl}(3)$
and initialized as $\vec{0}$ (here we also follow from (Lin et al. 2021) the
usage Lie algrebra to parameterize 2D homography transform). Analogously to
Equation 4, our objective is to jointly optimize the 2D image content
$F_{\text{2d}}:\mathbb{R}^{2}\to\mathbb{R}^{2}$ and per-patch homography warps
$\mathbf{P_{\text{2d}}}$ by the reconstruction loss. Joint optimization can be
formulated as:

$\begin{split}\medmath{\mathcal{L}_{\text{2d}}(F_{\text{2d}},\mathbf{P_{\text{2d}}})=\sum_{i=1}^{L}{\sum_{u\in\mathbf{U_{\text{2d}}}}{}}{\lVert
F_{\text{2d}}(\mathcal{W}_{\text{2d}}(P_{i},u))-I_{iu}\rVert}^{2}},\end{split}$
(9)

where $\mathbf{U_{\text{2d}}}$ is the set of all possible 2D coordinates in
the image patches, $I_{iu}$ is the color of pixel at location $u$ on input
image patch $I_{i}$, warp function
$\mathcal{W}_{\text{2d}}(P_{i},\\_):\mathbb{R}^{2}\to\mathbb{R}^{2}$ performs
2D homography transformation parameterized by $P_{i}\in\mathfrak{sl}(3)$ Lie
algebra, and $\mathcal{W}_{\text{2d}}(P_{i},u)$ maps 2D coordinate $u$ on
$I_{gt}$ into a transformed 2D coordinate on patch $I_{i}$. Notice the strong
structural correspondence among Eq. 5 (1D alignment), Eq. 9 (2D alignment),
and Eq. 4 (3D alignment), the three problems share similar computational
property.

We parameterize $F_{\text{2d}}$ by a 2D decomposed low-rank tensor
$\mathcal{T}_{\text{2d}}\in\mathbb{R}^{h\times w}$, where $w,h$ are the
dimensions of the image. Motivated by our analysis in Section 3.2, we filter
$\mathcal{T}_{\text{2d}}$ with 2D Gaussian kernel to avoid overfitting.

$\begin{split}\medmath{F_{\text{2d}}(\textbf{x})=(\mathcal{N}_{\text{2d}}\ast_{\text{2d}}\mathcal{T}_{\text{2d}})(\textbf{x})=(\mathcal{N}_{\text{2d}}\ast_{\text{2d}}(\sum^{R}_{r=1}\mathbf{v}^{X}_{r}\otimes\mathbf{v}^{Y}_{r}))(\textbf{x})},\end{split}$
(10)

where $\textbf{x}\in\mathbb{R}^{2}$ is 2D pixel coordinates,
$\mathcal{N}_{\text{2d}}$ is 2D gaussian kernel, $\ast_{\text{2d}}$ is the
convolution operator, and $\otimes$ denotes outer product between the 1D
vector components
$\mathbf{v}^{X}_{r}\in\mathbb{R}^{w},\mathbf{v}^{Y}_{r}\in\mathbb{R}^{h}$.
“$(\textbf{x})$” at the end of the expressions means bilinearly interpolating
the preceding discrete 2D volume with continuous coordinate x. Our method
outperforms the naïve tensor method and previous methods (Lin et al. 2021;
Chng et al. 2022), experiment results are shown at Sec. 4.1.

The width of Gaussian kernel $\mathcal{N}_{\text{2d}}$ is controlled by an
exponential coarse-to-fine training schedule that changes continuously (cf.
our supplement for details of such kernel schedule). In order to support
continuous changing width on a discrete Gaussian kernel, the kernel is
generated by the following rule:

$\displaystyle\medmath{\mathcal{N}_{\text{1d}}(\sigma)}$
$\displaystyle\medmath{=}\begin{cases}\medmath{\bigoplus^{L_{\mathcal{N}}/2}_{x=-{L_{\mathcal{N}}/2}}min(1,{1\over\sqrt{2\pi}\sigma}e^{-{x^{2}\over
2\sigma^{2}}})}&\medmath{\text{if }\sigma>0.0001}\\\
\medmath{\bigoplus^{L_{\mathcal{N}}/2}_{x=-{L_{\mathcal{N}}/2}}\delta[x]}&\medmath{\text{otherwise}},\end{cases}$
(11) $\displaystyle\medmath{\mathcal{N}_{\text{2d}}(\sigma)}$
$\displaystyle\medmath{=\mathcal{N}_{\text{1d}}(\sigma)\otimes\mathcal{N}_{\text{1d}}(\sigma)},$

where $L_{\mathcal{N}}$ is the size of the discrete kernel, 1D kernel
$\mathcal{N}_{\text{1d}}(\sigma)\in\mathbb{R}^{L_{\mathcal{N}}}$ is discretely
sampled from continuous Gaussian distribution and clamped to a max value of
$1.0$ before being concatenated into a vector by $\oplus$ operator. To avoid
numerical instability, when $\sigma<0.001$, we assign
$\mathcal{N}_{\text{1d}}(\sigma)$ to be discrete impluse function $\delta$. 2D
kernel $\mathcal{N}_{\text{2d}}(\sigma)\in\mathbb{R}^{L_{\mathcal{N}}\times
L_{\mathcal{N}}}$ is generated by outer product of two 1D kernels.

###  3.4 Decomposed Low-Rank Tensor

This section describes the decomposed low-rank tensor proposed by TensoRF
(Chen et al. 2022) which is the scene representation that our proposed method
is built upon. While there are two different types of tensor decomposition
considered in (Chen et al. 2022): CP-decomposition and VM-decomposition, in
our discussion we mainly focus on _VM-decomposition_ , although our method is
also naturally applicable to CP-decomposition.

To represent the 3D density field $F_{\sigma}$, we store the information in a
3D tensor $\mathcal{T}_{\sigma}\in\mathbb{R}^{I\times J\times K}$, in which
now $F_{\sigma}$ is defined simply as component-wise interpolation of
$\mathcal{T}_{\sigma}$.

$\medmath{\mathcal{T}_{\sigma}=\sum_{r=1}^{\mathbf{R}}{\mathbf{v}^{X}_{\sigma,r}}\otimes\mathbf{M}^{Y,Z}_{\sigma,r}+{\mathbf{v}^{Y}_{\sigma,r}}\otimes\mathbf{M}^{X,Z}_{\sigma,r}+{\mathbf{v}^{Z}_{\sigma,r}}\otimes\mathbf{M}^{X,Y}_{\sigma,r}},$
(12)

where $\mathbf{R}$ is the number of components in the decomposition,
$(\mathbf{v}^{X}_{r},\mathbf{v}^{Y}_{r},\mathbf{v}^{Z}_{r})\in(\mathbb{R}^{I},\mathbb{R}^{J},\mathbb{R}^{K})$
are 1D vector-components for axes $(X,Y,Z)$ repectively,
$(\mathbf{M}^{Y,Z}_{r},\mathbf{M}^{X,Z}_{r},\mathbf{M}^{X,Y}_{r})\in(\mathbb{R}^{J\times
K},\mathbb{R}^{I\times K},\mathbb{R}^{I\times J})$ are 2D matrix-components
for axes $(Y\textnormal{-}X,X\textnormal{-}Z,X\textnormal{-}Y)$ repectively,
operator $\otimes$ denotes the outer product between vector and matrix.

To represent the 3D color field $F_{c}$, the information
$\mathcal{T}_{c}(\textbf{x})\in\mathbb{R}^{G}$ queried from 3D feature tensor
$\mathcal{T}_{c}\in\mathbb{R}^{I\times J\times K\times G}$ is decoded by a
small MLP $S$ into RGB color value ($G$ is the input feature dimension of
$S$). The implementation can be formulated as

$\displaystyle\medmath{F_{c}(\textbf{x},\vec{d})=\mathbf{S}(\mathcal{T}_{c}(\textbf{x}),\vec{d})}$
(13)
$\displaystyle\medmath{\mathcal{T}_{c}=\medmath{\sum_{r=1}^{\mathbf{R}}{\mathbf{v}^{X}_{c,r}}\otimes\mathbf{M}^{Y,Z}_{c,r}\otimes\mathbf{b}^{X}_{r}+}}$
$\displaystyle\medmath{\quad\quad\quad\medmath{{\mathbf{v}^{Y}_{c,r}}\otimes\mathbf{M}^{X,Z}_{c,r}\otimes\mathbf{b}^{Y}_{r}+{\mathbf{v}^{Z}_{c,r}}\otimes\mathbf{M}^{X,Y}_{c,r}\otimes\mathbf{b}^{Z}_{r}}}.$

$\mathcal{T}_{c}(\textbf{x})$ denotes the component-wise linear-interpolation
of tensor volume $\mathcal{T}_{c}$ on 3D coordinate x. $\vec{d}$ is the
viewing direction of the current ray. $\mathbf{v}_{c,r}$ and
$\mathbf{M}_{c,r}$ have the same shape as their $\mathbf{v}_{\sigma,r}$ and
$\mathbf{M}_{\sigma,r}$ counterparts,
$\mathbf{b}^{X}_{r},\mathbf{b}^{Y}_{r},\mathbf{b}^{X}_{r}\in\mathbb{R}^{G}$
are feature components to expand the feature axis of $\mathcal{T}_{c}$.

###  3.5 Separable Component-Wise Convolution

As theoretically analyzed in Sec. 3.2 and empirically shown in Fig. 2(a),
naïvely applying low-rank decomposed tensor (which lacks internal bias that
limits the spectrum of learned signal, hence corresponds to the top raw of
Fig. 3) to joint camera pose optimization results in suboptimal reconstruction
quality and inaccurate poses. Therefore, we propose to limit the spectrum of
the radiance field $F_{\sigma}$ and $F_{c}$ with a coarse-to-fine training
schedule.

If we naïvely convolve the 3D Gaussian kernel with our 3D volume
$\mathcal{T}_{\sigma}$, (as in the 2D planar case of Eq. 10), we would have to
reconstruct the whole 3D tensor before applying convolution, destroying the
space compression advantage of decomposed low-rank tensor, see Eq. 14.

$\displaystyle\medmath{F_{\sigma}(x,y,z)=(\mathcal{N}_{\text{3d}}\ast_{\text{3d}}\mathcal{T}_{\sigma})(x,y,z)},$
(14)

where $\ast_{\text{3d}}$ denotes 3D convolution, $\mathcal{N}_{\text{3d}}$ is
the 3D Gaussian filter defined by
$\mathcal{N}_{1d}\otimes\mathcal{N}_{\text{2d}}$. Under this setting, the time
complexity and the space complexity are $O(I\cdot J\cdot K\cdot
L_{\mathcal{N}}^{3})$ and $O(I\cdot J\cdot K)$ respectively, where
$L_{\mathcal{N}}$ is the size of 3D Gaussian kernel in each dimension.

To achieve computationally efficient convolution on the 3D decomposed low-rank
tensor volume, we perform our proposed _separable component-wise convolution_
, by taking advantage of the following identity (whose correctness will be
proven in the supplementary material).

###### Theorem 4

$\displaystyle\tilde{\mathcal{T}}_{\sigma}=\medmath{\sum_{r=1}^{\mathbf{R}}{\mathbf{\tilde{v}}^{X}_{\sigma,r}}\otimes{\mathbf{\tilde{M}}^{Y,Z}_{\sigma,r}}+{\mathbf{\tilde{v}}^{Y}_{\sigma,r}}\otimes{\mathbf{\tilde{M}}^{X,Z}_{\sigma,r}}+{\mathbf{\tilde{v}}^{Z}_{\sigma,r}}\otimes{\mathbf{\tilde{M}}^{X,Y}_{\sigma,r}}},$
(15)

where
$\tilde{\mathcal{T}}_{\sigma}=(\mathcal{N}_{\text{3d}}\ast_{\text{3d}}\mathcal{T}_{\sigma})$
denotes the 3D Gaussian convoluted tensor volume,
${\mathbf{\tilde{v}}_{\sigma,r}}=(\mathcal{N}_{\text{1d}}\ast_{\text{1d}}\mathbf{\tilde{v}}_{\sigma,r})$
denotes the 1D Gaussian convoluted vector component, and
${\mathbf{\tilde{M}}_{\sigma,r}}=(\mathcal{N}_{\text{2d}}\ast_{\text{2d}}\mathbf{\tilde{M}}_{\sigma,r})$
denotes the 2D Gaussian convoluted matrix component. In other words, the 3D
convoluted tensor can be expressed as the composition of individually
convoluted components, which allows us to distribute the 3D Gaussian
convolution across the individual components of the decomposed low-rank
tensor. Similar to Sec. 3.4, the value of the density field is component-wised
linearly sampled from the Gaussian convoluted components, i.e.,
$\tilde{F_{\sigma}}(\textbf{x})=\tilde{\mathcal{T}_{\sigma}}(\textbf{x})$.
Similarly, the spectral restricted version of the color field $F_{c}$ can be
obtained as

$\displaystyle\medmath{\tilde{F_{c}}(\textbf{x},\vec{d})=\mathbf{S}(\tilde{\mathcal{T}}_{c}(\textbf{x}),\vec{d})}$
(16)
$\displaystyle\medmath{\tilde{\mathcal{T}}_{c}=\sum_{r=1}^{\mathbf{R}}{\mathbf{\tilde{v}}^{X}_{c,r}}\otimes\mathbf{\tilde{M}}^{Y,Z}_{c,r}\otimes\mathbf{b}^{X}_{r}+}$
$\displaystyle\quad\quad\quad\medmath{{\mathbf{\tilde{v}}^{Y}_{c,r}}\otimes\mathbf{\tilde{M}}^{X,Z}_{c,r}\otimes\mathbf{b}^{Y}_{r}+{\mathbf{\tilde{v}}^{Z}_{c,r}}\otimes\mathbf{\tilde{M}}^{X,Y}_{c,r}\otimes\mathbf{b}^{Z}_{r}}.$

With _separable component-wise convolution_ , the time complexity required is
$O(I\cdot J\cdot L_{\mathcal{N}}+J\cdot K\cdot L_{\mathcal{N}}+K\cdot I\cdot
L_{\mathcal{N}})$ for computing convoluted components (assuming that we
separate 2D Gaussian convolution on matrix components into 1D Gaussian
convolutions), and $O(\mathbf{R})$ for each query sample (same as the original
decomposed tensor in (Chen et al. 2022)), drastically reducing the computation
required for filtering 3D radiance fields $F_{\sigma}$ and $F_{c}$.

We stress here that our proposed component-wise convolution is different from
traditional technique of separated kernel convolution in signal processing
literature, in the sense that the common separated kernel technique only
separates the 3D kernel without utilizing the separability of the input signal
itself, and hence requires sequentially performing three 1D convolution
operation on 3D volume, the time complexity of traditional technique would be
$O(I\cdot J\cdot K\cdot L_{\mathcal{N}})$, and also requires a 3-dimensional
memory with space complexity of $(I\cdot J\cdot K)$ to store convolution
result.

###  3.6 Techniques for Increasing Pose Robustness

Here we summarize our improvements on naïve decomposed low-rank tensors that
improve joint camera pose optimization and radiance field reconstruction.

#### Coarse-to-Fine 3D schedule.

Using efficient 3D convolution algorithm in Sec. 3.5. During training, we
apply a coarse-to-fine schedule on the 3D radiance field
$\tilde{F}_{\sigma},\tilde{F}_{c}$ by controlling the kernel parameter
($\sigma$ of Eq. 11) of the Gaussian kernel, which is exponentially reduced to
0 at 10k iterations and remains 0 afterward (for detailed settings of
$\sigma$, please refer to the supplement).

#### Smoothed 2D Supervision.

Inspired by the analysis in Sec. 3.2, we discovered that blurring the 2D
training image with a parallel set of scheduled 2D Gaussian kernels also helps
the joint optimization. On the one hand, smoothed supervision images produce
smoothed image gradients and stabilize the camera alignment. On the other
hand, smoothed training image also helps to restrict the spectrum of the
learned 3D scene. The Gaussian schedule for smoothing 2D training images is
similar to that of the 3D radiance fields .

#### Randomly Scaled Kernel Parameter and Edge Guided Loss.

From the previous spectral analysis in Sec. 3.2, one may have the impression
that a larger kernel leads to stronger modulation, and hence always results in
more robust pose registration. However, this is not always true, because the
magnitude of $H(u,k)$ decreases linearly as $k$ approaches $0$. Notice that in
Fig. 3(b) the magnitude of modulated $\tilde{H}$ is weaker than that of $H$,
which means that $\frac{d}{du}\tilde{\mathcal{L}}_{\text{1d}}$ is weaker than
$\frac{d}{du}\mathcal{L}_{\text{1d}}$ and therefore is more easily influenced
by noise. In the 3D case, this _weak and noisy gradient problem_ caused by
overly aggressive filtering corresponds to the excessive blur effect that
destroys important edge signals in the training images, causing pose alignment
to fail. See Fig. 4(b) for a visualization of the image blurred by an over-
strength kernel, in which the thin edge information is eliminated, causing the
camera pose to randomly drift.

Figure 4: Visualization of 2D Randomly Sampled Kernel and Edge Guided Loss.
(a) Input supervision without kernel. Joint optimization using unblurred
images easily overfit to high-frequency noises (b) Input supervision blurred
by an overly aggressive kernel. Notice that the edge information is largely
destroyed by the blurring process, resulting in weak and noisy gradients,
causing the poses to drift around easily. (c) Same input supervision blurred
by four randomly scaled kernels. We empirically found that mixing different
filtering strengths results in a more robust joint optimization. (d) We select
edge area of a blurred image by Sobel filter with a threshold set to 1.25x of
the average value of the filtered edge-strength map.

Based on the effect of _weak and noisy gradient problem_ , when applying only
_coarse-to-fine 3D schedule_ and _smoothed 2D supervision_ , we found that it
is insufficient to use a single-size kernel on different real-world scene
structures (in which the same kernel may be overly aggressive in one scene,
but overly gentle in another scene). Therefore, we introduce _randomly scaled
kernel_ , which randomly scales the kernel by a factor uniformly sampled from
$[0,1]$. Random scales are sampled independently among 3D Gaussian kernels
(for the radiance field) and 2D Gaussian kernels (for training images),
allowing combinations of different-sized kernels to guide the joint
optimization. See Fig. 4(c) for a visualization of the same input image
filtered by a range of randomly sampled kernels. We observe that the training
schedule becomes more robust when we alternate between these randomly sampled
kernel scales.

Another way to mitigate the weak and noisy gradient problem is the _edge
guided loss_ , in which we increase the learning rate by 1.5x (and hence
amplify the gradient signal) on the pixels in the edge region, from which the
learning signal for pose alignment mainly comes. See visualization in Fig. 4
(d), where we color the edge area that is detected using the Sobel filter
(Kanopoulos, Vasanthavada, and Baker 1988) on the filtered 2D images in
yellow. Edge-guided rendering loss helps the joint optimization focuses more
on the edge area of the training images, resulting in more robust pose
optimization. Empirically we apply this edge-guided scale alternately on every
other training iteration.

##  4 Experiments

Although our method is applicable to various decomposed low-rank tensor
implementations, in this section, we validate our proposed method using
TensoRF (Chen et al. 2022) with inaccurate or unknown camera poses.

We evaluate our proposed method against three previous works BARF (Lin et al.
2021), GARF (Chng et al. 2022), and HASH (Heo et al. 2023). Since the
implementation of GARF and HASH are unavailable, we directly use the results
reported in their paper for comparison. We compare these methods on the planar
image alignment task and novel view synthesis task on NeRF-Synthetic and LLFF
dataset. We provide detailed implementation details and experimental setup in
the supplementary material.

###  4.1 Results

#### Planar Image Alignment (2D).

In Fig. 5 we compare our method (i.e., 2D TensoRF + 2D Gaussian) with naïve 2D
TensoRF implementation (Chen et al. 2022) and BARF (Lin et al. 2021).
Quantitative results are reported in Tab. 1, including $\mathfrak{sl}(3)$ warp
error and patch PSNR. These results demonstrate the effectiveness of Gaussian
filtering in joint optimization, verifying the analysis in Sec. 3.2.

Methods |  $\mathfrak{sl}(3)$ error $\downarrow$ | patch PSNR $\uparrow$
---|---|---
BARF | 0.0105 | 35.19
Naïve 2D TensoRF | 0.5912 | 20.80
2D TensoRF + 2D Gaussian | 0.0023 | 40.70
Table 1: Quantitative results of planar image alignment.

Figure 5: Qualitative comparisons of the 2D image patch alignment. _2D TensoRF
+ 2D Gaussian_ successfully registers accurate warping parameters, verifying
the analysis of Gaussian filtering on joint optimization.

Figure 6: Visual comparisons of novel view synthesis. Scene | Camera Pose Registration | View Synthesis Quality
---|---|---
Rotation $({}^{\circ})$ $\downarrow$ | Translation $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
GARF | BARF | HASH | Ours | GARF | BARF | HASH | Ours | GARF | BARF | HASH | Ours | GARF | BARF | HASH | Ours
Chair | 0.113 | 0.096 | 0.085 | 0.874 | 0.549 | 0.428 | 0.365 | 3.501 | 31.32 | 31.16 | 31.95 | 35.22 | 0.959 | 0.954 | 0.962 | 0.984
Drum | 0.052 | 0.043 | 0.041 | 0.037 | 0.232 | 0.225 | 0.214 | 0.118 | 24.15 | 23.91 | 24.16 | 25.78 | 0.909 | 0.900 | 0.912 | 0.934
Ficus | 0.081 | 0.085 | 0.079 | 0.050 | 0.461 | 0.474 | 0.479 | 0.173 | 26.29 | 26.26 | 28.31 | 31.37 | 0.935 | 0.934 | 0.943 | 0.978
Hotdog | 0.235 | 0.248 | 0.229 | 0.105 | 1.123 | 1.308 | 1.123 | 0.499 | 34.69 | 34.54 | 35.41 | 37.18 | 0.972 | 0.970 | 0.981 | 0.982
Lego | 0.101 | 0.082 | 0.071 | 0.049 | 0.299 | 0.291 | 0.272 | 0.100 | 29.29 | 28.33 | 31.65 | 34.23 | 0.925 | 0.927 | 0.973 | 0.981
Materials | 0.842 | 0.844 | 0.852 | 0.854 | 2.688 | 2.692 | 2.743 | 2.690 | 27.91 | 27.84 | 27.14 | 29.04 | 0.941 | 0.936 | 0.911 | 0.951
Mic | 0.070 | 0.071 | 0.068 | 1.177 | 0.293 | 0.301 | 0.287 | 5.000 | 31.39 | 31.18 | 32.33 | 32.50 | 0.971 | 0.969 | 0.975 | 0.976
Ship | 0.073 | 0.075 | 0.079 | 0.058 | 0.310 | 0.326 | 0.287 | 0.167 | 27.64 | 27.50 | 27.92 | 31.98 | 0.862 | 0.849 | 0.879 | 0.903
Mean | 0.195 | 0.193 | 0.189 | 0.400 | 0.744 | 0.756 | 0.722 | 1.533 | 28.96 | 28.84 | 29.86 | 32.07 | 0.935 | 0.930 | 0.943 | 0.961
Table 2:  Quantitative results on the NeRF-Synthetic dataset. Our method achieves the best average novel-view synthesis quality and the best pose error in 5 out of 8 scenes. Notice that our method converges within 40k iterations, while all previous methods train for 200k iterations.  Scene | Camera Pose Registration | View Synthesis Quality
---|---|---
Rotation $({}^{\circ})$ $\downarrow$ | Translation $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
GARF | BARF | HASH | Ours | GARF | BARF | HASH | Ours | GARF | BARF | HASH | Ours | GARF | BARF | HASH | Ours
Fern | 0.470 | 0.191 | 0.110 | 0.472 | 0.250 | 0.102 | 0.102 | 0.199 | 24.51 | 23.79 | 24.62 | 26.17 | 0.740 | 0.710 | 0.743 | 0.842
Flower | 0.460 | 0.251 | 0.301 | 1.375 | 0.220 | 0.224 | 0.211 | 0.389 | 26.40 | 23.37 | 25.19 | 25.62 | 0.790 | 0.698 | 0.744 | 0.810
Fortress | 0.030 | 0.479 | 0.211 | 0.449 | 0.270 | 0.364 | 0.241 | 0.419 | 29.09 | 29.08 | 30.14 | 29.68 | 0.820 | 0.823 | 0.901 | 0.882
Horns | 0.030 | 0.304 | 0.049 | 0.386 | 0.210 | 0.222 | 0.209 | 0.251 | 22.54 | 22.78 | 22.97 | 22.84 | 0.690 | 0.727 | 0.736 | 0.819
Leaves | 0.130 | 1.272 | 0.840 | 1.990 | 0.230 | 0.249 | 0.228 | 0.397 | 19.72 | 18.78 | 19.45 | 21.24 | 0.610 | 0.537 | 0.607 | 0.753
Orchids | 0.430 | 0.627 | 0.399 | 0.279 | 0.410 | 0.404 | 0.386 | 0.340 | 19.37 | 19.45 | 20.02 | 20.57 | 0.570 | 0.574 | 0.610 | 0.698
Room | 0.270 | 0.320 | 0.271 | 0.188 | 0.200 | 0.270 | 0.213 | 0.191 | 31.90 | 31.95 | 32.73 | 31.87 | 0.940 | 0.949 | 0.968 | 0.936
T-Rex | 0.420 | 1.138 | 0.894 | 0.523 | 0.360 | 0.720 | 0.474 | 0.416 | 22.86 | 22.55 | 23.19 | 24.19 | 0.800 | 0.767 | 0.866 | 0.878
Mean | 0.280 | 0.573 | 0.384 | 0.709 | 0.269 | 0.331 | 0.258 | 0.325 | 24.55 | 23.97 | 24.79 | 25.27 | 0.745 | 0.723 | 0.772 | 0.827
Table 3:  Quantitative results on the LLFF dataset. Our method achieves the
best average novel-view synthesis quality and best LPIPS in 7 out of 8 scenes.
Our method converges within 50k iterations, while all previous methods train
for 200k iterations.

#### NeRF (3D): Synthetic Object & Real World Objects.

Tab. 2 reports the pose error and novel-view synthesis quality of the NeRF-
Synthetic dataset. Our method achieves the smallest pose error in 5 out of 8
scenes and achieves the best reconstruction quality in all eight scenes, and
the quantitative results are shown in Fig. 6.

Tab. 3 reports the pose error and novel-view synthesis quality of the LLFF
dataset. Our method achieves pose error on par with previous methods and
produces the best average view synthesis quality. Our method also scores the
best LPIPS in 7 out of 8 scenes, indicating that our method produces
perceptually more natural novel-view synthesis.

Note that we achieve state-of-the-art results within only 20% to 25% of
training iterations, while all other competing methods train for 200k
iterations.

|  3D |  2D |  Random |  Edge | Rot. | Trans. | PSNR
---|---|---|---|---|---|---|---
|  Gauss. |  Gauss. |  Kernel |  Guided | $\downarrow$ | $\downarrow$ | $\uparrow$
(a) | ✓ | ✓ | ✓ | ✓ | 0.72 | 0.33 | 25.36
(b) | ✓ | ✓ |  | ✓ | 1.00 | 0.37 | 25.25
(c) | ✓ | ✓ |  |  | 1.91 | 0.93 | 25.12
(d) | ✓ |  |  |  | 33.00 | 12.7 | 20.10
(e) |  | ✓ |  |  | 26.25 | 8.9 | 19.73
(d) |  |  |  |  | 23.29 | 9.4 | 23.97
Table 4: Ablation study of the components of the proposed method on the real-world LLFF dataset. | Rot. $\downarrow$ | Trans. $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---|---
TensoRF + BARF | 45.47 | 0.17 | 20.71 | 0.630 | 0.314
TensoRF + GARF | 73.92 | 0.29 | 10.47 | 0.287 | 0.679
Ours | 0.43 | 0.003 | 26.92 | 0.872 | 0.104
Table 5: Ablation on Directly Applying BARF and GARF on TensoRF (Potential Baseline) Filter | Rot. $\downarrow$ | Trans. $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---|---
Box filter | 9.98 | 0.06 | 20.18 | 0.387 | 0.165
Gaussian filter | 0.46 | 0.004 | 29.49 | 0.874 | 0.063
Table 6: Ablation On Low-Pass Filters. Setting | Rot. $\downarrow$ | Trans. $\downarrow$ | PSNR $\uparrow$
---|---|---|---
w/ random kernels & edge guided loss | 0.06 | 0.002 | 34.34
w/o random kernels & edge guided loss | 0.28 | 0.010 | 34.43
Table 7: Ablation on Applying Randomly Scaled Kernel Parameter and Edge Guided Loss in Synthetic Scenes $\sigma$ | $0.125$ | $0.15$ | $0.175$ | $0.2$
---|---|---|---|---
BARF | Rotation $\downarrow$ | 0.094 | 0.068 | 0.100 | 0.108
Translation $\downarrow$ | 0.004 | 0.004 | 0.005 | 0.005
Ours | Rotation $\downarrow$ | 0.07 | 0.062 | 0.072 | 0.066
Translation $\downarrow$ | 0.003 | 0.003 | 0.003 | 0.002
Table 8: Ablation: Sensitivity Analysis On Gaussian Noise in Blender Chair.

###  4.2 Ablation

#### Component Analysis.

In Tab. 4, we report the effect of each proposed component on the pose error
and PSNR of the optimization results. The results are average across all real-
world scenes in the LLFF dataset. In (a) (b), we show the effect of _randomly
scaled kernel_ described in Sec. 3.6. In (b)(c), we show the effectiveness of
_edge guided loss_ (Sec. 3.6). Finally, in (c)(d)(e), we show the necessity of
Gaussian filtering on both 2D supervising images and 3D radiance field
represented by a decomposed tensor grid, which validates the analysis in Sec.
3.2.

#### Potential Baseline of TensoRF with BARF/GARF.

One may suspect that we can solve the joint optimization problem of
_decomposed low-rank tensor_ by simply applying the method of (Lin et al.
2021) or (Chng et al. 2022), we clarify that there exists no simple way of
integrating BARF (i.e., gradually activating higher-frequency components in
positional encoding) into TensoRF since the MLP decoder of TensoRF does not
take spatial coordinates as input (i.e., controlling spatial property in
TensoRF is hard to achieve by manipulating positional encoding). Nevertheless,
we make rough attempts to add a positional encoding schedule into the MLP
decoder input to simulate the setting of BARF or replace the decoder with a
GARF network. We conduct experiments on four randomly chosen scenes in the
LLFF dataset. The results are shown in Tab. 5, which demonstrate the efficacy
and pertinency of our proposed method to achieve successful training.

#### Using Other Low-Pass Filters.

As we would like to have identical filtering strength along all spatial
directions, we adopt the Gaussian filter in our method as it is the only
kernel that is both circularly symmetric and separable (a well-known property
in signal processing). Nevertheless, we experiment with other low-pass
filters. We report in Tab. 6 the performance of using the box filter (i.e., a
representative low-pass filter) on the LLFF Fortress scene, in which we
clearly observe the benefits of using the Gaussian filter.

#### Applying Randomly Scaled Kernel Parameter. and Edge Guided Loss on
Synthetic Scenes.

Although the two techniques are originally proposed to improve the robustness
of complex real-world scenes, they do not harm the performance of synthetic
ones and even slightly boost the pose estimation, as shown in Tab. 7.

#### Sensitivity w.r.t. Pose Initialization.

We adopt the Chair scenes in the Blender dataset to conduct sensitivity
analysis upon pose initialization via varying variance $\sigma$ of Gaussian
noise. The result is shown in Tab. 8, which demonstrates that both BARF and
our proposed method show certain robustness against the noisy initialization
of camera poses.

Figure 7: PSNR and training iterations comparison.

###  4.3 Time Complexity

In Fig. 7, we compare with previous methods on average PSNR and training
iterations in the _Synthetic NeRF_ dataset. The figure shows two advantages of
our method: (1) rapid convergence and (2) high-quality novel view synthesis.

The early-stage blurry supervision can hinder detailed structure
reconstruction later in the optimization, impacting the final result quality.
Our method resolves this problem by applying 3D filters with directly
controllable kernel parameters, which enables smooth and rapid transition (by
continuous exponential kernel schedule) of the 3D content across the spectrum
domains, as opposed to previous methods that use indirect methods (e.g.,
learning rate in (Heo et al. 2023), encoding magnitude in (Lin et al. 2021))
to influence learned 3D scene spectral property. Furthermore, our method is
carefully designed to use a single voxel grid, which is trained only once in
the coarse-to-fine schedule controlled by our proposed efficient component-
wised convolution algorithm, thus leading to faster convergence; in
comparison, (Heo et al. 2023), which also uses voxel-based representation,
requires sequential curriculum learning upon multiple voxel grids of different
resolutions, resulting in four times more training iterations than ours.

##  5 Conclusion

Our contributions is three fold: 1) _Theoretically_ , we provide insights into
the impact of 3D scene properties on the convergence of joint optimization
beyond the coarse-to-fine heuristic discussed in prior research (e.g., BARF,
Heo et al. 2023), thus offering a filtering strategy for improving the joint
optimization of camera pose and 3D radiance field. 2) _Algorithmically_ , we
introduce (and prove the equivalence of) an effective method for applying the
pilot study’s filtering strategy on the decomposed low-rank tensor, notice
that the proposed separable component-wise convolution is more efficient than
the traditionally well-known trick of separable convolution kernel as we
additionally utilize the separability of the input signal. Furthermore, we
also propose other techniques such as randomly-scaled kernel parameter,
blurred 2D supervision, and edge-guided loss mask to help our proposed method
better perform in complex real-world scenes. 3) Comprehensive evaluations
demonstrate our proposed framework’s state-of-the-art performance and rapid
convergence without known poses.

## Acknowledgments

This work is supported by National Science and Technology Council (NSTC)
111-2628-E-A49-018-MY4, 112-2221-E-A49-087-MY3, 112-2222-E-A49-004-MY2, and
Higher Education Sprout Project of the National Yang Ming Chiao Tung
University, as well as the Ministry of Education (MoE), Taiwan. In particular,
Yu-Lun Liu acknowledges the Yushan Young Fellow Program by the MoE in Taiwan.

## References

  * Chen et al. (2022) Chen, A.; Xu, Z.; Geiger, A.; Yu, J.; and Su, H. 2022.  Tensorf: Tensorial radiance fields.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Chen et al. (2023a) Chen, Y.; Chen, X.; Wang, X.; Zhang, Q.; Guo, Y.; Shan, Y.; and Wang, F. 2023a.  Local-to-global registration for bundle-adjusting neural radiance fields.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Chen et al. (2023b) Chen, Z.; Funkhouser, T.; Hedman, P.; and Tagliasacchi, A. 2023b.  MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Chng et al. (2022) Chng, S.-F.; Ramasinghe, S.; Sherrah, J.; and Lucey, S. 2022.  Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Fridovich-Keil et al. (2023) Fridovich-Keil, S.; Meanti, G.; Warburg, F. R.; Recht, B.; and Kanazawa, A. 2023.  K-Planes: Explicit Radiance Fields in Space, Time, and Appearance.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Fridovich-Keil et al. (2023) Fridovich-Keil, S.; Meanti, G.; Warburg, F. R.; Recht, B.; and Kanazawa, A. 2023.  K-Planes: Explicit Radiance Fields in Space, Time, and Appearance.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Fridovich-Keil et al. (2022) Fridovich-Keil, S.; Yu, A.; Tancik, M.; Chen, Q.; Recht, B.; and Kanazawa, A. 2022.  Plenoxels: Radiance Fields without Neural Networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Goel et al. (2022) Goel, R.; Dhawal, S.; Saini, S.; and Narayanan, P. J. 2022.  StyleTRF: Stylizing Tensorial Radiance Fields.  In _Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing_.
  * Han and Xiang (2023) Han, K.; and Xiang, W. 2023.  Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Hedman et al. (2021) Hedman, P.; Srinivasan, P. P.; Mildenhall, B.; Barron, J. T.; and Debevec, P. 2021.  Baking Neural Radiance Fields for Real-Time View Synthesis.  _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Heo et al. (2023) Heo, H.; Kim, T.; Lee, J.; Lee, J.; Kim, S.; Kim, H. J.; and Kim, J.-H. 2023.  Robust Camera Pose Refinement for Multi-Resolution Hash Encoding.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Hu et al. (2023) Hu, T.; Xu, X.; Chu, R.; and Jia, J. 2023.  TriVol: Point Cloud Rendering via Triple Volumes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kajiya and Von Herzen (1984) Kajiya, J. T.; and Von Herzen, B. P. 1984.  Ray tracing volume densities.  _ACM SIGGRAPH computer graphics_.
  * Kanopoulos, Vasanthavada, and Baker (1988) Kanopoulos, N.; Vasanthavada, N.; and Baker, R. L. 1988.  Design of an image edge detection filter using the Sobel operator.  _IEEE Journal of solid-state circuits_.
  * Kulhanek and Sattler (2023) Kulhanek, J.; and Sattler, T. 2023.  Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedra.  _arXiv preprint arXiv:2304.09987_.
  * Lin et al. (2021) Lin, C.-H.; Ma, W.-C.; Torralba, A.; and Lucey, S. 2021.  BARF: Bundle-Adjusting Neural Radiance Fields.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Liu et al. (2020) Liu, L.; Gu, J.; Lin, K. Z.; Chua, T.-S.; and Theobalt, C. 2020.  Neural Sparse Voxel Fields.  _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Liu et al. (2023) Liu, Y.-L.; Gao, C.; Meuleman, A.; Tseng, H.-Y.; Saraf, A.; Kim, C.; Chuang, Y.-Y.; Kopf, J.; and Huang, J.-B. 2023.  Robust Dynamic Radiance Fields.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Meuleman et al. (2023) Meuleman, A.; Liu, Y.-L.; Gao, C.; Huang, J.-B.; Kim, C.; Kim, M. H.; and Kopf, J. 2023.  Progressively Optimized Local Radiance Fields for Robust View Synthesis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Mildenhall et al. (2020) Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; and Ng, R. 2020.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Müller et al. (2022) Müller, T.; Evans, A.; Schied, C.; and Keller, A. 2022.  Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.  _ACM Transactions on Graphics (TOG)_.
  * Schönberger and Frahm (2016) Schönberger, J. L.; and Frahm, J.-M. 2016.  Structure-from-Motion Revisited.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Shao et al. (2023) Shao, R.; Zheng, Z.; Tu, H.; Liu, B.; Zhang, H.; and Liu, Y. 2023.  Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Sun, Sun, and Chen (2022) Sun, C.; Sun, M.; and Chen, H. 2022.  Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Tang et al. (2022) Tang, J.; Chen, X.; Wang, J.; and Zeng, G. 2022.  Compressible-Composable NeRF via Rank-residual Decomposition.  _Advances in Neural Information Processing Systems_.
  * Wang et al. (2022) Wang, L.; Zhang, J.; Liu, X.; Zhao, F.; Zhang, Y.; Zhang, Y.; Wu, M.; Yu, J.; and Xu, L. 2022.  Fourier plenoctrees for dynamic radiance field rendering in real-time.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wang et al. (2021) Wang, Z.; Wu, S.; Xie, W.; Chen, M.; and Prisacariu, V. A. 2021.  NeRF$--$: Neural Radiance Fields Without Known Camera Parameters.  _arXiv preprint arXiv:2102.07064_.
  * Xu et al. (2022) Xu, Q.; Xu, Z.; Philip, J.; Bi, S.; Shu, Z.; Sunkavalli, K.; and Neumann, U. 2022.  Point-nerf: Point-based neural radiance fields.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Xu et al. (2023) Xu, Y.; Wang, L.; Zhao, X.; Zhang, H.; and Liu, Y. 2023.  AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels.  In _ACM SIGGRAPH 2023 Conference Proceedings_.
  * Yu et al. (2021) Yu, A.; Li, R.; Tancik, M.; Li, H.; Ng, R.; and Kanazawa, A. 2021.  PlenOctrees for Real-time Rendering of Neural Radiance Fields.  _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Yüce et al. (2022) Yüce, G.; Ortiz-Jiménez, G.; Besbinar, B.; and Frossard, P. 2022.  A structured dictionary perspective on implicit neural representations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2020) Zhang, K.; Riegler, G.; Snavely, N.; and Koltun, V. 2020.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_.


# Evaluation of Pre-Trained CNN Models for Geographic Fake Image Detection

Sid Ahmed Fezza1, Mohammed Yasser Ouis1, Bachir Kaddar2, Wassim Hamidouche3,
Abdenour Hadid4 1National Higher School of Telecommunications and ICT, Oran,
Algeria
2University of Ibn Khaldoun, Tiaret, Algeria
3Univ. Rennes, INSA Rennes, CNRS, IETR - UMR 6164, Rennes, France
4Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi,
Abu Dhabi, UAE
<EMAIL_ADDRESS>
###### Abstract

Thanks to the remarkable advances in generative adversarial networks (GANs),
it is becoming increasingly easy to generate/manipulate images. The existing
works have mainly focused on deepfake in face images and videos. However, we
are currently witnessing the emergence of fake satellite images, which can be
misleading or even threatening to national security. Consequently, there is an
urgent need to develop detection methods capable of distinguishing between
real and fake satellite images. To advance the field, in this paper, we
explore the suitability of several convolutional neural network (CNN)
architectures for fake satellite image detection. Specifically, we benchmark
four CNN models by conducting extensive experiments to evaluate their
performance and robustness against various image distortions. This work allows
the establishment of new baselines and may be useful for the development of
CNN-based methods for fake satellite image detection.

###### Index Terms:

DeepFake, Satellite images, Convolutional neural network, Transfer learning,
Generative adversarial networks.

##  I Introduction

Deep learning has proven to be effective in solving various problems in the
field of computer vision [1]. In particular, with the development of new and
robust generative deep learning algorithms [2], a powerful new application has
emerged known as deepfake. Deepfake consists in generating synthetic media,
i.e., images, audios or videos, with manipulated content.

With the introduction of more advanced generative neural networks (such as
Autoencoders [3] or Generative Adversarial Networks [2]), the availability of
high-performance computational resources (e.g. GPU), and the easy access to a
large number of social medias and public databases [4, 5, 6], deepfake
generation has progressed significantly. Synthetic yet very realistic data
generation has indeed improved significantly [7], making it very difficult to
distinguish between real and synthesized videos, even for a human observer
[8].

Despite their various positive applications [9], there is a potential harm and
malicious use of deepfake, that can negatively affect privacy, democracy and
national security [10, 11]. Hence, the topic of deepfake detection has gained
more attention by the research community [12, 13]. For instance, the Defense
Advanced Research Project Agency (DARPA) initiated a novel research direction
aimed at focusing academic research efforts to accelerate the development of
new and more robust deepfake detection techniques. The National Institute of
Standards and Technology (NIST) and Facebook launched the Media Forensics
Challenge (MFC2018) and the Deepfake Detection Challenge (DFDC) to promote the
development of advanced solutions that detect AI-generated and manipulated
videos. As a result, the number of proposed solutions have increased
significantly to address this challenging problem of deepfake detection. [14].

Satellite images are a valuable and common source of data for cartography and
geographic information science, which are important domains that give humans
the ability to analyze and explore the real geographic world. Deepfake
geography [15], on the other hand, has emerged as a recent concern which can
pose a national security threat [16]. The aim of deepfake geography consists
of producing falsified satellite images by replacing an image at one location
with another image, while still maintaining a realistic appearance for the
entire scene [17]. Despite the remarkable advances in deepfakes detection,
research focus on fake satellite images detection is very limited.

Recently, with the widespread growth of the geographic deepfake problem [18],
researchers started looking at detecting whether satellites images have been
manipulated or not [19]. Among the first attempts to tackle this problem is
the introduction of public fake satellite image dataset [15]. The authors
proposed a GAN-based image-to-image translation approach to generate fake
satellite images. By learning the characteristics of an urban area, a fake
satellite image is then generated by mapping the learned characteristics onto
a different base map. Moreover, the authors proposed a method based on
handcrafted features to determine whether the satellite image is authentic or
fake. Following this work, another interesting method called Geo-DefakeHop
[19] has been proposed. This method is based on parallel subspace learning
(PSL), which consists of multiple filter banks used simultaneously. The
authors focused on the higher frequency components to learn the differences
between real and fake satellite images.

In this paper, we evaluate the performance of several deep convolutional
neural network (CNN) architectures, which have shown state-of-the-art accuracy
for general image classification tasks, for fake satellite images detection.
We conduct a thorough evaluation using the most promising CNNs with the aim of
establishing new baselines and promoting future research in this under-
explored area of geographic fake image detection.

The rest of this paper is structured as follows. The relevant related works
are presented in Section II. In Section III, the methodology is described.
Experimental results and discussions are given in Section IV, and the
conclusions are drawn in Section V.

##  II Related work

Images/videos of faces are by far the most targeted in the fake multimedia
field [20, 21, 6], and very few works deal with fake satellite images. Early
studies identify deepfakes by detecting specific artifacts that occurred on
the most common first-generation manipulated videos [22, 23, 24]. For
instance, in [24], the authors used statistical differences in color
components to distinguish synthetic deepfakes from authentic images. However,
it is difficult to handcraft all the most suitable and meaningful features.

Unlike previous deepfake generation techniques, GAN-based approaches are able
to produce more realistic deepfake videos with high visual quality [25, 26,
27, 28], making the deepfake detection task more challenging. Therefore, more
robust methods have been developed that focus on detecting subtle features and
inconsistencies between frames brought by the GAN model [29]. For instance, in
[30], the authors proposed a detection method based on color features and a
linear Support Vector Machine (SVM) for the final classification, achieving
promising results on the NIST MFC2018 dataset [31]. Zhang et al. [32] used the
bag-of-words method to extract a set of discriminating features that were fed
into various classifiers to discriminate manipulated face images from real
ones. A significant drawback of these approaches is that the models have to be
re-trained again when simple unseen image perturbation attacks (such as noise,
blur, cropping or compression) are presented.

In recent years, it has been shown that deep neural networks tend to perform
better than traditional image forensic tools [33, 34, 35]. Consequently, most
developed methods adopted it to automatically extract discriminating features
to detect deepfakes. These CNN-based approaches mainly rely on the presence of
visual artifacts or inconsistency of intrinsic features between fake and real
images/videos [36]. These artifacts can be detected by well established and
pre-trained CNN models such as VGG16 [37], ResNet50 [38], and so on. For
instance, Rossler et al. [5] used XceptionNet model that was trained on the
Faceforensics++ dataset to detect manipulated images. Nguyen et al. [39]
proposed the use of capsule networks that takes features obtained from the
VGG-19 network [37] to detect forged images and videos. Marra et al. performed
in [40] an interesting study in order to classify correctly unseen types of
GAN-based fake generated data. The authors proposed a multi-task incremental
learning detection method based on the XceptionNet model and achieved
promising results.

Despite the progress made on deepfake detection methods in face images and
videos, research on fake satellite image detection is still limited, mainly
due to the lack of labeled fake satellite image datasets. In [15], the authors
draw the public’s attention to the potential danger inherent in the
progressive evolution of this technology (GeoAI capabilities) and its impact
on our perception of the geographic world. For this purpose, they recently
proposed the first fake satellite image dataset using a GAN-based image-to-
image translation approach. This dataset consists of falsified satellite
images from three cities, including Tacoma, Seattle and Beijing. Furthermore,
the authors identified 26 salient hand-crafted features that show a
significant value difference between authentic and fake satellite images. They
classified these features into three classes, which are spatial, histogram and
frequency features. These salient features are concatenated and fed into an
SVM classifier trained to differentiate fake satellite images from the
authentic ones. Chen et al. [19] introduced a learning method for the
detection of fake satellite images called Geo-DefakeHop. The idea behind their
approach is that GANs can generate high-quality realistic images by
reproducing well the low-frequency responses of synthesized images. However,
it cannot do the same for high-frequency components due to the complexity
constraint. Thus, the authors proposed to use multiple filter banks, named
parallel subspace learning (PSL), selected a few discriminant features from
each filter bank and grouped them for binary classification.

Table I: The characteristics of the four considered CNNs pre-trained models.

Model Parameters Depth Top-1 Accuracy Top-5 Accuracy ResNet50[38] 25,636,712
168 0.749 0.921 VGG16[37] 138,357,544 23 0.713 0.901 InceptionV3[41]
23,851,784 159 0.779 0.937 Xception[42] 22,910,480 126 0.790 0.945

##  III Methodology

In this section, we start by briefly introducing the considered CNN
architectures. Then, we discuss the details of training and tuning these
models using the transfer learning technique for the fake satellite image
detection task.

###  III-A Deep Convolutional Neural Network Architectures

Taking advantage of the proliferation of large datasets in addition to
increasing in computational power, deep neural networks (DNNs) have become the
most efficient methods for many computer vision tasks. For instance, in some
visual recognition tasks, the DNNs are able to recognize images better than
humans.

Over the past few years, many DNNs models have been proposed and have shown
top results on the ImageNet dataset [43], which represents the large-scale
image classification dataset. Among them, we can cite VGG16 [37], InceptionV3
[41], ResNet50 [38] and Xception [42], which we considered in this study. The
number of parameters and the depth of each model are summarized in Table I. In
addition, top-1 and top-5 accuracy that refers to model’s performance on the
ImageNet validation dataset are also provided.

Figure 1: Learning process of transfer learning.

() Base map

() Beijing

() Seattle

Figure 2: Some samples of fake satellite images from the UW dataset [15]. A
neighborhood in Tacoma with landscape features from other cities. (a) The
CartoDB base map of Tacoma, fake satellite images generated with the visual
patterns of (b) Beijing and (c) Seattle.

Following the success of the AlexNet [44] model, VGG network [37] was the
first architecture that relied on small size filters, which improves the CNN
performance. This is due to the fact that two $3\times 3$ convolutions layers
have the same receptive field as a single $5\times 5$ convolution, while
having fewer parameters and taking less computation. VGGNet has two variants,
VGG16 and VGG19, which represent the number of convolutional layers in the
model. InceptionV3 [41] is based on some of the original ideas of GoogleNet
[45] and VGGNet. In InceptionV3, several techniques for optimizing the network
have been exploited, including factorized convolutions, regularization,
dimension reduction, and parallelized computations. Deep residual networks
[38], shortened ResNet, introduced a concept that allowed the development of
much deeper networks. Deeper networks can learn more complex functions and
representations of the inputs that usually lead to better performance.
However, a deep network still faces the challenges of network degradation as
well as exploding or vanishing gradients. In ResNet, residual blocks were
introduced, in which the inputs are added back to their outputs with the aim
of creating identity mappings. Such identity mappings are performed using the
so-called skip or residual connections. Xception [42] takes the principles of
Inception to the extreme. The main idea behind Xception is its depthwise
separable convolution, i.e., a depthwise convolution followed by a pointwise
convolution, which provides better performance than InceptionV3. Xception
adjusted the original inception block by mapping the spatial correlations
separately for each output channel instead of partitioning input data into
multiple compressed chunks and then performing a $1\times 1$ depth conversion
to capture cross-channel correlation.

###  III-B Transfer Learning for Geographic Fake Image Detection

Transfer Learning (TL) is a powerful machine learning technique that involves
reassigning learned classifiers to new tasks. In other words, the knowledge
acquired by the model after training on the data of the first task, encoded in
the weights of the model, can be exploited by a new target task, as
illustrated in Figure 1. Thus, in this work, we investigate the potential of
directly using CNN models pre-trained on ImageNet dataset for fake satellite
image detection. To reach this goal, for each considered model, the last FC
layer is removed and the rest of the network is considered as feature
extractor. Additionally, for classification purposes, three FC layers of size
1024, 512 and 2, respectively, are stacked. For the first three epochs, the
weights of feature extraction, i.e., those of pre-trained network, are
frizzed, after which they are fine-tuned by continuing end-to-end back
propagation. This way, we retrain all layers of the model with fake satellite
image dataset.

Figure 3: Trade-off between number of parameters and model’s F1 score.

##  IV Experimental Results

To assess the performance of the four considered pre-trained CNN models,
namely VGG16 [37], InceptionV3 [41], ResNet50 [38] and Xception [42], we
experimented with the UW fake satellite image dataset [15]. We compared the
performance of these four models to seven methods combining spatial, histogram
and frequency features, as proposed by Zhao et al. [15]. In addition, in order
to assess the robustness against various image distortions, we also considered
the effects of JPEG compression and Gaussian noise on the performance of each
model.

The training of the models is performed with back propagation using ADAM
optimization with a learning rate $lr$ of 0.0001. A dropout is applied with a
rate $r$ equals to 0.2 and each mini-batch contains 32 images.

We adopted three commonly used performance criteria to evaluate the models: F1
score, precision and recall [46].

###  IV-A Dataset

The UW dataset was introduced in 2021 by Zhao et al. [15] as the first
publicly available fake satellite image dataset. The fake satellite images
were generated by CycleGAN from three cities (i.e., Tacoma, Seattle and
Beijing), while its authentic satellite images are collected from Google
Earth’s satellite images. UW contains 8064 satellite images, of which 4032
images are authentic color satellite images of spatial resolution $256\times
256$, and their fake analog-generated images. Figure 2 illustrates some fake
satellite images from UW dataset.

We follow the same experimental setting as described in [15]. The dataset is
randomly split into training (90%) and test (10%) sets. In order to fine-tune
hyperparameters, the training set is further divided into training (70%) and
validation (20%) sets.

###  IV-B Results and Discussion

We compared the detection performance of the four models under three
scenarios: 1) original images from UW dataset, 2) images distorted using JPEG
compression, and finally, 3) images distorted using Gaussian noise. For the
second and third scenarios, it is important to note that distortions were only
applied to the test set.

Table II: Detection performance of the four considered CNNs models on UW
dataset (original images). The top result is highlighted in boldface.

Method Features$\backslash$Architectures F1 Precision Recall Zhao et al. [15]
Spatial 0.858 0.886 0.846 Histogram 0.759 0.749 0.816 Frequency 0.752 0.557
0.967 Spatial + Histogram 0.855 0.861 0.943 Spatial + Frequency 0.790 0.722
0.746 Histogram + Frequency 0.859 0.749 0.917 Spatial + Histogram + Frequency
0.850 0.848 0.928 CNNs TL VGG16 0.975 0.973 0.988 Resnet50 0.990 0.992 0.989
InceptionV3 0.951 0.974 0.942 Xception 0.970 0.971 0.968

Table III: Evaluation of detection performance for images corrupted by JPEG compression standard using three quality factors $QF$. The top result is highlighted in boldface. Method | Features$\backslash$Architectures | JPEG Quality Factor
---|---|---
$QF=90$ | $QF=80$ | $QF=70$
F1 | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall
Zhao et al. [15] | Spatial + Histogram + Frequency | 0.873 | 0.802 | 0.956 | 0.883 | 0.802 | 0.983 | 0.907 | 0.850 | 0.971
CNNs TL | VGG16 | 0.979 | 0.9807 | 0.978 | 0.974 | 0.980 | 0.968 | 0.978 | 0.980 | 0.976
Resnet50 | 0.991 | 0.985 | 0.997 | 0.988 | 0.985 | 0.990 | 0.990 | 0.985 | 0.995
InceptionV3 | 0.934 | 0.945 | 0.923 | 0.920 | 0.944 | 0.896 | 0.920 | 0.944 | 0.896
Xception | 0.948 | 0.972 | 0.925 | 0.933 | 0.971 | 0.899 | 0.912 | 0.970 | 0.860
Table IV: Evaluation of detection performance for images corrupted by additive white Gaussian noise using three standard deviation $\sigma$ values. The top result is highlighted in boldface. Method | Features$\backslash$Architectures | Gaussian Noise
---|---|---
$\sigma=0.02$ | $\sigma=0.06$ | $\sigma=1$
F1 | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall
Zhao et al. [15] | Spatial + Histogram + Frequency | 0.522 | 0.650 | 0.436 | 0.469 | 0.582 | 0.335 | 0.407 | 0.578 | 0.304
CNNs TL | VGG16 | 0.911 | 0.925 | 0.896 | 0.899 | 0.915 | 0.865 | 0.898 | 0.914 | 0.856
Resnet50 | 0.798 | 0.798 | 0.788 | 0.782 | 0.775 | 0.788 | 0.781 | 0.786 | 0.776
InceptionV3 | 0.812 | 0.695 | 0.976 | 0.803 | 0.652 | 0.976 | 0.795 | 0.628 | 0.938
Xception | 0.764 | 0.650 | 0.928 | 0.761 | 0.635 | 0.908 | 0.756 | 0.612 | 0.902

####  IV-B1 Detection performance on the original images

Table II shows the detection performance on the original UW test set. From
this table, we can see that the CNN-based methods achieves much better
performance than the methods of Zhao et al. [15]. In particular, ResNet-50
achieves the best results with an F1 score of 0.990 and outperforms both
InceptionV3 and Xception, although they have relatively the same number of
parameters (see Figure3). Similar high performance is achieved using the VGG16
and Xception models. However, by comparing the results obtained by the methods
based on handcrafted features, i.e., spatial, histogram, and frequency
features, we can observe a significant drop in performance compared to deep
learning based methods. For example, the combination of Histogram + Frequency,
which is the best performing method based on handcrafted features, achieves an
F1 score of 0.859.

####  IV-B2 Detection performance on the JPEG compression images

To examine the effect of compression distortion on the detection performance
of benchmarking methods, all images of test set were compressed by JPEG with
three different quality factors $QF=90$, $80$ and $70$. The results of the
experiments are depicted in Table III. As can be seen from these results, the
impact of JPEG compression on detection performance differs from one method to
another. From Table III, it is clear that the detection performance of
InceptionV3 and Xception deteriorates as the value of $QF$ decreases, while it
remains nearly unchanged for both VGG16 and ResNet-50 regardless of the $QF$.
Surprisingly, the detection performance of the hand-crafted-based methods
improves as the value of the $QF$ decreases. This may be explained by the
image quality metric features included in Zhao et al. [15] method to capture
more discriminant features in the case of compressed images. Another reason is
the fact that the method of Zhao et al. considers different frequency features
allowing high performance in this scenario.

####  IV-B3 Detection performance on the Gaussian noise images

Here, we measure the impact of Gaussian noise on detection performance. For
this, the images of the original test set are corrupted with three $sigma$
Gaussian noise levels. The result is shown in Table IV. As can be seen,
additive Gaussian noise has a negative impact and the detection performance of
all methods drop significantly on distorted images compared to the original
versions. The reason is that noise is usually dominant at high frequencies,
and we know that the most salient features for detecting fake images are at
high frequencies. Thus, the noise disturbs this prominent information.

##  V Conclusion

In this work, we evaluated the efficiency of CNN-based models for the
detection of fake satellite images. To this end, four CNN-based models, namely
VGG16, ResNet-50, InceptionV3 and Xception, were pre-trained on ImageNet
dataset and fine-tuned on UW dataset to distinguish between authentic and
counterfeit satellite images.

Evaluation results in terms of F1 scores, precision, and recall revealed that
the ResNet-50 outperformed all other methods, particularly InceptionV3 and
Xception. Moreover, we compared the performance of the CNN-based methods
against hand-crafted-based methods. Our experiments showed that CNN-based
approaches significantly outperform the hand-crafted-based methods. We also
analyzed the effect of image compression and noise on the detection
performance. Typically, these distortions cause a significant drop in
performance compared to using the original images. In particular, Gaussian
noise has the greatest negative impact on the detection performance compared
to JPEG compression. Surprisingly, the detection performance of the hand-
crafted-based methods seems to be the least affected by image compression. We
plan to release all the data and codes for public use to support the principle
of reproducible research and fair comparison.

## References

  * [1] J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep learning in computer vision: A critical review of emerging techniques and application scenarios,” _Machine Learning with Applications_ , vol. 6, p. 100134, 2021.
  * [2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, pp. 2672–2680, 2014.
  * [3] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in _Proceedings of the 25th international conference on Machine learning_ , 2008, pp. 1096–1103.
  * [4] P. Korshunov and S. Marcel, “Deepfakes: a new threat to face recognition? assessment and detection,” _arXiv preprint arXiv:1812.08685_ , 2018.
  * [5] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1–11.
  * [6] B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, “The deepfake detection challenge (dfdc) dataset,” _arXiv preprint arXiv:2006.07397_ , 2020.
  * [7] H. Ajder, G. Patrini, F. Cavalli, and L. Cullen, “The state of deepfakes: Landscape, threats, and impact,” _Amsterdam: Deeptrace_ , 2019.
  * [8] P. Korshunov and S. Marcel, “Subjective and objective evaluation of deepfake videos,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 2510–2514.
  * [9] B. Marr, “The best (and scariest) examples of ai-enabled deepfakes,” _Re-trieved from https://www. forbes. com/sites/bernardmarr/2019/07/22/the-best-and-scariest-examples-of-aienabled-deepfakes_ , 2019\.
  * [10] B. Chesney and D. Citron, “Deep fakes: a looming challenge for privacy, democracy, and national security,” _Calif. L. Rev._ , vol. 107, p. 1753, 2019\.
  * [11] C. Vaccari and A. Chadwick, “Deepfakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news,” _Social Media+ Society_ , vol. 6, no. 1, p. 2056305120903408, 2020\.
  * [12] R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, “Deepfakes and beyond: A survey of face manipulation and fake detection,” _arXiv preprint arXiv:2001.00179_ , 2020.
  * [13] Y. Mirsky and W. Lee, “The creation and detection of deepfakes: A survey,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 1, pp. 1–41, 2021.
  * [14] S. Lyu, “Deepfake detection,” in _Multimedia Forensics_. Springer, Singapore, 2022, pp. 313–331.
  * [15] B. Zhao, S. Zhang, C. Xu, Y. Sun, and C. Deng, “Deep fake geography? when geospatial data encounter artificial intelligence,” _Cartography and Geographic Information Science_ , vol. 48, no. 4, pp. 338–352, 2021.
  * [16] P. Tucker, “The newest ai-enabled weapon: Deep-faking photos of the earth,” _Defense One. https://www.defenseone.com/technology/2019/03/next-phase-ai-deep-faking-whole-world-and-china-ahead/155944/_ , 2019\.
  * [17] C. Xu and B. Zhao, “Satellite image spoofing: Creating remote sensing dataset with generative adversarial networks (short paper),” in _10th International conference on geographic information science (GIScience 2018)_. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2018.
  * [18] Z. Zhongming, L. Linong, Y. Xiaona, Z. Wangqiang, L. Wei _et al._ , “A growing problem of’deepfake geography’: How ai falsifies satellite images,” 2021\.
  * [19] H.-S. Chen, K. Zhang, S. Hu, S. You, and C.-C. J. Kuo, “Geo-defakehop: High-performance geographic fake image detection,” _arXiv preprint arXiv:2110.09795_ , 2021.
  * [20] L. Verdoliva, “Media forensics and deepfakes: an overview,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 14, no. 5, pp. 910–932, 2020.
  * [21] M. Masood, M. Nawaz, K. M. Malik, A. Javed, A. Irtaza, and H. Malik, “Deepfakes generation and detection: State-of-the-art, open challenges, countermeasures, and way forward,” _Applied Intelligence_ , pp. 1–53, 2022\.
  * [22] A. Agarwal, R. Singh, M. Vatsa, and A. Noore, “Swapped! digital face presentation attack detection via weighted local magnitude pattern,” in _2017 IEEE International Joint Conference on Biometrics (IJCB)_. IEEE, 2017, pp. 659–665.
  * [23] Z. Akhtar and D. Dasgupta, “A comparative evaluation of local feature descriptors for deepfakes detection,” in _2019 IEEE International Symposium on Technologies for Homeland Security (HST)_. IEEE, 2019, pp. 1–5.
  * [24] F. Matern, C. Riess, and M. Stamminger, “Exploiting visual artifacts to expose deepfakes and face manipulations,” in _2019 IEEE Winter Applications of Computer Vision Workshops (WACVW)_. IEEE, 2019, pp. 83–92.
  * [25] T. Karras, T. Aila, S. Laine, and J.-k. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [26] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normalization for generative adversarial networks,” _arXiv preprint arXiv:1802.05957_ , 2018\.
  * [27] M. G. Bellemare, I. Danihelka, W. Dabney, S. Mohamed, B. Lakshminarayanan, S. Hoyer, and R. Munos, “The cramer distance as a solution to biased wasserstein gradients,” _arXiv preprint arXiv:1705.10743_ , 2017.
  * [28] M. Bińkowski, D. J. Sutherland, M.-c. Arbel, and A. Gretton, “Demystifying mmd gans,” _arXiv preprint arXiv:1801.01401_ , 2018.
  * [29] A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics: A large-scale video dataset for forgery detection in human faces,” _arXiv preprint arXiv:1803.09179_ , 2018.
  * [30] S. McCloskey and M. Albright, “Detecting gan-generated imagery using color cues,” _arXiv preprint arXiv:1812.08247_ , 2018.
  * [31] H. Guan, M. Kozak, E. Robertson, Y. Lee, A. N. Yates, A. Delgado, D. Zhou, T. Kheyrkhah, J. Smith, and J. Fiscus, “Mfc datasets: Large-scale benchmark datasets for media forensic challenge evaluation,” in _2019 IEEE Winter Applications of Computer Vision Workshops (WACVW)_. IEEE, 2019, pp. 63–72.
  * [32] Y. Zhang, L. Zheng, and V. L. Thing, “Automated face swapping and its detection,” in _2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP)_. IEEE, 2017, pp. 15–19.
  * [33] B. Bayar and M. C. Stamm, “A deep learning approach to universal image manipulation detection using a new convolutional layer,” in _Proceedings of the 4th ACM workshop on information hiding and multimedia security_ , 2016, pp. 5–10.
  * [34] N. Rahmouni, V. Nozick, J. Yamagishi, and I. Echizen, “Distinguishing computer graphics from natural images using convolution neural networks,” in _2017 IEEE Workshop on Information Forensics and Security (WIFS)_. IEEE, 2017, pp. 1–6.
  * [35] F. Marra, D. Gragnaniello, D. Cozzolino, and L. Verdoliva, “Detection of gan-generated fake images over social networks,” in _2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_. IEEE, 2018, pp. 384–389.
  * [36] S. Tariq, S. Lee, H. Kim, Y. Shin, and S. S. Woo, “Detecting both machine and human created fake face images in the wild,” in _Proceedings of the 2nd international workshop on multimedia privacy and security_ , 2018, pp. 81–87.
  * [37] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pat-tern recognition_ , 2016, pp. 770–778.
  * [39] H. H. Nguyen, J. Yamagishi, and I. Echizen, “Capsule-forensics: Using capsule networks to detect forged images and videos,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 2307–2311.
  * [40] F. Marra, C. Saltori, G. Boato, and L. Verdoliva, “Incremental learning for the detection and classification of gan-generated images,” in _2019 IEEE International Workshop on Information Forensics and Security (WIFS)_. IEEE, 2019, pp. 1–6.
  * [41] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [42] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [43] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [44] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [46] M. Sokolova, N. Japkowicz, and S. Szpakowicz, “Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation,” in _Australasian joint conference on artificial intelligence_. Springer, 2006, pp. 1015–1021.

  *[GAN]: Generative Adversarial Network
  *[CNN]: convolutional neural network
  *[CNNs]: convolutional neural network