# Data augmentation for NeRF: a geometric consistent solution based on view
morphing

Matteo Bortolon1,2,3 Alessio Del Bue2 Fabio Poiesi1,2
1TeV  Fondazione Bruno Kessler 2PAVIS  Istituto Italiano di Tecnologia
3DISI  University of Trento
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

NeRF aims to learn a continuous neural scene representation by using a finite
set of input images taken from different viewpoints. The fewer the number of
viewpoints, the higher the likelihood of overfitting on them. This paper
mitigates such limitation by presenting a novel data augmentation approach to
generate geometrically consistent image transitions between viewpoints using
view morphing. View morphing is a highly versatile technique that does not
requires any prior knowledge about the 3D scene because it is based on general
principles of projective geometry. A key novelty of our method is to use the
very same depths predicted by NeRF to generate the image transitions that are
then added to NeRF training. We experimentally show that this procedure
enables NeRF to improve the quality of its synthesised novel views in the case
of datasets with few training viewpoints. We improve PSNR up to 1.8dB and
10.5dB when eight and four views are used for training, respectively. To the
best of our knowledge, this is the first data augmentation strategy for NeRF
that explicitly synthesises additional new input images to improve the model
generalisation.

##  1 Introduction

Novel View Synthesis (NVS) is the problem of synthesising unseen camera views
from a set of known views111Throughout the paper, we will use the term
viewpoint to refer to the camera pose, view to refer to the scene seen through
a certain viewpoint and to image to refer to the photometric content captured
from a view. [27, 7]. NVS is an enabling technology that can be used for
augmented or virtual reality [9], entertainment [5], and robotics [10]
applications. NVS has undergone a significant improvement after the
introduction of Neural Radiance Fields (NeRF) [18, 1], which consists of a
trainable implicit neural representation of a 3D scene that can
photorealistically render unseen (novel) views. NeRF is typically trained with
several images, e.g. about hundred images taken from different and uniformly
distributed camera viewpoints around an object of interest [18]. If viewpoints
are not uniformly distributed or they are few (e.g. four or eight), the
resulting NeRF model may fail to produce satisfactory renderings [11, 17].
This is a practical problem that may occur when non-experts acquire images of
an object in the wild [15] or when it is not possible to view objects from
every side. When NeRF is trained on few and non-uniformly distributed
viewpoints, the likelihood of overfitting on these viewpoints increases,
namely the few-shot view synthesis problem [11]. Approaches that aim at
addressing few-shot view synthesis include DietNeRF and pixelNeRF [11, 28].
The drawback of these methods is that they rely on auxiliary deep networks,
which have to be trained on other datasets, thus requiring additional loss
terms and increasing NeRF’s computational burden. Alternatively, data
augmentation solutions can be used to mitigate overfitting [24], however they
have not yet been explored for few-shot view synthesis.

Figure 1: Given a set of (1) known views (ground truth), (2) our approach
(DVM) generates image transitions between views (morph) that can be
effectively used as data augmentation (3) to train a NeRF model in the case of
few-shot view synthesis. (4) Results are of a higher quality when DVM is used.

In this paper, we propose a geometry-based data augmentation strategy that
enables NeRF to efficiently learn implicit representations of scenes captured
from few viewpoints. We reformulate the view morphing technique [23] to
operate alongside NeRF. View morphing allows conventional image morphing
methods to synthesise in-between viewpoint changes while ensuring realistic
image transitions. View morphing does not require any prior knowledge about
the captured 3D scene, and it can synthesise 3D projective transformations
(e.g. 3D rotations, translations, shears) of objects by operating entirely on
the input images. Because view morphing requires a dense pixel correspondence
map between viewpoint pairs, we exploit the depth predicted by NeRF to
retrieve the disparity map (Fig. 1). We implement a modified version of the
rectification algorithm proposed by Fusiello et al. [6] to rectify images and
disparity maps that are used by the view morphing process. The in-between
viewpoints are randomly sampled between each viewpoint pair to produce the
morphed views for training NeRF. We evaluate our approach by using the dataset
used in the original NeRF’s paper [18] and we show that PSNR improves up to
1.8dB and 10.5dB when eight and four views are used for training,
respectively. We also compare our approach with DietNeRF [11] and AugNeRF [4],
and show that our approach can produce higher-quality renderings.

To summarise, our contributions are:

  * •

A novel and effective data augmentation technique for NeRF for few-shot view
synthesis;

  * •

A new view morphing technique based on the NeRF depth output, named DVM;

  * •

We show that DVM enables NeRF to produce higher-quality rendered images.

##  2 Related work

3D scene synthesis can be solved either by using traditional 3D reconstruction
techniques [21] or by adopting methods based on neural rendering [25]. Neural
radiance fields (NeRF) is a recent neural rendering method that can learn a
volumetric representation of an unknown 3D scene approximating its radiance
and density fields from a set of known (ground-truth) views by using a
multilayer perceptron (MLP). This formulation enables differentiable rendering
of unseen views [18]. Because NeRF is designed to optimise its parameters on
one scene based on a set of known views, overfitting is likely to occur when
the views are few. In general, overfitting can be mitigated via data
augmentation, and, to the best of our knowledge, the only method that
addresses data augmentation for NeRF is AugNeRF [4]. AugNeRF aims at improving
NeRF generalisation by using adversarial data augmentation to enforce each ray
and its augmented version to produce the same result. AugNeRF perturbs NeRF at
the input coordinates, the intermediate features and the pre-rendering output.
We experimentally observed that AugNeRF underperforms NeRF when the known
input views are few (e.g. four, eight). Unlike AugNeRF, we do not perturb the
known input views, but we instead synthesise new views (novel 3D projective
transformations) between pairs of known views.

There exist solutions that specifically target the problem of training NeRF
when the input views are few, which are pixelNeRF [28], MVSNeRF [3], and
DietNeRF [11]. PixelNeRF [28] and MVSNeRF [3] condition the NeRF’s output
through features computed with auxiliary deep networks. During the rendering
phase, a feature is processed for each sampled 3D point that is provided as
input to the MLP for the conditioning. Unlike these approaches, DVM does not
rely on an auxiliary deep network, which has to be trained separately and ad-
hoc, to process the known input images. DietNeRF [11] extends NeRF design by
extracting feature vectors of the known images through a CLIP pre-trained
image encoder. DietNeRF training renders some random poses and processes them
using the very same CLIP image encoder. DietNeRF adds a loss component to
minimise the cosine distance between CLIP feature vectors of the images
rendered from random views and of ground-truth images. Unlike DietNeRF, we do
not need an additional consistency loss that is computed through the
additional CLIP Vision Transformer to extract semantic representations of
renderings, but we instead produce actual images from random viewpoints
through view morphing.

##  3 Preliminaries

###  3.1 NeRF overview

NeRF’s objective is to synthesise novel views of a scene by optimising a
volumetric function given a finite set of input views [18]. Let
$f_{\bm{\theta}}$ be the underlying function we aim to optimise. The input to
$f_{\bm{\theta}}$ is a 5D datum that encodes a point on a camera ray, i.e. a
3D spatial location $(x,y,z)$ and a 2D viewing direction $(\theta,\phi)$. Let
$\bm{c}\in\mathbb{R}^{3}$ be the view-dependent emitted radiance (colour) and
$\sigma$ be the volume density that $f_{\bm{\theta}}$ predicts at $(x,y,z)$.
Novel views are synthesised by querying 5D data along the camera rays.
Traditional volume rendering techniques can be used to transform $\bm{c}$ and
$\sigma$ into an image [12, 16]. Because volume rendering is differentiable,
$f_{\bm{\theta}}$ can be implemented as a fully-connected deep network and
learned.

Rendering a view from a novel viewpoint consists of estimating the integrals
of all 3D rays that originate from the camera optic centre and that pass
through each pixels of the camera image plane. Let $\bm{r}$ be a 3D ray. To
make rendering computationally tractable, each ray is represented as a finite
set of 3D spatial locations, which are defined between two clipping distances:
a near one ($t_{n}$) and a far one ($t_{f}$). Let $\Gamma$ be the number of 3D
spatial locations sampled between $t_{n}$ and $t_{f}$. Rendering the colour of
a pixel can be performed by computing

$\hat{\bm{c}}(\bm{r})=\sum_{i=1}^{\Gamma}s(i)\left(1-e^{-\sigma_{i}\delta_{i}}\right)\bm{c}_{i},\quad\text{
where }s(i)=e^{-\sum^{i-1}_{j=1}\sigma_{j}\delta_{j}},$ (1)

where $s(i)$ is the inverse of the volume density that is accumulated up to
the $i^{th}$ spatial location and $(1-e^{-\sigma_{i}\delta_{i}})$ is a
density-based weight component: the higher the density value of a point, the
larger the contribution on the final rendered colour.
$\delta_{i}=t_{i+1}-t_{i}$ is the distance between adjacent sampled 3D spatial
locations, and $\bm{c}_{i}$ and $\sigma_{i}$ are the colour and volume density
predicted by NeRF at the $i^{th}$ spatial location.

Rendering a depth value at a given pixel can be performed by substituting
$\bm{c}_{i}$ with the distance of the $i^{th}$ spatial location with respect
to the camera optic centre, i.e. $z_{i}$. Let $\hat{d}(\bm{r})$ be the
estimated depth value for the ray $\bm{r}$.

The input required to learn NeRF parameters is a set of $N$ images and their
corresponding camera information. Let $\mathcal{I}=\\{\bm{I}_{k}\\}_{k=1}^{N}$
be the training images, and $\mathcal{P}=\\{\bm{P}_{k}\\}_{k=1}^{N}$ and
$\mathcal{K}=\\{\bm{K}_{k}\\}_{k=1}^{N}$ be their corresponding camera poses
and intrinsic parameters, respectively.

Learning $f_{\bm{\theta}}$ is achieved by comparing each ground-truth pixel
$\bm{c}(\bm{r})$ with its predicted counterpart $\hat{\bm{c}}(\bm{r})$. Hence,
the goal is to minimise the following L2-norm objective function

$\mathcal{L}=\frac{1}{\lvert\mathcal{R}\rvert}\sum_{\bm{r}\in\mathcal{R}}\left(\lVert\bm{c}(\bm{r})-\hat{\bm{c}}_{c}(\bm{r})\rVert_{2}^{2}+\lVert\bm{c}(\bm{r})-\hat{\bm{c}}_{f}(\bm{r})\rVert_{2}^{2}\right),$
(2)

where $\hat{\bm{c}}_{c}(\bm{r})$ and $\hat{\bm{c}}_{f}(\bm{r})$ are the coarse
and fine predicted volume colours for ray $\bm{r}$, respectively. Please refer
to [18] for more details.

###  3.2 View morphing overview

View morphing objective is to synthetise realistic 2D transitions between an
image pair $(\bm{I}_{k},\bm{I}_{k^{\prime}})$. Firstly, the two images are
prewarped through rectification, i.e. their image planes are aligned without
changing their cameras’ optic centres. Secondly, the morph is computed between
these prewarped images to generate an in-between image whose viewpoint lies on
the line connecting the optic centres. Thirdly, the image plane of the in-
between image is transformed to a desired viewpoint through postwarping.

In practice, assuming the two views are prewarped, the morph requires the
knowledge of their camera poses $\bm{P}_{k},\bm{P}_{k^{\prime}}$, and the
pixel correspondence maps between the images, i.e.
$q_{k}:\bm{I}_{k}\Rightarrow\bm{I}_{k^{\prime}},q_{k^{\prime}}:\textbf{I}_{k^{\prime}}\Rightarrow\textbf{I}_{k}$
[22]. Pixel correspondences can be created by a user or by keypoint detectors,
subsequently densified via interpolation to create a map. Note that a
correspondence may be a one-to-many match. A warp function for each image can
be computed from the correspondence map through linear interpolation as

$\begin{split}&w_{k}(\bm{m}_{k},\alpha)=\left(1-\alpha\right)\bm{m}_{k}+\alpha
q_{k}\left(\bm{m}_{k}\right),\\\
&w_{k^{\prime}}(\bm{m}_{k^{\prime}},\alpha)=\left(1-\alpha\right)q_{k^{\prime}}\left(\bm{m}_{k^{\prime}}\right)+\alpha\bm{m}_{k^{\prime}},\\\
&\bm{P}_{\alpha}=(1-\alpha)\bm{P}_{0}+\alpha\bm{P}_{1},\\\ \end{split}$ (3)

where $\bm{m}_{k}$ is a pixel coordinate of image $k$,
$w_{k}(\bm{m}_{k},\alpha)$ is a pixel coordinate of the in-between image, and
$\alpha\in[0,1]$ regulates the position of the in-between image along the
connecting line. The in-between image $\bm{I}_{\alpha}$ can be computed by
averaging the pixel colours of the warped images. Please refer to [23] for
more details.

##  4 Our approach: Depth-based View Morphing for NeRF

Figure 2:  Block diagram of Depth-based View Morphing for NeRF main steps.
From the left, we (1) predict the depth with NeRF, (2) rectify the input
images and predicted depths, and (3) compute the image morphing of a view
randomly positioned between the view pair. $\alpha$ determines the new view
position and it is sampled from a Gaussian distribution.

The goal of Depth-based View Morphing (DVM) is to synthesise additional
training input views $\mathcal{G}=\\{\textbf{G}_{k}\\}_{k=1}^{W}$ by using the
predicted depth of nearby ground-truth view pairs and by linearly
interpolating their image content. Because the linear interpolation is
computed from corresponding pixels between views, the quality of the predicted
depth is key. Therefore, we let NeRF warm up by training on the known views
for $\lambda$ iterations before synthesising and injecting DVM-generated views
in the training process.

Fig. 2 shows the sequence of three operations of our data augmentation
approach for NeRF after the initial warmup stage. These three steps will be
described in the following sections: (i) how we compute the depth, (ii) how we
adapt the rectification algorithm proposed by Fusiello et al. [6] to handle
NeRF camera configurations, and (iii) how we morph the images given the
relative depths.

###  4.1 Rectification

We use Fusiello et al. [6] algorithm for rectification, as we experimentally
observed it can produce rectified images containing most of the photometric
information from both the unrectified images. Differently, Hartley and
Zisserman’s algorithm [8] produces rectified images whose content is more
biased towards one of the two unrectified images. Rectification algorithms are
typically based on the assumptions that viewpoints are aligned horizontally
and that the reference viewpoint is the left hand side camera (from an
observer positioned behind the cameras). This is not the typical configuration
in NeRF, as viewpoints may have arbitrary configurations. In such cases,
Fusiello et al. algorithm [6] produces rectified views that are inconsistent
with the initial camera poses and in the supplementary material we provide
examples where the method [6] fails at rectification. We upgrade [6] in order
to handle general stereo configurations by first determining the warping
matrices to transform the content of the original images, and then warping the
images and the depth maps on a common plane.

Warping matrices. Here the objective is to rotate $\bm{P}_{k}$ and
$\bm{P}_{k^{\prime}}$ around their optic centres until their image planes
become coplanar. This step allows us to compute the warping matrices
$\bm{W}_{k}$ and $\bm{W}_{k^{\prime}}$ that we can directly apply to the pixel
coordinates of the original images to produce the desired rectification
result. Let
$\tilde{\bm{R}_{k}}=[\bm{v}_{x,k}^{T},\bm{v}_{y,k}^{T},\bm{v}_{z,k}^{T}]$ be
the rotation matrix that we aim at computing to achieve these rotations.
Having image planes coplanar ensures that the baseline belongs to the shared
plane and that epipolar lines are parallel. Since the baseline must be
parallel to the new $x$ axis of both cameras, we can define it as
$\bm{v}_{x,k}=\bm{o}_{k}-\bm{o}_{k^{\prime}}$, where $\bm{o}_{k}$ is the optic
centre of camera $k$. The new $y$ axis can be defined as the cross product
between $\bm{v}_{x,k}$ and the original camera direction $\bm{z}_{k}$, i.e.
$\bm{v}_{y,k}=\bm{v}_{x,k}\times\bm{z}_{k}$. Note that this operation assumes
a specific horizontal stereo configuration, which is the constraint of [6]
that we want to generalise. Therefore, we firstly detect the configuration of
a specific camera pair (i.e. beyond the horizontal assumption), and then
modify the new $x$ and $y$ axes in order to apply [6] correctly, thus
producing coherent rectified images.

We assess the camera configuration through the application of four vectors
defined in the reference frame of camera $k$: $\bm{a}_{1}=(1,1,0)$,
$\bm{a}_{2}=(-1,1,0)$, $\bm{a}_{3}=(1,0,0)$, and $\bm{a}_{4}=(0,1,0)$. We use
these vectors to calculate the relative orientation of each camera (with
respect to camera $k$) and thus to determine whether to modify the camera axes

First, we check whether the two cameras are in a vertical configuration by
computing

$s=\left|\frac{\bm{a}_{1}\cdot\bm{v}_{x,k}}{\lVert\bm{a}_{1}\cdot\bm{v}_{x,k}\rVert}+\frac{\bm{a}_{2}\cdot\bm{v}_{x,k}}{\lVert\bm{a}_{2}\cdot\bm{v}_{x,k}\rVert}\right|.$
(4)

If $s\neq 0$, $\bm{v}_{x,k}$ and $\bm{v}_{y,k}$ are swapped. Then, we evaluate
the horizontal configuration, making sure the reference camera is on the left
hand side and compute

$\bm{v}_{x,k}=\bm{v}_{x,k}\underbrace{\frac{\bm{a}_{3}\cdot\bm{v}_{x,k}}{\lVert\bm{a}_{3}\cdot\bm{v}_{x,k}\rVert}}_{\in\\{1,-1\\}},\quad\bm{v}_{y,k}=\bm{v}_{y,k}\underbrace{\frac{\bm{a}_{4}\cdot\bm{v}_{y,k}}{\lVert\bm{a}_{4}\cdot\bm{v}_{y,k}\rVert}}_{\in\\{1,-1\\}}.$
(5)

The new $z$ axis is obtained as $\bm{v}_{z,k}=\bm{v}_{x,k}\times\bm{v}_{y,k}$.
We can then compute the warping matrices as

$\bm{W}_{k}=(\tilde{\bm{K}}\cdot\tilde{\bm{R}}_{k})\cdot(\bm{K}_{k}\cdot\bm{R}_{k})^{-1},\quad\bm{W}_{k^{\prime}}=(\tilde{\bm{K}}\cdot\tilde{\bm{R}}_{k})\cdot(\bm{K}_{k^{\prime}}\cdot\bm{R}_{k^{\prime}})^{-1},$
(6)

where $\tilde{\bm{K}}=\frac{1}{2}(\bm{K}_{k}+\bm{K}_{k^{\prime}})$ is the
matrix of intrinsic parameters. The new camera pose of $k$ is equal to
$\tilde{\bm{P}}_{k}=[\tilde{\bm{R}}_{k},\bm{t}_{k}]$, where $\bm{t}_{k}$ is
the translation of the original camera pose $\bm{P}_{k}$ (same for
$k^{\prime}$).

Image and depth warping. We warp both images and depth maps of both viewpoints
$k$ and $k^{\prime}$ using Eq. 6. Let
$\tilde{\bm{I}}_{k}=\bm{I}_{k}\circ\bm{W}_{k}$ be the warped image, where
$\circ$ is the operator that warps the pixel coordinates of $\bm{I}_{k}$ into
the rectified image. If we warp the depth maps in the same way as the images,
we would obtain distorted rectified depth maps, therefore we first convert
each depth value into its 3D coordinate (in the global reference frame)
counterpart by using the camera parameters, resulting in a map of 3D
coordinates of the same size as the original depth map. Let $\bm{D}_{k}$ be
the depth map predicted by NeRF of view $k$ and
$\bm{D}_{k}^{\scriptscriptstyle 3D}$ be its 3D coordinate map counterpart.
Then, we apply the warping as $\tilde{\bm{D}}_{k}^{\scriptscriptstyle
3D}=\bm{D}_{k}^{\scriptscriptstyle 3D}\circ\bm{W}_{k}$ and we transform
$\tilde{\bm{D}}_{k}^{\scriptscriptstyle 3D}$ to depth map $\tilde{\bm{D}}_{k}$
by using the camera parameters.

###  4.2 Image morphing

Image morphing fuses the rectified images to obtain the new image for data
augmentation during training. We begin by transforming images and depth maps
into coloured 3D point clouds. Specifically, we combine $\tilde{\bm{I}}_{k}$
and $\tilde{\bm{D}}_{k}^{\scriptscriptstyle 3D}$ to produce the relative
coloured point cloud $\bm{X}_{k}$ (same for $k^{\prime}$). As in Eq. 3, we
define the new camera pose as
$\tilde{\bm{P}}_{\alpha}=(1-\alpha)\tilde{\bm{P}_{k}}+\alpha\tilde{\bm{P}_{k^{\prime}}}$,
where $\alpha\in[0,1]$ regulates the position of the new viewpoint between the
two original views. We use $\tilde{\bm{P}}_{\alpha}$ to transform $\bm{X}_{k}$
and $\bm{X}_{k^{\prime}}$ in the local reference system of the new viewpoint.
Then, $\bm{X}_{k}$ and $\bm{X}_{k^{\prime}}$ are projected onto the new image
plane and if some 3D points are projected on the same pixels, they are fused
via coalescence by selecting the nearest one to the camera’s optic centre [2].
We use coalescence for merging pixels because it can be implemented on GPU
[13].

###  4.3 Applying DVM for data augmentation

DVM is subject to the same geometric constraints as the original view morphing
technique [23]. These constraints include singular camera configurations,
which occur when the optic centre of a camera is within the field of view of
another one [23]. We also discard cameras that are distant from each other
more than a threshold $\gamma$, as the in-between cameras may be on a
transition path that cross regions where the object of interest is not
visible.

Because view morphing allows the synthesis of a new view at any point on the
line that connects the known camera pair, we randomly sample new views using a
Gaussian distribution centred half-way through the camera pair. Specifically,
if we consider a normalised distance between the two cameras, the Gaussian
distribution is centred at 0.5 and the standard deviation $\sigma$ is chosen
such that $3\sigma\rightarrow 0$ at the optic centre positions. Therefore, we
sample $\alpha\sim\mathcal{N}(0.5,\sigma)$. For each valid camera pair, we
regenerate $M$ new views every $\eta$ training iterations as the predicted
depth improves over time during training.

##  5 Experiments

###  5.1 Experimental setup

We evaluate our method on three training setups by using the NeRF realistic
synthetic 360∘ dataset [18], which is composed of eight scenes, i.e. Chair,
Drums, Ficus, Lego, Materials, Ship, Mic, Hot, and Dog. First setup: We select
$N=8$ views out of 100 available for each scene using the Farthest Point
Sampling (FPS) [19] (the first view is used for FPS initialisation in each
scene). Second setup: we use the same $N=8$ views used in DietNeRF [11]. Third
setup: we select $N=4$ views using the previous FPS approach. We test each
trained model on all the test views of NeRF realistic synthetic 360∘. We
quantify the rendering results using the peak signal-to-noise ratio (PSNR)
score, structured similarity index measure (SSIM) [26] and learned perceptual
image patch similarity (LPIPS) [29].

We quantitatively compare our approach against DietNeRF [11] as being a recent
method for few-shot view synthesis, and against AugNeRF [4] because it is the
only data augmentation for NeRF method available, with the difference that it
perturbs input while we do new views generation. We also include a qualitative
comparison against these methods at the end.

We implement two oracle methods in order to report upper-bound results of DVM.
The first oracle method is called NeRF [18] \+ oracle image where we replace
the image synthesised with DVM by a ground-truth image rendered from the known
3D object, while the second method is NeRF [18] \+ DVM (oracle depth) where we
synthesise with DVM the new image using the ground-truth depth as opposed to
the predicted one.

We use the Chair scene for our ablation study, which consist of testing DVM on
four different, randomly-chosen, configurations of eight views and on the
DietNeRF configuration.

We implement NeRF and our approach in PyTorch Lightning, and run experiments
on a single Nvidia A40 with a batch size of 1024 rays. A single scene can be
trained in about two days. We use the original implementations of DietNeRF and
AugNeRF. We set the same training parameters as in [18], and set $\gamma=6$,
$\sigma=0.2$, $M=1$, $\eta=5$K, $\lambda=500$.

###  5.2 Analysis of the results

Tab. 1 shows the results averaged over the eight scenes. Per-scenes results
can be found in the supplementary material. Our NeRF implementation can
achieve nearly the same results reported in [18] on the 100-view setup, i.e.
PSNR equal to 31.21 (ours) compared to 31.01 ([18]). We can observe that when
we apply DVM on the eight- and four-view setups, the quality of the NeRF
renderings improves. The largest contribution is in the four-view setup where
we obtain an improvement of 5.92 PSNR on average. Here, the maximum
improvement occurs in the Lego scene where we obtain a +10.48 PSNR. However,
like DietNeRF, we do not improve in the Ship scene, i.e. we achieve -0.70 PSNR
as opposed to -0.91 PSNR of DietNeRF. The results also show that the
perturbation of the known input views, done by AugNeRF, has negative effects
in all the tested setups.

Table 1: Results on the NeRF realistic synthetic 360∘ dataset.

# views | Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---
100 [18] | NeRF [18] | 31.21 | 0.9513 | 0.0465
8 | NeRF [18] \+ oracle image | 26.14 | 0.9038 | 0.0893
NeRF [18] \+ DVM (oracle depth) | 24.22 | 0.8800 | 0.1076
8 | NeRF [18] | 23.45 | 0.8673 | 0.1303
DietNeRF [11] | 22.98 | 0.8545 | 0.1258
AugNeRF [4] | 10.04 | 0.5415 | 0.3866
NeRF [18] \+ DVM | 24.39 | 0.8768 | 0.1146
8 [11] | NeRF [11] | 20.09 | 0.822 | 0.179
DietNeRF [11] | 23.59 | 0.8740 | 0.0970
NeRF [18] \+ DVM | 24.14 | 0.8729 | 0.1180
4 | NeRF [18] | 10.98 | 0.6550 | 0.3620
DietNeRF [11] | 12.61 | 0.6591 | 0.3302
AugNeRF [4] | 8.14 | 0.3924 | 0.4802
NeRF [18] \+ DVM | 16.90 | 0.7563 | 0.2461

Table 2: Ablation study. Key. sX: view set X. avg. dist.: average distance
between view pairs.

# views | avg. dist. | Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---|---
8 [11] | 5.20 | DietNeRF [11] | 25.59 | 0.9120 | 0.0770
8 [11] | 5.20 | NeRF + DVM | 26.90 | 0.9180 | 0.0797
8 (s1) | 5.18 | NeRF + DVM | 27.87 | 0.9294 | 0.0693
8 (s2) | 4.38 | NeRF + DVM | 27.13 | 0.9178 | 0.0855
8 (s3) | 4.73 | NeRF + DVM | 27.48 | 0.9317 | 0.0676
8 (s4) | 4.69 | NeRF + DVM | 28.39 | 0.9365 | 0.0598

Fig. 3 shows some qualitative results on Chair, Hot Dog, and Lego, where we
can observe that DVM enables NeRF to produce higher definition results than
DietNeRF. We speculate that this lack in output details produced by DietNeRF
may be due to its CLIP-based approach that is introduced to leverage a
semantic consistency loss for regularisation [20]. In fact, the output of CLIP
is a low-dimensional (global) representation vector of the image, which may
hinder the learning of high-definition details. Differently, our approach
interpolates the original photometric information from two views to produce a
new input view, without losing information through the encoding of the low-
dimensional representation vector. Several more results and analyses can be
found in the supplementary material.

Ablation study We assess the stability of DVM by evaluating the rendering
quality when different combinations of views are used to train NeRF. Tab. 2
shows that the performance are fairly stable throughout different view
configurations. We also observed that the algorithm is robust to variations in
the distance between view pairs. As long as a view pair is not singular and
the distance between cameras is adequate to create acceptable 3D projective
transformations of the object, we can successfully synthesise new views with
DVM.

| Ground Truth | NeRF [18] | AugNeRF [4] | DietNeRF [11] | NeRF [18] \+ DVM
---|---|---|---|---|---
Chair |  | 23.52 | 11.94 | 22.74 | 25.80
|  |  |  |
|  |  |  |
Hot Dog |  | 27.59 | 12.45 | 28.76 | 31.29
|  |  |  |
|  |  |  |
Lego |  | 22.12 | 1.38 | 18.61 | 22.20
|  |  |  |
|  |  |  |  |

Figure 3:  Comparisons on test-set views of scenes of NeRF realistic synthetic
360∘. Unlike AugNeRF [4], DVM is an effective data augmentation method that
can be used for few-shot view synthesis problems. Unlike DietNeRF [11], DVM
enables NeRF to learn scenes with a higher definition. We report the PSNR that
we measured for each method and for each rendered image. AugNeRF
unsuccessfully learns Chair and Lego (white and black outputs).

##  6 Conclusions

We presented a novel data augmentation method for NeRF approaches that is
based on the view morphing technique [23]. View morphing requires no prior
knowledge of the 3D shape and it is based on general principles of projective
geometry. We showed how to synthesise random novel 3D projective
transformations of the object between viewpoint pairs using view morphing and
how to use them with NeRF. We evaluated our approach on the conventional
dataset used by NeRF-based approaches and showed that our data augmentation
approach enables NeRF to better learn 3D scenes in different few-shot view
synthesis setups. We would like to highlight that DVM is not meant to be a
replacement for DietNeRF, or other similar methods, but it is rather a data
augmentation method that can be used together with any NeRF-based approach. We
designed our approach to be fully differentiable, so an attractive research
direction is to embed our approach in NeRF and exploit it at the loss level by
aggregating photometric information from multiple cameras.

## References

  * Barron et al. [2021] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _ICCV_ , 2021.
  * Chaitin [1982] Gregory J Chaitin.  Register allocation & spilling via graph coloring.  _ACM Sigplan Notices_ , 17(6):98–101, June 1982\.
  * Chen et al. [2021] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo.  In _CVPR_ , 2021.
  * Chen et al. [2022] Tianlong Chen, Peihao Wang, Zhiwen Fan, and Zhangyang Wang.  Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations.  In _CVPR_ , 2022.
  * Devernay and Peon [2010] Frederic Devernay and Adrian Ramos Peon.  Novel view synthesis for stereoscopic cinema: detecting and removing artifacts.  In _Workshop on 3D Video Processing (ACMMM)_ , 2010.
  * Fusiello et al. [2000] Andrea Fusiello, Emanuele Trucco, and Alessandro Verri.  A compact algorithm for rectification of stereo pairs.  _Machine Vision and Applications_ , 12(1):16–22, Jul 2000.
  * Gallo et al. [2020] Orazio Gallo, Alejandro Troccoli, and Varun Jampani.  Novel view synthesis: From depth-based warping to multi-plane images and beyond.  Conference on Computer Vision and Pattern Recognition, 2020.  URL https://nvlabs.github.io/nvs-tutorial-cvpr2020/.
  * Hartley and Zisserman [2004] R. I. Hartley and A. Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, 2004.
  * Hedman et al. [2021] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec.  Baking neural radiance fields for real-time view synthesis.  In _ICCV_ , 2021.
  * Ichnowski et al. [2022] Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, and Ken Goldberg.  Dex-nerf: Using a neural radiance field to grasp transparent objects.  In _CRL_ , 2022.
  * Jain et al. [2021] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis.  In _ICCV_ , 2021.
  * Kajiya and Herzen [1984] J.T. Kajiya and B.P.V. Herzen.  Ray tracing volume densities.  In _SIGGRAPH_ , 1984.
  * Kanade et al. [2019] Varun Kanade, Frederik Mallmann-Trenn, and Thomas Sauerwald.  On coalescence time in graphs: When is coalescing as fast as meeting?: Extended Abstract.  In _Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms_ , 2019.
  * Lorensen and Cline [1987] W.E. Lorensen and H.E. Cline.  Marching Cubes: A High Resolution 3D Surface Construction Algorithm.  In _SIGGRAPH_ , 1987.
  * Martin-Brualla et al. [2021] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In _CVPR_ , 2021.
  * Max [1995] N. Max.  Optical models for direct volume rendering.  _IEEE Transaction on Visualization and Computer Graphics_ , 1(2):99–108, june 1995.
  * Mildenhall et al. [2019] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG)_ , 38(29):1–14, august 2019.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _ECCV_ , 2020.
  * Qi et al. [2019] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3d object detection in point clouds.  In _ICCV_ , 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Schönberger and Frahm [2016] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-Motion Revisited.  In _CVPR_ , 2016.
  * Seitz and Dyer [1996a] Steven M Seitz and Charles R Dyer.  View morphing.  In _Conference on Computer graphics and interactive techniques_ , 1996a.
  * Seitz and Dyer [1996b] Steven M Seitz and Charles R Dyer.  View morphing.  In _Conference on Computer graphics and interactive techniques_ , 1996b.
  * Shorten and Khoshgoftaar [2019] Connor Shorten and Taghi M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 6(1), July 2019.
  * Tewari and et al. [2022] A. Tewari and et al.  Advances in neural rendering.  _Computer Graphics Forum_ , 41(2):703–735, May 2022.
  * Wang et al. [2004] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, april 2004.
  * Xie et al. [2022] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.  Neural fields in visual computing and beyond.  _Computer Graphics Forum_ , 41(2):641–676, may 2022.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  PixelNeRF: Neural radiance fields from one or few images.  In _CVPR_ , 2021.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.

Data augmentation for NeRF: a geometric consistent solution based on view
morphing
Supplementary material

##  7 Introduction

The supplementary material is organised as follows.

  * •

Sec. 8 shows quantitative results for each scene of the NeRF realistic
synthetic 360∘ dataset;

  * •

Sec. 9 includes additional qualitative results when eight and four views are
used for training;

  * •

Sec. 10 shows two examples of the rectification problem we experienced with
Fusiello et al. [6] algorithm;

  * •

Sec. 11 includes an analysis of the depth learned by NeRF.

  * •

Sec. 12 includes some examples of view morphing results produced with DVM that
are generated with the depth predicted by NeRF.

##  8 Quantitative results per scene

We evaluate our method on three training setups by using the NeRF realistic
synthetic 360∘ dataset [18], which is composed of eight scenes, i.e. Chair,
Drums, Ficus, Lego, Materials, Ship, Mic, Hot, and Dog. First setup: We select
$N=8$ views out of 100 available for each scene using the Farthest Point
Sampling (FPS) (the first view is used for FPS initialisation in each scene).
Second setup: we use the same $N=8$ views used in DietNeRF [11]. Third setup:
we select $N=4$ views using the previous FPS approach. We test each trained
model on all the test views of NeRF realistic synthetic 360∘. We quantify the
rendering results using the peak signal-to-noise ratio (PSNR) score,
structured similarity index measure (SSIM) and learned perceptual image patch
similarity (LPIPS).

The numerical results for each scene are reported in Tab. 3 for PSNR, Tab. 4
for SSIM, and Tab. 5 for LPIPS. In terms of PSNR, we can see that our approach
outperforms the other approaches on almost all the scenes. In terms of SSIM
and LPIPS, we can see that our approach is typically on par with NeRF and
DietNeRF in the case of 8-views setting. This is mainly due to how these two
measures are formulated that are less sensitive to high-frequency variations,
but they rather aim at measuring the perceptual similarity between two images.
In the case of 4-views setting our approach often outperforms the other
approaches as it allows NeRF to correctly learn the geometry of the scene.

Table 3: PSNR metric per-scene results on the NeRF realistic synthetic 360∘
dataset.

# views | Method | Chair | Drums | Ficus | Hot Dog | Lego | Materials | Mic | Ship
---|---|---|---|---|---|---|---|---|---
100 | NeRF [18] | 34.57 | 25.71 | 29.05 | 36.88 | 31.47 | 29.49 | 33.05 | 29.48
8 | NeRF [18] \+ oracle image | 29.91 | 21.65 | 21.98 | 31.42 | 26.72 | 23.35 | 28.95 | 25.14
NeRF [18] \+ DVM (oracle depth) | 28.39 | 21.13 | 21.95 | 27.05 | 23.76 | 21.12 | 27.00 | 23.37
8 | NeRF [18] | 25.11 | 19.93 | 21.72 | 23.19 | 24.65 | 21.95 | 27.81 | 23.27
DietNeRF [11] | 26.01 | 18.79 | 20.89 | 26.01 | 21.55 | 21.28 | 27.32 | 21.99
AugNeRF [4] | 15.05 | 0.75 | 14.86 | 12.50 | 1.04 | 9.94 | 13.40 | 12.81
NeRF [18] \+ DVM | 27.87 | 20.20 | 21.36 | 27.15 | 24.84 | 22.35 | 27.61 | 23.71
4 | NeRF [18] | 14.11 | 10.45 | 14.29 | 10.35 | 9.38 | 8.28 | 13.13 | 7.85
DietNeRF [11] | 19.88 | 9.84 | 17.77 | 9.99 | 8.97 | 16.43 | 11.04 | 6.94
AugNeRF [4] | 13.57 | 11.18 | 14.16 | 1.05 | 9.82 | 1.12 | 11.94 | 2.27
NeRF [18] \+ DVM | 22.49 | 16.71 | 18.68 | 10.84 | 19.86 | 18.72 | 20.79 | 7.15

Table 4: SSIM metric per-scene results on the NeRF realistic synthetic 360∘
dataset.

# views | Method | Chair | Drums | Ficus | Hot Dog | Lego | Materials | Mic | Ship
---|---|---|---|---|---|---|---|---|---
100 | NeRF [18] | 0.9790 | 0.9288 | 0.9627 | 0.9785 | 0.9610 | 0.9557 | 0.9753 | 0.8694
8 | NeRF [18] \+ oracle image | 0.9505 | 0.8748 | 0.8789 | 0.9606 | 0.9129 | 0.8870 | 0.9625 | 0.8031
NeRF [18] \+ DVM (oracle depth) | 0.9418 | 0.8631 | 0.8753 | 0.9249 | 0.8633 | 0.8420 | 0.9498 | 0.7796
8 | NeRF [18] | 0.8829 | 0.8320 | 0.8720 | 0.8894 | 0.8820 | 0.8561 | 0.9540 | 0.7696
DietNeRF [11] | 0.9079 | 0.8020 | 0.8664 | 0.9101 | 0.8033 | 0.8453 | 0.9496 | 0.7515
AugNeRF [4] | 0.7576 | 0.0004 | 0.7760 | 0.7480 | 0.0012 | 0.6653 | 0.7497 | 0.6340
NeRF [18] \+ DVM | 0.9294 | 0.8302 | 0.8650 | 0.9266 | 0.8790 | 0.8576 | 0.9508 | 0.7758
4 | NeRF [18] | 0.8156 | 0.5657 | 0.8207 | 0.6507 | 0.5765 | 0.4765 | 0.8427 | 0.4912
DietNeRF [11] | 0.8130 | 0.5231 | 0.8195 | 0.6416 | 0.5249 | 0.7632 | 0.7187 | 0.7264
AugNeRF [4] | 0.6669 | 0.5330 | 0.7340 | 0.0009 | 0.5459 | 0.0008 | 0.6563 | 0.0011
NeRF [18] \+ DVM | 0.8439 | 0.7327 | 0.8247 | 0.7237 | 0.7737 | 0.7953 | 0.8819 | 0.4744

Table 5: LPIPS metric per-scene results on the NeRF realistic synthetic 360∘
dataset.

# views | Method | Chair | Drums | Ficus | Hot Dog | Lego | Materials | Mic | Ship
---|---|---|---|---|---|---|---|---|---
100 | NeRF [18] | 0.0225 | 0.0621 | 0.0339 | 0.0286 | 0.0347 | 0.0356 | 0.0254 | 0.1295
8 | NeRF [18] \+ oracle image | 0.0512 | 0.1067 | 0.1128 | 0.0498 | 0.0690 | 0.0830 | 0.0362 | 0.2060
NeRF [18] \+ DVM (oracle depth) | 0.0569 | 0.1139 | 0.1242 | 0.0862 | 0.1131 | 0.1151 | 0.0520 | 0.1994
8 | NeRF [18] | 0.1286 | 0.1623 | 0.1225 | 0.1380 | 0.0962 | 0.1197 | 0.0505 | 0.2247
DietNeRF [11] | 0.0857 | 0.1696 | 0.1029 | 0.1039 | 0.1638 | 0.1006 | 0.0473 | 0.2323
AugNeRF [4] | 0.2871 | 0.6423 | 0.2416 | 0.3042 | 0.6556 | 0.3360 | 0.3064 | 0.3195
NeRF [18] \+ DVM | 0.0693 | 0.1672 | 0.1243 | 0.0795 | 0.0985 | 0.1144 | 0.0657 | 0.1980
4 | NeRF [18] | 0.2267 | 0.4545 | 0.1822 | 0.4125 | 0.4285 | 0.4977 | 0.1967 | 0.4971
DietNeRF [11] | 0.1676 | 0.4652 | 0.1477 | 0.4094 | 0.4481 | 0.1879 | 0.3150 | 0.5009
AugNeRF [4] | 0.3820 | 0.4684 | 0.3016 | 0.6024 | 0.4561 | 0.6102 | 0.3875 | 0.6336
NeRF [18] \+ DVM | 0.1628 | 0.2657 | 0.1575 | 0.3527 | 0.1936 | 0.1771 | 0.1696 | 0.4899

##  9 Additional qualitative results

Figs. 4 and 5 show results of the other scenes that are not shown in the main
paper. In these examples we can see how the proposed method enables NeRF to
improve the quality of the learned scenes. In particular, in Fig. 5 we can see
how DVM allows NeRF to learn the correct geometry of the scenes Mic, Lego, and
Drums, as opposed to DietNeRF where it instead fails. Ship and Hot Dog are
challenging scenes that all the methods fail to learn correctly.

| Ground Truth | NeRF [18] | AugNeRF [4] | DietNeRF [11] | NeRF [18] \+ DVM
---|---|---|---|---|---
Drums |  | 22.15 | 0.89 | 21.33 | 22.34
|  |  |  |
|  |  |  |
Ficus |  | 20.14 | 14.33 | 19.41 | 19.74
|  |  |  |
|  |  |  |
Materials |  | 19.28 | 8.84 | 19.75 | 20.02
|  |  |  |
|  |  |  |
Mic |  | 27.13 | 11.19 | 25.10 | 27.27
|  |  |  |
|  |  |  |
Ship |  | 24.09 | 16.82 | 22.00 | 24.51
|  |  |  |
|  |  |  |

Figure 4:  Comparisons on test-set views of scenes of NeRF realistic synthetic
360∘, trained on 8-views. Unlike AugNeRF [4], DVM is an effective data
augmentation method that can be used for few-shot view synthesis problems.
Unlike DietNeRF [11], DVM enables NeRF to learn scenes with a higher
definition. We report the PSNR that we measured for each method and for each
rendered image.

| Ground Truth | NeRF [18] | AugNeRF [4] | DietNeRF [11] | NeRF + DVM
---|---|---|---|---|---
|  | 11.99 | 11.92 | 19.11 | 21.41
Chair |  |  |  |  |
|  | 11.99 | 11.92 | 10.87 | 20.06
Mic |  |  |  |  |
|  | 7.82 | 7.62 | 6.82 | 18.18
Lego |  |  |  |  |
|  | 10.84 | 10.63 | 10.5 | 17.11
Drums |  |  |  |  |
|  | 13.92 | 13.22 | 17.17 | 17.04
Ficus |  |  |  |  |
|  | 8.28 | 1.35 | 14.95 | 15.4
Materials |  |  |  |  |
|  | 8.21 | 2.13 | 5.88 | 6.17
Ship |  |  |  |  |
|  | 8.76 | 1.69 | 7.08 | 7.47
Hot Dog |  |  |  |  |

Figure 5: Comparisons on test-set views of scenes of NeRF realistic synthetic
360∘, trained on 4-views. Unlike other techniques DVM help to increase the
probability of building a consistent implict representation with very sparse
view. We report the PSNR that we measured for each method and for each
rendered image.

##  10 Rectification problem

Rectification algorithms are typically based on the assumptions that
viewpoints are aligned horizontally and that the reference viewpoint is the
left hand side camera (from an observer positioned behind the cameras). This
is not the case in NeRF as viewpoints may have arbitrary configurations. For
this reason, Fusiello et al. [6] algorithm produces rectified views that are
inconsistent with the initial camera poses. We upgrade the algorithm [6] in
order to handle general stereo configurations.

Fig. 6 shows the application of [6] to two stereo configurations where the
reference viewpoint is (i) the right-hand side camera (horizontal
configuration), and (ii) the top camera (vertical configuration). Here, we can
observe that when the algorithm in [6] is applied as is the 2D transitions are
incorrectly flipped. Instead, if we use our algorithm, the 2D transitions are
consistent with starting camera poses.

| Input images | [6] | Ours
---|---|---|---
| Horizontal
---
configuration
|  |  |  |  |
| Vertical
---
configuration
|  |  |  |  |

Figure 6: Cases where the algorithm in [6] fails to produce rectification
results that are consistent with the original camera configuration.

##  11 Analysis of the learnt depth

Fig. 7 shows qualitative results about the quality of the learned geometry on
the Chair and Lego scenes. We produced the meshed 3D reconstructions by using
the marching cubes algorithm [14]. We also include the predicted depth next to
each result. As we can observe from the results, the addition of the data
augmentation produced by DVM can significantly improve the estimated geometry
of the scene.

| Chair | Hot Dog | Lego
---|---|---|---
NeRF [18] |  |  |  |  |  |
NeRF + DVM |  |  |  |  |  |

Figure 7: Qualitative comparison between the geometry predicted by NeRF [18]
and by NeRF plus our data augmentation DVM. NeRF + DVM leads to smoother and
more accurate 3D models.

##  12 Examples of images produced with DVM

Fig. 8 shows examples of view morphing results produced with DVM that are
generated with the depth predicted by NeRF. We generated new views by using
the following values of $\alpha=0.08,0.23,0.58,$ and $0.82$. The images
include partial views of the object of interest because we feed NeRF with the
rectified views directly. These are novel views that help NeRF to better
generalise in the case of few-shot view synthesis.

Input Images | $\alpha=0.08$ | $\alpha=0.23$ | $\alpha=0.58$ | $\alpha=0.82$
---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

Figure 8: Examples of images generated by DVM using the depth predicted by
NeRF.


# Deep Koopman Operator with Control for Nonlinear Systems

Haojie Shi, Graduate Student Member, IEEE, Max Q.-H. Meng†, Fellow, IEEE
Manuscript received: February 13, 2022; Revised May 14, 2022; Accepted May 29,
2022.This paper was recommended for publication by Editor Dana Kulic upon
evaluation of the Associate Editor and Reviewers’ comments. This work was
partially supported by National Key R&D program of China with Grant No.
2019YFB1312400, Hong Kong RGC CRF grant C4063-18G, Hong Kong RGC GRF grant No.
14211420 and Hong Kong RGC TRS grant T42-409/18-R awarded to Max Q.-H. Meng.
(Corresponding author: Max Q.-H. Meng)Haojie Shi is from the Chinese
University of Hong Kong, Hong Kong<EMAIL_ADDRESS>Q.-H. Meng is
with Shenzhen Key Laboratory of Robotics Perception and Intelligence, and the
Department of Electronic and Electrical Engineering, Southern University of
Science and Technology, Shenzhen 518055, China, on leave from the Department
of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, and
also with the Shenzhen Research Institute of The Chinese University of Hong
Kong, Shenzhen 518057, China<EMAIL_ADDRESS>Object Identifier
(DOI): see top of this page.

###### Abstract

Recently Koopman operator has become a promising data-driven tool to
facilitate real-time control for unknown nonlinear systems. It maps nonlinear
systems into equivalent linear systems in embedding space, ready for real-time
linear control methods. However, designing an appropriate Koopman embedding
function remains a challenging task. Furthermore, most Koopman-based
algorithms only consider nonlinear systems with linear control input,
resulting in lousy prediction and control performance when the system is fully
nonlinear with the control input. In this work, we propose an end-to-end deep
learning framework to learn the Koopman embedding function and Koopman
Operator together to alleviate such difficulties. We first parameterize the
embedding function and Koopman Operator with the neural network and train them
end-to-end with the K-steps loss function.  Then, an auxiliary control network
is augmented to encode the nonlinear state-dependent control term to model the
nonlinearity in the control input. This encoded term is considered the new
control variable instead to ensure linearity of the modeled system in the
embedding system. We next deploy Linear Quadratic Regulator (LQR) on the
linear embedding space to derive the optimal control policy and decode the
actual control input from the control net. Experimental results demonstrate
that our approach outperforms other existing methods, reducing the prediction
error by order of magnitude and achieving superior control performance in
several nonlinear dynamic systems like damping pendulum, CartPole, and  the
seven DOF robotic manipulator.

###### Index Terms:

Deep Learning Methods, Model Learning for Control, Machine Learning for Robot
Control.

##  I INTRODUCTION

The prediction and control of nonlinear systems remain a challenging task.
Classical methods first require full knowledge of the system model to predict
the evolution of dynamical systems and then design state feedback control
laws[24] or optimization-based control policies[18, 1]. However, the hidden
dynamics of chaotic nonlinear systems are usually unknown or too complicated
to be modeled, prohibiting such controllers from taking effect.

For the prediction of nonlinear systems, great research interest has been put
into data-driven methods like SINDy [8]. The deep neural network also plays an
important role in nonlinear system modeling with its strong expressive
power[15]. Both of these methods discover hidden dynamics in a nonlinear form,
demanding nonlinear control methods such as iLQR[18] and NMPC[1].
Nevertheless, these optimization-based methods require tremendous
computational time with the increasing dimensions of the system state, which
is prohibitive for real-time control of high dimensional nonlinear systems.
Besides, model-based RL approaches illustrate great success in controlling
nonlinear systems. They first train the neural network to approximate the
dynamics and then apply policy search methods for trajectory optimization via
backpropagation[31, 28]. Model-free RL approaches also achieve strong
performance on continuous control of nonlinear systems[27, 27, 12]. However,
these methods suffer from sample efficiency issues and lack of generalization.

Figure 1:  Prediction Power vs. Linearity. The rank of prediction error is
computed by sorting the prediction error tested in six environments in
ascending order, and lower rank represents smaller prediction error. * means
our methods. KD, KRBF, KDNN and KRNN represents Koopman Derivative[22],
Koopman RBF[14], Deep neural network and Deep recurrent network methods.

Recently, the Koopman operator has shown great potential to alleviate such
difficulties for nonlinear control under unknown dynamics. As a data-driven
approach, the Koopman operator can automatically learn the system model with
the least square method. Furthermore, the Koopman operator can map nonlinear
dynamics to linear systems in the embedding space. And the embedded linear
system is ready to be controlled via linear methods like LQR[4, 22] and MPC[9,
14]. Still, selecting the embedding function to maintain prediction quality
remains a tough task. Recent approaches focus on learning the embedding
functions with deep neural networks [20, 2] and then apply linear control
methods[19, 13]. However, after lifting the state to embedded space, the
linear-quadratic form of the cost function can be distorted in the hidden
space, increasing the subsequent control difficulty. Moreover, previous
Koopman-based approaches only model nonlinear systems with linear control
input, destroying the prediction quality for general nonlinear systems.

In this work, we propose an end-to-end deep learning framework to learn the
Koopman embedding function and Koopman Operator together to alleviate such
difficulties. Our proposed embedding function concatenates the original state
and the neural network encodings so that the original state can be preserved
for further control. Hence, the cost function in the embedding space is
consistent with that in the original space. Furthermore, we propose an
auxiliary control network that both encodes the state and control to model the
nonlinear state-dependent term in control. Then the encoded term is considered
the new control variable for linear control. After deriving the optimal
control policy, the actual control input can be recovered from the encoded
term. In this way, we maintain both the prediction power and the linear
dynamics of the modeled system, suitable for linear control methods.

To further evaluate our results, we conduct experiments on several nonlinear
systems. For prediction, experimental results demonstrate that our approach
outperforms other Koopman-based approaches and achieves comparable or even
better performance than the deep recurrent neural network. Consequently,
better prediction performance ensures our approach to achieve better control
results in nonlinear dynamic environments like damping pendulum, CartPole, and
the seven DOF robotic manipulator. 111Source Code:
https://github.com/HaojieSHI98/DeepKoopmanWithControl

In summary, the main contributions of these papers are:

  1. 1.

We propose an end-to-end deep learning framework that trains the embedding
function and Koopman Operator together, maintaining both the prediction
quality and the consistency of cost function in embedded space.

  2. 2.

We design an auxiliary control network that models the state-dependent
nonlinearity in control input, improving the prediction performance while
retaining the linear dynamics of the modeled system so that it is feasible for
linear control methods.

  3. 3.

We evaluate our approach in several nonlinear dynamic systems and achieve
better prediction and control performance than existing Koopman-based methods
and deep learning-based approaches.

##  II RELATED WORK

Prediction of nonlinear systems: For unknown dynamics, data-driven methods
have been considered a leading tool for learning the hidden nonlinear dynamics
model. Dynamic model decomposition algorithms (DMD,EDMD) [25, 30, 33] seeks an
optimal linear operator matrix to fit the evolution of nonlinear systems with
spectral decomposition. Sparse identification of nonlinear dynamics algorithms
[8, 7] aim to discover the governing equations from data of nonlinear dynamics
via sparse regression of the dynamics matrix. Deep neural networks (DNNs) also
play an important role in directly learning the hidden nonlinear dynamics from
data[15], especially the deep recurrent neural networks that take advantage of
the temporal information of data[11]. And latent state space methods that
learn state representation with deep generative models succeed in modeling and
controlling nonlinear systems from raw pixels.[32, 16] Koopman-based
approaches first select the embedding function to lift the original state into
embedding space and then learn the linear operator with the least square
method to fit the linear dynamics in the lifted space. To choose the embedding
functions, standard methods utilize the radial basis function (RBF) networks
as the representation function[14]. The selection of RBF centers significantly
influences the approximation performance, and it requires tremendous manual
effort.[34]. Derivative-based Koopman[22] uses higher-order derivatives of
nonlinear dynamics as lift function, which achieves promising prediction
performance, but it requires the knowledge of the derivatives of unknown
dynamics. In contrast, deep learning approaches promote the learning of
embedding functions with deep neural networks[14, 2], improving the prediction
quality for a long time horizon.

Direct Nonlinear Control: Nonlinear control methods can be sorted as direct
control approaches and linearization approaches. As for direct control
approaches, they directly control the nonlinear model with backstepping
feedback control laws[10] or optimal control such as iLQR[18] and NMPC[1].
Backstepping methods require the analytical form of dynamic equations, which
is inaccessible for complex nonlinear systems. Moreover, direct optimal
control methods demand tremendous computational time for high-dimensional
systems, which is computationally prohibitive for real-time control. Besides,
RL-based methods train the neural network to approximate the optimal policy
via policy gradient or value iteration[12, 26, 17]. They can compute optimal
policy in real-time with feedforward networks but still suffer from sample
efficiency and generalization issues.

Nonlinear Control Linearization: Local linearization serves as the basic
approach for nonlinear control. It first linearizes the nonlinear function at
the current state and then controls the system with LQR or MPC[24], but the
stability is only guaranteed in a local region where the linearization
approximates the nonlinear dynamics well. Koopman-based approaches[14, 19, 22,
13] highly enlarge the stable area by globally mapping the nonlinear dynamics
to linear dynamics in embedding space. After lifting the state space to
embedding space, liner control methods like LQR and MPC are ready to be
deployed for real-time control.

##  III BACKGROUND

###  III-A Basics of Koopman Operator

Given the nonlinear systems with discrete dynamics:

$s_{k+1}=F(s_{k})$ (1)

Koopman Operator $\mathcal{K}$ is an infinite linear operator that evolves the
embedding functions $g$ of the state:

$\mathcal{K}g(s)=g\circ F(s)$ (2)

Considering the control input with the assumption that $u_{k+1}=u_{k}$, the
evolution flow follows as:

$g(x_{k+1},u_{k+1})=\mathcal{K}g(x_{k},u_{k})$ (3)

where $x\in\mathbb{R}^{n},u\in\mathbb{R}^{m},$ and
$g:\mathbb{R}^{n+m}\rightarrow\mathbb{R}^{d}$. Therefore, with the embedding
function $g$ lifting the state space to embedding space, Koopman Operator
$\mathcal{K}$ maps the nonlinear dynamics to linear dynamics. Readers can
refer to [6] for more details on Koopman Operator theory.

Previous methods simplify (3) by linearizing the control term. They first
seperate the embedding function into two parts that
$g(x,u)=[g_{x}(x,u);g_{u}(x,u)]$, and assume that
$g_{x}(x,u)=g_{x}(x),g_{u}(x,u)=u$. Then the Eq.(3) is simplified as:

$\left[\begin{array}[]{c}g_{x}(x_{k+1})\\\ u_{k+1}\\\
\end{array}\right]=\left[\begin{array}[]{cc}K_{xx}&K_{xu}\\\ K_{ux}&K_{uu}\\\
\end{array}\right]\left[\begin{array}[]{c}g_{x}(x_{k})\\\ u_{k}\\\
\end{array}\right]$ (4)

which can further be written as:

$g_{x}(x_{k+1})=K_{xx}g_{x}(x_{k})+K_{xu}u_{k}$ (5)

However, such simplification inevitably damages the prediction performance of
the nonlinear system and can further affect the control results.

###  III-B Learning Koopman Operator

Data-driven methods are deployed to learn the Koopman Operator $\mathcal{K}$.
Given the dataset $g_{k}=g(x_{k},u_{k}),k=[1,2,...,N]$, the Koopman Operator
$K_{d}$ is computed by optimizing the mean square loss:

$K_{d}=\operatorname*{arg\,min}_{K_{d}}{\Sigma_{i=1}^{N-1}||g_{i+1}-K_{d}g_{i}||}$
(6)

The least-square problem has a closed-form solution:

$K_{d}^{*}=PG^{\dagger}$ (7)

where $P=\frac{1}{N-1}\Sigma_{i=1}^{N-1}g_{i+1}g_{i}^{T}$,
$G=\frac{1}{N-1}\Sigma_{i=1}^{N-1}g_{i}g_{i}^{T}$, and ${\dagger}$ is the
Moore–Penrose pseudoinverse.

Figure 2: The overview of the neural network framework for K-steps
predictions.

##  IV METHODOLOGY

###  IV-A Framework

We first seperate the embedding function into two parts:
$g(x,u)=[g_{x}(x,u);g_{u}(x,u))]$. Without loss of generality, we assume that
the first term is only related to the system state, so we get
$g_{x}(x,u)=g_{x}(x)$. Based on Eq.(3), it is derived that:

$g_{x}(x_{k+1})=K_{xx}g_{x}(x_{k})+K_{xu}g_{u}(x_{k},u_{k})$ (8)

For the first part of embedding function, we use a deep neural network
$\theta$ to parameterize $g_{x}(x,k)$. Instead of directly encoding the state
like[20], which requires an extra decode network, we concatenate the origin
state and network encoding together:

$z_{k}=g_{x}(x_{k})=\left[\begin{array}[]{c}x_{k}\\\ g_{\theta}(x_{k})\\\
\end{array}\right]$ (9)

where $z_{k}$ is the embedded state,
$g_{\theta}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{d}$ is the parameterized
neural network. The advantage of this design is that the original state can be
easily recovered by:

$x_{k}=Cz_{k}$ (10)

where the matrix $C\in\mathbb{R}^{n\times(n+d)}$ holds the simple form as:

$C=\left[\begin{array}[]{cc}I_{n}&0\\\ \end{array}\right]$ (11)

In this way, we can retain the form of the cost function for further control.

For the second part of embedding function, we have three versions to represent
$g_{u}(x_{k},u_{k})$. The first version is called Deep KoopmanU with Control
(DKUC) algorithm. It just simplifies the function as
$g_{u}(x_{k},u_{k})=u_{k}$, then the evolution equation degrades to Eq.(5).

The sceond version is called Deep Koopman Affine with control (DKAC)
algorithm, it considers the evolution function as control affine form, so that
$g_{u}(x_{k},u_{k})=g_{u}(x_{k})u$, and the Eq.(3) is modified as:

$z_{k+1}=K_{xx}z_{k}+K_{xu}g_{u}(x_{k})u$ (12)

And we parameterize the function $g_{u}(x_{k})$ by designing an auxilliary
control network $\phi$, so that $g_{u}(x_{k})=g_{\phi}(x_{k})$.

For the final version, we use the control network $\phi$ to approximate the
function $g_{u}(x_{k},u_{k})$ directly as
$g_{u}(x_{k},u_{k})=g_{\phi}(x_{k},u_{k})$, and the evolution function remains
the same form as Eq.(8). It is called Deep Koopman Nonlinear (DKN) algorithm.

###  IV-B Feedforward Prediction

Given the current state $x_{t}$, we can easily predict future K-steps states
by the feedforward network. Besides the embedding function, we also
parameterize the Koopman Operator matrix $K_{x}x$ and $K_{x}u$ by one layer
linear network that $K_{xx}=A,K_{xu}=B$. Then the K-steps prediction follows:

$\displaystyle z_{t+k+1}=Az_{t+k}+Bg_{\phi}(x_{t+k},u_{t+k}),k=0,...,K-1$
$\displaystyle z_{t}=g_{x}(x_{t})=\left[\begin{array}[]{c}x_{t}\\\
g_{\theta}(x_{t})\\\ \end{array}\right]$ (15) $\displaystyle x_{t+k}=Cz_{t+k}$
(16)

where $g_{\phi}(x,u)=u$ for KPUC algorithm, $g_{\phi}(x,u)=g_{\phi}(x)u$ for
KPAC algorithm, and $g_{\phi}(x,u)=g_{\phi}(x,u)$ for KPN algorithm.

###  IV-C K-steps Loss Function

Standard Koopman-based algorithm first selects the embedding function and then
learns the Koopman Operator matrix with linear regression for only step
prediction. In this work, we learn the embedding function and Koopman Operator
end-to-end instead, and we design the K-steps prediction loss function for
long-term prediction. Given the dataset $\left[X_{i}\in\mathbb{R}^{N\times
n},U_{i}\in\mathbb{R}^{N\times m},i=0,1,2,...,K\right]$, we can compute the
real embedded state $Z_{i}=g_{x}(X)$, and predict the K-steps sates
$[\hat{Z}_{i},i=1,2,...,K]$ in embedding space from the initial state $X_{0}$
via Eq.(16). The loss function is computed as:

$L(\theta,\phi)=\Sigma_{i=1}^{K}\gamma^{i-1}MSE(Z_{i},\hat{Z}_{i})$ (17)

where $\gamma$ is the weight decay hyper-parameter, and $MSE$ is the mean
square loss function. Instead of only considering the following one-step
prediction error, the K-steps loss focuses on the weighted sum of K-steps
prediction error, conducive to prediction in a long time horizon.

Figure 3: The Framework of LQR Control with Deep Koopman Operator.

###  IV-D LQR Control

Without loss of generality, we consider quadratic loss function and optimal
control with infinite horizons and ignore constraints in this work. For
general nonlinear systems, given the desired state $x_{t}^{des}$, the standard
optimal control problem takes the form that:

$\displaystyle\min_{u_{t=1,...,\infty}}{\Sigma_{t=1}^{\infty}{(x_{t}-x_{t}^{des})}^{T}Q(x_{t}-x_{t}^{des})+u_{t}^{T}Ru_{t}}$
$\displaystyle\text{s.t. }x_{t}=F(x_{t-1},u_{t-1})$ (18)

where $Q,R$ is the cost matrix related to state and control input. Since the
evolution function is nonlinear, the optimal control problem is tough to be
solved directly.

On the contrary, the Koopman Operator maps the nonlinear systems to linear
dynamics in embedding space, ensuring the feasibility of linear control
methods. Therefore, we apply optimal control in embedding space and consider
$g_{\phi}(x,u)$ as the new control variable instead, in which case it degrades
to the LQR control problem. Based on Eq.(16),(10), the LQR problem can be
rewritten as:

$\displaystyle\min_{\hat{u}_{t=1,...,\infty}}\quad$
$\displaystyle\Sigma_{t=1}^{\infty}{(z_{t}-z_{t}^{des})}^{T}\hat{Q}(z_{t}-z_{t}^{des})+\hat{u}_{t}^{T}\hat{R}\hat{u}_{t}$
s.t. $\displaystyle\begin{aligned} z_{t+1}&=Az_{t}+B\hat{u}_{t},\quad
z_{0}=g_{x}(x_{0})\\\ \hat{u}_{t}&=g_{\phi}(Cz_{t},u_{t})\end{aligned}$ (19)

where $z_{t}^{des}=g_{\theta}(x_{t}^{des})$, $\hat{Q}=C^{T}QC$, $\hat{R}=R$
for KPUC algorithm, and
$\hat{R}_{t}=g_{\phi}(x_{t})^{-T}Rg_{\phi}(x_{t})^{-1}$ for KPAC algorithm.
Since $R$ is the pre-defined matrix and can be modified for different
algorithms, we just assume $\hat{R}=R$ for simplification.

Eq.(19) has the closed-form solution via the LQR algorithm. After computing
the LQR gain matrix $K_{LQR}$, the optimal control takes the form of:

$\hat{u}_{t}^{*}=K_{LQR}(z_{t}-z_{t}^{des})$ (20)

And we can recover the actual optimal control by:

$u_{t}^{*}=g_{\phi}^{-1}(x_{t},\hat{u}_{t}^{*})$ (21)

where $u_{t}^{*}=\hat{u}_{t}^{*}$ for KPUC algorithm, and
$u_{t}^{*}=g_{\phi}(x_{t})^{-1}\hat{u}_{t}^{*}$ for KPAC algorithm. The
detailed algorithm is illustrated in Algorithm 1.

Algorithm 1 Deep Koopman with Control

0: $g_{\theta}$, embedding network $g_{\phi}$, auxlliary control network
$A,B,C$, Koopman Operator matrix and recover matrix$Q,R$, cost matrix
$x_{0},x_{t}^{des}$, initial state and desired trajectory$T$, control
steps$F$, nonlinear evolution equation

1: initialize LQR gain matrix: $K_{LQR}\leftarrow LQR(A,B,Q,R)$

2: embed the desired state $z_{t}^{des}\leftarrow g_{x}(x_{t}^{des})$

3: reset the $env$ with initial state $x_{0}$

4: for each t in {0…T-1} do

5: embed current state $z_{t}\leftarrow g_{x}(x_{t})$

6: compute optimal control $\hat{u}_{t}^{*}\leftarrow
K_{LQR}(z_{t}-z_{t}^{des})$

7: recover actual optimal control $u_{t}^{*}\leftarrow
g_{\phi}^{-1}(x_{t},\hat{u}_{t}^{*})$

8: simulate the system forward $x_{t+1}\leftarrow F(x_{t},u_{t}^{*})$

9: end for

##  V EXPERIMENTS

In this section, we conduct complete experiments to answer the following
questions:

(1). How is the prediction performance of our approaches compared with
previous Koopman-based approaches and deep learning-based approaches? And how
is the sample efficiency of our proposed methods compared with others?

(2). How is the control performance of our approaches compared with previous
Koopman-based methods with linear control?

(3). Is our proposed control net conducive to nonlinear system prediction and
control? Specifically, do our DPAC and DKN algorithms outperform the DKUC
algorithm in prediction and control?

###  V-A Environments

Figure 4: Prediction Results. The log of mean prediction error in the next
thirty steps tested in six environments.

To evaluate our approach, we conduct experiments on six nonlinear dynamics
simulation envoriments, including (a) DampingPendulum, (b) Pendulum, (c)
MountainCarContious, (d) CartPole, (e) DoublePendulum, and (f) Franka. Among
above six envrionments, b)-d) are environments modified from OpenAI gym[5],
and f) Franka is the same as [23].

Besides, the dynamics of (a) DampingPendulum environment has the following
form:

$\frac{d}{dt}\left[\begin{array}[]{c}\theta\\\ \dot{\theta}\\\
\end{array}\right]=\left[\begin{array}[]{c}\dot{\theta}\\\
-glsin\theta+\frac{1}{ml}b\dot{\theta}+\frac{1}{ml}cos\theta u\\\
\end{array}\right]$ (22)

where $m$,$l$ is the mass and length of the pendulum, $b$ is the damping
ratio, and $u$ is the external force put horizontally (the control input).

The dynamics of (e) holds the following partial derivative equation[21]:

$M(\theta)\ddot{\theta}+c(\theta,\dot{\theta})+g(\theta)=u$ (23)

where $M(\theta)$ is the mass matrix, $c(\theta,\dot{\theta})$ is the
velocity-product term, $g(\theta)$ is the gravity term and
$u\in\mathbb{R}^{2}$ is the input force.

###  V-B K-steps Prediction

We evaluate the prediction performance of our approaches on the above six
environments and compare them with previous methods: Derivative-based Koopman
(KDerivative)[22], RBF-based Koopman (KRBF)[14], Deep neural networks (KDNN)
and Deep recurrent networks (KRNN). For KDerivative and KRBF, we train them
with linear regression as mentioned in Eq.(6). For KDNN and KRNN, we generally
train a neural network to approximate the nonlinear evolution function
$F(x,u)$ with K-steps Loss in Eq.(17). For DKAC and DKN, we choose the output
dimension of their control net the same as the dimension of its original
control input. We use three hidden layers and 128 hidden units, the same
learning rate, and 15-steps prediction loss for all deep learning-based
approaches.

The final 30-steps prediction results is plotted in Fig. 4. We run four times
for every environment and algorithm and plot the mean of log10 of the maximum
error in 30-time horizons. And the errors at the 15th step are shown in Table
I. KDerivative requires the $N$th derivative of the dynamics, which is too
complex or inaccessible for DoublePendulum and Franka, so we fail to conduct
these two experiments for KDerivative. From the above figure and tables, we
can find that: (1). KDerivative and KRBF work better at the first several
steps since the least square methods only optimize the first step prediction.
On the contrary, deep learning-based approaches achieve better performance
over a long time horizon. (2). The DKN algorithm performs best and even better
than KRNN in most cases, although KRNN takes advantage of the extra temporal
information of data. (3). Our algorithms DKUC and DKAC reduce the prediction
error by magnitudes than previous Koopman-based approaches (KDerivative &
KRBF) and achieve comparable or even better results than fully deep learning-
based approaches (KDNN & KRNN). (4). Our algorithm DKAC outperforms DKUC in
prediction, especially on DampingPendulum, where the 15th step prediction
error of DKAC is almost one-third of that of DKUC. It illustrates that our
auxiliary control net works for better prediction.

TABLE I: Prediction Results at the 15th Step

| DampingPendulum | Pendulum | MountainCar | CartPole | DoublePendulum | Franka
---|---|---|---|---|---|---
KDerivative | 6.760e-1±3.284e-2 | 9.634e-1±1.936e-2 | 6.725e-2±3.524e-4 | 8.626e-2±2.031e-3 | - | -
KRBF | 2.700e-2±9.814e-4 | 2.624e+0±9.173e-2 | 1.233e-1±9.268e-3 | 2.854e-2±2.031e-3 | 3.879e+0±7.232e-2 | 8.639e-3±8.657e-5
KDNN | 2.517e-2±4.462e-3 | 9.714e-2±2.020e-2 | 3.509e-2±5.250e-3 | 2.582e-2±2.923e-3 | 2.636e-2±1.486e-3 | 2.888e-2±8.885e-4
KRNN | 2.032e-2±4.695e-3 | 2.990e-1±8.550e-2 | 6.027e-4±2.128e-5 | 5.314e-2±7.360e-3 | 6.392e-2±9.584e-3 | 2.726e-3±1.187e-4
DKUC(ours) | 2.835e-2±5.166e-3 | 5.982e-1±3.138e-2 | 9.714e-4±7.404e-5 | 1.595e-2±1.042e-3 | 1.510e-1±1.005e-2 | 2.586e-3±3.630e-4
DKAC(ours) | 9.511e-3±6.465e-4 | 5.153e-1±1.809e-2 | 9.663e-4±4.724e-5 | 1.075e-2±7.291e-4 | 1.304e-1±2.537e-2 | 2.485e-3±6.110e-4
DKN(ours) | 2.575e-3±8.424e-4 | 5.026e-2±2.677e-3 | 1.341e-3±4.530e-4 | 8.423e-3±8.117e-4 | 7.247e-3±4.099e-3 | 2.995e-3±1.705e-4

###  V-C Sample Efficiency of Learning Algorithms

To evaluate the sample efficiency of our methods, we train them with various
numbers of trajectory samples in the DampingPendulum environment, and the
results are plotted in Fig 5. The trajectory samples vary in
[200,1000,5000,20000,50000], and each trajectory contains 15 time steps of
data. From the figure, we can see that the performance of DKUC and DKAC
algorithms is not sensitive to the number of training samples compared with
KDNN and KRBF methods. Moreover, our proposed methods can achieve good
prediction results after consuming 5000 trajectory samples (25 minutes of
data).

Figure 5: The mean and std of Log10 Error at the 15th step in the
DampingPendulum environment with different numbers of training samples. Figure
6:  Phase portraits of classical control results in six environments. The blue
star mark means the initial state, and the red star mark represents the goal
state. Control results of the KRBF algorithm running out of range are not
plotted. KDerivative does not apply to DoublePendulum without access to the
high order of derivatives.

###  V-D LQR Control

As for control performance, we compare our approaches with KRBF and
KDerivative, which are suitable for liner control methods like LQR, while KDNN
and KRNN methods are fully nonlinear, and linear control methods are
inapplicable to them. Moreover, we also added the iLQR[18] algorithm as a
baseline for nonlinear control methods. Besides, since we have not found an
invertible $g_{\phi}(x,u)$ of the DKN algorithm, we only deploy our proposed
DKUC and DKAC algorithms for control.

Figure 7: The total cost of all initial states in the DampingPendulum
environment. Each point in the plane represents the initial state
$(\theta,\dot{\theta})$, and the color represents the corresponding value.
Figure 8: Franka Control Results. The desired trajectory of the robot end-
effector generated in the y-z plane is plotted as the blue line. Trajectories
of WLS and KRBF are out of range. Top Tracking results without measuring
noise. Down Tracking results with Gaussian noise of $\mathcal{N}(0,0.01*I)$
added to the position of the robot end-effector.

####  V-D1 Classical Control

TABLE II: The Total Costs of Classical Control

| KDerivative | KRBF | DKUC | DKAC | iLQR
---|---|---|---|---|---
DampingPendulum | 1196.692 | 1265.189 | 1183.103 | 1053.759 | 1072.974
Pendulum | 307.728 | NaN | 305.387 | 309.232 | 301.101
MountainCar | 15.933 | NaN | 13.873 | 16.372 | 11.389
CartPole | 90.827 | NaN | 100.781 | 89.878 | 5265.195
DoublePendulum | - | NaN | 2919.976 | 2905.536 | 11774.169

TABLE III: The Tracking Error in Franka Control

| Franka-Eight | Franka-Star | Franka-Eight(noised) | Franka-Eight(noised)
---|---|---|---|---
LS | 2.968±90.051 | 3.395±90.289 | 7.505±0.339 | 93.217±0.771
SOC | 2.588±90.060 | 2.918±90.124 | 4.611±0.192 | 73.006±5.109
WLS | 233.834±914.601 | 262.825±916.344 | 237.889±16.652 | 461.962±44.285
CG | 2.179±90.661 | 2.496±90.118 | 3.137±0.088 | 71.110±5.655
KRBF | 359.744±9276.283 | 8165±922782 | 352.152±156.918 | 3483±7038
DKUC | 1.450±90.074 | 1.775±90.161 | 2.035±0.052 | 2.312±0.151

Environments from (a) to (e) are classical control problems with force as
input. The phase portrait control results are plotted in Fig. 6. Table II
shows the total cost defined in Eq. (18), where NaN means that the control
cost is out-of-range.

From the above plot and table, it can be seen that our approaches, DKUC and
DKAC, succeed in controlling all the systems and achieving good performance.
KRBF is quite unstable since it is highly dependent on the selection of
centers of RBF functions[34]. Furthermore, we can find that DKAC outperforms
DKUC and KDerivative in environments like DampingPendulum, where the control
term is highly nonlinear and related to the state. Only DKAC succeeds in
controlling the damping pendulum to achieve the zero point. iLQR method
achieves the best performance in low dimensional tasks like Pendulum-v1, but
it consumes more computing time and fails in higher dimensional tasks such as
CartPole-v1 and DoublePendulum.

To further illustrate the advantage of DKAC, we conduct control experiments in
the DampingPendulum environment of all the initial conditions
$\left\\{(\theta,\dot{\theta})|\theta\in[-4,4],\dot{\theta}\in[-4,4]\right\\}$,
and we plot the total cost given the corresponding initial states in Fig.7.
From the above plots, we can see that our DKAC algorithm succeeds in directing
the system to the desired state with less cost in most initial conditions,
while other methods fail when the initial state is far away from the desired
state.

####  V-D2 Franka Control

For Franka Environment, our control input is the desired joint velocity at
100Hz, and the actual force is computed via PID at 1000Hz. Our control goal is
to make the robot track the desired trajectory of the robot end-effector
$x_{t}^{des},t=1,2,...,N$ . The control term has seven DOF, and it is hard for
us to design the cost matrix $R$ for DKAC. Since the derivative is not
available for KDerivative, we only compare our DKUC algorithm with KoopmanRBF
and methods in previous work[23], including least square (LS), weighted least
square (WLS), constraint generation (CG), and SOC. The Franka control results
are plotted in Fig.8 and we compare the tracking error
$\Sigma_{t=1}^{N}||x_{t}-x_{t}^{des}||_{2}$ in 10 random initial states in
Table III. Furthermore, we add Gaussian noise $X\in\mathcal{N}(0,0.01*I)$ to
the position of the robot end-effector to evaluate the robustness of the
algorithms. Based on the experimental results, we can see that our algorithm
achieves the lowest tracking error and is robust in measuring noise.
Especially for Franka-Star (noised) task, only our algorithm succeeds in
tracking the desired trajectory.

##  VI LIMITATION AND FUTURE DIRECTION

Although our proposed algorithms outperform other state-of-the-art methods in
prediction and control from the experimental results, this work still has
several limitations. Firstly, though the DKN algorithm achieves the best
prediction performance, it is hard to find a unique inverse mapping function
to recover the real control signals from the output of its auxiliary control
network. To address this difficulty, designing an invertible network to decode
the output of the auxiliary control network would be the key problem. One
possible way could be to apply the autoencoder [29] to decode the output.
Training an invertible neural network [3] directly for the auxiliary control
network would also be a possible solution. Secondly, expanding our approaches
to higher dimensional nonlinear systems is quite promising for future study,
especially for soft robot systems whose dynamics are too complex to be
derived. Last but not least, our approach does not consider the input and
state constraints. Since our embedding state contains the original state, the
state constraints are easy to derive in the embedding space. However, the
input constraints are distorted by the control network of DKAC and DKN
algorithms in the embedding space. Therefore, our future study will also focus
on dealing with the input constraints in the embedding space.

##  VII CONCLUSION

In this work, we propose an end-to-end deep learning framework to learn the
Koopman embedding function and Koopman Operator together, improving the
prediction power of nonlinear systems and the control performance with LQR.
Furthermore, we design a control network to model the nonlinear state-
dependent term related to control, enhancing the prediction performance by
strengthening the expressive power and gaining even better prediction quality
than the recurrent neural network. Experimental results prove that this
approach is also conducive to the control of general systems with high
nonlinearity on control terms. We hope that our work could facilitate the
further study of real-time control on complex nonlinear systems.

## References

  * [1] Frank Allgöwer and Alex Zheng.  Nonlinear model predictive control, volume 26.  Birkhäuser, 2012.
  * [2] Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney.  Forecasting sequential data using consistent koopman autoencoders.  In International Conference on Machine Learning, pages 475–485. PMLR, 2020.
  * [3] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen.  Invertible residual networks.  In International Conference on Machine Learning, pages 573–582. PMLR, 2019.
  * [4] Alberto Bemporad, Manfred Morari, Vivek Dua, and Efstratios N Pistikopoulos.  The explicit linear quadratic regulator for constrained systems.  Automatica, 38(1):3–20, 2002.
  * [5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  arXiv preprint arXiv:1606.01540, 2016.
  * [6] Steven L Brunton, Marko Budišić, Eurika Kaiser, and J Nathan Kutz.  Modern koopman theory for dynamical systems.  arXiv preprint arXiv:2102.12086, 2021.
  * [7] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.
  * [8] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.  Sparse identification of nonlinear dynamics with control (sindyc).  IFAC-PapersOnLine, 49(18):710–715, 2016.
  * [9] Eduardo F Camacho and Carlos Bordons Alba.  Model predictive control.  Springer science & business media, 2013.
  * [10] Thor I Fossen and Aslaug Grovlen.  Nonlinear output feedback control of dynamically positioned ships using vectorial observer backstepping.  IEEE transactions on control systems technology, 6(1):121–128, 1998\.
  * [11] Ramazan Gencay and Tung Liu.  Nonlinear modelling and prediction with feedforward and recurrent networks.  Physica D: Nonlinear Phenomena, 108(1-2):119–134, 1997.
  * [12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In International conference on machine learning, pages 1861–1870. PMLR, 2018.
  * [13] Yiqiang Han, Wenjian Hao, and Umesh Vaidya.  Deep learning of koopman representation for control.  In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1890–1895. IEEE, 2020.
  * [14] Milan Korda and Igor Mezić.  Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control.  Automatica, 93:149–160, 2018.
  * [15] Rajesh Kumar, Smriti Srivastava, JRP Gupta, and Amit Mohindru.  Comparative study of neural networks for dynamic nonlinear systems identification.  Soft Computing, 23(1):101–114, 2019.
  * [16] Sascha Lange and Martin Riedmiller.  Deep auto-encoder neural networks in reinforcement learning.  In The 2010 international joint conference on neural networks (IJCNN), pages 1–8. IEEE, 2010.
  * [17] Sergey Levine and Vladlen Koltun.  Guided policy search.  In International conference on machine learning, pages 1–9. PMLR, 2013.
  * [18] Weiwei Li and Emanuel Todorov.  Iterative linear quadratic regulator design for nonlinear biological movement systems.  In ICINCO (1), pages 222–229. Citeseer, 2004.
  * [19] Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba.  Learning compositional koopman operators for model-based control.  arXiv preprint arXiv:1910.08264, 2019.
  * [20] Bethany Lusch, J Nathan Kutz, and Steven L Brunton.  Deep learning for universal linear embeddings of nonlinear dynamics.  Nature communications, 9(1):1–10, 2018.
  * [21] Kevin M Lynch and Frank C Park.  Modern robotics.  Cambridge University Press, 2017.
  * [22] Giorgos Mamakoukas, Maria L Castano, Xiaobo Tan, and Todd D Murphey.  Derivative-based koopman operators for real-time control of robotic systems.  IEEE Transactions on Robotics, 37(6):2173–2192, 2021.
  * [23] Giorgos Mamakoukas, Orest Xherija, and Todd Murphey.  Memory-efficient learning of stable linear dynamical systems for prediction and control.  Advances in Neural Information Processing Systems, 33:13527–13538, 2020.
  * [24] Shankar Sastry.  Nonlinear systems: analysis, stability, and control, volume 10.  Springer Science & Business Media, 2013.
  * [25] Peter J Schmid.  Dynamic mode decomposition of numerical and experimental data.  Journal of fluid mechanics, 656:5–28, 2010.
  * [26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [27] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In International conference on machine learning, pages 387–395. PMLR, 2014.
  * [28] Yuval Tassa, Tom Erez, and Emanuel Todorov.  Synthesis and stabilization of complex behaviors through online trajectory optimization.  In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4906–4913. IEEE, 2012.
  * [29] Michael Tschannen, Olivier Bachem, and Mario Lucic.  Recent advances in autoencoder-based representation learning.  arXiv preprint arXiv:1812.05069, 2018.
  * [30] Jonathan H Tu.  Dynamic mode decomposition: Theory and applications.  PhD thesis, Princeton University, 2013.
  * [31] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba.  Benchmarking model-based reinforcement learning.  arXiv preprint arXiv:1907.02057, 2019.
  * [32] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller.  Embed to control: A locally linear latent dynamics model for control from raw images.  Advances in neural information processing systems, 28, 2015.
  * [33] Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley.  A data–driven approximation of the koopman operator: Extending dynamic mode decomposition.  Journal of Nonlinear Science, 25(6):1307–1346, 2015.
  * [34] Rana MM Yousef and Khalil El-Hindi.  Locating Center Points for Radial Basis Function Networks Using Instance Reduction Techniques.  University of Jordan, 2004.