# R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-
coupled state Estimation and mapping package

Jiarong Lin and Fu Zhang J. Lin and F. Zhang are with the Department of
Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China.
$\\{$jiarong.lin<EMAIL_ADDRESS>
###### Abstract

In this letter, we propose a novel LiDAR-Inertial-Visual sensor fusion
framework termed R3LIVE, which takes advantage of measurement of LiDAR,
inertial, and visual sensors to achieve robust and accurate state estimation.
R3LIVE is contained of two subsystems, the LiDAR-inertial odometry (LIO) and
visual-inertial odometry (VIO). The LIO subsystem (FAST-LIO) takes advantage
of the measurement from LiDAR and inertial sensors and builds the geometry
structure of (i.e. the position of 3D points) global maps. The VIO subsystem
utilizes the data of visual-inertial sensors and renders the map’s texture
(i.e. the color of 3D points). More specifically, the VIO subsystem fuses the
visual data directly and effectively by minimizing the frame-to-map
photometric error. The developed system R3LIVE is developed based on our
previous work R2LIVE, with careful architecture design and implementation.
Experiment results show that the resultant system achieves more robustness and
higher accuracy in state estimation than current counterparts (see our
attached video111https://youtu.be/j5fT8NE5fdg).

R3LIVE is a versatile and well-engineered system toward various possible
applications, which can not only serve as a SLAM system for real-time robotic
applications, but can also reconstruct the dense, precise, RGB-colored 3D maps
for applications like surveying and mapping. Moreover, to make R3LIVE more
extensible, we develop a series of offline utilities for reconstructing and
texturing meshes, which further minimizes the gap between R3LIVE and various
of 3D applications such as simulators, video games and etc (see our demos
video222https://youtu.be/4rjrrLgL3nk). To share our findings and make
contributions to the community, we open source R3LIVE on our
Github333https://github.com/hku-mars/r3live, including all of our codes,
software utilities, and the mechanical design of our device.

Figure 1: (a) Our developed R3LIVE is able to reconstruct a dense, 3D, RGB-
colored point cloud of the traveled environment in real-time. The white path
is our traveling trajectory for collecting the data. (b) offline reconstructed
mesh, and the mesh after textured is rendered in (c).

##  I Introduction

Recently, LiDAR sensors have been increasingly used in various robotic
applications such as autonomous driving vehicles [1], drones [2, 3, 4], etc.
Especially with the emergence of low-cost solid-state LiDARs (e.g., [5]), more
applications [6, 7, 8, 9, 10] based on these LiDARs propel the development of
robotic community. However, for LiDAR-based SLAM systems, they would easily
fail in those scenarios with no enough geometry features, especially for
solid-state LiDARs which typically have limited FoV [11]. To address this
problem, fusing LiDAR with other sensors such as camera[12, 13, 14, 15] and
Ultra-wideband (UWB)[16, 17] can improve the system’s robustness and accuracy.
In particular, various of LiDAR-Visual fusion frameworks have been proposed
recently in the robotics community[18].

V-LOAM [19] proposed by Zhang and Singh is one of the early works of LiDAR-
Inertial-Visual systems, which leverage a loosely-coupled Visual-Inertial
odometry (VIO) as the motion model for initializing the LiDAR mapping
subsystem. Similarly, in [20], the authors propose a stereo visual-inertial
LiDAR SLAM which incorporates the tightly-coupled stereo visual-inertial
odometry with the LiDAR mapping and LiDAR enhanced visual loop closure. More
recently, Wang et al propose DV-LOAM [21], which is a direct Visual-LiDAR
fusion framework. The system first utilizes a two-staged direct visual
odometry module for efficient coarse state estimate, then the coarse pose is
refined with the LiDAR mapping module, and finally the loop-closure module is
leveraged for correcting the accumulated drift. The above systems fuse LiDAR-
inertial-visual sensors at a level of loosely-coupled, in which the LiDAR
measurement are not jointly optimized together with the visual or inertial
measurements.

More tightly-coupled LiDAR-Inertial-Visual fusion frameworks have been
proposed recently. For example, LIC-fusion[14] proposed by Zuo et al is a
tightly-coupled LiDAR-Inertial-Visual fusion framework that combines IMU
measurements, sparse visual features, LiDAR features with online spatial and
temporal calibration within the Multi-State Constrained Kalman Filter (MSCKF)
framework. To further enhance the robustness of the LiDAR scan matching, their
subsequent work termed LIC-Fusion 2.0[15] proposes a plane-feature tracking
algorithm across multiple LiDAR scans within a sliding-window and refines the
pose trajectories within the window. Shan et al in [13] propose LVI-SAM fuses
LiDAR-Visual-Inertial senor via a tightly-coupled smooth and mapping
framework, which is built atop a factor graph. The LiDAR-Inertial and Visual-
Inertial subsystems of LVI-SAM can function independently when failure is
detected in one of them, or jointly when enough features are detected. Our
previous work R2LIVE [12] tightly fuses the data of LiDAR-Inertial-Visual
sensors, extract LiDAR and sparse visual features, estimates the state by
minimizing the feature re-projection error within the framework of error-state
iterated Kalman-filter for achieving the real-time performance, meanwhile
improves the overall visual mapping accuracy with a sliding window
optimization. R2LIVE is able to run in various challenging scenarios with
aggressive motions, sensor failures, and even in narrow tunnel-like
environments with a large number of moving objects and with a small LiDAR FoV.

In this paper, we attack the problem of real-time simultaneous localization,
3D mapping, and map renderization based on tightly-coupled fusion of LiDAR,
inertial, and visual measurements. Our contributions are:

  * •

We propose a real-time simultaneous localization, mapping, and colorization
framework. The presented framework consists of a LiDAR-inertial odometry (LIO)
for reconstructing geometry structure and a visual-inertial odometry (VIO) for
texture rendering. The overall system is able to reconstruct a dense, 3D, RGB-
colored point cloud of the environment in real-time (Fig. 1 (a))

  * •

We propose a novel VIO system based on the RGB-colored point cloud map. The
VIO estimates the current state by minimizing the photometric error between
the RGB color of an observed map point and its measured color in the current
image. Such a process requires no salient visual feature in the environment
and saves the corresponding processing time (e.g. features detection and
extraction), which makes our proposed system more robust especially in
texture-less environments.

  * •

We implement the proposed methods into a complete system, R3LIVE, which is
able to construct the dense, precise, 3D, RGB-colored point cloud maps of the
environment in real-time and with low-drift. The overall system is validated
in various indoor and outdoor environments. Results show that our system
drifts only $0.16$ meters in translation and $3.9$ degree in rotation after
traveling up to 1.5 kilometers.

  * •

We open source our system on our Github. We also develop several offline tools
for reconstructing and texturing meshes from colorized point cloud (see Fig. 1
(b) and (c)). These software utilities and mechanical design of our devices
are also made open source to benefit the possible applications.

Figure 2: The overview of our proposed system.

##  II The system overview

The overview of our system is shown in Fig. 2, our proposed framework contains
two subsystems: the LIO subsystem (upper part) and the VIO subsystem (the
lower part). The LIO subsystem constructs the geometry structure of a global
map, which registers the input LiDAR scan, and estimates the system’s state by
minimizing the point-to-plane residuals. The VIO subsystem builds the map’s
texture, which renders the RGB color of each point with the input image,
updates the system state by minimizing the frame-to-frame PnP reprojection
error and the frame-to-map photometric error.

##  III Notation

Throughout this paper, we use notations shown in TABLE.
LABEL:table_I_NOMENCLATURE, which have been introduced in the previous work
R2LIVE[12].

TABLE I: NOMENCLATURE Notation |  Explanation
---|---
|  Expression
$\boxplus/\boxminus$ | The encapsulated “boxplus” and
| “boxminus” operations on manifold
${{}^{G}}(\cdot)$ | The coordinate of vector $(\cdot)$ in global frame
${{}^{C}}(\cdot)$ | The coordinate of vector $(\cdot)$ in camera frame
$\mathtt{Exp}(\cdot)/\mathtt{Log}(\cdot)$ | The Rodrigues’ transformation between the
| rotation matrix and rotation vector
$\delta\left(\cdot\right)$ | The estimated error of $(\cdot)$. It is the minimum
| parameterization that characterized in the tangent space.
$\bm{\Sigma}_{(\cdot)}$ | The covariance matrix of vector $(\cdot)$
|  Variable
$\mathbf{b}_{\mathbf{g}},\mathbf{b}_{\mathbf{a}}$ | The bias of gyroscope and accelerometer
${{}^{G}}\mathbf{g}$ | The gravitational acceleration in global frame
${{}^{G}}\mathbf{v}$ | The linear velocity in global frame
$({{}^{G}}\mathbf{R}_{I},{{}^{G}}\mathbf{p}_{I})$ | The attitude and position of the IMU w.r.t. global frame
$({{}^{I}}\mathbf{R}_{C},{{}^{I}}\mathbf{p}_{C})$ | The extrinsic value between camera and IMU
$\mathbf{x}$ | The full state vector
$\hat{\mathbf{x}}$ | The prior estimation of $\mathbf{x}$
$\check{\mathbf{x}}$ | The current estimate of $\mathbf{x}$ in each ESIKF iteration

###  III-A State vector

In our work, we define the full state vector $\mathbf{x}\in\mathbb{R}^{29}$
as:

$\displaystyle\mathbf{x}=\begin{bmatrix}{}^{G}\mathbf{R}_{I}^{T},{{}^{G}}\mathbf{p}_{I}^{T},{{}^{G}}\mathbf{v}^{T},\mathbf{b}_{\mathbf{g}}^{T},\mathbf{b}_{\mathbf{a}}^{T},{{}^{G}}\mathbf{g}^{T},{{}^{I}}\mathbf{R}_{C}^{T},{{}^{I}}\mathbf{p}_{C}^{T},{{}^{I}}t_{C},\bm{\phi}^{T}\end{bmatrix}^{T}$
(1)

where ${{}^{G}}\mathbf{g}\in\mathbb{R}^{3}$ is the gravity vector expressed in
global frame (i.e. the first LiDAR frame), ${{}^{I}}t_{C}$ is the time-offset
between IMU and camera while LiDAR is assumed to be synced with IMU already,
$\bm{\phi}=\begin{bmatrix}f_{x},f_{y},c_{x},c_{y}\end{bmatrix}^{T}$ is the
camera intrinsic vector, with $(f_{x},f_{y})$ the pixel focal length and
$(c_{x},c_{y})$ the offsets of the principal point from the top-left corner of
the image plane.

###  III-B Maps representation

Our map is made up of voxels and points, where points are contained in voxels
and are the minimum elements of maps.

####  III-B1 Voxels

For fast finding the points in maps for rendering and tracking (see Section.
V-C and Section. V-D) in our VIO subsystem, we design a fix-size (e.g.
$0.1m\times 0.1m\times 0.1m$) container named voxel. If a voxel has points
appended recently (e.g. in recent 1 second), we mark this voxel as activated.
Otherwise, this voxel is marked as deactivated.

####  III-B2 Points

In our work, a points $\mathbf{P}$ is a vector with size 6:

$\displaystyle\mathbf{P}=\left[{{}^{G}}\mathbf{p}_{x},{{}^{G}}\mathbf{p}_{y},{{}^{G}}\mathbf{p}_{z},\mathbf{c}_{r},\mathbf{c}_{g},\mathbf{c}_{b}\right]^{T}=\left[{{}^{G}}\mathbf{p}^{T},\mathbf{c}^{T}\right]^{T}$
(2)

where the first $3\times 1$ sub-vector
${{}^{G}}\mathbf{p}=\left[\mathbf{P}_{x},\mathbf{P}_{y},\mathbf{P}_{z}\right]^{T}$
denotes the point 3D position in the global frame, and the second $3\times 1$
vector
$\mathbf{c}=\left[\mathbf{c}_{r},\mathbf{c}_{g},\mathbf{c}_{b}\right]^{T}$ is
the point RGB color. Besides, we also record other informations of this point,
such as the $3\times 3$ covariance matrix $\bm{\Sigma}_{\mathbf{p}}$ and
$\bm{\Sigma}_{\mathbf{c}}$, which denote the estimated errors of
${{}^{G}}\mathbf{p}$ and $\mathbf{c}$, respectively, and the timestamps when
this point have been created and rendered.

##  IV LiDAR-Inertial odometry subsystem

As shown in Fig. 2, the LIO subsystem of R3LIVE constructs the geometry
structure of the global map. For an incoming LiDAR scan, the motion distortion
due to the continuous movement within the frame is compensated by IMU backward
propagation, as shown in [6]. Then, we leverage the error-state iterated
Kalman filter (ESIKF) minimizing the point-to-plane residuals to estimate the
system’s state. Finally, with the converged state, the points of this scan are
appended to the global map and mark the corresponding voxels as activated or
deactivated. The accumulated 3D points in the global map form the geometry
structure, which is also used for providing the depth for our VIO subsystem.
For the detailed implementations of the LIO subsystem in R3LIVE, we refer our
readers to our previous related works [22, 12].

##  V Visual-Inertial odometry subsystem

Our VIO subsystem renders the texture of a global map, with estimating the
system state by minimizing the photometric error. To be more specific, we
project a certain number of points (i.e. tracked points) from the global map
to the current image, then iteratively estimate the system state within the
framework of ESIKF by minimizing the photometric error of these points. The
tracked map points are sparse for the sake of efficiency, which usually
requires to build a
pyramid444https://en.wikipedia.org/wiki/Pyramid_(image_processing) of the
input image. The pyramid is however not invariant to either translation or
rotation that needs also to be estimated. In our proposed framework, we
utilize the color of a single map point to compute the photometric error. The
color, which is rendered simultaneously in the VIO, is an inherent property of
the map point and is invariant to both translation and rotation of the camera.
To ensure a robust and fast convergence, we design a two-step pipeline shown
in Fig. 2, where we firstly leverage a frame-to-frame optical flow to track
map points and optimize the system’s state by minimizing the Perspective-n-
Point (PnP) projection error of the tracked map points (Section V-A). Then, we
further refine the system’s state estimation by minimizing the frame-to-map
photometric error among the tracked points (Section V-B). With the converged
state estimate and the raw input image, we perform the texture rendering to
update the colors of points in the global map (Section V-C).

###  V-A Frame-to-frame Visual-Inertial odometry

Assume we have tracked $m$ map points
$\bm{\mathcal{P}}=\\{\mathbf{P}_{1},...,\mathbf{P}_{m}\\}$ in last image frame
$\mathbf{I}_{k-1}$, and their projection on the $\mathbf{I}_{k-1}$ are
$\\{\bm{\rho}_{1_{k-1}},...,\bm{\rho}_{m_{k-1}}\\}$, we leverage the
Lucas$-$Kanade optical flow to find out their location in the current image
frame $\mathbf{I}_{k}$, denoted as
$\\{\bm{\rho}_{1_{k}},...,\bm{\rho}_{m_{k}}\\}$. Then, we calculate and
optimize their projection error to estimate the state via an ESIKF.

####  V-A1 Perspective-n-Point projection error

Figure 3: Our frame-to-frame ESIKF VIO update the system’s state with
minimizing the PnP projection error.

As shown in Fig. 3, we take the $s$-th point
$\mathbf{P}_{s}=\left[{{}^{G}}\mathbf{p}_{s}^{T},\mathbf{c}_{s}^{T}\right]^{T}\in\bm{\mathcal{P}}$
as example, the projection error is calculated as follow
$\mathbf{r}\left(\check{\mathbf{x}}_{k},\bm{\rho}_{s}^{k},{{}^{G}\mathbf{p}_{s}}\right)$:

$\displaystyle\begin{split}{{}^{C}}\mathbf{p}_{s}&=\left[{{}^{C}}\mathbf{p}_{x_{s}},{{}^{C}}\mathbf{p}_{y_{s}},{{}^{C}}\mathbf{p}_{z_{s}}\right]^{T}=\left({{}^{G}\check{\mathbf{R}}_{I_{k}}}\cdot{{}^{I}\check{\mathbf{R}}_{C_{k}}}\right)^{T}\cdot{{{}^{G}}{\mathbf{p}}_{s}}\\\
&\quad\quad\quad-{{}^{I}\check{\mathbf{R}}_{C_{k}}^{T}}\cdot{{}^{I}\check{\mathbf{p}}_{C_{k}}}-\left({{}^{G}\check{\mathbf{R}}_{I_{k}}}\cdot{{}^{I}\check{\mathbf{R}}_{C_{k}}}\right)^{T}\cdot{{{}^{G}}\check{\mathbf{p}}_{I_{k}}}\end{split}$
(3) $\displaystyle\mathbf{r}$
$\displaystyle\left(\check{\mathbf{x}}_{k},\bm{\rho}_{s_{k}},{{}^{G}\mathbf{p}_{s}}\right)=\bm{\rho}_{s_{k}}-\bm{\pi}({{}^{C}\mathbf{p}_{s}},\check{\mathbf{x}}_{k})$
(4)

where $\check{\mathbf{x}}_{k}$ is the current state estimate of $\mathbf{x}$
in each ESIKF iteration,
$\bm{\pi}({{}^{C}\mathbf{p}_{s}},\check{\mathbf{x}}_{k})\in\mathbb{R}^{2}$ is
computed as follow:

$\displaystyle\begin{split}\bm{\pi}({{}^{C}\mathbf{p}_{s}},\check{\mathbf{x}}_{k})&=\left[\check{f}_{x_{k}}\dfrac{{{}^{C}}\mathbf{p}_{x_{s}}}{{{}^{C}}\mathbf{p}_{z_{s}}}+\check{c}_{x_{k}},~{}\check{f}_{y_{k}}\dfrac{{{}^{C}}\mathbf{p}_{y_{s}}}{{{}^{C}}\mathbf{p}_{z_{s}}}+\check{c}_{y_{k}}\right]^{T}\\\
&+\dfrac{{{}^{I}}\check{t}_{C_{k}}}{\Delta
t_{k-1,k}}(\bm{\rho}_{s_{k}}-\bm{\rho}_{s_{k-1}})\end{split}$ (5)

where $\Delta t_{k-1,k}$ is the time internal between last image frame
$\mathbf{I}_{k-1}$ and current image frame $\mathbf{I}_{k}$. Notice that in
(5), the first item is the pin-hole projection function and the second one is
the online-temporal correction factor [23].

The measurement noise in the residual (4) consists of two sources: one is the
pixel tracking error in $\bm{\rho}_{s_{k}}$ and the other lies in the map
point location error ${{}^{G}\mathbf{p}_{s}}$,

$\displaystyle{{}^{G}\mathbf{p}_{s}}={{}^{G}\mathbf{p}_{s}^{\mathtt{gt}}}+\mathbf{n}_{\mathbf{p}_{s}},~{}$
$\displaystyle\mathbf{n}_{\mathbf{p}_{s}}\sim\mathcal{N}(\mathbf{0},\bm{\Sigma}_{\mathbf{n}_{\mathbf{p}_{s}}})$
(6)
$\displaystyle\bm{\rho}_{s_{k}}={\bm{\rho}_{s_{k}}^{\mathtt{gt}}}+\mathbf{n}_{\bm{\rho}_{s_{k}}},~{}$
$\displaystyle\mathbf{n}_{\bm{\rho}_{s_{k}}}\sim\mathcal{N}(\mathbf{0},\bm{\Sigma}_{\mathbf{n}_{\bm{\rho}_{s_{k}}}})$
(7)

where ${{}^{G}\mathbf{p}_{s}^{\mathtt{gt}}}$ and
${\bm{\rho}_{s_{k}}^{\mathtt{gt}}}$ are the true values of
${{}^{G}\mathbf{p}_{s}}$ and $\bm{\rho}_{s_{k}}$, respectively. Then, we
obtain the first order Taylor expansion of the true zero residual
$\mathbf{r}(\mathbf{x}_{k},\bm{\rho}_{s_{k}}^{\mathtt{gt}},{{}^{G}\mathbf{p}_{s}^{\mathtt{gt}}})$
as:

$\begin{split}\mathbf{0}&=\mathbf{r}(\mathbf{x}_{k},\bm{\rho}_{s_{k}}^{\mathtt{gt}},{{}^{G}\mathbf{p}_{s}^{\mathtt{gt}}})\approx\mathbf{r}\left(\check{\mathbf{x}}_{k},\bm{\rho}_{s_{k}},{{}^{G}\mathbf{p}_{s}}\right)+\mathbf{H}^{r}_{s}\delta\check{\mathbf{x}}_{k}+\bm{\alpha}_{s}\end{split}$
(8)

where
$\bm{\alpha}_{s}\sim\mathcal{N}(\mathbf{0},{\bm{\Sigma}_{\bm{\alpha}_{s}}})$
and

$\displaystyle\mathbf{H}^{r}_{s}$
$\displaystyle=\left.\dfrac{\partial\mathbf{r}_{c}(\check{\mathbf{x}}_{k}\boxplus\delta{\check{\mathbf{x}}}_{k},\bm{\rho}_{s_{k}},{{}^{G}\mathbf{p}_{s}})}{\partial\delta{\check{\mathbf{x}}}_{k}}\right|_{\delta{\check{\mathbf{x}}}_{k}=\mathbf{0}}$
(9) $\displaystyle{\bm{\Sigma}_{\bm{\alpha}_{s}}}$
$\displaystyle=\bm{\Sigma}_{\mathbf{n}_{\bm{\rho}_{s_{k}}}}+{\mathbf{F}^{r}_{{{\mathbf{p}}}_{s}}}\bm{\Sigma}_{\mathbf{p}_{s}}{\mathbf{F}^{r}_{{{\mathbf{p}}}_{s}}}^{T}$
(10) $\displaystyle\mathbf{F}^{r}_{{{\mathbf{p}}}_{s}}$
$\displaystyle=\dfrac{\partial\mathbf{r}(\check{\mathbf{x}}_{k},\bm{\rho}_{s_{k}},{{}^{G}\mathbf{p}_{s}})}{\partial{{}^{G}{\mathbf{p}}}_{s}}.$
(11)

####  V-A2 Frame-to-frame VIO ESIKF update

Equation (8) constitutes an observation distribution for ${\mathbf{x}}_{k}$
(or equivalently
$\delta\check{\mathbf{x}}_{k}\triangleq{\mathbf{x}}_{k}\boxminus\check{\mathbf{x}}_{k}$),
which can be combined with the prior distribution from the IMU propagation to
obtain the maximum a posteriori (MAP) estimation of
$\delta\check{\mathbf{x}}_{k}$:

$\displaystyle\begin{split}\mathop{\min}_{\delta\check{\mathbf{x}}_{k}}&\left(\left\|\check{\mathbf{x}}_{k}\boxminus\hat{\mathbf{x}}_{k}+\bm{\mathcal{H}}\delta{\check{\mathbf{x}}}_{k}\right\|_{\bm{\Sigma}_{\delta\hat{\mathbf{x}}_{k}}}^{2}\right.\\\
+&\left.\sum\nolimits_{s=1}^{m}\left\|{\mathbf{r}\left(\check{\mathbf{x}}_{k},\bm{\rho}_{s_{k}},{{}^{G}\mathbf{p}_{s}}\right)+\mathbf{H}^{r}_{s}\delta\check{\mathbf{x}}_{k}}\right\|^{2}_{{\bm{\Sigma}_{\bm{\alpha}_{s}}}}\right)\end{split}$
(12)

where
$\left\|\mathbf{x}\right\|_{\bm{\Sigma}}^{2}=\mathbf{x}^{T}\bm{\Sigma}^{-1}\mathbf{x}$
is the squared Mahalanobis distance with covariance $\bm{\Sigma}$,
$\hat{\mathbf{x}}_{k}$ is the IMU propagated state estimate,
$\bm{\Sigma}_{\delta\hat{\mathbf{x}}_{k}}$ is the IMU propagated state
covariance, and $\bm{\mathcal{H}}$ is the Jacobian matrix when projecting the
state error from the tangent space of $\hat{\mathbf{x}}_{k}$ to the tangent
space of $\check{\mathbf{x}}_{k}$. The detailed derivation of first item in
(12) can be found in Section. E of R2LIVE[12].

Denote:

$\displaystyle\mathbf{H}$
$\displaystyle=\begin{bmatrix}{\mathbf{H}^{r}_{1}}^{T},\dots,{\mathbf{H}^{r}_{m}}^{T}\end{bmatrix}^{T}$
(13) $\displaystyle\mathbf{R}$
$\displaystyle=\text{diag}(\begin{matrix}\bm{\Sigma}_{\bm{\alpha}_{1}},\dots,\bm{\Sigma}_{\bm{\alpha}_{m}}\end{matrix})$
(14) $\displaystyle\check{\mathbf{z}}_{k}$
$\displaystyle=\left[\mathbf{r}\left(\check{\mathbf{x}}_{k},\bm{\rho}_{1_{k}},{{}^{G}\mathbf{p}_{1}}\right)\dots,\mathbf{r}\left(\check{\mathbf{x}}_{k},\bm{\rho}_{m_{k}},{{}^{G}\mathbf{p}_{m}}\right)\right]^{T}$
(15) $\displaystyle\mathbf{P}$
$\displaystyle=\left(\bm{\mathcal{H}}\right)^{-1}\bm{\Sigma}_{\delta\hat{\mathbf{x}}_{k}}{\left(\bm{\mathcal{H}}\right)}^{-T}$
(16)

Following [6], we have the Kalman gain computed as:

$\displaystyle\mathbf{K}=\left(\mathbf{H}^{T}\mathbf{R}^{-1}\mathbf{H}+\mathbf{P}^{-1}\right)^{-1}\mathbf{H}^{T}\mathbf{R}^{-1}$
(17)

Then we can update the state estimate as:

$\displaystyle\check{\mathbf{x}}_{k}=$
$\displaystyle\check{\mathbf{x}}_{k}\boxplus\left(-\mathbf{K}\check{\mathbf{z}}_{k}-\left(\mathbf{I}-\mathbf{KH}\right)\left(\bm{\mathcal{H}}\right)^{-1}\left(\check{\mathbf{x}}_{k}\boxminus\hat{\mathbf{x}}_{k}\right)\right)$
(18)

The above process (Section V-A1 to Section V-A2) is iterated until convergence
(i.e., the update is smaller than a given threshold). Notice that such an
iterated Kalman Filter is known to be equivalent to a Gauss-Newton
optimization [24].

Figure 4: Our frame-to-map ESIKF VIO update the system state by minimizing the
photometric error between the map point color and its measured color in the
curren timage.

###  V-B Frame-to-map Visual-Inertial odometry

####  V-B1 Frame-to-map photometric update

After the frame-to-frame VIO update, we have a good estimated state
$\check{\mathbf{x}}_{k}$, then we perform the frame-to-map VIO update by
minimizing the photometric error of the tracked points to lower the drift.As
shown in Fig. 4, take the $s$-th tracked point
$\mathbf{P}_{s}\in\bm{\mathcal{P}}$ as example, its frame-to-map photometric
error
$\mathbf{o}(\check{\mathbf{x}}_{k},{{}^{G}}\mathbf{p}_{s},\mathbf{c}_{s})$ is
calculated as follow:

$\displaystyle\mathbf{o}(\check{\mathbf{x}}_{k},{{}^{G}}\mathbf{p}_{s},\mathbf{c}_{s})=\mathbf{c}_{s}-\bm{\gamma}_{s}$
(19)

where $\mathbf{c}_{s}$ is the point color saved in the global map,
$\bm{\gamma}_{s}$ is the color observed in the current image $\mathbf{I}_{k}$.
To obtain the observed color $\bm{\gamma}_{s}$ and its covariance
$\bm{\Sigma}_{\mathbf{n}_{{\bm{\gamma}_{s}}}}$, we predict the point location
$\tilde{\bm{\rho}}_{s_{k}}=\bm{\pi}({{}^{C}\mathbf{p}_{s}},\check{\mathbf{x}}_{k})$
on the current image $\mathbf{I}_{k}$ and then linearly interpolate the RGB
colors of neighborhood pixels.

We also consider the measurement noise of $\bm{\gamma}_{s}$ and
$\mathbf{c}_{s}$:

$\displaystyle\bm{\gamma}_{s}=\bm{\gamma}^{gt}_{s}+\mathbf{n}_{{\bm{\gamma}_{s}}},~{}$
$\displaystyle\mathbf{n}_{{\bm{\gamma}_{s}}}\sim\mathcal{N}(\mathbf{0},\bm{\Sigma}_{\mathbf{n}_{{\bm{\gamma}_{s}}}})$
(20)
$\displaystyle\begin{split}\mathbf{c}_{s}=\mathbf{c}_{s}^{gt}+\mathbf{n}_{\mathbf{c}_{s}}+\bm{\eta}_{\mathbf{c}_{s}},~{}&\mathbf{n}_{\mathbf{c}_{s}}\sim{\mathcal{N}}(\mathbf{0},\bm{\Sigma}_{\mathbf{n}_{\mathbf{c}_{s}}})\\\
&\bm{\eta}_{\mathbf{c}_{s}}\sim{\mathcal{N}}(\mathbf{0},\bm{\sigma}^{2}_{s}\cdot\Delta
t_{\mathbf{c}_{s}})\end{split}$ (21)

where $\bm{\gamma}^{gt}_{s}$ is the ground true value of $\bm{\gamma}_{s}$ and
$\mathbf{c}_{s}^{gt}$ is the true value of $\mathbf{c}_{s}$, $\Delta
t_{\mathbf{c}_{s}}$ is the time internal between current frame and last
rendering time of $\mathbf{P}_{s}$. Notice that in (21), the measurement noise
of $\mathbf{c}_{s}$ consists of two parts: the estimated error
$\mathbf{n}_{\mathbf{c}_{s}}$ from the last time of rendering (see Section.
V-C), and the impulsed random walk process noise $\bm{\eta}_{\mathbf{c}_{s}}$,
which accounts for the change of environment illusion.

Combining (19), (20) and (21), we obtain the first order Taylor expansion of
the true zero residual
$\mathbf{o}(\mathbf{x}_{k},{{}^{G}}\mathbf{p}^{gt}_{s},\mathbf{c}^{gt}_{s})$:

$\displaystyle\begin{split}\mathbf{0}&=\mathbf{o}(\mathbf{x}_{k},{{}^{G}}\mathbf{p}^{gt}_{s},\mathbf{c}^{gt}_{s})\approx\mathbf{o}(\check{\mathbf{x}}_{k},{{}^{G}}\mathbf{p}_{s},\mathbf{c}_{s})+\mathbf{H}_{s}^{o}\delta\check{\mathbf{x}}_{k}+\bm{\beta}_{s}\end{split}$
(22)

where $\bm{\beta}_{s}\sim\mathcal{N}(\mathbf{0},\bm{\Sigma}_{\beta_{s}})$ and:

$\displaystyle\mathbf{H}^{o}_{s}$
$\displaystyle=\left.\dfrac{\partial\mathbf{o}(\check{\mathbf{x}}_{k}\boxplus\delta{\check{\mathbf{x}}}_{k},{{}^{G}}\mathbf{p}_{s},\mathbf{c}_{s})}{\partial\delta{\check{\mathbf{x}}}_{k}}\right|_{\delta{\check{\mathbf{x}}}_{k}=\mathbf{0}}$
(23) $\displaystyle{\bm{\Sigma}_{\bm{\beta}_{s}}}$
$\displaystyle=\bm{\Sigma}_{\mathbf{n}_{\mathbf{c}_{s}}}+\bm{\sigma}^{2}_{s}\cdot\Delta
t_{\mathbf{c}_{s}}+\bm{\Sigma}_{\mathbf{n}_{{\bm{\gamma}_{s}}}}+{\mathbf{F}^{o}_{{{\mathbf{p}}}_{s}}}\bm{\Sigma}_{\mathbf{p}_{s}}{\mathbf{F}^{o}_{{{\mathbf{p}}}_{s}}}^{T}$
(24) $\displaystyle{\mathbf{F}^{o}_{{{\mathbf{p}}}_{s}}}$
$\displaystyle=\dfrac{\partial\mathbf{o}(\check{\mathbf{x}}_{k},{{}^{G}}\mathbf{p}_{s},\mathbf{c}_{s})}{\partial{{}^{G}{\mathbf{p}}}_{s}}$
(25)

####  V-B2 Frame-to-map VIO ESIKF update

Equation (22) constitutes another observation distribution for
$\delta\check{\mathbf{x}}_{k}$, which is combined with the prior distribution
from the IMU propagation to obtain the maximum a posteriori (MAP) estimation
of $\delta\check{\mathbf{x}}_{k}$:

$\displaystyle\begin{split}\mathop{\min}_{\delta\check{\mathbf{x}}_{k}}&\left(\left\|\check{\mathbf{x}}_{k}\boxminus\hat{\mathbf{x}}_{k}+\bm{\mathcal{H}}\delta{\check{\mathbf{x}}}_{k}\right\|_{\bm{\Sigma}_{\delta\hat{\mathbf{x}}_{k}}}^{2}\right.\\\
+&\left.\sum\nolimits_{s=1}^{m}\left\|{\mathbf{o}(\check{\mathbf{x}}_{k},{{}^{G}}\mathbf{p}_{s},\mathbf{c}_{s})+\mathbf{H}^{o}_{s}\delta\check{\mathbf{x}}_{k}}\right\|^{2}_{{\bm{\Sigma}_{\bm{\beta}_{s}}}}\right)\end{split}$
(26)

Denote $\mathbf{H},\mathbf{R},\check{\mathbf{z}}_{k}$ and $\mathbf{P}$ similar
to (13) $\sim$ (16) :

$\displaystyle\mathbf{H}$
$\displaystyle=\begin{bmatrix}{\mathbf{H}^{o}_{1}}^{T},\dots,{\mathbf{H}^{o}_{m}}^{T}\end{bmatrix}^{T}$
(27) $\displaystyle\mathbf{R}$
$\displaystyle=\text{diag}(\begin{matrix}\bm{\Sigma}_{\bm{\beta}_{1}},\dots,\bm{\Sigma}_{\bm{\beta}_{m}}\end{matrix})$
(28) $\displaystyle\check{\mathbf{z}}_{k}$
$\displaystyle=\left[\mathbf{o}(\check{\mathbf{x}}_{k},{{}^{G}}\mathbf{p}_{1},\mathbf{c}_{1})\dots,\mathbf{o}(\check{\mathbf{x}}_{k},{{}^{G}}\mathbf{p}_{m},\mathbf{c}_{m})\right]^{T}$
(29) $\displaystyle\mathbf{P}$
$\displaystyle=\left(\bm{\mathcal{H}}\right)^{-1}\bm{\Sigma}_{\delta\hat{\mathbf{x}}_{k}}{\left(\bm{\mathcal{H}}\right)}^{-T}$
(30)

then we perform the state update similar to (17) and (18). This frame-to-map
VIO ESIKF update (Section V-B1 to Section V-B2) is iterated until convergence.
The converged state estimate is then used to: (1) render the texture of maps
(Section V-C); (2) update the current tracking point set $\bm{\mathcal{P}}$
for the next frame to use (Section V-D); and (3) serve as the starting point
of the IMU propagation in the next frame of LIO or VIO update.

$\displaystyle\hat{\mathbf{x}}_{k}=\check{\mathbf{x}}_{k},\hskip
5.69046pt\bm{\Sigma}_{\delta\hat{\mathbf{x}}_{k}}=\left(\mathbf{I}-\mathbf{KH}\right)\bm{\Sigma}_{\delta\check{\mathbf{x}}_{k}}$
(31)

###  V-C Render the texture of global map

Figure 5: We render the color of point $\mathbf{c}_{s}$ via Bayesian update.

After the frame-to-map VIO update, we have the precise pose of the current
image, and then we perform the rendering function to update the color of map
points.

First of all, we retrive all the points in all activated voxels (see Section.
IV). Assume the total number points is $n$ points and the point set
$\bm{\zeta}=\\{\mathbf{P}_{1},...,\mathbf{P}_{n}\\}$. Taking the color update
process of the $s$-th point
$\mathbf{P}_{s}=\left[{{}^{G}}\mathbf{p}_{s}^{T},\mathbf{c}_{s}^{T}\right]^{T}\in\bm{\zeta}$
as an example. If $\mathbf{P}_{s}$ falls in the current image
$\mathbf{I}_{k}$, we obtain its observation color $\bm{\gamma}_{s}$ and
covariance $\bm{\Sigma}_{\mathbf{n}_{{\bm{\gamma}_{s}}}}$ by linearly
interpolating the RGB values of the neighborhood pixels on $\mathbf{I}_{k}$.
The newly observed point color is fused with the existing color
$\mathbf{c}_{s}$ recorded in the map via Bayesian update (see Fig. 5), leading
to the updated color and the associated covariance of $\mathbf{c}_{s}$ as
follows:

$\displaystyle\bm{\Sigma}_{\mathbf{n}_{\tilde{\mathbf{c}}_{s}}}=\left(\left(\bm{\Sigma}_{\mathbf{n}_{\mathbf{c}_{s}}}+\bm{\sigma}^{2}_{s}\cdot\Delta
t_{\mathbf{c}_{s}}\right)^{-1}+\bm{\Sigma}_{\mathbf{n}_{\bm{\gamma}_{s}}}^{-1}\right)^{-1}$
(32)
$\displaystyle\tilde{\mathbf{c}}_{s}=\left(\left(\bm{\Sigma}_{\mathbf{n}_{\mathbf{c}_{s}}}+\bm{\sigma}^{2}_{s}\cdot\Delta
t_{\mathbf{c}_{s}}\right)^{-1}\mathbf{c}_{s}+\bm{\Sigma}_{\mathbf{n}_{\bm{\gamma}_{s}}}^{-1}\bm{\gamma}_{s}\right)^{-1}\bm{\Sigma}_{\mathbf{n}_{\tilde{\mathbf{c}}_{s}}}$
(33) $\displaystyle\hskip
42.67912pt{\mathbf{c}}_{s}=\tilde{\mathbf{c}}_{s},~{}~{}{\bm{\Sigma}}_{\mathbf{n}_{\mathbf{c}_{s}}}=\bm{\Sigma}_{\mathbf{n}_{\tilde{\mathbf{c}}_{s}}}\hskip
42.67912pt$ (34)

###  V-D Update of the tracking points of VIO subsystem

After the rendering of texture, we perform the update of tracked point set
$\bm{\mathcal{P}}$. Firstly, we remove the points from $\bm{\mathcal{P}}$ if
their projection error in (4) or photometric error in (19) are too large, and
also remove the points which does not fall into $\mathbf{I}_{k}$. Secondly, we
project each point in $\bm{\zeta}$ to the current image $\mathbf{I}_{k}$, and
add it to $\bm{\mathcal{P}}$ if there have no other tracked points located
nearby (e.g. in the radius of $50$ pixels).

##  VI Experiment and Results

###  VI-A Equipment setup

Figure 6: Our handheld device for data collection, (a) shows our minimum
system, with a total weight of $2.09$ Kg; (b) an additional D-GPS RTK system
and an ArUco marker board [25] are used to evaluate the system accuracy.

Our handheld device for data collection is shown in Fig. 6 (a), which includes
a power supply unit, the onboard DJI
manifold-2c555https://www.dji.com/manifold-2 computation platform (equipped
with an Intel i7-8550u CPU and 8 GB RAM), a FLIR Blackfly BFS-u3-13y3c global
shutter camera, and a LiVOX AVIA666https://www.livoxtech.com/avia LiDAR. The
FoV of the camera is $82.9^{\circ}\times 66.5^{\circ}$ while the FoV of the
LiDAR is $70.4^{\circ}\times 77.2^{\circ}$. To quantitatively evaluate the
precision of our algorithm (Section VI-D), we install a Differential-GPS
(D-GPS) real-time kinematic (RTK) system777https://www.dji.com/d-rtk on our
device, shown in Fig. 6 (b). In GPS denied environment (Section VI-B and
Section VI-C), we use the ArUco marker [25] for calculating the drift of our
odometry. Notice that all of the mechanical modules of this device are
designed as FDM
printable888https://en.wikipedia.org/wiki/Fused_filament_fabrication, and to
faciliate our reader to reproduce our work, we open source all of our
schematics on our github999https://github.com/ziv-lin/rxlive_handheld.

###  VI-B Experiment-1: Robustness evaluation in simultaneously LiDAR
degenerated and visual texture-less environments

In this experiment, we challenge one of the most difficult scenarios in SLAM,
we perform the robustness evaluation in simultaneously LiDAR degenerated and
visual texture-less environments. As shown in Fig. 8, our sensors pass through
a narrow “T”-shape passage while occasionally facing against the side walls.
When facing the wall which imposes only a single plane constraint, the LiDAR
is well-known degenerated for full pose estimation. Meanwhile, the visual
texture on the white walls are very limited (Fig. 8(a) and Fig. 8(c)),
especially for the wall-1, which has the only changes of illumination. Such
scenarios are challenging for both LiDAR-based and visual-based SLAM methods.

Taking advantage of using the raw image pixel color information, our proposed
algorithm can “survive” in such extremely difficult scenarios. Fig. 8 shows
our estimated pose, with the phases of passing through “wall-1” and “wall-2”
shaded with blue and yellow, respectively. The estimated covariance is also
shown in Fig. 8, which are bounded over the whole estimated trajectory,
indicating that our estimation quality is stable over the whole process. The
sensor is moved to the starting point, where an ArUco marker board is used to
obtain the ground-truth relative pose between the starting and end poses.
Compared with the ground-truth end pose, our algorithm drifts $1.62^{\circ}$
in rotation and $4.57$ cm in translation. We recommend the readers to the
accompanying video on YouTube101010https://youtu.be/j5fT8NE5fdg?t=6 for better
visualization of the experiment.

Figure 7: We test our algorithm in simultaneously LiDAR degenerated and visual
texture-less scenarios.

Figure 8: The estimated pose and its 3-$\sigma$ bound with 5 times
amplification for better visualization (the light-colored area around the
trajectory) of Experiment-1. The shaded areas in blue and yellow are the
phases of the sensors facing against the white “wall-1” and “wall-2”,
respectively. In the end, the algorithm drifts $1.62^{\circ}$ in rotation and
$4.57$ cm in translation. Figure 9: We show the dense 3D RGB-colored maps
reconstructed in real-time in Experiment-2. In (a), we merge our maps with the
Google-Earth satellite image and find them aligned well. Details of the map
are selectively shown in (b), (c), and (d), respectively. In (e), we show that
our algorithm can close the loop itself without any additional processing
(e.g. loop-detection and closure), with the return trajectory can return back
to the origin. We refer readers to the video on our
Youtube121212https://youtu.be/j5fT8NE5fdg?t=104for more details.

###  VI-C Experiment-2: High precision mapping large-scale indoor & outdoor
urban environment

Figure 10: The birdview of the 4 trajectories in Experiments-2, the total
length of the four trajectories are $1317$ m , $1524$ m, $1372$ m and $1191$
m, respectively.

Figure 11: The changes of altitude among the 4 trajectories in Experiment-2,
where the time are normalized to 1.

In this experiment, we show the capacity of our algorithm for reconstructing a
precise, dense, 3D, RGB-colored map of a large-scale campus environment in
real-time. We collect the data within the campus of the Hong Kong University
of Science and Technology (HKUST) 4 times with different traveling
trajectories (i.e. Traj 1$\sim$4), with their total length of $1317$, $1524$,
$1372$ and $1191$ meters, respectively. The bird-view (i.e. project on X$-$Y
plane) of these trajectories are shown Fig. 11 and the changes of their
altitude are plotted in Fig. 11. Without any additional processing (e.g. loop
closure), all of these four trajectories can close the loop itself (see Fig.12
(e)). Using the ArUco marker board placed at the starting point, the odometry
drifts are presented in Table. II, which indicates that our proposed method is
of high accuracy with small drifts over the long trajectories and in
complicated environments. Finally, we present the the reconstructed map in
“Traj-1” in Fig. 12. More visualization results are available on the project
page131313https://github.com/ziv-lin/r3live_preview.

TABLE II: Odometry drift in 4 trajectories of Experiment-2. | Traj-1 | Traj-2 | Traj-3 | Traj-4
---|---|---|---|---
Length of trajectory (m) | 1317 | 1524 | 1372 | 1191
Drift in translation (m) | 0.093 | 0.154 | 0.164 | 0.102
Drift in rotation (deg) | 2.140 | 0.285 | 2.342 | 3.925

###  VI-D Experiment-3: Quantitative evaluation of precision using D-GPS RTK

Figure 12: Comparison of the estimated trajectories in Experiments-3.

In this experiment, we collect two sequences of data in a seaport (Belcher Bay
Promenade141414https://goo.gl/maps/eikg6Kvg9k4gM3Sm9) with a real-time
Differential-GPS (D-GPS) Kinematic (RTK) system providing the ground-truth
trajectory. In these two sequences, the sensors often faces many pedestrians
and occasionally open seas, where the LiDAR has very few measurements.

We compare the trajectories estimated by R3LIVE with two different
configurations (“R3LIVE-HiRES” and “R3LIVE-RT”, see Table. III), “LVI-SAM”
(with modified its LiDAR front-end for Livox Avia LiDAR), “R2LIVE”[12], “VINS-
Mono” (IMU+Camera)[26], “Fast-LIO2” (IMU+LiDAR)[22] with the ground-truth in
Fig. 12, where we can see that our estimated trajectories agree the best with
the ground-truth in both sequences. To have more quantitative comparisons, we
compute the relative rotation error (RPE) and relative translation error (RTE)
[27] over all possible sub-sequences of length (50,100,150,…,300) meters are
shown in Table. III.

TABLE III: The relative pose error in Experiment-3. In configuration “R3LIVE-
HiRES”, we set the input image resolution as $1280\times 1024$ and the minimum
distance between two points in maps as $0.01$ meters. In configuration
“R3LIVE-RT”, we set the input image resolution as $320\times 256$ and the
minimum distance between two points in maps as $0.10$ meters  Relative pose
error in Experiments-3 (a)
---
Sub-sequence | 50 m | 100 m | 150 m | 200 m | 250 m | 300 m
RRE/RTE | deg / % | deg / % | deg / % | deg / % | deg / % | deg / %
R3LIVE-HiRes |  0.99 / 2.25 |  0.53 / 1.02 |  0.46 / 0.60 |  0.29 / 0.31 |  0.24 / 0.31 |  0.21 / 0.17
R3LIVE-RT |  1.48 / 2.21 | 0.54 / 1.06 | 0.49 / 0.60 | 0.31 / 0.41 |  0.25/ 0.31 | 0.22 / 0.23
LVI_SAM | 2.11 / 13.73 | 1.04 / 6.69 | 0.83 / 5.07 | 0.60 / 3.86 | 0.47 / 2.98 | 0.43 / 2.40
R2LIVE | 1.21 / 2.47 | 0.61 / 1.02 | 0.59 / 0.60 | 0.34 / 0.34 | 0.37 / 0.33 | 0.34 / 0.21
FAST-LIO2 | 1.36 / 2.35 | 0.66 / 0.82 | 0.47 / 0.60 | 0.37 / 0.36 | 0.37 / 0.31 | 0.21 / 0.20
VINS-Mono | 3.03 / 10.41 | 1.70 / 7.63 | 1.15 / 6.36 | 0.89 / 4.34 | 0.73 / 3.08 | 0.59 / 2.31
Relative pose error in Experiments-3 (b)
Sub-sequence | 50 m | 100 m | 150 m | 200 m | 250 m | 300 m
RRE/RTE | deg / % | deg / % | deg / % | deg / % | deg / % | deg / %
R3LIVE-HiRes |  1.06 / 1.98 |  0.58 / 1.34 |  0.43 / 0.83 |  0.32 / 0.59 |  0.27 / 0.27 | 0.25 / 0.16
R3LIVE-RT |  0.98 / 2.08 |  0.55 / 1.61 | 0.47 / 0.99 | 0.39 / 0.65 | 0.33 / 0.33 |  0.25 / 0.14
LVI_SAM | 2.04 / 3.33 | 1.05 / 2.37 | 0.81 / 1.53 | 0.57 / 1.38 | 0.49 / 1.13 | 0.44 / 1.01
R2LIVE | 1.13 / 2.06 | 0.65 / 1.45 | 0.49 / 0.88 | 0.31 / 0.57 | 0.26 / 0.30 | 0.29 / 0.16
FAST-LIO2 | 1.31 / 2.22 | 0.69 / 1.34 | 0.49 / 0.89 | 0.31 / 0.59 |  0.26 / 0.31 |  0.25 / 0.29
VINS-Mono | 2.44 / 9.35 | 1.31 / 7.28 | 0.99 / 4.63 | 0.66 / 3.68 | 0.56 / 3.24 | 0.48 / 2.73
TABLE IV: The average time consumption on two different platforms with different configurations.  VIO per-frame cost time |  LIO per-frame cost time
---|---
Image size |  320$\times$256 |  640$\times$512 |  1280$\times$1024
Pt_res (m) | 0.10 | 0.05 | 0.01 | 0.10 | 0.05 | 0.01 | 0.10 | 0.05 | 0.01
On PC (ms) | 7.01 | 10.41 | 33.55 | 11.53 | 14.68 | 39.75 | 13.63 | 17.58 | 43.71 | 18.40
On OB (ms) | 15.00 | 26.45 | 84.22 | 26.05 | 42.32 | 123.62 | 33.38 | 55.10 | 166.08 | 39.56

###  VI-E Run time analysis

We investigate the average time consumption among of our system all
experiments on two different platforms: the desktop PC (with Intel i7-9700K
CPU and 32GB RAM) and a UAV on-board computer (“OB”, with Intel i7-8550u CPU
and 8GB RAM). The detailed statistics are listed in Table. IV. The time
consumption of our VIO subsystem is affected by two main settings: the image
resolution and the point cloud map resolution (“Pt_res”).

##  VII Applications with R3LIVE

###  VII-A Mesh reconstruction and texturing

Figure 13:  Figure (a) show the RGB-colored 3D points reconstructed by R3LIVE.
Figure (b) and (c) show the wireframe and surface of our reconstructed mesh.
Figure (d) show the mesh after texture rendering.

While R3LIVE reconstructs the colored 3D map in real-time, we also develop
software utilities to mesh and texture the reconstructed map offline (see Fig.
13). For meshing, we make use of the Delaunay triangulation and graph cuts
[28] implemented in CGAL[29]151515https://www.cgal.org/. After the mesh
construction, we texture the mesh with the vertex colors, with is rendered by
our VIO subsystem.

Our developed utilities also export the colored point map from R3LIVE or the
offline meshed map into commonly used file formats such as
“pcd”161616http://pointclouds.org/documentation/tutorials/pcd_file_format.html,
“ply”171717https://en.wikipedia.org/wiki/PLY_(file_format),
“obj”181818https://en.wikipedia.org/wiki/Wavefront_.obj_file and etc. As a
result, the maps reconstructed by R3LIVE can be imported by various of 3D
softwares, including but not limited to CloudCompare[30], Meshlab[31],
AutoDesk 3ds Max191919https://www.autodesk.com.hk/products/3ds-max, etc.

###  VII-B Toward various of 3D applications

With the developed software utilities, we can export the reconstructed 3D maps
to Unreal Engine 202020https://www.unrealengine.com to enable a series of 3D
applications. For example, in Fig.15, we built the car and drone simulator
with the AirSim[32], and in Fig.15, we use our reconstructed maps for
developing the video games for desktop PC and mobile platform. For more
details about our demos, we refer the readers to watch our video on
YoutuBe212121https://youtu.be/4rjrrLgL3nk.

Figure 14: We use the maps reconstructed by R3LIVE to build the car (in (a))
and drone (in (b)) simulator with AirSim. The images in green and blue
frameboxes are of the depth, RGB image query from the airsim’s API,
respectively.

Figure 15: We use the maps built by R3LIVE to develop the video games for
mobile platform (see (a)) and desktop PC (see (b)). In (a), the player is
controling the actor to explore the campus of HKU. In (b), the player is
fighting against the dragon with shoting the rubber balls in the campus of
HKUST.

## References

  * [1] J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, _et al._ , “Towards fully autonomous driving: Systems and algorithms,” in _2011 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2011, pp. 163–168.
  * [2] A. Bry, A. Bachrach, and N. Roy, “State estimation for aggressive flight in gps-denied environments using onboard sensing,” in _2012 IEEE International Conference on Robotics and Automation_. IEEE, 2012, pp. 1–8.
  * [3] F. Gao, W. Wu, W. Gao, and S. Shen, “Flying on point clouds: Online trajectory generation and autonomous navigation for quadrotors in cluttered environments,” _Journal of Field Robotics_ , vol. 36, no. 4, pp. 710–733, 2019.
  * [4] F. Kong, W. Xu, Y. Cai, and F. Zhang, “Avoiding dynamic small obstacles with onboard sensing and computation on aerial robots,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 7869–7876, 2021.
  * [5] Z. Liu, F. Zhang, and X. Hong, “Low-cost retina-like robotic lidars based on incommensurable scanning,” _IEEE Transactions on Mechatronics_ , 2021, in press.
  * [6] W. Xu and F. Zhang, “Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman filter,” _IEEE Robotics and Automation Letters_ , 2021, in press.
  * [7] J. Lin, X. Liu, and F. Zhang, “A decentralized framework for simultaneous calibration, localization and mapping with multiple lidars,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 4870–4877.
  * [8] Z. Liu and F. Zhang, “Balm: Bundle adjustment for lidar mapping,” _IEEE Robotics and Automation Letters_ , 2021, in press.
  * [9] X. Liu and F. Zhang, “Extrinsic calibration of multiple lidars of small fov in targetless environments,” _IEEE Robotics and Automation Letters_ , 2021, in press.
  * [10] C. Yuan, X. Liu, X. Hong, and F. Zhang, “Pixel-level extrinsic self calibration of high resolution lidar and camera in targetless environments,” _arXiv preprint arXiv:2103.01627_ , 2021.
  * [11] J. Lin and F. Zhang, “Loam livox: A fast, robust, high-precision lidar odometry and mapping package for lidars of small fov,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 3126–3131.
  * [12] J. Lin, C. Zheng, W. Xu, and F. Zhang, “R2live: A robust, real-time, lidar-inertial-visual tightly-coupled state estimator and mapping,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 7469–7476, 2021\.
  * [13] T. Shan, B. Englot, C. Ratti, and D. Rus, “Lvi-sam: Tightly-coupled lidar-visual-inertial odometry via smoothing and mapping,” _arXiv preprint arXiv:2104.10831_ , 2021.
  * [14] X. Zuo, P. Geneva, W. Lee, Y. Liu, and G. Huang, “Lic-fusion: Lidar-inertial-camera odometry,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 5848–5854.
  * [15] X. Zuo, Y. Yang, J. Lv, Y. Liu, G. Huang, and M. Pollefeys, “Lic-fusion 2.0: Lidar-inertial-camera odometry with sliding-window plane-feature tracking,” in _IROS 2020_ , 2020.
  * [16] W. Zhen and S. Scherer, “Estimating the localizability in tunnel-like environments using lidar and uwb,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 4903–4908.
  * [17] H. Zhou, Z. Yao, and M. Lu, “Uwb/lidar coordinate matching method with anti-degeneration capability,” _IEEE Sensors Journal_ , vol. 21, no. 3, pp. 3344–3352, 2020.
  * [18] C. Debeunne and D. Vivet, “A review of visual-lidar fusion based simultaneous localization and mapping,” _Sensors_ , vol. 20, no. 7, p. 2068, 2020.
  * [19] J. Zhang and S. Singh, “Laser–visual–inertial odometry and mapping with high robustness and low drift,” _Journal of Field Robotics_ , vol. 35, no. 8, pp. 1242–1264, 2018.
  * [20] W. Shao, S. Vijayarangan, C. Li, and G. Kantor, “Stereo visual inertial lidar simultaneous localization and mapping,” _arXiv preprint arXiv:1902.10741_ , 2019.
  * [21] W. Wang, J. Liu, C. Wang, B. Luo, and C. Zhang, “Dv-loam: Direct visual lidar odometry and mapping,” _Remote Sensing_ , vol. 13, no. 16, p. 3340, 2021\.
  * [22] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang, “Fast-lio2: Fast direct lidar-inertial odometry,” _arXiv preprint arXiv:2107.06829_ , 2021.
  * [23] T. Qin and S. Shen, “Online temporal calibration for monocular visual-inertial systems,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3662–3669.
  * [24] B. M. Bell and F. W. Cathey, “The iterated kalman filter update as a gauss-newton method,” _IEEE Transactions on Automatic Control_ , vol. 38, no. 2, pp. 294–297, 1993.
  * [25] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez, “Automatic generation and detection of highly reliable fiducial markers under occlusion,” _Pattern Recognition_ , vol. 47, no. 6, pp. 2280–2292, 2014.
  * [26] T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” _IEEE Transactions on Robotics_ , vol. 34, no. 4, pp. 1004–1020, 2018.
  * [27] Z. Zhang and D. Scaramuzza, “A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 7244–7251.
  * [28] P. Labatut, J.-P. Pons, and R. Keriven, “Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts,” in _2007 IEEE 11th international conference on computer vision_. IEEE, 2007, pp. 1–8.
  * [29] A. Fabri and S. Pion, “Cgal: The computational geometry algorithms library,” in _Proceedings of the 17th ACM SIGSPATIAL international conference on advances in geographic information systems_ , 2009, pp. 538–539.
  * [30] D. Girardeau-Montaut, “Cloudcompare,” _France: EDF R &D Telecom ParisTech_, 2016.
  * [31] P. Cignoni, G. Ranzuglia, M. Callieri, M. Corsini, F. Ganovelli, N. Pietroni, and M. Tarini, “Meshlab,” 2011.
  * [32] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in _Field and service robotics_. Springer, 2018, pp. 621–635.


# VIVE3D: Viewpoint-Independent Video Editing using 3D-Aware GANs

Anna Frühstück2*, Nikolaos Sarafianos1, Yuanlu Xu1, Peter Wonka2, Tony Tung1
1 Meta Reality Labs Research, Sausalito 2 KAUST
afruehstueck.github.io/vive3D

###### Abstract

We introduce VIVE3D, a novel approach that extends the capabilities of image-
based 3D GANs to video editing and is able to represent the input video in an
identity-preserving and temporally consistent way. We propose two new building
blocks. First, we introduce a novel GAN inversion technique specifically
tailored to 3D GANs by jointly embedding multiple frames and optimizing for
the camera parameters. Second, besides traditional semantic face edits (_e.g_.
for age and expression), we are the first to demonstrate edits that show novel
views of the head enabled by the inherent properties of 3D GANs and our
optical flow-guided compositing technique to combine the head with the
background video. Our experiments demonstrate that VIVE3D generates high-
fidelity face edits at consistent quality from a range of camera viewpoints
which are composited with the original video in a temporally and spatially
consistent manner.

††*This work was conducted during an internship at Meta RL Research.

Figure 1: We propose VIVE3D, a novel method that creates a powerful
personalized 3D-aware generator using a low number of selected images of a
target person. Given a new video of that person, we can faithfully modify
several facial attributes as well as the camera viewpoint of the head crop.
Finally, we seamlessly composite the edited face with the source frame in a
temporally and spatially consistent manner, while retaining a plausible
composition with the static components of the frame outside of the generator’s
region. The dotted squares in the center frame denote the reference regions
for the three different camera poses in the column below.

##  1 Introduction

Semantic image editing has been an active research topic for the past few
years. Previous work [21] uses Generative Adversarial Networks (GANs) to
produce high-fidelity results in the image space. The most popular backbone is
StyleGAN [29, 28, 26, 27] as it generates high-resolution domain-specific
images while providing a disentangled latent space that can be utilized for
editing operations. To edit real photographs, there are typically two steps:
The first step maps the input image to the latent space of a pre-trained
generator. This is usually accomplished either through encoder-based embedding
or through optimization, such that generator can accurately reconstruct the
image from the latent code [53]. The second step is semantic image
manipulation, where one latent input representation is mapped to another to
obtain a certain attribute edit, (_e.g_. changing age, facial expression,
glasses, or hairstyle). While existing approaches produce impressive results
on single images, extending them to videos is far from straightforward. Among
the challenges that arise are: (1) people tend to move their heads freely in
videos (instead of assuming frontal image inputs), (2) the inversion of
multiple frames should be coordinated, (3) the inverted face and edits need to
be temporally consistent and (4) the compositing of the edited face with the
original frame must maintain boundary consistency.

A recent set of approaches has focused on 3D-aware GANs where a 2D face
generator is combined with a neural renderer. Given a latent code, a 2D image
and the underlying 3D geometry are generated, thus allowing for some camera
movement while rendering the head of the person.

In this paper, we tackle the problem of viewpoint-independent face editing in
videos. The edited face is rendered from novel views in a temporally-
consistent manner. Specifically, we use a 3D-aware GAN in the temporal domain
and apply facial image editing techniques per frame that are temporally smooth
regardless of the rendered view. Compared with other GAN-based video editing
approaches [48, 5], our method is the first to perform viewpoint-independent
video editing while showing the full upper body of the person in the video
with high fidelity.

VIVE3D takes a video of a person captured from a monocular camera as input.
The captured person can move freely across time, talk, and make facial
expressions while their body can be visible. Unlike all prior work that learns
a generator and performs edits on the exact same video, we disentangle these
steps. Hence the output of our approach can be a different video of the same
person or the same video. In both cases, the face has undergone one or more
attribute edits and is rendered from a novel view. To accomplish this
challenging task, we introduce several novel components, each addressing one
challenge of the problem at hand. Specifically, we first propose a simple yet
effective technique to create a personalized generator by inverting multiple
frames at the same time. The simultaneous inversion of $N$ frames exposes the
generator to a variety of facial poses and expressions, which results in a
larger capacity that we can then utilize. Our generator can generalize to new
unseen videos of the same identity where the person might be wearing a
different shirt, a result not demonstrated in the literature so far. In
addition, we propose to optimize the camera pose of the 3D-aware GAN during
inversion to obtain an accurate estimate which angle the face was captured
from. Finally, we introduce an optical flow-based compositing method to
properly place the novel view of the edited face back into the original frame
while ensuring that the end result is temporally and spatially consistent. Our
experimental work provides a wide range of qualitative and quantitative
results to demonstrate that VIVE3D accomplishes semantic video editing with
changing camera poses in a faithful way. In summary, our contributions are:

  * •

A new 3D GAN inversion technique that jointly embeds multiple images while
optimizing for their camera poses.

  * •

A complete attribute editing framework and an optical flow-based compositing
technique to replace the edited face in the original video.

  * •

VIVE3D is the first 3D GAN-based video editing method and the first that can
change the camera pose of the face.

Figure 2: VIVE3D Pipeline. To create an edited video, we first need to create
a personalized generator by jointly inverting selected faces and fine-tuning a
pre-trained generator. We then invert the cropped face regions from a source
video (which could be the same or a different video) into our personalized
generator and recover the latent codes and camera poses for each target frame.
We are able to perform semantic editing on the inverted stack of latent codes
using previously discovered latent space directions and we can freely change
the camera path around the face region. In order to composite the face with
the source frame in a consistent fashion, we use optical flow to correct the
position of the inset within the frame, which allows us to composite the
result in a seamless and temporally consistent fashion.

##  2 Related Work

GAN Inversion. GANs are a powerful tool for semantic editing. Most editing
techniques are tailored to StyleGAN, the state-of-the-art of 2D GANs [28, 29,
26, 27]. Several editing techniques [19, 18, 25, 10, 34] build upon StyleGAN
as it uses an intermediate disentangled latent space, usually referred to as
$\mathbf{w}$-space. Before editing, a latent space representation of the input
image has to be recovered using a process typically referred to as Inversion
or Projection [12, 1, 2, 50]. Refer to [52] for a survey of inversion
techniques. In contrast to optimization-based inversion techniques, learning-
based approaches attempt to obtain faster latent space correspondences by
training encoders [36, 47, 4]. In order to retain the generalization ability
of the $\mathbf{w}$-space while providing a high-quality inversion, Pivotal
Tuning [37] has successfully shown that trained generators can overfit to
target images while still maintaining a navigable latent space. Recent works
study 3D GAN inversion [31, 30], attempting to infer a 3D representation for a
reference image.

GAN-based Latent Space Editing. Once an appropriate latent space
representation of an input image has been recovered, semantic edits can be
applied by navigating the latent space manifold surrounding the inverted
latent code. Unsupervised techniques attempt to find interesting edits without
labeled data [24, 22, 41, 49]. InterfaceGAN [39, 40] is a simple and robust
supervised technique that is highly recommended for practical applications,
and as such we also employ it in our work. While there is a plethora of other
techniques [59, 55, 11, 3, 45, 44] the development of related latent space
manipulations itself is not the focus of our work. Another line of work is
text-based editing which gained immense popularity during the last year [20,
35].

3D-aware GANs. Recent GAN papers attempt to discover 3D information from large
collections of 2D images using Neural Radiance Fields (NeRFs) as shape
representations [9, 33, 14, 23, 58, 38, 8]. While most of these papers share
similar architectural ideas, EG3D [9] has emerged as a popular basis for
follow-up work (_e.g_. integration of a segmentation branch [43]). We chose to
build upon EG3D, but our work is also applicable to other generators with a
similar latent space. For more information on 3D GAN architectures, we refer
the interested reader to a recent survey [51].

Video Synthesis and Editing. One branch of work attempts to leverage 2D GANs
to generate video sequences [17, 46, 57, 42]. These ideas can be extended to
create 3D videos [6], which also rely on 3D NeRFs.

GAN-based Video Editing. GAN-based video editing is the core topic of this
paper. Duong _et al_. [15] employ deep reinforcement learning for automatic
face aging. LatentTransformer [55] encodes frames into the StyleGAN latent
space using an encoder. They train a transformer to do attribute editing on
single frames and blend the result with Poisson blending. The main competitor
to our work is Stitch it in Time (StiiT) [48], which crops the faces from a
video, edits them with 2D GAN techniques, and merges the edited result back to
the video with some blending. However, StiiT does not learn a 3D model of the
human head, overfits to a particular video, and is unable to provide edits to
the viewpoint of the human head. Recently, VideoEditGAN (VEG) [54] attempted
to improve the temporal consistency of StiiT by running a two-step
optimization approach focused on localized temporal coherence. Alaluf _et al_.
[5] use StyleGAN3 for video editing, to leverage its inherent alignment
capabilities and reduce texture sticking artifacts. Since this is an active
area of research, all these techniques are concurrent work to our method, yet
we do provide comparisons to showcase the benefits of our proposed approach.

##  3 Method

In this section, we introduce the key components of VIVE3D to perform frame-
by-frame video editing while allowing for rendering the edited face from new
views. We leverage a 3D-aware generator that infers 3D geometry and camera
positions while being trained solely on 2D images. We build a personalized
3D-aware generator by performing joint inversion on multiple frames and then
use it to perform attribute editing, apply camera viewpoint changes, and
finally composite the edited face rendered from a new view back into the
original frame. An overview of our proposed approach is depicted in Fig. 2
while the personalized generator architecture is shown in Fig. 3.

Figure 3: Personalized Generator. First, we run a joint inversion on $N$
selected target faces, where we optimize a shared target person latent
$\mathbf{w}_{\text{{ID}}}$ and an offset $\mathbf{o}_{\text{{n}}}$ for each
face. This ensures the inversions share information about the target.
Simultaneously, we jointly optimize for the camera pose $c_{\text{{n}}}$. We
then fine-tune the generator to ensure it captures the fine details of the
target identity. Note that the “default” latent (left column) implicitly
captures the identity of the target person without being explicitly optimized.

###  3.1 Personalized 3D-Aware Generator

Face Selection and Cropping. To create a personalized 3D-aware GAN model, we
start by processing a short range from the input video where $N$ frames are
selected such that they cover a range of orientations and facial expressions
of the target person. We detect the facial keypoints within these frames using
an off-the-shelf facial keypoint detector [7] and use them to determine the
face bounding box within the frame. This is achieved by calculating a rigid
transformation from the facial keypoints in the frame to the facial keypoints
in a generated example image, thereby aligning the keypoints at the center of
the crop in the same way as the generator’s original training data. We pick a
specific field of view for cropping the faces and optimizing the generator,
but the field of view remains a flexible parameter that can be adapted during
any later stage in the pipeline.

Simultaneous Inversion. We propose to perform multiple inversions
simultaneously. EG3D has two major components in its generator. The first
component uses a mapping network to map random vectors into a semantically
meaningful space, called $\mathbf{w}$-space. Vectors in this space control a
3D neural field that defines a 3D representation that is rendered using
volumetric rendering. The second component is a 2D upsampler that performs a
$4\times$ super-resolution on the original output. We invert all selected
faces simultaneously into the $\mathbf{w}$-space following a strategy similar
to [37] that we discuss in detail below.

In order to find a representation in $\mathbf{w}$-space, we define a “global”
$\mathbf{w}_{\text{{ID}}}$ aiming at capturing the global identity features of
the target person, and a “local” offset vector $\mathbf{o}_{\text{{n}}}$ for
each input expression $\mathbf{F}_{\text{{n}}}$ that encodes the differences
of each individual facial expression and position from the default
$\mathbf{w}_{\text{{ID}}}$. The length of each $\mathbf{o}_{\text{{n}}}$ is
regularized using an $\mathcal{L}_{\text{{L2}}}$ loss, aiming to keep the
difference as small as possible, and capturing all similarities between the
input images within the default person latent $\mathbf{w}_{\text{{ID}}}$. We
use a combination of a perceptual loss $\mathcal{L}_{\text{{LPIPS}}}$ and a
pixel loss $\mathcal{L}_{\text{{L1}}}$ for the inversion. Note that during
this stage, we calculate these losses on the raw output
$\mathcal{G}_{\text{{raw}}}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{n}}})$
of the EG3D neural renderer at 128$\times$128 resolution because we observed
that it yields sharper result quality rather than evaluating the loss at the
output of the super-resolution network. We downsample our target images to the
same resolution $\mathbf{D}_{\text{{128}}}({\mathbf{F}_{\text{{n}}}})$ to
compare. To ensure that we can faithfully capture the target person’s identity
and expression, we use BiSeNet [56] to obtain a segmentation
$\mathbf{S}_{\text{{exp}}}({\mathbf{F}_{\text{{n}}}})$ of the facial regions
encoding the expression (eyes, mouth, eyebrows, and nose) and add an
additional feature loss on this area to encourage consistent facial
expressions (_e.g_. closed eyes). To obtain the inversion, in each
optimization step, we sum up the losses for each face image
$\mathbf{F}_{\text{{n}}}$, therefore jointly optimizing all targets
simultaneously, yielding a total loss $\mathcal{L}_{\text{{inv}}}$.

$\displaystyle\textstyle\mathcal{L}_{\text{{inv}}}=\sum\nolimits_{\text{{n}}=0}^{N}\lambda_{\text{{LPIPS}}}\,$
$\displaystyle\mathcal{L}_{\text{{LPIPS}}}(\mathcal{G}_{\text{{raw}}}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{n}}}),\mathbf{D}_{\text{{128}}}({\mathbf{F}_{\text{{n}}}}))+$
$\displaystyle\lambda_{\text{{L1}}}\,$
$\displaystyle\mathcal{L}_{\text{{L1}}}(\mathcal{G}_{\text{{raw}}}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{n}}}),\mathbf{D}_{\text{{128}}}({\mathbf{F}_{\text{{n}}}}))+$
$\displaystyle\lambda_{\text{{seg}}}\,$
$\displaystyle\mathcal{L}_{\text{{LPIPS}}}(\mathbf{S}_{\text{{exp}}}({\mathcal{G}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{n}}})}),\mathbf{S}_{\text{{exp}}}({\mathbf{F}_{\text{{n}}}}))+$
$\displaystyle\lambda_{\text{{reg}}}\,$
$\displaystyle\mathcal{L}_{\text{{L2}}}(\mathbf{o}_{\text{{n}}})$

Due to the 3D awareness of the EG3D generator, the quality of the inversion
into the latent space is highly sensitive to the camera parameter settings.
Hence, in addition to optimizing for $\mathbf{w}_{\text{{ID}}}$ and
$\mathbf{o}_{\text{{n}}}$, we propose to also allow the inversion to optimize
for the camera parameters $c_{\text{{n}}}$ ($\text{{yaw}}_{\text{{n}}}$ and
$\text{{pitch}}_{\text{{n}}}$) for each input expression
$\mathbf{F}_{\text{{n}}}$, which reliably estimates the camera position that
the face is captured from.

A key advantage of this joint optimization is that the facial characteristics
of the person preserve their high fidelity even when seen from novel views.
When inverting a single image of a side-facing person into the EG3D latent
space, exploring other viewpoints of the inverted latent can lead to
significant distortions. Often, unseen features (_e.g_. hidden ears) can be
blurry or distorted, and the identity no longer resembles the input from a
different viewpoint. The joint inversion, however, ensures that the different
views are embedded closely enough in latent space such that even unseen views
yield consistently identity-preserving outputs.

Generator Fine-tuning. We propose a variant of Pivotal Tuning [37] to jointly
fine-tune the weights of the generator $\mathcal{G}_{\text{{EG3D}}}$ on all
input faces $\mathbf{F}_{\text{{n}}}$, while keeping the detected
$\mathbf{w}_{\text{{ID}}}$, $\mathbf{o}_{\text{{n}}}$ and camera poses
$c_{\text{{n}}}$ fixed. Here, we do not allow the weights of the upsampler of
the generator to be updated as we want to preserve the generalization
capabilities of the super-resolution network and prevent it from overfitting
to our target images. During this fine-tuning stage, we employ perceptual and
pixel losses described as follows:

$\displaystyle\textstyle\mathcal{L}_{\text{{tune}}}=\sum\nolimits_{n=0}^{N}\lambda_{\text{{LPIPS}}}\,$
$\displaystyle\mathcal{L}_{\text{{LPIPS}}}(\mathcal{G}_{\text{{ID}}}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{n}}}),\mathbf{F}_{\text{{n}}})+$
$\displaystyle\lambda_{\text{{L1}}}\,$
$\displaystyle\mathcal{L}_{\text{{L1}}}(\mathcal{G}_{\text{{ID}}}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{n}}}),\mathbf{F}_{\text{{n}}})$

Finally, we obtain a personalized EG3D generator $\mathcal{G}_{\text{{ID}}}$,
fine-tuned to a set of facial expressions of the target person. We verify that
the fine-tuned generator indeed provides a good generalized latent space for
the target person even though it was inverted and tuned based on a low number
of frames by exploring the person created by the “global” latent code, which
was not explicitly fine-tuned for, as well as through a latent space walk in
the fine-tuned latent space.

###  3.2 Frame-by-frame Video Inversion

With the personalized 3D-aware generator in hand, we are now given a video of
the same person as input which can be different from the one the generator was
trained on. To process our new target video, we extract the facial keypoints
from each frame f to determine the location of the box to indicate the face
crop within the frame. In order to stabilize the crop over time, which
supports the temporal coherence of the inversion, we perform a Gaussian
smoothing on the extracted facial keypoints along the temporal axis after
extraction. However, it is important to not over-smooth, because fast motions
in the video would yield distorted keypoint locations, deteriorating the
inversion quality.

We then perform a frame-by-frame inversion of the extracted face regions
$\mathbf{F}_{\text{{f}}}$ into the space of the fine-tuned generator
$\mathcal{G}_{\text{{ID}}}$. Like before, we optimize for an offset
$\mathbf{o}_{\text{{f}}}$ for each input frame $\mathbf{F}_{\text{{f}}}$, as
well as regularizing the offset length. After inverting the first frame, each
consecutive frame $\mathbf{F}_{\text{{f}}+1}$ is inverted starting from the
previous inversion and only needs a low number ($\scriptstyle\mathtt{\sim}$50)
of optimization steps.

$\displaystyle\textstyle\mathcal{L}_{\text{{vid}}}=\lambda_{\text{{LPIPS}}}\,$
$\displaystyle\mathcal{L}_{\text{{LPIPS}}}(\mathcal{G}_{\text{{ID}}}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{f}}}),\mathbf{F}_{\text{{f}}})+$
$\displaystyle\lambda_{\text{{L1}}}\,$
$\displaystyle\mathcal{L}_{\text{{L1}}}(\mathcal{G}_{\text{{ID}}}(\mathbf{w}_{\text{{ID}}}+\mathbf{o}_{\text{{f}}}),\mathbf{F}_{\text{{f}}})+\lambda_{\text{{reg}}}\,\mathcal{L}_{\text{{L2}}}(\mathbf{o}_{\text{{f}}})$

This inversion yields a stack of offsets $\mathbf{o}_{\text{{f}}}$ from the
identity latent $\mathbf{w}_{\text{{ID}}}$ as well as camera parameters
$c_{\text{{f}}}$ ($\text{{yaw}}_{\text{{f}}}$, $\text{{pitch}}_{\text{{f}}}$)
encoding the expression and camera position for each frame.

Figure 4: InterfaceGAN edits. We show InterfaceGAN editing directions
discovered in the latent space by applying them on our personalized generator.
The attribute edits are consistent in 3D.

###  3.3 Attribute Editing and Novel View Synthesis

Since EG3D is built on top of StyleGAN2, we can leverage existing latent space
editing techniques in order to discover semantic editing directions in the
latent space of EG3D. As a proof of concept, we implemented InterfaceGAN [40]
to find meaningful latent space direction vectors. We re-trained classifiers
on the CelebA dataset for several facial attributes such as age, smile,
gender, glasses, beard, and hair color and use these classifiers to classify
the reference outputs of a set of randomized latent codes from our generator.
Finally, we used an SVM to recover editing boundaries from these classified
latent codes, which allows us to perform attribute editing in the EG3D latent
space, as shown in Fig. 4. For a given latent space direction
$\mathbf{w}_{\text{{dir}}}$, we apply an edit as a linear combination of the
person latent $\mathbf{w}_{\text{{ID}}}$ with the direction, multiplied by an
empirically chosen weight $\alpha_{\text{{dir}}}$, which can be positive or
negative. For our video sequence, we use the edited person latent
${\mathbf{w}_{\text{{ID}}}}^{\prime}$ as the new identity to which we apply
our video offsets $\mathbf{o}_{\text{{f}}}$.

$\displaystyle{\mathbf{w}_{\text{{ID}}}}^{\prime}=\mathbf{w}_{\text{{ID}}}+\alpha_{\text{{dir}}}\times\mathbf{w}_{\text{{dir}}}$

In addition, we explore our temporal stack of latent encodings from novel
views, diverging from the input views the inversion discovered. This allows
us, to generate frontalized videos of the person by fixing the camera position
or to define arbitrary camera trajectories around the subject.

Figure 5: Border Composition. We calculate the composition border based on
face segmentations of the target image (a) and the edited inset (c). We unite
the masks and dilate the resulting joint mask to obtain a boundary around the
face regions (d) that should be optimized, which allows us to create the final
composition (e).

###  3.4 Compositing with Source Video

After editing the inverted video, we want to recomposite the edited faces back
with the source frames such that the edited video is temporally and spatially
consistent. We accomplish this by running an optimization to ensure that the
boundaries between the compositing of the edited face and the background of
the source frame are smooth. Since cluttered video backgrounds are hard to
reproduce consistently and without artifacts – especially for novel views – we
define a compositing boundary region in a similar manner to [48]. To
accomplish this, we need an accurate segmentation of the face and hair for
both the original frame as well as the edited face. Hence, we use BiSeNet
[56], a semantic segmentation technique, that accurately provides such face
semantics. We use the semantic regions for both the original and edited face
to form a union of their respective masks, obtaining a boundary region around
the relevant face region, as illustrated in Fig. 5. We run a small number of
optimization steps, optimizing for the boundary region of the edited image to
appear as close as possible to the boundary region of the input frame while
retaining the appearance of the edit. Finally, we use an affine transformation
to re-insert the cropped face region back into the original frame and we alpha
blend along the optimized boundary region to seamlessly composite the edit
with the source frame.

####  3.4.1 Flow-based View Adjustment

During the process of re-inserting the edited face
$\mathbf{F}_{\text{{edit}}}$ back into the source frame $S$, a major challenge
arises when the camera pose has been modified and the face is rendered from a
novel view. This is because the face is oriented within the bounding box based
on the facial keypoints as described in Sec. 3.1 while upon a camera pose
change, the face pivots around the keypoints. When, for instance, attempting
to replace a face viewed at an angle with a frontalized face while retaining
the original crop boundary of the inset, the keypoints are still roughly in
the same location, yet the mass of the head, and crucially, the neck is
shifted according to the face rotation, as seen in Fig. 6, which results in
the inset being disconnected from the rest of the body even when using a
boundary stitching technique.

Figure 6: View adjustment. After cropping (a) and inverting (b) a face, we
perform face editing (c) and change the camera viewpoint to an unseen angle
(d). When replacing the face in the original frame with this edit, it yields
poor quality (bottom center) even for small angular changes, because the
rotated face is in the wrong location with respect to the body. We address
this by estimating the optical flow (e) between the face crop and the edit and
use the flow direction to correct the location of the reference face based on
the prospective inset (f). This allows us to composite the edited face into
the frame in a natural-looking fashion (bottom right).

To alleviate this problem, we introduce a simple yet effective technique to
reposition the reference face region within the source frame. We discover the
optimal position of the updated inset with respect to the source frame by
estimating the optical flow between the face segmentation in the source frame
$S$ and the face segmentation in the inset region $\mathbf{F}_{\text{{edit}}}$
after camera rotation. We convert the images to grayscale and use Farnebäck
optical flow [16] to evaluate a dense flow field of the displacement between
the edited and target faces. The optical flow is defined as a 2D displacement
vector field $\mathbf{d}$ with the displacement vector at image position
$(x,y)$ given by $\mathbf{d}(x,y)=(u(x,y),v(x,y))$ where the correspondence
between the two images $\mathbf{F}_{\text{{f}}}$ and
$\mathbf{F}_{\text{{edit}}}$ is:

$\displaystyle\mathbf{F}_{\text{{edit}}}(x+u(x,y),y+v(x,y))=\mathbf{F}_{\text{{f}}}(x,y).$

We then compute vector magnitudes $\|\mathbf{d}\|=\sqrt{u^{2}+v^{2}}$ and
directions $\phi=\text{{atan2}}{(v,u)}$, respectively. After eliminating all
vectors with a magnitude smaller than a threshold $\epsilon$, we create a
histogram of all remaining directions. We define a dominant displacement
vector $\mathbf{d}_{\text{{dom}}}$ from the median direction of the histogram
bin with the largest count and the maximum vector length within that histogram
bin. This ensures that erroneous flow directions from features that are
present within one of the two images but not the other are not contributing to
the final output.

The displacement vector $\mathbf{d}_{\text{{dom}}}$ is reprojected from inset
space into frame space and is used to correct the location of the reference
face crop. To ensure a smooth transition between adjacent frames, we perform
temporal Gaussian smoothing of the recovered displacement vectors. We then
apply our inset optimization using the updated reference areas and obtain a
significantly more faithful result, allowing for large camera changes with
natural-looking results.

Table 1: Video Quality Metrics. We compare the quality of our inversion with
StiiT using reconstruction metrics on a subset of the VoxCeleb dataset. We
also evaluate the Fréchet Inception Distance (FID) of inversion and edits with
respect to the source video.

| Reconstruction Quality | Editing Quality
---|---|---
| PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$
Method | Inversion | Inversion | Age Edit | Angle Edit
StiiT | 38.0134 | 0.9798 | 6.8329 | 15.8021 | 19.0371
VIVE3D | 38.1259 | 0.9704 | 4.9852 |  9.3410 |  8.9953

##  4 Experiments

We conduct a wide range of quantitative and qualitative comparisons to
demonstrate the key contributions of our work along with ablation studies
against baselines and simplified variants where proposed modules are removed.
We showcase that VIVE3D is on par with StiiT in terms of reconstruction
quality while it also greatly outperforms prior work in editing quality and
identity preservation. However, a key novelty afforded by VIVE3D is the
ability to render the edited face from novel viewpoints within the existing
frame, a task for which comparisons are hard due to the absence of ground
truth. We showcase this with qualitative results and videos provided in the
supplementary material.

###  4.1 Quantitative Evaluation

We establish comparisons with StiiT [48], the key competitor to our work in
the field of GAN video editing, and with VEG [54] for which many components
(_e.g_. their stitching technique) are identical to StiiT, so we only compare
facial similarity metrics. Please see the supplementary material for a
detailed description of the comparison setup between our method and the
competitors.

Inversion Quality. We provide a quantitative comparison of our inversion
quality by measuring the reconstruction quality with respect to the input
video in Table 1. We evaluate PSNR and SSIM for our method and for StiiT on a
set of 16 videos from the VoxCeleb [32] dataset, inverting the face region and
recompositing it with the source video without edits. Both methods perform
well on the reconstruction of the input signal and the final reconstruction
quality of our technique is on par with StiiT.

Table 2: Face Similarity Metrics. We evaluate the identity preservation of
inversion and edits based on the cosine similarity of ArcFace features
extracted from generated face crops with respect to the face crops of the
source video. To evaluate coherence over time, we measure the dissimilarity
between consecutive frames.

| Method | Similarity to Source $\uparrow$ | Temporal Difference $\downarrow$
---|---|---|---
Inversion | e4e | 0.6923 | 6.2851
StiiT | 0.9261 | 1.2361
VIVE3D | 0.9203 | 1.0444
Age Editing | StiiT | 0.7891 | 1.4126
VEG | 0.6004 | 1.7159
VIVE3D | 0.8381 | 1.2257
Angle Editing | StiiT | 0.6695 | 1.3102
VEG | 0.4955 | 1.4502
VIVE3D | 0.8694 | 1.2761

Image Quality. To evaluate the image quality produced by the respective
techniques, we compute the Fréchet Inception Distance (FID), which is a
commonly used quality metric for GANs. To obtain the FID for each video, we
compare the set of all frames of the inverted video, as well as selected
edits, with all frames of the source video. Our method is able to score very
good FID scores overall (see Table 1 right), confirming the quality of our
results.

Face Fidelity. We calculate the fidelity of our inversion and edits based on a
facial similarity metric, ArcFace [13], which extracts a 512-D feature vector
capturing facial characteristics from a face region. We compute the metrics
based on the respective face crops of the final inset region, scaled to
512$\times$512px, for our method, e4e encoding, StiiT, and VEG in Table 2 and
average across a set of 16 videos from the VoxCeleb [32] dataset. The facial
similarity is evaluated both with respect to the input video (frame-by-frame)
as well as temporally by calculating the dissimilarity of adjacent frames in
order to survey the temporal coherence of facial characteristics. In all
cases, we use the cosine similarity between the extracted ArcFace deep
features. We observe that the quality of the inversion is good for both StiiT
and VIVE3D, both significantly improving upon e4e encoding. VEG uses the same
PTI-based inversion as StiiT and is therefore not listed. For latent space
editing, our proposed VIVE3D leads the competition. VEG exhibits good temporal
coherence, however, the edits contain artifacts, resulting in a deterioration
in the similarity to the source video. Additionally, our technique
reconstructs the facial identity faithfully even for angle edits, a task in
which StiiT and VEG fail to produce plausible results due to their methods’
inability to accommodate changes in the head rotation.

Table 3: Timings and Memory Requirements. We provide runtimes and Memory requirements for ours and competing methods. Method | Total | Precompute | Main | Postprocess | GPU
---|---|---|---|---|---
StiiT |  58m 53s |  28m 21s | 30m 19s | — | 22GB
VEG | 159m 54s | 107m 9s | 34m 41s | 18m 3s | 19GB
VIVE3D |  35m 43s |  6m 58s | 14m 54s | 13m 51s | 21GB

Resource Usage. We compare runtimes and memory requirements for the default
pipeline of related methods and our method, respectively, in Table 3. We split
each method, wherever applicable, into precomputation, main method, and
postprocessing steps and used the hyperparameter settings according to the
authors’ suggestions. Note that the precomputation step in our method has to
only be run once per identity and can then be applied to multiple videos. All
experiments are performed on an example video consisting of 200 frames at a
resolution of 1920$\times$1080px, using a single NVIDIA A100 GPU with 40GB
memory.

Ablation Study. We quantitatively verify the effectiveness of different
architecture choices we made, as shown in Table 4. We compare several metrics
with respect to the source video for our default implementation, and the
ablation experiments, respectively. We run four experiments on a set of 5
videos: (1) VIVE3D without generator fine-tuning, (2) VIVE3D without flow-
based adjustment, (3) VIVE3D without the joint $\mathbf{w}_{\text{{ID}}}$
latent and offset regularization, (4) using only a single input face for the
generator inversion and fine-tuning, which in practice is almost identical to
only frame-by-frame inversion in EG3D without any personalized generator. We
demonstrate that all experiments deteriorate the facial fidelity as well as
the reconstruction quality.

Table 4: Ablation Study. We demonstrate the effect of removing various
components of our pipeline on several quality and reconstruction metrics. We
measure the face similarity using the cosine similarity of ArcFace features of
the generated face crop, and the reconstruction metrics at the target video
resolution.

Ablation Type | Face Similarity $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---
VIVE3D, full | 0.9101 | 35.5367 | 0.9763
no generator fine-tuning | 0.7191 | 33.1202 | 0.9616
no flow correction | 0.8198 | 24.8845 | 0.9350
no regularization | 0.8007 | 25.6537 | 0.9162
single input for inversion | 0.7382 | 25.1950 | 0.9137

Figure 7: Comparisons with StiiT on attribute editing. We show an example of
our method and StiiT editing the subject’s age in the video frame (center
column). Both methods yield plausible but distinctly different results. Our
results (columns (a), age=–1.4 and (c), age=+2.3) vs StiiT (columns (b),
age=–8 and (d), age=+12).

###  4.2 Qualitative Evaluation

Semantic Edits. First, we demonstrate that the quality of semantic edits using
VIVE3D, adapting well-established latent space editing techniques for 3D GANs,
is on par with the editing quality of StiiT, as shown in Fig. 7. Note that
while both approaches discover latent space directions using InterfaceGAN
[40], the latent spaces and discovered directions are dissimilar, yet both
plausible.

Figure 8: Changing camera poses. Our method can freely alter the camera pose
and composite the result back with the source frame by fixing the divergence
between the source and target pose using our optical flow correction strategy.
The generated results look natural despite the static body pose.

Synthesizing novel views. We show that VIVE3D can accommodate changes in the
camera view with natural-looking results for a wide range of views regardless
of the input face orientation, as shown in Fig. 8. This is nontrivial as we
need to ensure that the person’s identity is consistent from multiple
viewpoints while the body in the source frame also defines a rigid constraint
to which the head alignment must be adjusted to. While StiiT produces good
results for attribute edits, it cannot composite images with angular changes,
despite the fact that a limited head pose change can be achieved by applying
latent space manipulations. In Fig. 9, we show a comparison where StiiT is
unable to generate a reasonable composition, whereas we can achieve natural-
looking results for a similar head rotation.

Figure 9: Comparisons with StiiT on viewpoint changes. VIVE3D ((a) and (c))
produces plausible results for both positive and negative yaw changes, whereas
StiiT ((b) and (d)) is unable to create natural compositions of the edit with
the target frame.

Compositing with challenging boundaries. When parts of the head or hair are
visible both inside and outside the face bounding box then compositing the
edited frame back into the original input is a challenging task. In most
cases, we address these scenarios by adjusting our camera parameters (_e.g_.,
choosing a wider field of view for our generator) but some configurations can
still be challenging, especially when different textures need to be matched.
We show in Fig. 10 that our technique attempts to produce plausible inset
optimizations even for such instances.

Figure 10: Spatial Consistency. VIVE3D composites images with challenging
boundaries such as long hair (right), yielding faithful hair color change
results. For hard boundary cases, such as matching with a static piece of hair
outside the boundary crop (left), it plausibly connects the contents of the
two images.

Limitations. Changing the camera parameters for the head in videos with fast
motion or discontinuities results in artifacts because the flow estimation
becomes unstable. Furthermore, we inherit the shortcomings of EG3D (see Fig.
11): stronger entanglement of attribute edits compared to StyleGAN2, extreme
camera poses are not captured in the training set, and texture sticking.
Finally, since the video outside the face region is immovable, the range of
possible changes is constricted to plausible compositions.

Figure 11: Limitations. VIVE3D inherits some limitations from the frameworks
we rely on. EG3D cannot capture extreme poses well (left). Large angle changes
cannot be composited naturally with the static body in the source frame. For
extreme edits (gender edits that change the hair structure (right)), it is
difficult to yield temporally consistent results, both due to the entanglement
of the latent space editing and the challenges of frame compositing.

##  5 Conclusion

In this paper, we introduced VIVE3D, a novel framework that uses prior
information encoded in 3D GANs for video editing. Our edits are identity-
preserving and temporally consistent. While we enable standard semantic edits,
such as age, or expressions, a distinguishing feature of our work is that we
facilitate edits that alter the view of the head. This capability is not
available in any 2D GAN-based prior work. The key building blocks of our work
are a new embedding algorithm that jointly embeds multiple frames and
optimizes for camera pose as well as flow-guided video compositing. In future
work, we aim to extend our framework to include a 3D GAN for head details and
another 3D GAN for the body. We also plan to investigate performance speedups
by replacing various optimizations with encoders.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2StyleGAN: How to embed images into the StyleGAN latent space?  In ICCV, 2019.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2StyleGAN++: How to edit the embedded images?  In CVPR, 2020.
  * [3] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka.  StyleFlow: Attribute-conditioned exploration of StyleGAN-generated images using conditional continuous normalizing flows.  ToG, 2021.
  * [4] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement.  In CVPR, 2021.
  * [5] Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski, and Daniel Cohen-Or.  Third time’s the charm? Image and video editing with StyleGAN3.  In ECCV Workshops, 2022.
  * [6] Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang, Gordon Wetzstein, Leonidas Guibas, Luc Van Gool, and Radu Timofte.  3D-aware video generation.  arXiv preprint arXiv:2206.14797, 2022.
  * [7] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks).  In ICCV, 2017.
  * [8] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool.  Pix2NeRF: Unsupervised conditional p-GAN for single image to neural radiance fields translation.  In CVPR, 2022.
  * [9] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3D generative adversarial networks.  In CVPR, 2022.
  * [10] Bindita Chaudhuri, Nikolaos Sarafianos, Linda Shapiro, and Tony Tung.  Semi-supervised synthesis of high-resolution editable textures for 3D humans.  In CVPR, 2021.
  * [11] Anton Cherepkov, Andrey Voynov, and Artem Babenko.  Navigating the GAN parameter space for semantic image editing.  In CVPR, 2021.
  * [12] Antonia Creswell and Anil Anthony Bharath.  Inverting the generator of a generative adversarial network.  IEEE transactions on neural networks and learning systems, 2018\.
  * [13] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  ArcFace: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [14] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong.  GRAM: Generative radiance manifolds for 3D-aware image generation.  In CVPR, 2022.
  * [15] Chi Nhan Duong, Khoa Luu, Kha Gia Quach, Nghia Nguyen, Eric Patterson, Tien D Bui, and Ngan Le.  Automatic face aging in videos via deep reinforcement learning.  In CVPR, 2019.
  * [16] Gunnar Farnebäck.  Two-frame motion estimation based on polynomial expansion.  In SCIA, 2003.
  * [17] Gereon Fox, Ayush Tewari, Mohamed Elgharib, and Christian Theobalt.  StyleVideoGAN: A temporal generative model using a pretrained StyleGAN.  In BMVC, 2021.
  * [18] Anna Frühstück, Ibraheem Alhashim, and Peter Wonka.  TileGAN: Synthesis of large-scale non-homogeneous textures.  In SIGGRAPH, 2019.
  * [19] Anna Frühstück, Krishna Kumar Singh, Eli Shechtman, Niloy J. Mitra, Peter Wonka, and Jingwan Lu.  InsetGAN for full-body image generation.  In CVPR, 2022.
  * [20] Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  StyleGAN-NADA: Clip-guided domain adaptation of image generators.  ToG, 2022.
  * [21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  NeurIPS, 2014.
  * [22] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  GANSpace: Discovering interpretable GAN controls.  NeurIPS, 2020.
  * [23] Fangzhou Hong, Zhaoxi Chen, Yushi Lan, Liang Pan, and Ziwei Liu.  EVA3D: Compositional 3D human generation from 2D image collections.  In ICLR, 2023.
  * [24] Ali Jahanian, Lucy Chai, and Phillip Isola.  On the ”steerability” of generative adversarial networks.  In ICLR, 2019.
  * [25] Cemre Efe Karakas, Alara Dirik, Eylül Yalçınkaya, and Pinar Yanardag.  Fairstyle: Debiasing StyleGAN2 with style channel manipulations.  In ECCV, 2022.
  * [26] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  In NeurIPS, 2020.
  * [27] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In NeurIPS, 2021.
  * [28] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [29] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In CVPR, 2020.
  * [30] Jaehoon Ko, Kyusun Cho, Daewon Choi, Kwangrok Ryoo, and Seungryong Kim.  3D GAN inversion with pose optimization.  In WACV, 2023.
  * [31] Connor Z. Lin, David B. Lindell, Eric R. Chan, and Gordon Wetzstein.  3D GAN inversion for controllable portrait image animation.  In ECCV Workshops, 2022.
  * [32] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  VoxCeleb: A large-scale speaker identification dataset.  In INTERSPEECH, 2017.
  * [33] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.  StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation.  In CVPR, 2022.
  * [34] Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, and Krishna Kumar Singh.  Spatially-adaptive multilayer selection for c inversion and editing.  In CVPR, 2022.
  * [35] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  StyleClip: Text-driven manipulation of StyleGAN imagery.  In ICCV, 2021.
  * [36] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in Style: a StyleGAN encoder for image-to-image translation.  In CVPR, 2021.
  * [37] Daniel Roich, Ron Mokady, Amit H. Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  ToG, 2022.
  * [38] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger.  VoxGRAF: Fast 3D-aware image synthesis with sparse voxel grids.  In NeurIPS, 2022.
  * [39] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of GANs for semantic face editing.  In CVPR, 2020.
  * [40] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  InterFaceGAN: Interpreting the disentangled face representation learned by GANs.  TPAMI, 2020.
  * [41] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in GANs.  In CVPR, 2021.
  * [42] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny.  StyleGAN-V a continuous video generator with the price, image quality and perks of StyleGAN2.  In CVPR, 2022.
  * [43] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu.  IDE-3D: Interactive disentangled editing for high-resolution 3D-aware portrait synthesis.  ToG, 2022.
  * [44] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt.  Pie: Portrait image embedding for semantic control.  ToG, 2020.
  * [45] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt.  StyleRig: Rigging StyleGAN for 3D control over portrait images.  In CVPR, 2020.
  * [46] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov.  A good image generator is what you need for high-resolution video synthesis.  In ICLR, 2021.
  * [47] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for StyleGAN image manipulation.  In SIGGRAPH, 2021.
  * [48] Rotem Tzaban, Ron Mokady, Rinon Gal, Amit H. Bermano, and Daniel Cohen-Or.  Stitch it in time: GAN-Based facial editing of real videos.  In SIGGRAPH Asia, 2022.
  * [49] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In ICML, 2020.
  * [50] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen.  High-fidelity GAN inversion for image attribute editing.  In CVPR, 2022.
  * [51] Weihao Xia and Jing-Hao Xue.  A survey on 3D-aware image synthesis.  arXiv preprint arXiv:2210.14267, 2022.
  * [52] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  GAN inversion: A survey.  TPAMI, 2022.
  * [53] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  GAN inversion: A survey.  TPAMI, 2022.
  * [54] Yiran Xu, Badour AlBahar, and Jia-Bin Huang.  Temporally consistent semantic video editing.  In ECCV, 2022.
  * [55] Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier.  A latent transformer for disentangled face editing in images and videos.  In ICCV, 2021.
  * [56] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  BiSeNet: Bilateral segmentation network for real-time semantic segmentation.  In ECCV, 2018.
  * [57] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin.  Generating videos with dynamics-aware implicit generative adversarial networks.  In ICLR, 2022.
  * [58] Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, and Alex Colburn.  Generative multiplane images: Making a 2D GAN 3D-aware.  In ECCV, 2022.
  * [59] Peiye Zhuang, Oluwasanmi Koyejo, and Alexander G. Schwing.  Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation.  In ICLR, 2021.