# Remote trajectory tracking of a rigid body in an incompressible fluid at low
Reynolds number

József J. Kolumbán  Institut für Mathematik, Universität Leipzig, D-04109,
Leipzig, Germany

###### Abstract.

In this paper we study the motion of a rigid body driven by Newton’s law
immersed in a stationary incompressible Stokes flow occupying a bounded simply
connected domain. The aim is that of trajectory tracking of the solid by the
means of a control in the form of Dirichlet boundary data on the outside
boundary of the fluid domain. We show that it is possible to exactly achieve
any smooth trajectory for the solid that stays away from the external
boundary, by the means of such a remote control. The proof relies on some
density methods for the Stokes system, as well as a reformulation of the solid
equations into an ODE.

###### Contents

  1. 1 Introduction and statement of results
  2. 2 Reformulation of the solid equations
  3. 3 A density result for controlled Stokes systems
  4. 4 Proof of Theorem 1.1

##  1\. Introduction and statement of results

We are interested in investigating the evolution of a rigid body immersed in
an incompressible fluid at low Reynolds number, under the influence of an
external boundary control. The model is given by the following coupled system
consisting of the stationary incompressible Stokes equation for the fluid:

(1.1) $\displaystyle\begin{split}-\Delta u+\nabla p&=0\text{ in
}\mathcal{F}(t),\\\ \operatorname{div}u&=0\text{ in }\mathcal{F}(t),\\\
u&=u_{S}\text{ on }\partial\mathcal{S}(t),\\\ u&=g\text{ on
}\partial\Omega,\end{split}$

and Newton’s balance law for the solid:

(1.2)
$\displaystyle\begin{split}mh^{\prime\prime}(t)&=-\int_{\partial\mathcal{S}(t)}\Sigma(u,p)n\,d\sigma,\\\
(\mathcal{I}(t)\theta^{\prime}(t))^{\prime}&=-\int_{\partial\mathcal{S}(t)}(x-h(t))\wedge\Sigma(u,p)n\,d\sigma,\end{split}$

for $t\in[0,T]$. Here we use the following notations:

  * •

$\Omega\subset\mathbb{R}^{3}$ is a bounded, open, simply connected domain,
which at all times $t\in[0,T]$ is split into a non-empty, closed, regular,
connected and simply connected set $\mathcal{S}(t)$ occupied by the solid, and
the remaining set $\mathcal{F}(t):=\Omega\setminus\mathcal{S}(t)$ occupied by
the fluid;

  * •

$u$ denotes the fluid velocity, taking values in $\mathbb{R}^{3}$, while $p$
is the scalar-valued fluid pressure, the Cauchy stress tensor is the symmetric
$3\times 3$ matrix given by

(1.3) $\displaystyle\Sigma(u,p)=-p\text{Id}_{3}+2D(u),\text{ where
}D(u)=\frac{1}{2}\nabla u+(\nabla u)^{T}\text{ is the symmetric gradient};$

  * •

$n$ always denotes the unit normal vector on $\partial\mathcal{F}(t)$ pointing
outside the fluid;

  * •

$g:[0,T]\times\partial\Omega\to\mathbb{R}^{3}$ plays the role of the control
(as Dirichlet boundary data), and is supported on a fixed, non-empty open part
$\Gamma$ of the outer boundary $\partial\Omega$, further satisfying the flux
condition $\int_{\Gamma}g\cdot n\,d\sigma=0$;

  * •

the solid $\mathcal{S}(t)$ is completely characterized by its center of mass
$h(t)\in\mathbb{R}^{3}$ and angle of rotation $\theta(t)\in\mathbb{R}^{3}$,
its evolution is obtained through rigid movement via

(1.4) $\displaystyle\mathcal{S}(t)=h(t)+R(\theta(t))(\mathcal{S}_{0}-h_{0}),$

with

(1.5) $\displaystyle
R(\theta)=\begin{pmatrix}\cos(\theta_{1})&-\sin(\theta_{1})&0\\\
\sin(\theta_{1})&\cos(\theta_{1})&0\\\
0&0&1\end{pmatrix}\begin{pmatrix}\cos(\theta_{2})&0&-\sin(\theta_{2})\\\
0&1&0\\\
\sin(\theta_{2})&0&\cos(\theta_{2})\end{pmatrix}\begin{pmatrix}1&0&0\\\
0&\cos(\theta_{3})&-\sin(\theta_{3})\\\
0&\sin(\theta_{3})&\cos(\theta_{3})\end{pmatrix}$

being the standard three-dimensional rotation matrix;

  * •

one then has that

(1.6) $\displaystyle\frac{d}{dt}R(\theta(t))=\theta^{\prime}(t)\wedge
R(\theta(t)),$

where $\theta^{\prime}(t)\wedge$ denotes the canonical skew-symmetric matrix
associated with the vector $\theta^{\prime}(t)$, and hence the solid velocity
is given by

(1.7) $\displaystyle
u_{S}(t,x)=h^{\prime}(t)+\theta^{\prime}(t)\wedge(x-h(t))\text{ for
}x\in\mathcal{S}(t);$

  * •

$m>0$ denotes the mass of the solid, $\mathcal{I}(t)$ is the positive definite
inertial matrix, evolving according to Sylvester’s law:

(1.8)
$\displaystyle\mathcal{I}(t)=R(\theta(t))\mathcal{I}_{0}R(\theta(t))^{T}.$

The problem we are interested in is that of trajectory tracking, i.e. steering
the solid along any given sufficiently regular trajectory by the means of the
external boundary control $g$.

Let us denote
$q:=(h,\theta),q^{\prime}:=(h^{\prime},\theta^{\prime})\in\mathbb{R}^{6}$. We
observe that the dependence with respect to $t$ in $\mathcal{F}(t)$ comes only
from the dependence with respect to $q$, hence from here on we will use the
(slightly abusive) notations $\mathcal{F}(q(t))$, $\mathcal{S}(q(t))$.
Furthermore, since we want to avoid the solid touching the outer boundary
$\partial\Omega$, we will consider only positions $q$ from the set

$\mathcal{Q}_{\delta}=\\{q\in\mathbb{R}^{6}:\ \mathcal{S}(q)\subset\Omega,\
d(\mathcal{S}(q),\partial\Omega)\geqslant\delta\\},$

for $\delta>0$.

We introduce the space in which we look for the spacial part of the control
$g(t,\cdot)$ as

$H^{1/2}_{m}(\Gamma):=\left\\{\phi\in
H^{1/2}(\partial\mathcal{F};\mathbb{R}^{3}):\ \text{supp }\phi\subset\Gamma,\
\int_{\Gamma}\phi\cdot n\,d\sigma=0\right\\}.$

Note that if $g\in C^{0}([0,T];H^{1/2}_{m}(\Gamma))$ and $q\in
C^{2}([0,T];\mathcal{Q}_{\delta})$ are given, system (1.1) has a unique
solution $u\in C^{0}([0,T];H^{1}(\mathcal{F}(q(t))))$, $p\in
C^{0}([0,T];L^{2}(\mathcal{F}(q(t)))/\mathbb{R})$, see for example [23], where

$L^{2}(A)/\mathbb{R}:=\left\\{\phi\in L^{2}(A):\
\int_{A}\phi(x)\,dx=0\right\\}\text{ for }A\subset\Omega.$

We observe that there is a slight abuse of notation in writing
$C^{0}([0,T];H^{1}(\mathcal{F}(q(t))))$ for a space of functions defined for
each $t$ in the fluid domain $\mathcal{F}(q(t))$. We will henceforth use the
convention that for a functional space $X$ of functions depending on the
variable $x$, the notation $C^{0}([0,T];X(\mathcal{F}(q(t))))$ signifies the
space of functions defined for each $t$ in the fluid domain
${\mathcal{F}}(q(t))$, which can be extended to functions in
$C^{0}([0,T];X(\mathbb{R}^{2}))$.

Furthermore, if $\psi\in H^{1}(\mathcal{F}(q(t));\mathbb{R}^{3})$ for some
fixed $t\in[0,T]$ with $\operatorname{div}\psi=0$, one may test the Stokes
equation in (1.1) with $\psi$ to obtain via integration by parts that

(1.9) $\displaystyle 0=\int_{\mathcal{F}(q(t))}(-\Delta u+\nabla
p)\cdot\psi\,dx=2\int_{\mathcal{F}(q(t))}D(u):D(\psi)\,dx-\int_{\partial\mathcal{S}(q(t))}\Sigma(u,p)n\cdot\psi\,d\sigma.$

Figure 1. The setting of the trajectory tracking problem.

The main result of the paper is the following trajectory tracking result for
the solid movement.

######  Theorem 1.1.

For any $\delta>0$, there exists a finite dimensional subspace $\mathcal{E}$
of $H^{1/2}_{m}(\Gamma)$ such that the following holds. Let $T>0$,
$\mathcal{K}\subset\mathbb{R}^{6}\times\mathbb{R}^{6}$ compact, then there
exists a control law
$\mathcal{C}\in\text{Lip}(\cup_{q\in\mathcal{Q}_{\delta}}\\{q\\}\times\mathcal{K};\mathcal{E}),$
which satisfies that for any given trajectory $q$ in
$C^{2}([0,T];\mathcal{Q}_{\delta})$ with
$(q^{\prime}(t),q^{\prime\prime}(t))\in\mathcal{K}$ for $t\in[0,T]$, there
exists a velocity field $u\in C^{0}([0,T];H^{1}(\mathcal{F}(q(t))))$ and
pressure $p\in C^{0}([0,T];L^{2}(\mathcal{F}(q(t)))/\mathbb{R})$ such that
$(q,u,p)$ is the unique solution of (1.1)-(1.2) with
$g(t,\cdot)=\mathcal{C}(q(t),q^{\prime}(t),q^{\prime\prime}(t))(\cdot)$ for
all $t$ in $[0,T]$. In addition, the cost of the control law $\mathcal{C}$ can
be estimated by

(1.10)
$\displaystyle\|\mathcal{C}(q,q^{\prime},q^{\prime\prime})(\cdot)\|_{H^{1/2}_{m}(\Gamma)}\leqslant
C_{\delta}(|q^{\prime\prime}|+|q^{\prime}|+|q^{\prime}|^{2}),\
\forall(q,q^{\prime},q^{\prime\prime})\in\mathcal{Q}_{\delta}\times\mathcal{K},$

where $C_{\delta}>0$ only depends on $\mathcal{Q}_{\delta}$.

Comparison to literature.

Note that similarly to the trajectory tracking result from [13] in the
inviscid case, our control is of feedback form, depending on the solid
position, velocity and acceleration. However, there is no circulation or
vorticity involved since we consider the low Reynolds number case, hence the
control only depends on the finite dimensional quantity
$(q,q^{\prime},q^{\prime\prime})$. This can be an advantage in terms of
possible practical implementation, compared to the discussion in the very last
paragraph of Section 3 of [13], where the disadvantages of having a feedback
law depending on the vorticity (as an infinite dimensional quantity) were
presented. Furthermore, the control takes values only in the finite-
dimensional subspace $\mathcal{E}$ of $H^{1/2}_{m}(\Gamma)$, which can be a
further advantage (in addition to the aforementioned feedback form) in such a
direction of practical use. However, one should note that in this paper we use
completely different methods of proof than in [13], where the authors used a
nonlinear method to solve linear perturbations of nonlinear equations
associated with a quadratic operator. Our proof rather relies on reformulating
the Newton equations into a quasi-linear ODE. We then prove exact trajectory
tracking for this ODE by using a density argument for our Stokes system which
can be seen as a generalization of the argument from Section 4 of [12]. In
addition, once more thinking in terms of practical implementation, one may
note similarly as in [12], that the density result Lemma 4.1 from Section 3
below can be advantageous in an approximation scheme. Finally, we note in
Remark 4.2 at the end of the paper that in fact if the trajectory $q$
corresponds to the solution of the uncontrolled system on some subinterval of
$[0,T]$, then the associated feedback control can be turned off on this
subinterval. Once again in this context we may highlight the fact that
although the fluid model is simpler than in [13] (linear Stokes versus
nonlinear Euler), we obtain more properties for our control law which can also
be of interest from an engineering point of view.

Let us compare our result to the local null-controllability results obtained
in the case of a rigid body evolving in a fluid modelled by the Navier-Stokes
equations (with Dirichlet boundary conditions) from [5, 6, 17]. While our
model is simpler due to the fact that we only consider the stationary Stokes
equations (hence there is no time derivative or nonlinearity involved in the
PDE part), if one thinks only on the level of the solid, our result of exact
trajectory tracking is also stronger than merely local null-controllability.
However, in the three papers mentioned above, the authors actually achieve
local null-controllability for the solid position and the velocities of both
the solid and the fluid, while in our current paper we only track the
trajectory of the solid. In fact one should note that in our model there is
always some fluid stuck to the solid boundary, so exactly controlling the
fluid velocity in the whole domain at the same time as the prescribing the
solid trajectory would run into difficulties. However, Lagrangian
controllability of parts of the fluid away from the solid could be possible at
the same time, using a strategy as in [12]. On a different note, while the
papers [5, 6, 17] rely on on Carleman estimates for the linearized equation,
one can compare this to our use of a unique continuation argument for an
appropriate Stokes system to prove the density result in Section 3 below.

Let us also recall the difficult open problem of global exact controllability
of the final position and velocity of a body evolving in a viscous fluid
modelled by the Navier-Stokes equations with Dirichlet boundary conditions,
due to the effect of boundary layers. However, if one considers the so-called
Navier slip-with-friction boundary conditions instead, such a controllability
has been achieved in [18], by generalizing certain methods introduced in [9]
in the case of a fluid alone. However, as mentioned in [18], the methods used
in that paper are not applicable in order to achieve trajectory tracking,
since they rely on a time-rescale of the system.

In a different direction, one may also think of comparing our result to the
controllability of swimmers (i.e. deformable bodies) moving in viscous fluids,
where the control is no longer on the outside boundary of the fluid domain,
but consists of the deformation of the solid itself. For the case where the
fluid is modelled the Navier-Stokes equations (either stationary or non-
stationary), see the papers [8, 11, 19, 21], while the case where the fluid is
at low Reynolds number (hence modelled by the quasi-static Stokes equation) is
handled in [1, 2, 20, 22]. In particular, some of the main steps of [22] can
be compared to those of our paper. For instance, the authors also use an ODE
reformulation (which differs from ours due to the fact that their solid
boundary deforms with the control, while ours does not), and their Lemma 4.1
can be paralleled to our Proposition 3.1 and its use in Section 4 to
approximate certain elementary functions.

Generalizations and open problems.

A few remarks are in order regarding certain generalizations of Theorem 1.1.
One may establish analogous results in the following cases:

  * •

if $\Omega$ is not simply connected;

  * •

if there are multiple solids evolving in the fluid instead of just one (see
e.g. [13]);

  * •

if there is a background flow present (see [16]);

  * •

or if one considers the problem in the two-dimensional setting.

The respective assumptions were only made in order to simplify the
presentation. Particularly in the view of tracking the trajectory of multiple
solids, in terms of possible practical applications, one can mention similar
problems as those in [13], such as regrouping or dispersing solids, but in the
case of low Reynolds number fluids. Such examples include cleaning up
undesirable solids from the fluid or delivering some medication (solid
particles) via an external fluid.

We further note the case of trajectory tracking for a solid in an unsteady
Stokes flow as an interesting open problem. However, the extra $\partial_{t}u$
term seems incompatible with our current strategy, and for the moment we have
not found any other approach to counteract this.

As mentioned above when comparing to the existing literature, trajectory
tracking when the fluid is governed by the Navier-Stokes equations with
Dirichlet boundary conditions is an even more challenging open problem. In
fact, one should note that if one can prove exact trajectory tracking, then
exact controllability of the position and velocity follows trivially, by
picking an appropriate trajectory having the desired final position and
velocity.

While we specifically introduced the set of admissible positions
$\mathcal{Q}_{\delta}$ in order to avoid the solid touching the outer boundary
$\partial\Omega$, one can also be interested in achieving controlled
collisions in this sense. In such a direction one should mention the papers
[14, 15] where the authors have shown that at least in the uncontrolled case,
in the setting when the solid is a ball and the outer boundary is flat, there
can be no collision (both in case of the Navier-Stokes and) in the case of the
stationary Stokes system. However, a priori this does not rule out the
possibility that in some other controlled regime, one could make the solid
touch the outer boundary.

Structure of the paper. In Section 2 we reformulate the Newton equations into
a quasi-linear ODE. In Section 3 we show a density argument for controlled
solutions of the Stokes system. Finally in Section 4 we prove our main result
by appropriately combining these two parts together.

##  2\. Reformulation of the solid equations

In this Section we will reformulate the system (1.1)-(1.2) by splitting the
fluid velocity $u$ into two parts, one corresponding to the control and the
other to the movement of the solid. For similar reformulation methods for
rigid movement in Stokesian dynamics, see for instance the references [3] and
[7].

More precisely, for $q\in\mathcal{Q}_{\delta}$, we introduce the so-called
”elementary rigid velocities”

(2.3) $\displaystyle\phi_{i}(q,x)=\left\\{\begin{array}[]{ll}e_{i}\text{ for
}i=1,2,3,\\\ e_{i-3}\wedge(x-h)\text{ for }i=4,5,6,\end{array}\right.$

where $(e_{1},e_{2},e_{3})$ is the canonical basis of $\mathbb{R}^{3}$. We may
then associate the so-called ”elementary Stokes solutions” as the unique
smooth solutions $(V_{i}(q,\cdot),P_{i}(q,\cdot))$ of the problem

(2.4) $\displaystyle\begin{split}-\Delta_{x}V_{i}+\nabla_{x}P_{i}&=0\text{ in
}\mathcal{F}(q),\\\ \operatorname{div}_{x}V_{i}&=0\text{ in
}\mathcal{F}(q),\\\ V_{i}&=\phi_{i}\text{ on }\partial\mathcal{S}(q),\\\
V_{i}&=0\text{ on }\partial\Omega,\end{split}$

for $i=1,\ldots,6$. We will also use the notation
$V(q,\cdot):=(V_{1},\ldots,V_{6})(q,\cdot)$ and
$P(q,\cdot):=(P_{1},\ldots,P_{6})(q,\cdot)$.

We further consider the contribution due to the control $g\in
H^{1/2}_{m}(\Gamma)$, denoted by $(u^{c}[g],p^{c}[g])\in
H^{1}(\mathcal{F}(q))\times(L^{2}(\mathcal{F}(q))/\mathbb{R})$, defined as the
solution of the problem

(2.5) $\displaystyle\begin{split}-\Delta u^{c}+\nabla p^{c}&=0\text{ in
}\mathcal{F}(q),\\\ \operatorname{div}u^{c}&=0\text{ in }\mathcal{F}(q),\\\
u^{c}&=0\text{ on }\partial\mathcal{S}(q),\\\ u^{c}&=g\text{ on
}\partial\Omega.\end{split}$

Clearly $g\mapsto(u^{c}[g],p^{c}[g])$ is linear.

It is then easy to see that the solution $(u,p)$ of (1.1) can be written as

(2.6) $\displaystyle u(t,x)=q^{\prime}(t)\cdot V(q(t),x)+u^{c}(t,x),\quad
p(t,x)=q^{\prime}(t)\cdot P(q(t),x)+p^{c}(t,x).$

We may further define the mass matrix

(2.7) $\displaystyle\mathcal{M}(q):=\begin{pmatrix}m\text{Id}_{3}&0_{3}\\\
0_{3}&\mathcal{I}(q),\end{pmatrix}$

as well as the so-called ”Stokes resistance matrix”

(2.8)
$\displaystyle\mathcal{K}(q):=\left(\int_{\partial\mathcal{S}(q)}\Sigma(V_{i}(q,\cdot),P_{i}(q,\cdot))n\cdot\phi_{j}(q,\cdot)\,d\sigma\right)_{i,j=1,\ldots,6}=2\left(\int_{\mathcal{F}(q)}D(V_{i}(d,\cdot)):D(V_{j}(d,\cdot))\,dx\right)_{i,j=1,\ldots,6},$

where we have used an integration by parts similar to (1.9) to obtain the
second equality. It can be checked that $\mathcal{K}(q)$ is symmetric,
positive definite and invertible, and the map
$q\in\mathcal{Q}_{\delta}\mapsto\mathcal{K}(q)$ is Lipschitz (see Section 5 in
[16]).

Then, using (1.3) and (2.6), the Newton equations (1.2) become

(2.9)
$\displaystyle\frac{d}{dt}(\mathcal{M}(q)q^{\prime})=-\mathcal{K}(q)q^{\prime}-\left(\int_{\partial\mathcal{S}(q)}\Sigma(u^{c},p^{c})n\cdot\phi_{i}(q,\cdot)\,d\sigma\right)_{i=1,\ldots,6}$

Note that if $g=0$, this reduces to a quasi-linear ordinary differential
equation.

##  3\. A density result for controlled Stokes systems

In this Section we generalize the density result Theorem 4.1 from [12] to our
setting.

Let us consider the following simplified model where we neglect the solid
displacement. We fix some $q\in\mathcal{Q}_{\delta}$ and introduce the space

(3.1) $\displaystyle H^{1/2}_{m}(\partial\mathcal{S}(q)):$
$\displaystyle=\left\\{\phi\in
H^{1/2}(\partial\mathcal{S}(q);\mathbb{R}^{3}):\
\int_{\partial\mathcal{S}(q)}\phi\cdot n\,d\sigma=0\right\\},$

and look at the problem

(3.2) $\displaystyle\begin{split}-\Delta u+\nabla p&=0\text{ in
}\mathcal{F}(q),\\\ \operatorname{div}u&=0\text{ in }\mathcal{F}(q),\\\
u&=0\text{ on }\partial\mathcal{S}(q),\\\ u&=\alpha\text{ on
}\partial\Omega,\end{split}$

for some given $\alpha\in H^{1/2}_{m}(\Gamma)$.

The main result of this section is the following proposition.

######  Proposition 3.1.

For any $q\in\mathcal{Q}_{\delta}$, the set

$\\{\Sigma(u,p)n|_{\partial\mathcal{S}(q)}:\ (u,p)\text{ solves
}\eqref{eq:modst}\text{ for some }\alpha\in H^{1/2}_{m}(\Gamma)\\}$

is dense in $(H^{1/2}_{m}(\partial\mathcal{S}(q)))^{\prime}$.

###### Proof.

We shall prove that

(3.3) $\displaystyle\left\\{w\in H^{1/2}_{m}(\partial\mathcal{S}(q)):\text{
for any }\alpha\in H^{1/2}_{m}(\Gamma)\text{ the solution }(u,p)\text{ of
}\eqref{eq:modst}\text{ satisfies
}\int_{\partial\mathcal{S}(q)}\Sigma(u,p)n\cdot w\,d\sigma=0\right\\}$

contains only $0$, from which the claim follows by duality.

With any $w$ in the above set, we may associate the solution of the following
system:

(3.4) $\displaystyle\begin{split}-\Delta v+\nabla q&=0\text{ in
}\mathcal{F}(q),\\\ \operatorname{div}v&=0\text{ in }\mathcal{F}(q),\\\
v&=w\text{ on }\partial\mathcal{S}(q),\\\ v&=0\text{ on
}\partial\Omega,\end{split}$

which satisfies $v\in H^{1}(\mathcal{F}(q))$.

Therefore, for given $\alpha\in H^{1/2}_{m}(\Gamma)$, we may test (3.4) with
the solution $u$ of (3.2) to obtain using integration by parts that

$\displaystyle
2\int_{\mathcal{F}(q)}D(u):D(v)\,dx=\int_{\Gamma}\Sigma(v,q)n\cdot\alpha\,d\sigma.$

On the other hand, testing (3.2) with $v$ similarly gives

$\displaystyle
2\int_{\mathcal{F}(q)}D(u):D(v)\,dx=\int_{\partial\mathcal{S}(q)}\Sigma(u,p)n\cdot
w\,d\sigma=0,$

due to $w$ belonging to (3.3).

Hence we deduce that

$\displaystyle\int_{\Gamma}\Sigma(v,q)n\cdot\alpha\,d\sigma=0,\
\forall\alpha\in H^{1/2}_{m}(\Gamma),$

from where it follows that $\Sigma(v,q)n=0$ in
$(H^{1/2}_{m}(\Gamma))^{\prime}$.

By unique continuation for the Stokes-system (see for instance Corollary 1.1
in [10] or Corollary 1.2 in [4]), since $v=0$ on $\Gamma$, it follows that
$v=0$ in $\mathcal{F}(q)$, and hence $w=0$.

∎

##  4\. Proof of Theorem 1.1

We are now in position to prove our main result regarding the trajectory
tracking of a solid immersed in a stationary Stokes flow.

In order to obtain a finite family of controls to span the subspace
$\mathcal{E}$ we are looking for, we will make use of the following lemma.

######  Lemma 4.1.

Let $\delta>0$, $\varepsilon>0$, there exists a finite dimensional subspace
$\mathcal{E}_{\varepsilon}$ of $H^{1/2}_{m}(\Gamma)$ and Lipschitz mappings

$q\in\mathcal{Q}_{\delta}\mapsto\bar{g}^{\varepsilon}_{i}(q,\cdot)\in\mathcal{E}_{\varepsilon}\text{
for }i=1,\ldots,6,$

such that for any $q\in\mathcal{Q}_{\delta}$ there holds

(4.1)
$\displaystyle\left|\left(\int_{\partial\mathcal{S}(q)}\Sigma(u^{c}[\bar{g}^{\varepsilon}_{i}(q,\cdot)],p^{c}[\bar{g}^{\varepsilon}_{i}(q,\cdot)])n\cdot\phi_{j}(q,\cdot)\,d\sigma\right)_{i,j=1,\ldots,6}-\mathcal{K}(q)\right|\leqslant\varepsilon.$

###### Proof.

For any $q\in\mathcal{Q}_{\delta}$, $\varepsilon>0$, $i\in\\{1,\ldots,6\\}$
one may apply Proposition 3.1 to deduce that there exist
$g^{\varepsilon}_{i}(q,\cdot)\in H^{1/2}_{m}(\Gamma)$ such that

$\|\Sigma(V_{i}(q,\cdot),P_{i}(q,\cdot))n-\Sigma(u^{c}[g^{\varepsilon}_{i}(q,\cdot)],p^{c}[g^{\varepsilon}_{i}(q,\cdot)])n\|_{(H^{1/2}_{m}(\partial\mathcal{S}(q)))^{\prime}}\leqslant\varepsilon,$

where we recall that $V_{i}$ was defined in (2.4), respectively $u^{c}[\cdot]$
in (2.5). Using integration by parts we may see that the functions $\phi_{i}$
defined in (2.3) are actually in $H^{1/2}_{m}(\partial\mathcal{S}(q))$, so we
may then conclude that

(4.2)
$\displaystyle\left|\left(\int_{\partial\mathcal{S}(q)}\Sigma(u^{c}[g^{\varepsilon}_{i}(q,\cdot)],p^{c}[g^{\varepsilon}_{i}(q,\cdot)])n\cdot\phi_{j}(q,\cdot)\,d\sigma\right)_{i,j=1,\ldots,6}-\mathcal{K}(q)\right|\leqslant
C\varepsilon,$

for some $C>0$ independent of $q\in\mathcal{Q}_{\delta}$.

We may associate for $q,\tilde{q}\in\mathcal{Q}_{\delta}$, the unique solution
$(\tilde{u},\tilde{p})\in H^{1}\times L^{2}/\mathbb{R}$ of the Stokes problem

$\displaystyle-\Delta\tilde{u}(q,\tilde{q},\cdot)+\nabla\tilde{p}(q,\tilde{q},\cdot)$
$\displaystyle=0\text{ in }\mathcal{F}(\tilde{q}),$
$\displaystyle\operatorname{div}\tilde{u}(q,\tilde{q},\cdot)$
$\displaystyle=0\text{ in }\mathcal{F}(\tilde{q}),$
$\displaystyle\tilde{u}(q,\tilde{q},\cdot)$ $\displaystyle=0\text{ on
}\partial\mathcal{S}(\tilde{q}),$ $\displaystyle\tilde{u}(q,\tilde{q},\cdot)$
$\displaystyle=g^{\varepsilon}_{i}(q,\cdot)\text{ on }\partial\Omega.$

Arguing once more as in Section 5 of [16], one can obtain that the map

$\tilde{q}\in\mathcal{Q}_{\delta}\mapsto\int_{\mathcal{F}(\tilde{q})}D(\tilde{u}(q,\tilde{q},\cdot)):D(V_{i}(\tilde{q},\cdot))\,dx$

is Lipschitz. Using integration by parts similarly to (1.9), the fact that
$q\in\mathcal{Q}_{\delta}\mapsto\mathcal{K}(q)$ is Lipschitz, and the estimate
(4.2), we may deduce that for any $q\in\mathcal{Q}_{\delta}$, there exists
$r_{q}=r_{q}(\varepsilon)>0$ such that

(4.3)
$\displaystyle\left|\left(\int_{\partial\mathcal{S}(q)}\Sigma(\tilde{u}(\hat{q},q,\cdot),\tilde{p}(\hat{q},q,\cdot))n\cdot\phi_{j}(q,\cdot)\,d\sigma\right)_{i,j=1,\ldots,6}-\mathcal{K}(q)\right|\leqslant\tilde{C}\varepsilon,\text{
for }|q-\hat{q}|<r_{q},$

where once more $\tilde{C}>0$ is independent of $q\in\mathcal{Q}_{\delta}$.

Using the compactness of $\mathcal{Q}_{\delta}$, one may extract a finite
subcover of balls $\\{B(q_{l},r_{l})\\}_{l=1,\ldots,N_{\delta}}$, and use a
partition of unity $\\{\rho_{l}\\}_{l=1,\ldots,N_{\delta}}$ adapted to this
subcover to obtain that

$\bar{g}_{i}^{\varepsilon}(q,\cdot):=\sum_{l=1}^{N_{\delta}}\rho_{l}(q)g^{\varepsilon}_{i}(q_{l},\cdot)$

satisfies (4.1) with $\hat{C}\varepsilon$ instead of $\varepsilon$ on the
right-hand side. Reparametrizing with respect to $\varepsilon$ and setting
$\mathcal{E}_{\varepsilon}:=\text{span}\\{g^{\varepsilon}_{i}(q_{l},\cdot):\
i=1,\ldots,6,\ l=1,\ldots,N_{\delta}\\}$ allows us to finish the proof of the
lemma.

∎

We then continue working with the functions $\bar{g}_{i}^{\varepsilon}$ given
by Lemma 4.1. Since the Stokes resistance matrix defined in (2.8) is
invertible, there exists $\bar{\varepsilon}>0$ small enough such that the
matrix

(4.4)
$\displaystyle\left(\int_{\partial\mathcal{S}(q)}\Sigma(u^{c}[\bar{g}^{\bar{\varepsilon}}_{i}],p^{c}[\bar{g}^{\bar{\varepsilon}}_{i}])n\cdot\phi_{j}(q,\cdot)\,d\sigma\right)_{i,j=1,\ldots,6}$

is also invertible. Note that a priori $\bar{\varepsilon}$ would depend on
$q$, but since $q\in\mathcal{Q}_{\delta}\mapsto\mathcal{K}(q)$ is continuous
and $Q_{\delta}$ is compact, $\bar{\varepsilon}>0$ can be chosen to be uniform
for $q\in\mathcal{Q}_{\delta}$.

We will then take the finite dimensional subspace $\mathcal{E}$ we are looking
for to be $\mathcal{E}_{\bar{\varepsilon}}$. We may look for a control of the
form

$g(q,\cdot)=\sum_{i=1}^{6}\mu_{i}\bar{g}_{i}^{\bar{\varepsilon}}(q,\cdot),$

where $\mu\in\mathbb{R}^{6}$. Plugging this into (2.9) and using the linearity
of $\Sigma(u^{c}[\cdot],p^{c}[\cdot])$ with respect to $g$, the trajectory
tracking problem reduces to finding a vector $\mu$ such that

$\displaystyle\frac{d}{dt}(\mathcal{M}(q)q^{\prime})+\mathcal{K}(q)q^{\prime}=-\left(\int_{\partial\mathcal{S}(q)}\Sigma(u^{c}[\bar{g}^{\bar{\varepsilon}}_{i}],p^{c}[\bar{g}^{\bar{\varepsilon}}_{i}])n\cdot\phi_{j}(q,\cdot)\,d\sigma\right)_{i,j=1,\ldots,6}\mu.$

However, since we have shown above that the matrix in (4.4) is invertible, one
deduces the existence of such $\mu=\mu(q,q^{\prime},q^{\prime\prime})$ as
given by

(4.5)
$\displaystyle\mu(q,q^{\prime},q^{\prime\prime})=-\left(\int_{\partial\mathcal{S}(q)}\Sigma(u^{c}[\bar{g}^{\bar{\varepsilon}}_{i}],p^{c}[\bar{g}^{\bar{\varepsilon}}_{i}])n\cdot\phi_{j}(q,\cdot)\,d\sigma\right)_{i,j=1,\ldots,6}^{-1}\left(\frac{d}{dt}(\mathcal{M}(q)q^{\prime})+\mathcal{K}(q)q^{\prime}\right).$

It is further easy to check that $\mu$ is Lipschitz with respect to
$(q,q^{\prime},q^{\prime\prime})\in\mathcal{Q}_{\delta}\times\mathcal{K}$.
This concludes the proof of the existence part of Theorem 1.1 by setting

$\mathcal{C}(q,q^{\prime}q^{\prime\prime})(\cdot):=\sum_{i=1}^{6}\mu(q,q^{\prime},q^{\prime\prime})\bar{g}_{i}^{\bar{\varepsilon}}(q,\cdot)).$

Note that for any given control $g$, the uniqueness of system (1.1)-(1.2)
follows from the reformulation given in (2.6) and (2.9).

Finally, the estimate (1.10) on the cost of the control follows from (4.5),
the form of $\mathcal{M}$ in (2.7), the boundedness of $\mathcal{K}$, and the
boundedness of $\bar{g}_{i}^{\varepsilon}$ from Lemma 4.1, by writing

$\displaystyle\|\mathcal{C}(q,q^{\prime},q^{\prime\prime})(\cdot)\|_{H^{1/2}_{m}(\Gamma)}\leqslant
C_{\delta}\left|\frac{d}{dt}(\mathcal{M}(q)q^{\prime})+\mathcal{K}(q)q^{\prime}\right|,$

which concludes the proof of Theorem 1.1.

######  Remark 4.2 (Switching off the control).

Note that if the target trajectory $q$ satisfies the uncontrolled ODE

$\frac{d}{dt}(\mathcal{M}(q)q^{\prime})+\mathcal{K}(q)q^{\prime}=0$

on some subinterval of $[0,T]$, then by construction the corresponding control
$g$ given by $\mathcal{C}(q,q^{\prime},q^{\prime\prime})$ in fact vanishes for
all times in this subinterval.

## References

  * [1] F. Alouges, A. DeSimone, A. Lefebvre, Swimming at low Reynolds number at optimal strokes: An example, J. Nonlin. Sci. 3 (2008) 277–302.
  * [2] F. Alouges, L. Giraldi, Enhanced controllability of low reynolds number swimmers in the presence of a wall, Acta Appl. Math. 128 (2013) 153–179.
  * [3] G. Bossis, J. F. Brady, Dynamic simulation of sheared suspensions. I. General method, J. Chem. Phys. 80, 5141-5154 (1984)
  * [4] M. Boulakia, A. C. Egloffe, C. Grandmont, Stability estimates for the unique continuation property of the Stokes system and for an inverse boundary coefficient problem, Inverse Problems 29 (2013): 115001.
  * [5] M. Boulakia, S. Guerrero, Local null controllability of a fluid-solid interaction problem in dimension 3, J. European Math Society, 15 (2013), no. 3, 825–856.
  * [6] M. Boulakia, A. Osses, Local null controllability of a two-dimensional fluid-structure interaction problem, ESAIM Control Optim. Calc. Var., 14 (2008), no. 1, 1–42.
  * [7] H. Brenner, M. E. O’Neill, On the Stokes resistance of multiparticle systems in a linear shear field, Chemical Engineering Science, Volume 27, Issue 7, 1972, Pages 1421-1439.
  * [8] J. Carling, T. Williams, G. Bowtell, Self-propelled anguilliform swimming: simultaneous solution of the two-dimentional Navier-Stokes equations and Newton’s laws of motion, J. of Experimental Biology, 201:3143-3166, 1998.
  * [9] J.-M. Coron, F. Marbach, F. Sueur, Small time global exact null controllability of the Navier- Stokes equation with Navier slip-with-friction boundary conditions, Journal of the European Mathematical Society, Volume 22, Issue 5, 2020, pp. 1625–1673.
  * [10] C. Fabre, G. Lebeau, Prolongement unique des solutions de l’équation de Stokes, Comm. Partial Differential Equations, 21(3-4): 573–596, 1996.
  * [11] G. P. Galdi, On the steady self-propelled motion of a body in a viscous incompressible fluid, Arch. Ration. Mech. Anal., 148(1):53-88, 1999.
  * [12] O. Glass, T. Horsin, Lagrangian controllability at low Reynolds number, ESAIM. Control, Optimisation and Calculus of Variations, vol. 22, no4, p. 1040–1053.
  * [13] O. Glass, J. J. Kolumbán, F. Sueur, Remote trajectory tracking of rigid bodies immersed in a 2D perfect incompressible fluid, to appear in Pure and Applied Analysis, 2022.
  * [14] M. Hillairet, Lack of collision between solid bodies in a 2D incompressible viscous flow, Commun. Partial Differ. Equ. 32, 1345–1371 (2007)
  * [15] M. Hillairet, T. Takahashi, Collisions in three-dimensional fluid structure interaction problems, SIAM J. Math. Anal. 40, 2451–2477, 2009.
  * [16] R. M. Höfer, C. Prange, F. Sueur, Motion of several slender rigid filaments in a Stokes flow, Journal de l’Ecole Polytechnique, Tome 9, 327-380, 2022.
  * [17] O. Imanuvilov, T. Takahashi, Exact controllability of a fluid-rigid body system. J. Math. Pures Appl. 87, Issue 4 (2007), 408-437.
  * [18] J. J. Kolumbán.  Control at a distance of the motion of a rigid body immersed in a two-dimensional viscous incompressible fluid, Journal of Differential Equations, Volume 269, Issue 1, 15 (2020), Pages 764-831.
  * [19] H. Liu and K. Kawachi, A numerical study of undulatory swimming, J. comput. phys., 155(2):223-247, 1999.
  * [20] J. Lohéac and A. Munnier, Controllability of 3D low Reynolds number swimmers, ESAIM COCV 20 (2014) 236–268
  * [21] J. San Martin, J. F. Scheid, T. Takahashi, M. Tucsnak, An initial and boundary problem modeling fish-like swimming, Arch. Ration. Mech. Anal., June 2008, Volume 188, Issue 3, pp 429-455.
  * [22] J. San Martin, T. Takahashi, M. Tucsnak, A control theoretic approach to the swimming of microscopic organisms, Quart. Appl. Math., 65(3):405-424, 2007.
  * [23] R. Temam, Navier-Stokes Equations: Theory and numerical analysis, North-Holland Publications, 1979\.


# Dynamic Neural Portraits

Michail Christos Doukas${}^{1,2}\quad$ Stylianos Ploumpis${}^{2}\quad$
Stefanos Zafeiriou1,2
1Imperial College London, UK$\quad{}^{2}$Huawei Technologies, London, UK
$\\{$michail.christos.doukas, stylianos.ploumpis,
<EMAIL_ADDRESS>
###### Abstract

We present Dynamic Neural Portraits, a novel approach to the problem of full-
head reenactment. Our method generates photo-realistic video portraits by
explicitly controlling head pose, facial expressions and eye gaze. Our
proposed architecture is different from existing methods that rely on GAN-
based image-to-image translation networks for transforming renderings of 3D
faces into photo-realistic images. Instead, we build our system upon a 2D
coordinate-based MLP with controllable dynamics. Our intuition to adopt a
2D-based representation, as opposed to recent 3D NeRF-like systems, stems from
the fact that video portraits are captured by monocular stationary cameras,
therefore, only a single viewpoint of the scene is available. Primarily, we
condition our generative model on expression blendshapes, nonetheless, we show
that our system can be successfully driven by audio features as well. Our
experiments demonstrate that the proposed method is 270 times faster than
recent NeRF-based reenactment methods, with our networks achieving speeds of
24 fps for resolutions up to $1024\times 1024$, while outperforming prior
works in terms of visual quality.

##  1 Introduction

Controllable video portrait synthesis is an interesting research topic that
captures the attention of both Computer Graphics and Computer Vision
communities. A video portrait is defined as a sequence of frames that depict a
single individual performing diverse head movements and facial expressions.
The person’s entire head is contained within the frame’s borders, along with a
small part of the upper body, i.e. neck and torso, while the subject usually
stands in front of a static background. Recent attempts to video portraits
synthesis using neural networks have shown very promising results, based
either on Generative Adversarial Networks (GANs) [20] or Neural Radiance
Fields (NeRFs) [31]. The applications of such systems are numerous, ranging
from video editing and movie dubbing to teleconference, virtual assistance,
social media, VR and games.

A number of learning-based solutions for generating video portraits rely on
GAN-based image-to-image translation models, with an encoder-decoder
architecture. For instance, Deep Video Portraits (DVP) [26] employ a network
that learns a mapping from coloured renderings of 3D faces to realistic
portraits. Similarly, Head2Head [27] translates images of Projected Normalized
Coordinate Codes (PNCCs) [53] into photo-realistic frames, using a video-to-
video framework. The renderings of 3D faces that serve as conditional input to
generative neural networks depend on expression blendshapes that are obtained
after fitting 3D Morphable Models (3DMMs) [5, 35, 7, 8, 29] to videos. The
fitting step is followed by a physically-based rendering process, which
creates the 2D renderings. On the contrary, we propose a multi-layer
perceptron (MLP) that conditions synthesis directly on non-spatial data (e.g.
expression parameters), and thus does not require renderings of 3D faces.

More recently, NerFACE [16] capitalised on the photo-realism achieved by NeRFs
[31] and produced synthetic video portraits of higher quality, in larger
resolutions, such as $512\times 512$. Nonetheless, due to ray casting and
volume sampling, image rendering with NerFACE requires several seconds per
frame. Moreover, the authors make the assumption that the tracked head pose
parameters coincide with the camera viewpoints of the scene, which causes
significant inconsistencies in torso synthesis, as in practice the camera is
static and therefore only a single viewpoint exists for the scene. AD-NeRF
[22] proposes an audio-driven method for portrait synthesis and solves the
camera problem by taking advantage of face segmentation maps [28]. They split
the portrait into head, torso and background and devise two separate NeRF
models: one for the head, which uses head poses as camera viewpoints, and
another one for the torso, that treats head poses as simple inputs to the MLP,
while considering camera viewpoints fixed. However, AD-NeRF is even slower
during inference, as it evaluates two MLPs.

(a) Image-based rendering of video portraits [26, 27](b) NeRF-based video
portrait rendering [16, 22](c) Our paradigm for video portrait rendering
Figure 1: In contrast to traditional GAN-based image-to-image translation
approaches to full-head reenactment, such as DVP [26] and Head2Head [27] (a),
or recent NeRF-based video portrait rendering methods, namely NerFACE [16] and
AD-NeRF [22] (b), we propose a novel paradigm for controllable video portrait
synthesis, composed of an MLP and a CNN-based decoder (c).

In this paper, we take a completely different approach and present Dynamic
Neural Portraits, a fast and efficient framework for reenacting human faces.
Our method draws inspiration from implicit neural representations (INR), as we
leverage neural networks to parameterise video portraits. We follow practices
from both conditionally independent pixel synthesis (CIPS) [2] and neural
rendering with convolutional networks. To be more specific, we perform video
portrait synthesis using a 2D coordinate-based MLP with controllable dynamics.
That is, we condition an MLP network on pixel coordinates, expression, pose
and gaze parameters, without relying on renderings of 3D faces. Nonetheless,
instead of directly predicting pixel colours, we adopt practices from 3D aware
GANs [32, 21, 52, 9] and propose an MLP that produces feature vectors, which
are computed across all 2D spatial locations and up-sampled with a CNN-based
decoder. We optimise our ”hybrid” MLP-CNN architecture jointly on the task of
video portrait reconstruction. To the best of our knowledge, we are the first
to condition a 2D coordinate-based MLP on expression, pose and gaze parameters
for explicitly controlling video portraits, without relying on renderings of
3D faces, GANs, or NeRFs. Moreover, unlike previous methods that focus
exclusively either on expression blendshapes or audio signals as driving data,
our work demonstrates how to leverage both modalities using the same
architecture. Our approach combines high-quality samples with unparalleled
execution performance, $270$ times faster than recent NeRF-based state-of-the-
art reenactment methods [16, 22]. The contributions of this paper can be
summarised as follows:

  * •

We propose a new approach to full-head reenactment, with a generator that
consists of a 2D coordinate-based MLP with controllable dynamics and a CNN
decoder.

  * •

We show that our architecture can be driven either by expression blendshapes
or audio-based features.

  * •

Our comprehensive experiments demonstrate that our method outperforms related
state-of-the-art systems, both in terms of execution speed and image quality.

##  2 Related work

3D Face Modeling and Face Reenactment. Since their introduction, 3DMMs [5, 35,
7, 8, 29] have been widely used to represent human faces, as they are strong
statistical models that enable explicit control over the shape and texture of
3D facial meshes. Various 3D face reconstruction methods depend on 3DMMs to
recover 3D faces from visual data. Such methods are classified either as
optimisation-based [46, 6], as they fit 3DMMs to visual data and estimate
parameters in an analysis-by-synthesis fashion, or learning-based [18, 15]
that rely on neural networks to reconstruct 3D faces. Apart from capturing
coarse meshes, the later have shown very promising results in extracting even
local fine details. Recovering facial shape and expression information from
video data has been proven very useful for numerous face reenactment methods
[17, 42, 43]. The work of Garrido et al. [17] is one of the first attempts to
reenact faces while relying on 3D face modeling. A succeeding graphics-based
method, namely Face2Face [43], performs real-time expression transfer from a
driving sequence of frames to a target identity, by re-writing the interior
facial region of the target video. A more recent approach, HeadOn [44],
achieves full-head reenactment that includes pose and gaze transfer, based on
RGB-D video data.

Learning-based Talking Head Synthesis. In contrary to the graphics-based
systems described above, most of the latest face re-targeting approaches are
learning-based [39, 26, 27]. Suwajanakorn et al. [39] are among the first to
propose an audio-driven neural network for mapping acoustic signals to photo-
realistic frames with accurate lip motions. Following up, Neural Voice
Puppetry [40] employs a module for translating audio features into expression
blendshapes, as an intermediate representation. Regarding video-driven
methods, Deep Video Portraits (DVP) [26] is one of the earliest GAN-based
approaches to full-head reenactment. It relies on an image-to-image
translation network that receives as inputs synthetic face renderings of a
parametric 3D face model and generates images of the target subject. In a
similar direction, Head2Head [27, 12] translates PNCCs [53] into photo-
realistic frames, with the help of a video-based GAN for better temporal
stability. Deferred Neural Rendering [41] takes a different step, by combining
traditional graphics with learnable neural textures, embedded on the 3D facial
mesh. Apart from the aforementioned person-specific approaches, there is a
plethora of person-generic methods, which require only a few frames of the
target identity [48, 37, 38, 4, 19, 50, 49, 23, 13].

Neural Representations for Scenes and Faces. With the introduction of NeRFs
[31], much research has been focused on neural scene representations [30, 36],
with various attempts to model human faces. More specifically, Nerfies [33]
and HyperNeRF [34] have shown incredible results for reconstructing non-rigid
scenes of moving heads. Despite their impressive generative ability, such
systems are not able to control neither head poses nor facial movements. In a
different direction, NerFACE [16] proposes to control a dynamic NeRF with the
assistance of expression blendshapes. In a similar line of work, AD-NeRF [22]
proposes an audio-driven NeRF-like model, based on acoustic features extracted
with DeepSpeech [24, 1].

##  3 Method

In this section, we first describe our baseline approach to the problem of
full-head reenactment (Sec. 3.1), that is a 2D coordinate-based MLP with
controllable dynamics. Then, we couple the MLP with a CNN-based decoder to
build our full model for video portrait synthesis (Sec. 3.2), namely Dynamic
Neural Portraits. Finally, we show an extension of our system that supports
audio-driven synthesis (Sec. 3.3).

###  3.1 2D MLP with Controllable Dynamics

Let $\mathbf{I}$ be an image of a fixed resolution $H\times W$. We can
represent $\mathbf{I}$ with a neural network by training a fully-connected MLP
to reconstruct the image from its 2D coordinates [14]. For the synthesis of
each pixel, we pass its coordinates $\mathbf{x}=(x,y)$ through the MLP
network, which returns the colour of the pixel $\mathbf{c}$. We optimise the
network by penalising the distance between the predicted and true colour. In
order to compute the entire image, the MLP is evaluated at each position
$(x,y)$ of the coordinate grid.

The model described above is quite limited, as it learns only to reconstruct a
single static image from its pixel coordinates. We extend the 2D coordinate-
based MLP for handling temporally varying data, such as video portraits of
human faces. Here, we focus on RGB videos captured by a monocular and
stationary camera. A time-varying representation can be obtained by
conditioning the MLP network on facial information that changes between
frames. Let $\mathbf{I}_{1:T}$ be a sequence of frames and $\mathbf{p}_{1:T}$,
$\mathbf{e}_{1:T}$ be the corresponding head pose and facial expression
parameters respectively, which have been recovered with a face tracking
system. An intuitive solution for modeling the video portrait with a neural
network would be to estimate the pixel’s RGB value from the $i$-th video frame
with the MLP, using the pixel’s coordinates $\mathbf{x}$ and tracked
parameters $\mathbf{p}_{i},\mathbf{e}_{i}$, as

$\mathbf{c}=C(\mathbf{x},\mathbf{p}_{i},\mathbf{e}_{i}).$ (1)

Here, $\mathbf{c}$ is the colour at 2D location $\mathbf{x}\in[0,1]^{2}$,
$\mathbf{p}_{i}\in\mathbb{R}^{6}$ is the head pose that is given by rotation
(Euler angles) and translation parameters, and
$\mathbf{e}_{i}\in\mathbb{R}^{n_{exp}}$ are the expression parameters, which
correspond to non-rigid facial deformations. Please note that the pose
parameters $\mathbf{p}_{i}$ describe the rigid motions of the face with
respect to the camera and do not include information regarding torso
movements. The position of the camera remains fixed throughout the frames.

###  3.2 Dynamic Neural Portraits

Figure 2: An overview of Dynamic Neural Portraits framework during training.
As opposed to previous full-head reenactment methods that rely on image-to-
image translation networks, our model is made up of an MLP encoder and a CNN
decoder. Instead of facial expression parameters, we can drive synthesis using
audio features, recovered from acoustic signals.

Even though the 2D coordinate-based MLP with controllable dynamics is a
straightforward approach for modeling video portraits with a single MLP
network, in practice we found that the photo-realism of generated samples and
the rendering speed significantly degrade as we increase the resolution of
videos. As shown in our experiments, we obtain superior results in terms of
visual quality by combing the MLP with a convolutional decoder network.
Following the paradigm of recent 3D aware GANs [32, 21, 52, 9], instead of
predicting RGB colour values we propose an MLP that maps its input to a visual
feature vector $\mathbf{f}\in\mathbb{R}^{n_{f}}$. Given the 2D spatial
location $\mathbf{x}$ along with pose $\mathbf{p}_{i}$, expression
$\mathbf{e}_{i}$ and gaze information $\mathbf{g}_{i}$, our MLP now predicts a
feature vector

$\mathbf{f}=F(\gamma(\mathbf{x}),\gamma(\mathbf{p}_{i}),\gamma(\mathbf{g}_{i}),\mathbf{e}_{i},\mathbf{v}_{i}).$
(2)

We observed that we acquire more accurate eye movements by adding eye gaze
angles $\mathbf{g}_{i}\in\mathbb{R}^{2}$ as an extra input to our MLP network.
Moreover, we noticed that introducing per-image learnable latent variables
$\mathbf{v}_{i}$ as inputs, a technique which has been previously adopted in
NeRF methods [30, 33, 34, 16], improves the stability of our network, since it
enables the MLP to learn variations among frames that are not modeled by the
pose and expression parameters (e.g. torso movements, illumination changes,
small background motions). Furthermore, following established practices from
NeRF-based systems and CIPS [2], we adopt positional encoding for the MLP
inputs, and more specifically on the position $\mathbf{x}$, pose
$\mathbf{p}_{i}$ and gaze $\mathbf{g}_{i}$ vectors, which are low-dimensional.
The standard encoding function

$\displaystyle\begin{split}\gamma(x)=[x,&\sin(2\pi x),\cos(2\pi x),\dots,\\\
&\sin(2^{N-1}\pi x),\cos(2^{N-1}\pi x)]^{\top}\end{split}$ (3)

proposed in [31] is applied to all values of $\mathbf{x}$, $\mathbf{p}_{i}$
and $\mathbf{g}_{i}$, mapping each number $x\in\mathbb{R}$ to a higher
dimension embedding $\gamma(x)\in\mathbb{R}^{2\cdot N+1}$. Replacing the
original inputs with their embeddings allows our model to achieve high
frequency details in the generated images.

The MLP network described above learns to estimate a feature vector
$\mathbf{f}$ for each spatial location of the image plane independently, while
considering head pose, facial expression and eye gaze information. In order to
render frame $i$, we first evaluate the MLP network at each spatial position
$\mathbf{x}\in\mathbf{X}$ of the coordinate grid that corresponds to
resolution $H_{f}\times W_{f}$, while keeping all other inputs fixed. After
that, we accumulate the resulting features in a visual feature map
$\mathbf{F}_{i}\in\mathbb{R}^{H_{f}\times W_{f}\times n_{f}}$. Then, we employ
a decoding network $D$, which receives the feature map and performs up-
sampling, in order to synthesise the output frame
$\tilde{\mathbf{I}}_{i}=D(\mathbf{F}_{i})$ at the target resolution $H\times
W$. An overview of our proposed framework is shown in Fig. 2. More details on
the architecture of networks are available in the supplementary material.

Objective Function and Optimisation. We train our proposed model on the task
of reconstruction. Given the generated image $\tilde{\mathbf{I}}_{i}$ and the
corresponding ground truth frame $\mathbf{I}_{i}$, we define the
reconstruction loss as the L2-distance between the predicted and true image.
We experimented with L1, perceptual [47] and adversarial [20] losses, but all
of them produced substantially inferior results. However, we found that we can
improve our method’s performance by adding an extra output layer to the MLP,
which predicts the colour $\mathbf{c}\in\mathbb{R}^{3}$ side-by-side with
visual features $\mathbf{f}$, and minimise the distance of the predicted
colour from the true one. To that end, we accumulate the colour outputs for
all 2D spatial locations in $\mathbf{C}_{i}\in\mathbb{R}^{H_{f}\times
W_{f}\times 3}$, and penalise the L2-distance from the ground truth image
$\mathbf{I}_{i}^{\prime}$ down-scaled to resolution $H_{f}\times W_{f}$, in
order to match the resolution of $\mathbf{C}_{i}$. The overall loss term for
frame $i$ is given as

$\mathcal{L}=\mathcal{L}_{rec}+\mathcal{L}_{rec}^{\prime}=||\tilde{\mathbf{I}}_{i}-\mathbf{I}_{i}||_{2}^{2}+||\mathbf{C}_{i}-\mathbf{I}_{i}^{\prime}||_{2}^{2}$
(4)

We optimise the MLP network and CNN-based decoder jointly, in an end-to-end
fashion.

###  3.3 Audio-driven Portrait Synthesis

Our choice to inject the driving signals (i.e. pose, expression and gaze
parameters) into our system through an MLP network enables to easily adapt our
method to other driving modalities, such as acoustic signals. Unlike previous
works [22, 16, 40, 26], which focus exclusively either on expression
blendshapes or audio data as driving signals, we show that the same
architecture can be used effectively for both modalities. More specifically,
we can replace the expression blendshapes with an audio feature vector
$\boldsymbol{\alpha}_{i}$ as input to the MLP network, in case the audio
stream is available. We extract high-level features from acoustic signals with
the widely-adopted DeepSpeech model [24, 1]. As a first step, we assign a
29-dimensional vector estimated by DeepSpeech to every video frame. Then we
create a window of vectors with size $w=16$ around each frame, taken from its
neighboring (past and future) time steps. In this way each frame $i$ is
coupled with a DeepSpeech feature $\mathbf{A}_{i}\in\mathbb{R}^{16\times 27}$.
We follow a similar approach with AD-NeRF [22] and utilise a 1D-convolutional
network $N_{aud}$ that learns to compute per-frame latent codes
$\mathbf{a}_{i}\in\mathbb{R}^{n_{a}}$ from $\mathbf{A}_{i}$. Next, we employ a
self-attention network $N_{att}$, as proposed in [40] and [22], which operates
as a temporal filter on subsequent audio codes $\mathbf{a}_{i-u+1:i+u}$ and
mixes them up with the assistance of predicted attention weights
$w_{i-u+1:i+u}$, to form the final audio feature vector
$\boldsymbol{\alpha}_{i}=\sum_{j=i-u+1}^{i+u}w_{j}\mathbf{a}_{j}$. We set
$u=4$, which results in a window of $2u=8$ time steps.

ground truthoursNerFACE [16]DVP [26]FOMM [38] Figure 3: Visual comparison with
baselines on the task of reconstruction. Our method consistently generates
more realistic samples with finer details than its counterparts. For the
evaluation of FOMM [38], images had to be further cropped. Please zoom in for
details. driveroursDVP [26]HeadGAN [13]FOMM [38] Figure 4: Visual comparison
with baselines on reenactment. Our approach transfers pose, expression and
gaze more reliably than both person-specific DVP [26] and person-generic
methods HeadGAN [13] and FOMM [38]. Please note that for the correct
evaluation of HeadGAN and FOMM, images had to be cropped closer to the face.

##  4 Experiments

Dataset. Our networks are optimised on monocular RGB videos, of various
resolutions: $256^{2}$, $512^{2}$ and $1024^{2}$. We train a new model for
each video portrait (different individual). As a pre-processing step, we crop
frames around the target face and compute pose, expression and gaze
parameters. We experiment with videos from 5K to 20K frames. In the
supplementary material we provide more details of the adopted video database
and face tracking systems.

Method | Portrait |  L1 ($\downarrow$) |  LPIPS ($\downarrow$) |  FID ($\downarrow$) |  FVD ($\downarrow$)
---|---|---|---|---|---
FOMM | ID. 1 | 8.79 | 0.095 | 25.48 | 331.0
ID. 2 | 7.60 | 0.084 | 37.24 | 338.3
ID. 3 | 9.48 | 0.130 | 24.72 | 254.10
DVP | ID. 1 | 6.95 | 0.152 | 49.35 | 195.4
ID. 2 | 9.08 | 0.079 | 37.58 | 464.3
ID. 3 | 8.01 | 0.123 | 51.30 | 196.7
NerFACE | ID. 1 | 6.19 | 0.136 | 74.22 | 278.3
ID. 2 | 10.98 | 0.143 | 74.02 | 357.7
ID. 3 | 6.28 | 0.067 | 34.64 | 81.17
Ours | ID. 1 | 6.45 | 0.094 | 23.76 | 169.8
ID. 2 | 5.21 | 0.071 | 24.60 | 222.1
ID. 3 | 5.15 | 0.051 | 23.94 | 78.06
Table 1: Numerical comparison with FOMM [38], DVP [26] and NerFACE [16] for
three different portraits on the task of reconstruction (self-reenactment).

###  4.1 Comparison with State-of-the-Art

Reconstruction. First, we carry out a comparison with Deep Video Portraits
(DVP) [26] and NerFACE [16], the best-performing person-specific reenactment
methods, as well as First Order Motion Model (FOMM) [38], a representative of
person-generic models. We evaluate the generative performance of methods on
the task of reconstruction, also know as self-reenactment. We assess the
fidelity of reconstruction quantitatively, with the assistance of L1-distance
between generated and ground truth test frames, as well as with Learned
Perceptual Image Patch Similarity (LPIPS) [51], which tests the perceptual
similarity between images. Moreover, we determine photo-realism with Fréchet
Inception Distance (FID) [25] and Fréchet Video Distance (FVD) [45] metrics,
which appear to correlate well with human perception. The results displayed in
Table 1 indicate that our method outperforms all baselines, for three
different video portraits. Please note that the reported numbers refer to the
average scores, computed across all test frames for each video portrait. An
exception to that are FID [25] and FVD [45] metrics, as they are computed on
pairs of videos. All scores suggest that our approach produces superior
samples in terms of visual quality and photo-realism. This observation is
confirmed visually in Fig. 3. As can be seen, our method generates more crispy
images with finer details and more consistent eye gaze. For a better
visualisation of our results, we urge the reader to refer to our supplementary
video.

Method | CSIM ($\uparrow$) | Expression | Pose | Gaze
---|---|---|---|---
Dist. ($\downarrow$) |  Dist. ($\downarrow$) |  Dist. ($\downarrow$)
FOMM | 0.840 | 13.28 |  6.31∘ |  9.42∘
HeadGAN | 0.755 | 14.17 |  3.26∘ |  6.35∘
DVP | 0.861 | 11.95 |  4.93∘ |  8.37∘
Ours | 0.885 | 10.69 | 2.57∘ | 4.74∘
Table 2: Numerical comparison with FOMM [38], HeadGAN [13] and DVP [26] on the
task of cross-identity reenactment.

Reenactment. We further validate our model’s performance on the task of cross-
identity motion transfer (reenactment) in a quantitative way. This task
involves passing on the head pose, facial expression and eye gaze from a
driving actor to another target person, while preserving the identity of the
later, as the two subjects are now different. We compare our approach with DVP
[26], FOMM [38] and HeadGAN [13]. To that end, we use the source and target
video portraits provided by the authors of DVP. For the numerical comparison,
we measure the target’s identity preservation with Cosine Similarity (CSIM),
based on identity embeddings extracted with the assistance of ArcFace [11].
Moreover, we use DECA [15] to regress pose and expression parameters, both
from the driving and generated frames. Then, we compute the L1-distance
between expression parameters, as well as the head rotation distance in
degrees. Finally, we use the gaze estimator from [10] to regress gaze vectors,
and calculate their angular distance. In Table 2, we show that our method
achieves better scores than all three baselines, across all metrics related to
successful reenactment. In Fig. 4, we showcase examples of frames where our
method transfers pose, expression and gaze more accurately than DVP [26],
HeadGAN [13] and FOMM [38].

Audio-driven reconstruction. Except for conditioning our MLP network on
expression blendshapes, we demonstrate the generative performance of our
system when driven by acoustic signals. For that, we conducted a side-by-side
comparison with AD-NeRF [22], the state-of-the-art model on this task. As
shown in Table 3, our quantitative analysis reveals that our approach performs
better than AD-NeRF, both in terms of reconstruction and image quality. Our
findings can be observed also visually in Fig. 5.

Method | Portrait |  L1 ($\downarrow$) |  LPIPS ($\downarrow$) |  FID ($\downarrow$) |  FVD ($\downarrow$)
---|---|---|---|---|---
AD-NeRF | Obama | 4.11 | 0.083 | 16.67 | 225.7
May | 5.13 | 0.143 | 47.31 | 272.2
Ours | Obama | 4.04 | 0.054 | 9.12 | 110.3
May | 5.72 | 0.087 | 26.72 | 96.1
Table 3: Numerical comparison with AD-NeRF [22] on audio-driven video
reconstruction. ground truthoursAD-NeRF [22] Figure 5: Visual comparison with
AD-NeRF [22] on audio-driven reconstruction. We generate more accurate lips
motions and higher quality details compared to AD-NeRF. Please zoom in for
details.

Execution Speed. Owing to its lightweight architecture, our method is able to
render frames in a resolution up to $1024\times 1024$, in 24 fps. For lower
resolutions (e.g. $256\times 256$ and $512\times 512$) our pipeline operates
in speeds faster than real time. In comparison to recent NeRF-based state-of-
the-art methods [16, 22], our system achieves a significant speed-up,
generating images nearly 270 times faster, which makes it much more efficient
for real-world applications. In Table 4, we report the execution times of
different methods, measured on NVIDIA’s Tesla V100 PCIe 32 GB. For DVP* [26]
we use the numbers recorded by its authors.

Method | $256\times 256$ | $512\times 512$ | $1024\times 1024$
---|---|---|---
time (fps) | time (fps) | time (fps)
AD-NeRF [22] | - | 9630 (0.10) | -
NerFACE [16] | - | 8465 (0.12) | -
DVP* [26] | 65 (15.4) | 196 (5.1) | -
HeadGAN [13] | 41 (24.5) | - | -
FOMM [38] | 21 (47.2) | - | -
Ours |  11 (90.9) |  31 (32.3) |  42 (24.2)
Table 4: Comparison of the execution time between our generative model and
related methods. Time is reported in milliseconds (msec). Please note that all
reported numbers refer to the forward pass time of models during inference,
without considering data pre-processing.

###  4.2 Ablation study

ground truth2D MLP contr. dyn. (A)CNN-based decoder (B)our full model (F) Figure 6: Qualitative results of our ablation study. Our full model (F) improves the quality of frames synthesised by our baseline (A) (2D coordinate-based MLP with controllable dynamics) or the unaided CNN-based decoder (B) by a noteworthy margin.  Variation | Portrait |  L1 ($\downarrow$) |  LPIPS ($\downarrow$) |  FID ($\downarrow$) |  FVD ($\downarrow$) |  Gaze Dist. ($\downarrow$)
---|---|---|---|---|---|---
(A): | 2D coordinate-based MLP with | Biden | 5.33 | 0.072 | 10.69 | 197.2 | -
controllable dynamics (baseline, Section 3.1) | Obama | 3.74 | 0.077 | 14.33 | 132.9 |
(B): | CNN-based decoder | Biden | 6.67 | 0.089 | 11.51 | 263.8 | -
Obama | 4.27 | 0.068 | 12.78 | 127.4 |
(C): | MLP + decoder | Biden | 5.55 | 0.061 | 8.76 | 114.2 | -
Obama | 3.58 | 0.060 | 9.37 | 103.4 |
(D): | (C) + latent codes | Biden | 5.68 | 0.062 | 8.38 | 92.05 | -
Obama | 3.10 | 0.047 | 9.37 | 83.1 |
(E): | (D) + $\mathcal{L}_{rec}^{\prime}$ | Biden | 5.14 | 0.057 | 9.80 | 100.9 | 2.69
Obama | 2.80 | 0.041 | 9.97 | 82.1 | 2.31
(F): | (E) + gaze input | Biden | 5.03 | 0.054 | 9.42 | 107.2 | 1.86
(full model, Section 3.2) | Obama | 2.86 | 0.041 | 9.33 | 79.3 | 2.18
Table 5: Quantitative results of our ablation study on two separate video
portraits (Biden and Obama), of resolution $512\times 512$.

Next, we evaluate our design choices and validate the significance of
different components that make up our model. We conduct quantitative and
qualitative experiments for six variations of our system. To that end, we test
all variations on the task of reconstruction, for two separate portraits
(Biden and Obama). We start with a 2D coordinate-based MLP with controllable
dynamics, described in Section 3.1, which is used as the baseline model (A) of
our ablation study. In order to demonstrate the advantages of our proposed
architecture, we further experiment with a CNN-based decoding network only,
without the MLP network (B). Then, we couple the MLP with the decoding
network, which leads to variation (C). We form the next variations by first
adding the learnable latent variables input (D), then including the
reconstruction loss term $\mathcal{L}_{rec}^{\prime}$ (E) and finally the gaze
input (F), ending up with our full model as presented in Section 3.2. Our
numerical analysis presented in Table 5 reveals the importance of each
component, especially the huge impact of the MLP network when coupled with a
CNN-based decoding network. As can be observed, the latent variables increase
FVD scores as they help to stabilise movements between frames. Furthermore,
according to LPIPS, the $\mathcal{L}_{rec}^{\prime}$ loss improves the
perceptual similarity with ground truth data. Finally, the gaze input corrects
eye motions, something that becomes more apparent when inspecting our
supplementary video. In Fig. 6, we illustrate some examples of the visual
differences between samples generated by variations (A), (B) and our full
model (F).

##  5 Discussion

3D multi-view consistency. Theoretically, our method is not 3D aware.
Nonetheless, for the purposes of portrait reenactment this is not essential
for achieving consistent results in various head poses. Given that training
videos are captured by stationary cameras, they provide access to a single
view of the scene. This makes 3D NeRF-based methods an over-parametrisation
for such data, which end up violating 3D consistency for upper torso and
producing severe shoulder trembling [16]. We argue that 3D modeling would
require a video portrait captured by a moving camera from multiple viewpoints,
as demonstrated most recently by RingNeRF [3]. For videos captured by
stationary cameras, our 2D model is able to generate frames with consistent
appearance in diverse head poses and surpass 3D NeRF-based methods [16, 22] in
visual quality and inference speed.

Large changes in head pose. The reenactment results we presented are mainly
frontal. This is attributed to the training data, which consist mostly of
frontal poses. Actually, both 2D and 3D-based methods are limited by the head
pose variation that exists in the training video. We observed that all systems
struggle to synthesise poses out of the training data span. In Fig. 7, we
illustrate the training frame with the most extreme head pose, in terms of yaw
angle. Next to it, we show the most extreme pose synthesised by NerFACE [16]
and our method, without a substantial drop in quality. Our approach achieves
higher photo-realism.

Ethical considerations. Generative models that offer explicit control on
movements of faces could be potentially used for unethical purposes. For
example, they could be used to synthesise videos of politicians acting in a
provocative way. We would like to clarify that the intention of this paper is
to make advancements realistic full-head reenactment for benevolent
applications. We do not condone using our work to produce fake news or deceive
the public.

max yaw (training)NerFACE [16]ours Figure 7: Evaluation of larger changes in
head pose.

##  6 Conclusion

We described Dynamic Neural Portraits, a novel method for controllable video
portrait synthesis. In contrast to previous attempts, our approach does not
require renderings of 3D faces to drive synthesis and does not rely on GANs or
NeRFs. Our experiments demonstrate the superiority of our model in terms of
visual quality and run-time performance, compared to state-of-the-art video or
audio-driven systems.

Acknowledgement. The work of Stefanos Zafeiriou was partially funded by the
EPSRC Fellowship DEFORM: Large Scale Shape Analysis of Deformable Models of
Humans (EP/S010203/1).

## References

  * [1] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.  Deep speech 2: End-to-end speech recognition in english and mandarin.  In International conference on machine learning, pages 173–182. PMLR, 2016.
  * [2] Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov.  Image generators with conditionally-independent pixel synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14278–14287, 2021.
  * [3] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu.  Rignerf: Fully controllable neural 3d portraits.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20364–20373, 2022.
  * [4] Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, and Michael F Cohen.  Bringing portraits to life.  ACM Transactions on Graphics (TOG), 36(6):1–13, 2017.
  * [5] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194, 1999.
  * [6] James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou.  3d face morphable models” in-the-wild”.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 48–57, 2017.
  * [7] James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou.  Large scale 3d morphable models.  International Journal of Computer Vision, 126(2):233–254, 2018\.
  * [8] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.
  * [9] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  * [10] Zhaokang Chen and Bertram E Shi.  Appearance-based gaze estimation using dilated-convolutions.  In Asian Conference on Computer Vision, pages 309–324. Springer, 2018.
  * [11] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [12] Michail Christos Doukas, Mohammad Rami Koujan, Viktoriia Sharmanska, Anastasios Roussos, and Stefanos Zafeiriou.  Head2head++: Deep facial attributes re-targeting.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(1):31–43, 2021.
  * [13] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska.  Headgan: One-shot neural head synthesis and editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14398–14407, 2021.
  * [14] Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet.  Coin: Compression with implicit neural representations.  arXiv preprint arXiv:2103.03123, 2021.
  * [15] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  volume 40, 2021.
  * [16] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8649–8658, June 2021.
  * [17] Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormahlen, Patrick Perez, and Christian Theobalt.  Automatic face reenactment.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4217–4224, 2014.
  * [18] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou.  Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [19] Jiahao Geng, Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou.  Warp-guided gans for single-photo facial animation.  ACM Transactions on Graphics (TOG), 37(6):1–12, 2018.
  * [20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, 2014.
  * [21] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis.  arXiv preprint arXiv:2110.08985, 2021.
  * [22] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang.  Ad-nerf: Audio driven neural radiance fields for talking head synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5784–5794, 2021.
  * [23] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim.  Marionette: Few-shot face reenactment preserving identity of unseen targets.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10893–10900, 2020.
  * [24] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al.  Deep speech: Scaling up end-to-end speech recognition.  arXiv preprint arXiv:1412.5567, 2014.
  * [25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [26] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt.  Deep video portraits.  ACM Transactions on Graphics (TOG), 37(4):163, 2018.
  * [27] Mohammad Rami Koujan, Michail Christos Doukas, Anastasios Roussos, and Stefanos Zafeiriou.  Head2head: Video-based neural head synthesis.  In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 16–23. IEEE, 2020.
  * [28] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5548–5557, 2020.
  * [29] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194–1, 2017.
  * [30] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210–7219, 2021.
  * [31] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [32] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [33] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  * [34] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  arXiv preprint arXiv:2106.13228, 2021.
  * [35] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.  A 3d face model for pose and illumination invariant face recognition.  In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pages 296–301. Ieee, 2009.
  * [36] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [37] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  Animating arbitrary objects via deep motion transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2377–2386, 2019.
  * [38] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [39] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman.  Synthesizing obama: learning lip sync from audio.  ACM Transactions on Graphics (ToG), 36(4):1–13, 2017.
  * [40] Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner.  Neural voice puppetry: Audio-driven facial reenactment.  ECCV 2020, 2020.
  * [41] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  ACM TOG 2019, 2019.
  * [42] Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt.  Real-time expression transfer for facial reenactment.  ACM Trans. Graph., 34(6):183–1, 2015.
  * [43] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2016\.
  * [44] Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Nießner.  Headon: Real-time reenactment of human portrait videos.  ACM Transactions on Graphics (TOG), 37(4):1–13, 2018.
  * [45] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  arXiv preprint arXiv:1812.01717, 2018.
  * [46] Thomas Vetter and Volker Blanz.  Estimating coloured 3d face models from single images: An example based approach.  In In Proceedings, European Conference on Computer Vision, pages 499–513. Springer, 1998.
  * [47] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [48] Olivia Wiles, A Koepke, and Andrew Zisserman.  X2face: A network for controlling face generation using images, audio, and pose codes.  In Proceedings of the European conference on computer vision (ECCV), pages 670–686, 2018.
  * [49] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  In European Conference on Computer Vision, pages 524–540. Springer, 2020.
  * [50] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9459–9468, 2019.
  * [51] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [52] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian.  Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis.  arXiv preprint arXiv:2110.09788, 2021.
  * [53] Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li.  Face alignment across large poses: A 3d solution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 146–155, 2016.