# Importance of higher harmonics and $v_{4}$ puzzle in quark-gluon plasma
tomography

Dusan Zigic  Institute of Physics Belgrade, University of Belgrade, Serbia
Jussi Auvinen  Institute of Physics Belgrade, University of Belgrade, Serbia
Igor Salom  Institute of Physics Belgrade, University of Belgrade, Serbia
Pasi Huovinen  Institute of Physics Belgrade, University of Belgrade, Serbia
Incubator of Scientific Excellence—Centre for Simulations of Superdense
Fluids, University of Wrocław, Poland  Magdalena Djordjevic111E-mail:
<EMAIL_ADDRESS>Institute of Physics Belgrade, University of Belgrade, Serbia

###### Abstract

QGP tomography aims to constrain the parameters characterizing the properties
and evolution of Quark-Gluon Plasma (QGP) formed in heavy-ion collisions, by
exploiting low and high-$p_{\perp}$ theory and data. Higher-order harmonics
$v_{n}$ ($n>2$) are an important—but seldom explored—part of this approach.
However, to take full advantage of them, several issues have to be addressed:
i) consistency of different methods for calculating $v_{n}$, ii) importance of
event-by-event fluctuations to high-$p_{\perp}$ $R_{AA}$ and $v_{2}$
predictions, iii) sensitivity of higher harmonics to the initial state of
fluid-dynamical evolution. We obtain that i) several methods for calculating
harmonics are compatible with each other, ii) event-by-event calculations are
important in mid-central collisions, and iii) various initializations of the
evolution of the medium lead to quantitatively and qualitatively different
predictions, likely to be distinguished by future measurements. We also find
that the present high-$p_{\perp}$ $v_{4}$ data cannot be reproduced using
initial states for fluid-dynamical evolution given by state-of-the-art models.
We call this discrepancy high-$p_{\perp}$ $v_{4}$ puzzle at the LHC.

###### pacs:

12.38.Mh; 24.85.+p; 25.75.-q

##  I Introduction

During the past two decades, an impressive experimental and theoretical effort
has been invested in generating and exploring a new form of matter called
Quark-Gluon Plasma (QGP) QGP1 ; QGP2 ; QGP3 ; QGP4 . This form of matter
consists of interacting and no longer confined quarks, antiquarks, and gluons
Collins ; Baym  and is created at extremely high energy densities achieved in
ultra-relativistic heavy ion collisions at the Relativistic Heavy Ion Collider
(RHIC) and the Large Hadron Collider (LHC) experiments. An unprecedented
amount of data for different collision systems (large and small), collision
energies, types of particles, momentum regions, centralities, etc., are
generated in these experiments, and one of the major current goals is to
optimally use these data to investigate the properties of this exciting form
of matter.

As one of the latest experimental achievements, the high momentum
(high-$p_{\perp}$) higher harmonics have recently become available at RHIC and
the LHC. For example, for charged hadrons, the data are available up to the
$7^{th}$ harmonic (for ATLAS ATLAS_CH_vn ) and cover the $p_{\perp}$ region up
to 100 GeV (for CMS CMS_CH_vn ). For heavy flavor, the coverage is not that
extensive (for both harmonics and momentum region)—still, the upcoming
experimental data at high-luminosity LHC Run3 should provide these data for
both light and heavy flavor with much higher precision. In the upcoming RHIC
(sPHENIX and STAR) experiments, similar quality data is expected, with
$p_{\perp}$ coverage up to 20 GeV. Even if the $p_{\perp}$ range accessible at
RHIC is narrower than at the LHC, it is particularly useful for QGP tomography
due to the pronounced difference between light and heavy flavor in that
region. While these data (will) represent the state-of-the-art in the
experimental sector, theoretically the higher harmonics at high-$p_{\perp}$
have not been well explored.

To use these data for QGP tomography, i.e., for exploring the bulk QGP
properties through high-$p_{\perp}$ theory and data, one should first identify
and address potential limitations, in particular related to coverage and
design of different experiments. For example, four different methods are
commonly used in the literature to evaluate $v_{n}$: two-particle cumulant
$v_{n}\\{2\\}$, four-particle cumulant $v_{n}\\{4\\}$, event plane
$v_{n}\\{\mathrm{EP}\\}$, and scalar product $v_{n}\\{\mathrm{SP}\\}$ methods
(see section II.3 for more details). Do these methods provide consistent
results, especially when different experimental collaborations even define
$v_{n}\\{\mathrm{SP}\\}$ in different ways?

Furthermore, in experimental analysis, the scalar product method correlates
the particle of interest at midrapidity with the bulk medium constituents at
higher rapidity regions to avoid non-flow effects on measured $v_{n}$
ATLAS_CH_vn ; CMS_CH_vn . From theoretical perspective, this means the use of
the experimental definition for $v_{n}\\{\mathrm{SP}\\}$ necessitates 3+1D
hydrodynamic modeling for event-by-event simulations. However, 3+1D
simulations are computationally several orders of magnitude more demanding
than 2+1D simulations and consequently time-wise impractical for high
precision QGP tomography. Thus, the question arises whether it would be
plausible to compare $v_{n}\\{\mathrm{SP}\\}$ obtained in boost-invariant 2+1D
simulations to experimental data in a model where the high- and
low-$p_{\perp}$ particles have separate sources (fragmenting jets and a
thermal fireball, respectively), and are thus uncorrelated.

Next, for the second harmonic, $v_{2}$, event-by-event fluctuations are
expected to either have a significant effect on $v_{2}$ values
NoronhaHostler_ebe , or to be small enough to be considered negligible He_ebe
; CIBJET . However, these studies were done in limited and different
centrality regions. It is expected Betz_ebe  that the effects of event-by-
event fluctuations increase with decreasing centrality. Thus, it is important
to systematically investigate and quantify these effects for the
high-$p_{\perp}$ region at different centralities.

Therefore, the study presented in this manuscript has the following main
goals:

  1. (i)

Explore to what extent the different methods for calculating higher harmonics
are compatible with each other.

  2. (ii)

Explore the importance of event-by-event fluctuations and correlations to
high-$p_{\perp}$ $v_{2}$ and $R_{AA}$.

  3. (iii)

Explore the qualitative and quantitative effects of different medium evolution
scenarios on high-$p_{\perp}$ higher harmonics, and how well the existing
high-$p_{\perp}$ data can be reproduced without further tuning of parameters.

Overall, this study explores whether and how high-$p_{\perp}$ higher
harmonics, with an adequate theoretical framework, can provide further
constraints to the bulk QGP properties.

##  II Methods

###  II.1 Outline of DREENA-A framework

To use the high-$p_{\perp}$ particles to explore the bulk properties, we
developed a fully optimized modular framework DREENA-A DREENAA , where
“DREENA” stands for Dynamical Radiative and Elastic ENergy loss Approach,
while “A” stands for Adaptive. We further optimized the framework for this
study to efficiently incorporate any, arbitrary, event-by-event fluctuating
temperature profile within the dynamical energy loss formalism. Due to the
very large amount of temperature profile data processed in event-by-event
calculations, we optimized file handling and formats. Also, we reorganized the
parallelization of computation, as well as ensured that spatio-temporal
resolution and calculation precision are optimal and adjusted to the event-by-
event type of profiles.

The framework does not have fitting parameters within the energy loss model
(i.e., all parameters used in the model correspond to standard literature
values), which allows to systematically compare the data and predictions
obtained by the same formalism and parameter set. Therefore different
temperature profiles (which are the only input in the DREENA-A framework)
resulting from different initial states, and QGP properties, can be
distinguished by the high-$p_{\perp}$ observables they lead to, and the bulk
QGP properties can be further constrained by studying low and high-$p_{\perp}$
theory and data jointly.

The dynamical energy loss formalism MD_PRC ; DH_PRL ; MD_Coll  has several
important features, all of which are needed for accurate predictions BD_JPG :
i) QCD medium of finite size and temperature consisting of dynamical (i.e.,
moving) partons. ii) Calculations are based on generalized Hard-Thermal-Loop
approach Kapusta , with naturally regulated infrared divergences MD_PRC ;
MD_Coll ; DG_TM . iii) Both radiative MD_PRC ; DH_PRL  and collisional MD_Coll
energy losses are calculated in the same theoretical framework and apply to
both light and heavy flavors. iv) The framework is generalized toward running
coupling MD_PLB  and finite magnetic mass MD_MagnMass . We have also
investigated the validity of the widely used soft-gluon approximation sga ,
but found it a very good approximation which does not need to be relaxed.

The initial quark spectrum, for light and heavy partons, is computed at next
to leading order Vitev0912 . We use DSS DSS  fragmentation functions to
generate charged hadrons, and BCFY BCFY  and KLP KLP  fragmentation functions
for D and B mesons, respectively. To generate high-$p_{\perp}$ predictions, we
use the same parameter set as in DREENA-A DREENAA . Specifically, we assume
effective light quark flavors $n_{f}{\,=\,}3$ and $\Lambda_{QCD}=0.2$ GeV. The
temperature-dependent Debye mass $\mu_{E}$ is obtained by applying the
procedure from Peshier  and leads to results compatible with the lattice QCD
LatticeMass . For the gluon mass, we assume $m_{g}=\mu_{E}/\sqrt{2}$ DG_TM ,
and for light quark mass $M{\,=\,}\mu_{E}/\sqrt{6}$. The charm mass is
$M{\,=\,}1.2$ GeV and the bottom mass is $M{\,=\,}4.75$ GeV. For magnetic to
electric mass ratio, we use $\mu_{M}/\mu_{E}=0.5$ Maezawa ; Nakamura .

###  II.2 Modeling the bulk evolution

We investigate three different event-by-event initializations for the bulk
evolution. The first is Monte Carlo Glauber (MC-Glauber) initialization at
initial time $\tau_{0}=1.0$ fm without initial transverse flow. We assign the
binary collision points at halfway between the two colliding nucleons and
convert these points to a continuous binary collision density using 2-D
Gaussian distributions

$n_{BC}(x,y)=\frac{1}{2\pi\sigma_{BC}^{2}}\sum_{i=1}^{N_{BC}}\exp\left(-\frac{(x-x_{i})^{2}+(y-y_{i})^{2}}{2\sigma_{BC}^{2}}\right)$
(1)

with a width parameter $\sigma_{BC}=0.35$ fm. The binary collision density is
then converted to energy density with the formula

$\epsilon(x,y)=C_{0}(n_{BC}+c_{1}n_{BC}^{2}+c_{2}n_{BC}^{3}),$ (2)

and further extended in the longitudinal direction using the LHC
parametrization from Ref. Molnar:2014zha . The evolution of the fluid is
calculated using a 3+1D viscous fluid code Molnar:2014zha , with a constant
shear viscosity over entropy density ratio $\eta/s=0.03$ and no bulk
viscosity. The equation of state (EoS) parametrization is $s95p$-PCE-v1
Huovinen:2009yb . The model parameters were tuned to ALICE charged particle
multiplicity Adam:2015ptt  and $v_{n}(p_{\perp})$ data Adam:2016izf  for
10-20%, 20-30% and 30-40% centrality classes in Pb+Pb collisions at
$\sqrt{s_{NN}}=5.02$ TeV.

The second model is the TRENTo initialization Moreland:2014oya  with a free
streaming stage until $\tau_{0}=1.16$ fm, further evolved using the VISH2+1
code Song:2007ux  as described in Bernhard:2018hnz ; Bernhard:2019bmu . The
parameters in this calculation are based on a Bayesian analysis of the data at
Pb+Pb collisions at $\sqrt{s_{NN}}=2.76$ and $5.02$ TeV Bernhard:2019bmu . In
particular the calculation includes temperature dependent shear and bulk
viscosity coefficients with the minimum value of $\eta/s=0.081$ and maximum of
$\zeta/s=0.052$. The EoS Bernhard:2018hnz  is based on the lattice results by
the HotQCD collaboration HotQCD:2014kol .

The third investigated initialization model is IP-Glasma ipglasma ; ipglasma2
. The calculated event-by-event fluctuating initial states Schenke:2020mbo
are further evolved Chun-private  using the MUSIC code Schenke:2010nt ;
Schenke:2010rr ; Schenke:2011bn  constrained to boost-invariant expansion. In
these calculations, the switch from Yang-Mills to fluid-dynamical evolution
takes place at $\tau_{\mathrm{switch}}=0.4$ fm, shear viscosity over entropy
density ratio is constant $\eta/s=0.12$, and the temperature-dependent bulk
viscosity coefficient over entropy density ratio has the maximum value
$\zeta/s=0.13$. The equation of state is based on the HotQCD lattice results
HotQCD:2014kol  as presented in Ref. Moreland:2015dvc .

###  II.3 Flow analysis

####  II.3.1 Scalar product and event plane methods

We start by defining the low-$p_{\perp}$ normalized flow vector for $n$-th
harmonic based on $M$ particles as

$Q_{n}=\frac{1}{M}\sum_{j=1}^{M}e^{in\phi_{j}}\equiv|v_{n}|e^{in\Psi_{n}},$
(3)

where $\Psi_{n}$ is the event plane angle:
$\Psi_{n}=\arctan(\frac{\mathrm{Im}\,Q_{n}}{\mathrm{Re}\,Q_{n}})/n$.

Similarly to low $p_{\perp}$, we can define the flow vector for a high
$p_{\perp}$ bin as
($R_{AA}(p_{\perp})=\frac{1}{2\pi}\int_{0}^{2\pi}R_{AA}(p_{\perp},\phi)\,d\phi$)

$q_{n}^{\mathrm{hard}}=\frac{\frac{1}{2\pi}\int_{0}^{2\pi}\,e^{in\phi}R_{AA}(p_{\perp},\phi)\,d\phi}{R_{AA}(p_{\perp})},$
(4)

and single-event high-$p_{\perp}$ flow coefficients $v_{n}^{\mathrm{hard}}$ as
NoronhaHostler_ebe

$v_{n}^{\mathrm{hard}}=\frac{\frac{1}{2\pi}\int_{0}^{2\pi}\,\cos[n(\phi-\Psi_{n}^{\mathrm{hard}}(p_{\perp}))]R_{AA}(p_{\perp},\phi)\,d\phi}{R_{AA}(p_{\perp})}$
(5)

where the event plane angle $\Psi_{n}^{\mathrm{hard}}(p_{\perp})$ is defined
as

$\Psi_{n}^{\mathrm{hard}}(p_{\perp})=\frac{1}{n}\arctan\left(\frac{\int_{0}^{2\pi}\,\sin(n\phi)R_{AA}(p_{\perp},\phi)\,d\phi}{\int_{0}^{2\pi}\,\cos(n\phi)R_{AA}(p_{\perp},\phi)\,d\phi}\right).$
(6)

The high-$p_{\perp}$ $v_{n}$ is then calculated by correlating $q_{n}$ with
$Q_{n}$ NoronhaHostler_ebe ; Andres:2019eus ; He_ebe :

$v_{n}^{\mathrm{hard}}\\{\mathrm{SP}\\}=\frac{\langle\mathrm{Re}\,(q_{n}^{\mathrm{hard}}(Q_{n})^{*})\rangle_{\mathrm{ev}}}{\sqrt{\langle
Q_{n}(Q_{n})^{*}\rangle_{\mathrm{ev}}}}\\\
=\frac{\langle|v_{n}^{\mathrm{hard}}||v_{n}|\cos[n(\Psi_{n}^{\mathrm{hard}}(p_{\perp})-\Psi_{n})]\rangle_{\mathrm{ev}}}{\sqrt{\langle|v_{n}|^{2}\rangle_{\mathrm{ev}}}}.$
(7)

We may also simply calculate the high-$p_{\perp}$ anisotropy with respect to
the event plane $\Psi_{n}$, which we shall denote as the “event plane” $v_{n}$
He_ebe :

$v_{n}\\{\mathrm{EP}\\}=\langle\langle\cos[n(\phi^{\mathrm{hard}}-\Psi_{n})]\rangle\rangle_{ev}=\langle
v_{n}^{\mathrm{hard}}\cos[n(\Psi^{\mathrm{hard}}_{n}-\Psi_{n})]\rangle_{ev}.$
(8)

For our theoretical $v_{n}\\{\mathrm{SP}\\}$, the reference flow vector
$Q_{n}$ is calculated using only midrapidity particles. In order to reduce
non-flow effects, it is common in experiments to introduce a rapidity gap
between the particles of interest and the reference flow particles. ATLAS
defines the scalar product $v_{n}$ as ATLAS_CH_vn :

$v_{n}\\{\mathrm{SP}_{\mathrm{ATLAS}}\\}=\frac{\mathrm{Re}\,\langle\langle
e^{in\phi}(Q_{n}^{-|+})^{*}\rangle\rangle_{\mathrm{ev}}}{\sqrt{\langle
Q_{n}^{-}(Q_{n}^{+})^{*}\rangle}_{\mathrm{ev}}}$ (9)

where $Q_{n}^{-}=\frac{1}{M^{-}}\sum_{j=1}^{M^{-}}e^{in\phi_{j}}$ refers to
particles in the rapidity interval $-4.9<\eta<-3.2$ and $Q_{n}^{+}$ similarly
to particles in the interval $3.2<\eta<4.9$, while $e^{in\phi}$ is associated
with particles in midrapidity $|\eta|<2.5$. $Q_{n}^{-|+}$ indicates that
particle of interest with $\eta<0$ are coupled to $Q_{n}^{+}$ and particles
with $\eta>0$ to $Q_{n}^{-}$ to maximize the rapidity gap 222Since our
high-$p_{\perp}$ particles are produced at $\eta=0$, the choice of $Q_{n}^{+}$
or $Q_{n}^{-}$ for the correlation is arbitrary..

CMS definition for the scalar product is CMS_CH_vn

$v_{n}\\{\mathrm{SP}_{\mathrm{CMS}}\\}=\frac{\mathrm{Re}\,\langle
Q_{n}Q_{nA}^{*}\rangle_{\mathrm{ev}}}{\sqrt{\frac{\langle
Q_{nA}Q_{nB}^{*}\rangle_{\mathrm{ev}}\langle
Q_{nA}Q_{nC}^{*}\rangle_{\mathrm{ev}}}{\langle
Q_{nB}Q_{nC}^{*}\rangle_{\mathrm{ev}}}}}$ (10)

where the flow vector $Q_{n}=\sum_{j=1}^{M}e^{in\phi_{j}}$ consists of
particles of interest in midrapidity $|\eta|<1.0$, vectors
$Q_{nA},Q_{nB}=\sum_{j=1}^{M_{A,B}}E_{T}\,e^{in\phi_{j}}$ are measured from
the HF calorimeters at $2.9<|\eta|<5.2$, one at the negative and the other at
the positive rapidity, and the third reference vector
$Q_{nC}=\sum_{j=1}^{M_{C}}p_{\perp}\,e^{in\phi_{j}}$ is obtained from tracks
with $|\eta|<0.75$. If the particle of interest comes from the positive-$\eta$
side of the tracker, then $Q_{nA}$ is calculated using the negative-$\eta$
side of HF, and vice versa.

####  II.3.2 Cumulant method

For 2- and 4-particle cumulant analysis, we use the unnormalized flow vector:

$\tilde{Q}_{n}=\sum_{j=1}^{M}e^{in\phi_{j}}.$ (11)

The low-$p_{\perp}$ integrated reference flow is calculated using Eqs.
(7)-(18) from Ref. Bilandzic:2010jr : The 2-particle cumulant $v_{n}$ is
defined as

$v_{n}\\{2\\}=\sqrt{c_{n}\\{2\\}},$ (12)

where the second order cumulant $c_{n}\\{2\\}$ equals the event-averaged
2-particle correlation $\langle\langle 2\rangle\rangle_{\mathrm{ev}}$. The
4-particle cumulant $v_{n}$ is

$v_{n}\\{4\\}=\sqrt[4]{-c_{n}\\{4\\}},$ (13)

where $c_{n}\\{4\\}$ is the 4th order cumulant $\langle\langle
4\rangle\rangle_{\mathrm{ev}}-2\langle\langle
2\rangle\rangle^{2}_{\mathrm{ev}}$.

For a single event, the 2-particle correlation is

$\langle 2\rangle=\frac{|\tilde{Q}_{n}|^{2}-M}{W_{2}}$ (14)

with a combinatorial weight factor $W_{2}=M(M-1)$ and the single-event
4-particle correlation is

$\begin{split}\langle
4\rangle=&\frac{|\tilde{Q}_{n}|^{4}+|\tilde{Q}_{2n}|^{2}-2\mathrm{Re}|\tilde{Q}_{2n}\tilde{Q_{n}^{*}}\tilde{Q_{n}^{*}}|}{W_{4}}\\\
&-2\frac{2(M-2)|\tilde{Q}_{n}|^{2}-M(M-3)}{W_{4}}\end{split}$ (15)

with $W_{4}=M(M-1)(M-2)(M-3)$.

Using the weight factors defined above, the weighted average of a $k$-particle
correlation over multiple events is then

$\langle\langle
k\rangle\rangle_{\mathrm{ev}}=\frac{\sum\limits_{i=1}^{N_{\mathrm{events}}}W_{k,i}\,\langle
k\rangle_{i}}{\sum\limits_{i=1}^{N_{\mathrm{events}}}W_{k,i}}.$ (16)

Once the reference flow has been determined, the $p_{T}$-differential flow can
be calculated using Eqs. (20)-(35) of Bilandzic:2010jr . Here we denote the
flow vector in a $p_{\perp}$ bin with $m_{q}$ particles as

$q_{n}=\sum_{j=1}^{m_{q}}e^{in\phi_{j}}.$ (17)

For high-$p_{\perp}$ particles, $q_{n}$ is calculated from the distribution

$q_{n}=\int_{0}^{2\pi}e^{in\phi}\frac{dN}{dp_{\perp}d\phi}d\phi$ (18)

with the associated multiplicity

$m_{q}=\int_{0}^{2\pi}\frac{dN}{dp_{\perp}d\phi}d\phi.$ (19)

For high-$p_{\perp}$ differential flow, none of the particles in a $p_{\perp}$
bin are included in the calculation of the reference flow, so the weight
factors are $W^{\prime}_{2}=m_{q}M$ and $W^{\prime}_{4}=m_{q}M(M-1)(M-2)$, and
the 2-particle correlation is simply

$\langle 2^{\prime}\rangle=\frac{q_{n}\tilde{Q}_{n}^{*}}{W^{\prime}_{2}},$
(20)

while the 4-particle correlation is

$\langle
4^{\prime}\rangle=\frac{q_{n}\tilde{Q}_{n}\tilde{Q}_{n}^{*}\tilde{Q}_{n}^{*}-q_{n}\tilde{Q}_{n}\tilde{Q}_{2n}^{*}-2Mq_{n}\tilde{Q}_{n}^{*}+2q_{n}\tilde{Q}_{n}^{*}}{W^{\prime}_{4}}.$
(21)

With the knowledge of the correlations, we can calculate the differential
cumulants

$\begin{split}d_{n}\\{2\\}&=\langle\langle
2^{\prime}\rangle\rangle_{\mathrm{ev}},\\\ d_{n}\\{4\\}&=\langle\langle
4^{\prime}\rangle\rangle_{\mathrm{ev}}-2\langle\langle
2^{\prime}\rangle\rangle_{\mathrm{ev}}\langle\langle
2\rangle\rangle_{\mathrm{ev}}\end{split}$ (22)

and the differential flow

$\begin{split}v^{\prime}_{n}\\{2\\}=\frac{d_{n}\\{2\\}}{\sqrt{c_{n}\\{2\\}}},\\\
v^{\prime}_{n}\\{4\\}=-\frac{d_{n}\\{4\\}}{(-c_{n}\\{4\\})^{3/4}}.\end{split}$
(23)

##  III Results and discussion

###  III.1 Compatibility of analysis methods

Figure 1: Charged hadron $v_{2}$ (left), $v_{3}$ (middle) and $v_{4}$ (right)
in Pb+Pb collisions at $\sqrt{s_{\mathrm{NN}}}=5.02$ TeV for 20-30% centrality
class, computed using different analysis methods: 2-particle cumulant,
4-particle cumulant, event plane, midrapidity scalar product, ATLAS-defined
scalar product, and CMS defined scalar product, each described in the section
II.3. Energy loss calculation was performed on MC-Glauber+3d-hydro temperature
profiles, with $\mu_{M}/\mu_{E}=0.5$.

In Fig. 1, we compare $v_{n}(p_{\perp})$ for high-$p_{\perp}$ particles
obtained using six different methods: 2-particle cumulant $v_{n}\\{2\\}$ given
by Eq. (12), 4-particle cumulant $v_{n}\\{4\\}$ given by Eq. (13), event plane
$v_{n}\\{\mathrm{EP}\\}$ defined by Eq. (8), midrapidity scalar product
$v_{n}\\{\mathrm{SP}\\}$ calculated using Eq. (7), scalar product
$v_{n}\\{\mathrm{SP}_{\mathrm{ATLAS}}\\}$ as defined by the ATLAS
collaboration (Eq. (9)), and scalar product
$v_{n}\\{\mathrm{SP}_{\mathrm{CMS}}\\}$ as defined by the CMS collaboration
(Eq. (10)). High-$p_{\perp}$ $R_{AA}$ and $v_{n}$ predictions were obtained
using generalized DREENA-A framework with the temperature profiles calculated
using the combination of 3+1D viscous fluid code and MC-Glauber initial
conditions (i.e., the first bulk model described in the section II.2).

As illustrated in Fig. 1, different scalar product methods for evaluating the
$v_{n}$ coefficients, and the 2-particle cumulant method, lead to the same
results with $\approx 5$% level accuracy. In agreement with Refs. He_ebe ;
Shen_Photons ; CIBJET , the event plane results are also comparable to the
scalar product results deviating only $\approx 10$%, i.e., less than the
current experimental uncertainty. The only method with significantly different
results is the four-particle cumulant method $v_{n}\\{4\\}$, which is expected
to differ from $v_{n}\\{2\\}$ in the presence of event-by-event fluctuations
Betz_ebe ; Voloshin:2007pc . The equivalence of different approaches
simplifies comparison between theoretical predictions and experimental
results, since a theoretical prediction calculated using any method (with the
exception of the 4-particle cumulant method) can be directly compared to
experimental data analyzed using any method. We have also checked that, in the
scalar product method, the rapidity of particles used to calculate the
reference flow vector has a negligible impact on high-$p_{\perp}$ particle
$v_{n}$ in our framework and setup, allowing us to make meaningful
$v_{n}\\{\mathrm{SP}\\}$ data comparisons using the boost-invariant hydro
simulations. However, it must be remembered that the scalar product method
with large rapidity gap can be affected by the event plane decorrelation at
different rapidities Jia:2014vja ; CMS:2015xmx . In our approach the event
plane is the same independent of rapidity, and thus the effect of
decorrelation is not included. How the event plane depends on rapidity depends
on the model used to create the longitudinal structure of the initial state,
and since there are very few theoretical constraints for it, we leave these
studies for a later work.

###  III.2 Event-by-event fluctuations

Figure 2:  Upper panels: charged hadron $R_{AA}$ calculated using event-by-
event (ebe) fluctuating temperature profiles compared to $R_{AA}$ calculated
using a smooth temperature profile (avg). Lower panels: charged hadron
$v_{n}\\{2\\}$ and $v_{n}\\{4\\}$ calculated using event-by-event (ebe)
fluctuating temperature profiles compared to $v_{n}\\{2\\}$ and $v_{n}\\{4\\}$
calculated using a smooth temperature profile (avg). Calculation was done for
Pb+Pb collisions at $\sqrt{s_{\mathrm{NN}}}=5.02$ TeV, $\mu_{M}/\mu_{E}=0.5$,
using MC-Glauber+3d-hydro bulk evolution. Each column represents different
centrality class (from left to right: 10-20%, 20-30%, 30-40% and 40-50%).

To investigate the influence of event-by-event fluctuations on
high-$p_{\perp}$ observables, MC-Glauber initial conditions for all events
within a single centrality class were averaged (we kept reaction planes
aligned, and averaged binary collision densities before converting to energy
density, (Eq. 2)) and then evolved using the 3+1D viscous fluid code (in a
single run, instead of one run for each event). Obtained smooth temperature
profile was used to calculate high-$p_{\perp}$ predictions, and $R_{AA}$ as
well as $v_{2}\\{2\\}$ and $v_{2}\\{4\\}$ results were compared to those
obtained using full event-by-event calculations (evolved separately for each
event), see Fig 2.

We see that event-by-event fluctuations increase both $R_{AA}$ and $v_{2}$.
While the effect on the $R_{AA}$ values is rather small ($\approx 7\%$) and
does not have clear centrality dependence, the effect on $v_{2}\\{2\\}$ is
more pronounced and increases with decreasing centrality. Quantitatively, we
obtain that the average difference between event-by-event $v_{2}\\{2\\}$ and
$v_{2}\\{2\\}$ calculated using smooth temperature profile goes from 14% for
the 40-50% centrality class to 32% in the 10-20% centrality class. The
observed centrality dependence can be explained by the fact that with the
increase in centrality, the influence of geometry on $v_{2}\\{2\\}$ becomes
larger, while at low centralities, event-by-event fluctuations have the
dominant impact on $v_{2}\\{2\\}$. We also observe a $p_{\perp}$ dependence of
these differences (generally decreasing with increasing $p_{\perp}$) and no
notable difference between $v_{2}\\{2\\}$ and $v_{2}\\{4\\}$ when calculated
on the smooth temperature profile, where initial state eccentricity
fluctuations are absent.

###  III.3 Effects of initial state

Figure 3: Charged hadron $R_{AA}$ (first row) $v_{2}$ (second row), $v_{3}$
(third row) and $v_{4}$ (fourth row) in Pb+Pb collisions at
$\sqrt{s_{\mathrm{NN}}}=5.02$ TeV for different initializations of the QGP
evolution (indicated in the legend). Theoretical predictions, obtained using
SP method, are compared to CMS CMS_CH_RAA ; CMS_CH_vn  (blue squares), ALICE
ALICE_CH_RAA ; ALICE_CH_vn  (red circles) and ATLAS ATLAS_CH_RAA ; ATLAS_CH_vn
(green triangles) data. Columns 1-4 correspond to, respectively, 10-20%,
20-30%, 30-40% and 40-50% centrality classes. $\mu_{M}/\mu_{E}=0.5$.

To demonstrate the applicability of high-$p_{\perp}$ theoretical predictions
as a QGP tomography tool, we generated three different sets of temperature
profiles using three different initial conditions and hydrodynamics codes.
Generalized DREENA-A DREENAA  was then used to calculate high-$p_{\perp}$
predictions, which are compared to experimental data and, for charged hadrons,
presented in Fig. 3, and for D and B mesons in Fig. 4. As can be seen,
different initializations of fluid-dynamical evolution lead to different
high-$p_{\perp}$ predictions for both $R_{AA}$ and $v_{2}$, $v_{3}$ and
$v_{4}$, even though they all provide good agreement with low-$p_{\perp}$
data. Specific differences are visible already on the level of $R_{AA}$
values, where the IP-Glasma model results in discernibly stronger suppression.
The differences in predictions become even higher when we consider the $v_{2}$
observable, with TRENTo leading to lower $v_{2}$ than IP-Glasma, while MC-
Glauber predictions are far above the two. A similar magnitude of relative
differences is also obtained for $v_{3}$ and $v_{4}$ predictions, with an
additional qualitative signature appearing for these observables: we notice
that some initializations lead to negative values of high-$p_{\perp}$ $v_{3}$
and $v_{4}$, i.e., models can differ even in the expected sign of the flow
coefficients.

Since DREENA-A does not have fitting parameters in the energy loss (the only
inputs are the temperature profile and binary collisions, which come as a
direct output from fluid-dynamical calculation and the initial state model),
Figs. 3 and 4 demonstrate that high-$p_{\perp}$ $R_{AA}$ and higher harmonics
can distinguish between different initializations and temperature profiles,
and subsequently further constrain their parameters. Furthermore, Fig. 4
suggests that heavy flavor high-$p_{\perp}$ observables are even more
sensitive to different temperature profiles than the light flavor. We also see
that predictions for high-$p_{\perp}$ higher harmonics can be either positive
or negative. Thus, the high-$p_{\perp}$ sector can provide both quantitative
and qualitative constraints for different initial states.

Figure 4: D meson (left $4\times 2$ panel) and B meson (right $4\times 2$
panel) predictions in Pb+Pb collisions at $\sqrt{s_{\mathrm{NN}}}=5.02$ TeV
for different initializations of QGP evolution (indicated in the legend). In
each $4\times 2$ panel, first row corresponds to $R_{AA}$, the second, third,
fourth to $v_{2}$, $v_{3}$, $v_{4}$, respectively, while the left (right)
column corresponds to 10-30% (30-50%) centrality class. D meson theoretical
predictions are compared to CMS CMS_D_vn  (blue squares) and ALICE ALICE_D_RAA
; ALICE_D_vn  (red circles) data, while B meson predictions are compared to
preliminary CMS CMS_B_vn  (blue squares) and preliminary ALICE ALICE_B_RAA
(red circles) data for non-prompt D meson from b decay. $\mu_{M}/\mu_{E}=0.5$.

Presently, of the considered models, the best agreement is observed for MC-
Glauber. This result is compatible with our earlier findings Stojku:2020wkh ,
where the best agreement with high-$p_{\perp}$ data was found by delaying the
start of transverse expansion and energy loss to time $\tau_{0}\approx 1.0$
fm. However, all models seem to vastly underestimate the $v_{4}$ values,
though the error bars for the available $v_{4}$ data are quite large. If this
tendency is preserved in future high luminosity experiments (e.g., in LHC run
3), it will present a new “high-$p_{\perp}$ $v_{4}$ puzzle”, whose solution
will require modifications to the present initial state models and/or energy
loss mechanisms. Additionally, better quality heavy flavor data are needed,
especially D and B-meson data, as they present valuable constraint to the
evolution of the medium.

##  IV Summary

We obtained four main conclusions in this work: i) We found that different
methods to calculate higher harmonics at high-$p_{\perp}$ are compatible with
each other within $\approx 5$–10$\%$ accuracy, which is less than the current
experimental uncertainties. ii) Event-by-event calculations are particularly
important for high-$p_{\perp}$ $v_{2}$ in mid-central collisions. iii)
Predictions for high-$p_{\perp}$ observables, and especially for higher
harmonics, are sensitive to the initial state of fluid-dynamical evolution,
and can distinguish between different initial state models. iv) All initial
state models lead to way smaller high-$p_{\perp}$ $v_{4}$ than experimentally
observed, and this disparity deserves to be called a “$v_{4}$ puzzle”.
Overall, the higher harmonics provide an exciting opportunity to obtain
further constraints to the QGP properties and its evolution in heavy-ion
collisions by combining new theoretical developments (with the corresponding
predictions) and upcoming higher luminosity experimental measurements.

Acknowledgments: We thank Chun Shen for sharing his results with us. We thank
Marko Djordjevic, Bojana Ilic and Stefan Stojku for useful discussions. This
work is supported by the European Research Council, grant ERC-2016-COG:
725741, and by the Ministry of Science and Technological Development of the
Republic of Serbia. PH was also supported by the program Excellence
Initiative–Research University of the University of Wrocław of the Ministry of
Education and Science.

## References

  * (1) E. V. Shuryak, Nucl. Phys. A 750, 64 (2005); Rev. Mod. Phys. 89, 035001 (2017).
  * (2) M. Gyulassy and L. McLerran, Nucl. Phys. A 750, 30 (2005).
  * (3) B. Jacak and P. Steinberg, Phys. Today 63, 39 (2010).
  * (4) B. Muller, J. Schukraft and B. Wyslouch, Ann. Rev. Nucl. Part. Sci. 62, 361 (2012).
  * (5) J. C. Collins and M. J. Perry, Phys. Rev. Lett. 34, 1353 (1975).
  * (6) G. Baym and S. A. Chin, Phys. Lett. B 62, 241 (1976).
  * (7) M. Aaboud, et al. [ATLAS], Eur. Phys. J. C 78, 997 (2018).
  * (8) A. M. Sirunyan, et al. [CMS], Phys. Lett. B 776, 195 (2018).
  * (9) J. Noronha-Hostler, B. Betz, J. Noronha and M. Gyulassy, Phys. Rev. Lett. 116, 252301 (2016).
  * (10) Y. He, W. Chen, T. Luo, S. Cao, L. G. Pang and X. N. Wang, [arXiv:2201.08408 [hep-ph]].
  * (11) S. Shi, J. Liao and M. Gyulassy, Chin. Phys. C 43, 044101 (2019).
  * (12) B. Betz, M. Gyulassy, M. Luzum, J. Noronha, J. Noronha-Hostler, I. Portillo and C. Ratti, Phys. Rev. C 95, 044901 (2017).
  * (13) D. Zigic, I. Salom, J. Auvinen, P. Huovinen and M. Djordjevic, [arXiv:2110.01544 [nucl-th]].
  * (14) M. Djordjevic, Phys. Rev. C 80, 064909 (2009).
  * (15) M. Djordjevic and U. Heinz, Phys. Rev. Lett. 101, 022302 (2008).
  * (16) M. Djordjevic, Phys. Rev. C 74, 064907 (2006).
  * (17) B. Blagojevic and M. Djordjevic, J. Phys. G 42, 075105 (2015).
  * (18) J. I. Kapusta, Finite-Temperature Field Theory (Cambridge University Press, 1989).
  * (19) M. Djordjevic and M. Gyulassy, Phys. Rev. C 68, 034914 (2003).
  * (20) M. Djordjevic and M. Djordjevic, Phys. Lett. B 734, 286 (2014).
  * (21) M. Djordjevic, Phys. Lett. B 709, 229 (2012).
  * (22) B. Blagojevic, M. Djordjevic and M. Djordjevic, Phys. Rev. C 99, 024901 (2019).
  * (23) Z. B. Kang, I. Vitev and H. Xing, Phys. Lett. B 718, 482 (2012); R. Sharma, I. Vitev and B.W. Zhang, Phys. Rev. C 80, 054902 (2009).
  * (24) D. de Florian, R. Sassot and M. Stratmann, Phys. Rev. D 75, 114010 (2007).
  * (25) M. Cacciari, P. Nason, JHEP 0309, 006 (2003), E. Braaten, K.-M. Cheung, S. Fleming and T. C. Yuan, Phys. Rev. D 51, 4819 (1995).
  * (26) V. G. Kartvelishvili, A.K. Likhoded, V.A. Petrov, Phys. Lett. B 78, 615 (1978).
  * (27) A. Peshier, hep-ph/0601119 (2006).
  * (28) O. Kaczmarek, F. Karsch, F. Zantow and P. Petreczky, Phys. Rev. D 70, 074505 (2004); O. Kaczmarek and F. Zantow, Phys. Rev. D 71, 114510 (2005).
  * (29) Yu. Maezawa et al. [WHOT-QCD Collaboration], Phys. Rev. D 81 091501 (2010).
  * (30) A. Nakamura, T. Saito and S. Sakai, Phys. Rev. D 69, 014506 (2004).
  * (31) E. Molnar, H. Holopainen, P. Huovinen and H. Niemi, Phys. Rev. C 90, 044904 (2014).
  * (32) P. Huovinen and P. Petreczky, Nucl. Phys. A 837, 26-53 (2010).
  * (33) J. Adam et al. [ALICE Collaboration], Phys. Rev. Lett. 116, 222302 (2016).
  * (34) J. Adam et al. [ALICE Collaboration], Phys. Rev. Lett. 116, 132302 (2016).
  * (35) J. S. Moreland, J. E. Bernhard and S. A. Bass, Phys. Rev. C 92, 011901 (2015).
  * (36) H. Song and U. W. Heinz, Phys. Rev. C 77, 064901 (2008).
  * (37) J. E. Bernhard, arXiv:1804.06469 [nucl-th].
  * (38) J. E. Bernhard, J. S. Moreland and S. A. Bass, Nature Phys. 15, no.11, 1113-1117 (2019).
  * (39) A. Bazavov et al. [HotQCD], Phys. Rev. D 90, 094503 (2014).
  * (40) B. Schenke, P. Tribedy and R. Venugopalan, Phys. Rev. Lett. 108, 252301 (2012).
  * (41) B. Schenke, P. Tribedy and R. Venugopalan, Phys. Rev. C 86, 034908 (2012).
  * (42) B. Schenke, C. Shen and P. Tribedy, Phys. Rev. C 102, 044905 (2020)
  * (43) C. Shen, private communication (2020).
  * (44) B. Schenke, S. Jeon and C. Gale, Phys. Rev. C 82, 014903 (2010).
  * (45) B. Schenke, S. Jeon and C. Gale, Phys. Rev. Lett. 106, 042301 (2011).
  * (46) B. Schenke, S. Jeon and C. Gale, Phys. Rev. C 85, 024901 (2012).
  * (47) J. S. Moreland and R. A. Soltz, Phys. Rev. C 93, 044913 (2016).
  * (48) C. Andres, N. Armesto, H. Niemi, R. Paatelainen and C. A. Salgado, Phys. Lett. B 803, 135318 (2020).
  * (49) A. Bilandzic, R. Snellings and S. Voloshin, Phys. Rev. C 83, 044913 (2011).
  * (50) J. F. Paquet, C. Shen, G. S. Denicol, M. Luzum, B. Schenke, S. Jeon and C. Gale, Phys. Rev. C 93, 044906 (2016).
  * (51) S. A. Voloshin, A. M. Poskanzer, A. Tang and G. Wang, Phys. Lett. B 659, 537 (2008).
  * (52) J. Jia and P. Huo, Phys. Rev. C 90, 034905 (2014).
  * (53) V. Khachatryan et al. [CMS], Phys. Rev. C 92, 034911 (2015).
  * (54) V. Khachatryan et al. [CMS], JHEP 04, 039 (2017).
  * (55) S. Acharya et al. [ALICE], JHEP 11, 013 (2018).
  * (56) S. Acharya et al. [ALICE], JHEP 07, 103 (2018).
  * (57) [ATLAS], ATLAS-CONF-2017-012.
  * (58) A. M. Sirunyan et al. [CMS], Phys. Lett. B 816, 136253 (2021).
  * (59) S. Acharya et al. [ALICE], JHEP 01, 174 (2022).
  * (60) S. Acharya et al. [ALICE], Phys. Lett. B 813, 136054 (2021).
  * (61) contribution link: https://indico.cern.ch/event/895086/contributions/4314625/
  * (62) contribution link: https://indico.cern.ch/event/895086/contributions/4715758/
  * (63) S. Stojku, J. Auvinen, M. Djordjevic, P. Huovinen and M. Djordjevic, Phys. Rev. C 105, L021901 (2022).


# Scaling Neural Face Synthesis to High FPS and Low Latency by Neural Caching

Frank Yu

<EMAIL_ADDRESS>Sid Fels
University of British Columbia (UBC)
<EMAIL_ADDRESS>

Helge Rhodin

<EMAIL_ADDRESS>
###### Abstract

Recent neural rendering approaches greatly improve image quality, reaching
near photorealism. However, the underlying neural networks have high runtime,
precluding telepresence and virtual reality applications that require high
resolution at low latency. The sequential dependency of layers in deep
networks makes their optimization difficult. We break this dependency by
caching information from the previous frame to speed up the processing of the
current one with an implicit warp. The warping with a shallow network reduces
latency and the caching operations can further be parallelized to improve the
frame rate. In contrast to existing temporal neural networks, ours is tailored
for the task of rendering novel views of faces by conditioning on the change
of the underlying surface mesh. We test the approach on view-dependent
rendering of 3D portrait avatars, as needed for telepresence, on established
benchmark sequences. Warping reduces latency by 70$\%$ (from 49.4ms to 14.9ms
on commodity GPUs) and scales frame rates accordingly over multiple GPUs while
reducing image quality by only 1$\%$, making it suitable as part of end-to-end
view-dependent 3D teleconferencing applications. Our project page can be found
at: https://yu-frank.github.io/lowlatency/.

##  1 Introduction

Figure 1:  Neural Rendering with Warping. While a single frame method requires
a deep neural network to synthesize a realistic head image from rough
geometry, our implicit warping yields low latency for a new frame at $t+1$
with a shallow network.

Telepresence via photo-realistic 3D avatars promises to better connect people.
Recent advances in neural rendering already enable near photo-realistic image
quality, but the underlying deep neural networks limit the best possible
latency with their sequential, layer-wise processing. This is a problem for
virtual reality applications as these require low latency upon head motion of
the user to convey an immersive experience. For example, [34] creates a high-
fidelity system using 120Hz projectors and user viewpoint tracking with a
tracker having 60Hz updates and 3ms latency to minimize users’ perception of
warping of the scene when they move. However, none of the existing neural
renderers reaches the required _motion to photons latency_ , i.e., the time it
takes from the user input, such as moving the head in VR, until the display
updates. It is henceforth an open problem to find improved neural models that
strike a better trade-off between speed and image quality.

We develop a parallel implementation, inspired by previous approaches for
improving the frame rate for other video processing tasks [3] by spreading the
computation over several frames over multiple GPUs. By itself, parallel
execution only improves frame rate, not the latency that is so crucial for VR,
since the effective network depth of sequentially executed layers remains the
same. To nevertheless reach the desired latency reduction and frame rate, we
combine parallel execution with a dedicated warp layer that is tailored for
neural face synthesis and acts as a skip connection between consecutive
frames.

This neural caching approach re-uses information from the previous time step
to improve latency from the current one with a shallow network for image
generation and a deep network for computing the cache while waiting for the
next input frame. Note that this caching strategy by itself already improves
latency on a single GPU and scales in frame rate on increasingly parallel
hardware by breaking the sequential layer dependencies and offloading the
image cache generation into multiple threads.

The difficulty lies in finding the right representation for the cache to
succeed with a low-capacity network. Our approach is inspired by classical
low-latency rendering on VR displays [28], which compute only the position of
objects for in-between frames and warp color from the previous. However,
explicit warping does not apply well to neural renderers where the underlying
geometry is approximate and the neural texture is high-dimensional making
warping operations much more costly.

To warp the neural representation, we introduce an implicit warp that provides
a skip connection that is tailored for neural rendering by taking into account
the head model parameterization. The result is a warp network that models the
image-space motion from one frame to the next given the desired viewpoint and
head model parameters. Figure 1 outlines the main components.

Our design is geared towards _novel-view-synthesis_ of a talking head given a
dynamically changing viewpoint, such as the user’s head motion in VR. We build
upon the deferred neural renderer (DNR) [25] that uses a neural texture
learned at training time. Our contributions towards scaling frame rate and
latency on parallel hardware are:

  * •

Demonstrating that the proposed neural caching can reduce latency by up to 70%
with minimal degradation in image quality (only 1% PSNR).

  * •

Extending a DNR to generate a high-resolution output with low latency via
caching and an implicit warping.

  * •

Developing a parallel scheduler that supports warping and synchronizes
multiple threads using queues.

  * •

Refining the representation of facial expression, head pose, and camera angle
to improve the implicit warp.

  * •

Adding head-stabilization and tweaking the backbone and training strategy for
noisy real-world conditions.

Our experiments highlight the importance of how to cache as well as what and
how information from the current frame is passed to the shallow _implicit
warping network_. Figure 2 compares the most related methods. Ours strikes the
best latency and FPS improvement with the least image quality trade-off.

Figure 2:  Performance vs. latency (left) and fps (right) for our models and
the baselines on the _beard_ dataset. Optimal performance is in the top-left
and top-right corners, indicated by the yellow star in each respective figure.
The vertical blue line marks the latency and fps of [33]. Models suitable for
parallel execution are run using two GPUs.

##  2 Related Work

In this section we discuss recent high-quality image generation methods and
contrast with those that optimize runtime with a focus on face generation.

Photo-realistic synthesis. High-quality photorealistic rendering is booming,
using either implicit scene representations such as Neural Radiance Fields
(NeRFs) [20] or deep neural networks trained on GAN objectives [15, 16], which
can also be conditioned on viewpoint and pose changes [5, 24]. However, these
implicit models all rely on very deep neural networks that do not run at high-
enough frame rates or are limited to static scenes via pre-computed
acceleration structures [32].

Talking head models. For dynamic faces, it is most common to start from a
parametric face model that parameterizes identity and expression in the form
of blend shapes that are linearly combined with the full mesh [1, 18]. The
coefficients of these models are low-dimensional and therefore suitable for
driving avatars in telepresence and for reenactment by mixing identity and
pose information of two subjects. While earlier models focus on only modeling
the facial region [30, 26, 27, 23, 25] conditioning on a complete head model,
such as FLAME [18], enables novel view synthesis of side views [10] suitable
for view-dependent 3D systems[36]. We follow this line of work and extend it
by reducing the rendering latency for interactive applications.

Another line of work developed subject-agnostic face synthesis [22, 21, 9, 8]
that is conditioned on single or multiple images instead of a subject-specific
3D model. However, this comes at the price of reduced details and entanglement
of expressions and head motion [31], particularly if the reference and target
poses differ largely, and may require explicit warping operations that are
relatively costly and ill-defined for occluded regions [33]. Although [33] is
already extremely fast on medium image resolution with 4ms runtime, there is
still room for improvement when rendering high-resolution images and medium
resolution on embedded devices.

Caching approaches. Re-using information from the previous time step has been
used for many computer vision tasks, among them: object detection [19], video
action recognition [6], and segmentation [11]. Carreira et al. [3] gives an
excellent overview of different architectures for video processing, including:
_i)_ _depth parallel_ architectures that execute a deep neural network over
several iterations, leading to a delay equal to the depth of the layers; _ii)_
_depth parallel + skip_ in which the head (last couple of network layers) of a
depth parallel network is updated with the new input via a skip connection;
and, _iii)_ _multi-rate clock_ architectures in which the input features for
the head are not updated at every time step and the head and backbone operate
at different clock rates. Our caching approach follows the multi-rate clock
pattern. However, none of the existing parallel models has demonstrated image
generation. Our contribution is to tailor this general concept to the problem
of novel-view-synthesis of faces via neural rendering by a suitable form of
warping.

##  3 Preliminaries

Our goal is an efficient neural renderer that outputs an image
$\mathbf{I}\in\mathbb{R}^{H\times W}$ of a face parameterized as a surface
mesh with vertices $\mathbf{v}\in\mathbb{R}^{3\times K}$, triangle indices
$\mathbf{i}\in\mathbb{R}^{3\times K}$, and associated texture coordinates
$\mathbf{u}\in\mathbb{R}^{2\times K}$ and neural texture
$\mathbf{N}\in\mathbb{R}^{D\times H\times W}$ [25]. These are the same inputs
that a normal renderer would expect, except that the texture $\mathbf{N}$ is
$D$-dimensional instead of storing three color values.

Deferred neural rendering. Our starting point is the deferred neural renderer
introduced by Thies et al. [25], which approximates the complex and
computationally expensive rendering equation with a convolutional neural
network $G$. Figure 1 gives an overview including the differences of [25]
applied to our full model including caching. Initially, a rasterizer renders
UV maps $\mathbf{U}\in\mathbb{R}^{2\times H\times W}$ of the textured mesh.
These are of the same dimension as the output image and store for every pixel
the corresponding texture coordinate. Sampling these locations from
$\mathbf{N}$ gives the feature map $\mathbf{F}\in\mathbb{R}^{D\times H\times
W}$. For classical deferred rendering, we would sample from a color texture
and combine it with light position information to form the final image. In the
case of the neural renderer, the texture has more than three channels forming
learnable features. The network $G$ turns $\mathbf{F}$ into the final image,
replacing geometric illumination computations and material shaders in
classical rendering. This forward pass is traced with blue arrows in Figure 1
while the backwards information flow during training is marked in green.

Training and face reconstruction. The parameters of the involved neural
network $G$ as well as the neural texture itself are trained on a large
dataset that has examples of the input 3D mesh and a high resolution image of
the face—the desired output. We use real videos of the person as input and
reconstruct vertices $\mathbf{v}\in\mathbb{R}^{3\times K}$, expression
coefficients $\mathbf{e}\in\mathbb{R}^{50}$, PCA shape parameters
$\mathbf{s}\in\mathbb{R}^{100}$, and head pose $\theta\in\mathbb{R}^{6}$ of
the FLAME parametric model [18] alongside camera position
$\mathbf{p}\in\mathbb{R}^{3}$ using the off-the-shelf estimator DECA [7].
Internally, it is using the 2D keypoint detector from [2]. The reconstructed
face overlays well with the image when re-projected, but details, such as hair
and ears are often misaligned, which puts a larger burden on the neural
renderer to synthesize these. This reconstruction step is marked with a red
arrow in Figure 1.

The generator $G$ is a U-Net and the neural texture $\mathbf{N}$ is
initialized at random and subsequently optimized by backpropagation to store
details of the training object locally. The loss function is the L1 difference
between the rendered and the reference image in the dataset. This backwards
pass is traced with green arrows in Figure 1. Training is on cropped images,
which speeds up training.

Base architecture. We use the 10-layer U-Net as in [12] and a multi-scale
neural texture with four levels of detail as in [25]. Furthermore, to better
model viewpoint-dependent effects, the view direction is projected to 9
spherical harmonics coefficients which are subsequently multiplied to channels
4 through 12 of the feature map. This enables explicit encoding of view-
dependent effects similar to positional encoding [29].

Real-time viewpoint-dependent rendering. Our primary application fields is 3D
teleconferencing, where a person must be rendered at a high frame rate, with
low latency, from the viewpoint of the user that is roughly frontal. Given a
new view direction, e.g., from an eye-tracker, our focus is on generating a
natural looking image of this novel view as quickly as possible to mitigate
motion sickness, reduce warping artifacts [35], and avoid discomfort. The
object motion capture could be performed offline or through a slower channel,
as usually only the viewpoint-dependent rendering demands the low-latency. In
telepresence applications, bandwidth is dictated by the size of the FLAME
model; estimated on the source side, transferred, and rendered by our system
given a new view from the receiver.

Figure 3:  Our low latency pipeline caches the previous frame (features from
the generator (left) marked orange and red) and the pose of the person to
generate the subsequent frames with only two additional up-convolution layers
(right) as soon as a new view direction is available. The generator and
warping networks are learned end-to-end and can be applied in parallel at
inference time.

##  4 Method

In this section, we introduce our neural caching approach, and propose two
variants that operate on single and multi-GPU systems and are respectively
tuned for latency and frame rate. We cache information from the immediately
preceding frames. Thereby, the motion that must be bridged from the cached
information to the current frame is small, which allows us to introduce an
implicit warp that attains maximum performance.

#### Neural Cache.

For our neural caching, we first run the deep and slow image generator
$G(\mathbf{N}_{t},\mathbf{U}_{t},\allowbreak\mathbf{p}_{t})$ that is
conditioned on the neural texture $\mathbf{N}$, UV-map $\mathbf{U}_{t}$ and
view direction (e.g., user’s head motion in VR) of the current frame $t$.
Figure 3 provides a detailed overview of our pipeline. We cache features
$\mathbf{C}^{(3)}_{t}$, $\mathbf{C}^{(4)}_{t}$, and $\mathbf{C}^{(5)}_{t}$
from the last 3 layers of the generator together with camera position
$\mathbf{p}_{t}$, and spherical harmonics (SH) encoding of the pose,
$\mathbf{h}_{t}^{\text{obj}}$. Furthermore, we add the UV map
$\mathbf{U}_{t}$, expression $\mathbf{e}_{t}$, and the pose $\theta_{t}$ that
are specific to rendering faces. Formally we write the combined cache
$\mathbf{C}$ as

$\mathbf{C}_{t}:=[\mathbf{C}^{(3)}_{t},\mathbf{C}^{(4)}_{t},\mathbf{C}^{(5)}_{t},\theta_{t},\mathbf{p}_{t},\mathbf{e}_{t},\mathbf{h}_{t}^{\text{obj}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color<EMAIL_ADDRESS>(1)

Implicit Warping. Previous work used an explicit warp operation from a
reference frame, which requires preceding neural network layers to predict the
warp and is implemented as a texture sampling step that is relatively slow due
to random access patterns for every pixel. We propose an implicit warp with a
neural network $W$ that is composed of only two up-convolution layers. These
two layers take in cached information $\mathbf{C}_{t}$, rendered UV map and
the new camera, pose, and expression from the new frame $t+1$, to reconstruct
the image $I_{t+1}$. This network is intentionally kept shallow to decrease
the latency of image generation.

In addition, we found in a detailed ablation study that giving as input the
new camera position $\mathbf{p}_{t+1}$, object pose $\theta_{t+1}$, expression
$\mathbf{e}_{t+1}$ and their differences to the previous frame, via a single-
layer MLP, $M$, works best. This yields

$\bar{\theta}_{t+1}=M(\mathbf{p}_{t+1},\Delta\mathbf{p}_{t+1},\theta_{t+1},\Delta\theta_{t+1},\mathbf{e}_{t+1}\Delta,\mathbf{e}_{t+1},\mathbf{h}_{t+1}^{\text{obj}}),$
(2)

where the $\Delta$ refers to the change in a quantity between two frames, here
$\theta_{t+1}-\theta_{t}$. Additionally, we also include the UV map
$\mathbf{U}_{t+1}$. Together with the cached features processing by the first
warping layer $W_{1}$ gives

$\mathbf{F}_{1}=W_{1}(\mathbf{C}^{(3)}_{t},\mathbf{C}^{(4)}_{t}\Delta\mathbf{h}_{t+1}^{\text{obj}},\bar{\theta}_{t+1},\Delta\mathbf{U}_{t+1})$
(3)

where $\Delta\mathbf{h}_{t+1}$ and $\bar{\theta}_{t+1}$ are broadcasted to the
resolution of the cache and $\Delta\mathbf{h}_{t+1}$ modulates the feature map
by multiplication, similar to how positional encoding works, but here for
rotational changes. These features are further processed to output the image
with
$I_{t+1}=W_{2}(\mathbf{F}_{1},\mathbf{C}^{(5)}_{t}\Delta\mathbf{h}_{t+1}^{\text{obj}})$.
Note that $W$ warps the previous to the current frame, but without a spatial
transformation layer—it does so implicitly via a local approximation of how
the image changes with respect to changes in pose.

Figure 4:  Our rendering pipelines for sequential (Top) and parallel execution
(Bottom). A network $W$ warps a cache from the previous frame as soon as a new
viewpoint $V$ is ready for rendering, producing the output image much quicker
than the full generator $G$ could. Leveraging two threads further increases
the output frame rate.

###  4.1 Operation Modes and Parallel Execution

Our approach scales easily between single and parallel GPU execution. When
only a single GPU is available, Figure 4-top visualizes the principle of
reducing latency with a shallow warp network, compared to running the slower
generator. Here, the cache is updated in sequence with the warp, thereby
reducing latency but not frame rate.

When multiple GPUs are available, we can combine the proposed implicit warp
with parallel execution, thereby rendering at a two or more times higher frame
rate by warping a single or multiple images
$\mathbf{I}_{t},\mathbf{I}_{t+1},\cdots$ while the generator runs on a
separate GPU on a separate thread. Figure 4-right showcases this scheme.
Notably, even though the warping is an approximation, multiple warping does
not reduce image quality when operating online on high-frame rate video
streams. The higher processing speed of warping twice in parallel reduces the
distance between frames that can be processed and thereby makes the two-frame
warp as difficult as a single-frame warp operating at half the frame rate.

We experimented with different job assignment and synchronization schemes
between the two threads. We found that threads alternating between image
generation and warping is most efficient and eases implementation. In this
model, the main thread distributes newly arriving viewpoint information to the
two worker threads via a queue, each associated to one GPU. These in turn wait
for new data in the queue. Upon receiving new data, they alternate between
caching and warping as visualized in Figure 4-bottom. Thereby, the warping is
executed on the same GPU as the generator, such that the cache can remain on
the same GPU. The alternative of using a dedicated warp and cache thread has a
much lower performance since the cache would have to be moved from one GPU to
CPU and then again from CPU to the target GPU. The required synchronization of
ques has a negligible overhead in our implementation on two RTX 2080s with
only 0.25ms/frame. Moreover, our preliminary attempt of using manual locks
instead of queues was more complex without improving performance. We will make
our implementation publicly available to facilitate further research.

###  4.2 Improved Neural Head Rendering

Our starting point, DNR [25], is a general rendering approach. In the
following, we explain our architectural changes towards tuning it for face
synthesis.

Head stabilization To reduce jitter and flicker of the head, we ensure that
the virtual camera that we use to generate our UV masks is always centered on
the subject’s head and has consistent scale when generating videos by
centering the camera on the head midpoint and scaling by the projected ear-to-
ear distance. Because the driving motion capture signal is often unstable, we
further smooth the global head position $\mathbf{p}_{t}$ with a delayed
Gaussian filter of size five and fix the identity $\mathbf{s}$ of the FLAME
model to be the mean identity estimated by DECA on the training set. To
maintain facial expressions and lip motion faithfully, the jaw orientation in
$\theta$ and expression parameters $\mathbf{e}$ are left untouched.

Loss Function For our baselines which only predict the image at time $t$ we
define the loss function as

$L_{\text{train}}=\lambda_{\text{tex}}L_{\text{tex}}+\lambda_{\text{img}}L_{\text{img}}+\lambda_{\text{p}}L_{\text{p}},$
(4)

which measures the L1 distance between the first three channels of the sampled
neural texture and the ground truth image ($L_{\text{tex}}$), the L1
photometric loss ($L_{\text{img}}$), and the perceptual loss [13] between the
predicted image and ground truth image ($L_{p}$). We weight these loss terms
with coefficients 1, 1, and 0.1. For the warping we add $\lambda_{\text{img}}$
and $L_{\text{p}}$ on the future frame ${t+1}$ and down-weight the existing
$L_{\text{img}}$ by 0.1 to put much more weight on the prediction for
$I_{t+1}$ that is used at inference time.

Architectural Changes. Based on the work of [14] we make the following
improvements to the base DNR [25] network to improve output image quality.
First, we replace the transpose convolutions in the latter half of of the
U-Net with a bilinear upsample layer followed by a 2D convolution (up-
convolution). This has been shown to increase the final image quality of
output and reduce grid-like noise in reconstructed images. Furthermore, we
apply a Gaussian low pass filter (LPF) on the smallest spacial features of the
U-Net architecture. We refer to the baselines that utilize all these
improvements as DNR+.

##  5 Experiments

We evaluate our Neural Warping technique with respect to our goal of
maximizing image quality and minimizing the latency by applying our Neural
Warping. Figure 2 summarizes our main results on the possible trade-offs
between accuracy vs. latency and accuracy vs. fps, comparing our two variants
to the most related work and showing an improvement in fps of up to $300\%$
and a reduction in latency of $70\%$. We also provide an ablation study to
identify specific trade-offs with respect to individual components of our
warping network. The supplemental material contains additional results.

Baselines. DNR [25] is the backbone we use as our reference. DNR+ improves DNR
with recent neural network architecture improvements from [14]. We furthermore
add DNR++ that uses the current and past frame as input. DNR+ and DNR++ both
act as a theoretical upper bound for our model’s image quality and therefore,
provide a good measure of the effectiveness and efficiency of our approach. We
also compare against the recent method from Wang et al. [31], using their
online interface (http://imaginaire.cc/vid2vid-cameo/) and in terms of runtime
to Zakharov et al. [33] using the same $512\times 512$ image resolution. We do
not provide PSNR numbers as it was designed for a much smaller resolution of
$256\times 256$. In addition, we create a _naive_ baseline where we shift
input frames by one to emulate the delay that incures when running a large
image generator without warping. It serves as an expected lower bound on
accuracy.

Datasets. We use the talking head sequences from [25] to compare against prior
work as well as a _beard_ and a _high-fps_ sequence with more difficult facial
hair that was recorded with ethics board approval. Videos for the _beard_ and
_high-fps_ dataset were recorded at 1920x1080 and split into, respectively,
2604/558/558 and 3600/500/1000 frames for training/validation/testing. _Trump_
[25] sequences is a 1280x720 of 431 frames. The _Obama_ [25] sequence is a
512x512 of 2412 frames. The _male_ and _female_ [25] sequences are both
768x768 with 2380 frames. The _high-fps_ recording has 60 fps, all others run
at 30 fps.

Training Setup. We train all our models for 150 epochs using Adam [17] with
betas equal to {0.9, 0.999}, learning rate of 1e-4 for both the Generator, $G$
and Warp network $W$, and 1e-3 for the neural texture.

Figure 5:  Frames and their error maps for our model (1x and 2x warp), DNR+,
and naive baseline. Based on the L1 error we can see all models perform
similarly, differences are visible in the error maps for high-frequency
details.

Metrics. We evaluate our image reconstruction accuracy using an L1
reconstruction error between the ground truth and reconstructed image as well
as PSNR and SSIM. Furthermore, to show our increase in speed, we report the
latency and frames-per-second (fps) of our models and baselines. For all
methods, we only measure the time it takes to process the input UV maps and
the corresponding skeleton (pose), expression, and camera (extrinsics)
information; excluding the time it takes to render the UV map because they are
implementation dependent and with negligible overhead when implemented in a
rasterizer. Similarly, we only account for the time the inference generator
takes in [33], ignoring the processing of the conditional keypoint image. All
latency and fps metrics are computed on NVIDIA RTX 2080 GPUs. For our
approaches we report metrics and timings when we cache every frame (1x warp)
and every second frame (2x warp) to show our method is robust to varying
deviations between input frames.

###  5.1 Latency and Runtime Improvement

The generator backbone has a runtime of 47.02ms and equivalent latency. Our
warp net has a runtime of 14.62ms and therefore deduces latency by a factor of
3.2. Running on multiple GPUs improves frame rate from 28.5 to 67.6 while
inducing only a negligible 0.25 ms increase in latency due to the required
synchronization. Note that parallel execution across multiple GPUs can by
itself not improve the latency of a sequential process. When processing even
and odd frames on different GPUs, the time from input view to rendering output
remains the same. The impact of different model configurations on latency is
evaluated in Table 3.

###  5.2 Offline Reconstruction Quality

Model | #GPU | L1 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Latency [ms] $\downarrow$ | FPS $\uparrow$
---|---|---|---|---|---|---
timing baseline [33] | 1 | - | - | - | 16.60 | 60.2
DNR+ (Naive) | 1 | 0.0278 | 25.56 | 0.8970 | 46.81 | 21.4
DNR+ | 1 | 0.0240 | 26.59 | 0.9165 | 46.81 | 21.4
DNR++ | 1 | 0.0237 | 26.67 | 0.9168 | 49.37 | 20.3
ExWarp as in [33] (1x warp) | 2 | 0.0257 | 26.33 | 0.9108 | 18.84 | 26.4
ExWarp as in [33] (2x warp) | 2 | 0.0260 | 26.24 | 0.9094 | 18.84 | 60.8
Ours (1x warp) | 1 | 0.0244 | 26.66 | 0.9107 | 14.62 | 16.3
Ours (1x warp) | 2 | 0.0244 | 26.66 | 0.9107 | 14.87 | 28.5
Ours (2x warp) | 1 | 0.0251 | 26.45 | 0.9069 | 14.62 | 26.3
Ours (2x warp) | 2 | 0.0251 | 26.45 | 0.9069 | 14.87 | 67.6

Table 1:  Offline evaluation on the beard dataset. As expected, our model does
not achieve the best metrics on the image reconstruction metrics, but they
outperform the baselines in terms of latency and fps. Timing results for _Ours
2x warp_ are using parallel execution.

To test image generation quality, we use a held-out test video and drive the
trained models using the FLAME head model reconstructed on the reference
video. The results from Figure 5 and Table 1 evaluate the difference to the
reference video in offline processing mode. When comparing individual frames
between the baseline and our model, Figure 5 shows that single (1x warp) and
multi-frame warping (2x warp) work nearly as well in terms of error. Table 1
reveals, somewhat surprisingly that 1x warp slightly outperforms the DNR+
baseline in terms of PSNR despite having to warp with a shallow network. This
is possible since it has access to the current and past frame that helps to
correct errors in the facial expression estimation. To this end, we introduced
the DNR++ baseline as a new upper bound that also has access to the two
previous frames. In summary, at a small reduction in accuracy compared to the
baseline models, latency and frame rate are greatly improved by a factor of
two or more.

###  5.3 Online Reconstruction Quality

Online reconstruction requires the algorithm to run at the native frame rate
of the video. This has a significant influence on the performance of our
algorithm as the warping operation becomes simpler for high-frame rate videos
where the motion between two frames is reduced, leading to even higher
performance gains than for the previous offline evaluation. We test this
effect on the _high-fps_ sequence shown in Fig. 6. Table 2 reveals that
running online on 60 fps videos with Ours (2x warp) improves on Ours (1x
warp), as the latter can only process every other frame requiring larger
warps. Hence, warping multiple times is beneficial, jointly improving in
latency, runtime, and image quality, when parallel hardware is available. The
basic DNR baselines do not even run at 30 fps and are hence not comparable in
the high-fps online setting.

Figure 6:  Online application. When running at their native frame rate, high-
fps models (2x warp) improve as they have to bridge a smaller gap between
frames.

Operation Mode | L1 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---
Ours (1x warp @ 30fps) | 0.0361 | 23.19 | 0.8148
Ours (2x warp @ 60fps) | 0.0359 | 23.23 | 0.8162

Table 2:  _high-fps_ comparison for our warping network using realistic
operation settings on the _high-fps_ dataset. 2x warp not only improves speed
but even slightly improves the rendering quality.

Note that absolute PSNR numbers differ across subjects and scenes since faces
are smaller/bigger and also contain more or less high-frequency details,
including the texture on the shoulder region that is of little concern for the
facial feature reconstruction fidelity. Hence, it does not make sense to
relate absolute but only relative numbers between the _beard_ and _high-fps_
scores.

###  5.4 Novel View Synthesis Quality

Reproducing a pre-recorded sequence does not necessarily need low latency
since the entire video could be cached. Yet, latency is crucial for rendering
a face from a novel viewpoint to account for the user’s head motion in VR and
in general for viewpoint-dependent displays. We generate novel views of
characters by rotating their underlying 3D mesh (which is used to generate our
input UV maps) while holding other parameters fixed. Figure 8 shows the
retargeting of poses and views across videos and 7 shows synthetically
generated views and compares it to the results from [31]. Because we condition
on a full 3D face model, our rotation is more precise and keeps the pose
unchanged compared to the learned 3D features from Wang et al. that lead to
opening of the mouth and up-rotation. The quality by [31] is expected to be
slightly lower as it is not person-specific, which serves our motivation to
train a person-specific model.

Figure 7:  Comparison against Wang et al. [31]. While both models show similar
levels of detail, ours is anchored in a 3D representation which gives us more
fine-grained and independent control.  Figure 8:  Novel views while
retargeting. Our model is capable of generating realistic, view-dependent
novels views (Section 5.4) while mimicking the source’s facial expressions
(Section 5.5). These examples are generated using Ours (1x warp) using the
target actors from [25].

###  5.5 Retargeting

To show the flexibility of our approach, we retarget facial and head motions
from one person to another on in-the-wild videos that were established as a
benchmark in [25]. In this setting we train the neural texture and renderer on
our target subject and use these learned models at inference time, driven by
the head motion reconstructed from a source sequence with a different actor.
In Figure 9 we show that our approach is capable of retargeting with just
transferring expressions (as in [25]) and also when mapping the global head
orientation. The original results, provided from [25] do not include head
stabilization. Nevertheless, this comparison shows that our approach (1x warp)
is reproducing or even outperforming their image quality.

Figure 9:  Retargeting, without and with transfer of the global head motion,
including the comparison to [25].

Concat UV | Use $\Theta$ | Use MLP | SH Pose | SH Skips | ExWarp | Exp. | L1 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Latency [ms] $\downarrow$ | Rel. Latency [ms] $\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---
$\checkmark$ |  |  |  |  |  |  | 0.0276 | 25.6799 | 0.8987 | 12.80 | -
$\checkmark$ | $\checkmark$ |  |  |  |  |  | 0.0275 | 25.6994 | 0.8989 | 13.24 | 0.44
$\checkmark$ | $\checkmark$ | $\checkmark$ |  |  |  |  | 0.0271 | 25.8378 | 0.9012 | 13.61 | 0.81
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  |  | 0.0260 | 26.1627 | 0.9038 | 13.79 | 0.99
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | 0.0244 | 26.5705 | 0.9105 | 14.40 | 1.60
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 0.0257 | 26.3326 | 0.9108 | 18.84 | 6.03
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | 0.0244 | 26.6562 | 0.9107 | 14.62 | 1.82

Table 3:  Ablation study results for our warping network using the sequential
variant (1x warp), showing the relative improvement resulting from including
each component on the _beard_ dataset.

Furthermore, as we condition on a full 3D head model, we are able to generate
novel views while performing the retargeting as shown in Figure 8. Since our
approach only approximates the background for each scene, we use a static
background in our predictions.

###  5.6 Ablation Study

Network Backbones. To show our method’s generality to other backbones, we
compare our UNet to the widely successful ResNet-backbone using residual
blocks. As expected, warping is effective in reducing latency from 113 ms to
26 ms, but the ResNet does not outperform the UNet architectures. On the
_beard_ sequence, the ResNet backbone operating using single warps gives a
PSNR of 23.70 (UNet 26.66) and 23.18 PSNR using two warps (UNet 26.45).

Model Components. Adding the parameters and transformations used in _Our_ full
model one-by-one increases image quality (L1, PSNR, and SSIM) while only
incurring a slight decrease in latency and fps (see relative latency and
relative fps column, the relative latency compared to our full model). Table 3
presents the results for the sequential operation mode on the _beard_ sequence
over the following components:

  * •

Concat UV: using the difference of the current and cached UV maps in
$\mathbf{C}_{3}$.

  * •

Use $\mathbf{\theta}$: concatenating the cached pose and camera extrinsics to
$\mathbf{C}_{3}$.

  * •

Use MLP: passing $\theta$, $\mathbf{e}$, $\mathbf{p}$ through an MLP $M$
before concatenation with $\mathbf{C}_{3}$.

  * •

SH Pose: includes the spherical harmonics $S^{obj}$ in $\theta$.

  * •

SH Skips: ’rotationally’ encode $\mathbf{C}_{4},\mathbf{C}_{5}$ using
$\Delta\mathbf{S}^{obj}$.

  * •

ExWarp: explicit warping from a reference by sampling the learned neural
texture with $\mathbf{U}_{t+1}$ and concatenating it with $\mathbf{C}_{3}$.

  * •

Exp: cache and concatenate the expression $\mathbf{e}$ with $\theta$.

The explicit warping (_ExWarp_ , second-last row) adds a large latency
increase while not improving the quality metrics consistently. Hence, we
favour the implicit warp in our full model. When applied in parallel on
multiple GPUs, these reduced latencies translate directly to improved frame
rate. The synchronization overhead in the parallel implementation is only
0.25ms/frame, which we measured by running the sequential model with the same
threading and queue synchronization as for the parallel mode and taking their
latency difference.

##  6 Limitations

Because our generator, $G$, and warping network, $W$, learn how to generate an
image without an explicit rendering equation, we require a diverse set of
training views to ensure that we can perform novel view synthesis and accurate
warping at inference time. We can see in Figure 8 that the novel views break
at extreme angles around the ears of the target subject as these are not seen
during our training videos. The limiting factor is here the face
reconstruction algorithm that becomes unreliable when large parts of the face
are occluded. Eyes and a wide-open mouths can also pose a problem since they
are represented as holes in the underlying FLAME model and therefore the
direction of the eye gaze and tongue cannot be modeled. Furthermore, because
the FLAME model only estimates 3D models for the head, the network struggles
in cases where users have glasses or expressions that it cannot express.
Improving image quality in these directions is largely orthogonal to our
contributions towards low latency and high frame rate.

##  7 Conclusion

We introduced an implicit warping method that reduces the latency and, if
parallel hardware is available, increases the frame rate of neural face
rendering. We believe that such parallel execution to reduce latency and
increase frame rate will gain importance with VR and AR emerging on the
consumer market at scale, as our caching approach is compatible with the
deeper neural network architectures required to meet the ever-increasing
demand for output resolution. Thus, our work makes an important step towards
end-to-end VR and 3D telepresence using view-dependent displays.

#### Acknowledgements

This work was sponsored in part by the Natural Sciences and Engineering
Research Council (NSERC), Huawei, and Compute Canada. We also want to thank
Justus Thies for providing their dataset and baseline.

## Appendix

##  Appendix A Implementation details

Meshes are generated using the forward function of the FLAME model implemented
in python [18]. UV maps are generated by rendering the FLAME head model using
the PyTorch3D rasterizer, which implements rasterization on the GPU sing
custom cuda kernels but is not as fast as classical rasterization. The
explicit warp is implemented using the pytorch _grid_sample_ function, which
is the same as used by [4] for their explicit warp.

#### Generating Novel Views

Our method relies on both the processed UV maps, initialized with the 3D mesh
estimated by DECA [7]. Therefore, to generate novel views for our method we
require to modify these parameters. In our tests, we found that the best
generalization to novel views is obtained by rotating the vertices outputed by
the FLAME model, as modifying the joint angles in FLAME model to extreme poses
may lead to parameter ranges not seen during training.

#### Multithreading Library

Our multi-gpu operation modes are implemented using the PyTorch multithreading
library and two threads. For synchronization and communication between threads
we use the _torch.multiprocessing.mp.Queue_ ques: one input queue for each
thread (two for two GPUs), one image output que, and one que to signal
completion of caching. Listing 1 provides pseudocode for how caching and
warping threads are orchestrated from the main thread, making sure that the
output images arrive in sequence and that the caching completes before warping
on the same worker while also ensuring that the parallel workers are
independent otherwise for maximal performance. For simplicity, the pseudocode
is simplified for 2x GPU and 2x warping for every caching operation. A
different number of GPUs and warp operations only requires to change the
condition in line 4 and line 8 that distribute caching and warping operations
over worker threads. We will publish our multithreading code so others can
build upon it with their own neural models.

1

/* Main thread */

Queue viewpoint_queue[NGPU=2], cache_queue, out_queue

// Queues used for synchronization

cache_tid $\leftarrow$ 0

// Pointer to assign worker responsible for caching

2 for  _t=0; t++;_ do

Input: viewpoint, expr

3 if  _t % NUM_WARPS == 0_ then

cache_queue.get()

// Wait for caching to complete

cache_tid $\leftarrow$ (cache_tid+1) % NGPU

// Switch role of threads

// Post viewpoint to worker and assign caching role

4 (viewpoint, expr, warp=False) $\rightarrow$ viewpoint_queue[cache_tid]

// Post viewpoint to the other worker and assign warping role

5 (viewpoint, expr, warp=True) $\rightarrow$ viewpoint_queue[(cache_tid+1) %
NGPU]

// Display output image once available

Output: out_queue.get()

6

7

/* Worker thread, one for each GPU */

8

9do in parallel

NeuralCache cache

// variable local to each thread

(viewpoint, expression, warp) $\leftarrow$ viewpoint_queue.get()

// Wait for new input to process

10 if  _warp = True_ then

image $\leftarrow$ runWarpNet(viewpoint, expression, cache)

// Generate the image

image $\rightarrow$ out_queue

// Return output image via queue

11

12 else

mesh $\leftarrow$ runFaceModel(viewpoint, expression)

// Face mesh from expr. parameters

cache $\leftarrow$ runDecoder(viewpoint, expression, mesh)

// Cache neural representation

”caching done” $\rightarrow$ cache_queue

// Notify cache completion via queue

13

14

15

Algorithm 1 Parallel Implementation (2x GPUs, 2x warp)

#### Algorithm Time Complexity

Using the code provided by pytorch-OpCounter, we measured the FLOPs for both
the generator and warping network which are 364G and 87G FLOPs respectively.
This factor of 4x corresponds well with the 3.2x latency improvement that we
reported. Note that our approach decouples the complexity of the generator and
warp networks, i.e., the warp keeps the same time complexity when coupled with
deeper generators (c.f. experiment with ResNet in ablation study).

##  Appendix B Limitations of Capture Algorithm

Figure 10:  Limitations of Head Reconstruction We can see that at extreme
angles and glasses present a problem to the capture algorithm and produces
inaccurate UV maps and parameters.

Due to limitations with the DECA approach [7], extreme head poses and cases
when the subject is wearing glasses are not accurately reconstructed. Figure
10 shows such a case with glasses and a perpendicular angle to the camera,
leading to a misalignment of model and image around the mouth, ears, and
forehead.

##  Appendix C Additional Ablation on Caching Layers

In this additional ablation, we sequentially remove the caching layers (C3,
C4, C5) and measure the reconstruction quality in terms of L1 loss, PSNR, and
SSIM. Each configuration was trained on the beard dataset for 50 epochs, and
metrics were computed on the held-out test set. Table 4 shows that adding the
C4 cache layer is the most impactful, but since performance increases after
adding all layers, we include all cache layers in our final configuration.

C3 | C4 | C5 | L1 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---|---|---
$\checkmark$ |  |  | 0.0261 | 26.13 | 0.9048
$\checkmark$ | $\checkmark$ |  | 0.0247 | 26.48 | 0.9096
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.0246 | 26.54 | 0.9102
Table 4:  Ablating caching layers for our warping network using the sequential
variant (1x warp), showing the improvement after adding each component on the
_beard_ dataset.

##  Appendix D Additional Ablation on the Warp Distance

In the experiments in the main document, we present the results of our warping
network by warping either 1 or 2 frames ahead. This ablation study measures
the performance drop from warping more frames ahead. Figure 11 shows that the
reconstruction quality decreases linearly with the warp distance on the
_beard_ dataset, which was recorded at 30fps. Please note that this increase
of error is expected as the proposed neural caching and warping is a local
approximation designed for high-fps videos where the warp distance is small.

Figure 11:  Ablation on warp distance as we increase the warp distance, or the
number of warping operations per caching operation, the reconstruction quality
decreases linearly.

## References

  * [1] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3D faces.  1999\.
  * [2] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks).  In International Conference on Computer Vision, 2017.
  * [3] Joao Carreira, Viorica Patraucean, Laurent Mazare, Andrew Zisserman, and Simon Osindero.  Massively parallel video networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 649–666, 2018.
  * [4] Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li.  Fast, accurate and lightweight super-resolution with neural architecture search.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 59–64. IEEE, 2021.
  * [5] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong.  Disentangled and controllable face image generation via 3D imitative-contrastive learning.  In CVPR, 2020.
  * [6] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203–213, 2020.
  * [7] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  volume 40, 2021.
  * [8] Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala.  Text-based editing of talking-head video.  ACM TOG, 2019.
  * [9] Jiahao Geng, Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou.  Warp-guided GANs for single-photo facial animation.  ACM TOG, 2018.
  * [10] Partha Ghosh, Pravir Singh Gupta, Roy Uziel, Anurag Ranjan, Michael J Black, and Timo Bolkart.  Gif: Generative interpretable faces.  In 2020 International Conference on 3D Vision (3DV), pages 868–878. IEEE, 2020.
  * [11] Ping Hu, Fabian Caba, Oliver Wang, Zhe Lin, Stan Sclaroff, and Federico Perazzi.  Temporally distributed networks for fast video semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8818–8827, 2020.
  * [12] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  2017\.
  * [13] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [14] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  arXiv preprint arXiv:2106.12423, 2021.
  * [15] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [16] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In CVPR, 2020.
  * [17] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [18] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194–1, 2017.
  * [19] Mateusz Malinowski, Grzegorz Swirszcz, Joao Carreira, and Viorica Patraucean.  Sideways: Depth-parallel training of video models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11834–11843, 2020.
  * [20] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [21] Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, and Hao Li.  paGAN: real-time avatars using dynamic textures.  ACM TOG, 2018.
  * [22] Kyle Olszewski, Zimo Li, Chao Yang, Yi Zhou, Ronald Yu, Zeng Huang, Sitao Xiang, Shunsuke Saito, Pushmeet Kohli, and Hao Li.  Realistic dynamic facial textures from a single image using GANs.  In ICCV, 2017.
  * [23] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman.  Synthesizing Obama: learning lip sync from audio.  ACM TOG, 2017.
  * [24] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt.  StyleRig: Rigging StyleGAN for 3d control over portrait images.  In CVPR, 2020.
  * [25] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  ACM Transactions on Graphics (TOG), 38(4):1–12, 2019.
  * [26] Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt.  Real-time expression transfer for facial reenactment.  ACM TOG, 2015.
  * [27] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2Face: Real-time face capture and reenactment of rgb videos.  In CVPR, 2016.
  * [28] JMP Van Waveren.  The asynchronous time warp for virtual reality on consumer hardware.  In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, pages 37–46, 2016.
  * [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [30] Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popovic.  Face transfer with multilinear models.  ACM TOG, 2005.
  * [31] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.  One-shot free-view neural talking-head synthesis for video conferencing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10039–10049, 2021.
  * [32] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  arXiv preprint arXiv:2103.14024, 2021.
  * [33] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  In ECCV, 2020.
  * [34] Qian Zhou, Georg Hagemann, Dylan Fafard, Ian Stavness, and Sidney Fels.  An evaluation of depth and size perception on a spherical fish tank virtual reality display.  IEEE transactions on visualization and computer graphics, 2019.
  * [35] Qian Zhou, Gregor Miller, Kai Wu, Ian Stavness, and Sidney Fels.  Analysis and practical minimization of registration error in a spherical fish tank virtual reality system.  volume 10114, pages 519–534, 03 2017.
  * [36] Qian Zhou, Kai Wu, Gregor Miller, Ian Stavness, and Sidney Fels.  3dps: An auto-calibrated three-dimensional perspective-corrected spherical display.  In 2017 IEEE Virtual Reality (VR), pages 455–456. IEEE, 2017.