# Outage Probability Analysis of HARQ-Aided Terahertz Communications

Ziyang Song1, Zheng Shi1, Qingping Dou1, Guanghua Yang1, Yunfei Li2, and
Shaodan Ma2
1The School of Intelligent Systems Science and Engineering, Jinan University,
China
2The State Key Laboratory of Internet of Things for Smart City, University of
Macau, China

###### Abstract

Although terahertz (THz) communications can provide mobile broadband services,
it usually has a large path loss and is vulnerable to antenna misalignment.
This significantly degrades the reception reliability. To address this issue,
the hybrid automatic repeat request (HARQ) is proposed to further enhance the
reliability of THz communications. This paper provides an in-depth
investigation on the outage performance of two different types of HARQ-aided
THz communications, including Type-I HARQ and HARQ with chase combining (HARQ-
CC). Moreover, the effects of both fading and stochastic antenna misalignment
are considered in this paper. The exact outage probabilities of HARQ-aided THz
communications are derived in closed-form, with which the asymptotic outage
analysis is enabled to explore helpful insights. In particular, it is revealed
that full time diversity can be achieved by using HARQ assisted schemes.
Besides, the HARQ-CC-aided scheme performs better than the Type-I HARQ-aided
one due to its high diversity combining gain. The analytical results are
eventually validated via Monte-Carlo simulations.

###### Index Terms:

Diversity order, hybrid automatic repeat request (HARQ), outage probability,
terahertz (THz) communciations.

##  I Introduction

The past decade has witnessed the explosive growth of wireless devices and the
increasing number of bandwidth-consuming internet services, which increase the
demand of low-latency high data rate and lead to the spectrum scarcity. To
fulfill these challenges, terahertz (THz) communications are envisioned to be
a promising wireless technology for beyond fifth generation (B5G) networks,
because its wide frequency band allows the data transmission rate up to
terabits per second (Tbps) [1, 2, 3]. Unfortunately, the conventional low-
frequency channel model is inapplicable to the THz communications due to its
unique propagation characteristics, e.g., atmospheric influence, antenna
misalignment, and rain attenuation [4]. It has been reported in [5, 6, 7] that
these unfavorable factors significantly degrades the reliability performance
of THz communications. Thus, it is of urgent necessity to address how to
enhance the reliability of THz communications.

There are a few works that have been proposed for the reliable THz
communications [8, 9, 10, 11]. In particular, In [8], highly directional
antennas were proposed to establish reliable THz links. The strategy of
integrating THz communications with mobile heterogenous networks was developed
to ensure the reliability in [9], and the performance of the bit error rate
(BER) was then studied. In [10], a mixed THz-radio frequency wireless
architecture was devised to provide a promising reliable backhaul-fronthaul
solution. Moreover, by noticing that hybrid automatic repeat request (HARQ) is
a reliable transmission technique, HARQ assisted THz communications were
proposed to investigate the relationship between the BER and the energy
efficiency in [11], in which the discussions are conducted based on system-
level simulations. To the best of our knowledge, it lacks of the theoretical
analysis for the performance investigation on HARQ-aided THz communications in
the literature. To substantially exploit the time diversity gain from HARQ, it
is very imperative to study the performance of HARQ-aided THz communications
from the information-theoretical perspective.

Motivated by this, we thoroughly study the outage performance of HARQ-aided
THz communications in this paper. The closed-form expressions are derived for
the outage probabilities of two types of HARQ-aided THz communications,
including Type-I HARQ and HARQ with chase combining (HARQ-CC). With the
analytical results, the asymptotic outage analysis is performed to uncover
meaningful insights. For example, both Type-I HARQ and HARQ-CC-aided THz
schemes can achieve full time diversity. Moreover, the numerical results show
that the HARQ-CC-aided THz scheme outperforms the Type-I HARQ-aided one due to
its high diversity gain.

The rest of this paper is organized as follows. Section II presents the system
model for HARQ-aided THz communications, based on which Section III derives
the closed-form expressions for exact and asymptotic outage probabilities for
two different types of HARQ-aided THz communications. Section IV then shows
simulation and numerical results. Finally, Section V concludes this paper.

##  II System Model

In this paper, we consider a point-to-point HARQ-aided THz communication
system. To enhance the reception reliability, HARQ technique is utilized to
assist THz communications. The received signal at the $k$-th HARQ round can be
expressed as

${{\bf y}_{k}}=\sqrt{P}{h_{k}}{\bf s}_{k}+{{\bf w}_{k}},$ (1)

where ${P}$ and $h_{k}$ represents the transmit power and the equivalent THz
channel coefficient in the $k$-th HARQ round, respectively, ${\bf s}_{k}$ is
the transmitted symbols with unity power, and ${{\bf w}_{k}}$ is the complex
additive white Gaussian noise (AWGN) with mean zero and variance $N_{0}$.
According to [5], the channel coefficient $h_{k}$ is modeled according to
${h_{k}}={h_{l}}{h_{pf,k}}$, where $h_{l}$ is the deterministic THz path gain
and keeps constant during all HARQ rounds, and ${h_{pf,k}}$ captures the joint
effect of antenna misalignment and multipath fading. From [5], ${h_{l}}$ is
given by

${h_{l}}=\frac{{c\sqrt{{G_{t,1}}{G_{r,1}}}}}{{4\pi{f_{1}}{d_{1}}}}\exp\left({-\frac{1}{2}\kappa({f_{1}},T,\psi,p){d_{1}}}\right),$
(2)

where $c$, $f_{1}$, and $d_{1}$ stand for the speed of light, the carrier
frequency, and the transmission distance, respectively, ${{G_{t,1}}}$ and
${{G_{r,1}}}$ represent the transmit and receive antenna gains, respectively,
$\kappa({f_{1}},T,\psi,p)$ characterizes the molecular absorption coefficient,
which is determined by the temperature $T$, the relative humidity $\psi$ and
the atmospheric pressure $p$. The explicit expression of
$\kappa({f_{1}},T,\psi,p)$ is given by [5, eq. (4)], which is omitted here to
conserve space.

Moreover, as proved in [5], the probability density function (PDF) and
cumulative distribution function (CDF) of ${h_{pf,k}}$ are respectively
expressed as

${f_{\left|{{h_{pf,k}}}\right|}}(x)=\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}{x^{\phi-1}}}}{{S_{0}^{\phi}\hat{h}_{f}^{\phi}\Gamma(\mu)}}\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha},\frac{{\mu{x^{\alpha}}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}}}}\right),$
(3)
${F_{\left|{{h_{pf,k}}}\right|}}(x)=1-\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}{x^{\phi}}}}{{\alpha
S_{0}^{\phi}\hat{h}_{f}^{\phi}}}\sum\limits_{n=0}^{\mu-1}{\frac{1}{{n!}}}\Gamma\left({\frac{{\alpha
n-\phi}}{\alpha},\frac{{\mu{x^{\alpha}}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}}}}\right),$
(4)

where $\Gamma\left(a,x\right)$ denotes the upper incomplete Gamma function,
$\alpha$ is the distribution parameter, $\mu$ and ${\hat{h}_{f}}$ denote the
variance and the $\alpha$-root mean value of the fading channel envelope,
respectively, ${S_{0}}$ is the fraction of the maximum collected power and is
given by ${S_{0}}={\left|{{\rm erf}(\zeta)}\right|^{2}}$ and
$\zeta=\sqrt{\pi}{w_{d1}}/\left({\sqrt{2}{r_{1}}}\right)$, $r_{1}$ and
${w_{d1}}$ denote the radius of the receive antenna effective area and the
transmission beam footprint radius at reference distance ${d_{1}}$,
respectively, $\phi=w_{e}^{2}/4\sigma_{s}^{2}$, ${w_{e}}$ and ${\sigma_{s}}$
are the equivalent beam width radius and the doubled spatial jitter standard
deviation, respectively, $w_{e}^{2}=\left({w_{d1}^{2}\sqrt{\pi}\rm
erf\left(\zeta\right)}\right)/\left({2\zeta\exp\left({-{\zeta^{2}}}\right)}\right)$.

To prevent network congestion in the presence of possible deep fading, the
maximum number of HARQ rounds is limited up to $M$. In this paper, we consider
two types of HARQ schemes, i.e., Type-I HARQ and HARQ-CC. Their difference
lies in the diversity combining technique applied at the receiver. More
specifically, for Type-I HARQ-aided THz communications, the receiver attempts
to recover the message by solely relying on the current packet. The
erroneously received packets will be discarded. From the information
theoretical perspective [12], the accumulated mutual information of Type-I
HARQ-aided THz communications after $M$ HARQ rounds is given by

${I^{Type-I}}=\mathop{\max}\limits_{k=1,\cdots,M}{\log_{2}}\left({1+\rho{{\left|{{h_{l}}}\right|}^{2}}{{\left|{{h_{pf,k}}}\right|}^{2}}}\right),$
(5)

where $\rho$ refers to the transmit signal-to-noise ratio (SNR), i.e.
$\rho={P}/{N_{0}}$.

Furthermore, for HARQ-CC-aided THz communications, the previously failed
packets are stored for the combination with the subsequent packet by using
maximum ratio combining (MRC). According to [12], the accumulated mutual
information of HARQ-CC-aided THz communications after $M$ HARQ rounds is given
by

${I^{CC}}={\log_{2}}\left({1+\sum\limits_{k=1}^{M}{\rho{{\left|{{h_{l}}}\right|}^{2}}{{\left|{{h_{pf,k}}}\right|}^{2}}}}\right),$
(6)

##  III Outage Probability Analysis

To study the transmission reliability of HARQ-aided THz wireless systems, the
outage probability is the most important performance metric. To be specific,
the outage probability is defined as the probability of the event that the
accumulated mutual information is less than the transmission rate $R$ [13].
Accordingly, the outage probability for HARQ is given by

${P_{out}}=\Pr\left\\{{I<R}\right\\},$ (7)

where $I\in\left\\{{{I^{Type-I}},{I^{CC}}}\right\\}$. By considering different
formulations of the accumulated mutual information for the two types of HARQ
schemes, the outage analyses for the Type-I and the HARQ-CC-aided THz
communications will be undertaken individually in the following subsections.

###  III-A Type-I HARQ-Aided THz Communications

In what follows, the exact outage analysis will be first carried out for
Type-I HARQ-aided THz communications, with which the asymptotic outage
analysis is enabled to gain useful insights.

####  III-A1 Exact analysis

By substituting (5) into (7), ${P_{out}}$ is then expressed as

$P_{out}^{Type-I}=\Pr\left\\{{\mathop{\max}\limits_{k=1,\cdots,M}{{\log}_{2}}\left({1+\rho{{\left|{{h_{l}}}\right|}^{2}}{{\left|{{h_{pf,k}}}\right|}^{2}}}\right)<R}\right\\}.$
(8)

We assume fast fading channels where the channel coefficients are independent
across all HARQ rounds, ${P_{out}}$ can be expressed in terms of the CDF of
${h_{pf,k}}$ as

$\displaystyle P_{out}^{Type-I}$
$\displaystyle=\prod\limits_{k=1}^{M}{\Pr\left\\{{{{\log}_{2}}(1+\rho{{\left|{{h_{l}}}\right|}^{2}}{{\left|{{h_{pf,k}}}\right|}^{2}})<R}\right\\}}$
$\displaystyle=\prod\limits_{k=1}^{M}{{F_{\left|{{h_{pf,k}}}\right|}}\left({\frac{1}{{\left|{{h_{l}}}\right|}}\sqrt{\frac{{{2^{R}}-1}}{\rho}}}\right)}.$
(9)

By substituting the CDF of $h_{pf,k}$ in (4) into (III-A1), a closed-form
expression of the outage probability of Type-I HARQ-aided THz communications
can be expressed as (10), as shown at the top of the next page.

$P_{out}^{Type-I}=\prod\limits_{k=1}^{M}{\left({1-\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}{{\left({\sqrt{\frac{{{2^{R}}-1}}{\rho}}}\right)}^{\phi}}}}{{\alpha
S_{0}^{\phi}\hat{h}_{f}^{\phi}{{\left|{{h_{l}}}\right|}^{\phi}}}}\sum\limits_{n=0}^{\mu-1}{\frac{1}{{n!}}}\Gamma\left({\frac{{\alpha
n-\phi}}{\alpha},\frac{{\mu{{\left({\sqrt{\frac{{{2^{R}}-1}}{\rho}}}\right)}^{\alpha}}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}{{\left|{{h_{l}}}\right|}^{\alpha}}}}}\right)}\right)}$
(10)

####  III-A2 Asymptomatic analysis

It is clear that (10) is very complicated to extract more meaningful insights.
To avoid this, the asymptotic analysis is performed for the outage probability
in the high SNR regime, i.e., $\rho\to\infty$. As proved in Appendix A, the
asymptotic outage probability of Type-I HARQ-aided THz communications is given
by

$\displaystyle P_{out}^{Type-I}\simeq$
$\displaystyle\left\\{{\begin{array}[]{*{20}{c}}{{{\left({\frac{{\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha}}\right){\mu^{\frac{\phi}{\alpha}}}{{\left({{2^{R}}-1}\right)}^{\frac{\phi}{2}}}}}{{\Gamma(\mu)\hat{h}_{f}^{\phi}S_{0}^{\phi}{{\left|{{h_{l}}}\right|}^{\phi}}{\rho^{\frac{\phi}{2}}}}}}\right)}^{M}}}&{,\mu\alpha-\phi>0}\\\
{{{\left({\frac{{\phi{\mu^{\mu-1}}{{\left({{2^{R}}-1}\right)}^{\frac{{\alpha\mu}}{2}}}}}{{\Gamma(\mu)\left({\phi-\alpha\mu}\right)\hat{h}_{f}^{\alpha\mu}S_{0}^{\alpha\mu}{{\left|{{h_{l}}}\right|}^{\alpha\mu}}{\rho^{\frac{{\alpha\mu}}{2}}}}}}\right)}^{M}}}&{,\mu\alpha-\phi<0}\end{array}}\right.$
(13)

Clearly from (III-A2), the diversity order attained by Type I-HARQ-aided THz
communications can be obtained, where the diversity order is used to gauge the
number of degrees of freedom in the communication system. More precisely, the
diversity order is defined as the ratio of the outage probability to the
transmit SNR on a log-log scale as [14]

$d=-\mathop{\lim}\limits_{\rho\to\infty}\frac{{\log{{P_{out}}}}}{{\log\rho}}.$
(14)

By substituting (III-A2) into (14), the diversity order of Type-I HARQ-aided
THz communications is derived as

$\displaystyle{d^{Type-I}}=\left\\{\begin{array}[]{l}\frac{{\phi
M}}{2},\alpha\mu-\phi>0\\\ \frac{{\alpha\mu
M}}{2},\alpha\mu-\phi<0\end{array}\right.$ (17)

Since the diversity order ${d^{Type-I}}$ is linearly proportional to the
maximum number of transmissions, i.e., $M$, full time diversity can be
achieved by employing Type-I HARQ.

###  III-B HARQ-CC-Aided THz Communications

Similarly, the exact and the asymptotic outage analyses are performed for
HARQ-CC aided THz communications.

####  III-B1 Exact analysis

By plugging (6) into (7) together with some rearrangements, the outage
probability ${P_{out}}$ of HARQ-CC aided THz communications can be rewritten
as

$P_{out}^{CC}=\Pr\left\\{{\underbrace{\sum\limits_{k=1}^{M}{\rho{{\left|{{h_{l}}}\right|}^{2}}{{\left|{{h_{pf,k}}}\right|}^{2}}}}_{\gamma}<{2^{R}}-1}\right\\}.$
(18)

It is found that $P_{out}^{CC}$ is equivalent to determining the distribution
of a summation of multiple random variables, i.e., $\gamma$. Therefore, the
method of moment generating function (MGF) is adopted to derive the
distribution of $\gamma$. Particularly, the MGF of the PDF of $\gamma$ can be
obtained as

$\displaystyle{\mathbb{E}_{\gamma}}\left\\{{{e^{t\gamma}}}\right\\}=$
$\displaystyle\prod\limits_{k=1}^{M}{\mathbb{E}\left\\{{{e^{\rho{{\left|{{h_{l}}}\right|}^{2}}{{\left|{{h_{pf,k}}}\right|}^{2}}t}}}\right\\}}$
$\displaystyle=$
$\displaystyle\prod\limits_{k=1}^{M}{\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}}}{{S_{0}^{\phi}\hat{h}_{f}^{\phi}\Gamma(\mu)}}\int_{\rm{0}}^{\infty}{{x_{k}}^{\phi-1}{e^{\rho{{\left|{{h_{l}}}\right|}^{2}}{x_{k}}^{2}t}}}}$
$\displaystyle\times\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha},\frac{{\mu
x_{k}^{\alpha}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}}}}\right)d{x_{k}},$ (19)

where $\mathbb{E}\\{\cdot\\}$ stands for the expectation operator. Then, the
CDF of $\gamma$ can be obtained by inverse Laplace transform as

${F_{\gamma}}(\gamma)=\frac{1}{{2\pi\rm i}}\int_{\alpha-\rm
i\infty}^{\alpha{\rm{+\rm i}}\infty}{{e^{\gamma
t}}\frac{1}{t}{\mathbb{E}_{\gamma}}\left\\{{{e^{-t\gamma}}}\right\\}dt}.$ (20)

where $\alpha>0$ and $\rm i=\sqrt{-1}$. By substituting (III-B1) into (20), it
follows that

$\displaystyle{F_{\gamma}}(\gamma)=\frac{1}{{2\pi\rm i}}\int_{\alpha-\rm
i\infty}^{\alpha{\rm{+\rm i}}\infty}{{e^{\gamma
t}}\frac{1}{t}\prod\limits_{k=1}^{M}{\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}}}{{S_{0}^{\phi}\hat{h}_{f}^{\phi}\Gamma(\mu)}}}}$
$\displaystyle\times\int_{\rm{0}}^{\infty}{{x_{k}}^{\phi-1}{e^{-\rho{{\left|{{h_{l}}}\right|}^{2}}{x_{k}}^{2}t}}\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha},\frac{{\mu
x_{k}^{\alpha}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}}}}\right)}d{x_{k}}dt.$
(21)

By applying Parseval’s Type Property of Mellin transform [15, Eq. 8.3.22] to
the inner integral of (III-B1), one has

$\displaystyle{F_{\gamma}}(\gamma)=$ $\displaystyle\frac{1}{{2\pi\rm
i}}\int_{\alpha{\rm{-i}}\infty}^{\alpha{\rm{+i}}\infty}{{e^{\gamma
t}}\frac{1}{t}\prod\limits_{k=1}^{M}{\frac{\phi}{{2\hat{h}_{f}^{\alpha-\phi}\Gamma(\mu)}}}}$
$\displaystyle\times\frac{1}{{2\pi\rm
i}}\int_{\alpha{\rm{-i}}\infty}^{\alpha{\rm{+i}}\infty}{\frac{{\Gamma\left({\frac{s}{2}}\right)\Gamma\left({\phi-s}\right)\Gamma\left({\mu-\frac{s}{\alpha}}\right)}}{{\Gamma\left({1+\phi-s}\right)}}}$
$\displaystyle\times{{\left({{{\left({\rho{{\left|{{h_{l}}}\right|}^{2}}t}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{\hat{h}_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}}\right)}^{-s}}dsdt,$
(22)

where (III-B1) holds by using the following two Mellin transforms [15, Eq.
8.2.5], [16, Eq. 6.455, Eq. 3.326.2]

$\displaystyle\int_{\rm{0}}^{\infty}{{x_{k}}^{s-1}{e^{-\rho{{\left|{{h_{l}}}\right|}^{2}}x_{k}^{2}t}}d}{x_{k}}=\frac{1}{2}{\left({\rho{{\left|{{h_{l}}}\right|}^{2}}t}\right)^{-\frac{s}{2}}}\Gamma\left({\frac{s}{2}}\right),$
(23)
$\int_{\rm{0}}^{\infty}{{x_{k}}^{s-1}\Gamma\left(\frac{{\alpha\mu-\phi}}{\alpha},\frac{{\mu
x_{k}^{\alpha}}}{{S_{\rm{0}}^{\alpha}h_{f}^{\alpha}}}\right)d}{x_{k}}\\\
=\frac{1}{{s}}{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)^{-\frac{{s}}{\alpha}}}\Gamma\left(\frac{{\alpha\mu+s-\phi}}{\alpha}\right).$
(24)

By recognizing the inner integral as a Mellin-Barnes integral and identifying
it with the extended Fox’s H function, we derive ${F_{\gamma}}(\gamma)$ in a
compact form, where the definition of the extended Fox’s H function is given
by (25) [17, Eq.T.I.1], as shown at the top of the next page.

$H_{p,q}^{m,n}\left[{z|_{{{\left({{\beta_{j}},{B_{j,}}{b_{j}}}\right)}_{1,q}}}^{{{\left({{\alpha_{j}},{A_{j}},{a_{j}}}\right)}_{1,p}}}}\right]=\frac{1}{{2\pi\rm
i}}\oint_{C}{\frac{{\prod\nolimits_{j=1}^{n}{{{\left({\Gamma\left({1-{\alpha_{j}}+{A_{j}}s}\right)}\right)}^{{a_{j}}}}\prod\nolimits_{j=1}^{m}{{{\left({\Gamma\left({{\beta_{j}}-{B_{j}}s}\right)}\right)}^{{b_{j}}}}}}}}{{\prod\nolimits_{j=n+1}^{p}{{{\left({\Gamma\left({{\alpha_{j}}-{A_{j}}s}\right)}\right)}^{{a_{j}}}}\prod\nolimits_{j=m+1}^{q}{{{\left({\Gamma\left({1-{\beta_{j}}+{B_{j}}s}\right)}\right)}^{{b_{j}}}}}}}}}{z^{s}}ds,$
(25)

As a consequence, by substituting $\gamma={2^{R}}-1$ into
${F_{\gamma}}(\gamma)$, we arrive at the final expression of the outage
probability of HARQ-CC aided THz communications as (26), as shown at the top
of the next page.

$P_{out}^{CC}={F_{\gamma}}({2^{R}}-1)={\left({\frac{\phi}{{2\Gamma(\mu)}}}\right)^{M}}\frac{1}{{2\pi\rm
i}}\int_{\alpha-\rm i\infty}^{\alpha{\rm{+\rm
i}}\infty}{\frac{{{e^{\left({{2^{R}}-1}\right)t}}}}{t}\prod\limits_{k=1}^{M}{H_{0,1}^{1,2}\left[{{{\left({\rho{{\left|{{h_{l}}}\right|}^{2}}t}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}\left|{{}_{\left({0,\frac{1}{2}}\right),\left({-\phi,1}\right)}^{\left({1-\phi,1}\right),\left({1-\mu,\frac{1}{\alpha}}\right)}}\right.}\right]}dt},$
(26)

####  III-B2 Asymptomatic analysis

Similarly to Section III-A2, the asymptomatic outage analysis is conducted for
(26) under high SNR, i.e., $\rho\to\infty$. As proved in Appendix B, as the
transmit SNR $\rho$ approaches to infinity, the outage probability is
asymptotic to

$\displaystyle P_{out}^{CC}\simeq\eta P_{out}^{Type-I},$ (27)

where $\eta$ is given by

$\eta=\left\\{{\begin{array}[]{*{20}{c}}{\frac{{{{\left({\Gamma\left({\frac{\phi}{2}+1}\right)}\right)}^{M}}}}{{\Gamma\left({\frac{{\phi
M}}{2}+1}\right)}},}&{\mu\alpha-\phi>0}\\\
{\frac{{{{\left({\Gamma\left({\frac{{\mu\alpha}}{2}+1}\right)}\right)}^{M}}}}{{\Gamma\left({\frac{{\mu\alpha
M}}{2}+1}\right)}},}&{\mu\alpha-\phi<0}\end{array}}\right.$ (28)

By substituting (27) into (14), the diversity order of HARQ-CC-aided THz
communications is the same as Type-I HARQ-aided ones, i.e.,

$\displaystyle{d^{CC}}={d^{Type-I}}.$ (29)

This indicates that both the two HARQ-aided THz communication schemes achieve
the full time diversity. Moreover, it is not hard to prove that $\eta$ is less
than 1 for $M>1$. From (27), this result justifies that the outage performance
of HARQ-CC aided THz communications outperforms that of Type-I HARQ-aided THz
communications due to its high diversity combining gain.

##  IV Numerical Results

In this section, the simulation results are presented to verify the analytical
results. Unless otherwise stated, the system parameters are set as follows,
$\alpha=2$, $\mu=1$, $\psi=50\%$, $T=296{\rm{{}^{\circ}}}$ K, $p=101325$ Pa,
$M=3$, ${\sigma_{\rm{s}}}=1$ and the carrier frequency is $f_{1}=275$ THz.
Moreover, both the transmit and the receive antenna gains equal to $55$ dBi,
the transmission distance is assumed to be $20$ m. By considering the effect
of the sign of $\alpha\mu-\phi$ on the asymptotic analysis, we assume that the
equivalent beamwidth ${w_{e}}$ is set as $1$ for $\alpha\mu-\phi>0$, while
${w_{e}}$ is set as $3$ for $\alpha\mu-\phi<0$.

Fig. 1 investigates the relationship between the outage probability
${P_{out}}$ and the transmission rate $R$, where the labels ’Sim.’, ’Exa.’,
and ’Asy.’ represent the simulated, the exact and the asymptotic outage
probabilities, respectively. It can be seen that the exact results perfectly
coincide with the simulated results. Moreover, it is as expected that the
outage probability increases with transmission rate. Furthermore, it is easily
observed that HARQ-CC-aided THz communications is superior to Type-I HARQ-
aided THz communications due to the high diversity combining gain no matter if
$\alpha\mu-\phi>0$ or not. This observation clearly confirms the validity of
our asymptotic results.

Figure 1: The outage probability $P_{out}$ versus the transmission rate $R$
with $\rho=30$ dB.

Fig. 2 depicts the relationship between the outage probability ${P_{out}}$ and
the average transmit SNR for the two HARQ-aided THz communication schemes.
Similarly, the exact and the simulated results match well with each other, and
they converge to the asymptotic ones in the high SNR regime no matter whether
$\alpha\mu-\phi>0$ or not. Besides, it is found that the decreasing slopes of
both schemes are the same, that is, the outage curves of the two schemes are
parallel to each other. This observation is consistent with the analysis of
the diversity order.

Figure 2: The outage probability $P_{out}$ versus the average SNR $\rho$ with
$R=2$ bps/Hz.

##  V Conclusions

In this paper, we investigated the outage performance of two different types
of HARQ-aided THz communications, including Type-I HARQ and HARQ-CC. The main
contribution of this paper was devoted to quantify the effect of the fading
and antenna misalignment on the outage performance through the exact and the
asymptotic analyses. The results filled the gap for the performance analysis
of HARQ-aided THz communications from information-theoretical perspective. It
revealed that the Type-I HARQ-aided THz communication scheme are inferior to
the HARQ-CC-aided one due to its low diversity combining gain, but the
analysis of the diversity order indicated that both Type-I HARQ and HARQ-CC-
aided schemes can achieve full time diversity. The analytical results were
finally validated through Monte Carlo simulations.

##  Appendix A Proof of (III-A2)

With the PDF of $|{h_{pf,k}}|$ given by (3), (III-A1) can be rewritten as

$\displaystyle
P_{out}^{Type-I}=\prod\limits_{k=1}^{M}{\int\limits_{0}^{\frac{1}{{\left|{{h_{l}}}\right|}}\sqrt{\frac{{{2^{R}}-1}}{\rho}}}{{f_{\left|{{h_{pf,k}}}\right|}}(x)dx}}$
$\displaystyle=\prod\limits_{k=1}^{M}{\int\limits_{0}^{\frac{1}{{\left|{{h_{l}}}\right|}}\sqrt{\frac{{{2^{R}}-1}}{\rho}}}{\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}{x^{\phi-1}}}}{{S_{0}^{\phi}\hat{h}_{f}^{\phi}\Gamma(\mu)}}\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha},\frac{{\mu{x^{\alpha}}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}}}}\right)dx}}.$
(30)

By noticing that the upper incomplete Gamma function in (A) has different
asymptotic behaviors for $\alpha\mu-\phi>0$ and $\alpha\mu-\phi<0$, the
asymptotic expressions of the outage probability in these two cases are
derived one by one.

###  A-A $\alpha\mu-\phi>0$

By using the first mean value theorem [18, Eq. 1.4.29], we have

$\displaystyle
P_{out}^{Type-I}=\prod\limits_{k=1}^{M}{\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha},\frac{{\mu{\varsigma^{\alpha}}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}}}}\right)\int\limits_{0}^{\frac{1}{{\left|{{h_{l}}}\right|}}\sqrt{\frac{{{2^{R}}-1}}{\rho}}}{\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}{x^{\phi-1}}}}{{S_{0}^{\phi}\hat{h}_{f}^{\phi}\Gamma(\mu)}}dx}}$
$\displaystyle=$
$\displaystyle\prod\limits_{k=1}^{M}{\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha},\frac{{\mu{\varsigma^{\alpha}}}}{{S_{0}^{\alpha}\hat{h}_{f}^{\alpha}}}}\right)\frac{{{\mu^{\frac{\phi}{\alpha}}}}}{{S_{0}^{\phi}\hat{h}_{f}^{\phi}\Gamma(\mu)}}}{{\left({\frac{1}{{\left|{{h_{l}}}\right|}}\sqrt{\frac{{{2^{R}}-1}}{\rho}}}\right)}^{\phi}},$
(31)

where
$\varsigma\in\left[{0,\sqrt{{({{2^{R}}-1})}/{\rho}}/{{\left|{{h_{l}}}\right|}}}\right]$.
In addition, it is found in the high SNR regime that $\varsigma$ tends to $0$
for $\rho\to\infty$. Thus, if $\alpha\mu-\phi>0$,
$\Gamma\left({\left({\alpha\mu-\phi}\right){\alpha^{-1}},\mu{\varsigma^{\alpha}}\hat{h}_{f}^{-\alpha}S_{0}^{-\alpha}}\right)$
approaches to
$\Gamma\left({\left({\alpha\mu-\phi}\right){\alpha^{-1}}}\right)$. As a
consequence, the asymptotic expression of the outage probability in the case
of $\alpha\mu-\phi>0$ can be obtained as (III-A2).

###  A-B $\alpha\mu-\phi<0$

By using the asymptotic expression of the upper incomplete gamma function
$\Gamma\left(a,x\right)\simeq-x^{a}/a$ for $x\to 0$ if $a<0$ [18, Eq. 8.7.3],
the asymptotic outage probability in the high SNR regime can be obtained as

$\displaystyle P_{out}^{Type-I}$
$\displaystyle=\prod\limits_{k=1}^{M}{\int\limits_{0}^{\frac{1}{{\left|{{h_{l}}}\right|}}\sqrt{\frac{{{2^{R}}-1}}{\rho}}}{\frac{{\phi{\mu^{\frac{\phi}{\alpha}}}{x^{\phi-1}}}}{{S_{0}^{\phi}\hat{h}_{f}^{\phi}\Gamma(\mu)}}\frac{{\alpha{{\left({\frac{\mu}{{\hat{h}_{f}^{\alpha}S_{0}^{\alpha}}}}\right)}^{\frac{{\alpha\mu-\phi}}{\alpha}}}}}{{\phi-\alpha\mu}}{x^{\alpha\mu-\phi}}dx}}$
$\displaystyle={{{\left({\frac{{\phi{\mu^{\mu-1}}{{\left({{2^{R}}-1}\right)}^{\frac{{\alpha\mu}}{2}}}}}{{\left({\phi-\alpha\mu}\right)S_{0}^{\alpha\mu}\widehat{h}_{f}^{\alpha\mu}{\rho^{\frac{{\alpha\mu}}{2}}}\Gamma(\mu){{\left|{{h_{l}}}\right|}^{\alpha\mu}}}}}\right)}^{M}}}.$
(32)

Finally, the asymptotic expressions of the outage probability for Type I-HARQ-
aided THz communications can be derived as (III-A2).

##  Appendix B Proof of (27)

To obtain the asymptotic expression of (26) under high SNR, it is obligatory
to derive the asymptotic expression of the extended Fox’s H function for
$\rho\to\infty$. By using the residue theorem, the extended Fox’s H function
in (26) is asymptotic to (35), as shown at the top of the next page, where
${\mathop{{\rm{Res}}}\limits_{s=a}}\\{f(s)\\}$ denotes the residue of $f(s)$
at $s=a$ and the second step holds by using dominant term approximation under
the condition of $\rho\to\infty$.

$\displaystyle
H_{0,1}^{1,2}\left[{\left.{{{\left({\rho{{\left|{{h_{l}}}\right|}^{2}}t}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}}\right|\begin{array}[]{*{20}{c}}{\left({1-\phi,1}\right),\left({1-\mu,\frac{1}{\alpha}}\right)}\\\
{\left({0,\frac{1}{2}}\right),\left({-\phi,1}\right)}\end{array}}\right]$ (35)
$\displaystyle=-\sum\nolimits_{\scriptstyle\left\\{{a:a=\phi+n,n\in[0,\infty]}\right\\}\hfill\atop\scriptstyle\bigcup{\left\\{{a:a=\mu\alpha+n\alpha,n\in[0,\infty]}\right\\}}\hfill}{\mathop{{\rm{Res}}}\limits_{s=a}\left\\{{\frac{{\Gamma\left({\frac{s}{2}}\right)\Gamma\left({\phi-s}\right)\Gamma\left({\mu-\frac{s}{\alpha}}\right)}}{{\Gamma\left({1+\phi-s}\right)}}{{\left({{{\left({\rho{{\left|{{h_{l}}}\right|}^{2}}t}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}}\right)}^{-s}}}\right\\}}$
$\displaystyle\simeq-\mathop{\lim}\limits_{s\to\phi}(s-\phi)\frac{{\Gamma\left({\frac{s}{2}}\right)\Gamma\left({\phi-s}\right)\Gamma\left({\mu-\frac{s}{\alpha}}\right)}}{{\Gamma\left({1+\phi-s}\right)}}{\left({{{\left({\rho{{\left|{{h_{l}}}\right|}^{2}}t}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}}\right)^{-s}}$
$\displaystyle-\mathop{\lim}\limits_{s\to\mu\alpha}(s-\mu\alpha)\frac{{\Gamma\left({\frac{s}{2}}\right)\Gamma\left({\phi-s}\right)\Gamma\left({\mu-\frac{s}{\alpha}}\right)}}{{\Gamma\left({1+\phi-s}\right)}}{\left({{{\left({\rho{{\left|{{h_{l}}}\right|}^{2}}t}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}}\right)^{-s}}$
$\displaystyle={\left({\rho
t}\right)^{-\frac{\phi}{2}}}\underbrace{\Gamma\left({\frac{\phi}{2}}\right)\Gamma\left({\mu-\frac{\phi}{\alpha}}\right){{\left({{{\left({{{\left|{{h_{l}}}\right|}^{2}}}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}}\right)}^{-\phi}}}_{{B_{k}}}+{\left({\rho
t}\right)^{-\frac{{\mu\alpha}}{2}}}\underbrace{\frac{\alpha}{{\phi-\mu\alpha}}\Gamma\left({\frac{{\mu\alpha}}{2}}\right){{\left({{{\left({{{\left|{{h_{l}}}\right|}^{2}}}\right)}^{\frac{1}{2}}}{{\left({\frac{\mu}{{h_{f}^{\alpha}S_{\rm{0}}^{\alpha}}}}\right)}^{-\frac{1}{\alpha}}}}\right)}^{-\mu\alpha}}}_{{C_{k}}},$
(36)

By substituting (35) into (26) and by change of variable as by $y=\rho t$, it
leads to

$P_{out}^{CC}\simeq{\left({\frac{\phi}{{2\Gamma(\mu)}}}\right)^{M}}\frac{1}{{2\pi\rm
i}}\int_{\alpha{\rm{-i}}\infty}^{\alpha{\rm{+i}}\infty}{\frac{{{e^{\left({{2^{R}}-1}\right)\frac{y}{\rho}}}}}{y}}\\\
\times\prod\limits_{k=1}^{M}{\left({{B_{k}}{y^{-\frac{\phi}{2}}}+{C_{k}}{y^{-\frac{{\mu\alpha}}{2}}}}\right)}dy.$
(37)

(37) can be further expanded as

$\displaystyle P_{out}^{CC}$
$\displaystyle\simeq{\left({\frac{\phi}{{2\Gamma(\mu)}}}\right)^{M}}\frac{1}{{2\pi\rm
i}}\int_{\alpha{\rm{-i}}\infty}^{\alpha{\rm{+i}}\infty}{\frac{{{e^{\left({{2^{R}}-1}\right)\frac{y}{\rho}}}}}{y}}$
$\displaystyle\times\sum\limits_{{\bf{a}}\in{\bf{\Omega}}}{\prod\limits_{k=1}^{M}{{{\left({{B_{k}}{y^{-\frac{\phi}{2}}}}\right)}^{{a_{k}}}}{{\left({{C_{k}}{y^{-\frac{{\mu\alpha}}{2}}}}\right)}^{1-{a_{k}}}}}}dy$
$\displaystyle={\left({\frac{\phi}{{2\Gamma(\mu)}}}\right)^{M}}\sum\limits_{{\bf{a}}\in{\bf{\Omega}}}{\prod\limits_{k=1}^{M}{{B_{k}}^{{a_{k}}}{C_{k}}^{1-{a_{k}}}}}$
$\displaystyle\times\frac{1}{{2\pi\rm
i}}\int_{\alpha{\rm{-i}}\infty}^{\alpha{\rm{+i}}\infty}{{e^{\left({{2^{R}}-1}\right)\frac{y}{\rho}}}{y^{-\sum\limits_{k=1}^{M}{\left({\frac{{\mu\alpha}}{2}\left({1-{a_{k}}}\right)+\frac{\phi}{2}{a_{k}}}\right)}-1}}dy},$
(38)

where
${\bf{\Omega}}=\left\\{{{\left({{a_{1}},\cdots,{a_{M}}}\right)}:{a_{k}}=\left\\{{0,1}\right\\},1\leq
k\leq M}\right\\}$. By means of inverse Laplace transform [16, Eq. 3.382.7],
(B) can be simplified as

$\displaystyle P_{out}^{CC}\simeq$
$\displaystyle{\left({\frac{\phi}{{2\Gamma(\mu)}}}\right)^{M}}\sum\limits_{{\bf{a}}\in{\bf{\Omega}}}{\frac{{\prod\limits_{k=1}^{M}{{B_{k}}^{{a_{k}}}{C_{k}}^{1-{a_{k}}}}}}{{\Gamma\left({{M\frac{{\mu\alpha}}{2}+{\frac{{\phi-\mu\alpha}}{2}}\sum\limits_{k=1}^{M}{{a_{k}}}}+1}\right)}}}$
$\displaystyle\times{{\left({\frac{{{2^{R}}-1}}{\rho}}\right)}^{M\frac{{\mu\alpha}}{2}+{\frac{{\phi-\mu\alpha}}{2}}\sum\limits_{k=1}^{M}{{a_{k}}}}}.$
(39)

In the high SNR regime, i.e., $\rho\to\infty$, to get the dominant terms
$\rho^{-{\left(M\frac{{\mu\alpha}}{2}+{\frac{{\phi-\mu\alpha}}{2}}\sum\nolimits_{k=1}^{M}{{a_{k}}}\right)}}$
amounts to finding the minimum value of the corresponding exponent
${M\frac{{\mu\alpha}}{2}+{\frac{{\phi-\mu\alpha}}{2}}\sum\nolimits_{k=1}^{M}{{a_{k}}}}$.
Clearly, the minimum of the exponent depends on the sign of
${\phi-\mu\alpha}$. More specifically, the minimum of the exponent is attained
with $a_{1}=\cdots=a_{M}=0$ if ${\phi-\mu\alpha}>0$ and $a_{1}=\cdots=a_{M}=1$
otherwise. With this result, (B) can be finally asymptotically expressed as

$\displaystyle P_{out}^{CC}\simeq$
$\displaystyle\left\\{{\begin{array}[]{*{20}{c}}{\frac{{{{\left({\Gamma\left({\frac{\phi}{2}+1}\right)}\right)}^{M}}}}{{\Gamma\left({\frac{{\phi
M}}{2}+1}\right)}}{{\left({\frac{{\Gamma\left({\frac{{\alpha\mu-\phi}}{\alpha}}\right){\mu^{\frac{\phi}{\alpha}}}{{\left({{2^{R}}-1}\right)}^{\frac{\phi}{2}}}}}{{\Gamma(\mu)\hat{h}_{f}^{\phi}S_{0}^{\phi}{{\left|{{h_{l}}}\right|}^{\phi}}{\rho^{\frac{\phi}{2}}}}}}\right)}^{M}},}&{\mu\alpha-\phi>0}\\\
{\frac{{{{\left({\Gamma\left({\frac{{\mu\alpha}}{2}+1}\right)}\right)}^{M}}}}{{\Gamma\left({\frac{{\mu\alpha
M}}{2}+1}\right)}}{{\left({\frac{{\phi{\mu^{\mu-1}}{{\left({{2^{R}}-1}\right)}^{\frac{{\alpha\mu}}{2}}}}}{{\Gamma(\mu)\left({\phi-\alpha\mu}\right)\hat{h}_{f}^{\alpha\mu}S_{0}^{\alpha\mu}{{\left|{{h_{l}}}\right|}^{\alpha\mu}}{\rho^{\frac{{\alpha\mu}}{2}}}}}}\right)}^{M}},}&{\mu\alpha-\phi<0}\end{array}}\right.$
(42)

By comparing (B) with (III-A2), the asymptotic outage probability of HARQ-CC-
aided THz communications can be rewritten as (27) in the end.

## References

  * [1] H. Zhang, L. Zhang, and X. Yu, “Terahertz band: Lighting up next-generation wireless communications,” _China Commun._ , vol. 18, no. 5, pp. 153–174, May 2021.
  * [2] H. Tataria, M. Shafi, A. F. Molisch, M. Dohler, H. Sjöland, and F. Tufvesson, “6G wireless systems: Vision, requirements, challenges, insights, and opportunities,” _Proc. IEEE_ , vol. 7, no. 7, pp. 1166–1199, Mar. 2021.
  * [3] I. F. Akyildiz, A. Kak, and S. Nie, “6G and beyond: The future of wireless communications systems,” _IEEE Access_ , vol. 8, pp. 133 995–134 030, Jul. 2020.
  * [4] S. Liu, X. Yu, R. Guo, Y. Tang, and Z. Zhao, “THz channel modeling: Consolidating the road to THz communications,” _China Commun._ , vol. 18, no. 5, pp. 33–49, May 2021.
  * [5] A.-A. A. Boulogeorgos, E. N. Papasotiriou, and A. Alexiou, “Analytical performance assessment of THz wireless systems,” _IEEE Access_ , vol. 7, no. 1, pp. 11 436–11 453, Jan. 2019.
  * [6] E. N. Papasotiriou, A.-A. A. Boulogeorgos, and A. Alexiou, “Performance analysis of THz wireless systems in the presence of antenna misalignment and phase noise,” _IEEE Commun. Lett._ , vol. 24, no. 6, pp. 1211–1215, Mar. 2020.
  * [7] J. Ye, S. Dang, B. Shihada, and M.-S. Alouini, “Modeling co-channel interference in the THz band,” _IEEE Trans. Veh. Technol._ , pp. 1–1, Jun. 2021.
  * [8] Q. Xia and J. M. Jornet, “Expedited neighbor discovery in directional Terahertz communication networks enhanced by antenna side-lobe information,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 8, pp. 7804–7814, Jun. 2019.
  * [9] Z. Li, L. Guan, C. Li, and A. Radwan, “A secure intelligent spectrum control strategy for future THz mobile heterogeneous networks,” _IEEE Commun. Mag._ , vol. 56, no. 6, pp. 116–123, Jun. 2018.
  * [10] A.-A. A. Boulogeorgos and A. Alexiou, “Error analysis of mixed THz-RF wireless systems,” _IEEE Commun. Lett._ , vol. 24, no. 2, pp. 277–281, Dec. 2019.
  * [11] L. Lopacinski, M. H. Eissa, G. Panic, M. Brzozowski, A. Hasani, and R. Kraemer, “Implementation of a multi-core data link layer processor for THz communication,” in _Proc. IEEE. Veh. Technol. Conf. (VTC Spring)_ , Porto, Portugal, Jul. 2018, pp. 1–5.
  * [12] G. Caire and D. Tuninetti, “The throughput of hybrid-ARQ protocols for the Gaussian collision channel,” _IEEE Trans. Inf. Theory_ , vol. 47, no. 5, pp. 1971–1988, Jul. 2001.
  * [13] G. Yang, H. Zhang, Z. Shi, S. Ma, and H. Wang, “Asymptotic outage analysis of spatially correlated Rayleigh MIMO channels,” _IEEE Trans. Broadcast._ , vol. 67, no. 1, pp. 263–278, Mar. 2021.
  * [14] Z. Shi, C. Zhang, Y. Fu, H. Wang, G. Yang, and S. Ma, “Achievable diversity order of HARQ-aided downlink NOMA systems,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 1, pp. 471–487, Jan. 2020.
  * [15] L. Debnath and D. Bhatta, _Integral Transforms and Their Applications_. Boca Raton, FL, USA: CRC Press, 2010.
  * [16] I. S. Gradshteyn and I. M. Ryzhik, _Table of integrals, series, and products_. NewYork, NY, USA: Academic Press, 2014.
  * [17] I. S. Ansari, F. Yilmaz, M.-S. Alouini, and O. Kucur, “New results on the sum of Gamma random variates with application to the performance of wireless communication systems over Nakagami-m fading channels,” _Trans. Emerg. Telecommun. Technol._ , vol. 28, no. 1, p. e2912, Jan. 2017.
  * [18] F. W. Olver, _NIST handbook of mathematical functions_. NewYork, NY, USA: Cambridge University Press, 2010.


# Low Bandwidth Video-Chat Compression using Deep Generative Models

Maxime Oquab⋆, Pierre Stock⋆∗, Oran Gafni⋆, Daniel Haziza⋆, Tao Xu⋆, Peizhao
Zhang⋆, Onur Celebi⋆,
Yana Hasson†, Patrick Labatut⋆, Bobo Bose-Kolanu⋆, Thibault Peyronel⋆, Camille
Couprie⋆
⋆ Facebook, † INRIA, work achieved during internship at Facebook AI Research
{qas, pstock, oran, dhaziza, xutao, stzpz, celebio, plabatut, bobobose,
peyronel<EMAIL_ADDRESS><EMAIL_ADDRESS>Contributed equally

###### Abstract

To unlock video chat for hundreds of millions of people hindered by poor
connectivity or unaffordable data costs, we propose to authentically
reconstruct faces on the receiver’s device using facial landmarks extracted at
the sender’s side and transmitted over the network. In this context, we
discuss and evaluate the benefits and disadvantages of several deep
adversarial approaches. In particular, we explore quality and bandwidth trade-
offs for approaches based on static landmarks, dynamic landmarks or
segmentation maps. We design a mobile-compatible architecture based on the
first order animation model of Siarohin et al. In addition, we leverage SPADE
blocks to refine results in important areas such as the eyes and lips. We
compress the networks down to about 3 MB, allowing models to run in real time
on iPhone 8 (CPU). This approach enables video calling at a few kbits per
second, an order of magnitude lower than currently available alternatives.

##  1 Introduction

For many smartphone users around the world, video-calling remains unavailable
or unaffordable. These users are driven out of this fundamental connectivity
experience by the prohibitive cost of data plans or because they depend on
outdated technologies and infrastructures. For instance, networks might suffer
from congestion, poor coverage, power fluctuations and datarate limits – 2G
networks allow for a maximum of 30 kbits/s. However, with current
technologies, an acceptable video-call quality requires at least a stable 200
kbits/s connection.

Figure 1: We propose to authentically reconstruct faces in real-time on mobile
devices using a stream of compressed facial landmarks extracted from _driving_
or target frames. The identity of the sender is transmitted in one shot to the
receiver at the beginning of the call through a _reference_ or _source_ frame.
This approach is compatible with end-to-end encryption (E2EE).

Meanwhile, the research in generative models has now come to a point where the
quality of synthetic faces are sometimes indistinguishable from real videos
[13]. To name a few, we may cite Deep video portraits [18], X2Face [42], FSGAN
[27], Neural Talking Heads [46], the Bilayer model [45] and the First Order
Model [33]. This unprecedented performance can now be exploited to the benefit
of higher quality video calls. However, there remain important challenges to
address before generative models can offer ultra-low data-rate video-calling.
In particular, to unlock duplex video-calling for users with last-mile
connectivity issues or limited data plans, the models need to be light and
fast enough to run on mobile handsets. In addition, to deliver a more seamless
and authentic experience, the models should adapt to the current appearance of
the user without additional training. In this work, we focus on identifying
the best generative strategy compatible with real-time inference on device. We
discuss the following approaches:

  * •

The Neural Talking Heads model [46], which requires sending a stream of
landmarks in addition to an initial face embedding.

  * •

The Bilayer model [45], where the face is reconstructed from a stream of
landmarks and a reference frame sent once.

  * •

The SegFace model, a novel architecture based on SPADE [28], adapted to face
animation, which requires sending an initial face embedding and a stream of
semantic segmentation maps.

  * •

The First Order Model (FOM) [33], which requires sending ten landmarks, their
associated motion matrices, and one reference frame.

Analysing the FOM in depth, we observe that only sending the landmarks
compressed with Huffman coding (no motion matrices) achieves sufficient
quality and leads to an outstanding data-rate reduction. Compared to other
approaches, this model allows for good identity and background preservation.
Our contributions are the following:

  * •

We provide a comparative analysis of leading generative approaches for the
specific use-case of enabling ultra-low data-rate video calling.

  * •

We develop a strong baseline leveraging the SPADE architecture and
segmentation maps.

  * •

We propose a warping based approach leveraging SPADE blocks to refine
important face attributes such as eyes and lips.

  * •

While previous approaches were tested on specialized hardware (servers, mobile
GPU), we provide first real-time results on mobile CPU.

##  2 Related work

###  2.1 Face compression before deep learning

The idea of face-specific video compression is not novel and appeared with
classical computer vision tools, for instance morphings using Delaunay
triangulations, Eigenfaces, or 3D models. The first reference we found on the
topic is the work of Lopez et al. [24] that proposes to encode only pose
parameters of a 3D head model, which is projected to reproduce a video
sequence.

Previous work [20] use PCA to model the current frame as a linear combination
of three basis frames sent prior to the call. The authors rely on known
control points on the face boundaries and landmarks. The principal drawback of
the approach is the presence of triangulation artefacts, even when a large
number of control points is used. The achieved bandwidth is 1500 bits/frame.
Similar usage of Eigenspaces are suggested in [37, 36, 34]. Among these
proposals using Eigenspaces, one claims an extremely low bit-rates achievement
of 100 bits/s [35]. However the proposed solution is hard to scale, as it
requires storing personal galleries of face images to reconstruct videos at
the receiver side.

###  2.2 Deep compression

The emergence of Generative Adversarial Networks (GANs) stimulated the
application of deep learning to video compression. Super-resolution has been
an active field of research leveraging GANs for image and video compression.
There have been a number of research works tackling this problem [8, 38, 4].
However, for compressing faces, these reconstructions methods are limited to
restoring personal traits from low level images and only work well for limited
upscaling factors (around $2\times$ in resolution). The power of GANs for
lossy image compression started to be demonstrated in the Generative
compression work of Santukar et al. [32], using an auto-encoder combined with
adversarial training. The state-of-the-art has since improved with the Extreme
Learned Image Compression work of Agustsson et al. [1], thanks to a multi-
scale architecture and the usage of semantic segmentation information, among
other tricks used by the authors. The work of Liu et al. [22] surveys deep
learning-based approaches for general purpose video compression. Among them,
Learned Video Compression [29] demonstrates for the first time the superior
capacity of an end-to-end machine learning approach over standard codecs. By
focusing on faces only, we can lower the bandwidth, improve the quality and
compress models compared to using more generic methods. Therefore, we review
next deep face videos reconstruction approaches and their adequacy to video
chat compression.

###  2.3 Deep talking head approaches

3D based approaches produce realistic avatars which can be animated in real-
time [5]. However, such methods require to capture a set of images of the user
(a few dozens) to build their personal face model. PAGAN [25] generates key
face expression textures that can be deformed and blended in real-time on
mobile from a single frame. However, the reconstruction of certain features,
notably the hair, is still problematic in 3D model-based approaches. Deep
video portraits [18] is handling this issue using a rendering-to-video
translation network, but the approach needs about a thousand images per
subject for training. Stimulated by advancements in face swapping pipelines
[19, 42], a number of deep generative re-enactment approaches arose. Contrary
to warping based re-enactment [2], learning faces reconstructions enables
extra robustness in presence of large head angles. The Face Swapping GAN [27]
relies on several steps: landmarks extraction, segmentation, interpolation and
inpainting. This complex pipeline may result in robustness issues and limited
bandwidth gain due to the need of sending both compressed segmentations and
landmarks. Similarly, the vid2vid approach ([41], [40]) requires sending a
“sketch” (edge map) for each frame in order to re-enact a face, which has a
relatively high bandwidth cost. In the next section, we discuss and compare
the learning based approaches which yield the most promising results in terms
of bandwidth, visual quality and inference time.

Figure 2: Scheme of principle for the different deep generative approaches
discussed in this study. In particular, we detail two novel architectures,
SegFace and Hybrid Motion-SPADE (right) and compare them to existing NTH [46]
and FOM [33] models (left). For all models, we assume the generation is
performed by the encoder-decoder pair on the receiver device, while the
emitter sends a reference frame (or several) at the beginning of inference,
and streams a series of landmarks or segmentation maps.

##  3 Generative models

In this section we describe in-depth several recent face animation algorithms
that we have implemented and studied. We share our understanding of these
works and present our two model contributions, namely SegFace and Hybrid
Motion-SPADE. An overview of these different models appears in Fig. 2. For
this self-reenactment task, unless mentioned otherwise, the goal of all these
approaches is to generate a frame based on (i) one fixed _reference_ or
_source_ frame and (ii) position information (e.g. landmarks) from a stream of
_driving_ or _target_ frames (see Figure 1).

###  3.1 Talking heads (NTH) and Bilayer model

The “Talking Heads” work of Zakharov et al. [46] learns to synthesize videos
of people from facial landmarks given one or few reference images. It follows
an encoder-decoder architecture with a style transfer component:

  * •

A set of style parameters is computed for the set of reference images.

  * •

Facial landmarks are plotted as images and processed by an encoder network.

  * •

The resulting code is decoded with style transfer, using Adaptive Instance
Normalization [15] layers, adjusting the mean and standard deviation of each
feature map with the style parameters.

The networks are trained end-to-end with adversarial and perceptual losses on
a dataset of videos. The best results are achieved by performing a fine-tuning
training phase on the generator to match the reference frames as precisely as
possible. This fine-tuning phase requires several minutes on a modern server
GPU. Without fine-tuning, the identity is not preserved as well in the
generated frames. In practice, a few hundreds of frames would have to be sent
at the beginning of the call.

This work was further improved in the Bilayer Synthesis approach [45], where
the fine-tuning step is not required anymore, and leads to visually appealing
and sharp results. In our observations (see Figure 3), the identity
preservation suffers from a stronger uncanny valley effect.

In terms of bandwidth, the NTH and Bilayer approaches require sending 68
compressed landmarks.

###  3.2 First order model for image animation (FOM)

The “First Order Model” approach of Siarohin et al. [33] deforms a reference
source frame to follow the motion of a driving video. While this method works
on various types of videos (Tai-chi, cartoons), we focus here on the face
animation application. FOM follows an encoder-decoder architecture with a
motion transfer component:

  * •

A landmark extractor is learned using an equivariant loss, without explicit
labels.

  * •

Two sets of ten learned landmarks are computed for the source and driving
frames.

  * •

A dense motion network uses the landmarks and the source frame to produce a
dense motion field and an occlusion map.

  * •

The encoder encodes the source frame.

  * •

The resulting feature map is warped using the dense motion field (using a
differentiable grid-sample operation [17]), then multiplied with the occlusion
map.

  * •

The decoder generates an image from the warped map.

The networks are trained end-to-end on video frames, using perceptual losses,
and are then optionally fine-tuned with an adversarial discriminator. The
self-supervised landmarks do not necessarily match precise locations of the
face. Instead, they correspond to point coordinates that are optimized to
achieve the best deformation of the source frame. [33] describes how to
improve motion approximation in landmark areas by estimating Jacobian matrices
to model motion in their neighborhood. In our observations (see Table 5), this
approach preserves identities better than NTH and is at least on par with the
follow-up Bilayer synthesis approach. Next, we study variants of this
approach.

#### Variants

First, our implementation does not use the Jacobian component, as we do not
observe a strong effect the quality of the results. We refer to the resulting
model as “Motion Net (MN-10)” as it no longer uses first order approximation
anymore and employs a set of ten landmarks.

Second, we explore using off-the-shelf facial landmarks extraction to replace
the unsupervised landmarks. In this case, we only stream 20 or 68 compressed
landmarks.

Third, we explore a combined strategy employing both 10 self-supervised
landmarks and 20 supervised ones, that we note MN-10+20. We introduce a fourth
variant in Section 3.4, after detailing our SegFace approach below.

###  3.3 SegFace

This approach builds upon [28]. Unlike MaskGAN [21], we propose to use a face
descriptor computed on a source frame, and decode it by conditioning on face
segmentation maps from a driving frame. It follows an encoder-decoder
architecture described as follows:

  * •

A face descriptor is computed on a source frame.

  * •

This face descriptor is given to a decoder network, that applies SPADE
normalization blocks at each layer using the face segmentation maps of the
driving frame, ensuring all parts of the face are correctly placed.

The decoder network is trained using VGGFace2 face embeddings [6], and
segmentation maps from [44] as inputs. Its objective during training is to
reconstruct the same source frame. The optimization is done using losses from
[28], and the face perceptual loss from [14]. This method operates on
independent frames, and thus allows to use high-resolution training data,
leading to high quality results. Training is achieved using CelebA [23] and
Flickr-Faces-HQ datasets.

#### Bandwidth

The model requires a segmentation map labeled for 15 categories (eyes, hairs,
ears etc.). Sending compressed segmentation maps would require 18/25 kbits/s
at resolutions $48\times$/$64\times$, knowing that there is a trade-off
between the resolution of the transmitted segmentation maps and the quality of
the generated faces. We do not build on this method further for low-bandwidth
video-chat because the cost of running a face parser inference step and the
bandwidth requirements are too high. The SegFace implementation, however,
allows us to observe that the generated images respect the segmentation map
labels almost perfectly, consistently with the conclusions of [21]. We will
build on this property in the next subsection with our Hybrid Motion-SPADE
approach.

###  3.4 Hybrid Motion-SPADE model

Important quality criteria for compressed video-chat include a good
synchronization between the lips and the speech, and a good rendering of the
eyes and eyebrows; therefore, it is crucial to generate these facial parts
precisely.

We propose an improvement over the FOM-based Motion Net method, by adding
SPADE normalization layers in the upsampling blocks of the decoder network (in
the last step of the FOM approach). We draw polygons for the eyes, eyebrows,
lips and inner mouth using 60 extracted face landmarks, and use these as
semantic maps for SPADE.

The dense motion network receives (i) a downsampled reference frame with (ii)
the positions of $N$ landmarks for that frame, and (iii) the positions of the
same landmarks for a driving frame. It outputs a motion field $M$ and an
occlusion map $O$. The encoder network outputs a feature map $F_{s}$. The
decoder warps $F_{s}$ with the result of the dense motion network $M$ and
multiplies it element-wise with the occlusion map $O$, to obtain $F_{w}$.
Then, $F_{w}$ is processed by a stack of five residual blocks and three
upsampling blocks that apply the SPADE normalization using a set of 60
landmarks.

Training is performed with a multiscale perceptual loss (based on a VGG-19
architecture) with a weight $\lambda_{p}=10$ in addition to an equivariance
loss with a weight $\lambda_{eq}=1$ for the unsupervised landmark detector
when applicable, following the procedure described in [33].

#### Bandwidth

The necessary segmentation maps are obtained by plotting the polygons of the
facial landmarks extracted using a landmark detector (see Figure 2), rather
than running a face segmentation network. Moreover, landmark coordinates are
inexpensive to transmit, while rasterized segmentation maps are more difficult
to compress, especially at higher resolutions. In terms of bandwidth, this
approach requires sending $N+60$ compressed landmarks. We experiment with
$N=10,20,$ and 30.

##  4 Compression

Model variant | Inputs | FPS | #Params | #FLOPS | int8 size | Raw BW | Compressed BW
---|---|---|---|---|---|---|---
Motion Net | 10 U | 18 | 2.9 M | 1411 M | 3.1 MB | 3.9 kbits/s | 1.4 kbits/s
Motion Net | 20 L | 19 | 2.3 M | 1293 M | 2.5 MB | 7.8 kbits/s | 2.2 kbits/s
Motion Net | 10 U + 20 L | 14 | 3.0 M | 1505 M | 3.4 MB | 11.7 kbits/s | 3.6 kbits/s
Motion SPADE | 10 U | 16 | 2.9 M | 1198 M | 3.2 MB | 27.3 kbits/s | 8.0 kbits/s
Motion SPADE | 20 L | 19 | 2.3 M | 1029 M | 2.5 MB | 41.2 kbits/s | 8.8 kbits/s
Motion SPADE | 10 U + 20 L | 13 | 3.0 M | 1292 M | 3.4 MB | 35.3 kbits/s | 10.2 kbits/s
Table 1: Comparison of our approaches running on mobile in terms of
compression for both model size and stream.“10 U” (resp. “20 L”) means that 10
unsupervised landmarks (resp. 20 facial landmarks) are used as inputs to the
dense motion network. SPADE variants require 60 extra facial landmarks to draw
the facial label maps. Notes: the “int8 size” is the full combined size of the
models. The number of frames per second (FPS) is measured for the whole
int8-quantized pipeline running on an iPhone 8, including landmark detection,
grid-samples and face alignment. The #FLOPS count is for the dense motion,
decoder, and unsupervised landmark extractor networks. The bandwidth (BW) is
measured at 25 FPS, without (Raw BW) and with Huffman encoding (Compressed
BW).

In this section, we explain different strategies to make architectures – and
in particular our novel hybrid Motion-SPADE – compatible with low-bandwidth
video calls on mobile. We first detail the architectures and then the
compression aspects for the models and the bandwidth. Results are displayed in
Table 1.

###  4.1 Mobile architectures

#### Base blocks

We rely on the open-source FbNet family of architectures [43, 39, 11] to
design mobile-capable models for our Motion Net and Motion-SPADE approaches.
These networks typically build on blocks combining $1\times 1$ point-wise and
$3\times 3$ depth-wise convolutions [31] that require less floating-point
operations than traditional $3\times 3$ convolutions found in residual blocks.
We provide further architecture details in Figure 5.

#### Mobile SPADE normalization blocks

When applicable, we perform a SPADE normalization after the last $1\times 1$
point-wise convolution, with kernel sizes of $1\times 1$, and 32 hidden
channels. We have found these parameters to provide a good trade-off between
speed and quality while preserving the fidelity of the SPADE approach.

###  4.2 Landmark stream compression

We compress the landmarks with Huffman encoding [16]. In this approach, the
landmark displacements are first binarized into 32 bins plus one sign bit, and
we encode the bin index with Huffman coding. This compression leads to an
average rate of 90 bits/frame for 20 landmarks, hence 2.2 kbits/s at 25 FPS
(see Table 1 for details). For reference, bandwidth requirements for audio are
around 10 kbits/s, while the AV1 video codec (not widely hardware-supported to
date) aims at 30 kbits/s [10]. Therefore, we did not explore other variants
such as Arithmetic Coding [30] since the audio part takes most of the
bandwidth of a call with the proposed approach.

###  4.3 Model quantization

We rely on int8 post-training quantization. This technique simply consists in
uniformly quantizing both weights and activations over 8 bits, thus reducing
the model size by a factor $4$. Moreover, int8 models traditionally benefit
from a $\times 2-3$ speed-up compared to their fp32 counterparts for both
server and mobile CPUs. The scale and zero-point parameters111The affine
transform coefficients that allow converting an 8-bit quantized tensor
(integer-valued in $[0,255]$) to its floating-point counterpart. of the
quantized layers are calibrated after training using a few batches of training
data. When not properly calibrated, we found that the decoder generates an
image with a small amount of grain or noise, resulting in a loss of visual
quality.

To compress the Motion based models, we only rely on int8 since the non-
compressed models are already small. The models are converted to TorchScript
and run on the phone’s CPU. These results are displayed in Table 1.

###  4.4 Implementation details

Our mobile models are trained on the DFDC dataset [13] rather than the
VoxCeleb [26] dataset, in contrast to the original work of [33] (though we
provide evaluation numbers for comparison and reference). We split different
identities following a 90%-10% ratio, resulting in a total of 21899 training
videos, and 2369 validation videos. We choose DFDC in this work because the
videos are higher-quality and not cropped as tight, allowing for different
face alignment procedures: (i) cropping around the face, or (ii) cropping
after rotation using the facial landmarks such that the eyes are horizontally
aligned. We have notably found that for smaller Motion Net models, this
alignment makes the task easier and improves the results. The alignment
procedure is reproduced on mobile at inference time to match the training
distribution.

We perform training on 8 GPUs using a Distributed Data Parallel pipeline in
Pytorch, with a batch size of 48, for 265K steps. We use the Adam optimizer
with learning rate $2.10^{-4}$ on all networks in all experiments.

##  5 Experiments

###  5.1 Evaluation metrics

We evaluate the models using the perceptual LPIPS [47] and multi-scale LPIPS-
like metrics employed in [33], that we name msVGG. Second, as argued in [7],
the cosine similarity CSIM computed between features of the pre-trained face
embedding network ArcFace [12] is one of the most effective metric to assess
quality of talking heads models, we therefore report it. Finally, we quantify
facial landmarks mismatch by running a landmark detector on the true and
generated videos and computing the Mean Square Error between each pair of
landmarks. This metric is classically referred to as the Normalized Mean Error
(NME) of head pose [3]. All the generative approaches considered in this work
are trained using different alignments and close-ups (see Figure 3), so we
compute our metrics using the optimal modified videos for each method as
ground truth.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
Source | Target | Seg2Face | NTH | Bilayer | FOM adv | Mob MS-20L
|  | #param: 126M | #param: 34M | #param: 144M | #param: 47M | #param:3M
Figure 3: Comparison of different results using Seg2Face ($48\times 48$), NTH,
Bilayer, and FOM adv. Each model generates the face using the fixed source
frame and the facial information (such as landmarks) of the driving frame. We
pasted ground truth backgrounds to have a fair evaluation. The last column
showcases the results using our Mobile Motion-SPADE that runs at 18 FPS on an
iPhone 8, whereas the other models run on server, have at least 10$\times$
more parameters and are not necessarily compatible with low-bandwidth video
calling. Note that the alignment procedure may differ between the models,
hence the head is not centered the same on the generated faces.

###  5.2 Quality evaluation: ablation studies

For evaluation, we assembled a set of 28 videos of diverse persons in terms of
gender, age, skin color from the validation set of VoxCeleb2 [9], and a
similar set of 50 videos from the validation split of the DFDC dataset [13].

We begin our analysis of the FOM by computing the quality of reconstruction
without first order motion approximation and without adversarial training in
Table 2. While it is clear that the adversarial fine-tuning boosts the
performance, we experiment without it in the remaining of our ablation study
around this model to reduce training time for each model. Removing the first
order approximation only slightly degrades the LPIPS but not the msVGG
perceptual metric. Interestingly, the CSIM metric which is the one supposed to
best reflect the identify preservation, is slightly increased by dropping this
component. A second observation is that the fidelity of facial landmarks to
the target video is negatively affected by this removal. Since the drop of
performance induced by discarding first order motion approximation leads to
important bandwidth savings and limited loss in performance, we conduct our
experiments without it. We refer to this approach as the Motion Net approach.
Next, we explore the replacement of the self-supervised landmarks of the
Motion Net approach by off-the-shelf landmarks from a state-of-the art
detector. Results appear in Table 3. Note that the results presented in this
table are obtained by our re-implementation of the MotionNet approach, and are
slightly better than these of Table 2 obtained with the original code.

| FOM adv | FOM w/o adv | MN
---|---|---|---
msVGG $\downarrow$ | 85.6 | 87.5 | 87.9
LPIPS $\downarrow$ | 0.226 | 0.233 | 0.236
NME $\downarrow$ | 0.51 | 0.53 | 0.54
CSIM $\uparrow$ | 0.83 | 0.81 | 0.82
Table 2: Ablation study for FOM on VoxCeleb2-28. MN: FOM without first order
approximation nor adversarial fine-tuning.

We compare in Table 3 different variants of the Motion Net approach, using 20
input landmarks, 68 input landmarks, self-supervised landmarks with dense
architectures and with mobile architectures. All these dense architectures
employ a latent space of $256\times 64\times 64$, and were trained on
VoxCeleb. Using standard facial landmarks instead of unsupervised motion
landmarks degrades the scores of perceptual metrics, but improves NME. Using
68 landmarks is only very slightly improving the quality over 20. With mobile
architectures, we reduce the latent space to $256\times 32\times 32$. In
addition to using 10 motion landmarks or 20 landmarks alone, combining these
two sets helps boost all quality metrics. Finally, we observe that adding the
SPADE blocks preserves the perceptual quality and brings a large improvement
in NME.

| LPIPS $\downarrow$ | NME $\downarrow$ | CSIM $\uparrow$
---|---|---|---
Dense MN-10 U | 0.221 | 0.59 | 0.83
Dense MN-20 L | 0.242 | 0.50 | 0.80
Dense MN-68 L | 0.240 | 0.49 | 0.81
Mob MN-10 U | 0.225 | 0.52 | 0.79
Mob MN-20 L | 0.244 | 0.48 | 0.78
Mob MN-10 U + 20 L | 0.218 | 0.46 | 0.80
Mob M-SPADE-10 U | 0.217 | 0.47 | 0.81
Mob M-SPADE-20 L | 0.242 | 0.44 | 0.79
Mob M-SPADE-10 U + 20 L | 0.215 | 0.46 | 0.81
Table 3: Evaluation results for Motion Net approaches without adversarial
fine-tuning on the VoxCeleb2-28 video subset. Mob : Mobile models. Dense
models ($64\times 64$ latent space) are trained on VoxCeleb. Mobile models
($32\times 32$) are trained on the DFDC aligned dataset. U: unsupervised
landmarks; L: facial landmarks.

###  5.3 Quantitative comparative evaluation

We compare the quality/bandwidth trade-off of different dense face animation
approaches in Table 4. As the different models were trained using different
data pre-processing (different crops, alignment), we evaluate each one in the
setting allowing the best generations. This means that synthesized videos need
to be compared to different source videos. Therefore, we paste the ground
truth video background on the generated result so that the metrics focus on
evaluating face differences only. We observe that the NTH results lead to
better NME, and FOM to better LPIPS and CSIM metrics. SegFace numerical
results lower, partly due to a color shift appearing in the results.
Interestingly, our mobile results have better NME and msVGG scores than the
original FOM dense approach.

| NTH | Bilayer | SegFace | FOM | MS20L
---|---|---|---|---|---
msVGG* $\downarrow$ | 56.3 | 68.6 | 84.4 | 58.7 | 57.9
LPIPS* $\downarrow$ | 0.165 | 0.200 | 0.304 | 0.153 | 0.167
NME* $\downarrow$ | 0.38 | 0.55 | 0.55 | 0.50 | 0.44
CSIM* $\uparrow$ | 0.83 | 0.85 | 0.76 | 0.87 | 0.84
kbits/s $\downarrow$ | $9.7^{+}$ | 9.7 | 18 | 4.0 | 8.8
Table 4: Comparison of Bilayer, SegFace ($48\times 48$), FOM adv, NTH in terms
of quality / bandwidth (kbits/s with 25 fps) trade-offs on VoxCeleb2-28. * :
Metrics were computed using ground truth backgrounds. We also include our best
mobile model, Motion-SPADE-20L (MS20L) in the comparison. +: Bandwidth
computed without considering the mandatory frames transmission necessary to
the fine-tuning step.

Sources |  Targets |  MN-10 |  MN-20 |  MN-10+20 |  M-SP-10 | M-SP-10+20 |  M-SP-20 | H264 9kb/s
---|---|---|---|---|---|---|---|---

Figure 4: Qualitative results using Motion based variants on mobile
architectures, using a $32\times 32\times 256$ latent space. Each model
generates the face given the fixed source frame and the landmarks of the
target frame. All models run in real-time on an iPhone 8.

###  5.4 Qualitative evaluation and human study

Figure 3 compares results obtained using SegFace, Bilayer, NTH and FOM with
adversarial finetuning. We observe skin tones/lightning differences between
targets and SegFace results and distortions of personal traits. The Bilayer,
NTH and FOM models are qualitatively better. In the last column, we observe a
side by side comparison with a Mobile Motion-SPADE model.

Table 5 provides a quality assessment of different models by human raters.
Participants are asked to rate images produced by the different models by
comparing them in terms of identity and expression preservation, on a scale
from 1 to 5. In a first round of evaluations, we display side by side the four
main dense models results, and in the second round, Motion Net and Motion-
SPADE results using six different mobile architectures. We collect in each
case 500 pairwise evaluations, each from five different participants. For
dense models results, human scores seem to agree with metrics, ranking FOM
first and NTH second. Mobile models results differences are more subtle, but
the Hybrid Motion-SPADE using 10 landmarks model seems preferred. The addition
of SPADE blocks brings significant improvement in most cases.

Dense models
---
| human identity | human expression
model | score | score
NTH | 3.71$\pm$0.041 | 3.77$\pm$0.040
Bilayer | 3.70$\pm$0.046 | 3.62$\pm$0.046
SegFace | 3.11$\pm$0.047 | 3.00$\pm$0.051
FOM adv |  3.99$\pm$0.042 |  4.00$\pm$0.041

Overall human ratings of Mobile models
---
| MN-10 | MN-20 | MN-10+20
no SPADE | 3.44$\pm 0.034$ | 3.40$\pm 0.034$ | 3.46$\pm 0.034$
with SPADE | 3.50$\pm 0.034$ | 3.46$\pm 0.035$ | 3.45$\pm 0.034$

Table 5: Quality assessment of different dense models on DFDC-50 - Human
study. Average scores (Higher is better) with confidence intervals.

Figure 4 illustrates the quality performance reached on Mobile. The two last
lines display challenging cases for the motion based approach. We note that
the quality of results degrades in presence of large head rotation. In the
last line, a fan on top of the head combined with a bad lightning causes
errors in hair reconstruction. Still, the Motion-SPADE results are visually
close to the targets, particularly it renders lips and teeth better. The H264
compression results are displayed given a bandwidth of 9 kbit/s, to be
compared to the ones of the Motion-SPADE 20 model that runs the fastest on
mobile. This illustrates that at this bandwidth, video transmission is hardly
possible using standard codecs, whereas our mobile approach would make the
video call possible.

##  6 Conclusions

Our exploration of state-of-the art animation models led us to the following
observations: The Neural Talking head results are qualitatively satisfactory,
but the fine-tuning step requirement makes the approach complex to implement
in practice. Using a full face segmentation approach seems unfit to a low
bandwidth application. The Bilayer approach and the FOM methods perform best
towards reaching a correct low-bandwidth/quality trade-off on mobile. Our
human study shows that FOM results are preferred in terms of identity and
expression preservation. Focusing on this best candidate approach, we design a
novel hybrid architecture taking advantage of the high fidelity to the target
thanks to the warping principle, and enhancing the quality of important
attributes with SPADE blocks. Only exploiting polygons induced segments allows
this approach to improve quality without high transmission cost. The obtained
image quality is close to the one reached by dense models while running in
real-time on Mobile CPU. The bandwidth required to send a video is lower than
the one required for sending audio. There are a number of interesting
challenges to tackle next to improve quality of the generations, e.g.
generating large head rotation movements, hands, or using pupils tracking.

## Acknowledgements

We would like to thank Elif Albuz, Bryan Anenberg, Stephanie Guáman, Hervé
Jegou, Stéphane Lathuilière, Aliaksandr Siarohin, Jakob Verbeek and Qiang
Zhang for their precious help.

## References

  * [1] Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool.  Generative adversarial networks for extreme learned image compression.  In ICCV, 2019.
  * [2] Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, and Michael F Cohen.  Bringing portraits to life.  Transactions on Graphics, 36(6):1–13, 2017.
  * [3] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks).  In ICCV, 2017.
  * [4] Adrian Bulat and Georgios Tzimiropoulos.  Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans.  In CVPR, 2018.
  * [5] Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou.  Real-time facial animation with image-based dynamic avatars.  Transactions on Graphics, 35(4), 2016.
  * [6] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  In International Conference on Automatic Face & Gesture Recognition, 2018.
  * [7] Lele Chen, Guofeng Cui, Ziyi Kou, Haitian Zheng, and Chenliang Xu.  What comprises a good talking-head video generation?: A survey and benchmark.  arXiv 2005.03201, 2020.
  * [8] Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang.  Fsrnet: End-to-end learning face super-resolution with facial priors.  In CVPR, 2018.
  * [9] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman.  Voxceleb2: Deep speaker recognition.  In INTERSPEECH, 2018.
  * [10] Dave Citron.  “four new google duo features to help you stay connected”, April 2020\.
  * [11] Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al.  Fbnetv3: Joint architecture-recipe search using neural acquisition function.  arXiv 2006.02049, 2020.
  * [12] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [13] Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer.  The deepfake detection challenge (dfdc) dataset.  arXiv 2006.07397, 2019.
  * [14] Oran Gafni, Lior Wolf, and Yaniv Taigman.  Live face de-identification in video.  In CVPR, 2019.
  * [15] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In ICCV, 2017.
  * [16] David A. Huffman.  A method for the construction of minimum-redundancy codes.  Proceedings of the Institute of Radio Engineers, 40(9):1098–1101, 1952.
  * [17] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In NeurIPS, pages 2017–2025, 2015.
  * [18] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt.  Deep video portraits.  Transactions on Graphics, 37(4), 2018.
  * [19] Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis.  Fast face-swap using convolutional neural networks.  In ICCV, 2017.
  * [20] Ioannis Koufakis and Bernard F Buxton.  Very low bit rate face video compression using linear combination of 2D face views and principal components analysis.  Image and Vision computing, 17(14):1031–1051, 1999.
  * [21] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  In CVPR, 2020.
  * [22] Dong Liu, Yue Li, Jianping Lin, Houqiang Li, and Feng Wu.  Deep learning-based video coding: A review and a case study.  Computing Surveys, 53(1):1–35, 2020.
  * [23] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In ICCV, 2015.
  * [24] Ricardo Lopez and Thomas S Huang.  Head pose computation for very low bit-rate video coding.  In International Conference on Computer Analysis of Images and Patterns, 1995.
  * [25] Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, and Hao Li.  pagan: real-time avatars using dynamic textures.  In SIGGRAPH Asia, page 258, 2018.
  * [26] Arsha Nagrani, Joon Son. Chung, and Andrew Zisserman.  Voxceleb: a large-scale speaker identification dataset.  In INTERSPEECH, 2017.
  * [27] Yuval Nirkin, Yosi Keller, and Tal Hassner.  FSGAN: Subject agnostic face swapping and reenactment.  In CVPR, 2019.
  * [28] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In CVPR, 2019.
  * [29] Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander Anderson, and Lubomir Bourdev.  Learned video compression.  In ICCV, 2019.
  * [30] Jorma Rissanen and Glen G. Langdon.  Arithmetic coding.  IBM Journal of Research and Development, 23(2):149–162, 1979.
  * [31] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In CVPR, 2018.
  * [32] Shibani Santurkar, David Budden, and Nir Shavit.  Generative compression.  In Picture Coding Symposium, 2018.
  * [33] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  In NeurIPS, 2019.
  * [34] Ulrik Söderström.  Very low bitrate facial video coding: based on principal component analysis.  PhD thesis, Tillämpad fysik och elektronik, 2006.
  * [35] Le-Hung Son, Ulrik Söderström, and Haibo Li.  Ultra low bit-rate video communication: video coding= pattern recognition, 2006.
  * [36] Luis Torres and Daniel Prado.  A proposal for high compression of faces in video sequences using adaptive eigenspaces.  In ICIP, 2002.
  * [37] Mihran Tuceryan and Bruce E Flinchbaugh.  Model based faced coding and decoding using feature detection and eigenface coding, Mar. 28 2000.  US Patent 6,044,168.
  * [38] Evgeniya Ustinova and Victor S. Lempitsky.  Deep multi-frame face hallucination for face identification.  arXiv, 1709.03196, 2017.
  * [39] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Peter Vajda, and Joseph E. Gonzalez.  Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions, 2020.
  * [40] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro.  Few-shot video-to-video synthesis.  In NeurIPS, 2019.
  * [41] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Video-to-video synthesis.  In NeurIPS, 2018.
  * [42] Olivia Wiles, A. Sophia Koepke, and Andrew Zisserman.  X2face: A network for controlling face generation using images, audio, and pose codes.  In ECCV, 2018.
  * [43] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer.  Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search, 2019.
  * [44] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In ECCV, 2018.
  * [45] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  ECCV, 2020.
  * [46] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  ICCV, 2019.
  * [47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.

##  7 Appendix

The details of our mobile architectures are provided in Figure 5.

---
(a) Mobile architectures (dense motion net and landmark extractor)
(b) Generator architecture (Input: driving frame, output: generation)
|
(c) Inverted residual (IRF) blocks | (d) SPADE blocks used in Motion-SPADE
Figure 5: Schemes of mobiles architectures for the Motion Net ((a), (c)) and
Motion-SPADE approaches ((a), (b), (c), (d)).