# PHOTON MASS AND VERY LONG BASELINE INTERFEROMETRY111This article is an
extended version of the essay ‘COMBINING GENERAL RELATIVITY, MASSIVE QED AND
VERY LONG BASELINE INTERFEROMETRY TO GRAVITATIONALLY CONSTRAIN THE PHOTON
MASS’ (A. Accioly, J. Helayël-Neto and E. Scatena, Phys. Lett. A 374 (2010)
3806) which was awarded an “honorable mention” in the 2010 Essay Competition
of the Gravity Research Foundation.

ANTONIO ACCIOLY${}^{\dagger,\;\star,\;\amalg}$ JOSÉ HELAYËL-
NETO${}^{\dagger,\;\diamond}$ and ESLLEY SCATENA${}^{\star,\;\ddagger}$
†Laboratório de Física Experimental (LAFEX),
Centro Brasileiro de Pesquisas Físicas (CBPF),
Rua Dr. Xavier Sigaud 150, Urca, 22290-180,
Rio de Janeiro, RJ, Brazil
⋆Instituto de Física Teórica (IFT),
São Paulo State University (UNESP),
R. Dr. Bento Teobaldo Ferraz 271, Bl. II - Barra Funda,
01140-070, São Paulo, SP, Brazil
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
(Day Month Year; Day Month Year)

###### Abstract

A relation between the photon mass, its frequency, $\nu$, and the deflection
parameter, $\gamma$, determined by experimentalists (which characterizes the
contribution of space curvature to gravitational deflection) is found. This
amazing result allows us to conclude that the knowledge of the parameters
$\nu$ and $\gamma$ is all we need to set up gravitational bounds on the photon
mass. By considering as inputs the most recent measurements of the solar
gravitational deflection of radio waves obtained via the Very Long Baseline
Interferometry, upper bounds on the photon mass are estimated.

###### keywords:

Gravitational deflection; massive QED; photon mass, Very Long Baseline
Interferometry.

Managing Editor

##  1 Introduction

As a consequence of the substantial improvement in the interferometry
techniques that took place over the last few decades, a new light was shed on
the physical phenomena in which gravitational and quantum effects are
interwoven. Consider, for instance, the quantum mechanical phase shift of
neutrons caused by their interaction with Earth’s gravitational field which
was observed in the Colella-Overhauser-Werner[1] and Bonse-Wrobleski[2]
experiments performed with neutron interferometry. These seminal and
pioneering scientific tests have helped dramatically to convince physicists
that there exist some experiments whose outcome necessarily depends upon both
the gravitational constant and the Planck’s constant. The fact that in the
great majority of phenomena of interest in terrestrial physics, gravity and
quantum mechanics do not simultaneously play an important role, is certainly
responsible for the wrong idea that gravity and quantum mechanics cannot be
closely intertwined in some special circumstances. In the last four decades
the COW experiments have become more sophisticated. The latest neutron
interferometry experiments[3] report a statistically significant discrepancy
between the theoretically predicted and experimentally measured values of the
neutron phase shift due to gravity. Supposing the equality of inertial and
gravitational masses for the neutron the experimenters found the neutron phase
factor to be $1\%$ lower than predicted,[3] which clearly signals a possible
violation of the classical equivalence principle. At first sight, it seems
that these latest neutron interferometry experiments are in conflict with the
more precise tests of the classical equivalence principle conducted via atomic
interferometry, and with those based on torsion pendulum. Adunas, Rodriguez-
Milla and Ahluwalia,[4] showed, however, that each of the aforementioned
experiments probes a different aspect of gravity; and that current experiment
techniques, when coupled to solar-neutrino data, may be able to explore
quantum mechanically induced violations of the classical equivalence
principle. They also predicted a quantum violation of the classical
equivalence principle for the next generation of atomic interferometry
experiments. Actually, from an operational point of view one cannot claim,
even in principle, that there exists, for certain quantum systems, an exact
equality of gravitational and inertial masses.[4, 5] Therefore, we come to the
conclusion that quantum mechanics and the classical equivalence principle
cannot coexist peacefully.[6, 7]

Recently, an interesting experiment, not directly related to interferometry
techniques, but that attests to the fact that neutrons can reside in quantum
stationary states formed in the gravitational field of the Earth, was carried
out. In this experiment, the lowest quantum state of neutrons in the Earth’s
gravitational field was identified in the measurement of neutron transmission
between a horizontal mirror on the bottom and an absorber/scatter on top.[8,
9] This result motivate Ernest to do a careful and through research on
gravitational eigenstates in weak gravity.[10, 11] Actually, despite the
almost universal study of quantum theory applied to atomic and molecular
states, very little work has been done to investigate the properties of the
hypothetical stationary states that should exist in similar types of
gravitational central potentials wells, particularly those with large quantum
numbers.

The preceding experiments show clearly that physical phenomena in which
gravity (specifically, Earth’s gravitational field) and quantum effects are
merged together, are no more beyond our reach. We remark that in all these
investigations the gravitational field of the Earth is described by Newton’s
gravity.

On the other hand, since its original publication in 1915, Einstein’s general
theory of relativity continues to be an active area of both theoretical and
experimental research. Presently, the theory successfully accounts for all
data gathered to date.[12] In addition, among its so-called classical tests
there is one, namely, light bending, that has been confirmed with an accuracy
that increases as time goes by. In reality, it is expected that a series of
improved designed experiments with the Very Long Baseline Interferometry
(VLBI) will increase the present accuracy of the deflection parameter $\gamma$
by at least a factor of 4.[13] By the way, the current value for $\gamma$ is
0.9998$\pm$0.0003 (68$\%$ confidence level),[13] in agreement with general
relativity. Besides, it is also a generally acknowledged fact that the
gravitational deflection of light by the sun can be measured more accurately
at radio wavelengths with interferometry techniques than at visible wavelength
with available optical techniques.14-17 Indeed, at present the VLBI is the
most accurate technique we have at our disposal for measuring radio-wave
gravitational deflection.[13] Now, taking into account that the search for
upper bounds on the photon mass222In all these researches, the photon is
described by massive QED, which is nothing but the most straightforward
extension of standard QED. Its Lagrangian can be written as
$\mathcal{L}=-\frac{1}{4}F^{2}_{\mu\nu}+\frac{1}{2}m^{2}A^{2}_{\mu}-J^{\mu}A_{\mu},$
(1) where $F_{\mu\nu}$($=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}$) is the
field strength, $J^{\mu}$ is the conserved (electric) current and $m$ is the
photon mass. Here, indices are raised and lowered with $\eta^{\mu\nu}$ and
$\eta_{\mu\nu}$, respectively. In reality, massive QED is theoretically
simpler than the standard theory,[18] besides being renormalizable.[19] has
increased over the past several decades,20-22 it would be interesting to
estimate gravitational bounds for the photon mass by considering the most
recent measurements of the solar gravitational deflection of radio waves
obtained by means of the VLBI. This is precisely the goal of this work.

The article is organized as follows. In Sec. 2 we show that there exists a
constraint on the photon mass, its frequency and the deflection parameter
determined by experimenters (which characterizes the contribution of space
curvature to gravitational deflection). From this amazing result, upper limits
on the photon mass are found in Sec. 3, by considering as inputs the most
recent measurements of the solar gravitational deflection of radio waves
obtained via the VLBI.[13] To conclude, we discuss in Sec. 4, whether or not
the bounds we have estimated can be improved.

In our convention $\hbar=c=1$, and the signature is (+ - - -).

##  2 Combining General Relativity, Massive QED and Very Long Baseline
Interferometry to Gravitationally Constrain the Photon Mass

To start off, we recall that the Lagrangian for the gravitational minimally
coupled massive photon field is

$\displaystyle{\cal{L}}=\sqrt{-g}\left[-\frac{1}{4}g^{\mu\alpha}g^{\nu\beta}F_{\mu\nu}F_{\alpha\beta}+\frac{m^{2}}{2}g^{\mu\nu}A_{\mu}A_{\nu}\right].$
(2)

On the other hand, for small fluctuations around the Minkowski metric $\eta$,
the full metric can be written as

file=foton-graviton.eps,width=5.9cm

Figure 1: Feynman graph for the interaction between a massive photon and an
external gravitational field.

$\displaystyle g_{\mu\nu}=\eta_{\mu\nu}+\kappa h_{\mu\nu},$ (3)

where $\kappa^{2}=32\pi G$. Here $G$ is Newton’s constant.

Supposing then that the massive photon is scattered by an external weak
gravitational field (See Fig. 1), we promptly obtain from Eqs. (2) and (3)
that the Lagrangian for the interaction under discussion is

$\displaystyle{\cal{L}}_{\mathrm{int}}=\frac{\kappa}{2}h_{\mathrm{ext}}^{\mu\nu}\left[\eta^{\alpha\beta}F_{\alpha\mu}F_{\beta\nu}-\frac{1}{4}F_{\alpha\beta}^{2}\eta_{\mu\nu}+\frac{m^{2}}{2}A_{\alpha}^{2}\eta_{\mu\nu}-m^{2}A_{\mu}A_{\nu}\right].$
(4)

Note that now the indices are raised (lowered) with
$\eta^{\mu\nu}(\eta_{\mu\nu})$.

Accordingly, the vertex function for the alluded process is, in momentum
space, given by

$\displaystyle V_{\alpha\beta}(p,p^{\prime})$ $\displaystyle=$
$\displaystyle\frac{\kappa}{2}h^{\mu\nu}_{\mathrm{ext}}(\mathbf{k})\Big{[}(m^{2}-p\cdot
p^{\prime})(\eta_{\mu\nu}\eta_{\alpha\beta}-2\eta_{\mu\alpha}\eta_{\nu\beta})+p^{\prime}_{\alpha}p_{\beta}\eta_{\mu\nu}$
(5)
$\displaystyle+\;2(-p^{\prime}_{\alpha}p_{\nu}\eta_{\mu\beta}-p^{\prime}_{\mu}p_{\beta}\eta_{\nu\alpha}+p^{\prime}_{\mu}p_{\nu}\eta_{\alpha\beta})\Big{]}.$

Now, assuming that the external weak gravitational field is generated by a
point mass $M$ at $\bf{r}$=$\bf{0}$, we immediately find the expression for
the external gravitational field by solving Einstein’s linearized field
equations in the de Donder gauge. The resulting expression is

$h^{\mu\nu}_{\mathrm{ext}}(\mathbf{r})=\frac{\kappa M}{16\pi
r}(\eta^{\mu\nu}-2\eta^{\mu 0}\eta^{\nu 0}).$ (6)

Consequently, the momentum space gravitational field,
$h_{\mathrm{ext}}^{\mu\nu}({\bf k})$, is given by

$\displaystyle h^{\mu\nu}_{\mathrm{ext}}(\mathbf{k})$ $\displaystyle=$
$\displaystyle\int{d^{3}\mathbf{r}e^{-i\mathbf{k}\cdot\mathbf{r}}h^{\mu\nu}_{\mathrm{ext}}(\mathbf{r})}$
(7) $\displaystyle=$ $\displaystyle\frac{\kappa
M}{2\mathbf{k}^{2}}\Big{(}\frac{\eta^{\mu\nu}}{2}-\eta^{\mu 0}\eta^{\nu
0}\Big{)}.$

We are now ready to compute the unpolarized differential cross section for the
process displayed in Fig. 1. To do that we recall that the expression for the
mentioned cross section is

$\frac{d\sigma}{d\Omega}=\frac{1}{(4\pi)^{2}}\frac{1}{3}\sum^{3}_{r=1}\sum^{3}_{r^{\prime}=1}\mathcal{M}^{2}_{rr^{\prime}},$
(8)

with

${\cal{M}}_{rr^{\prime}}=\epsilon_{r}^{\alpha}({\bf
p})\epsilon_{r^{\prime}}^{\beta}({\bf
p^{\prime}})V_{\alpha\beta}(p,p^{\prime}),$ (9)

where $\epsilon_{r}^{\alpha}({\bf p})$ and $\epsilon_{r^{\prime}}^{\beta}({\bf
p^{\prime}})$ are the polarizations vectors for the ingoing and outgoing
vectorial bosons, respectively. These vectors, in turn, satisfy the relation

$\sum_{r=1}^{3}\epsilon_{r}^{\mu}(\mathbf{p})\epsilon^{\nu}_{r}(\mathbf{p})=-\eta^{\mu\nu}+\frac{p^{\mu}p^{\nu}}{m^{2}}.$
(10)

After algebraic manipulations, we find that the expression for the unpolarized
differential cross section we are searching for reads

$\frac{d\sigma}{d\Omega}=\frac{1}{6}\frac{M^{2}G^{2}}{\Big{(}\sin^{2}{\frac{\theta}{2}}\Big{)}^{2}}\Bigg{[}3\mathbf{p}^{4}+\frac{3}{2}m^{4}+2\mathbf{p}^{2}m^{2}+2\mathbf{p}^{2}(\mathbf{p}^{2}+2m^{2})\cos{\theta}+\mathbf{p}^{4}\cos^{2}{\theta}\Bigg{]},$
(11)

where $\theta$ is the scattering angle. For small angles, it reduces to

$\frac{d\sigma}{d\Omega}=\frac{16M^{2}G^{2}}{\theta^{4}}\left[\frac{1-\frac{m^{2}}{2E^{2}}}{1-\frac{m^{2}}{E^{2}}}\right]^{2},$
(12)

where $E$ is the energy of the ingoing photon.

The above differential cross section can be related to a classical trajectory
with impact parameter $b$ via the relation

$\displaystyle bdb=-\frac{d\sigma}{d\Omega}\theta d\theta.$ (13)

From Eqs. (12) and (13), we arrive at the conclusion that

$\displaystyle\theta=\frac{4MG}{b}\left(\frac{1-\frac{m^{2}}{2E^{2}}}{1-\frac{m^{2}}{E^{2}}}\right),$
(14)

which, in the ultrarelativistic limit, i.e., $E\gg m$, leads to

$\displaystyle\theta$ $\displaystyle=$
$\displaystyle\theta_{\mathrm{E}}\left(1+\frac{m^{2}}{2E^{2}}\right)$ (15)
$\displaystyle=$
$\displaystyle\theta_{\mathrm{E}}\left(1+\frac{m^{2}}{8\pi^{2}\nu^{2}}\right),$
(16)

where $\nu$ is the frequency of the massive photon and
$\theta_{\mathrm{E}}\equiv\frac{4MG}{b}$.

Before going on, some comments are in order.

Recently, it was shown that the unpolarized differential cross sections for
the scattering of different quantum particles are spin dependent, which is in
disagreement with the classical equivalence principle[7] (See Table 1). This
result raises an important question: Why the gravitational field perceives the
spin? Because there is the presence of a nonzero momentum transfer (k) in the
scattering, responsible for probing the internal structure (spin) of the
particle. Nevertheless, if we choose any two expressions from those listed in
Table 1, we get that the difference between them is always extremely small for
typical deflection angles. To show this for the massless particles, for
instance, we study the behavior of

$\displaystyle\frac{\Delta(\frac{d\sigma}{d\Omega})}{(\frac{d\sigma}{d\Omega})_{s=0}}\equiv\frac{(\frac{d\sigma}{d\Omega})_{s}-(\frac{d\sigma}{d\Omega})_{s=0}}{(\frac{d\sigma}{d\Omega})_{s=0}},$
(17)

as a function of the scattering angle $\theta$ (See Fig. 2). It is trivial to
show that for small angles the preceding expression reduces to

$\displaystyle\frac{\Delta(\frac{d\sigma}{d\Omega})}{(\frac{d\sigma}{d\Omega})_{s=0}}\approx-\frac{s\theta^{2}}{2}.$
(18)

For a typical deflection angle, say $\theta\sim 10^{-6}$, we found
$\frac{\Delta(\frac{d\sigma}{d\Omega})}{(\frac{d\sigma}{d\Omega})_{s=0}}\sim
10^{-12}$. The detection of so small an effect is, of course, beyond todays
technology. Consequently, for these tiny deflection angles, the cross sections
will be unaffected by the spin of the particle.

Unpolarized differential cross-sections for the scattering of different
quantum particles by an external weak gravitational field generated by a
static point particle of mass $M$. Here $m$ is the particle mass, $s$ the
spin, $\theta$ the scattering angle, and
$\lambda\equiv\frac{m^{2}}{\mathbf{p}^{2}}=\frac{1-\mathbf{v}^{2}}{\mathbf{v}^{2}}$,
with $\mathbf{v}$ and $\mathbf{p}$ being the velocity and three-momentum, in
this order, of the incident particle.  $m$ $s$ $\frac{d\sigma}{d\Omega}$ 0 0
$\Big{(}\frac{GM}{\sin^{2}{\frac{\theta}{2}}}\Big{)}^{2}$ $\neq 0$ 0
$\Big{(}\frac{GM}{\sin^{2}{\frac{\theta}{2}}}\Big{)}^{2}\Big{(}1+\frac{\lambda}{2}\Big{)}^{2}$
0 $\frac{1}{2}$
$\Big{(}\frac{GM}{\sin^{2}{\frac{\theta}{2}}}\Big{)}^{2}\cos^{2}{\frac{\theta}{2}}$
$\neq 0$ $\frac{1}{2}$
$\Big{(}\frac{GM}{\sin^{2}{\frac{\theta}{2}}}\Big{)}^{2}\Big{[}\cos^{2}{\frac{\theta}{2}}+\frac{\lambda}{4}\Big{(}1+\lambda+3\cos^{2}{\frac{\theta}{2}}\Big{)}\Big{]}$
0 1
$\Big{(}\frac{GM}{\sin^{2}{\frac{\theta}{2}}}\Big{)}^{2}\cos^{4}{\frac{\theta}{2}}$
$\neq 0$ 1
$\Big{(}\frac{GM}{\sin^{2}{\frac{\theta}{2}}}\Big{)}^{2}\Big{[}\frac{1}{3}+\frac{2}{3}\cos^{4}{\frac{\theta}{2}}-\frac{\lambda}{3}\Big{(}1-\frac{3\lambda}{4}-4\cos^{2}{\frac{\theta}{2}}\Big{)}\Big{]}$
0 2
$\Big{(}\frac{GM}{\sin^{2}{\frac{\theta}{2}}}\Big{)}^{2}\Big{(}\sin^{8}{\frac{\theta}{2}}+\cos^{8}{\frac{\theta}{2}}\Big{)}$

In order to recover Einstein’s geometrical results from Table 1, we must have
${\bf k}\rightarrow{\bf 0}$; in other words, in the nontrivial limit of small
momentum transfer, which corresponds to a nontrivial small angle limit since
$\left|{\bf k}\right|=2\left|{\bf p}\right|\sin\frac{\theta}{2}$, the massive
(massless) particles behave in the same way, regardless the spin. In fact,
when the spin is ‘switched off’, i.e, for small angles, we obtain from Table 1
that for $m=0$,

$\displaystyle\frac{d\sigma}{d\Omega}\approx\frac{16G^{2}M^{2}}{\theta^{4}},$
(19)

while for $m\neq 0$,

$\displaystyle\frac{d\sigma}{d\Omega}\approx\frac{16G^{2}M^{2}}{\theta^{4}}\left(1+\frac{\lambda}{2}\right)^{2}.$
(20)

Using Eq. (12) we conclude that for $m=0$,

$\displaystyle\theta\approx\frac{4GM}{b},$ (21)

while for $m\neq 0$,

$\displaystyle\theta\approx\frac{4GM}{b}\left(1+\frac{\lambda}{2}\right).$
(22)

The former equation gives the gravitational deflection angle for a massless
particle — a result foreseen by Einstein a long time ago; whereas the latter
coincides with the prediction of general relativity for the deflection of a
massive classical test particle by an external weak gravitational field.[23]
It is worth noticing that Eqs. (15) and (22) are exactly the same. In short,
for small angles the results of Table 1 not only reduce to those predicted by
Einstein’s geometrical theory, they are also in agreement with the classical
equivalence principle.

file=spin3.eps,width=10cm

Figure 2:
$\frac{\Delta(\frac{d\sigma}{d\Omega})}{(\frac{d\sigma}{d\Omega})_{s=0}}$ as a
function of the scattering angle $\theta$.

At first glance it seems that Eq. (16) predicts a dispersive deflection angle
for the massive photons. Actually, this is a false impression; indeed, it is
straightforward to show that Eq. (16) can be rewritten as

$\displaystyle\theta=\theta_{\mathrm{E}}\left(\frac{3-{\bf v^{2}}}{2}\right).$
(23)

After these important digressions, we return to the analysis of Eq. (16). The
first term in the expression (16) coincides with that obtained by Einstein in
1916, by solving the equation of light propagation in the field of a static
body,[24] while the second one is the most important correction to the mass
$m$ of the massive photon.

On the other hand, the angle of gravitational bending determined by the
experimental groups is in general expressed through the relation[25]

$\displaystyle\theta_{\mathrm{exp}}=\frac{1+\gamma}{2}\theta_{\mathrm{E}},$
(24)

where $\gamma$ is the deflection parameter, precisely and unambiguously
determined by experimenters by measuring the deflection of electromagnetic
radiation in the gravitational field of the sun.

From Eqs. (16) and (24), we then get

$\displaystyle m<2\pi\nu\sqrt{\left|1-\gamma\right|}.$ (25)

This amazing result clearly shows that there exists a constraint between the
photon mass and the parameters $\gamma$ and $\nu$. In addition, it tells us
that the knowledge of these parameters is all we need to set up upper limits
on the photon mass.

##  3 Finding Gravitational Upper Bounds on the Photon Mass

We are now finally ready to find gravitational bounds on the photon mass. To
accomplish this goal, we make use of the recent measurements of the solar
gravitational deflection of radio waves found by Fomalont _et al._[13] using
the VLBI. The solutions for $\gamma$ obtained from these measurements, as well
as the corresponding bounds on the photon mass we have estimated using Eq.
(25), are displayed in Table 2.

In order to determine the optimum value of $(\gamma-1)$ from the experimental
results, the aforementioned authors minimized the (normalized) chi-squared
expression

$\chi_{k}^{2}=\frac{1}{k}\sum_{d,i}\left(\frac{P_{d}(i)-0.5(\gamma-1)D_{d}(i)}{\sigma_{d}(i)}\right)^{2},$
(26)

where $P_{d}(i)$ and $\sigma_{d}(i)$ are the measured position offset and
error estimate, in this order. The term $D_{d}(i)$ is the differential general
relativity gravitational bending prediction averaged over the session. The
$\sigma_{\gamma}$’s and $\chi_{k}^{2}$’s directly related to the the solutions
for $\gamma$ are shown in Table 2.

Uppers bounds on the photon mass estimated using the solutions for $\gamma$
found by Fomalont _et al._ 13 Solution Type $(\gamma-1)\times 10^{-4}$
$\sigma_{\gamma}\times 10^{-4}$ $\chi^{2}_{k}$ $m\times 10^{-11}$($MeV$)
$43\;GHz$ data (corona-free) -2.4 3.2 0.9 1.7 $43\;GHz$ data only -1.0 2.6 2.2
1.1 $43\;GHz$ data only - Oct05 -3.2 2.8 1.1 2.0 $23\;GHz$ data only - Oct05
-2.0 2.4 4.7 0.8

In Table 2, the $43\;GHz$ corona-free fit is the most accurate since it has
the lowest $\chi^{2}$. This is due to the lessening of some coronal effects,
and the increase of the position errors. The two $43\;GHz$ only solutions
(with no removal of the ionosphere reduction) show the effect of the Oct05
session that was made relatively close to the sun; the first solution was
found using data obtained during sessions that lasted several days, while the
second one is based on data that were got within the space of one day only.
Finally, the $23\;GHz$ only solution suggests that coronal refraction, which
is four times larger than that at $43\;GHz$, is dominating the sensitivity of
the experiment at $23\;GHz$.

It is worth mentioning that Fomalont $et\;al.$[13] have used for the results
in their paper an average of the four solutions exhibited in Table 2 to obtain
$\gamma=0.9998\pm 0.0003$. From this result and assuming that the massive
photon passing near the solar limb has a frequency $\nu=43GHz$, which is
perfectly justifiable since their data came mainly from $43GHz$ observations
where the refraction effects of the solar corona were negligible beyond 3
degrees from the sun, we obtain another gravitational bound on the photon
mass, namely, $m\sim 3.4\times 10^{-11}MeV$.

For reasons already discussed, we come to the conclusion that among the
gravitational bounds on the photon mass we have found, the most reliable is
$m\sim 1.7\times 10^{-11}MeV$. In fact, the associated $\gamma$ was
determined, on the one hand, using one single frequency; on the other, it has
the best chi-squared. Furthermore, the data used for computing $\gamma$ came
mainly from observations where the refraction effects of the solar corona were
negligible.

##  4 Discussion

Certainly, the bounds we have found on the photon mass are considerably higher
than the recently recommended limit published by the Particle Data Group
($m<1\times 10^{-18}eV$).[26] They are nevertheless comparable to other
existing bounds.20-22 Let us then discuss whether or not a better limit on the
photon mass might be obtained using Eq. (25). First, if the deflections
measured using the VLBI could be made with greater accuracy the value of
$\sqrt{\left|1-\gamma\right|}$ would be reduced giving, as a result, a better
gravitational estimate. According to Fomalont et al.,[13] a series of designed
experiments with the VLBI could increase the accuracy of the future
experiments by at least a factor of 4. Second, if deflection measurements can
be obtained at lower frequencies, while maintaining the value of the
deflection parameter $\gamma$, the gravitational bound will be improved in
direct proportion to the frequency. This point, however, is very delicate. In
fact, as we have already mentioned, up till now the best results obtained for
the gravitational deflection via the VLBI are those that come mainly from
$43GHz$ where the refraction effects of the solar corona were negligible
beyond 3 degrees from the sun. Incidentally, the lowest frequency employed by
the radio astronomers was $2GHz$. However, the measurements made at this
frequency are less reliable because of the refraction effects of the solar
corona. Actually, the radio astronomers use in their experiments a mixing of
different frequencies but the most significant contributions come in general
from $\sim 43GHz$. The possibility of improving the gravitational limit on the
photon mass in this case is thence very limited.

We remark that up to now only two attempts to constrain the photon mass using
gravitational deflection measurements were made.[27, 28] Unfortunately, these
estimates are not very reliable because in both of them the values of the
deflection parameter $\gamma$ are overestimated, while the values of the
frequency are underestimated. In reality, the $\nu$-values used in the
mentioned estimates are in the neighborhood of the the lowest frequency
employed by radio astronomers ($\approx 2GHz$).

We call attention to the fact that the gravitational estimates on the photon
mass we have obtained, like the great majority of estimates made with the
purpose of limiting the photon mass and which are available in the literature,
are essentially order-of-magnitude arguments. Nonetheless, our efforts in this
work are based on a new conceptual approach to the subject; besides, in the
calculation of the bounds, the most accurate experimental data currently
available have been taken as inputs.

To conclude, we remark that recently we have found a quantum bound on the
photon mass ($m\sim 1.6\times 10^{-10}MeV$) based on the computation of the
anomalous electron magnetic moment in the framework of Proca
electrodynamics.29,30 To the best of our knowledge, this is the firs time a
quantum bound on the photon mass is estimated.333Note that the bound on the
photon mass obtained by Boulware and Deser31 via the Aharonov-Bohm effect
(which is present in massive (finite-range) electrodynamics) is a
semiclassical limit; our bound, nonetheless, unlike the Bolware-Deser limit,
is based on truly (loop) quantum effects. It is worth noticing that this
quantum bound is one order of magnitude higher than the bound derived from the
gravitational scattering.

## Acknowledgments

The authors are very grateful to FAPERJ, CNPq, and CAPES (Brazilian funding
agencies) for the financial support.

## References

  * [1] R. Colella, A. Overhauser and S. Werner, Phys. Rev. Lett. 34 (1975) 1472.
  * [2] U. Bonse and T. Wroblewski, Phys. Rev. Lett. 51 (1983) 1401.
  * [3] K. Littrell, B. Allman and S. Werner, Phys. Rev. A 56 (1997) 1767.
  * [4] G. Adunas, E. Rodriguez-Milla and D. V. Ahluwalia-Khalilova, Gen. Relativ. Gravit. 33 (2001) 183.
  * [5] G. Adunas, E. Rodriguez-Milla and D. V. Ahluwalia-Khalilova, Phys. Lett. B 485 (2000) 215.
  * [6] A. Accioly and R. Paszko, Phys. Rev. D 78 (2008) 064002.
  * [7] A. Accioly and R. Paszko, Adv. Studies Theor. Phys. 3 (2009) 65.
  * [8] V. Nesvizhevsky et al., Nature 415 (2002) 297.
  * [9] V. Nesvizhevsky et al., Phys. Rev. D 67 (2003) 102002.
  * [10] A. Ernest, J. Phys. A: Math. Theor. 42 (2009) 115207.
  * [11] A. Ernest, J. Phys. A: Math. Theor. 42 (2009) 115208.
  * [12] S. Turyshev, Usp. Fiz. Nauk 179 (2009) 3.
  * [13] E. Fomalont et al., Astrophys. J. 699 (2009) 1395.
  * [14] D. Robertson, W.Carter and W. Dillenger, Nature 349 (1991) 768.
  * [15] B. Bertotti, L. Iess, and P. Tortora, Nature 425 (2003) 374.
  * [16] S. Shapiro et al., Phys. Rev. Lett. 92 (2005) 121101.
  * [17] S. Lambert and C. Le Poncin-Lafitte, Astron. Astrophys. 499 (2009) 331.
  * [18] A. Ignatiev and G. Joshi, Phys. Rev. D 53 (1996) 984.
  * [19] D. Boulware, Ann. Phys. 56 (1970) 140.
  * [20] A. Goldhaber and M. Nieto, Rev. Mod. Phys. 43 (1971) 277.
  * [21] Liang-Cheng Tu, J. Luo, and G. Gilles, Rep. Prog. Phys. 68 (2005) 77.
  * [22] A. Goldhaber and M. Nieto, Rev. Mod. Phys. 82 (2010) 939.
  * [23] A. Accioly and S. Ragusa, Class. Quant. Grav. 19 (2002) 5429 [Erratum Class. Quant. Grav. 20 (2003) 4963].
  * [24] A. Einstein, Ann. Phys. 49 (1916) 769.
  * [25] C. Will, Theory and Experiment in Gravitational Physics (Cambridge: Cambridge Univ. Press, 1993).
  * [26] C. Amsler et al. (Particle Data Group), Phys. Lett. B 667 (2008) 1.
  * [27] D. Lowenthal, Phys. Rev. D 8 (1973) 2349.
  * [28] A. Accioly and R. Paszko, Phys. Rev. D 69 (2004) 107501.
  * [29] A. Accioly, J. Helayël-Neto and E. Scatena, Phys. Rev. D  82 (2010) 065026.
  * [30] A. Accioly, J. Helayël-Neto, R. Turcati, J. Morais and E. Scatena, Classical Quantum Gravity 27 (2010) 205010.
  * [31] D. Boulware and S. Deser, Phys. Rev. Lett. 63 (1989) 2319.


# Disentangled Face Identity Representations
for joint 3D Face Recognition and Expression Neutralisation

Anis Kacem
SnT, University of Luxembourg
Kseniya Cherenkova
SnT, University of Luxembourg, Artec3D
Djamila Aouada
SnT, University of Luxembourg

###### Abstract

In this paper, we propose a new deep learning based approach for disentangling
face identity representations from expressive 3D faces. Given a 3D face, our
approach not only extracts a disentangled identity representation, but also
generates a realistic 3D face with a neutral expression while predicting its
identity. The proposed network consists of three components; (1) a Graph
Convolutional Autoencoder (GCA) to encode the 3D faces into latent
representations, (2) a Generative Adversarial Network (GAN) that translates
the latent representations of expressive faces into those of neutral faces,
(3) and an identity recognition sub-network taking advantage of the
neutralized latent representations for 3D face recognition. The whole network
is trained in an end-to-end manner. Experiments are conducted on three
publicly available datasets showing the effectiveness of the proposed
approach.

##  1 Introduction

Automatic recognition of faces is a non-intrusive technology that, if
performed accurately, can open the door to many innovative applications and
revolutionize the interactions of humans with infrastructures and services.
With the recent advances in acquisition systems, impressive face recognition
systems have been deployed in real-world applications [10, 15, 14, 18, 16, 21,
1]. In particular, 3D sensors played an important role in making these
advances thanks to the rich geometrical information and their robustness to
multiple real-world constraints (e.g., pose variations, illumination changes,
etc) [19].

Another important constraint in real-world scenarios consists of recognizing
faces under different facial deformations usually termed as facial expressions
[10, 2]. Indeed, the captured information when acquiring faces encodes
different attributes, e.g., identity, expression, pose, etc [3, 2, 13].
Recognizing faces usually requires decoupling identity attributes from other
attributes. Following this direction, multiple works tried to build 3D face
parametric models disentangling identity attributes from expression ones. In
particular, 3D Morphable Models (3DMM) have been widely used in different
forms to achieve this goal [3, 2, 4]. Such models usually assume linear
combination of the different attributes which limits their disentanglement
abilities due to the natural non-linearity of face shape variations. Aware of
such limitations, recent works proposed non-linear models to model 3D face
shapes [13, 8, 5, 20]. [20] proposed a Graph Convolutional Autoencoder (GCA)
making use of spectral graph convolutions [6] to encode 3D face shapes into
non-linear latent representations. [8] presented an intrinsic Generative
Adversarial Network (GAN) architecture named as MeshGAN operating directly on
3D face meshes using a similar strategy as [20]. Their proposed method allows
the generation of new identities and expressions. Similarly, the authors in
[13] decoupled identity and expression representations by building two
separate branches of networks, one for identity and the other for expression.
Their proposed networks were based on spectral graph convolutions and were
applied on deformation representations computed on the face meshes in advance.
In order to control the distribution in latent space, they used a variational
strategy when training each branch. Taking another direction, [5] introduced
spiral convolutions on meshes and used them within a GAN-based architecture to
synthesise face meshes.

Figure 1: Approach Overview. In training phase: pairs of expressive and
neutral 3D faces of the same identity are collected. These pairs are fed to a
two-branch autoencoder sharing the same weights. A GAN-based module consisting
of a generator and a discriminator is introduced in the latent space to learn
a valid translation from expressive to neutral representation. An identity
recognition module is adopted to learn better identity representations. In
testing phase: An expressive 3D face is given as input to the network. The
encoder encodes the 3D face into a latent representation. Then, the generator
of the GAN module translates this latent representation into neutral one which
is then decoded into a neutral 3D face. The output features of the face
recognition module can be also used to conduct face recognition. Best viewed
in color.

The main focus of this paper is to effectively disentangle identity attributes
from expression ones and use them within a face recognition scenario.
Accordingly, we learn a non-linear mapping translating expressive 3D faces to
neutral ones. This is achieved at the level of latent representations obtained
using a Graph Convolutional Autoencoder (GCA) [20]. The latent space
translation is learned using a Generative Adversarial Network (GAN) applied on
pairs of expressive and neutral latent representations [12]. Moreover,
additional modules and constraints are proposed to better preserve identity
features when neutralizing expressions. Figure 1 shows an overview of the
proposed approach. The main contributions of this paper are:

  * •

An end-to-end network allowing joint face expression neutralization and face
recognition. Given an expressive 3D face, the proposed network can also
synthesise the corresponding realistic neutral 3D face;

  * •

A novel data-driven disentangled identity representation obtained by a
realistic translation from expressive faces to neutral ones;

  * •

An experimental validation of the proposed approach on three publicly
available datasets.

The rest of the paper is organized as follows: Section 2 describes the
proposed approach. In Section 3, we provide the network architecture and
details on the training. The experimental evaluation is reported in Section 4.
Finally, Section 5 concludes the paper and draws some perspectives.

##  2 Proposed Approach

Let us consider a set of $m$ face meshes $\\{F_{j}\\}^{m}_{j=1}$. A face mesh
is defined as a graph structure, $F=(V,A)$. Here,
$V=\left[v_{1},v_{2},\dots,v_{n}\right]$ denotes the 3D coordinates of $n$
vertices and $A$ represents an $n\times n$ adjacency matrix with $A_{ij}=1$ if
an edge connection exists between vertex $v_{i}$ and $v_{j}$, and $A_{ij}=0$
otherwise.

A set of pairs of expressive and neutral 3D faces are collected such that each
pair $(F^{e}_{i},F^{n}_{i})$ involves an expressive 3D face $F^{e}_{i}$ and
its neutral counterpart $F^{n}_{i}$ (_i.e._ , a neutral 3D face of the same
identity). The main objective of our approach is to translate the expressive
face $F^{e}_{i}$ into a neutral face $\tilde{F^{n}_{i}}$ that approximates
well the ground truth neutral face $F^{n}_{i}$, while allowing the prediction
of an identity code $\tilde{y_{i}}$ that should match the ground truth
identity label $y_{i}$ of both $F^{e}_{i}$ and $F^{n}_{i}$.

###  2.1 Graph Convolutional Autoencoder

The first step to our approach is to to embed the input 3D face meshes into a
low-dimensional latent space that encodes the expression and identity
attributes of the input face meshes. To do so, we use a mesh-based autoencoder
consisting of an encoder $Enc$ and a decoder $Dec$. The encoder $Enc$ maps the
input face pair $(F^{e}_{i},F^{n}_{i})$ into latent representations
$(z_{i}^{e},z_{i}^{n})$. The decoder $Dec$ directly maps back the neutral
latent representation $z_{i}^{n}$ into a neutral face mesh $\tilde{F^{n}_{i}}$
and also maps the neutralised version of the expressive latent representation
$z_{i}^{e}$ into the same neutral face mesh $\tilde{F^{n}_{i}}$. The
neutralization of the latent representation will be explained in the next
section. From an implementation point of view, the proposed auto-encoder can
be seen as two branches of Encoder-Decoder sharing the same weights.

Due to the graph structure of the input face meshes, standard convolutions
cannot be used in $Enc$ and $Dec$. As a solution, we use the spectral graph
convolutions proposed in [6]. In what follows, we recall the background
related to these convolutions.

Spectral Graph Convolutions: These convolutions are defined on the graph
Fourier transform domain. The Fourier transform of the vertices $V$ is given
by $V_{w}=U^{t}V$, where $U\in\mathbb{R}^{n\times n}$ whose columns are the
orthogonal eigenvectors of the associated Laplacian matrix $L$, while the
inverse Fourier transform is defined as $V=UV_{w}$. The associated Laplacian
to a graph $F$ is defined as $L=\Lambda-A$, where $\Lambda$ is a diagonal
matrix representing the degree of each vertex in $V$ with diagonal elements
$\Lambda_{ii}=\sum_{j=1}^{n}A_{ij}$. The convolution operator in Fourier space
can be defined as a Hadamard product of the Fourier transforms of the graph
and the filter. For efficiency matters, it is common to use a recursive
Chebyshev polynomial of order $P$ [20, 9] to approximate a kernel
$k_{\theta}(L)=\sum_{p=0}^{P-1}\theta_{p}T_{p}(\hat{L})$, where
$\hat{L}=2L/e_{max}-I_{n}$ is the scaled Laplacian matrix, $e_{max}$ maximal
eigenvalue of $L$, $\theta\in\mathbb{R}^{P}$ is a vector of Chebyshev
coefficients that should be learned, and $T_{p}\in\mathbb{R}^{n\times n}$ is
the Chebyshev polynomial of order $p$ that can be computed recursively [9].
The spectral convolution is finally defined as,

$c_{j}=\sum_{i=1}^{n_{f}}k_{\theta_{i,j}}(L)V_{:,i}\in\mathbb{R}^{n},$ (1)

where $c_{j}$ is the $j$-th feature of the output $c\in\mathbb{R}^{n\times
n_{of}}$ ($n_{of}$ is the number of output features), $V_{:,i}$ is the $i$-th
feature of the input $V\in\mathbb{R}^{n\times n_{f}}$ ($n_{f}$ is the number
of input features), and $\theta_{i,j}\in\mathbb{R}^{P}$ are the Chebyshev
trainable coefficients. For more details about spectral graph convolutions,
readers are refereed to [9, 20].

For notation simplicity, we will use in the following
$z_{i}^{e}=Enc(F^{e}_{i})$ to denote the expressive latent representation and
$z_{i}^{n}=Enc(F^{n}_{i})$ to represent the neutral latent representation.

###  2.2 Latent Space Translation

Given a set of pairs of latent representations $(z_{i}^{e},z_{i}^{n})$ learned
from the expressive and neutral face meshes $(F^{e}_{i},F^{n}_{i})$, we aim to
find a mapping function $\mathcal{G}:z_{i}^{e}\rightarrow z_{i}^{n}$ that
translates the expressive latent representation into its corresponding neutral
latent representation. To approximate this function we use a neural network
$\mathcal{G}$ on the expressive latent representation $z_{i}^{e}$ and
constrain its output to be close to the neutral latent representation
$z_{i}^{n}$ by minimizing the L1 distance:

$\mathcal{L}_{L1}=\left\lVert\mathcal{G}(z_{i}^{e})-z_{i}^{n}\right\rVert_{1}.$
(2)

For a more realistic translation, we further constrain the distribution of
expressive latent representations to match that of neutral latent
representations. This is achieved by employing a conditional Generative
Adversarial Network (cGAN) [17, 12] on these latent representations. The
generator of this cGAN is given by the network $\mathcal{G}$ that takes the
expressive latent representations as input and tries to generate the
corresponding neutral latent representations. The discriminator $\mathcal{D}$
is defined as another network that considers the neutral latent
representations $\mathcal{G}(z_{i}^{e})$ as fake examples, while taking the
neutral latent representations $z_{i}^{n}$ as real examples. Furthermore, the
discriminator $\mathcal{D}$ is conditioned using the expressive latent
representation $z_{i}^{e}$ as proposed in [12] for image-to-image translation.
This condition is established by concatenating the condition $z_{i}^{e}$ to
the inputs of the discriminator (_i.e._ , $z_{i}^{n}$ or
$\mathcal{G}(z_{i}^{e})$). The adversarial loss to be minimized is then
defined as,

$\begin{multlined}\mathcal{L}_{GAN}=\mathbb{E}_{z_{i}^{e},z_{i}^{n}}\left[\log\mathcal{D}(z_{i}^{n},z_{i}^{e})\right]+\\\
\mathbb{E}_{z_{i}^{e},z_{i}^{n}}\left[\log(1-\mathcal{D}(\mathcal{G}(z_{i}^{e}),z_{i}^{e}))\right].\end{multlined}\mathcal{L}_{GAN}=\mathbb{E}_{z_{i}^{e},z_{i}^{n}}\left[\log\mathcal{D}(z_{i}^{n},z_{i}^{e})\right]+\\\
\mathbb{E}_{z_{i}^{e},z_{i}^{n}}\left[\log(1-\mathcal{D}(\mathcal{G}(z_{i}^{e}),z_{i}^{e}))\right].$
(3)

The total loss function for latent space translation is given by,

$\mathcal{L}_{l2l}=\lambda_{L1}\mathcal{L}_{L1}+\lambda_{GAN}\mathcal{L}_{GAN},$
(4)

where $\lambda_{L1}$ and $\lambda_{GAN}$ denote the weights of
$\mathcal{L}_{L1}$ and $\mathcal{L}_{GAN}$, respectively.

###  2.3 Face Recognition Module

In order to conduct face recognition, we take advantage of the generated
neutral latent representation for an expression-robust face recognition.
Specifically, we use a neural network $\mathcal{R}$ on the generated neutral
latent representation $\mathcal{G}(z_{i}^{e})$ to predict the identity
$\tilde{y_{i}}=\mathcal{R}(\mathcal{G}(z_{i}^{e}))$ of the expressive face
$F^{e}_{i}$. The same network is also used on the real neutral latent
representation $z_{i}^{n}$ to obtain a prediction
$\tilde{y_{i}}=\mathcal{R}(z_{i}^{n})$. The standard cross-entropy loss is
used for training over the identities as follows,

$\mathcal{L}_{id}=-\sum^{s}_{j=1}y^{j}_{i}\log\tilde{y}^{j}_{i},$ (5)

where $s$ denotes the number of subjects (_i.e._ , classes) and $y^{j}_{i}$ is
is the $j$-th value of the ground truth label $y_{i}$.

It is important to mention that the face recognition module is not only used
for predicting identities but also for a better preservation of the identity
information when neutralizing the expression. We experimentally validate this
claim in Section 4.1.

##  3 Final Objective and Network Architecture

In addition to the loss functions of latent space translation and face
recognition module defined in Eq. 4 and Eq. 5, respectively, we further define
two loss functions at the end of the decoder $Dec$. The first one enforces the
autoencoder to reconstruct an input neutral face $F^{n}_{i}$, while the second
constrains the whole network to reconstruct, starting from an expressive face
$F^{e}_{i}$, an output neutral face that is close to the input neutral face
$F^{n}_{i}$. These reconstruction losses are formulated using an $L_{1}$ loss
as follows,

$\begin{multlined}\mathcal{L}_{rec}=\left\lVert
Dec(Enc(F_{i}^{n}))-F_{i}^{n}\right\rVert_{1}+\\\ \left\lVert
Dec(\mathcal{G}(Enc(F_{i}^{e})))-F_{i}^{n}\right\rVert_{1},\end{multlined}\mathcal{L}_{rec}=\left\lVert
Dec(Enc(F_{i}^{n}))-F_{i}^{n}\right\rVert_{1}+\\\ \left\lVert
Dec(\mathcal{G}(Enc(F_{i}^{e})))-F_{i}^{n}\right\rVert_{1},$ (6)

The final objective to be minimised is finally defined as,

$\mathcal{L}_{tot}=\lambda_{l2l}\mathcal{L}_{l2l}+\lambda_{id}\mathcal{L}_{id}+\lambda_{rec}\mathcal{L}_{rec},$
(7)

where $\lambda_{l2l}$, $\lambda_{id}$, and $\lambda_{rec}$ represent the
weights of the loss functions $\mathcal{L}_{l2l}$, $\mathcal{L}_{id}$, and
$\mathcal{L}_{rec}$, respectively.

Inference details: As mentioned in Section 2, it is crucial for the training
phase of our approach to collect pairs of expressive face meshes and their
corresponding neutral faces. These pairs are used to train the latent space
translator introduced in Section 2.2. During testing phase, the only required
input is an expressive face mesh $F^{e}_{i}$ that is encoded using $Enc$ to
obtain an expressive latent representation $z_{i}^{e}$. The latter is then fed
to the generator $\mathcal{G}$ to generate a neutral latent representation
$z_{i}^{n}$. The generated neutral representation is finally passed to the
face recognition module $\mathcal{R}$ to predict the identity if the input
expressive face and also decoded to produce a neutralized face mesh.

Figure 2: Face expression neutralization results on FaceScape, COMA, and FWH
datasets. The first row shows different input expressive 3D faces, the second
row presents the predicted neutralized 3D faces, the third row depicts the
corresponding ground-truth neutral 3D faces, and the last row depicts the
errors of the neutralization computed using per-vertex Euclidean distances
between predicted neutral (second row) and ground-truth neutral (third row) 3D
faces. Best viewed in color.

Network architecture: The encoder $Enc$ has four Chebyshev convolutional
layers. The first three layers have $16$ output features per vertex and the
last layer has $32$ output features. The output of the last convolutional
layer is fed into a (Fully Connected) FC layer to obtain a vector latent
representation of size $25$. The decoder $Dec$ consists of a FC layer that
transforms back the latent representation to a graph representation of $n$
vertices with $32$ features per vertex. The output of this FC is then passed
to four Chebyshev convolutional layers with a number of output features $32$,
$16$, $16$, and $3$, respectively. Each Chebyshev convolutional layer in $Enc$
and $Dec$ is approximated using $P=6$ polynomials and is followed by a biased
ReLU [11]. Regarding the latent representation generator $\mathcal{G}$, we use
a succession of four FC layers with a number of output features $100$, $200$,
$50$, and $25$, respectively. Each layer is followed by a ReLU activation. A
similar architecture is employed for the discriminator $\mathcal{D}$ which
consists of four FC layers of output size $100$, $200$, $50$, and $1$,
respectively. The first three FC layers are followed by Leaky ReLU activations
and the last layer is followed by a Sigmoid activation. Finally, the face
recognition module $\mathcal{R}$ consists of two FC layers, the first one has
an output size $100$ and is followed by a ReLU activation, and the second one
has an output size equal to the number of training subjects and is followed by
a Softmax layer.

In all our experiments we empirically set the weights of the loss function
terms as follows: $\lambda_{l2l}=0.5$, $\lambda_{L1}=0.4$, $\lambda_{GAN}=1$,
$\lambda_{id}=0.05$, and $\lambda_{rec}=2$

##  4 Experiments

We experimentally validate the proposed approach on three publicly available
datasets. On one hand, we evaluate the face neutralization and compare it with
some state-of-art methods. On the other hand, we conduct face recognition
based on the disentagled identity representations and compare our results with
baselines. The used datasets are the following:

COMA dataset [20]: is a dataset designed for dynamic 3D face analysis. It
contains more than $20$k 3D faces collected from $12$ identities. Each subject
performed $12$ facial expressions showing different challenges.

FaceWareHouse (FWH) dataset [7]: is a 3D face shape dataset including $47$
expressions (including neutral) performed by $150$ different identities. The
intensity of the expressions involved in this dataset varies from very weak to
very strong expressions.

FaceScape dataset [22]: consists of a dataset of $938$ subjects, each one
performed $21$ expressions inspired from the FWH dataset (including neutral).

###  4.1 Face Expression Neutralization

We evaluate the effectiveness of the expression neutralization by comparing
the neutralized expressive face to its ground-truth neutral counterpart.

Experimental protocol: We split the three datasets into training and testing
set following a cross-subject protocol. $70$% of the subjects were used for
training and the rest for testing. The fitted base models given by the
considered datasets have been employed since the proposed approach assumes
registered 3D face shapes. The pairs of expressive and neutral 3D faces are
directly given by the FWH and FaceScape datasets since both involve static 3D
face shapes with different expressions (including neutral). In the dynamic
COMA dataset, we take the first frame of each 3D face sequence as neutral face
and select $12$ expressive faces from other frames.

Results and discussions: In Figure 2, we report qualitative results of the
face expression neutralization process. In particular, given some input
expressive faces selected from the three considered datasets, we present the
predicted neutralized faces and compare them to the corresponding ground-truth
neutral faces using a per-vertex Euclidean distance. Overall, our approach can
perfectly neutralize weak and strong expressions on all the datasets. However,
we can observe that the neutralization on FaceScape dataset preserves better
the identity features. For example, the facial wrinkles were not properly
reconstructed on some examples of COMA and FWH datasets. This can be explained
by the larger number of identities included in the FaceScape dataset compared
to FWH and COMA datasets. This suggests that the generalization ability of the
proposed network to unseen subjects can be impacted by the number of
identities considered in the training.

In Table 1, we quantitatively evaluate the expression neutralization results
by computing the mean errors on the testing set of all datasets between
predicted neutral 3D faces and ground-truth ones. In accordance with the
observed qualitative results, the lowest error was recorded on FaceScape
dataset reaching a mean error of 2.02 millimeters (mm). The highest error was
obtained the FWH dataset corresponding to 3.07 mm. On COMA dataset, we
obtained an error of 2.73 mm.

We also compare our results to an adapted version of the Graph Convolutional
Autoencoder (GCA) method [20]. In particular, we trained the GCA network by
considering the prepared pairs of expressive and neutral faces as input and
ground-truth faces, respectively. The mean errors of the adapted GCA were
clearly outperformed by our approach on all the datasets. A further comparison
has been conducted with the Disentagled Representation Learning (DRL) approach
[13]. Specifically, we considered the output of the identity branch of the DRL
network as predicted neutral face and evaluated the mean errors following the
same protocol as our approach. In Table 1, we can note that the DRL approach
was more accurate than ours by more than 1 mm on the FWH dataset. It is
important to note that DRL method relies on an intermediate representation
called Deformation Representation (DR) which needs to be computed before
training and/or inference. Another important step in DRL approach is the data
augmentation performed on the DR representations to increase the number of
identities in the dataset. In particular, 10k faces were added on top of the
original FWH dataset used in our approach.

Finally, we conduct an ablation study of the two loss function terms
$\mathcal{L}_{l2l}$ and $\mathcal{L}_{id}$ defined in Eq. 4 and Eq. 5,
respectively. In Table. 1, we can observe that dropping one of these terms
results in a lower performance showing their importance in the training.

Method | COMA | FWH | FaceScape
---|---|---|---
GCA [20] | 3.28 | 3.56 | 2.88
DLR [13] | - | 1.83 | -
Ours w/o $\mathcal{L}_{l2l}$ | 3.45 | 3.92 | 3.29
Ours w/o $\mathcal{L}_{id}$ | 3.06 | 3.35 | 2.56
Ours | 2.73 | 3.07 | 2.02
Table 1: Comparison of mean error of expression neutralization on different
datasets. All errors are in millimeters.

###  4.2 Face Recognition

A direct application of the disentangled identity representation obtained
using our method consists of face recognition. As mentioned in Section 2.3, we
introduced a face recognition module that improved the face neutralization by
enhancing identity preservation as shown in Section 4.1. Such module can also
be used to predict the identity of an input face mesh under different facial
expressions. To validate this claim, we conducted face identification
experiments based on the embedding of the face recognition module
$\mathcal{R}$. In particular, we consider the identity features obtained by
the last FC layer of the face recognition module (before applying softmax) as
an identity representation of an input 3D face. These representations are then
compared using cosine similarity measure to achieve face identification.

Experimental protocol: For face identification, only FWH and FaceScape
datasets were retained since COMA dataset consists of only $12$ subjects
making not useful to evaluate face identification. As done for expression
neutralization experiments, the two considered datasets were split into
training and testing sets following a cross-subject protocol. The training of
the whole network was achieved on the training set and the testing set of
unseen subjects was considered to evaluate face identification. We considered
neutral 3D faces of the testing set as gallery and the remaining expressions
as probe for the two datasets. The evaluation metric was the rank-1 accuracy.

Method | FWH (%) | FaceScape (%)
---|---|---
GCA [20] | 90.48 | 95.64
Ours w/o $\mathcal{L}_{l2l}$ | 88.75 | 97.83
Ours | 92.34 | 99.88
Table 2: Comparison of rank-1 accuracies of face identification on FWH and
FaceScape datasets.

Results and discussions: In Table 2, we report the rank-1 accuracies obtained
using our method on FaceScape and FWH datasets. On the FaceScape dataset which
includes a large number of identities, we achieved a rank-1 accuracy of
99.88%. A lower performance consisting of 92.34% was obtained on the FWH
dataset which involves lower number of identities than the FaceScape dataset.
As noticed for expression neutralization in Section 4.1, face identification
results suggest that the performance of the proposed method is better when a
large number of identities are involved in the training. Similarly to
expression neutralization experiments in Section 4.1, we compare our face
identification results with the adapted GCA method [20]. The considered
identity features to conduct face identification with GCA were the latent
representations of the input expressive faces (i.e., bottleneck of the
autoencoder). Our approach outperformed the GCA method by around 4% and 2% on
FaceScape and FHW datasets, respectively.

We also evaluate our results when discarding the $\mathcal{L}_{l2l}$ loss
term. This means that expression neutralization is only ensured by the
reconstruction losses defined in Eq. 6. Unsurprisingly, face identification
accuracies were lower than those achieved by the retained approach on the two
considered datasets.

##  5 Conclusion And Future Works

In this paper, we proposed a neural network method for learning disentangled
identity representations. Given an input expressive 3D face, the proposed
network neutralizes the facial expression and also provides relevant identity
features for face identification. This is achieved by translating expressive
latent space representations of 3D faces obtained using spectral graph
convolutions to neutral ones. The obtained results of face expression
neutralization and face identification on three publicly available datasets
showed the effectiveness of the proposed approach. However, some limitations
were also noticed when the number of identities is not sufficient which
resulted in less generalization abilities to unseen subjects. A data
augmentation step increasing face shape variations before training can be
introduced in the future to overcome this problem. As additional future work,
the collection of expressive and neutral pairs can be avoided by considering
an unpaired translation of the latent space such as in [23].

## Acknowledgement

This work was funded by the National Research Fund (FNR), Luxembourg, under
the project reference CPPP17/IS/11643091/IDform/Aouada

## References

  * [1] Face ID apple inc.  https://support.apple.com/en-us/HT208108.  Accessed: 2021-04-21.
  * [2] Brian Amberg, Reinhard Knothe, and Thomas Vetter.  Expression invariant 3d face recognition with a morphable model.  In 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition, pages 1–6. IEEE, 2008.
  * [3] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194, 1999.
  * [4] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway.  A 3d morphable model learnt from 10,000 faces.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5543–5552, 2016.
  * [5] Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael Bronstein, and Stefanos Zafeiriou.  Neural 3d morphable models: Spiral convolutional networks for 3d shape representation learning and generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7213–7222, 2019.
  * [6] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun.  Spectral networks and locally connected networks on graphs.  arXiv preprint arXiv:1312.6203, 2013.
  * [7] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.
  * [8] Shiyang Cheng, Michael Bronstein, Yuxiang Zhou, Irene Kotsia, Maja Pantic, and Stefanos Zafeiriou.  Meshgan: Non-linear 3d morphable models of faces.  arXiv preprint arXiv:1903.10384, 2019.
  * [9] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  In Advances in neural information processing systems, pages 3844–3852, 2016.
  * [10] Hassen Drira, Boulbaba Ben Amor, Anuj Srivastava, Mohamed Daoudi, and Rim Slama.  3d face recognition under expressions, occlusions, and pose variations.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(9):2270–2283, 2013.
  * [11] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315–323, 2011.
  * [12] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [13] Zi-Hang Jiang, Qianyi Wu, Keyu Chen, and Juyong Zhang.  Disentangled representation learning for 3d face shape.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11957–11966, 2019.
  * [14] Yinjie Lei, Mohammed Bennamoun, Munawar Hayat, and Yulan Guo.  An efficient 3d face recognition approach using local geometrical signatures.  Pattern Recognition, 47(2):509–524, 2014.
  * [15] Huibin Li, Di Huang, Jean-Marie Morvan, Yunhong Wang, and Liming Chen.  Towards 3d face recognition in the real: a registration-free approach using fine-grained matching of 3d keypoint descriptors.  International Journal of Computer Vision, 113(2):128–142, 2015\.
  * [16] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212–220, 2017.
  * [17] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [18] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman.  Deep face recognition.  2015\.
  * [19] Dijana Petrovska-Delacrétaz, Gérard Chollet, and Bernadette Dorizzi.  Guide to biometric reference systems and performance evaluation.  Springer, 2009.
  * [20] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black.  Generating 3d faces using convolutional mesh autoencoders.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 704–720, 2018.
  * [21] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [22] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao.  Facescape: A large-scale high quality 3d face dataset and detailed riggable 3d face prediction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [23] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.