# Distortion modes in halide perovskites: to twist or to stretch, a matter of
tolerance and lone pairs

Santosh Kumar Radhaa, Churna Bhandarib, and Walter R. L. Lambrechta a
Department of Physics, Case Western Reserve University, 10900 Euclid Avenue,
Cleveland, OH-44106-7079  b Department of Physics and Astronomy, University of
Missouri, Columbia, Missouri 65211, USA

###### Abstract

Using first-principles calculations, we show that CsBX3 halides with B=Sn or
Pb undergo octahedral rotation distortions, while for B=Ge and Si, they
undergo a ferro-electric rhombohedral distortion accompanied by a rhombohedral
stretching of the lattice. We show that these are mutually exclusive at their
equilibrium volume although different distortions may occur as function of
lattice expansion. The choice between the two distortion modes is in part
governed by the Goldschmidt tolerance factor. However, another factor
explaining the difference between Sn and Pb compared with Ge and Si is the
stronger lone-pair character of Ge and Si when forced to be divalent as is the
case in these structures. The lone-pair chemistry is related to the off-
centering. While the Si-based compounds have not yet been synthesized, the Ge
compounds have been established experimentally. As a final test of the
importance of the tolerance factor we consider RbGeX3, which has smaller
tolerance factor than the corresponding CsGeX3 because Rb is smaller than Cs.
We find that it can lower its energy by both rotations or rhombohedral off-
centering distortions but the latter lower the energy slightly more
efficiently.

##  I Introduction

The cubic perovskite structure is well known from the oxide perovksites to
exhibit various possible phase transitions. These fall in two main categories:
ferro-electric distortions, in which the $B$ atom in ABX3 is displaced within
its surrounding octahedron, and antiferro-electric distortions, in which the
octahedra rotate, possibly about multiple axes. Depending on the type of
displacement, for example along a cubic axis such as [001], or along two cubic
axis or a [110] direction, or three cubic axes, corresponding to the [111]
axis, the resulting symmetry becomes tetragonal, orthorhombic or rhombohedral.
Likewise for the rotation type instabilities, rotation about one cubic axis
leads to a tetragonal structure, about two orthogonal axes leads to an
orthorhombic phase.

The halide perovskites with B=Pb, Sn, Ge have recently garnered a lot of
attention, mostly driven by the hybrid organic/inorganic halides’ demonstrated
potential for solar cell applications.Kojima  _et al._ (2009); Lee  _et al._
(2012); Ball  _et al._ (2013); Eperon  _et al._ (2014); Liu  _et al._ (2013);
Burschka  _et al._ (2013); Kim  _et al._ (2014); Park (2015) In particular
methyl ammonium lead iodide (CH3NH3PbI3 or (MA)PbI3 or MAPI) and closely
related materials have reached larger than 20 % efficiencies in solar cells in
a record development time frame. The interplay between the dipole character
and the orientation of the organic component and the inorganic framework leads
to interesting effects on the above mentioned phase transitions.Quarti  _et
al._ (2014); Li and Rinke (2016) However, similar phase transitions also occur
in the purely inorganic CsBX3 family. While these distortions, which are
related to soft-phonon mode instabilities,Huang and Lambrecht (2014) lead to
minor changes in the band structure, related to bond angle distortions, other
phases are known in the halides, which are far more disruptive of the band
structure. These latter phases include edge-sharing octahedra and exhibit band
structures with much wider band gaps than their perovskite counterparts.Huang
and Lambrecht (2013) As an example, the structural phases in CsSnI3 were
studied in detail by In Chung et al. Chung  _et al._ (2012). They fall
generally in a set of three “black phases”, cubic, tetragonal and
orthorhombic, which correspond to rotated octahedral structures, and another
orthorhombic “yellow phase”, which has 1D chains of edge-sharing octahedra
forming Sn2I${}_{6}^{2-}$ structural motifs. It is notable that the
transitions from cubic to tetragonal to orthorhombic perovksite each time
increase the density and the yellow phase has an even higher density. The
orthorhombic $\gamma$-phase is stable with respect to soft-phonons, but has
been calculated to have an energy either lowerHuang and Lambrecht (2016) than
or very closeda Silva  _et al._ (2015) to that of the yellow phase.

Because the driving force for these transitions appears to be the increasing
density, the occurrence of the edge-sharing octahedral structures, which
appears to be detrimental for many of the sought applications, may perhaps be
already inferred from the behavior of the material under octahedral rotations,
which in turn is related to the relative sizes of the ions. For example, for
the CsGeX3 compounds, the sequence of tetragonal, orthorhombic octahedral
rotations is not observed and, to the best of our knowledges, no edge-sharing
octahedral phase is known to occur, although a different, monoclinic phase
occurs for the Cl members of the family. Instead of octahedral rotation
phases, a ferro-electric rhombohedral distortion is found to occur in these
materials, consisting of the displacement of the Ge along the body diagonal of
the cubic unit cell, accompanied by a rhombohedral stretch of the unit cell.

In this paper we examine the behavior of a family of halide perovskites
computationally under both octahedral rotation and rhombohedral ferro-electric
distortions. Hence the phrase in the title: “to rotate or to stretch”. We find
that the Sn and Pb members of the family of cubic perovskites are unstable
toward rotation of the octahedra but stable with respect to ferro-electric
distortions. In contrast, the Ge and Si based compounds show the opposite
behavior: they are unstable towards ferro-electric distortion but are stable
with respect to rotations. Furthermore we relate this distinct behavior to the
Goldschmidt tolerance factor,Goldschmidt (1926) which provides a convenient
way to summarize the relative ionic sizes. Notably, we include here the Si
based halide perovskites, which have, as far as we known, not yet been
synthesized.

The remainder of the paper is organized as follows. The details of our
computational approach are given in Sec. II. The relationships between the
different crystal structures and distortions to be studied are given in Sec.
III. The results Sec. IV is divided in several subsections. First, we give a
qualitative discussion in Sec. IV.1 establishing the different behavior of Sn
and Pb vs. Si and Ge. Next, we consider full relaxations of the rotationally
distorted structures of Sn and Pb based compounds in Sec. IV.2, then the full
relaxations of the rhomobohedral structures of the Ge and Si based compounds
in Sec. IV.3. In Sec. IV.4 we study the competion between both types of
distortion as function of lattice expansion for the Sn and Pb based systems.
Finally in Sec. IV.5 we consider the RbGeX3 compounds and end with a summary
of the results in Sec.V.

##  II Computational Methods

The calculations are performed within density functional theory in the local
density (LDA) and/or generalized gradient (GGA) approximations. Specifially,
we use the Perdew-Burke-Ernzerhof (PBE) form of GGA.Perdew  _et al._ (1996)
The full-potential linearized muffin-tin orbital (FP-LMTO) band-structure
method is utilized.Methfessel  _et al._ (2000); Kotani and van Schilfgaarde
(2010) Within this method, the basis set consists of Bloch sums of atom
centered spherical waves as envelope functions, described by smoothed Hankel
functions,Bott  _et al._ (1998) which are then augmented with solutions of the
radial Schrödinger equation inside muffin-tin spheres and their energy
derivatives. For the present calculations, a large basis set of $spdf-spd$
with two sets of Hankel function decay constants $\kappa$ and smoothing radii
is used. Inside the sphere, augmentation is done to an angular momentum cut-
off of $l_{max}=4$. The Cs $5p$ states are treated as valence electrons.
Likewise for Rb, the semicore $4p$ are treated as local orbitals. The
Brillouin zone integrations are done with a $6\times 6\times 6$
$\Gamma$-centered mesh.

The LDA turns out to significantly underestimate the lattice constants in
these materials, much more than the GGA overestimates them. Although our
initial study of the rotation or distortion patterns used the experimental
lattice constants of the cubic phase, our final full relaxation are done
within GGA-PBE.

##  III Crystal structures

Figure 1: Rotation of octahedra in perovskite structure, large circles: A
atom, small open circle: B atom, smallest filled circle: X atom, the black
small circle corresponds to the cubic perovskite position, the red one the
rotated one. The blue dash-dotted trangle indicates the rotation angle
$\theta$.

We start from the cubic perovskite structure. In this structure, with a simple
cubic Bravais lattice, for the composition ABX3, the $B$ atom occurs in the
center of the cubic unit cell and is octahedrally surrounded by $X$ atoms on
the face centers. The $A$ atoms occupy the corners of the cubic cell. The
stability is governed among other by the Goldschmidt tolerance factor,
$t=(R_{A}+R_{X})/\sqrt{2}(R_{B}+R_{X})$, where $R_{A}$, $R_{B}$, $R_{X}$ are
the ionic radii. Hence for $t=1$ the ionic spheres are touching and hence
Goldschmidt’s original idea was that $t$ should not deviate too far from $1$
for the perovskite structure to be stable. When $t<1$, the $A$ ion is somewhat
too small for the interstitial space between the octahedra. This is what leads
to the rotations, which tighten the space for the $A$ ion. In contrast, when
$t>1$, the octahedral space is too large for the $B$ ion, which might then be
expected to shift in its surroundings to make stronger bonds with a subset of
the 6 neighbors. On the other hand, it is not so clear a-priori whether this
is related to the tolerance factor or to the lone-pair character of the
B-cation.

In terms of octahedral rotations, we consider both the in-phase and out-of-
phase rotations about a single cubic axis. These both lead to a tetragonal
structure, the first one having the space group No. 127, $P4/mbm$ or
$D_{4h}^{5}$, the second one space group No. 140 , $I4/mcm$ or $D_{4h}^{18}$.
They correspond to the Glazer tilt systemsGlazer (1972); Woodward (1997)
$a^{0}a^{0}c^{+}$ and $a^{0}a^{0}c^{-}$ respectively. Although other Glazer
tilt systems are possible and in fact occur in the Sn-halide perovskites,Huang
and Lambrecht (2014) (a rotation about a second axis (Glazer $a^{+}b^{-}b^{-}$
leads to the orthorhombic $Pnma$ or $D_{2h}^{16}$ $\gamma$-phase), we here are
primarily concerned with the instability either with respect to rotation of
octahedra or ferro-electric distortions and thus consider the tetrahedral
rotation as the trigger toward rotation behavior. So, we do not consider other
tilt systems. In the tetragonal $P4/mbm$ structure, the Wyckoff positions for
the B-atoms is 2a, for the A-atom is 2c, for the X atoms, 2b and 4h. The $x$
parameter of the 4h positions is related to the rotation angle of the
octahedron by $\tan{\theta}=1-4x$ as can be seen in Fig. 1. In fact, the blue
rectangular triangle marked by one corner at position $(x,x+\frac{1}{2})$ has
sides $(x-\frac{1}{4})\sqrt{2}$ and $\sqrt{2}/4$ in units of the lattice
constant $a$ and hence their ratio gives $\tan{\theta}$.

As far as the ferro-electric distortions, we only consider the rhombohedral
structure corresponding to a displacement of the central $B$ ion along the
[111] direction. In the prototypical ferro-electric oxide BaTiO3 this phase
occurs at the lowest temperatures, with an orthorhombic and tetragonal phase
occuring at higher temperatures and eventually a cubic phase. Cooling from
high temperature, the displacement thus acquires successively more components
along the cubic axes which deviate from the central position. While we
presently do not exclude these other potential phases, our choice is guided by
the CsGeX3 compounds, which have been found to exhibit this rhombohedral phase
at low temperatures and a cubic phase at high temperatures but no other phases
in between. The rhombohedral symmetry distortion of the ion is accompanied by
a rhombohedral shear of the lattice vectors. Thus we will study the energy as
function of displacement of the ion for varying rhombohedral strain.

Table 1: Shannon ionic radii ($R_{i}$) and tolerance factors ($t$) of cubic perovksites. The last column indicates whether the cubic structure is unstable toward octahedron rotation.  Ion |  $R_{i}$ (Å) |
---|---|---
Cs | 1.88 |
Rb | 1.52 |
Si | 0.4 |
Ge | 0.53 |
Sn | 0.69 |
Pb | 0.775 |
Cl | 1.81 |
Br | 1.96 |
I | 2.2 |
Compound | $t$ | rotations
CsSiI3 | 1.10 | no
CsGeI3 | 1.057 | no
CsSnI3 | 0.998 | yes
CsPbI3 | 0.970 | yes
CsSiBr3 | 1.151 | no
CsGeBr3 | 1.090 | no
CsSnBr3 | 1.025 | yes
CsPbBr3 | 0.993 | yes
CsSiCl3 | 1.181 | no
CsGeCl3 | 1.115 | no
CsSnCl2 | 1.044 | yes
CsPbCl3 | 1.009 | yes
RbGeCl3 | 1.006 | yes
RbGeBr3 | 0.988 | yes
RbGeI3 | 0.964 | yes

The occurrence of this distortion in Ge based halides but not in Sn or Pb
based systems, which we will demonstrate later, is not only related to the
Goldschmidt ratio of ionic sizes but is also related to the lone-pair
character of the bonding. As one goes down the column of group-IV atoms, the
valence $s$ states become increasingly deeper relative to the valence $p$
states. That is why carbon has $s$ and $p$ orbitals of similar extent and is
extremely flexible in choosing different hybridization schemes: $sp^{2}$ in
graphite, $sp^{3}$ in diamond and so on. Si and Ge clearly prefer $sp^{3}$
hybridization and thus tend to be tetravalent, while Sn and Pb become
increasingly divalent. Nonetheless, in the halide perovskite crystal
structure, it is clear that even Ge behaves as a divalent ion. Whether Si can
also be forced to be divalent in these compounds remains to be seen. However,
the $s$-electrons then behave as a stereochemically active lone-pair, which
promotes off-centering of the Ge in its surrounding octahedron with an
asymmetric bonding configuration in which the lone pair electrons are located
opposite to the direction of the displacement of the ion.Walsh  _et al._
(2011) The lone-pair related trends in the series Pb-Sn-Ge have been addressed
by Waghmare et al. Waghmare  _et al._ (2003) in the context of IV-VI
compounds. We will show that even in the Sn-case this happens under lattice
expansion, as was previously shown by Fabini et al. Fabini  _et al._ (2016)
According to the latest insights into lone-pair chemistry, the hybridization
with the anion $p$-orbitals play a crucial role in this. The important role of
the Sn-$s$ halogen-$p$ hybridization on the band structure of CsSnX3 halides
was already pointed out in our earlier work.Huang and Lambrecht (2013) We
point out here that competition between rotation instabilities and lone-pair
off-centering was previously studied in CsPbF3 by Smith et al. Smith  _et al._
(2015) Lone pair physics related to Pb also occurs when Pb is the A-cation in
oxide perovskites.Waghmare and Rabe (1997); Ghosez  _et al._ (1999)

Finally, we should mention that the tolerance factor depends on the choice of
ionic radii. Usually we use the ShannonShannon (1976) ionic radii for this
purpose. However, these are themselves based on an analysis of bonding in
different coordinations and for example do not give us any information on the
behavior under hydrostatic pressure. One might conceivably think of the
relative ion sizes to change with pressure or wish to include other aspects
than pure ionic size to predict structural stability.Brehm  _et al._ (2014);
Travis  _et al._ (2016) With these precautions, we used the Shannon ionic
radii calculated tolerance factors as a guide to our study. They are
summarized in Table 1. We note that our goal with the tolerance factor is not
so much to predict structural maps in the sense of separating perovskite
versus non-perovskite forming compounds but rather the type of structural
distortion occurring within the perovskite. Also, because Shannon only
provides ionic radii for Pb(II) in the divalent state, but not for Sn, Ge or
Si, we used instead the tetravalent radii for octahedral environment. This may
seem to contradict the fact that in these structures the B ion is supposed to
be divalent. On the other hand, we should recognize that the bonding is
partially covalent anyways. We find that within each group of a given anion,
the tolerance factor decreases along the sequence Si-Ge-Sn-Pb. The dividing
critical value between octahedral rotations being favored or not, depends
actually on which anion (a similar point was also made by Travis et al. Travis
_et al._ (2016)), but is close to 1 for all the Cs compounds. For the Cl
compounds, it would be between 1.115 and 1.044. For Rb which has a smaller
radius, the value 1.006 is thus definitely on the small side and hence
predicts rotations to occur.

Our goal in this paper is to study the instability of the cubic structure to
these two types of distortion as function of the $B$ atom and to correlate
them with the tolerance factors in the above Table 1.

##  IV Results

###  IV.1 Qualitative discussion

First, we consider the CsSnI3 compound. In Fig. 2a we show its total energy as
function of rotation angle $\theta$ of the octahedra. This calculation is done
at the cubic experimental volume although we know that the observed
$\beta$-structure, corresponding to the $P4/mbm$ space group has higher
density. We consider both the in-phase and out-of-phase rotations. The figure
shows that their energy is almost indistinguishable. More importantly, it
shows clearly that the system prefers a rotation angle of about 6.9∘. Of
course, the rotation can be either clockwise or counterclockwise. The energy
barrier between the two is of the order of a few meV/formula unit. So, this
agrees with the well-established fact that CsSnI3 undergoes octahedral
rotations of this type although the equilibrium optimum angle appears to be
somewhat underrestimated compared to the experimental angle which is 9∘,
corresponding to the Wyckoff parameter $x=0.21$. This is because in this
initial calculation, we kept the cubic structure of the lattice and did not
allow yet for a full relaxaton. Full relaxation results are given later in
Table 2 and are discussed in subsection IV.2. The present result shows that
the rotation instability is already present even at the volume of the cubic
structure.

Figure 2: left: Total energy (per formula unit in meV) vs. octahedral rotation
angle $\theta$ (∘) in CsSnI3 left. Here $\theta+$ stands for out-of-phase
rotation and $\theta-$ for in-phase rotation respectively. Right: total energy
vs. displacement of Sn from body center in unit of the cubic lattice constant
$a$.

Next, we consider the behavior of CsSnI3 under the ferro-electric rhombohedral
distortion. We do this at zero strain, so keeping the cubic lattice vectors.
Clearly there is only one minimum at exactly the central position of the Sn in
its octahedral cage. So, there is no evidence for a ferro-electric
instability. Nonetheless, the curves are clearly not parabolic but show a
rather flat energy minimum region for the position of the central atom.

Figure 3: Left: Total energy vs. octahedral rotation angle $\theta$ (∘) in
CsGeI3 left. Here $\theta+$ stands for out-of-phase rotation and $\theta-$ for
in-phase rotation respectively. Right: total energy vs. displacement of Ge
from body center in units of the lattice constant $a$.

In contrast, if we consider CsGeI3, (Fig. 3a) for its rotational stability, we
find no evidence at all of a rotational instability. The preferred angle is 0.
This is true for both in-phase and out-of-phase rotations. On the other hand,
in Fig.3b we see that now there is a clear instability against the ferro-
electric displacement. Again, it is symmetric with respect to the central
position. The displacement is given in units of the lattice constant $a$ of
the cubic cell. The optimimum position lies between 0.52 and 0.54 or 0.46 and
0.48. In this case, we study the optimum position and the energy barrier as
function of rhombohedral strain but initially keeping the volume fixed at that
of the cubic structure. This is quantified by the parameter $\eta$ which gives
the stretch along the [111] direction (when $\eta>1$) and is compensated by a
compression in the orthogonal directions, which conserves the volume. Thus, we
applied here a pure shear or traceless strain at fixed volume. We can see that
the optimum position varies slightly with the strain. The lowest overall
energy occurs for a strain of $\eta=1.03$ and $\delta u=0.035$. A full
structural relaxation within the rhombohedral symmetry requires not only
optimizing $u$ and $\eta$ but also the volume and the results of such a full
relaxation are given in Table 3 in Sec. IV.3.

We thus see a mutually exclusive behavior of the two type of distortion modes.
Either the material is unstable under rotations, or it is unstable under the
ferro-electric distortion but not both. We found that these structural
instabilities already occur at the cubic structure equilibrium volume but once
the distortion takes place and full relaxation is allowed, a new equilibrium
is found. We should remember though that the mutual exclusivity correspond to
the experimental volume. This might change as function of pressure. For
example, in SrTiO3, Zhong and VanderbiltZhong and Vanderbilt (1995) predict an
interplay between the two types of distortions, leading eventually to a
complex phase diagram as function of pressure and temperature. We will discuss
the distortion behavior for CsSnI3 as function of lattice constant later.

Having established the basic two types of behavior, we now consider the
variation with anion. In the CsSnBr3 and CsSnCl3 cases, we again find the
structure to be stable against ferro-electric distortion, but unstable toward
rotations. The energies as function of rotation angle are given in
Suppelementary Material.sup  For the CsGeBr3, CsGeCl3 cases, we find the
structures to be stable under rotation as expected but we do find a ferro-
electric distortion in both cases.sup  Next we show that Pb behaves similar to
Sn and Si behaves similar to Ge.sup  For the Si case, where no experimental
results are known, we initially used the LDA optimized lattice constants for
the cubic CsSiX3 case but in the next section for our fully relaxed
structures, we use GGA-PBE for improved accuracy.

###  IV.2 Full structural relaxation for Sn and Pb based rotations.

In this section we study the fully relaxed tetragonal $P4/mbm$ structure
corresponding to the rotational distortions. The optimum rotation angles are
summarized in Table 2. Because we found LDA to underestimate the lattice
constants more than GGA overstimates them, we performed the full structural
relaxations in GGA-PBE. In Table 2 we show both the results for the rotation
angle when fixing the lattice constants to be “rotated cubic” and fully
relaxing the tetragonal structure, i.e. also relaxing $c/a$. By “rotated
cubic” we mean we consider a $\sqrt{2}\times\sqrt{2}$ superlattice in which
the octahedrons can rotate as shown in Fig. 1 but keep the $c/a$ ratio exactly
at a factor $\sqrt{2}$ and keep the volume at the cubic volume. These results
are also presented in Fig. 4 to visualize the trends with halogen.

Table 2:  Structural relaxation results for rotation for the CsSnX3 and CsPbX3 compounds: $\alpha^{\prime}$ means “rotated cubic” and $\beta$ means fully relaxed tetragonal. All results obtained within GGA-PBE. Volume is per formula unit. $\Delta E$ is the energy barrier between the optimum angle structure and the cubic structure at rotation angle $\theta=0$. Compound | CsSnI3 | CsSnBr3 | CsSnCl3
---|---|---|---
Structure | $\alpha^{\prime}$ | $\beta$ | $\alpha^{\prime}$ | $\beta$ | $\alpha^{\prime}$ | $\beta$
$a$ (Å) | 8.935 | 8.800 | 8.372 | 8.282 | 8.033 | 7.942
$c$ (Å) | 6.318 | 6.300 | 5.920 | 5.944 | 5.78 | 5.710
$V$ (Å3) | 252.19 | 243.92 | 207.47 | 203.84 | 183.25 | 180.10
$\Delta V/V$ (%) |  | -3.28 |  | -1.75 |  | -1.72
$\theta$ (∘) | 6.93 | 10.1 | 3.61 | 8.85 | 2.49 | 8.32
$\Delta E$ (meV) | 9.9 | 11.2 | 4.6 | 6.6 | 0.7 | 8.9
Compound | CsPbI3 | CsPbBr3 | CsPbCl3
Structure | $\alpha^{\prime}$ | $\beta$ | $\alpha^{\prime}$ | $\beta$ | $\alpha^{\prime}$ | $\beta$
$a$ (Å) | 9.065 | 8.610 | 8.514 | 8.367 | 8.160 | 8.034
$c$ (Å) | 6.410 | 6.245 | 6.020 | 6.085 | 5.77 | 5.82
$V$ (Å3) | 263.37 | 231.520 | 218.17 | 213.039 | 192.10 | 187.98
$\Delta V/V$ (%) |  | -12.09 |  | -2.35 |  | -2.15
$\theta$ (∘) | 10.75 | 12.36 | 9.13 | 12.36 | 8.58 | 11.77
$\Delta E$ (meV) | 33 | 258 | 22 | 50 | 17 | 39
Figure 4: Optimized volume and rotation angle as function of halogen in CsSnX3
and CsPbX3.

For CsSnI3, there are two sets of experimental results, by Yamada et al.
Yamada  _et al._ (1991) and by In Chung et al. Chung  _et al._ (2012). Yamada
et al. give $a=8.772$ Å, $c=6.261$ Å, $V=240.815$ Å3 for the $\beta$-structure
and $a=6.219$ Å, $V=240.526$ Å3 for the cubic structure, in other words,
almost the same volume. In contrast, In Chun et al. Chung  _et al._ (2012)
give $a=8.7182$ Å, $c=6.1908$ Å, $V=235.27$ Å3 for the tetragonal and and
$a=6.2057$ Å, $V=238.99$ Å3 for the cubic structure. These results correspond
to 500 and 380 K respectively and clearly show a smaller volume for the
tetragonal structure. Our calculated results agrees qualitatively better with
those of In Chun et al. Chung  _et al._ (2012) in finding a volume reduction
induced by the $\alpha\rightarrow\beta$ transition. Our GGA calculations
overestimate the experimental volumes by about 5.6% and 3.6 % for the cubic
and tetragonal structures compared to In Chung et al. Chung  _et al._ (2012)
We find systematically the same trend in volumes for the other compounds. We
may note that the optimum rotation angle depends strongly on volume. It is
typically larger in the relaxed tetragonal $\beta$-structure than if we keep
the volume fixed at the cubic volume. We may also note that it decreases with
decreasing volume along the series CsSnI3, CsSnBr3, CsSnCl3 and similarly in
the Pb based series. The rotation angles are larger in the Pb-based compounds
than in the Sn-based compounds. This means the smaller the tolerance factor,
the larger the rotation. Our optimum angle of octahedral rotation for CsSnI3
agrees well with the experimental value of 9.09∘.Yamada  _et al._ (1991)

Finally we may note that for the larger cubic volumes, the rotation angle for
the Sn-based compounds becomes rather small. Below, in Sec. IV.4, we show that
under lattice constant expansion it actually goes to zero, and, at some
critical volume, the ferro-electric distortion becomes preferable instead.

The energy barriers $\Delta E$ between the tetragonal energy minimum and the
cubic unrotated structure are seen to be significantly larger for the Pb
compounds than the Sn compounds and within each family decrease from I to Br
to Cl, except for the fully relaxed CsSnBr3 and CsSnCl3.

Figure 5: Trends in structural relaxation parameters for the CsGeX3 and CsSiX3
halides corresponding to the data in Table 3

###  IV.3 Full structural relaxation for Ge and Si based rhombohedral
distortions.

Table 3: Optimized cubic and rhomobohedral structures for CsBX3 with B=Ge, Si. The $\Delta E$ are the barriers between the cubic structure with $\delta u=0$ and $\eta=1$ and the optimized rhombohedral structure each at their own equilibrim volume.  Compound | CsGeCl3 | CsGeBr3 | CsGeI3
---|---|---|---
cubic $V$ GGA (Å3) | 155.72 | 177.50 | 216.00
cubic $V$ Expt.(Å3) | 163.67 | 184.22 | 221.44
rhombohedral $V$ (Å3) | 168.19 | 189.11 | 228.46
rhombohedral $a$ (Å) GGA | 5.52 | 5.74 | 6.11
rhombohedral $a$ (Å) Expt | 5.434 | 5.635 | 5.983
$\delta u$ | 0.027 | 0.026 | 0.028
$\delta_{1}$ | 0.015 | 0.011 | 0.008
$\delta_{2}$ | 0.022 | 0.013 | 0.004
$\eta$ | 1.014 | 1.023 | 1.024
$\alpha$ GGA | 89.17 | 88.64 | 88.61
$\alpha$ Expt. | 89.72 | 88.74 | 88.61
$\Delta E$ (meV) | 75 | 65 | 56
$E_{g}$ (eV) GGA | 2.01 | 1.31 | 1.05
Compound | CsSiCl3 | CsSiBr3 | CsSiI3
cubic $V$ | 145.531 | 166.375 | 203.297
rhombohedral $V$ (Å3) | 170.53 | 187.06 | 222.91
rhombohedral $a$ (Å) | 5.54 | 5.71 | 6.06
$\delta u$ | 0.038 | 0.029 | 0.052
$\delta_{1}$ | 0.007 | 0.010 | 0.007
$\delta_{2}$ | -0.057 | 0.033 | 0.018
$\eta$ | 1.021 | 1.016 | 1.034
$\alpha$ | 88.79 | 89.05 | 88.00
$\Delta E$ (meV) | 357 | 235 | 142
$E_{g}$ (eV) GGA | 2.02 | 1.31 | 0.605

In this section we further study the fully relaxed rhombohedrally distorted
structures. In Table 3 we first give the optimum GGA volume of the cubic
structure. It is compared with the experimental values at elevated temperature
where that phase is stable, from Thiele et al. Thiele  _et al._ (1987) at 170,
270 and 300 ∘C respectively for the Cl, Br, I cases. Clearly these values are
larger than our GGA because of the lattice expansion at elevated temperature.
Next, we applied a rhombohedral strain along the cubic structure, allowed the
central Ge atom to go off-center by a displacement $\delta u$ and allowed the
volume to relax. The strain $\eta=1+2\epsilon$ is applied along the [111]
cubic direction while perpendicular to it, the distances are multiplied by
$1/\sqrt{\eta}\approx 1-\epsilon$, thus maintaining the volume. The strain
tensor can be written to linear order

$\bm{\epsilon}=\left(\begin{array}[]{ccc}0&\epsilon&\epsilon\\\
\epsilon&0&\epsilon\\\ \epsilon&\epsilon&0\end{array}\right)$

The cubic lattice vectors $a[1,0,0]$ are thus distorted into vectors of
$a(1,2\epsilon,2\epsilon)$ with length $a\sqrt{1+2\epsilon^{2}}$ which to
first order in $\epsilon$ means they stay unchanged. The results for the Ge
and Si based compounds are given in Table 3. We can see that for the Br and I
cases, our relaxed lattice constant for the rhombohedral phase in GGA slightly
overestimates the experimental value, even though the latter is measured at
20∘C while our calculated volume is in principle at 0 K. For the Cl case the
calculated lattice constant is slightly underestimated.

The displacement from the 0.5 value is almost the same in all cases. The
rhombohedral angle extracted from the shear $\eta$ using
$\cos{\alpha}=\eta_{min}-1$ to linear order in strain agrees well with the
experimental values. For the Si compounds, all values are obtained within GGA
and no experimental values are available to compare with.

The full relaxation also requires the anions to move. For example the anion
which in the cubic case is located at (0.5, 0, 0.5) moves to
$(0.5-\delta_{1},-\delta_{2},0.5+\delta_{1})$, in other words, it moves inward
toward the displaced Ge as shown in Fig.7. The motion of the other anions is
similarly determined by symmetry. The corresponding parameters are given in
Table 3. Table 4 shows that the B-X bond lengths are shortened upon relaxation
in spite of the overall volume being expanded in the rhombohedral distortion.

Table 4: B-X bond length ( in Å) compared between the perfect cubic structure and the relaxed structure, where B=Ge,Si and X=Cl,Br,I Compound | cubic | relaxed | % change
---|---|---|---
CsGeCl3 | 2.69 | 2.49 | -7.88%
CsGeBr3 | 2.81 | 2.65 | -5.93%
CsGeI3 | 3 | 2.86 | -4.77%
CsSiCl3 | 2.63 | 2.31 | -13.91%
CsSiBr3 | 2.75 | 2.50 | -9.80%
CsSiI3 | 2.94 | 2.68 | -9.62%

The energy differences $\Delta E$ between the cubic undistorted structure and
the rhombohedral optimized structure each at their own equilibrium volume are
also shown in Table 3. They indicate an increase from Cl to Br to I and much
larger values for the Si then the Ge based compounds.

Figure 6: Unit cell of the relaxed structure of CsSiCl3 with the colored atoms
at the relaxed positions and the gray atoms at the unrelaxed cubic positions.
Green pink and violet spheres represent Cs, Cl and Si respectively.

The band gaps, which must be underestimates because of the GGA, are also
included in Table 3 and show the expected trend of decreasing from Cl to Br to
I, in other words decreasing with decreasing ionicity. They are also smaller
in the Si than the Ge compounds. The gaps in the $GW$ approximation at the
experimental rhombohedral structures for the CsGeX3 compounds were given in
Ref. Huang and Lambrecht, 2016 and are 4.304, 2.654 and 1.619 eV for the Cl,
Br and I cases respectively. For the Si- based compounds, they remain to be
determined but assuming a similar gap correction, we can already see that both
CsSiI3 and CsSiBr3 may have gaps suitable for photovoltaics. The trends of the
data in Table 3 are visualized in Fig. 5.

Although the energy barriers increase from Cl to Br to I, they do not show a
clear correlation with the transition temperatures, which are 277-283 ∘C,
238-242 ∘C, and 155 ∘C respectively for CsGeI3, CsGeBr3, CsGeCl3. The problem
here is that our calculations consider a homogeneous transformaton, which is
forced to be the same in each unit cell. In the actual phase transition, there
is a competition between the interaction energies of atoms in neighboring
cells and the double-well anharmonic potential well in each unit cell. The
phase transition could be either displacive or order-disorder type.Dove (1997)
In the former case, corresponding to a large interaction energy between
neighboring cells, the positions of the atoms vibrate about an average near
the barrier maximum (corresponding to the cubic structure) at high temperature
and settle into one or the other minimum below the transition temperature. A
nucleation process occurs where groups of neighboring atoms settle into one of
the two local minima. In contrast in the order-disorder model, corresponding
to a strong double well potential but weaker intercellular interactions, the
atoms are always in one of the two minima but at high temperature, they are
equally likely to be in the left or right well. From our present calculations,
we do not have access to the inter-cell energies in such a model, and thus we
cannot draw conclusions about the nature of the phase transition.
Experimentally, it was established by Thiele et al. Thiele  _et al._ (1987)
that for CsGeBr3 and CsGeI3 the phase transition is first-order, while for the
Cl-case it is second order. This would indicate a displacive transition for
the latter case but an order-disorder type for the former.

###  IV.4 Rotation and rhombohedral distortion under volume expansion

Figure 7: Relative energy as function of rotation angle at various lattice
expansions for CsSnI3 as % expansions in lattice constant The energies are
considered relative to the un-rotated energy at each lattice parameter

As we already mentioned, the tendency toward octahedral rotation in the Sn and
Pb halides decreases, that is to say the rotation angle decreases, with
increasing volume for a given material. We therefore further studied the
behavior under lattice expansion and compression, which one might think of as
occurring by thermal expansion and under high-pressure respectively. First, we
show the energy curves for CsSnI3 as function of rotation angle for various
lattice expansions in Fig. 7. Even without volume expansion, we see that the
curves shows two local minima, one at zero angle and one at about 7∘. As we
increase the volume, the local minimum corresponding to the finite rotation
moves up in relative energy and eventually,beyond 3 % expansion of the lattice
constants, it disappears, at which point the curve becomes very flat. Although
they still show a very shallow finite angle minimum, we may essentially
consider this as a sign that the rotation is no longer preferred. On the other
hand, under compression, the local minimum appears to shift toward smaller
angle and becomes deeper relative to the unrotated structure.

Figure 8: Percentage lattice-constant expansion vs. angle of rotation and
distance of Sn from $[.5,.5,.5]$ for CsSnI3. The size and darkness of the
rotation markers (circular ones) represent the size of energy barrier $w.r.t$
the perfect cubic perovskite structure

The optimum angle of rotation is shown as the red curve as function of lattice
expansion in Fig. 8. The increasing values for lattice expansion actually
correspond to a very low energy barrier, as is indicated by the small sign of
the symbols marking each point and may to first approximation be ignored.
Under compression, the rotation angle clearly is reduced and the barrier
increases, meaning the energy of the rotated minimum becomes deeper.

Next, we examine the possibility of off-centering of the Sn atom as function
of lattice expansion. As we can see in Fig. 8 in the blue dashed curve, the
off-centering displacement stays zero until 1 % expansion at which point it
starts increasing linearly. Eventually it collapses again beyond 6 %
expansion. Similar results are also obtained for the other halogens and for
the Pb compounds as shown in Fig. 9. In summary, we find that beyond a given
lattice expansion the CsSnX3 and CsPbX3 materials undergo a rhombohedral
distortion with off-centering of the Sn (or Pb) rather than the octahedral
rotation.

Figure 9: Displacement of Sn/Pb from the center $[.5,.5,.5]$ for CsSnI3,
CsSnCl3, CsSnBr3 and CsPbI3 as function of lattice constant expansion.

This type of behavior was reported earlier for CsSnBr3 by Fabini et al. Fabini
_et al._ (2016) and related to the active lone-pair behavior of the $s$
electrons which was studied in detail. We thus see that there is indeed a
competition between the two types of distortion behavior, rotation or
rhombohedral off-centering. The lone pair character promotes the off-centering
and is strongest for Ge and Si if the latter are required, as in this
structure, to behave divalent but it also occurs in Sn and to a smaller degree
in Pb. However, in Sn and Pb this mechanism of distortion is in competition
with rotations, while in Ge and Si it is not. Finally, it should be pointed
out that the off-centering in CsSnBr3 was experimentally observed by Fabini et
al. Fabini  _et al._ (2016) but occurs dynamically. It was observed only
through analysis of the pair distribution functions. In other words, it does
not occur coherently throughout the sample, which means that a rhombohedral
crystallographic phase is not found for this compound. Instead it is hidden in
the cubic phase but is apparent from the large atomic displacements which are
coherent only on a local scale. This is an important difference from Ge where
the rhombohedral phase is the actually observed equilibrium crystal structure.

###  IV.5 Rb instead of Cs

Table 5: Energy barrier and angle of rigidly rotated RbGeX3 with X=Cl, Br, I.The Energy barrier is the barriers between the cubic structure with 0∘ and local/global minimum at the given angle  Compound | RbGeCl3 | RbGeBr3 | RbGeI3
---|---|---|---
Angle ($\theta$) | 4.71 | 6.93 | 10.21
Energy barier (meV) | -70.0 | 13.5 | 36.4

In this section we consider the RbGeX3 compounds compared with the CsGeX3
compounds. From Table 1 we expect that because of the smaller size of the Rb
ion, these compounds would be unstable toward octahedral rotation. The results
for rotation in Table 5 show indeed that octahedral rotation lowers the energy
for a finite rotation angle for I and Br but not for Cl. In the latter case,
there is still a local minimum at a finite angle but its energy is actually
higher than at the zero angle rotation. The energy lowering is comparable and
even larger than for the corresponding CsSnX3 compounds and the angle of
rotation is larger for I than for Br.

Table 6: Optimized cubic and rhomobohedral structures for RbGeX3 with X=Cl, Br, I. The $\Delta E$ are the barriers between the cubic structure with $\delta u=0$ and $\eta=1$ and the optimized rhombohedral structure each at their own equilibrim volume.  Compound | RbGeCl3 | RbGeBr3 | RbGeI3
---|---|---|---
Cubic $a$ (Å) GGA | 5.345 | 5.57 | 5.97
Cubic $V$ (Å3) | 152.70 | 172.80 | 212.77
Cubic bond length (Å) | 2.67 | 2.78 | 2.98
Rhombohedral $a$ (Å3) | 5.44 | 5.65 | 5.99
Rhombohedral $V$ (Å3) | 161.27 | 180.44 | 214.98
Rhombohedral bond length (Å) | 2.31 | 2.46 | 2.69
$\Delta V/V$ (%) | 5.31% | 4.23% | 1.02%
Change in bond length (%) | -13.37% | -11.55% | -9.84%
$\delta u$ | 0.035 | 0.038 | 0.036
$\delta_{1}$ | 0.001 | 0.002 | 0.001
$\delta_{2}$ | 0.011 | 0.025 | 0.014
$\eta$ | 1.026 | 1.052 | 1.039
$\alpha$ | 88.47 | 87.00 | 87.71
$\Delta E$ (meV) GGA | 455.8 | 367.2 | 304.7
Band Gap (eV) GGA | 1.72 | 1.05 | 0.43

On the other hand, from the previous sections, it is also clear that from the
point of view of lone-pair physics, Ge is prone to off-centering. Therefore we
also study the possibility of lowering the energy by the rhombohedral
disotortion. The results are shown in Table 6. This shows that the off-
centering and related rhombohedral distortion lowers the energy significantly
more efficiently than the octahedral rotation. For RbGeCl3 the rotation
actually does not lower the energy, while the distortion does. For RbGeBr3 and
RbGeI3, the energy lowering by the off-centering is significatnly larger than
by rotation of the octahedra. Thus comparing to the Cs case, this indicates
that the off-centering of Ge is not so much determined by the tolerance factor
but rather by the lone-pair physics. The relaxation parameters, barriers and
energy gaps in GGA for these compounds are given in Table 6 in the same way as
for the other compounds.

Finally, we illustrate the lone-pair character in this case by plotting the
charge density for this in Figs. 10, 11. The firts one shows a 3D view of
isosurfaces, the second one shows the valence charge density along the body
diagonal.

Figure 10: Total valence charge density for the relaxed distorted RbGeCl3
shown as a superposition of 8 isosurfaces with values ranging frm 0.058 to
0.071 $e/a_{0}^{3}$. Each isosorface is shown as a mesh of different color.
One can clearly distinguish the Ge-$s$ like lobe in the direction opposite to
the displacemnt. The pink sphere is Rb, the blue one Ge and the green ones Cl.
Figure 11: Plot of the valence charge density along the [111] body diagonal.
One can see the asymmetry of the charge density near the Ge position, at a
position larger than 0.5, again reflecting the lone-pair character.

##  V Conclusions

In this paper we have explored the stability of inorganic halide perovskites
ABX3 with $X$ a halogen (Cl, Br, I), $A$ a large alkali ion(Cs or Rb) and B a
group IV element, (Si, Ge, Sn, Pb), under two types of distortion: an
antiferroelectric distortion corresponding to octahedral rotation and a ferro-
electric off-centering of the central IV ion inside its halogen octahedron. At
first, we find that there is a clear trend that the Pb and Sn cases prefer
rotation while Ge and Si prefer ferro-electric distortion. We also find that
the rotation, when fully optimizing the structures, is accompanied by a
reduction of the volume. The off-centering is accompanied by rhombohedral
distortion and volume increase. The tendency toward rotation is clearly
related to the Goldschmidt tolerance factor. On the other hand, we find that
upon volume expansion, the rotation angle decreases and beyond a certain
expansion off-centering becomes favorable even for Sn and Pb based compounds.
The origin of the off-centering is thus more related to the lone-pair physics.
The Ge and Si based compounds, in which Ge and or Si are forced to behave as a
divalent ion, strongly favor lone-pair induced off-centering or ferro-electric
distortion. In the Rb case, both distortion modes tend to lower the energy
(except for the Cl case) but the ferro-electric distortion nonetheless lowers
the energy signiricantly more efficiently. Thus the lone-pair physics
dominates the RbGeX3 based compounds rather than the tolerance factor related
rotation. The two distortion mechanisms can thus be in competition with each
other and the off-centering for the Si and Ge cases occurs even if the
tolerance factor would allow for rotations as a mechanism to lower the energy.

###### Acknowledgements.

This work was supported by the U.S. Department of Energy (Basic Energy
Sciences) DOE-BES under grant No. DE-SC0008933. The calculations made use of
the High Performance Computing Resource in the Core Facility for Advanced
Research Computing at Case Western Reserve University.

## References

  * Kojima  _et al._ (2009) A. Kojima, K. Teshima, Y. Shirai, and T. Miyasaka, Journal of the American Chemical Society 131, 6050 (2009), pMID: 19366264, http://dx.doi.org/10.1021/ja809598r .
  * Lee  _et al._ (2012) M. M. Lee, J. Teuscher, T. Miyasaka, T. N. Murakami, and H. J. Snaith, Science 338, 643 (2012).
  * Ball  _et al._ (2013) J. M. Ball, M. M. Lee, A. Hey, and H. J. Snaith, Energy Environ. Sci. 6, 1739 (2013).
  * Eperon  _et al._ (2014) G. E. Eperon, V. M. Burlakov, P. Docampo, A. Goriely, and H. J. Snaith, Advanced Functional Materials 24, 151 (2014).
  * Liu  _et al._ (2013) M. Liu, M. B. Johnston, and H. J. Snaith, Nature 501, 395 (2013).
  * Burschka  _et al._ (2013) J. Burschka, N. Pellet, S.-J. Moon, R. Humphry-Baker, P. Gao, M. K. Nazeeruddin, and M. Grätzel, Nature 499, 316 (2013).
  * Kim  _et al._ (2014) H.-S. Kim, S. H. Im, and N.-G. Park, The Journal of Physical Chemistry C 118, 5615 (2014).
  * Park (2015) N.-G. Park, Materials Today 18, 65 (2015).
  * Quarti  _et al._ (2014) C. Quarti, E. Mosconi, and F. D. Angelis, Chem. Mater. 26, 6557 (2014).
  * Li and Rinke (2016) J. Li and P. Rinke, Phys. Rev. B 94, 045201 (2016).
  * Huang and Lambrecht (2014) L.-y. Huang and W. R. L. Lambrecht, Phys. Rev. B 90, 195201 (2014).
  * Huang and Lambrecht (2013) L.-y. Huang and W. R. L. Lambrecht, Phys. Rev. B 88, 165203 (2013).
  * Chung  _et al._ (2012) I. Chung, J.-H. Song, J. Im, J. Androulakis, C. D. Malliakas, H. Li, A. J. Freeman, J. T. Kenney, and M. G. Kanatzidis, Journal of the American Chemical Society 134, 8579 (2012), http://pubs.acs.org/doi/pdf/10.1021/ja301539s .
  * Huang and Lambrecht (2016) L.-y. Huang and W. R. L. Lambrecht, Phys. Rev. B 93, 195211 (2016).
  * da Silva  _et al._ (2015) E. L. da Silva, J. M. Skelton, S. C. Parker, and A. Walsh, Phys. Rev. B 91, 144107 (2015).
  * Goldschmidt (1926) V. M. Goldschmidt, Naturwissenschaften 14, 477 (1926).
  * Perdew  _et al._ (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * Methfessel  _et al._ (2000) M. Methfessel, M. van Schilfgaarde, and R. A. Casali, in  _Electronic Structure and Physical Properties of Solids. The Use of the LMTO Method_ , Lecture Notes in Physics, Vol. 535, edited by H. Dreyssé (Berlin Springer Verlag, 2000) p. 114.
  * Kotani and van Schilfgaarde (2010) T. Kotani and M. van Schilfgaarde, Phys. Rev. B 81, 125117 (2010).
  * Bott  _et al._ (1998) E. Bott, M. Methfessel, W. Krabs, and P. C. Schmidt, Journal of Mathematical Physics 39, 3393 (1998), http://dx.doi.org/10.1063/1.532437 .
  * Glazer (1972) A. M. Glazer, Acta Crystallographica Section B 28, 3384 (1972).
  * Woodward (1997) P. M. Woodward, Acta Cryst. B53, 44 (1997).
  * Walsh  _et al._ (2011) A. Walsh, D. J. Payne, R. G. Egdell, and G. W. Watson, Chem. Soc. Rev. 40, 4455 (2011).
  * Waghmare  _et al._ (2003) U. V. Waghmare, N. A. Spaldin, H. C. Kandpal, and R. Seshadri, Phys. Rev. B 67, 125111 (2003).
  * Fabini  _et al._ (2016) D. H. Fabini, G. Laurita, J. S. Bechtel, C. C. Stoumpos, H. A. Evans, A. G. Kontos, Y. S. Raptis, P. Falaras, A. Van der Ven, M. G. Kanatzidis, and R. Seshadri, Journal of the American Chemical Society 138, 11820 (2016), pMID: 27583813, http://dx.doi.org/10.1021/jacs.6b06287 .
  * Smith  _et al._ (2015) E. H. Smith, N. A. Benedek, and C. J. Fennie, Inorganic Chemistry 54, 8536 (2015), pMID: 26295352, https://doi.org/10.1021/acs.inorgchem.5b01213 .
  * Waghmare and Rabe (1997) U. V. Waghmare and K. M. Rabe, Phys. Rev. B 55, 6161 (1997).
  * Ghosez  _et al._ (1999) P. Ghosez, E. Cockayne, U. V. Waghmare, and K. M. Rabe, Phys. Rev. B 60, 836 (1999).
  * Shannon (1976) R. D. Shannon, Acta Cryst. A 32, 751 (1976).
  * Brehm  _et al._ (2014) J. A. Brehm, J. W. Bennett, M. R. Schoenberg, I. Grinberg, and A. M. Rappe, The Journal of Chemical Physics 140, 224703 (2014), http://dx.doi.org/10.1063/1.4879659 .
  * Travis  _et al._ (2016) W. Travis, E. N. K. Glover, H. Bronstein, D. O. Scanlon, and R. G. Palgrave, Chem. Sci. 7, 4548 (2016).
  * Zhong and Vanderbilt (1995) W. Zhong and D. Vanderbilt, Phys. Rev. Lett. 74, 2587 (1995).
  * (33) A more complete set of results as function of rotation and/or ferro-electric distortion is provided in the Supplementary Materials accompanying this paper.
  * Yamada  _et al._ (1991) K. Yamada, S. Funabiki, H. Horimoto, T. Matsui, T. Okuda, and S. Ichiba, Chemistry Letters 20, 801 (1991).
  * Thiele  _et al._ (1987) G. Thiele, H. W. Rotter, and K. D. Schmidt, Z. anorg. allg. Chem. 545, 148 (1987).
  * Dove (1997) M. T. Dove, American Mineralogist 82, 213 (1997).


Comparing facial emotion recognition methods R. El Cheikh et al. A comparative
study of emotion recognition methods using facial expressions Rim El Cheikh,
Hélène Tran
Issam Falih, Engelbert Mephu Nguifo Understanding the facial expressions of
our interlocutor is important to enrich the communication and to give it a
depth that goes beyond the explicitly expressed. In fact, studying one’s
facial expression gives insight into their hidden emotion state. However, even
as humans, and despite our empathy and familiarity with the human emotional
experience, we are only able to guess what the other might be feeling. In the
fields of artificial intelligence and computer vision, Facial Emotion
Recognition (FER) is a topic that is still in full growth mostly with the
advancement of deep learning approaches and the improvement of data
collection. The main purpose of this paper is to compare the performance of
three state-of-the-art networks, each having their own approach to improve on
FER tasks, on three FER datasets. The first and second sections respectively
describe the three datasets and the three studied network architectures
designed for an FER task. The experimental protocol, the results and their
interpretation are outlined in the remaining sections. Comprendre les
expressions faciales de notre interlocuteur est important pour enrichir la
communication et lui donner une profondeur qui va au-delà de l’explicitement
exprimé. En fait, étudier l’expression du visage donne un aperçu de l’état
émotionnel caché. Cependant, même en tant qu’êtres humains, et malgré notre
empathie et familiarité avec l’expérience émotionnelle humaine, nous ne
pouvons que deviner ce qu’une personne pourrait ressentir. Dans les domaines
de l’intelligence artificielle et de la vision par ordinateur, la
reconnaissance des émotions faciales (FER) est un sujet qui est encore en
pleine croissance, principalement avec l’avancement des approches
d’apprentissage en profondeur et l’amélioration de collecte de données.
L’objectif principal de cet article est de comparer les performances de trois
réseaux de l’état de l’art, chacun ayant sa propre approche pour améliorer les
tâches FER, sur trois jeux de données FER. Les première et deuxième sections
décrivent respectivement les trois jeux de données et les trois architectures
de réseaux étudiées pour effectuer une tâche FER. Le protocole expérimental,
les résultats et leur interprétation sont décrites dans les sections
restantes.

##  1 Introduction

Automatic Emotion Recognition is an area of research that has been active for
many decades. Researchers are not only trying to mimic human logical-
mathematical intelligence but have been for many years now trying to explore
emotional functions. However, despite many impressive results in terms of
predictive ability, automatic methods still have a long way to go when it
comes to offering a satisfying model that can capture the complexity,
heterogeneity, and subjectivity of the emotional experience.

The interest in this topic stems from different concerns. FER has applications
in various fields. For instance, given that understanding the emotional state
of our interlocutor is essential for a deep and rich communication,
recognizing emotions from facial expressions could be seen as a wish to
further develop the human-machine interaction. Also, having an automatic FER
that offers reliable results on complex emotional states could be a
breakthrough in the fields of healthcare (Kashif et al., 2021) and psychology
(Tadalagi and Joshi, 2021).

A key element for building a robust machine learning model is the availability
and quality of data. In the context of emotion recognition using images of
facial expressions, collecting a big amount of data is a manageable task as
proven by the multitude of available FER datasets. The samples in these
datasets are collected from different sources. For example, images of faces
could be collected from the internet and can be considered as expressions
captured "in-the-wild". The samples could be pre-processed to only keep the
facial region (Barsoum et al., 2016; Mollahosseini et al., 2017) or we can
choose to also incorporate and analyze the context in which the image is
captured (Kosti et al., 2017). Other datasets contain images taken in the lab,
where subjects are asked to act the facial expressions in a controlled
environment (Lyons et al., 1998; Lucey et al., 2010). Other differences
between datasets could come from the existence of noisy data, the lighting,
the encoding of the samples, the diversity of the faces in terms of age,
gender, race, ethnicity, etc. It is also important to insist of the
subjectivity in expressing and identifying emotions, which introduces bias and
ambiguity (Tran et al., 2022) in the annotation step, as well as during the
interpretation of the results step. In fact, one way to try to build a more
accurate and precise emotion recognition model is to combine different types
of data that could be informative on the emotional state of a person, such as
facial expression, speech, tone, body posture, physiological signal, etc. In
this situation, processing and combining data from different sources creates
an additional challenge to the design of a robust model.

As a first step to building such systems, we choose to focus this work on
methods that predict emotions from facial expression images only. Three state-
of-the-art models for FER tasks have been selected for experiments. These
models diverge not only in their architectures but also in the level on which
they intervene to better the prediction quality. Our experiments provide a
fair comparison of their performance on three datasets that differ in terms of
their size, the setting in which the samples are captured, and the
distribution of classes. The paper starts by giving an overview of three
existing facial expression datasets used for emotion prediction and that were
chosen for our experiments: FER+, AffectNet and CK+. Then, three neural
network architectures are described in section 2, followed by a presentation
of the experimental protocol in section 3. In sections 4 and 5, results are
respectively presented and analyzed. The code used for the experiments is
publicly available at https://anonymous.4open.science/r/FE-rec-0F0B/.

##  2 Dataset Description

This section describes the three datasets, FER+, AffectNet and CK+, used for
our experiments.

###  2.1 FER+

FER+, introduced by Barsoum et al. (2016), is one of the most known and used
datasets for FER tasks. FER+ is annotated with 8 emotions: Neutral, Happiness,
Surprise, Sadness, Anger, Disgust, Fear, and Contempt, as well as Unknown and
NF (Not a Face). The images are collected from the internet and contain only
the facial region. They are gray-scale and are of size 48x48 pixels, which
could be considered as very low resolution. To avoid affecting the quality of
predictions, we resized the images to 96x96 pixels.

FER+ can be considered as an improvement of the FER-2013 dataset, which was
introduced by Goodfellow et al. (2013). While the labeling for FER-2013 was
done by two persons only (the authors of the dataset), FER+ took advantage of
the increasingly popular scheme of crowdsourcing in order to collect ground
truth labels and the images of FER-2013 were relabeled by 10 crowdsourced
taggers. Each image has an emotion probability distribution by using the
annotations given by the 10 taggers. The FER+ dataset is therefore a multi-
labeled set of images that better reflects the ambiguity and diversity of
facial emotions. In our single-label classification experiments, the label
affected to a sample is the one with the highest number of votes. If the label
with the highest number of votes is Unknown or NF, we choose to exclude the
sample from the experiments.

###  2.2 AffectNet

The AffectNet dataset (Mollahosseini et al., 2017) provides 291,651 images of
faces annotated with 8 categories of emotions: Neutral, Happy, Sad, Surprise,
Fear, Disgust, Anger and Contempt. A larger version of AffectNet provides
annotations such as None, Uncertain, and No-Face that were not taken into
account in the implementation but are important to ensure the quality of the
annotated dataset. The images present in AffectNet were collected from the
internet by the means of queries such as "joyful girl", "astonished senior",
etc. Only the face region was kept and 12 annotators were asked to annotate
the samples. The authors provide the agreement percentages between two
annotators over the annotations. It is interesting to note that all classes
resulted in agreements ranging from 50% to 70%, except Happy with 79.6%. These
percentages are similar to the accuracy reportedly found by recognition models
in the FER literature that are trained on this dataset (Siqueira et al., 2020;
Farzaneh and Qi, 2021).

The images are in RGB and of size 224x224 pixels. In addition to the discrete
emotion categories, AffectNet gives a continuous annotation of the faces in a
two dimensional space, valence-arousal. The valence dimension is an indicator
for the pleasantness of the emotion and arousal is a measure of the emotion
intensity. These annotations were not used for our experiments, but provide
precious information that enables us to extend our research to emotion
recognition using continuous modeling.

###  2.3 CK+

The Extended Cohn-Kanade, or CK+, dataset (Lucey et al., 2010), contains 593
sequences of facial expressions captured from 123 subjects in a lab. A
sequence starts with a neutral expression and ends with the peak expression
where the facial Action Units (AUs) are coded. Action Units were proposed by
Friesen and Ekman (1978) as a way to model facial expressions by encoding the
movements of facial muscles. From these 593 sequences, only 327 are labeled
with a category of emotions: Anger, Contempt, Disgust, Fear, Happy, Sadness
and Surprise. The unlabeled sequences are considered as non fitting for the
prototypical definition of the emotions taken into account and are not used
for supervised training. The labeling is done by assigning an emotion to the
facial expression if one or more AUs are detected, with respect to the Facial
Action Coding System manual (Friesen and Ekman, 1978).

Given the small number of available sequences, we chose to take three images
from each sequence instead of only the peak expression. This way, the number
of samples is increased for each class of emotions. Moreover, the neutral
class is created by taking the first frame of each sequence, so that the set
of emotions used for CK+ is the same as the other two datasets. The images are
sized either 640x490 or 640x480 pixels. Some were gray-scale while others were
RGB. In our experiments, the images were fed to the networks as gray-scale
640x490 pixel arrays.

##  3 Studied networks

This section describes the methods chosen for experiments in order to compare
their performance on the previously presented datasets. The choice was based
on the fact that each method provides a different approach in terms of
improving on facial emotion recognition tasks. Availability of implementations
of these models was also an important factor in choosing them.

###  3.1 ESR

In the paper titled “Efficient Facial Feature Learning with Wide Ensemble-
based Convolutional Neural Networks”, by Siqueira et al. (2020) introduced the
Ensembles with Shared Representations (ESR) network. Ensemble learning
combines the predictions obtained by different classifiers that are usually
trained in an individual and independent manner, hence producing a more
accurate output than a single model would. The ESR network makes use of the
translation-invariance property of the patterns learned by each convolutional
layer. In fact, the patterns that are learned in the early layers of the
network (low-level features) can be considered in some way as common to all
the images that the network might encounter. These patterns can be oriented
lines, edges, or colors. As we go deeper into the layers, the patterns that
the network is learning become more complex and specific to each image. These
features, in the case of facial expressions, could be the shape of the eyes,
of the mouth, and of the nose. Therefore, in ESR we find two main building
pieces:

  1. 1.

The base of the network: A line-up of convolutional layers that are
responsible for learning the low-level features. As mentioned before, patterns
learned at this level of the network are very general and therefore can be
shared by multiple branches. This base uses a transfer learning mechanism that
speeds up the learning process as the ensemble grows, while improving the
performance, since the best configuration from the training of each branch is
reloaded as base.

  2. 2.

The independent convolutional branches: A branching of the convolutional
layers allows each branch to learn its own individual high-level features.

Finally, the optimization of the network consists in minimizing a loss
function that combines the loss function of each branch.

In the case of in-the-lab databases, the ESR network contains only 4
independent convolutional branches, as input data is expected to be of good
quality (good lighting, adjusted head pose, etc.). By contrast, when dealing
with in-the-wild datasets, the number of branches is increased to 9.

###  3.2 SCN

In the Self-Cure Network (Wang et al., 2020), the authors addressed the
problem of uncertainty that comes with the labeling of the facial expression
datasets. One of the reasons behind is the problem of subjectiveness when
categorizing the human emotional experience. Furthermore, in datasets where
images are captured in-the-wild, the uncontrolled setting is definitely a
source of inconsistency and uncertainty. To overcome this problem, the SCN
network proposes a relabeling step before the recognition task to perform
robust feature learning with uncertainty. First, the features are extracted
from the images using a “backbone” convolutional network, which can be any
traditional CNNs. In the implementation provided by the authors and as well as
in our experiments, the backbone used is ResNet-18 (He et al., 2016) pre-
trained on the MS-Celeb-1M face recognition dataset (Guo et al., 2016).
Second, if a sample is considered uncertain, it is relabeled. The relabeling
can be summarized in three steps:

  1. 1.

Self-attention importance weighting: Importance weights are computed for each
sample using a linear fully-connected layer with a sigmoid activation
function. The assigned importance weight reflects the contribution that each
image has on the classifier training.

  2. 2.

Ranking regularization: The weights are regularized in order to reduce the
importance given to uncertain samples. The samples are ranked by importance
and two groups of samples are created: the low-importance (30%) and the high-
importance images (70%).

  3. 3.

Relabeling: This module assigns a new emotion label (the one with highest
predicted probability produced by the model) to the images from the low-
importance group. For that purpose, softmax probabilities are used to
determine which image is actually incorrectly labeled. If the difference
between the predicted probability for the label initially given to the sample,
and the highest predicted probability produced by the model, is higher than a
given threshold (set to 0.2 by the authors), then the data is relabeled
according to the highest probability found by the model.

###  3.3 DACL

DACL, or Deep Attentive Center Loss (Farzaneh and Qi, 2021), is a facial
expression recognition method that uses an attention mechanism to estimate
attention weights that are correlated with feature importance. In fact, in the
training phase, learning irrelevant features is harmful for the performance of
the network. Therefore, the authors proposed the integration of a Deep Metric
Learning (DML) approach that enhances the learning of discriminative features
by the model.

The first step is to feed a convolutional neural network with the input images
in order to generate the feature maps, followed by the DACL component composed
of two building pieces:

  1. 1.

Context Encoder Unit: This generates latent representations for each spatial
feature map that is outputted by the backbone CNN. All these feature maps
represent the context, and therefore the obtained latent feature vector is
reduced in dimension and is devoid of the noise, only containing relevant
information from the initial features extracted by the CNN. The linear layer
weights were initialized according to the Kaiming initialization (He et al.,
2015), which is appropriate when using the ReLU activation function, as it
helps to capture non-linear relationships between layers.

  2. 2.

Multi-head binary classification module: This component estimates the
attention weights from the latent representation outputted by the Context
Encoder Unit. At this point, the problem is considered as a multi-label
classification problem, where an attention weight (softmax) is computed at
each component for the latent feature vector.

Finally, these attention weights are used to compute the sparse center loss,
which in turn is fractionally used, alongside the loss computed by the CNN
backbone to compute the final loss. The authors empirically gave 1% of the
final loss to the sparse center loss.

##  4 Experimental Setting

In order to fairly compare the performance of the three networks mentioned
above on the FER+, AffectNet and CK+ datasets, we followed an identical
experimental protocol for training and testing them. A 5-fold cross-validation
is used to evaluate the model. In fact, the three datasets are constructed in
different ways: FER+ has training, validation and testing subsets, AffectNet
has training and validation subsets, and CK+ has no subsets. Therefore, in the
case of FER+ and AffectNet the subsets are mixed before performing cross-
validation. The dataset is split as follows: 80% for training, 10% for
validation, and 10% for test. Table 1 shows the distribution of the eight
emotions for the three datasets. The training is performed over 60 epochs,
with Adam optimizer and a 0.001 learning rate.

FER+ AffectNet CK+ Train Valid Test Total Train Valid Test Total Train Valid
Test Total Neutral 8796 1148 1052 10996 3737 464 471 4672 262 28 37 327 Happy
7231 912 896 9039 6283 756 815 7854 166 16 25 207 Sad 3001 351 399 3751 1509
192 185 1886 67 9 8 84 Surprise 3153 378 410 3941 1043 147 113 1303 199 31 19
249 Fear 546 67 69 682 691 81 92 864 60 6 9 75 Disgust 126 15 17 158 587 72 75
734 141 23 13 177 Anger 2125 253 278 2656 1449 188 174 1811 108 14 13 135
Contempt 120 13 17 150 563 83 58 704 43 4 7 54 Total 25098 3137 3138 31373
15862 1983 1983 19828 1046 131 131 1308

Table 1: Class distribution in the three studied dataset.

###  4.1 Evaluation Metrics

This subsection presents the metrics used for the model evaluation, we define
TP as the value of true positives and FP is the false positives.

#### Accuracy and Balanced Accuracy

The overall accuracy is reported in our results. However, it does not take
into account the highly imbalanced label distribution found in the three
datasets. For this reason, we report the balanced accuracy defined as the
average of recall (see further in this section) obtained on each class. This
definition proposed by Mosley (2013) is equivalent to the most commonly used
formula for accuracy where each sample is weighted by the prevalence of its
true label. The chosen formula computes an aggregated score of the
measurements of the predictive quality for each class independently.

#### Precision and Recall

For our experiments, we compute precision score, which measures the
predictor’s ability not to label a negative sample as positive, weighted by
the support of the labels. Similarly, the weighted recall score is computed by
averaging the recall score for each label after weighting it by the support of
the label.

#### F1-score

We also report the F1-score as the harmonic mean of the weighted precision and
the weighted recall.

#### AUC ROC

The AUC ROC score reflects the discrimination ability of the classifier
between the different classes. In our experiments, we report the prevalence-
weighted average of AUC ROC scores computed for each class against all the
others.

##  5 Results

In this section we present the results of the five-fold cross-validation.
First, we evaluate and compare the models on the whole dataset in terms of
predictive performance. Second, the models are compared by taking into account
their capability to discriminate each emotion class.

###  5.1 Overall classification

Table 2 reports the computed metrics on the test subsets averaged over the 5
folds. This fairly quantifies the predictive performance of the models while
taking into account the imbalance in the datasets. When trained on FER+, DACL
shows a better accuracy than ESR and SCN. Weighted accuracy is decreased
compared to overall accuracy, which is expected given the imbalance in the
classes distribution in FER+. Also, DACL shows a better precision and recall
compared to the other models, as well as a higher ability to discriminate
between labels (AUC ROC). For AffectNet, DACL also shows the best performance
in terms of recall, precision and AUC ROC scores. However, the best balanced
accuracy is achieved by ESR while staying relatively close to the balanced
accuracy of DACL. Regarding CK+, ESR shows the best performance with a really
high accuracy of 91.5% and a very satisfying balanced accuracy compared to the
other models. The F1-score is also the highest for ESR on this dataset.
However, SCN produces the best AUC ROC score which shows that it is the best
model in terms of discrimination between the categories.

|  | acc | bal acc | pr | rec | F1 | AUC ROC
---|---|---|---|---|---|---|---
FER+ | ESR |  0.857 ± 0.013 |  0.617 ± 0.029 |  0.855 ± 0.015 |  0.857 ± 0.013 |  0.856 ± 0.014 |  0.937 ± 0.023
SCN |  0.810 ± 0.012 |  0.520 ± 0.028 |  0.808 ± 0.010 |  0.810 ± 0.012 |  0.809 ± 0.011 |  0.956 ± 0.002
DACL |  0.867 ± 0.005 |  0.647 ± 0.018 |  0.863 ± 0.005 |  0.867 ± 0.005 |  0.865 ± 0.005 |  0.973 ± 0.002
AffectNet | ESR |  0.648 ± 0.002 |  0.439 ± 0.001 |  0.626 ± 0.006 |  0.648 ± 0.002 |  0.637 ± 0.004 |  0.821 ± 0.002
SCN |  0.651 ± 0.002 |  0.390 ± 0.011 |  0.622 ± 0.005 |  0.651 ± 0.002 |  0.636 ± 0.003 |  0.894 ± 0.004
DACL |  0.664 ± 0.016 |  0.429 ± 0.033 |  0.633 ± 0.017 |  0.664 ± 0.016 |  0.648 ± 0.017 |  0.901 ± 0.007
CK+ | ESR |  0.915 ± 0.018 |  0.888 ± 0.034 |  0.922 ± 0.014 |  0.915 ± 0.018 |  0.918 ± 0.016 |  0.945 ± 0.008
SCN |  0.820 ± 0.030 |  0.703 ± 0.072 |  0.798 ± 0.043 |  0.820 ± 0.030 |  0.808 ± 0.032 |  0.962 ± 0.010
DACL |  0.846 ± 0.125 |  0.790 ± 0.175 |  0.843 ± 0.154 |  0.846 ± 0.125 |  0.844 ± 0.140 |  0.951 ± 0.051
Table 2: Average performance metrics computed on the test subsets afetr the
five-fold cross-validation. The value after (±) symbol represents the standard
deviation across five folds. "acc" is accuracy, "bal acc" is balanced
accuracy, "pr" is weighted precision, "rec" is weighted recall, "F1" is the
F1-score, "AUC ROC" is the Area under the ROC Curve.

###  5.2 Emotion-specific classification

In order to have a closer look on the performance of each model for each of
the 8 emotion categories, normalized confusion matrices are shown in figure 1.
When training on FER+, we find a higher TP value for the classes that are
over-represented. For example, a high percentage of images that are labeled
"Happy" in the dataset were correctly classified. However, not more than 1.2%
of "Contempt" images were predicted as such by the three models. We can
clearly see that for FER+ the imbalance in the training sets impacts the
learning process by the models. Also, "Contempt" expressions are more often
than not misclassified as "Neutral" by all three of the models. When training
on AffectNet, we can see that "Neutral" and "Happy" have a high classification
accuracy for the three models. This is not the case for all other classes,
which have a high probability of being misclassified as "Neutral" (and as
"Happy" for "Contempt" samples). As the distribution classes in AffectNet
shows, "Happy" has a higher frequency than "Neutral" in AffectNet, this was
translated by a higher positive values for "Happy" than for "Neutral".
Finally, when training on CK+, ESR predicts correct labels for over 80% of the
samples for all classes except for "Contempt" which is accurately predicted in
77.1% of cases. However, this is not the case for SCN and DACL, where a high
accuracy is only found for classes that are over-represented in CK+ (Happy,
Neutral and Surprise). This said, all three models trained on CK+ still manage
to provide a better accuracy on almost all under-represented classes compared
to the other two datasets.

(a) Confusion matrices of ESR predictions on FER+(left), AffectNet(middle) and
CK+(right)

(b) Confusion matrices of SCN predictions on FER+(left), AffectNet(middle) and
CK+(right)

(c) Confusion matrices of DACL predictions on FER+(left), AffectNet(middle)
and CK+(right)

Figure 1: Normalized confusion matrices, each computed by averaging the
confusion matrices obtained over the 5 folds.

##  6 Discussion

DACL provides the best scores when training on FER+ and AffectNet. Both these
datasets contain facial expressions captured in-the-wild, where the subjects
expressed their emotions in a natural way. This is why samples from FER+ and
AffectNet might require a representation that takes into account the
characteristics of the whole face region. It could be said that the Context
Encoder unit of DACL is helping to represent the overall context from the face
region of these samples and the multi-head attention mechanism allow the
network to retain the most important information that are useful to infer the
emotion. ESR gives the best accuracy and F1-score for CK+. Therefore, we can
say that the exploitation of shared representations that is characteristic of
ESR, proves effective when the samples are taken in a controlled environment
and when the expressions are posed and intentional.

In figure 1, it is noteworthy that training on FER+ induces a lot of mistakes
by predicting "neutral" for samples that are labeled "sad" (using ESR, 59.4%
of "sad" are correctly predicted and 31.9% are predicted as "neutral"). This
is not the case for samples that are labeled "surprise" (using ESR, 87.4% of
"surprise" are correctly classified), despite the fact that both classes,
"sad" and "surprise" are represented in the dataset in very close proportions
(3751 and 3941 samples respectively). This raises the question of what is it
about "sad" expressions that are less discernible than "surprise" expressions.
However, both ESR and DACL manage to correctly classify 31.9% of the "disgust"
samples, which is an under-represented class. This is not the case for SCN. In
fact, we can see that SCN’s characteristic of dealing with uncertainty in the
annotations of the training set does not translate well on the predictions
done on a test set. As for training on AffectNet, the classification of a
higher number of samples from other classes into "neutral" than into "happy"
is proof that this misclassification comes from the ambiguity of emotions and
not necessarily from the distribution in the dataset. We can say that
"neutral" is a safe choice to categorize an emotion when we are somewhat
uncertain. Similar to AffectNet, we find that when SCN and DACL are trained on
CK+, a high number of misclassified samples are in fact predicted as
"neutral". This shows that even a controlled manner of capturing emotions
induces the mistake of safely labelling an emotion as "neutral".

In our experiments, we chose to compare the performance of three different
neural networks, ESR, SCN and DACL. They all have different architectures and
different approaches to deal with FER tasks. However, they cannot represent
the whole class of methods to which they belong (attention-based, dealing with
uncertainty, or ensemble learning). Other architectures for each method class
(Li et al., 2020; Hao et al., 2020) could be studied in order to have a more
comprehensive view of their performance in the context of FER applications.
The same could be said about datasets, where we can extend our experiments to
other datasets (Lyons et al., 1998; Li et al., 2017) that share
characteristics with the ones we discussed in this paper.

##  7 Conclusion

This paper compares the performance of three neural networks that have
different approaches to tackle FER challenges: the first uses ensemble
learning and transfer learning to learn facial emotion patterns, the second
addresses the problem of subjectivity and uncertainty when labelling emotions,
and the third makes use of an attention mechanism to learn from relevant
features. To evaluate these models, we used three datasets that are different
in terms of data collection and capture setting. FER+ and AffectNet are both
in-the-wild datasets with facial expressions captured in an uncontrolled
setting. The model that uses an attention mechanism provides the best results
on images that are captured in the wild, which was expected as this type of
images is very noisy and it would have been difficult to recognize the emotion
if the model was not guided in focusing on the relevant parts of the images.
On CK+, containing images with posed expressions, the model based on ensemble
and transfer learning is the one that performs the best: the network is build
to exploit low-level features, which is suitable in a setting where noisy
parameters are mitigated (face in front of the camera with a neutral
background). Moreover, our experiments show that models often mistake some
emotion classes (e.g. "contempt" and "neutral") in in-the-wild datasets,
showing that emotion ambiguity alters the model’s discrimination abilities. In
our comparative study, many challenges were identified, such as the under-
represented emotion classes found in many FER datasets and the ambiguity of
facial expressions. Also, extending the experiments to more models and
datasets would provide a reliable benchmark to choose an adapted FER model in
terms of the application at hand.

## References

  * Barsoum et al. (2016) Barsoum, E., C. Zhang, C. C. Ferrer, and Z. Zhang (2016).  Training deep networks for facial expression recognition with crowd-sourced label distribution.  In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 279–283.
  * Farzaneh and Qi (2021) Farzaneh, A. H. and X. Qi (2021).  Facial expression recognition in the wild via deep attentive center loss.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2402–2411.
  * Friesen and Ekman (1978) Friesen, E. and P. Ekman (1978).  Facial action coding system: a technique for the measurement of facial movement.  Palo Alto 3(2), 5.
  * Goodfellow et al. (2013) Goodfellow, I. J., D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, et al. (2013).  Challenges in representation learning: A report on three machine learning contests.  In International conference on neural information processing, pp. 117–124. Springer.
  * Guo et al. (2016) Guo, Y., L. Zhang, Y. Hu, X. He, and J. Gao (2016).  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In European conference on computer vision, pp. 87–102. Springer.
  * Hao et al. (2020) Hao, M., W.-H. Cao, Z.-T. Liu, M. Wu, and P. Xiao (2020).  Visual-audio emotion recognition based on multi-task and ensemble learning with multiple features.  Neurocomputing 391, 42–51.
  * He et al. (2015) He, K., X. Zhang, S. Ren, and J. Sun (2015).  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pp. 1026–1034.
  * He et al. (2016) He, K., X. Zhang, S. Ren, and J. Sun (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Kashif et al. (2021) Kashif, M., A. Hussain, A. Munir, A. B. Siddiqui, A. Abbasi, M. Aakif, A. J. Malik, F. E. Alazemi, and O.-Y. Song (2021).  A machine learning approach for expression detection in healthcare monitoring systems.  Computers, Materials & Continua 67(2), 2123–2139.
  * Kosti et al. (2017) Kosti, R., J. M. Alvarez, A. Recasens, and A. Lapedriza (2017).  Emotion recognition in context.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1667–1675.
  * Li et al. (2020) Li, J., K. Jin, D. Zhou, N. Kubota, and Z. Ju (2020).  Attention mechanism-based cnn for facial expression recognition.  Neurocomputing 411, 340–350.
  * Li et al. (2017) Li, S., W. Deng, and J. Du (2017).  Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Lucey et al. (2010) Lucey, P., J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews (2010).  The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression.  In 2010 ieee computer society conference on computer vision and pattern recognition-workshops, pp. 94–101. IEEE.
  * Lyons et al. (1998) Lyons, M., M. Kamachi, and J. Gyoba (1998).  The Japanese Female Facial Expression (JAFFE) Dataset.
  * Mollahosseini et al. (2017) Mollahosseini, A., B. Hasani, and M. H. Mahoor (2017).  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing 10(1), 18–31.
  * Mosley (2013) Mosley, L. (2013).  A balanced approach to the multi-class imbalance problem.  Doctor of Philosophy Thesis, Iowa State University of Science and Technology, USA.
  * Siqueira et al. (2020) Siqueira, H., S. Magg, and S. Wermter (2020).  Efficient facial feature learning with wide ensemble-based convolutional neural networks.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), pp. 1–1.
  * Tadalagi and Joshi (2021) Tadalagi, M. and A. M. Joshi (2021).  Autodep: automatic depression detection using facial expressions based on linear binary pattern descriptor.  Medical & biological engineering & computing 59(6), 1339–1354.
  * Tran et al. (2022) Tran, H., L. Brelet, I. Falih, X. Goblet, and E. M. Nguifo (2022).  L’ambiguïté dans la représentation des émotions: état de l’art des bases de données multimodales.  In Conférence Extraction et Gestion de Connaissances.
  * Wang et al. (2020) Wang, K., X. Peng, J. Yang, S. Lu, and Y. Qiao (2020).  Suppressing uncertainties for large-scale facial expression recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6897–6906.