20XX Vol. X No. XX, 000–000

11institutetext: Key Laboratory of Dark Matter and Space Astronomy, Purple
Mountain Observatory, Chinese Academy of Sciences, Nanjing 210023, P. R.
China<EMAIL_ADDRESS>
22institutetext: School of Astronomy and Space Science, University of Science
and Technology of China, Hefei 230026, P. R. China

# Injection spectra of different species of cosmic rays from AMS-02, ACE-CRIS
and Voyager-1

Xu Pan 1122 Qiang Yuan 1122

###### Abstract

Precise measurements of energy spectra of different cosmic ray species were
obtained in recent years, by particularly the AMS-02 experiment on the
International Space Station. It has been shown that apparent differences exist
in different groups of the primary cosmic rays. However, it is not
straightforward to conclude that the source spectra of different particle
groups are different since they will experience different propagation
processes (e.g., energy losses and fragmentations) either. In this work, we
study the injection spectra of different nuclear species using the
measurements from Voyager-1 outside the solar system, and ACR-CRIS and AMS-02
on top of atmosphere, in a physical framework of cosmic ray transportation.
Two types of injection spectra are assumed, the broken power-law and the non-
parametric spline interpolation form. The non-parametric form fits the data
better than the broken power-law form, implying that potential structures
beyond the constrained spectral shape of broken power-law may exist. For
different nuclei the injection spectra are overall similar in shape but do
show some differences among each other. For the non-parametric spectral form,
the helium injection spectrum is the softest at low energies and the hardest
at high energies. For both spectral shapes, the low-energy injection spectrum
of neon is the hardest among all these species, and the carbon and oxygen
spectra have more prominent bumps in $1-10$ GV in the $R^{2}dN/dR$
presentation. Such differences suggest the existence of differences in the
sources or acceleration processes of various nuclei of cosmic rays.

###### keywords:

cosmic rays, astroparticle physics

##  1 Introduction

Although the exact origin of cosmic rays (CRs) is not clear yet, it is
generally believed that CRs with energies below PeV originate from supernova
remnants. Energetic CRs were accelerated by diffusive shocks and then injected
into the interstellar space. Theoretically the accelerated spectrum can be
simply described by a power-law form $dN/dR\propto R^{-n}$, with $R$ being the
particle rigidity and $n$ being the index (Fermi, 1949; Bell, 1978; Blandford
& Ostriker, 1978; Bell, 2014). Extension of the conventional diffusive shock
acceleration mechanism with test particle assumption to consider the
interaction between accelerated particles and the surrounding fluid results in
nonlinear effects and deviation from the simple power-law spectrum (Malkov &
Drury, 2001; Bell, 2004; Caprioli et al., 2010). From the observational point
of view, complicated spectral structures of CRs were also revealed by many
measurements (Panov et al., 2009; Ahn et al., 2010; Adriani et al., 2011;
Aguilar et al., 2015b, a, 2017; Atkin et al., 2018; An et al., 2019; Adriani
et al., 2019, 2020; Alemanno et al., 2021, 2022). Particularly, apart from the
breaks around a few GV, remarkable hardenings around hundreds of GV and
subsequent softenings around 10 TV were shown by the data. The spectra also
differs among different nuclei. The helium spectrum is found to be clearly
harder than that of protons (Adriani et al., 2011; Aguilar et al., 2015b, a).
The AMS-02 measurements further showed that the high-energy spectra of neon
(Ne), magnesium (Mg), and silicon (Si) are different from those of helium
(He), carbon (C), and oxygen (O), and suggested that different types of
primary sources exist (Aguilar et al., 2020). These results may indicate that
the origin and acceleration of CRs are more complicated.

It should be noted that after the acceleration, CR particles are injected into
the interstellar space, and experience complex propagation processes. The
energy losses and fragmentation cross sections of various nuclei differ from
each other, making the propagated spectra become diverse even for the same
injection spectra. Therefore, the apparent differences of the spectra are not
directly reflecting the differences at injection. To properly address this
issue needs a thorough consideration of the CR propagation (Yuan et al., 2017;
Boschini et al., 2018; Derome et al., 2019; Boschini et al., 2020a, b; Yuan,
2019; Wu & Chen, 2019; Niu, 2022; Korsmeier & Cuoco, 2022).

Here we investigate the source injection spectra of different primary nuclei
including He, C, O, Ne, Mg, Si, and Fe, based mainly on the AMS-02 data
(Aguilar et al., 2017, 2020, 2021a, 2021b). At low energies the fluxes will be
suppressed due to the solar modulation effect. We use the force-field
approximation to account for the solar modulation (Gleeson & Axford, 1967). To
break the degeneracy between the injection and the solar modulation effects,
the measurements at low energies outside the solar system by Voyager-1 will
also be included (Cummings et al., 2016). We further use the ACE-CRIS
111http://www.srl.caltech.edu/ACE/ASC/level2/lvl2DATA_CRIS.html measurements
at the same time periods of the AMS-02 to better constrain the low-energy
spectra. The GALPROP code is employed to calculate the propagation of CRs
(Strong & Moskalenko, 1998; Moskalenko & Strong, 1998). The Markov Chain Monte
Carlo (MCMC) method is used to do the fit (Liu et al., 2012).

Compared with previous works along the line of studying the injection spectra
of CRs (Yuan, 2019; Korsmeier & Cuoco, 2022; Niu, 2022), this work differs in
either more species of nuclei used (e.g., Ne, Mg, Si, Fe) or the low-energy
ACE and Voyager data included which better constrain the wide-band spectral
shape.

##  2 Cosmic ray injection and propagation

Given more and more complicated structures of the CR spectra were revealed by
recent precise measurements, it is expected that simple empirical functions
may not be proper enough to describe the injection spectra of CRs in a wide
energy range. In this work, we use a non-parametric interpolation (NPI)
spectrum determined by a cubic spline interpolation method (Ghelfi et al.,
2016; Zhu et al., 2018), which has more freedom to reveal multiple structures
of the spectra. The interpolation is done in the $\log(R)-\log(J)$ parameter
space, where $R=pc/Ze$ is the particle rigidity in unit of MV and $J$ is the
flux. Specifically, we set the following rigidity knots in the analysis:

$\left\\{\log(R_{1}),...,\log(R_{7})\right\\}=\left\\{2.50,3.11,3.72,4.29,4.91,5.49,6.10\right\\}.$
(1)

The corresponding fluxes at these rigidity knots, $\log(J_{i})$, are fitted as
free parameters.

To capture the main features of the spectra, we also consider a broken power-
law (BPL) form of injection spectrum for comparison

$q(R)\propto\begin{cases}\left(R/R_{\mathrm{br0}}\right)^{-\gamma_{0}}\left(R_{\mathrm{br0}}/R_{\mathrm{br1}}\right)^{-\gamma_{1}},&R\leq
R_{\mathrm{br0}}\\\
\left(R/R_{\mathrm{br1}}\right)^{-\gamma_{1}},&R_{\mathrm{br0}}<R\leq
R_{\mathrm{br1}},\\\ \left(R/R_{\mathrm{br1}}\right)^{-\gamma_{2}},&R\geq
R_{\mathrm{br1}}\end{cases}$ (2)

where $\gamma_{0}$, $\gamma_{1}$, and $\gamma_{2}$ are spectral indices in
different rigidity ranges, $R_{\mathrm{br0}}$ and $R_{\mathrm{br1}}$ are break
rigidities.

Following the distribution of supernova remnants, the source distribution of
CRs is parameterized as

$f(r,z)=\left(\frac{r}{r_{\odot}}\right)^{\alpha}\exp\left[-\frac{\beta\left(r-r_{\odot}\right)}{r_{\odot}}\right]\exp\left(-\frac{|z|}{z_{s}}\right),$
(3)

where $r_{\odot}=8.5$ kpc, $z_{s}=0.2$ kpc, $\alpha=1.25$, $\beta=3.56$
(Trotta et al., 2011).

The propagation of nuclei in the Milky Way includes mainly the diffusion in
the random magnetic field, the energy losses due to ionization and Coulomb
collisions, the fragmentation due to inelastic collisions with the
interstellar medium (ISM), and possible convective transportation and
reacceleration (Ginzburg & Syrovatskii, 1964; Strong et al., 2007). The
propagation can be described by a set of differential equations for all
species of nuclei, which self-consistently predict the fluxes of both primary
and secondary nuclei. The general propagation equations can not be solved
analytically, and numerical solutions were developed and widely employed
(Strong & Moskalenko, 1998; Moskalenko & Strong, 1998).

The propagation parameters we adopt are determined through fitting to the
newest measurements of secondary and primary CRs (Yuan et al., 2020). We work
in the diffusion-reacceleration framework, and the convection velocity is set
to be 0. The main parameters include: the spatial diffusion coefficient
$D_{xx}=D_{0}\beta^{\eta}(R/4~{}{\rm GV})^{\delta}$, with $D_{0}=7.69\times
10^{28}$ cm2 s-1, $\eta=-0.05$ which phenomenologically describes the possible
resonant interactions of CRs with the magnetohydrodynamic (MHD) waves (Ptuskin
et al., 2006), $\delta=0.362$, the Alfvenic velocity $v_{A}=33.76$ km s-1
which characterizes the reacceleration of particles during the propagation,
and the half height of the propagation halo $z_{h}=6.27$ kpc.

After entering the solar system, CRs would be further affected by the magnetic
field carried by the solar wind, and experience flux suppression at low
energies (below tens GV). This so-called solar modulation results in an anti-
correlation of the low-energy CR fluxes with solar activities. Although more
sophisticated modulation models were developed (e.g., Potgieter (2013)), the
simple force-field approximation (Gleeson & Axford, 1967) is employed in this
work. Since the particles discussed here are all positively charged with mass-
to-charge ratio $A/Z\approx 2$, we expect that their relatively differences
are less sensitive to the solar modulation model.

##  3 Analysis method

In this work we focus on the primary CR nuclei with $A/Z\approx 2$, including
He, C, O, Ne, Mg, Si, and Fe. The proton spectrum which shows clear difference
from that of He is not discussed (Zhang et al., 2017). The CosRayMC code (Liu
et al., 2012) which combines the CR propagation and the MCMC sampler is used.
According to the Bayes’ theorem, the posterior probability of a model
described by parameters $\boldsymbol{\theta}$ can be obtained as

$P(\boldsymbol{\theta}|D)\propto{\mathcal{L}}(\boldsymbol{\theta})P(\boldsymbol{\theta}),$
(4)

where $D$ denotes the data,
${\mathcal{L}}(\boldsymbol{\theta})=P(D|\boldsymbol{\theta})$ is the
likelihood of the model for data $D$, and $P(\boldsymbol{\theta})$ is the
prior probability of the model. The likelihood function can be calculated as

${\mathcal{L}}(\boldsymbol{\theta})\propto\exp\left(-\frac{1}{2}{\sum_{i=1}^{n}\frac{[J(E_{i};\boldsymbol{\theta})-J_{i}]}{\sigma_{i}^{2}}}\right),$
(5)

where $J(E_{i};\boldsymbol{\theta})$ is the model predicted flux, $J_{i}$ and
$\sigma_{i}$ are the observational flux and error of the $i$-th energy bin.

The AMS-02 and Voyager-1 data can be directly obtained from the publications
(Aguilar et al., 2021b; Cummings et al., 2016). The total uncertainties used
are the quadratic sum of the statistical ones and systematic ones. For the
ACE-CRIS data, we extract them from the online data server. The systematic
uncertainties of ACE-CRIS data include the geometry factor (2%), the
scintillating optical fiber trajectory efficiency (2%), and the correction of
spallation in the instrument ($1\%\sim 5\%$ depending on the charge and energy
bin) (George et al., 2009). For He nuclei, no ACE-CRIS data are available. For
Fe nuclei, the ACE-CRIS data are not included in the likelihood calculation
due to the possible excess compared with the AMS-02 data (see the discussion
in Boschini et al. (2021)).

##  4 Results

Figure 1: The injection spectra of different nuclei. In the top panels we
normalize all spectra at 10 GV, and in the bottom panels they are shown for 4
different groups. The left panels are for the BPL form, and the right panels
are for the NPI form. Table 1: The parameters of BPL form of injection
spectra.

| He | C | O | Ne | Mg | Si | Fe
---|---|---|---|---|---|---|---
$\gamma_{0}$ | $1.27^{+0.02}_{-0.02}$ | $0.50^{+0.09}_{-0.10}$ | $0.82^{+0.07}_{-0.08}$ | $0.30^{+0.14}_{-0.20}$ | $0.62^{+0.12}_{-0.16}$ | $0.56^{+0.11}_{-0.11}$ | $1.37^{+0.11}_{-0.12}$
$\gamma_{1}$ | $2.41^{+0.01}_{-0.01}$ | $2.43^{+0.00}_{-0.00}$ | $2.41^{+0.00}_{-0.00}$ | $2.44^{+0.01}_{-0.01}$ | $2.45^{+0.01}_{-0.01}$ | $2.41^{+0.01}_{-0.01}$ | $2.41^{+0.01}_{-0.01}$
$\gamma_{2}$ | $2.25^{+0.02}_{-0.03}$ | $2.21^{+0.04}_{-0.06}$ | $2.14^{+0.09}_{-0.08}$ | $2.14^{+0.11}_{-0.09}$ | $2.13^{+0.12}_{-0.09}$ | $2.27^{+0.01}_{-0.01}$ | $2.46^{+0.24}_{-0.16}$
$R_{\rm br0}$ (GV) | $2.78^{+0.07}_{-0.07}$ | $1.63^{+0.05}_{-0.06}$ | $1.71^{+0.06}_{-0.06}$ | $1.94^{+0.40}_{-0.39}$ | $1.93^{+0.08}_{-0.08}$ | $2.03^{+0.06}_{-0.04}$ | $3.29^{+0.39}_{-0.29}$
$R_{\rm br1}$ (GV) | $306.30^{+36.22}_{-34.78}$ | $389.72^{+73.95}_{-59.53}$ | $692.38^{+140.72}_{-154.58}$ | $669.79^{+159.01}_{-143.47}$ | $667.32^{+162.19}_{-152.56}$ | $791.60^{+138.20}_{-207.51}$ | $705.66^{+199.56}_{-314.79}$
$\phi$ | $0.70^{+0.01}_{-0.01}$ | $0.71^{+0.01}_{-0.01}$ | $0.67^{+0.01}_{-0.01}$ | $0.76^{+0.02}_{-0.02}$ | $0.70^{+0.01}_{-0.01}$ | $0.72^{+0.02}_{-0.02}$ | $1.00^{+0.04}_{-0.04}$
$\chi^{2}/\mathrm{d.o.f.}$ | 91.3/77 | 60.0/80 | 54.3/81 | 81.2/76 | 80.3/76 | 69.6/76 | 24.1/48

Table 2: The parameters of NPI form of injection spectra.

| He | C | O | Ne | Mg | Si | Fe
---|---|---|---|---|---|---|---
$\log(J_{1})$ | $-2.65\pm 0.02$ | $-3.11\pm 0.05$ | $-3.01\pm 0.05$ | $-3.02\pm 0.10$ | $-2.99\pm 0.10$ | $-2.86\pm 0.10$ | $-2.89\pm 0.12$
$\log(J_{2})$1 | $-3.5$ | $-3.5$ | $-3.5$ | $-3.5$ | $-3.5$ | $-3.5$ | $-3.5$
$\log(J_{3})$ | $-4.80\pm 0.01$ | $-4.78\pm 0.01$ | $-4.77\pm 0.01$ | $-4.61\pm 0.02$ | $-4.70\pm 0.01$ | $-4.72\pm 0.02$ | $-4.69\pm 0.02$
$\log(J_{4})$ | $-6.14\pm 0.01$ | $-6.17\pm 0.01$ | $-6.16\pm 0.01$ | $-5.99\pm 0.02$ | $-6.09\pm 0.02$ | $-6.08\pm 0.02$ | $-6.04\pm 0.02$
$\log(J_{5})$ | $-7.56\pm 0.01$ | $-7.62\pm 0.02$ | $-7.62\pm 0.01$ | $-7.44\pm 0.02$ | $-7.57\pm 0.02$ | $-7.54\pm 0.02$ | $-7.46\pm 0.02$
$\log(J_{6})$ | $-8.97\pm 0.01$ | $-9.06\pm 0.02$ | $-9.07\pm 0.01$ | $-8.92\pm 0.02$ | $-9.05\pm 0.02$ | $-9.02\pm 0.02$ | $-8.93\pm 0.02$
$\log(J_{7})$ | $-10.35\pm 0.02$ | $-10.39\pm 0.02$ | $-10.44\pm 0.01$ | $-10.25\pm 0.03$ | $-10.43\pm 0.03$ | $-10.43\pm 0.03$ | $-10.36\pm 0.05$
$\phi$ | $0.56\pm 0.01$ | $0.60\pm 0.01$ | $0.56\pm 0.02$ | $0.51\pm 0.02$ | $0.50\pm 0.02$ | $0.50\pm 0.02$ | $0.69\pm 0.05$
$\chi^{2}/\mathrm{d.o.f.}$ | 45.2/75 | 24.7/78 | 14.2/79 | 20.0/74 | 24.2/74 | 29.5/74 | 13.3/46

  * 1

The $\log(J_{2})$ of all nuclei is normalized to the same constant value.

The best-fit parameters and the $1\sigma$ uncertainties for the BPL and NPI
injection spectra are presented in Tables 1 and 2. For all species, the NPI
form show smaller reduced $\chi^{2}$ values compared with those of the BPL
form. Figure 1 displays the injection spectra of various nuclei (the $1\sigma$
bands) obtained from the fitting. Top panels are for the injection spectra
normalized at 10 GV, and bottom panels display the spectra which are grouped
into 4 groups, He, C-O, Ne-Mg-Si, and Fe, respectively.These injection spectra
show a general similarity among each other. Specifically, the injection
spectra for all nuclei experience softenings around several GV rigidities and
hardenings around a few hundred GV. For the BPL form, our results of C, O, Ne,
and Mg are consistent with those given in Niu (2022), despite the
methodologies are different. The relative spectral shapes among different
nuclei are different for the BPL and NPI forms. Since the NPI form introduces
less constraints on the injection spectra, and the fittings are much better
than the BPL form, we take the results from the NPI fitting as benchmark. The
helium spectrum is the softest at low energies and the hardest at high
energies. The Ne spectrum is the hardest in the low-energy range among all
species. For C and O, their injection spectra show prominent bumps in the
$1-10$ GV range compared with other nuclei. The Fe spectrum is similar to that
of Si at low energies, but is slightly harder above 10 GV.

Thanks to the observations by Voyager-1 outside the solar system and the low-
energy fluxes by ACE-CRIS, the degeneracy between the injection spectrum and
the solar modulation can be effectively broken. Fig. 2 shows the probability
distributions of the solar modulation potentials $\phi$ for different nuclei.
For both the BPL and NPI forms, Fe has the largest $\phi$ value. The remaining
nuclei exhibit similar modulation potentials within $\sim 2\sigma$
uncertainties. The difference of modulation potentials between Fe and the
other nuclei may be due to the low-energy structures of the Fe spectrum as
revealed by combining the ACE-CRIS and AMS-02 data (see the discussion below).

Figure 2: The one-dimensional probability density distributions of the solar
modulation potentials, for the BPL form (left) and NPI form (right).

Fig. 3 shows the comparisons between the best-fit spectra and the
measurements. The higher curve in each panel represents the local interstellar
spectrum (LIS) before the solar modulation, and the lower one shows the
spectrum at the top of atmosphere (TOA) of the Earth. Good consistency between
the fitting results and the data can be seen. We also show that the ACE-CRIS
measurement for the fluxes of Fe nuclei can not connect smoothly with the
AMS-02 data, as already being pointed out in Boschini et al. (2021). The
combined AMS-02 and ACE-CRIS data may indicate a bump structure at $\sim 2$
GV, which may be due to a past supernova explosion in the local bubble
(Boschini et al., 2021).

Figure 3: Comparison of the best-fit results of the spectra with the
measurements (Cummings et al., 2016; Aguilar et al., 2021b). In each panel the
higher line is the LIS and the lower one is the TOA spectrum. The residuals
are depicted in the lower sub-panel, defined as $\chi=({\rm data}-{\rm
model})/{\rm error}$ (stars are for BPL form and circles are for the NPI
form).

##  5 Conclusion and discussion

New measurements of the energy spectra of CRs with precent-level precision
enable us to investigate crucially the acceleration and propagation processes
of particles. The measured spectra contain mixed effects of the acceleration
and propagation, and thus cannot be directly used to infer the injection
spectra of different CR particles. In this work, we thus derive the source
injection spectra of a series of primary nuclei under the framework of a
physical propagation model. Our results show that, even these nuclei have
similar $A/Z\approx 2$, their injection spectra show diverse behaviors. As a
test, we assume identical injection spectra for all these nuclei using the NPI
form, and find a reduced chi-squared value of $\chi^{2}$/d.o.f. $=9424.3/586$.
If we choose two injection spectra, one is applied to He, C, and O nuclei, and
the other is applied to Ne, Mg, and Si nuclei, we get $\chi^{2}$/d.o.f.
$=650.3/275$ for the fitting to He, C, and O, and $\chi^{2}$/d.o.f.
$=223.5/265$ for the fitting to Ne, Mg, and Si. The fitting to C and O gives
$\chi^{2}$/d.o.f $=71.2/165$, indicating that the injection spectrum of C and
O should be different from that of He. When we add Fe to Ne-Mg-Si group, we
get $\chi^{2}$/d.o.f. $=672.0/319$. The injection spectrum of Fe appears to be
similar to those of Mg and Si at low energies, but is harder at high energies.
The combined fitting of Fe and He-C-O gives $\chi^{2}$/d.o.f. $=1164.8/339$.
These tests show that we can perhaps classify the injection spectra into four
groups, He, C-O, Ne-Mg-Si, and Fe as shown in Fig. 1. Assuming the same
injection spectra for different groups results in poor fittings to the data,
indicating the intrinsic difference of their injection spectra.

The diversity of the derived injection spectra may be related with the
acceleration processes. Various acceleration models were proposed to explain
the spectral differences of protons and helium nuclei. For example, it was
proposed that the reverse shock acceleration of different supernova shocks
(e.g., Type I where hydrogen is absent, and Type II where hydrogen is
abundant) could explain the harder spectrum of helium nuclei (Ptuskin et al.,
2013). Ohira & Ioka (2011) proposed that the acceleration in chemically
enriched regions with outward-decreasing abundance could naturally result in
different spectra of different species. Those models may be extended to
account for the differences of injection spectra of heavy nuclei as found in
this work. In addition, models including different ionization histories of
nuclei (Casse & Goret, 1978) and condensation of different elements into
grains (Ellison et al., 1997) could also explain the diversity of the inferred
injection spectra.

Note that we have assumed a single source population in derive the injection
spectra. The results may reflect the fact that there are multiple source
components of CRs. For example, it has been proposed that a nearby source with
element abundance different from that of the average background sources may
result in different spectral shapes of various nuclei (Yuan et al., 2021).

Finally, we assume a spatially uniform propagation in this work. However, a
number of new observations may suggest a spatially-dependent propagation model
of CRs (Tomassetti, 2012; Liu et al., 2018; Guo & Yuan, 2018; Zhao et al.,
2021). Due to the differences of the energy loss rates and fragmentation cross
sections of different nuclei, they experience different propagation lengths in
the Milky Way. In the spatially-dependent propagation model, such an effect
results in additional spectral differences on the results based on homogeneous
propagation assumption. Whether or not the observed spectral differences can
be reproduced in a realistic spatially-dependent propagation model needs
future studies.

###### Acknowledgements.

We acknowledge the use of the ACE-CRIS data provided by the ACE Science
Center. This work is supported by the National Key Research and Development
Program of China (No. 2021YFA0718404), the National Natural Science Foundation
of China (No. 12220101003) and the Project for Young Scientists in Basic
Research of Chinese Academy of Sciences (No. YSBR-061).

## References

  * Adriani et al. (2011) Adriani, O., Barbarino, G. C., Bazilevskaya, G. A., et al. 2011, Science, 332, 69
  * Adriani et al. (2019) Adriani, O., Akaike, Y., Asano, K., et al. 2019, Phys. Rev. Lett., 122, 181102
  * Adriani et al. (2020) Adriani, O., Akaike, Y., Asano, K., et al. 2020, Phys. Rev. Lett., 125, 251102
  * Aguilar et al. (2015a) Aguilar, M., Aisa, D., Alpat, B., et al. 2015a, Phys. Rev. Lett., 115, 211101
  * Aguilar et al. (2015b) Aguilar, M., Aisa, D., Alpat, B., et al. 2015b, Phys. Rev. Lett., 114, 171103
  * Aguilar et al. (2017) Aguilar, M., Ali Cavasonza, L., Alpat, B., et al. 2017, Phys. Rev. Lett., 119, 251101
  * Aguilar et al. (2020) Aguilar, M., Ali Cavasonza, L., Ambrosi, G., et al. 2020, Phys. Rev. Lett., 124, 211102
  * Aguilar et al. (2021a) Aguilar, M., Cavasonza, L. A., Allen, M. S., et al. 2021a, Phys. Rev. Lett., 126, 041104
  * Aguilar et al. (2021b) Aguilar, M., Ali Cavasonza, L., Ambrosi, G., et al. 2021b, Phys. Rep., 894, 1
  * Ahn et al. (2010) Ahn, H. S., Allison, P., Bagliesi, M. G., et al. 2010, ApJ, 714, L89
  * Alemanno et al. (2021) Alemanno, F., An, Q., Azzarello, P., et al. 2021, Phys. Rev. Lett., 126, 201102
  * Alemanno et al. (2022) Alemanno, F., An, Q., Azzarello, P., et al. 2022, Science Bulletin, 67, 2162
  * An et al. (2019) An, Q., Asfandiyarov, R., Azzarello, P., et al. 2019, Science Advances, 5, eaax3793
  * Atkin et al. (2018) Atkin, E., Bulatov, V., Dorokhov, V., et al. 2018, Soviet Journal of Experimental and Theoretical Physics Letters, 108, 5
  * Bell (1978) Bell, A. R. 1978, MNRAS, 182, 147
  * Bell (2004) Bell, A. R. 2004, MNRAS, 353, 550
  * Bell (2014) Bell, A. R. 2014, Brazilian Journal of Physics, 44, 415
  * Blandford & Ostriker (1978) Blandford, R. D., & Ostriker, J. P. 1978, ApJ, 221, L29
  * Boschini et al. (2018) Boschini, M. J., Della Torre, S., Gervasi, M., et al. 2018, ApJ, 858, 61
  * Boschini et al. (2020a) Boschini, M. J., Torre, S. D., Gervasi, M., et al. 2020a, ApJ, 889, 167
  * Boschini et al. (2020b) Boschini, M. J., Della Torre, S., Gervasi, M., et al. 2020b, ApJS, 250, 27
  * Boschini et al. (2021) Boschini, M. J., Della Torre, S., Gervasi, M., et al. 2021, ApJ, 913, 5
  * Caprioli et al. (2010) Caprioli, D., Amato, E., & Blasi, P. 2010, Astroparticle Physics, 33, 307
  * Casse & Goret (1978) Casse, M., & Goret, P. 1978, ApJ, 221, 703
  * Cummings et al. (2016) Cummings, A. C., Stone, E. C., Heikkila, B. C., et al. 2016, ApJ, 831, 18
  * Derome et al. (2019) Derome, L., Maurin, D., Salati, P., et al. 2019, A&A, 627, A158
  * Ellison et al. (1997) Ellison, D. C., Drury, L. O., & Meyer, J.-P. 1997, ApJ, 487, 197
  * Fermi (1949) Fermi, E. 1949, Physical Review, 75, 1169
  * George et al. (2009) George, J. S., Lave, K. A., Wiedenbeck, M. E., et al. 2009, ApJ, 698, 1666
  * Ghelfi et al. (2016) Ghelfi, A., Barao, F., Derome, L., & Maurin, D. 2016, A&A, 591, A94
  * Ginzburg & Syrovatskii (1964) Ginzburg, V. L., & Syrovatskii, S. I. 1964, The Origin of Cosmic Rays (New York: Macmillan)
  * Gleeson & Axford (1967) Gleeson, L. J., & Axford, W. I. 1967, ApJ, 149, L115
  * Guo & Yuan (2018) Guo, Y.-Q., & Yuan, Q. 2018, Phys. Rev. D, 97, 063008
  * Korsmeier & Cuoco (2022) Korsmeier, M., & Cuoco, A. 2022, Phys. Rev. D, 105, 103033
  * Liu et al. (2012) Liu, J., Yuan, Q., Bi, X.-J., Li, H., & Zhang, X. 2012, Phys. Rev. D, 85, 043507
  * Liu et al. (2018) Liu, W., Yao, Y.-h., & Guo, Y.-Q. 2018, ApJ, 869, 176
  * Malkov & Drury (2001) Malkov, M. A., & Drury, L. O. 2001, Reports on Progress in Physics, 64, 429
  * Moskalenko & Strong (1998) Moskalenko, I. V., & Strong, A. W. 1998, ApJ, 493, 694
  * Niu (2022) Niu, J.-S. 2022, ApJ, 932, 37
  * Ohira & Ioka (2011) Ohira, Y., & Ioka, K. 2011, ApJ, 729, L13
  * Panov et al. (2009) Panov, A. D., Adams, J. H., Ahn, H. S., et al. 2009, Bulletin of the Russian Academy of Science, Phys., 73, 564
  * Potgieter (2013) Potgieter, M. S. 2013, Living Reviews in Solar Physics, 10, 3
  * Ptuskin et al. (2006) Ptuskin, V. S., Moskalenko, I. V., Jones, F. C., Strong, A. W., & Zirakashvili, V. N. 2006, ApJ, 642, 902
  * Ptuskin et al. (2013) Ptuskin, V., Zirakashvili, V., & Seo, E.-S. 2013, ApJ, 763, 47
  * Strong & Moskalenko (1998) Strong, A. W., & Moskalenko, I. V. 1998, ApJ, 509, 212
  * Strong et al. (2007) Strong, A. W., Moskalenko, I. V., & Ptuskin, V. S. 2007, Annu. Rev. Nucl. Part. Sci., 57, 285
  * Tomassetti (2012) Tomassetti, N. 2012, ApJ, 752, L13
  * Trotta et al. (2011) Trotta, R., Jóhannesson, G., Moskalenko, I. V., et al. 2011, ApJ, 729, 106
  * Wu & Chen (2019) Wu, J., & Chen, H. 2019, Physics Letters B, 789, 292
  * Yuan (2019) Yuan, Q. 2019, Science China Physics, Mechanics, and Astronomy, 62, 49511
  * Yuan et al. (2017) Yuan, Q., Lin, S.-J., Fang, K., & Bi, X.-J. 2017, Phys. Rev. D, 95, 083007
  * Yuan et al. (2021) Yuan, Q., Qiao, B.-Q., Guo, Y.-Q., Fan, Y.-Z., & Bi, X.-J. 2021, Frontiers of Physics, 16, 24501
  * Yuan et al. (2020) Yuan, Q., Zhu, C.-R., Bi, X.-J., & Wei, D.-M. 2020, J. Cosmology Astropart. Phys, 2020, 027
  * Zhang et al. (2017) Zhang, Y., Liu, S., & Yuan, Q. 2017, ApJ, 844, L3
  * Zhao et al. (2021) Zhao, M.-J., Fang, K., & Bi, X.-J. 2021, Phys. Rev. D, 104, 123001
  * Zhu et al. (2018) Zhu, C.-R., Yuan, Q., & Wei, D.-M. 2018, ApJ, 863, 119


# Vision Transformer with Attentive Pooling for
Robust Facial Expression Recognition

Fanglei Xue, Qiangchang Wang, Zichang Tan, Zhongsong Ma, and Guodong Guo  This
work was done when Fanglei Xue and Qiangchang Wang were interns at Institute
of Deep Learning, Baidu Research. The first two authors contribute equally.
Corresponding authors: Zhongsong Ma and Guodong Guo. Fanglei Xue and Zhongsong
Ma are with University of Chinese Academy of Sciences, and also with the Key
Laboratory of Space Utilization, Technology and Engineering Center for Space
Utilization, Chinese Academy of Sciences, Beijing, China.
<EMAIL_ADDRESS><EMAIL_ADDRESS>Qiangchang Wang is with
West Virginia University, Morgantown, USA<EMAIL_ADDRESS>Zichang Tan
and Guodong Guo are with Institute of Deep Learning, Baidu Research, and also
with National Engineering Laboratory for Deep Learning Technology and
Application, Beijing, China<EMAIL_ADDRESS><EMAIL_ADDRESS>Manuscript received April 19, 2005; revised August 26, 2015.

###### Abstract

Facial Expression Recognition (FER) in the wild is an extremely challenging
task. Recently, some Vision Transformers (ViT) have been explored for FER, but
most of them perform inferiorly compared to Convolutional Neural Networks
(CNN). This is mainly because the new proposed modules are difficult to
converge well from scratch due to lacking inductive bias and easy to focus on
the occlusion and noisy areas. TransFER, a representative transformer-based
method for FER, alleviates this with multi-branch attention dropping but
brings excessive computations. On the contrary, we present two attentive
pooling (AP) modules to pool noisy features directly. The AP modules include
Attentive Patch Pooling (APP) and Attentive Token Pooling (ATP). They aim to
guide the model to emphasize the most discriminative features while reducing
the impacts of less relevant features. The proposed APP is employed to select
the most informative patches on CNN features, and ATP discards unimportant
tokens in ViT. Being simple to implement and without learnable parameters, the
APP and ATP intuitively reduce the computational cost while boosting the
performance by ONLY pursuing the most discriminative features. Qualitative
results demonstrate the motivations and effectiveness of our attentive
poolings. Besides, quantitative results on six in-the-wild datasets outperform
other state-of-the-art methods.

###### Index Terms:

Facial expression recognition, attentive pooling, vision transformer, affect,
deep learning.

##  1 Introduction

Facial expression is one of the most important ways for people to express
their emotions [1]. Facial Expression Recognition (FER) requires that the
computer program could automatically recognize the expression from an input
face image. The FER task has attracted broad interest in the computer vision
community [2, 3, 4] due to its wide applications in human-computer
interactions, medical progress monitoring, driver fatigue monitoring, and many
other fields.

However, FER is a very challenging task, especially in the wild. This is
mainly because of the significant intra-class variances and inter-class
similarities among expression categories, which differ from the general image
classification task. For example, the same people in the same illumination and
pose may have different expressions, while people with different identities,
ages, gender, and pose may express the same emotion. In the past few years,
with the development of the convolutional neural network, many methods [4, 5,
6, 7, 8, 9] have been proposed and greatly improve the performance of FER.

Figure 1: An illustration of attention maps in TransFER [10] (left) and the
attentive pooling results of APP (right). Our APP directly discards the
background features to reduce the influence of noisy areas and save
computation time.

Recently, the Vision Transformer (ViT) was proposed for image classification
[11] and achieved promising performance with the non-local self-attention
mechanism. It shows great potential for solving visual tasks. Some researchers
adopt the ViT for FER [12, 13, 10]. However, the performance is inferior to
the state-of-the-art CNNs except TransFER [10].  The main issue is that the
ViT needs a large amount of data to train due to a large number of parameters
and lacks the inductive bias [11]. Existing FER datasets are much smaller
compared to general image classification datasets (i.e. ImageNet [14]), making
it hard for newly proposed Transformer-based modules to converge well and are
easy to mistakenly focus on some occlusion or background areas. Many
regularization [15, 16] and attention [6, 17, 8, 10, 13] methods have been
proposed to address this issue.  The TransFER model generates an attention map
and multiplies it with the feature maps to reduce the impact of noisy
features. We investigated the TransFER model and find that the model has
learned to distinguish informative areas from noisy areas (as illustrated in
Fig. 1). However, the noisy features are still fed into the downstream models
in TransFER. Hence, we raise a question: Why do we still compute noisy
features even though we have already known they are noises?

Benefiting from the flexible design of the Transformer model, it could adopt
any number of tokens as input without changing the model parameters. Inspired
by this, we propose the APP module to discard the noisy features directly. As
illustrated in the right part of Fig. 1, the noisy features are now directly
pooled (denoted as a small white grid) instead of multiplying with a small
value.

As for the Transformer block, it is built based on the attention mechanism,
making it more intuitive to perform attentive pooling. Recently, CPVT [18]
found that using the global average pooling (GAP) to replace the [class] token
could produce an even better performance. DeepViT [19] further investigated
this phenomenon, finding that the attention maps become similar after
particular layers. To reduce the redundancy in deep blocks, DeepViT [19]
proposed a Re-attention method to increase the diversity of different layers,
LV-ViT [20] proposed a token labelling strategy to give a label to every token
to supervise. Different from that, and inspired by the Max Pooling in CNNs (as
illustrated in Fig. 2 (a)) to reduce the size of feature maps and obtains an
abstract and robust representation, we directly drop the less essential tokens
and only remain a small number of tokens to embed the information.
Specifically, the ATP module utilizes the attention mechanism in the
Transformer and directly discards the less important tokens in deep blocks.
For a better understanding, the differences between these methods are
illustrated in Fig. 2.

Fig. 2 (b) demonstrates the vanilla ViT model where the patch token number
remains constant. To interrupt the propagation of noisy features, the ATP
gradually decreases the token number after a specific layer, as illustrated in
Fig. 2 (c). Compared with the Max Pooling in CNN models, which can only be
pooled to a quarter at least, the ATP decreases the feature map size in a more
controllable way. Our proposed APViT model that combines APP and ATP is
illustrated in Fig. 2 (d).

Our contributions include the following:

  1. 1.

An Attentive Patch Pooling (APP) module is proposed to select the most
distinguishable local patches from the CNN feature maps, which could prevent
the noisy patches from passing through to the downstream modules to affect the
recognition performance.

  2. 2.

An Attentive Token Pooling (ATP) module is developed to utilize the attention
mechanism of the Transformer model only to pay attention to the top-k relevant
tokens to reduce the influence caused by noises and occlusions and save the
computation time.

  3. 3.

Experimental results on several challenging in-the-wild datasets demonstrate
the advantages of our APViT method over the state of the arts.

Figure 2: Illustration of the feature map size or token number of different
architectures. (a) denotes the Max Pooling in CNN to quarter the feature map
size. (b) denotes the vanilla ViT with a fixed token number. (c) denotes our
ATP, which gradually reduces the number of tokens. (d) denotes our APViT,
which consists of the APP and ATP to pool CNN features and ViT tokens
simultaneously.  Figure 3: The overall architecture of our APViT model.
Firstly, feature maps are extracted from the facial images by a CNN extractor
(e.g. ResNet[21], MobileNet [22]). Secondly, an attention map is calculated by
the attention module ($attn\\_f$), which guides the APP module to select
corresponding informative tokens. Thirdly, the selected tokens are flattened
and fed into $M$ stacked ATP enhanced Transformer Blocks (denote as ATP
Blocks) with an additional learnable [class] token attached. Similar to the
APP, the ATP Blocks further select the most discriminative tokens with the
multi-head self-attention mechanism. Thus, the patch information would be
gathered to the [class] token as the number of kept patch tokens decreases
gradually. Finally, an MLP Head is attached to generate the final
classification result from the [class] token. The left bottom blue dashed
rectangle indicates the pooling process from the view of feature information
flow. The APP and ATP select the most distinguished patches and tokens,
avoiding the noisy and occluded regions passing to the downstream calculation.

##  2 Related Work

Facial expression recognition (FER) frameworks mainly consist of three
processes: face detection and alignment, feature extraction, and
classification. To reduce the impact of noisy background, a face detector
(such as MTCNN [23] and RetinaFace [24]) first detects faces from the image
and aligns them to canonical alignments with corresponding detected landmarks.
After that, a feature extractor is responsible for extracting discriminative
features from the facial image. These features are then classified into
expression categories. In the past decades, FER usually relied on well-
designed hand-crafted features, which could be further grouped into texture-
based and geometry-based features. For example, LBP [2], Gabor [25], HOG [26],
and SIFT [27] are proposed to utilize texture-based features. On the other
hand, many geometry-based methods [28, 29, 30] take consider of features of
some typical areas (such as eyes, noses, and mouths) based on landmarks. Most
of these methods perform well on laboratory-collected databases. However, when
facing in-the-wild facial images, such as various poses and occlusions, their
performances are not as good as recently proposed learning-based methods [4,
5, 6, 7, 8, 9]. With the help of public large-scale in-the-wild facial
expression datasets, deep learning methods have achieved many breakthroughs.
Li _et al_. [4] proposed the DLP-CNN method to extract discriminative features
while Cai _et al_. [5] proposed a island loss function to achieve the similar
goal. Li _et al_. [6] utilized an attention mechanism to let the CNN network
focus on the most discriminative un-occluded regions to overcome the occlusion
problem.

Our proposed attentive pooling modules are mainly based on the patch attention
mechanism and could affect tokens in the Transformer model. So we review the
previous works in these two respects as follows:

###  2.1 FER Based on Patch Attention

For high-resolution images, ATS [31] samples locations of informative patches
from an attention distribution based on a lower resolution image to save
computation costs. NTS-Top [32] improves this attention sampling with a
differentiable top-K patch selection module. STTS [33] selects tokens in both
temporal and spatial dimensions to save computation in video tasks based on a
scorer network and a differentiable top-K selection. For FER task,  the facial
expression is invoked by the movements of several facial muscles. Some
researchers found that few patch areas contribute mostly to expression
recognition [34]. Many FER methods [35, 28, 36] split the face image into
patches and select discriminative regions with an attention mechanism. Zhong
_et al_. [35] proposed a two-stage multi-task sparse learning framework to
locate common patches shared by different expressions and specific patches
useful for a particular expression. Happy  _et al_. [28] adopted the one-
against-one classification method to determine salient facial patches.
Nineteen patches around the eyes, nose, and mouth are pre-selected as
candidate patches. A patch is selected if it can classify the two expressions
accurately.

Recent deep learning methods utilize attention mechanisms that learn an
attention weight and multiply this weight with the features. Li  _et al_. [6]
selected 24 patches based on facial landmarks and proposed the Patch-Gated
Unit to perceive occluded patches. Zhao  _et al_. [37] divided extracted
feature maps into several local feature maps and employed a Convolution Block
Attention Module [38] to generate attention maps. The generated attention maps
are multiplied by the original local feature maps to propagate gradients. MViT
[13] utilizes two transformer modules where one generates a mask to filter out
complex backgrounds and occlusion patches, and the other is the vanilla ViT
model to perform the classification. However, background features in these
methods still participate in the following calculation even though we know
that they are less critical and may cause misfocus to be prejudicial to
recognition. Different from this, our Attentive Pooling modules directly
discard these unimportant features, which could prevent the noisy information
from passing to the following module and guide the model to ONLY focus on the
most informative features.  Another difference between our method and these
methods is that we find the hand-designed mask generator performs better than
the learning-based one. And benefitting this, our APViT could apply the origin
top-K operation to replace the fancy differentiable ones.

###  2.2 Vision Transformers on FER

The Transformer architecture [39] was first proposed for machine translation
based on attention mechanisms. And it is interesting to explore the
Transformer to visual tasks [40, 41]. Vision Transformer (ViT) [11] is the
first work to directly apply the vanilla Transformer with few modifications
for image classification. By pre-training on large datasets (e.g. ImageNet-1k,
ImageNet-21k [14], and JFT [42]) and fine-turning on the downstream target
dataset, ViT has achieved excellent results compared to the CNN networks [43,
44, 45].

Some researchers also introduced ViTs to FER. Ma  _et al_. [12] proposed an
attention selective fusion module to leverage feature maps which are generated
by two-branch CNNs, and applied the vanilla ViT module to explore
relationships between visual tokens. Li  _et al_. [13] proposed the MViT,
utilizing two transformer modules where one generates a mask to filter out
complex backgrounds and occlusion patches, and the other one is the vanilla
ViT model to perform the classification. A squeeze and excitation module [46]
was proposed before the MLP head in the ViT module to optimize the learning
process on small facial expression datasets. TransFER [10] extracts abundant
attention information with multi-branch local CNNs and the multi-head self-
attention in the ViT and forces these attention features to be diverse by a
multi-attention dropping module.

Unlike these methods, we proposed two attentive pooling methods without
bringing extra training parameters to the ViT model, fully using the pre-
trained weights. The pooling operation also forces the model only to pay
attention to the most discriminative features, reducing the influence of
occlusions and noises and saving the computation time.

##  3 Proposed Method

###  3.1 Overview

We propose two attentive pooling modules to gradually pool features in the
convolutional neural network and vision transformer. As discussed above, our
primary motivation is to select informative features and drop the features of
background and occlusion with as few additional parameters as possible to
overcome the over-fitting problem caused by limited data.

Fig. 3 illustrates the framework of our proposed attentive pooling methods
based on the hybrid ViT-small model [11]. A facial image is first input into a
CNN backbone to extract feature maps, then flattened and fed into multiple
Transformer Blocks.

Besides generating the patch tokens, the feature maps extracted from the CNN
are encoded to an attention map by the attention generating method (denoted as
$attn\\_f$). The proposed APP module takes this attention map as input and
selects the top-k patch tokens according to their corresponding attention
values.

The selected patch tokens and the [class] token are jointly fed into the
Transformer blocks [39] which consist of alternating layers of multi-head
self-attention (MSA) and MLP blocks to explore the relationship between
different patches in a global scope. The ATP module is proposed for the
Transformer Blocks to gradually decrease the token number, forcing the model
to pay attention to discriminative tokens and eliminate the noisy tokens about
background and occlusions. After the Transformer encoder, the [class] token is
used to generate the final predictions through a single fully connected layer.

###  3.2 Attentive Patch Pooling

Since the CNNs slide through the whole face, features from uninformative areas
may introduce noises to the feature maps, which may be harmful to recognition.
Our APP module is employed to locate informative areas and drop the
uninformative ones to boost recognition performance.

The shape of the feature maps extracted from the CNN backbone is $[H,W,C]$
where $H$, $W$, and $C$ represent the feature maps’ height, width, and channel
number, respectively. The APP module takes these attention maps as input and
selects the top-k most important patches from the total $H\times W$ patches
according to the important weight of each position. To achieve this, we need
to define a pooling criterion to guide where to pool and a pooling method to
process pooling.

Pooling Criterion. To determine which parts to pool, we need a pooling
criterion to denote the importance of every pixel. Each pixel in the feature
map denotes a small patch of the original facial input because the Max Pooling
layers enlarge the reception field. So given the feature maps in $[H,W,C]$, an
attention map with a size $[H,W,1]$ is needed to denote the relevant weight.
In other words, the pooling criterion $\mathcal{F}$ is responsible for
reducing the channel dimension of the feature maps to one.

$\mathcal{F}:R^{H\times W\times C}\xrightarrow[]{}R^{H\times W\times 1}$ (1)

There are a few methods that could complete this task. We group them into
hand-designed generators and learning-based generators.

Figure 4: Illustration of the two kinds of pooling criteria. (a) denotes the
hand-designed generator, which calculates the sum of the absolute values of
the feature maps. (b) denotes the learning-based generator. Taking the LANet
[47] as an example, two $1\times 1$ convolutional layers gradually decrease
the channel dimension to 1.

  1. 1.

Hand-designed generators utilize mathematical operations to reduce the channel
dimension. For simplicity, the sum of values ($F_{SUM}$), the sum of absolute
values ($F_{ABS}$), and the max of values ($F_{MAX}$) [48] are introduced
here. As illustrated in Fig. 4 (a), the $F_{ABS}$ sums the absolute values of
the feature maps along the channel dimension. The generated attention map puts
high weight on locations where multiple channels respond with high
activations. Similarly, the $F_{MAX}$ takes the maximal value among all neuron
activations.

  2. 2.

Learning-based generators utilize neural networks to learn the relationship
between the feature maps of the attention map. The $1\times 1$ convolutional
layer is widely used for this operation to reduce the channel dimension
without changing the spatial resolution. The LANet [47] illustrated in Fig. 4
(b) is one of the examples of these generators. It combines two convolutional
layers to gradually reduce the channel dimension. The sigmoid layer at the end
of the original LANet is discarded here because only the relative relationship
matters.

Pooling Method. After getting the attention map, we need a pooling method to
drop the less relative patches and retain the most relative ones. A common
method is to multiply the feature maps by the attention map, like in [47, 36].
However, this method fails to discard the noisy patches. Although locations
with lower attention weights have smaller feature values, they are still
computed in the incoming modules, which may bring noise and lead to misfocus.

The proposed APP module takes the top-k attention activation positions and
selects the corresponding features. The detailed process is illustrated in
Fig. 3.

###  3.3 Attentive Token Pooling

The APP module is efficient in pooling the uninformative and noise patches by
the attention mechanism, but it lacks the global scope because of the small
reception field of CNN. As illustrated in Fig. 3, the Transformer encoder
enhanced by ATP (denoted as ATP Block) is utilized to explore the relationship
among selected patches in a global scope. We start with a brief description of
the Transformer Encoder.

Transformer Encoder. The Transformer model is firstly proposed to process
natural language processing (NLP) tasks and then utilized for image
recognition [11]. It contains a stack of encoder blocks, with each block
mainly consisting of Multi-head Self-Attention (MSA), Multi-Layer Perception
(MLP), and Layer Normalization (LN). The MSA module takes a series of tokens
as input, and linearly transforms them into three parts, denoted as queries
($Q\in\mathbb{R}^{(n+1)\times d}$), keys ($K\in\mathbb{R}^{(n+1)\times d}$),
and values ($V\in\mathbb{R}^{(n+1)\times d}$), where $n$ is the sequence
length of the tokens, and $d$ is the embedding dimensions. The [class] token
is concatenated to the input tokens, so the total length of the input is
$n+1$. Every token is updated based on the correlation with each other. It can
be considered a weighted sum calculated by the dot product of $Q$ and $K$.
This process can be formulated as:

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d}})V$ (2)

After the Transformer encoder, only the [class] token is converted by a
single-layer MLP layer to generate the output score.

As we can see, the Transformer encoder does not change the length of the input
sequence, nor do the MLP or the LN. It is different from the CNN models, where
the Max Pooling is widely used to reduce the spatial resolution and increase
the generalization ability. So, we propose the ATP directly drop the less
important ones and only remain a small number of tokens to embed the
information. For a better understanding, the differences between these methods
are illustrated in Fig. 2.

Pooling Criterion. Like the APP described above, the ATP also needs a pooling
criterion to guide where to pool. However, unlike the CNN, the Transformer
architecture is built based on the attention mechanism. The Transformer model
is much more intuitive to find where to pay more attention without an
additional attention-generating module.

In the self-attention module, every token’s query will compute a relation with
the key of other tokens by a dot product, as formulated in Eq. (2). The output
of a token is calculated by a weighted sum of the values of input tokens.
Moreover, the weight is determined by the query and key. The higher the
$QK^{T}$ value is, the more relevant between these two tokens. As described
above, the [class] token is utilized to represent the whole image. The more
relative between some patch tokens and the [class] token, the more important
these patch tokens are to recognize the expression. So we take the $QK^{T}$ of
the [class] token as the pooling criterion to remove the less relative tokens.
As there are multiple heads, the sum of all heads is utilized.

Pooling Method. The pooling method in ATP is similar to APP. We only consider
the top-k tokens and remove the rest based on the pooling criterion. It is
worth noting that the pooling only acts on patch tokens, and the [class] token
is always reserved. Because there are several stacked attention blocks in the
Transformer encoder, we propose a keep rate $r$ to adjust the pooling process,
which is defined as follows:

$keep\\_num=\lfloor r*input\\_token\\_number\rfloor$ (3)

where $\lfloor x\rfloor$ denotes the floor operation. As discovered in
DeepViT, the attention collapse issue only happens in deep blocks, so the ATP
is only applied to the second half of Transforme blocks. We set $r$ in these
blocks to the same value for convenience. Specifically, denote there are $M$
blocks. The output token number will be about $k\cdot r^{M/2}$, ignoring the
influence of the [class] token.

##  4 Experiments

###  4.1 Datasets

We evaluate our proposed method on six in-the-wild FER datasets: RAF-DB [4],
FERPlus [49], AffectNet [3], SFEW [50], ExpW [51], and Aff-Wild2 [52].

RAF-DB contains 29,672 real-world facial images collected by searching on
Flickr and was labelled by 40 trained human workers. In our experiments, the
basic annotation images are utilized, resulting in 12,271 images for training
and 3,068 images for testing, with six basic expressions and neutral.

FERPlus is extended from FER2013 [53], providing more accurate annotations
relabeled by ten workers to eight expression categories (six basic
expressions, neutral, and contempt). It contains 28,709 training images, 3,589
validation images, and 3,589 test images.

AffectNet contains about 1M facial images collected by three major search
engines, where about 420K images were manually annotated. Following the
settings in [54], we only used about 280K training images and 3,500 validation
images (500 images per category) with seven emotion categories for a fair
comparison.

SFEW 2.0 was created by extracting static frames from the Acted Facial
Expressions in the Wild (AFEW) [55], a temporal facial expressions database.
It was used as a benchmark in the third Emotion Recognition in the Wild
Challenge (EmotiW2015), with 958 facial images for training and 436 372 images
for validation and testing. Since the label on the test set is not public, the
performance on the validation set is reported.

ExpW was collected by searching combined emotion-related keywords (such as
”excited”, ”panic”) and occupation nouns (such as ”student”, ”layer”) on the
Google image search. Facial images are filtered by a facial detector and
labelled into six basic expressions and neutral. There is no official training
set split approach. For a fair comparison, we follow [56] to split the dataset
into 75% for training, 10% for validation, and 15% for testing with random
sampling. As a result, there are 68,845, 9,179, and 13,769 images for
training, validation, and testing.

Aff-Wild2 [52, 57] is a recently proposed, large-scale facial expression
recognition database. It is an extension of the Aff-Wild database [58] and by
far is the largest public FER dataset in the wild. It contains 564 videos of
around 2.8M frames and labels them with three representations, including
Categorical Emotions (CE), Action Units (AU), and Valence Arousal (VA).
Following [59], we adopt the training set of both CE and AU tasks for training
and the validation set of the CE task for testing. For a fair comparison, we
take the same head design of [59], but with a different loss weight.
Specifically, we sum the expression classification loss and action unit loss
with weights 10 and 1 to strengthen the classification performance.

TABLE I: Evaluation (%) of APP, ATP, [class] token (CLT), global average pooling (GAP) on RAF-DB and AffectNet, and corresponding FLOPs compared with the baseline. APP | ATP | Head | RAF-DB | AffectNet | FLOPs
---|---|---|---|---|---
|  | GAP | 90.77 | 65.66 | 100%
|  | CLT | 91.26 | 65.91 | 100%
| $\checkmark$ | CLT | 91.52 | 66.65 | 94%
$\checkmark$ |  | GAP | 90.91 | 66.34 | 88%
$\checkmark$ |  | CLT | 91.65 | 66.51 | 88%
$\checkmark$ | $\checkmark$ | CLT | 91.98 | 66.94 | 83%
TABLE II: Evaluation (%) of APP and ATP on Aff-Wild2 and corresponding FLOPs compared with the baseline. APP |  ATP | Aff-Wild2 | FLOPs
---|---|---|---
|  | 85.02 | 100%
| $\checkmark$ | 85.70 | 81%
$\checkmark$ |  | 85.39 | 61%
$\checkmark$ | $\checkmark$ | 86.03 | 54%

###  4.2 Implementation Details

All images were aligned and cropped to $112\times 112$ pixels by MTCNN [23]
before being input into the model. The first three stages of IR50 [60], pre-
trained on Ms-Celeb-1M 111The pre-trained weight is downloaded from
https://github.com/ZhaoJ9014/face.evoLVe.PyTorch\\#Model-Zoo. [61] are used as
the feature extractor. The ViT-small [11] model with 8 Transformer blocks pre-
trained on ImageNet222The pre-trained weight is downloaded from
https://github.com/rwightman/pytorch-image-models/. is used as Transformer
Encoders. The keep number ($k$) of the APP is set to 160 for RAF-DB,
AffectNet, and FERPlus; and 120 for SFEW. The keep rate ($r$) of ATP is set to
0.9 for these four databases. It is worth noting that, ExpW and Aff-Wild2
achieve the best performance with only $k=80$ and $r=0.6$, indicating that our
APViT works more efficiently on large-scale datasets. It is worth noting that
there are 105 tokens reserved before the classification head when $k=160$ and
$r=0.9$, and only 10 tokens reserved when $k=80$ and $r=0.6$. Our APViT
maintains the performance with about half computation costs. Our model was
trained with the SGD optimizer to minimize the cross-entropy loss. The
momentum was set to 0.9, and the weight decay was set to 5e-4. The mini-batch
size was set to 128, and the learning rate was set to 1e-3 for RAF-DB,
FERPlus, SFEW, Aff-Wild2, and 5e-4 for AffectNet and ExpW with cosine decay
[62]. We also found that gradient clipping of 10 based on the L2-norm is
helpful, as described in [11]. During the training process, we follow the same
augmentation methods as TransFER [10]. At the test time, only the resize
operation is reserved. We train our model with one NVIDIA V100 GPU, 20 epochs
for FERPlus, 40 epochs for RAF-DB and SFEW, 6k iterations for AffectNet, 9k
iterations for ExpW, and 150k iterations for Aff-Wild2. The overall accuracy
is reported by default.

###  4.3 Ablation Studies

Figure 5: Evaluation (%) of different keep numbers ($k$) and keep rates ($r$)
on AffectNet. As the keep number and keep rate decade, the performance first
increases and then decreases. From the top left to the bottom right, our AP
modules reduce more than half of the computations, and the performance is
robust without hard decreases, indicating the effectiveness of our AP modules.

Effectiveness of the Proposed Modules. To evaluate the effect of the proposed
modules, we designed the ablation study on RAF-DB and AffectNet to better
understand the impact of the proposed APP and ATP. Our two pooling modules are
designed to resolve the over-fitting problem when transferring the pre-trained
Transformer model to small datasets and restrain the impact of noisy patches.

As is shown in Tab. I, both modules increase the performance in FER compared
with the baseline strategy. The APP and ATP contribute similarly when applied
solely. The APP improves the performance by 0.39% and 0.60% in RAF-DB and
AffectNet, respectively. The ATP brings a similar performance gain, 0.26%, and
0.74%, respectively. However, the APP takes effect more early, thus decreasing
more FLOPs than ATP. The best improvement is obtained when combining two
pooling methods. Specifically, the baseline is improved from 91.26% to 91.98%
on RAF-DB and from 65.91% to 66.94% on AffectNet with only 83% FLOPs333The
number of FLOPs is calculated using the fvcore toolkit provided by Facebook AI
Research.  We also compared the [class] token (CLT) with global average
pooling (GAP) with different strategies. Although GAP outperforms CLT in CVPT
[18], it fails to achieve a better performance here. We suspect different pre-
trained strategies cause this. The model should change as little as its pre-
trained for downstream finetuning.  These experiments illustrate that the APP
and ATP could help the model to only focus on more discriminative patches and
reduce the over-fitting problem caused by the noisy patches.

Impact of the Keep Number $k$ in APP and Keep Rate $r$ in ATP. The keep number
($k$) and keep rate ($r$) are two hyper-parameters of our proposed model to
control how many patches to keep. To explore the impact of these two
parameters, we study different values from 196 to 40 for $k$, and 1 to 0.6 for
$r$ on AffectNet. When $k=196$ and $r=1$, it represents the baseline model
with no pooling taking effect. The results are shown in Fig. 5, the best
performance (66.94%) could be obtained when $k=160$ and $r=0.9$. The smaller
$k$ and $r$ are, the fewer patches are kept for recognition, which forces the
model to focus on more discriminative patches and take less computation. Too
less information, however, may make it harder to recognize, so the performance
decreases when $k$ and $r$ are too small. It is worth noting that the minimal
model ($k=40$ and $r=0.6$) here only has five tokens reserved but still
outperforms the baseline model. Specifically, it boosts the performance from
65.91% to 66.48%, but with only 45.12% FLOPs compared to the baseline model.

Evaluation of Pre-trained Weights. To evaluate the effects of pre-trained
weights on CNN and ViT, we conduct experiments on RAF-DB and AffectNet. As
illustrated in Tab. III, the pre-trained weight is more critical for CNN,
which boosts the performance from 81.85% to 90.84% on RAF-DB. This indicates
that CNN pre-trained on facial recognition databases can extract helpful
features for FER. In addition, pre-training on ViT has a negative influence
when no CNN pre-trained weight is loaded but could help the model to achieve
better performance with pre-trained CNN.

TABLE III: Performances with or without pre-trained weights. Pre-trained CNN | Pre-trained ViT | RAF-DB | AffectNet
---|---|---|---
|  | 81.55 | 53.88
$\checkmark$ |  | 90.84 | 65.34
| $\checkmark$ | 78.91 | 51.56
$\checkmark$ | $\checkmark$ | 91.98 | 66.91
TABLE IV: Performances (%) with gradually decayed and constant $k$ and $r$. Database | Gradually decay | Constant
---|---|---
RAF-DB | 91.46 | 91.98
FERPlus | 90.19 | 90.86
AffectNet | 66.08 | 66.91

Comparison of Gradually Decay $k$ and $r$. As AP modules rely on accurate
attention maps, however training from scratch cannot get accurate attention
maps right from the start. Intuitively, not to drop patches in the early stage
of training, gradually increasing the proportion of dropped patches as the
network learns is a more appropriate training strategy. To evaluate the impact
of this strategy with the constant one, we designed experiments with gradual
decay of the $k$ and $r$ on RAF-DB, FERPlus, and AffectNet. Specifically, the
$k$ and $r$ are initially set to 196 and 1.0 at the beginning and linearly
decayed to the database’s corresponding values. Surprisingly, as we can see
from Tab. IV, the performances with gradually decayed $k$ and $r$ are slightly
inferior to the constant ones. This may be because inaccurate attention maps
perform like random drop out at the beginning of the training period, pushing
the model to find the informative features and preventing the model from over-
fitting to noisy backgrounds. However, the gradual decay strategy lacks this
benefit.

Comparison of Different Attention Generate Methods. As discussed in the Method
section, there are several different methods to generate attention maps. We
run an ablation study to specify the different impacts of these methods on
RAF-DB and AffectNet. As illustrated in Tab. V, when $F_{SUM}$ is adopted as
the method in ATP, $F_{LANet}$ (adopted by TransFER [10]) performs the worst
(91.23% and 66.28%) and even worse than the baseline (91.26% ) on RAF-DB. This
may be because the LANet needs to multiply the attention map with the features
to backward the gradient. The multiplication operation changes the magnitude
of features and makes the downstream network harder to adapt. Hand-designed
methods do not have this problem and achieve better performance. The best
performance (91.98% and 66.94%) is achieved by $F_{ABS}$, which indicates that
the negative values in CNN features also contain rich semantic information.

We also compare three hand-designed methods in ATP. The $F_{ABS}$ here does
not achieve comparable performance as in APP did and is much lower than
$F_{SUM}$. It is because the dot product attention mechanism has a strong
mathematical meaning: small values (including negative ones) indicate a poor
relationship. While taking the absolute value as attention value breaks this
mechanism. The $F_{MAX}$ takes the max value among multiple heads, but certain
heads may dominate the result, which reduces the discriminative ability.

TABLE V: Evaluation (%) of different pooling methods on RAF-DB, the method action on APP and ATP are denoted as APP method and ATP method, respectively. APP Method | ATP Method | RAF | AffectNet
---|---|---|---
LANet | SUM | 91.23 | 66.28
MAX | 91.36 | 66.51
SUM | 91.72 | 66.66
ABS | 91.98 | 66.94
ABS | ABS | 91.33 | 66.34
MAX | 91.39 | 66.54
TABLE VI: Performances with “soft” and “hard” pooling approaches. Database | Soft | Hard (Ours)
---|---|---
RAF-DB | 91.07 | 91.98
FERPlus | 90.55 | 90.86
AffectNet | 65.80 | 66.91
TABLE VII: Performance comparison (%) with SOTA methods on RAF-DB and AffectNet. “*” indicates Transformer-based models. Method | RAF-DB | AffectNet
---|---|---
DLP-CNN [4] | 80.89 | 54.47
gACNN [6] | 85.07 | 58.78
IPA2LT [7] | 86.77 | 55.71
LDL-ALSG [63] | 85.53 | 59.35
RAN [8] | 86.90 | 59.50
CovPool [64] | 87.00 | -
SCN [65] | 87.03 | 60.23
DLN [66] | 86.40 | 63.70
DACL [67] | 87.78 | 65.20
KTN [54] | 88.07 | 63.97
EfficientFace [68] | 88.36 | 59.89
PAT-Res101 + attr [69] | 88.43 | -
FDRL [70] | 89.47 | -
PT [71] | 89.57 | 58.54
VTFF* [12] | 88.14 | 61.85
MViT* [13] | 88.62 | 64.57
TransFER* [10] | 90.91 | 66.23
APViT* (Ours) | 91.98 | 66.91
TABLE VIII: Per-class performance comparison (%) with the state-of-the-art methods on RAF-DB. “*” indicates Transformer-based models. Method | Anger | Disgust | Fear | Sadness | Happiness | Surprise | Neutral | Average
---|---|---|---|---|---|---|---|---
VGG [54] | 53.09 | 33.75 | 14.87 | 77.41 | 92.41 | 67.48 | 82.35 | 60.19
ResNet [54] | 62.58 | 18.13 | 22.97 | 59.21 | 90.04 | 72.04 | 82.50 | 58.35
CL-CNN [72] | 71.61 | 14.38 | 9.46 | 83.26 | 94.09 | 86.63 | 83.68 | 63.30
DLP-CNN [4] | 71.60 | 52.15 | 62.16 | 80.13 | 92.83 | 81.16 | 80.29 | 74.20
IL-CNN [5] | 70.99 | 45.63 | 32.43 | 76.78 | 91.48 | 78.42 | 75.74 | 67.35
Boosting-POOF [73] | 73 | 57 | 64 | 74 | 89 | 80 | 76 | 73.19
STSN [54] | 82.72 | 66.25 | 59.46 | 87.87 | 94.35 | 86.63 | 85.00 | 80.32
KTN [54] | 81.48 | 65.62 | 68.92 | 87.24 | 94.60 | 83.28 | 88.53 | 81.38
PT [71] | 81 | 55 | 54 | 87 | 96 | 87 | 92 | 78.86
PAT-Res101 + attr [69] | 77.16 | 60.00 | 67.57 | 86.61 | 95.53 | 89.67 | 88.38 | 80.70
MViT* [13] | 78.40 | 63.75 | 60.81 | 87.45 | 95.61 | 87.54 | 89.12 | 80.38
VTFF* [12] | 85.80 | 68.12 | 64.86 | 87.24 | 94.09 | 85.41 | 87.50 | 81.20
TransFER* [10] | 88.89 | 79.37 | 68.92 | 88.70 | 95.95 | 89.06 | 90.15 | 85.86
APViT* (Ours) | 86.42 | 73.75 | 72.97 | 88.70 | 97.30 | 93.31 | 92.06 | 86.36
Figure 6: Confusion matrices of our APViT model on RAF-DB, AffectNet, FERPlus,
ExpW, SFEW, and Aff-Wild2 databases.

Comparison of ”Soft” Pooling with Our ”Hard” Pooling. In most deep-learning-
based models, like the LANet in the TransFER model, multiplying the mask with
the features (denoted as ”soft” pooling) is the only choice to propagate
gradients to the mask-generating branch. We overcome this restriction by
utilizing the hand-designed attention generator and found that its performance
is good enough and could reduce the computation. To compare these two
approaches, we conduct experiments on three databases, as illustrated in Tab.
VI. As we can see, the ”soft” pooling approach performs much worse than the
”hard” one, especially for the noisy database (AffectNet). The conclusion is
the same as the LANet, which has been illustrated in Tab. V. This indicates
that our proposed ”hard” pooling is more efficient in helping the model to
find the informative features.

###  4.4 Comparison with the State-of-the-Art Methods

In this section, we compare our proposed method with several state-of-the-art
(SOTA) methods on six major in-the-wild datasets in Tab. VII \- XII. We also
illustrate the confusion matrices of these databases on Fig. 6.

Results on RAF-DB. The comparison with other progressive methods on RAF-DB is
illustrated in Tab. VII. TFE [74] adopts joint learning with identity and
emotion information. DLP-CNN [4], DACL [67], and KTN [54] utilize new proposed
loss functions to enhance the discriminative power of deep features; IPA2LT
[7], LDL-ALSG [63], and SCN [65] tend to increase the performance by solving
the uncertain problem in the FER task. VTFF [12], MViT [13] and TransFER [10]
use Transformer to obtain global-scoop attentions. Among that, TransFER
obtained a big performance gain with 90.91%. Based on their architecture, our
proposed APViT further improves the recognition performance with the attentive
pooling modules by restraining the noisy influence. We also find that
realigning the facial images in the official RAF-DB database could further
improve the performance. Our model outperforms the state-of-the-art models
with 91.98%.

The per-class results and the average accuracies on RAF-DB are illustrated in
Tab. VIII. As we can see, our APViT outperforms other methods in most classes.
Happiness has the highest accuracy with 97.30%, and fear has the lowest
accuracy with 72.97% (but still outperforms other methods). The RAF-DB
database is collected in the wild and is class-imbalanced because negative
expressions (such as anger, disgust, fear, and sadness) are rare and
challenging to collect in daily life. As is illustrated in Fig. 6 (a), thanks
to the reliable label annotation, our method performs well in negative
expressions (the left top part of the confusion matrix). We believe this is
because the attentive pooling module forces the model to find and focus on the
most class-relative patch features.

TABLE IX: Performance comparison (%) with SOTA methods on FERPlus. “*” indicates Transformer-based models. Method | FERPlus
---|---
TFE-joint learning [74] | 84.29
PLD [49] | 85.10
RAN [8] | 88.55
SeNet50 [75] | 88.80
RAN-VGG16 [8] | 89.16
SCN [65] | 89.35
PT [71] | 86.60
KTN [54] | 90.49
VTFF* [12] | 88.81
MViT* [13] | 89.22
TransFER* [10] | 90.83
APViT* (Ours) | 90.86

Results on AffectNet. Since the testing set of AffectNet is not public. We
evaluate and compare the performance of the validation set. As the training
set of the AffectNet is imbalanced and the validation set is balanced, we
employ a heavy over-sampling strategy as RAN [8], SCN [65], MA-Net [37], and
TransFER does. For a fair comparison, the 7-categories results are illustrated
in the third column of Tab. VII. As we can see, we obtain 66.91% FER accuracy,
which outperforms all other state-of-the-art methods. The confusion matrix,
illustrated in Fig. 6 (b), demonstrates that our model has a similar
recognition capacity (57% to 69%) in all seven expression categories, while
happiness has the highest accuracy with 88%.

Results on FERPlus. Different from other databases, FERPlus has eight
expression categories with an additional contempt. As illustrated in Tab. IX,
our method performs best with an accuracy of 90.86%. Compared with TransFER,
we maintain the performance with 83% computations. The confusion matrix of
FERPlus is illustrated in Fig. 6 (c). As we can see, contempt performs worst
and is hard to distinguish from neutral. This is because there are only 0.47%
contempt facial images versus about 29% neutral images in the training set,
making model predictions biased to neutral as illustrated in the right bottom
of Fig. 6 (c).

TABLE X: Performance comparison (%) with SOTA methods on SFEW. Method | SFEW 2.0 (Val)
---|---
DLP-CNN [4] | 51.05
Kim’s CNN [76] | 52.5
LBAN-IL [77] | 55.28
RAN(VGG16+ResNet18) [8] | 56.40
APM [78] | 57.57
DAN [79] | 57.88
AHA [80] | 58.89
MA-Net [37] | 59.40
PAT-Res101 [69] | 53.90
PAT-Res101 + attr [69] | 57.57
APViT (Ours) | 61.92

Results on SFEW. We compare our APViT with several state-of-the-art methods on
SFEW in Tab. X. LBAN-IT [77] proposed a local binary attention network and an
islets loss to discover local changes in the face. Since the training samples
of SFEW are limited (less than 1,000 images for training), MA-Net [37] pre-
trained the model in FER2013, while APM [78] pre-trained the model in RAF-DB.
With fine-tuning, they achieved performance with 59.40% and 57.57%. PAT-Res101
[69] extend the training dataset with RAF-DB to have 50% RAF-DB samples in
every mini-batch. PAT-Res101+attr [69] further improved the performance by
introduce attribute-annotated samples to 57.57%. However, we only use the pre-
trained weights on the RAF-DB database and outperform these methods without
additional data with an accuracy of 61.92%. As we can see from the confusion
matrix, which is illustrated in Fig. 6 (d), the accuracy of disgust and fear
are both 9% only. Since the distribution of SFEW is extremely imbalanced,
there are only 54 disgust samples and 81 fear samples in the training set. The
poor performance in these two categories is reasonable and acceptable.

TABLE XI: Performance comparison (%) with SOTA methods on ExpW. Method | ExpW
---|---
HOG + SVM [51] | 60.66
Baseline DCN [51] | 65.06
DCN + AP [51] | 70.06
Lian _et al_.[81] | 71.90
Pham _et al_.[82] | 72.84
PAT-Res101 [69] | 69.29
PAT-Res101 + attr [69] | 72.93
SchiNet [56] | 73.10
APViT (Ours) | 73.48
TABLE XII: Performance comparison (%) with SOTA methods on Aff-Wild2.  Method | Aff-Wild2 (val) | Rank
---|---|---
Zhang _et al_.[83] | 85.6 | 1
Jin _et al_.[84] | 63.0 | 2
Thinh _et al_.[59] | 82.6 | 3
Wang _et al_.[85] | 50.1 | 4
Deng, Wu, and Shi [86] | 57.79 | 5
APViT (Ours) | 86.03 | -

Results on ExpW. Same as SFEW, due to the severe imbalanced problem, the
performance in disgust and fear are poor, and predictions are biased to
neutral as illustrated in Tab. 6 (e). The comparison of our APViT with other
state-of-the-art methods on ExpW is illustrated in Tab. XI. Lian  _et al_.
[81] divided the face into six areas to explore the authenticity of <emotion,
region>pairs. They only chose the facial images with confident scores greater
than 60 in the database and achieved 71.90% accuracy. Pham  _et al_. [82] used
action unit features to find the images with a similar emotion for
oversampling. SchiNet [56] combined ExpW with other databases (CEW [87],
CelebA [88], and EmotionNet [89]) and achieved a promising accuracy, 73.10%.
With only training on the target database, our APViT outperforms these methods
with 73.48% accuracy. Moreover, benefiting from the large-scale database, our
APViT achieves the best performance with only about half of (54.07%)
computations.

Results on Aff-Wild2. We compare our proposed APViT with other state-of-the-
art methods in the second ABAW2 Competition [52], as illustrated in Tab. XII.
As the test set is private, we only report the classification accuracy on the
validation set. The rank column indicates their leader board rank on the test
set. [90, 85, 86] utilized all three annotations, including CE, AU, and VA,
achieving 85.6%, 50.1%, and 57.79% on the validation set, respectively. [59]
utilized only CE and AU and achieved a promising performance of 82.6%. [91]
took the audio information as another modal and achieved 63.0%. Following
[59], we only take the CE and AU as input. With the help of our proposed two
attentive pooling modules to reduce the influence of the noisy features, our
APViT outperforms these state-of-the-art methods with an accuracy of 86.03% on
the validation set. And as illustrated in Fig. 6 (f), the accuracy of disgust
and fear is higher than that of SFEW and ExpW due to more training samples in
these categories.

TABLE XIII: Comparison of accuracy and computation cost against state-of-the-art methods on RAF-DB. (160, 0.9) indicates $k=160$, $r=0.9$ in APViT. Method | GFLOPs | RAF-DB
---|---|---
TransFER [10] | 15.30 | 90.91
APViT (160, 0.9) | 12.67 | 91.98
PAT-CNN [69] | 7.64 | 88.43
KTN [54] | 7.60 | 88.07
VTFF [12] | 6.08 | 88.14
MViT [13] | 5.95 | 88.62
APViT (10, 0.6) | 5.89 | 90.67
TABLE XIV: Performance comparison (%) on lighter Transformer models on RAF-DB. Model | Input | GFLOPs | RAF-DB
---|---|---|---
Hybrid-DeiT-Tiny | 1122 | 6.72 | 90.91
Hybrid-DeiT-Tiny + AP | 1122 | 6.10 | 91.26
DeiT-Tiny | 2242 | 1.26 | 85.72
DeiT-Tiny + AP | 2242 | 0.96 | 86.70

###  4.5 Further Analysis

Accuracy and computation comparison with state-of-the-art methods. To better
demonstrate the effectiveness of our proposed attentive pooling modules, we
illustrate the GFLOPs and accuracy on RAF-DB in Tab. XIII. PAT-CNN [69] and
KTN [54] are CNN-based methods; TransFER [10], VTFF [12], and MViT [13] are
based on the Transformer. Our APViT (160, 0.9) achieves the best performance
(91.98%) with 82.81% FlOPs compared with TransFER. And when we pool more
patches and tokens to keep comparable FLOPs with other methods, our APViT (10,
0.6) outperforms them with 2.05% higher accuracy. This indicate that our
attentive pooling module drops background and noises and forces the model to
only focus on discriminative information.

Experiments on Lighter Transformer models. Tab. XIV shows the results on RAF-
DB by exploring our two AP modules to two versions of lighter transformer
models. The Hybrid-DeiT-Tiny model replaces the ViT-Small in APViT with DeiT-
Tiny [43]. Our AP reports 0.35% and 0.98% performance improvements and 9.18%,
23.68% computation reduction on Hybrid-DeiT-Tiny and DeiT-Tiny, respectively.
The $k$ and $r$ for Hybrid-DeiT are 120 and 0.9, respectively; and the $r$ for
DeiT-Tiny is 0.8. Without the APP module, the ATP in DeiT-Tiny needs to pool
more noisy features to get the best performance.

Experiments on CIFAR-10. Most FER methods might not be suitable for general
image classification tasks. This is mainly because face images are cropped and
aligned before input into the FER model, making faces’ position fixed and
unified. While for general image classification tasks, foreground objects may
occur at any place and scale. To investigate the generalization ability of our
APViT model, we conduct experiments on CIFAR-10 [92]. As shown in Tab. XV, our
APViT maintains the performance with only 54.07% computation costs and could
slightly increase the performance from 96.02% to 96.23%, indicating the good
generalization of the proposed method.

TABLE XV: Performance and GFLOPs comparison (%) on CIFAR-10. $k$ | $r$ | CIFAR-10 | FLOPs
---|---|---|---
baseline | 96.02 | 100%
120 | 0.9 | 96.23 | 70.85%
80 | 0.6 | 96.11 | 54.07%

Attention Visualization. In Fig. 7, we visualize the attention map generated
by our APViT on RAF-DB. The first column is the input images, and the second
to fourth columns are reserved patches with our APP with different keep
numbers. The pooled patches are erased to white background. The last two
columns are the attention maps extracted by the method [93] to demonstrate the
focus area of the model without and with the proposed AP modules. The selected
images have complex conditions with occlusion and pose variation, varying with
race and age. We can find that these attention maps are robust and accurate,
and the pooled areas are noisy for the FER task. Thus 80 patches are enough to
represent the facial expression. Compared with the baseline (w/o AP), AP
modules could guide the model to pool the occluded and background areas and
only focus on the distinguished ones, _e.g._ the hand in the first row and the
chin in the second row. These visualizations also indicate that our approach
is meaningful theoretically.

Figure 7: Visualization of attention maps in complex conditions. The second to
fourth columns demonstrate attention maps in APP with different keep numbers
of 160, 120, and 80, respectively. The last two columns are the attention maps
in ATP without and with our proposed two AP modules, indicating the focused
areas by Transformer.

##  5 Conclusion

In this paper, we have proposed two Attentive Pooling (AP) modules with Vision
Transformer (APViT) to utilize the pre-trained Transformer model for the
limited size of the Facial Expression Recognition dataset. The proposed APViT
could focus on the most discriminative features and discard the less relevant
ones. This could prevent the model from focusing on occlusion or other noisy
areas. Experiments on six major in-the-wild FER datasets demonstrated that our
APViT model outperforms the state-of-the-art methods. Visualization showed the
intuition and robustness of our attentive poolings. We believe that decreasing
the number of tokens is a promising direction to pursue. We hope our work can
further inspire researchers to explore this paradigm with more effective
methods.

## References

  * [1] C. Darwin and P. Prodger, _The expression of the emotions in man and animals_. Oxford University Press, USA, 1998.
  * [2] C. Shan, S. Gong, and P. W. McOwan, “Facial expression recognition based on local binary patterns: A comprehensive study,” _Image and vision Computing_ , vol. 27, no. 6, pp. 803–816, 2009.
  * [3] A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,” _IEEE T-AC_ , vol. 10, no. 1, pp. 18–31, 2017.
  * [4] S. Li, W. Deng, and J. Du, “Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild,” in _CVPR_ , vol. 28. IEEE, Jul. 2017, pp. 2584–2593.
  * [5] J. Cai, Z. Meng, A. S. Khan, Z. Li, J. O’Reilly, and Y. Tong, “Island Loss for Learning Discriminative Features in Facial Expression Recognition,” in _2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018)_ , May 2018, pp. 302–309.
  * [6] Y. Li, J. Zeng, S. Shan, and X. Chen, “Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism,” _IEEE T-IP_ , vol. 28, no. 5, pp. 2439–2450, 2019.
  * [7] J. Zeng, S. Shan, and X. Chen, “Facial expression recognition with inconsistently annotated datasets,” in _ECCV_ , 2018, pp. 222–237.
  * [8] K. Wang, X. Peng, J. Yang, D. Meng, and Y. Qiao, “Region attention networks for pose and occlusion robust facial expression recognition,” _IEEE T-IP_ , vol. 29, pp. 4057–4069, 2020.
  * [9] Y. Fan, V. Li, and J. C. Lam, “Facial Expression Recognition with Deeply-Supervised Attention Network,” _IEEE T-AC_ , pp. 1–1, 2020.
  * [10] F. Xue, Q. Wang, and G. Guo, “TransFER: Learning Relation-aware Facial Expression Representations with Transformers,” in _ICCV_ , Mar. 2021.
  * [11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [12] F. Ma, B. Sun, and S. Li, “Facial expression recognition with visual transformers and attentional selective fusion,” _IEEE T-AC_ , pp. 1–1, 2021\.
  * [13] H. Li, M. Sui, F. Zhao, Z. Zha, and F. Wu, “MViT: Mask Vision Transformer for Facial Expression Recognition in the wild,” _arXiv preprint arXiv:2106.04520_ , Jun. 2021.
  * [14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _CVPR_ , Jun. 2009, pp. 248–255.
  * [15] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [16] G. Ghiasi, T.-Y. Lin, and Q. V. Le, “Dropblock: A regularization method for convolutional networks,” in _NeurIPS_ , 2018.
  * [17] Z. Tan, Y. Yang, J. Wan, H. Hang, G. Guo, and S. Z. Li, “Attention-based pedestrian attribute analysis,” _T-IP_ , vol. 28, no. 12, pp. 6126–6140, 2019.
  * [18] X. Chu, Z. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen, “Conditional Positional Encodings for Vision Transformers,” _arXiv preprint arXiv:2102.10882_ , feb 2021.
  * [19] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng, “DeepViT: Towards Deeper Vision Transformer,” _arXiv preprint arXiv:2103.11886_ , mar 2021.
  * [20] Z. Jiang, Q. Hou, L. Yuan, D. Zhou, Y. Shi, X. Jin, A. Wang, and J. Feng, “All Tokens Matter: Token Labeling for Training Better Vision Transformers,” in _NeurIPS_ , 2021.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [22] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [23] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [24] J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, “RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild,” in _CVPR_ , 2020, pp. 5203–5212.
  * [25] C. Liu and H. Wechsler, “Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition,” _IEEE T-IP_ , vol. 11, no. 4, pp. 467–476, 2002.
  * [26] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _CVPR_ , vol. 1. IEEE, 2005, pp. 886–893.
  * [27] P. C. Ng and S. Henikoff, “Sift: Predicting amino acid changes that affect protein function,” _Nucleic acids research_ , vol. 31, no. 13, pp. 3812–3814, 2003.
  * [28] S. Happy and A. Routray, “Automatic facial expression recognition using features of salient facial patches,” _IEEE T-AC_ , vol. 6, no. 1, pp. 1–12, 2014.
  * [29] D. Ghimire and J. Lee, “Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines,” _Sensors_ , vol. 13, no. 6, pp. 7714–7734, 2013.
  * [30] A. Saeed, A. Al-Hamadi, R. Niese, and M. Elzobi, “Frame-based facial expression recognition using geometrical features,” _Advances in human-computer interaction_ , vol. 2014, 2014.
  * [31] A. Katharopoulos and F. Fleuret, “Processing megapixel images with deep attention-sampling models,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 3282–3291.
  * [32] J.-B. Cordonnier, A. Mahendran, A. Dosovitskiy, D. Weissenborn, J. Uszkoreit, and T. Unterthiner, “Differentiable patch selection for image recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2351–2360.
  * [33] J. Wang, X. Yang, H. Li, Z. Wu, and Y.-G. Jiang, “Efficient video transformers with spatial-temporal token selection,” _arXiv preprint arXiv:2111.11591_ , 2021.
  * [34] G. Duchenne de Bologne, “Mechanisme de la physionomie humaine,” _Jules Renouard Libraire. Paris_ , 1862.
  * [35] L. Zhong, Q. Liu, P. Yang, B. Liu, J. Huang, and D. N. Metaxas, “Learning active facial patches for expression analysis,” in _CVPR_ , 2012, pp. 2562–2569.
  * [36] S. Xie, H. Hu, and Y. Wu, “Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition,” _Pattern Recognition_ , vol. 92, pp. 177–191, 2019.
  * [37] Z. Zhao, Q. Liu, and S. Wang, “Learning Deep Global Multi-scale and Local Attention Features for Facial Expression Recognition in the Wild,” _IEEE T-IP_ , pp. 1–1, 2021.
  * [38] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _ECCV_ , 2018, pp. 3–19.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _arXiv preprint arXiv:1706.03762_ , 2017.
  * [40] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-End Object Detection with Transformers,” in _ECCV_. Springer, May 2020.
  * [41] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” _arXiv preprint arXiv:2010.04159_ , 2020.
  * [42] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era,” in _ICCV_ , 2017, pp. 843–852.
  * [43] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” _arXiv preprint arXiv:2012.12877_ , 2020.
  * [44] S. D’Ascoli, H. Touvron, M. L. Leavitt, A. S. Morcos, G. Biroli, and L. Sagun, “ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases,” in _ICML_. PMLR, 2021, pp. 2286–2296.
  * [45] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” in _ICCV_.
  * [46] M. Aouayeb, W. Hamidouche, C. Soladie, K. Kpalma, and R. Seguier, “Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition,” _arXiv:2107.03107 [cs]_ , Jul. 2021.
  * [47] Q. Wang, T. Wu, H. Zheng, and G. Guo, “Hierarchical pyramid diverse attention networks for face recognition,” _CVPR_ , pp. 8323–8332, 2020.
  * [48] S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” in _ICLR_. International Conference on Learning Representations, ICLR, dec 2017. [Online]. Available: https://github.com/szagoruyko/attention-transfer.
  * [49] E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” in _Proceedings of the 18th ACM International Conference on Multimodal Interaction_ , 2016, pp. 279–283.
  * [50] A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” _ICCV_ , pp. 2106–2112.
  * [51] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “From Facial Expression Recognition to Interpersonal Relation Prediction,” _IJCV_ , vol. 126, no. 5, pp. 550–569.
  * [52] D. Kollias, I. Kotsia, E. Hajiyev, and S. Zafeiriou, “Analysing affective behavior in the second abaw2 competition,” _arXiv preprint arXiv:2106.15318_ , 2021.
  * [53] I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee _et al._ , “Challenges in representation learning: A report on three machine learning contests,” in _International conference on neural information processing_. Springer, 2013, pp. 117–124.
  * [54] H. Li, N. Wang, X. Ding, X. Yang, and X. Gao, “Adaptively learning facial expression representation via cf labels and distillation,” _IEEE T-IP_ , vol. 30, pp. 2016–2028, 2021.
  * [55] A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Acted facial expressions in the wild database,” _Australian National University, Canberra, Australia, Technical Report TR-CS-11_ , vol. 2, p. 1, 2011.
  * [56] M. Bishay, P. Palasek, S. Priebe, and I. Patras, “SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis,” _IEEE T-AC_ , vol. 12, no. 4, pp. 949–961.
  * [57] D. Kollias and S. Zafeiriou, “Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface,” _arXiv preprint arXiv:1910.04855_ , 2019.
  * [58] S. Zafeiriou, D. Kollias, M. A. Nicolaou, A. Papaioannou, G. Zhao, and I. Kotsia, “Aff-wild: Valence and arousal ‘in-the-wild’challenge,” in _CVPR Workshops_. IEEE, 2017, pp. 1980–1987.
  * [59] P. T. D. Thinh, H. M. Hung, H.-J. Yang, S.-H. Kim, and G.-S. Lee, “Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique.”
  * [60] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _CVPR_ , 2019, pp. 4690–4699.
  * [61] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “Ms-celeb-1m: A dataset and benchmark for large-scale face recognition,” in _ECCV_. Springer, 2016, pp. 87–102.
  * [62] I. Loshchilov and F. Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts,” _ICLR_ , aug 2016.
  * [63] S. Chen, J. Wang, Y. Chen, Z. Shi, X. Geng, and Y. Rui, “Label Distribution Learning on Auxiliary Label Space Graphs for Facial Expression Recognition,” in _CVPR_ , 2020, pp. 13 984–13 993.
  * [64] D. Acharya, Z. Huang, D. Pani Paudel, and L. Van Gool, “Covariance pooling for facial expression recognition,” in _CVPR Workshops_ , 2018, pp. 367–374.
  * [65] K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” in _CVPR_ , 2020, pp. 6897–6906.
  * [66] W. Zhang, X. Ji, K. Chen, Y. Ding, and C. Fan, “Learning a Facial Expression Embedding Disentangled From Identity,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6759–6768.
  * [67] A. H. Farzaneh and X. Qi, “Facial expression recognition in the wild via deep attentive center loss,” in _WACV_ , 2021, pp. 2402–2411.
  * [68] Z. Zhao, Q. Liu, and F. Zhou, “Robust lightweight facial expression recognition network with label distribution training,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, 2021, pp. 3510–3519.
  * [69] J. Cai, Z. Meng, A. S. Khan, Z. Li, J. O’Reilly, and Y. Tong, “Probabilistic Attribute Tree Structured Convolutional Neural Networks for Facial Expression Recognition in the Wild,” _IEEE T-AC_ , pp. 1–1.
  * [70] D. Ruan, Y. Yan, S. Lai, Z. Chai, C. Shen, and H. Wang, “Feature decomposition and reconstruction learning for effective facial expression recognition,” in _CVPR_ , 2021, pp. 7660–7669.
  * [71] J. Jiang and W. Deng, “Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher,” _IEEE T-AC_ , pp. 1–1.
  * [72] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in _European conference on computer vision_. Springer, 2016, pp. 499–515.
  * [73] Z. Liu, S. Li, and W. Deng, “Boosting-poof: boosting part based one vs one feature for facial expression recognition in the wild,” in _2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)_. IEEE, 2017, pp. 967–972.
  * [74] M. Li, H. Xu, X. Huang, Z. Song, X. Liu, and X. Li, “Facial expression recognition with identity and emotion joint learning,” _IEEE T-AC_ , vol. 12, no. 2, pp. 544–550, 2021.
  * [75] S. Albanie, A. Nagrani, A. Vedaldi, and A. Zisserman, “Emotion recognition in speech using cross-modal transfer in the wild,” in _Proceedings of the 26th ACM international conference on Multimedia_ , 2018, pp. 292–301.
  * [76] B.-K. Kim, H. Lee, J. Roh, and S.-Y. Lee, “Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition,” in _Proceedings of the 2015 ACM on International Conference on Multimodal Interaction_ , ser. ICMI ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 427–434. [Online]. Available: https://doi.org/10.1145/2818346.2830590
  * [77] H. Li, N. Wang, Y. Yu, X. Yang, and X. Gao, “Lban-il: A novel method of high discriminative representation for facial expression recognition,” _Neurocomputing_ , vol. 432, pp. 159–169, 2021.
  * [78] Z. Li, S. Han, A. S. Khan, J. Cai, Z. Meng, J. O’Reilly, and Y. Tong, “Pooling map adaptation in convolutional neural network for facial expression recognition,” in _2019 IEEE international conference on multimedia and expo (ICME)_. IEEE, 2019, pp. 1108–1113.
  * [79] Z. Wen, W. Lin, T. Wang, and G. Xu, “Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition.”
  * [80] J. Weng, Y. Yang, Z. Tan, and Z. Lei, “Attentive hybrid feature with two-step fusion for facial expression recognition,” in _2020 25th International Conference on Pattern Recognition (ICPR)_ , 2021, pp. 6410–6416.
  * [81] Z. Lian, Y. Li, J.-H. Tao, J. Huang, and M.-Y. Niu, “Expression Analysis Based on Face Regions in Real-world Conditions,” _International Journal of Automation and Computing_ , vol. 17, no. 1, pp. 96–107.
  * [82] T. T. D. Pham and C. S. Won, “Facial Action Units for Training Convolutional Neural Networks,” _IEEE Access_ , vol. 7, pp. 77 816–77 824.
  * [83] W. Zhang, Z. Guo, K. Chen, L. Li, Z. Zhang, Y. Ding, R. Wu, T. Lv, and C. Fan, “Prior aided streaming network for multi-task affective analysis,” in _ICCV Workshops_ , 2021, pp. 3532–3542.
  * [84] Y. Jin, T. Zheng, C. Gao, and G. Xu, “MTMSN: Multi-Task and Multi-Modal Sequence Network for Facial Action Unit and Expression Recognition,” in _ICCV Workshops_ , pp. 3590–3595.
  * [85] L. Wang, S. Wang, J. Qi, and K. Suzuki, “A multi-task mean teacher for semi-supervised facial affective behavior analysis,” 2021.
  * [86] D. Deng, L. Wu, and B. E. Shi, “Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition,” in _ICCV Workshops_ , pp. 3557–3566.
  * [87] F. Song, X. Tan, X. Liu, and S. Chen, “Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients,” _Pattern Recognition_ , vol. 47, no. 9, pp. 2825–2838, 2014.
  * [88] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _ICCV_ , 2015, pp. 3730–3738.
  * [89] C. Fabian Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, “Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild,” in _CVPR_ , 2016, pp. 5562–5570.
  * [90] W. Zhang, Z. Guo, K. Chen, L. Li, Z. Zhang, and Y. Ding. Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition.
  * [91] Y. Jin, T. Zheng, C. Gao, and G. Xu. A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition.
  * [92] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [93] H. Chefer, S. Gur, and L. Wolf, “Transformer interpretability beyond attention visualization,” in _CVPR_ , 2021, pp. 782–791.