# Does the Reionization Model Influence the Constraints on Dark Matter Decay
or Annihilation?

Lu Chen1<EMAIL_ADDRESS>and Ke Wang2<EMAIL_ADDRESS>333Corresponding author 1 School of Physics and Electronics,
Shandong Normal University, Jinan 250014, China
2 Institute of Theoretical Physics & Research Center of Gravitation,
Lanzhou University, Lanzhou 730000, China

###### Abstract

If dark matter decay or annihilate, a large amount of energy and particles
would be released into the cosmic plasma. Therefore, they could modify the
thermal and ionization history of our universe, then leave footprints on the
cosmic microwave background power spectra. In this paper, we take dark matter
annihilation as an example and investigate whether different reionization
models influence the constraints on dark matter annihilation. We consider the
ionization history including both dark matter annihilation and star formation,
then put constraints on DM annihilation. Combining the latest Planck data, BAO
data, SNIa measurement, $Q_{\text{HII}}$ constraints from observations of
quasars, as well as the star formation rate density from UV and IR data, the
optical depth is $\tau=0.0571^{+0.0005}_{-0.0006}$ at 68$\%$C.L. and the upper
limit of $\epsilon_{0}f_{d}$ reads $2.7765\times 10^{-24}$ at 95$\%$C.L.. By
comparison, we also constrain dark matter annihilation in the instantaneous
reionization model from the same data combination except the $Q_{\text{HII}}$
constraints and star formation rate density. We get
$\tau=0.0559^{+0.0069}_{-0.0076}$ at 68$\%$C.L. and the upper limit of
$\epsilon_{0}f_{d}$ is $2.8468\times 10^{-24}$ at 95$\%$C.L.. This indicates
various reionization models have little influence ($\lesssim 2.5\%$) on
constraining parameters of dark matter decay or annihilation.

###### pacs:

???

##  I Introduction

The existence of dark matter (DM) has been confirmed by cosmological
observations, for example, gravitational lensing of clusters and galaxies
Hoekstra:2002nf ; Koopmans:2002qh ; Metcalf:2003sz ; Moustakas:2002iz , the
large-scale structure formation Davis:1985rj ; Navarro:1995iw ; Navarro:2003ew
; Davis:1992ui ; Blumenthal:1984bp , rotation curves of galaxies
vandenBosch:2000rza ; Begeman:1991iy ; deBlok:2001hbg , the cosmic microwave
background (CMB) Ade:2015xua ; Aghanim:2018eyx  and so on Jungman:1995df ;
Bertone:2004pz ; Clowe:2006eq ; Peter:2012rz . However, its nature is still a
puzzle after decades of researches and observations Lewin:1995rx ;
Gaskins:2016cha ; Kahlhoefer:2017dnp ; Freese:2012xd ; Cirelli:2012tf . Many
theories predict that DM particles can decay or annihilate into standard model
particles (e.g. photons, protons, electrons and so on) which are well known in
particle physics Geringer-Sameth:2014yza ; Steigman:2012nb ; Gondolo:1999ef ;
Galli:2009zc ; Galli:2011rz . It is clear that the thermal and ionization
history of our universe would be influenced because annihilating or decay DM
release extra particles and energy into the cosmic plasma Zhang:2006fr ;
Dvorkin:2013cga ; Weniger:2013hja ; Chluba:2009uv . If it occurs during the
epoch of recombination, DM decay or annihilation would lead to extra
ionizations and excitations, as well as the increase of plasma temperature. As
a result, recombination would be modified. Moreover, DM decay or annihilation
occurred during the reionization epoch could be an extra ionized source
besides star formation. Eventually, the whole process would leave footprints
on the CMB power spectra and provide a method to constrain DM using CMB
observations. Actually, the recombination redshift $z_{*}\sim 1089$ is
constrained well from observational data in various recombination models
Ade:2015xua ; Aghanim:2018eyx . On the other hand, observations of the Gunn-
Peterson effect in the spectra of high-redshift quasars reveal that the
reionization epoch of our universe ended before redshift $z\sim 6$
Robertson:2015uda ; Gruppioni:2011vi ; Chornock:2013una ; Madau:2014bja . But
the history during $z\sim 6-1089$ is still under debate, especially the onset
and process of reionization.

Although previous works indicate that DM decay or annihilation contributed
much less ionized photons than stars and galaxies during the epoch of
reionization Lopez-Honorez:2013cua ; Carucci:2018tzf ; Liu:2016cnk ;
Mapelli:2006ej ; Hazra:2018eib ; Paoletti:2020ndu , that occurred during
recombination has the ability to influence the reionization history by
providing the initial conditions. The most popular reionization model is the
instantaneous reionization model with a simple parameterization of hyperbolic
tangent function $\tanh$ (named the TANH model hereafter) 444 In the TANH
model, the ionized fraction $x_{e}=N_{e}/N_{\text{H}}\simeq
x_{\text{p}}+x_{\text{HeII}}$ is assumed to be Lewis:2008wr
$\displaystyle
x_{e}=\dfrac{1+N_{\text{He}}/N_{\text{H}}}{2}\left[1+\tanh\left(\dfrac{y(z_{\text{re}})-y(z)}{\Delta
y}\right)\right],$
where $y(z)=(1+z)^{3/2}$, $\Delta y=\dfrac{3}{2}(1+z_{\text{re}})^{1/2}\Delta
z$, $\Delta z=0.5$. And the optical depth is
$\displaystyle\tau(z)=\int_{0}^{z}x_{e}N_{\text{H}}\sigma_{\text{T}}H^{-1}(z^{\prime})(1+z^{\prime})^{-1}dz^{\prime}.$
. Planck Collaboration gave the reionization optical depth $\tau=0.054\pm
0.007$ at 68$\%$C.L. in 2018 (hereafter Planck 2018) Aghanim:2018eyx . An
extended model named Poly-reion model Hazra:2017gtx  has also been well
studied. The ionized fraction is patameterized by a polynomial function and
higher values of $\tau$ are acquired, which are similar with values obtained
using the principle-component analysis (PCA) Heinrich:2016ojb . However, there
is another much more direct approach to study the cosmic reionization history
via star formation history (SFH). Since early stars and galaxies are
considered as ionized sources of reionization Robertson:2015uda ;
Khusanova:2019cxr ; Madau:2014bja ; Robertson:2010an ; Robertson:2013bq , we
can reconsider the reionization history using star formation rate (SFR)
density from UV and IR data Madau:2014bja  at lower redshifts directly (named
the SFH model hereafter). In this paper, we take DM annihilation into
consideration and put constraints on parameters in the TANH and SFH
reionization models from observational data.

This paper is organized as follows. In section II, the ionization history of
universe is reconsidered including DM annihilation and SFR rate. In subsection
II.1, we review the modified ionized fraction after taking DM annihilation
into account. In subsection II.2, the SFR density and its influence on the
ionization of the intergalactic medium (IGM) during the epoch of reionization
are presented. In subsection II.3, we give the required equations for
calculating the optical depth. Our constraints on DM annihilation are shown in
section III. In the SFH model, we make use of the data combination including
CMB measurement from Planck 2018 Aghanim:2018eyx , SFR density from UV data
and IR data Madau:2014bja , $Q_{\text{HII}}$ constraints from observations of
quasars Bouwens:2015vha ; Fan:2005es ; Schenker:2014tda , baryon acoustic
oscillation (BAO) data Beutler:2011hx ; Ross:2014qpa ; Alam:2016hwk , as well
as the measurement of Type Ia supernova Scolnic:2017caz . In the TANH model,
the same data combination excluding SFR density is used. Finally, a brief
summary is included in section IV.

##  II Ionization History

###  II.1 Effects of Annihilating Dark Matter on Recombination and
Reionization

In this subsection, we give a brief summary of the ionized fraction related to
DM annihilation Chluba:2009uv . Considering self-annihilating DM particles
$\chi$ and their anti-particles $\bar{\chi}$, the total energy rate released
via annihilation per unit volume usually depends on the involved annihilation
channels, mass of DM particles $M_{\chi}=M_{\bar{\chi}}$ and their averaged
cross-section $<\sigma v>$. But only a fraction of them can be deposited into
the cosmic plasma. The energy deposition efficiency $f_{\text{d}}$ is related
to redshift $z$ and cosmological models. In brief, the total rate of energy
deposited to the cosmic plasma per unit volume is given by

$\displaystyle\dfrac{dE_{\text{d}}}{dt}\Big{|}_{\chi\bar{\chi}}=\epsilon_{0}f_{\text{d}}N_{\text{H}}(z)(1+z)^{3}\text{eV}\cdot\text{s}^{-1},$
(1)

where $N_{\text{H}}(z)$ is the comoving number density of hydrogen nuclei in
our universe and $\epsilon_{0}$ is a dimensionless parameter given by

$\displaystyle\epsilon_{0}=1.5\times
10^{-24}\Big{[}\dfrac{M_{\chi}c^{2}}{100\text{GeV}}\Big{]}^{-1}\Big{[}\dfrac{\Omega_{\chi}h^{2}}{0.13}\Big{]}^{2}\Big{[}\dfrac{<\sigma
v>}{3\times 10^{-26}\text{cm}^{3}\cdot\text{s}^{-1}}\Big{]},$ (2)

where $c$ is the speed of light, $\Omega_{\chi}h^{2}$ is the physical density
of DM today. In this paper, $\epsilon_{0}f_{\text{d}}$ is regarded as a
constant roughly.

The energy absorbed by the cosmic plasma will heat, ionize and excite atoms.
To measure the effects of annihilating DM on the ionization history, the
evolution equations for the net ionization rate from ground states of neutral
hydrogen and helium results from DM annihilation, as well as the plasma
temperature evolution equation, are written as Chluba:2009uv ; Lopez-
Honorez:2013cua

$\displaystyle\dfrac{dx_{\text{p}}}{dt}\Big{|}_{\chi\bar{\chi}}$
$\displaystyle=$
$\displaystyle-\dfrac{1}{N_{\text{H}}[1+f_{\text{He}}]}\dfrac{g_{\text{p}}(z)}{E^{\text{HI}}_{\text{ion}}}\dfrac{dE_{d}}{dt}\Big{|}_{\chi\bar{\chi}}\,,$
$\displaystyle\dfrac{dx_{\text{HeII}}}{dt}\Big{|}_{\chi\bar{\chi}}$
$\displaystyle=$
$\displaystyle-\dfrac{f_{\text{He}}}{N_{\text{H}}[1+f_{\text{He}}]}\dfrac{g_{\text{He}}(z)}{E^{\text{HeI}}_{\text{ion}}}\dfrac{dE_{d}}{dt}\Big{|}_{\chi\bar{\chi}}\,,$
$\displaystyle\dfrac{dT_{\text{m}}}{dt}\Big{|}_{\chi\bar{\chi}}$
$\displaystyle=$
$\displaystyle\dfrac{1}{N_{\text{H}}[1+f_{\text{He}}+x_{\text{e}}(z)]}\dfrac{2g_{\text{h}}(z)}{3k_{\text{B}}}\dfrac{dE_{d}}{dt}\Big{|}_{\chi\bar{\chi}}\,,$

where $x_{\text{p}}\equiv N_{\text{p}}/N_{\text{H}}$, $x_{\text{HeII}}\equiv
N_{\text{HeII}}/N_{\text{H}}$ are the ionized fraction from the ground states
of hydrogen and helium related to DM annihilation respectively,
$f_{\text{He}}\equiv N_{\text{He}}/N_{\text{H}}\simeq 0.08$ is the fraction of
helium, $E^{\text{HI}}_{\text{ion}}=13.6\ \text{eV}$ and
$E^{\text{HeI}}_{\text{ion}}=24.6\ \text{eV}$ are the ionization potentials of
hydrogen and helium respectively, $k_{\text{B}}$ is the Boltzmann constant.
$g_{\text{h}}$, $g_{\text{p}}$ and $g_{\text{He}}$ represent the partial
contributions of the total deposited energy going into heating the plasma,
ionizations of hydrogen and helium. Here we use the following approximations,

$\displaystyle g_{\text{p}}$ $\displaystyle\approx$
$\displaystyle\dfrac{1-x_{\text{p}}(z)}{3}\,,$ $\displaystyle g_{\text{He}}$
$\displaystyle\approx$ $\displaystyle\dfrac{1-Z_{\text{HeII}}(z)}{3}\,,$
$\displaystyle g_{\text{h}}$ $\displaystyle\approx$
$\displaystyle\dfrac{1}{3}\left(1+\dfrac{2x_{\text{e}}(z)}{1+f_{\text{He}}}\right)\,,$

where $Z_{\text{HeII}}=N_{\text{HeII}}/N_{\text{He}}$ is the fraction of
singly ionized helium atoms. For simplification, $T_{\text{m}}$ is assumed to
be a constant during the epoch of reionization, which is different from the
redshift-dependent temperature $T_{\text{m}}$ during recombination . This is
reasonable because the duration of recombination is much longer than
reionization and the change of $T_{\text{m}}$ during reionization epoch is
small enough to be ignored. Notice that we have ignored the contribution of
excitations because electrons are tend to stay at lower levels and it has a
weak influence relative to ionizations Lopez-Honorez:2013cua ; Galli:2011rz ;
Hutsi:2011vx .

###  II.2 Effects of Star Formation on Reionization

On the other hand, the mainstream considers early star formation of galaxies
most likely supplies the ionizing photons for the reionization of our universe
Madau:2014bja ; Robertson:2015uda . Naturally, SFR involves significant
information on the onset and process of reionization. We use the following
parameterization of cosmic SFR density $\rho_{\text{SFR}}$ (in units of
$\text{M}_{\odot}\cdot\text{yr}^{-1}\cdot\text{Mpc}^{-3}$) Madau:2014bja ;
Bowman:2018yin ,

$\displaystyle\rho_{\text{SFR}}=\dfrac{a_{\text{p}}(1+z)^{b_{\text{p}}}}{1+[(1+z)/c_{\text{p}}]^{d_{\text{p}}}}-\dfrac{a_{\text{p}}(1+22)^{b_{\text{p}}}}{1+[(1+22)/c_{\text{p}}]^{d_{\text{p}}}},$
(5)

with $a_{\text{p}}$, $b_{\text{p}}$, $c_{\text{p}}$ and $d_{\text{p}}$ are the
four coefficients which ought to be determined by observational data. The
second term results from our assumption that there’s no star formation before
$z\sim 22$. We choose the value of 22 because there is almost no 21 cm
absorption signal before $z\sim 22$ as the figure of best-fit 21 cm absorption
profiles shows in Bowman:2018yin . This parameterization not only can provide
the bump at about $z\sim 2$, but also behaves as
$\log\rho_{\text{SFR}}\rightarrow-\infty$ near the starting redshift of
reionization. Therefore, $\log\rho_{\text{SFR}}(z\gtrsim 10)$ performs a rapid
downtrend which is consistent with Fig. 8 of Ishigaki_2018 . If Lyman
continuum photons released from star forming galaxies play a dominant role in
the reionization, their influence on the ionized rate depends on the photon
production rate $\xi_{\text{ion}}$ and the escaping rate $f_{\text{esc}}$ of
galaxies besides $\rho_{\text{SFR}}$, which can be written as the growth rate
of number density for ionized photons,

$\displaystyle\dot{N}_{\text{ion}}=f_{\text{esc}}\xi_{\text{ion}}\rho_{\text{SFR}},$
(6)

with the overdot indicates the differentiation of cosmic time $t$. Here, we
adopt the fiducial values $f_{\text{esc}}=0.2$ and
$\log_{10}\xi_{\text{ion}}=53.14\
[\text{Lyc}\cdot\text{photons}\cdot\text{s}^{-1}\cdot\text{M}_{\odot}^{-1}\cdot\text{yr}]$.

###  II.3 Thomson Optical Depth

The Tomson optical depth is given by

$\displaystyle\tau(z)=c\sigma_{\text{T}}\int_{0}^{z}(1+f_{\text{He}})Q_{\text{HII}}(z^{\prime})N_{\text{H}}(z^{\prime})(1+z^{\prime})^{-1}H^{-1}(z^{\prime})dz^{\prime},$
(7)

where $c$ is the speed of light, $\sigma_{\text{T}}$ is the Thomson scattering
cross section, $H(z)$ is the redshift-dependent Hubble parameter. And
$Q_{\text{HII}}$ indicates the ionized fraction (equivalent to $x_{\text{p}}$
physically) which can be calculated by evolving the differential equation

$\displaystyle\dot{Q}_{\text{HII}}=-\dfrac{Q_{\text{HII}}}{t_{\text{rec}}}+\dfrac{dx_{\text{p}}}{dt}\Big{|}_{\chi\bar{\chi}}+\dfrac{\dot{N}_{\text{ion}}}{N_{\text{H}}}.$
(8)

Here $t_{\text{rec}}$ is the recombination time as follows Bouwens:2015vha

$\displaystyle t_{\text{rec}}=0.88\
\text{Gyr}\left(\dfrac{1+z}{7}\right)^{-3}\left(\dfrac{T_{0}}{2\times
10^{4}\text{K}}\right)^{-0.7}\left(\dfrac{C_{\text{HII}}}{3}\right)^{-1},$ (9)

where $T_{0}\sim 2\times 10^{4}$ K is the temperature of the ionizing hydrogen
gas and $C_{\text{HII}}\sim 3$ is the clumping factor of ionized hydrogen. The
second term of Eq. 8 represents the effect of DM annihilation and the last
term represents the effect of SFR. Note we have ignored the contribution of He
ionization in the second equation of Eq.II.1. The initial value of
$Q_{\text{HII}}$ at the start of reionization depends on DM annihilation
occurred during the recombination epoch and that is why we can constrain
$\epsilon_{0}f_{\text{d}}$.

##  III Results

In this section, we use the observational data to reconstruct the ionization
history of our universe and put constraints on DM annihilation. CAMB
Lewis:1999bs  and CosmoRec Shaw:2011ez  are used to calculate the optical
depth and CMB power spectra. Then the Markov Chain Monte Carlo sampler—CosmoMC
Lewis:2002ah ; Lewis:2013hha  is applied to constrain the free parameters
using the maximum likelihood determination. In the SFH model, we refer to the
data combination of Planck 2018 TT,TE,EE$+$lowE$+$lensing Aghanim:2018eyx ,
baryon acoustic oscillation (BAO) data at redshifts
$z=0.106,0.15,0.32,0.57,1.52$ (named 6dFGS Beutler:2011hx , MGS Ross:2014qpa ,
DR12 Alam:2016hwk  respectively), the latest SNIa measurement (named Pantheon)
Scolnic:2017caz , $Q_{\text{HII}}$ constraints between $5.0\leq z\leq 8.0$
from observations of Gunn-Peterson optical depth and prevalence of Ly-$\alpha$
emission in galaxies Bouwens:2015vha ; Fan:2005es ; Schenker:2014tda ,as well
as the SFR density from UV and IR data Madau:2014bja . Notice that the SFR
data used here extend to $z\sim 8$. Actually, Bouwens:2015vha ; Ishigaki_2018
provide UV data extending to $z\sim 10$, but the uncertainties of the data are
large between $8<z<10$. So our choice would not impact the results
significantly. There are ten free parameters in this model,
$\\{\Omega_{\text{b}}h^{2},\Omega_{\text{c}}h^{2},100\theta_{\text{MC}},\text{ln}\left(10^{10}A_{\text{s}}\right),n_{\text{s}},10^{-23}\epsilon_{0}f_{\text{d}},a_{\text{p}},b_{\text{p}},c_{\text{p}},d_{\text{p}}\\}$.
Here five of them are parameters from the $\Lambda$CDM model:
$\Omega_{\text{b}}h^{2}$ and $\Omega_{\text{c}}h^{2}$ are the density of
baryons and cold dark matter today respectively, $100\theta_{\text{MC}}$ is
100 times the ratio of angular diameter distance to the large scale structure
sound horizon, $A_{\text{s}}$ is the amplitude of the power spectrum of
primordial curvature perturbations, and $n_{\text{s}}$ is the scalar spectrum
index. Besides, $\epsilon_{0}f_{\text{d}}$ and
$a_{\text{p}},b_{\text{p}},c_{\text{p}},d_{\text{p}}$ are parameters of DM
annihilation and SFR density as described in section II. For comparison, the
TANH model with previous data combination except the SFR density data has been
run, too. In this model, we have seven free parameters
$\\{\Omega_{\text{b}}h^{2},\Omega_{\text{c}}h^{2},100\theta_{\text{MC}},\text{ln}\left(10^{10}A_{\text{s}}\right),n_{\text{s}},\tau,10^{-23}\epsilon_{0}f_{\text{d}}\\}$.
Moreover, we have to mention that the optical depth $\tau$ is a free parameter
in the TANH model, but it’s a derived parameter in the SFH model because SFR
density data are used to put constraints on the reionization epoch directly,
replacing the parameterization in the TANH model.

Our results are summarized in Tab. I. We have listed the 68$\%$ limits of the
free parameters and necessary derived parameters in these two models. The
upper limit of $\epsilon_{0}f_{\text{d}}$ is $1.3309\times 10^{-24}$ at 68$\%$
C.L. and $2.7765\times 10^{-24}$ at 95$\%$ C.L. in the SFH model, while it
reads $1.4253\times 10^{-24}$ at 68$\%$ C.L. and $2.8468\times 10^{-24}$ at
95$\%$ C.L. in the TANH model. The constraint on $\epsilon_{0}f_{\text{d}}$
improves 6.6$\%$ at 68$\%$ C.L. and 2.5$\%$ at 95$\%$ C.L.. The likelihood is
shown vividly in Fig. 1. Our results reveal modifying reionization epoch has
little influence on constraining DM annihilation. In the SFH model, the
optical depth $\tau$ is $0.0571^{+0.0005}_{-0.0006}$ at 68$\%$C.L.. It is in
consistent with the value of $0.0559_{-0.0076}^{+0.0069}$ in the TANH model,
as well as $0.054\pm 0.007$ in the $\Lambda$CDM model released by Planck
Collaboration in 2018 Aghanim:2018eyx . The mean values of $\tau$ do not vary
almostly, leading to similar constraints on $\epsilon_{0}f_{\text{d}}$. This
is because they are related directly as shown in Sec. II. However, the error
bar of $\tau$ is reduced by around an order of magnitude in the SFH model.
Notice previous reionization analysis including both CMB and SFR data gave
uncertainties of $\tau$ about $\sigma_{\tau}\sim 0.001-0.002$ Gorce:2017glg ;
Krishak:2021fxp . Our error bars are about half smaller on account of the
$Q_{\text{HII}}$ constraints of Gunn-Peterson optical depth between
$5.03<z<5.85$, which provide strict restrictions of $Q_{\text{HII}}$ with
uncertainties of order $\sigma\sim 0.00001-0.00005$ Bouwens:2015vha ;
Fan:2005es . Besides, the ionized fraction $x_{e}(z)$ in these two models are
illustrated in Fig. 2. It shows that the reionization started at about $z\sim
20$ in the SFH model, which is higher than the TANH model.

Table 1: The 68$\%$ limits for the cosmological parameters in the SFH and TANH model. Note we give the upper limits of $10^{-23}\epsilon_{0}f_{\text{d}}$ at 95$\%$C.L..  |  SFH |  TANH
---|---|---
$\Omega_{\text{b}}h^{2}$ |  $0.02240\pm 0.00013$ |  $0.02245_{-0.00013}^{+0.00014}$
$\Omega_{\text{c}}h^{2}$ |  $0.1192\pm 0.0008$ |  $0.1193\pm 0.0009$
$100\theta_{\text{MC}}$ |  $1.04094_{-0.00029}^{+0.00030}$ |  $1.04095_{-0.00029}^{+0.00030}$
$\ln(10^{10}\text{A}_{\text{s}})$ |  $3.0529_{-0.0071}^{+0.0063}$ |  $3.0509\pm 0.0144$
$n_{s}$ |  $0.9675\pm 0.0036$ |  $0.9677\pm 0.0037$
$10^{-23}\epsilon_{0}f_{\text{d}}(95\%$ C.L.) |  $<0.27765$ |  $<0.28468$
$a_{\text{p}}$[$\text{M}_{\odot}\cdot\text{yr}^{-1}\cdot\text{Mpc}^{-3}$] |  $0.01769_{-0.00069}^{+0.00068}$ |  -
$b_{\text{p}}$ |  $2.967\pm 0.118$ |  -
$c_{\text{p}}$ |  $2.555_{-0.100}^{+0.087}$ |  -
$d_{\text{p}}$ |  $5.131_{-0.095}^{+0.096}$ |  -
$\tau$ |  $0.0571_{-0.0006}^{+0.0005}$ |  $0.0559_{-0.0076}^{+0.0069}$
$10^{-4}Q_{\text{HII}}(z=22)$ |  $2.666_{-0.511}^{+0.220}$ |  -
$H_{0}$[km $\cdot$ s-1 $\cdot$ Mpc-1] |  $67.66_{-0.38}^{+0.39}$ |  $67.69\pm 0.42$
Figure 1: The likelihood of $10^{-23}\epsilon_{0}f_{\text{d}}$ in the SFH and
TANH model. The blue solid curve indicates the SFH model and the red dashed
curve indicates the TANH model. Figure 2: The evolution of ionized fraction
$x_{e}(z)$ in the SFH and TANH model. The blue solid curve illustrate the SFH
model and the red dashed curve corresponds to the TANH model. Note that the
values of $\epsilon_{0}f_{\text{d}}$ are set to the upper limits at 95$\%$C.L.
and we set the other necessary parameters to their mean values in Tab. I.

##  IV Summary and discussion

In this paper, we investigate whether the reionization model influences
constraining parameters of DM annihilation or decay. Take DM annihilation as
an example. Energy released by annihilating DM would be absorbed by the cosmic
plasma, resulting extra heating, ionization and excitation of gas. Therefore,
both the recombination and reionization history of our universe are
influenced, leaving footprints on the CMB power spectra. On the other hand,
early star forming galaxies are thought to be sources of reionization epoch.
This provides a way to reconstruct the ionization history with the SFR density
directly. Combining the measurement of Planck 2018, BAO data, PANTHEON
samples, $Q_{\text{HII}}$ constraints, as well as the SFR density from UV and
IR data, we put constraints on the ionization history with DM annihilation and
star formation by performing a maximum likelihood determination with
CAMB$+$CosmoMC packages. The upper limits of $\epsilon_{0}f_{d}$ show
$1.3309\times 10^{-24}$ at 68$\%$ C.L. and $2.7765\times 10^{-24}$ at 95$\%$
C.L. in this model. By comparison, we also research the instantaneous
reionization model including DM annihilation from the same data combination
except SFR density data and $Q_{\text{HII}}$ constraints. We get similar
constraints and $\epsilon_{0}f_{d}$ reads $1.4253\times 10^{-24}$ at 68$\%$
C.L. and $2.8468\times 10^{-24}$ at 95$\%$ C.L.. So the reionization models
have little influence ($\lesssim 2.5\%$) on constraining DM annihilation.
Similarly, DM decay has analogous influence on ionization history of our
universe and the decay rate plays a similar role as
$\epsilon_{0}f_{\text{d}}$. Therefore, we arrive at a conclusion that various
reionization models have little effects on DM decay or annihilation. Besides,
the optical depth is $\tau=0.0571^{+0.0005}_{-0.0006}$ at 68$\%$ in the SFH
model, improving significantly than $\tau=0.0559^{+0.0069}_{-0.0076}$ in the
TANH model.

In our models, we have made some simplifications. We use the tipical values of
parameters, such as the clumping factor of ionized hydrogen
$C_{\text{HII}}\sim 3$, the escaping rate $f_{\text{esc}}=0.2$ and so on.
Actually, other comparable values or redshift-varying functions of them are
also acceptable Ouchi:2009ys ; Carucci:2018tzf ; Robertson:2010an . However,
we do not think these comparable values would change our conclusion
significantly. More discussion about complicated models can be found in
Carucci:2018tzf ; Robertson:2013bq ; Robertson:2010an ; Gorce:2017glg . We
also assume $\epsilon_{0}f_{d}$ to be a constant though it varies with
redshift, cosmological models, as well as DM annihilation channels. The plasma
temperature evolution and excitation during the epoch of reionization are also
small enough to be ignored. Moreover, DM annihilation in halos may also
influence the reionization history of our universe Lopez-Honorez:2013cua ;
Liu:2016cnk . But we have ignored the enhancement of halos in this paper.

Acknowledgments

We acknowledge the use of HPC Cluster of Tianhe II in National Supercomputing
Center in Guangzhou. Ke Wang is supported by grants from NSFC (grant No.
12005084) and grants from the China Manned Space Project with NO. CMS-
CSST-2021-B01. Lu Chen is supported by grants from NSFC (grant No. 12105164).
This work has also received funding from project ZR2021QA021 supported by
Shandong Provincial Natural Science Foundation.

## References

  * (1) H. Hoekstra, H. Yee and M. Gladders, New Astron. Rev. 46, 767-781 (2002) doi:10.1016/S1387-6473(02)00245-2 [arXiv:astro-ph/0205205 [astro-ph]].
  * (2) L. V. E. Koopmans and T. Treu, Astrophys. J. 583, 606-615 (2003) doi:10.1086/345423 [arXiv:astro-ph/0205281 [astro-ph]].
  * (3) R. B. Metcalf, L. A. Moustakas, A. J. Bunker and I. R. Parry, Astrophys. J. 607, 43-59 (2004) doi:10.1086/383243 [arXiv:astro-ph/0309738 [astro-ph]].
  * (4) L. A. Moustakas and R. B. Metcalf, Mon. Not. Roy. Astron. Soc. 339, 607 (2003) doi:10.1046/j.1365-8711.2003.06055.x [arXiv:astro-ph/0206176 [astro-ph]].
  * (5) M. Davis, G. Efstathiou, C. S. Frenk and S. D. M. White, Astrophys. J. 292, 371-394 (1985) doi:10.1086/163168
  * (6) J. F. Navarro, C. S. Frenk and S. D. M. White, Astrophys. J. 462, 563-575 (1996) doi:10.1086/177173 [arXiv:astro-ph/9508025 [astro-ph]].
  * (7) J. F. Navarro, E. Hayashi, C. Power, A. Jenkins, C. S. Frenk, S. D. M. White, V. Springel, J. Stadel and T. R. Quinn, Mon. Not. Roy. Astron. Soc. 349, 1039 (2004) doi:10.1111/j.1365-2966.2004.07586.x [arXiv:astro-ph/0311231 [astro-ph]].
  * (8) G. R. Blumenthal, S. M. Faber, J. R. Primack and M. J. Rees, Nature 311, 517-525 (1984) doi:10.1038/311517a0
  * (9) M. Davis, F. J. Summers and D. Schlegel, Nature 359, 393-396 (1992) doi:10.1038/359393a0
  * (10) K. G. Begeman, A. H. Broeils and R. H. Sanders, Mon. Not. Roy. Astron. Soc. 249, 523 (1991)
  * (11) W. J. G. de Blok, S. S. McGaugh, A. Bosma and V. C. Rubin, Astrophys. J. Lett. 552, L23-L26 (2001) doi:10.1086/320262 [arXiv:astro-ph/0103102 [astro-ph]].
  * (12) F. C. van den Bosch and R. A. Swaters, Mon. Not. Roy. Astron. Soc. 325, 1017 (2001) doi:10.1046/j.1365-8711.2001.04456.x [arXiv:astro-ph/0006048 [astro-ph]].
  * (13) P. A. R. Ade et al. [Planck], Astron. Astrophys. 594, A13 (2016) doi:10.1051/0004-6361/201525830 [arXiv:1502.01589 [astro-ph.CO]].
  * (14) N. Aghanim et al. [Planck], [arXiv:1807.06209 [astro-ph.CO]].
  * (15) G. Jungman, M. Kamionkowski and K. Griest, Phys. Rept. 267, 195-373 (1996) doi:10.1016/0370-1573(95)00058-5 [arXiv:hep-ph/9506380 [hep-ph]].
  * (16) G. Bertone, D. Hooper and J. Silk, Phys. Rept. 405, 279-390 (2005) doi:10.1016/j.physrep.2004.08.031 [arXiv:hep-ph/0404175 [hep-ph]].
  * (17) D. Clowe, M. Bradac, A. H. Gonzalez, M. Markevitch, S. W. Randall, C. Jones and D. Zaritsky, Astrophys. J. Lett. 648, L109-L113 (2006) doi:10.1086/508162 [arXiv:astro-ph/0608407 [astro-ph]].
  * (18) A. H. G. Peter, [arXiv:1201.3942 [astro-ph.CO]].
  * (19) J. D. Lewin and P. F. Smith, Astropart. Phys. 6, 87-112 (1996) doi:10.1016/S0927-6505(96)00047-3
  * (20) K. Freese, M. Lisanti and C. Savage, Rev. Mod. Phys. 85, 1561-1581 (2013) doi:10.1103/RevModPhys.85.1561 [arXiv:1209.3339 [astro-ph.CO]].
  * (21) J. M. Gaskins, Contemp. Phys. 57, no.4, 496-525 (2016) doi:10.1080/00107514.2016.1175160 [arXiv:1604.00014 [astro-ph.HE]].
  * (22) F. Kahlhoefer, Int. J. Mod. Phys. A 32, no.13, 1730006 (2017) doi:10.1142/S0217751X1730006X [arXiv:1702.02430 [hep-ph]].
  * (23) M. Cirelli, Pramana 79, 1021-1043 (2012) doi:10.1007/s12043-012-0419-x [arXiv:1202.1454 [hep-ph]].
  * (24) A. Geringer-Sameth, S. M. Koushiappas and M. Walker, Astrophys. J. 801, no.2, 74 (2015) doi:10.1088/0004-637X/801/2/74 [arXiv:1408.0002 [astro-ph.CO]].
  * (25) G. Steigman, B. Dasgupta and J. F. Beacom, Phys. Rev. D 86, 023506 (2012) doi:10.1103/PhysRevD.86.023506 [arXiv:1204.3622 [hep-ph]].
  * (26) P. Gondolo and J. Silk, Phys. Rev. Lett. 83, 1719-1722 (1999) doi:10.1103/PhysRevLett.83.1719 [arXiv:astro-ph/9906391 [astro-ph]].
  * (27) S. Galli, F. Iocco, G. Bertone and A. Melchiorri, Phys. Rev. D 80, 023505 (2009) doi:10.1103/PhysRevD.80.023505 [arXiv:0905.0003 [astro-ph.CO]].
  * (28) S. Galli, F. Iocco, G. Bertone and A. Melchiorri, Phys. Rev. D 84, 027302 (2011) doi:10.1103/PhysRevD.84.027302 [arXiv:1106.1528 [astro-ph.CO]].
  * (29) L. Zhang, X. L. Chen, Y. A. Lei and Z. G. Si, Phys. Rev. D 74, 103519 (2006) doi:10.1103/PhysRevD.74.103519 [arXiv:astro-ph/0603425 [astro-ph]].
  * (30) C. Dvorkin, K. Blum and M. Zaldarriaga, Phys. Rev. D 87, no.10, 103522 (2013) doi:10.1103/PhysRevD.87.103522 [arXiv:1302.4753 [astro-ph.CO]].
  * (31) C. Weniger, P. D. Serpico, F. Iocco and G. Bertone, Phys. Rev. D 87, no.12, 123008 (2013) doi:10.1103/PhysRevD.87.123008 [arXiv:1303.0942 [astro-ph.CO]].
  * (32) J. Chluba, Mon. Not. Roy. Astron. Soc. 402, 1195 (2010) doi:10.1111/j.1365-2966.2009.15957.x [arXiv:0910.3663 [astro-ph.CO]].
  * (33) C. Gruppioni, F. Pozzi, G. Zamorani and C. Vignali, Mon. Not. Roy. Astron. Soc. 416, 70-86 (2011) doi:10.1111/j.1365-2966.2011.19006.x [arXiv:1105.1955 [astro-ph.CO]].
  * (34) R. Chornock, E. Berger, D. B. Fox, R. Lunnan, M. R. Drout, W. F. Fong, T. Laskar and K. C. Roth, Astrophys. J. 774, 26 (2013) doi:10.1088/0004-637X/774/1/26 [arXiv:1306.3949 [astro-ph.CO]].
  * (35) P. Madau and M. Dickinson, Ann. Rev. Astron. Astrophys. 52, 415-486 (2014) doi:10.1146/annurev-astro-081811-125615 [arXiv:1403.0007 [astro-ph.CO]].
  * (36) B. E. Robertson, R. S. Ellis, S. R. Furlanetto and J. S. Dunlop, Astrophys. J. Lett. 802, no.2, L19 (2015) doi:10.1088/2041-8205/802/2/L19 [arXiv:1502.02024 [astro-ph.CO]].
  * (37) I. P. Carucci and P. S. Corasaniti, Phys. Rev. D 99, no.2, 023518 (2019) doi:10.1103/PhysRevD.99.023518 [arXiv:1811.07904 [astro-ph.CO]].
  * (38) L. Lopez-Honorez, O. Mena, S. Palomares-Ruiz and A. C. Vincent, JCAP 07, 046 (2013) doi:10.1088/1475-7516/2013/07/046 [arXiv:1303.5094 [astro-ph.CO]].
  * (39) H. Liu, T. R. Slatyer and J. Zavala, Phys. Rev. D 94, no.6, 063507 (2016) doi:10.1103/PhysRevD.94.063507 [arXiv:1604.02457 [astro-ph.CO]].
  * (40) M. Mapelli, A. Ferrara and E. Pierpaoli, Mon. Not. Roy. Astron. Soc. 369, 1719-1724 (2006) doi:10.1111/j.1365-2966.2006.10408.x [arXiv:astro-ph/0603237 [astro-ph]].
  * (41) D. K. Hazra, D. Paoletti, F. Finelli and G. F. Smoot, JCAP 09, 016 (2018) doi:10.1088/1475-7516/2018/09/016 [arXiv:1807.05435 [astro-ph.CO]].
  * (42) D. Paoletti, D. K. Hazra, F. Finelli and G. F. Smoot, JCAP 09, 005 (2020) doi:10.1088/1475-7516/2020/09/005 [arXiv:2005.12222 [astro-ph.CO]].
  * (43) A. Lewis, Phys. Rev. D 78, 023002 (2008) doi:10.1103/PhysRevD.78.023002 [arXiv:0804.3865 [astro-ph]].
  * (44) D. K. Hazra and G. F. Smoot, JCAP 11, 028 (2017) doi:10.1088/1475-7516/2017/11/028 [arXiv:1708.04913 [astro-ph.CO]].
  * (45) C. H. Heinrich, V. Miranda and W. Hu, Phys. Rev. D 95, no.2, 023513 (2017) doi:10.1103/PhysRevD.95.023513 [arXiv:1609.04788 [astro-ph.CO]].
  * (46) Y. Khusanova, O. Le Fèvre, P. Cassata, O. Cucciati, B. C. Lemaux, L. A. M. Tasca, R. Thomas, B. Garilli, V. Le Brun and D. Maccagni, et al. Astron. Astrophys. 634, A97 (2020) doi:10.1051/0004-6361/201935400 [arXiv:1903.01884 [astro-ph.GA]].
  * (47) B. E. Robertson, R. S. Ellis, J. S. Dunlop, R. J. McLure and D. P. Stark, Nature 468, 49 (2010) doi:10.1038/nature09527 [arXiv:1011.0727 [astro-ph.CO]].
  * (48) B. E. Robertson, S. R. Furlanetto, E. Schneider, S. Charlot, R. S. Ellis, D. P. Stark, R. J. McLure, J. S. Dunlop, A. Koekemoer and M. A. Schenker, et al. Astrophys. J. 768, 71 (2013) doi:10.1088/0004-637X/768/1/71 [arXiv:1301.1228 [astro-ph.CO]].
  * (49) R. J. Bouwens, G. D. Illingworth, P. A. Oesch, J. Caruana, B. Holwerda, R. Smit and S. Wilkins, Astrophys. J. 811, no.2, 140 (2015) doi:10.1088/0004-637X/811/2/140 [arXiv:1503.08228 [astro-ph.CO]].
  * (50) X. H. Fan, M. A. Strauss, R. H. Becker, R. L. White, J. E. Gunn, G. R. Knapp, G. T. Richards, D. P. Schneider, J. Brinkmann and M. Fukugita, Astron. J. 132, 117-136 (2006) doi:10.1086/504836 [arXiv:astro-ph/0512082 [astro-ph]].
  * (51) M. A. Schenker, R. S. Ellis, N. P. Konidaris and D. P. Stark, Astrophys. J. 795, no.1, 20 (2014) doi:10.1088/0004-637X/795/1/20 [arXiv:1404.4632 [astro-ph.CO]].
  * (52) F. Beutler, C. Blake, M. Colless, D. H. Jones, L. Staveley-Smith, L. Campbell, Q. Parker, W. Saunders and F. Watson, Mon. Not. Roy. Astron. Soc. 416, 3017-3032 (2011) doi:10.1111/j.1365-2966.2011.19250.x [arXiv:1106.3366 [astro-ph.CO]].
  * (53) A. J. Ross, L. Samushia, C. Howlett, W. J. Percival, A. Burden and M. Manera, Mon. Not. Roy. Astron. Soc. 449, no.1, 835-847 (2015) doi:10.1093/mnras/stv154 [arXiv:1409.3242 [astro-ph.CO]].
  * (54) S. Alam et al. [BOSS], Mon. Not. Roy. Astron. Soc. 470, no.3, 2617-2652 (2017) doi:10.1093/mnras/stx721 [arXiv:1607.03155 [astro-ph.CO]].
  * (55) D. M. Scolnic, D. O. Jones, A. Rest, Y. C. Pan, R. Chornock, R. J. Foley, M. E. Huber, R. Kessler, G. Narayan and A. G. Riess, et al. Astrophys. J. 859, no.2, 101 (2018) doi:10.3847/1538-4357/aab9bb [arXiv:1710.00845 [astro-ph.CO]].
  * (56) G. Hutsi, J. Chluba, A. Hektor and M. Raidal, Astron. Astrophys. 535, A26 (2011) doi:10.1051/0004-6361/201116914 [arXiv:1103.2766 [astro-ph.CO]].
  * (57) J. D. Bowman, A. E. E. Rogers, R. A. Monsalve, T. J. Mozdzen and N. Mahesh, Nature 555, no.7694, 67-70 (2018) doi:10.1038/nature25792 [arXiv:1810.05912 [astro-ph.CO]].
  * (58) Masafumi Ishigaki and Ryota Kawamata and Masami Ouchi and Masamune Oguri and Kazuhiro Shimasaku and Yoshiaki Ono, The Astrophysical Journal, 854:73 (20pp), 2018 February 10, doi:10.3847/1538-4357/aaa544.
  * (59) A. Lewis, A. Challinor and A. Lasenby, Astrophys. J. 538, 473-476 (2000) doi:10.1086/309179 [arXiv:astro-ph/9911177 [astro-ph]].
  * (60) J. R. Shaw and J. Chluba, Mon. Not. Roy. Astron. Soc. 415, 1343 (2011) doi:10.1111/j.1365-2966.2011.18782.x [arXiv:1102.3683 [astro-ph.CO]].
  * (61) A. Lewis and S. Bridle, Phys. Rev. D 66, 103511 (2002) doi:10.1103/PhysRevD.66.103511 [arXiv:astro-ph/0205436 [astro-ph]].
  * (62) A. Lewis, Phys. Rev. D 87, no.10, 103529 (2013) doi:10.1103/PhysRevD.87.103529 [arXiv:1304.4473 [astro-ph.CO]].
  * (63) A. Krishak and D. K. Hazra, [arXiv:2106.01728 [astro-ph.CO]].
  * (64) A. Gorce, M. Douspis, N. Aghanim and M. Langer, Astron. Astrophys. 616, A113 (2018) doi:10.1051/0004-6361/201629661 [arXiv:1710.04152 [astro-ph.CO]].
  * (65) M. Ouchi, B. Mobasher, K. Shimasaku, H. C. Ferguson, M. S. Fall, Y. Ono, N. Kashikawa, T. Morokuma, K. Nakajima and S. Okamura, et al. Astrophys. J. 706, 1136-1151 (2009) doi:10.1088/0004-637X/706/2/1136 [arXiv:0908.3191 [astro-ph.CO]].


# Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty
Estimation for Facial Expression Recognition

Jiahui She∗1,2 Yibo Hu∗2 Hailin Shi2 Jun Wang2 Qiu Shen†1 Tao Mei2
1Nanjing University 2JD AI Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>{wangjun492<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Due to the subjective annotation and the inherent inter-class similarity of
facial expressions, one of key challenges in Facial Expression Recognition
(FER) is the annotation ambiguity. In this paper, we proposes a solution,
named DMUE, to address the problem of annotation ambiguity from two
perspectives: the latent Distribution Mining and the pairwise Uncertainty
Estimation. For the former, an auxiliary multi-branch learning framework is
introduced to better mine and describe the latent distribution in the label
space. For the latter, the pairwise relationship of semantic feature between
instances are fully exploited to estimate the ambiguity extent in the instance
space. The proposed method is independent to the backbone architectures, and
brings no extra burden for inference. The experiments are conducted on the
popular real-world benchmarks and the synthetic noisy datasets. Either way,
the proposed DMUE stably achieves leading performance.

††footnotetext: $*$ These authors contributed equally to this
work.††footnotetext: ${\dagger}$ Corresponding author

##  1 Introduction

Facial expression plays an essential role in human’s daily life. Automatic
Facial Expression Recognition (FER) is crucial in real world applications,
such as service robots, driver fragile detection and human computer
interaction. In recent years, with the emerge of large-scale datasets,
AffectNet [DBLP:journals/affectnet], RAF-DB [DBLP:conf/cvpr/raf] and EmotioNet
[DBLP:conf/cvpr/emotionet], many deep learning based FER approaches
[DBLP:conf/cvpr/LDL-ALSG, DBLP:conf/cvpr/SCN, DBLP:conf/eccv/IPA2LT] have been
proposed and achieved promising performance.

However, the ambiguity problem remains an obstacle that hinders the FER
performance. Usually, facial images are annotated to one of several basic
expressions for training the FER model. Yet the definition with respect to the
expression category may be inconsistent among different people. For better
understanding, we randomly pick two images from AffectNet
[DBLP:journals/affectnet] and conduct a user study. As shown in Fig. 1, for
the image annotated with Anger, the most possible class decided by volunteers
is Neutral. For the other image, the confidence gap between the most and
secondary possible classes is only 20%, which means annotating it to a
specific class is not suitable. In other words, a label distribution that
depicts the possibility belonging to each class can better describe the visual
feature. There are two reasons leading to the above phenomenon: (1) It is
subjective for people to define which type of expression a facial image is.
(2) With a large amount of images in large-scale FER datasets, it is expensive
and time-consuming to provide label distribution of images. As there exists a
considerable portion of ambiguous samples in large-scale datasets, the models
are prevented from learning the robust visual features with respect to a
certain type of expression, thus the performance has reached a bottleneck. The
previous approaches tried to address this issue by introducing label
distribution learning [DBLP:conf/cvpr/LDL-ALSG] or suppressing uncertain
samples [DBLP:conf/cvpr/SCN]. However, they still suffer from the ambiguity
problem revealed in data that cannot be directly solved from the single
instance perspective.

Figure 1: User study results by 50 volunteers on two randomly picked images.
The red (purple) bar represents the most (secondary) possible class given by
the volunteers. The results provide insights that the annotations may be
inconsistent among the users.

In this paper, we propose a solution to address the ambiguity problem in FER
from two perspectives, the latent Distribution Mining and the pairwise
Uncertainty Estimation (DMUE). For the former, several temporary auxiliary
branches are introduced to discover the label distributions of samples in an
online manner. The iteratively updated distributions can better describe the
visual features of expression images in the label space. Thus, it can provide
the model informative semantic features to flexibly handle ambiguous images.
For the latter, we design an elaborate uncertainty estimation module based on
pairwise relationships between samples. It jointly utilizes the original
annotations and the statistics of relationships to reflect the ambiguity
extent of samples. The estimated uncertain level encourages the model to
dynamically adjust learning focus between the mined label distribution and
original annotations. Note that our proposed framework is end-to-end training
and has no extra cost for inference. All the auxiliary branches and the
uncertainty estimation module will be removed during deployment. Overall, the
main contributions can be summarized as follows:

  * •

We propose a novel end-to-end solution to investigate the ambiguity problem in
FER by exploring the latent label distribution of the given sample, without
introducing extra burden on inference.

  * •

An elaborate uncertainty estimation module is designed based on the statistics
of relationships, which provides guidance for the model to dynamically adjust
learning focus between the mined label distribution and annotations from
sample level.

  * •

Our approach is evaluated on the popular real-world benchmarks and synthetic
noisy datasets. Particularly, it achieves the best performance by 89.42% on
RAF-DB and 63.11% on AffectNet, setting new records.

Figure 2: Overview of the DMUE. $\bm{y}$ denotes the set of annotations of
images in a batch. $\bm{\widetilde{y}}$ denotes the set of mined latent
distributions of images in a batch.

##  2 Related Work

###  2.1 Facial Expression Recognition

Numerous FER algorithms [SeNet50, DBLP:journals/tip/gaCNN, DBLP:journals/SIFT,
DBLP:conf/mm/normal_acm1] have been proposed, which can be grouped into
handcraft and learning-based methods. Early attempts
[DBLP:conf/fgr/Gabor_Wavelets, DBLP:journals/SIFT, DBLP:journals/LBP] rely on
handcraft features that reflect folds and geometry changes caused by
expression. With the development of deep learning, learning-based methods
[DBLP:journals/corr/R3A1, DBLP:conf/mm/normal_acm1, DBLP:conf/cvpr/DeRl]
become the majority, such as decoupling the identity information
[DBLP:conf/mm/normal_acm1] or exploiting the difference between expressive
images [DBLP:conf/cvpr/DeRl].

In recent years, several attempts try to address ambiguity problem in FER.
Zeng et al. [DBLP:conf/eccv/IPA2LT] consider annotation inconsistency and
introduce multiple training phases. Chen et al. [DBLP:conf/cvpr/LDL-ALSG]
build nearest neighbor graphs for training data in advance and investigate
label distribution of samples in a semi-online way. Previous leading
performance has been achieved by Wang et al. [DBLP:conf/cvpr/SCN]. They focus
on finding the confidence weight and the latent truth of each sample to
suppress harmful influence from ambiguous data. However, the compound
expressions [DBLP:conf/cvpr/raf] and the original annotations could be jointly
considered in estimating ambiguity.

###  2.2 Learning with Ambiguity Label

Mislabelled annotations and low data quality may result in ambiguity problem.
For the former, learning with noisy label
[DBLP:journals/corr/noisy_label_survey] is one of the most popular directions.
Another direction is the uncertainty estimation [DBLP:conf/nips/metaweightnet,
DBLP:conf/iccv/noisy_2], such as MentorNet [DBLP:conf/icml/MentorNet] and
CleanLab [DBLP:journals/corr/cleanlab]. In recent years, a promising way to
handle mislabelled annotations is to find the latent truth
[DBLP:conf/iclr/noiselayer], such as utilizing the prediction of model
[DBLP:conf/nips/MixMatch, DBLP:conf/iclr/noiselayer,
DBLP:conf/iccv/label_noisy1_phototype, DBLP:conf/iclr/dividemix,
DBLP:journals/ijcv/AberHu2] or introducing auxiliary embeddings
[DBLP:conf/eccv/AberHu1, DBLP:conf/cvpr/label_noisy1]. For the latter, an
universal way is to enhance the label [genxin1, genxin2] of low quality images
by the temperature softmax [DBLP:conf/nips/MixMatch] or inject the artificial
uncertainty [DBLP:conf/cvpr/propface2, DBLP:conf/iccv/propface,
DBLP:conf/cvpr/uncertainty1]. Unlike prior methods [genxin1, genxin2], the
ambiguity problem in FER, compound expressions [DBLP:conf/cvpr/raf] exists in
a more subjective way. The label description of a compound expression image is
various among the users.

##  3 Method

Notation. Given a FER dataset $(\mathcal{X},\mathcal{Y})$ in which each image
$\bm{x}$ belongs to one of $C$ classes, we denote $y_{x}\in\\{1,2,\cdots,C\\}$
as its annotated deterministic class. However, as shown in Fig. 1, the exact
type of $\bm{x}$ is inapparent or uncertain. We employ latent distribution
$\widetilde{\bm{y}}_{x}$ to represent the probability distribution for
$\bm{x}$ belonging to all possible classes except $y_{x}$. That is,
$\widetilde{\bm{y}}_{x}\in\mathbb{R}^{C-1}$ is a distribution vector,
$\|\widetilde{\bm{y}}_{x}\|_{1}=1$.

###  3.1 Overview of DMUE

To address the annotation ambiguity, we mine $\widetilde{\bm{y}}_{x}$ for each
$\bm{x}$ and regularize the model to learn jointly from
$\widetilde{\bm{y}}_{x}$ and $y_{x}$. Benefited from the semantic features of
ambiguous samples, the performance of the model can be greatly improved.

For a toy experiment, we train a ResNet-18 on AffectNet
[DBLP:journals/affectnet] and present its prediction for a mislabelled
training sample in Fig. 10, where a crying baby (Sad) is labelled with
Neutral. We can observe that the predicted distribution reflects the truth
class of the mislabelled sample. It inspires us to employ the predictions from
a trained model to help a new model in training phase, where such mislabelled
image may be tagged by a distribution reflecting its true class. By imposing
the latent distribution $\widetilde{\bm{y}}_{x}$ as the additional
supervision, model can utilize the latent semantic features to better deal
with ambiguity samples, thus improve the performance. We employ the classifier
trained by samples from negative classes of $\bm{x}$, samples from other $C-1$
classes except for $y_{x}$, to find its $\widetilde{\bm{y}}_{x}$, based on
qualitative and quantitative analyses in Section 4.7. Moreover, to balance the
learning between the annotation and the mined $\widetilde{\bm{y}}_{x}$, an
uncertainty estimation module is elaborately designed to guide the model to
learn more from $\widetilde{\bm{y}}_{x}$ than $y_{x}$ for those ambiguous
samples.

An overview of DMUE is depicted in Fig. 2. The DMUE contains: (1) latent
distribution mining with C auxiliary branches and one target branch that have
the same architecture (the last stage of ResNet), and (2) pairwise uncertainty
estimation, where an uncertainty estimation module is established by two fully
connected (FC) layers. Each auxiliary branch is served as an individual
$(C-1)$-class classifier aiming to find $\widetilde{\bm{y}}_{x}$ for the
corresponding $\bm{x}$. $\widetilde{\bm{y}}_{x}$ and $y_{x}$ are joint
together to guide the target branch. Furthermore, we regularize the branches
to predict consistent relationships of images by their similarity matrices.
Note that all auxiliary branches and the uncertainty estimation module will be
removed, and only the target branch will be reserved for deployment.
Therefore, our framework is end-to-end and can be flexibly integrated into
existing network architectures without extra cost on inference.

###  3.2 Latent Distribution Mining

As $\widetilde{\bm{y}}_{x}$ is predicted by the classifier trained with
samples from negative classes of $\bm{x}$, If there are total $C$ classes,
then $C$ classifiers need to be trained to predict the latent distribution of
each sample. Considering the computational efficiency and the shared low-level
features [DBLP:conf/aaai/online3, DBLP:conf/nips/online2], we propose a multi-
branch architecture to construct these classifiers. As shown in Fig. 2, $C$
auxiliary branches are introduced to predict the latent distributions and a
target branch is employed for final prediction. Given a batch, the $j$-th
branch predicts $\widetilde{\bm{y}}_{x}$ for $\bm{x}$ annotated to the $j$-th
class. Thus, we can obtain $\widetilde{\bm{y}}_{x}$ for each $\bm{x}$ in batch
by $C$ auxiliary branches. Note all branches have the same structure (the last
stage of ResNet) and share the common lower layers (the first three stages of
ResNet). Classifier $j,j\in\\{1,\cdots,C\\}$ is $(C-1)$-class and Classifier
$0$ (the target classifier) is $C$-class for final deployment.

A comprehensive description of mini-batch training is presented in Algorithm
1. Given a batch, we use images not annotated to the $j$-th category to train
the $j$-th auxiliary branch. In other words, each image $\bm{x}$ is utilized
to train other $C-1$ auxiliary branches than the $y_{x}$-th branch. The Cross-
Entropy(CE) loss $L_{CE}^{aux}$ is employed for optimization:

$\vspace{-1em}\centering{\rm{L}}_{CE}^{aux}=\frac{1}{C}\sum\limits_{j=1}^{C}{L_{CE}^{au{x_{j}}},}\vspace{-0.5em}\@add@centering$
(1) $\centering{\rm{L}}_{CE}^{au{x_{j}}}=-\frac{\begin{array}[]{l}\\\
1\end{array}}{N_{j}}\sum\limits_{p=1}^{N_{j}}{\sum\limits_{k=1,k\neq
j}^{C}{{y_{x_{p},k}}\log{f_{j}}({\bm{x}_{p}};\theta)_{k}},}\@add@centering$
(2)

where $L_{CE}^{aux_{j}}$ is the CE loss for training the $j$-th branch,
$N_{j}$ is the number of $\bm{x}$ not annotated to $j$ in the batch and $p$ is
index. $y_{x_{p},k}$ is the label of $\bm{x}_{p}$ belonging to the $k$-th
class and ${f_{j}}{({\bm{x}_{p}};\theta)_{k}}$ is the possibility of
$\bm{x}_{p}$ belonging to the $k$-th class predicted by the $j$-th branch.

As described above, the prediction of the $j$-th auxiliary branch for $\bm{x}$
with annotation $j$, is used as its latent distribution
$\widetilde{\bm{y}}_{x}\in\mathbb{R}^{C-1}$. One additional step, called
Sharpen [DBLP:conf/nips/MixMatch, DBLP:conf/iclr/dividemix,
DBLP:journals/corr/online1], is adopted before regularizing the target branch:

$\centering
Sharpen(\widetilde{\bm{y}}_{x},T)_{i}=\widetilde{y}_{x,i}^{\frac{1}{T}}/\sum\nolimits_{j}^{C-1}{\widetilde{y}_{x,j}^{\frac{1}{T}},}\@add@centering$
(3)

where $\widetilde{{y}}_{x,i}$ the $i$-th element of $\widetilde{\bm{y}}_{x}$
and $T$ is the temperature. Sharpen function provides the flexibility to
slightly adjust the entropy of $\widetilde{\bm{y}}_{x}$. When $T>1$, the
output $Sharpen(\widetilde{\bm{y}}_{x},T)$ will be more flatten than the
original $\widetilde{\bm{y}}_{x}$.

After sharpening, we utilize $L_{2}$ loss to minimize the deviation between
the prediction of target branch and the sharpened $\widetilde{\bm{y}}_{x}$,
which is defined as:

$\centering{L_{soft}}=\frac{1}{{N(C-1)}}\sum\limits_{p=1}^{N}{\sum\limits_{k=1,k\neq
i}^{C}{{{({\widetilde{y}_{{x_{p}},k}}-{f_{target}}{{({\bm{x}_{p}};\theta)}_{k}})}^{2}}},}\@add@centering$
(4)

where $N$ is the batch size. $\widetilde{y}_{{x_{p}},k}$ is the possibility of
$\bm{x}_{p}$ belonging to the $k$-th class in the latent distribution
$\widetilde{\bm{y}}_{x_{p}}$, and ${f_{target}}{{({\bm{x}_{k}};\theta)}_{j}}$
is the prediction of target branch. The reason employing L2 loss is that
unlike Cross-Entropy, the L2 loss is bounded and less sensitive to inaccurate
predictions. We do not back propagate gradients through computing
$\widetilde{\bm{y}}$.

Similarity Preserving. Inspired by [DBLP:conf/iccv/SP-KD], we find it is
beneficial to regularize all the branches to predict consistent relationship
when given a pair of images. This is because CE loss only utilizes samples
individually from the label space. However, the relationship between samples
is another knowledge paradigm. For instance, given a pair of smiling images,
besides telling network their annotations Happy, the similarities of their
semantic features extracted by different branches should be consistent. Thus,
we generalize [DBLP:conf/iccv/SP-KD] to the context of multi-branch
architecture as multi-branch similarity preserving ($MSP$), defined as:

$\centering{L_{sp}}=MSP(G_{aux}^{1},\cdots,G_{aux}^{C},G_{tar}),\@add@centering$
(5)

where $G_{aux}^{i}\in\mathbb{R}^{B_{i}\times{B_{i}}}$ and
$G_{tar}\in\mathbb{R}^{B\times{B}}$ are the similarity matrices calculated by
the semantic features in auxiliary and target branches, respectively. Their
elements reflect the pairwise relationships between samples. $L_{sp}$ aims at
sharing the relation information across branches. Specific computation is
provided in the supplementary material.

Figure 3: Uncertainty estimation module. $y_{a}$ is the one-hot form of anchor
image’s annotation. $S_{a}$ and $y_{a}$ are concatenated to reflect how
ambiguous the anchor image is.

###  3.3 Pairwise Uncertainty Estimation

To handle the ambiguous samples, we introduce latent distribution mining.
However, the target branch should also be benefitted from clean samples.
Directly employing CE loss may lead to improvement degradation due to the
existing of ambiguous samples. Accordingly, we impose a modulator term into
the standard CE loss to trade-off between the latent distribution and
annotation in the sample space. Specifically, we estimate the confidence
scores of the samples based on the statistics of their relationships. Lower
score will be assigned to more ambiguous samples, further reducing the CE
loss. Thus, the latent distribution will provide more guidance.

For better understanding, we choose an anchor image in a given batch to
illustrate the uncertainty estimation module. As shown in Fig. 3, we denote
the semantic feature and one-hot label of the anchor image as
$(\bm{f}_{a},\bm{y}_{a})$, while others in the batch as
$(\bm{f}_{i},\bm{y}_{i})$, where $\bm{f}$ is the feature before classifier in
the target branch, $i$ is the index. We calculate the average cosine
similarity of $\bm{f}_{a}$ with each of $\bm{f}_{i}$ annotated with $j$-th
category as $S_{a,j}$, and vector $\bm{S}_{a}=[S_{a,1},\cdots,S_{a,C}]$. After
that, $\bm{S}_{a}$ is concatenated with $\bm{y}_{a}\in\mathbb{R}^{C}$ to form
$\bm{SV}_{a}\in\mathbb{R}^{2C}$, which reflects how ambiguous the anchor
sample is:

$\centering\bm{SV}_{a}=concat(\bm{S}_{a},\bm{y}_{a}),\@add@centering$ (6)

$\centering{S_{a,j}}=\frac{1}{{{N_{j}}}}\sum\limits_{i=1}^{{N_{j}}}{\frac{{\left<{{\bm{f}_{a}},{\bm{f}_{i}}}\right>}}{{\left\|{{\bm{f}_{a}}}\right\|\left\|{{\bm{f}_{i}}}\right\|}},}\@add@centering$
(7)

where $<\bm{f}_{a},\bm{f}_{i}>$ is the dot product of $\bm{f}_{a}$ and
$\bm{f}_{i}$. $N_{j}$ is the number of samples whose annotation is $j$-th
class in the batch and $i$ is the index.

Here, We provide two perspectives to understand this delicate design: (1) For
a mislabelled sample $(\bm{x},y_{x})$ in the given batch (the semantic feature
of $\bm{x}$ belongs to $i$-th class but $y_{x}=j$), the average similarity of
semantic features between $\bm{x}$ and the images in $i$-th class should be
high. However, the concatenated $y_{x}$ indicates $\bm{x}$ is annotated with
$i$-th class. (2) A clear $\bm{x}(y_{x}=i)$ should only capture the typical
semantic feature of $i$-th class. The average similarity of its semantic
feature with other types of images should be discriminatively lower than with
$i$-th class samples. Thus, $\bm{SV}_{x}$ can reveal the ambiguity information
of $\bm{x}$.

Let
$\bm{SV}=[\bm{SV}_{1},\bm{SV}_{2},\cdots,\bm{SV}_{N}]\in\mathbb{R}^{2C\times{N}}$
denotes the ambiguity information feature of a batch, the uncertainty
estimation module takes $\bm{SV}$ as the input and outputs a confidence scalar
$\alpha_{i}\in(0,1)$ for each image. The module consists of two FC layers with
a PRelu non-linear function and a sigmoid activation:

$\centering{\alpha}=Sigmoid(\bm{W}_{2}^{T}\sigma(\bm{W}_{1}^{T}\bm{SV}),\@add@centering$
(8)

where $\bm{W}_{1}\in\mathbb{R}^{2C\times{C}}$ and
$\bm{W}_{2}\in\mathbb{R}^{C\times{1}}$ are the parameters of two FC
layers,$\sigma$ is the PReLU activation.

With the estimated confidence score, we perform weighted training in the
target branch. Directly multiplying the score with CE loss may obstruct the
uncertainty estimation, because it will make the estimated score to be zero
[DBLP:conf/cvpr/SCN]. Therefore, we alternatively multiply the score with the
output logit of the classifier in the target branch. The weighted CE loss
[DBLP:conf/cvpr/face1, DBLP:conf/cvpr/SCN] is formulated as:

$\centering L_{WCE}^{t\arg
et}=-\frac{1}{N}\sum\limits_{i=1}^{N}{\log\frac{{{e^{{\alpha_{i}}W_{{y_{i}}}^{T}{f_{i}}}}}}{{\sum\nolimits_{j=1}^{C}{{e^{{\alpha_{i}}W_{j}^{T}{f_{i}}}}}}}.}\@add@centering$
(9)

Obviously, $L_{WCE}^{target}$ has positive correlation with the score $\alpha$
[DBLP:conf/cvpr/Sphereface]. Thus, for ambiguous samples, the estimated scores
are small, reducing the impact of CE loss, and the target branch learns more
from the mined latent distributions.

###  3.4 Overall Loss function

The overall objective of DMUE is:

$\centering
L_{total}=w_{u}(e)(L_{WCE}^{target}+\omega{L_{soft}}+\gamma{L_{sp}})+w_{d}(e){L_{CE}^{aux}},\@add@centering$
(10)

where $\omega$, $\gamma$ are the hyperparameters. $w_{u}$ and $w_{d}$ are the
weighted ramp functions [DBLP:conf/iclr/ramp] the epoch e, which is formulated
as:

$w_{u}(e)=\left\\{{\begin{array}[]{*{20}{c}}{\exp(-{{(1-\frac{e}{\beta})}^{2}})}&{e\leq\beta}\\\
1&{e>\beta}\end{array}}\right.,$ (11)
$w_{d}(e)=\left\\{{\begin{array}[]{*{20}{c}}1&{e\leq\beta}\\\
{\exp(-{{(1-\frac{\beta}{e})}^{2}})}&{e>\beta}\end{array}}\right.,$ (12)

where $\beta$ is the epoch threshold for functions. where $\beta$ is the epoch
threshold. The Eq. 11 and 12 are introduced to benefit training from two
aspects: (1) At the beginning of training, the latent distributions mined by
auxiliary branches are not stable enough. Thus, we focus on training the
auxiliary branches. (2) When the auxiliary branches are well trained, we then
divert our attention to train the target branch.

It worth noting that we remove all the auxiliary branches and the uncertainty
estimation module for deployment. Our framework is end-to-end and can be
flexibly integrated with existing network architectures, without extra cost on
inference.

##  4 Experiments

We verify the effectiveness of DMUE on synthetic noisy datasets and 4 popular
in-the-wild benchmarks, and further validate the contribution of each
component of DMUE. Extensive ablation studies with respect to the
hyperparameters and the different backbone architectures are carried out to
confirm the advantage of our method.

Input: Training Images $\mathcal{X}$ and annotations $\mathcal{Y}$ with $C$
classes, $MaxEpoch$, $num\\_iters$

Output: Trained model with target branch $\theta^{0}$ and $C$ auxiliary
branches $\theta^{j},j\in\\{1,\cdots,C\\}$

/* Training */

1 Initialize $\theta^{0}$ and $\theta^{j}$ with random values,
$j\in\\{1,2,\cdots,C\\},e=1$

2 while  _$e <MaxEpoch$_ do

3 for  _$k=0$ to num_iters_ do

From $(\mathcal{X},\mathcal{Y})$, sample a batch $set_{batch}$;

// Note samples in $j$-th class as $set_{j}$

Compute $\mathcal{L}_{ce}^{aux}$ ;

// use $set_{batch}\backslash set_{j}$ to compute $\mathcal{L}_{ce}^{aux_{j}}$
for $\theta^{j},j\in\\{1,\cdots,C\\}$

4

Compute latent distribution for $set_{batch}$ ;

// Use $\theta^{j}$ predict for $set_{j},$ $j\in\\{1,\cdots,C\\}$

5

6 Compute $\mathcal{L}_{soft}$ and $\mathcal{L}_{sp}$

Compute $\mathcal{L}_{wce}^{target}$ in $\theta^{0}$ ;

// use $set_{batch}$

7

8 Update all branches $\theta^{j},j\in\\{0,1,2,\cdots,C\\}$

9 $e=e+1$;

/* Testing */

Deploy model only with the target branch $\theta^{0}$

Algorithm 1 DMUE.

###  4.1 Datasets and Metrics

RAF-DB [DBLP:conf/cvpr/raf] is constructed by 30,000 facial images with basic
or compound annotations. In the experiment, we choose the images with seven
basic expressions (neutral, happiness, surprise, sadness, anger, disgust and
fear), of which 12,271 are used for training, and the remaining 3,068 for
testing. AffectNet [DBLP:journals/affectnet] is currently the largest FER
dataset, including 440,000 images. The images are collected from the Internet
by querying the major search engines with 1,250 emotion-related keywords. Half
of the images are annotated with eight basic expressions, providing 280K
training images and 4K testing images. FERPlus [DBLP:conf/icmi/ferplus] is an
extension of FER2013 [DBLP:conf/iconip/fer2013], including 28,709 training
images and 3,589 testing images resized to 48$\times$48 gray-scale pixels.
Each image is labelled by 10 crowd-sourced annotators to one of eight
categories. For a fair comparison, the most voting category is picked as the
annotation for each image following [DBLP:conf/icmi/ferplus, resvgg,
DBLP:conf/cvpr/SCN, DBLP:journals/tip/RAN]. SFEW [DBLP:conf/sfew] contains the
images from movies with seven basic emotions, including 958 images for
training and 436 images for testing. For each dataset, we report the overall
accuracy on the testing set.

###  4.2 Implementation Details

By default, we use ResNet-18 as the backbone network pretrained on MS-Celeb-1M
with the standard routine [DBLP:conf/cvpr/SCN, DBLP:journals/tip/RAN] for a
fair comparison. The last stage and the classifier of ResNet-18 are separated
tor form auxiliary branches, while the remaining low-level layers are shared
across auxiliary and target branches. The facial images are aligned and
cropped with three landmarks [DBLP:conf/iccv/adaptiveWing], resized to
256$\times$256 pixels, augmented by random cropping to 224$\times$224 pixels
and horizontal flipped with a probability of $0.5$. During training, the batch
size is 72, and each batch is constructed to ensure every class is included.
We use Adam with weight decay of $10^{-4}$. The initial learning rate is
$10^{-3}$, which is further divided by 10 at epoch 10 and 20. The training
ends at epoch 40. Only the target branch is kept during testing. By default,
the hyperparameters are set as $T=1.2,\omega=0.5,\beta=6$ and $\gamma=10^{3}$,
according to the ablation studies. All experiments are carried out on a single
Nvidia Tesla P40 GPU which takes 12 hours to train AffectNet with 40 epochs.

###  4.3 Evaluation on Synthetic Ambiguity

The annotation ambiguity in FER mainly lies in two aspects: mislabelled
annotations and uncertain visual representation. We quantitatively evaluate
the improvement of DMUE against the mislabelled annotations on RAF-DB and
AffectNet. Specifically, a portion (10%, 20% and 30%) of the training samples
are randomly chosen, of which the labels are flipped to other random
categories. We choose ResNet-18 as the baseline and the backbone of DMUE, and
compare the performance with SCN [DBLP:conf/cvpr/SCN], which is the state-of-
the-art noise-tolerant FER method. SCN reckons uncertainty in each sample by
its visual feature, and aims to find their deterministic latent truth. Each
experiment is repeated three times, then the mean accuracy and standard
deviation on the testing set are reported. To make fair comparison, SCN is
pretrained on MS-Celeb-1M with the backbone of ResNet-18.

As shown in Table 1, the DMUE outperforms each baseline and SCN
[DBLP:conf/cvpr/SCN] consistently. With noise ratio of 30%, DMUE improves the
accuracy by 4.29% and 4.21% on RAF-DB and AffectNet, respectively. This
attributes to the mined latent distribution that can flexibly describe both
synthetic noisy samples and compound expressions in the label space. Thus, it
guides the model to overcome the harmful influence from noisy annotations.

Visualization of $\widetilde{\bm{y}}$. Qualitative results are presented in
the supplementary material to demonstrate that our approach can obtain the
latent truth for mislabelled samples, and thereby achieve performance
improvement.

###  4.4 Component Analysis

We conduct experiments on RAF-DB and AffectNet to analyse the contribution of
latent distribution mining, uncertainty estimation and similarity preserving.
As shown in Table 2, some observations can be found: (1) Latent distribution
mining plays a more important role than others. When only one component
employed, it outperforms similarity preserving and uncertainty estimation by
2.09% and 0.4% on AffectNet, 1.19% and 0.13% on RAF-DB, respectively. It
proves the benefits provided by the latent distribution, as the semantic
features of ambiguous images are well utilized. (2) When combining uncertainty
estimation and latent distribution, we achieve performance improvement by
0.74% and 0.91% over only using the latent distribution on AffectNet and RAF-
DB, respectively. It attributes to the uncertainty estimation module providing
guidance for the target branch. Thus, the target branch can flexibly adjust
the learning focus between the annotation and the latent distribution,
according to the ambiguous extent of samples. (3) Similarity preserving also
brings some improvements, while its contribution is relatively small than
others. As it benefits the learning mainly by making different branches
predict consistent relationships for image pairs, speeding up the training
convergence. We present more results of similarity preserving in the
supplementary material.

Table 1: Mean Accuracy and standard deviation ($\%$) on RAF-DB and AffectNet
with synthetic noisy annotations.

Method | Noisy(%) | RAF-DB | AffectNet
---|---|---|---
Baseline | 10 | 80.43$\pm$0.72 | 57.21$\pm$0.31
SCN [DBLP:conf/cvpr/SCN] | 10 | 81.92$\pm$0.69 | 58.48$\pm$0.62
DMUE | 10 | 83.19$\pm$0.83 | 61.21$\pm$0.36
Baseline | 20 | 78.01$\pm$0.29 | 56.21$\pm$0.31
SCN [DBLP:conf/cvpr/SCN] | 20 | 80.02$\pm$0.32 | 56.98$\pm$0.28
DMUE | 20 | 81.02$\pm$0.69 | 59.06$\pm$0.34
Baseline | 30 | 75.12$\pm$0.78 | 52.67$\pm$0.45
SCN [DBLP:conf/cvpr/SCN] | 30 | 77.46$\pm$0.64 | 55.04$\pm$0.54
DMUE | 30 | 79.41$\pm$0.74 | 56.88$\pm$0.56
|  |  |

Table 2: Accuracy ($\%$) comparison of the different components. SP denotes
the similarity preserving. Confidence denotes involving the uncertainty
estimation module for the weighted training in target branch.

Latent distribution | SP | Confidence | AffectNet | RAF-DB
---|---|---|---|---
- | - | - | 58.85 | 86.33
✓ | - | - | 61.76 | 87.84
- | ✓ | - | 59.67 | 86.65
- | - | ✓ | 61.36 | 87.71
✓ | ✓ | - | 62.34 | 88.23
- | ✓ | ✓ | 61.65 | 87.98
✓ | - | ✓ | 62.50 | 88.45
✓ | ✓ | ✓ | 62.84 | 88.76
|  |  |  |

Figure 4: The mined latent distribution is compared with the subjective
results. Each image is tagged with annotation and the KL-divergence between
two distributions. The generated latent distribution is consistent with
intuition. Best viewed in color. Zoom in for better view.
(Ne=Neutral,Ha=Happy,Sa=Sad,Su=Surprise,Fe=Fear,Di=Disgust,An=Anger,Co=Contempt).
Figure 5: (a) A Sad training image mislabelled with Neutral in AffectNet. (b)
The prediction from model trained on AffectNet can show the right class. (c)
The prediction from model trained on all classes except for the Neutral better
reflects the truth discriminatively. Best viewed in color. Zoom in for better
view.

###  4.5 Comparison with the State-of-the-art

We compare DMUE with existing state-of-the-art methods on 4 popular in-the-
wild benchmarks in Table 3(d).

Results. In Table 3(d), both CAKE [DBLP:conf/bmvc/cake], SCN
[DBLP:conf/cvpr/SCN] and RAN [DBLP:journals/affectnet] utilize ResNet-18 as
the backbone. SCN and RAN are pretrained on MS-Celeb-1M according to their
original papers. RAN mainly deals with the occlusion and head pose problem in
FER. As shown in Tabel 3(d), DMUE achieves current leading performance on
AffectNet. For RAF-DB, all three LDL-ALSG, IPA2LT and SCN are noise-tolerant
FER methods considering ambiguity, among which SCN achieves state-of-the-art
results. We further improve the performance of ambiguous FER by mining latent
distribution and considering annotations in uncertainty estimation. Table 3(d)
also shows the results on FERPlus and SFEW, respectively. Without bells and
whistles, our method achieves better performance than the counterparts.

###  4.6 Visualization Analysis

To further diagnose our method, we conduct visualizations of the discovered
latent distribution and the estimated confidence score.

Latent Distribution. In Section 4.3, we quantitatively demonstrate the
effectiveness of DUME to deal with mislabelled images. In this section, we
further conduct user study for qualitative analysis of how latent distribution
cope with uncertain expressions. Specifically, 20 images are randomly picked
from RAF-DB and AffectNet, and labelled by 50 voters. As latent distribution
reflects the sample’s probability distribution among its negative classes, we
set the number of votes on sample’s positive class to be zero. The normalized
subjective results are compared with the mined latent distribution.

In Fig. 4, KL-divergence between the subjective result and the latent
distribution is reported for reference. It is interesting to see people have
different views of the specific type of expressions. Furthermore, our approach
obtains qualitatively consistent results with human intuition. Although there
exists differences in details, it is worth noting that the results can already
qualitatively explain that the latent distribution benefits the model by
reinforcing the supervision information.

Confidence Score. To corporate with latent distribution mining, a confidence
score is estimated by the uncertainty estimation module for each image given a
batch. The more ambiguous a sample is, the lower its confidence score will be.
Thus, the target branch will learn more from its latent distribution. We
qualitatively analyse the uncertainty estimation module by visualizing images
with the original annotation and the scaled confidence score. Moreover, we
rank images by their confidence scores and report their ranks in a batch of 72
images.

In Fig. 6, we choose two typical anchor images and report their results in
three different batches. The confident samples are assigned with higher score,
while the ambiguous ones are the opposite. Furthermore, both the scores and
ranks of anchor images are consistent within three different batches. It shows
the robustness of our pairwise uncertainty estimation module. More analyses
are provided in the supplementary material.

Figure 6: From top to bottom: images from three different batches with their
annotations. Red (Green) bounding box denotes bad (good) anchor image. The
upper left (right) corner of each picture is tagged with its confidence score
(rank) in the batch. The estimated score is robust and consistent with
intuition. Best viewed in color. Zoom in for better view. Table 3: Comparison
with the state-of-the-art results. Res denotes ResNet. + denotes both
AffectNet and RAF-DB are used as the training set. ∗ means using extra
distribution data instead of single category annotation. † denotes the method
is trained and tested with 7 classes on AffectNet.

(a) Comparison on AffectNet

Method | Acc.
---|---
Upsample [DBLP:journals/affectnet] | 47.01
IPA2LT+ [DBLP:conf/eccv/IPA2LT] | 55.71
RAN [DBLP:journals/affectnet] | 59.50
CAKE† [DBLP:conf/bmvc/cake] | 61.70
SCN [DBLP:conf/cvpr/SCN] | 60.23
Ours(Res-18) | 62.84
Ours(Res-50IBN) | 63.11
|

(b) Comparison on RAF-DB

Method | Acc.
---|---
gaCNN [DBLP:journals/tip/gaCNN] | 85.07
LDL-ALSG+ [DBLP:conf/cvpr/LDL-ALSG] | 85.53
IPA2LT+ [DBLP:conf/eccv/IPA2LT] | 86.77
SCN [DBLP:conf/cvpr/SCN] | 87.03
SCN+ [DBLP:conf/cvpr/SCN] | 88.14
Ours(Res-18) | 88.76
Ours(Res-50IBN) | 89.42
|

(c) Comparison on FERPlus

Method | Acc.
---|---
PLD∗ [DBLP:conf/icmi/ferplus] | 85.10
Res+VGG [resvgg] | 87.40
SCN | 88.01
SeNet50∗ [SeNet50] | 88.80
RAN [DBLP:journals/tip/RAN] | 88.55
Ours(Res-18) | 88.64
Ours(Res-50IBN) | 89.51
|

(d) Comparison on SFEW

Method | Acc.
---|---
IdentityCNN [DBLP:conf/icip/identitycnn] | 50.98
Island loss [DBLP:conf/fgr/island] | 52.52
Incept-ResV1 [DBLP:conf/cvpr/cov_pool] | 51.90
MultiCNNs [DBLP:conf/icmi/multicnns] | 55.96
RAN [DBLP:journals/tip/RAN] | 56.40
Ours(Res-18) | 57.12
Ours(Res-50IBN) | 58.34
|

###  4.7 Ablation Study

We conduct extensive ablation studies on AffectNet, as it is the largest
dataset. Some of them are provided in the supplementary material, due to the
page limitation.

Mining latent distribution. Quantitative and qualitative experiments on
AffectNet are conducted to analyze the way of mining latent distribution. For
the former, given a batch, we train each auxiliary branch with all the
samples, where the ($C-1$)-class classifier is switched to $C$-class. To make
the latent distribution, their predictions are averaged to increase the
robustness. For simplification, we denote latent distribution mined in this
way as LD-A, while the original in DUME as LD-N.

As shown in Table 4, LD-N guides the target branch better. Because it can
reflect more discriminative latent truth. More analyses are provided in
supplementary material.

Table 4: Ablation study of ways to mine latent distribution.

Methods | Baseline | LD-A | LD-N
---|---|---|---
Acc. (%) | 58.85 | 60.03 | 61.32
|  |  |

(a)

(b)

Figure 7: (a) The accuracy (%) with different $\omega$. (b) The accuracy (%)
with different $T$.

Trade-off Weight $\omega$. $\omega$ balances the learning of target branch
between $\widetilde{\bm{y}}_{x}$ and annotation. Fig. 7 shows that too small
$\omega$ causes trouble for target branch to learn $\widetilde{\bm{y}}_{x}$.
When $\omega$ is too large, it is hard for uncertain estimation module to
adjust learning focus, as the sensitivity to $\widetilde{\bm{y}}_{x}$ is
enlarged.

Sharpen Temperature $T$. $T$ provides the flexibility to slightly modify the
entropy of $\widetilde{\bm{y}}_{x}$. Fig. 7 shows the effect with different
$T$. When $T<1$, the distribution becomes steep quickly, damaging the fine-
grained label information. Using $T>1$ flattens $\widetilde{\bm{y}}_{x}$,
relieving model’s sensitivity to incorrect predictions. Yet, the performance
will be degraded if $T$ is too large, as the pattern of
$\widetilde{\bm{y}}_{x}$ is suppressed.

Epoch Threshold $\beta$. The first $\beta$-th epoch is dedicated to
pretraining the auxiliary branches in prior, to make them provide stable
latent distribution. After the $\beta$-th epoch, attention is paid more on
optimizing the target branch. Table 5 shows the accuracy with different
$\beta$.

Similarity Preserving factor $\gamma$. We generalized the similarity
preserving to the context of multi-branch architecture. $\gamma$ adjusts the
contribution ratio of the mechanism. Fig. 8 reflects the performance of model
with different $\gamma$.

Table 5: The accuracy (%) with different $\beta$.

$\beta$ | 2 | 3 | 6 | 10 | 14
---|---|---|---|---|---
Acc. (%) | 62.28 | 62.54 | 62.84 | 62.50 | 62.41
|  |  |  |  |

Figure 8: Accuracy(%) sensitivity to $\gamma$.

##  5 Conclusion

In order to address the ambiguity problem in FER, we propose DMUE, with the
design of latent distribution mining and pairwise uncertainty estimation. On
one hand, the mined latent distribution describes the ambiguous instance in a
fine-grained way to guide the model. On the other hand, pairwise relationships
between samples are fully exploited to estimate the ambiguity degree. Our
framework imposes no extra burden on inference, and can be flexibly integrated
with the existing network architectures. Experiments on popular benchmarks and
synthetic ambiguous datasets show the effectiveness of DMUE.

## Acknowledgements

This work is supported by the National Key R&D Program of China under Grant
No. 2020AAA0103800 and by the National Natural Science Foundation of China
under Grant U1936202 and 62071216. This work is mainly done at JD AI Research.

##  6 Appendix

###  6.1 The Generalized Similarity Preserving Loss

In this section, we give the generalized formulation of $L_{sp}$, which is
defined as the following in the main text:

$\centering{L_{sp}}=MSP(G_{aux}^{1},\cdots,G_{aux}^{C},G_{tar}),\@add@centering$
(13)

where $G_{aux}^{i}\in\mathbb{R}^{N_{i}\times{N_{i}}}$ and
$G_{tar}\in\mathbb{R}^{N\times{N}}$ denote the similarity matrices calculated
by the auxiliary and target branches, respectively. $N$ is the batch size, and
$N_{i}$ is the number of images that are not annotated to the $i$-th class in
the batch. For coding simplicity, given an image batch, we define
$\bm{A}_{tar}\in\mathbb{R}^{N\times{N}}$ and
$\bm{A}_{aux}^{i}\in\mathbb{R}^{N\times{N}}$ ($i\in\\{1,2,\cdots,C\\}$), and
the $j$-th row $\bm{a}_{tar_{j}}$ of $\bm{A}_{tar}$ and $\bm{a}_{aux_{j}}^{i}$
of $\bm{A}_{aux}^{i}$ are denoted as:

$\centering{\bm{a}_{tar_{j}}}={\frac{{{{\bm{f}_{tar_{j}}}\cdot{\bm{f}_{tar}^{T}}}}}{{\left\|{{{\bm{f}_{tar_{j}}}\cdot{\bm{f}_{tar}^{T}}}}\right\|_{2}}},}\@add@centering$
(14)
$\centering{\bm{a}_{aux_{j}}^{i}}={\frac{{{{\bm{f}_{aux_{j}}^{i}}\cdot{\bm{f}{{}_{aux}^{i}}^{T}}}}}{{\left\|{{\bm{f}_{aux_{j}}^{i}}\cdot{\bm{f}{{}_{aux}^{i}}^{T}}}\right\|_{2}}},}\@add@centering$
(15)

where $\bm{f}_{tar}\in\mathbb{R}^{N\times{d}}$ and
$\bm{f}{{}_{aux}^{i}}\in\mathbb{R}^{N\times{d}}$ are the semantic features in
the target branch and the auxiliary branch $i$, respectively,
$\bm{f}_{tar_{j}}\in\mathbb{R}^{1\times{d}}$ and
$\bm{f}_{aux_{j}}^{i}\in\mathbb{R}^{1\times{d}}$ are the $j$-th row of
$\bm{f}_{tar}$ and $\bm{f}{{}_{aux}^{i}}$, respectively, and $d$ is the
feature dimension. Then, we implement $L_{sp}$ by masking $\bm{A}_{tar}$ and
$\bm{A}_{aux}^{i}$, which $L_{sp}$ can be rewritten as:

$\centering{L_{sp}}=\frac{1}{C}\sum\limits_{j=1}^{C}{\frac{1}{{N_{i}^{2}}}}\left\|\bm{M}^{i}\ast{\bm{A}_{tar}-\bm{M}^{i}\ast\bm{A}_{aux}^{i}}\right\|_{F}^{2},\@add@centering$
(16)

where $\ast$ is the element-wise product. The $q$-th row and $p$-th column
element $m_{q,p}^{i}$ of $\bm{M}^{i}{\in\mathbb{R}^{N\times{N}}}$ is defined
as:

$m_{q,p}^{i}=\left\\{{\begin{array}[]{*{20}{c}}0&{y_{p}=i\ or\ y_{q}=i}\\\
1&{Others}\end{array}}\right.,$ (17)

where $y_{p}$ and $y_{q}$ denote the annotations of the $p$-th and $q$-th
images in the batch, respectively ($y_{p},y_{q}\in\\{1,\cdots,C\\}$). $L_{sp}$
is easy to be implemented by a few lines of code††Our source code and pre-
trained models will be released..

Benefits of $L_{sp}$. The non-zero elements in ${\bm{A}}_{tar}$ and
${\bm{A}}_{aux}^{i}$ represent the predicted similarity values of image pairs.
With the constraint of $L_{sp}$, all the branches are regularized to predict
consistent similarity value for an image pair. As shown in Fig. 9, similarity
preserving makes the training more stable. The loss value rises up a little
around the $20k$-th iteration step because the ramp functions gradually assign
larger weight for $L_{wce}^{target}$ to train the target branch.

Figure 9: The loss curves of training with $L_{sp}$ (the red curve) and
without $L_{sp}$ (the blue curve) along iterations. Experiments are conducted
on AffectNet with ResNet-18 as the backbone architecture.

###  6.2 Evaluation of Synthetic Ambiguity

To better demonstrate the superiority of latent distribution mining, we
conduct qualitative analyses on synthetic mislabelled samples. Specifically, a
portion of training samples are randomly chosen of which the labels are
flipped to other categories. Then, we use DMUE to train the network on the
synthetic mislabelled data and visualize the mined latent distributions in
Fig. 11. We can observe that the mined latent distribution is able to correct
the noisy annotation well. Taking the last image in Fig. 11 as an example,
although its true label Anger is flipped to Happy, the latent distribution
well reflects its true class Anger. Moreover, the latent distribution also has
the capacity to reflect the second possible class for compound expressions.
For the first image, the latent distribution reflects its second possible
class Anger, which is in line with the subjective perception. By imposing the
latent distribution as the additional supervision, DMUE effectively utilizes
the semantic features of samples.

###  6.3 Ablation Study

Different Backbone Networks. As described in the main text, DMUE is
independent to the backbone architectures. We further apply DMUE to
ShuffleNetV1 [DBLP:conf/cvpr/shufflenet] and MobileNetV2
[DBLP:conf/cvpr/mobilenet] to demonstrate the universality of DMUE, where the
last stage of ShuffleNetV1 and the last two stages of MobileNetV2 are
separated for latent distribution mining. The results on AffectNet and RAF-DB
are presented in Table 6. We observe that DMUE can stably improve the
performance of all the architectures, including ShuffleNetV1, MobileNetV2,
ResNet-18 and ResNet-50IBN, by an average of 4.30% and 2.47% on AffectNet and
RAF-DB, respectively. In addition, the ResNet50-IBN achieves the best record
among these architectures because of the large number of parameters and the
IBN module. Furthermore, we report the results of training DMUE from scratch
on AffectNet and RAF-DB in Table 7. Similar observations can also be found
without using the pre-trained model on MS-Celeb-1M.

Backbone Architecture | DMUE | AffectNet | RAF-DB
---|---|---|---
ShuffleNetV1 (group=3;2.0$\times$) | - | 56.51 | 86.20
ShuffleNetV1 (group=3;2.0$\times$) | ✓ | 60.87 | 88.73
MobileNetV2 | - | 57.65 | 86.01
MobileNetV2 | ✓ | 62.34 | 87.97
ResNet-18 | - | 58.85 | 86.33
ResNet-18 | ✓ | 62.84 | 88.76
ResNet50-IBN | - | 58.94 | 86.57
ResNet50-IBN | ✓ | 63.11 | 89.51
|  |  |

Table 6: Accuracy ($\%$) on RAF-DB and AffectNet with pre-training on MS-
Celeb-1M.

Backbone Architecture | DMUE | AffectNet | RAF-DB
---|---|---|---
ShuffleNetV1 (group=3;2.0$\times$) | - | 55.02 | 85.65
ShuffleNetV1 (group=3;2.0$\times$) | ✓ | 59.67 | 88.10
MobileNetV2 | - | 54.94 | 85.44
MobileNetV2 | ✓ | 60.43 | 88.15
ResNet-18 | - | 55.22 | 86.01
ResNet-18 | ✓ | 61.22 | 88.33
ResNet50-IBN | - | 55.52 | 85.67
ResNet50-IBN | ✓ | 60.54 | 88.94
|  |  |

Table 7: Accuracy ($\%$) on RAF-DB and AffectNet without pre-training on MS-
Celeb-1M.

Mining latent distribution. In the main text Table 4, we describe the
quantitative comparison aiming at investigating which way to mine latent
distribution is better. In the quantitative comparison, we train the auxiliary
branches with the whole image batch, and their predictions for each image are
averaged, denoted as LD-A. As shown in Fig. 12, LD-A reflects the visual
feature of images to some extent. But the second and the third possible class
of a compound expression is not discriminative in LD-A.

In Fig 10, we provide a toy example of latent label space some intermediate
iterations. The latent distribution is completely random at the beginning. It
gradually reflects the visual feature of the sample during iterations.

###  6.4 More Results of User Study

As described in the main text, we pick 20 images from FER datasets and have
them labelled by 50 volunteers. We provide more visualization results of the
mined latent distribution and the perception from volunteers in Fig. 13.
Accordingly, we draw the following conclusions: (1) One inherent property of
facial expression is that compound facial expressions may exist. It is easy
for volunteers to have disagreements with the exact type of images whose
annotations in dataset are Fear, Disgust, Sad and Anger. One reason may be
that folds in the region of eyebrow are often involved in those easily
confused expressions. Thus, they may share some common visual features, making
it hard to define the exact expression type from a static image. (2) The main
goal of latent distribution is to provide reasonable guidance to the target
branch, rather than finding the exact label distribution of a facial
expression image. Thus, we utilize the $L_{2}$ loss to minimize the deviation
because it is bounded and less sensitive to the incorrect prediction.

Why auxiliary branches work? Based on the conclusions above, we find the one-
hot label is hard to represent the visual features of expressions. The
annotation of face expression is subjective and difficult, because different
expressions naturally entangle each other in the visual space. Auxiliary
branches are proposed to disentangle such connections. Each auxiliary branch
is a classifier that maps images to their latent classes. By doing so, we
disentangle the ambiguity in the label space.

Figure 10: The iteratively updated latent distribution of the sample from Fig.
4 in the main text. Best viewed in color. Zoom in for better view.

###  6.5 More Results and Mathematical Reason for the Uncertainty Estimation

Assume a class center feature $\bm{c}_{i}$ of the $i$-th class, the angle
between a $i$-th class sample’s feature $\bm{x}$ and $\bm{c}_{i}$ is $\theta$,
that $\theta=acos(\left\langle\bm{x},\bm{c}_{i}\right\rangle)$. Without losing
generality, assume $\theta\sim\mathcal{N}(0,\sigma^{2})$ in [$-\pi,\pi$] (as
the ambiguity increases, the number of samples decreases). Given a sample
$\bm{a}$ with semantic feature $\bm{f}$, that
$\left\langle\bm{f},\bm{c}_{i}\right\rangle=cos\alpha$. We have:

$\centering{{S}_{a,i}}=\mathop{E}\limits_{\theta\sim\mathcal{N}(0,{\sigma^{2}}),\theta\in[-\pi,\pi]}\\{<\bm{x},\bm{f}>\\},\@add@centering$
(18)

where $\|\bm{c}_{i}\|=\|\bm{x}\|=\|\bm{f}\|=1$. We first study the equation in
a special case where $\alpha=\frac{\pi}{2}$. We denote $\bm{x}^{\bot}$ as the
projection from $\bm{x}$ to the linear subspace $\bm{W}$:

$\centering\bm{x}=\bm{c}_{i}+\bm{x}^{\bot},\@add@centering$ (19)

$\mathbb{R}^{n}$ = $span\\{\bm{c}_{i}\\}\oplus W$, $\oplus$ is the direct sum.
Let $\\{\bm{z}_{1},\cdots,\bm{z}_{n-1}\\}$ is a set of basis of $\bm{W}$. We
have $\bm{x}^{\bot}$ under uniform distribution as prior for Softmax or other
angle-based loss, that is $E\\{\left<\bm{x}^{\bot},\bm{z}_{k}\right>\\}=0$,
with $1\leq k\leq n-1$. As $\alpha=\frac{\pi}{2}$ in this special case,
$\bm{f}$ can be rewritten as:

$\centering\bm{f}=\sum\nolimits_{k}^{n-1}{\omega_{k}\bm{z}_{k}}.\@add@centering$
(20)

We have $\bm{f}\bot\bm{c}_{i}$, and have:

$\displaystyle{E\\{<\bm{x}^{\bot},\bm{f}>\\}}$
$\displaystyle=E\\{\bm{x}^{\bot}\cdot\sum\nolimits_{k}^{n-1}{\omega_{k}\bm{z}_{k}}\\}$
(21)
$\displaystyle=\sum\nolimits_{k}^{n-1}{\omega_{k}E\\{\bm{x}^{\bot}\cdot{\bm{z}_{k}}}\\}$
$\displaystyle=0,$ $\displaystyle E\\{<\bm{x},\bm{f}>\\}$ $\displaystyle=0.$

Now, for the general case, we construct:

$\left\\{\begin{array}[]{lr}\bm{f}_{1}=<\bm{f},\bm{c}_{i}>\bm{c_{i}},&\\\
\bm{f}_{2}=\bm{f}-\bm{f}_{1},&\end{array}\right.$ (22)

$\centering<\bm{x},\bm{f}>=<\bm{x},\bm{f}_{1}>+<\bm{x},\bm{f}_{2}>.\@add@centering$
(23)

We notice that $\bm{f}_{2}\bot\bm{c}_{i}$:

$\displaystyle<\bm{f}_{2},\bm{c}_{i}>$
$\displaystyle=<\bm{f},\bm{c}_{i}>-<\bm{f}_{1},\bm{c}_{i}>$ (24)
$\displaystyle=<\bm{f},\bm{c}_{i}>-<<\bm{f},\bm{c}_{i}>\bm{c_{i}},\bm{c}_{i}>$
$\displaystyle=<\bm{f},\bm{c}_{i}>-<\bm{f},\bm{c}_{i}>\|\bm{c}_{i}\|$
$\displaystyle=0.$

From Eq. 21, we have $E\\{\left<\bm{x},\bm{f}_{2}\right>\\}=0$, so we have:

$\displaystyle{{S}_{a,i}}$
$\displaystyle=\mathop{E}\limits_{\theta\sim\mathcal{N}(0,{\sigma^{2}}),\theta\in[-\pi,\pi]}\\{<\bm{x},\bm{f}>\\}$
(25)
$\displaystyle=\mathop{E}\limits_{\theta\sim\mathcal{N}(0,{\sigma^{2}}),\theta\in[-\pi,\pi]}\\{<\bm{x},\bm{f}_{1}>\\}$
$\displaystyle=\mathop{E}\limits_{\theta\sim\mathcal{N}(0,{\sigma^{2}}),\theta\in[-\pi,\pi]}\\{<\bm{x},<\bm{f},\bm{c}_{i}>\bm{c_{i}}>\\}$
$\displaystyle=<\bm{f},\bm{c}_{i}>\mathop{E}\limits_{\theta\sim\mathcal{N}(0,{\sigma^{2}}),\theta\in[-\pi,\pi]}\\{<\bm{x},\bm{c_{i}}>\\}$
$\displaystyle=\cos\alpha\mathop{E}\limits_{\theta\sim\mathcal{N}(0,{\sigma^{2}}),\theta\in[-\pi,\pi]}\\{\cos\theta\\}.$

Obviously, if $\bm{a}$ is the $j$-th class sample mislabelled to the $i$-th
class, then $|\alpha|$ is large, ${S}_{a,i}$ becomes small and ${S}_{a,j}$
becomes large, which is contrary to the concatenated label. Thus, we can
estimate the uncertainty from $\bm{SV}_{a}$ as it carries ambiguity
information.

We present more visualization results of the estimated uncertainty score in
Fig. 14, where lower scores mean more ambiguous images. It is obvious that the
estimated uncertainty level is in line with the subjective perception. With
the uncertainty estimation module, DMUE is able to suppress the adverse
influence from ambiguous data, encouraging the network to utilize the semantic
features and learn the latent distribution for the ambiguous image.

Figure 11: DMUE yields the latent truth for noisy samples. The bottom of each
image is tagged by its original annotation. The top of each image is tagged by
manually flipped noisy label. DMUE is adopted to train network on synthetic
noisy datasets. We visualize the mined latent distribution for synthetic noisy
samples at the right of each image. The mined latent distribution is in line
with the human subjective perception, where the most possible class reflected
by latent distribution is corresponding to the original annotation.
(Neu=Neutral, Hap=Happy, Sad=Sad, Sur=Surprise, Fea=Fear, Dis=Disgust,
Ang=Anger, Con=Contempt) Figure 12: Qualitative comparison between LD-A and
LD-N. The red bar denotes the positive class predicted in LD-A. The purple bar
denotes the second possible class predicted in LD-A and LD-N for ambiguous
images. (a) Images tagged by their original annotation. (b) The LD-A can
reflect the visual feature to a certain extent, yet the images’ possibility
distribution among its negative classes is not discriminative. (c) The LD-N
that we used in DMUE, describes an image on its negative classes
discriminatively. (Ne=Neutral, Ha=Happy, Sa=Sad, Su=Surprise, Fe=Fear,
Di=Disgust, An=Anger, Co=Contempt) Figure 13: More visualizations of the mined
latent distributions and subjective survey results. The age of 50 volunteers
range from 17 to 51. Each image is tagged with its annotation. The orange bar
denotes the mined latent distribution and the blue bar denotes the subjective
survey results. To process the votes from volunteers, we set the number of
votes on each sample’s positive class as zero. Then we normalize the results,
which reflect the probability that image belonging to each negative class. As
we can see, the mined latent distribution is consistent with human intuition
in general. (Ne=Neutral, Ha=Happy, Sa=Sad, Su=Surprise, Fe=Fear, Di=Disgust,
An=Anger, Co=Contempt) Figure 14: More visualization results of the estimated
confidence score. From top to bottom, each row presents images from the same
batch. The bottom of each image is tagged with its annotation from the
dataset. The upper left of each image is tagged with its estimated uncertainty
score. Lower scores are assigned to those more ambiguous images. The upper
right of each image is tagged with its confidence rank in the batch. From left
to right, we present images with their confidence scores in an ascending
order. Images near the right side of the figure are less ambiguous, while
images near the left side are in the opposite. In general, we observe that the
estimated uncertainty score is in line with the subjective perception.
Moreover, we insert an anchor image which is annotated to Fear in two
different batches (the green bounding box in the $5$-th and $7$-th row). The
uncertainty estimation module predicts consistent confidence score for this
anchor image, which indicates the stability of our uncertainty estimation
module.