# Thermodynamics of Fractal Universe

Ahmad<EMAIL_ADDRESS>Zeinab Teimoori 3 and Bin Wang
<EMAIL_ADDRESS>1 Physics Department and Biruni Observatory, College of
Sciences, Shiraz University, Shiraz 71454, Iran

2 Research Institute for Astronomy and Astrophysics of Maragha (RIAAM), P.O.
Box 55134-441, Maragha, Iran
3 Department of Physics, Shahid Bahonar University, P.O. Box 76175, Kerman,
Iran
4 INPAC and Department of Physics, Shanghai Jiao Tong University, Shanghai
200240, China

###### Abstract

We investigate the thermodynamical properties of the apparent horizon in a
fractal universe. We find that one can always rewrite the Friedmann equation
of the fractal universe in the form of the entropy balance relation $\delta
Q=T_{h}d{S_{h}}$, where $\delta Q$ and $T_{h}$ are the energy flux and Unruh
temperature seen by an accelerated observer just inside the apparent horizon.
We find that the entropy $S_{h}$ consists two terms, the first one which obeys
the usual area law and the second part which is the entropy production term
due to nonequilibrium thermodynamics of fractal universe. This shows that in a
fractal universe, a treatment with nonequilibrium thermodynamics of spacetime
may be needed. We also study the generalized second law of thermodynamics in
the framework of fractal universe. When the temperature of the apparent
horizon and the matter fields inside the horizon are equal, i.e. $T=T_{h}$,
the generalized second law of thermodynamics can be fulfilled provided the
deceleration and the equation of state parameters ranges either as $-1\leq
q<0$, $-1\leq w<-1/3$ or as $q<-1$, $w<-1$ which are consistent with recent
observations. We also find that for $T_{h}=bT$, with $b<1$, the GSL of
thermodynamics can be secured in a fractal universe by suitably choosing the
fractal parameter $\beta$.

##  I Introduction

Nowadays, it is a general belief that there is a deep connection between
thermodynamics and gravity. The story started with the discovery of black
holes thermodynamics in $1970$’s by Hawking and Bekenstein Haw1 ; Bek ; Haw ;
Bek1 . According to their discovery, a black hole can be regarded as a
thermodynamic system, with temperature and entropy proportional to its surface
gravity and horizon area, respectively. After that, people were speculating
that maybe there is a direct connection between thermodynamics and Einstein
equation, a hyperbolic second order partial differential equation for the
spacetime metric. In $1995$, Jacobson Jac  was indeed able to derive the
Einstein equation from the requirement that the Clausius relation $\delta
Q=T\delta S$ holds for all local acceleration horizons through each spacetime
point, where $\delta S$ is one-quarter the horizon area change in Planck units
and $\delta Q$ and $T$ are the energy flux across the horizon and the Unruh
temperature seen by an accelerating observer just inside the horizon.
Jacobson’s derivation of the Einstein field equation from thermodynamics
opened a new window for understanding the thermodynamic nature of gravity.
After Jacobson, a lot of works have been done to disclose the profound
connection between gravity and thermodynamics. It was shown that the
gravitational field equations in a wide range of theories, can be rewritten in
the form of the first law of thermodynamics and vice versa Elin ; Cai1 ; Akb ;
Pad1 ; Pad2 ; Pad3 ; Par ; Kot ; Pad4 . The studies were also generalized to
the cosmological setup, where it was shown that the differential form of the
Friedmann equation in the Friedmann-Robertson-Walker (FRW) universe can be
transformed to the first law of thermodynamics on the apparent horizon Cai2 ;
Cai3 ; CaiKim ; Fro ; Dan ; Abdalla ; WANG ; Myu ; Cai4 ; Shey1 ; Shey2 ;
Shey3 .

On the other side, the second law of black hole mechanics expresses that the
total area of the event horizon of any collection of classical black holes can
never decrease, even if they collide and swallow each other. This is
remarkably similar to the second law of thermodynamics where the area is
playing the role of entropy. Note that the second law of black hole
thermodynamics can be violated if one take into account the quantum effect,
such as the Hawking radiation. To overcome this difficulty, Bekenstein Bek ;
Bek1  introduced the so-called total entropy $S_{\rm tot}$ which is defined as

$S_{\rm tot}=S_{h}+S_{m},$ (1)

where $S_{h}$ and $S_{m}$ are, respectively, the black hole entropy and the
entropy of the surrounding matter. According to Bekenstein’s argument, in
general, the total entropy should be a non decreasing function. This statement
is known as the generalized second law (GSL) of thermodynamics,

$\triangle S_{\rm tot}\geq 0.$ (2)

Besides, if thermodynamical interpretation of gravity near the apparent
horizon is a generic feature, one needs to verify whether the results may hold
not only for more general spacetimes but also for the other principles of
thermodynamics, especially for the GSL of thermodynamics. The GSL of
thermodynamics is a universal principle governing the evolution of the
universe. It was argued that in the accelerating universe the GSL is valid
provided the boundary of the universe is chosen the apparent horizon wang1 ;
wang2 ; Shey4 ; Shey5 ; Shey6 .

In this paper, we would like to extend the study to the fractal universe.
Fractal cosmology was recently proposed by Calcagni Calc1 ; Calc2  for a
power-counting renormalizable field theory living in a fractal spacetime. It
is interesting to see whether the Friedmann equation of a fractal universe can
be written in the form of the first law of thermodynamics. As we will see, in
a fractal universe, the Friedmann equation can be transformed to Clausius
relation, but a treatment with nonequilibrium thermodynamics of spacetime is
needed.

In the next section we review the basic equations in the framework of fractal
cosmology. In section III, we show that the Friedmann equation of a fractal
universe can be written in the form of the fundamental relation $\delta
Q=T_{h}dS_{h}$, where $\delta Q$ and $T_{h}$ are, respectively, the energy
flux and Unruh temperature seen by an accelerated observer just inside the
apparent horizon. In section IV, we check the validity of the GSL of
thermodynamics for a fractal cosmology. The last section is devoted to some
concluding remarks.

##  II Fractal universe

The total action of Einstein gravity in a fractal spacetime is given by Calc1
; Calc2

$S=S_{G}+S_{m},$ (3)

where the gravitational part of the action is given by

$S_{G}=\frac{1}{16\pi G}\int
d\varrho(x)\sqrt{-g}(R-2\Lambda-\omega\partial_{\mu}\upsilon\partial^{\mu}\upsilon),$
(4)

and the matter part of the action is

$S_{m}=\int d\varrho(x)\sqrt{-g}\mathcal{L}_{m}.$ (5)

Here $g$ is the determinant of the dimensionless metric $g_{\mu\nu}$,
$\Lambda$ and $R$ are, respectively, the cosmological constant and Ricci
scalar. $\upsilon$ is the fractional function and $\omega$ is the fractal
parameter. The standard measure $d^{4}x$ replaced with a Lebesgue-Stieltjes
measure $d\varrho(x)$. The derivation of the Einstein equations goes almost
like in scalar-tensor models. Taking the variation of the action (3) with
respect to the FRW metric $g_{\mu\nu}$, one can obtain the Friedmann equations
in a fractal universe as Calc2

$H^{2}+\frac{k}{a^{2}}+H\frac{\dot{\upsilon}}{\upsilon}-\frac{\omega}{6}\dot{\upsilon}^{2}=\frac{8\pi
G}{3}\rho+\frac{\Lambda}{3},$ (6)
$\dot{H}+H^{2}-H\frac{\dot{\upsilon}}{\upsilon}+\frac{\omega}{3}\dot{\upsilon}^{2}-\frac{1}{2}\frac{\square\upsilon}{\upsilon}=-\frac{8\pi
G}{6}(\rho+3p)+\frac{\Lambda}{3},$ (7)

where $H={\dot{a}}/{a}$ is the Hubble parameter, $\rho$ and $p$ are the total
energy density and pressure of the ideal fluid composing the universe,
respectively. The curvature constant $k=0,1,-1$ corresponding to a flat,
closed and open universe, respectively. The continuity equation in a fractal
universe takes the form Calc2

$\dot{\rho}+\left(3H+\frac{\dot{\upsilon}}{\upsilon}\right)(\rho+p)=0.$ (8)

It is clear that for $\upsilon=1$, the standard Friedmann equations are
recovered. We further assume that only the time direction is fractal, while
spatial slices have usual geometry. Indeed, in the framework of fractal
cosmology, classically fractals can be timelike $[\upsilon=\upsilon(t)]$ or
even spacelike $[\upsilon=\upsilon(x)]$ (see Ref. Calc2  for details). These
two cases lead to different classical physics, but at quantum level all
configurations should be taken into account, so there is no quantum analogue
of space or timelike fractals. In this paper we take a timelike fractal. Thus,
those parameters that depend on time change and those parts that related to
$x$ remain fixed.

Assuming a timelike fractal profile $\upsilon=t^{-\beta}$ Calc2 , where
$\beta=4(1-\alpha)$ is the fractal dimension, the Friedmann equations (6) and
(7) in the absence of the cosmological constant can be written as

$H^{2}+\frac{k}{a^{2}}-\frac{\beta}{t}H-\frac{\omega\beta^{2}}{6t^{2(\beta+1)}}=\frac{8\pi
G}{3}\rho,$ (9)
$\dot{H}+H^{2}-\frac{\beta}{2t}H+\frac{\beta(\beta+1)}{2t^{2}}+\frac{\omega\beta^{2}}{3t^{2(\beta+1)}}=-\frac{8\pi
G}{6}(\rho+3p),$ (10)

while, the continuity equation (8) takes the form

$\dot{\rho}+\left(3H-\frac{\beta}{t}\right)(\rho+p)=0.$ (11)

From the definition of the fractional integral Calc2 ; Hilfer , we know that
$\alpha$ ranges as $0<\alpha\leqslant 1$. Thus for $\alpha=1$, we obtain
$\beta=0$ which physically means that the universe does not have any fractal
structure and one can recovers the well-known Friedmann equations in standard
cosmology. As one can see from Friedmann equations (9) and (10), we have no
limit $t\rightarrow 0$ for a timelike fractal profile, since in this case the
Friedmann equations diverge unless $\beta=0$. This implies that at the early
stages of the universe, we could not have the timelike fractal structure.

In the remaining part of this paper we show that the differential form of the
Friedmann equation (9) can be written in the form of the fundamental relation
$\delta Q=T_{h}dS_{h}$, where $S_{h}$ is the entropy associated with the
apparent horizon. We also investigate the validity of the GSL of
thermodynamics for the fractal universe surrounded by the apparent horizon.

##  III First law of thermodynamics in fractal cosmology

For a homogenous and isotropic FRW universe the line elements can be written

$ds^{2}={h}_{\mu\nu}dx^{\mu}dx^{\nu}+\tilde{r}^{2}(d\theta^{2}+\sin^{2}\theta
d\phi^{2}),$ (12)

where $\tilde{r}=a(t)r$, $x^{0}=t,x^{1}=r$, and $h_{\mu\nu}$=diag
$(-1,a^{2}/(1-kr^{2}))$ is the two dimensional metric. The dynamical apparent
horizon, a marginally trapped surface with vanishing expansion, is determined
by the relation $h^{\mu\nu}\partial_{\mu}\tilde{r}\partial_{\nu}\tilde{r}=0$.
Straightforward calculation gives the apparent horizon radius for the FRW
universe as Shey5

$\tilde{r}_{A}=\frac{1}{\sqrt{H^{2}+k/a^{2}}}.$ (13)

The associated temperature $T$ with the apparent horizon is given by

$T_{h}=\frac{1}{2\pi\tilde{r}_{A}},$ (14)

where $A=4\pi\tilde{r}_{A}^{2}$ is the apparent horizon area and we have
assumed the apparent horizon radius is fixed. We shall assume the matter
source in the fractal universe has a perfect fluid form with stress-energy
tensor

$T_{\mu\nu}=(\rho+p)u_{\mu}u_{\nu}+pg_{\mu\nu},$ (15)

where $\rho$ and $p$ are the energy density and pressure, respectively.
Assuming the total energy inside the apparent horizon is given by $E=\rho V$,
where $V=\frac{4}{3}\pi\tilde{r}_{A}^{3}$ is the volume of the 3-sphere of
radius $\tilde{r}_{A}$. Taking differential form of energy, we can obtain the
energy flux $dE=Vd\rho$. Here we have assumed the volume enveloped by the
apparent horizon is fixed during the infinitesimal internal of time $dt$. Thus
$-dE$ is actually just the heat flux $\delta Q$ in Jac  crossing the apparent
horizon within an infinitesimal internal of time $dt$, it is not the change in
the matter energy inside the apparent horizon due to the volume change, so
there is no term of volume change. Hence we can write

$dE=V\dot{\rho}dt.$ (16)

Note that by taking $\upsilon=t^{-\beta}$, from the continuity equation (11)
we have $\rho\sim(a^{3}\upsilon(t))^{-(1+w)}$, which shows that the measure
weight $\upsilon$ is hidden in $\rho$ and thus it is not necessary to consider
its variation in Eq. (16) separately.

Substituting $\dot{\rho}$ from the continuity equation (11), we get

$dE=-4\pi
H\tilde{r}_{A}^{3}(\rho+p)dt+\frac{4\pi}{3}\frac{\beta}{t}(\rho+p)\tilde{r}_{A}^{3}dt.$
(17)

Differentiating Friedmann equation (9) with respect to the cosmic time $t$ and
using the continuity equation(11), we find

$\displaystyle
H\left(\dot{H}-\frac{k}{a^{2}}\right)-\frac{\beta}{2t}\dot{H}+\frac{\beta}{2t^{2}}H+\frac{\omega\beta^{2}(\beta+1)}{6t^{2\beta+3}}=$
$\displaystyle-4\pi GH(\rho+p)+\frac{4\pi
G}{3}\left(\frac{\beta}{t}\right)(\rho+p).$ (18)

Multiplying the factor $(-\tilde{r}_{A}^{3})$ on both sides of Eq.(III), we
reach

$\displaystyle-H\left(\dot{H}-\frac{k}{a^{2}}\right)\tilde{r}_{A}^{3}+\frac{\beta}{2t}\dot{H}\tilde{r}_{A}^{3}-\frac{\beta}{2t^{2}}H\tilde{r}_{A}^{3}-\frac{\omega\beta^{2}(\beta+1)}{6t^{2\beta+3}}\tilde{r}_{A}^{3}$
$\displaystyle=4\pi GH(\rho+p)r_{A}^{3}-\frac{4\pi
G}{3}\frac{\beta}{t}(\rho+p)\tilde{r}_{A}^{3}.$

Differentiating Eq.(13) with respect to the cosmic time $t$, we obtain

$\dot{\tilde{r}}_{A}=-H\left(\dot{H}-\frac{k}{a^{2}}\right)\tilde{r}_{A}^{3}.$
(20)

Substituting Eq.(20) into (III) we can rewrite it as

$\displaystyle\frac{1}{G}\left[d\tilde{r}_{A}-\beta\left(\frac{H}{2t^{2}}-\frac{\dot{H}}{2t}\right)\tilde{r}_{A}^{3}dt-\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}\tilde{r}_{A}^{3}dt\right]$
(21) $\displaystyle=4\pi
H(\rho+p)\tilde{r}_{A}^{3}dt-\frac{4\pi}{3}\frac{\beta}{t}(\rho+p)\tilde{r}_{A}^{3}dt.$

Combining Eq. (17) with (21) and using the fact that $\delta Q=-dE$ is just
the energy flux crossing through the apparent horizon, we have

$\delta
Q=\frac{1}{G}\left[d\tilde{r}_{A}-\beta\left(\frac{H}{2t^{2}}-\frac{\dot{H}}{2t}\right)\tilde{r}_{A}^{3}dt-\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}\tilde{r}_{A}^{3}dt\right].$
(22)

Eq. (22) can be further rewritten as

$\displaystyle\delta
Q=\frac{1}{2\pi\tilde{r}_{A}}\frac{2\pi\tilde{r}_{A}}{G}\left[d\tilde{r}_{A}-\beta\left(\frac{H}{2t^{2}}-\frac{\dot{H}}{2t}\right)\tilde{r}_{A}^{3}dt\right.$
$\displaystyle\left.-\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}\tilde{r}_{A}^{3}dt\right].$
(23)

Using definition (14) for the temperature, one can see that Eq. (III) is just
the entropy balance relation,

$\delta Q=T_{h}dS_{h},$ (24)

provided we define

$d{S_{h}}=\frac{2\pi\tilde{r}_{A}d\tilde{r}_{A}}{G}+\frac{2\pi\beta}{G}\left[\left(\frac{\dot{H}}{2t}-\frac{H}{2t^{2}}\right)-\frac{\omega}{6}\frac{\beta(\beta+1)}{t^{2\beta+3}}\right]\tilde{r}_{A}^{4}dt.$
(25)

Integrating, we find

$\displaystyle
S_{h}=\frac{A}{4G}+\frac{2\pi\beta}{G}\int\left[\left(\frac{\dot{H}}{2t}-\frac{H}{2t^{2}}\right)-\frac{\omega\beta(\beta+1)}{6t^{2\beta+3}}\right]\tilde{r}_{A}^{4}dt.$

As one can see, in a fractal universe, the entropy $S_{h}$ associated with the
apparent horizon consists two parts, the first one obeys the usual area law
and the second part is the entropy term developed internally in the system as
a result of being out of equilibrium Elin ; noneq . The entropy production
rate vanishes for standard cosmology where $\beta=0$. It is worth mentioning
that even in standard cosmology one can still have non-equilibrium
thermodynamics depending on the assumptions Elin .

##  IV GSL of thermodynamics in fractal universe

In this section we investigate the validity of the GSL of thermodynamics in a
region enclosed by the apparent horizon in the framework of the fractal
universe. Let us put $k=0$ for simplicity, so we have
${\tilde{r}}_{A}={1}/{H}$,
$\dot{H}=-{\dot{\tilde{r}}_{A}}/{\tilde{r}_{A}^{2}}$. The total entropy
associated with the apparent horizon, $S_{h}$, can be written

$\displaystyle
dS_{h}=\frac{2\pi}{G}\left[\tilde{r}_{A}d\tilde{r}_{A}-\frac{\beta}{2t}\tilde{r}_{A}^{2}d{\tilde{r}_{A}}-\frac{\beta}{2t^{2}}\tilde{r}_{A}^{3}dt\right.$
$\displaystyle\left.-\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}\tilde{r}_{A}^{4}dt\right].$
(27)

Dividing Eq.(IV) by $dt$, we arrive at

$\displaystyle\dot{S}_{h}=\frac{2\pi}{G}\left[\tilde{r}_{A}\dot{\tilde{r}}_{A}-\frac{\beta}{2t}\tilde{r}_{A}^{2}\dot{\tilde{r}}_{A}-\frac{\beta}{2t^{2}}\tilde{r}_{A}^{3}\right.$
$\displaystyle\left.-\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}\tilde{r}_{A}^{4}\right].$
(28)

Eq. (IV), can be written as

$\displaystyle\dot{S}_{h}=\frac{2\pi}{G}\tilde{r}_{A}\dot{\tilde{r}}_{A}\left(1-\frac{\beta}{2t}\tilde{r}_{A}\right)-\frac{2\pi}{G}\frac{\beta}{2t^{2}}\tilde{r}_{A}^{3}$
$\displaystyle-\frac{2\pi}{G}\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}\tilde{r}_{A}^{4}.$
(29)

The Friedmann equation (9), for a flat universe, becomes

$\frac{1}{\tilde{r}_{A}^{2}}-\frac{\beta}{t}\frac{1}{\tilde{r}_{A}}-\frac{\omega\beta^{2}}{6t^{2(\beta+1)}}=\frac{8\pi
G}{3}\rho.$ (30)

Differentiating the above equation with respect to the cosmic time and using
the continuity equation(11), after some simplification, we get

$\displaystyle\frac{\dot{\tilde{r}}_{A}}{\tilde{r}_{A}^{3}}\left(1-\frac{\beta}{2t}\tilde{r}_{A}\right)-\frac{\beta}{2t^{2}}\frac{1}{\tilde{r}_{A}}-\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}$
$\displaystyle=4\pi GH(\rho+p)-\frac{4\pi G}{3}\frac{\beta}{t}(\rho+p).$ (31)

Solving this equation for $\dot{\tilde{r}}_{A}$, we find

$\displaystyle\dot{\tilde{r}}_{A}=\left(1-\frac{\beta}{2t}\tilde{r}_{A}\right)^{-1}\tilde{r}_{A}^{3}\left[4\pi
GH(\rho+p)\right.$ $\displaystyle\left.-\frac{4\pi
G}{3}\frac{\beta}{t}(\rho+p)+\frac{\beta}{2t^{2}}\frac{1}{\tilde{r}_{A}}+\frac{\omega}{6}\frac{\beta^{2}(\beta+1)}{t^{2\beta+3}}\right].$
(32)

Substituting $\dot{\tilde{r}}_{A}$ from Eq.(IV) into (IV), we obtain

$\dot{S}_{h}=\frac{2\pi}{G}H^{-4}\left[4\pi GH(\rho+p)-\frac{4\pi
G}{3}\frac{\beta}{t}(\rho+p)\right],$ (33)

which can also be rewritten in the following form

$\dot{S}_{h}=8\pi^{2}H^{-3}(\rho+p)\left(1-\frac{\beta}{3Ht}\right),$ (34)

where we have used ${\tilde{r}}_{A}={H}^{-1}$ for the flat universe. Let us
discuss the two cases in which $\dot{S_{h}}\geq 0$. In the first case we
assume the dominant energy condition valid, $\rho+p\geq 0$, therefore
$\dot{S_{h}}\geq 0$, provided $\beta\leq 3Ht$. However, in an accelerating
universe the dominant energy condition may violate, $\rho+p<0$. In this case
$\dot{S_{h}}\geq 0$ provided $\beta\geq 3Ht$. However, as we will see below,
the GSL can be still fulfilled in an accelerating fractal universe.

For latter convenience we also calculate $T_{h}\dot{S_{h}}$,

$T_{h}\dot{S_{h}}=4\pi H^{-2}(\rho+p)\left(1-\frac{\beta}{3Ht}\right).$ (35)

Next, we study the GSL of thermodynamics, namely the time evolution of the
total entropy including the entropy $S_{h}$ associated with the apparent
horizon together with the matter field entropy $S_{\rm in}$ inside the
apparent horizon. The entropy of the universe inside the horizon can be
related to its energy and pressure in the horizon by the Gibbs equation Pavon2

$TdS_{\rm in}=d(\rho V)+pdV=Vd\rho+(\rho+p)dV.$ (36)

We assume the temperature of the perfect fluid inside the apparent horizon
scales as the temperature of the horizon, which for flat universe is
$T_{h}=H/(2\pi)$. In general, if the temperature of the horizon differs much
from that the fluid, then the energy would spontaneously flow between the
horizon and the fluid, something at variance with FRW universe Pavon2 ; Muba .
Thus we suppose that the temperature $T_{h}$ associated with the apparent
horizon is $T_{h}=bT$ wang2 , where $b$ is a real proportional constant. Since
at the present time the horizon temperature is lower than that of the CMB by
many orders of magnitude, we will not consider the case $b>1$. We limit
ourselves to the assumption of the local equilibrium hypothesis, that the
energy would not spontaneously flow between the horizon and the fluid. Indeed,
this will certainly be the situation at late times, that is when the universe
fluids and the horizon will have interacted for a long time, it is ambiguous
if it will be the case at early or intermediate times jamil1 . However, in
order to avoid nonequilibrium thermodynamical calculations, which would lead
to lack of mathematical simplicity and generality, the assumption of
equilibrium, although restricting, has widely accepted for studying the GSL in
the literature jamil1 . Thus, we follow this assumption and we notice that our
results are valid only at the late stages of the universe evolution where the
universe fluids and the horizon will interact for a long time.

Let us first consider the case where $b=1$. Physically, this means that we
have assumed during the infinitesimal internal time $dt$, the temperature of
the perfect fluid is equal to the temperature $T_{h}$ associated with the
apparent horizon. Therefore, from Gibbs equation, after using the continuity
equation (11), we get

$\displaystyle T\dot{S}_{\rm in}$ $\displaystyle=$ $\displaystyle-4\pi
H^{-2}(\rho+p)\left(1-\frac{\beta}{3Ht}\right)$ (37) $\displaystyle+4\pi
H^{-2}(\rho+p)\dot{\tilde{r}}_{A}.$

To check the GSL of thermodynamics, we have to examine the evolution of the
total entropy $S_{h}+S_{\rm in}$. Adding equations (35) and (37), we get

$T_{h}(\dot{S}_{h}+\dot{S}_{\rm in})=4\pi
H^{-2}(\rho+p)\dot{\tilde{r}}_{A}=A(\rho+p)\dot{\tilde{r}}_{A},$ (38)

where $A=4\pi H^{-2}$ is the apparent horizon area. The deceleration parameter
$q$ can be written as

$q=-1-\frac{\dot{H}}{H^{2}}.$ (39)

Using the fact that in flat FRW universe we have
$\dot{\tilde{r}}_{A}=-\dot{H}/H^{2}$, Eq.(38) can be written as

$T_{h}(\dot{S}_{h}+\dot{S}_{\rm in})=A\rho(1+w)(1+q),$ (40)

where $w={p}/{\rho}$ is the equation of state parameter. From the above
equation we see that for $(1+q)(1+w)\geq 0$ the GSL of thermodynamics is
fulfilled in a region enclosed by the apparent horizon. Let us consider two
cases separately. In the first case where $q\geq-1$ and $w\geq-1$, the GSL
holds. Recent observations from type Ia supernova show that our universe is
currently undergoing a phase of accelerated expansion. For an accelerating
universe we have $q<0$ and $w<-1/3$. Therefore, in an accelerating universe
the GSL is fulfilled for $-1\leq q<0$ and $-1\leq w<-1/3$ which are consistent
with recent observations. In the second case where $q<-1$ and $w<-1$, the GSL
is again fulfilled, but in this case the equation of state parameter crosses
the phantom line, $w=-1$, which again some cosmological data confirm it. Now
we consider the general case where $b<1$. Since at the present time the
horizon temperature is lower than that of the CMB by many orders of magnitude,
we will not consider the case $b>1$. Adding equations (35) and (37) with
$T_{h}=bT$, we obtain

$\displaystyle T_{h}(\dot{S}_{h}+\dot{S}_{\rm in})=4\pi
H^{-2}(\rho+p)\left(1-\dfrac{\beta}{3Ht}\right)(1-b)$ (41) $\displaystyle+4\pi
bH^{-2}(\rho+p)(1+q)$ $\displaystyle=$ $\displaystyle 4\pi
H^{-2}(\rho+p)\left[\left(1-\dfrac{\beta}{3Ht}\right)\left(1-b\right)+b(1+q)\right]$
$\displaystyle=$ $\displaystyle 4\pi
H^{-2}\rho(1+w)\left[1+bq-\dfrac{\beta}{3Ht}\left(1-b\right)\right].$

Equation (41) shows that for $w\geq-1$, the GSL can be secured provided that

$\displaystyle\beta(1-b)\leq 3Ht(1+bq).$ (42)

Thus, for $b<1$, Eq. (42) can be translated to

$\displaystyle\beta\leq 3Ht\left(\frac{1+bq}{1-b}\right).$ (43)

Since $\beta$ is a positive definite, that is $\beta>0$, hence the above
condition can also be written

$\displaystyle 0<\beta\leq 3Ht\left(\frac{1+bq}{1-b}\right).$ (44)

The right hand side of inequality (44) should be positive, which leads to
$q>-b^{-1}$. In an accelerating universe $q<0$, and hence the condition for
keeping the GSL in a fractal universe reduced to $-b^{-1}<q<0$.

On the other hand, if equation of state parameter crosses the phantom line,
which some observational data support it, then we have $w<-1$. In this case,
the GSL holds provided

$\displaystyle\beta(1-b)>3Ht(1+bq).$ (45)

For $b<1$, the GSL can be fulfilled if

$\displaystyle\beta>3Ht\left(\frac{1+bq}{1-b}\right).$ (46)

We conclude that the requirement of the GSL of thermodynamics in a fractal
universe leads to constraint on the fractal dimension parameter $\beta$. It is
worth noting that our local equilibrium hypothesis, which we used for checking
the GSL, only valid at the late time. Thus, the result obtained in this secion
have no $t\rightarrow 0$ limit.

Finally, it is instructive to discuss the sign of the entropy and temperature,
when the universe lies in the phantom phase $(\omega<-1)$. A lot of works have
been done in the literature, showing that in the absence of chemical potential
in phantom regime, the temperature must be negative, while the energy density
and the entropy should be positive Myn ; Gon . In Lim ; Pere  a negative
chemical potential was assumed for the phantom fluid, and showed that the
temperature, entropy and density must be positive. In Sar  it was shown that
with an arbitrary chemical potential the density and entropy are always
positive, while the temperature of a phantom universe ($w<-1$) is negative,
and that of quintessence universe ($w>-1$) is positive. The temperature
negativity can only be interpreted in the quantum framework Sar . In Pavon2
it was found that the phantom temperature is positive and its entropy
negative. Finally, in Bre ; Noj  it was argued that one can describe the
phantom universe either with negative temperature and positive entropy, or
with negative entropy and positive temperature. Since the horizon temperature
is always positive, it is deduced that the universe temperature will be
positive even if it lies in the phantom phase. Thus we should have a negative
universe entropy in this case. Although the total entropy is always positive.
Because negative entropy of the universe ingredients is overcome by the
positive horizon entropy.

##  V concluding remarks

In summary, in this paper we studied thermodynamics of the apparent horizon in
the framework of fractal cosmology. We found that the Friedmann equation of a
fractal universe can be transformed to the fundamental relation $\delta
Q=T_{h}d{S_{h}}$ on the apparent horizon, where $\delta Q$ and $T_{h}$ are the
energy flux and Unruh temperature seen by an accelerated observer just inside
the apparent horizon. We showed that the entropy $S_{h}$ consists two terms,
the first one which obeys the usual area law and the second part which is the
entropy production term and appears due to the nonequilibrium thermodynamics
of the fractal universe. This indicates that in a fractal universe, a
treatment with nonequilibrium thermodynamics of spacetime maybe needed. We
also investigated the time evolution of the total entropy including the
entropy $S_{h}$ associated with the apparent horizon together with the matter
field entropy $S_{\rm in}$ inside the apparent horizon. We assumed the
temperature of the apparent horizon is proportional to the matter fields
temperature inside the horizon, i.e. $T_{h}=bT$. We studied several cases
including $b<1$ and $b=1$, in which the GSL of thermodynamics can be secured
in a fractal universe. Interestingly enough, we found that for an accelerating
fractal universe the GSL can be preserved, at least for the late times where
the local equilibrium hypothesis holds. The fulfillment of the GSL of
thermodynamics in an accelerating fractal universe leads also to constraints
on the fractal dimension parameter $\beta$.

###### Acknowledgements.

We thank the anonymous referee for constructive comments which helped us to
improve the paper significantly. This work has been supported financially by
Research Institute for Astronomy and Astrophysics of Maragha (RIAAM) and also
by Shiraz University Research Council.

## References

  * (1) J. M. Bardeen, B. Carter and S.W. Hawking, Commun. Math. Phys. 31, 161 (1973).
  * (2) J. D. Bekenstein, Phys. Rev. D 7, 2333 (1973).
  * (3) S. W. Hawking, Nature 248, 30 (1974).
  * (4) J. D. Bekenstein, Phys. Rev. D 9, 12 (1974).
  * (5) T. Jacobson, Phys. Rev. Lett. 75, 1260 (1995).
  * (6) C. Eling, R. Guedens, and T. Jacobson, Phys. Rev. Lett. 96, 121301 (2006).
  * (7) M. Akbar and R. G. Cai, Phys. Lett. B 635, 7 (2006).
  * (8) M. Akbar and R. G. Cai, Phys. Lett. B 648, 243 (2007).
  * (9) T. Padmanabhan, Class. Quantum Grav 19, 5387 (2002).
  * (10) T. Padmanabhan, Phys. Rep. 406, 49 (2005).
  * (11) T. Padmanabhan, Int. J. Mod. Phys. D 15 ,1659 (2006).
  * (12) A. Paranjape, S. Sarkar and T. Padmanabhan, Phys. Rev. D 74, 104015 (2006).
  * (13) D. Kothawala, S. Sarkar and T. Padmanabhan, Phys. Lett. B 652 ,338 (2007).
  * (14) T. Padmanabhan and A. Paranjape, Phys. Rev. D 75, 064004 (2007).
  * (15) M. Akbar and R. G. Cai, Phys. Rev. D 75, 084003 (2007).
  * (16) R. G. Cai and L. M. Cao, Phys.Rev. D 75, 064008 (2007).
  * (17) R. G. Cai and S. P. Kim, JHEP 0502, 050 (2005).
  * (18) A. V. Frolov and L. Kofman, J. Cosmol. Astropart. Phys. 0305, 009 (2003).
  * (19) U. K. Danielsson, Phys. Rev. D 71, 023516 (2005).
  * (20) B. Wang, E. Abdalla and R. K. Su, Phys. Lett. B 503, 394 (2001).
  * (21) B. Wang, E. Abdalla and R. K. Su, Mod. Phys. Lett. A 17, 23 (2002).
  * (22) R. G. Cai and Y. S. Myung, Phys. Rev. D 67, 124021 (2003).
  * (23) R. G. Cai and L. M. Cao, Nucl. Phys. B 785, 135 (2007).
  * (24) A. Sheykhi, B. Wang and R. G. Cai, Nucl. Phys. B 779, 1 (2007).
  * (25) A. Sheykhi, B. Wang and R. G. Cai, Phys. Rev. D 76, 023515 (2007).
  * (26) A. Sheykhi, J. Cosmol. Astropart. Phys. 05, 019 (2009);
A. Sheykhi, Eur. Phys. J. C 69, 265 (2010).

  * (27) B. Wang, Y. Gong, E. Abdalla, Phys. Rev. D 74, 083520 (2006).
  * (28) J. Zhou, B. Wang, Y. Gong, E. Abdalla, Phys. Lett. B 652, 86 (2007).
  * (29) A. Sheykhi, B. Wang, Phys. Lett. B 678, 434 (2009).
  * (30) A. Sheykhi, Class. Quantum Grav 27, 025007 (2010).
  * (31) A. Sheykhi, B. Wang, Mod. Phys. Lett. A 25(14), 1199 (2010).
  * (32) G. Calcagni, Phys. Rev. Lett. 104, 251301 (2010).
  * (33) G. Calcagni, JHEP 03, 120 (2010).
  * (34) R. Hilfer (Ed.), Applications of Fractional Calculus in Physics (Word Scientific Publishing, New Jersey, 2000).
  * (35) S. R. de Groot and P. Mazur, Non-Equilibrium Thermodynamics (North-Holland, Amsterdam, 1962).
  * (36) G. Izquierdo and D. Pavon, Phys. Lett. B 633,420 (2006).
  * (37) M. Jamil, E. N. Saridakis, M. R. Setare, Phys. Rev. D 81, 023007, (2010).
  * (38) M. Jamil, E. N. Saridakis and M. R. Setare, J. Cosmol. Astropart. Phys. 1011, 032, (2010).
  * (39) Y. S. Myung, Phys. Lett. B 671, 216 (2009).
  * (40) P. F. Gonzalez-Diaz and C. L. Siguenza, Nucl. Phys. B 697, 363 (2004).
  * (41) J. A. S. Lima and S. H. Pereira, Phys. Rev. D 78, 083504 (2008).
  * (42) S. H. Pereira and J. A. S. Lima, Phys. Lett. B 669, 266 (2008).
  * (43) E. N. Saridakis, P. F. Gonzalez-Diaz and C. L. Siguenza, Class. Quantum Grav 26, 165003 (2009).
  * (44) I. H. Brevik, S. Nojiri, S. D. Odintsov and L. Vanzo, Phys. Rev. D 70, 043520 (2004).
  * (45) S. Nojiri and S. D. Odintsov, Phys. Rev. D 70, 103522 (2004).


# Class adaptive threshold and negative class guided noisy annotation robust
Facial Expression Recognition

Darshan Gera
SSSIHL, Brindavan Campus
Bengaluru, Karnataka, India
<EMAIL_ADDRESS>
Badveeti Naveen Siva Kumar
SSSIHL, Prasanthi Nilayam Campus
Sri Sathya Sai District, Andhra Pradesh, India
<EMAIL_ADDRESS>
Bobbili Veerendra Raj Kumar
SSSIHL, Prasanthi Nilayam Campus
Sri Sathya Sai District, Andhra Pradesh, India
<EMAIL_ADDRESS>
S Balasubramanian
SSSIHL, Prasanthi Nilayam Campus
Sri Sathya Sai District, Andhra Pradesh, India
<EMAIL_ADDRESS>

###### Abstract

The hindering problem in facial expression recognition (FER) is the presence
of inaccurate annotations referred to as noisy annotations in the datasets.
These noisy annotations are present in the datasets inherently because the
labeling is subjective to the annotator, clarity of the image, etc. Recent
works use sample selection methods to solve this noisy annotation problem in
FER. In our work, we use a dynamic adaptive threshold to separate confident
samples from non-confident ones so that our learning won’t be hampered due to
non-confident samples. Instead of discarding the non-confident samples, we
impose consistency in the negative classes of those non-confident samples to
guide the model to learn better in the positive class. Since FER datasets
usually come with 7 or 8 classes, we can correctly guess a negative class by
85% probability even by choosing randomly. By learning "which class a sample
doesn’t belong to", the model can learn "which class it belongs to" in a
better manner. We demonstrate proposed framework’s effectiveness using
quantitative as well as qualitative results. Our method performs better than
the baseline by a margin of 4% to 28% on RAFDB and 3.3% to 31.4% on FERPlus
for various levels of synthetic noisy labels in the aforementioned datasets.

_K_ eywords Negative class $\cdot$ Facial Expression Recognition $\cdot$

Figure 1: This figure shows a) weakly augmented image, b)strongly augmented
view, and c) the prediction probabilities by the ResNet-18 model trained on
10% synthetic label noise dataset of RAFDB. On the top set of images, the
model predicted the correct label on both weak and strong augmentations
whereas, in the bottom set of images, the model predicted different labels on
different augmentations. In spite of not having an overlap in predictions of
the most probable class, the model does have an overlap in dissimilarity
(least probable class) in both cases.

##  1 Introduction

In recent years, significant progress has been made towards developing deep
learning (DL) based robust facial expression recognition (FER) systems [1, 2,
3, 4, 5]. Making machines understand human emotions and intentions through the
use of facial features is the goal of automatic facial expression recognition.
Numerous real-world applications of FER exist, including the detection of
driver weariness [6], mental health analysis [7], increasing student-teacher
interaction in distance learning environments [8], virtual assistants [9],
social robots [10], and others. In a supervised setting, learning positive
class is quite faster but if there are noisy labeled samples(i.e samples with
inaccurate labels) in the dataset, then the model starts learning incorrect
features leading to poor generalization. We can subjugate this if we train our
model only on the confident samples but getting clean data is a very expensive
and tiring process. Labels are subjective to the annotator as well as quality
of the image. Because of these factors, datasets inherently have noisy labels.

Several methods tried to alleviate the ill effects of noisy labels on the
learning process of the DL model, while training, by choosing samples that are
relatively clean. This is done by choosing samples with low loss [11, 12, 13,
14] but these methods lose on learning from clean samples with large loss
value might be due to difference in pose or illumination,etc. These methods
also use multiple networks for training and [11, 12, 13] need to know the
noise rate present in the dataset beforehand which won’t be available in real-
world cases. [15] uses a single threshold for all the classes. This method
doesn’t need to know the noise rate beforehand. The threshold that is used for
selecting clean samples is based on JS-divergence between two different
augmentations of same sample.

Recent methods such as SCN[16] use weighted cross-entropy loss (CE loss) where
the model itself learns the weights. It groups the weights into two, high
importance group and low importance group in a 7:3 ratio after sorting the
weights. It relabels the samples among the low-importance group based on a
threshold. RUL[17] uses two branches, one for learning feature vectors and
another for uncertainty vectors. It learns uncertainty vectors by comparing
two samples of the same batch. DMUE[18] is an ensemble technique that uses
multiple auxiliary branches to mine the latent distribution of a sample.
EAC[19] uses an attention consistency mechanism between the class activation
maps of every class and uses the original image and its flipped version. It
uses the CE loss function to learn from all the samples but the CE loss is not
robust to the noisy labels.

None of the above-mentioned models cater to inter-class size imbalance
problems and intra-class difficulty. To address this issue, we use a dynamic
adaptive threshold which we calculate at every mini-batch. So, we don’t need
to have prior knowledge of noise rate and we use only a single network unlike
some of the earlier methods.

We conducted experiments using a ResNet-18 model trained on a 30% synthetic
noise dataset of RAFDB with CE loss. From the experiment, We made an
observation that the model is consistent on the low prediction
probabilities(least probable classes) for different augmentations but not
always on the most probable class( see Fig1, on the below set of images the
model is consistent on only three least probable classes they are Surprise,
Fear, and Anger. whereas in the top set of images, the prediction
probabilities are consistent over all the expression classes i.e least being
neutral, second least being happy,…, and highest being sad). In other words,
even though the amount of similarity in the most probable class is not the
same, there is an overlap in dissimilarity. Inspired by this phenomenon, we
want to make use of these least probable classes as negative classes (We are
given only with ground truth label, all the others are negative labels for
us). Since strong augmentation is a complex augmentation technique, it is not
as easy for the model to predict the similar probability distribution of
labels as that of weak augmentations’. To do so, the model must learn the
correct features or memorize both weak augmentation and strong augmentation
together along with the label.

To make use of negative classes, we can use the prediction probabilities of a
single classifier to get a positive class and negative classes as well or we
can use another classifier that learns the negative classes separately. To see
which is a better option, we conducted another experiment with the ResNet-18
model. We used CE loss only on the confident samples obtained after applying a
dynamic adaptive threshold and using consistency loss on the remaining non-
confident samples with different augmentations. Based on other observation
from Fig1 that the amount of overlap of dissimilarity may differ for different
samples (for eg. in Fig1, on the top set of images overlap of dissimilarity
extended from Neutral to Anger but in the bottom set of images, the overlap of
dissimilarity is only for 3 expression classes namely Surprise, Fear and
Anger). We would wish to get an optimal overlap of dissimilarity of
predictions for the majority of the samples. So we masked the least-k classes
where k is a hyper-parameter and used only the prediction probabilities of
those that are masked in consistency loss. Expression recognition performance
in accuracy is shown in table1.

Table 1: Performance evaluation on RAFDB in the presence of different label noise levels when ResNet-18 is trained with confident samples selected based on dynamic adaptive threshold using cross-entropy loss and on non-confident samples using consistency loss. Here k refers to the number of top-k negative classes used for consistency loss. We give performance in x/y format where x refers to maximum accuracy obtained in training and y refers to average performance on last 5 epochs(i.e from 35 to 40). - | k=1 | k=2 | k=3 | k=4 | k=5 | k=6 | k=7
---|---|---|---|---|---|---|---
RAFDB | 83.409/82.483 | 83.767/83.148 | 83.833/82.764 | 83.409/83.018 | 83.442/82.822 | 82.007/67.959 | 82.431/38.631
10%noise | 83.246/82.157 | 83.148/82.464 | 82.887/82.235 | 82.953/82.255 | 83.05/81.89 | 82.073/64.152 | 82.203/55.189
60%noise | 74.185/69.511 | 75.977/70.684 | 74.12/70.241 | 76.499/70.225 | 74.315/70.897 | 75.195/41.03 | 76.108/23.122

It has been observed that on higher levels of label noise, performance is
greater than the performance of the model that learns only from confident
samples obtained by using a dynamic adaptive threshold. But on the lower level
noises, there is a performance drop by 5% on RAFDB and around 4% on the 10%
synthetic label noise dataset of RAFDB. This shows us the learning of negative
labels for non-confident is interfering with the learning of positive class
when we use only prediction probabilities from a single classifier. Using two
different classifiers, one that predicts positive class and another that
predicts negative class proved to be a better option. We show the
effectiveness of this approach in section4.

Overall our contributions can be summarized as follow :

  * •

We use a single network model unlike previous methods, ensuring that our model
is not computationally expensive.

  * •

We deal with inter-class similarities and intra-class difficulties by using
dynamic adaptive threshold [20].

  * •

We utilize all the samples in learning the expression features for some it
learns what class it is directly from the positive class classifier and for
some, it learns what it is not from the negative class classifier which in
turn can help in learning what it is eventually.

  * •

Our method is an end-to-end- framework that achieves superior performance when
the label noise rate is very high in the dataset. In addition, it is also
backbone independent.

We showed the effectiveness of our model on synthetic noisy label datasets
generated from RAFDB[21] and FERPlus[22], real-world noisy dataset
(automatically annotated subset of AffectNet with 0.459M images)[23]. Compared
to the baseline model, our method improves the performance in the range of 4%
to 28% on RAFDB and in the range of 3.3% to 31.4% improvement on FERPlus for
various synthetic noisy label datasets generated from RAFDB and FERPlus
respectively. Compared to the model that learns using only confident samples
obtained by using a dynamic adaptive threshold, our model performs better by a
range of 0.03 to 11.7% on RAFDB and in the range of 0.01 to 1.63% on FERPlus
for various synthetic noisy label datasets generated from RAFDB and FERPlus
respectively.

The remainder of this paper is organized in the following manner: Section 2
gives an overview of other related methods in this field. Section 3 describes
the proposed method and provides the architecture and algorithm for training
our framework. Section 4 shows the effectiveness of the framework using
quantitative and qualitative results. We conclude the paper in section 5.

##  2 Related Work

###  2.1 General Noisy Label Problem

Figure 2: Performance of ResNet-18 trained on RAFDB in the presence of 30%
synthetic noise on RAFDB.

If we train a DL model in the presence of noisy labels, due to the strong
memorizing capacity of the DL models, the performance of the model hampers as
shown in Fig. 2. Several ways are explored in order to solve the noisy label
problem. These works can be grouped into few categories such as sample re-
weighting, label cleansing, roust loss functions, and selecting clean based on
small loss. [24, 11, 12, 14] uses small loss samples to train but they need
the prior knowledge of noise rate in the dataset except for [14], getting to
know the noise rate beforehand is not always possible in real-world scenarios.
They also use multiple networks using peer/joint training either with
agreement [11]on the samples selected or by disagreement[12]. [25] uses a
negative class to deal with noisy labels but it has multiple phases including
a semi-supervised learning phase and the loss function in this paper is based
on the inversion of prediction probabilities from classifier with a negative
label (chosen among the classes other than ground truth label). But when we
are training for fewer epochs like 40 (which we do in all our experiments),
selective negative learning will lead to prediction probabilities of all
classes to be greater than 1/(number of classes) leading to no class being
greater than 0.5 when it comes to selective positive learning phase. Our
method works based on sample selection but we don’t need to know the noise
rate beforehand like the aforementioned methods. We use a dynamic adaptive
threshold [20] generated from posterior prediction probabilities from a
positive classifier.

###  2.2 Noisy Label Problem in FER

Due to the availability of benchmark datasets like RAFDB[21], FERPlus[22],
AffectNet[23], etc. FER has become a well-explored field of research. In spite
of this, obtaining good performance when trained on the noisy annotated FER
dataset is not a trivial task. Recent works like SCN [16], DMUE [18], IPA2LT
[26], CCT [27], and RUL [17] have attempted to handle the noisy labels in FER
datasets. IPA2LT [26] actually deals with inconsistent labels for an image,
the real label is learned by maximizing the log-likelihood of multiple
inconsistent human-made annotations and machine-predicted annotations. But for
this model, we need to have multiple annotations from different annotators.
SCN[16] uses weighted CE loss where weights are learned by the model itself.
It tries to segregate the top 70% and below 30% of the learned weights as a
high-importance group and low-importance group and imposes a loss function
such that the model separates these groups by a margin. Among these low-
importance groups, it relabels some of the samples based on a criterion. This
model suffers from self-confirmation bias and since datasets can have any
amount of noise rate in them, separating the learned weights in a $7:3$ ratio
is not always optimal.

RUL[17] model has two branches to the backbone model where one branch gives
the feature vector and another branch gives the uncertainty vector for each
image. For a given batch of images using some random permutation, this model
performs a mix-up of feature vectors based on the learned uncertainty values
and the model imposes an add-up loss function which forces to predict both the
labels correctly from the mixed feature vector. They use the CE loss for this
add-up loss function which is not that effective in the presence of label
noise. DMUE[18] uses one main branch and as many auxiliary branches as the
number of classes for a dataset. The main branch learns from all the samples
whereas $i_{t}h$ auxiliary branches learn from the samples other than those
that are annotated with $i$ as their label. Consistency is maintained between
auxiliary branches and the main branch. It uses CE loss on all the auxiliary
branches and weighted CE loss on the main branch where weights are learned
based on pairwise uncertainties between the samples of the same mini-batch.
This model is computationally very expensive. And the architecture while
training is wholly dependent on the number of classes present in the dataset.
If we were to have a large number of classes, it would blow up the requirement
for computations.

EAC[19] uses attention consistency on class activation maps for consistency
loss. Activation maps for each class are obtained by performing the
multiplication of weights from the fully connected layer that predicts the
class label and the feature maps that are extracted from the penultimate layer
of the model. It uses an imbalanced framework where it uses CE loss only on
the predictions of the original image but not its flipped version so as to
ensure that the model doesn’t memorize both the original image and its flipped
version to reduce the overall loss. To reduce the memorization effect of the
model further, it performs random erasing on the original image. In spite of
its simplicity, the weakness of this model is it uses CE loss on all the
images. Cross entropy loss drives the predictions toward the ground truth,
which can hamper the learning process.

Unlike DMUE, our proposed model is independent of the number of classes and
not expensive in computations. Compared to other methods, our model differs in
utilizing only confident samples to be fed to CE loss for positive class
prediction, where we obtain confident samples from the dynamic adaptive
threshold. None of the above-mentioned methods deal with the intra-class
sample size imbalances and inter-class difficulties, we address them by using
dynamic adaptive threshold, and instead of discarding the non-confident
samples, we learn from them as well by imposing consistency on the negative
classes. All the above-mentioned methods try to use loss functions that try to
make even the non-confident samples learn the positive class (which can be
wrong due to noisy annotations). Here we take an indirect approach to use "not
this" philosophy and learn "which class the sample doesn’t belong to" on the
non-confident samples which in turn can help us to learn "which class it
belongs to".

Figure 3: Architecture of proposed model. Here a batch of $x_{w}andx_{s}$
(weak and strong augmentations) are sent through the backbone(ResNet-18).
Extracted features are sent to the positive class classifier (pcc) as well as
negative class classifier(ncc). On the predictions of weak augmentation from
pcc, we calculate the adaptive threshold and get confident and non-confident
indices for the given batch of samples. Only the confident samples are used
for supervised loss$L_{s}$ and only the non-confident samples are sent to
consistency loss $L_{c}$

##  3 Proposed Method

In this section, we first provide the motivation for the proposed method. Then
we list out the details of our method. Following it, we provide a training
algorithm. The architecture for our model is shown in Fig3

###  3.1 Motivation and overview

Our method is based on the idea of selecting confident samples to learn from
in order to ensure that the model doesn’t learn incorrect features to predict
the labels. Unlike selecting samples having a small loss that leaves out hard
samples which can be crucial for the model to better learn features for a
given class to generalize better, we use posterior probabilities and generate
a threshold to get the confident samples. Datasets do come with multiple
challenges like variation in poses, variation in illumination,…, etc. Due to
these reasons even in a given class, there can be easy samples and hard
samples because of which the prediction probabilities which DL model gives may
vary. This is the intra-class difficulty. In order to tackle the inter-class
similarities and intra-class difficulties, we use dynamic adaptive threshold
[20] to get the confident samples from the positive class classifier. We have
conducted an experiment and observed the prediction probability distributions
on different augmentations. We have used weak augmentation that includes
horizontal flip and center crop. We have used RandAugment[28] for strong
augmentation. We observed that the prediction probabilities of the least
probable classes are more consistent compared to the most probable class. Even
though the amount of similarity in prediction probabilities of the most
probable class is not the same, there is an overlap in dissimilarity in
prediction probabilities of negative classes. Examples of this phenomenon are
shown in Fig1. Even if we predict randomly, we can be correct in guessing the
negative label by 85% given that the datasets that we use are having only 7 or
8 classes. If we ensure the consistency between different augmentations, as
training progresses, we can get consistent predictions even in positive
classes. Based on an experiment, where we trained the ResNet-18 model with the
cross-entropy loss on confident samples obtained using dynamic adaptive
threshold and consistency loss on non-confident samples on the negative
classes based on the prediction probabilities of the classifier, we have
observed that the performance(accuracy) is lower than the performance of the
model that uses only confident samples. So consistency loss on negative
classes is affecting the learning process of positive classes when we use a
single classifier. So we use two classifiers. One is to predict the positive
class of the samples where we use a dynamic adaptive threshold. And another
negative class classifier that predicts the negative class of the samples.
Among these, we apply consistency loss defined in equation4 only on the non-
confident samples.

###  3.2 Problem Formulation

Given a batch of N samples $S=\\{(x_{i},y_{i})\\}_{i=1}^{N}$ where each face
image $x_{i}$ has an expression label $y_{i}\in\\{1,2,...,C\\}$ here C denotes
the number of expression classes. The shared backbone network is ResNet-18. It
is parameterized by $\theta$. Features from the backbone are classified using
two fully connected layers (FC) followed by softmax to obtain prediction
probabilities. The first FC layer is to predict the positive class and the
other FC layer predicts the negative class. We use two different augmentations
one weak-augmented image and another strongly-augmented image which is denoted
by $x_{w}$ and $x_{s}$. Prediction probabilities obtained by passing $x_{w}$
and $x_{s}$ through the positive class classifier are denoted as $p_{w}^{p}$
and $p_{s}^{p}$ respectively and through the negative class classifier are
denoted as $p_{w}^{n}$ and $p_{s}^{n}$ respectively. Standard cropping along
with horizontal flipping with a probability of 50% is used for weak
augmentations. And for strong augmentation, Randaugment [28] is used. During
training, for each mini-batch $S_{n}$, dynamic adaptive threshold $T_{c}$ is
calculated from the predictions of positive class classifier $p_{w}^{n}$ as
per equation1, where $X_{c}$ is a set of samples in the current mini-batch
with class c.

$T_{c}=\frac{1}{X_{c}}\sum_{x\in S_{c}}p^{c}(x;\theta)$ (1)

After finding the dynamic adaptive threshold $T_{c}$, We choose those images
whose ground truth prediction probabilities are greater than $T_{c}$ for that
ground truth label. This can be represented by equation 2

$X_{c}^{clean}=\\{x\in X_{c}\ni p^{c}\geq T_{c}\\}$ (2)

. By propagating the losses only from these confident samples, we learn better
from the positive class. Instead of discarding all the other samples from the
mini-batch, we use them for feature learning using consistency loss. For this,
we take the prediction probabilities from negative class classifier
$p_{w}^{n}$ and $p_{s}^{n}$ and use masked CE loss defined in section3.3. We
say it is masked because the performance of the model dehances if we were to
use prediction probabilities of all classes, we choose top K classes from the
negative class classifier predictions and do consistency loss only on those
classes. K is a hyper-parameter obtained based on ablation study4.4.

###  3.3 Loss functions

We use Cross-Entropy loss on a positive class classifier defined by equation3.
Here $L_{s}$ represents supervised loss

$L_{s}=\\{-\sum_{c=1}^{C}y_{i=1}^{c}\log(p^{c}(x_{i};\theta)\\}_{i=1}^{N}$ (3)

The consistency loss denoted by $L_{c}$ on negative class classifier
predictions can be obtained using equation4. where $M_{k}$ represents the mask
where $M_{k}$ becomes zero if the prediction probability for a given class is
not among top k else it is one.

$L_{c}=-\frac{1}{N}\sum_{i=1}^{N}M_{k}p_{w}^{n}log(p_{s}^{n})$ (4)

Overall loss is denoted as $L_{overall}$5

$L_{overall}=L_{s}+L_{c}$ (5)

Input: Given a model f with parameters $\theta$, dataset
$S=\\{(x_{i},y_{i})\\}_{i=1}^{N}$, mini-batch size (b), learning rate($\eta$),
number of expression classes (C), total epochs $E_{max}$, warm-up epochs
($E_{warm}$)

Output: Updated model parameters $\theta$

1 Initialize $\theta$ randomly.

2

3for  _$e=1,2,..,E_{max}$_ do

4 Shuffle training samples $\\{(x_{i},y_{i})\\}_{i=1}^{N}$

5 Sample mini-batch $S_{n}$ from S

6 for  _each class $c\in\\{1,2,..,C\\}$_ do

7 if  _ $e<E_{warm}$ _ then

8 Compute loss $L_{s}$ on all samples using Eq. 3

9

10 else

11 Compute dynamic adaptive threshold $T_{c}$ using Eq. 1

12 Select confident samples $S_{c}^{clean}$ from current mini-batch using Eq.
2

13

14 Compute supervision loss $L_{s}$ on above selected confident samples using
Eq. 3

15 From those non-confident samples, based on negative class classifier
prediction probabilities, Compute consistency loss $L_{c}$ using Eq. 4

16 Compute total loss $L$ using Eq. 5

17

18

19

20 Update model parameters $\theta=\theta-\eta\nabla L_{\theta}$ as per
gradient descent rule

21

return $\theta$

Algorithm 1 Training algorithm

##  4 Experiments

###  4.1 Datasets

We evaluate our model on three popular real-world benchmark FER datasets
RAFDB, FERPlus, and AffectNet.

  * •

RAFDB [21, 29]: The Real-world Affective Face Database (RAFDB) has a basic
emotion set of 12271 images for training and 3068 images for testing. Both
train and test sets are imbalanced w.r.t sample sizes of different expression
classes.

  * •

FERPlus [22]: FERPlus is an extended version of FER2013 [30]. It consists of
images with the 8 basic emotions (with contempt), of which 28709 are used for
training, 3589 are used for validation and the remaining 3589 for testing.

  * •

AffectNet [23]: AffectNet is the large dataset with 0.44M manually annotated
and 0.459M automatically annotated facial expression images for 8 emotions. We
use an Automatically annotated subset of seven classes for training under real
noisy conditions and tested on the validation set constituting 3500 images.

  * •

Synthetic noisy annotated datasets: We randomly change 10%, 30%, 50%, 60%,
70%, 80% labels of training images from RAFDB, FERPlus to create synthetic
noisy annotated datasets. The performance of our model is reported on the
corresponding clean test/validation sets.

###  4.2 Implementation Details

In the experiments below, we used MTCNN [31] to recognize and resize the
images of facial expressions to 224x224. The PyTorch DL toolbox is used to
build our technique, and a single Tesla K40C GPU with 11.4GB RAM is used to
run our experiments. The backbone network utilized is ResNet-18, which was
previously trained on the MS-Celeb-1M[32] face recognition dataset. In
addition to random cropping with 4 pixels and resizing to 224x224, random
horizontal flipping with a chance of 0.5 is employed for weak augmentation.
RandAugment [28] is used for strong augmentation. From a selection of
transformations including contrast adjustment, rotation, color inversion,
translation, etc. RandAugment chooses two augmentations at random. Similar to
[4, 16, 1], oversampling is used to solve class imbalance issues in AffectNet.
The batch size used for training is 128. The model is optimized using the Adam
optimizer, with a learning rate of 0.0001 for the backbone and 0.001 for the
positive class classifier and negative class classifier (FC layers). We have
run the model for 40 epochs.

###  4.3 Experiment Results on Synthetic Noisy Annotated Datasets

In all the experiment results below, we have given performance in accuracy as
x/y, where x is the maximum accuracy obtained during the process of training
in 40 epochs. y represents the average accuracies of the last 5 epochs (i.e
from 36 to 40).

Table 2: Performance evaluation comparison on synthetic symmetric label noise on FERPlus FERPlus | 10%noise | 20%noise | 30%noise | 50%noise | 60%noise | 70%noise | 80%noise
---|---|---|---|---|---|---|---
SCN [16] | 84.28 | 84.99 | 82.47 | 75.33 | 68.06 | 39.43 | 37.62
RUL[17] | 85.94 | 84.99 | 82.75 | 77.18 | 73.54 | 64.07 | 43.39
EAC[19] | 87.03 | 86.07 | 85.44 | 81.48 | 79.82 | 74.98 | 62.19
NCCTFER | 86.29 | 85.66 | 84.79 | 81.73 | 80.20 | 75.17 | 68.03
Table 3: Performance evaluation comparison on synthetic symmetric label noise on RAFDB RAFDB | 10%noise | 20%noise | 30%noise | 50%noise | 60%noise | 70%noise | 80%noise
---|---|---|---|---|---|---|---
SCN[16] | 82.18 | 79.79 | 77.46 | 73.5 | 59.55 | 41.98 | 38.82
RUL[17] | 86.22 | 83.35 | 82.06 | 73.5 | 69.62 | 57.66 | 36.34
EAC[19] | 88.02 | 86.05 | 84.42 | 80.54 | 76.37 | 68.9 | 47.46
NCCTFER | 86.7 | 86.147 | 85.169 | 81.486 | 79.73 | 71.9 | 48.89

Synthetic symmetric noise is manually added on RAFDB, and FERPlus datasets by
randomly changing labels in the ratio of 10,30,50,60,70,80%. We compare our
model (referred to as baseline+pc+nc(pc: positive classifier, nc: negative
classifier) ) with ResNet-18 with pre-trained weights of MS-Celeb checkpoint,
trained using CE loss (referred to as Baseline)23 and against ResNet-18 with
pre-trained weights of MS-Celeb checkpoint, trained using CE loss but only on
the confident samples selected using dynamic adaptive threshold(referred to as
baseline+pc(positive classifier))23. Automatically Annotated subset of
AffectNet-7 is a challenging noisy dataset because of heavy class imbalance as
well as intra-class difficulty and the annotations are generated automatically
without human intervention. No method is able to achieve good performance. Our
method gave 56.66% accuracy when tested on the validation set of the AffectNet
dataset. The confusion plot for this model is shown in Fig.4

Figure 4: Confusion Matrix of model performance when trained on real noisy
data-subset of AffectNet(i.e Automatically Annotated image subset with 0.459M
images).

Figure 5: Confusion Matrix of model performance when trained on RAFDB(left)
and FERPlus(right) respectively

####  4.3.1 Performance on asymmetric noise

Apart from symmetric noise, the effectiveness of the model is shown on
synthetic asymmetric noise on the RAFDB dataset. For a given expression in
RAFDB, we have replaced the expression with its most confused pair in the
required percentage. The most confused pairs in RAFDB, based on confusion
plots of some SOTA methods are Surprise-Anger, Fear-Surprise, Disgust-Anger,
Happy-Neutral, Sad-Neutral, Anger-Happy, and Neutral-Sad. From these confusion
pairs, if the former is the ground truth label, we replaced it with the latter
for the required amount of noise rate. We have conducted experiments with 10%
to 50% of asymmetric noise rates as per the above-mentioned method. Our method
improves from 1.076 to 7.21% over the baseline. The results are shown in the
table4.

Table 4: Performance evaluation in the presence of synthetic asymmetric label noise on RAFDB RAFDB | 10%noise | 20%noise | 30%noise | 40%noise | 50%noise
---|---|---|---|---|---
Baseline | 84.81 | 80.73 | 77.28 | 68.87 | 48.10
NCCTFER | 85.886 | 84.713 | 81.799 | 75.912 | 55.31
improvement | 1.076 | 3.983 | 4.519 | 7.042 | 7.21

####  4.3.2 Visualizations

We have visualized our model’s performance as a confusion matrix as shown in
Fig5 and 4. It can be observed that Happy, Surprise, and Neutral are easy
classes to predict mainly because of the availability of more samples with
these labels. Whereas Disgust and Fear are most confused among all three
datasets Fear is most confused with Surprise and Disgust is confused with
neutral in FERPlus and RAFDB but Disgust is most confused with Anger in the
automatically annotated subset of AffectNet dataset. Contempt in the FERPlus
dataset is most confused with Neutral, Anger, and Sad in the mentioned order.

(a) t-sne plot of learned feature vectors from FERPlus dataset

(b) t-sne plot of learned feature vectors from RAFDB dataset

###  4.4 Ablation study

We have seen the effectiveness of masking different number of classes
(choosing only top k classes for consistency loss) among the non-confident
samples whose prediction probabilities obtained from the positive class
classifier failed to be greater than the dynamic adaptive threshold. The
hyper-parameter k is used to determine how many classes we are going to use
for consistency loss. Results are shown in Fig. 7. We have adjusted the value
for k as 4 because when we mask the top 4 negative classes, we are achieving
good performance over various levels of noisy labels.

Figure 7: Effect of k shown on synthetic symmetric label noise of a)FERPlus
and b)RAFDB

Apart from this, we have verified the importance of each component in learning
by removing the other components from the model and checking for their
performance. Table 5 shows that baseline is far from below compared to the
proposed model and effectiveness of learning from negative classes on the non-
confident samples is also shown by the increase in performance compared to
model that learns only from confident samples.

Table 5: Ablation study on the importance of learning from positive class and negative class on synthetic symmetric noise on RAFDB (Here Baseline+pc refers to model learning from only confident samples and Baseline+nl refers to baseline with loss functions defined in[25] ) RAFDB | 10%noise | 30%noise | 50%noise | 60%noise | 70%noise | 80%noise
---|---|---|---|---|---|---
Baseline | 84.7 | 81.2 | 77.1 | 69.2 | 60.26 | 41.59
Baseline+nl | 84.2 | 80.54 | 75.488 | 71.35 | 62.9 | 39
Baseline+pc | 86.7 | 83.3 | 78.94 | 75.4 | 61.3 | 33.1
NCCTFER | 86.7 | 85.169 | 81.486 | 79.73 | 71.9 | 48.89
Table 6: Performance on real-world datasets RAFDB and FERPlus (* represents trained on AffectNet and RAFDB combined.) Models$\backslash$ Datasets | RAFDB | FERPlus
---|---|---
IPA2LT[26] | 86.77* | -
RAN[4] | 86.90 | 88.55
SCN[16] | 88.14* | 88.01
DMUE[18] | 88.76 | 88.64
RUL[17] | 88.98 | 88.75
NCCTFER | 87.97 | 88.21

####  4.4.1 Attention Maps

Attention Maps are the weighted feature maps,such that more weight is given to
the salient regions which are more concentrated on by the model to predict the
label. In order to investigate these salient regions focused by the proposed
model, the attention-weighted activation maps are visualized using Grad-CAM
[33] for Baseline trained on RAFDB and proposed method trained on 30% and 60%
synthetic label noise on RAFDB. Darker color indicates high attention while
lighter color indicates negligible attention. The baseline sometimes focuses
on irrelevant parts or misses out on relevant parts. In comparison to
Baseline, the proposed model attends to non-occluded and relevant parts for
expression recognition. These visualizations validate the effectiveness of our
framework in the prediction of the correct label instead of over-fitting to
the noisy label. In Fig. 8, the emotion given below each image is the label
that is predicted by model and green represents correct whereas rad represents
incorrect prediction. On top of the every image, we have given the prediction
probability with which, the model predicted the label. Clearly, Our method is
able to learn robust features in the presence of noisy labels.

Figure 8: The attention maps of the Baseline trained on clean RAFDB, our
proposed framework trained on 30% and our proposed framework trained on 60%
synthetic label noise of RAFDB on test images from RAFDB using Grad-CAM are
compared in this figure. The emotion label in red color mean the prediction is
incorrect and emotion label in green mean the prediction is correct. On top of
every image, we have given the probability with which the model predicted the
label. Figure 9: Prediction scores compared to baseline trained on 30% label
noise dataset on RAFDB and our model trained on 30% label noise dataset on
RAFDB. Above every image, we have given the prediction of the model mentioned
on first column of each row. The emotion label in green represents correct and
in red represents incorrect label. Just below the label, we have given the
probability with which the model predicted the label. Below the down set of
images, we have given the noisy label with which the image was trained.
Inspite of giving different incorrect label, model is able to learn the
correct label.

####  4.4.2 Confidence Scores

In order to quantitatively demonstrate the effectiveness of our model with
noisy labeled images, we visualize the prediction/confidence scores on images
of different expression classes from the RAFDB dataset. These are shown in
Fig. 9. The more uncertain the annotation of a sample is, the lower will be
its confidence score. Given any noisy label, our model DNFER predicts
correctly the true label with high probability for almost all cases.

##  5 Conclusions

In this paper, we have proposed a new method to handle the problem of noisy
annotations in FER datasets. Our model uses posterior prediction probabilities
from a positive class classifier and uses a dynamic adaptive threshold to get
confident and non-confident samples. On the confident samples, we use CE loss,
and on the non-confident samples using the prediction probabilities from the
negative class classifier, we use consistency loss in each mini-match. Recent
sample selection algorithms are computationally expensive since they employ
many networks for joint or peer training and/or require knowledge of the noise
rate beforehand. In contrast to previous models, our model doesn’t need to
know the noise rates nor needs to learn using multiple networks, nor need a
separate supplement of clean data. Our model uses all the samples either for
consistency loss on the predictions of the negative class classifier or
supervised loss on the predictions of the positive class classifier. By using
the dynamic adaptive threshold, It also handles the class imbalance problem,
It also caters to inter-class similarities and intra-class difficulties.
Instead of imposing the non-confident samples to learn the positive class
which might be wrong in our case due to noisy labels, we impose consistency
only on the negative classes of these non-confident samples but not on the
positive class. To summarize, Our method on lower noise rates performs on par
with the model that uses only confident samples based on the dynamic adaptive
threshold one RAFDB but performs better in the case of FERPlus but it is more
effective as the noise rate increases in the dataset as we can see that the
improvement over the former by 0.433 to 4.537% on FERPlus and 2.56 to 11.71%
on RAFDB datasets.

## Acknowledgments

We dedicate this work to Bhagawan Sri Sathya Sai Baba, Divine Founder
Chancellor of Sri Sathya Sai Institute of Higher Learning, Prasanthi Nilayam,
A.P., India.

## References

  * [1] Hui Ding, Peng Zhou, and Rama Chellappa.  Occlusion-adaptive deep network for robust facial expression recognition.  In 2020 IEEE International Joint Conference on Biometrics (IJCB), pages 1–9. IEEE, 2020.
  * [2] Darshan Gera and S Balasubramanian.  Landmark guidance independent spatio channel attention and complementary context information based facial expression recognition.  Pattern Recognition Letters, 145:58–66, 2021.
  * [3] Shasha Mao, Guanghui Shi, Shuiping Gou, Dandan Yan, Licheng Jiao, and Lin Xiong.  Adaptively lighting up facial expression crucial regions via local non-local joint network.  arXiv preprint arXiv:2203.14045, 2022.
  * [4] Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, and Yu Qiao.  Region attention networks for pose and occlusion robust facial expression recognition.  IEEE Transactions on Image Processing, 29:4057– 4069, January 2020\.
  * [5] Delian Ruan, Yan Yan, Shenqi Lai, Zhenhua Chai, Chunhua Shen, and Hanzi Wang.  Feature decomposition and reconstruction learning for effective facial expression recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7660–7669, 2021.
  * [6] Zhongmin Liu, Yuxi Peng, and Wenjin Hu.  Driver fatigue detection based on deeply-learned facial expression representation.  Journal of Visual Communication and Image Representation, 71:102723, 2020.
  * [7] Carmen Bisogni, Aniello Castiglione, Sanoar Hossain, Fabio Narducci, and Saiyed Umer.  Impact of deep learning approaches on facial expression recognition in healthcare industries.  IEEE Transactions on Industrial Informatics, 18(8):5619–5627, 2022\.
  * [8] Waleed Maqableh, Faisal Y Alzyoud, and Jamal Zraqou.  The use of facial expressions in measuring students’ interaction with distance learning environments during the covid-19 crisis.  Visual informatics, 7(1):1–17, 2023.
  * [9] Lingyu Yan, Menghan Sheng, Chunzhi Wang, Rong Gao, and Han Yu.  Hybrid neural networks based facial expression recognition for smart city.  Multimedia Tools and Applications, pages 1–24, 2022.
  * [10] Andrea F. Abate, Carmen Bisogni, Lucia Cascone, Aniello Castiglione, Gerardo Costabile, and Ilenia Mercuri.  Social robot interactions for social engineering: Opportunities and open issues.  In 2020 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), pages 539–547, 2020.
  * [11] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama.  Co-teaching: robust training of deep neural networks with extremely noisy labels.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8536–8546, 2018.
  * [12] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama.  How does disagreement help generalization against label corruption?  In International Conference on Machine Learning, pages 7164–7173. PMLR, 2019.
  * [13] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An.  Combating noisy labels by agreement: A joint training method with co-regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13726–13735, 2020.
  * [14] Fahad Sarfraz, Elahe Arani, and Bahram Zonooz.  Noisy concurrent training for efficient learning under label noise.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3159–3168, 2021.
  * [15] Darshan Gera, G Vikas, and S Balasubramanian.  Handling ambiguous annotations for facial expression recognition in the wild.  In Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing, pages 1–9, 2021.
  * [16] Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao.  Suppressing uncertainties for large-scale facial expression recognition.  In CVPR, pages 6897–6906, 2020.
  * [17] Yuhang Zhang, Chengrui Wang, and Weihong Deng.  Relative uncertainty learning for facial expression recognition.  Advances in Neural Information Processing Systems, 34:17616–17627, 2021.
  * [18] Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen, and Tao Mei.  Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6248–6257, 2021.
  * [19] Yuhang Zhang, Chengrui Wang, Xu Ling, and Weihong Deng.  Learn from all: Erasing attention consistency for noisy label facial expression recognition.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, pages 418–434. Springer, 2022.
  * [20] Darshan Gera, Naveen Siva Kumar Badveeti, Bobbili Veerendra Raj Kumar, and S Balasubramanian.  Dynamic adaptive threshold based learning for noisy annotations robust facial expression recognition.  arXiv preprint arXiv:2208.10221, 2022.
  * [21] Shan Li and Weihong Deng.  Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition.  IEEE Transactions on Image Processing, 28(1):356–370, 2019.
  * [22] E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang.  Training deep networks for facial expression recognition with crowdsourced label distribution.  In Proceedings of the 18th ACM International Conference on Multimodal Interaction, page 279–283, 2016.
  * [23] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing, 2017.
  * [24] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li Jia Li, and Li Fei Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In International Conference on Machine Learning, pages 2304–2313. PMLR, 2018.
  * [25] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim.  Nlnl: Negative learning for noisy labels.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 101–110, 2019.
  * [26] Jiabei Zeng, Shiguang Shan, and Xilin Chen.  Facial expression recognition with inconsistently annotated datasets.  In Proceedings of the European conference on computer vision (ECCV), pages 222–237, 2018.
  * [27] Darshan Gera and S Balasubramanian.  Noisy annotations robust consensual collaborative affect expression recognition.  In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 3578–3585, 2021.
  * [28] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
  * [29] Shan Li, Weihong Deng, and JunPing Du.  Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2584–2593. IEEE, 2017.
  * [30] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al.  Challenges in representation learning: A report on three machine learning contests.  In International conference on neural information processing, pages 117–124. Springer, 2013.
  * [31] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE Signal Processing Letters, 23(10):1499–1503, 2016.
  * [32] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  ECCV, 2016.
  * [33] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.