# Hall Coefficient and Resistivity in the Doped Bilayer Hubbard Model

Yin Shi<EMAIL_ADDRESS>Department of Materials Science and Engineering, The
Pennsylvania State University, University Park, Pennsylvania 16802  Jonathan
Schirmer<EMAIL_ADDRESS>Department of Physics, The Pennsylvania State
University, University Park, Pennsylvania 16802  Long-Qing Chen  Department of
Materials Science and Engineering, The Pennsylvania State University,
University Park, Pennsylvania 16802

###### Abstract

Finding and understanding non-Fermi liquid transport behaviors are at the core
of condensed matter physics. Most of the existing studies were devoted to the
monolayer Hubbard model, which is the simplest model that captures essential
features of high-temperature superconductivity. Here we discover a new type of
non-Fermi liquid behavior emergent in the hole-doped bilayer Hubbard model,
using dynamical mean-field theory with a full consideration of the short-range
interlayer electron correlation. We find that at low temperatures, the Hall
coefficient has a strong nonmonotonic dependence on temperature, leading to a
double or quadruple reversal of its sign depending on the doping level. At the
same time, the resistivity exhibits two plateaus rather than linearity in its
temperature dependence. We show that these intriguing transport behaviors stem
from the formation of coherent interlayer singlets, which scatter off gapped
collective modes arising from short-range interlayer antiferromagnetic
fluctuations.

††preprint: APS/123-QED

##  I Introduction

Studying the magnetotransport properties of electron systems is a valuable way
to learn about their electronic structure. For example, in high-temperature
cuprate superconductors, the direct-current (DC) Hall resistivity has a strong
temperature ($T$) dependence and changes its sign in the heavily overdoped
regime [1, 2]. Meanwhile, the DC longitudinal resistivity in the normal state
has linear temperature dependence and exceeds the Mott-Ioffe-Regel criterion
[3, 4, 5], known as strange metallicity. In an atomically thin cuprate van der
Waals heterostructure during cooling, the Hall resistivity decreases and
changes from positive to negative and then reverses sign again before
vanishing at low temperatures. This was explained by the vortex dynamics-based
description of the Hall effect in high-temperature superconductors [6]. These
behaviors are incompatible with the Fermi liquid theory of weakly interacting
electrons and manifest the intricate nature of strongly correlated electron
systems.

In efforts to understand the non-Fermi liquid behaviors, various authors have
calculated the magnetotransport properties of the hole-doped Hubbard model
using the quantum Monte Carlo method for small square lattices [7], the
dynamical mean-field theory (DMFT) approximation for hypercubic [8] and square
[9, 10, 11] lattices, and an expansion formula of the Hall coefficient for
small square lattices [12]. A double sign change of the $T$-dependent DC Hall
coefficient similar to that in cuprate superconductors has been observed [8,
12]. Recent numerical calculations for the square-lattice Hubbard model also
revealed the $T$-linear DC longitudinal resistivity exceeding the Mott-Ioffe-
Regel limit [13] and a $T$-linear electron scattering rate at low temperatures
[14].

These works motivate us to investigate further the magnetotransport properties
of a more complicated lattice model, the Hubbard model on a bilayer square
lattice, in which electrons can form disordered interlayer singlets with a
spin gap [15, 16, 17]. Accurately computing the conductivities of strongly
correlated systems is notoriously difficult [18], and is frequently hindered
by small lattice sizes, infinite expansion summations, or the omission of
vertex corrections. We use the DMFT [19] to calculate the resistivities of the
hole-doped bilayer Hubbard model. The DMFT works for the thermodynamic limit,
but in this theory the vertex corrections to the in-plane conductivities
cancel out due to the neglect of in-plane nonlocal correlations [20]. However,
the short-range out-of-plane correlation is entirely accounted for in the Kubo
bubble. This distinguishes our calculation from those for the monolayer
Hubbard model.

We find that the Hall coefficient has a strong nonmonotonic $T$ dependence at
low temperatures and can change its sign twice or four times with decreasing
temperature, depending on the doping level. Concomitantly, the longitudinal
resistivity as a function of $T$ acquires two plateaus that smoothly cross
over to each other. These unfamiliar transport behaviors are shown to be
associated with the formation of coherent interlayer singlets, which scatter
off gapped collective modes arising from short-range interlayer
antiferromagnetic fluctuations.

Figure 1: Hall coefficient (left panel) and longitudinal resistivity (right
panel) of the bilayer Hubbard model as functions of the temperature at various
doping levels. The insets are close-up views of the lowest-temperature data.
The solid lines are the guide to the eye. The dashed lines in the inset of the
right panel are quadratic fits $\rho_{xx}=\mathrm{const.}\times T^{2}$. The
error bars represent Monte Carlo sampling errors and errors arising from DMFT
iterations [21], determined by four iterations starting from a converged
solution.

##  II Model and methods

The bilayer square-lattice Hubbard model consists of two square lattices
stacked site-to-site. We consider only the nearest-neighbor intralayer hopping
$t$ and interlayer hopping $t_{\perp}$. The Hamiltonian is

$\displaystyle H$ $\displaystyle=$
$\displaystyle-\sum_{\ell=1}^{2}\sum_{\langle\imath,\jmath\rangle,\sigma}tc_{\ell\imath\sigma}^{\dagger}c_{\ell\jmath\sigma}-\sum_{\imath,\sigma}(t_{\perp}c_{1\imath\sigma}^{\dagger}c_{2\imath\sigma}+\text{H.c.})$
$\displaystyle+\sum_{\ell=1}^{2}\sum_{\imath}Un_{\ell\imath\uparrow}n_{\ell\imath\downarrow}.$

Here $c_{\ell\imath\sigma}$ ($c_{\ell\imath\sigma}^{\dagger}$) annihilates
(creates) an electron of spin $\sigma$ ($=\uparrow,\downarrow$) on the site
$\imath$ in the layer $\ell$. $U$ is the onsite Coulomb repulsion and
$n_{\ell\imath\sigma}=c_{\ell\imath\sigma}^{\dagger}c_{\ell\imath\sigma}$ is
the electron number operator. The Kubo formulae for the longitudinal and Hall
conductivities (sheet conductances) in a vanishing out-of-plane magnetic field
$B_{z}\rightarrow 0$ can be directly extended from those for the monolayer
Hubbard model [8, 22, 23],

$\displaystyle\sigma_{xx}$ $\displaystyle=$
$\displaystyle\frac{e^{2}\pi}{\hbar
N}\sum_{\mathbf{k},\sigma}\left(\frac{\partial\epsilon_{\mathbf{k}}}{\partial
k_{x}}\right)^{2}\int
d\omega\operatorname{Tr}[\hat{A}_{\mathbf{k}\sigma}(\omega)^{2}]\left[-\frac{df(\omega)}{d\omega}\right],$
$\displaystyle\frac{\sigma_{xy}}{B_{z}}$ $\displaystyle=$
$\displaystyle\frac{2\pi^{2}e^{3}a^{2}}{3\hbar^{2}N}\sum_{\mathbf{k},\sigma}\left(\frac{\partial\epsilon_{\mathbf{k}}}{\partial
k_{x}}\right)^{2}\frac{\partial^{2}\epsilon_{\mathbf{k}}}{\partial k_{y}^{2}}$
$\displaystyle\times\int
d\omega\operatorname{Tr}[\hat{A}_{\mathbf{k}\sigma}(\omega)^{3}]\left[-\frac{df(\omega)}{d\omega}\right].$

Here $e$ is the elementary charge magnitude, $\hbar$ is the reduced Planck
constant, $a$ is the in-plane lattice constant, $N$ is the number of unit
cells in the lattice, and $\mathbf{k}=(k_{x},k_{y})$ is the reciprocal vector
in the first Brillouin zone. $f(\omega)=(1+e^{\hbar\omega/T})^{-1}$ is the
Fermi distribution function. The energy of the bonding or antibonding band up
to a constant shift is $\epsilon_{\mathbf{k}}=-2t(\cos k_{x}+\cos k_{y})$.
$\hat{A}$ is the spectral function, which is a matrix in the layer index
space.

We choose $t_{\perp}=1.2t$ and $U=10t$, which are relevant to the material VO2
[15], a prototypical strongly correlated oxide with the vanadium dimer as the
basic unit [24]. In the DMFT, we consider an interlayer dimer embedded in a
self-consistent noninteracting electron bath, thereby fully taking into
account the short-range interlayer electron correlation. We use the
continuous-time auxiliary-field Monte Carlo method [25, 26] to solve the
corresponding quantum impurity problem, accurately measuring the self-energy
at all Matsubara frequencies. We then employ the recently developed maximum
quantum entropy method [27, 28] to analytically continue the self-energy
matrix to the real-frequency axis.

##  III Transport coefficients

Figure 1 shows the calculated Hall coefficient
$R_{H}=\sigma_{xy}/(B_{z}\sigma_{xx}^{2})$ and longitudinal resistivity
$\rho_{xx}=\sigma_{xx}^{-1}$ as functions of the temperature at various hole
doping levels $p=1-\sum_{\sigma}\langle n_{\ell\imath\sigma}\rangle$. For
$T\gtrsim 0.1t$, the $T$ dependence of $R_{H}$ is similar for all doping
levels shown, but $R_{H}$ shifts downward with increasing doping. In this
temperature range, as $T$ increases, $R_{H}$ decreases in $0.1t\lesssim
T\lesssim 0.13t$, then increases in $0.13t\lesssim T\lesssim 0.5t$, and then
decreases again for $T\gtrsim 0.5t$. In $0.67t\lesssim T\lesssim 1t$,
$R_{H}(T)$ changes more slowly with increasing doping and becomes almost flat
at $p=0.3$. Depending on the doping level, $R_{H}$ can be totally below zero
($p=0.3$), or change sign once ($p=0.2$) or three times ($p=0.25$) in this
range $T\gtrsim 0.1t$.

For $T\lesssim 0.1t$, the behaviors of $R_{H}(T)$ at different doping levels
are radically different. In this temperature range, the $T$ dependence of
$R_{H}$ quickly weakens with increasing doping. At $p=0.2$ and $p=0.25$,
$R_{H}$ changes sign once due to its strong dependence on $T$. But for a
heavier doping $p=0.3$, $R_{H}(T)$ is almost a negative constant. Therefore,
the total number of times $R_{H}(T)$ changes its sign counts to zero at
$p=0.3$, two at $p=0.2$, and as many as four at $p=0.25$, in contrast to the
single or double sign reversal normally observed in high-temperature
superconductors [2, 29, 6] and the single-orbital Hubbard model [8, 12].

Figure 2: Single-particle excitation spectra $A(\mathbf{k},\omega)$ along a
path connecting high-symmetry reciprocal points, and corresponding densities
of states $A(\omega)$ at indicated temperatures for $p=0.25$. The black dotted
lines are the noninteracting bands for the same doping. The Fermi level lies
at $\omega=0$.

The longitudinal resistivity $\rho_{xx}(T)$ also shows unfamiliar behavior
(Fig. 1, right panel). There are two temperature ranges, $0.26t\lesssim
T\lesssim 0.5t$ and $T\lesssim 0.1t$, where $\rho_{xx}(T)$ is almost constant,
and these ranges become broader for heavier doping. Especially at low
temperatures, $\rho_{xx}(T)$ deviates significantly from the quadratic fits
$\rho_{xx}=\mathrm{const.}\times T^{2}$ expected for a Fermi liquid (Fig. 1,
right panel, inset). Nevertheless, the quadratic fit is improved for heavier
doping, along with the weaker $T$ dependence of $R_{H}$ for heavier doping
(Fig. 1, left panel, inset), demonstrating that the system at low temperatures
approaches the Fermi liquid phase as doping increases. The constant value of
$\rho_{xx}$ in $T\lesssim 0.1t$ does not change much as the doping level is
varied, whereas at high temperatures, $\rho_{xx}$ exceeds the Mott-Ioffe-Regel
limit ($\sim\sqrt{2\pi}\hbar/e^{2}\approx 2.5\hbar/e^{2}$ [5]) and is lower
for heavier doping consistent with more charge carriers.

##  IV Mechanisms

To understand the anomalous behaviors of the Hall coefficient and longitudinal
resistivity, we plot in Fig. 2 the single-particle excitation spectra
$A(\mathbf{k},\omega)=\sum_{\sigma}\operatorname{Tr}\hat{A}_{\mathbf{k}\sigma}(\omega)$
and densities of states $A(\omega)=\sum_{\mathbf{k}}A(\mathbf{k},\omega)/N$ at
various temperatures for $p=0.25$. The noninteracting band structure is also
superimposed (dotted lines). In the noninteracting limit, at light doping, the
bonding (lower-lying) band has a hole pocket at the $M$ point and the
antibonding (higher-lying) band has a smaller electron pocket at the $\Gamma$
point, which is the case for $p=0.25$. At heavy doping, both the bonding and
antibonding bands have an electron pocket at the $\Gamma$ point.

At a high temperature $T=2t$, the spectrum is highly incoherent and
continuous, showing only two broad Hubbard bands centered around $\omega=-1t$
and $\omega=10t$, respectively, with a pseudogap in between. Charge excitation
across this pseudogap produces a negative $R_{H}$ [8]. As $T$ is lowered to
$1t$, the spectrum near the Fermi level becomes more coherent, and the peak of
the density of states of the lower Hubbard band moves to a higher energy.
Fewer electrons are excited onto the upper Hubbard band, leading to a more
holelike $R_{H}$, that is, a larger $R_{H}$.

Figure 3: Left panel: interlayer singlet correlation as a function of the
temperature at various doping levels. The vertical dotted lines mark the
temperatures of the maximum correlation strength, $T_{m}$. The dashed lines
are extrapolations to zero temperature by fitting the data for $T<T_{m}$ to
cubic functions. The arrows indicate the approximate boundaries of the two
resistivity plateaus. Right panel: imaginary part of the self-energy in the
low-energy range at $p=0.25$ for the various temperatures in correspondence to
Fig. 2.

At a moderate temperature $T=0.5t$, two dispersive quasiparticle bands develop
near and above the Fermi level, whereas the two incoherent Hubbard bands
persist. The quasiparticle bands are renormalized by the Kondo screening [15]
to be approximately twice as narrow as their noninteracting counterparts. To
see the interlayer Kondo screening, we show in the left panel of Fig. 3 the
nearest-neighbor interlayer spin correlation function $\langle
S_{1\imath}^{z}S_{2\imath}^{z}\rangle$ as a function of temperature at various
doping levels, where
$S_{\ell\imath}^{z}=n_{\ell\imath\uparrow}-n_{\ell\imath\downarrow}$ is the
spin density operator. The neighboring interlayer spins do tend to be
antiparallel screening each other. The singlet correlation strength peaks at a
nonzero temperature that increases with increasing doping. For $T\lesssim
0.5t$, the singlet correlation strength is significant [14] and considerably
larger than that at $T=2t$, compatible with the observed crossover from the
totally incoherent spectrum at $T=2t$ to the coherent bands at $T=0.5t$, which
can thus be viewed as interlayer singlet bands [15]. Compared to the spectrum
at $T=1t$, the hole pocket at the $M$ point is relatively well defined thanks
to the coherent singlet bands, while the electron pocket at the $\Gamma$ point
remains incoherent, leading to an overall positive $R_{H}$ (Fig. 1, left
panel).

As $T$ is lowered from $0.5t$ to $0.267t$, the spectral function does not
change much; especially the smearing of the quasiparticle bands at these two
temperatures is similar, both being $\sim 0.5t$ wide and having a spectral
weight of $\sim 3t^{-1}$. This is consistent with the Kondo resonance, in
which the width and height of the resonance peak are temperature-independent
and are only determined by one energy scale, the Kondo temperature [30]. In
this temperature range, $T$ is less than the smearing width of the
quasiparticle bands, resulting in approximately $T$-constant resistivity (Fig.
1, right panel) reminiscent of resistivity saturation in the Kondo problem.
There are two slight, yet visible changes in the spectral function as $T$ is
lowered from $0.5t$ to $0.267t$. One is that the electron pocket at the
$\Gamma$ point becomes more well defined and deeper on the way to the complete
formation of the coherent singlet bands. The other is the appearance of a weak
incoherent flat band with a smearing width of $\sim 1t$ at the Fermi level,
which hosts electrons. These two changes tend to reduce $R_{H}$ (Fig. 1, left
panel).

At a relatively low temperature $T=0.1t$, a small single-particle gap _in the
incoherent spectrum_ ($\sim 0.4t$) opens at the Fermi level. Concomitantly,
the coherent singlet bands within the incoherent spectrum gap (spectral weight
$\sim 30t^{-1}$) become much sharper and flatter than those at $T=0.267t$
(spectral weight $\sim 3t^{-1}$). This results in a very sharp peak in the
density of states at the Fermi level, which is the manifestation of a strong
Kondo resonance. The opening of the gap in the incoherent spectrum depopulates
the upper Hubbard band, and the hole pocket of the well-defined singlet bands
is larger than its electron pocket, resulting in an increase in $R_{H}(T)$
from $T\simeq 0.13t$ to $T\simeq 0.1t$ (Fig. 1, left panel).

As $T$ decreases from $0.1t$ to $0.05t$, the quasiparticle bands again remain
almost unchanged, with a $\sim 0.1t$ smearing width and $\sim 30t^{-1}$
spectral weight at the Fermi level. Therefore, the resistivity in $T\lesssim
0.1t$ should also be approximately temperature-independent (Fig. 1, right
panel). The two plateaus in $\rho_{xx}(T)$ are smoothly connected by a
crossover spanning from $T\simeq 0.1t$ to $T\simeq 0.26t$. At $T=0.05t$, a
flat band (not the quasiparticle band) with weak intensity ($\sim 0.1t^{-1}$)
appears at the Fermi level, which hosts electrons leading to a decrease in
$R_{H}(T)$ from $T\simeq 0.1t$ to $T\simeq 0.05t$ (Fig. 1, left panel).

The non-Fermi liquid behavior down to very low temperatures also has a
manifestation in the interlayer spin correlation, which does not vanish at
zero temperature (Fig. 3, left panel). Instead, the singlet correlation
extrapolated to zero temperature is $\sim-0.06$, which is very close to the
nearest-neighbor spin correlation responsible for the non-Fermi liquid
scattering in the overdoped monolayer square-lattice Hubbard model at low
temperatures [14]. A dynamical cluster approximation study of the doped
bilayer Hubbard model suggested that there exists non-Fermi liquid behavior
even in the absence of finite scattering rate at vanishing temperature,
attributed to short-range interlayer antiferromagnetic fluctuation [31]. This
observation is consistent with our results, nevertheless we showed that the
resulting non-Fermi liquid behavior is not the $T$-linear resistivity but a
resistivity plateau.

To show where the Kondo saturation comes from, we depict in the right panel of
Fig. 3 the imaginary part of the self-energy,
$\Sigma(\omega)=\sum_{\sigma}\operatorname{Tr}\hat{\Sigma}_{\sigma}(\omega)$,
where $\hat{\Sigma}_{\sigma}(\omega)$ is the self-energy matrix of spin
$\sigma$ in the layer-index space. At the lowest temperature shown, $T=0.05t$,
$-\Im\Sigma(\omega)$ has two peaks in the low-energy range located at
$\omega\simeq 0$ and $\omega\simeq 0.9t$, respectively, which are separated by
a gap spanning $0.05t\lesssim\omega\lesssim 0.3t\equiv\omega_{g}$. These two
peaks are absent in the DMFT result of the single-orbital Hubbard model
[following the Fermi liquid behavior at low temperatures,
$-\Im\Sigma(\omega)\sim\omega^{2}$ at small $\omega$ [23, 19]] and represent
two low-energy scattering modes arising from short-range interlayer
antiferromagnetic fluctuations. The zero-energy mode should be the
simultaneous flip of spins in the inert interlayer singlet costing no energy,
while the second mode should be the flip of a single spin in the singlet.
There is also a pseudogap at $\omega\simeq 2t$ that separates the second
scattering mode and a broad peak at a high energy $\omega\simeq 7.8t$ (not
shown) corresponding to scattering off doubly-occupied-site states.

The zero-energy scattering mode is responsible for nonzero resistivity at
vanishing temperatures. For $T\lesssim 0.1t\simeq\omega_{g}/3$, electrons near
the Fermi level thermally fluctuate so weakly that they cannot cross the gap
to scatter off the second scattering mode [32]. Therefore, the scattering rate
$-\Im\Sigma(0)$ at $T=0.1t$ saturates and is close to that at $T=0.05t$,
leading to the lower resistivity plateau at $T\lesssim 0.1t$. Similarly, the
pseudogap renders the scattering rates $-\Im\Sigma(0)$ at $T\simeq 0.267t$ and
$T\simeq 0.5t$ near and both close to that of the second scattering mode
[$-\Im\Sigma(0.9t)$ at $T=0.05t$], forming the higher resistivity plateau at
$0.26t\lesssim T\lesssim 0.5t$. But because the pseudogap is not a genuine
gap, the higher resistivity plateau is not as flat as the lower resistivity
plateau (Fig. 1, right panel). For heavier doping, the Kondo resonance peak
will get wider [23, 19], i.e., the (pseudo-) gap in $-\Im\Sigma(\omega)$ will
become wider, resulting in wider resistivity plateaus.

##  V Conclusion

In conclusion, we have calculated the Hall coefficient and longitudinal
resistivity of the hole-doped bilayer Hubbard model and found that its
transport properties are very different from those of the monolayer or single-
orbital Hubbard model. The Hall coefficient has a strong nonmonotonic
dependence on temperature at low temperatures, and it can change sign four
times for some range of doping. The resistivity at low temperatures is not
linear in temperature like that of strange metals or quadratic in temperature
like that of Fermi liquids. Rather, it exhibits two plateaus with a smooth
crossover between them. These anomalous transport behaviors can be traced back
to the formation of coherent interlayer singlets, which scatter off gapped
collective modes arising from short-range interlayer antiferromagnetic
fluctuations.

Although we did not account for vertex corrections to the conductivity, the
corresponding analysis for the single-band Hubbard model hints at the effect
of the vertex corrections. Inclusion of the vertex corrections shifts the
longitudinal resistivity downward, but preserves its temperature dependence
[33, 34]. Neither does it alter the trend of the Hall coefficient as a
function of temperature [7, 8, 12]. Therefore, we do not expect the vertex
corrections to give rise to resistivity behaviors qualitatively different from
our conclusions.

##  VI Acknowledgements

This work was supported as part of the Computational Materials Sciences
Program funded by the U.S. Department of Energy, Office of Science, Basic
Energy Sciences, under Award No. DE-SC0020145. JS was supported by the U.S.
Department of Energy, Office of Basic Energy Sciences, under Grant No. DE-
SC-0005042. This research used resources of the National Energy Research
Scientific Computing Center (NERSC), a U.S. Department of Energy Office of
Science User Facility located at Lawrence Berkeley National Laboratory,
operated under Contract No. DE-SC-0020145 using NERSC award BES-ERCAP0023632.

## References

  * Hwang  _et al._ [1994] H. Y. Hwang, B. Batlogg, H. Takagi, H. L. Kao, J. Kwo, R. J. Cava, J. J. Krajewski, and W. F. Peck, Scaling of the temperature dependent hall effect in ${\mathrm{la}}_{2\mathrm{-}\mathit{x}}$${\mathrm{sr}}_{\mathit{x}}$${\mathrm{cuo}}_{4}$, Phys. Rev. Lett. 72, 2636 (1994).
  * Tsukada and Ono [2006] I. Tsukada and S. Ono, Negative hall coefficients of heavily overdoped ${\mathrm{la}}_{2-x}{\mathrm{sr}}_{x}\mathrm{Cu}{\mathrm{o}}_{4}$, Phys. Rev. B 74, 134508 (2006).
  * Gurvitch and Fiory [1987] M. Gurvitch and A. T. Fiory, Resistivity of ${\mathrm{la}}_{1.825}$${\mathrm{sr}}_{0.175}$${\mathrm{cuo}}_{4}$ and ${\mathrm{yba}}_{2}$${\mathrm{cu}}_{3}$${\mathrm{o}}_{7}$ to 1100 k: Absence of saturation and its implications, Phys. Rev. Lett. 59, 1337 (1987).
  * Ando  _et al._ [2004] Y. Ando, S. Komiya, K. Segawa, S. Ono, and Y. Kurita, Electronic phase diagram of high-${T}_{c}$ cuprate superconductors from a mapping of the in-plane resistivity curvature, Phys. Rev. Lett. 93, 267001 (2004).
  * Gunnarsson  _et al._ [2003] O. Gunnarsson, M. Calandra, and J. E. Han, Colloquium: Saturation of electrical resistivity, Rev. Mod. Phys. 75, 1085 (2003).
  * Zhao  _et al._ [2019] S. Y. F. Zhao, N. Poccia, M. G. Panetta, C. Yu, J. W. Johnson, H. Yoo, R. Zhong, G. D. Gu, K. Watanabe, T. Taniguchi, S. V. Postolova, V. M. Vinokur, and P. Kim, Sign-reversing hall effect in atomically thin high-temperature ${\mathrm{bi}}_{2.1}{\mathrm{sr}}_{1.9}{\mathrm{cacu}}_{2.0}{\mathrm{o}}_{8+\delta}$ superconductors, Phys. Rev. Lett. 122, 247001 (2019).
  * Assaad and Imada [1995] F. F. Assaad and M. Imada, Hall coefficient for the two-dimensional hubbard model, Phys. Rev. Lett. 74, 3868 (1995).
  * Lange and Kotliar [1999] E. Lange and G. Kotliar, Magnetotransport in the doped mott insulator, Phys. Rev. B 59, 1800 (1999).
  * Markov  _et al._ [2019] A. A. Markov, G. Rohringer, and A. N. Rubtsov, Robustness of the topological quantization of the hall conductivity for correlated lattice electrons at finite temperatures, Phys. Rev. B 100, 115102 (2019).
  * Vučičević and Žitko [2021a] J. c. v. Vučičević and R. Žitko, Universal magnetic oscillations of dc conductivity in the incoherent regime of correlated systems, Phys. Rev. Lett. 127, 196601 (2021a).
  * Kuchinskii  _et al._ [2022] E. Z. Kuchinskii, N. A. Kuleeva, D. I. Khomskii, and M. V. Sadovskii, Hall effect in a doped mott insulator: Dmft approximation, JETP Letters 115, 402 (2022).
  * Wang  _et al._ [2020] W. O. Wang, J. K. Ding, B. Moritz, E. W. Huang, and T. P. Devereaux, Dc hall coefficient of the strongly correlated hubbard model, npj Quantum Materials 5, 51 (2020).
  * Huang  _et al._ [2019] E. W. Huang, R. Sheppard, B. Moritz, and T. P. Devereaux, Strange metallicity in the doped hubbard model, Science 366, 987 (2019).
  * Wú  _et al._ [2022] W. Wú, X. Wang, and A.-M. Tremblay, Non-fermi liquid phase and linear-in-temperature scattering rate in overdoped two-dimensional hubbard model, Proceedings of the National Academy of Sciences 119, e2115819119 (2022).
  * Nájera  _et al._ [2017] O. Nájera, M. Civelli, V. Dobrosavljević, and M. J. Rozenberg, Resolving the ${\mathrm{vo}}_{2}$ controversy: Mott mechanism dominates the insulator-to-metal transition, Phys. Rev. B 95, 035113 (2017).
  * Nájera  _et al._ [2018] O. Nájera, M. Civelli, V. Dobrosavljević, and M. J. Rozenberg, Multiple crossovers and coherent states in a mott-peierls insulator, Phys. Rev. B 97, 045108 (2018).
  * Mou  _et al._ [2022] Y. Mou, R. Mondaini, and R. T. Scalettar, Bilayer hubbard model: Analysis based on the fermionic sign problem, Phys. Rev. B 106, 125116 (2022).
  * Auerbach [2018] A. Auerbach, Hall number of strongly correlated metals, Phys. Rev. Lett. 121, 066601 (2018).
  * Georges  _et al._ [1996] A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Dynamical mean-field theory of strongly correlated fermion systems and the limit of infinite dimensions, Rev. Mod. Phys. 68, 13 (1996).
  * Vučičević and Žitko [2021b] J. Vučičević and R. Žitko, Electrical conductivity in the hubbard model: Orbital effects of magnetic field, Phys. Rev. B 104, 205101 (2021b).
  * Gull  _et al._ [2007] E. Gull, P. Werner, A. Millis, and M. Troyer, Performance analysis of continuous-time solvers for quantum impurity models, Phys. Rev. B 76, 235123 (2007).
  * Voruganti  _et al._ [1992] P. Voruganti, A. Golubentsev, and S. John, Conductivity and hall effect in the two-dimensional hubbard model, Phys. Rev. B 45, 13945 (1992).
  * Pruschke  _et al._ [1995] T. Pruschke, M. Jarrell, and J. Freericks, Anomalous normal-state properties of high-t c superconductors: intrinsic properties of strongly correlated electron systems?, Advances in Physics 44, 187 (1995).
  * Biermann  _et al._ [2005] S. Biermann, A. Poteryaev, A. I. Lichtenstein, and A. Georges, Dynamical singlets and correlation-assisted peierls transition in ${\mathrm{v}\mathrm{o}}_{2}$, Phys. Rev. Lett. 94, 026404 (2005).
  * Gull  _et al._ [2008] E. Gull, P. Werner, O. Parcollet, and M. Troyer, Continuous-time auxiliary-field monte carlo for quantum impurity models, Europhysics Letters 82, 57003 (2008).
  * Gull  _et al._ [2011] E. Gull, A. J. Millis, A. I. Lichtenstein, A. N. Rubtsov, M. Troyer, and P. Werner, Continuous-time monte carlo methods for quantum impurity models, Rev. Mod. Phys. 83, 349 (2011).
  * Sim and Han [2018] J.-H. Sim and M. J. Han, Maximum quantum entropy method, Phys. Rev. B 98, 205102 (2018).
  * Bergeron and Tremblay [2016] D. Bergeron and A.-M. S. Tremblay, Algorithms for optimized maximum entropy and diagnostic tools for analytic continuation, Phys. Rev. E 94, 023303 (2016).
  * Li  _et al._ [2019] D. Li, K. Lee, B. Y. Wang, M. Osada, S. Crossley, H. R. Lee, Y. Cui, Y. Hikita, and H. Y. Hwang, Superconductivity in an infinite-layer nickelate, Nature 572, 624 (2019).
  * Mahan [2000] G. D. Mahan,  _Many-particle physics_ , 3rd ed. (Klumer Academic/Plenum Publishers, New York, 2000).
  * Lee  _et al._ [2014] H. Lee, Y.-Z. Zhang, H. O. Jeschke, and R. Valentí, Competition between band and mott insulators in the bilayer hubbard model: A dynamical cluster approximation study, Phys. Rev. B 89, 035139 (2014).
  * Hartnoll and Mackenzie [2022] S. A. Hartnoll and A. P. Mackenzie, Colloquium: Planckian dissipation in metals, Rev. Mod. Phys. 94, 041002 (2022).
  * Vučičević  _et al._ [2019] J. Vučičević, J. Kokalj, R. Žitko, N. Wentzell, D. Tanasković, and J. Mravlje, Conductivity in the square lattice hubbard model at high temperatures: Importance of vertex corrections, Phys. Rev. Lett. 123, 036601 (2019).
  * Vranić  _et al._ [2020] A. Vranić, J. Vučičević, J. Kokalj, J. Skolimowski, R. Žitko, J. Mravlje, and D. Tanasković, Charge transport in the hubbard model at high temperatures: Triangular versus square lattice, Phys. Rev. B 102, 115142 (2020).


# SMU: smooth activation function for deep networks using smoothing maximum
technique

Koushik Biswas
&Sandeep Kumar
&Shilpak Banerjee
&Ashish Kumar Pandey

###### Abstract

Deep learning researchers have a keen interest in proposing two new novel
activation functions which can boost network performance. A good choice of
activation function can have significant consequences in improving network
performance. A handcrafted activation is the most common choice in neural
network models. ReLU is the most common choice in the deep learning community
due to its simplicity though ReLU has some serious drawbacks. In this paper,
we have proposed a new novel activation function based on approximation of
known activation functions like Leaky ReLU, and we call this function Smooth
Maximum Unit (SMU). Replacing ReLU by SMU, we have got 6.22% improvement in
the CIFAR100 dataset with the ShuffleNet V2 model.

_Keywords_ Smooth Activation Function $\cdot$ Deep Learning $\cdot$ Neural
Network

##  1 Introduction

Deep Neural network has emerged a lot in recent years and has significantly
impacted our real-life applications. Neural networks are the backbone of deep
learning. An activation function is the brain of the neural network, which
plays a central role in the effectiveness & training dynamics of deep neural
networks. Hand-designed activation functions are quite a common choice in
neural network models. ReLU [1] is a widely used hand-designed activation
function. Despite its simplicity, ReLU has a major drawback, known as the
dying ReLU problem in which up to 50% neurons can be dead during network
training. To overcome the shortcomings of ReLU, a significant number of
activations have been proposed in recent years, and Leaky ReLU [2], Parametric
ReLU [3], ELU [4], Softplus [5], Randomized Leaky ReLU [6] are a few of them
though they marginally improve performance of ReLU. Swish [7] is a non-linear
activation function proposed by the Google brain team, and it shows some good
improvement of ReLU. GELU [8] is an another popular smooth activation
function. It can be shown that Swish and GELU both are a smooth approximation
of ReLU. Recently, a few non-linear activations have been proposed which
improves the performance of ReLU, Swish or GELU. Some of them are either hand-
designed or smooth approximation of Leaky ReLU function, and Mish [9], ErfAct
[10], Padé activation unit [11], Orthogonal Padé activation unit [12] are a
few of them.

##  2 Related Work and Motivation

In a deep neural network, activations are either fixed before training or
trainable. Researchers have proposed several activations in recent years by
combining known functions. Some of these functions have hyperparameters or
trainable parameters. In the case of trainable activation functions,
parameters are optimized during training. Swish is a popular activation
function that can be used as either a constant or trainable activation
function, and it shows some good performance in a variety of deep learning
tasks like image classification, object detection, machine translation etc.
GELU shares similar properties like the Swish activation function, and it
gains popularity in the deep learning community due to its efficacy in natural
language processing tasks. GELU has been used in BERT [13], GPT-2 [14], and
GPT-3 [15] architectures. Padé activation unit (PAU) has been proposed
recently, and it is constructed from the approximation of the Leaky ReLU
function by rational polynomials of a given order. Though PAU improves network
performance in the image classification problem over ReLU, its variants, and
Swish, it has a major drawback. PAU contains many trainable parameters, which
significantly increases the network complexity and computational cost.

Motivated from these works, we propose activation functions using the
smoothing maximum technique. The maximum function is non-smooth at the origin.
We want to explore how the smooth approximation of the maximum function (which
can be used as an activation function) affects a network’s training dynamics
and performance. Our experimental evaluation shows that our proposed
activation functions are comparatively more effective than ReLU, Mish, Swish,
GELU, PAU etc., across different deep learning tasks. We summarise the paper
as follows:

  1. 1.

We have proposed activation functions by smoothing the maximum function. We
show that it can approximate GELU, ReLU, Leaky ReLU or the general Maxout
family.

  2. 2.

We show that the proposed functions outperform widely used activation
functions in a variety of deep learning tasks.

Figure 1: Approximation of ReLU using SMU ($\alpha=0$) for different values of
$\mu$. As $\mu\rightarrow\infty$, SMU smoothly approximate ReLU

Figure 2: Approximation of Leaky ReLU ($\alpha=0.25$) using SMU for different
values of $\mu$. As $\mu\rightarrow\infty$, SMU smoothly approximate Leaky
ReLU

Figure 3: First order derivatives of SMU for $\alpha=0.25$ and different
values of $\mu$.

##  3 Smooth Maximum Unit

We present Smooth Maximum Unit (SMU), smooth activation functions from the
smooth approximation of the maximum function. Using the smooth approximation
of the $|x|$ function, one can find a general approximating formula for the
maximum function, which can smoothly approximate the general Maxout[16]
family, ReLU, Leaky ReLU or its variants, Swish etc. We also show that the
well established GELU [8] function can be obtained as a special case of SMU.

###  3.1 Smooth approximation of the maximum function

Note that the maximum function can be expressed as following two different
ways:

$\displaystyle max(x_{1},x_{2})$ $\displaystyle=\begin{cases}x_{1}&\text{ if
}x_{1}\geq x_{2}\\\ x_{2}&\text{otherwise}\end{cases}$
$\displaystyle=\frac{(x_{1}+x_{2})+|x_{1}-x_{2}|}{2}$ (1)

Note that the max function is not differentiable at the origin. Using
approximations of the $|x|$ function by a smooth function, we can create
approximations to the maximum functions. There are many known approximations
to $|x|$, but for the rest of this article, we will focus on two specific
approximations of $|x|$, namely $x\text{erf}(\mu x)$ and
$\sqrt{x^{2}+{\mu}^{2}}$. We noticed that the activations constructed using
these two functions provide good performance on standard datasets on different
deep learning problems (for more details, see the supplementary section). Note
that $\sqrt{x^{2}+{\mu}^{2}}$ as $\mu\rightarrow 0$ approximate $|x|$ from
above while $x\text{erf}(\mu x)$. as $\mu\rightarrow\infty$ gives an
approximation of $|x|$ from below. The approximation is uniform on compact
subsets of the real line. Here $\operatorname{erf}$ is the Gaussian error
function defined as follows:

$\displaystyle\operatorname{erf}(x)=\dfrac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^{2}}\,dt.$

Now, replacing the $|x|$ function by $x\text{erf}(\mu x)$ in equation (3.1),
we have the smooth approximation formula for maximum function as follows:

$\displaystyle f_{1}(x_{1},x_{2};\mu)=\frac{(x_{1}+x_{2})+(x_{1}-x_{2})\
\text{erf}(\mu(x_{1}-x_{2}))}{2}.$ (2)

Similarly, we can derive the the smooth approximation formula for the maximum
function from equation (3.1) by replacing the $|x|$ function by
$\sqrt{x^{2}+{\mu}^{2}}$ as follows:

$\displaystyle
f_{2}(x_{1},x_{2};\mu)=\frac{(x_{1}+x_{2})+\sqrt{(x_{1}-x_{2})^{2}+\mu^{2}}}{2}$
(3)

Note that as $\mu\rightarrow\infty$, $f_{1}(x_{1},x_{2};\mu)\rightarrow$
max$(x_{1},x_{2})$ and as $\mu\rightarrow 0$,
$f_{2}(x_{1},x_{2};\mu)\rightarrow$ max$(x_{1},x_{2})$. For particular values
of $x_{1}$ and $x_{2}$, we can approximate known activation functions. For
example, consider $x_{1}=ax$, $x_{2}=bx$, with $a\neq b$ in (2), we get:

$\displaystyle f_{1}(ax,bx;\mu)=\frac{(a+b)x+(a-b)x\
\text{erf}(\mu(a-b)x)}{2}.$ (4)

This is a simple case from the Maxout family [16] while more complicated cases
can be found by considering nonlinear choices of $x_{1}$ and $x_{2}$. We can
similarly get smooth approximations to ReLU and Leaky ReLU. For example,
consider $x_{1}=x$ and $x_{2}=0$, we have smooth approximation of ReLU as
follows:

$\displaystyle f_{1}(x,0;\mu)=\frac{x+x\ \text{erf}(\mu x)}{2}.$ (5)

We know that GELU [8] is a smooth approximation of ReLU. Notice that, if we
choose $\mu=\frac{1}{\sqrt{2}}$ in equation (5), we can recover GELU
activation function which also show that GELU is smooth approximation of ReLU.
Also, considering $x_{1}=x$ and $x_{2}=\alpha x$, we have a smooth
approximation of Leaky ReLU or Parametric ReLU depending on whether $\alpha$
is a hyperparameter or a learnable parameter.

$\displaystyle f_{1}(x,\alpha x;\mu)=\frac{(1+\alpha)x+(1-\alpha)x\
\text{erf}(\mu(1-\alpha)x)}{2}.$ (6)

Note that, equation (5) and equation (6) approximate ReLU or Leaky ReLU from
below. Similarly, we can derive approximating function from equation (3) which
will approximate ReLU or Leaky ReLU from above.

The corresponding derivatives of equation (6) for input variable $x$ is

$\displaystyle\frac{d}{dx}f_{1}(x,\alpha
x;\mu)=\frac{1}{2}[(1+\alpha)+(1-\alpha)\ \text{erf}(\mu(1-\alpha)x)$
$\displaystyle+\frac{2}{\sqrt{\pi}}\mu(1-\alpha)^{2}xe^{-(\mu(1-\alpha)x)^{2}}]$
(7) $\displaystyle\text{where}\ \
\frac{d}{dx}\text{erf}(x)=\frac{2}{\sqrt{\pi}}e^{-x^{2}}.\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad$

Figures 3, 3, and 3 show the plots for $f_{1}(x,0;\mu)$, $f_{1}(x,0.25x;\mu)$,
and derivative of $f_{1}(x,0.25x;\mu)$ for different values of $\mu$. From the
figures it is clear that as $\mu\rightarrow\infty$, $f_{1}(x,\alpha x;\mu)$
smoothly approximate ReLU or Leaky ReLU depending on value of $\alpha$. We
call the function in equation (6) as Smooth Maximum Unit (SMU). Similarly, We
can derive a function by replacing $x_{1}=x$ and $x_{2}=\alpha x$ in equation
(3) and we call this function SMU-1. For all of our experiments, we will use
SMU and SMU-1 as our proposed activation functions.

###  3.2 Learning activation parameters via back-propagation

Trainable activation function parameters are updated using backpropagation
[17] technique. We implemented forward pass in both Pytorch [18] & Tensorflow-
Keras [19] API, and automatic differentiation will update the parameters.
Alternatively, CUDA [20] based implementation (see [2]) can be used and the
gradients of equations 8 and 9) for the parameters $\alpha$ and $\mu$ can be
computed as follows:

$\displaystyle\frac{\partial f_{1}}{\partial\alpha}=\frac{1}{2}[x-x\
\text{erf}(\mu(1-\alpha)x)-(1-\alpha)\mu x^{2}e^{-(\mu(1-\alpha)x)^{2}}]$ (8)
$\displaystyle\frac{\partial
f_{1}}{\partial\mu}=\frac{1}{2}(1-\alpha)^{2}x^{2}e^{-(\mu(1-\alpha)x)^{2}}$
(9)

$\alpha$ and $\mu$ can be either hyperparameters or trainable parameters. For
classification problem, we consider $\alpha=0.25$, a hyperparameter. We
consider $\mu$ as trainable parameter and initialise at $1.0$ and
$4.352665993287951e^{-09}$ for SMU and SMU-1 respectively. For object
detection and semantic segmentation problem, we consider $\alpha=0.01$, a
hyperparameter. We consider $\mu$ as trainable parameter and initialise at
$2.5$ and $4.332461424154261e-09$ for SMU and SMU-1 respectively.
Now, note that the class of neural networks with SMU and SMU-1 activation
function is dense in $C(K)$, where $K$ is a compact subset of $\mathbb{R}^{n}$
and $C(K)$ is the space of all continuous functions over $K$.
The proof follows from the following proposition (see [11]).

Proposition 1. (Theorem 1.1 in Kidger and Lyons, 2020 [21]) :- Let
$\rho:\mathbb{R}\rightarrow\mathbb{R}$ be any continuous function. Let
$N_{n}^{\rho}$ represent the class of neural networks with activation function
$\rho$, with $n$ neurons in the input layer, one neuron in the output layer,
and one hidden layer with an arbitrary number of neurons. Let
$K\subseteq\mathbb{R}^{n}$ be compact. Then $N_{n}^{\rho}$ is dense in $C(K)$
if and only if $\rho$ is non-polynomial.

##  4 Experiments

In the next subsections, we will provide a detailed experimental evaluation to
compare performance for our proposed activation function and some widely used
activation functions on 3 different deep learning problems like image
classification, object detection, and semantic segmentation.

###  4.1 CIFAR

We report the top-1 accuracy on Table 1 and Table 2 on CIFAR10 [22] and
CIFAR100 [22] datasets respectively. We consider MobileNet V2 [23], Shufflenet
V2 [24], PreActResNet [25], ResNet [26], VGG [27] (with batch-normalization
[28]), and EfficientNet B0 [29].

Model ReLU SMU SMU-1 Top-1 accuracy Top-1 accuracy Top-1 accuracy ShuffleNet
V2 1.0x 90.81 $\pm$ 0.24 92.72 $\pm$ 0.18 92.42 $\pm$ 0.20 ShuffleNet V2 2.0x
91.70 $\pm$ 0.20 93.61 $\pm$ 0.14 93.40 $\pm$ 0.16 PreActResNet 34 94.21 $\pm$
0.17 95.12 $\pm$ 0.13 94.93 $\pm$ 0.14 ResNet 34 94.22 $\pm$ 0.18 94.91 $\pm$
0.16 94.77 $\pm$ 0.17 SeNet 34 94.42 $\pm$ 0.20 95.27 $\pm$ 0.15 94.89 $\pm$
0.17 MobileNet V2 94.22 $\pm$ 0.15 95.50 $\pm$ 0.09 95.27 $\pm$ 0.10
EffitientNet B0 95.10 $\pm$ 0.15 96.23 $\pm$ 0.10 96.11 $\pm$ 0.12 Xception
90.51 $\pm$ 0.22 93.25 $\pm$ 0.17 92.59 $\pm$ 0.20 VGG16 93.59 $\pm$ 0.18
94.54 $\pm$ 0.14 93.32 $\pm$ 0.15

Table 1: Comparison between our proposed activations and other baseline
activations in the CIFAR10 dataset. We report the results for the mean of 15
different runs. mean$\pm$std is reported in the table.

Model ReLU SMU SMU-1 Top-1 accuracy Top-1 accuracy Top-1 accuracy Shufflenet
V2 1.0x 64.41 $\pm$ 0.25 70.60 $\pm$ 0.21 69.96 $\pm$ 0.22 Shufflenet V2 2.0x
67.52 $\pm$ 0.25 73.74 $\pm$ 0.20 73.45 $\pm$ 0.23 PreActResNet 34 73.41 $\pm$
0.24 76.21 $\pm$ 0.20 75.87 $\pm$ 0.21 ResNet 34 73.33 $\pm$ 0.27 75.77 $\pm$
0.20 75.59 $\pm$ 0.21 SeNet 34 75.12 $\pm$ 0.22 76.79 $\pm$ 0.18 75.79 $\pm$
0.21 MobileNet V2 74.17 $\pm$ 0.24 76.31 $\pm$ 0.19 76.03 $\pm$ 0.19 Xception
71.22 $\pm$ 0.26 74.62 $\pm$ 0.23 74.11 $\pm$ 0.23 EffitientNet B0 76.60 $\pm$
0.27 79.10 $\pm$ 0.22 78.77 $\pm$ 0.23 VGG16 71.87 $\pm$ 0.30 73.26 $\pm$ 0.22
72.79 $\pm$ 0.23

Table 2: Comparison between our proposed activations and other baseline
activations in the CIFAR100 dataset. We report the results for the mean of 15
different runs. mean$\pm$std is reported in the table.

###  4.2 Object Detection

In this section, we report results on Pascal VOC dataset [30] on Single Shot
MultiBox Detector(SSD) 300 [31] with VGG-16(with batch-normalization) [27] as
the backbone network. We report the mean average precision (mAP) in Table 3
for the mean of 6 different runs.

Activation Function |  mAP
---|---
SMU |  78.1$\pm$0.09
SMU-1 |  77.9$\pm$0.10
ReLU | 77.2$\pm$0.14
Leaky ReLU | 77.2$\pm$0.18
PReLU | 77.2$\pm$0.17
ReLU6 | 77.1$\pm$0.15
ELU | 75.1$\pm$0.22
Softplus | 74.2$\pm$0.25
Swish | 77.3$\pm$0.11
GELU | 77.3$\pm$0.12
Table 3: A detailed comparison between SMU activation and other baseline
activations in Pascal VOC dataset. mean$\pm$std is reported in the table

###  4.3 Semantic Segmentation

We report our experimental results in this section on the popular CityScapes
dataset [32]. The U-net model [33] is considered as the segmentation
framework, and we report the mean of 6 different runs for Pixel Accuracy and
the mean Intersection-Over-Union (mIOU) on test data on table 4.

Activation Function |  Pixel Accuracy | mIOU
---|---|---
SMU |  81.71$\pm$0.38 |  71.45$\pm$0.30
SMU-1 |  81.17$\pm$0.41 |  71.10$\pm$0.30
ReLU | 79.49$\pm$0.46 | 69.31$\pm$0.28
PReLU | 78.95$\pm$0.42 | 68.88$\pm$0.41
ReLU6 | 79.58$\pm$0.41 | 69.70$\pm$0.42
Leaky ReLU | 79.41$\pm$0.41 | 69.64$\pm$0.42
ELU | 79.48$\pm$0.50 | 68.19$\pm$0.40
Softplus | 78.45$\pm$0.52 | 68.08$\pm$0.49
Swish | 80.22$\pm$0.46 | 69.81$\pm$0.30
GELU | 80.14$\pm$0.37 | 69.59$\pm$0.40
Table 4: A detailed comparison between SMU activation and other baseline
activations in CityScapes dataset. mean$\pm$std is reported in the table.

##  5 Conclusion

This work uses the maximum smoothing technique to approximate Leaky ReLU, a
well-established non-smooth (not differentiable at 0) function by two smooth
functions. We call these two functions SMU and SMU-1, and we use them as a
potential candidate for activation functions. Our experimental evaluation
shows that the proposed functions beat the traditional activation functions in
well-known deep learning problems and have the potential to replace them.

## References

  * [1] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 807–814. Omnipress, 2010.
  * [2] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng.  Rectifier nonlinearities improve neural network acoustic models.  In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
  * [3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
  * [4] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus), 2016.
  * [5] Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li.  Improving deep neural networks using softplus units.  In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1–4, 2015.
  * [6] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.  Empirical evaluation of rectified activations in convolutional network, 2015.
  * [7] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for activation functions, 2017.
  * [8] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus), 2020.
  * [9] Diganta Misra.  Mish: A self regularized non-monotonic activation function, 2020.
  * [10] Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey.  Erfact and pserf: Non-monotonic smooth trainable activation functions, 2021.
  * [11] Alejandro Molina, Patrick Schramowski, and Kristian Kersting.  Padé activation units: End-to-end learning of flexible activation functions in deep networks, 2020.
  * [12] Koushik Biswas, Shilpak Banerjee, and Ashish Kumar Pandey.  Orthogonal-padé activation functions: Trainable activation functions for smooth and faster convergence in deep networks, 2021.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [14] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [15] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [16] Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.  Maxout networks, 2013.
  * [17] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural Computation, 1(4):541–551, 1989.
  * [18] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library, 2019\.
  * [19] François Chollet et al.  Keras.  https://keras.io, 2015.
  * [20] J. Nickolls, I. Buck, M. Garland, and K. Skadron.  Scalable parallel programming.  In 2008 IEEE Hot Chips 20 Symposium (HCS), pages 40–53, 2008.
  * [21] Patrick Kidger and Terry Lyons.  Universal approximation with deep narrow networks, 2020.
  * [22] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * [23] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks, 2019.
  * [24] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design, 2018.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks, 2016.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [27] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015\.
  * [28] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
  * [29] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
  * [30] Mark Everingham, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  Int. J. Comput. Vision, 88(2):303–338, June 2010.
  * [31] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  Ssd: Single shot multibox detector.  Lecture Notes in Computer Science, page 21–37, 2016.
  * [32] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding, 2016.
  * [33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015\.