††thanks: Corresponding author<EMAIL_ADDRESS>§ These authors
contributed equally to this work††thanks: Corresponding author:
<EMAIL_ADDRESS>Corresponding author<EMAIL_ADDRESS>
# Spin excitations in the kagome-lattice metallic antiferromagnet
Fe0.89Co0.11Sn

Tao Xie§ Neutron Scattering Division, Oak Ridge National Laboratory, Oak
Ridge, Tennessee 37831, USA  Qiangwei Yin§ Laboratory for Neutron Scattering,
and Beijing Key Laboratory of Optoelectronic Functional Materials MicroNano
Devices, Department of Physics, Renmin University of China, Beijing 100872,
China  Qi Wang  Laboratory for Neutron Scattering, and Beijing Key Laboratory
of Optoelectronic Functional Materials MicroNano Devices, Department of
Physics, Renmin University of China, Beijing 100872, China  A. I. Kolesnikov
Neutron Scattering Division, Oak Ridge National Laboratory, Oak Ridge,
Tennessee 37831, USA  G. E. Granroth  Neutron Scattering Division, Oak Ridge
National Laboratory, Oak Ridge, Tennessee 37831, USA  D. L. Abernathy  Neutron
Scattering Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee
37831, USA  Dongliang Gong  Department of Physics and Astronomy, University of
Tennessee, Knoxville, Tennessee 37996, USA  Zhiping Yin  Center for Advanced
Quantum Studies and Department of Physics, Beijing Normal University, Beijing
100875, China  Hechang Lei  Laboratory for Neutron Scattering, and Beijing Key
Laboratory of Optoelectronic Functional Materials MicroNano Devices,
Department of Physics, Renmin University of China, Beijing 100872, China  A.
Podlesnyak  Neutron Scattering Division, Oak Ridge National Laboratory, Oak
Ridge, Tennessee 37831, USA

###### Abstract

Kagome-lattice materials have attracted tremendous interest due to the broad
prospect for seeking superconductivity, quantum spin liquid states, and
topological electronic structures. Among them, the transition-metal kagome
lattices are high-profile objects for the combination of topological
properties, rich magnetism, and multiple-orbital physics. Here we report an
inelastic neutron scattering study on the spin dynamics of a kagome-lattice
antiferromagnetic metal Fe0.89Co0.11Sn. Although the magnetic excitations can
be observed up to $\sim$250 meV, well-defined spin waves are only identified
below $\sim$90 meV and can be modeled using Heisenberg exchange with
ferromagnetic in-plane nearest-neighbor coupling $J_{1}$, in-plane next-
nearest-neighbor coupling $J_{2}$, and antiferromagnetic (AFM) interlayer
coupling $J_{c}$ under linear spin-wave theory. Above $\sim$90 meV, the spin
waves enter the itinerant Stoner continuum and become highly damped particle-
hole excitations. At the $K$ point of the Brillouin zone, we reveal a possible
band crossing of the spin wave, which indicates a potential Dirac magnon. Our
results uncover the evolution of the spin excitations from the planar AFM
state to the axial AFM state in Fe0.89Co0.11Sn, solve the magnetic Hamiltonian
for both states, and confirm the significant influence of the itinerant
magnetism on the spin excitations.

##  I Introduction

The magnetic kagome lattice, a two-dimensional (2D) network of corner-sharing
triangles surrounding hexagons, provides an ideal platform to search for
exotic states such as quantum spin liquids Balents (2010); Wen  _et al._
(2019); Broholm  _et al._ (2020); Lhotel  _et al._ (2020) and other
topological quantum states Nakatsuji  _et al._ (2015); Wang  _et al._ (2018);
Liu  _et al._ (2018); Ye  _et al._ (2018); Yin  _et al._ (2018); Liu  _et al._
(2020, 2021); Wang  _et al._ (2021); Kang  _et al._ (2020); Yin  _et al._
(2021). Theoretical studies have shown that the typical electronic bands of
kagome lattices contain linearly dispersive Dirac bands and nondispersive flat
bands Guo and Franz (2009); Mazin  _et al._ (2014). When a kagome lattice is
occupied by 3$d$ transition-metal atoms, the combination of the rich
magnetism, topological electronic bands, and multiple-orbital characteristics
will induce abundant novel phenomena such as the anomalous Hall effect
Nakatsuji  _et al._ (2015); Ye  _et al._ (2018); Yin  _et al._ (2018); Wang
_et al._ (2018); Liu  _et al._ (2018, 2021); Wang  _et al._ (2021).

In modern magnetic theory, the interaction between electron spins can be
described by a local moment picture or itinerant electron model Moriya (1985);
Borovik-Romanov and Sinha (1988); Melnikov and Reser (2018); Coey and Parkin
(2021). Although the former case is usually appropriate in the magnetic
insulators and the latter model always comes into play in metallic magnets,
some metallic systems can be analyzed by the local moment model Ishikawa
(1978); Endoh and B$\rm\ddot{o}$ni (2006). What is more, both the local moment
and the itinerant electron scenarios can coexist in some systems Zhao  _et
al._ (2009); Dai (2015); Lee  _et al._ (2018). For example, in iron-based
superconductors, the spin waves can be reproduced by an effective Heisenberg
Hamiltonian taking into consideration the anisotropic spin-wave damping
characteristics of an itinerant electron system Zhao  _et al._ (2009); Dai
(2015). In some itinerant magnets, well-defined spin-wave excitations only can
be observed in the low-energy region before entering the Stoner continuum
[Fig. 1(e)], in which the spin waves decay into damped particle-hole
excitations Mook and Nicklow (1971); Mook  _et al._ (1974); Ishikawa (1978);
Borovik-Romanov and Sinha (1988); Melnikov and Reser (2018); Chen  _et al._
(2020); Coey and Parkin (2021).

Recently, Dirac fermions and flat electronic bands have been reported in
kagome-lattice metallic antiferromagnet FeSn, paramagnet CoSn, and the doped
compounds Fe1-xCoxSn by angle-resolved photoemission spectroscopy studies Kang
_et al._ (2020); Liu  _et al._ (2020); Lin  _et al._ (2020); Moore  _et al._
(2022). The family including FeSn and CoSn has a hexagonal structure with
$P$6/$mmm$ space group. The Fe and Co atoms form the kagome lattice with
hexagonal holes filled with Sn atoms [Fig. 1(a)]. In antiferromagnetic (AFM)
FeSn, below the Néel temperature $T_{N}$ = 365 K, the magnetic moments of Fe
in each kagome layer align ferromagnetically, and the adjacent ferromagnetic
(FM) layers stack in an antiparallel manner along the $c$ axis [Fig. 1(b)]
Yamaguchi and Watanabe (1967); Häggström  _et al._ (1975); Kakihana  _et al._
(2019); Sales  _et al._ (2019); Lin  _et al._ (2020); Kang  _et al._ (2020).
With Co substitution at the Fe site in FeSn, the ordered moments’ direction
can be tuned from that in the $ab$ plane (planar AFM) [Fig. 1(b)] to along the
$c$ axis (axial AFM) [Fig. 1(c)] continuously by crossing an intermediate
state (tilted AFM) Meier  _et al._ (2019); Sales  _et al._ (2021). During this
process, the magnetic moments of the neighboring FM layers remain antiparallel
to each other. At some specific levels of Co doping, these different AFM
states can be obtained by simply changing the temperature Meier  _et al._
(2019).

Figure 1:  (a) The Fe-Co kagome layer with the hexagonal holes filled with Sn
in Fe1-xCoxSn. The in-plane nearest-neighbor and next-nearest-neighbor
exchange couplings are indicated by the red and green arrows, respectively.
(b) and (c) The magnetic structures for the planar and axial AFM states in
Fe1-xCoxSn. The orange arrows represent the interlayer nearest-neighbor
exchange coupling $J_{c}$. (d) A schematic of the 2D Brillouin zone of
Fe1-xCoxSn. (e) A schematic of the spin wave and Stoner continuum in some
metallic magnets. (f) A typical photo of Fe0.89Co0.11Sn single crystals on
1-mm grid paper. The long axis of the rodlike crystals is the crystallographic
$c$ axis. (g) The magnetization as a function of temperature of our
Fe0.89Co0.11Sn sample. FC, field cooled; ZFC, zero-field cooled. (h) and (i)
Zero-energy ($E$ = 0 $\pm$ 0.2 meV) 2D slices in ($H$ 0 $L$) with $E_{i}$ = 10
meV at 18 K (planar AFM state) and 280 K (axial AFM state), respectively. The
arcs at the lower right corner and the upper right corner are the scattering
intensity from the aluminum sampler holder. (j) and (k) Zero-energy ($E$ = 0
$\pm$ 1 meV) 2D slices in ($H$ $K$ 0.5) with $E_{i}$ = 80 meV at 10 K (planar
AFM state) and 280 K (axial AFM state), respectively. The dashed lines
indicate the boundary of the Brillouin zones in the ($H$ $K$ 0) plane.

Although the recent inelastic neutron scattering (INS) studies on FeSn show
some differences, both reports confirm the non-negligible effect of itinerant
electrons on the spin excitations Xie  _et al._ (2021); Do  _et al._ (2022),
which suggests that the combination of localized and itinerant magnetism
should be considered in this kind of metallic kagome-lattice AFM. While
theoretical calculations suggested the existence of a magnetic flat band in
FeSn Xie  _et al._ (2021), INS studies did not observe it Xie  _et al._
(2021); Do  _et al._ (2022). In addition, a damped Dirac magnon has been
suggested to exist in FeSn Do  _et al._ (2022). The Co substitution in FeSn
enriches the magnetism and may change the itinerancy of the electrons Sales
_et al._ (2019); Meier  _et al._ (2019), which makes Fe1-xCoxSn a good
candidate to study the topological magnon, the magnetic flat band, and their
interplay with the itinerant electrons as well as the evolution of these
properties with spin orientations.

In this paper, we select Fe0.89Co0.11Sn, which contains axial AFM, tilted AFM,
and planar AFM states in different temperature regions, as our research
object. By employing magnetization and neutron scattering measurements, we
first confirm the existence of the different AFM states in Fe0.89Co0.11Sn. We
subsequently obtain the in-plane FM spin excitations from zero energy to
$\sim$250 meV together with the out-of-plane AFM spin wave below $\sim$25 meV
in both planar and axial AFM states, which suggests quasi-2D magnetism in
Fe0.89Co0.11Sn. The magnetic excitations below $\sim$80–90 meV can be
described by linear spin-wave theory (LSWT) simulation. Above $\sim$90 meV,
the spin waves enter the Stoner continuum and decay into the highly damped
particle-hole excitations. Evidence of the existence of the Dirac magnon is
also observed at the $K$ point of the Brillouin zone (BZ), albeit its upper
part is obscured due to the interaction with the Stoner continuum from
itinerant magnetism.

##  II EXPERIMENTAL DETAILS

We prepared high-quality single crystals of Fe0.89Co0.11Sn using the self-flux
method. Details can be found in the Supplemental Material (SM) SI  (see also
Refs. Xu  _et al._ (2013); Shirane  _et al._ (2002) therein). The crystals are
long bars along the crystalline $c$ axis with a hexagonal cross section [Fig.
1(f)]. Magnetization measurements were performed using a Quantum Design (QD)
Magnetic Properties Measurement System (MPMS3). We coaligned about 100 single
crystals in the ($H$ 0 $L$) scattering plane on thin aluminum plates to obtain
a mosaic sample with a mass of about 2 g and mosaicity below 1.5$\degree$ SI .
The neutron scattering experiments were performed on the time-of-flight Wide
Angular-Range Chopper Spectrometer (ARCS) Abernathy  _et al._ (2012), Fine-
Resolution Fermi Chopper Spectrometer (SEQUOIA) Granroth  _et al._ (2010), and
Cold Neutron Chopper Spectrometer (CNCS) Ehlers  _et al._ (2011, 2016) at the
Spallation Neutron Source, Oak Ridge National Laboratory. Measurements were
carried out with a series of incident neutron energies $E_{i}$ = 3.32, 10, 80,
150, 250, 300, and 400 meV in both the planar AFM (at $T$ = 10 and 18 K) and
axial AFM (at $T$ = 280 K) states. The sample was rotated along the vertical
axis in a wide angle range (except for the $E_{i}$ = 400 meV measurement,
where the beam was fixed with the $k_{i}$ $\parallel$ $c$ axis) to make a
complete survey in the energy and momentum space. We used the software
packages Mantid Arnold  _et al._ (2014) and Horace Ewings  _et al._ (2016) for
neutron scattering data reduction and analysis. The neutron scattering
intensities are normalized to the same scale with arbitrary units using the
incoherent elastic scatterings of the sample SI . Throughout this paper, a
wave vector $\bf{Q}$ will be shown in reciprocal-lattice units (r. l. u.), in
which $\bf{Q}$ = ($H$, $K$, $L$) means $\bf{Q}$ = $H$$\bf{a}^{*}$ \+
$K$$\bf{b}^{*}$ \+ $L$$\bf{c}^{*}$, where $\bf{a}^{*}$, $\bf{b}^{*}$, and
$\bf{c}^{*}$ are basis vectors in reciprocal space.

##  III Neutron Scattering Results

###  III.1 Elastic neutron scattering

The magnetization measurements of Fe0.89Co0.11Sn show three characteristic
temperatures, named $T_{N}$ $\approx$ 340 K, $T_{N}^{{}^{\prime}}$ $\approx$
250 K, and $T_{N}^{{}^{\prime\prime}}$ $\approx$ 70 K [Fig. 1(g)], which
correspond to the phase-transition temperatures from the paramagnetic (PM)
state to the axial AFM state, the axial AFM state to the tilted AFM state, and
the tilted AFM state to the planar AFM state, respectively Meier  _et al._
(2019). In order to confirm the magnetic phases in our sample, we first check
the elastic neutron scattering results and then compare the results with those
in Ref. Meier  _et al._ (2019).

Figures 1(h)–1(k) present several zero-energy 2D slices in the ($H$ 0 $L$) and
($H$ $K$ 0.5) planes. In Fig. 1(h), we can see strong peaks appear at $Q$ =
(0, 0, $n$/2) ($n$ = integer) and $Q$ = (1, 0, $n$/2). The peaks at integer
$L$ are nuclear peaks, while the peaks at half-integer $L$ are magnetic Bragg
peaks, which correspond to a propagation vector $\mathbf{q}$ = (0, 0, 1/2).
The peaks in the ($H$ $K$ 0.5) plane shown in Fig. 1(j) further confirm the
fact that the ordered moments align ferromagnetically in the $ab$ plane. These
observations are consistent with the previous neutron diffraction results
Meier  _et al._ (2019). Since neutron scattering measurements probe the
magnetic moment components that are normal to the wave vector $\mathbf{Q}$, we
expect no magnetic Bragg peaks at $Q$ = (0, 0, $n$ \+ 1/2) for an axial AFM
state due to the parallel direction between magnetic moments and
$\mathbf{Q}$’s [Fig. 1(c)]. However, weak magnetic Bragg peaks still can be
observed at $Q$ = (0, 0, $n$ \+ 1/2) at $T$ = 280 K [Figs. 1(i) and 1(k)]. A
similar phenomenon has been reported in Ref. Meier  _et al._ (2019) and was
explained as the tails of inelastic scattering by low-energy transverse
magnons. In our study, we can rule out this possibility in Fe0.89Co0.11Sn
clearly (see details in Sec. IIIB) and confirm the existence of the small in-
plane magnetic moment components which result in the weak magnetic peaks at
$Q$ = (0, 0, $n$ \+ 1/2) in the axial AFM state. At last, we calculate the
component of the ordered moment along the $c$ axis, $m_{c}\approx
1.39~{}\mu_{B}$, and the small in-plane component $m_{ab}\approx
0.12~{}\mu_{B}$, which corresponds to a small canting angle
($\sim$4.84$\degree$) of the ordered spins away from the $c$ axis (see details
in the SM SI ).

Figure 2:  Dispersion of the spin wave. (a) and (b) Spin-wave dispersion along
the [$H$, 0, 0.5] direction [the red path in Fig. 1(d)] measured with $E_{i}$
= 80 meV, $T$ = 10 K, and along the [0, 0, $L$] direction measured with
$E_{i}$ = 150 meV, $T$ = 10 K. (c) Spin-wave dispersion along the [0, 0, $L$]
direction measured with $E_{i}$ = 3.32 meV, $T$ = 10 K. (d) and (e) Spin-wave
dispersion along the [$H$, 0, 0.5] and [0, 0, $L$] directions, respectively,
measured with $E_{i}$ = 80 meV, $T$ = 280 K. (f) Spin-wave dispersion along
the [0, 0, $L$] direction measured with $E_{i}$ = 3.32 meV, $T$ = 280 K. The
extra intensities in (a) and (d) around 10 and 30 meV in the high-$Q$ region
are the phonon signal. (g) Spin-wave dispersion along the [$H$,
0]$\rightarrow$[$H$, $H$] path at 10 K measured with $E_{i}$ = 150 meV. The
extra intensity around 10 meV is the intensity of phonon scattering. (h) and
(i) 1D constant-energy curves along the [$H$, 0, 0.5] direction and the [0, 0,
$L$] direction at different energies in both planar and axial AFM states. The
solid lines are fittings with the Gaussian function. (j) 1D constant-momentum
cuts at $Q$ = (0, 0, 0.5) in both planar and axial AFM states. The integration
range along the [$-K$/2 $K$ 0] direction for all the panels is $K$ = [$-$0.1,
0.1]. The dashed lines in (a), (b), (d), (e), and (g) are results of LSWT
fittings (see details in Sec. IV). The data points with error bars in (a),
(d), and (g) are the peak positions of the 1D constant-energy curves shown in
(h) and (i). The vertical error bars are the energy resolution of the
instrument, and the horizontal error bars are the full width at half maximum
(FWHM) of the Gaussian fittings there. The data points in (b) and (e) are peak
positions of the 1D constant-momentum curves. The vertical error bars are the
line width (FWHM) of spectra, and the horizontal error bars are the
integration momentum range.

###  III.2 Low- and intermediate-energy spin waves

Now the spin dynamics of Fe0.89Co0.11Sn are discussed. Figure 2 presents the
spin-wave results in the low and intermediate energy range, which were
collected with $E_{i}$ = 3.32 meV (at CNCS), $E_{i}$ = 80 meV (at SEQUOIA),
and $E_{i}$ = 150 meV (at ARCS, only $T$ = 10 K). There are three magnetic (Fe
or Co) atoms in one unit cell [Figs. 1(b) and 1(c)], which will give rise to
three magnon branches. As shown in Figs. 2(a) and 2(d), we can see the steep
dispersion of the acoustic magnon along the [$H$, 0, 0.5] direction for both
planar (10 K) and axial (280 K) AFM states. While the top of the acoustic
spin-wave band of the planar AFM state cannot be clearly seen in the measured
energy range with $E_{i}$ = 80 meV, the energy band top for the axial AFM
state seems to appear at $\sim$67 meV. Figure 2(g) shows spin-wave dispersion
along the [$H$, 0]$\rightarrow$[$H$, $H$] path at 10 K measured with $E_{i}$ =
150 meV. The top of the acoustic spin-wave band at the $M$ point appears
around 82 meV, above which the weak spin-excitation intensity continues up to
130 meV. On the other hand, the spin wave dispersion along the [0, 0, $L$]
direction reaches the band top at about 21 and 13 meV for the planar and axial
AFM states, respectively [Figs. 2(b) and 2(e)]. We extracted 1D constant-
energy curves from the spin-wave dispersion in Figs. 2(a), 2(b), 2(d), and
2(e), and fitted the curves with Gaussian functions. Some of the 1D constant-
energy curves are shown in Figs. 2(h) and 2(i). The data points in Figs. 2(a),
2(d), and 2(g) are the peak positions of these 1D constant-energy curves.

Figure 3:  Constant-energy 2D slices in the ($H$ $K$) plane and the ($H$ 0
$L$) plane. (a)–(e) The evolution of the spin wave with the increasing energy
in the ($H$ $K$) plane at $T$ = 10 and 18 K (planar AFM state). (f) Constant-
energy slice in the ($H$ 0 $L$) plane at $T$ = 10 K and $E$ = 50 $\pm$ 1 meV.
(g)–(k) The evolution of the spin wave with the increasing energy in the ($H$
$K$) plane at $T$ = 280 K (axial AFM state). (l) Constant-energy slice in the
($H$ 0 $L$) plane at $T$ = 280 K and $E$ = 50 $\pm$ 1 meV. The white dashed
lines represent the boundary of the Brillouin zones.  Figure 4:  The high-
energy spin wave and LSWT fittings with SPINW. (a)–(d) Experimental INS
spectra along the [$H$, 0] [the red path in Fig. 1(d)] and [$H$, $H$] [the
green path in Fig. 1(d)] directions measured with $E_{i}$ = 300 meV, $T$ = 18
K and the corresponding LSWT calculations. (e) Comparison of the experimental
spin-wave dispersion and the LSWT calculation of the spin-wave dispersion
along [0, 0, $L$] for the planar AFM state. (f)–(i) INS spectra along the
[$H$, 0] and [$H$, $H$] directions measured with $E_{i}$ = 250 meV, $T$ = 280
K and the corresponding LSWT calculations. (j) Comparison of the experimental
spin-wave dispersion and the LSWT calculation of the spin-wave dispersion
along [0, 0, $L$] for the axial AFM state. Some extra intensities that are
away from the main dispersion around 50 meV in (a), (c), (f), and (h) are
residual intensities due to the imperfect background subtraction. The gray
data points with error bars in (a), (c), (f), and (h) are fitted peak
positions of constant-energy curves. The vertical error bars are the energy
resolution of the instrument, and the horizontal error bars are the FWHM of
the Gaussian fittings. The dashed lines are LSWT calculations with the best
fitting parameters in Table 1. The black dashed lines indicate the acoustic
magnons, and the red and white dashed lines indicate the optical magnons.

In order to figure out whether the spin wave of Fe0.89Co0.11Sn is gapped or
gapless, we measured the low-energy excitations with $E_{i}$ = 3.32 meV (with
an energy resolution $\sim$0.11 meV at the elastic position SI ). From Figs.
2(c) and 2(f), we can see sharp spin waves stem from $Q$ = (0, 0, 0.5) for
both states. The 1D energy cuts at $Q$ = (0, 0, 0.5) show the evolution of the
spin-wave intensity more intuitively [Fig. 2(j)]. At 10 K, we do not see the
abrupt intensity decrease with the decreasing energy at $Q$ = (0, 0, 0.5),
which was observed and considered as evidence of the spin gap in FeSn Xie  _et
al._ (2021). This indicates that the spin wave of the planar AFM state in
Fe0.89Co0.11Sn is gapless in our resolution limit. At 280 K, the intensity of
the spin wave first gradually increases with the decreasing energy above 1
meV, then abruptly decreases with the decreasing energy below $\sim$0.5 meV
until the spin wave touches the tail of the Bragg peak, and upturns below
$\sim$0.1 meV [Fig. 2(j)]. Having this high-resolution data at the axial AFM
state, two important issues can be figured out clearly. First, in the axial
AFM state, there is a small spin gap below $\sim$0.5 meV, although the gap is
not fully opened. Second, the weak magnetic Bragg peaks observed at $Q$ = (0,
0, $n$ \+ 1/2) (see Sec. IIIA) are intrinsic magnetic peaks for the existence
of in-plane magnetic moments in the axial AFM state. These peaks cannot be the
tails of inelastic scattering by low-energy transverse magnons Meier  _et al._
(2019) because the inelastic tail should have a rather low intensity below
$\sim$0.2 meV for the opening of the spin gap. If the elastic peaks at $Q$ =
(0, 0, $n$ \+ 1/2) are inelastic tails, their intensity should be lower than
(or comparable to) the intensity of the excitations at $\sim$0.2 meV. With
this, there cannot be the obviously upward intensity below $\sim$0.1 meV as
shown in Fig. 2(j). Thus, we demonstrate that the weak magnetic Bragg peaks
observed at $Q$ = (0, 0, $n$ \+ 1/2) at 280 K are intrinsic, and the magnetic
moments are not perfectly aligned along the $c$ axis in the axial AFM state of
Fe0.89Co0.11Sn.

###  III.3 High-energy spin excitations

To cover the high-energy spin excitations of Fe0.89Co0.11Sn, we measured the
spin dynamics with higher incident neutron energies: $E_{i}$ = 300 meV (at
ARCS and $T$ = 18 K) and 250 meV (at SEQUOIA and $T$ = 280 K). Figure 3
presents some constant-energy 2D slices in the ($H$ $K$) plane and the ($H$ 0
$L$) plane. In the ($H$ $K$ 0.5) plane, the spin waves stem from the same
positions where the magnetic Bragg peaks are observed in Figs. 1(j) and 1(k).
The small spots then evolve to be circles with increasing energy. From Figs.
3(a)–3(c) and 3(g)–3(i), we can see that the sizes of the spots or circles in
the axial AFM state (280 K) are always larger than those in the planar AFM
state (10 or 18 K), which means that the spin excitation can reach the BZ
boundary at a lower energy in the axial AFM state and is consistent with the
analysis in Sec. IIIB. When the neutron energy transfer approaches $\sim$80
meV, the spin excitations evolve to the edge of the BZs, which indicates the
energy top of the acoustic magnon band. Furthermore, we found that the spin
excitation has no obvious intensity modulation along the [0, 0, $L$] direction
for energies above $\sim$30 meV [Figs. 3(f) and 3(l)]. Thus the results above
30 meV shown in Figs. 3 and 4 were extracted by integrating a wide range of
$L$ ($-$5 $\leq$ $L$ $\leq$ 5), and we will omit the $L$ indices for these
cases.

The INS spectra up to 215 meV are shown in Fig. 4. For the planar AFM state,
the dispersion along the [$H$, 0] direction shows strong intensity below
$\sim$100 meV, above which the signals become diffusive and rather weak, but
still can be identified up to $\sim$200 meV [Fig. 4(a)]. For the dispersion
along the [$H$, $H$] direction [Fig. 4(c)], a sharp spin wave below $\sim$100
meV and an obvious intensity decrease above $\sim$100 meV are also observed.
In the axial AFM state [Figs. 4(f) and 4(h)], the high-energy spin excitations
are similar to those in the planar AFM state, while the energy scale is
smaller than in the planar AFM state. It is worth noting that here we only can
observe the clear acoustic spin-wave mode and the weak diffusive spin
excitations (between $\sim$90 and $\sim$200 meV); any indications of the other
two expected magnon modes cannot be identified.

Table 1: Exchange coupling parameters (meV) for the magnetic Hamiltonian (IV) obtained from the LSWT fittings. Ordering type | $SJ_{1}$ | $SJ_{2}$ | $SJ_{c}$ | $SK_{c}$ | $SK_{a}$
---|---|---|---|---|---
Planar AFM | -18.15 $\pm$ 4.9 | -4.50 $\pm$ 1.89 | 10.87 $\pm$ 2.21 | 0 $<K_{c}\leq$ 0.0038 |
Axial AFM | -15.90 $\pm$ 4.32 | -3.95 $\pm$ 1.55 | 6.37 $\pm$ 1.54 | -0.009 | -0.009

##  IV LSWT simulations and discussions

To understand the experimentally observed magnetic excitations, we employ the
LSWT simulations using the SPINW library Toth and Lake (2015). We use the
following Heisenberg Hamiltonian:

$\displaystyle\mathcal{H}=J_{1}\sum_{\langle
i,j\rangle}\mathbf{S}_{i}\cdot\mathbf{S}_{j}+J_{2}\sum_{\langle\langle
i,j\rangle\rangle}\mathbf{S}_{i}\cdot\mathbf{S}_{j}+J_{c}\sum_{\langle
i,j\rangle}\mathbf{S}_{i}\cdot\mathbf{S}_{j}$
$\displaystyle+K_{c}\sum_{\langle i\rangle}(S^{z}_{i})^{2}+K_{a}\sum_{\langle
i\rangle}(S^{{\hat{e}}_{\mathbf{r}}}_{i})^{2},$ (1)

where $J_{1}$ is the in-plane nearest-neighbor (NN) exchange coupling, $J_{2}$
is the in-plane next-nearest-neighbor (NNN) exchange coupling, and $J_{c}$ is
the NN interlayer exchange coupling. The last two terms represent the single-
ion anisotropy. It should be noted that the last term is only applicable for
the axial AFM state. With $K_{a}<0$,
$K_{a}(S^{{\hat{e}}_{\mathbf{r}}}_{i})^{2}$ represents the in-plane easy-axis
anisotropy following the lattice symmetry, which is responsible for the small
canting angle of the ordered spins. The in-plane easy-axis direction
${\hat{e}}_{\mathbf{r}}$ depends on the position ($\mathbf{r}$) of the
magnetic atoms (see details in the SM SI ). Due to the itinerant properties of
such a metallic system, the spin value could be ambiguous. The effective spin
value may also change from the planar AFM state to the axial AFM state. To
describe the magnetic Hamiltonian smoothly, we thus use the combination of the
spin value and exchange coupling parameters $SJ_{1}$, $SJ_{2}$, $SJ_{c}$,
$SK_{c}$, and $SK_{a}$ hereinafter.

We first cut the experimental spectra and fitted them to get the dispersion
relation and intensity of the spin wave. Then we fit the extracted data using
SPINW with Hamiltonian (IV) to get the exchange coupling parameters (see
details in the SM SI ). The best fitting parameters are summarized in Table 1.
The fitted spin-wave dispersion curves for the acoustic magnon are shown as
dashed lines in Figs. 2(a), 2(b), 2(d), 2(e), and 2(g), which indeed can
describe the data points from the experimental results perfectly. The
calculated dynamical spin structure factors with the parameters in Table 1 are
shown in Figs. 4(b), 4(d), 4(e), 4(g), 4(i) and 4(j).

Note that the single-ion anisotropy parameters $SK_{c}$ and $SK_{a}$ in Table
1 are determined separately from $SJ_{1}$, $SJ_{2}$, and $SJ_{c}$ SI . This is
a reasonable approach since the anisotropy terms are rather small and have
negligible influence on the spin-wave dispersion here. Although no spin gap
can be identified in the planar AFM state, it is still necessary to include an
easy-plane single-ion anisotropy ($SK_{c}>0$) to confine the ordered spins in
the $ab$ plane. As for the axial AFM state, a minor spin gap below $~{}0.5$
meV has been identified, which requires a nonzero single-ion anisotropy to
open the gap. However, as the small canting angle ($\sim$4.84$\degree$) has
been confirmed by our previous analysis in Secs. IIIA and IIIB, a single easy-
axis anisotropy term ($SK_{c}<0$) cannot stabilize such a special magnetic
structure. Our solution is to add the $K_{a}\sum_{\langle
i\rangle}(S^{{\hat{e}}_{\mathbf{r}}}_{i})^{2}$ term ($SK_{a}<0$) in the
Hamiltonian. At last, we estimate the single-ion anisotropy parameters to be
$0<SK_{c}\leq$ 0.0038 meV in the planar AFM state and $SK_{c}$ = $SK_{a}$ =
$-$0.009 meV in the axial AFM state (see details in the SM SI ).

From Table 1, we can see that the generalized exchange coupling parameters
$SJ_{1}$, $SJ_{2}$, and $SJ_{c}$ decrease on different levels from the planar
AFM state to the axial AFM state. Specifically,
($SJ_{1}$)${}_{\text{axial}}/$($SJ_{1}$)${}_{\text{planar}}\approx$ 0.876,
($SJ_{2}$)${}_{\text{axial}}/$($SJ_{2}$)${}_{\text{planar}}\approx$ 0.878,
($SJ_{c}$)${}_{\text{axial}}/$($SJ_{c}$)${}_{\text{planar}}\approx$ 0.586. If
the effective local spin value in the Hamiltonian is supposed to be
proportional to the ordered magnetic moment of the ground state, we can get
the ratio of the effective spin value between the planar and the axial AFM
states: $S^{\text{axial}}/S^{\text{\text{planar}}}$ =
$m^{\text{axial}}/m^{\text{\text{planar}}}\approx$ 0.760 (see details in the
SM SI ). Then the ratios of the real exchange coupling parameters:
$J^{\text{axial}}_{1}/J^{\text{planar}}_{1}\approx$ 1.153,
$J^{\text{axial}}_{2}/J^{\text{planar}}_{2}\approx$ 1.155,
$J^{\text{axial}}_{c}/J^{\text{planar}}_{c}\approx$ 0.771. This means that
from the planar AFM state to the axial AFM state, the in-plane parameters
$J_{1}$ and $J_{2}$ increase synchronously, while the out-of-plane parameter
$J_{c}$ decreases. These interesting evolutions of the effective exchange
couplings should reflect the changes in the electronic band structure for the
planar-to-axial phase transition, which have been demonstrated by theoretical
calculations of the electronic structure on a similar compound, Fe0.94Co0.06Sn
Moore  _et al._ (2022). To explain the observed changes in the effective
exchange coupling parameters on the basis of the change in the electronic
structure, one should project the exchange coupling interactions into orbital-
resolved contributions. Exchange coupling parameters can be calculated from
the electronic structure using the known formalisms, e.g., the local spin
density functional Liechtenstein  _et al._ (1987) or the real-space linear-
muffin-tin method van Schilfgaarde and Antropov (1999); Frota-Pessôa  _et al._
(2000), which require extensive computational work that is out of the scope of
this paper.

Figure 5:  (a) and (b) Spin-excitation dispersion along the [$H-$1/3, 2/3]
direction [the blue path in Fig. 1(d)] in planar (a) and axial (b) AFM states.
The gray data points with error bars are fitted peak positions of constant-
energy curves. The vertical error bars are the energy resolution of the
instrument, and the horizontal error bars are the FWHM of the Gaussian
fittings. The black solid lines are the calculated spin-wave dispersion using
the LSWT with the parameters in Table 1. (c) and (d) High-energy spin
excitations measured with $E_{i}$ = 400 meV.

By comparing the experimental and LSWT simulation results, it is clear that
the LSWT calculation works well only for the acoustic spin wave. For the spin
excitations above the acoustic magnon, the data look ambiguous, and we cannot
identify the residual two optical magnon modes. The LSWT simulation cannot
cover the experimental results. The weak and ambiguous excitations above the
acoustic magnon and the disappearance of the optical magnon modes can be
explained by the interaction between the spin wave and the Stoner excitations
from the itinerant magnetism Mook and Nicklow (1971); Mook  _et al._ (1974);
Ishikawa (1978); Borovik-Romanov and Sinha (1988); Melnikov and Reser (2018);
Chen  _et al._ (2020); Coey and Parkin (2021); Xie  _et al._ (2021); Do  _et
al._ (2022); Zhang  _et al._ (2022). As depicted in Fig. 1(e), we only can
observe well-defined spin wave at the relatively low-energy region (below
$\sim$90 meV here) just before touching the lower boundary of the Stoner
continuum. After entering the Stoner continuum (above $\sim$90–100 meV here),
the spin waves decay into the particle-hole excitations, which makes the
optical magnon modes invisible and only leaves us the observable weak damped
excitations up to $\sim$250 meV. A recent $ab~{}initio$ study on FeSn
indicates that the Stoner continuum appears above $\sim$ 80–100 meV and
overlaps with the high-energy magnon spectra, which results in the strong
damping of the magnon Zhang  _et al._ (2022). Our results in Fe0.89Co0.11Sn
here are qualitatively consistent with this calculation and the experimental
INS results in FeSn Do  _et al._ (2022).

We further check the data across the $K$ points of BZs [the blue path shown in
Fig. 1(d)] in both AFM states. In Fig. 5(a), we can see that the spectrum
shows a downward-cone-like shape with the vertex appearing at $\sim$100 meV at
$T$ = 18 K. Above the downward cone, the excitation intensity becomes weak and
diffusive and is similar to the aforementioned results shown along the red and
green paths in Fig. 1(d). Similarly, at $T$ = 280 K, a downward cone with a
slightly lower vertex ($\sim$85 meV) can also be identified [Fig. 5(b)],
although it is not as clear as that at $T$ = 18 K. This feature is not easy to
understand at first glance. However, if we consider the aforementioned
interaction between the spin wave and the itinerant Stoner continuum, the
downward-cone-like excitation here should be the lower part of two crossed
spin-wave branches, with the upper part of the crossed branches becoming weak
and diffusive from entering the so-called Stoner continuum. The band-crossing-
like features can be further supported by the LSWT. In Figs. 5(a) and 5(b),
the black solid lines are the calculations from the LSWT using the parameters
in Table 1. The calculations indeed show band crossings at the vertices of the
measured downward cones. Such a kind of band crossing is known as the
criterion of Dirac magnons Yao  _et al._ (2018); Bao  _et al._ (2018); Yuan
_et al._ (2020); Elliot  _et al._ (2021); McClarty (2022). This indicates that
we may have found experimental evidence for the existence of Dirac magnons in
Fe0.89Co0.11Sn. However, the existence of the itinerant Stoner continuum stops
us from investigating this feature further. We note that a similar band
crossing feature was also observed in pure FeSn and argued to be damped Dirac
magnons Do  _et al._ (2022).

Another noteworthy point is that a magnetic flat band from the quasiparticle
excitations between the spin-up flat electronic band (majority electrons) and
spin-down flat electronic band (minority electrons) has been proposed for 2D
FM metals with a kagome lattice Xie  _et al._ (2021). In the case of FeSn,
despite the presence of AFM order below $T_{N}$, it was treated as a quasi-2D
FM metal for the weak AFM coupling between the adjacent FM planes Xie  _et
al._ (2021); Do  _et al._ (2022). Starting from such a quasi-2D FM metal, the
theoretical calculations predicted a magnetic flat band of the spin
excitations in FeSn. However, according to the INS results such a flat band is
absent up to $\sim$300 meV in FeSn Xie  _et al._ (2021); Do  _et al._ (2022).
Compared with the pure FeSn, Fe0.89Co0.11Sn has a similar AFM transition
temperature ($T_{N}$ $\approx$ 340 K), the same planar AFM order (below
$T_{N}^{{}^{\prime\prime}}$ $\approx$ 70 K), and comparable ordered magnetic
moment SI . Therefore we would expect that Fe0.89Co0.11Sn has a similar
electronic structure and itineracy to FeSn. This means that the predicted
magnetic flat band for FeSn Xie  _et al._ (2021) is expected to exist in
Fe0.89Co0.11Sn. Since such a flat band has not been observed up to $\sim$210
meV (measurements with $E_{i}$ = 300 meV) in our Fe0.89Co0.11Sn sample, we
then measured the higher-energy spin excitations with $E_{i}$ = 400 meV to see
if there is a magnetic flat band in higher-energy regions. We found the strong
spin-wave dispersion below $\sim$100 meV and the weak Stoner continuum
intensity up to $\sim$250 meV only [Figs. 5(c) and 5(d)]. Our results show
that there is no sign of the localized magnetic flat band in Fe0.89Co0.11Sn up
to $\sim$320 meV. The absence of such a flat band in experiments could have
three possible reasons: (i) The flat band is too weak to be visible. (ii) The
flat band may mix with the general Stoner continuum from other transition
channels and could lose its flatness and narrowness Zhang  _et al._ (2022).
Together with the possibly low intensity as mentioned in reason (i), the flat
band could become indistinguishable. (iii) The flat band does not exist, which
is inconsistent with the theoretical prediction Xie  _et al._ (2021).

##  V Summary

We have performed systematic neutron scattering measurements on the kagome-
lattice AFM metal Fe0.89Co0.11Sn. The planar and axial AFM ordered states are
confirmed by neutron diffraction and magnetization results. The careful
analyses of the diffraction and the low-energy spin-wave results demonstrate
that the weak magnetic Bragg peaks at $Q$ = (0, 0, $n$ \+ 1/2) ($n$ = integer)
of the axial state are intrinsic and come from the small in-plane magnetic
moment components. Although it has been well confirmed that the ordered
moments stack antiferromagnetically along the $c$ axis, the spin-excitation
spectra are dominated by the in-plane FM spin excitation, which indicates
quasi-2D magnetism. The INS shows a sharp spin wave below $\sim$90 meV, above
which the spin excitations become weak and diffusive, but persist up to
$\sim$250 meV. The sharp acoustic spin-wave band can be described in the frame
of LSWT by a Heisenberg $J_{1}$-$J_{2}$-$J_{c}$ model considering weak single-
ion anisotropy. In the axial AFM state, although the generalized exchange
coupling parameters $SJ_{1}$, $SJ_{2}$, and $SJ_{c}$ are smaller than those in
the planar state, $J_{1}$ and $J_{2}$ may show the opposite behavior if the
change in effective spin value is considered. Above the acoustic magnon, the
Stoner continuum appears, which makes the optical magnons highly damped and
invisible. At the $K$ points of the BZs, we give evidence for the existence of
the Dirac magnon with the upper part of the Dirac cone becoming weak and
decayed in the Stoner continuum. The magnetic flat band is demonstrated to be
absent in Fe0.89Co0.11Sn up to $\sim$320 meV. Our results give a comprehensive
overview of the INS experiments and LSWT calculations on kagome-lattice AFM
metal Fe0.89Co0.11Sn. The absence of the two optical magnon branches and the
upper part of the Dirac cone highlights the indispensable role of the
itinerant electrons in understanding the magnetism in Fe0.89Co0.11Sn.

## Acknowledgments

We thank Dr. Victor Fanelli for help with the experiment at the SEQUOIA
spectrometer, Dr. Jong Keum for help with the x-ray Laue measurements, and Dr.
Matthew Stone for suggestions about the neutron beam time application. Work at
Oak Ridge National Laboratory (ORNL) was supported by the U.S. Department of
Energy (DOE), Office of Science, Basic Energy Sciences, Materials Science and
Engineering Division. H.L. was supported by the National Key R$\&$D Program of
China (Grant No. 2018YFE0202600, 2022YFA1403800), the Beijing Natural Science
Foundation (Grant No. Z200005), and the National Natural Science Foundation of
China (12274459). This research used resources at the Spallation Neutron
Source, a DOE Office of Science User Facility operated by the Oak Ridge
National Laboratory. X-ray Laue measurements were conducted at the Center for
Nanophase Materials Sciences (CNMS) (CNMS2019-R18) at ORNL, which is a DOE
Office of Science User Facility.

## References

  * Balents (2010) L. Balents, “Spin liquids in frustrated magnets,” Nature 464, 199–208 (2010).
  * Wen  _et al._ (2019) Jinsheng Wen, Shun-Li Yu, Shiyan Li, Weiqiang Yu, and Jian-Xin Li, “Experimental identification of quantum spin liquids,” npj Quantum Materials 4, 12 (2019).
  * Broholm  _et al._ (2020) C. Broholm, R. J. Cava, S. A. Kivelson, D. G. Nocera, M. R. Norman, and T. Senthil, “Quantum spin liquids,” Science 367, eaay0668 (2020).
  * Lhotel  _et al._ (2020) Elsa Lhotel, Ludovic DC Jaubert, and Peter CW Holdsworth, “Fragmentation in frustrated magnets: A review,” J. Low Temp. Phys. 201, 710–737 (2020).
  * Nakatsuji  _et al._ (2015) Satoru Nakatsuji, Naoki Kiyohara, and Tomoya Higo, “Large anomalous Hall effect in a non-collinear antiferromagnet at room temperature,” Nature 527, 212–215 (2015).
  * Wang  _et al._ (2018) Qi Wang, Yuanfeng Xu, Rui Lou, Zhonghao Liu, Man Li, Yaobo Huang, Dawei Shen, Hongming Weng, Shancai Wang, and Hechang Lei, “Large intrinsic anomalous Hall effect in half-metallic ferromagnet Co3Sn2S2 with magnetic Weyl fermions,” Nat. Commun. 9, 3681 (2018).
  * Liu  _et al._ (2018) Enke Liu, Yan Sun, Nitesh Kumar, Lukas Muechler, Aili Sun, Lin Jiao, Shuo-Ying Yang, Defa Liu, Aiji Liang, Qiunan Xu, _et al._ , “Giant anomalous Hall effect in a ferromagnetic kagome-lattice semimetal,” Nat. Phys. 14, 1125 (2018).
  * Ye  _et al._ (2018) Linda Ye, Mingu Kang, Junwei Liu, Felix Von Cube, Christina R Wicker, Takehito Suzuki, Chris Jozwiak, Aaron Bostwick, Eli Rotenberg, David C Bell, _et al._ , “Massive Dirac fermions in a ferromagnetic kagome metal,” Nature 555, 638–642 (2018).
  * Yin  _et al._ (2018) Jia-Xin Yin, Songtian S Zhang, Hang Li, Kun Jiang, Guoqing Chang, Bingjing Zhang, Biao Lian, Cheng Xiang, Ilya Belopolski, Hao Zheng, _et al._ , “Giant and anisotropic many-body spin–orbit tunability in a strongly correlated kagome magnet,” Nature 562, 91–95 (2018).
  * Liu  _et al._ (2020) Zhonghao Liu, Man Li, Qi Wang, Guangwei Wang, Chenhaoping Wen, Kun Jiang, Xiangle Lu, Shichao Yan, Yaobo Huang, Dawei Shen, Jia-Xin Yin, Ziqiang Wang, Zhiping Yin, Hechang Lei, and Shancai Wang, “Orbital-selective Dirac fermions and extremely flat bands in frustrated kagome-lattice metal CoSn,” Nat. Commun. 11, 4002 (2020).
  * Liu  _et al._ (2021) Chang Liu, JianLei Shen, JiaCheng Gao, ChangJiang Yi, Di Liu, Tao Xie, Lin Yang, Sergey Danilkin, GuoChu Deng, WenHong Wang, _et al._ , “Spin excitations and spin wave gap in the ferromagnetic Weyl semimetal Co3Sn2S2,” Sci. China: Phys. Mech. Astron 64, 217062 (2021).
  * Wang  _et al._ (2021) Qi Wang, Kelly J. Neubauer, Chunruo Duan, Qiangwei Yin, Satoru Fujitsu, Hideo Hosono, Feng Ye, Rui Zhang, Songxue Chi, Kathryn Krycka, Hechang Lei, and Pengcheng Dai, “Field-induced topological Hall effect and double-fan spin structure with a $c$-axis component in the metallic kagome antiferromagnetic compound $\mathrm{Y}{\mathrm{Mn}}_{6}{\mathrm{Sn}}_{6}$,” Phys. Rev. B 103, 014416 (2021).
  * Kang  _et al._ (2020) Mingu Kang, Linda Ye, Shiang Fang, Jhih-Shih You, Abe Levitan, Minyong Han, Jorge I. Facio, Chris Jozwiak, Aaron Bostwick, Eli Rotenberg, Mun K. Chan, Ross D. McDonald, David Graf, Konstantine Kaznatcheev, Elio Vescovo, David C. Bell, Efthimios Kaxiras, Jeroen van den Brink, Manuel Richter, Madhav Prasad Ghimire, Joseph G. Checkelsky, and Riccardo Comin, “Dirac fermions and flat bands in the ideal kagome metal FeSn,” Nat. Mater. 19, 163 (2020).
  * Yin  _et al._ (2021) Jia-Xin Yin, Shuheng H Pan, and M Zahid Hasan, “Probing topological quantum matter with scanning tunnelling microscopy,” Nat. Rev. Phys. 3, 249–263 (2021).
  * Guo and Franz (2009) H.-M. Guo and M. Franz, “Topological insulator on the kagome lattice,” Phys. Rev. B 80, 113102 (2009).
  * Mazin  _et al._ (2014) I. I. Mazin, Harald O. Jeschke, Frank Lechermann, Hunpyo Lee, Mario Fink, Ronny Thomale, and Roser Valentí, “Theoretical prediction of a strongly correlated Dirac metal,” Nat. Commun. 5, 4161 (2014).
  * Moriya (1985) T. Moriya,  _Spin Fluctuations in Itinerant Electron Magnetism_ (Springer, Berlin, Heidelberg, 1985).
  * Borovik-Romanov and Sinha (1988) A. S. Borovik-Romanov and S. K. Sinha,  _Spin Waves and Magnetic Excitations_ (Elsevier, Amsterdam, 1988).
  * Melnikov and Reser (2018) N. B. Melnikov and B. I. Reser,  _Dynamic Spin Fluctuation Theory of Metallic Magnetism_ (Springer, New York, 2018).
  * Coey and Parkin (2021) J. M. D. Coey and Stuart S. P. Parkin,  _Handbook of Magnetism and Magnetic Materials_ (Springer, New York, 2021).
  * Ishikawa (1978) Y. Ishikawa, “Neutron scattering studies of itinerant electron magnetism,” J. Appl. Phys. 49, 2125–2130 (1978).
  * Endoh and B$\rm\ddot{o}$ni (2006) Yasuo Endoh and Peter B$\rm\ddot{o}$ni, “Magnetic excitations in metallic ferro- and antiferromagnets,” J. Phys. Soc. Japan 75, 111002 (2006).
  * Zhao  _et al._ (2009) Jun Zhao, DT Adroja, Dao-Xin Yao, R Bewley, Shiliang Li, XF Wang, G Wu, XH Chen, Jiangping Hu, and Pengcheng Dai, “Spin waves and magnetic exchange interactions in CaFe2As2,” Nat. Phys. 5, 555–560 (2009).
  * Dai (2015) Pengcheng Dai, “Antiferromagnetic order and spin dynamics in iron-based superconductors,” Rev. Mod. Phys. 87, 855–896 (2015).
  * Lee  _et al._ (2018) Jooseop Lee, Masaaki Matsuda, John A. Mydosh, Igor Zaliznyak, Alexander I. Kolesnikov, Stefan Süllow, Jacob P. C. Ruff, and Garrett E. Granroth, “Dual Nature of Magnetism in a Uranium Heavy-Fermion System,” Phys. Rev. Lett. 121, 057201 (2018).
  * Mook and Nicklow (1971) H. A Mook and R. M. Nicklow, “Spin waves and Stoner modes in iron,” J. Phys. Colloq. 32, C1–1177 (1971).
  * Mook  _et al._ (1974) H. A. Mook, J. W. Lynn, and R. M. Nicklow, “Magnetic Excitations in Nickel and Iron,” AIP Conference Proceedings 18, 781 (1974).
  * Chen  _et al._ (2020) Xiang Chen, Igor Krivenko, Matthew B Stone, Alexander I Kolesnikov, Thomas Wolf, Dmitry Reznik, Kevin S Bedell, Frank Lechermann, and Stephen D Wilson, “Unconventional Hund metal in a weak itinerant ferromagnet,” Nat. Commun. 11, 3076 (2020).
  * Lin  _et al._ (2020) Zhiyong Lin, Chongze Wang, Pengdong Wang, Seho Yi, Lin Li, Qiang Zhang, Yifan Wang, Zhongyi Wang, Hao Huang, Yan Sun, Yaobo Huang, Dawei Shen, Donglai Feng, Zhe Sun, Jun-Hyung Cho, Changgan Zeng, and Zhenyu Zhang, “Dirac fermions in antiferromagnetic FeSn kagome lattices with combined space inversion and time-reversal symmetry,” Phys. Rev. B 102, 155103 (2020).
  * Moore  _et al._ (2022) Robert G. Moore, Satoshi Okamoto, Haoxiang Li, William R. Meier, Hu Miao, Ho Nyung Lee, Makoto Hashimoto, Donghui Lu, Elbio Dagotto, Michael A. McGuire, and Brian C. Sales, “Topological electronic structure evolution with symmetry-breaking spin reorientation in $({\mathrm{Fe}}_{1-x}{\mathrm{Co}}_{x})\mathrm{Sn}$,” Phys. Rev. B 106, 115141 (2022).
  * Yamaguchi and Watanabe (1967) Kazuyuki Yamaguchi and Hiroshi Watanabe, “Neutron Diffraction Study of FeSn,” J. Phys. Soc. Japan 22, 1210–1213 (1967).
  * Häggström  _et al._ (1975) L. Häggström, T. Ericsson, R. Wäppling, and K. Chandra, “Studies of the magnetic structure of FeSn using the Mössbauer Effect,” Physica Scripta 11, 47–54 (1975).
  * Kakihana  _et al._ (2019) Masashi Kakihana, Kengo Nishimura, Dai Aoki, Ai Nakamura, Miho Nakashima, Yasushi Amako, Tetsuya Takeuchi, Takanori Kida, Time Tahara, Masayuki Hagiwara, Hisatomo Harima, Masato Hedo, Takao Nakama, and Yoshichika Onuki, “Electronic states of antiferromagnet FeSn and Pauli paramagnet CoSn,” J. Phys. Soc. Japan 88, 014705 (2019).
  * Sales  _et al._ (2019) Brian C. Sales, Jiaqiang Yan, William R. Meier, Andrew D. Christianson, Satoshi Okamoto, and Michael A. McGuire, “Electronic, magnetic, and thermodynamic properties of the kagome layer compound FeSn,” Phys. Rev. Materials 3, 114203 (2019).
  * Meier  _et al._ (2019) William R. Meier, Jiaqiang Yan, Michael A. McGuire, Xiaoping Wang, Andrew D. Christianson, and Brian C. Sales, “Reorientation of antiferromagnetism in cobalt doped FeSn,” Phys. Rev. B 100, 184421 (2019).
  * Sales  _et al._ (2021) B. C. Sales, W. R. Meier, A. F. May, J. Xing, J.-Q. Yan, S. Gao, Y. H. Liu, M. B. Stone, A. D. Christianson, Q. Zhang, and M. A. McGuire, “Tuning the flat bands of the kagome metal CoSn with Fe, In, or Ni doping,” Phys. Rev. Mater. 5, 044202 (2021).
  * Xie  _et al._ (2021) Yaofeng Xie, Lebing Chen, Tong Chen, Qi Wang, Qiangwei Yin, J. Ross Stewart, Erxi Feng, Huibo Cao, Hechang Lei, Zhiping Yin, Allan H. MacDonald, and Pengcheng Dai, “Spin excitations in metallic kagome lattice FeSn and CoSn,” Commun. Phys. 4, 240 (2021).
  * Do  _et al._ (2022) Seung-Hwan Do, Koji Kaneko, Ryoichi Kajimoto, Kazuya Kamazawa, Matthew B. Stone, Jiao Y. Y. Lin, Shinichi Itoh, Takatsugu Masuda, German D. Samolyuk, Elbio Dagotto, William R. Meier, Brian C. Sales, Hu Miao, and Andrew D. Christianson, “Damped Dirac magnon in the metallic kagome antiferromagnet FeSn,” Phys. Rev. B 105, L180403 (2022).
  * (39) See Supplemental Material for crystal growth and characterizations, data normalization and background subtraction of the neutron scattering results, and the details of the LSWT fittings.
  * Xu  _et al._ (2013) Guangyong Xu, Zhijun Xu, and J. M. Tranquada, “Absolute cross-section normalization of magnetic neutron scattering data,” Rev. Sci. Instrum. 84, 083906 (2013).
  * Shirane  _et al._ (2002) Gen Shirane, Stephen M Shapiro, and John M Tranquada,  _Neutron Scattering with a Triple-Axis Spectrometer: Basic Techniques_ (Cambridge University Press, Cambridge, 2002).
  * Abernathy  _et al._ (2012) D. L. Abernathy, M. B. Stone, M. J. Loguillo, M. S. Lucas, O. Delaire, X. Tang, J. Y. Y. Lin, and B. Fultz, “Design and operation of the wide angular-range chopper spectrometer ARCS at the Spallation Neutron Source,” Rev. Sci. Instrum. 83, 015114 (2012).
  * Granroth  _et al._ (2010) G. E. Granroth, A. I. Kolesnikov, T. E. Sherline, J. P. Clancy, K. A. Ross, J. P. C. Ruff, B. D. Gaulin, and S. E. Nagler, “SEQUOIA: a newly operating chopper spectrometer at the SNS,” J. Phys.: Conf. Ser. 251, 012058 (2010).
  * Ehlers  _et al._ (2011) G. Ehlers, A. Podlesnyak, J. L. Niedziela, E. B. Iverson, and P. E. Sokol, “The new cold neutron chopper spectrometer at the Spallation Neutron Source: Design and performance,” Rev. Sci. Instrum. 82, 085108 (2011).
  * Ehlers  _et al._ (2016) G. Ehlers, A. Podlesnyak, and A. I. Kolesnikov, “The cold neutron chopper spectrometer at the Spallation Neutron Source–A review of the first 8 years of operation,” Rev. Sci. Instrum. 87, 093902 (2016).
  * Arnold  _et al._ (2014) O. Arnold, J. C. Bilheux, J. M. Borreguero, A. Buts, S. I. Campbell, L. Chapon, M. Doucet, N. Draper, R. F. Leal, M. A. Gigg, _et al._ , “Mantid – Data analysis and visualization package for neutron scattering and $\mu$SR experiments,” Nucl. Instrum. Methods Phys. Res. Sect. A 764, 156 (2014).
  * Ewings  _et al._ (2016) R. A. Ewings, A. Buts, M. D. Le, J. van Duijn, I. Bustinduy, and T. G. Perring, “HORACE: software for the analysis of data from single crystal spectroscopy experiments at time-of-flight neutron instruments,” Nucl. Instrum. Methods Phys. Res. Sect. A 834, 3132–142 (2016).
  * Toth and Lake (2015) S. Toth and B. Lake, “Linear spin wave theory for single-Q incommensurate magnetic structures,” J. Phys. Condens. Matter 27, 166002 (2015).
  * Liechtenstein  _et al._ (1987) A.I. Liechtenstein, M.I. Katsnelson, V.P. Antropov, and V.A. Gubanov, “Local spin density functional approach to the theory of exchange interactions in ferromagnetic metals and alloys,” Journal of Magnetism and Magnetic Materials 67, 65–74 (1987).
  * van Schilfgaarde and Antropov (1999) M. van Schilfgaarde and V. P. Antropov, “First-principles exchange interactions in Fe, Ni, and Co,” Journal of Applied Physics 85, 4827–4829 (1999).
  * Frota-Pessôa  _et al._ (2000) S. Frota-Pessôa, R. B. Muniz, and J. Kudrnovský, “Exchange coupling in transition-metal ferromagnets,” Phys. Rev. B 62, 5293–5296 (2000).
  * Zhang  _et al._ (2022) Yi-Fan Zhang, Xiao-Sheng Ni, Trinanjan Datta, Meng Wang, Dao-Xin Yao, and Kun Cao, “$Ab~{}initio$ study of spin fluctuations in the itinerant kagome magnet FeSn,” Phys. Rev. B 106, 184422 (2022).
  * Yao  _et al._ (2018) Weiliang Yao, Chenyuan Li, Lichen Wang, Shangjie Xue, Yang Dan, Kazuki Iida, Kazuya Kamazawa, Kangkang Li, Chen Fang, and Yuan Li, “Topological spin excitations in a three-dimensional antiferromagnet,” Nat. Phys. 14, 1011–1015 (2018).
  * Bao  _et al._ (2018) Song Bao, Jinghui Wang, Wei Wang, Zhengwei Cai, Shichao Li, Zhen Ma, Di Wang, Kejing Ran, Zhao-Yang Dong, DL Abernathy, _et al._ , “Discovery of coexisting Dirac and triply degenerate magnons in a three-dimensional antiferromagnet,” Nat. Commun. 9, 2591 (2018).
  * Yuan  _et al._ (2020) Bo Yuan, Ilia Khait, Guo-Jiun Shu, F. C. Chou, M. B. Stone, J. P. Clancy, Arun Paramekanti, and Young-June Kim, “Dirac Magnons in a Honeycomb Lattice Quantum $\mathit{XY}$ Magnet ${\mathrm{CoTiO}}_{3}$,” Phys. Rev. X 10, 011062 (2020).
  * Elliot  _et al._ (2021) M Elliot, Paul A McClarty, D Prabhakaran, RD Johnson, HC Walker, P Manuel, and R Coldea, “Order-by-disorder from bond-dependent exchange and intensity signature of nodal quasiparticles in a honeycomb cobaltate,” Nat. Commun. 12, 3936 (2021).
  * McClarty (2022) Paul A. McClarty, “Topological magnons: A review,” Annual Review of Condensed Matter Physics 13, 171–190 (2022).


# Competition-based Adaptive ReLU
for Deep Neural Networks

Junjia Chen and Zhibin Pan The authors are with the School of Electronic and
Information Engineering, Xi’an Jiaotong University, Xi’an 710049, P. R. China.
(e-mail<EMAIL_ADDRESS>
###### Abstract

Activation functions introduce nonlinearity into deep neural networks. Most
popular activation functions allow positive values to pass through while
blocking or suppressing negative values. From the idea that positive values
and negative values are equally important, and they must compete for
activation, we proposed a new Competition-based Adaptive ReLU (CAReLU). CAReLU
scales the input values based on the competition results between positive
values and negative values. It defines two parameters to adjust the scaling
strategy and can be trained uniformly with other network parameters. We verify
the effectiveness of CAReLU on image classification, super-resolution, and
natural language processing tasks. In the experiment, our method performs
better than other widely used activation functions. In the case of replacing
ReLU in ResNet-18 with our proposed activation function, it improves the
classification accuracy on the CIFAR-100 dataset. The effectiveness and the
new perspective on the utilization of competition results between positive
values and negative values make CAReLU a promising activation function.

###### Index Terms:

Adaptive activation function, ReLU, deep learning, competition-based

##  I Introduction

Deep learning is one of the most popular techniques demonstrating superior
performance across multiple applications and has been the focus of academic
and engineering communities for years. Deep learning techniques are evolving
fast and each year hundreds of new models are proposed. Its applications
quickly dominate in many areas such as computer vision (CV)[1, 2] and natural
language processing (NLP)[3, 4].

$y=g(\bm{w}^{\rm T}\bm{x}+b).$ (1)

A deep neural network contains thousands of neural units. Every neural unit
gathers the input signals from other neurons and generates a signal for other
neurons to process. The structure of a deep neural network is a directed
acyclic graph where every vertex represents a mapping from signals of incoming
arcs into new signals of outgoing arcs. Eq. (1) is one of the most common
vertices in deep neural networks, which applies an affine transformation and a
non-linear mapping $g(\cdot)$ to the input tensor, where
$\bm{x}\in\mathbb{R}^{d}$ is the $d$-dimension input gathered from incoming
arcs, $\bm{w}\in\mathbb{R}^{d}$ is the weight vector, $b\in\mathbb{R}$ is the
bias, and $y\in\mathbb{R}$ is the vertex’s output. $\bm{w}$ and $b$ can be
obtained through training. This non-linear mapping is called the activation
function, without which a deep neural network degenerates into a linear
regression model.

${\rm ReLU}(\bm{w}^{\rm T}\bm{x}+b)={\rm max}(\bm{w}^{\rm T}\bm{x}+b,0).$ (2)

In the early stage of deep learning research, S-shape functions like the
Sigmoid function and the hyperbolic tangent function (tanh) were proposed and
became popular but they were later found ineffective. Rectified Linear Unit
(ReLU)[5, 6] shown in Eq. (2) was proposed that demonstrates top-tier
convergence speed, better generalization and ease of implementation. But ReLU
completely shuts down negative values, which led to the development of
LeakyReLU[7] that allows the negative values to pass through. Parametric ReLU
(PReLU)[8] improves model fitting by parameterizing the LeakyReLU’s slope of
the negative part. Multi-phase ReLU[9] utilizes six different phases from the
input. Sparse regularization[10] increases the sparsity of ReLU’s input.

Activation functions with more complicated formulas are also explored. Dan
proposed Gaussian Error Linear Unit (GeLU) and Sigmoid Linear Unit (SiLU)[11],
which are widely used in natural language processing. SiLU was also found and
named Swish[12] by Prajit by utilizing meta-learning techniques. Inspired by
SiLU’s self-gating property, Mish[13] and Serf[14] are proposed to further
improve performance. Brosnan developed the Universal Activation Function (UAV)
[15], which generalizes some popular activation functions by using five
trainable parameters. Shui-Long proposed the tanhLU[16] by integrating tanh
into a linear unit. Mathew proposed the Growing Cosine Unit (GCU)[17] that can
improve gradient flow and reduce network size.

Most activation functions differently process the negative part and the
positive part of the input tensor. For example, ReLU completely blocks
negative values; LeakyReLU, PReLU, and SiLU scale down negative values. It
implies that positive values are far more important than negative values. From
the viewpoint of signal processing, positive and negative values are equally
important and they must compete for activation instead of simply shutting down
or suppressing negative values.

In this paper, we propose a new Competition-based Adaptive ReLU (CAReLU). 1) A
CAReLU utilizes the competition results between positive values and negative
values in the input tensor. 2) A CAReLU has two parameters that can be trained
uniformly with other network parameters. 3) Different competition indicators
can cooperate with CAReLU to generate different activation functions for
different tasks. We evaluate our method and find that our method constantly
performs better compared to the most popular activation functions.

##  II Proposed Method

###  II-A Construction of Competition-based Adaptive ReLU

Our proposed method stems from the idea that both positive values and negative
values are equally important and they must compete for activation. We choose
energy as an indicator of the competition. The rule is that the competitor
with higher energy wins the qualification for activation. Let
$\bm{z}=(z_{1},z_{2},...,z_{d})\in\mathbb{R}^{d}$ be the output of the
previous affine transformation. The percentage of positive values’ energy
$p_{E}$ is defined as follows:

$\displaystyle p_{E}$ $\displaystyle=\frac{\sum_{j=1}^{d}[{\rm
max}(z_{j},0)]^{2}}{\|\bm{z}\|^{2}+\epsilon},$ (3)

where a small positive constant value $\epsilon$ is added in the denominator
to prevent the division by zero. We define $s$ as follows:

$\displaystyle s$ $\displaystyle={\rm sgn}(2p_{E}-1),$ (4)

where ${\rm sgn}(\cdot)$ is the sign function. If $p_{E}<0.5$ is true, then we
have $s=-1$, which means the negative values win the competition; If
$p_{E}>0.5$ is true, then we have $s=1$, which means the positive values win
the competition. By multiplying $\bm{z}$ with $s$ when passing it to the ReLU,
we can flip the input if negative values have higher energy:

$y_{i}={\rm ReLU}(sz_{i}).$ (5)

However, this vanilla version of our idea does not perform well. First, there
is no degeneracy into regular ReLU when prioritizing higher energy impairs
performance. Second, the sign function creates discontinuities in the model’s
parameter space, which harms the gradient-based optimization. To address the
first issue, we replace the fixed scaling factor of $2$ and the bias of $-1$
in Eq. (4) with trainable parameters $\alpha$ and $\beta$ per layer:

$\displaystyle s$ $\displaystyle={\rm sgn}(\alpha p_{E}+\beta).$ (6)

If $\alpha$ is close or equal to $0$ after training, this mapping degenerates
into ReLU with a factor of $1$ or $-1$. Only 2 extra parameters are introduced
per layer, which is negligible when considering the total number of weights.

To address the second issue, we replace the sign function with the tanh
function as follows:

$\displaystyle s$ $\displaystyle={\tanh}(\alpha p_{E}+\beta).$ (7)

The tanh function has a codomain ranging from $-1$ to $1$ and it is a
continuous function with a non-zero gradient, which can be viewed as a smooth
version of the sign function.

There are other kinds of indicators for competition between positive values
and negative values other than energy. The other two indicators we experiment
with are:

$\begin{aligned} p_{L1}&=\frac{\sum_{j=1}^{d}|{\rm
max}(z_{j},0)|}{\|\bm{z}\|_{1}+\epsilon}\\\ \end{aligned},$ (8)
$\begin{aligned} p_{c}&=\frac{\sum_{j=1}^{d}H(z_{j})}{d}\\\ \end{aligned},$
(9)

where $H(\cdot)$ is the Heaviside step function, Eq. (8) is the percentage of
positive values’ L1-norm, and Eq. (9) is the percentage of the number of
positive values.

Competition-based Adaptive Scaling (CAS) can be defined as follows to
adaptively scale the input $\bm{z}$ based on competition results:

${\rm CAS}(\bm{z})\triangleq K[{\tanh}(\alpha p+\beta)]\bm{z},\\\ $ (10)

where $K$ is a constant and $p\in\\{p_{E},p_{L1},p_{c}\\}$. Since
$|\tanh(\alpha p+\beta)|$ is less than 1, the magnitude of feature vectors
will shrink layer by layer in a sequence model. To combat this phenomenon, the
constant $K$ is placed to scale up the results of the tanh function. By
concatenating CAS and ReLU, we construct Competition-based Adaptive ReLU as
follows:

${\rm CAReLU}(\bm{z})\triangleq{\rm ReLU}({\rm CAS}(\bm{z})).$ (11)

###  II-B Working with Batch Normalization

Batch Normalization (BN) [18] is widely used in convolutional neural networks.
BN normalizes and rescales the input with 2 parameters on a mini-batch before
activation. Since CAReLU requires competition between positive values and
negative values, the BN’s normalization might impair this process. To improve
our method’s compatibility with BN, we define $\rm BN\text{-}CAReLU$ as
follows by placing Eq. (10) before BN so that the neural network can obtain
competition results before the normalization:

${\rm BN\text{-}CAReLU}(\bm{z})\triangleq{\rm ReLU}({\rm BN}({\rm
CAS}(\bm{z}))).$ (12)

###  II-C Gradients of the Competition-based Adaptive Scaling

Let $\alpha_{0}$ be the initial value of $\alpha$ and $\beta_{0}$ be the
initial value of $\beta$. Instead of running a grid search for initial values,
we simply set these values as follows:

$\begin{dcases}\alpha_{0}&=0,\\\ \beta_{0}&=1,\\\ K&=1/\tanh(\beta_{0})\approx
1.313.\end{dcases}$ (13)

where $\alpha_{0}=0$ means that deep neural networks don’t utilize competition
results at the beginning. $\beta_{0}=1$ makes the tanh function starts at the
non-saturating area, where the gradient is large enough to initiate the
update. The constant $K$ is set to $1/\tanh(\beta_{0})$ so that the CAS is
initialized as an identity mapping when the training begins.

To update the parameters, we need to compute gradients of the loss function
with respect to $\alpha$, $\beta$, and input value $z_{i}$. Let
$\hat{\bm{z}}=({\hat{z}}_{1},{\hat{z}}_{2},...,{\hat{z}}_{d})\in\mathbb{R}^{d}$
be the output of CAS as follows:

$\hat{\bm{z}}={\rm CAS}(\bm{z}).$ (14)

Gradients can be derived from the chain rule as follows:

$\displaystyle\frac{\partial\mathcal{L}}{\partial\alpha}=$ $\displaystyle
K\frac{4p}{({\rm e}^{\alpha p+\beta}+{\rm e}^{-\alpha
p-\beta})^{2}}\sum_{j=1}^{d}\frac{\partial\mathcal{L}}{\partial\hat{z}_{j}}z_{j},$
(15) $\displaystyle\frac{\partial\mathcal{L}}{\partial\beta}=$ $\displaystyle
K\frac{4}{({\rm e}^{\alpha p+\beta}+{\rm e}^{-\alpha
p-\beta})^{2}}\sum_{j=1}^{d}\frac{\partial\mathcal{L}}{\partial\hat{z}_{j}}z_{j},$
$\displaystyle\frac{\partial\mathcal{L}}{\partial z_{i}}=$ $\displaystyle
K[\frac{4\alpha}{({\rm e}^{\alpha p+\beta}+{\rm e}^{-\alpha
p-\beta})^{2}}\frac{\partial p}{\partial
z_{i}}(\sum_{j=1}^{d}\frac{\partial\mathcal{L}}{\partial\hat{z}_{j}}z_{j})$
$\displaystyle+\frac{\partial\mathcal{L}}{\partial\hat{z}_{i}}\tanh(\alpha
p+\beta)],$

where $\mathcal{L}$ is the loss function and
$\frac{\partial\mathcal{L}}{\partial\hat{z}_{i}}$ is obtained from
backpropagation. $\frac{\partial p}{\partial z_{i}}$ is the partial
differential of the competition indicator $p$ with respect to the input. For
$p_{E}$, $p_{L1}$ and $p_{c}$, we ignore the small positive constant value
$\epsilon$ and the gradients are derived as follows:

$\displaystyle\frac{\partial p_{E}}{\partial z_{i}}$
$\displaystyle=\frac{2{\rm
max}(z_{i},0)\|\bm{z}\|^{2}-2z_{i}\sum_{j=1}^{d}[{\rm
max}(z_{j},0)]^{2}}{(\sum_{j=1}^{d}z_{j}^{2})^{2}},$ (16)
$\displaystyle\frac{\partial p_{L1}}{\partial z_{i}}$
$\displaystyle=\begin{dcases}\frac{\sum_{j=1}^{d}{\rm
max}(z_{j},0)}{\|\bm{z}\|_{1}^{2}},&z_{i}<0\\\
\frac{\|\bm{z}\|_{1}-\sum_{j=1}^{d}{\rm
max}(z_{j},0)}{\|\bm{z}\|_{1}^{2}},&z_{i}>0\end{dcases},$
$\displaystyle\frac{\partial p_{c}}{\partial z_{i}}$ $\displaystyle=0.$

Thus, we obtained all gradients of CAS for gradient descent and
backpropagation. Since Eq. (11) and Eq. (12) are just different concatenations
of CAS, ReLU, and BN, which are all differentiable with respect to the input
and parameters, the gradients can be derived from the chain rule.

##  III Experiments

In this section, we evaluate CAReLU across different applications. For models
without BN, we directly replace existing activation functions with Eq. (11) in
every layer. For models with BN, we also replace $\rm BN\text{-}ReLU$
combination with Eq. (12) in every layer. We respectively use $\rm
CAReLU_{E}$, $\rm CAReLU_{L1}$, and $\rm CAReLU_{c}$ to denote CAReLU
implemented with $p_{E}$ in Eq. (3), $p_{L1}$ in Eq. (8), and $p_{c}$ in Eq.
(9). The following experiments show that our method outperforms other popular
activation functions in multiple applications.

###  III-A CIFAR-100 Image Classification

We compare our methods to the most popular activation functions on CIFAR-100
imagine classification task [19]. CIFAR-100 is a dataset that has 100 classes,
containing 500 training images and 100 test images for each class. We use
ResNet-18 [20], GoogLeNet [21], and VGG-13 [22] networks to evaluate our
activation functions. A stochastic gradient descent (SGD) optimizer with a
momentum of $0.9$ and a weight decay of $5\times 10^{-4}$ is used to train all
networks. The learning rate starts at $0.1$ and is divided by $5$ in $50$th,
$120$th, and $160$th epochs. The training ends at $200$th epoch.

TABLE I: Top-1 Accuracy (%) on CIFAR-100 Test Set Methods | ResNet-18 | GoogLeNet | VGG-13
---|---|---|---
${\rm ReLU}$ | $76.14\pm 0.21$ | $78.53\pm 0.19$ | $72.53\pm 0.20$
${\rm LeakyReLU}$ | $76.18\pm 0.14$ | $78.43\pm 0.08$ | $72.27\pm 0.11$
${\rm PReLU}$ | $74.26\pm 0.14$ | $76.14\pm 0.37$ | $71.02\pm 0.15$
${\rm Swish\text{-}1}$ | $75.74\pm 0.15$ | $75.74\pm 0.08$ | $71.36\pm 0.07$
${\rm Swish}$ | $76.30\pm 0.25$ | $77.31\pm 0.24$ | $72.10\pm 0.25$
${\rm BN\text{-}CAReLU_{E}}$ | $76.50\pm 0.20$ | $\bm{79.23\pm 0.13}$ | $\bm{72.85\pm 0.19}$
${\rm CAReLU_{E}}$ | $\bm{76.62\pm 0.23}$ | $79.21\pm 0.15$ | $72.62\pm 0.14$
${\rm BN\text{-}CAReLU_{L1}}$ | $76.44\pm 0.15$ | $78.94\pm 0.18$ | $72.47\pm 0.32$
${\rm CAReLU_{L1}}$ | $76.43\pm 0.15$ | $78.89\pm 0.22$ | $72.58\pm 0.36$
${\rm BN\text{-}CAReLU_{c}}$ | $76.29\pm 0.28$ | $78.60\pm 0.28$ | $72.46\pm 0.23$
${\rm CAReLU_{c}}$ | $76.13\pm 0.21$ | $78.65\pm 0.11$ | $72.24\pm 0.26$

Table I shows the results of the classification experiment. Most
implementations of our method have overall better performance compared to
other baseline methods. ${\rm CAReLU_{E}}$ has the highest accuracy in
ResNet-18 experiments and ${\rm BN\text{-}CAReLU_{E}}$ has the highest
accuracy in GoogLeNet and VGG-13 experiments. When comparing different
implementations of our method, ${\rm BN\text{-}CAReLU}$ performs better than
${\rm CAReLU}$ except for the ${\rm CAReLU_{E}}$ in ResNet-18 experiments and
${\rm CAReLU_{c}}$ in GoogLeNet, which proves the previous assumption that
BN’s normalization impairs the effectiveness of our method. Energy-based
implementations generally perform better than the other two implementations.
The reason is that the gradient of ${\rm CAReLU_{E}}$ is a continuous function
and it’s smoother than the gradient of ${\rm CAReLU_{L1}}$; ${\rm CAReLU_{c}}$
contains Heaviside step functions, which results in a bumpy loss landscape
[23] and thus impairs the performance.

TABLE II: Values of $\tanh(\alpha p+\beta)$ from the First 10 CAReLUs of
Best Trained Models $\rm CAS$ | ResNet-18 | GoogLeNet | VGG-13
---|---|---|---
#1 | $0.5418\pm 0.0172$ | $0.1775\pm 0.0007$ | $0.1861\pm 0.0028$
#2 | $0.1915\pm 0.0113$ | $0.2028\pm 0.0169$ | $0.1447\pm 0.0078$
#3 | $0.4540\pm 0.0218$ | $0.1449\pm 0.0049$ | $0.2670\pm 0.0265$
#4 | $0.0678\pm 0.0065$ | $0.1591\pm 0.0079$ | $0.1489\pm 0.0077$
#5 | $0.2266\pm 0.0060$ | $0.1558\pm 0.0250$ | $0.1388\pm 0.0045$
#6 | $0.1327\pm 0.0100$ | $0.5597\pm 0.0305$ | $0.9923\pm 0.0000$
#7 | $0.5954\pm 0.0067$ | $0.3055\pm 0.0020$ | $0.0962\pm 0.0085$
#8 | $0.1049\pm 0.0046$ | $0.2143\pm 0.0399$ | $0.4725\pm 0.0060$
#9 | $0.1795\pm 0.0033$ | $0.1317\pm 0.0080$ | $0.1553\pm 0.0245$
#10 | $0.1673\pm 0.0149$ | $0.4171\pm 0.0008$ | $0.2692\pm 0.0849$

Figure 1: Histograms of $\alpha p+\beta$ obtained from the best trained model.
(a) ResNet-18/${\rm CAReLU_{E}}$. (b) GoogLeNet/${\rm BN\text{-}CAReLU_{E}}$.
(c) VGG-13/${\rm BN\text{-}CAReLU_{E}}$.

We also evaluate values of $\tanh(\alpha p+\beta)$ in CASs on each layer. In
Table II, we show the means and standard deviations of $\tanh(\alpha p+\beta)$
from the first 10 CASs of best models on the test set. Small deviations
indicate that CASs on trained models do not generate significantly different
values of $\tanh(\alpha p+\beta)$ for different samples. Instead, input values
are scaled uniformly and then fine-tuned for each sample according to the
input’s competition results. Some degenerate into constant scaling such as
CAS#6 of VGG-13 in Table II. Allowing degeneracy into ReLU enables our method
to utilize competition results without compromising the performance whenever
the original approach is optimal. Histograms of $\alpha p+\beta$ obtained from
the best models on the test set are shown in Fig. 1. Despite our initial
design being a binary scale factor described in Eq. (4), values of $\alpha
p+\beta$ mostly land on non-saturating regions.

###  III-B BSD-300 Image Super Resolution

We compare our methods to the most popular activation functions on an image
super-resolution task on the Berkeley segmentation dataset (BSD300), which
contains 200 training images and 100 test images [24]. The network we used for
this experiment is an efficient sub-pixel convolutional neural network (ESPCN)
featuring a network comprising several convolutional layers and a pixel
shuffle layer [25]. Training images are cropped to $256\times 256$ and scaled
down to $(256/r)\times(256/r)$ where $r$ is the upscale factor. The ESPCN
network upscales the down-scaled images back to $256\times 256$. An Adam
optimizer [26] with a learning rate of $1\times 10^{-3}$ is used for training
the network in 200 epochs. We run every setting 5 times and show experiment
results in Table III.

The data show that ${\rm CAReLU_{E}}$ and ${\rm CAReLU_{L1}}$ surpass other
activation functions in PSNR. ${\rm CAReLU_{c}}$ outperforms PReLU, Swish-1,
and Swish, but does not perform as well as ${\rm CAReLU_{E}}$ and ${\rm
CAReLU_{L1}}$ due to its discontinuity introduced by the Heaviside function.

TABLE III: PSNR (dB) on BSD300 Super-resolution Task Methods | $r=3$ | $r=4$
---|---|---
${\rm ReLU}$ | $25.001\pm 0.014$ | $23.634\pm 0.008$
${\rm LeakyReLU}$ | $25.005\pm 0.008$ | $23.635\pm 0.010$
${\rm PReLU}$ | $24.980\pm 0.015$ | $23.635\pm 0.014$
${\rm Swish\text{-}1}$ | $24.934\pm 0.010$ | $23.585\pm 0.010$
${\rm Swish}$ | $24.947\pm 0.013$ | $23.590\pm 0.011$
${\rm CAReLU_{E}}$ | $\bm{25.018\pm 0.006}$ | $23.642\pm 0.008$
${\rm CAReLU_{L1}}$ | $25.015\pm 0.003$ | $\bm{23.643\pm 0.009}$
${\rm CAReLU_{c}}$ | $25.000\pm 0.012$ | $23.631\pm 0.013$

###  III-C Natural Language Inference on SNLI

In this section, we evaluate our methods on the Stanford Natural Language
Inference (SNLI) corpus [27]. SNLI corpus is a collection of 570k human-
written English sentence pairs labeled for entailment, contradiction, and
neutral. A premise and a hypothesis comprise a sentence pair. The model we
used for this task comprises an embedding layer, a Long short-term memory
(LSTM) [28] encoder, and a sequence of fully connected layers. Hypothesis and
premise go through the embedding layer and the encoder independently. We
concatenate the encoded hypothesis feature and premise feature, then send them
through the fully connected layers for classification. We use the Adam
optimizer with a learning rate of $0.001$ to train the parameters for 50
epochs on the SNLI training set. We run every setting 5 times and show
experiment results in Table IV.

[ht]

TABLE IV: Classification Accuracy (%) on SNLI Test Set

Methods |  Acc
---|---
${\rm ReLU}$ |  $77.69\pm 0.25$
${\rm LeakyReLU}$ |  $77.12\pm 0.48$
${\rm PReLU}$ |  $77.25\pm 0.33$
${\rm Swish\text{-}1}$ |  $-$1
${\rm Swish}$ |  $-$
${\rm CAReLU_{E}}$ |  $\bm{78.67\pm 0.25}$
${\rm CAReLU_{L1}}$ |  $78.41\pm 0.39$
${\rm CAReLU_{c}}$ |  $78.40\pm 0.25$

  * 1

”$-$” indicates that the training with this activation function does not
converge.

Data suggest that our methods perform better than other activation functions.
Swish and Swish-1 are unable to converge in this task. ${\rm CAReLU_{E}}$
achieve the highest classification accuracy in this experiment. Though mean
accuracies of ${\rm CAReLU_{L1}}$ and ${\rm CAReLU_{c}}$ are approximately
identical, ${\rm CAReLU_{c}}$ is more robust because it has a smaller standard
deviation. All three implementations of our method outperform other baseline
activation functions.

##  IV Conclusion

Stemming from the idea that both positive values and negative values are
equally important and they must compete for activation, we developed the
Competition-based Adaptive ReLU (CAReLU) activation function by introducing
the competition between positive values and negative values. A CAReLU has 2
parameters that can be trained uniformly with other network parameters using
gradient descent. By respectively implementing CAReLU with each one of 3
competition indicators, we developed 3 different activation functions: $\rm
CAReLU_{E}$, $\rm CAReLU_{L1}$, and $\rm CAReLU_{c}$. We also developed a
technique when working with Batch Normalization to have extra performance
gain.

We evaluated our method on different tasks and achieved consistent performance
improvements. $\rm CAReLU_{E}$ is generally more effective, but $\rm
CAReLU_{L1}$ and $\rm CAReLU_{c}$ can also achieve great performance in
certain scenarios. The effectiveness and the new perspective on the
competition between positive values and negative values make CAReLU promising
in deep learning tasks.

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Commun. ACM_ , vol. 60, no. 6, p. 84–90, may 2017.
  * [2] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017\.
  * [4] D. W. Otter, J. R. Medina, and J. K. Kalita, “A survey of the usages of deep learning for natural language processing,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 2, pp. 604–624, 2021.
  * [5] K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, “What is the best multi-stage architecture for object recognition?” in _2009 IEEE 12th International Conference on Computer Vision_ , 2009, pp. 2146–2153.
  * [6] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Proceedings of the 27th International Conference on Machine Learning (ICML-10)_ , 2010, pp. 807–814.
  * [7] A. L. Maas, A. Y. Hannun, A. Y. Ng _et al._ , “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. ICML_ , vol. 30, no. 1. Atlanta, Georgia, USA, 2013, p. 3.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _2015 IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 1026–1034.
  * [9] C. Banerjee, T. Mukherjee, and E. Pasiliao Jr, “The multi-phase relu activation function,” in _Proceedings of the 2020 ACM Southeast Conference_ , 2020, pp. 239–242.
  * [10] H. Ide and T. Kurita, “Improvement of learning for CNN with ReLU activation by sparse regularization,” in _2017 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2017, pp. 2684–2691.
  * [11] D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [12] P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for activation functions,” _arXiv preprint arXiv:1710.05941_ , 2017.
  * [13] D. Misra, “Mish: A self regularized non-monotonic neural activation function,” _arXiv preprint arXiv:1908.08681_ , vol. 4, no. 2, pp. 10–48 550, 2019.
  * [14] S. Nag, M. Bhattacharyya, A. Mukherjee, and R. Kundu, “Serf: Towards better training of deep neural networks using log-softplus error activation function,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 5324–5333.
  * [15] B. Yuen, M. T. Hoang, X. Dong, and T. Lu, “Universal activation function for machine learning,” _Scientific Reports_ , vol. 11, no. 1, pp. 1–11, 2021\.
  * [16] S.-L. Shen, N. Zhang, A. Zhou, and Z.-Y. Yin, “Enhancement of neural networks with an alternative activation function tanhlu,” _Expert Syst. Appl._ , vol. 199, no. C, aug 2022.
  * [17] M. M. Noel, A. Trivedi, P. Dutta _et al._ , “Growing cosine unit: A novel oscillatory activation function that can speedup training and reduce parameters in convolutional neural networks,” _arXiv preprint arXiv:2108.12943_ , 2021.
  * [18] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _ICML_ , 2015.
  * [19] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” Univ. of Toronto, Tech. Rep., 2009.
  * [20] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [21] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [22] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [23] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018.
  * [24] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , vol. 2, 2001, pp. 416–423 vol.2.
  * [25] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 1874–1883.
  * [26] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [27] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, 2015.
  * [28] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.