# Constraining spinning primordial black holes with global 21-cm signal

Pravin Kumar Natwariya ${}^{\href https://orcid.org/0000-0001-9072-8430}$
<EMAIL_ADDRESS><EMAIL_ADDRESS>Physical Research Laboratory, Theoretical
Physics Division, Ahmedabad, Gujarat 380 009, India  Department of Physics,
Indian Institute of Technology, Gandhinagar, Palaj, Gujarat 382 355, India
Alekha C. Nayak ${}^{\href https://orcid.org/0000-0001-6087-2490}$
<EMAIL_ADDRESS>National Institute of Technology, Meghalaya, Shillong,
Meghalaya 793 003, India  Tripurari Srivastava ${}^{\href
https://orcid.org/0000-0001-6856-9517}$<EMAIL_ADDRESS>Department
of Physics and Astrophysics, University of Delhi, Delhi 110007, India

###### Abstract

Abstract

We study the upper projected bounds on the dark matter fraction in the form of
the primordial black holes (PBHs) with a non-zero spin by using the absorption
feature in the global 21-cm signal at redshift $z\approx 17$. The mass and
spin are fundamental properties of a black hole, and they can substantially
affect the evaporation rate of the black hole. The evaporating black hole can
inject energy into the intergalactic medium and heat the gas. Subsequently, it
can modify the absorption amplitude in the global 21-cm signal. Therefore, the
absorption feature in the 21-cm signal can provide a robust bound on PBHs. We
analyse the projected constraints on the dark matter fraction in the form of
both spinning and non-spinning PBHs. The constraints are more stringent for
spinning PBHs than non-spinning ones. We also compare these bounds with other
observations and find the most stringent lower constraint on PBHs mass, which
is allowed to constitute the entire dark matter to $6.7\times 10^{17}$ g for
extremal spinning PBHs.

Primordial Black Hole, Dark Matter, 21-cm signal

##  I Introduction

About 85 percent of the total matter content in the Universe is dominated by
the dark matter (DM) Planck Collaboration VI (2020). In the last decade, many
DM models, such as collision-less cold DM Peebles (1982), fuzzy cold DM Hu
_et al._ (2000), warm DM Dodelson and Widrow (1994); Boyarsky  _et al._
(2019); Bulbul  _et al._ (2014), self-interacting DM Spergel and Steinhardt
(2000); Natwariya  _et al._ (2020), have been proposed to explain various
astrophysical observations. However, the microscopic nature of dark matter is
still unknown. One interesting, well-motivated proposal is the fraction/all of
DM in the form of PBHs (Carr and Kühnel (2020); Dasgupta  _et al._ (2020);
Frampton  _et al._ (2010); Khlopov (2010); Belotsky  _et al._ (2019) and
references therein). Recently, PBHs have been gathered much attention after
the black hole binary merger detection by LIGO and Virgo collaboration. These
events suggest that PBHs may constitute a fraction of DM Bird  _et al._
(2016); Abbott et al. (2016a, b, 2017a, 2017b); Sasaki  _et al._ (2016).

PBHs may have originated in the early Universe due to initial inhomogeneities
Zel’dovich and Novikov (1967); Hawking (1971); Carr and Hawking (1974); Carr
(1975), Higgs potential instability at a scale of $10^{11}$ GeV Espinosa  _et
al._ (2018), hybrid inflation Frampton  _et al._ (2010); Clesse and García-
Bellido (2015), etc. Depending on the origin time (t), PBHs can have a wide
range of masses $M_{\rm PBH}\sim 10^{15}\,\big{[}t/(10^{-23}\rm sec)\big{]}$ g
Carr  _et al._ (2010). PBHs having mass larger than $10^{15}$ g can survive
the Hawking evaporation and account for present-day DM density Hawking (1975).
The presence of PBHs can be responsible for the ultra-luminous X-ray sources,
seeds for supermassive black holes at the centre of galaxies Clesse and
García-Bellido (2015), and it may provide seeds for present-day observed
structures Clesse and García-Bellido (2015); García-Bellido (2017). There are
several hints that indicate the presence of PBHs, such as dynamics and star
clusters of ultra-faint-dwarf-galaxies, correlations between X-ray and
infrared cosmic backgrounds, etc. (for a detailed review, see Ref. Clesse and
García-Bellido (2018)). The presence of evaporating PBHs can explain the
galactic/extra-galactic $\gamma$-ray background radiation Wright (1996);
Lehoucq  _et al._ (2009); Carr (1976); Page and Hawking (1976), short-duration
$\gamma$-ray bursts Cline  _et al._ (1997); Green (2001), and reionization by
injection of $\gamma$ and $e^{\pm}$ radiations into Inter-Galactic-Medium
(IGM) Belotsky  _et al._ (2014); Belotsky and Kirillov (2015).

During the cosmic dawn era, the evolution of the gas temperature and
ionization fraction of the Universe are well-known Seager  _et al._ (1999,
2000). The addition of any exotic source of energy during the cosmic dawn era
can significantly impact the ionization and thermal history of the Universe.
Therefore, we can constrain the properties of such exotic sources from the
observations during the cosmic dawn era. Evaporating PBHs can heat the gas and
modify the free electron fraction in the IGM Laha  _et al._ (2021); Kim
(2021). Rotating PBHs can emit more particles into IGM and substantially
affect the IGM evolution compared to non-rotating PBHs Chandrasekhar and
Detweiler (1977); Page and Hawking (1976); Page (1976a). Therefore, it is
important to study the properties of spinning PBHs. In the present work, we
consider the Hawking emission of PBHs into background radiations (photons and
electron/positron) and provide the projected constraints on the fraction of DM
in the form of PBHs, $f_{\rm PBH}=\Omega_{\rm PBH}/\Omega_{\rm DM}$, as a
function of mass and spin. Here, $\Omega_{\rm PBH}$ and $\Omega_{\rm DM}$ are
the dimensionless density parameters for PBHs and DM.

Recently, EDGES observation detected a large absorption signal in the 21-cm
line in the redshift range $15-20$ Bowman  _et al._ (2018); Pritchard and Loeb
(2012). The 21-cm signal appears to be a treasure trove to provide constraints
on various cosmological phenomena such as the formation of the first stars,
galaxies or any exotic source of energy injection. The 21 cm line corresponds
to the wavelength of the hyperfine transition between the $1S$ singlet and
triplet states of the neutral hydrogen atom. The EDGES absorption signal is
nearly two times larger than the theoretical prediction based on the standard
model of cosmology ($\Lambda$CDM) at the redshift $z\approx 17$ Bowman  _et
al._ (2018); Pritchard and Loeb (2012). In the $\Lambda$CDM model, the gas and
cosmic-microwave-background (CMB) temperatures vary adiabatically as $T_{\rm
gas}\propto(1+z)^{2}$ and $T_{\rm CMB}\propto(1+z)$ during the cosmic dawn
era. Subsequently, at $z=17$, one gets the gas and CMB temperatures to be
$\sim 6.7$ K and $\sim 49.1$ K, respectively, and it implies
$T_{21}\approx-220$ mK Seager  _et al._ (1999, 2000). While EDGES
collaboration reported $T_{21}=-0.5_{-0.5}^{+0.2}$ K with 99% confidence
intervals at a centre frequency of $78\pm 1$ MHz or $z\simeq 17$ Bowman  _et
al._ (2018). To resolve this discrepancy, either one has to increase the
background radio radiation or decrease the gas temperature. Both the scenarios
have been explored in several literatures Ewall-Wice  _et al._ (2018); Jana
_et al._ (2018); Feng and Holder (2018); Lawson and Zhitnitsky (2019);
Natwariya (2021); Lawson and Zhitnitsky (2013); Levkov  _et al._ (2020);
Natwariya and Bhatt (2020); Brandenberger  _et al._ (2019); Chianese  _et al._
(2019); Bhatt  _et al._ (2020); Tashiro  _et al._ (2014); Barkana (2018);
Sikivie (2019); Mirocha and Furlanetto (2019); Ghara and Mellema (2019).
However, increasing the background radio radiation or cooling the IGM gas by
the non-standard mechanisms are not well known and are debatable issues Muñoz
and Loeb (2018); Sean Fraser et al. (2018); Bransden  _et al._ (1958); Barkana
_et al._ (2018); Berlin  _et al._ (2018); Kovetz  _et al._ (2018); Muñoz  _et
al._ (2018); Slatyer and Wu (2018); D’Amico  _et al._ (2018); Mitridate and
Podo (2018). Therefore, we have not considered any methods to increase the
background radiation above CMB or cooling of the IGM gas. In this work, we
study projected bounds on spinning PBHs such that 21-cm differential
brightness temperature does not change more than a factor of 1/4 from
$\Lambda$CDM framework based theoretical prediction.

The fraction of DM in the form of PBHs is constrained from various
astrophysical observations and theoretical predictions. PBHs with mass smaller
than $\sim\mathcal{O}(10^{15}~{}{\rm g})$ may have evaporated as of now and
can be constrained from the impact on big bang nucleosynthesis by evaporated
particles, background radiation etc. Higher mass PBHs can be constrained by
the effect on large-scale structures, gravitational wave and lensing, and
impact on thermal and ionization history of the IGM (for details, see the
recent reviews Carr  _et al._ (2021); Green and Kavanagh (2021); Carr and
Kühnel (2020) and references therein). In the context of the 21-cm signal, the
upper bound on the $f_{\rm PBH}$ can be found in Refs. Hektor  _et al._
(2018); Clark  _et al._ (2018); Mena  _et al._ (2019); Yang (2020a); Halder
and Banerjee (2021); Tashiro and Kadota (2021); Yang (2020b); Villanueva-
Domingo and Ichiki (2021). Angular momentum is a fundamental property of a
black hole, and it can modify the Hawking evaporation drastically. In the case
of rotating PBHs, authors of the Refs. Dasgupta  _et al._ (2020); Laha  _et
al._ (2021) have reported the various types of bound on $f_{\rm PBH}$ as a
function of PBHs mass and spin. Future collaboration, All-sky Medium Energy
Gamma-ray Observatory (AMEGO)111https://asd.gsfc.nasa.gov/amego/index.html
will be able to constrain some parameter space for the rotating PBHs Ray  _et
al._ (2021).

##  II Thermal History Of IGM

A rotating black hole with angular momentum $J_{\rm PBH}$ and having mass
$M_{\rm PBH}$ can be defined with a rotation parameter, $a_{*}=J_{\rm
PBH}/(G\,M_{\rm PBH}^{2})$ Page (1976a), where $G$ is the gravitational
constant. Black holes can get their spin depending on generation mechanisms,
merger or accretion Kesden  _et al._ (2010); Cotner and Kusenko (2017); Harada
_et al._ (2021); Luca  _et al._ (2019, 2020); Harada  _et al._ (2017); Kühnel
(2020); Flores and Kusenko (2021); Arbey  _et al._ (2020a); He and Suyama
(2019); Cotner  _et al._ (2019). PBHs with higher mass can have a lifetime
larger/comparable than the age of the Universe. Therefore, they have enough
time to accrete mass and spin up Dong  _et al._ (2016). Rotating black hole
with higher spin ($a_{*}\rightarrow 1$) injects more energy into IGM and
evaporates faster than non-rotating BHs Chandrasekhar and Detweiler (1977);
Taylor  _et al._ (1998); Arbey  _et al._ (2020b, 2021). Therefore, we expect
that the bounds on $f_{\rm PBH}$ to be more stringent compared to non-rotating
PBHs. The energy injection per unit volume per unit time due to $e^{\pm}$ and
photons into IGM, for monochromatic mass distribution of PBHs, can be written
as Laha  _et al._ (2021); Mittal  _et al._ (2021),

$\displaystyle\Gamma_{\rm PBH}^{e^{\pm}}(z,a_{*})$
$\displaystyle=2\int\left[f_{c}^{e}(E-m_{e},z)\,(E-m_{e})\left(\frac{d^{2}N_{e}}{dt\,dE}\right)\,\right]\,n_{\rm
PBH}\,dE\,,$ (1) $\displaystyle\Gamma_{\rm PBH}^{\gamma}(z,a_{*})$
$\displaystyle=\int\left[\
f_{c}^{\gamma}(E,z)\,E\,\left(\frac{d^{2}N_{\gamma}}{dt\,dE}\right)\
\right]\,n_{\rm PBH}\ dE\,.$ (2)

Energy injection into IGM happens by three processes: heating, ionization, and
excitation of the gas Slatyer (2016a, b); Liu  _et al._ (2020). $f_{c}^{i}$
represents the energy deposition efficiency into IGM. Here, $c$ stands for
above-mentioned three channels and $i\equiv({\rm electron/positron,\,photon})$
stands for different types of injected particles. The factor of 2 in equation
(1) accounts for the total contribution of electrons and positrons. $n_{\rm
PBH}=f_{\rm PBH}\,(\rho_{\rm DM}/M_{\rm PBH})$ is the number density of the
PBHs, and $\rho_{\rm DM}$ is the dark matter energy density.
$d^{2}N^{i}/(dt\,dE)\equiv d^{2}N^{i}/(dt\,dE)\,\big{(}E,M_{\rm
PBH},a_{*}\big{)}$ represents the number of particles emitted by black hole
per unit time per unit energy Page (1976a); MacGibbon and Webber (1990); Laha
_et al._ (2021); Arbey and Auffinger (2019). We use the BlackHawk
code222https://blackhawk.hepforge.org/ to calculate the spectra due to photons
and electrons/positrons; we take both the primary and secondary Hawking
evaporation spectra into account Arbey and Auffinger (2019, 2021).

In the presence of Hawking radiation, the thermal evolution of the gas Chluba
_et al._ (2015); D’Amico  _et al._ (2018),

$\displaystyle\frac{dT_{\rm gas}}{dz}=2\,\frac{T_{\rm
gas}}{1+z}+\frac{\Gamma_{c}}{(1+z)\,H}(T_{\rm gas}-T_{\rm CMB})-\frac{2\
\,\Gamma_{\rm PBH}\,}{3\,N_{\rm tot}(1+z)\,H}\,,$ (3)

here, $\Gamma_{\rm PBH}=\Gamma_{\rm PBH}^{e^{\pm}}+\Gamma_{\rm PBH}^{\gamma}$
is the total energy injection per unit time and per unit volume into IGM,
$N_{\rm tot}=N_{\rm H}\,(1+f_{\rm He}+X_{e})$ is the total number density of
the gas, $N_{\rm H}$ is the hydrogen number density, $f_{\rm He}=N_{\rm
He}/N_{\rm H}$, $N_{\rm He}$ is the helium number density, $X_{e}=N_{e}/N_{\rm
H}$ is the free electron fraction and $N_{e}$ is the free electron number
density. $\Gamma_{c}$ stands for the Compton scattering rate Schleicher  _et
al._ (2008); Natwariya and Bhatt (2020). We consider the following numerical
values of the cosmological parameters: $h=0.674$, $\Omega_{\rm M}=0.315$,
$\Omega_{\rm b}=0.049$ and $T_{\rm CMB}|_{z=0}=2.725$ K Planck Collaboration
VI (2020); Fixsen (2009). To compute the energy deposition efficiency, thermal
and ionization history of the Universe, we use
DarkHistory333https://darkhistory.readthedocs.io/en/master/ package with
necessary modifications Liu  _et al._ (2020).

(a)

(b)

Figure 1:  The gas temperature evolution with redshift for evaporating
primordial black hole. The red dashed lines represent the CMB temperature
evolution. The black dashed lines depicts the $T_{\rm gas}$ when there is no
PBHs. The shaded region corresponds to the redshift $15\leq z\leq 20$ (EDGES
observed signal). In plot (1a), we consider PBHs mass and $f_{\rm PBH}$ to
$1\times 10^{15}$ g and $10^{-7}$, respectively, and vary the spin of PBHs. In
plot (1b), we keep $M_{\rm PBH}=1\times 10^{15}$ g and $a_{*}=0.5$ constant
and vary $f_{\rm PBH}$. Figure 2:  The caption is the same as in Figure (1),
except, here, we vary the mass of PBHs and keep spin and $f_{\rm PBH}$ to
$0.5$ and $10^{-7}$, respectively.

##  III Results and Discussion

Following the Refs. Mesinger and Furlanetto (2007); Mesinger  _et al._ (2011);
Pritchard and Loeb (2012); Mittal and Kulkarni (2020), we write the global
21-cm differential brightness temperature as,

$\displaystyle T_{21}=27\,X_{\rm HI}\,\left(1-\frac{T_{\rm R}}{T_{\rm
S}}\right)\,\left(\frac{0.15}{\Omega_{\rm
m}}\,\frac{1+z}{10}\right)^{1/2}\left(\frac{\Omega_{\rm
b}h}{0.023}\right)~{}{\rm mK}\,,$ (4)

here, $X_{\rm HI}=N_{\rm HI}/N_{\rm H}$ is the fraction of neutral hydrogen in
the Universe, and $N_{\rm HI}$ is the neutral hydrogen number density. $T_{\rm
S}$ is the spin temperature, and it is characterized by the number density
ratio of $1S$ triplet and singlet hyperfine states of the neutral hydrogen
atom. In the cosmological scenarios, there are mainly three processes that can
affect the spin temperature: background radio radiation, Ly$\alpha$ radiation
from the first stars and collisions of a hydrogen atom with another hydrogen
atom, residual electron or proton. In the detailed balance between the
population of $1S$ singlet and triplet state, one can write the spin
temperature as Field (1958); Pritchard and Loeb (2012),

$\displaystyle T_{\rm S}^{-1}=\frac{T_{\rm
R}^{-1}+x_{\alpha}\,T_{\alpha}^{-1}+x_{c}\,T_{\rm
gas}^{-1}}{1+x_{\alpha}+x_{c}}\,,$ (5)

here, $T_{\alpha}$ is the Ly$\alpha$ colour temperature. $x_{\alpha}$ is the
Ly$\alpha$ coupling coefficient due to Wouthuysen-Field effect Wouthuysen
(1952); Field (1958). $x_{c}$ is the collisional coupling coefficient due to
scattering between hydrogen atoms or scattering of hydrogen atoms with other
species such as electrons and protons (Pritchard and Loeb (2012) and
references therein). The colour temperature can be taken as gas temperature,
$T_{\alpha}\simeq T_{\rm gas}$, due to the repeated scattering between
Ly$\alpha$ photons and gas Field (1958, 1959); Pritchard and Loeb (2012).
After the formation of the first stars ($z\sim 30$), their Ly$\alpha$
radiation causes the hyperfine transition in the neutral hydrogen atom, and
the $x_{\alpha}$ starts dominating over other couplings Pritchard and Loeb
(2012). Therefore, at the redshift 17.2, spin temperature can be approximated
as $T_{\rm S}\simeq T_{\rm gas}$, when the background radiation temperature
$T_{\rm R}=T_{\rm CMB}$ Pritchard and Loeb (2012); Natwariya (2021). In the
standard cases, the background radiation contribution is assumed solely by CMB
radiation, $T_{\rm R}=T_{\rm CMB}$. In the present work, we do not consider
X-ray heating of the gas due to the uncertainty of the known physics of the
first stars. The inclusion of X-ray heating will further strengthen our
projected constraints. Here, it is to be noted that the gas temperature may
increase due to the energy transfer from the background radiation to the
thermal motions of the gas mediated by Ly$\alpha$ radiation from the first
stars Venumadhav  _et al._ (2018). However, due to the uncertainty in known
physics of the first star formation, we do not include this effect also. The
inclusion of this effect will further strengthen our projected bounds on
$f_{\rm PBH}$. Depending on the ratio $T_{\rm CMB}/T_{\rm S}$, there can be
three scenarios: absorption ($T_{\rm CMB}>T_{\rm S}$), emission ($T_{\rm
CMB}<T_{\rm S}$) or no signal ($T_{\rm CMB}=T_{\rm S}$). At redshift $17.2$,
to get $T_{21}\leq-150$ mK, we require $T_{\rm gas}\leq 9.62$ K. Here, $X_{\rm
HI}\simeq 1-X_{e}$, and in our case at required redshift we get
$X_{e}\lesssim\mathcal{O}(10^{-3})$. Therefore, $X_{\rm HI}$ can be regarded
as unity.

(a)

(b)

Figure 3: The upper projected bounds on the dark fraction of matter in the
form PBHs ($f_{\rm PBH}=\Omega_{\rm PBH}/\Omega_{\rm DM}$) as a function of
PBHs mass for varying spin ($a_{*}$) of PBHs. The shaded regions are excluded
from our analysis for $f_{\rm PBH}$ when $a_{*}=0$ (dotted black line), 0.5
(dot-dashed black line), 0.9 (dashed black line) and 0.9999 (solid black
line). The dashed blue curve depicts the upper constraint on $f_{\rm PBH}$ by
observations of the diffuse Isotropic Gamma-Ray Background (IGRB) for
$a_{*}=0.9$ Arbey  _et al._ (2020b). The double-dot-dashed blue curve
represents the upper constraint on $f_{\rm PBH}$ from Diffuse Supernova
Neutrino Background (DSNB) searches at Super-Kamiokande, while the solid blue
line represents the INTErnational Gamma-Ray Astrophysical Laboratory
(INTEGRAL) observation of 511 KeV $\gamma$-ray lines at Galactic centre
constraint on $f_{\rm PBH}$ for $a_{*}=0.9$ Dasgupta  _et al._ (2020). The
double-dot-dashed magenta (red) line represents the AMEGO forecast for
$a_{*}=0\ (a_{*}=0.9999)$ Ray  _et al._ (2021). Near future, AMEGO
collaboration will be able to probe the parameter-space above the magenta
(red) double-dot-dashed curve for $a_{*}=0\ (a_{*}=0.9999)$. The solid green
line stands for 95% confidence level bound from INTEGRAL observation of
Galactic gamma-ray flux for non-spinning PBHs Laha  _et al._ (2020). Solid
cyan curve depicts the upper bound from observing the 511 KeV $\gamma$-ray
lines at the Galactic centre by assuming all the PBHs within a 3 Kpc radius of
the Galactic centre for non-spinning PBHs Laha (2019). The magenta solid line
represents the Planck constraint Clark  _et al._ (2017). The red solid line
depicts the dwarf galaxy Leo T constraint Kim (2021) and the green dashed line
shows the COMPTEL bound Coogan  _et al._ (2021) for non-spinning PBHs.

In Figures (1) and (2), we present IGM gas temperature evolution as a function
of redshift for different PBH masses, spins and fractions of DM in the form of
PBHs. The shaded region corresponds redshift range, $15-20\,$. The red dashed
curves in all plots depict the CMB temperature evolution, while the black
dashed line represents the gas temperature when there are no evaporating PBHs.
In Figure (1a), we keep mass and $f_{\rm PBH}$ to $1\times 10^{15}$ g and
$10^{-7}$, respectively, and vary the spin of PBHs. As expected, when we
increase the spin of PBHs, the gas temperature rises significantly in the
shaded region. The solid violet curve represents the case when the spin of
PBHs is 0. Increasing the spin to 0.5 (solid green line), the gas temperature
increases. Continuously increasing spin to 0.99 (solid cyan line), the gas
temperature rises further. In Figure (1b), we keep $M_{\rm PBH}=1\times
10^{15}$ g, spin to 0.5 and vary $f_{\rm PBH}$. In this plot, as we increase
the $f_{\rm PBH}$ from $10^{-8}$ (solid cyan line) to $10^{-6}$ (solid violet
line), the IGM heating rises rapidly. If the gas temperature becomes larger
than the CMB temperature in the shaded region, it can erase the 21 cm
absorption signal; instead, it may give an emission signal. Therefore, at
desired redshift (in our scenario $z=17.2$), one has to keep $T_{\rm
gas}<T_{\rm CMB}$ to get an absorption signal. Increasing $f_{\rm PBH}$, for a
given mass, the number density of PBHs increases, resulting in more energy
injection into IGM by PBHs Hawking evaporation. Therefore, increasing the
$f_{\rm PBH}$, the gas temperature rises. In Figure (2), we vary the mass of
PBHs and keep spin and $f_{\rm PBH}$ constants to $0.5$ and $10^{-7}$,
respectively. In this plot, as we increase the mass of PBHs from $1\times
10^{15}$ g (solid violet line) to $5\times 10^{15}$ g (solid cyan line), the
gas temperature decreases. It happens for two reasons: (i) Increasing the mass
of PBHs leads to a decrease in the total power contributions from Hawking
evaporation of PBHs MacGibbon and Webber (1990). (ii) Ignoring the integral
dependency in equations (1) and (2), $\Gamma_{\rm PBH}^{e^{\pm}}$ and
$\Gamma_{\rm PBH}^{\gamma}$ are proportional to $n_{\rm PBH}=f_{\rm
PBH}\,(\rho_{\rm DM}/M_{\rm PBH})$. For a fixed dark-matter energy density and
$f_{\rm PBH}$, the number density of PBHs increases by decreasing the black
hole mass. Thus, energy injection into IGM per unit volume and time
($\Gamma_{\rm PBH}$) increases, and one gets more heating of the gas.

In Figure (3), we plot the upper projected bounds on the fraction of DM in the
form of PBHs as a function of PBHs mass for different spins. Here, we have
considered that 21-cm differential brightness temperature, $T_{21}$, remains
$-150$ mK at redshift $z=17.2$. We vary the mass of PBHs from $10^{15}$ g to
$10^{18}$ g. The shaded regions in both the plots are excluded for the
corresponding PBH spins. The dashed blue curve represents the upper constraint
on $f_{\rm PBH}$ by observations of the diffuse Isotropic Gamma-Ray Background
(IGRB) Arbey  _et al._ (2020b). The double-dot-dashed blue curve represents
the upper constraint on $f_{\rm PBH}$ from Diffuse Supernova Neutrino
Background (DSNB) searches at Super-Kamiokande, while the solid blue line
represents the INTErnational Gamma-Ray Astrophysical Laboratory (INTEGRAL)
observation of 511 KeV $\gamma$-ray line at Galactic centre constraint on
$f_{\rm PBH}$ for $a_{*}=0.9$ Dasgupta  _et al._ (2020). For $a_{*}=0$, the
observation at the Jiangmen Underground Neutrino Observatory (JUNO) will be
able to place a 20 times stronger bound on the upper allowed value of $f_{\rm
PBH}$ for $M_{\rm PBH}=10^{15}$ g compared to Super-Kamiokande Wang  _et al._
(2021); Dasgupta  _et al._ (2020). The double-dot-dashed magenta (red) line
represents the AMEGO forecast for $a_{*}=0\ (a_{*}=0.9999)$ Ray  _et al._
(2021). In the near future, AMEGO collaboration will be able to probe the
parameter-space above the magenta (red) double-dot-dashed curve for $a_{*}=0\
(a_{*}=0.9999)$. Solid green line stands for 95% confidence level bound from
INTEGRAL observation of Galactic $\gamma$-ray flux for non-spinning PBHs Laha
_et al._ (2020). The solid cyan curve depicts the upper bound from the
observation of 511 KeV $\gamma$-ray lines at the Galactic centre by assuming
all the PBHs within a 3 Kpc radius of the Galactic centre for non-spinning
PBHs Laha (2019). For the comparison, we have also plotted the bounds from
Planck Clark  _et al._ (2017), Leo T Kim (2021) and COMPTEL Coogan  _et al._
(2021) observations for non-spinning PBHs. In Figure (3a), $f_{\rm PBH}$
varies from $1\times 10^{-10}$ to $1\times 10^{-5}$, while, in Figure (3b), it
varies from $1\times 10^{-5}$ to its maximum allowed value 1 ($\Omega_{\rm
PBH}=\Omega_{\rm DM}$). In Figure (3), as we increase the value of spin from
$0$ to its extremal value, $0.9999$, the upper bounds become more stringent.
This is due to an increment in evaporation of PBHs, and it results in more
energy injection into the IGM Page (1976a, b, 1977). As discussed earlier,
increasing the mass of PBHs, energy injection into IGM decreases.
Subsequently, one gets more window to increase the gas temperature or $f_{\rm
PBH}$, and the upper bound becomes weaker. Therefore, in Figure (3), the upper
bound on $f_{\rm PBH}$ weakens as we increase the mass. Our upper projected
constraint on $f_{\rm PBH}$ for $a_{*}=0.9$ is comparable to the INTEGRAL
observation of 511 KeV $\gamma$-ray lines for PBHs mass larger than $\sim
8\times 10^{16}$ and becomes stronger for smaller PBH masses. Also, compared
to IGRB Arbey  _et al._ (2020b) and DSNB Dasgupta  _et al._ (2020), our
projected bounds are stringent for the considered mass range of PBHs. We find
the most robust lower projected constraint on the mass of PBHs, which is
allowed to constitute the entire dark matter, to $1.5\times 10^{17}$ g,
$1.9\times 10^{17}$ g, $3.9\times 10^{17}$ g and $6.7\times 10^{17}$ g for PBH
spins 0, 0.5, 0.9 and 0.9999, respectively. The lower bound on $M_{\rm PBH}$
for $\Omega_{\rm PBH}=\Omega_{\rm DM}$, for extremal spinning PBHs is nearly
four times larger than non-spinning PBHs.

##  IV Conclusions

Spinning primordial black holes can substantially affect the ionization and
thermal history of the Universe. Subsequently, it can modify the 21-cm
absorption signal in the cosmic dawn era by injecting energy due to Hawking
evaporation. We study the upper projected bounds on the fraction of dark
matter in the form of PBHs as a function of mass and spin, considering that
the 21-cm differential brightness temperature does not change more than a
factor of 1/4 from the theoretical prediction based on the $\Lambda$CDM
framework. Our projected constraints are stringent compared to DSNB, INTEGRAL
observation of the 511 KeV line, IGRB, Planck, Leo T and COMPTEL. In the near
future, AMEGO collaboration will be able to probe some parameter space in our
considered mass range of PBHs. In the present work, we have considered the
monochromatic mass distribution of PBHs. The allowed parameter space can also
be explored for different PBHs mass distributions such as log-normal, power-
law, critical collapse, etc. Arbey and Auffinger (2019). Here, it is to be
noted that we have not considered heating of IGM gas due to X-ray from the
first stars in the vague of known physics of the first stars. The inclusion of
X-ray heating will further strengthen our projected bounds.

##  V Acknowledgements

The authors would like to acknowledge Prof. Jitesh R Bhatt and Ranjan Laha for
valuable comments and suggestions, and the TEQIP-III sponsored Workshop on
Astroparticle Physics and Cosmology at the National Institute of Technology
Meghalaya. We thank Alexandre Arbey and Jérémy Auffinger for providing the new
version of the BlackHawk code in advance. T. S. would like to acknowledge the
support from the Dr. D. S. Kothari Postdoctoral fellowship scheme No.
F.4-2/2006 (BSR)/PH/20-21/0163 Finally, the authors would like to thank the
Referee for the suggestions and a detailed report that significantly improved
the quality of the manuscript.

## References

  * Planck Collaboration VI (2020) Planck Collaboration VI, Astronomy & Astrophysics 641, A6 (2020).
  * Peebles (1982) P. J. E. Peebles, ApJL 263, L1 (1982).
  * Hu  _et al._ (2000) W. Hu, R. Barkana, and A. Gruzinov, Phys. Rev. Lett. 85, 1158 (2000).
  * Dodelson and Widrow (1994) S. Dodelson and L. M. Widrow, Phys. Rev. Lett. 72, 17 (1994).
  * Boyarsky  _et al._ (2019) A. Boyarsky, M. Drewes, T. Lasserre, S. Mertens, and O. Ruchayskiy, Progress in Particle and Nuclear Physics 104, 1 (2019).
  * Bulbul  _et al._ (2014) E. Bulbul, M. Markevitch, A. Foster, R. K. Smith, M. Loewenstein, and S. W. Randall, The Astrophysical Journal 789, 13 (2014).
  * Spergel and Steinhardt (2000) D. N. Spergel and P. J. Steinhardt, Phys. Rev. Lett. 84, 3760 (2000).
  * Natwariya  _et al._ (2020) P. K. Natwariya, J. R. Bhatt, and A. K. Pandey, Eur. Phys. J. C 80 (2020), 10.1140/epjc/s10052-020-8341-8.
  * Carr and Kühnel (2020) B. Carr and F. Kühnel, Annual Review of Nuclear and Particle Science 70, 355 (2020), https://doi.org/10.1146/annurev-nucl-050520-125911 .
  * Dasgupta  _et al._ (2020) B. Dasgupta, R. Laha, and A. Ray, Phys. Rev. Lett. 125, 101101 (2020).
  * Frampton  _et al._ (2010) P. H. Frampton, M. Kawasaki, F. Takahashi, and T. T. Yanagida, J. Cosmol. Astropart. Phys. 2010, 023 (2010).
  * Khlopov (2010) M. Y. Khlopov, Research in Astronomy and Astrophysics 10, 495 (2010).
  * Belotsky  _et al._ (2019) K. M. Belotsky, V. I. Dokuchaev, Y. N. Eroshenko, E. A. Esipova, M. Y. Khlopov, L. A. Khromykh, A. A. Kirillov, V. V. Nikulin, S. G. Rubin, and I. V. Svadkovsky, Eur. Phys. J. C 79 (2019), 10.1140/epjc/s10052-019-6741-4.
  * Bird  _et al._ (2016) S. Bird, I. Cholis, J. B. Muñoz, Y. Ali-Haïmoud, M. Kamionkowski, E. D. Kovetz, A. Raccanelli, and A. G. Riess, Phys. Rev. Lett. 116, 201301 (2016).
  * Abbott et al. (2016a) B. Abbott et al. (LIGO Scientific Collaboration and Virgo Collaboration), Phys. Rev. Lett. 116, 061102 (2016a).
  * Abbott et al. (2016b) B. Abbott et al. (LIGO Scientific Collaboration and Virgo Collaboration), Phys. Rev. Lett. 116, 241103 (2016b).
  * Abbott et al. (2017a) B. Abbott et al. (LIGO Scientific and Virgo Collaboration), Phys. Rev. Lett. 118, 221101 (2017a).
  * Abbott et al. (2017b) B. Abbott et al. (LIGO Scientific Collaboration and Virgo Collaboration), Phys. Rev. Lett. 119, 141101 (2017b).
  * Sasaki  _et al._ (2016) M. Sasaki, T. Suyama, T. Tanaka, and S. Yokoyama, Phys. Rev. Lett. 117, 061101 (2016).
  * Zel’dovich and Novikov (1967) Y. B. Zel’dovich and I. D. Novikov, Soviet Astronomy 10, 602 (1967).
  * Hawking (1971) S. Hawking, MNRAS 152, 75 (1971).
  * Carr and Hawking (1974) B. J. Carr and S. W. Hawking, Monthly Notices of the Royal Astronomical Society 168, 399 (1974).
  * Carr (1975) B. J. Carr, ApJ 201, 1 (1975).
  * Espinosa  _et al._ (2018) J. R. Espinosa, D. Racco, and A. Riotto, Phys. Rev. Lett. 120, 121301 (2018).
  * Clesse and García-Bellido (2015) S. Clesse and J. García-Bellido, Phys. Rev. D 92, 023524 (2015).
  * Carr  _et al._ (2010) B. J. Carr, K. Kohri, Y. Sendouda, and J. Yokoyama, Phys. Rev. D 81, 104019 (2010).
  * Hawking (1975) S. W. Hawking, Commun. Math. Phys. 43, 199 (1975), [Erratum: Commun.Math.Phys. 46, 206 (1976)].
  * García-Bellido (2017) J. García-Bellido, Journal of Physics: Conference Series 840, 012032 (2017).
  * Clesse and García-Bellido (2018) S. Clesse and J. García-Bellido, Physics of the Dark Universe 22, 137 (2018).
  * Wright (1996) E. L. Wright, ApJ 459, 487 (1996), arXiv:astro-ph/9509074 [astro-ph] .
  * Lehoucq  _et al._ (2009) R. Lehoucq, M. Cassé, J. M. Casandjian, and I. Grenier, A&A 502, 37 (2009), arXiv:0906.1648 [astro-ph.HE] .
  * Carr (1976) B. J. Carr, ApJ 206, 8 (1976).
  * Page and Hawking (1976) D. N. Page and S. W. Hawking, ApJ 206, 1 (1976).
  * Cline  _et al._ (1997) D. B. Cline, D. A. Sanders, and W. Hong, The Astrophysical Journal 486, 169 (1997).
  * Green (2001) A. M. Green, Phys. Rev. D 65, 027301 (2001).
  * Belotsky  _et al._ (2014) K. M. Belotsky, A. E. Dmitriev, E. A. Esipova, V. A. Gani, A. V. Grobov, M. Y. Khlopov, A. A. Kirillov, S. G. Rubin, and I. V. Svadkovsky, Modern Physics Letters A 29, 1440005 (2014).
  * Belotsky and Kirillov (2015) K. Belotsky and A. Kirillov, J. Cosmol. Astropart. Phys. 2015, 041 (2015).
  * Seager  _et al._ (1999) S. Seager, D. D. Sasselov, and D. Scott, Ast. J. 523, L1 (1999).
  * Seager  _et al._ (2000) S. Seager, D. D. Sasselov, and D. Scott, ApJ 128, 407 (2000).
  * Laha  _et al._ (2021) R. Laha, P. Lu, and V. Takhistov, Physics Letters B 820, 136459 (2021).
  * Kim (2021) H. Kim, Monthly Notices of the Royal Astronomical Society 504, 5475 (2021).
  * Chandrasekhar and Detweiler (1977) S. Chandrasekhar and S. L. Detweiler, Proc. Roy. Soc. Lond. A 352, 325 (1977).
  * Page (1976a) D. N. Page, Phys. Rev. D 13, 198 (1976a).
  * Bowman  _et al._ (2018) J. D. Bowman, A. E. E. Rogers, R. A. Monsalve, T. J. Mozdzen, and N. Mahesh, Nature 555, 67 (2018).
  * Pritchard and Loeb (2012) J. R. Pritchard and A. Loeb, Rep. Prog. Phys 75, 086901 (2012).
  * Ewall-Wice  _et al._ (2018) A. Ewall-Wice, T.-C. Chang, J. Lazio, O. Doré, M. Seiffert, and R. A. Monsalve, The Astrophysical Journal 868, 63 (2018).
  * Jana  _et al._ (2018) R. Jana, B. B. Nath, and P. L. Biermann, Monthly Notices of the Royal Astronomical Society 483, 5329 (2018).
  * Feng and Holder (2018) C. Feng and G. Holder, The Astrophysical Journal 858, L17 (2018).
  * Lawson and Zhitnitsky (2019) K. Lawson and A. Zhitnitsky, Physics of the Dark Universe 24, 100295 (2019).
  * Natwariya (2021) P. K. Natwariya, Eur. Phys. J. C 81 (2021), 10.1140/epjc/s10052-021-09155-z.
  * Lawson and Zhitnitsky (2013) K. Lawson and A. R. Zhitnitsky, Physics Letters B 724, 17 (2013).
  * Levkov  _et al._ (2020) D. G. Levkov, A. G. Panin, and I. I. Tkachev, Phys. Rev. D 102, 023501 (2020).
  * Natwariya and Bhatt (2020) P. K. Natwariya and J. R. Bhatt, MNRAS: Letters 497, L35 (2020).
  * Brandenberger  _et al._ (2019) R. Brandenberger, B. Cyr, and R. Shi, J. Cosmol. Astropart. Phys. 2019, 009 (2019).
  * Chianese  _et al._ (2019) M. Chianese, P. Di Bari, K. Farrag, and R. Samanta, Physics Letters B 790, 64 (2019).
  * Bhatt  _et al._ (2020) J. R. Bhatt, P. K. Natwariya, A. C. Nayak, and A. K. Pandey, Eur. Phys. J. C 80, 334 (2020).
  * Tashiro  _et al._ (2014) H. Tashiro, K. Kadota, and J. Silk, Phys. Rev. D 90, 083522 (2014).
  * Barkana (2018) R. Barkana, Nature 555, 71 (2018).
  * Sikivie (2019) P. Sikivie, Physics of the Dark Universe 24, 100289 (2019).
  * Mirocha and Furlanetto (2019) J. Mirocha and S. R. Furlanetto, MNRAS 483, 1980 (2019).
  * Ghara and Mellema (2019) R. Ghara and G. Mellema, MNRAS 492, 634 (2019).
  * Muñoz and Loeb (2018) J. B. Muñoz and A. Loeb, Nature 557, 684 (2018).
  * Sean Fraser et al. (2018) Sean Fraser et al., Phys. Lett. B 785, 159 (2018).
  * Bransden  _et al._ (1958) B. H. Bransden, A. Dalgarno, T. L. John, and M. J. Seaton, Proc. Phys. Soc. 71, 877 (1958).
  * Barkana  _et al._ (2018) R. Barkana, N. J. Outmezguine, D. Redigolo, and T. Volansky, Phys. Rev. D 98, 103005 (2018).
  * Berlin  _et al._ (2018) A. Berlin, D. Hooper, G. Krnjaic, and S. D. McDermott, Phys. Rev. Lett. 121, 011102 (2018).
  * Kovetz  _et al._ (2018) E. D. Kovetz, V. Poulin, V. Gluscevic, K. K. Boddy, R. Barkana, and M. Kamionkowski, Phys. Rev. D 98, 103529 (2018).
  * Muñoz  _et al._ (2018) J. B. Muñoz, C. Dvorkin, and A. Loeb, Phys. Rev. Lett. 121, 121301 (2018).
  * Slatyer and Wu (2018) T. R. Slatyer and C.-L. Wu, Phys. Rev. D 98, 023013 (2018).
  * D’Amico  _et al._ (2018) G. D’Amico, P. Panci, and A. Strumia, Phys. Rev. Lett. 121, 011103 (2018).
  * Mitridate and Podo (2018) A. Mitridate and A. Podo, J. Cosmol. Astropart. Phys. 2018, 069 (2018).
  * Carr  _et al._ (2021) B. Carr, K. Kohri, Y. Sendouda, and J. Yokoyama, Rep. Prog. Phys. 84, 116902 (2021).
  * Green and Kavanagh (2021) A. M. Green and B. J. Kavanagh, Journal of Physics G: Nuclear and Particle Physics 48, 043001 (2021).
  * Hektor  _et al._ (2018) A. Hektor, G. Hütsi, L. Marzola, M. Raidal, V. Vaskonen, and H. Veermäe, Phys. Rev. D 98 (2018), 10.1103/physrevd.98.023503.
  * Clark  _et al._ (2018) S. J. Clark, B. Dutta, Y. Gao, Y.-Z. Ma, and L. E. Strigari, Phys. Rev. D 98 (2018), 10.1103/physrevd.98.043006.
  * Mena  _et al._ (2019) O. Mena, S. Palomares-Ruiz, P. Villanueva-Domingo, and S. J. Witte, Phys. Rev. D 100 (2019), 10.1103/physrevd.100.043540.
  * Yang (2020a) Y. Yang, Phys. Rev. D 102 (2020a), 10.1103/physrevd.102.083538.
  * Halder and Banerjee (2021) A. Halder and S. Banerjee, Phys. Rev. D 103 (2021), 10.1103/physrevd.103.063044.
  * Tashiro and Kadota (2021) H. Tashiro and K. Kadota, Phys. Rev. D 103 (2021), 10.1103/physrevd.103.123532.
  * Yang (2020b) Y. Yang, The European Physical Journal Plus 135 (2020b), 10.1140/epjp/s13360-020-00710-3.
  * Villanueva-Domingo and Ichiki (2021) P. Villanueva-Domingo and K. Ichiki, “21 cm forest constraints on primordial black holes,” (2021), arXiv:2104.10695 [astro-ph.CO] .
  * Ray  _et al._ (2021) A. Ray, R. Laha, J. B. Muñoz, and R. Caputo, Phys. Rev. D 104, 023516 (2021).
  * Kesden  _et al._ (2010) M. Kesden, G. Lockhart, and E. S. Phinney, Phys. Rev. D 82 (2010), 10.1103/physrevd.82.124045.
  * Cotner and Kusenko (2017) E. Cotner and A. Kusenko, Phys. Rev. D 96 (2017), 10.1103/physrevd.96.103002.
  * Harada  _et al._ (2021) T. Harada, C.-M. Yoo, K. Kohri, Y. Koga, and T. Monobe, The Astrophysical Journal 908, 140 (2021).
  * Luca  _et al._ (2019) V. D. Luca, V. Desjacques, G. Franciolini, A. Malhotra, and A. Riotto, J. Cosmol. Astropart. Phys. 2019, 018 (2019).
  * Luca  _et al._ (2020) V. D. Luca, G. Franciolini, P. Pani, and A. Riotto, J. Cosmol. Astropart. Phys. 2020, 052 (2020).
  * Harada  _et al._ (2017) T. Harada, C.-M. Yoo, K. Kohri, and K.-I. Nakao, Phys. Rev. D 96 (2017), 10.1103/physrevd.96.083517.
  * Kühnel (2020) F. Kühnel, Eur. Phys. J. C 80 (2020), 10.1140/epjc/s10052-020-7807-z.
  * Flores and Kusenko (2021) M. M. Flores and A. Kusenko, Phys. Rev. D 104 (2021), 10.1103/physrevd.104.063008.
  * Arbey  _et al._ (2020a) A. Arbey, J. Auffinger, and J. Silk, Monthly Notices of the Royal Astronomical Society 494, 1257 (2020a).
  * He and Suyama (2019) M. He and T. Suyama, Phys. Rev. D 100 (2019), 10.1103/physrevd.100.063520.
  * Cotner  _et al._ (2019) E. Cotner, A. Kusenko, M. Sasaki, and V. Takhistov, J. Cosmol. Astropart. Phys. 2019, 077 (2019).
  * Dong  _et al._ (2016) R. Dong, W. H. Kinney, and D. Stojkovic, J. Cosmol. Astropart. Phys. 2016, 034 (2016).
  * Taylor  _et al._ (1998) B. E. Taylor, C. M. Chambers, and W. A. Hiscock, Phys. Rev. D 58 (1998), 10.1103/physrevd.58.044012.
  * Arbey  _et al._ (2020b) A. Arbey, J. Auffinger, and J. Silk, Phys. Rev. D 101 (2020b), 10.1103/physrevd.101.023010.
  * Arbey  _et al._ (2021) A. Arbey, J. Auffinger, P. Sandick, B. Shams Es Haghi, and K. Sinha, Phys. Rev. D 103 (2021), 10.1103/physrevd.103.123549.
  * Mittal  _et al._ (2021) S. Mittal, A. Ray, G. Kulkarni, and B. Dasgupta, “Constraining primordial black holes as dark matter using the global 21-cm signal with x-ray heating and excess radio background,” (2021), arXiv:2107.02190 [astro-ph.CO] .
  * Slatyer (2016a) T. R. Slatyer, Phys. Rev. D 93, 023521 (2016a).
  * Slatyer (2016b) T. R. Slatyer, Phys. Rev. D 93, 023527 (2016b).
  * Liu  _et al._ (2020) H. Liu, G. W. Ridgway, and T. R. Slatyer, Phys. Rev. D 101 (2020), 10.1103/physrevd.101.023530.
  * MacGibbon and Webber (1990) J. H. MacGibbon and B. R. Webber, Phys. Rev. D 41, 3052 (1990).
  * Arbey and Auffinger (2019) A. Arbey and J. Auffinger, Eur. Phys. J. C 79 (2019), 10.1140/epjc/s10052-019-7161-1.
  * Arbey and Auffinger (2021) A. Arbey and J. Auffinger, Eur. Phys. J. C 81 (2021), 10.1140/epjc/s10052-021-09702-8.
  * Chluba  _et al._ (2015) J. Chluba, D. Paoletti, F. Finelli, and J. A. Rubiño-Martín, MNRAS 451, 2244 (2015).
  * Schleicher  _et al._ (2008) D. R. G. Schleicher, R. Banerjee, and R. S. Klessen, Phys. Rev. D 78, 083005 (2008).
  * Fixsen (2009) D. J. Fixsen, ApJ 707, 916 (2009).
  * Mesinger and Furlanetto (2007) A. Mesinger and S. Furlanetto, ApJ 669, 663 (2007).
  * Mesinger  _et al._ (2011) A. Mesinger, S. Furlanetto, and R. Cen, MNRAS 411, 955 (2011).
  * Mittal and Kulkarni (2020) S. Mittal and G. Kulkarni, Monthly Notices of the Royal Astronomical Society 503, 4264 (2020).
  * Field (1958) G. B. Field, Proceedings of the IRE 46, 240 (1958).
  * Wouthuysen (1952) S. A. Wouthuysen, ApJ 57, 31 (1952).
  * Field (1959) G. B. Field, ApJ 129, 536 (1959).
  * Venumadhav  _et al._ (2018) T. Venumadhav, L. Dai, A. Kaurov, and M. Zaldarriaga, Phys. Rev. D 98, 103513 (2018).
  * Laha  _et al._ (2020) R. Laha, J. B. Muñoz, and T. R. Slatyer, Phys. Rev. D 101, 123514 (2020).
  * Laha (2019) R. Laha, Phys. Rev. Lett. 123, 251101 (2019).
  * Clark  _et al._ (2017) S. J. Clark, B. Dutta, Y. Gao, L. E. Strigari, and S. Watson, Phys. Rev. D 95 (2017), 10.1103/physrevd.95.083006.
  * Coogan  _et al._ (2021) A. Coogan, L. Morrison, and S. Profumo, Physical Review Letters 126 (2021), 10.1103/physrevlett.126.171101.
  * Wang  _et al._ (2021) S. Wang, D.-M. Xia, X. Zhang, S. Zhou, and Z. Chang, Phys. Rev. D 103 (2021), 10.1103/physrevd.103.043010.
  * Page (1976b) D. N. Page, Phys. Rev. D 14, 3260 (1976b).
  * Page (1977) D. N. Page, Phys. Rev. D 16, 2402 (1977).


# Estimating Multiplicative Relations in Neural Networks

Bhaavan Goel<EMAIL_ADDRESS>
###### Abstract

Universal approximation theorem suggests that a shallow neural network can
approximate any function. The input to neurons at each layer is a weighted sum
of previous layer neurons and then an activation is applied. These activation
functions perform very well when the output is a linear combination of input
data. When trying to learn a function which involves product of input data,
the neural networks tend to overfit the data to approximate the function. In
this paper we will use properties of logarithmic functions to propose a pair
of activation functions which can translate products into linear expression
and learn using backpropagation. We will try to generalize this approach for
some complex arithmetic functions and test the accuracy on a disjoint
distribution with the training set.

###### Index Terms:

Universal approximation theorem, activation function, neural network,
logarithm, product

##  I Introduction

The quality of a model is typically measured by its ability to generalize from
a training set to previously unseen data from the similar distribution. Models
based on neural network architecture with linear activation functions provide
good accuracy on testing data for functions which are linear combination of
inputs described by:

$y=c+\sum\limits_{i}a_{i}x_{i}$

To estimate non linear outputs, the activation functions can be carefully set
to some non-linear functions. The non linear activation functions have to be
differential in the relevant domain for back propagation learning[1] to work.
Generally these non linear activation functions can approximate a wide range
of practical problems and work very well on previously unseen real life data.
However, we focus on a very specific problem of trying to approximate a
product function:

$y=c\prod\limits_{i}x_{i}$

Multiplication cannot be generalized using some linear combination of
variables, thus it becomes necessary to think of some other approach to
accommodate these product functions within the existing neural networks. In
this paper we will:

  1. 1.

introduce customized logarithm-exponential activation pair which could learn
multiplication

  2. 2.

test the accuracy on data which is disjoint with the training set

  3. 3.

compare the results with other activation functions

The following section describes the data used in the experiments. Afterwards,
we introduce our method and discuss the architecture, its training and
relation to prior art. Throughout our discussion, we will refer to $\log_{e}$
as $\log$ for convenience.

##  II Data

We will generate some uniformly distributed synthetic data. We will have a set
of variables $\\{x_{1},x_{2},...,x_{n}\\}$ as input and a single value $y$ as
output. Each input variable will be in the range $[10,100)$ for training and
in the range $[100,1000)$ for testing. Using disjoint sets for training and
testing, we will get a better picture if a model is over-fitting the training
data. The histogram of the training and testing data is shown in figure 2 and
2 respectively. As a primary objective, we will try to approximate three
functions:

$y=\frac{x_{1}x_{2}}{N}$ $y=\frac{x_{1}x_{2}x_{3}}{N}$
$y=\frac{x_{1}x_{2}x_{3}x_{4}}{N}$

and compare the results with various activation functions. Here $N=10^{n}$
represents the normalizing factor to keep the scale of output $y$ comparable
to the inputs $x_{i}$. The normalizing factor N will be kept variable to see
how it impacts the accuracy of the model.

Figure 1: Training dataset

Figure 2: Testing dataset

##  III Symmetric Logarithm-Exponential activation pair

The output of a product function scales very quickly compared to the
individual inputs, which creates the problem of gradient explosion during
backpropagation. In this paper, we try to leverage the well-known property of
logarithm function:

$\log\left(\prod\limits_{i}x_{i}\right)=\sum\limits_{i}\log x_{i},\forall x>0$

We will set logarithm as the activation of a layer to capture the sense of
multiplication through addition of log values. Now using the property of
exponential function:

$e^{\sum\limits_{i}x_{i}}=\prod\limits_{i}e^{x_{i}}$

we get the initial inputs as a product with their corresponding weights as
exponents.

$w_{1}\log x_{1}+w_{2}\log x_{2}\xrightarrow{\exp}x_{1}^{w_{1}}*x_{2}^{w_{2}}$

Backpropagation should update the weights $w_{1}$ and $w_{2}$ to 1 if
multiplication is suitable for reducing the loss and -1 if division can reduce
the loss.

###  III-A Symmetric Logarithm Unit

Since $\log$ is only defined for positive inputs, we need to define how the
activation function would behave for 0 and negative values. Keeping in mind
the discussed property of $\log$, we will try to define the activation such
that it is:

  * •

defined for real number space $\mathbb{R}$

  * •

differentiable in the whole domain

  * •

symmetric about origin

The first two conditions allow backpropagation to work for all the real inputs
and symmetricity about origin avoids bias shift. Thus we define activation
function for the first layer as:

$f(x)=\begin{cases}\log(x+1)&x\geq 0\\\ -\log(1-x)&x<0\end{cases}$ (1)

The differentiability of function 1 $\forall x\in\mathbb{R}-\\{0\\}$ follows
from differentiability of $\log x$, $\forall x>0$.

Figure 3: Symmetric $\log$ function defined in eq.1

Considering the critical point $x=0$

$\lim_{x=0,\delta\to
0^{+}}\frac{\log(x+\delta+1)-\log(x+1)}{\delta}=\frac{\log(1+\delta)}{\delta}=1$
$\lim_{x=0,\delta\to
0^{-}}\frac{-\log(-x-\delta+1)+\log(-x+1)}{\delta}=-\frac{\log(1-\delta)}{\delta}=1$

shows that the function is differentiable for $\forall x\in\mathbb{R}$.

###  III-B Symmetric Exponential Unit

Similarly based on the properties discussed for Symmetric Logarithm unit, we
will define the exponential activation function for the second layer which is
continuous and differentiable $\forall x\in\mathbb{R}$:

$f(x)=\begin{cases}e^{x}-1&x\geq 0\\\ -e^{-x}+1&x<0\end{cases}$ (2)

Figure 4: Symmetric exponential function defined in eq.2 Figure 5:
Architecture of shallow NN model, A1 and A2 represent activation functions

##  IV Experimental Setup

Keras[5] with Google TensorFlow backend[6] was used to implement the deep
learning algorithms in this study with the aid of other scientific computing
libraries: matplotlib[7], numpy[8], and Scikit-learn[9]. All experiments in
this study were conducted on a laptop computer with Intel Core (TM) i7-9750H
CPU @ 2.60GHz, 16GB of DDR3 RAM, and NVIDIA GeForce GTX 1650 4GB.

##  V The Model

We will prepare a shallow feed-forward neural network containing 2 hidden
layers and test various combinations of activation functions on our prepared
dataset. The final output dense layer will use linear activation which is
default for Keras[5]. The abstract architecture of the model has been
described in Fig. 5 and the implemented Keras[5] architecture in figure 6. For
comparison, we will consider activation functions from {elu[10], hard sigmoid,
linear, relu[12], selu[13], sigmoid, softmax, softplus, softsign, swish[14]
and tanh}. These 11 unique activation functions make a total of $121(11\times
11)$ activation pairs. We will test the score of our proposed activation
function against all the $121$ possible pairs.

Figure 6: Keras model

(a) $y=x_{1}x_{2}$

(b) $y=\frac{x_{1}x_{2}}{10}$

(c) $y=x_{1}x_{2}x_{3}$

(d) $y=\frac{x_{1}x_{2}x_{3}}{100}$

(e) $y=x_{1}x_{2}x_{3}x_{4}$

(f) $y=\frac{x_{1}x_{2}x_{3}x_{4}}{1000}$

Figure 7: Training loss over 100 epochs for different input size and
normalization. Here ”lg_model” is our proposed model.

##  VI Results

###  VI-A Product Functions

####  VI-A1 Training Loss

The model is trained to minimize the Mean Absolute Error (MAE) using adaptive
momentum estimation (Adam)[11] optimizer for our experiments. We have trained
the model for product functions with up to 4 input variables and different
normalization factors. Normalization is essential to ensure that the product
of inputs does not increase unboundedly. The plots in figure 7 show $7$ (out
of $121$) activation pairs with minimum training error after $100$ epochs
along with our proposed logarithm – exponential activation pair. In all cases
our model not only achieves the minimum training loss but also converges in
minimum number of epochs. In cases where the output is not normalized to the
scale of input variables, other activation functions show much larger training
loss than our model.

####  VI-A2 Testing Error

We will consider the usual MAE and Mean Percentage Error ($\%_{err}$) which is
calculated as:

$\%_{err}=\frac{100}{n}\sum\limits_{i}\frac{|y^{i}_{pred}–y^{i}_{actual}|}{|y^{i}_{actual}|}$
(3)

The table depicts that the proposed model has much lower $\%_{err}$ compared
to other activation pairs for the test data. It also suggests that in cases
where other activation functions showed lower training error were over-fitting
the training data.

TABLE I: Testing results for different product functions

| Our Model | Next Best
---|---|---
Expression | $\%_{error}$ | MAE | Activation Pair | $\%_{error}$
$x_{1}x_{2}$ | 15.453791 | 47284.32 | relu_linear | 61.191017
$x_{1}x_{2}/10$ | 7.801138 | 1565.0941 | relu_selu | 69.28548
$x_{1}x_{2}x_{3}$ | 16.815907 | 27887974 | elu_elu | 98.30871
$x_{1}x_{2}x_{3}/100$ | 24.82434 | 394495.03 | softplus_relu | 93.41624
$x_{1}x_{2}x_{3}x_{4}$ | 28.907835 | 3.4e+10 | softplus_selu | 99.79
$x_{1}x_{2}x_{3}x_{4}/1000$ | 39.76962 | 34425464 | elu_linear | 99.70

###  VI-B Complex function

We will also show the effectiveness of our method for estimating more complex
arithmetic functions. The function we try to estimate is:

$y=x_{1}(x_{2}+x_{3})+x_{4}$ (4)

####  VI-B1 Training Loss

The training loss for the function described by eq.4 is also minimum for our
model and also converges quickly. The training loss is depicted in figure 8.

Figure 8: Training loss for $y=x_{1}(x_{2}+x_{3})+x_{4}$. Here ”lg_model” is
our proposed model

####  VI-B2 Testing Error

The mean percentage error without any architectural changes for the complex
formula is 22.794909. which is much less than the next best 97.09597 for selu-
linear activation pair.

##  VII Related Work

Durbin and Rumelhart, 1989 [3] tries to achieve the results of product
function by taking weighted product of inputs. This approach achieves desired
results but requires changes in the commonly used training loop where we
generally take weighted sum of the inputs. Georg Martius and Christoph H.
Lampert [4] propose a cell that takes two inputs at a time and calculates
their product. It also performs a weighted product of the inputs. NLRELU [2]
introduces an activation similar to the activation function used in the first
layer of our model. NLRELU has various advantages but it alone cannot estimate
a product function very well.

##  VIII Conclusion

We presented and studied the behavior of logarithm paired with an exponential
activation function for estimating product functions. For data which have
multiplicative relations between the inputs, we saw that our customized
activation pair achieved minimum scoring error compared to other activation
functions and also avoids over-fitting on the training data with a very
shallow feed-forward neural network. We proved the differentiability of our
functions which allows end-to-end training using backpropagation and thus can
be integrated within deep neural networks to estimate hidden product
relations.

##  IX Acknowledgement

We thank Soham Pal and Shivam Gupta for continuous encouragement and feedback
throughout this paper.

## References

  * [1] David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams ”Learning representations by back-propagating errors”
  * [2] Y. Liu, J. Zhang, C. Gao, J. Qu and L. Ji, ”Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks,” 2019 IEEE 5th International Conference on Computer and Communications (ICCC), Chengdu, China, 2019, pp. 2000-2008, doi: 10.1109/ICCC47050.2019.9064398.
  * [3] Durbin, R. M., and Rumelhart, D. E. (1989). Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Computation, 1(1), 133–142. https://doi.org/10.1162/neco.1989.1.1.133
  * [4] Martius, G. and Lampert, C. H., “Extrapolation and learning equations”, $\it arXive-prints\it$, 2016.
  * [5] François Chollet et al. 2015. Keras. https://github.com/keras-team/keras. (2015).
  * [6] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015\. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). http://tensorflow.org/ Software available from tensorflow.org.
  * [7] J. D. Hunter. 2007. Matplotlib: A 2D graphics environment. Computing In Science and Engineering 9, 3 (2007), 90–95. https://doi.org/10.1109/MCSE.2007.55
  * [8] Stéfan van der Walt, S Chris Colbert, and Gael Varoquaux. 2011. The NumPy array: a structure for efficient numerical computation. Computing in Science and Engineering 13, 2 (2011), 22–30.
  * [9] Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.
  * [10] Clevert, Djork-Arné and Unterthiner, Thomas and Hochreiter, Sepp. (2016). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs).
  * [11] Kingma, Diederik and Ba, Jimmy. (2014). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations.
  * [12] A. F. Agarap, “Deep Learning using Rectified Linear Units (ReLU).,” CoRR, vol. abs/1803.08375, 2018.
  * [13] Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S., “Self-Normalizing Neural Networks”, $\it arXive-prints\it$, 2017.
  * [14] Ramachandran, P., Zoph, B., and Le, Q. V., “Searching for Activation Functions”, $\it arXive-prints\it$, 2017.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CLEF 2021 – Conference and Labs of the Evaluation Forum, September 21–24,
2021, Bucharest, Romania

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

# Recognizing bird species in diverse soundscapes under weak supervision

Christof Henkel  NVIDIA, Munich, Germany  Pascal Pfeiffer  RWTH Aachen
University, Aachen, Germany  Philipp Singer  H2O.ai, Mountain View, USA  All
authors contributed equally.

(2021)

###### Abstract

We present a robust classification approach for avian vocalization in complex
and diverse soundscapes, achieving second place in the BirdCLEF2021 challenge.
We illustrate how to make full use of pre-trained convolutional neural
networks, by using an efficient modeling and training routine supplemented by
novel augmentation methods. Thereby, we improve the generalization of weakly
labeled crowd-sourced data to productive data collected by autonomous
recording units. As such, we illustrate how to progress towards an accurate
automated assessment of avian population which would enable global
biodiversity monitoring at scale, impossible by manual annotation.

###### keywords:

BirdCLEF2021 LifeCLEF audio bioacoustics fine-grained classification bird
recognition Kaggle

##  1 Introduction

Avian species are important members of our ecosystem providing regulating
services like pollinating plants, dispersing seeds and controlling pests. To
understand the functioning of a local ecosystems, it is important to detect
and identify individual bird species. Since manual monitoring of avian species
is impossible at a large scale it is crucial to develop autonomous systems for
monitoring and classification.

The yearly Bird Recognition challenge (_BirdCLEF_) puts attention on the
reliable detection of avian vocalizations in various soundscapes. The
_BirdCLEF2021_ edition [1] as part of _LifeCLEF2021_ [2, 3, 4, 5] especially
focuses on generalization of audio data from the crowd-sourced bird sound
database _xeno-canto_ [6] towards analyzing complex and long soundscape
recordings generated in different contexts.

The goal of this year’s competition is to predict the presence of individual
bird species in five second intervals for recordings across four distinct
locations in North and South America. Contrarily, the provided training data
only contains short recordings of individual bird calls extracted from user
uploads on the xeno-canto platform. Consequently, competitors had to address a
weak label problem to bridge the gap between training and testing data
discrepancy.

In this work, we discuss an efficient model architecture and training routine
as part of the second place solution of the BirdCLEF2021 competition. Our
solution builds upon a robust bootstrapped validation setup, and consists of
an ensemble of convolutional neural networks (CNN) trained on extracted
spectrograms of short audio clips. Our developed modeling architecture
successfully bridges the gap between the weakly labeled crowd-sourced training
data and the more accurately hand-labeled test data with hard labels at five
second intervals. To further account for label noise in training data, we
employ several augmentation and training methods such as novel mixup
variations, the addition of background noise, or label smoothing. Final
predictions are supplemented by a binary bird presence classifier and several
post processing steps. Implementation details can be found on
github111https://github.com/ChristofHenkel/kaggle-birdclef2021-2nd-place.

##  2 Material

###  2.1 Competition Data

The final test set of this competition includes approximately 80 soundscape
recordings across the four unique locations Columbia (COL), Costa Rica (COR),
Sierra Nevada (SNE) in California, USA and Sapsucker Woods area (SSW) in New
York, USA. Each recording lasts ten minutes and predictions had to be provided
for each five second interval of each audio file addressing the presence or
absence of individual bird species. Additional meta data including the time of
recording as well as the approximate coordinates of the location is provided
for the test data.

Contrarily, the training data only contains weak labels for around $63,000$
short recordings of individual bird calls shared by users on the xeno-canto
platform. Overall, $397$ bird species are captured in this training set, out
of which only an unknown set is also present in the test set. For each
recording, a primary label is specified, as well as an optional list of
secondary labels. Additional meta data including the date of recording, geo-
location, the user uploading the data, as well as a quality rating from $0-5$,
where $5$ indicates best quality, is present.

To allow for better local validation of models, the competition data also
includes a separate validation soundscape dataset which is similarly
structured as the final test dataset. However, it includes only $20$
soundscape files across two out of the four test data locations and does not
necessarily capture the same bird species as the final test set. Yet, it
allows to roughly bridge the gap between the difference of weak labels used
for training, and hard labels used for testing, and allows for better
validation as discussed in Section 3.

###  2.2 External Data

To improve the generalization of any model trained on the crowd-sourced
database xeno-canto and apply it to new unseen data we augmented the training
data with background noise not containing avian calls. In particular, we used
two additional external datasets freefield1010 [7] and BirdVox-DCASE-20k [8]
which had binary labels for the presence/absence of bird sounds and were
advertised in the DCASE2018 Challenge [9]. Moreover, we extracted consecutive
30 second clips containing no avian sound from the validation set of the
previous BirdCLEF2020 competition [10] which was hosted on aicrowd
222https://www.aicrowd.com/challenges/lifeclef-2020-bird-monophone.

##  3 Methods

###  3.1 Validation

In Kaggle code competitions, the test dataset is hidden in such a way, that
only very limited information can be gathered about it other than the public
leaderboard score, which resembles the score for 35% of the total test data
described in Section 2, also called _public test data_. The final score is
calculated on the remaining 65% of the data, also called _private test data_.
In this particular machine learning competition, the training data is very
different to the test data, as the audio files only contain weak labels
without time information and differ in length. Nevertheless, for two of the
four test sites, a small validation set resembling the test set is provided by
the hosts of the competition to give competitors a general understanding of
the test data’s nature. The metric of choice according to which competitors
are evaluated in this competition is the row-wise micro averaged F1 score.

Consequently, our models are only trained on the short training clips, and
locally evaluated on the provided validation soundscapes. This validation
scheme was further improved by omitting three full soundscapes (out of 20)
that did not contain any bird calls at all. In the following, this validation
scheme is referred to as CV-3. To receive an even more robust feedback from
validation, a bootstrap sampling method was introduced. Our final validation
setup included the following steps:

  * •

Remove three songs without calls.

  * •

For k times (e.g., $10$) sample $80\%$ of the remaining songs—this should
emulate the full test dataset (public+private).

  * •

Apply any kind of threshold selection technique or post processing on this
data resembling the application in inference for the full public and private
test data.

  * •

For j times (e.g., $50$) sample $65\%$ of the remaining songs—this should
emulate the private test dataset.

  * •

Calculate the score on each of these j samples.

  * •

Report average, median, min, max, std scores across all k times j (e.g.,
$500$) subsets

###  3.2 Modeling

Our final solution is a an ensemble of CNNs which were trained on extracted
spectrograms of the audio clips. Their predictions were further post-processed
using a binary CNN for classifying the presence/absence of avian sound and
available meta data with respect to date and region of the test soundscapes.
In the following, we discuss design and training routine of the CNNs. All our
models were trained on GPUs utilizing the Pytorch framework and backbones that
were pre-trained on ImageNet and that are available at timm [11]. In order to
perform computational efficient training, we also performed the spectrogram
transformation on GPU using
torchaudio333https://pytorch.org/audio/stable/index.html and mixed precision.

####  3.2.1 Bird Classifier

The core part of our solution addresses the main task-specific challenges:
Firstly, training data is weakly labeled with bird sound classes being
annotated on audio clips of varying length, while prediction needs to be
performed on five second time windows of several minute long recordings.
Secondly, due to the complex soundscapes and broad range of data contributors,
the training labels contain a significant amount of low quality annotations.
Lastly, the test data is of significantly different context and of different
quality.

##### Preprocessing

The training data contains short sound clips of different length with weak
primary and secondary labels. To preprocess the training data for modeling, we
randomly cropped a 30 second time window from a recording and applied a mel-
spectrogram transformation. Hereby, 30 seconds proved to be a good compromise
between label accuracy and generalization provided by the augmentation. To
account for the five second time window required for inference of the test
data, we reshaped the crops into six parts. Before feeding the spectrograms
into the CNN-backbone, we applied a two dimensional version of mixup [12]. In
particular, we not only mixed between different recordings (up to two times),
but also within a recording by mixing the six parts. Our preprocessing was
performed on GPU using functionalities of torchaudio for mel-spectrogram
transformation and a custom implementation for mixup. We illustrate our
preprocessing pipeline in Figure 1.

Figure 1: Visualization of preprocessing steps for the bird classification
model — Input is a 30 second wav crop. After mel-spectrogram transformation,
the input data is split into six equal five second parts, before mixup
augmentation is applied within and between recordings.

##### Architecture

Given a batch of mel-spectrograms, the model extracts features with a CNN-
backbone pretrained on ImageNet, where in our final solution we used backbones
from the resnet [13] efficientnet v2 [14] and nfnet [15] family, which are all
available in the timm repository [11]. We re-arrange the tensors to the 30
second representation by concatenating the respective time segments and use
generalized mean pooling (GeM) [16] of time and frequency dimension before
forwarding through a simple one layer head which results in a prediction of
397 bird classes. For inference we directly fed five second snippets to the
CNN-backbone and applied the head without reshaping. See Figure 2 for an
illustration.

Figure 2: Illustration of bird classification model — five second mel-
spectrograms are fed into a pretrained CNN-backbone and re-arranged to 30
second time segments before GeM pooling is applied. A single linear layer
outputs the bird classes.

##### Training routine

We did not differentiate between primary and secondary labels and set the
multilabel target as the union of both. We trained our models with a binary
cross-entropy loss using Adam optimizer and a 11 to 20 epochs lasting cosine
annealing schedule with a batch size varying from 16 to 32 per model. While
the model is not very sensitive to hyperparameters of optimizer and learning
rate schedule, several adjustments lead to significant improvement in
classification quality.

Firstly, we used the quality rating, which is given as meta information for
the training data for weighting a recording’s contribution to the loss. The
assumption is that recordings with a lower rating have worse quality with
respect to audio and label and should contribute less to model training. In
detail, we weight each sample by rating/max(ratings).

Secondly, we used one-sided label smoothing, i.e., adding $0.01-0.025$ across
all negative labels while positive class is unchanged. This accounts for noisy
annotations and absence of birds in “unlucky“ 30 second crops. Similar to
solutions of past BirdCLEF challenges, we mixed the training data with
additional background noise containing no avian calls. Specifically, we used
recordings from the freefield dataset labeled for bird absence, three
recordings of this years validation set containing no bird calls and 30 second
crops of past years validation data containing no bird calls according to
their annotations; for details, please refer to Section 2.

####  3.2.2 Binary classifier

A binary classifier was trained to distinguish any bird call from other
environment sounds and predictions were used in the ensembling/post-processing
step together with the bird classifier models. For this task, we only used
external data, the binary labeled freefield1010 [7] and BirdVox-DCASE-20k [8]
datasets, both containing ten second audio recordings with or without avian
sounds (see Section 2).

Figure 3 gives an overview of the binary classifier. After transforming the
recordings to mel-spectrograms, mixup augmentation is applied before feeding
into a pretrained CNN-backbone. The output feature map is mean pooled on the
frequency axis before applying attention pooling on time segments. A single
linear layer is applied to output a binary bird presence / absence prediction.

Figure 3: Illustration of binary classification model — Ten second recordings
are transformed to spectrograms before augmented by mixup. Using a pre-trained
CNN-backbone features are extracted which are pooled by mean and attention
before a single linear layer outputs a binary bird presence / absence
prediction

###  3.3 Postprocessing

As F1 metric is evaluated on hard predictions without taking probabilities
into account, appropriate thresholds for detections in a five second segment
in soundscapes had to be found. Optimizing the thresholds on CV-3 and applying
it to the test set already indicated good correlation. Nevertheless,
submitting to the public leaderboard with lower thresholds (higher number of
predictions) yielded even better F1 scores. Additionally, one major issue we
observed was when blending new models, the threshold needs to be re-optimized
each time. Hard thresholds are not necessarily consistent across models, as
each one can exhibit certain shifts in probabilities and their averages.
Consequently, it is challenging to properly judge if new models work well in
an ensemble based on the merit of the models, or only based on arbitrary
probability / threshold shifts that emerged from it. For a more robust
approach regarding shifts in model probabilities, we decided to apply a
percentile based thresholding approach. In detail, we flattened all
predictions and calculated the threshold for a certain percentile which we
optimized independently for maximum F1 score on CV-3 and on public
leaderboard.

The more birds a set contains, the lower the percentile should be if
predictions are decently ranked. With this approach, we kept the percentile
fixed, thus always predicting the same amount of bird calls, and just
exchanged models, blends and other post processing steps. If the quality in
our ranking of predictions improved, also the score improved.

On our bootstrapped CV-3, we found a percentile of $0.9987$ to be optimal.
Based on public leaderboard feedback, we decided to decrease the percentile a
bit further to $0.9981$ for our test predictions expecting a few more birds to
be predicted. In retrospective, a slightly higher percentile would have
further improved our solution. By and large, our percentile based thresholding
approach appeared to be robust across CV and LB and generalized well to the
unseen private part of the test set.

Additionally, we employed several smaller post processing steps to improve
predictions including: (1) increasing the probability of birds in songs based
on their average prediction probability, (2) smoothing neighboring
predictions, and (3) we also removed some unlikely predictions based on
distance in space and time given the provided metadata. Additionally, we
utilized our binary models predicting whether a bird song is present in a
given window or not and adjusted our predictions by
$p_{bird}=p_{bird}*(1+p_{binary}*0.8)$. Overall, we fitted 15 binary models
and averaged them.

###  3.4 Ensembling

The ensembling [17] of our models was straightforward since all outputs have
the same shapes; we took the arithmetic mean of the predictions. Post-
processing steps (1) and (2) as described in Section 3.3 were applied
individually on each model, all other steps were applied after blending. Our
final solution contains nine models differing only on hyperparameters and
backbones. Each type of model was fitted with six different seeds accounting
for variability in predictions. Our final inference Kaggle
kernel444https://www.kaggle.com/ilu000/2nd-place-birdclef2021-inference/ ran
only one hour out of three hours allowed giving us further potential room for
improvements. In the following Section 4, we describe according results and go
into more detail about the individual models.

Figure 4: Visualization of bootstrapped validation results — This figure depicts the distribution of scores over 500 iterations based on our bootstrap validation routine. Table 1: Model results — This table highlights individual model and final ensemble results sorted by CV score. All models exhibit same postprocessing steps, but differ in utilized backbones and hyperparameters. The main hyperparameter differences stem from melspec settings, background augmentation, and label smoothing (LS). For melspec settings, S1 refers to window_size=1024, hop_size=320, fmin=50, fmax=14000, mel_bins=256, power=2, and top_db=80, while S2 refers to window_size=2048, hop_size=512, fmin=16, fmax=16386, mel_bins=64, power=2, and top_db=None. Model | Backbone | Melspec |  Back ground | LS | CV | Public LB | Private LB
---|---|---|---|---|---|---|---
ps-6-v2 | resnet34 | S1 | no | 0.000 | 0.7387 | 0.7649 | 0.6592
ps-12-v8 | resnet34 | S1 | yes | 0.000 | 0.7698 | 0.7665 | 0.6832
ps-12-v13 | tf_efficientnetv2_m_in21k | S2 | yes | 0.000 | 0.7716 | 0.7610 | 0.6719
ps-12-v21 | eca_nfnet_l0 | S2 | yes | 0.000 | 0.7803 | 0.7892 | 0.6846
ps-12-v11 | tf_efficientnetv2_s_in21k | S1 | yes | 0.000 | 0.7807 | 0.7848 | 0.6751
ps-12-v30 | tf_efficientnetv2_s_in21k | S2 | yes | 0.025 | 0.7861 | 0.7841 | 0.6667
ch-12-v25a | tf_efficientnetv2_s_in21k | S2 | yes | 0.010 | 0.7901 | 0.7917 | 0.6785
ps-12-v32 | tf_efficientnetv2_s_in21k | S2 | yes | 0.010 | 0.7855 | 0.7838 | 0.6656
ch-12-v25g | tf_efficientnetv2_s_in21k | S2 | yes | 0.010 | 0.7975 | 0.7869 | 0.6845
Ensemble |  |  |  |  | 0.8000 | 0.7998 | 0.6893
Table 2: Leaderboard results — This table shows the ranking and scores of the top five teams on the private leaderboard including their respective public leaderboard scores. The team of this article is highlighted in bold. Team | Public LB | Private LB
---|---|---
Dr.北村の愉快な仲間たち | 0.7736 (#17) | 0.6932 (#1)
new baseline | 0.7998 (#3) | 0.6893 (#2)
Shiro | 0.7919 (#4) | 0.6891 (#3)
Third time’s the charm. | 0.7897 (#6) | 0.6864 (#4)
Kramarenko Vladislav | 0.7897 (#7) | 0.6820 (#5)

##  4 Results and discussion

As described in Section 3, our final solution is based on an ensemble of
multiple models with different hyperparameter settings and seeds. In Figure 4,
we visualize our bootstrapped CV-3 validation results; the x-axis depicts the
score, and the y-axis respective density across all 500 bootstrap iterations.
The distribution exhibits a mean of $0.800$, a median of $0.801$, a minimum of
$0.706$, a maximum of $0.906$, as well as a standard deviation of $0.0365$. As
mentioned, the percentile for validation was set to $0.9987$, while for test
we used a percentile of $0.9981$. Post-processing parameters are optimized on
validation, and accordingly set for test inference.

In Table 1, we highlight all individual models as well as the final blend with
according settings, and validation and test scores. For all models, we
employed the same thresholding, binary prediction adjustments, as well as
post-processing, meaning that individual results only differ on the trained
models used. Again, note that each individual model averages six different
seeds to better account for individual randomness. For further details please
also refer to respective configurations in our code base 555See footnote 1.

The results further highlight the difficulty of robust local validation
generalizing well to the unseen test data. While we can observe a trend of
higher CV leading to higher test scores, the best individual local validation
models are not necessarily also the best models on public and private
leaderboard scores. Also, while validation and public LB scores are more
aligned, private LB scores exhibit significantly lower scores even exceeding
the lower bounds of our bootstrapped validation scores as shown in Figure 4.
With test data being hidden, we can only speculate that some of the
differences can be explained by the presence of two additional test sites (COL
& SNE) for which we can neither robustly judge the presence / absence of
certain bird species nor the amount of bird segments in respective data
strongly driving absolute scores. It is also unclear, whether public and
private test data is split randomly, or based on a certain logic. However,
note that we could gather these insights only after the competition ended, as
we were only able to acquire limited insights into public LB scores (two
submissions per day), and no insights about private LB during competition.
Yet, these observations further strengthen the importance of robust solutions
in this competition, that are able to generalize well to partly unseen data
and locations.

While we can observe some (partly random) fluctuations in individual model
scores (although each one consists of six individual seeds), our final
ensemble exhibits the best local validation score, best public LB score, and
best private LB score. The ensemble successfully captures the weaknesses and
strengths of each individual model and was a robust choice to generalize well
on the unseen test data. A further testament of the robustness of our solution
is given by the fact that we improved our rank on private leaderboard compared
to our initial rank on public leaderboard compared to other contestants. The
final leaderboard results are highlighted in Table 2.

##  5 Conclusions

In this work, we discussed how to address challenges of using the crowd-
sourced dataset xeno-canto for productive bird sound recognition as part of
the second place solution to BirdCLEF2021 [1]. Specifically, we showed how to
efficiently finetune a pre-trained CNN with weakly labeled data by reshaping
input and output tensors and how to handle noisy labels by using quality
rating, as well as label smoothing. Furthermore, we demonstrated that mixing
train dataset recordings internally as well as with external data
significantly improves generalization to distinct acoustic environments. Thus,
we gave suggestions to enhance autonomous monitoring of avian population which
is essential for assessing the biodiversity and healthiness of ecosystems and
a sustainable development of humanity.

###### Acknowledgements.

We would like to thank Kaggle and the competition hosts for conducting this
interesting competition as well as all other competitors on Kaggle for the
challenge. Additionally, a big thank you goes to all users of xeno-canto
generously uploading and sharing their bird recordings with the world. Without
them this work would not have been possible.

## References

  * Kahl et al. [2021] S. Kahl, T. Denton, H. Klinck, H. Glotin, H. Goëau, W.-P. Vellinga, R. Planqué, A. Joly,  Overview of BirdCLEF 2021: Bird call identification in soundscape recordings,  in: Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, 2021.
  * Joly et al. [2021] A. Joly, H. Goëau, S. Kahl, L. Picek, T. Lorieul, E. Cole, B. Deneu, M. Servajean, R. Ruiz De Castañeda, I. Bolon, H. Glotin, R. Planqué, W.-P. Vellinga, A. Dorso, H. Klinck, T. Denton, I. Eggel, P. Bonnet, H. Müller,  Overview of LifeCLEF 2021: a system-oriented evaluation of automated species identification and species distribution prediction,  in: Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021), 2021.
  * Goëau et al. [2021] H. Goëau, P. Bonnet, A. Joly,  Overview of PlantCLEF 2021: cross-domain plant identification,  in: Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, 2021.
  * Picek et al. [2021] L. Picek, A. M. Durso, R. Ruiz De Castañeda, I. Bolon,  Overview of SnakeCLEF 2021: Automatic snake species identification with country-level focus,  in: Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, 2021.
  * Lorieul et al. [2021] T. Lorieul, E. Cole, B. Deneu, M. Servajean, A. Joly,  Overview of GeoLifeCLEF 2021: Predicting species distribution from 2 million remote sensing images,  in: Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, 2021.
  * canto Foundation [2021] X. canto Foundation, xeno-canto, https://www.xenocanto.org, 2021\.
  * Stowell and Plumbley [2014] D. Stowell, M. D. Plumbley,  freefield1010 - an open dataset for research on audio field recording archives,  in: Proceedings of the Audio Engineering Society 53rd Conference on Semantic Audio (AES53), Audio Engineering Society, 2014.
  * Lostanlen et al. [2018] V. Lostanlen, J. Salamon, A. Farnsworth, S. Kelling, J. P. Bello,  Birdvox-full-night: a dataset and benchmark for avian flight call detection,  in: Proc. IEEE ICASSP, 2018.
  * Stowell et al. [2018] D. Stowell, Y. Stylianou, M. Wood, H. Pamuła, H. Glotin,  Automatic acoustic detection of birds through deep learning: the first bird audio detection challenge,  Methods in Ecology and Evolution (2018). URL: https://arxiv.org/abs/1807.05812]. arXiv:1807.05812.
  * Kahl et al. [2020] S. Kahl, M. Clapp, W. Hopping, H. Goëau, H. Glotin, R. Planqué, W.-P. Vellinga, A. Joly,  Overview of BirdCLEF 2020: Bird sound recognition in complex acoustic environments,  in: CLEF 2020, 2020.
  * Wightman [2019] R. Wightman, Pytorch image models, https://github.com/rwightman/pytorch-image-models, 2019\. doi:10.5281/zenodo.4414861.
  * Zhang et al. [2017] H. Zhang, M. Cisse, Y. N. Dauphin, D. Lopez-Paz,  mixup: Beyond empirical risk minimization,  arXiv preprint arXiv:1710.09412 (2017).
  * He et al. [2015] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, 2015. arXiv:1512.03385.
  * Tan and Le [2021] M. Tan, Q. V. Le, EfficientNetV2: Smaller models and faster training, 2021\. arXiv:2104.00298.
  * Brock et al. [2021] A. Brock, S. De, S. L. Smith, K. Simonyan, High-performance large-scale image recognition without normalization, 2021. arXiv:2102.06171.
  * Radenović et al. [2018] F. Radenović, G. Tolias, O. Chum, Fine-tuning CNN image retrieval with no human annotation, 2018. arXiv:1711.02512.
  * Hansen and Salamon [1990] L. K. Hansen, P. Salamon,  Neural network ensembles,  IEEE transactions on pattern analysis and machine intelligence 12 (1990) 993–1001.