# A Chiral Triplet Quasi-Two-Dimensional Superconductor in a Parallel Magnetic
Field

A.G. Lebed∗ Department of Physics, University of Arizona, 1118 E. 4-th Street,
Tucson, AZ 85721, USA

###### Abstract

We calculate the parallel upper critical magnetic field $H_{\parallel}(0)$ for
an in-plane isotropic quasi-two-dimensional (Q2D) chiral triplet
superconductor at zero temperature, $T=0$. In particular, the ratio
$H_{\parallel}(0)/(|dH^{GL}_{\parallel}/dT|_{T=T_{c}}T_{c})=0.815$ is defined,
where $|dH^{GL}_{\parallel}/dT|_{T=T_{c}}$ is the so-called Ginzburg-Landau
slope of the upper critical magnetic field, $T_{c}$ is a superconducting
transition temperature at $H=0$. We show that the theoretically obtained above
mentioned value strongly contradicts to the experimentally measured ones in a
candidate for a chiral triplet superconductivity Sr2RuO4, which provides one
more argument against the chiral triplet scenario of superconductivity in this
compound. Our results may be useful for establishing chiral triplet
superconductivity in other Q2D candidates for this phenomenon.

††preprint: Lebed-PRL

Since discovery of superconductivity in the quasi-two-dimensional (Q2D)
conductor Sr2RuO4 [1], it has been intensively investigated for more than 25
years (for reviews, see Refs.[2,3]). Some analogy of this Q2D superconductor
with the superfluid 3He was recognized from the beginning and the existence of
a chiral triplet superconducting phase in the Sr2RuO4 was suggested [4]. This
scenario of superconducting pairing was supported by the observations of no
change of the Knight shift between normal and superconducting phases [5,6] and
breaking of the time reversal symmetry in the superconducting phase [7,8]. On
the other hand, there were arguments against the chiral triplet
superconductivity scenario, which were almost ignored that time by scientific
community. One of the first argument was the paramagnetic limitation of the
parallel upper critical magnetic field in Sr2RuO4 [9,10]. In addition, the
predicted in the chiral triplet scenario edge currents were not found in the
Sr2RuO4 [11,12] but were found zeros of superconducting gap on Q2D Fermi
surface (FS) [13,14], which is against the fully gaped chiral triplet scenario
[4]. Recently, the strongest experimental argument against the triplet
scenario of superconductivity in Sr2RuO4 was published [15], where strong drop
of the Knight shift in superconducting state of the above mentioned material
was experimentally discovered.

As seen from the above discussion, the situation with the chiral triplet
scenario of superconductivity in Sr2RuO4 is still rather controversial. The
goal of our Letter is two-fold. First, we improve and make our pioneering
argument [9] in favor of singlet superconductivity in Sr2RuO4 to be firm. The
point is that in Ref.[9] (see also recent Ref.[16]) we calculated the ratio
$H_{\parallel}(0)/(|dH^{GL}_{\parallel}/dT|_{T=T_{c}}T_{c})=0.75$, where
$|dH^{GL}_{\parallel}/dT|_{T=T_{c}}$ is the so-called Ginzburg-Landau (GL)
slope, for $s$-wave Q2D superconductivity and compared it with the
experimental one, 0.45-0.5 [17-19]. To make our argument against the chiral
triplet scenario to be firm, below we calculate the above mentioned ratio
exactly for the in-plane isotropic chiral triplet superconductor with ${\bf
d}$ vector order parameter [4,20],

${\bf d}={\bf z}\ \Delta_{0}\ (k_{x}\pm ik_{y}),$ (1)

and obtain even stronger inconsistency,

$H_{\parallel}(0)=0.815\ |dH^{GL}_{\parallel}/dT|_{T=T_{c}}\ T_{c},$ (2)

with the experimental values [17-19], where, to the best of our knowledge,
Eq.(2) is the first time obtained in the Letter. The second our goal is to
suggest one more test for chiral triplet superconductivity, which may already
exist in the slightly in-plane anisotropic Q2D triplet superconductor UTe2
[21] and, as we hope, will be discovered in some other Q2D compounds in the
future.

Note that superconducting phase (1) is not destructed by the Pauli
paramagnetic spin-splitting effects in a parallel magnetic field [9,20,22].
Moreover, in accordance with Refs.[23-28], it has also to restore
superconductivity in very high magnetic fields due to quantum effects of the
Bragg reflections from boundaries of the Brillouin zone. In this Letter, we
consider the so-called quasi-classical parallel orbital upper critical
magnetic field [29,16,30], $H_{\parallel}(0)$, which destroys
superconductivity in Q2D superconductors at low temperatures in an
intermediate region of the fields.

Let us consider a layered superconductor with the following in-plane isotropic
Q2D electron spectrum:

$\epsilon({\bf p})=\epsilon(p_{x},p_{y})-2t_{\perp}\cos(p_{z}c^{*}),\ \ \
t_{\perp}\ll\epsilon_{F},$ (3)

where

$\epsilon(p_{x},p_{y})=\frac{(p^{2}_{x}+p^{2}_{y})}{2m}\ ,\ \ \
\epsilon_{F}=\frac{p^{2}_{F}}{2m}\ .$ (4)

[In Eqs.(3) and (4), $t_{\perp}$ is the integral of overlapping of electron
wave functions in a perpendicular to the conducting planes direction, $m$ is
the in-plane electron mass, $\epsilon_{F}$ and $p_{F}$ are the Fermi energy
and Fermi momentum, respectively; $\hbar\equiv 1$.] In a parallel magnetic
field, which is applied along ${\bf x}$ axis,

${\bf H}=(H,0,0)\ ,$ (5)

it is convenient to choose the vector potential of the field in the form:

${\bf A}=(0,0,Hy)\ .$ (6)

For electron motion within the conducting planes and between the planes, we
make use of the the so-called Peierls substitution method [22] in the electron
energy spectrum (3) and (4):

$p_{x}\rightarrow-i\biggl{(}\frac{\partial}{\partial x}\biggl{)},\ \
p_{y}\rightarrow-i\biggl{(}\frac{\partial}{\partial y}\biggl{)},$ (7)

and

$p_{z}c^{*}\rightarrow
p_{z}c^{*}-\biggl{(}\frac{\omega_{c}}{v_{F}}\biggl{)}y,\ \
\omega_{c}=\frac{ev_{F}c^{*}H}{c},$ (8)

where $e$ is the electron charge, $c$ is the velocity of light.

After the Peierls substitutions in a magnetic field (7) and (8), the electron
Hamiltonian (3) and (4) becomes:

$\hat{H}=-\frac{1}{2m}\biggl{(}\frac{\partial^{2}}{\partial
x^{2}}+\frac{\partial^{2}}{\partial
y^{2}}\biggl{)}-2t_{\perp}\cos\biggl{(}p_{z}c^{*}-\frac{\omega_{c}}{v_{F}}y\biggl{)}.$
(9)

If we take into account that $\omega_{c}\ll\epsilon_{F}$, we can define the
following electron wave functions in the mixed, $(p_{x},y,p_{z})$,
representation:

$\displaystyle\Psi^{\pm}_{\epsilon}(x,y,z)=\ \exp(ip_{x}x)\ \exp[\pm
ip^{0}_{y}(p_{x})y]\ \exp(ip_{z}z)$
$\displaystyle\times\psi_{\epsilon}^{\pm}(p_{x},y,p_{z}),\ \
p^{0}_{y}(p_{x})=\sqrt{p_{F}^{2}-p^{2}_{x}},$ (10)

where energy $\epsilon$ is counted from the Fermi energy $\epsilon_{F}$. Note
that Eq.(9) is very general. For instance, it takes into account the quantum
effects for open electron orbits in a magnetic field - the Bragg reflections
from boundaries of the Brillouin zones (see, Refs. [23-28]). As we already
mentioned, below we calculate the so-called quasi-classical upper critical
magnetic field [29,16,30] and, thus, do not take into account possible
stabilization of superconductors in very high magnetic fields [23-28], which
is possibly observed in the Q2D superconductor UTe2.

It is easy to prove that, in this case, we can represent the electron
Hamiltonian (9) for the wave functions $\psi_{\epsilon}^{\pm}(p_{x},y,p_{z})$
in Eq.(10) as

$\displaystyle\biggl{\\{}\frac{1}{2m}\biggl{[}p^{2}_{F}\pm
2ip^{0}_{y}(p_{x})\frac{d}{dy}\biggl{]}-2t_{\perp}\cos\biggl{(}p_{z}c^{*}-\frac{\omega_{c}}{v_{F}}y\biggl{)}\biggl{\\}}$
$\displaystyle\times\psi_{\epsilon}^{\pm}(p_{x},y,p_{z})=(\epsilon+\epsilon_{F})\
\psi_{\epsilon}^{\pm}(p_{x},y,p_{z}).$ (11)

Note that it is easy to represent Eq.(11) in the following more convenient
way:

$\displaystyle\biggl{[}\pm
iv^{0}_{y}(p_{x})\frac{d}{dy}-2t_{\perp}\cos\biggl{(}p_{z}c^{*}-\frac{\omega_{c}}{v_{F}}y\biggl{)}\biggl{]}\psi_{\epsilon}^{\pm}(p_{x},y,p_{z})$
$\displaystyle=\epsilon\ \psi_{\epsilon}^{\pm}(p_{x},y,p_{z}),\ \ \
v^{0}_{y}(p_{x})=p^{0}_{y}(p_{x})/m.$ (12)

We point out that Eq.(12) is still too general. For instance, for a pure case,
it contains the above discussed quantum effects of an electron motion in a
magnetic field. As mentioned before, below, we study the quasi-classical case,
where, as shown in Refs. [16] and [30], it is possible to take into account
only linear term with respect to magnetic field. As a result, we obtain:

$\displaystyle\biggl{[}\pm
iv^{0}_{y}(p_{x})\frac{d}{dy}-2t_{\perp}\cos(p_{z}c^{*})-\biggl{(}\frac{2t_{\perp}\omega_{c}y}{v_{F}}\biggl{)}\sin(p_{z}c^{*})\biggl{]}$
$\displaystyle\times\ \psi_{\epsilon}^{\pm}(p_{x},y,p_{z})=\epsilon\
\psi_{\epsilon}^{\pm}(p_{x},y,p_{z}).$ (13)

In this Letter, we use the so-called Matsubara’s variant of the Green’s
functions, $G^{\pm}(i\omega_{n};p_{x};y,y_{1};p_{z})$, method [31] for
interacting electrons in the mixed $(p_{x},y,p_{z})$ representation. For non
interacting electrons in the magnetic field, equation for Matsubara’s Green’s
function, can be written as [32]

$\displaystyle\biggl{[}i\omega_{n}\mp
iv^{0}_{y}(p_{x})\frac{d}{dy}+2t_{\perp}\cos(p_{z}c^{*})+\biggl{(}\frac{2t_{\perp}\omega_{c}y}{v_{F}}\biggl{)}\sin(p_{z}c^{*})\biggl{]}$
$\displaystyle\times G^{\pm}(i\omega_{n};p_{x};y,y_{1};p_{z})=\delta(y-y_{1})\
,$ (14)

where $\omega_{n}$ is the so-called Matsubara’s frequency [31]. It important
that Eq.(14) can be solved analytically [16,25]. As a result, we obtain:

$\displaystyle
G^{\pm}(i\omega_{n};p_{x};y,y_{1};p_{z})=-i\frac{sgn(\omega_{n})}{v^{0}_{y}(p_{x})}\exp\biggl{[}\pm\frac{\omega_{n}(y-y_{1})}{v^{0}_{y}(p_{x})}\biggl{]}$
$\displaystyle\times\exp\biggl{[}\mp
i\frac{2t_{\perp}\cos(p_{z}c^{*})(y-y_{1})}{v^{0}_{y}(p_{x})}\biggl{]}$
$\displaystyle\times\exp\biggl{[}\mp
i\frac{t_{\perp}\omega_{c}(y^{2}-y^{2}_{1})}{v^{0}_{y}(p_{x})v_{F}}\sin(p_{z}c^{*})\biggl{]}.$
(15)

Let us consider the so-called Gor’kov’s gap equation for two component order
parameter (see, for example, Ref. [33]),

$\Delta({\bf k},{\bf q})=\Delta_{x}({\bf q})\delta_{x}({\bf
k})+\Delta_{y}({\bf q})\delta_{y}({\bf k}),$ (16)

where $[\phi_{x}({\bf k}),\phi_{y}({\bf k})]$ are basis functions of two-
dimensional representation of tetragonal group transforming as
$(k_{x},k_{y})$. The Gor’kov’s equation in this case can be written as [33]:

$\displaystyle\Delta({\bf k},{\bf q})=T\sum_{n}\int\frac{d^{3}{\bf
k_{1}}}{(2\pi)^{3}}\ V\sum_{i=x,y}\delta_{i}({\bf k})\delta_{i}({\bf k_{1}})$
$\displaystyle\times G^{\pm}(i\omega_{n};{\bf k_{1}})\
G^{\mp}(-i\omega_{n};{\bf-k_{1}+q})\Delta({\bf k_{1}},{\bf q}).$ (17)

Using Fourier transformations of the known Green’s functions (15), it is
possible to derive the so-called gap equation in the mixed representation,
determining the upper critical magnetic field for two component order
parameter (16). Below, we use the Q2D Fermi surface (3) with isotropic in-
plane electron 2D spectrum (4) and the following isotropic chiral triplet
electron electron superconducting interactions:

$\displaystyle V\sum_{i=x,y}\delta_{i}({\bf k})\delta_{i}({\bf k_{1}})$
$\displaystyle=2g[\cos\phi\cos\phi_{1}+\sin\phi\sin\phi_{1}]=2g\cos(\phi-\phi_{1}).$
(18)

As a result of lengthly but straightforward calculations, we obtain the
Gor’kov’s gap equation in the form:

$\displaystyle\Delta(\phi,y)=\int_{0}^{2\pi}\frac{d\phi_{1}}{2\pi}g\cos(\phi-\phi_{1})\int^{\infty}_{|y-y_{1}|>d|\sin\phi_{1}|}$
$\displaystyle\times\frac{2\pi
Tdy_{1}}{v_{F}|\sin\phi_{1}|\sinh\biggl{(}\frac{2\pi
T|y-y_{1}|}{v_{F}|\sin\phi_{1}|}\biggl{)}}$ $\displaystyle\times
J_{0}\biggl{[}\frac{2t_{\perp}\omega_{c}}{v^{2}_{F}|\sin\phi_{1}|}(y^{2}-y^{2}_{1})\biggl{]}\
\Delta(\phi_{1},y_{1})\ ,$ (19)

where $g$ is the effective electron coupling constant, $d$ is the cut-off
distance, $J_{0}(...)$ is the zero order Bessel function. In Eq.(19) the
superconducting gap $\Delta(\phi,y)$ depends on a center of mass of the BCS
pair, $y$, and on the position on the cylindrical FS (4), where $\phi$ and
$\phi_{1}$ are the polar angles counted from ${\bf x}$ axis.

Using the general Eq.(16), we need to seek two solutions of Eq.(19) in the
form,

$\Delta(\phi,y)=\Delta_{x}(y)\cos(\phi)+\Delta_{y}(y)\sin(\phi),$ (20)

for given direction of a magnetic field. Since we consider the isotropic 2D FS
(4) and the isotropic chiral triplet electron-electron interactions (18), it
is possible to make sure that the in-plane upper critical magnetic field does
not depend on direction of the field. Therefore, we can consider magnetic
field applied along ${\bf x}$ axis (5), as we suggested before. For such
magnetic field, it is easy to show that there are the following two solutions:

$\Delta_{1}(\phi,y)=\Delta_{x}(y)\cos(\phi)$ (21)

and

$\Delta_{2}(\phi,y)=\Delta_{y}(y)\sin(\phi),$ (22)

which obey the following integral equations:

$\displaystyle\Delta_{x}(y)=\int_{0}^{2\pi}\frac{d\phi_{1}}{2\pi}g\cos^{2}(\phi_{1})\int^{\infty}_{|y-y_{1}|>d|\sin\phi_{1}|}$
$\displaystyle\times\frac{2\pi
Tdy_{1}}{v_{F}|\sin\phi_{1}|\sinh\biggl{(}\frac{2\pi
T|y-y_{1}|}{v_{F}|\sin\phi_{1}|}\biggl{)}}$ $\displaystyle\times
J_{0}\biggl{[}\frac{2t_{\perp}\omega_{c}}{v^{2}_{F}|\sin\phi_{1}|}(y^{2}-y^{2}_{1})\biggl{]}\
\Delta_{x}(y_{1})\ ,$ (23)

and

$\displaystyle\Delta_{y}(y)=\int_{0}^{2\pi}\frac{d\phi_{1}}{2\pi}g\sin^{2}(\phi_{1})\int^{\infty}_{|y-y_{1}|>d|\sin\phi_{1}|}$
$\displaystyle\times\frac{2\pi
Tdy_{1}}{v_{F}|\sin\phi_{1}|\sinh\biggl{(}\frac{2\pi
T|y-y_{1}|}{v_{F}|\sin\phi_{1}|}\biggl{)}}$ $\displaystyle\times
J_{0}\biggl{[}\frac{2t_{\perp}\omega_{c}}{v^{2}_{F}|\sin\phi_{1}|}(y^{2}-y^{2}_{1})\biggl{]}\
\Delta_{y}(y_{1})\ ,$ (24)

correspondingly. Note that the solutions (21) and (22) degenerate in the
absence of a magnetic field. In the presence of the magnetic field (5) with
the vector potential (6), Eqs.(23) and (24) are not degenerated anymore and
correspond to different solutions, $\Delta_{x}(y)\neq\Delta_{y}(y)$, with
different upper critical magnetic fields, $H^{x}_{\parallel}\neq
H^{y}_{\parallel}$. The actual upper critical field will be the maximum value
of them. Therefore, below, we need to solve and study the both integral
equations, (23) and (24).

Here, we introduce more convenient variable of the integrations,

$y_{1}-y=z|\sin\phi_{1}|,$ (25)

and rewrite the integral equations (23) and (24) in the following ways:

$\displaystyle\Delta_{x}(y)=\int_{0}^{2\pi}\frac{d\phi_{1}}{2\pi}g\cos^{2}(\phi_{1})\int^{\infty}_{|z|>d}\frac{2\pi
Tdz}{v_{F}\sinh\biggl{(}\frac{2\pi T|z|}{v_{F}}\biggl{)}}$
$\displaystyle\times
J_{0}\biggl{[}\frac{2t_{\perp}\omega_{c}}{v^{2}_{F}}[z(2y+z|\sin\phi_{1}|)]\biggl{]}$
$\displaystyle\times\Delta_{x}(y+z|\sin\phi_{1}|)$ (26)

and

$\displaystyle\Delta_{y}(y)=\int_{0}^{2\pi}\frac{d\phi_{1}}{2\pi}g\sin^{2}(\phi_{1})\int^{\infty}_{|z|>d}\frac{2\pi
Tdz}{v_{F}\sinh\biggl{(}\frac{2\pi T|z|}{v_{F}}\biggl{)}}$
$\displaystyle\times
J_{0}\biggl{[}\frac{2t_{\perp}\omega_{c}}{v^{2}_{F}}[z(2y+z|\sin\phi_{1}|)]\biggl{]}$
$\displaystyle\times\Delta_{y}(y+z|\sin\phi_{1}|).$ (27)

Let us first study Eq.(26). Here, we derive the GL equation for (26) to
determine the so-called GL slope of the upper critical magnetic field,
$|dH^{x(GL)}_{\parallel}/dT|_{T=T_{c}}$. In order to derive the GL equation,
we expand the superconducting gap and the Bessel function in Eq.(26) with
respect to small parameter, $|z|\ll v_{F}/(\pi T_{c})$:

$\displaystyle\Delta_{x}(y+z|\sin\phi_{1}|)\approx\Delta_{x}(y)+\frac{1}{2}z^{2}\sin^{2}\phi_{1}\biggl{[}\frac{d^{2}\Delta_{x}(y)}{dy^{2}}\biggl{]},$
$\displaystyle
J_{0}\biggl{\\{}\frac{2t_{\perp}\omega_{c}}{v^{2}_{F}}[z(z|\sin\phi_{1}|+2y)]\biggl{\\}}\approx
1-\frac{4t^{2}_{\perp}\omega^{2}_{c}}{v^{4}_{F}}z^{2}y^{2}.$ (28)

Next step is to substitute the obtained expansions (28) into the integral gap
equation (26) and average over the polar angle $\phi_{1}$:

$\displaystyle-\frac{1}{8}\biggl{[}\frac{d^{2}\Delta_{x}(y)}{dy^{2}}\biggl{]}\int_{0}^{\infty}\frac{2\pi
T_{c}z^{2}dz}{v_{F}\sinh\biggl{(}\frac{2\pi T_{c}z}{v_{F}}\biggl{)}}$
$\displaystyle+y^{2}\Delta_{x}(y)\frac{4t^{2}_{\perp}\omega^{2}_{c}}{v^{4}_{F}}\int_{0}^{\infty}\frac{2\pi
T_{c}z^{2}dz}{v_{F}\sinh\biggl{(}\frac{2\pi T_{c}z}{v_{F}}\biggl{)}}$
$\displaystyle+\Delta_{x}(y)\biggl{[}\frac{1}{g}-\int_{d}^{\infty}\frac{2\pi
Tdz}{v_{F}\sinh\biggl{(}\frac{2\pi Tz}{v_{F}}\biggl{)}}\biggl{]}=0.$ (29)

Note that in the absence of a magnetic field the superconducting transition
temperature, $T_{c}$, obeys the following equation:

$\frac{1}{g}=\int_{d}^{\infty}\frac{2\pi
T_{c}dz}{v_{F}\sinh\biggl{(}\frac{2\pi T_{c}z}{v_{F}}\biggl{)}}\ .$ (30)

After substitution of Eq.(30) into Eq.(29), we obtain the following
differential equation:

$-\xi^{2}_{\parallel}\biggl{[}\frac{d^{2}\Delta_{x}(y)}{dy^{2}}\biggl{]}+\biggl{(}\frac{2\pi
H}{\phi_{0}}\biggl{)}^{2}\xi^{2}_{\perp}y^{2}\Delta_{x}(y)-\tau\Delta_{x}(y)=0,$
(31)

which is called the GL one. During the derivation of the GL equation,
determining the upper critical magnetic field slope near $T_{c}$,
$\tau=(T_{c}-T)/T_{c}\ll 1$, we introduce the so-called parallel and
perpendicular GL coherent lengths:

$\xi_{\parallel}=\frac{\sqrt{7\zeta(3)}v_{F}}{8\pi T_{c}},\ \ \
\xi_{\perp}=\frac{\sqrt{7\zeta(3)}t_{\perp}c^{*}}{2\sqrt{2}\pi T_{c}}.$ (32)

Note that $\zeta(x)$ is the so-called Riemann function [34],

$\int^{\infty}_{0}\frac{z^{2}dz}{\sinh(z)}=\frac{7}{3}\zeta(3),$ (33)

whereas $\phi_{0}=\frac{\pi c}{e}$ is the magnetic flux quantum. It is
important that $\xi_{\parallel}$ in Eqs.(31) and (32) is different from that
in Refs.[16,30], since in this Letter we consider two-component
superconducting gap. To obtain the parallel upper critical field for Eq.(26)
near Tc, it is necessary to solve the Scrödinger-like equation (31) and to
find its lowest energy level. Since mathematically Eq.(31) is equivalent to
the Scrödinger equation for a harmonic oscillator, we obtain the following
equation for the parallel upper critical field in the GL region, $\tau\ll 1$:

$H^{x(GL)}_{\parallel}(T)=\biggl{(}\frac{\phi_{0}}{2\pi\xi_{\parallel}\xi_{\perp}}\biggl{)}\tau=\biggl{[}\frac{8\sqrt{2}\pi^{2}cT^{2}_{c}}{7\zeta(3)ev_{F}t_{\perp}c^{*}}\biggl{]}\tau,$
(34)

which, as expected, differs from that in Refs.[16,30].

More complicated problem is to solve Eq.(26) at $T=0$ and, thus, to find the
upper critical magnetic field at zero temperature, $H^{x}_{\parallel}(0)$.
This is possible to do only by means of numerical calculations. To this end,
let us introduce more convenient variables,

$\tilde{z}=\frac{\sqrt{2t_{\perp}\omega_{c}}}{v_{F}}z,\ \ \
\tilde{y}=\frac{\sqrt{2t_{\perp}\omega_{c}}}{v_{F}}y,$ (35)

and rewrite the gap Eq.(26) for the case $T=0$:

$\displaystyle\tilde{\Delta}_{x}(\tilde{y})=g\int_{-\pi/2}^{\pi/2}\frac{d\phi_{1}}{\pi}\int^{\infty}_{d}\frac{d\tilde{z}}{\tilde{z}}J_{0}[\tilde{z}(2\tilde{y}+\tilde{z}\sin\phi_{1})]$
$\displaystyle\times
2\cos^{2}(\phi_{1})\tilde{\Delta}_{x}(\tilde{y}+\tilde{z}\sin\phi_{1})\ ,$
(36)

where

$\tilde{\Delta}_{x}(\tilde{y})=\Delta\biggl{(}\frac{v_{F}\tilde{y}}{\sqrt{2t_{\perp}\omega_{c}}}\biggl{)}.$
(37)

Here, we summarize procedure of the numerical solution of the integral Eq.(36)
and obtain the following new result:

$H^{x}_{\parallel}(0)=10.78\frac{cT^{2}_{c}}{ev_{F}t_{\perp}c^{*}}.$ (38)

Note that solution for the superconducting gap,
$\tilde{\Delta}_{x}(\tilde{y})$ of Eq.(36) is not of an exponential shape and
changes its sign several times in space, in contrast to the 3D case [29,35].
Using Eqs. (34) and (38), we have

$H^{x}_{\parallel}(0)=0.815\ |dH^{x(GL)}_{\parallel}/dT|_{T=T_{c}}\ T_{c}.$
(39)

Now, let us summarize the investigation of Eq.(24), which is done in the same
way as our study of Eq.(26). In particular, it is possible to show that the GL
field near transition temperature is equal to

$H^{y(GL)}_{\parallel}(T)=\biggl{[}\frac{8\sqrt{2}\pi^{2}cT^{2}_{c}}{7\zeta(3)\sqrt{3}ev_{F}t_{\perp}c^{*}}\biggl{]}\tau,$
(40)

which is $\sqrt{3}$ times smaller than in Eq.(34). Solution of the Eq.(27) at
$T=0$ also gives value of the upper critical field, which is approximately two
times less than the upper critical field (38), obtained for Eq.(36):

$H^{y}_{\parallel}(0)=0.5\times H^{x}_{\parallel}(0).$ (41)

On the above mentioned grounds, we can conclude that the solutions (21) of
Eq.(23) are essential for our calculations of the parallel upper critical
magnetic field in the chiral triplet in-plane isotropic Q2D superconductor.
Therefore, experimentally measured ratio has to be close to the value
(2),(39). It is important that the theory developed by us above cannot be
obtained in the framework of the Gor’kov [27] and Werthamer-Helfand-Hohenberg
[33] approaches. There are two reasons for that. First, electrons in a Q2D
superconductor in a parallel magnetic field move along open trajectories in
${\bf p}$ space, in contrast to closed electron orbits of Refs.[29,35].
Second, we have two-component order parameter, instead of one-component
[29,35].

As we already mentioned, in the candidate for the chiral triplet in-plane
isotropic superconductivity, Sr2RuO4, the corresponding experimental
coefficients [17-19] are almost two times smaller than the calculated in this
Letter (2), which is a strong argument against the chiral triplet scenario. In
this context, it is important that the parallel upper critical magnetic field
in the above mentioned compound is a good measurable quantity, unlike the
upper critical magnetic fields in some other Q2D superconductors. In our
opinion, the experimental value of the $H_{\parallel}(0)$ at low temperatures
are restricted by the paramagnetic spin-splitting effects, which are always
present in the singlet and some triplet superconducting phases. This point of
view is also supported by the first order nature of the phase transition in
parallel magnetic fields at low temperatures [17-19]. The recent Knight shift
experiments [15] also strongly demonstrate the existence of the paramagnetic
spin-splitting effects in superconducting phase of the Sr2RuO4. In conclusion,
let us discuss the physical model we have used for the calculations of the
parallel upper critical magnetic field. It is known [36] that in the Sr2RuO4
perpendicular to the conducting layers coherence length (32) is much larger
than the inter-plane distance. Therefore, we have used Eq.(36), instead of the
so-called Lawrence-Doniach model [37,38]. We have also make use of the fact
that metallic phase of the Sr2RuO4 is a good Fermi liquid [2,3].

The author is thankful to N.N. Bagmet (Lebed) for useful discussions.

∗Also at: L.D. Landau Institute for Theoretical Physics, RAS, 2 Kosygina
Street, Moscow 117334, Russia.

## References

  * (1) Y. Maeno, H Hashimoto, K Yoshida, S Nishizaki, T Fujita, J.G Bednorz, F Lichtenberg, Nature 372, 532 (1994).
  * (2) A.P. Mackenzie and Y. Maeno, Rev. Mod. Phys. 75, 657 (2003).
  * (3) C. Kallin, Rep. Progr. Phys. 75, 042501 (2012).
  * (4) T.M. Rice and M. Sigrist, J. Phys.: Condens. Matter 7, L643 (1995).
  * (5) K. Ishida, H. Mukuda, Y. Kitaoka, K. Asayama, Z. Q. Mao, Y. Mori, Y. Maeno, Nature 396, 658 (1998).
  * (6) J.A. Duffy, S. M. Hayden, Y. Maeno, Z. Mao, J. Kulda, and G. J. McIntyre, Phys. Rev. Lett. 85, 5412 (2000).
  * (7) G.M. Luke, Y. Fudamoto, K.M. Kojima, M.I. Larkin, J. Merrin, B. Nachumi, Y.J. Uemura, Y. Maeno, Z.Q. Mao, Y. Mori, H. Nakamura, M. Sigrist, Nature 394, 558 (1998).
  * (8) J. Xia, Y. Maeno, P.T. Beyersdorf, M.M. Fejer, and A. Kapitul’nik, Phys. Rev. Lett. 97, 167002 (2006).
  * (9) A.G. Lebed and N. Hayashi, Physica C 341-348, 1677 (2000).
  * (10) K. Machida, JPS Conf. Proc. 30, 011038 (2020).
  * (11) P.G. Bjrnsson, Y. Maeno, M.E. Huber, and K.A. Moler, Phys. Rev. B 72, 167002 (2005).
  * (12) C. A. Watson, A. S. Gibbs, A.P. MacKenzie, C.W. Hicks, and K. A. Moler, Phys. Rev. B 98, 094521 (2018).
  * (13) E. Hassinger, P. Bourgeois-Hope, H. Taniguchi, S. Rene de Cotret, G. Grissonnanche, M. S. Anwar, Y. Maeno, N. Doiron-Leyraud, and L. Taillefer, Phys. Rev. X 7, 011032 (2017).
  * (14) S. Kittaka, J. Phys. Soc. Jpn 87, 093703 (2018).
  * (15) A. Pustogow, Y. Luo, A. Chronister, Y.-S. Su, D. Sokolov, F. Jerzembeck, A. P. MacKenzie, C. W. Hicks, N. Kikugawa, S. Radhu, E. D. Bauer, and S. E. Brown, Nature 574, 72 (2019).
  * (16) A.G. Lebed, JETP Lett., 110, 173 (2019) [Pisma v Zh. Eksp. Teor. Fiz. 110, 163 (2019)].
  * (17) Z.Q. Mao, Y. Maeno, S. NishiZaki, T. Akima, and T. Ishiguro, Phys. Rev. Lett. 84, 991 (2000).
  * (18) S. Kittaka, T. Nakamura, Y. Aono, S. Yonezawa, K. Ishida, and Y. Maeno, Phys. Rev. B 80, 147514 (2009).
  * (19) S. Yonezawa, T. Kajikawa, and Y. Maeno, Phys. Rev. Lett. 110, 077008 (2013).
  * (20) V.P. Mineev and K.V. Samokhin, Introduction to Unconventional Superconductivity (Gordon and Breach Science Publisher, Australia, 1999).
  * (21) L. Jiao, S. Howard, S. Ran, Z. Wang, J. O. Rodriguez, M. Sigrist, Z. Wang, N. P. Butch, and V. Madhavan, Nature 579, 523 (2020).
  * (22) See, for example, book A.A. Abrikosov, Fundamentals of Theory of Metals (Elsevier Science, Amsterdam, 1988).
  * (23) A.G. Lebed, JETP Lett. 44, 114 (1986) [Pisma Zh. Eksp. Teor. Fiz. 44, 89 (1986)].
  * (24) N. Dupuis, G. Montambaux, and C.A.R. Sa de Melo, Phys. Rev. Lett. 70, 2613 (1993).
  * (25) A.G. Lebed and K. Yamaji, Phys. Rev. Lett. 80, 2697 (1998).
  * (26) V.P. Mineev, J. Phys. Soc. Jpn. 69, 3371 (2000).
  * (27) V.P. Mineev, JETP Lett. 111, 715 (2020).
  * (28) A.G. Lebed, Mod. Phys. Lett. B 34, 2030007 (2020).
  * (29) L.P. Gor’kov, Sov. Phys. JETP, 37(10), 42 (1960).
  * (30) A.G. Lebed, JETP Lett. 111, 239 (2020) [Pisma v ZhETF, 111, 249 (2020)].
  * (31) A.A. Abrikosov, L.P. Gor’kov, and I.E. Dzyaloshinskii, Methods of Quantum Field Theory in Statistical Mechanics (Dover, New York, 1963).
  * (32) L.P. Gor’kov and A.G. Lebed, J. Phys. (Paris) Lett. 45, L433 (1984).
  * (33) V.P. Mineev, Phys. Rev. B 89, 134519 (2014).
  * (34) I.S. Gradshteyn and I.M. Ryzhik, Table of Integrals, Series, and Products (6-th edition, Academic Press, London, United Kingdom, 2000).
  * (35) N.R. Werthamer, E. Helfand, and P.C. Hohenberg, Phys. Rev. 147, 295 (1966).
  * (36) Y. Maeno, Physica C 282-287, 286 (1997).
  * (37) L.N. Bulaevskii and A.A. Guseinov, Pisma Zh. Eksp. Teor. Fiz. 19, 742 (1974) [JETP Lett. 19, 382 (1974)].
  * (38) R.A. Klemm, A. Luther, and M.R. Beasley, Phys. Rev. B 12, 877 (1975).


Dan Stowell, Machine Listening Lab, Centre for Digital Music, Queen Mary
University of London, London, UK<EMAIL_ADDRESS>EPSRC fellowship
EP/L020505/1; NERC grant NE/L000520/1; ERASMUS+; AGH Research Grant No
15.11.130.642

# Automatic acoustic detection of birds through deep learning: the first Bird
Audio Detection challenge

Dan Stowell  Machine Listening Lab, Centre for Digital Music, Queen Mary
University of London.  Mike Wood  Ecosystems and Environment Research Centre,
School of Environment and Life Sciences, University of Salford.  Hanna Pamuła
Department of Mechanics and Vibroacoustics, AGH University of Science and
Technology, 30-059 Kraków  Yannis Stylianou  Computer Science Department,
University of Crete.  Hervé Glotin  LSIS UMR CNRS, University of Toulon, Inst.
Univ. de France

###### Abstract

Assessing the presence and abundance of birds is important for monitoring
specific species as well as overall ecosystem health. Many birds are most
readily detected by their sounds, and thus passive acoustic monitoring is
highly appropriate. Yet acoustic monitoring is often held back by practical
limitations such as the need for manual configuration, reliance on example
sound libraries, low accuracy, low robustness, and limited ability to
generalise to novel acoustic conditions. Here we report outcomes from a
collaborative data challenge showing that with modern machine learning
including deep learning, general-purpose acoustic bird detection can achieve
very high retrieval rates in remote monitoring data — with no manual
recalibration, and no pre-training of the detector for the target species or
the acoustic conditions in the target environment. Multiple methods were able
to attain performance of around 88% AUC (area under the ROC curve), much
higher performance than previous general-purpose methods. We present new
acoustic monitoring datasets, summarise the machine learning techniques
proposed by challenge teams, conduct detailed performance evaluation, and
discuss how such approaches to detection can be integrated into remote
monitoring projects.

Keywords: bird, sound, machine learning, deep learning, passive acoustic
monitoring

##  1 Introduction

Worldwide, bird populations have exhibited steep declines since the 1970s,
largely due to changes in land management [31, 29]. Bird populations are also
expected to change in number and distribution as the impacts of climate change
play out in coming years [16]. It is thus crucial to monitor avian
populations, for the purposes of conservation, scientific research and
ecosystem management. This has traditionally been performed via manual
surveying, often including the use of volunteers to help address the
challenges of scale [17, 19]. However, manual observation remains limited,
especially in areas that are physically challenging to access, or night-time
behaviour. Many bird species are readily detectable by their sounds, often
more so than by vision, and so with modern remote monitoring stations able to
capture continuous audio recordings the prospect opens up of massive-scale
spatio-temporal monitoring of birds [2, 10, 14, 25, 9, 20].

The first wave of such technology performs automatic recording but not
automatic detection, relying on manual after-the-fact study of sound
recordings [10, 9]. Later projects have employed some form of automatic
detection, which might be based on low-complexity signal processing such as
energy thresholds or template matching [38, 6], or on machine learning
algorithms [2]. However, when used for field deployments, practitioners face a
common hurdle. With the current state of the art, all methods require manual
tuning of algorithm parameters, customisation of template libraries and/or
post-processing of results, often necessitating some degree of expertise in
the underlying method. The methods are not inherently able to generalise to
new conditions—whether those conditions be differing species balances, noise
conditions, or recording equipment. Many methods also exhibit only moderate
accuracy, which is tolerable in small surveys but leads to unfeasible amounts
of false-negatives and -positives in large surveys [24]. A further common
limitation is the lack of robustness in particular to weather noise: sounds
due to rain and wind are commonly observed to dramatically affect detector
performance, and as a result surveys may need to treat weather-affected
recording periods as missing data.

Recent decades have witnessed extremely strong growth in the abilities of
machine learning. The advances are due to increased dataset sizes and
computational power but also due to deep learning methods that can learn to
make predictions in extremely nonlinear problem settings, such as speech
recognition or visual object recognition [22]. These methods have indeed been
applied to bioacoustic audio tasks [12, 32, 20], and it is clear that their
use could enable many organisations to work more cheaply and efficiently [18].
However, even with the strong performance of modern machine learning there
remain important questions about generalisability [20]. Machine learning
workflows use a “training set” of data from which the algorithm “learns”,
optionally a “validation set” used to determine when the learning has achieved
a satisfactory level, and then a “testing set” which is used for the actual
evaluation, to estimate the algorithm’s typical performance on unseen data.
Such evaluation is typically performed in matched conditions, meaning the
training and testing sets are drawn from the same pool of data, and thus
general properties of the datasets—such as the number of positive versus
negative cases—are expected to be similar. This enables users to test that the
algorithm can generalise to new items drawn from the same distribution.
However, in practical deployments of machine learning the new items are rarely
drawn from the same distribution: conditions drift, or the tool is applied to
new data for which no training data are available [37, 20]. This is one reason
that accuracy results obtained in research papers might not translate to the
field.

In order to address such problems, we designed a public evaluation campaign
focused on a highly-general version of the bird detection task, intended
specifically to encourage detection methods which are able to generalise well:
agnostic to species, and able to work in unseen acoustic environments. In this
work we present the new acoustic datasets we collated and annotated for the
purpose, the design of the challenge, and its outcomes, with new deep learning
methods able to achieve strong results despite the difficult task. We analyse
the submitted system outputs for their detection ability as well as their
robust calibration, we perform a detailed error analysis to inspect the sound
types that remain difficult for machine learning detectors, and apply the
leading system to a separate held-out dataset of night flight calls. We
conclude by discussing the new state of the art represented by the deep
learning methods that excelled in our challenge, the quality of their outputs
and the feasibility of deployment in remote monitoring projects.

##  2 Materials and methods

To conduct the evaluation campaign we designed a detection task to be
solved—specific but illustrative of general-purpose detection issues—gathered
multiple datasets and annotated them, and then led a public campaign
evaluating the results submitted by various teams. After the campaign, we
performed detailed analysis of the system outputs, inspecting questions of
accuracy, generality and calibration.

Our aim to facilitate general-purpose robust bird detection, agnostic to any
specific application, was key to how we designed the challenge specification.
The task of ‘detecting’ birds in audio can be operationalised in multiple
ways: for example, a system that emits a trigger signal in continuous time
representing the onset of each bird call, a system that identifies regions of
pixels in a spectrogram representation (time-frequency ‘boxes’), or a system
that estimates the number of calling individuals in a given time region [3].
For any given application, the choice of approach will depend on the
requirements for downstream processing. We selected an option which we
consider gave wide relevance, while also being a task that could be solved by
diverse methods, from simple energy detection, through to template matching or
machine learning. This was that audio should be divided into ten-second clips,
and the task specification would be to label each clip with a binary label
indicating the presence or absence of birds.

This approach quantises time such that any positive detection should be time-
localisable within $\pm 10$ seconds, which is sufficient for most purposes. It
also restricts such that there is no indication of the absolute number of bird
calls detected within a positively-labelled clip; however this is hard to
ground-truth accurately. Also via statistical ecology methods relative
abundances may still be inferred from the distribution of positive detections
[24]. A concrete advantage of this approach was that it was much quicker to
gather manual data annotations than would be the case for more complex
labelling.

###  2.1 Datasets

We gathered and annotated datasets from multiple sources. The purpose of this
was twofold: firstly to provide better evaluation of the generality of
algorithms, and secondly to provide challenge participants with development
data (e.g. to perform trial runs, or to train machine learning algorithms) in
addition to testing data.

We used audio data from remote monitoring projects and also from crowdsourced
audio recordings. These two dataset types differ from each other in many ways,
for example: remote monitoring audio was passively gathered, while
crowdsourced audio recordings were actively captured; the ratio of positive
and negative items was different; remote monitoring used fixed and known
recording equipment, while crowdsourcing used uncontrolled equipment. These
differences were deliberately introduced for their use in ensuring that the
challenge would be a strong test of generalisation.

Chernobyl dataset:

Our primary remote-monitoring dataset was collected in the Chernobyl Exclusion
Zone (CEZ) for a project to investigate the long-term effects of the Chernobyl
accident on local ecology [40, 11]. The project had captured over 10,000 hours
of audio since June 2015, across various CEZ environments, using Wildlife
Acoustics SM2 units. For the present work we selected six of recording
locations representing different environments (Table 1), and from those
selected a deterministic subsample: continuous 5-minute audio segments at
hourly intervals, across multiple days. Annotators manually labelled all time
intervals in which birds were heard (using Raven Pro software), and then we
split recordings and metadata automatically into ten-second segments. The
number of files per location is uneven because of limited annotator time,
giving us 6,620 items in total. No weather filtering or other rejection of
difficult regions was applied.

Codename | Habitat | Radiation
---|---|---
Buryakovka | (TBC) | Low
S2 | Deciduous forest | Medium
S11 | Meadow area | Medium
S37 | Pine forest | High
S60 | Shrub area | Low
S93 | Mixed forest | High
Table 1:  Recording locations in Chernobyl dataset.

Warblr dataset:

Our first crowdsourced dataset came from a UK-wide project Warblr. Warblr is a
software app available for Android and Apple smartphones, which offers
automatic bird species classification (using the method of [34]) for members
of the public via the submission of ten-second audio recordings. We extracted
a dataset of 10,000 audio files gathered in 2015–2016. The audio files were
thus actively collected, recorded on diverse mobile phone devices, and likely
to contain various human noise such as speech and handling noise. No
assumptions can be made that the data were a representative sample of
geographic locations, weather conditions, or bird species. Metadata for the
files indicated that they covered all the UK seasons, many times of day (with
a bias towards weekends and mornings) and geographically spread all around the
UK, with a bias toward population centres.

All recordings were selected that fell within the time window of available
data, limited to a maximum of 10,000. No selection or filtering of the data
were performed beyond the self-selection inherent in crowdsourcing.

Although the data included automatic estimates of which bird species were
present, these were not precise enough to be converted to ground-truth data
for the detection challenge. We thus performed manual annotation, with each
item being labelled as positive or negative according to the challenge
specification. Most items were single-annotated, although we were able to
obtain double-annotation for a small number of items, which allowed us to
estimate inter-rater reliability. Annotation was performed by experienced
listeners using headphone listening and a simple web interface.

freefield1010 dataset:

Our second crowdsourced dataset was an existing public dataset called
freefield1010 [35]. This consists of 7690 audio clips selected from the
Freesound online audio archive. To create this dataset the audio clips had
been selected such that they were labelled with the ‘field-recording’ tag in
the database, and trimmed to ten seconds duration. The data were of different
origin than Warblr: they covered a global geographic range, and the recording
devices used were almost never documented, but likely to include hand-held
audio recorders as used by pro-amateur sound recordists, as well as some
mobile phones and some higher-end recording devices. The Freesound database is
crowdsourced and thus largely uncontrolled.

These data did not come with labels suitable for our challenge; instead, each
item came with a set of freely-chosen tags to indicate the content generally.
We investigated the ‘birdsong’ tag, one of the most commonly used (2.6% of
items), but found this insufficiently accurate. We therefore had these audio
annotated through the same process as the Warblr data.

PolandNFC dataset:

The last dataset contains recordings from one author’s (HP) project of
monitoring autumn nocturnal bird migration. The recordings were collected
every night, from September to November 2016 on the Baltic Sea coast, near
Darlowo, Poland. We used Song Meter SM2 units with weather-resistant,
directional Night Flight Calls microphones from Wildlife Acoustics Inc.,
mounted on 3–5 m poles. The amount of collected data (>3200 h of recordings)
exceeded what human expert can annotate manually in reasonable time. Therefore
we subjectively chose and manually annotated the subset consisting of 22 half-
hour recordings from 15 nights with different weather conditions and
background noise including wind, rain, sea noise, insect calls, human voice
and deer calls. No other selection criterion or weather filtering was applied.
Manual annotation was performed by visual inspection of a spectrogram and
listening to the audio files. Only the passerine migrant calls were annotated
(voices in 5–10 kHz range), so it may happen that some low pitched bird
species (e.g. resident owl calls) obtained no-bird label. However, such calls
were extremely rare in described dataset, so this did not bias results
significantly.

The selected recordings were then split into 1 s clips. The chosen clip
length, different from other datasets, was chosen due to nocturnal bird calls
typical duration (10–300 ms).

More details about the dataset structure (and analysis of the effect of audio
clip duration) may be found in [30].

All sound files used in the public challenge were normalised in amplitude and
saved as 16-bit single-channel PCM at 44.1 kHz sampling rate,111Normalisation
via the sox tool using gain -n -2. and are available under open licences (see
Data Accessibility statement).

Our data annotation process was designed after early community discussions
about how the challenge should be conducted. We resolved that the annotations
should reflect plausible annotation conditions as encountered in applications.
In particular, they should be well-annotated, yet any mislabellings discovered
in the groundtruth data as the challenge progressed should not be eliminated,
since training data in practice do contain some errors and are not subject to
the same scrutiny as in a data challenge. A good detection algorithm must be
able to cope with a small level of imprecision in the annotation data.

However, it was possible at the end of the challenge to perform further
analysis and inspect the degree of machine errors and human errors. To make
good use of annotator time we used mismatch between automatically inferred
decisions and manual annotations to search for mislabelled items in the
dataset. For this we used the mean decision from the strongest three
submissions to the challenge. All items in the testing set with a negative
groundtruth label but a mean decision greater than 0.2, and all items with a
positive groundtruth label but a mean decision less than 0.3, were examined
and relabeled if needed. One might expect the threshold for re-validation to
be 0.5: the asymmetry is because systems generally exhibited a bias towards
low confidence, as will be seen later (Section 3.1). This re-validation
process refined the testing set, but also allowed us to calculate a value for
the inter-rater agreement for manual annotation, which we will express as an
AUC for comparison against the results of automatic detection. Note that the
re-validation process requires the time of expert listeners, and so it was not
feasible to perform mass crowdsourcing on the whole collection.

###  2.2 Baseline classifiers

To establish baseline performance against which to compare new methods, we
used two existing machine-learning based classification algorithms.

The first (code-named smacpy) was the same baseline classifier as used in a
2013 challenge on “detection and classification of acoustic scenes and events”
(“DCASE”) [33]. This baseline classifier represents a well-studied method used
in many audio classification tasks: audio is converted to a representation
called mel frequency cepstral coefficients (MFCCs), and the distributions of
MFCCs are then modelled using Gaussian mixture models (GMMs). Such an approach
is simple, efficient and adaptable to many sound recognition tasks. It has
been superseded for accuracy in general-purpose sound recognition by more
advanced methods [33]. We selected it to provide a common low-complexity
baseline, and also because its simplicity meant it might successfully avoid
overfitting to the training data, i.e. avoid becoming overspecialised, given
that the training and test data would have different characteristics.

The second baseline (code-named skfl) was a recent and more powerful
classifier introduced for bird species recognition [34]. This was the
strongest audio-only bird species classifier in a 2014 international
evaluation campaign. Relative to smacpy it innovated in both the feature
representation and the classification algorithm. The feature representation
was an automatically-learnt data transformation: two layers of “unsupervised
feature learning” applied to Mel spectrogram input. For classification the
method used a random forest, an ensemble learning method based on decision
trees that has emerged as powerful and robust for many tasks in machine
learning [4]. Both of these components are known to work well with difficult
classification scenarios, such as multi-modal classes, unbalanced datasets,
and outliers. We thus selected this second baseline as a representative of
modern and flexible machine learning, designed for bird sounds. In principle
it could be more vulnerable to overfitting than the first baseline. However,
because of the inherent difficulty of the task, we expected skfl to perform
more strongly than smacpy, and to provide a high performing baseline.

Python code for each of the baseline classifiers has previously been published
[34, 33]. Initial results using the baseline classifiers were published online
as a guide to the challenge participants.222http://machine-
listening.eecs.qmul.ac.uk/2016/10/bird-audio-detection-baseline-
generalisation/

###  2.3 The public challenge

Conduct of the public challenge followed the design of previous successful
contests on related topics [33, 12]. Having already determined the need for
the challenge from remote monitoring literature and practitioners [36], we
announced intentions and led community discussion on the task design via a
dedicated mailing list.

We collated our audio datasets into three portions: development data to be
publicly shared (7,690 items of freefield1010 plus 8,000 items from Warblr),
testing data whose true labels were to be kept private (10,000 items from
Chernobyl plus 2,000 items from Warblr), and a separate set not used for the
challenge itself but for further study of algorithm generalisation
(PolandNFC). Providing two distinct development data sets allowed participants
to test generalisation from one to the other, as part of their own algorithm
development process, while keeping some datasets fully private allowed us to
evaluate generalisation without concern that algorithms might have been
configured to the specifics of a given dataset.

The public development datasets were distributed in September 2016, both audio
and groundtruth annotations. Teams could then begin to develop methods and
train their systems. In December 2016 we released the testing data (audio
only), with a one month deadline for the submission of inferred detection
labels. The short time horizon of one month was intended to minimise the
opportunity for overly adapting the methods to the characteristics of the
testing data. During this period, teams could make multiple submissions, but
limited to a maximum of one per day. "Preview" results, calculated from 15% of
the testing data, were provided in an interactive online plot, in order to
give approximate feedback on performance (Figure 2).

Participants were allowed to run their software on their own machines and then
to submit merely the outputs (as opposed to the software code), which our
online system would then score without revealing the groundtruth labels for
the testing data. Given that this open approach has potential
vulnerabilities—such as recruiting manual labellers rather than developing
automatic methods—we required the highest-scoring teams to send in their code
which we inspected and re-ran on our own systems, to ensure a fair outcome.

Aside from intrinsic motivation, incentives for participants were cash prizes:
one for the strongest scoring system , and one judges’ award decided according
to the use of interesting or novel methodology. This was done to stimulate
conceptual development in the field, as opposed to the mere application of
off-the-shelf deep learning. Participants were further required to submit
technical notes describing their method, and later were invited to submit
peer-reviewed conference papers to a special session at the European Signal
Processing Conference (EUSIPCO) 2017. Of these, 8 challenge-related papers
were accepted and presented.

The challenge organisation was thus designed to achieve the following: public
benchmarking of methods against a common task and data, specifically tailored
to fully-automatic configuration-free bird detection in unseen conditions;
public documentation of the methods used to achieve leading results; and
greater attention from machine learning researchers on data analysis tasks in
environmental sound monitoring.

###  2.4 Evaluation

Our goal was to evaluate algorithms for their ability to perform general-
purpose bird detection, within the selected format of binary decisions for
ten-second audio clips. A strong algorithm is one that can reliably separate
the two classes “bird(s) present” and “no bird present”. However, since our
evaluation was general and not targeted at a specific application, we wished
to generalise over the possible tradeoffs of precision versus recall (the
relative cost of false-positive detections versus false-negative detections).
This strongly motivated our design such that participants should return
probabilistic or graded outputs—a real-valued prediction for each audio clip
rather than simply a 1 or 0—and our evaluation would use the well-studied area
under the ROC curve (AUC) as the primary quality metric. The AUC measure has
numerous qualities that make it well-suited to evaluation such classification
tasks: it generalises over all the possible thresholds that one might apply to
real-valued detector outputs; unlike raw accuracy, it is not affected by
“unbalanced” datasets having an uneven mixture of positive and negative items;
chance performance for AUC is always 50% irrespective of dataset; and it has a
probabilistic interpretation, as the probability that a given algorithm will
rank a randomly-selected positive instance more highly than a randomly-
selected negative instance [8].

The ranking interpretation just mentioned highlights another aspect of the AUC
statistic: it treats detector outputs essentially as ranked values and is thus
invariant to any monotonic mapping of the outputs, in particular to whether
the outputs are well-calibrated probabilities or not. Well-calibrated, in this
context, implies that when a detector outputs "0.75" for an item, this matches
the empirical probability that in three out of four such cases the item is
indeed a positive instance [28].

Thus AUC does not evaluate calibration. But the need for calibration depends
on the application: if a detector is being used to select a subset of strong
detections, or to rank items for further manual inspection, there may be no
need for calibration. However if the detections are to be used in some
probabilistic model e.g. for modelling a population distribution, it is
desirable for a detector to output well-calibrated probabilities. If a
detector performs well in the sense evaluated by AUC, then its outputs can be
mapped to probabilities by a post-processing step [28]. Hence, we used AUC as
our primary measure of quality, and separately we analysed the calibration of
the submitted algorithms using the method of calibration plots, which are
histogram plots comparing outputs against empirical probabilities [28].

###  2.5 Further analysis via PolandNFC dataset

After the challenge concluded we took the highest-scoring algorithm and
applied it to the PolandNFC dataset, an unseen and difficult dataset
containing night flight calls, often brief and distant. We used this in two
ways: (a) trained on a held-out portion (72.7 %) of the PolandNFC data and
tested on the remaining 27.3%; (b) trained using the main challenge
development data and again tested on the 27.3% of the PolandNFC data.

This allowed us to evaluate further the generalisation capability learnt by
the network. For variant (a), the training dataset consisted of sixteen 30 min
recordings collected over 11 nights (split into 28,784 1 s clips), and the
testing dataset had six recordings from 4 nights (split into 10,793 1 s
clips). Training and testing recording dates were disjoint sets. The testing
set was held the same across variant (a) and (b) to ensure comparability of
results. Important to note is the specific structure of the PolandNFC
dataset—it contains mostly negatively annotated examples (only 3.2 % for
testing and 1.6 % for training set were positive).

##  3 Results

In re-validating the testing set we examined those items with the strongest
mismatch between manual and automatic detection, to determine which was in
error: 500 presumed negative and 1243 presumed positive items. This showed
inter-rater disagreement in 16.6% of such cases—predominantly, the most
ambiguous cases with barely-audible bird sounds with amplitude close to the
noise threshold. Note that this percentage is not representative of
disagreement across the whole dataset, but only on the ‘controversial’ cases.
We also observed that a strong mismatch according to the automatic detectors
did not necessarily imply human mislabelling: some perceptually obvious data
items could be consistently misjudged by algorithms. We will discuss algorithm
errors further below. Through re-validation the inter-rater reliability,
measured via the AUC, was measured as 96.7%. This value provides an
approximate upper limit for machine performance since it reflects the extent
of ambiguity in the data according to human listeners’ perception.

The two baseline classifiers gave relatively good performance on the
development data, the strongest at over 85% AUC in matched conditions, but
generalised poorly. The simpler GMM-based baseline classifier showed
consistently lower results than the more advanced classifier, as expected. It
also showed strong resistance to overfitting in the sense that its performance
on its training set was a very good predictor of its performance on a matched-
conditions testing set. However, this was not sufficient to allow it to
generalise to mismatched conditions, in which its performance degraded
dramatically (Figure 1). The more advanced baseline classifier also degraded
when tested in mismatched conditions, though to a lesser extent, attaining 79%
AUC.

Figure 1: Tests using baseline classifiers on the two development datasets.
The middle column shows testing results under matched conditions, as commonly
reported in machine learning; the rightmost column shows testing results under
mismatched conditions, i.e. training with one dataset and testing with the
other.

###  3.1 Challenge outcomes

Thirty different teams submitted results to the challenge, from various
countries and research disciplines, with many submitting multiple times during
the one-month challenge period (Figure 2). Around half of the teams also
submitted system descriptions, of which the majority were based on deep
learning methods, often convolutional neural networks (CNNs) (Figure 3). To
preprocess the audio for use in deep learning, most teams used a spectrogram
representation, often a "mel spectrogram", which is a spectrogram with its
frequency axis warped to an approximation of human nonlinear frequency-band
sensitivity. Many teams also used data augmentation, meaning that they
artificially increased the amount of training data by copying and modifying
data items in small ways, such as adding noise or shifting the audio in time.
These strategies are in line with other work using machine learning for bird
sound [12, 32].

Figure 2: Timeline of preview scores attained by every submission for each
team. Error bars are estimated by bootstrap sampling.

Figure 3: Overview of the methods used by the submitted systems, and their
scores.

Most teams were able to achieve over 80% AUC, but none over 90%: the strongest
score was 88.7% AUC, attained by team ‘bulbul’ (Thomas Grill) on the final day
of challenge submission (Figure 4). The team has given further details of
their approach in a short conference publication [13] and with open-source
code available
online.333https://jobim.ofai.at/gitlab/gr/bird_audio_detection_challenge_2017/tree/master
Inspecting the highest-scoring entries, and estimating their variation through
bootstrap sampling, we found that four teams’ results were within the
confidence interval of the highest score (Figure 5).

Figure 4: Final scores attained by the highest-performing submission for each
team. Error bars are estimated by bootstrap sampling. Figure 5: Final scores
attained by the 20 highest-scoring submissions. Error bars are estimated by
bootstrap sampling. Figure 6: ROC plots for the systems attaining the four
highest and one lowest AUC score.

AUC scores are summary statistics of ROC plots. Inspecting the detail of the
ROC plots for these four highest-scoring systems and for the lowest-scoring
system (Figure 6), we find some tendency for the curves to be biased towards
an asymmetry (close to the left-hand edge of the plot but not the upper edge).
This implies a spread of ‘difficulty’ for the test items: there were some
positive items that were easy to detect without incurring extra false-
positives as a side-effect, while many remained difficult to detect. The most
balanced of the ROC plots inspected was that of ‘cakir’, implying that this
system had a more balanced distribution of its discriminative power across the
easy and difficult cases.

Since the testing data consisted of items from multiple ‘sites’—i.e. known
sites in the Chernobyl Exclusion Zone, plus the Warblr (UK) data considered as
a separate single site—we were able to calculate the AUC scores on a per-site
basis (Figure 7, left). These showed a strong site-dependency of algorithm
performance. Whereas Warblr data could be detected with an AUC of over 95%,
Chernobyl sites showed varying difficulty for the detectors overall, some as
low as 80% even for the leading algorithms. However, the overall AUC was very
highly predictive of the average of the per-site AUCs (Pearson $R^{2}=0.76$;
Figure 7, right). Note that the two AUC calculations are not independent and
so some correlation is expected. The observed correlation validates that the
overall AUC is usable as a summary of the per-site performance. The ‘bulbul’
system remained the strongest performer even under the per-site analysis. The
rank ordering of systems was not highly preserved: for example the second-
placed ‘cakir’ system would have been ranked tenth if using the average per-
site AUC.

Figure 7: Performance (AUC) of submitted algorithms analysed on a per-site
basis. Warblr data (from around the UK) is treated as one site, while the
other sites are recorders in the CEZ. Left-hand plot shows the per-site
results; right-hand plot shows how the AUC scores compare when calculated over
the whole pooled dataset, versus as a mean of the stratified per-site AUCs.

Applying PCA to the system outputs, we recovered an abstract ‘similarity
space’ of systems, which showed that some of the lowest-scoring systems formed
two outlier clumps in terms of their predictions, while the stronger systems
formed something of a continuum (Figure 8). The very strongest scoring systems
did not cluster tightly together, indicating that there remains some diversity
in the strategies implicit in these high-performing detectors.

Figure 8: Similarity space comparing the top-scoring submission by each team
(PCA projection of the submitted predictions after rank-transformation).
Submissions are close together if their predictions were similar, irrespective
of their accuracy. Submissions obtaining higher AUC scores are darker in
colour.

We measured calibration curves separately for the Warblr and Chernobyl testing
data (Figure 9). Calibration was generally better for Warblr, as one might
expect given the availability of Warblr training data. Notably, the highest-
scoring submission ‘bulbul’ had by far the worst calibration on the Chernobyl
data: around 80% of cases it assigned a prediction value of 0.25 were indeed
positive (versus around 30% on the Warblr data). The second-highest-scoring
submission ‘cakir’ exhibited quite different behaviour, remaining relatively
well calibrated even when assessing the unseen Chernobyl data.

Figure 9: Calibration plots for the strongest submission by each team,
separately for the Warblr test data (first plot) and Chernobyl test data
(second plot). For legibility we have limited this to the submissions
attaining at least 80% AUC. A submission whose outputs are well-calibrated
probabilities should yield a line close to the identity diagonal.

###  3.2 Error analysis

Category | False positives | False negatives
---|---|---
Clear | 1 | 68
Dontknow | 7 | 40
Faint (e.g. v distant) | 0 | 179
Short call | 0 | 69
Noise-masking (inc wind, river) | 0 | 67
Insect | 26 | 52
Human (speech, laughter, tv, imitation) | 31 | 13
Rain (inc drops) | 26 | 5
Unusual bird sound | 0 | 29
Misc distractor | 0 | 11
Misc mammal | 2 | 0
Table 2:  Inferred reasons for mistakes made by the strongest-performing
systems, annotated for the 500 items for which the systems showed the
strongest deviation from groundtruth. Note that the count data sum to more
than 500, since multiple reasons could potentially be attributed to each item.
“Clear” means the item was perceptually clear and should have been correctly
labelled; “Dontknow” means that no obvious reason for a mistake is evident,
even if the item is not particularly clear; all other rows are categories of
presumed reasons for machine errors.

We inspected the 500 data items for which the predictions of the strongest
systems exhibited mismatch with the re-validated ground truth, to characterise
typical errors made by even the strongest machine learning systems in bird
audio detection (Table 2). Such inspection is heuristic, relying on perceptual
judgment to estimate the causes of errors; however repeated tendencies give us
indications about the performance of the current state of the art.

For false negatives, by far the most common observation was that positive
items contained very faint bird sound (e.g. distant), often needing multiple
listens to be sure it was present. These faint sounds had low SNR and were
often also quite reverberated. Low SNR was also a factor in the third most
common presumed cause, noise masking. This category included general broadband
‘pink’ noise sources including wind and rivers. There were other more specific
categories of sound that appeared to act as masker or distractor causing
systems to overlook the bird sound: insect noise was common in the CEZ data,
while human sounds such as speech, whistling, or TVs were present in the
Warblr data. The second most common presumed cause of false negatives was
however the presence of extremely short calls: often a single “chink” sound,
which might perhaps be overlooked or confused with rain-drop sounds. Some
sounds were presumed to be missed because they were unusual for the dataset
(e.g. goose honking), although this was not seen as a major factor.

False positives occurred at a much lower rate in the top 500 most mismatched
items. They appeared to be caused in roughly equal proportion by insects,
human sounds, and rain sounds (individual drops or diffuse rainfall).

###  3.3 Further analysis via PolandNFC dataset results

We then applied the highest-scoring method (‘bulbul’) to the separate and
unseen acoustic monitoring dataset PolandNFC. The bulbul algorithm performed
its inference in two stages: the first stage applied the pre-trained neural
network to make initial predictions; the second stage allowed the neural
network to adapt to the observed data conditions, by feeding back the most
confident predictions as new training data [13]. We evaluated the outputs from
each stage (Table 3). AUC results were of high quality and were most strongly
affected by the choice of training data, the matched-conditions training
yielding much more accurate predictions. The second stage adaptation offered
some improvement in the case where the training set came from mismatched
conditions. However in matched-conditions training the second stage actually
incurred a slight reduction in performance. Calibration plots for this test
show that the detector was also better calibrated when trained in matched
conditions, and that the second stage re-training did not have a strong effect
on calibration (Figure 10).

| AUC (%)
---|---
Training data | First stage | Second stage
Challenge (mismatched) | 83.9 (+/- 0.4) | 87.76 (+/- 0.07)
PolandNFC (matched) | 94.95 (+/- 0.01) | 93.78 (+/- 0.01)
Table 3:  Performance obtained when applying the highest-scoring challenge
system to remote monitoring audio data from Poland, average of results from
two runs of the system.

Figure 10: Calibration plots obtained when applying the highest-scoring
challenge system to remote monitoring audio data from Poland (cf. Figure 9).
First plot: Challenge (mismatched) training; second plot: PolandNFC (matched)
training.

##  4 Discussion

Two broad observations emerge from this study:

  1. 1.

Machine learning methods, primarily deep learning, are able to achieve very
high recognition rates on remote monitoring acoustic data, despite weather
noise, low SNRs, wide variation in bird call types, and even with mismatched
training data. The AUC results presented here are a dramatic advance in the
state of the art, and machine learning methods are of practical use in remote
monitoring projects.

  2. 2.

However, there remains a significant gap between performance in matched
conditions and in mismatched conditions. True generalisation remains
difficult, and further work is needed. Projects are thus recommended to obtain
some amount of matched-conditions training data where possible, and to treat
automatic detection results with some caution especially with regard to the
calibration of the outputs if they are to be treated as probabilities or if a
fixed detection threshold is used. (Post-processing such as Platt scaling can
ameliorate calibration issues [28].) If a ranked-results approach is used
(e.g. keeping the strongest $N$ detections), which circumvents questions of
calibration, then performance can remain strong even in mismatched conditions
(Fig 7; Table 3).

In practical applications there are differing tradeoffs of precision versus
recall, of false negatives versus false positives. The AUC statistics
summarise over these but tell us that whatever position on the ROC curve is
chosen, the improvement in the state of the art which we report can give a
much better tradeoff (Fig 6), which corresponds for example to a much lower
amount of manual postprocessing time in filtering out false-positive results
[30].

The strongest machine learning methods in this study were convolutional and/or
recurrent neural nets (CNNs, RNNs, or CRNNs), as has been observed in other
domains [22]. In order to ensure methods could work in conditions different
from those in the training data, various participants explored self-
adaptation, in which a trained network is fine-tuned upon exposure to the new
conditions (without needing any additional ground-truth information) [13, 5,
1]. Participants reported mixed results of this, some observing no benefit. We
found little benefit of self-adaptation for matched conditions; however, in
cases where matched-conditions training data is not available, we found that
it can reduce the adverse effect of the mismatch (Table 3).

A further practical question is the feasibility of implementation on low-power
devices for long-term deployment in the field. Deep learning experiments often
require hardware acceleration, primarily for the training phase. After
training, deep learning algorithms can be deployed onto smaller embedded units
[23]. However, the self-adaptation methods considered here are essentially
additional rounds of training, albeit conducted with unlabelled data, and thus
would incur quite some cost for use in the field. A pragmatic version of this
would be to perform training or ‘pre-training’ using mismatched data, then
collecting a small amount of matched data from the target field conditions to
perform self-adaptation, before fixing the algorithm parameters for use on-
device.

Our error analysis showed that the most common error for the present
generation of algorithms appears to be false-negatives due to the failure to
detect faint/distant bird sound. The next most salient concern is robustness
to masking noise (wind, river, insects, speech). There were many entries which
were very hard to decide, as a listener: faint, short, and masked sounds
together constitute a large portion of items. This perceptual difficulty,
reflected in inter-rater disagreements, reminds us that some cases may be
inherently ambiguous and thus may always be difficult for machine recognition.
However, there is still a fair proportion (more than 1 in 10 of the top 500
items inspected) for which the sound recording was judged to be perceptually
clear, meaning that the reason for those false negatives is due to a detector
failing to model bird sound correctly, providing scope for algorithm
improvements.

A further specific cause of detector errors stems from the ambiguity between
very short ‘chink’ bird calls and sounds such as individual rain drops which
have similar effects in a spectrogram. A related issue was observed in bats,
with the very short calls of species in the Myotis genus being the most
difficult to disambiguate according to Walters et al. [39]. If the observable
attributes of multiple sources overlap entirely then it is not possible to
distinguish them even in principle. However, at least in our case human
listeners can tell the difference, whether from context or from fine detail of
signals. We thus expect that future work on higher-resolution input
features—such as waveform data rather than spectrogram data—will be able to
improve on this issue.

Hutto and Stutzman [15] previously performed an analysis of human sound
detection of birds. Their comparison was between humans and “autonomous
recording units”: however, note that in the latter case the detection was
performed manually by inspecting spectrograms and listening to recordings,
contrasted against a human listener in the field. Their results are thus not
directly comparable to ours; however they too found that distant bird sounds
were the predominant cause of missed detections for remote sensing units.
Furnas and Callas [10] likewise studied in-field versus audio-based detection
using manual annotation, with similar results. They noted that detection
probability could vary according to situational factors such as elevation and
tree canopy cover. Digby et al. [7] compared evaluated automated detection in
audio against in-field manual detection, for a single species (the little
spotted kiwi, Apteryx owenii), finding detection rates of around 40% with a
relatively simple detection algorithm; despite this, they concluded that the
high efficiency of automatic methods led to a large reduction in person-hours
and thus was recommended. They found that wind noise exerted a larger
influence on automatic detection than on manual detection.

Overall, our study design via a data challenge has been successful in moving
forward the state of the art in acoustic remote monitoring. The design as a
binary classification task, evaluated by AUC, is recommended as a way to
generalise over some diversity in requirements among remote monitoring
projects, with the calibration analysis as an important addition to AUC
evaluation. The use of multiple test sets sourced from different projects is a
robust approach for general-purpose evaluation of algorithms, and we further
recommend the use of per-site stratified AUC to account for per-site
differences.444The second edition of the Bird Audio Challenge, launched at
time of writing, incorporates these recommendations, using per-site stratified
AUC as well as adding further test sets to the challenge.
http://dcase.community/challenge2018/task-bird-audio-detection This
complements the task-specific evaluation that a well-resourced individual
project should undertake (cf. Knight et al. [20]). Going beyond the “yes/no”
binary classification task, in some cases is desirable to identify individual
bird calls: the binary classification paradigm can in fact enable this,
through a procedure of “weakly supervised learning” [26, 27, 21]. In future
evaluations we recommend the exploration of such approaches, combining broad-
scale detection with the elucidation of finer detail.

## Acknowledgments

We would like to thank Paul Kendrick for preparation and annotation of
Chernobyl data, Luciana Barçada for annotation of Chernobyl data, and Julien
Ricard for programming and administering the challenge submission website. We
also thank the many challenge participants for their enthusiastic effort.

## Author contributions

DS, MW, YS and HG designed the data challenge study and its analysis; DS, MW
and HP provided datasets; HG led the creation of the challenge submission
website; DS and HP performed tests of machine learning systems; DS, HG and HP
analysed the results; DS led the writing of the manuscript, with some sections
by HP. All authors contributed critically to the drafts and gave final
approval for publication.

## Data accessibility

  * •

Development datasets: both audio and annotations are available under CC-BY-4.0
licences.
warblrb10kaudio: https://archive.org/details/warblrb10k_public
ff1010bird audio: https://archive.org/details/ff1010bird
Annotations: https://doi.org/10.6084/m9.figshare.3851466.v1

  * •

Testing data (Chernobyl/warblrb10k): audio public, but annotations held back
for future challenges.
Audio: https://archive.org/details/birdaudiodetectionchallenge_test

  * •

PolandNFC data: held back, in preparation for future challenge.

  * •

Source code for baseline classifiers:
GMM ‘smacpy’ classifier (MIT licence): https://github.com/danstowell/smacpy.
skfl feature-learning is available as supplementary information in Stowell and
Plumbley [34] (MIT licence).

  * •

Source code for the online submission website (MIT licence):
https://github.com/julj/submission

  * •

Source code and papers for various of the systems submitted by challenge teams
are available via http://c4dm.eecs.qmul.ac.uk/events/badchallenge_results/
(various licences).

## References

  * Adavanne et al. [2017] Adavanne, S., Parascandolo, G., Drossos, K., Virtanen, T. et al. (2017) Convolutional recurrent neural networks for bird audio detection.  In Proceedings of EUSIPCO 2017.  Special Session on Bird Audio Signal Processing.
  * Aide et al. [2013] Aide, T. M., Corrada-Bravo, C., Campos-Cerqueira, M., Milan, C., Vega, G. and Alvarez, R. (2013) Real-time bioacoustics monitoring and automated species identification.  PeerJ, 1, e103.
  * Benetos et al. [2018] Benetos, E., Stowell, D. and Plumbley, M. D. (2018) Approaches to complex sound scene analysis.  In Computational Analysis of Sound Scenes and Events, chap. 8, 215–242. Springer.
  * Breiman [2001] Breiman, L. (2001) Random forests.  Machine Learning, 45, 5–32.
  * Çakır et al. [2017] Çakır, E., Parascandolo, G., Heittola, T., Huttunen, H. and Virtanen, T. (2017) Convolutional recurrent neural networks for polyphonic sound event detection.  IEEE Transactions on Audio, Speech and Language Processing, Special Issue on Sound Scene and Event Analysis.  ArXiv preprint arXiv:1702.06286.
  * Colonna et al. [2015] Colonna, J. G., Cristo, M., Salvatierra, M. and Nakamura, E. F. (2015) An incremental technique for real-time bioacoustic signal segmentation.  Expert Systems with Applications.
  * Digby et al. [2013] Digby, A., Towsey, M., Bell, B. D. and Teal, P. D. (2013) A practical comparison of manual and autonomous methods for acoustic monitoring.  Methods in Ecology and Evolution, 4, 675–683.
  * Fawcett [2006] Fawcett, T. (2006) An introduction to ROC analysis.  Pattern Recognition Letters, 27, 861–874.
  * Frommolt [2017] Frommolt, K.-H. (2017) Information obtained from long-term acoustic recordings: applying bioacoustic techniques for monitoring wetland birds during breeding season.  Journal of Ornithology, 1–10.
  * Furnas and Callas [2015] Furnas, B. J. and Callas, R. L. (2015) Using automated recorders and occupancy models to monitor common forest birds across a large geographic region.  The Journal of Wildlife Management, 79, 325–337.  URL: http://dx.doi.org/10.1002/jwmg.821.
  * Gashchak et al. [2017] Gashchak, S., Gulyaichenko, Y., Beresford, N. A. and Wood, M. D. (2017) European bison (bison bonasus) in the chornobyl exclusion zone (ukraine) and prospects for its revival.  Proceedings of the Theriological School, 15, 58–66.
  * Goëau et al. [2016] Goëau, H., Glotin, H., Vellinga, W.-P., Planqué, R. and Joly, A. (2016) LifeCLEF bird identification task 2016: The arrival of deep learning.  In Working Notes of CLEF 2016-Conference and Labs of the Evaluation forum, Évora, Portugal, 5-8 September, 2016., 440–449.
  * Grill and Schlüter [2017] Grill, T. and Schlüter, J. (2017) Two convolutional neural networks for bird detection in audio signals.  In Proceedings of EUSIPCO 2017.  Special Session on Bird Audio Signal Processing.
  * Hill et al. [2017] Hill, A. P., Prince, P., Covarrubias, E. P., Doncaster, C. P., Snaddon, J. L. and Rogers, A. (2017) AudioMoth: Evaluation of a smart open acoustic device for monitoring biodiversity and the environment.  Methods in Ecology and Evolution.  URL: https://doi.org/10.1111%2F2041-210x.12955.
  * Hutto and Stutzman [2009] Hutto, R. L. and Stutzman, R. J. (2009) Humans versus autonomous recording units: a comparison of point-count results.  Journal of Field Ornithology, 80, 387–398.
  * Johnston et al. [2013] Johnston, A., Ausden, M., Dodd, A. M., Bradbury, R. B., Chamberlain, D. E., Jiguet, F., Thomas, C. D., Cook, A. S., Newson, S. E., Ockendon, N. et al. (2013) Observed and predicted effects of climate change on species abundance in protected areas.  Nature Climate Change, 3, 1055–1061.
  * Johnston et al. [2014] Johnston, A., Newson, S. E., Risely, K., Musgrove, A. J., Massimino, D., Baillie, S. R. and Pearce-Higgins, J. W. (2014) Species traits explain variation in detectability of uk birds.  Bird Study, 61, 340–350.
  * Joppa [2017] Joppa, L. N. (2017) Comment: The case for technology investments in the environment.  Nature.  URL: https://www.nature.com/articles/d41586-017-08675-7.
  * Kamp et al. [2016] Kamp, J., Oppel, S., Heldbjerg, H., Nyegaard, T. and Donald, P. F. (2016) Unstructured citizen science data fail to detect long-term population declines of common birds in denmark.  Diversity and Distributions.
  * Knight et al. [2017] Knight, E., Hannah, K., Foley, G., Scott, C., Brigham, R. and Bayne, E. (2017) Recommendations for acoustic recognizer performance assessment with application to five common automated signal recognition programs.  Avian Conservation and Ecology, 12.
  * Kong et al. [2017] Kong, Q., Xu, Y. and Plumbley, M. (2017) Joint detection and classification convolutional neural network on weakly labelled bird audio detection.  In Proceedings of EUSIPCO 2017.  Special Session on Bird Audio Signal Processing.
  * LeCun et al. [2015] LeCun, Y., Bengio, Y. and Hinton, G. (2015) Deep learning.  Nature, 521, 436–444.
  * Mac Aodha et al. [2018] Mac Aodha, O., Gibb, R., Barlow, K. E., Browning, E., Firman, M., Freeman, R., Harder, B., Kinsey, L., Mead, G. R., Newson, S. E., Pandourski, I., Parsons, S., Russ, J., Szodoray-Paradi, A., Szodoray-Paradi, F., Tilova, E., Girolami, M., Brostow, G. and Jones, K. E. (2018) Bat detective—deep learning tools for bat acoustic signal detection.  PLOS Computational Biology, 14, 1–19.  URL: https://doi.org/10.1371/journal.pcbi.1005995.
  * Marques et al. [2012] Marques, T. A., Thomas, L., Martin, S. W., Mellinger, D. K., Ward, J. A., Moretti, D. J., Harris, D. and Tyack, P. L. (2012) Estimating animal population density using passive acoustics.  Biological Reviews.
  * Matsubayashi et al. [2017] Matsubayashi, S., Suzuki, R., Saito, F., Murate, T., Masuda, T., Yamamoto, K., Kojima, R., Nakadai, K. and Okuno, H. G. (2017) Acoustic monitoring of the great reed warbler using multiple microphone arrays and robot audition.  Journal of Robotics and Mechatronics, 29, 224–235.
  * Morfi and Stowell [2017] Morfi, V. and Stowell, D. (2017) Deductive refinement of species labelling in weakly labelled birdsong recordings.  In Proc ICASSP 2017, 656–660.
  * Morfi and Stowell [2018] Morfi, V. and Stowell, D. (2018) Deep Learning for Audio Transcription on Low-Resource Datasets.  ArXiv e-prints.
  * Niculescu-Mizil and Caruana [2005] Niculescu-Mizil, A. and Caruana, R. (2005) Predicting good probabilities with supervised learning.  In Proceedings of the 22nd international conference on Machine learning, 625–632. ACM.
  * North American Bird Conservation Initiative [2016] North American Bird Conservation Initiative (2016) State of North America’s birds 2016.  Tech. rep., Ottawa, Ontario.  URL: http://www.stateofthebirds.org/2016/state-of-the-birds-2016-pdf-download/.
  * Pamuła et al. [2017] Pamuła, H., Kłaczynski, M., Remisiewicz, M., Wszołek, W. and Stowell, D. (2017) Adaptation of deep learning methods to nocturnal bird audio monitoring.  In LXIV Open Seminar on Acoustics (OSA) 2017. Piekary Ṡla̧skie, Poland.
  * RSPB [2013] RSPB (2013) The state of nature in the uk and its overseas territories.  Tech. rep., RSPB and 24 other UK organisations.  URL: http://www.rspb.org.uk/ourwork/projects/details/363867-the-state-of-nature-report.
  * Salamon and Bello [2017] Salamon, J. and Bello, J. P. (2017) Deep convolutional neural networks and data augmentation for environmental sound classification.  IEEE Signal Processing Letters, 24, 279–283.
  * Stowell et al. [2015] Stowell, D., Giannoulis, D., Benetos, E., Lagrange, M. and Plumbley, M. D. (2015) Detection and classification of acoustic scenes and events.  IEEE Transactions on Multimedia, 17, 1733–1746.
  * Stowell and Plumbley [2014a] Stowell, D. and Plumbley, M. D. (2014a) Automatic large-scale classification of bird sounds is strongly improved by unsupervised feature learning.  PeerJ, 2, e488.
  * Stowell and Plumbley [2014b] — (2014b) An open dataset for research on audio field recording archives: freefield1010.  In Proceedings of the Audio Engineering Society 53rd Conference on Semantic Audio (AES53). Audio Engineering Society.
  * Stowell et al. [2016] Stowell, D., Wood, M., Stylianou, Y. and Glotin, H. (2016) Bird detection in audio: a survey and a challenge.  In Proceedings of MLSP 2016.
  * Sugiyama and Kawanabe [2012] Sugiyama, M. and Kawanabe, M. (2012) Machine learning in non-stationary environments: Introduction to covariate shift adaptation.  MIT press.
  * Towsey et al. [2012] Towsey, M., Planitz, B., Nantes, A., Wimmer, J. and Roe, P. (2012) A toolbox for animal call recognition.  Bioacoustics, 21, 107–125.
  * Walters et al. [2012] Walters, C. L., Freeman, R., Collen, A., Dietz, C., Brock Fenton, M., Jones, G., Obrist, M. K., Puechmaille, S. J., Sattler, T., Siemers, B. M. et al. (2012) A continental-scale tool for acoustic identification of european bats.  Journal of Applied Ecology, 49, 1064–1074.
  * Wood and Beresford [2016] Wood, M. and Beresford, N. (2016) The wildlife of chernobyl: 30 years without man.  Biologist, 63, 16–19.


# GreenDB - A Dataset and Benchmark for Extraction of Sustainability
Information of Consumer Goods

Sebastian Jäger  Alexander Flick  Jessica Adriana Sanchez Garcia  Kaspar von
den Driesch  Karl Brendel  Felix Biessmann

###### Abstract

The production, shipping, usage, and disposal of consumer goods have a
substantial impact on greenhouse gas emissions and the depletion of resources.
Machine Learning (ML) can help to foster sustainable consumption patterns by
accounting for sustainability aspects in product search or recommendations of
modern retail platforms. However, the lack of large high quality publicly
available product data with trustworthy sustainability information impedes the
development of ML technology that can help to reach our sustainability goals.
Here we present _GreenDB_ , a database that collects products from European
online shops on a weekly basis. As proxy for the products’ sustainability, it
relies on sustainability labels, which are evaluated by experts. The GreenDB
schema extends the well-known _schema.org Product_ definition and can be
readily integrated into existing product catalogs. We present initial results
demonstrating that ML models trained with our data can reliably (F1 score 96%)
predict the sustainability label of products. These contributions can help to
complement existing e-commerce experiences and ultimately encourage users to
more sustainable consumption patterns.

Machine Learning, ICML

##  1 Introduction

Climate change is one of the most important challenges of our generation. The
increasing demand for consumer goods and their implications for globalized
supply chains, shipping, and disposal are major factors in increased
greenhouse gas emissions (Ritchie & Roser, 2020; Umweltbundesamt, 2021a) and
the depletion of resources. With every purchase decision made, people can
decide for – or against – more sustainable consumption behavior. While many
consumers would like to choose sustainable options (PwC, 2019; Economist
Intelligence Unit, 2021; Umweltbundesamt, 2021b) at the point of sale, the
relevant information is not available or difficult to evaluate for the
consumers due to lacking standards. There are attempts to take sustainability
information and user preferences into account for recommender systems (Tomkins
et al., 2018). Our research and discussions with researchers, managers, and
engineers working for large online retailers have shown that the most
important factor hindering the integration of sustainability aspects in their
shopping experiences is the availability of trustworthy sustainability data.

To foster more sustainable shopping behavior, we need to build ML methods that
are aware of sustainability. This requires large high-quality datasets that
are recent, structured, and trustworthy111Many retailers and manufacturers
provide sustainability information that is primarily intended for marketing
purposes (greenwashing).. To the best of our knowledge, the largest publicly
available database containing products and their respective sustainability
information is the _GreenDB_ (Jäger et al., 2022). As proxy for the products’
sustainability, it relies on sustainability labels, which are evaluated by
experts regarding their _credibility_ , _environmental_ , and _socio-economic_
aspects. Currently, the GreenDB includes about $230,000$ unique products from
$4$ shops in $2$ European countries. However, it is growing continuously and
is updated at a weekly cadence. Its $26$ product categories are selected based
on a careful analysis of search logs of several millions of users of one of
the largest search engines in Europe. An overview of the data in the GreenDB
is shown in Figure 1. For detailed information about the GreenDB data
collection system and sustainability evaluation process, we refer the reader
to (Jäger et al., 2022).

##  2 Related Work

To better assess the value and contributions made with the GreenDB, we present
an overview of related public datasets that complement our work. The Web Data
Commons (WDC) (Primpeli et al., 2019) leverages structured metadata from
websites HTML to create product datasets for product matching and product
categorization. Based on the product corpus, they have published the WDC-25
Gold Standard for Product Categorization, consisting of $24,000$ product
offers from different e-shops, which were manually assigned to 25 product
categories. The Amazon ML Challenge aims at amazon browse node (product
category) classification based on product metadata like _name_ , _description_
, and _brand_. The dataset consists of about $2.9\>million$ products, each
assigned to one of 9,919 browse nodes. (Challenge, 2022) The Amazon Review
Data focuses on product reviews. However, it also contains product attributes
such as _color_ or _size_ and offers many product categories. Since the data
spans from 1996 to 2018, the dataset size is enormous (about $34GB$). (Ni et
al., 2019) The Klarna Product-Page Dataset was collected between 2018 and 2019
and is intended to benchmark representation learning algorithms on the task of
web element prediction on e-commerce websites. It consists of $51,701$ product
pages as HTML and screenshots of the website and has been manually labeled,
specifically, the product _price_ , _name_ , _image_ , _add-to-cart_ , and
_go-to-cart buttons_. In addition, the data comes from 8 different markets
(US, GB, SE, NL, FI, NO, DE, AT). (Hotti et al., 2021) Flipkart Products is
most similar to GreenDB. It is a tabular dataset with $20,000$ products and 15
columns. These attributes are partially very similar to ours, e.g.,
_product_name_ , _product_url_ , _retail_price_ , _description_ , and _brand_.
(Flipkart, 2022) However, none of them integrates products’ sustainability
information, nor are they updated regularly.

More recently, sustainability certification institutions aim at publishing
their data, such as the _European Product Database for Energy Labeling
(EPREL)_ (European Union, 2022), _Blue Angel_ (Ger, 2022), or _Higg
Sustainability Profiles_ (Hig, 2022). These data sets are rather small and
contain very specific selections of products.

Figure 1: GreenDB overview. We follow the terminology by (Gossen et al., 2022) and differentiate between _private_ and _third-party labels_. Third-party labels require independent verification of the label’s environmental and/or socio-economic requirements. However, private labels are self-declared claims and not based on independent verification. Depending on the online shops and product categories, the number (and fraction) of preferable third-party labels varies and is generally much lower.  | precision | recall | f1
---|---|---|---
BETTER_COTTON_INITIATIVE | 0.89 | 0.93 | 0.91
BIORE | 0.98 | 0.98 | 0.98
BLUESIGN_APPROVED | 0.92 | 0.92 | 0.92
BLUESIGN_PRODUCT | 0.95 | 0.97 | 0.96
COTTON_MADE_IN_AFRICA | 0.95 | 0.98 | 0.97
CRADLE_TO_CRADLE_BRONZE | 0.33 | 0.14 | 0.20
CRADLE_TO_CRADLE_GOLD | 0.54 | 0.24 | 0.33
CRADLE_TO_CRADLE_SILVER | 0.75 | 0.27 | 0.40
ECOCERT | 0.94 | 0.68 | 0.79
EU_ECOLABEL_TEXTILES | 1.00 | 0.91 | 0.95
FAIRTRADE_COTTON | 0.97 | 0.85 | 0.90
FAIRTRADE_TEXTILE_PRODUCTION | 1.00 | 0.80 | 0.89
GLOBAL_RECYCLED_STANDARD | 0.93 | 0.85 | 0.89
GOOD_CASHMERE_STANDARD | 0.00 | 0.00 | 0.00
GOTS_MADE_WITH_ORGANIC_MATERIALS | 0.84 | 0.40 | 0.54
GOTS_ORGANIC | 0.85 | 0.79 | 0.82
GREEN_BUTTON | 0.96 | 0.88 | 0.92
LEATHER_WORKING_GROUP | 0.93 | 0.93 | 0.93
MADE_IN_GREEN_OEKO_TEX | 0.98 | 0.95 | 0.96
ORGANIC_CONTENT_STANDARD_100 | 0.79 | 0.59 | 0.67
ORGANIC_CONTENT_STANDARD_BLENDED | 0.62 | 0.41 | 0.49
PRIVATE_LABEL | 0.97 | 0.98 | 0.98
RECYCLED_CLAIM_STANDARD_100 | 0.77 | 0.50 | 0.61
RECYCLED_CLAIM_STANDARD_BLENDED | 1.00 | 0.93 | 0.97
RESPONSIBLE_DOWN_STANDARD | 0.94 | 0.72 | 0.81
RESPONSIBLE_WOOL_STANDARD | 0.72 | 0.54 | 0.62
accuracy | 0.96 | 0.96 | 0.96
macro avg | 0.83 | 0.69 | 0.74
weighted avg | 0.96 | 0.96 | 0.96
Table 1: Prediction results of sustainability labels based on the GreenDB
product attributes _name_ , _description_ , _brand_ , _category_ , _color_ ,
_merchant_ , and _price_. The extraction of these attributes is part of the
GreenDB data collection system (Jäger et al., 2022).

##  3 Predicting Sustainability Information

To establish a simple baseline for automated inference of sustainability
information from online products, we used the _TabularPredictor_ module of the
AutoML package _AutoGluon_ (AutoGluon, ; Erickson et al., 2020). The model was
trained on the entire GreenDB dataset (Jäger et al., 2022). We simplified the
prediction problem, without loss of generality, from a multilabel problem
(each product can have more than one sustainability label) to a multiclass
prediction problem: For each product, we consider the first sustainability
label (sorted in alphabetical order). In Table 1, we list the results. Note
that our simple benchmark achieves an F1 score (class frequency weighted) of
0.96, indicating that the high data quality of the GreenDB enables researchers
to quickly achieve production-ready ML-based automated sustainability
inference.

## References

  * Ger (2022) The German Ecolabel, 2022.  URL https://www.blauer-engel.de/en.  [Online; accessed 23-May-2022].
  * Hig (2022) Higg Sustainability Profiles, 2022.  URL https://profiles.higg.com/.  [Online; accessed 23-May-2022].
  * (3) AutoGluon.  https://github.com/awslabs/autogluon.
  * Challenge (2022) Challenge, A. M., 2022.  URL https://www.hackerearth.com/en-us/challenges/competitive/amazon-ml-challenge/.  [Online; accessed 23-May-2022].
  * Economist Intelligence Unit (2021) Economist Intelligence Unit.  An eco-wakening - measuring global awareness, engagement and action for nature, 2021.  URL https://files.worldwildlife.org/wwfcmsprod/files/Publication/file/93ts5bhvyq_An_EcoWakening_Measuring_awareness__engagement_and_action_for_nature_FINAL_MAY_2021.pdf.
  * Erickson et al. (2020) Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A.  Autogluon-tabular: Robust and accurate automl for structured data.  _arXiv preprint arXiv:2003.06505_ , 2020.
  * European Union (2022) European Union.  Product database, 2022.  URL https://ec.europa.eu/info/energy-climate-change-environment/standards-tools-and-labels/products-labelling-rules-and-requirements/energy-label-and-ecodesign/product-database_de.  [Online; accessed May 19, 2022].
  * Flipkart (2022) Flipkart, 2022.  URL https://www.kaggle.com/PromptCloudHQ/flipkart-products.  [Online; accessed 23-May-2022].
  * Gossen et al. (2022) Gossen, M., Jäger, S., Hoffmann, M. L., Bießmann, F., Korenke, R., and Santarius, T.  Nudging Sustainable Consumption: A Large-Scale Data Analysis of Sustainability Labels for Fashion in German Online Retail.  _Frontiers in Sustainability_ , 3:922984, June 2022.  ISSN 2673-4524.  doi: 10.3389/frsus.2022.922984.  URL https://www.frontiersin.org/articles/10.3389/frsus.2022.922984/full.
  * Hotti et al. (2021) Hotti, A., Risuleo, R. S., Magureanu, S., Moradi, A., and Lagergren, J.  The klarna product page dataset: A realistic benchmark for web representation learning.  _CoRR_ , abs/2111.02168, 2021.  URL https://arxiv.org/abs/2111.02168.
  * Jäger et al. (2022) Jäger, S., Greene, J., Jakob, M., Korenke, R., Santarius, T., and Biessmann, F.  GreenDB: Toward a Product-by-Product Sustainability Database.  Technical Report arXiv:2205.02908, arXiv, May 2022.  URL http://arxiv.org/abs/2205.02908.
  * Jäger et al. (2022) Jäger, S., Bießmann, F., Flick, A., Sanchez Garcia, J. A., von den Driesch, K., and Brendel, K.  GreenDB: A Product-by-Product Sustainability Database, February 2022\.  URL https://doi.org/10.5281/zenodo.6576662.  Supported by the Federal Ministry for the Environment, Nature Conservation and Nuclear Safety based on a decision of the German Bundestag. Förderkennzeichen: 67KI2022B.
  * Ni et al. (2019) Ni, J., Li, J., and McAuley, J. J.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pp. 188–197. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/D19-1018.  URL https://doi.org/10.18653/v1/D19-1018.
  * Primpeli et al. (2019) Primpeli, A., Peeters, R., and Bizer, C.  The wdc training dataset and gold standard for large-scale product matching.  In _Companion Proceedings of The 2019 World Wide Web Conference_ , WWW ’19, pp. 381–386, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450366755.  doi: 10.1145/3308560.3316609.
  * PwC (2019) PwC.  Surviving the retail apocalypse, 2019.  URL https://www.pwc.de/de/handel-und-konsumguter/studie-surviving-the-retail-apocalypse.pdf.
  * Ritchie & Roser (2020) Ritchie, H. and Roser, M.  Co2 and greenhouse gas emissions.  _Our World in Data_ , 2020.  https://ourworldindata.org/co2-and-other-greenhouse-gas-emissions.
  * Tomkins et al. (2018) Tomkins, S., Isley, S., London, B., and Getoor, L.  Sustainability at scale: towards bridging the intention-behavior gap with sustainable recommendations.  In Pera, S., Ekstrand, M. D., Amatriain, X., and O’Donovan, J. (eds.), _Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018_ , pp. 214–218. ACM, 2018.  doi: 10.1145/3240323.3240411.  URL https://doi.org/10.1145/3240323.3240411.
  * Umweltbundesamt (2021a) Umweltbundesamt.  Konsum und umwelt: Zentrale handlungsfelder, 2021a.  URL https://www.umweltbundesamt.de/themen/wirtschaft-konsum/konsum-umwelt-zentrale-handlungsfelder.  [Online; accessed February 03, 2022].
  * Umweltbundesamt (2021b) Umweltbundesamt.  Umweltbewusstsein in deutschland, 2021b.  URL https://www.umweltbundesamt.de/themen/nachhaltigkeit-strategien-internationales/umweltbewusstsein-in-deutschland.  [Online; accessed February 03, 2022].