# BEANS: The Benchmark of Animal Sounds

Masato Hagiwara, Benjamin Hoffman, Jen-Yu Liu, Maddie Cusimano
Felix Effenberger, Katie Zacarian
Earth Species Project
{masato, benjamin, jenyu, maddie, felix<EMAIL_ADDRESS>
###### Abstract

The use of machine learning (ML) based techniques has become increasingly
popular in the field of bioacoustics over the last years. Fundamental
requirements for the successful application of ML based techniques are
curated, agreed upon, high-quality datasets and benchmark tasks to be learned
on a given dataset. However, the field of bioacoustics so far lacks such
public benchmarks which cover multiple tasks and species to measure the
performance of ML techniques in a controlled and standardized way and that
allows for benchmarking newly proposed techniques to existing ones. Here, we
propose BEANS (the BEnchmark of ANimal Sounds), a collection of bioacoustics
tasks and public datasets, specifically designed to measure the performance of
machine learning algorithms in the field of bioacoustics. The benchmark
proposed here consists of two common tasks in bioacoustics: classification and
detection. It includes 12 datasets covering various species, including birds,
land and marine mammals, anurans, and insects. In addition to the datasets, we
also present the performance of a set of standard ML methods as the baseline
for task performance. The benchmark and baseline code is made publicly
available 111https://github.com/earthspecies/beans in the hope of establishing
a new standard dataset for ML-based bioacoustic research.

##  1 Introductions

Figure 1: Overview of BEANS

Due to their increasing affordability, recording and storage devices are now
widely used to collect bioacoustic data. These devices enable animal welfare
and wildlife conservation applications, such as passive acoustic monitoring
(PAM), which offers tools for wildlife population assessment and conservation
research in a non-invasive and unbiased manner [46]. However, they produce a
large amount of bioacoustic data, and the manual processing and analysis of
these data has become a bottleneck [45, 47].

For this reason, machine learning (ML) has increasingly been used to automate
the processing and analysis of bioacoustics tasks. It has been successfully
applied to a variety of tasks such as classification of species, individuals,
and various other characteristics of calls [29], detection and recognition of
vocalizations in passive recordings [26], and automatic discovery of
vocalization units [39]. Furthermore, the so-called “deep learning” (DL)
models based on deep artificial neural networks have drastically reduced error
rates and are increasingly being used for these tasks in recent years.

However, typical research studies in bioacoustics focus only on a small number
of species and/or specific types of methodology [9, 42, 21]. This narrow focus
has led to a proliferation of ML models and algorithms that perform well on
the tasks in question on a given data set, but not necessarily outside of
their scope [43]. Moreover, the lack of publicly available, agreed-upon
standard datasets in bioacoustics [1, 28] makes it difficult to reproduce and
compare different approaches in a standardized way.

If we turn our attention to other fields of machine learning, much of the
recent progress has been driven by standard “benchmarks”. A benchmark is a
collection of datasets along with tasks to be learned to perform on the data,
specifically designed to measure the performance of ML algorithms in a
standardized way. For example, standardized datasets such as MNIST [12],
CIFAR-10/100 [25], ImageNet [11], and more recently, VISSL [17] have been used
to measure the performance of image classification algorithms in computer
vision for decades. Other examples include GLUE [50] and SuperGLUE [49] for
natural language processing, and SUPERB [52] and HEAR [48] for human
speech/audio processing. Benchmarks are often accompanied by competitions and
leaderboards that show a ranking of different ML models based on their
performance on those benchmarks. These standardized benchmarks and
leaderboards allow for an objective and quantitative comparison of different
approaches, the identification of strengths and weaknesses of different
methodologies [32], and additionally for an assessment of the overall progress
made in the field of research. Moreover, benchmarks played an important role
in the developments and evaluation of some of the recent progress in ML,
including the BERT [13] and GPT-3 [5] models.

In this paper, we propose BEANS (the BEnchmark of ANimal Sounds), a collection
of publicly available bioacoustics datasets along with tasks to be performed
on those, specifically designed to measure the performance of ML algorithms in
the bioacoustics domain in a standardized manner. The benchmark includes two
common bioacoustics tasks, classification and detection, and consists of
twelve datasets covering diverse species, including birds, land and marine
mammals, anurans, and insects. We run various non-DL and DL algorithms as the
baseline on BEANS and show that there is considerable room for improvement,
especially for the detection task. We release the entire code and the baseline
implementations as open source 222https://github.com/earthspecies/beans to
encourage the further development of generic bioacoustic methods.

##  2 Benchmark Design

Figure 2: Tasks included in BEANS

The goal of BEANS is to accurately measure and compare ML models through a
collection of bioacoustic datasets covering diverse species. We wish to
encourage the development of ML models that work well not only on specific
species and dataset, but also on a diverse set of species with as little
species-specific modification or training data as possible. Such models are of
interest to the bioacoustic community due to better generalizability and lower
development cost.

In this benchmark, we focus on classification and detection tasks, which are
the two tasks most commonly considered in the bioacoustics literature [45].
Classification is a task in which each sample is assigned one or more labels
from a set of predefined classes, such as a set of individuals or a set of
species. Here, we use a regular single-label, multi-class setting (Figure 2
left).

Detection is a task in which one uses ML algorithms to identify subsections of
interest and their properties from long recordings (often obtained from
passive acoustic monitoring). We adopt a sliding window approach, as commonly
done for detection tasks [26, 14], where long recordings are broken up into
short (potentially overlapping) segments and the ML algorithm makes a
prediction per segment (Figure 2 right). In order to address multiple
overlapping vocalizations, more than one label can be assigned to a segment,
making it a multi-label, multi-class classification setting.

Due to the design of these tasks, from the perspective of the ML algorithm,
the two tasks can be both solved by the same classification model. This
homogeneity and simplicity in terms of the structure of the tasks allows the
benchmark users to use almost the same algorithm with little modification and
encourages the use of generic models.

As metrics for evaluating task performance, we use accuracy $A$ for
classification tasks and mean average precision ${\rm mAP}$ [15] for detection
tasks defined below. Specifically, let $N$ be the total number of samples in
the dataset, $C$ be the number of classes, and ${\it tp}_{c},{\it
fp}_{c},P_{c}(r)$ be the number of true positives, false positives, the
interpolated precision at recall $r$ for class $c$, respectively. The metrics
are then defined as

$\displaystyle A=\frac{\sum_{c}({\it tp}_{c}+{\it tn}_{c})}{CN},\quad{\rm
mAP}=\frac{1}{11C}\sum_{r\in\\{0,0.1,...,1.0\\}}P_{c}(r)$ (1)

The benchmark comes with predefined train, validation, and test splits, as
well as a baseline implementation, to encourage consistent comparison and
reproducibility. Note that we do not encourage the reporting of a single
aggregated score across different datasets and tasks for the benchmark, as a
single score does not necessarily reflect the difficulties of different tasks,
a fact often criticized in the literature [10].

##  3 Datasets

Dataset | Description |  # Train / Valid / Testa | # Labels (type) | Sample Rate | License
---|---|---|---|---|---
Classification |
watkins | aquat. mamm. | 1017 / 339 / 339 | 31 (species) | 44.1kHz |  freeb
bats | bats | 6000 / 2000 / 2000 | 10 (individual) | 250kHz | CC-BY-ND
cbi | birds | 14207 / 3548 / 3620 | 264 (species) | 44.1kHz | CC-BY-NC-SA
dogs | dogs | 415 / 139 / 139 | 10 (individual) | 44.1kHz |  allowedc
humbugdb | mosquito | 9293 / 1859 / 1859 | 14 (species) | 44.1kHz | CC-BY
Detection |
dcase | birds & mamm. | 701 / 233 / 232 | 20 (species) | various | CC-BY
enabirds | birds | 230 / 76 / 76 | 34 (species) | 32kHz | CC0
hiceas | cetaceans | 406 / 134 / 134 | 1 (species) | 500kHz |  freed
rfcx | birds & frogs | 2835 / 944 / 945 | 24 (species) | 48kHz |  freee
gibbons | gibbons | 959 / 479 / 639 | 3 (call type) | 9.6kHz | CC-NC-SA
Auxiliary |
esc-50 | misc. sound | 1200 / 400 / 400 | 50 (sound type) | 16kHz | CC-BY-NC
sc | human | 84843 / 9981 / 11006 | 35 (word) | 44.1kHz | CC-BY
Table 1: Datasets included in the benchmark. a The numbers of samples for
classification and the number of 1-minute “chunks” for detection (see Section
4 for more details). b free for personal or academic use, c academic use
allowed through personal correspondence, d data are free for use without
restriction, e usage allowed for academic research.

In this section, we describe the datasets that we included in the benchmark.
To make this choice, we surveyed many bioacoustics datasets from the
literature and chose 5 datasets for classification and another 5 for detection
tasks (Table 1) based on the following criteria:

  * •

Availability: Is the dataset publicly and freely available for research
purposes?

  * •

Difficulty: Is the dataset moderately difficult for ML algorithms to solve?

  * •

Size: is the dataset large enough for ML algorithms to learn meaningful
patterns from it? Is the dataset small enough so that the training is within
the reach of an average compute budget of typical users (biologists and
machine learning researchers)?

  * •

Diversity: Does the benchmark represent a diverse collection of sound-making
animal species?

We also included two “auxiliary” datasets that are commonly used to evaluate
environmental sound detection and speech classification systems, two domains
closely related to bioacoustics, in order to encourage the development of ML
models that generalize beyond bioacoustics. These two datasets are not
officially part of the benchmark, but benchmark users can choose to report
performance numbers on them as a reference.

Below, we describe the specifics of the five classification datasets:

  * •

watkins [40]: The Watkins Marine Mammal Sound Database is a database of marine
mammal sounds. We used the preprocessed dataset hosted on the Internet Archive
333https://archive.org/details/watkins_202104 which contains the recordings of
32 species from the ‘Best of cuts’ section (except for weddell seal
recordings, which had only a few samples). We randomly split the dataset into
6:2:2 train:valid:test portions with stratification. All recordings are
resampled to 44.1kHz.

  * •

bats [36]: The original dataset contains annotated recordings of Egyptian
fruit bats (Rousettus aegyptiacus) vocalizations recorded at a sampling rate
of 250kHz. We used the preprocessed dataset consisting of individual calls of
up to a few seconds long. The target label is the emitter ID (individual).

  * •

cbi [31]: This is the dataset from the Cornell Bird Identification competition
hosted on Kaggle. The training set consists of bird recordings uploaded to
xeno-canto444https://xeno-canto.org/ by volunteer users. Since the test set
labels are hidden, we split the train set into 6:2:2 train:valid:test portions
in such a way that there is no overlap in recordists between splits.

  * •

dogs [54]: This dataset consists of barks recorded from 10 individual domestic
dogs in different situations (disturbance, isolation, and play) originally at
48 kHz and resampled to 44.1kHz. Each recording is annotated with the
individual and the situation, but we used the individual as the target label.
We randomly split the dataset into 6:2:2 train:valid:test portions with
stratification.

  * •

humbugdb [23]: HumBugDB is a collection of wild and cultured mosquito wingbeat
sounds recorded in various settings (including sounds when the animals were
located in cups and under bednets). The purpose is to detect and classify
species that can be vectors of diseases such as malaria. We took their species
classification dataset, while collapsing any species with 100 or fewer samples
into an “OTHER” category, resulting in a grouping into 14 classes. We randomly
split the dataset into 6:2:2 train:valid:test portions with stratification.

The following are the details of the five detection datasets:

  * •

dcase [30]: This is the dataset used for DCASE 2021 Task 5: Few-shot
Bioacoustic Event Detection. It contains mammal and bird multi-species
recordings annotated with species, onset, and offset times. We repurposed
their few-shot development dataset as a (regular) detection dataset by
partitioning long recordings into 1-minute chunks, and used the first 60% for
training, the next 20% for validation, and the final 20% for testing. We only
retained positive (POS) labels. The dataset contains files recorded at various
sample rates, but we up/down sampled them all at 16kHz.

  * •

enabirds [8]: The dataset contains recordings of bird dawn chorus, annotated
with the onset/offset time, the frequency range, and the species. We used the
33 most frequent species labels and treated all infrequent labels as “OTHERS.”
We partitioned the data set into training, validation, and testing portions as
described in dcase.

  * •

hiceas [6]: The dataset consists of a subset of passive acoustic data
collected using a multi-channel towed hydrophone during the Hawaiian Islands
Cetacean and Ecosystem Assessment Survey (HICEAS) in 2017. We used the human-
audible Minke whale “boing” vocalization annotations of the data set,
rendering this a single-class (binary) detection task. We sampled 1/20th of
all the files in the dataset, downsampled them from 500kHz to 22.5kHz, and
assigned them to train, valid and test splits randomly with a 6:2:2 ratio. We
redistribute the preprocessed dataset as part of the BEANS repository.

  * •

rfcx [26]: This is a dataset of continuous soundscape recordings of 24 species
of frogs and birds collected by Rainforest Connection (RFCx). The data were
annotated with the onset / offset time, as well as the frequency range. We
randomly assigned the files into train, valid, and test splits with a 6:2:2
ratio. We did not use the false positive annotation.

  * •

hainan-gibbons [14]: The dataset contains continuous recordings of Hainan
gibbon calls. The data were annotated with onset/offset times and call types
(one pulse, multiple pulse, duet). There are a total of 14 files, each
corresponding to 8 hours of recording for a particular day. Due to its large
size, we sub-sampled 1/3 of the dataset after splitting into chunks. We used
the first 6 files for training, the next 3 files for validation, and the
remaining ones for testing.

Finally, we included two auxiliary datasets in the benchmark as described
above. Note that both are classification tasks.

  * •

esc50 [35]: A dataset of environmental audio recordings including animal
sounds (e.g., dogs, roosters), nature sounds (e.g., rain, sea waves), human
(non-speech) sounds (e.g., crying, sneezing), interior/domestic sounds (e.g.,
door knocks, mouse clicks), and exterior/urban sounds (e.g., helicopter,
chainsaw). The dataset is commonly used as a benchmark for environmental sound
classification and comes with predefined splits. We used split 4 for
validation, split 5 for testing, and the rest for training.

  * •

speech-commands [51]: A dataset of single-word utterances covering 35 English
words including digits (e.g., “zero”, “one”, …), commands (e.g., “yes”, “no”,
“up”, “left”, “stop”), and others (e.g., “bird”, “happy”, “wow”), spoken by
multiple speakers. The dataset is commonly used to benchmark speech systems.
The dataset comes with predefined train, validation, and test splits. We used
version 0.02 of the data set.

##  4 Experiments

###  4.1 Experimental Setup

We ran a wide range of traditional non-DL and DL algorithms to establish a
baseline for BEANS. All the waveforms were mixed to a single channel (mono)
with 16 bit depth. For classification, each waveform was padded with silence
at the end if it was shorter than the minimum duration threshold set for the
dataset, or truncated if it was longer. For detection, a sliding window
algorithm was applied to partition the chunks into instances. The length of
sliding windows was 2 seconds for dcase and enabirds, 10 seconds for hiceas
and rfcx, and 4 seconds for hainan-gibbons. An instance is marked positive if
the amount of overlap with any annotation is more than 20%.

We consider the following non-DL algorithms, often used for classification of
tabular data:

  * •

Logistic regression (LR)

  * •

Support vector machine (SVM)

  * •

Decision tree (DT)

  * •

Gradient-boosted decision tree (GBDT)

  * •

XGBoost (XGB, [7])

We used the official library for XGBoost
555https://xgboost.readthedocs.io/en/stable/ and the implementations in
scikit-learn [34] for the other four algorithms. We first obtained 20 MFCCs
features from power mel-spectrogram [53]. The length of the FFT window and the
hop length were chosen as 50ms and 10ms, respectively. We computed four
summary statistics, mean, standard deviation, min, and max of each MFCC
dimension over time, resulting in a 80-dimension feature vector per sample.

Furthermore, we consider the following DL-based models. They are all based on
convolutional neural networks (CNNs), a DL architecture which has been
successfully applied in most fields of ML, and also the fields of bioacoustics
[38], environmental sound classification [3], and human speech [24].

  * •

ResNet [19] (ResNet18, ResNet50, ResNet152, both random and pretrained weights
on ImageNet)

  * •

VGGish [20], a VGG-like architecture [44] pretrained on a large YouTube audio
dataset

The VGGish model has been widely used as a strong baseline model for various
audio classification and detection tasks [45]. The input to those models were
power mel-spectrograms computed with torchaudio [53], using the same
parameters as above. We downsampled the input to 16kHz for VGGish due to its
preprocessing pipeline. We used the ResNet models implemented in torchvision
[27]. The representations were average-pooled before being fed to the
classification layer. For classification tasks, a linear and a softmax layer
were added on top of the classification layer (for ResNet) and the embedding
layer (for VGGish), and the entire network was optimized using a cross-entropy
loss function. For detection tasks, the softmax layer was replaced with a
sigmoid layer and the network was optimized with a binary cross-entropy loss
function.

Each model was fine-tuned on the training portion of each dataset using the
Adam [22] optimizer with $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=1.0\times
10^{-8}$. We swept the learning rate over $1.0\times 10^{-5},5.0\times
10^{-5},1.0\times 10^{-4}$ for 50 epochs, and picked the best model based on
the validation metric.

###  4.2 Main Results

| Classification | Detection | Auxiliary
---|---|---|---
| wtkn | bat | cbi | hbdb | dogs | dcase | enab | hiceas | rfcx | gib | esc | sc
lr | 0.776 | 0.661 | 0.156 | 0.751 | 0.885 | 0.143 | 0.247 | 0.221 | 0.030 | 0.006 | 0.428 | 0.535
svm | 0.870 |  0.720 | 0.139 |  0.779 | 0.914 | 0.146 | 0.299 | 0.218 | 0.038 | 0.039 | 0.478 | 0.572
dt | 0.661 | 0.474 | 0.025 | 0.700 | 0.626 | 0.095 | 0.183 | 0.223 | 0.009 | 0.007 | 0.240 | 0.230
gbdt | 0.758 | 0.674 | 0.038 | 0.759 | 0.827 | 0.104 | 0.235 | 0.216 | 0.009 | 0.007 | 0.338 | 0.481
xgb | 0.808 | 0.692 | 0.097 | 0.772 | 0.842 | 0.124 | 0.270 | 0.214 | 0.012 | 0.007 | 0.403 | 0.525
rn18 | 0.752 | 0.443 | 0.357 | 0.697 | 0.662 | 0.161 | 0.325 | 0.280 | 0.064 | 0.164 | 0.500 | 0.926
rn50 | 0.799 | 0.548 | 0.295 | 0.696 | 0.633 | 0.183 | 0.282 |  0.304 | 0.055 | 0.215 | 0.235 | 0.936
rn152 | 0.743 | 0.483 | 0.330 | 0.645 | 0.511 | 0.154 | 0.280 | 0.255 | 0.069 |  0.248 | 0.365 | 0.929
rn18p | 0.735 | 0.532 | 0.509 | 0.649 | 0.705 | 0.223 |  0.462 | 0.262 | 0.079 | 0.316 |  0.590 | 0.936
rn50p | 0.735 | 0.560 |  0.548 | 0.673 | 0.763 | 0.178 | 0.424 | 0.284 |  0.087 | 0.155 | 0.545 |  0.946
rn152p | 0.720 | 0.544 | 0.573 | 0.662 | 0.741 | 0.198 | 0.429 | 0.273 | 0.085 | 0.230 | 0.540 |  0.946
vggish |  0.847 | 0.743 | 0.440 | 0.808 |  0.906 | 0.335 | 0.535 | 0.463 | 0.140 | 0.150 | 0.705 | 0.948
Table 2: Main results. As measure for task performance, we used accuracy for
classification and auxiliary tasks, and mean average precision (mAP) for
detection tasks. The best and second best performing models are highlighted
and underlined, respectively, for each dataset.

The main results are shown in Table 2. In terms of model performance, VGGish
works best overall, followed by pretrained ResNet models. Among the non-DL
based ML models, SVM shows surprisingly good performance and beats neural
models in some classification tasks. Note that ResNets are pretrained on
ImageNet, while VGGish is pre-trained on a YouTube dataset [20]. We
hypothesize that if we were to pretrain a ResNet on a large-scale audio
datasets such as AudioSet [16], that the performance attainable by ResNets
could be better. We leave this for future work.

Cornell Bird Identification (CBI) is the most challenging classification task
in BEANS, due to the way the data is split (with no overlapping recordists)
and the sheer number of target labels (264 species), which is close to how
machine learning models are used in some real-world bioacoustic settings.

In terms of tasks, performance on some detection datasets (e.g., rfcx,
gibbons) was lower than others. This might be due to the fact that a typical
detection dataset is very sparse, meaning that only a minor portion of a
typical recording is animal vocalization and has fewer training annotations
per class than classification. Modern regularization and data augmentation
techniques, such as mixup [55] and SpecAugment [33] may help improve the
results.

For most of the datasets contained in BEANS, the performance of our baselines
is well below 90%, which means that there is large room for improvement for
future, more specialized ML algorithms.

Note that the way we padded the instances could inadvertently affect the
classification performance if there exists systematic difference in instance
lengths between classes irrespective of true vocalization lengths, although we
didn’t find any evidence that padding is leading to an overestimation of
classification performance.

##  5 Discussion

#### Sample rate

Bioacoustics data can have a wide range of sample rates, which can be
challenging for ML models that are usually trained on data captured at a
single and fixed sample rate. Even within BEANS, vocalizations range from
human-audible calls that can be captured with a sampling rate around 10kHz to
ultrasonic echolocation of bats that require sampling rates of 250kHz. Methods
that can be adopted to different sampling frequencies (e.g., S4 [18]) could be
a promising future direction.

#### Overlapping calls

Currently, we cannot address the cases where multiple individuals are
vocalizing at the same time unless those individuals are explicitly and
separately annotated. Sound separation models (e.g., BioCPPNet [2]) are a
promising preprocessing step to address this issue.

#### Bias

Some animal species and/or geographical locations can be over- or under-
represented in a dataset. For example, North American birds are over-
represented in BEANS, included in at least three datasets (cbi, dcase, and
enabirds). We acknowledge that the curating process of datasets is inherently
subjective [41, 37] and wish to address this bias in future iterations of this
benchmark.

#### Limitations

We acknowledge that a benchmark is only a proxy for progress that we measure
and can face construct validity issues [37, 4]. We also note the inherent
fragility of the ML benchmarking process [10]. Therefore, we in particular do
not recommend reporting a single metric averaged over datasets and tasks as a
performance measure of an ML algorithm on this benchmark. Instead, BEANS is
intended as a diagnostic tool to encourage the development of generic, well-
balanced bioacoustics approaches.

## References

  * [1] Ed Baker and Sarah Vincent.  A deafening silence: a lack of data and reproducibility in published bioacoustics research?  Biodiversity Data Journal, 7:e36783, 2019.
  * [2] Peter C. Bermant.  BioCPPNet: automatic bioacoustic source separation with deep neural networks.  Scientific Reports, 11, 2021.
  * [3] Venkatesh Boddapati, Andrej Petef, Jim Rasmusson, and Lars Lundberg.  Classifying environmental sounds using image recognition networks.  Procedia Computer Science, 112:2048–2056, 2017.  Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 21st International Conference, KES-20176-8 September 2017, Marseille, France.
  * [4] Samuel R. Bowman and George Dahl.  What will it take to fix benchmarking in natural language understanding?  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843–4855, Online, June 2021. Association for Computational Linguistics.
  * [5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [6] NOAA Pacific Islands Fisheries Science Center.  Hawaiian islands cetacean and ecosystem assessment survey (hiceas) towed array data.  Edited and annotated for the 9th International Workshop on Detection, Classification, Localization, and Density Estimation of Marine Mammals Using Passive Acoustics (DCLDE 2022), 2022.
  * [7] Tianqi Chen and Carlos Guestrin.  XGBoost: A scalable tree boosting system.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794, New York, NY, USA, 2016. Association for Computing Machinery.
  * [8] Lauren M. Chronister, Tessa A. Rhinehart, Aidan Place, and Justin Kitzes.  An annotated set of audio recordings of Eastern North American birds containing frequency, time, and species information.  Ecology, 102(6):e03329, 2021.
  * [9] Kevin R. Coffey, Russell G. Marx, and John F. Neumaier.  DeepSqueak: a deep learning-based system for detection and analysis of ultrasonic vocalizations.  Neuropsychopharmacology, 44:859–868, 2019.
  * [10] Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals.  The benchmark lottery.  ArXiv, abs/2107.07002, 2021.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [12] Li Deng.  The MNIST database of handwritten digit images for machine learning research [best of the web].  IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [14] Emmanuel Dufourq, Ian Durbach, James P. Hansford, Amanda Hoepfner, Heidi Ma, Jessica V. Bryant, Christina S. Stender, Wenyong Li, Zhiwei Liu, Qing Chen, Zhaoli Zhou, and Samuel T. Turvey.  Automated detection of Hainan gibbon calls for passive acoustic monitoring.  Remote Sensing in Ecology and Conservation, 7(3):475–487, 2021\.
  * [15] Mark Everingham and John Winn.  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) development kit, 2012.  http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/devkit_doc.html.
  * [16] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio Set: An ontology and human-labeled dataset for audio events.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776–780, 2017.
  * [17] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra.  VISSL.  https://github.com/facebookresearch/vissl, 2021.
  * [18] Albert Gu, Karan Goel, and Christopher Ré.  Efficiently modeling long sequences with structured state spaces.  In The International Conference on Learning Representations (ICLR), 2022.
  * [19] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [20] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin Wilson.  CNN architectures for large-scale audio classification.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 131–135, 2017.
  * [21] Stefan Kahl, Connor M. Wood, Maximilian Eibl, and Holger Klinck.  BirdNET: A deep learning solution for avian diversity monitoring.  Ecological Informatics, 61:101236, 2021.
  * [22] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Ivan Kiskin, Marianne E. Sinka, Adam D. Cobb, Waqas Rafique, Lawrence Wang, Davide Zilli, Benjamin Gutteridge, Theodoros Marinos, Yunpeng Li, Emmanuel Wilson Kaindoa, Gerard F Killeen, Katherine J. Willis, and S. Roberts.  HumBugDB: a large-scale acoustic mosquito dataset.  In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, 2021.
  * [24] Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang.  Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6124–6128, 2020.
  * [25] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  pages 32–33, 2009.
  * [26] Jack LeBien, Ming Zhong, Marconi Campos-Cerqueira, Julian P. Velev, Rahul Dodhia, Juan Lavista Ferres, and T. Mitchell Aide.  A pipeline for identification of bird and frog species in tropical soundscape recordings using a convolutional neural network.  Ecological Informatics, 59:101113, 2020.
  * [27] Sébastien Marcel and Yann Rodriguez.  Torchvision the machine-vision package of torch.  In Proceedings of the 18th ACM International Conference on Multimedia, MM ’10, page 1485–1488, New York, NY, USA, 2010. Association for Computing Machinery.
  * [28] Michael P. Mcloughlin, Rebecca Stewart, and Alan G. McElligott.  Automated bioacoustics: methods in ecology and conservation and their potential for animal welfare monitoring.  Journal of The Royal Society Interface, 16(155):20190225, 2019.
  * [29] Veronica Morfi, Robert F. Lachlan, and Dan Stowell.  Deep perceptual embeddings for unlabelled animal sound events.  The Journal of the Acoustical Society of America, 150(1):2–11, 2021\.
  * [30] Veronica Morfi, Inês Nolasco, Vincent Lostanlen, Shubhr Singh, Ariana Strandburg-Peshkin, Lisa F. Gill, Hanna Pamula, David Benvent, and Dan Stowell.  Few-shot bioacoustic event detection: A new task at the dcase 2021 challenge.  In Detection and Classification of Acoustic Scenes and Events 2021, 2021.
  * [31] Cornell Lab of Ornithology.  Cornell birdcall identification, 2020.  https://www.kaggle.com/competitions/birdsong-recognition.
  * [32] Randal S. Olson, W. L. Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore.  PMLB: a large benchmark suite for machine learning evaluation and comparison.  BioData Mining, 10, 2017.
  * [33] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le.  SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition.  In Proc. Interspeech 2019, pages 2613–2617, 2019.
  * [34] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12(85):2825–2830, 2011.
  * [35] Karol J. Piczak.  ESC: Dataset for Environmental Sound Classification.  In Proceedings of the 23rd Annual ACM Conference on Multimedia, pages 1015–1018. ACM Press.
  * [36] Yosef Prat, Mor Taub, Ester Pratt, and Yossi Yovel.  An annotated dataset of egyptian fruit bat vocalizations across varying contexts and during vocal ontogeny.  Scientific Data, 4, 2017.
  * [37] Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily L. Denton, and A. Hanna.  AI and the everything in the whole wide world benchmark.  ArXiv, abs/2111.15366, 2021.
  * [38] Zachary J. Ruff, Damon B. Lesmeister, Leila S. Duchac, Bharath K. Padmaraju, and Christopher M. Sullivan.  Automated identification of avian vocalizations with deep convolutional neural networks.  Remote Sensing in Ecology and Conservation, 6(1):79–92, 2020.
  * [39] Tim Sainburg, Marvin Thielk, and Timothy Q Gentner.  Latent space visualization, characterization, and generation of diverse vocal communication signals.  bioRxiv, 2020.
  * [40] Laela Sayigh, Mary Ann Daher, Julie Allen, Helen Gordon, Katherine Joyce, Claire Stuhlmann, and Peter Tyack.  The Watkins marine mammal sound database: An online, freely accessible resource.  Proceedings of Meetings on Acoustics, 27(1):040013, 2016.
  * [41] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley.  No classification without representation: Assessing geodiversity issues in open data sets for the developing world.  In NIPS 2017 workshop: Machine Learning for the Developing World, 2017.
  * [42] Yu Shiu, K. J. Palmer, Marie A. Roch, Erica Fleishman, Xiaobai Liu, Eva-Marie Nosal, Tyler Helble, Danielle Cholewiak, Douglas Gillespie, and Holger Klinck.  Deep neural networks for automated detection of marine mammal species.  Scientific Reports, 10(607), 2020.
  * [43] Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  Information Fusion, 81:84–90, 2022.
  * [44] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [45] Dan Stowell.  Computational bioacoustics with deep learning: a review and roadmap.  PeerJ, 10:e13152, 2022.
  * [46] Larissa Sayuri Moreira Sugai, Thiago Sanna Freire Silva, Jr Ribeiro, José Wagner, and Diego Llusia.  Terrestrial passive acoustic monitoring: Review and perspectives.  BioScience, 69(1):15–25, 11 2018.
  * [47] Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie Weygandt Mathis, Frank van Langevelde, Tilo Burghardt, Roland Kays, Holger Klinck, Martin Wikelski, Iain D Couzin, Grant van Horn, Margaret C Crofoot, Charles V Stewart, and Tanya Berger-Wolf.  Perspectives in machine learning for wildlife conservation.  Nature Communications, 13(1), February 2022.
  * [48] Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin, and Yonatan Bisk.  Hear: Holistic evaluation of audio representations.  In Douwe Kiela, Marco Ciccone, and Barbara Caputo, editors, Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pages 125–145. PMLR, 06–14 Dec 2022.
  * [49] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  In NeurIPS, 2019.
  * [50] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In BlackboxNLP@EMNLP, 2018.
  * [51] P. Warden.  Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.  ArXiv e-prints, April 2018.
  * [52] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdel rahman Mohamed, and Hung yi Lee.  SUPERB: Speech processing universal performance benchmark.  In Interspeech, 2021.
  * [53] Yao-Yuan Yang, Moto Hira, Zhaoheng Ni, Anjali Chourdia, Artyom Astafurov, Caroline Chen, Ching feng Yeh, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z. Yang, Jason Lian, Jay Mahadeokar, Jeff Hwang, Ji Chen, Peter Goldsborough, Prabhat Roy, Sean Narenthiran, Shinji Watanabe, Soumith Chintala, Vincent Quenneville-B’elair, and Yangyang Shi.  TorchAudio: Building blocks for audio and speech processing.  In ICASSP, 2022.
  * [54] Sophia Yin and Brenda McCowan.  Barking in domestic dogs: context specificity and individual identification.  Animal Behaviour, 68(2):343–355, 2004.
  * [55] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  International Conference on Learning Representations, 2018.