Automatic acoustic identification of individual animals: Improving
generalisation across species and recording conditions

Dan Stowell1*, Tereza Petrusková2, Martin Šálek3,4, Pavel Linhart5

1 Machine Listening Lab, Centre for Digital Music, Queen Mary University of
London, UK
2 Department of Ecology, Faculty of Science, Charles University, Prague, Czech
Republic
3 Institute of Vertebrate Biology, The Czech Academy of Sciences, Brno, Czech
Republic
4 Faculty of Environmental Sciences, Czech University of Life Sciences Prague,
Prague, Czech Republic
5 Department of Behavioural Ecology, Faculty of Biology, Adam Mickiewicz
University, Poznań, Poland

*<EMAIL_ADDRESS>
## Abstract

Many animals emit vocal sounds which, independently from the sounds’ function,
embed some individually-distinctive signature. Thus the automatic recognition
of individuals by sound is a potentially powerful tool for zoology and ecology
research and practical monitoring. Here we present a general automatic
identification method, that can work across multiple animal species with
various levels of complexity in their communication systems. We further
introduce new analysis techniques based on dataset manipulations that can
evaluate the robustness and generality of a classifier. By using these
techniques we confirmed the presence of experimental confounds in situations
resembling those from past studies. We introduce data manipulations that can
reduce the impact of these confounds, compatible with any classifier. We
suggest that assessment of confounds should become a standard part of future
studies to ensure they do not report over-optimistic results. We provide
annotated recordings used for analyses along with this study and we call for
dataset sharing to be a common practice to enhance development of methods and
comparisons of results.

Keywords: animal communication; individual differences; individuality;
acoustic monitoring; song repertoire; vocalisation.

##  1 Introduction

Animal vocalisations exhibit consistent individually-distinctive patterns,
often referred to as acoustic signatures. Individual differences in acoustic
signals have been reported universally across vertebrate species (e.g., fish
[1], amphibians [2], birds [3], mammals [4]). Individual differences may arise
from various sources, for example: distinctive fundamental frequency and
harmonic structure of acoustic signal can result from individual vocal tract
anatomy [4, 5]; distinct temporal or frequency modulation patterns of vocal
elements may result from inaccurate matching of innate or learned template or
can occur de novo through improvisation [6]. Such individual signatures
provide individual recognition cues for other conspecific animals, and
individual recognition based on acoustic signals is widespread among animals
[7]. Long-lasting individual recognition spanning over one or more year has
also been often demonstrated [8, 9, 10]. External and internal factors such
as, for example, sound degradation during transmission [11, 12], variable
ambient temperature [13], inner motivation state [14, 15], acquisition of new
sounds during life [16], may potentially increase variation of acoustic
signals. Despite these potential complications, robust individual signatures
were found in many taxa.

Besides being studied for their crucial importance in social interactions [17,
18, 19], individual signatures can become a valuable tool for monitoring
animals. Acoustic monitoring of individuals of various species based on vocal
cues could become a powerful tool in conservation (reviewed in [3, 20, 21]).
Classical capture-mark methods of individual monitoring involve physically
disturbing the animals of interest and might have a negative impact on health
of studied animals or their behaviour (e.g. [22, 23, 24, 25]). Also, concerns
have been raised about possible biases in demographic and behavioural studies
resulting from trap boldness or shyness of specific individuals [26].
Individual acoustic monitoring offers a great advantage of being non-invasive,
and thus can be deployed across species with fewer concerns about effect on
behaviour [3]. It also may reveal complementary or more detailed information
about species behaviour than classical methods [27, 28, 29, 30].

Despite many pilot studies [31, 28, 32, 33], automatic acoustic individual
identification is still not routinely applied. It is usually restricted to a
particular research team or even to a single research project, and eventually,
might be abandoned altogether for a particular species. Part of the problem
probably lies in the fact that methods of acoustic individual identification
were closely tailored to a single species (software platform, acoustic
features used, etc.). This is good in order to obtain the best possible
results for a particular species but it also hinders general, widespread
application because methods need to be developed from scratch for each new
species or even project. Little attention has been paid to developing general
methods of automatic acoustic individual identification (henceforth “AAII”)
which could be used across different species.

A few studies in the past have proposed to develop a general, call-type-
independent acoustic identification, working towards approaches that could be
used across different species, having simple as well as complex vocalisations
[34]. Despite promising results, most of the published papers included
vocalisations recorded within very limited periods of time (a few hours in a
day) [34, 35, 36, 37]. Hence, these studies might have failed to separate
effects of target signal and potentially confounding effects of particular
recording conditions and background sound, which have been reported as notable
problems in case of other machine learning tasks [38, 39]. Reducing such
confounds directly, by recording an animal in different backgrounds, may not
be achievable in field conditions since animals typically live within limited
home ranges and territories. However, acoustic background can change during
the breeding season due to vegetation changes or cycles in activity of
different bird species. Also, song birds may change territories in subsequent
years or even within a single season [27]. Some other studies of individual
acoustic identification, on the other hand, provided evidence that machine
learning acoustic identification can be robust in respect to possible long-
term changes in the acoustic background but did not provide evidence of being
generally usable for multiple species [30, 32]. Therefore, the challenge of
reliable generalisation of machine learning approach in acoustic individual
identification across different conditions and different species has not yet
been satisfactorily demonstrated.

###  1.1 Previous methods for automatic classification of individuals from
their vocalisations

We briefly review studies representing methods for automatic classification of
individuals. Note that in the present work, as in many of the cited works, we
set aside questions of automating the prior steps of recording focal birds and
isolating the recording segments in which they are active. It is common, in
preparing data sets, for recordists to collate recordings and manually trim
them to the regions containing the “foreground” individual of interest (often
with some background noise), discarding the regions containing only background
sound. In the present work we will make use of both the foreground and
background clips, and our method will be applicable whether such segmentation
is done manually or automatically.

Matching a signal against a library of templates is a well-known bioacoustic
technique, most commonly using spectrogram (sonogram) representations of the
sound, via spectrogram cross-correlation [40]. For identifying individuals,
template matching will work in principle when the individuals’ vocalisations
are strongly stereotyped with stable individual differences—and in practice
this can give good recognition results for some species [41]. However,
template matching is only applicable to a minority of species. It is strongly
call-type dependent and requires a library covering all of the vocalisation
units that are to be identified. It is unlikely to be useful for species which
have a very large vocabulary, high variability, or whose vocabulary changes
substantially across seasons.

An approach which can be more independent of call type is that of Gaussian
mixture models (GMMs), previously used extensively in human speech technology
[42, 30]. These do not rely on a strongly fixed template but rather build a
statistical model summarising the observations (e.g. the spectral shapes) that
are likely to be produced from each individual. A particularly useful aspect
of the GMM paradigm is that it can straightforwardly incorporate the concept
of a “universal background model” (UBM), which represents not “background” as
ordinarily understood but a universal pool of the sounds that might be
produced by individuals known and unknown. It therefore allows for the
practical possibility that a given sound might come from unknown individuals
that are not part of the target set [42]. This approach has been used in
songbirds, although without testing across multiple seasons [42], and for
orangutan including across-season evaluation [30].

The GMM is a very basic statistical model, which does not incorporate any
notion of temporal structure. It thus misses out on making use of a large
amount of information in the signal. One way to improve on this, again well-
developed in human speech technology, is to apply hidden Markov models (HMMs).
HMMs are statistical models of temporal structure and have more flexibility
than template-matching. However, in general they are likely to be call-type-
dependent since they do encode the temporal structure observed in each
vocalisation. Adi et al. used HMMs for recognising individual songbirds, in
this case ortolan buntings, with a pragmatic approach to call-type dependence
[32]. They first applied HMMs to infer the call type active in a given
recording (independent of individual), and then given the call type, applied
GMMs to infer which individual was active.

Other computational approaches have been studied. Cheng et al. compared four
classifier methods, aiming to develop call-type-independent recognition across
three passerine species [37]. They found HMM and support vector machines to be
favourable among the methods they tested. However, the data used in this study
was relatively limited: it was based on single recording sessions per
individual, and thus could not test across-year performance; and the authors
deliberately curated the data to select clean recordings with minimal noise,
acknowledging that this would not be representative of realistic recordings.
Fox et al. also focused on the challenge of call-independent identification,
across three other passerine species [35, 34]. They used a neural network
classifier, and achieved good performance for their species. However, again
the data for this study was based on a single session per individual, which
makes it unclear how far the findings generalise across days and years, and
also does not fully test whether the results may be affected by confounding
factors such as recording conditions.

Computational methods for various automatic recognition tasks have recently
been dominated and dramatically improved by new trends in machine learning,
including deep learning. Within that broad field, the challenge of reliable
generalisation is far from solved, and is an active research topic. Within
bioacoustics this has recently been studied for detection of bird sounds [43].
In deep learning, it was discovered that even the best-performing deep neural
networks might be surprisingly non-robust, and could be forced to change their
decisions by the addition of tiny imperceptible amounts of background noise to
an image [38].

Note that deep learning systems also typically require very large amounts of
data to train, meaning they may currently be infeasible for tasks such as
acoustic individual ID in which the number of recordings per individual is
necessarily limited. For deep learning, “data augmentation” has been used to
expand dataset sizes. Data augmentation refers to the practice of
synthetically creating additional data items by modifying or recombining
existing items. In the audio domain, this could be done for example by adding
noise, filtering, or mixing audio clips together [44]. However, simple
unprincipled data augmentation does not reduce issues such as undersampling
(e.g. some vocalisations unrepresented in data set) or confounding factors.

There thus remains a gap in applying machine learning for automatic individual
identification as a general-purpose tool that can be shown to be reliable for
multiple species and can generalise correctly across recording conditions.

In the work reported in this paper, we tested generalisation of machine
learning across species and across recording conditions in context of
individual acoustic identification. We used extensive data for three different
bird species, including repeated recordings of the same individuals within and
across two breeding seasons. As well as directly evaluating across seasons, we
also introduced ways to modify the evaluation data to probe the generalisation
properties of the classifier. We then improved on the baseline approach by
developing novel methods which help to improve generalisation performance,
again by modifying the data used. Although tested with selected species and
classifiers, our approach of modifying the data rather than the classification
algorithm was designed to be compatible with a wide variety of automatic
identification workflows.

##  2 Materials and methods

###  2.1 Data collection

For this study we chose three bird species of varying vocal complexity (Figure
1), in order to explore how a single method might apply to the same task at
differing levels of difficulty and variation. Little owl (Athene noctua)
represents a species with simple vocalisation (Figure 1a): territorial call is
a single syllable which is individually unique and it is held to be stable
over time (Linhart and Šálek unpubl. data) as was shown in several other owl
species (e.g. [31, 45]). Then, we selected two passerine species, which
exhibit vocal learning: chiffchaff (Phylloscopus collybita) and tree pipit
(Anthus trivialis). Tree pipit songs are also individually unique and stable
over time [27]; but male on average uses 11 syllable types (6-18) which are
repeated in phrases that can be variably combined to create a song ([46],
Figure 1b). Chiffchaff song, when visualised, may seem simpler than that of
the pipit. However, the syllable repertoire size might actually be higher—9 to
24 types—and, contrary to the other species considered, chiffchaff males may
change syllable composition of their songs over time ([47], (Figure 1c).
Selected species also differ in their ecology. While little owls are sedentary
and extremely faithful to their territories [48], tree pipits and chiffchaffs
belong to migratory species with high fidelity to their localities. Annual
returning rates for both are 25% to 30% ([27], Linhart unpubl. data).

Figure 1: Example spectrograms representing our three study species:
(a) little owl (b) tree pipit (c) chiffchaff.

For each of these species, we used targeted recordings of single vocally
active individuals. Distance to the recorded individual varied across
individuals and species according to their tolerance towards people. We tried
to get the best recording and minimise distance to each singing individual
without disturbing its activities. Recordings were always done under favorable
weather conditions (no rain, no strong wind). In general, signal-to-noise
ratio is very good in all of our recordings (not rigorously assessed), but
there are also environmental sounds, sounds from other animals or conspecifics
in the recording background. All three species were recorded with following
equipment: Sennheiser ME67 microphone, Marantz PMD660 or 661 solid-state
recorder (sampling frequency 44.1 kHz, 16 bit, PCM).

Little owl (Linhart and Šálek 2017) [49]: Little owls were recorded in two
Central European farmland areas: northern Bohemia, Czech Republic (50°23’N,
13°40’E), and eastern Hungary (47°33ˈN, 20°54ˈE). Recordings were made from
sunset until midnight between March and April of 2013—2014. Territorial calls
of each male were recorded for up to three minutes after a short playback
provocation (1 min) inside their territories from up to 50 m distance from the
individuals. Identity of the males could not be explicitly checked because
only a small proportion of males were ringed. Therefore, we inferred identity
by the territory location combined with the call frequency modulation pattern
which is distinctive per individual.

Chiffchaff (Průchová et al 2017 [47], Ptáček et al 2016 [42]): Chiffchaff
males were recorded in a former military training area on the outer boundary
of České Budějovice town, the Czech Republic (48°59.5ˈN, 14°26.5ˈE). Males
were recorded for the purposes of various studies from 2008 up to and
including 2011. Recordings were done from 05:30 to 11:00 hours in the morning.
Only spontaneously singing males were recorded from within about 5–15 m
distance. Identity of males was confirmed by colour rings.

Tree Pipit (Petrusková et al. 2015 [27]): Tree Pipit males were recorded at
the locality Brdská vrchovina, the Czech Republic (49°84N, 14°10E) where the
population has been continuously studied since 2011. Spontaneously singing
males were recorded throughout whole day according to the natural singing
activity of Tree pipits from mid-April to mid-July. Males were identified
either based on colour ring observations or their song structure [27].

Audio files were divided into regions during which the focal individual was
vocally active (“foreground”) and inactive (“background”). The total numbers
of individuals and sound files in each dataset are summarised in Table 1.

Table 1: Details of the audio recording datasets used. Evaluation scenario |  Num. of inds |  Foreground # audio files (train : eval) |  Foreground total minutes (train : eval) |  Background # audio files (train : eval) |  Background total minutes (train : eval)
---|---|---|---|---|---
Chiffchaff within-year | 13 | 5107 : 1131 | 451 : 99 | 5011 : 1100 | 453 : 92
Chiffchaff only-15 | 13 |  195 : 1131 | 18 : 99 |  195 : 1100 | 21 : 92
Chiffchaff across-year | 10 | 324 : 201 | 32 : 20 | 304 : 197 | 31 : 24
Little owl across-year | 16 | 545 : 407 | 11 : 8 | 546 : 409 | 34 : 27
Pipit within-year | 10 | 409 : 303 | 27 : 21 | 398 : 293 | 49 : 47
Pipit across-year | 10 | 409 : 313 | 27 : 19 | 398 : 306 | 49 : 37

###  2.2 Structured data augmentation

“Data augmentation” in machine learning refers to creating artificially large
or diverse data sets by synthetically manipulating items in data sets to
create new items—for example, by adding noise or performing mild distortions.
These artificially enriched data sets, used for training, often lead to
improved automatic classification results, helping to mitigate the effects of
limited data availability [50, 51]. Data augmentation is increasingly used in
machine learning applied to audio. Audio-specific manipulations used might
include filtering or pitch-shifting, or the mixing together of audio files
(i.e. summing their signals together) [52, 53]. Some of the highest-performing
automatic species recognition systems rely in part on such data augmentations
to attain their strongest results [44].

In this work, we describe two augmentation methods used specifically to
evaluate and to reduce the confounding effect of background sound. These
structured data augmentations are based on audio mixing but with the
combinations of files to mix selected based on foreground and background
identity metadata. We make use of the fact that when recording audio from
focal individuals in the wild, it is common to obtain recording clips in which
the focal individual is vocalising (Figure 2a), as well as ‘background’
recordings in which the vocal individual is silent (Figure 2b). The latter are
commonly discarded. We used them as follows:

Adversarial data augmentation:

To evaluate the extent to which confounding from background information is an
issue, we created datasets in which each foreground recording has been mixed
with one background recording from some other individual (Figure 2c). In the
best case, this should make no difference, since the resulting sound clip is
acoustically equivalent to a recording of the foreground individual, but with
a little extra irrelevant background noise. In fact it could be considered a
synthetic test of the case in which an individual is recorded having travelled
out of their home range. In the worst case, a classifier that has learnt
undesirable correlations between foreground and background will be misled by
the modification, either increasing the probability of classifying as the
individual whose territory provided the extra background, or simply confusing
the classifier and reducing its general ability to classify well. In our
implementation, each foreground item was used once, each mixed with a
different background item. Thus the evaluation set remains the same size as
the unmodified set. We evaluated the robustness of a classifier by looking at
any changes in the overall correctness of classification, or in more detail
via the extent to which the classifier outputs are modified by the adversarial
augmentation.

Stratified data augmentation:

We can use a similar principle during the training process, to create an
enlarged and improved training data set. We created training datasets in which
each training item had been mixed with an example of background sound from
each other individual (Figure 2d). If there are $K$ individuals this means
that each item is converted into $K$ synthetic items, and the data set size
increases by a factor of $K$. Stratifying the mixing in this way, rather than
selecting background samples purely at random, is intended to expose a
classifier to training data with reduced correlation between foreground and
background, and thus reduce the chance that it uses confounding information in
making decisions.

(a) ‘Foreground’ recordings, which also contain some signal content coming
from the background habitat. The foreground and background might not vary
independently, especially in the case of territorial animals.

(b) ‘Background’ recordings, recorded when the focal animal is not vocalising

$\rightarrow$classify vs. $\rightarrow$classify

(c) In adversarial data augmentation, we mix each foreground recording with a
background recording from another individual, and measure the extent to which
this alters the classifier’s decision.

… $\rightarrow$train

(d) In stratified data augmentation, each foreground recording is mixed with a
background recording from each other class. This creates a to reduce the
confounding correlation in the training data.

Figure 2: Explanatory illustration of our data augmentation interventions.

To implement the foreground and background audio file mixing, we used the sox
processing tool v14.4.1.

###  2.3 Using background items directly

Alongside our data augmentation, we can also consider simple interventions in
which the background sound recordings are used alone without modification.

One way of diagnosing confounding-factor issues in AAII is to apply the
classifier to background-only sound recordings. If there are no confounds in
the trained classifier, trained on foreground sounds, then it should be unable
to identify the corresponding individual for any given background-only sound
(identifying ‘a’ or ‘b’ in Figure 2b). Automatic identification (“AAII”) for
background-only sounds should yield results at around chance level.

A second use of using the background-only recordings is to create an explicit
‘wastebasket’ class during training. As well as training the classifier to
recognise individual labels A, B, C, …, we created an additional ‘wastebasket’
class which should be recognised as ‘none of the above’, or in this case,
explicitly as ‘background’. The explicit-background class may or may not be
used in the eventual deployment of the system. Either way, its inclusion in
the training process could help to ensure that the classifier learns not to
make mistaken associations with the other classes. This approach is related to
the universal background model (UBM) used in open-set recognition methods
[42]. Note that the ‘background’ class is likely to be different in kind from
the other classes, having very diverse sounds. In methods with an explicit
UBM, the background class can be handled differently than the others [42].
Here, we chose to use methods that can work with any classifier, and so the
background class was simply treated analogously to the classes of interest.

###  2.4 Automatic classification

(a) A standard workflow for automatic audio classification. The upper portion
shows the training procedure, and the lower shows the application or
evaluation procedure.

(b) Workflow for our automatic classification experiments. Dashed boxes
represent steps which we enable/disable as part of our experiment. The upper
portion shows the training procedure, and the lower shows the evaluation
procedure. The two portions are very similar. However, note that the purpose
and method of augmentation is different in each, as is the use of background-
only audio: in the training phase the ‘concatenation’ block creates an
enlarged training set as the union of the background items and the foreground
items, while in the evaluation phase the ‘choose’ block select only one of the
two, for the system to make predictions about.

Figure 3: Classification workflows.

In this work, we started with a standard automatic classification processing
workflow (Figure 3a), and then experimented with inserting our proposed
improvements. We modified the feature processing stage, but our main
innovations in fact came during the data set preparation stage, using the
foreground and/or background data sets in various combinations to create
different varieties of training and testing data (Figure 3b).

As in many other works, the audio files—which in this case may be the
originals or their augmented versions—were not analysed in their raw waveform
format, but were converted to a mel spectrogram representation: ‘mel’
referring to a perceptually-motivated compression of the frequency axis of a
standard spectrogram. We used audio files (44.1 kHz mono) converted into
spectrograms using frames of length 1024 (23 ms), with Hamming windows, 50%
frame overlap, and 40 mel bands. We applied median-filtering noise reduction
to the spectrogram data.

Following the findings of [54], we also applied unsupervised feature learning
to the mel spectrogram data as a preprocessing step. This procedure scans
through the training data in unsupervised fashion (i.e. neglecting the data
labels), finding a linear projection that provides an informative
transformation of the data. We evaluated the audio feature data with and
without this feature learning step, to evaluate whether the data
representation had an impact on the robustness and generalisability of
automatic classification. In other words, as input to the classifier we used
either the mel spectrograms, or the learned representation obtained by
transforming the mel spectrogram data.

The automatic classifier we used was one based on a random forest classifer
that was previously tested successfully for bird species classification, but
had not been tested for AAII [54].

###  2.5 Evaluation

As is standard in automatic classification evaluation, we divided our datasets
into portions used for training the system, and portions used for evaluating
system performance. Items used in training were not used in evaluation, and
the allocation of items to the training or evaluation sets was done to create
a partitioning through time: evaluation data came from different days within
the breeding season, or subsequent years, than the training data. This
corresponds to a plausible use-case in which a system is trained with existing
recordings and then deployed; the partitioning also helps to reduce the
probability of over-estimating performance.

To quantify performance we used receiver operating curve (ROC) analysis, and
as a summary statistic the area under the ROC curve (AUC). The AUC summarises
classifier performance and has various desirable properties for evaluating
classification [55].

We evaluated the classifiers following the standard paradigm used in machine
learning. Note that during evaluation, we optionally modified the evaluation
data sets in two possible ways, as already described: adversarial data
augmentation, and background-only classification. In all cases we used AUC as
the primary evaluation measure. However, we also wished to probe the effect of
adversarial data augmentation in finer detail: even when the overall decisions
made by a classifier are not changed by modifying the input data, there may be
small changes in the full set of probabilities it outputs. A classifier that
is robust to adversarial augmentation should be one whose probabilities change
little if at all. Hence for the adversarial augmentation test, we also took
the probabilities output from the classifier and compared them against their
equivalent probabilities from the same classifier in the non-adversarial case.
We measured the difference between these sets of probabilities simply by their
root-mean-square error (RMS error).

###  2.6 Phase one: testing with chiffchaff

For our first phase of testing, we wished to compare the effectiveness of the
different proposed interventions, and their relative effectiveness on data
tested within-year or across-year. We chose to use the chiffchaff datasets for
these tests, since the chiffchaff song has an appropriate level of complexity
to elucidate the differences between classifier performance, in particular the
possible change of syllable composition across years. The chiffchaff dataset
is also by far the largest.

We wanted to explore the difference in estimated performance when evaluating a
system with recordings from the same year, separated by days from the training
data, versus recordings from a subsequent year. In the latter case, the
background sounds may have changed intrinsically, or the individual may have
moved to a different territory; and of course the individual’s own
vocalisation patterns may change across years. This latter effect may be an
issue for AAII with a species such as the chiffchaff, and also impose limits
to the application of previous approaches such as template-based matching.
Hence we wanted to test whether this more flexible machine learning approach
could detect individual signature in the chiffchaff even when applied to data
from a different field season. We thus evaluated performance on ‘within-year’
data—recordings from the same season—and ‘across-year’ data—recordings from
the subsequent year, or a later year.

Since the size of data available is often a practical constraint in AAII, and
since dataset size can have a strong influence on classifier performance, we
further performed a version of the ‘within-year’ test in which the training
data had been restricted to only 15 items per individual. The evaluation data
was not restricted.

To evaluate formally the effect of the different interventions, we applied
generalised linear mixed models (GLMM) to our evaluation statistics, using the
glmmadmb package within R version 3.4.4 [56, 57]. Since AUC is a continuous
value constrained to the range $[0,1]$, we used a beta link function. Since
RMSE is a non-negative error measure, we used a gamma family with a
logarithmic link function. For each of these two evaluation measures, we
applied a GLMM, using the data from all three evaluation scenarios (within-
year, cross-year, only-15). The evaluation scenario was included as a random
effect. Since the same evaluation-set items were reused in differing
conditions, this was a repeated-measures model with respect to the individual
song recordings.

###  2.7 Phase two: testing multiple species

In the second phase of our investigations, we evaluated the selected approach
across the three species separately: chiffchaff, pipit and little owl. For
each of these we compared the most basic version of the classifier (using mel
features, no augmentation, and no explicit-background) against the improved
version that was selected from phase one of the investigation. For each
species separately, and using within-year and across-year data according to
availability, we evaluated the basic and the improved classifier for the
overall performance (AUC measured on foreground sounds). We also evaluated
their performance on background-only sounds, and on the adversarial data
augmentation test, both of which checked the relationship between improved
classification performance and improvements or degradations in the handling of
confounding factors.

For both of these tests (background-only testing and adversarial
augmentation), we applied GLMM tests similar to those already stated. In these
cases we entered separate factors for the testing condition and for whether
the improved classifier was in use, as well as an interaction term between the
two factors. This therefore tested for an effect of whether our improved
classifier indeed mitigated the problems that the tests were designed to
expose.

##  3 Results

###  3.1 Phase one: chiffchaff

AAII performance over the 13 chiffchaff individuals was strong, above 85% AUC
in all variants of the within-year scenario (Figure 4). For interpretation,
note that this corresponds to over 85% probability that a random true-positive
item is ranked higher than a random true-negative item by the system [55].
This reduced to around 70–80% when the training set was limited to 15 items
per individual, and reduced even further to around 60% in the across-year
evaluation scenario. Recognising chiffchaff individuals across years remains a
challenging task even under the studied interventions.

The focus of our study is on discriminating between individuals, but our
“explicit-background” configuration additionally made it possible for the same
classifier to discriminate between cases where a focal individual was singing,
and cases where it was not. Across all three of the conditions mentioned
above, foreground-vs-background discrimination (aka “detection” of any focal
individual) for chiffchaff was strong at over 95% AUC. Mel spectral features
performed slightly better for this (range 96.6–98.6%) than learnt features
(range 95.3–96.7%). Given this, in the remainder of the results we focus on
our main question of discriminating between individuals.

Figure 4: Performance of classifier (AUC) across the three chiffchaff
evaluation scenarios, and with various combinations of configuration:
with/without augmentation (‘aug’), learnt features, and explicit-background
(‘exbg’) training.

We tested the GLMM residuals for the two evaluation measures (AUC, RMSE) and
found no evidence for overdispersion. We also tested all possible reduced
models with factors removed, comparing among models using AIC. In both cases,
the full model as well as a model with ‘exbg’ (explicit-background training)
removed gave the best fit, with the full model less than 2 units above the
exbg-reduced model and leading to no difference in significance estimates. We
therefore report results from the full models.

Feature-learning and structured data augmentation were both found to
significantly improve classifier performance (Table 2) as well as robustness
to adversarial data augmentation (Table 3). Explicit-background training was
found to lead to mild improvement but this was a long way below significance.

Table 2: Results of GLMM test for AUC, across the three chiffchaff evaluation scenarios. | Estimate | p-value
---|---|---
(Intercept) | 0.8199 | 0.041 *
Feature-learning | 0.3093 | 0.014 *
Augmentation | 0.2509 | 0.048 *
Explicit-bg class | 0.0626 | 0.621
Table 3: Results of GLMM fit for RMSE in the adversarial data augmentation test, across the three chiffchaff evaluation scenarios. | Estimate | p-value
---|---|---
(Intercept) | 1.8543 | 1.9e-05 ***
Feature-learning | -0.5044 | 1.9e-08 ***
Augmentation | -0.8734 |  $<$ 2e-16 ***
Explicit-bg class | -0.0141 | 0.87

###  3.2 Phase two: multiple species

Based on the results of our first study, we took forward an improved version
of the classifier (using stratified data augmentation, and learnt features,
but not explicit-background training) to test across multiple species.

Applying this classifier to the different species and conditions, we found
that it led in most cases to a dramatic improvement in recognition performance
of foreground recordings, and little change in the recognition of background
recordings (Figure 5, Table 4). This suggests that the improvement is based on
the individuals’ signal characteristics and not confounding factors.

Figure 5: Our selected interventions—data augmentation and feature-
learning—improve classification performance, in some cases dramatically (left-
hand pairs of points), without any concomitant increase in the background-only
classification (right-hand pairs of points) which would be an indication of
counfounding.

Our adversarial augmentation, intended as a diagnostic test to adversarially
reduce classification performance, did not have strong overall effects on the
headline performance indicated by the AUC scores (Figure 6, Table 4). Half of
the cases examined—the across-year cases—were not adversely impacted, in fact
showing a very small increase in AUC score. The chiffchaff within-year tests
were the only to show a strong negative impact of adversarial augmentation,
and this negative impact was removed by our improved classification method.

We also conducted a more fine-grained analysis of the effect of augmentation,
by measuring the amount of deviation induced in the probabilities output from
the classifier. On this measure we observed a consistent effect, with our
improvements reducing the RMS error by ratios of approx 2–6, while the overall
magnitude of the error differed across species (Figure 7).

Figure 6: Adversarial augmentation has a varied impact on classifier performance (left-hand pairs of points), in some cases giving a large decline. Our selected interventions vastly reduce the impact of this adversarial test, while also generally improving classification performance (right-hand pairs of points). Table 4: Results of GLMM test for AUC, across all three species, to quantify the general effect of our improvements on the foreground test and the background test (cf. Figure 5). | Estimate | p-value
---|---|---
(Intercept) | 0.792 | 0.00150 **
Use of improved classifier | 0.852 | 0.00032 ***
Background-only testing | -0.562 | 0.00624 **
Interaction term | -0.896 | 0.00391 **
Table 5: Results of GLMM test for AUC, across all three species, to quantify the general effect of our improvements on the adversarial test (cf. Figure 6). | Estimate | p-value
---|---|---
(Intercept) | 0.873 | 0.0121 *
Use of improved classifier | 0.820 | 0.0027 **
Adversarial data augmentation | -0.333 | 0.1713
Interaction term | 0.225 | 0.5520
Figure 7: Measuring in detail how much effect the adversarial augmentation has
on classifier decisions: RMS error of classifier output, in each case applying
adversarial augmentation and then measuring the differences compared against
the non-adversarial equivalent applied to the exact same data. In all five
scenarios, our selected interventions lead to a large decrease in the RMS
error.

##  4 Discussion

We demonstrate that a single approach to automatic acoustic identification of
individuals (AAII) can be successfully used across different species with
different complexity of vocalisations. One exception to this is the hardest
case, chiffchaff tested across years, in which automatic classification
performance remains modest. The chiffchaff case (complex song, variable song
content), in particular, highlights the need for proper assessment of
identification performance. Without proper assessment we cannot be sure if
promising results reflect the real potential of proposed identification
method. We document that our proposed improvements to the classifier training
process are able, in some cases, to improve the generalisation performance
dramatically and, on the other hand, reveal confounds causing over–optimistic
results.

We evaluated spherical k-means feature-learning as previously used for species
classification [54]. We found that for individual identification it provides
an improvement over plain Mel spectral features, not just in accuracy (as
previously reported) but also in resistance to confounding factors (ibid.). We
believe this is due to the feature-learning having been tailored to reflect
fine temporal details of bird sound; if so, this lesson would carry across to
related systems such as convolutional neural networks. Our machine-learning
approach may be particularly useful for automatic identification of
individuals in species with more complex songs, such as pipits (note huge
increase in performance over mel features in Figure 5), or chiffchaffs (on
short-time scale though).

Using silence-regions from focal individuals to create an “explicit-
background” training category provided only a mild improvement in the
behaviour of the classifier, under various evaluations. Also, we found that
the best-performing configuration used for detecting the presence/absence of a
focal individual was not the same as the best-performing configuration for
discriminating between individuals. Hence, it seems generally preferable not
to combine the detection and AAII tasks into one classifier.

By contrast, using silence-regions to perform dataset augmentation of the
foreground sounds was found to give a strong boost to performance as well as
resistance against confounding factors. Background sounds are useful in
training a system for AAII, through data augmentation (rather than explicit-
background training).

We found that adversarial augmentation provided a useful tool to diagnose
concerns about the robustness of an AAII system. In the present work we found
that the classifier was robust against this augmentation (and thus we can
infer that it was largely not using background confounds to make its
decision), except for the case of chiffchaff with the simple mel features
(Figure 6). This latter case exhorts us to be cautious, and suggests that
results from previous call-type independent methods may have been over-
optimistic in assessing performance [34, 35, 36, 37, 42]. Our adversarial
augmentation method can help to test for this even in the absense of across-
year data.

Background-only testing was useful to confirm that when the performance of a
classifier was improved, the confounding factors were not aggravated in
parallel, i.e. that the improvement was due to signal and not confound (Figure
5). However, the performance on background sound recordings was not reduced to
chance, but remained at some level reflecting the foreground-background
correlations in each case, so results need to interpreted comparatively
against the foreground improvement, rather than in isolation. This individual
specificity of the background may be related to the time interval between
recordings. This is clear from the across-year outcomes; within-year, we note
that there was one day of temporal separation for chiffchaffs (close to 70
percent AUC on background-only sound), while an interval of weeks for pipits
(chance-level classification of background). These effects surely depend on
characteristics of the habitat.

Our improved classifier performs much more reliably than the standard one;
however, the most crucial factor still seems to be a targeted species. For the
little owl we found good performance, and least affected by modifications in
methods - consistent with the fact that it is the species with the simplest
vocalisations. Little owl represents a species well suited for template
matching individual identification methods which have been used in past for
many species with similar simple, fixed vocalisations (discriminant analysis,
cross-correlation). For these cases, it seems that our automatic
identification method does not bring advantage regarding improved
classification performance. However, a general classifier such as ours,
automatically adjusting a set of features for each species, would allow common
users to start individual identification right away without the need to choose
an appropriate template-matching method (e.g. [49]).

We found that feature learning gave the best improvement in case of pipits
(Figure 5). Pipits have more complex song, where simple template matching
cannot be used to identify individuals. In pipits, each song may have
different duration and may be composed of different subsets of syllable
repertoire, and so any a single song cannot be used as template for template
matching approach. This singing variation likely also prevents good
identification performance based on Mel features in pipits. Nevertheless, a
singing pipit male will cycle through the whole syllable repertoire within a
relatively low number of songs and individual males can be identified based on
their unique syllable repertoires ([27]). We think that our improvements to
the automatic identification might allow the system to pick up correct
features associated with stable repertoire of each male. This extends the use
of the same automatic identification method to the large part of songbird
species that organise songs into several song types and, at the same time, are
so-called closed-ended learners ([58]).

Our automatic identification, however, cannot be considered fully independent
of song content in a sense defined earlier (e.g.[34, 36]). Such content-
independent identification method should be able to classify across-year
recordings of chiffchaffs in which syllable repertoires of males differ almost
completely between the two years [47]. Due to vulnerability of Mel feature
classification to confounds reported here and because performance of content
independent identification has been only tested on short-term recordings, we
believe that the concept of fully content-independent individual
identification needs to be reliably demonstrated yet.

Our approach seems to be definitely suitable for species with individual
vocalisation stable over time, even if that vocalisation is complex—a very
wide range of species—in general outdoor conditions. For such species it might
be successfully used for individual automatic acoustic monitoring, although
this needs to be tested at larger scale: in various species and in large
populations. In future work these approaches should also be tested with ‘open-
set’ classifiers allowing for the possibility that new unknown individuals
might appear in data. This is well-developed in the “universal background
model” (UBM) developed in GMM-based speaker recognition [42], and future work
in machine learning is needed to develop this for the case of more powerful
classifiers.

Important for further work in this topic is open sharing of data in standard
formats. Only this way can diverse datasets from individuals be used to
develop/evaluate automatic recognition that works across many taxa and
recording conditions.

We conclude by listing the recommendations that emerge from this work for
users of automatic classifiers, in particular for acoustic recognition of
individuals:

  1. 1.

Record ‘background’ segments, for each individual (class), and publish
background audio samples alongside the trimmed individual audio samples.
Standard data repositories can be used for these purposes (e.g. Dryad,
Zenodo).

  2. 2.

Improve robustness by:
(a) suitable choice of input features;
(b) structured data augmentation, using background sound recordings.

  3. 3.

Probe your classifier for robustness by:
(a) background-only recognition: higher-than-chance recognition strongly
implies confound;
(b) adversarial distraction with background: a large change in classifier
outputs implies confound;
(c) across-year testing (if such data are available): a stronger test than
within-year.

  4. 4.

Be aware of how species characteristics will affect recognition. The
vocalisation characteristics of the species will influence the ease with which
automatic classifiers can identify individuals. Songbirds whose song changes
within and between seasons will always be harder to identify reliably - as is
also the case in manual identification.

  5. 5.

Best practice is to test manual features and learned features since the
generalisation and performance characteristics are rather different. In the
present work we compare basic features against learned features; for a
different example see [12]. Manual features are usually of lower accuracy, but
with learned features more care must be taken with respect to confounds and
generalisation.

## Ethics

Our study primarily involved only non-invasive recording of vocalising
individuals. In the case of ringed individuals (all chiffchaffs and some tree
pipits and little owls), ringing was done by experienced ringers (PL, MŠ, TP)
who all held ringing licences at the time of study. Tree pipits and chiffchaff
males were recorded during spontaneous singing. Only for little owls short
playback recording (1 min) was used to provoke calling. Playback provocations
as well as handling during ringing were kept as short as possible and we are
not aware of any consequences for subjects’ breeding or welfare.

## Data Accessibility

Our audio data and the associated metadata files are available online under
the Creative Commons Attribution licence (CC BY 4.0) at
http://doi.org/10.5281/zenodo.1413495

## Competing Interests

We have no competing interests.

## Authors’ Contributions

DS and PL conceived and designed the study. PL, TP and MŠ recorded audio. PL
processed the audio recordings into data sets. DS carried out the
classification experiments and performed data analysis. DS, PL and TP wrote
the manuscript. All authors gave final approval for publication.

## Funding

DS was supported by EPSRC Early Career research fellowship EP/L020505/1. PL
was supported by the National Science Centre, Poland, under Polonez fellowship
reg. no UMO-2015/19/P/NZ8/02507 funded by the European Union’s Horizon 2020
research and innovation programme under the Marie Skłodowska-Curie grant
agreement No 665778. TP was supported by the Czech Science Foundation (project
P505/11/P572).MŠ was supported by the research aim of the Czech Academy of
Sciences (RVO 68081766).

## References

  * 1. Amorim MCP, Vasconcelos RO.  Variability in the mating calls of the Lusitanian toadfish Halobatrachus didactylus: cues for potential individual recognition.  Journal of Fish Biology. 2008;73:1267–1283.
  * 2. Bee MA, Gerhardt HC.  Neighbour-stranger discrimination by territorial male bullfrogs (Rana catesbeiana): I. Acoustic basis.  Animal Behaviour. 2001;62:1129–1140.
  * 3. Terry AM, Peake TM, McGregor PK.  The role of vocal individuality in conservation.  Frontiers in Zoology. 2005;2(1):10.
  * 4. Taylor AM, Reby D.  The contribution of source-filter theory to mammal vocal communication research.  Journal of Zoology. 2010;280(3):221–236.  Available from: http://onlinelibrary.wiley.com/doi/10.1111/j.1469-7998.2009.00661.x/abstract.
  * 5. Gamba M, Favaro L, Araldi A, Matteucci V, Giacoma C, Friard O.  Modeling individual vocal differences in group-living lemurs using vocal tract morphology.  CURRENT ZOOLOGY. 2017;63(4):467–475.
  * 6. Janik V, Slater PB.  Vocal Learning in Mammals.  vol. Volume 26. Academic Press; 1997. p. 59–99.  Available from: http://www.sciencedirect.com/science/article/pii/S0065345408603770.
  * 7. Wiley RH.  Specificity and multiplicity in the recognition of individuals: implications for the evolution of social behaviour.  Biological Reviews. 2013;88(1):179–195.  WOS:000317066700011.
  * 8. Boeckle M, Bugnyar T.  Long-Term Memory for Affiliates in Ravens.  Current Biology. 2012;22(9):801–806.  Available from: http://www.sciencedirect.com/science/article/pii/S0960982212003107.
  * 9. Insley SJ.  Long-term vocal recognition in the northern fur seal : Article : Nature.  Nature. 2000;406(6794):404–405.  Available from: http://www.nature.com/nature/journal/v406/n6794/full/406404a0.html.
  * 10. Briefer EF, de la Torre MP, McElligott AG.  Mother goats do not forget their kids' calls.  Proceedings of the Royal Society B: Biological Sciences. 2012 jun;279(1743):3749–3755.
  * 11. Slabbekoorn H.  Singing in the wild: the ecology of birdsong.  In: Marler P, Slabbekoorn H, editors. Nature’s music: the science of birdsong. Elsevier Academic Press; 2004. p. 178–205.
  * 12. Mouterde SC, Elie JE, Theunissen FE, Mathevon N.  Learning to cope with degraded sounds: Female zebra finches can improve their expertise at discriminating between male voices at long distance.  The Journal of Experimental Biology. 2014;p. jeb–104463.
  * 13. Gambale PG, Signorelli L, Bastos RP.  Individual variation in the advertisement calls of a Neotropical treefrog (Scinax constrictus).  Amphibia-Reptilia. 2014;35(3):271–281.  Available from: http://booksandjournals.brillonline.com/content/journals/10.1163/15685381-00002949.
  * 14. Collins SA.  Vocal fighting and flirting: the functions of birdsong.  In: Marler PR, Slabbekoorn H, editors. Nature’s music: the science of birdsong. Elsevier Academic Press; 2004. p. 39–79.
  * 15. Linhart P, Jaška P, Petrusková T, Petrusek A, Fuchs R.  Being angry, singing fast? Signalling of aggressive motivation by syllable rate in a songbird with slow song.  Behavioural Processes. 2013;100:139–145.  Available from: http://www.sciencedirect.com/science/article/pii/S0376635713001927.
  * 16. Kroodsma DE.  The diversity and plasticity of bird song.  In: Marler PR, Slabbekoorn H, editors. Nature’s music: the science of birdsong. Elsevier Academic Press; 2004. p. 108–131.
  * 17. Thom MDF, Dytham C.  Female Choosiness Leads to the Evolution of Individually Distinctive Males.  Evolution. 2012;66(12):3736–3742.  WOS:000312218200008.
  * 18. Bradbury JW, Vehrencamp SL.  Principles of animal communication.  1st ed. Sinauer Associates; 1998.
  * 19. Crowley PH, Provencher L, Sloane S, Dugatkin LA, Spohn B, Rogers L, et al.  Evolving cooperation: the role of individual recognition.  Biosystems. 1996;37(1):49–66.  Available from: http://www.sciencedirect.com/science/article/pii/0303264795015469.
  * 20. Mennill DJ.  Individual distinctiveness in avian vocalizations and the spatial monitoring of behaviour.  Ibis. 2011;153(2):235–238.  Available from: http://onlinelibrary.wiley.com/doi/10.1111/j.1474-919X.2011.01119.x/abstract.
  * 21. Blumstein DT, Mennill DJ, Clemins P, Girod L, Yao K, Patricelli G, et al.  Acoustic monitoring in terrestrial environments using microphone arrays: applications, technological considerations and prospectus.  Journal of Applied Ecology. 2011;48(3):758–767.  Available from: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2664.2011.01993.x/abstract.
  * 22. Johnsen A, Lifjeld J, Rohde PA.  Coloured leg bands affect male mate-guarding behaviour in the bluethroat.  Animal Behaviour. 1997;54(1):121–130.
  * 23. Gervais JA, Catlin DH, Chelgren ND, Rosenberg DK.  Radiotransmitter mount type affects burrowing owl survival.  The Journal of wildlife management. 2006;70(3):872–876.
  * 24. Linhart P, Fuchs R, Poláková S, Slabbekoorn H.  Once bitten twice shy: long-term behavioural changes caused by trapping experience in willow warblers Phylloscopus trochilus.  Journal of avian biology. 2012;43(2):186–192.
  * 25. Rivera-Gutierrez HF, Pinxten R, Eens M.  Songbirds never forget: long-lasting behavioural change triggered by a single playback event.  Behaviour. 2015;152(9):1277–1290.  Available from: http://booksandjournals.brillonline.com/content/journals/10.1163/1568539x-00003278.
  * 26. Camacho C, Canal D, Potti J.  Lifelong effects of trapping experience lead to age-biased sampling: lessons from a wild bird population.  Animal Behaviour. 2017;130:133–139.  Available from: http://www.sciencedirect.com/science/article/pii/S0003347217301938.
  * 27. Petrusková T, Pišvejcová I, Kinštová A, Brinke T, Petrusek A.  Repertoire-based individual acoustic monitoring of a migratory passerine bird with complex song as an efficient tool for tracking territorial dynamics and annual return rates.  Methods in Ecology and Evolution. 2015 nov;7(3):274–284.  Available from: https://doi.org/10.1111%2F2041-210x.12496.
  * 28. Laiolo P, Vögeli M, Serrano D, Tella JL.  Testing acoustic versus physical marking: two complementary methods for individual-based monitoring of elusive species.  Journal of Avian Biology. 2007;38(6):672–681.
  * 29. Kirschel ANG, Cody ML, Harlow ZT, Promponas VJ, Vallejo EE, Taylor CE.  Territorial dynamics of Mexican Ant-thrushes Formicarius moniliger revealed by individual recognition of their songs.  Ibis. 2011;153:255–268.
  * 30. Spillmann B, van Schaik CP, Setia TM, Sadjadi SO.  Who shall I say is calling? Validation of a caller recognition procedure in Bornean flanged male orangutan (Pongo pygmaeus wurmbii) long calls.  Bioacoustics. 2017;26(2):109–120.
  * 31. Delport W, Kemp AC, Ferguson JWH.  Vocal identification of individual African Wood Owls Strix woodfordii: a technique to monitor long-term adult turnover and residency.  Ibis. 2002;144:30–39.
  * 32. Adi K, Johnson MT, Osiejuk TS.  Acoustic censusing using automatic vocalization classification and identity recognition.  Journal of the Acoustical Society of America. 2010 FEB;127(2):874–883.
  * 33. Terry AMR, McGregor PK.  Census and monitoring based on individually identifiable vocalizations: the role of neural networks.  Animal Conservation. 2002;5:103–111.
  * 34. Fox EJS.  A new perspective on acoustic individual recognition in animals with limited call sharing or changing repertoires.  Animal Behaviour. 2008 MAR;75(3):1187–1194.
  * 35. Fox EJS, Roberts JD, Bennamoun M.  Call-independent individual identification in birds.  Bioacoustics. 2008;18(1):51–67.
  * 36. Cheng J, Sun Y, Ji L.  A call-independent and automatic acoustic system for the individual recognition of animals: A novel model using four passerines.  Pattern Recognition. 2010 NOV;43(11):3846–3852.
  * 37. Cheng J, Xie B, Lin C, Ji L.  A comparative study in birds: call-type-independent species and individual recognition using four machine-learning methods and two acoustic features.  Bioacoustics. 2012 JUN;21(2):157–171.
  * 38. Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, et al.  Intriguing properties of neural networks.  arXiv preprint arXiv:13126199. 2013;.
  * 39. Mesaros A, Heittola T, Virtanen T.  Acoustic Scene Classification: an Overview of DCASE 2017 Challenge Entries.  In: 16th International Workshop on Acoustic Signal Enhancement (IWAENC). Tokyo, Japan; 2018. .
  * 40. Khanna H, Gaunt S, McCallum D.  Digital spectrographic cross-correlation: tests of sensitivity.  Bioacoustics. 1997;7(3):209–234.
  * 41. Foote JR, Palazzi E, Mennill DJ.  Songs of the Eastern Phoebe, a suboscine songbird, are individually distinctive but do not vary geographically.  Bioacoustics. 2013;22(2):137–151.
  * 42. Ptáček L, Machlica L, Linhart P, Jaška P, Muller L.  Automatic recognition of bird individuals on an open set using as-is recordings.  Bioacoustics. 2016;25(1):55–73.
  * 43. Stowell D, Stylianou Y, Wood M, Pamuła H, Glotin H.  Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge.  ArXiv e-prints. 2018 Jul;.
  * 44. Lasseck M.  Audio-based Bird Species Identification with Deep Convolutional Neural Networks.  Working Notes of CLEF. 2018;2018.
  * 45. Grava T, Mathevon N, Place E, Balluet P.  Individual acoustic monitoring of the European Eagle Owl Bubo bubo.  Ibis. 2008;150:279–287.
  * 46. Petrusková T, Osiejuk TS, Linhart P, Petrusek A.  Structure and Complexity of Perched and Flight Songs of the Tree Pipit (Anthus trivialis).  Annales Zoologici Fennici. 2008 apr;45(2):135–148.  Available from: https://doi.org/10.5735%2F086.045.0205.
  * 47. Průchová A, Jaška P, Linhart P.  Cues to individual identity in songs of songbirds: testing general song characteristics in Chiffchaffs Phylloscopus collybita.  Journal of Ornithology. 2017 apr;Available from: https://doi.org/10.1007%2Fs10336-017-1455-6.
  * 48. Nieuwenhuyse DV, Génot JC, Johnson DH.  The Little Owl: Conservation, Ecology and Behavior of Athene noctua.  Cambridge University Press; 2008.
  * 49. Linhart P, Šálek M.  The assessment of biases in the acoustic discrimination of individuals.  PloS one. 2017;12(5):e0177206.
  * 50. Krizhevsky A, Sutskever I, Hinton GE.  ImageNet classification with deep convolutional neural networks.  In: Advances in neural information processing systems (NIPS); 2012. p. 1097–1105.  Available from: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.
  * 51. Cireşan D, Meier U, Schmidhuber J.  Multi-column deep neural networks for image classification.  arXiv preprint arXiv:12022745. 2012;.
  * 52. Schlüter J, Grill T.  Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks.  In: Proceedings of the International Conference on Music Information Retrieval (ISMIR); 2015. p. 121–126.
  * 53. Salamon J, Bello JP.  Deep convolutional neural networks and data augmentation for environmental sound classification.  IEEE Signal Processing Letters. 2017;24(3):279–283.
  * 54. Stowell D, Plumbley MD.  Automatic large-scale classification of bird sounds is strongly improved by unsupervised feature learning.  PeerJ. 2014;2:e488.
  * 55. Fawcett T.  An introduction to ROC analysis.  Pattern Recognition Letters. 2006;27(8):861–874.
  * 56. Fournier DA, Skaug HJ, Ancheta J, Ianelli J, Magnusson A, Maunder MN, et al.  AD Model Builder: using automatic differentiation for statistical inference of highly parameterized complex nonlinear models.  Optim Methods Softw. 2012;27:233–249.
  * 57. Skaug H, Fournier D, Bolker B, Magnusson A, Nielsen A. Generalized Linear Mixed Models using ’AD Model Builder’; 2016-01-19.  R package version 0.8.3.3.
  * 58. Beecher MD, Brenowitz EA.  Functional aspects of song learning in songbirds.  Trends in Ecology & Evolution. 2005;20(3):143–149.  Available from: http://www.sciencedirect.com/science/article/pii/S0169534705000054.


# Separable convolutions for optimizing 3D Stereo Networks

###### Abstract

Deep learning based 3D stereo networks give superior performance compared to
2D networks and conventional stereo methods. However, this improvement in the
performance comes at the cost of increased computational complexity, thus
making these networks non-practical for the real-world applications.
Specifically, these networks use 3D convolutions as a major work horse to
refine and regress disparities. In this work first, we show that these 3D
convolutions in stereo networks consume up to $94$% of overall network
operations and act as a major bottleneck. Next, we propose a set of
“plug-&-run” separable convolutions to reduce the number of parameters and
operations. When integrated with the existing state of the art stereo
networks, these convolutions lead up to $7\times$ reduction in number of
operations and up to $3.5\times$ reduction in parameters without compromising
their performance. In fact these convolutions lead to improvement in their
performance in the majority of cases111Code link..

Index Terms—  Stereo Matching, Separable Convolutions, Disparity Estimation,
Computational Efficiency, CNNs

##  1 Introduction

Estimating depth from an input stereo pair is an actively pursued goal in the
computer vision domain. This typically involves two steps: (i) image
rectification based on epipolar constraints and (ii) disparity estimation. A
typical disparity estimation pipeline mainly consists of matching cost
computation and cost aggregation steps [1]. In the matching cost step,
features are computed from both images and matched to generate a potential set
of matches. In the cost aggregation step, global [2, 3] or local neighbourhood
[4, 1] constraints are applied to refine the potential matches and generate
the disparity values.

Fig. 1: Relative comparison between 3D convolutions, FwSCs, FDwSCs _w.r.t_.
number of operations and parameters. Here $\times$ represents the reduction
factor.

With the renaissance of deep learning, many new methods have been proposed to
improve the overall density estimation pipeline. These methods target either a
specific step [5, 6, 7] or work in end-to-end fashion [8, 9, 10] to improve
the overall pipeline. End-to-end deep learning methods, in general, have two
main stages. In the first stage, a backbone network is used to compute
features from both the input images. In the second stage, features from both
the images are merged to form a cost volume; this cost volume is then further
processed via a sequence of convolutional layers to regress the final
disparities. Broadly speaking, these end-to-end methods can be classified to
2D or 3D methods – depending on their methodology of merging features, types
of layers and convolutions in the second stage – _c.f_. Sec. 2. Overall, 2D
networks use 2D convolutions after merging features, whereas 3D methods use 3D
convolutions over 4D cost volumes to regress and refine disparities.

Even though 3D methods give far superior performance compared to 2D methods,
this performance improvement comes at the cost of extra computational budget.
In fact, these 3D methods are orders of magnitude slower than 2D ones. A lot
of efforts in the community have been recently targeted to make these methods
computationally efficient [11, 12, 13]. Although majority of these methods
target optimization of the cost-volume construction step, they still use the
costly 3D convolutions for disparity regression. In this work, we aim to
further reduce the overall computational footprint of the 3D methods by
optimizing 3D convolutions.

For this, we first do a thorough empirical investigation of the state of the
art 3D stereo methods [10, 9] to identify major contributing factors to the
overall computational cost of these methods. As expected, our investigations
conclude that 3D convolutions are by far the most costly operations in these
methods. For instance, in GANetdeep, 3D convolutions consume around 94% of
operations – _c.f_. Table 1. Next, we propose methods for optimizing 3D
convolutions, _i.e_. how we can optimize 3D convolutions without sacrificing
3D stereo networks performance.

To this end, motivated by the recent performance of light-weight 3D networks
for visual recognition [14, 15], we propose to replace costly 3D convolution
operations in 3D stereo networks with their light-weight counterparts (_i.e_.
separable 3D convolutions) to build efficient stereo networks. For this we
design and explore three different versions of separable 3D convolutions: (i)
Feature-wise Separable Convolutions (FwSCs); (ii) Disparity-wise Separable
Convolutions (DwSCs); and their extremely separable version (iii)
Feature-&-Disparity-wise Separable Convolutions (FDwSCs). We empirically
evaluate these separable 3D convolution versions as “plug-&-run” replacement
for existing 3D convolutions.

Overall, we make the following contributions: (i) we present an in-depth cost
analysis of major building blocks of state of the art 3D stereo networks; (ii)
we design and propose separable 3D convolutions for stereo networks; (iii) we
empirically evaluate and show the “plug-&-run” characteristic of proposed
convolutions for state of the art methods on standard datasets. Our results
show that FwSCs and FDwSCs can be reliably used as a replacement for 3D
convolutions in stereo networks without sacrificing their performance while
reducing the number of operations (up to $7.2\times$) and number of parameters
(up to $3.5\times$) – _c.f_. Fig. 1 & Table 3. To the best of our knowledge,
this is the first work to explicitly optimize 3D convolutions involved in 3D
stereo networks.

##  2 Related Work

Generally speaking, deep learning stereo methods can be broadly categorized
into 2D or 3D learning methods [16] based on how they merge information across
views and then process it. For instance, 2D methods merge features from the
left and right images into a 3D cost volume and then apply 2D convolutions.
DispNet-C [17], iResNet [18], MADNet [19], SegStereo [20], EdgeStereo [21] are
some examples of 2D methods. On the other hand, 3D methods first generate a 4D
cost volume by merging features from the left and right images (under stereo
constraints) and use 3D convolutional layers to refine and regress dense
disparities. GC-Net [8], PSMNet [9], GANet [10], AnyNet [11] and DeepPruner
[13] are some examples of 3D methods. Overall 3D methods produce better
results but at the expense of extra computational cost. Although recent
methods propose to optimize the 3D cost volume construction step, they still
use costly 3D convolutions for cost aggregations, refinement and disparity
regression. For instance, [11] use hierarchical 3D cost volumes to reduce 4D
volume construction cost while [12] introduce bottleneck matching modules to
efficiently match features across disparities. In another work [13] employs
patch-matching to sample a range of disparities during volume construction.

##  3 Methodology

Generally, an end-to-end 3D stereo pipeline consists of these modules: (i)
Feature extraction: a 2D shared backbone-network to extract features from left
and right images of a rectified stereo pair; (ii) Volume construction: a 4D
cost volume constructed by merging left and right images features maps; and
(iii) Cost aggregation & disparity regression: a 3D network to aggregate cost
volume and then regress and refine disparities from aggregated cost volume.

Table 1: Detailed computational cost profiling of 3D stereo networks.

Model | Metric | Feature Network |  | Cost Aggregation and
---
Disparity Regression
Total
2D CNNs | 3D CNNs |
GANet11 | # Params (M) | $\numprint{2.37}$ | $\numprint{53.33}\%$ | $\numprint{0.76}$ | $\numprint{17.7}\%$ | $\numprint{4.48}$
# Ops (GMACs) | $\numprint{4.80}$ | $\numprint{2.03}\%$ | $\numprint{217.44}$ | $\numprint{91.92}\%$ | $\numprint{236.56}$
GANetdeep | # Params (M) | $\numprint{2.37}$ | $\numprint{36.31}\%$ | $\numprint{1.80}$ | $\numprint{27.21}\%$ | $\numprint{6.58}$
# Ops (GMACs) | $\numprint{4.80}$ | $\numprint{1.25}\%$ | $\numprint{358.85}$ | $\numprint{93.51}\%$ | $\numprint{383.75}$
PSMNet | # Params (M) | $\numprint{3.34}$ | $\numprint{63.52}\%$ | $\numprint{1.89}$ | $\numprint{36.5}\%$ | $\numprint{5.22}$
# Ops (GMACs) | $\numprint{58.19}$ | $\numprint{31.40}\%$ | $\numprint{127.12}$ | $\numprint{68.6}\%$ | $\numprint{185.31}$
11footnotetext: Note: Here $\%$ represents the fraction of parameters (Params
in Millions) and operations (Ops in GMACs) relative to complete network (image
size for GANet models is $240\times 528$ and PSMNet is $256\times 512$
pixels.). We can see that although 3D CNNs have fewer parameters compared to
2D CNNs, they still consume the majority of operations.

###  3.1 Profiling 3D Stereo Networks

As the first step, we profile the overall computational requirements of the
state of the art 3D stereo networks [10, 9]. Specifically, we record the
number of parameters and operations involved in feature extraction and cost
aggregation & disparity regression modules – _c.f_. Table 1. Here operations
are reported as the number of MACs (Multiply-ACcumulate operations) where
$1\mbox{MAC}=1$ multiplication + $1$ addition operations. We can observe from
Table 1 that despite having fewer parameters than 2D networks, 3D networks
consume more operations in all the networks and thus represent clear
bottlenecks. For instance, in GANet11 [10] 3D CNN contains $0.8\times 10^{6}$
($17\%$ relative to complete network) parameters compared to 2D feature
extraction module’s $2.4\times 10^{6}$ parameters ($52.8\%$). However, it
consumes $217$ GMACs operations ($91.9\%$ of overall operations) compared to
$4.8$ GMACs ($2$ %) of the 2D network. Similar observations can be made about
other networks.

Once we have built a detailed insight into the reasons behind these networks
computational overhead, we set out to network design choices to reduce the
computational overhead without compromising their performance.

###  3.2 Separable 3D Stereo Networks

|
---|---
Fig. 2: Pictorial description of steps involved in the computation of FwSCs
(left) and FDwSCs (right).

Model | #3D | Metric | 3D | FwSCs | FDwSCs
---|---|---|---|---|---
Convs | Convs |  | Reduction |  | Reduction
GANet11 | $11$ | # Params (M) | $\numprint{0.76}$ | $\numprint{0.26}$ | $2.9\times$ | $\numprint{0.25}$ | $3.0\times$
# Ops (GMACs) | $\numprint{217.44}$ | $\numprint{43.05}$ | $5.1\times$ | $\numprint{40.60}$ | $5.4\times$
GANetdeep | $15$ | # Params (M) | $\numprint{1.80}$ | $\numprint{0.53}$ | $3.4\times$ | $\numprint{0.52}$ | $3.5\times$
# Ops (GMACs) | $\numprint{358.85}$ | $\numprint{64.03}$ | $5.6\times$ | $\numprint{60.06}$ | $6.0\times$
PSMNet | $22$ | # Params (M) | $\numprint{1.89}$ | $\numprint{0.58}$ | $3.3\times$ | $\numprint{0.57}$ | $3.3\times$
# Ops (GMACs) | $\numprint{127.12}$ | $\numprint{19.06}$ | $6.7\times$ | $\numprint{17.70}$ | $7.2\times$

Table 2: Comparison of 3D convolutions parameters and operations with
separable 3D Convs.

Recently light-weight convolutional networks (like MobileNet, ShuffleNet,
_etc_. [22, 23]) have been proposed to improve the computational efficiency of
2D visual recognition convolutional networks. These methods propose to use
depth-wise separable variants of 2D convolutions to reduce the number of
operations. However, in comparison to a 2D convolution kernel, a 3D
convolution kernel works on a 4D volume and aggregates information across
spatial, depth and channels dimensions, and thus there are different possible
ways a 3D convolution can be made separable. For instance, a 3D kernel
($k\times k\times k$, _e.g_. $k=5$) working on an input 4D cost volume of size
$h_{i}\times w_{i}\times d_{i}\times c_{i}$ (input height, width, depth
(disparity), and channels) aggregates information from a window of size
$k\times k\times k\times c_{i}$ in each of its applications. Furthermore, as a
3D convolutional layer contains many 3D kernels each producing a new feature
map so overall there are $(h_{i}\times w_{i}\times d_{i}\times(k\times k\times
k\times c_{i}))\times c_{o}$ operations performed in each layer – here $c_{o}$
is number of output channels (number of kernels) in a layer. From this, we can
see that these 3D convolutions can be made separable across either $d_{i}$ or
$c_{i}$ dimension or even simultaneously both across $d_{i}$ and $c_{i}$. We
intend to reduce computational requirement by making 3D convolutions separable
in one or more of the given dimensions.

####  3.2.1 Feature-wise Separable Convolutions (FwSCs):

As a first replacement for 3D convolutions in stereo networks, we propose to
split the convolutions across the feature (or channel) dimension. FwSCs works
in two steps: in the first step, the 4D cost volume of size $h_{i}\times
w_{i}\times d_{i}\times c_{i}$ is split into $c_{i}$ cubes, and $c_{i}$
kernels of size $k\times k\times k$ are applied to generate $c_{i}$ output
cubes – _c.f_. Fig. 2 (left).

In the second step, $c_{o}$ point-wise kernels of size $1\times 1\times
1\times c_{i}$ are applied to aggregate information across the $c_{i}$ cubes.
These steps lead to a significant reduction in the number of operations, as
total the number of operations reduces to $(h_{i}\times w_{i}\times
d_{i}\times k\times k\times k)\times c_{i}+(h_{i}\times w_{i}\times
d_{i}\times 1\times 1\times 1\times c_{i})\times c_{o}$ compared to 3D
convolutions. When applied in the state of the art stereo networks, FwSCs lead
up to $6.7\times$ reduction in GMACs (see Table 2)222Please note that these
calculations also include MACs operations for biases and batch-normalization.
with better performance than 3D convolutions in the majority of cases (Sec.
4).

####  3.2.2 Disparity-wise Separable Convolutions (DwSCs):

DwSCs are identical to FwSCs, except the separable operation is performed in
the disparity dimension. Precisely, in the first step the 4D cost volume of
size $h_{i}\times w_{i}\times d_{i}\times c_{i}$ is split into $d_{i}$ cubes
(after permuting $h_{i}\times w_{i}\times d_{i}\times c_{i}$ to $h_{i}\times
w_{i}\times c_{i}\times d_{i}$) and $d_{i}$ kernels of size $k\times k\times
k$ are applied to produce $d_{i}$ output cubes.

In the second step, $c_{o}$ kernels of size $1\times 1\times 1\times d_{i}$
are applied to aggregate information across the $d_{i}$ cubes. Finally, the
output volume is permuted back. Although DwSCs lead to a reduction in the
number of operations, the overall reduction is not as significant as in FwSCs,
because usually in stereo networks size of disparity dimension is greater than
the size of feature dimension. For instance, in GANetdeep the size of
disparity dimension is 48 whereas in almost all the layers feature dimension
is 32.

Moreover, for a given pixel first aggregating the information across the
feature dimension and then across the disparity dimension does not appear to
be the optimal choice. This was confirmed by our initial experiments, which
returned much worse results thus we do not report further experimental results
on the DwSCs.

####  3.2.3 Feature-Disparity-wise Separable Convolutions (FDwSCs)

FDwSCs are extremely separable variants of 3D convolutions and built by adding
an extra layer of separability on FwSC. These convolutions are built in three
steps (see Fig. 2 (right)). In the first step, similar to FwSCs, the input
cost volume is split into $c_{i}$ cubes, and $c_{i}$ disparity-wise separable
kernels of size $k\times k\times 1$ are applied to each cube to aggregate
spatial information and generate $c_{i}$ output cubes. In the second step,
$c_{i}$ point-wise kernels of size $1\times 1\times k$ are applied to each
cube independently to aggregate disparity information. In the final step
$c_{o}$ kernels of size $1\times 1\times 1\times c_{i}$ are applied to
aggregate information across feature dimension. The total number of operations
involved in FDwSCs is equal to $(h_{i}\times w_{i}\times d_{i}\times(k\times
k\times 1))\times c_{i}+(h_{i}\times w_{i}\times d_{i}\times(1\times 1\times
k))\times c_{i}+(h_{i}\times w_{i}\times d_{i}\times 1\times 1\times 1\times
c_{i})\times c_{o}$.

As a result, FDwSCs lead to a further reduction in the number of parameters
and operations than all other types of convolutions discussed. Precisely, they
lead to a reduction of $3.3\times$ in parameters and $7.2\times$ in GMACs for
PSMNet and $3.5\times$ in parameters and $6.0\times$ in GMACs for GANetdeep –
_c.f_. Table 2. Surprisingly, this reduction in the number of parameters and
operations does not significantly impact the performance – _c.f_. Table 3.

|  | SceneFlow | KITTI 2015
---|---|---|---
Model | Metric | 3D Convs | FwSCs | FDwSCs | 3D Convs | FwSCs | FDwSCs
GANet11 |  $3$ px (%) | $4.21$ | $4.02$ | $3.94$ | $2.01$ | $1.94$ | $2.07$
D1 error (%) | $3.49$ | $3.36$ | $3.25$ | $1.92$ | $1.85$ | $1.95$
EPE | $0.99$ | $0.92$ | $0.94$ | $0.67$ | $0.67$ | $0.68$
GANetdeep |  $3$ px (%) | $4.01$ | $3.74$ | $4.13$ | $1.67$ | $1.67$ | $1.73$
D1 error (%) | $3.29$ | $3.03$ | $3.45$ | $1.61$ | $1.57$ | $1.65$
EPE | $1.01$ | $0.90$ | $0.99$ | $0.63$ | $0.64$ | $0.71$
PSMNet |  $3$ px (%) | $4.20$ | $3.72$ | $3.76$ | $2.10$ | $2.57$ | $2.18$
D1 error (%) | $3.48$ | $3.08$ | $3.10$ | $2.00$ | $2.50$ | $2.08$
EPE | $0.99$ | $0.90$ | $0.90$ | $0.88$ | $0.92$ | $0.90$

Table 3: Quantitative results of networks trained using 3D and separable 3D
convolutions based stereo networks on benchmark datasets.

Fig. 3: Qualitative Results of GANet11 model on sample images from KITTI 2015
(left) and SceneFlow (right) datasets. Here the second and fourth columns
represent the error maps _w.r.t_. ground truth. Darker red and blue colors
represent higher and lower disparity errors, respectively.

##  4 Experimental Results

We trained and evaluated our networks on two popular stereo benchmark
datasets.
SceneFlow [17] is a large scale synthetic dataset and contains $35,454$
training images and $4,370$ test images of size $540\times 960$ with dense
annotations. For our experiments, we further divide the training dataset to
$32,454$ training and $3000$ validation pairs following [10] .
KITTI 2015 [24] is a real dataset of driving scenes. It only contains $200$
training and $200$ test image pairs of size $376\times 1240$ pixels with
sparse annotations. For our experiments, we further divide the training
dataset into $150$ training and $50$ validation pairs as in [10].

###  4.1 Training Details

For our baseline networks, we use the publicly released code of GANetdeep and
PSMNet. For fair comparison and to show the “plug-&-run” capabilities of
separable 3D convolutions we use identical training settings for both baseline
networks (_i.e_. networks with 3D convolutions) and optimized networks (_i.e_.
networks with separable 3D convolutions). Furthermore, it is important to
mention that we do not perform any type of hyperparameters tuning and train
all the networks from scratch (given in Table 3) with default values as in
base networks.
GANet: All the input images are cropped to the size of $240\times 528$ while
the maximum disparity is set to $192$. Furthermore, all the images are
standard normalized using the mean and standard deviation of the training set.
For training, we use the Adam Optimizer ($\lambda=1\text{e-}3$,
$\beta_{1}=0.9$ and $\beta_{2}=0.999$) with a training batch size of $4$. For
KITTI 2015, we transfer learned the models trained on SceneFlow for $650$
epochs with the learning rate reduced to $1\text{e-}4$ after $300$ epochs.
PSMNet: The majority of settings are identical to the baseline except that the
images are randomly cropped to $256\times 512$ during training, and we use a
batch size of $8$. Moreover, for transfer learning on KITTI 2015 the learning
rate is reduced after $200$ epochs.

###  4.2 Results

For quantitative comparison, following the SceneFlow and KITTI 2015 evaluation
protocols, we report three evaluation metrics including $3$-pixels, D1 and
End-Point-Errors (EPE) [17, 24] for all the networks . On the SceneFlow
dataset (Table 3 (left)) we notice that for most of the cases, networks with
FwSCs outperform baseline networks with 3D convolutions despite having around
$6.7\times$ less operations and $3.3\times$ fewer parameters. The better
performance of FwSCs, compared to 3D convolutions, can be attributed to the
fact that a cost volume contains a lot of redundant information (the majority
of an image neighbouring disparities do not change) and these separable 3D
convolutions contain enough representation power to model disparity
variations. Surprisingly, even extremely separable FDwSCs perform better than
regular 3D convolutions in most of the cases. However, their performance is a
little inferior to FwSC, which is somewhat expected, due to the loss of their
correlation modeling capabilities in disparity dimension.

By looking at the quantitative results from the KITTI 2015 dataset from Table
3 (right) we can draw similar conclusions as on the SceneFlow dataset. FwSCs
still outperform 3D convolutional networks with a relatively smaller number of
3D convolutional layers, _e.g_. GANetdeep has $15$ convolutional layers.
However, for the networks with a far higher number of 3D convolutional layers,
_e.g_. PSMNet ($22$ convolutional layers) we see a slight deterioration in
their performance – this might be because KITTI 2015 is a small and sparse
dataset. FDwSCs also give comparable performance to FwSCs and 3D convolutions.

Fig. 3 shows qualitative results on sample examples from the KITTI 2015
validation and SceneFlow test set. Here once again we can verify that
separable 3D convolutions are giving comparable results to their 3D
convolutions counterparts.

##  5 Conclusions

In this work, we have shown that 3D convolutional layers are the major
bottleneck in overall execution of 3D stereo networks. To circumvent this we
have proposed and empirically shown that separable 3D convolutions can
considerably reduce the number of operations and parameters for stereo
matching in state of the art networks. These proposed convolutions not only
lead to leaner networks but they also consistently lead to better performance
of these networks. We believe that these convolutions “plug-&-run” nature can
lead to their integration into many other 3D stereo networks.

## References

  * [1] Daniel Scharstein and Richard Szeliski,  “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,”  International journal of computer vision, vol. 47, no. 1-3, pp. 7–42, 2002.
  * [2] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz,  “Fast cost-volume filtering for visual correspondence and beyond,”  IEEE TPAMI, vol. 35, no. 2, pp. 504–511, 2012.
  * [3] Kuk-Jin Yoon and In So Kweon,  “Adaptive support-weight approach for correspondence search,”  IEEE TPAMI, vol. 28, no. 4, pp. 650–656, 2006.
  * [4] Dongbo Min, Jiangbo Lu, and Minh N Do,  “A revisit to cost aggregation in stereo matching: How far can we reduce its computational redundancy?,”  in 2011 International Conference on Computer Vision. IEEE, 2011, pp. 1567–1574.
  * [5] Jure Žbontar and Yann LeCun,  “Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches,”  arXiv:1510.05970 [cs], May 2016,  arXiv: 1510.05970.
  * [6] Akihito Seki and Marc Pollefeys,  “SGM-Nets: Semi-Global Matching with Neural Networks,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, July 2017, pp. 6640–6649, IEEE.
  * [7] Wenjie Luo, Alexander G Schwing, and Raquel Urtasun,  “Efficient deep learning for stereo matching,”  in Proceedings of the IEEE CVPR, 2016, pp. 5695–5703.
  * [8] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry,  “End-to-end learning of geometry and context for deep stereo regression,” 2017.
  * [9] Jia-Ren Chang and Yong-Sheng Chen,  “Pyramid Stereo Matching Network,”  in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, June 2018, pp. 5410–5418, IEEE.
  * [10] Feihu Zhang, Victor Prisacariu, Ruigang Yang, and Philip H. S. Torr,  “GA-Net: Guided Aggregation Net for End-to-end Stereo Matching,”  arXiv:1904.06587 [cs], Apr. 2019,  arXiv: 1904.06587.
  * [11] Yan Wang, Zihang Lai, Gao Huang, Brian H. Wang, Laurens van der Maaten, Mark Campbell, and Kilian Q. Weinberger,  “Anytime Stereo Image Depth Estimation on Mobile Devices,”  arXiv:1810.11408 [cs], Mar. 2019,  arXiv: 1810.11408.
  * [12] Stepan Tulyakov, Anton Ivanov, and Francois Fleuret,  “Practical deep stereo (pds): Toward applications-friendly deep stereo matching,” 2018.
  * [13] Shivam Duggal, Shenlong Wang, Wei-Chiu Ma, Rui Hu, and Raquel Urtasun,  “Deeppruner: Learning Efficient Stereo Matching Via Differentiable Patchmatch,”  arXiv:1909.05845 [cs], Sept. 2019,  arXiv: 1909.05845.
  * [14] Zhaofan Qiu, Ting Yao, and Tao Mei,  “Learning spatio-temporal representation with pseudo-3d residual networks,” 2017.
  * [15] Rongtian Ye, Fangyu Liu, and Liqiang Zhang,  “3D Depthwise Convolution: Reducing Model Parameters in 3D Vision Tasks,”  arXiv:1808.01556 [cs], Aug. 2018,  arXiv: 1808.01556.
  * [16] Matteo Poggi, Fabio Tosi, Konstantinos Batsos, Philippos Mordohai, and Stefano Mattoccia,  “On the Synergies between Machine Learning and Stereo: a Survey,”  arXiv:2004.08566 [cs], Apr. 2020,  arXiv: 2004.08566.
  * [17] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox,  “A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 2016, pp. 4040–4048, IEEE.
  * [18] Zhengfa Liang, Yiliu Feng, Yulan Guo, Hengzhu Liu, Wei Chen, Linbo Qiao, Li Zhou, and Jianfeng Zhang,  “Learning for Disparity Estimation through Feature Constancy,”  arXiv:1712.01039 [cs], Mar. 2018,  arXiv: 1712.01039.
  * [19] Alessio Tonioni, Fabio Tosi, Matteo Poggi, Stefano Mattoccia, and Luigi Di Stefano,  “Real-time self-adaptive deep stereo,”  arXiv:1810.05424 [cs], Oct. 2018,  arXiv: 1810.05424.
  * [20] Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, and Jiaya Jia,  “SegStereo: Exploiting Semantic Information for Disparity Estimation,”  arXiv:1807.11699 [cs], July 2018,  arXiv: 1807.11699.
  * [21] Xiao Song, Xu Zhao, Hanwen Hu, and Liangji Fang,  “EdgeStereo: A Context Integrated Residual Pyramid Network for Stereo Matching,”  arXiv:1803.05196 [cs], Sept. 2018,  arXiv: 1803.05196.
  * [22] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen,  “Mobilenetv2: Inverted residuals and linear bottlenecks,”  in Proceedings of the IEEE CVPR, 2018, pp. 4510–4520.
  * [23] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun,  “Shufflenet: An extremely efficient convolutional neural network for mobile devices,”  in Proceedings of the IEEE CVPR, 2018, pp. 6848–6856.
  * [24] Moritz Menze and Andreas Geiger,  “Object scene flow for autonomous vehicles,”  in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, June 2015, pp. 3061–3070, IEEE.