# A Survey of Sound Source Localization with Deep Learning Methods

Pierre-Amaury Grumiaux
Nantes Université, École Centrale Nantes, CNRS, LS2N
2 chemin de la Houssinière
F-44332 Nantes, France
<EMAIL_ADDRESS>& Srđan Kitić
Orange Labs
4 Rue du Clos Courtel
35510 Cesson-Sévigné, France
<EMAIL_ADDRESS>& Laurent Girin
Univ. Grenoble Alpes, Grenoble-INP, GIPSA-lab
11 Rue des Mathématiques
38400 Saint-Martin-d’Hères, France
<EMAIL_ADDRESS>& Alexandre Guérin
Orange Labs
4 Rue du Clos Courtel
35510 Cesson-Sévigné, France
<EMAIL_ADDRESS>During the writing of this paper, Pierre-Amaury
Grumiaux was also at Orange Labs, 4 Rue du Clos Courtel, F-35510 Cesson-
Sévigné, France, and at Univ. Grenoble Alpes, Grenoble-INP, CNRS, GIPSA-lab,
11 Rue des Mathématiques, F-38400 Saint-Martin-d’Hères, France.

###### Abstract

This article is a survey of deep learning methods for single and multiple
sound source localization, with a focus on sound source localization in indoor
environments, where reverberation and diffuse noise are present. We provide an
extensive topography of the neural network-based sound source localization
literature in this context, organized according to the neural network
architecture, the type of input features, the output strategy (classification
or regression), the types of data used for model training and evaluation, and
the model training strategy. Tables summarizing the literature survey are
provided at the end of the paper, allowing a quick search of methods with a
given set of target characteristics.

##  1 Introduction

Sound source localization (SSL) is the problem of estimating the position of
one or several sound sources relative to some arbitrary reference position,
which is generally the position of the recording microphone array, based on
the recorded multichannel acoustic signals. In most practical cases, SSL is
simplified to the estimation of the sources’ direction of arrival (DoA),
_i.e._ , it focuses on the estimation of azimuth and elevation angles, without
estimating the distance to the microphone array (therefore, unless otherwise
specified, in this article we use the terms “SSL” and “DoA estimation”
interchangeably). SSL has numerous practical applications – for instance, in
source separation, e.g., (Chazan et al., 2019), automatic speech recognition
(ASR), e.g., (Lee et al., 2016), speech enhancement, e.g., (Xenaki et al.,
2018), human-robot interaction, e.g., (Li et al., 2016b), noise control, e.g.,
(Chiariotti et al., 2019), and room acoustic analysis, e.g., (Amengual Garí et
al., 2017). As detailed in the following, in this paper, we focus on sound
sources in the audible range (typically speech and audio signals) in indoor
(office or domestic) environments.

Although SSL is a longstanding and widely researched topic (Gerzon, 1992;
DiBiase et al., 2001; Argentieri et al., 2015; Cobos et al., 2017; Benesty et
al., 2008; Knapp and Carter, 1976; Brandstein and Ward, 2001; Nehorai and
Paldi, 1994; Hickling et al., 1993), it remains a very challenging problem to
date. Traditional SSL methods are based on signal/channel models and signal
processing (SP) techniques. Although they have shown notable advances in the
domain over the years, they are known to perform poorly in difficult yet
common scenarios where noise, reverberation, and several simultaneously
emitting sound sources may be present (Blandin et al., 2012; Evers et al.,
2020). In the last decade, the potential of data-driven deep learning (DL)
techniques for addressing such difficult scenarios has received an increasing
interest. As a result, an increasing number of SSL systems based on deep
neural networks (DNNs) have been proposed in the recent years. Most of the
reported works have indicated the superiority of DNN-based SSL methods over
conventional (i.e., SP-based) SSL methods. For example, Chakrabarty and Habets
(2017a) showed that, in low signal-to-noise ratio conditions, using a CNN led
to a two-fold increase in overall DoA classification accuracy compared to
using the conventional method called steered response power with phase
transform (SRP-PHAT) (see Section 3). In Perotin et al. (2018b), the authors
were able to obtain a $25\%$ increase of DoA classification accuracy when
using a convolutional recurrent neural network (CRNN) over a method based on
independent component analysis (ICA). Finally Adavanne et al. (2018) proved
that employing a CRNN can reduce the average angular error by $50\%$ in
reverberant conditions compared to the conventional MUSIC algorithm (see
Section 3).

This kind of results has further motivated the expansion of scientific papers
on DL applied to SSL. In the meantime, there has been no comprehensive survey
of the existing approaches, which would be very useful for researchers and
practitioners in the domain. Although we can find reviews mostly focused on
conventional methods, e.g., (Argentieri et al., 2015; Cobos et al., 2017;
Evers et al., 2020; Gannot et al., 2019), to the best of our knowledge only a
very few have explicitly targeted SSL with DL methods. Ahmad et al. (2021)
presented a short survey of several existing DL models and datasets for SSL
before proposing a DL architecture of their own. Bianco et al. (2019) and
Purwins et al. (2019) presented an interesting overview of machine learning
applied to various problems in audio and acoustics. Nevertheless, only a short
portion of each of these two reviews is dedicated to SSL with DNNs.

###  1.1 Aim of the paper

The goal of this paper is to fill this gap, and to provide a thorough survey
of the SSL literature using DL techniques. More precisely, we examined and
review 156 papers published from 2011 to 2021. We classify and discuss the
different approaches in terms of characteristics of the employed methods and
addressed configurations (e.g., single-source vs. multi-source localization
setup or neural network architecture; the exact list is given in Section 1.3).
In other words, we present a taxonomy of the DL-based SSL literature published
in the last decade. At the end of the paper, we present a summary of this
survey in the form of four tables (one for the period 2011–2018, and one for
each of the years 2019, 2020 and 2021). All of the methods that we reviewed
are reported in these tables with a summary of their characteristics presented
in different columns. This enables the reader to rapidly select the subset of
methods having a given set of characteristics, if they are interested in that
particular type of method.

Note that in this survey paper, we do not aim to evaluate and compare the
performance of the different systems. Due to the large number of DNN-based SSL
papers and the diversity of configurations, such a contribution would be very
difficult and cumbersome (albeit very useful), especially because the
discussed systems are often trained and evaluated on different datasets. As we
will see later, listing and commenting on these different datasets is,
however, part of our survey effort. Note also that we do not consider SSL
systems that exploit other modalities in addition to sound, e.g., audio-visual
systems (Ban et al., 2018; Wu et al., 2021a; Masuyama et al., 2020). Finally,
we do consider DL-based methods for joint sound event localization and
detection (SELD), which is a combination of sound event detection (SED; here
detection actually means classification) and SSL, and in that case, we focus
on the localization task. In particular, we include in the review the SELD
methods presented to the DCASE Challenge (and/or to the corresponding DCASE
Workshop) in 2019, 2020 and 2021 (see the DCASE website at
https://dcase.community/). One of the task of this challenge is precisely
dedicated to SELD, which has contributed to make the DL-based SSL (and SED)
problem a popular research topic over the recent years.

###  1.2 General principle of DL-based SSL

Figure 1: General pipeline of a DL-based SSL system.

The general principle of DL-based SSL methods and systems can be schematized
with a simple pipeline, as illustrated in Fig. 1. A multichannel input signal
recorded with a microphone array is processed by a feature extraction module
to provide input features. These input features are fed into a DNN, which
delivers an estimate of the source location or DoA. As discussed later in the
paper, a recent trend is to skip the feature extraction module to directly
feed the network with multichannel raw data. In any case, the two fundamental
reasons behind the design of such SSL are detailed below.

First, multichannel signals recorded with an array of $I$ microphones
distributed in space contain information about the location of the source(s).
Indeed, when the microphones are close to each other compared to their
distance to the source(s), the microphone signal waveforms, although appearing
similar from a distance, exhibit more or less notable and complex differences
in terms of delay and amplitude, depending on the experimental setup. These
interchannel differences are due to distinct propagation paths from the source
to the different microphones, for both the direct path (line of sight between
source and microphone) and the numerous reflections that compose the
reverberation in an indoor environment. In other words, a source signal
$s_{j}(t)$ is convolved with different room impulse responses (RIRs)
$a_{i,j}(t)$, which depend on the source position, microphone position and
directivity ($i$ denotes the microphone index in the array), and acoustic
environment configuration (e.g., room shape):

$\displaystyle x_{i}(t)=a_{i,j}(t)\star
s_{j}(t)+n_{i}(t)=\sum_{\tau=0}^{T-1}a_{i,j}(\tau)s_{j}(t-\tau)+n_{i}(t),$ (1)

where $x_{i}(n)$ denotes the resulting recorded signal at microphone $i$,
$n_{i}(t)$ is the noise signal at microphone $i$ (diffuse, “background” noise
and possibly some sensor noise), and $\star$ denotes the convolution (note
that we work with digital signals and $t$ and $\tau$ are discrete time
indexes; $T$ is the effective length of the RIR). Therefore, the recorded
signal contains information on the relative source-to-microphone array
position. The microphone signals are often expressed in the time-frequency
(TF) domain, using the short-term Fourier transform (STFT), where the
convolution in Eq. (1) is assumed to transform into a product between the STFT
of the source signal $S_{j}(f,n)$ and the acoustic transfer function (ATF)
$A_{i,j}(f)$, which is the (discrete) Fourier transform of the corresponding
RIR and is thus encoding the source spatial information ($f$ denotes the
frequency bin, and $n$ is the STFT frame index) (Gannot et al., 2017; Vincent
et al., 2018):

$X_{i}(f,n)=A_{i,j}(f)S_{j}(f,n)+N_{i}(f,n).$ (2)

When several, say $J$, sources are present, the recorded signal is the sum of
their contribution (plus the noise):

$x_{i}(t)=\sum_{j=1}^{J}a_{i,j}(t)\star s_{j}(t)+n_{i}(t).$ (3)

This latter equation is often reformulated in the TF domain in matrix form:

$\mathbf{X}(f,n)=\mathbf{A}(f)\mathbf{S}(f,n)+\mathbf{N}(f,n),$ (4)

where $\mathbf{X}(f,n)=[X_{1}(f,n),...,X_{I}(f,n)]^{\top}$ is the microphone
signal vector, $\mathbf{A}(f)$ is the matrix gathering the ATFs,
$\mathbf{S}(f,n)=[S_{1}(f,n),...,S_{J}(f,n)]^{\top}$ is the source signal
vector, and $\mathbf{N}(f,n)=[N_{1}(f,n),...,N_{I}(f,n)]^{\top}$ is the noise
vector. In that multi-source case, the difficulty of the SSL problem is that
the contributions of the different sources generally overlap in time. SSL then
requires to proceed to some kind of source clustering, which is generally
easier to proceed in the frequency or time-frequency domain due to the natural
sparsity of audio sources in that domain (Rickard, 2002). In this paper, we do
not describe the foundations of source-to-microphone propagation in more
detail. They can be found in several references on general acoustics, e.g.,
(Jacobsen and Juhl, 2013; Rossing, 2007), room acoustics, e.g., (Kuttruff,
2016), array signal processing, e.g., (Brandstein and Ward, 2001; Benesty et
al., 2008; Jarrett et al., 2017; Rafaely, 2019), speech enhancement and audio
source separation, e.g., (Gannot et al., 2017; Vincent et al., 2018), and many
papers on conventional SSL.

The second reason for designing DNN-based SSL systems is that even if the
relationship between the information contained in the multichannel signal and
the location of the source(s) is generally complex (especially in a
multisource reverberant and noisy configuration, see Eqs. (3) and (4)), DNNs
are powerful models that are able to automatically identify and exploit this
relationship, given that they are provided with a sufficiently large number of
representative training examples. This ability of data-driven DL methods to
replace conventional methods based on a signal/channel model and SP techniques
—or at least a part of them, since the feature extractor module can be based
on conventional processing— makes them attractive for addressing problems such
as SSL. While some conventional methods can adapt to the observed signals,
e.g., Dvorkind and Gannot (2005); Li et al. (2016c); Laufer-Goldshtein et al.
(2020); Li et al. (2016b), they are all intrinsically based on certain (more
or less plausible) modeling assumptions, which can limit their effectiveness
when exposed to the complexity of real-world acoustics. Deep learning models
do not _explicitly_ impose any such assumptions, and instead they efficiently
adapt to the presented training data. This is, however, also the major
drawback of the DNN-based approaches, as they are less generic than
traditional methods. A deep model designed for and trained in a given
configuration (e.g., a given microphone array geometry) will not provide
satisfying localization results if the setup changes (Liu et al., 2018; Le
Moing et al., 2021), unless some relevant adaptation method can be used, which
is still an open problem in DL in general.

###  1.3 Outline of the paper

The remainder of the paper is organized as follows. In Section 2, we specify
the context and scope of the survey in terms of the considered acoustic
environment and sound source configurations. In Section 3, we briefly present
the most common conventional SSL methods, for two reasons: first, they are
often used as a baseline for the evaluation of DL-based methods; and second,
we will see that several types of features extracted by conventional methods
can be used in DL-based methods. Section 4 aims to classify the different
neural network architectures used for SSL. Section 5 presents the various
types of input features used for SSL with neural networks. In Section 6, we
explain the two output strategies employed in DL-based SSL: classification and
regression. We then discuss in Section 7 the datasets used for training and
evaluating the models. In Section 8, learning paradigms such as supervised or
semi-supervised learning are discussed from the SSL perspective. Section 9
provides the two summary tables and concludes the paper. Note that, due to the
large number of acronyms used in this survey paper, we provide a list of these
acronyms in Table 1.

Table 1: Table of acronyms.

ACCDOA | activity-coupled Cartesian direction of arrival
---|---
AE | autoencoder
ATF | acoustic transfer function
ASR | automatic speech recognition
BGRU | bidirectional gated recurrent unit
BIR | binaural impulse response
BRIR | binaural room impulse response
CC | cross-correlation
CNN | convolutional neural network
CRNN | convolutional recurrent neural network
CPS | cross power spectrum
DCASE | Detection and Classification
| of Acoustic Scenes and Events
DIRHA | distant-speech interaction
| for robust home applications
DL | deep learning
DNN | deep neural network
DoA | direction of arrival
DP-RTF | direct-path relative transfer function
DRR | direct-to-reverberant ratio
EM | expectation maximization
ESPRIT | Estimation of Signal Parameters
| via Rotational Invariance Techniques
EVD | eigenvalue decomposition
FFNN | feed-forward neural network
FOA | first-order Ambisonics
GAN | generative adversarial network
GCC | generalized cross-correlation
GLU | gated linear unit
GMM | Gaussian mixture models
GMR | Gaussian mixture regression
GPU | graphical processing unit
GRU | gated recurrent unit
HATS | head-and-torso simulator
HOA | higher-order Ambisonics

HRTF | head-related transfer function
---|---
ICA | independent component analysis
ILD | interaural level difference
IPD | interaural phase difference
ITD | interaural time difference
ISM | image source method
LSTM | long short-term memory
MHSA | multihead self-attention
MLP | multiLayer Perceptron
MOT | multi-object tracking
MUSIC | MUltiple SIgnal Classification
NLP | natural language processing
NoS | number of sources
PHAT | PHAse Transform
RIR | room impulse response
RNN | recurrent neural network
RTF | relative transfer function
SA | self-attention
SCM | spatial covariance matrix
SED | sound event detection
SELD | sound event localization and detection
SH | spherical harmonics
SMIR | spherical microphone impulse response
SMN | sequence matching network
SP | signal processing
SPS | spatial pseudo-spectrum
SRP | steered power response
SSL | sound source localization
STFT | short-term Fourier transform
TCN | temporal convolutional network
TDoA | time difference of arrival
TF | time-frequency
VAD | voice activity detection
VAE | variational autoencoder
WDO | W-disjoint orthogonality

##  2 Acoustic environment and sound source configurations

SSL has been applied in different configurations, depending on the
application. In this section, we specify the scope of our survey in terms of
acoustic environment (noisy, reverberant, or even multi-room) and the nature
of the considered sound sources (their type, number, and static/mobile
status).

###  2.1 Acoustic environments

In this paper, we focus on SSL in an indoor environment, _i.e._ , when the
microphone array and the sound source(s) are present in a closed room,
generally of moderate size, typically an office room or a domestic
environment. This implies reverberation: in addition to the direct source-to-
microphone propagation path, the recorded sound contains many other multi-path
components of the same source. All of these components form the room impulse
response (RIR), which is defined for each source position and microphone array
position (including orientation) and for a given room configuration.

In a general manner, the presence of reverberation is seen as a notable
perturbation that makes SSL more difficult compared to the simpler (but
somewhat unrealistic) anechoic case, which assumes the absence of
reverberation, as is obtained in the free field propagation setup. Another
important adverse factor to take into account in SSL is noise. On the one
hand, noise can come from interfering sound sources in the surrounding
environment: TV, background music, pets, street noise passing through open or
closed windows, etc. Often, noise is considered as diffuse, _i.e._ , it does
not originate from a clear direction. On the other hand, the imperfections of
the recording devices are another source of noise that are generally
considered as artifacts.

Early works on using neural networks for DoA estimation most often considered
direct-path propagation only (the anechoic setting), e.g., (Rastogi et al.,
1987; Goryn and Kaveh, 1988; Jha et al., 1988; Jha and Durrani, 1989, 1991;
Falong et al., 1993; Yang et al., 1994; Southall et al., 1995; El Zooghby et
al., 2000), though a model of the acoustical environment was used to generate
simulated data to train the neural network of Datum et al. (1996). Most of
these works are from the pre-deep-learning era, using “shallow” neural
networks with only one or two hidden layers Goodfellow et al. (2016). We do
not detail these works in our survey, although we acknowledge them as
pioneering contributions to the neural network-based DoA estimation problem. A
few more recent works based on more “modern” neural network architectures also
focused on anechoic propagation only, or did not consider sound sources in the
audible bandwidth (Liu et al., 2018; Ünlerşen and Yaldiz, 2016; Bialer et al.,
2019; Elbir, 2020; Choi and Chang, 2020).

###  2.2 Source types

In the SSL literature, a great proportion of systems focuses on localizing
speech sources because of their importance in related tasks such as speech
enhancement or speech recognition. Examples of speaker localization systems
can be found in papers by Chakrabarty and Habets (2019b); Grumiaux et al.
(2021b); He et al. (2021a); Hao et al. (2020). In such systems, the neural
networks are trained to estimate the DoA of speech sources so that they are
somewhat specialized in this type of source. Other systems, in particular
those participating in the DCASE Challenge, consider a variety of sound source
types (Politis et al., 2020b). Depending on the challenge task and its
corresponding dataset, these methods are capable of localizing alarms, crying
babies, crashes, barking dogs, female/male screams, female/male speech,
footsteps, knockings on doors, ringings, phones, and piano sounds. Note that
the localization of such sources, even if they overlap in time, is not
necessarily a more difficult problem than localization of several overlapping
speakers, since the former usually have distinct spectral characteristics that
neural models may exploit for better detection and localization.

###  2.3 Number of sources

The number of sources (NoS) in a recorded mixture signal is an important
parameter for SSL. In the SSL literature, the NoS might be considered as known
(as a working hypothesis). Alternatively, it can be estimated along with the
source location, in which case the SSL problem is a combination of detection
and localization. Examples of conventional (non-deep) SSL works including NoS
estimation can be found in papers by Arberet et al. (2009); Landschoot and
Xiang (2019).

Many DNN-based works have considered only one source to localize, as it is the
simplest scenario to address, e.g., (Perotin et al., 2018b; Bologni et al.,
2021; Liu et al., 2021). We refer to this scenario as single-source SSL. In
this case, the networks are trained and evaluated on datasets with only at
most one active source (a source is said to be active when emitting sound and
inactive otherwise). In terms of NoS, we thus have here either 1 or 0 active
source. The activity of the source in the processed signal, which generally
contains background noise, can be artificially controlled, _i.e._ , the
knowledge of source activity is a working hypothesis. This is a reasonable
approach at training time when using synthetic data, but it is quite
unrealistic at test time on real-world data. Alternatively, the source
activity can be estimated, which is a more realistic approach at test time. In
the latter case, there are two ways of dealing with the source activity
detection problem. The first is to employ a source detection algorithm
beforehand and then apply the SSL method only on the signal portions with an
active source. For example, a voice activity detection (VAD) technique has
been used in the SSL systems of Kim and Hahn (2018); Chang et al. (2018);
Sehgal and Kehtarnavaz (2018); Li et al. (2016d). The other way is to detect
the activity of the source at the same time as the localization algorithm. For
example, an additional neuron was added by Yalta et al. (2017) to the output
layer of their DNN, which outputted $1$ when no source was active (in that
case, all other localization neurons were trained to output $0$), and $0$
otherwise.

Multi-source localization is a much more difficult problem than single-source
SSL. Current state-of-the-art DL-based methods address multi-source SSL in
adverse environments. In this survey, we consider as multi-source localization
the scenario in which several sources overlap in time (_i.e._ , they are
simultaneously emitting), regardless of their type (e.g., there could be
several speakers or several distinct sound events). The specific case of a
multi-speaker conversation with or without speech overlap is strongly
connected to the speaker diarization problem (“who speaks when?”) (Tranter and
Reynolds, 2006; Anguera et al., 2012; Park et al., 2021b). Speaker
localization, diarization, and (speech) source separation are intrinsically
connected problems, as the information retrieved from solving each of them can
be useful for addressing the others (Vincent et al., 2018; Kounades-Bastian et
al., 2017; Jenrungrot et al., 2020). An investigation of these connections is
beyond the scope of this survey.

In the multi-source scenario, the source detection problem transposes to a
source counting problem, but the same considerations as in the single-source
scenario hold. In some works, the knowledge of the NoS is a working
hypothesis, e.g., (Grumiaux et al., 2021b; Ma et al., 2015; He et al., 2019a;
Perotin et al., 2019b; Fahim et al., 2020; Bohlender et al., 2021; Grumiaux et
al., 2021a) and the sources’ DoA can be directly estimated. If the NoS is
unknown, one can apply a source counting system beforehand, e.g., with a
dedicated DNN (Grumiaux et al., 2020). For example, Tian (2020) trained a
separate neural network to estimate the NoS in the recorded mixture signal,
after which he used this information along with the output of the DoA
estimation neural network. Alternatively, the NoS can be estimated alongside
the DoAs, as in the single-source scenario, based on the SSL network output.
When using a classification paradigm, the network output generally predicts
the probability of the presence of a source within each discretized region of
the space (see Section 8). One can thus set a threshold on this estimated
probability, which implicitly provides source counting.111Note that this
problem is common to DL-based multi-source SSL methods and conventional
methods for which a source activity profile is estimated and peak-picking
algorithms are typically used to select the active sources. Otherwise, the
ground-truth or estimated NoS is typically used to select the corresponding
number of classes having the highest probability.

Finally, several DNN-based systems were purposefully designed to estimate the
NoS alongside the DoAs. For example, the method proposed by Nguyen et al.
(2020a) uses a neural architecture with two output branches: the first branch
is used to estimate the NoS (up to four sources; the problem is formulated as
a classification task), while the second branch is used to classify the
azimuth into several regions. In the same spirit, we can mention the numerous
systems presented at the DCASE Challenge, in which the SED task, jointly
conducted with SSL, intrinsically provides an estimate of the NoS. Note that
many DCASE Challenge candidate systems will be reviewed in the core of this
survey.

###  2.4 Moving sources

Source tracking is the problem of estimating the evolution of the sources’
position(s) over time, especially when the sources are mobile. In this survey
paper, we do not address the problem of tracking on its own, which is usually
done in a separate algorithm using the sequence of DoA estimates obtained by
applying SSL on successive time windows (Vo et al., 2015). Still, several DL-
based SSL systems have been shown to produce more accurate localization of
moving sources when they were trained on a dataset that includes this type of
source (Adavanne et al., 2019c; Diaz-Guerra et al., 2021a; Guirguis et al.,
2020; He et al., 2021b). In other cases, as the number of real-world datasets
with moving sources is limited and the simulation of signals with moving
sources is cumbersome, a number of systems trained on static sources have been
shown to retain fair to good performance for moving sources, e.g., (Opochinsky
et al., 2021; Grumiaux et al., 2021a; Sundar et al., 2020).

##  3 Conventional SSL methods

Before the advent of DL, a set of signal processing techniques were developed
to address SSL. A detailed review of these techniques was made by DiBiase et
al. (2001). A review in the specific robotics context was made by Argentieri
et al. (2015). In this section, we briefly present the most common
conventional SSL methods. As briefly stated in the introduction, the reason
for this is twofold: first, conventional SSL methods are often used as
baselines for DL-based methods; and second, many DL-based SSL methods use
input features extracted with conventional methods (see Section 5).

When the geometry of the microphone array is known, DoA estimation can be
performed by estimating the time-difference of arrival (TDoA) of the sources
between the microphones (Xu et al., 2013). The generalized cross-correlation
with phase transform (GCC-PHAT) is one of the most employed method when
dealing with a 2-microphone array (Knapp and Carter, 1976). It is computed as
the inverse Fourier transform of a weighted version of the cross-power
spectrum (CPS) between the signals of the two microphones:

$r_{1,2}(\tau)=\sum_{f=0}^{F-1}\frac{X_{1}(f)X_{2}(f)^{*}}{|X_{1}(f)X_{2}(f)^{*}|}e^{j2\pi\frac{f\tau}{N}},$
(5)

where $X_{i}(f)$ are the $N$-point Fourier transform of the microphone signals
$x_{i}(t)$, and $X_{1}(f)X_{2}(f)^{*}$ is the CPS (∗ denotes the complex
conjugate). The TDoA estimate is then obtained by finding the time delay
between the microphone signals that maximizes the GCC-PHAT function:

$\hat{\tau}=\arg\max_{\tau}r_{1,2}(\tau).$ (6)

The GCC approach has been extended to arrays with more than two microphones,
showing in particular that the localization could be improved by taking
advantage of the multiple microphone pairs (DiBiase et al., 2001; Benesty et
al., 2008).

Building an acoustic power map $P(\mathbf{x})$, with $\mathbf{x}$ the spatial
coordinates, usually a regular grid, is another way to retrieve the DoA of one
or multiple sources, as local maxima of this map mainly correspond to the
sources’ DoA. The Steered-Response Power (SRP) map has been extensively used:
it consists in pointing delay and sum beamformers towards each of the
candidate grid positions and measuring the energy that arises from these
directions. Its PHAT version, which reveals more robust to reverberation, is
certainly the most popular. Practically, it can be derived from the average of
the GCC-PHAT computed on all microphone pairs (DiBiase et al., 2001):

$P(\mathbf{x})=\sum_{m_{1}=1}^{M}\sum_{m_{2}=m_{1}+1}^{M}r_{1,2}(\tau_{m_{1},m_{2}}(\mathbf{x})),$
(7)

where $\tau_{m_{1},m_{2}}(\mathbf{x})$ is the delay between the microphones
$m_{1}$ and $m_{2}$ associated to the spatial position $\mathbf{x}$.

An alternative to building the SRP-based acoustic map – which happens to be
computationally expensive as it usually amounts to a grid search – is
localization by exploiting the sound intensity. The use of sound intensity for
source localization has a long history, e.g., (Nehorai and Paldi, 1994;
Hickling et al., 1993; Jarrett et al., 2010; Tervo, 2009; Raangs and
Druyvesteyn, 2002; Basten et al., 2008). In favorable acoustic conditions,
sound intensity is parallel to the direction of the propagating sound wave
(see Section 5.5), and hence the DoA can be efficiently estimated.
Unfortunately, its accuracy quickly degrades in the presence of acoustic
reflections (Daniel and Kitić, 2020).

Subspace methods are another classical family of localization algorithms.
These methods rely on the computation of the (time-averaged) CPS matrix
$\mathbf{R}(f)$ defined by:

$\mathbf{R}(f)=\sum_{n=1}^{N}\mathbf{X}(f,n)\mathbf{X}(f,n)^{H},$ (8)

where $\mathbf{X}(f,n)$ is the STFT (or more generally a local discrete
Fourier transform) of the multichannel signal vector defined in (4) (H denotes
the Hermitian operator), and its eigenvalue decomposition (EVD). Assuming that
the target source signals and noise are uncorrelated, the multiple signal
classification (MUSIC) method (Schmidt, 1986) applies EVD to estimate the
signal and noise subspaces. After Eq. (4), the signal subspace bases are
assumed to correspond to the columns of the mixing matrix $\mathbf{A}(f)$,
which are the multichannel ATFs of the sources (often referred to as _steering
vectors_ in this context). The signal or noise subspace bases are then used to
probe a given direction for the presence of a source, i.e. apply _spatial
filtering_ or _beamforming_ (Van Veen and Buckley, 1988; Benesty et al.,
2008). This time-demanding search can be relaxed using the Estimation of
Signal Parameters via Rotational Invariance Technique (ESPRIT) algorithm (Roy
and Kailath, 1989), which exploits the structure of the source subspace to
directly infer the source DoA. However, this often comes at the cost of
producing less accurate predictions than MUSIC (Mabande et al., 2011). MUSIC
and ESPRIT assume narrowband signals, although wideband extensions have been
proposed, e.g., (Dmochowski et al., 2007; Hogg et al., 2021). Subspace methods
are robust to noise and can produce highly accurate estimates, but they are
sensitive to reverberation.

Methods based on probabilistic generative mixture models have been proposed
by, e.g., Roman and Wang (2008); Mandel et al. (2009); May et al. (2011);
Woodruff and Wang (2012); Schwartz and Gannot (2013); Dorfan and Gannot
(2015); Li et al. (2017). Typically, the models are variants of Gaussian
mixture models (GMMs), with one Gaussian component per source to be localized
or per candidate source position. In a very few papers (e.g., (May et al.,
2011)), the model is trained offline with a dedicated training dataset. But
most often, the model parameters are directly estimated “at test time,” that
is using the multichannel signal containing the sources to localize. This is
done by maximizing the data likelihood function with histogram-based or
expectation-maximization (EM) algorithms exploiting the sparsity of sound
sources in the time-frequency (TF) domain (Rickard, 2002), which can be
computationally intensive. A GMM variant functioning directly in regression
mode, _i.e._ , a form of Gaussian mixture regression (GMR), was proposed for
single-source localization by Deleforge and Horaud (2012) and later extended
to multi-source localization (and possibly separation) (Deleforge et al.,
2013, 2015). The GMR is locally linear but globally non-linear and the
estimation of the model parameters is done offline on training data. Hence the
spirit is close to DNN-based SSL. White noise signals convolved with synthetic
RIRs were used for training. The method was shown to generalize well to speech
signals, which are sparser than noise in the TF domain, thanks to the use of a
latent variable modeling the signal activity in each TF bin.

Mixture models are strongly connected to Bayesian inference, which considers
the posterior distribution of model parameters given the observed data (hence
involving both the likelihood function and a prior distribution of the model
parameters). Escolano et al. (2014) considered applying Bayesian inference on
a Laplacian source mixture model, using GCC-PHAT features in a two-microphone
array set-up. Interestingly, they used two levels of Bayesian inference: one
for the estimation of the NoS (which is an hyper-parameter of the model),
using Bayesian model selection, and one for the estimation of the model
parameters (and thus the corresponding source DoAs), using posterior
distribution evaluation. In this work, the evaluation of the involved
distributions was done with sampling techniques, e.g., Markov Chain Monte
Carlo (MCMC) methods. The same methodology was further applied by Bush and
Xiang (2018) with a coprime array consisting of two superimposed, spatially
undersampled, uniform linear arrays (Vaidyanathan and Pal, 2010), and by
Landschoot and Xiang (2019) in the spherical harmonics domain using a
spherical microphone array (see Section 5).

Compressive sensing and sparse recovery methods are widely used in acoustics
for different purposes Gerstoft et al. (2018); Xenaki et al. (2014), including
SSL Yang et al. (2018). The main premise is that many high-dimensional signals
admit a low-dimensional representation, which can be viewed through, e.g.,
_sparse synthesis_ Candes et al. (2006) or _sparse analysis_ Nam et al. (2013)
model. Concerning the SSL problem, the sparsity assumption is usually assumed
in the spatial (or spatial beam) domain, e.g., Chardon and Daudet (2012);
Noohi et al. (2013); Fortunati et al. (2014); Kitić et al. (2014); Gerstoft et
al. (2016), and the resulting problem is addressed by convex optimization,
greedy or Bayesian methods, e.g., Foucart and Rauhut (2013); Gerstoft et al.
(2018). This concept has lead to prominent localization methods achieving
remarkable performance. Nonetheless, despite their strong theoretical
guarantees, compressive sensing methods suffer from two drawbacks. For one, it
is usually required that the sources coincide with points of some pre-defined
grid, although grid-free methods have been proposed in some specific cases,
e.g., Xenaki and Gerstoft (2015); Yang and Xie (2015). The second issue is
shared with other conventional methods, _i.e._ , the strong modeling
assumptions reflected in, for example, the known structure of the (sub-
Gaussian) dictionary matrix. Dictionary learning techniques have been proposed
to alleviate the latter problem to some extent, e.g., Wang et al. (2018);
Hahmann et al. (2021b); Zea and Laudato (2021). Sparse Bayesian learning (SBL)
is a combination of the Bayesian framework with the principles of sparse
representations and compressed sensing. It usually involves using sparse
arrays such as the coprime array mentioned above and nested arrays (Pal and
Vaidyanathan, 2010). SBL has been used for SSL by, e.g., Nannuru et al.
(2018); Zhang et al. (2014); Liu et al. (2012); Gerstoft et al. (2016); Xenaki
et al. (2018); Ping et al. (2020).

Finally, ICA is a class of algorithms aimed at retrieving the different source
signals comprising a mixture by assuming and exploiting their mutual
statistical independence. ICA has most often been used in audio processing for
blind source separation, but it has also proven to be useful for multi-source
SSL (Sawada et al., 2003). As briefly stated before, in the multi-source
scenario, SSL is closely related to the source separation problem, since
localization can help separation, and separation can help localization (Gannot
et al., 2017; Vincent et al., 2018).

##  4 Neural network architectures for SSL

In this section, we discuss the neural network architectures that have been
proposed in the literature to address the SSL problem. However, we do not
present the basics of these neural networks since they have been extensively
described in the general DL literature, e.g., (LeCun et al., 2015; Goodfellow
et al., 2016; Chollet, 2017). The design of DNNs for a given application often
requires investigating (and possibly combining) different architectures and
tuning their hyperparameters. This was the case for SSL over the last decade,
and the evolution of DL-based SSL techniques has followed the general
evolution of DNNs toward more and more complex architectures or new efficient
models adopted by the DL and SP communities at large, i.e., largely beyond the
SSL problem (e.g., attention models). In other words, the DNN architectures
used in SSL are often inherited from other works in other (connected or more
distant) domains, simply because they were shown to work well on audio signals
or other types or signals. In the same spirit, different models are often
combined (in parallel and/or sequentially).

We have thus organized the presentation according to the type of layers used
in the networks, with a progressive and “inclusive” approach in terms of
complexity: a network within a given category can contain layers from another
previously presented category. We thus first present systems based on
feedforward neural networks (FFNNs). We then focus on convolutional neural
networks (CNNs) and recurrent neural networks (RNNs), which generally
incorporate some feedforward layers. Next, we review architectures combining
CNNs with RNNs, namely convolutional recurrent neural networks (CRNNs). Then,
we focus on neural networks with residual connections and with attention
mechanisms. Finally, we present SSL systems with an encoder-decoder
architecture.

###  4.1 Feedforward neural networks

Figure 2: The MLP architecture used by Takeda _et al._ in several papers
(Takeda and Komatani, 2016b, a, 2017; Takeda et al., 2018). Multiple subband
feedforward layers, indexed by $w$, are trained to extract features from the
CPS matrix eigenvectors $\mathbf{e}_{w,i}$, which are used as directional
activation functions. The obtained subband vectors $\mathbf{X}_{2,w}$ are
integrated across subbands progressively via other feedforward layers, giving
$\mathbf{X}_{3,w}$ and then $\mathbf{X}_{4}$. The output layer finally
classifies its input in one of the candidate DoAs (the entries of the vector
$\mathbf{X}_{5}$). Note: Reprinted from (Takeda and Komatani, 2016a);
copyright by IEEE; reprinted with permission.

The FFNN was the first and simplest type of artificial neural network to be
designed. In such a network, data move in one direction from the input layer
to the output layer, possibly via a series of hidden layers (Goodfellow et
al., 2016; LeCun et al., 2015). Non-linear activation functions are usually
used after each layer (possibly except for the output layer). While this
definition of FFNN is very general and may include architectures such as CNNs
(discussed in the next subsection), here we mainly focus on architectures made
of fully-connected layers known as Perceptron and Multi-Layer Perceptron (MLP)
(Goodfellow et al., 2016; LeCun et al., 2015). A Perceptron has no hidden
layer, while the notion of MLP is a bit ambiguous: some authors state that an
MLP has one hidden layer, while others allow more hidden layers. In this
paper, we call an MLP an FFNN with one or more hidden layers.

A few pioneering SSL methods using shallow neural networks (Perceptron or
1-hidden layer MLP) and applied in “unrealistic” setups (e.g., assuming
direct-path sound propagation only) have been briefly mentioned in Section
2.1. One of the first uses of an MLP for SSL was proposed by Kim and Ling
(2011), who actually considered several MLPs. One network estimates the NoS,
after which a distinct network is used for SSL for each considered NoS. The
authors evaluated their method on reverberant data even though they assumed an
anechoic setting. Tsuzuki et al. (2013) proposed using a complex-valued MLP in
order to process complex two-microphone-based features, which led to better
results than using a real-valued MLP. Youssef et al. (2013) also used an MLP
to estimate the azimuth of a sound source from a binaural recording made with
a robot head. The interaural time difference (ITD) and the interaural level
difference (ILD) values (see Section 5) were separately fed into the input
layer and were each processed by a specific set of neurons. A single-hidden-
layer MLP was used by Xiao et al. (2015), taking GCC-PHAT-based features as
inputs and tackling SSL as a classification problem (see Section 8), which
showed an improvement over conventional methods on simulated and real data. A
similar approach was proposed by Vesperini et al. (2016), but the localization
was done by regression in the horizontal plane.

Naturally, MLPs with deeper architecture (_i.e._ , more hidden layers) have
also been investigated for SSL. Roden et al. (2015) compared the performance
of an MLP with two hidden layers and different input types, the number of
hidden neurons being linked to the type of input features (see Section 5 for
more details). Yiwere and Rhee (2017) used an MLP with three hidden layers
(tested with different numbers of neurons) to output source azimuth and
distance estimates. An MLP with four hidden layers was tested by He et al.
(2018a) for multi-source localization and speech/non-speech classification,
showing similar results as a 4-layer CNN (see Section 4.2).

Ma et al. (2015) proposed using a different MLP for different frequency sub-
bands, with each MLP having eight hidden layers. This idea is based on the
assumption that, in the presence of multiple sources, each frequency band is
mostly dominated by a single source, which enables the training to be done
exclusively on single-source data. The output of each sub-band MLP corresponds
to a probability distribution on azimuth regions, and the final azimuth
estimations are obtained by integrating the probability values over the
frequency bands. Another system in the same vein was proposed by Takeda _et
al._ in several papers (Takeda and Komatani, 2016b, a, 2017; Takeda et al.,
2018). In these works, the eigenvectors of the recorded signal interchannel
correlation matrix were separately fed per frequency band into parallel
branches of the network, particularly into specific fully-connected layers.
Then, several additional fully-connected layers progressively integrated the
frequency-dependent outputs (see Fig. 2). The authors showed that this
specific architecture outperforms a more conventional 7-layer MLP and the
classical MUSIC algorithm on anechoic and reverberant single- and multi-source
signals. Opochinsky et al. (2019) proposed a small 3-layer MLP to estimate the
azimuth of a single source using the relative transfer function (RTF, see
Section 5.1.1) of the signal. Their approach is weakly supervised since one
part of the loss function is computed without the ground truth DoA labels (see
Section 8).

An indirect use of an MLP was explored by Pak and Shin (2019), who used a
3-layer MLP to enhance the interaural phase difference (IPD) (see Section 5)
of the input signal, which was then used for DoA estimation.

###  4.2 Convolutional neural networks

CNNs are a popular class of DNNs widely used for pattern recognition due to
their property of being translation equivariant (Cohen et al., 2019;
Goodfellow et al., 2016). They have been successfully applied to various
tasks, such as image classification, e.g., (Krizhevsky et al., 2017), natural
language processing (NLP), e.g., (Kim, 2014) or automatic speech recognition,
e.g., (Waibel et al., 1989). CNNs have also been used for SSL, as detailed
below.

To our knowledge, Hirvonen (2015) was the first to use a CNN for SSL. He
employed this architecture to classify an audio signal containing one speech
or musical source into one of eight spatial regions (see Fig. 3). This CNN is
composed of four convolutional layers to extract feature maps from
multichannel magnitude spectrograms (see Section 5), followed by four fully-
connected layers for classification. Classical pooling is not used because,
according to the author, it does not seem relevant for audio representations.
Instead, a 4-tap stride with a 2-tap overlap is used to reduce the number of
parameters. This approach shows good performance on single-source signals and
is capable of adapting to different configurations without hand-engineering.
However, two topical issues of such a system were pointed out by the author:
the robustness of the network with respect to a shift in source location, and
the difficulty of interpreting the hidden features.

Figure 3: The CNN architecture proposed by Hirvonen (2015) for SSL. The input
is an 8-channel signal. For each short-term frame, the 8 magnitude spectra (of
$128$ frequency bins) are concatenated to form a $1024\times 1$ tensor, which
is fed into a series of four convolutional layers with $500$ or $600$
learnable kernels. The extracted features then pass through several
feedforward layers containing $500$ or $300$ neurons. The output layer
contains $8$ neurons (or $16$ if the source type is also considered) and
estimates the probability of a source being present in eight candidate DoAs
using a softmax activation function. Note: Reprinted from (Hirvonen, 2015);
copyright by the author; reprinted with permission.

Chakrabarty and Habets also designed a CNN to predict the azimuth of one
(Chakrabarty and Habets, 2017a) or two (Chakrabarty and Habets, 2019b, 2017b)
speakers in reverberant environments. The input features are the multichannel
short-time Fourier transform (STFT) phase spectrograms (see Section 5). In
(Chakrabarty and Habets, 2017a), they proposed using three successive
convolutional layers with $64$ filters of size $2\times 2$ to consider
neighboring frequency bands and microphones. In (Chakrabarty and Habets,
2017b), they reduced the filter size to $2\times 1$ ($1$ in the frequency
axis) because of the W-disjoint orthogonality (WDO) assumption for speech
signals, which assumes that several speakers are not simultaneously active in
a same TF bin (Rickard, 2002). In (Chakrabarty and Habets, 2019b), they
demonstrated that for an $M$-microphone array, the optimal number of
convolutional layers for exploiting phase correlations between the neighboring
microphones is $M-1$.

He et al. (2018a) compared a 4-layer MLP and a 4-layer CNN for the multi-
speaker detection and localization task. The results showed similar accuracy
for both architectures. A deeper architecture was proposed by Yalta et al.
(2017), with 11 to 20 convolutional layers depending on the experiments. These
deeper CNNs showed robustness against noise compared to MUSIC, as well as
smaller training time, but this was partly due to the presence of residual
blocks (see Section 4.5). A similar architecture was presented by He et al.
(2018b), with many convolutional layers and some residual blocks, although
with a specific multi-task configuration. The end of the network was split
into two convolutional branches, one for azimuth estimation, and the other for
speech/non-speech signal classification.

While most localization systems aim to estimate the azimuth or both the
azimuth and elevation, Thuillier et al. (2018) investigated the estimation of
only the elevation angle using a CNN with binaural input features: the
ipsilateral and contralateral head-related transfer function (HRTF) magnitude
responses (see Section 5). Vera-Diaz et al. (2018) chose to apply a CNN
directly on raw multichannel waveforms, assembled side by side as an image, to
predict the Cartesian coordinates $(x,y,z)$ of a single static or moving
speaker. The successive convolutional layers contain around a hundred filters
from size $7\times 7$ for the first layers to $3\times 3$ for the last layer.
Ma and Liu (2018) also used a CNN to perform regression, but they used the CPS
matrix as an input feature (see Section 5). To estimate both the azimuth and
elevation, Nguyen et al. (2018) used a relatively small CNN (two convolutional
layers) in regression mode, with binaural input features. A similar approach
was considered by Sivasankaran et al. (2018) for speaker localization based on
a CNN. They showed that injecting a speaker identifier, particularly a mask
estimated for the speaker uttering a given keyword, alongside the binaural
features at the input layer improved the DoA estimation.

A joint VAD and DoA estimation CNN was developed by Vecchiotti et al. (2018).
They showed that both problems can be handled jointly in a multi-room
environment using the same architecture, although considering separate input
features (GCC-PHAT and log-mel-spectrograms) in two separate input branches.
These branches are then concatenated in a further layer. Vecchiotti et al.
(2019b) extended this work by exploring several variant architectures and
experimental configurations, and Vecchiotti et al. (2019a) developed an end-
to-end auditory-inspired system based on a CNN, with Gammatone filter layers
included in the neural architecture. A method based on mask estimation was
proposed by Zhang et al. (2019b), in which a TF mask was estimated and used to
either clean or be appended to the input features, facilitating the DoA
estimation by a CNN.

Nguyen et al. (2020a) presented a multi-task CNN containing 10 convolutional
layers with average pooling, inferring both the NoS and the sources’ DoA. They
evaluated their network on signals with up to four sources, showing very good
performance on both simulated and real environments. A small 3-layer CNN was
employed by Varanasi et al. (2020) to infer both azimuth and elevation using
signals decomposed with third-order spherical harmonics (see Section 5). The
authors tried several combinations of input features, including using only the
magnitude and/or the phase of the spherical harmonic decomposition.

In the context of hearing aids, a CNN was applied to both VAD and DoA
estimation by Varzandeh et al. (2020). This system is based on two input
features, GCC-PHAT and periodicity degree, both fed separately into two
convolutional branches. These two branches are then concatenated in a further
layer, which is followed by feedforward layers. Fahim et al. (2020) applied an
8-layer CNN to the so-called modal coherence of first-order Ambisonics input
features (see Section 5) for the localization of multiple sources in a
reverberant environment. They proposed a new method to train a multi-source
DoA estimation network with only single-source training data, showing an
improvement over the system of Chakrabarty and Habets (2019b), especially for
signals with three speakers. Hao et al. (2020) investigated a real-time
implementation of SSL using a CNN with a relatively small architecture (three
convolutional layers).

Krause et al. (2020a) investigated the use of several types of convolution.
They reported that networks using 3D convolutions (on the time, frequency, and
channel axes) achieved better localization accuracy compared to those based on
2D convolutions, complex convolutions, and depth-wise separable convolutions
(all of them on the time and frequency axes), but with a high computational
cost. They also showed that the use of depth-wise separable convolutions leads
to a good trade-off between accuracy and model complexity (to our knowledge,
they were the first to explore this type of convolutions).

Bologni et al. (2021) proposed a neural network architecture including a set
of 2D convolutional layers for frame-wise feature extraction, followed by
several 1D convolutional layers in the time dimension for temporal
aggregation. Diaz-Guerra et al. (2021a) applied 3D convolutional layers on
SRP-PHAT power maps computed for both azimuth and elevation estimation. They
also used a couple of 1D causal convolutional layers at the end of the network
to perform single-source tracking. Their whole architecture was designed to
function in fully causal mode so that it can be adapted for real-time
applications. Wu et al. (2021b) proposed using a supervised image mapping
approach inspired from computer vision works and referred to as image
translation. The used a CNN (completed with residual layers, see Section 4.5)
to map an input 2D image (DoA features extracted by conventional beamforming
and reshaped as a function of Cartesian coordinates $(x,y)$) into an output 2D
image of the target source position (in which the pixel intensity is
decreasing rapidly with the distance to the source), from which the source
location is obtained.

As mentioned in the introduction, the DCASE Challenge includes a SELD task,
and CNNs have also been used in some of the challenge candidate systems
(Politis et al., 2020b). Chytas and Potamianos (2019) used convolutional
layers with hundreds of filters of size $4\times 10$ for azimuth and elevation
estimation in a regression mode. Kong et al. (2019) compared different numbers
of convolutional layers for SELD, while an 8-layer CNN was proposed by Noh et
al. (2019) to improve the results over the baseline.

An indirect use of a CNN was proposed by Salvati et al. (2018). They trained
the neural network to estimate a weight for each of the narrow-band SRP
components fed at the input layer in order to compute a weighted combination
of these components. In their experiments, they showed on a few test examples
that this allowed for a better fusion of the narrow-band components and
reduced the effects of noise and reverberation, leading to better localization
accuracy.

In the DoA estimation literature, a few works have explored the use of dilated
convolutions in DNNs. Dilated convolutions, also known as atrous convolutions,
are a type of convolutional layer in which the convolution kernel is wider
than the classical one but zeros are inserted so that the number of parameters
remains the same. Formally, a 1D dilated convolution with a dilation factor
$l$ is defined by

$(x*k)(n)=\sum_{i}x(n-li)k(i),$ (9)

where $x$ is the input and $k$ the convolution kernel. The conventional linear
convolution is obtained with $l=1$. This definition extends to
multidimensional convolution.

Chakrabarty and Habets (2019a) demonstrate that incorporating dilated
convolutions with gradually increasing dilation factors reduces the optimal
number of convolutional layers of their original CNN architecture (Chakrabarty
and Habets, 2019b) (discussed previously in this section). This leads to an
architecture with similar SSL performance and lower computational cost.

###  4.3 Recurrent neural networks

RNNs are neural networks designed for modeling temporal sequences of data
(LeCun et al., 2015; Goodfellow et al., 2016). Particular types of RNNs
include long short-term memory (LSTM) cells (Hochreiter and Schmidhuber, 1997)
and gated recurrent units (GRUs) (Cho et al., 2014). These two types of RNNs
have become very popular thanks to their capability to circumvent the training
difficulties that regular RNNs face, in particular the vanishing and exploding
gradient problems (LeCun et al., 2015; Goodfellow et al., 2016).

There are few published works on SSL using only RNNs, as recurrent layers are
often combined with convolutional layers (see Section 4.4). Nguyen et al.
(2021a) used an RNN to align SED and DoA predictions, which were obtained
separately for each possible sound event type. The RNN was ultimately used to
determine which SED prediction matched which DoA estimation. A bidirectional
LSTM network was used by Wang et al. (2019) to estimate a TF mask to enhance
the signal, further facilitating DoA estimation by conventional methods such
as SRP or subspace methods.

###  4.4 Convolutional recurrent neural networks

CRNNs are neural networks containing one or more convolutional layers and one
or more recurrent layers. CRNNs have been regularly exploited for SSL since
2018 because of the respective capabilities of these layers: The convolutional
layers have proven to be suitable for extracting relevant features for SSL,
and the recurrent layers are well designed for integrating the information
over time.

In the series of papers (Adavanne et al., 2019c, 2018, a), Adavanne _et al._
used a CRNN for SELD, in a multi-task configuration, with first-order
Ambisonics (FOA) input features (see Section 5). In (Adavanne et al., 2018),
their architecture contained a series of successive convolutional layers, each
followed by a max-pooling layer and two bidirectional GRU (BGRU) layers. Then,
a feedforward layer provided an estimation of the spatial pseudo-spectrum
(SPS) provided by the MUSIC algorithm (Schmidt, 1986), acting as an
intermediary output (see Fig. 4). This SPS was then fed into the second part
of the neural network, which was composed of two convolutional layers, a dense
layer, two BGRU layers, and a final feedforward layer for azimuth and
elevation estimation by classification. The use of an intermediary SPS output
has been proposed to help the neural network learn a representation that has
proven to be useful for SSL using traditional methods.

In (Adavanne et al., 2019a) and (Adavanne et al., 2019c), this intermediary
output was no longer used. Instead, the DoA was directly estimated using a
block of convolutional layers, a block of BGRU layers, and a feedforward
layer. This system is able to localize and detect several sound events even if
they overlap in time, provided they are of different types (e.g., speech and
car, see the discussion in Section 2.2). This CRNN was the baseline system for
Task 3 of the DCASE Challenge in 2019 and 2020. Therefore, it has inspired
many other works, and many DCASE Challenge candidate systems were built on the
system of Adavanne et al. (2019a) with various modifications and improvements.

For example, Lin and Wang (2019) added Gaussian noise to the input
spectrograms to train the network to be more robust to noise. Lu (2019)
integrated some additional convolutional layers and replaced the BGRU layers
with bidirectional LSTM layers. Leung and Ren (2019) used the same
architecture with all combinations of cross-channel power spectra, whereas the
replacement of input features with group delays was tested by Nustede and
Anemüller (2019). GCC-PHAT features were added as input features by Maruri et
al. (2019). Zhang et al. (2019a) used data augmentation during training and
averaged the output of the network for a more stable DoA estimation. Xue et
al. (2019) sent the input features separately into different branches of
convolutional layers, log-mel, and constant Q-transform features on the one
hand, and phase spectrograms and CPS features on the other hand (see Section
5). Cao et al. (2019b) concatenated the log-mel spectrogram and GCC-PHAT
features and fed them into two separate CRNNs for SED and DoA estimation (they
also incorporated the intensity vector in (Cao et al., 2019a)). In contrast to
the baseline of Adavanne et al. (2019a), more convolutional layers and one
single BGRU layer were used. The convolutional part of the DoA network was
transferred from the SED CRNN, which was followed by fine-tuning of the DoA
branch, labelling this method as two-stage. This led to a notable improvement
in localization performance over the DCASE Challenge baseline of Adavanne et
al. (2019a). Small changes to this baseline were also tested by Pratik et al.
(2019), such as the use of Bark-scale spectrograms as input features, the
modification of the activation function or pooling layers, and the use of data
augmentation, resulting in noticeable improvements for some experiments.

The same baseline neural architecture of Adavanne et al. (2019a) was used by
Kapka and Lewandowski (2019), with one separate (but identical, except for the
output layer) CRNN instance for each subtask: source counting (up to two
sources), DoA estimation of source 1 (if applicable), DoA estimation of source
2 (if applicable), and sound type classification. The authors showed that
their method was more efficient than the baseline. Krause and Kowalczyk (2019)
explored different manners of splitting the SED and DoA estimation tasks in a
CRNN. While some configurations showed an improvement in SED, the localization
accuracy was below the baseline for the reported experiments. Park et al.
(2019b) investigated a combination of a gated linear unit (GLU, a
convolutional block with a gated mechanism) and a trellis network (containing
convolutional and recurrent layers, see the paper by Bai et al. (2019) for
details), yielding better results than the baseline. The authors extended this
work for the DCASE 2020 Challenge by improving the overall architecture and
investigating other loss functions (Park et al., 2020). A non-direct DoA
estimation scheme was also derived by Grondin et al. (2019), who estimated the
TDoA using a CRNN, from which they inferred the DoA.

We also found propositions of CRNN-based systems in the 2020 edition of the
DCASE Challenge. Singla et al. (2020) used the same CRNN as in the baseline of
Adavanne et al. (2019a), except that they did not use two separated output
branches for SED and DoA estimation. Instead, they concatenated the SED output
with the output of the previous layer to estimate the DoA. Song (2020) used
separated neural networks similar to the one of Adavanne et al. (2019a) to
address NoS estimation and DoA estimation in a sequential way. Multiple CRNNs
were trained by Tian (2020): one to estimate the NoS (up to two sources),
another to estimate the DoA assuming one active source, and another (same as
the baseline) to estimate the DoAs of two simultaneously active sources. Cao
et al. (2020) designed an end-to-end CRNN architecture to detect and estimate
the DoA of possibly two instances of the same sound event. The addition of
one-dimensional convolutional filters was investigated by Ronchini et al.
(2020) to exploit the information along the feature axes. Sampathkumar and
Kowerko (2020) augmented the baseline system of Adavanne et al. (2019a) by
providing the network with more input features (log-mel spectrograms, GCC-
PHAT, and intensity vector, see Section 5).

Figure 4: The CRNN architecture of Adavanne et al. (2018, 2019c, 2019a), which
has inspired numerous SELD systems. The input is the multichannel STFT-domain
FOA magnitude and phase spectrogram. First, features are extracted by four
successive convolutional layers with $64$ $3\times 3$ kernels, each followed
by a max-pooling layer. Then two BGRU layers with $64$ units each and tanh
activations are used to capture the temporal evolution of the extracted
features. An intermediate SPS output is then computed using a time distributed
feedforward layer (i.e., this layer is computed separately on each vector of
the temporal axis). Then, two $16$-kernel convolution layers followed by a
$32$-unit time distributed feedfoward layer and two $16$-units BGRU layers
process the estimated SPS. A final $432$-unit time distributed feedforward
layer with sigmoid activation function is employed to infer the DoA. Note:
Reprinted from (Adavanne et al., 2018); copyright by IEEE; reprinted with
permission.

Independently of the DCASE Challenge, the CRNN of Adavanne et al. (2019a) was
adapted by Comminiello et al. (2019) to receive quaternion FOA input features,
which slightly improved the CRNN performance. Perotin _et al._ proposed using
a CRNN with bidirectional LSTM layers on the FOA pseudo-intensity vector to
localize one (Perotin et al., 2018b) or two (Perotin et al., 2019b) speakers.
They showed that this architecture achieves very good performance in simulated
and real reverberant environments with static speakers (both types of input
features are discussed in Section 5). This work was extended by Grumiaux et
al. (2021a), who obtained a substantial improvement in performance over the
CRNN of Perotin et al. (2019b) by adding more convolutional layers with less
max-pooling, to localize up to three simultaneous speakers.

Non-square convolutional filters and a unidirectional LSTM layer were used in
the CRNN architecture of Li et al. (2018). Xue et al. (2020) presented a CRNN
with two types of input features: the phase of the CPS and the signal
waveforms. The former was first processed by a series of convolutional layers
before being concatenated with the latter. Another improvement of the network
of Adavanne et al. (2019a) was proposed by Komatsu et al. (2020), who replaced
the classical convolutional blocks with GLUs, based on the hypothesis that
GLUs are better suited for extracting relevant features from phase
spectrograms. This has led to a notable improvement of localization
performance compared to the baseline of Adavanne et al. (2019a). Bohlender et
al. (2021) proposed an extension of the system of Chakrabarty and Habets
(2019b), in which LSTMs and temporal convolutional networks (TCNs) replaced
the last dense layer of the former architecture. A TCN was made of successive
1D dilated causal convolutional layers with increasing dilated factors (Lea et
al., 2017). The authors showed that taking the temporal context into account
with such temporal layers actually improves the localization accuracy.

Finally, we can mention the original approach of Nguyen et al. (2020c) in
which a two-step hybrid approach with two CRNNs is used: In the first step, a
first CRNN is used for SED and a single-source histogram-based (conventional)
method is used for DoA estimation. In the second step, a second CRNN-based
network, referred to as sequence matching network (SMN), is used to match the
estimated sequences from the SED and DoA branches. This approach is motivated
by the fact that overlapping sounds often have different onsets and offsets,
and by matching the outputs of the two branches, an estimated DoA can be
associated with the corresponding sound class. This approach was extended to
localize moving sources in the framework of the DCASE 2020 Challenge (Nguyen
et al., 2020b), by adapting the resolution of the azimuth and elevation
histograms and by using an ensemble of SMNs.

###  4.5 Residual neural networks

Residual neural networks were originally introduced by He et al. (2016), who
pointed out that designing very deep networks can lead the gradients to
explode or vanish due to the non-linear activation functions, as well as the
degradation of the overall performance. Residual connections are designed to
enable a feature to bypass a layer block in parallel to the conventional
process through this layer block. This allows the gradients to flow directly
through the network, usually leading to a better training.

To our knowledge, the first use of a network with residual connections for SSL
was proposed by Yalta et al. (2017). As illustrated in Fig. 5, this network
includes three residual blocks, which are stacks of layers with one of the
layers having residual connections with another layer deeper in the stack.
Each of these blocks is made of three convolutional layers, the first and last
of which are designed with $1\times 1$ filters, with the middle layer designed
with $3\times 3$ filters. A residual connection is used between the input and
output of each residual block. The same type of residual block was used for
SSL by He et al. (2019a, 2018b) in parallel to sound classification as speech
or non-speech. Suvorov et al. (2018) used a series of 1D convolutional layers
with several residual connections for single-source localization, directly
from the multichannel waveform.

Pujol et al. (2019, 2021) integrated residual connections alongside 1D dilated
convolutional layers with increasing dilation factors. They used the
multichannel waveform as the network input. After the input layer, the
architecture was divided into several subnetworks containing the dilated
convolutional layers, which functioned as filter banks. Ranjan et al. (2019)
combined a modified version of the original ResNet architecture (He et al.,
2016) with recurrent layers for SELD. This was shown to reduce the DoA error
by more than 20° compared to the baseline of Adavanne et al. (2019a).
Similarly, Bai et al. (2021) also used the ResNet model of (He et al., 2016)
followed by two GRU layers and two fully-connected layers for SELD. Kujawski
et al. (2019) also adopted the original ResNet architecture and applied it to
the single-source localization problem.

Another interesting architecture containing residual connections was proposed
by Naranjo-Alcazar et al. (2020) for the DCASE 2020 Challenge. Before the
recurrent layers (consisting of two BGRUs), three residual blocks successively
processed the input features. These residual blocks contained two residual
convolutional layers, followed by a squeeze-excitation module (Hu et al.,
2020). These modules aim to improve the modeling of interdependencies between
input feature channels compared to classical convolutional layers. Similar
squeeze-excitation mechanisms were used by Sundar et al. (2020) for multi-
source localization. Another combination of a residual network with squeeze-
excitation blocks was reported by Huang and Perez (2021), who implemented it
in the framework of a sample-level CNN (i.e., a CNN applied on the time-domain
signal samples) (Lee et al., 2017). The resulting blocks are further followed
by two Conformer blocks (see the next subsection). The motivation for
combining these different models was their observed effectiveness in other
audio processing tasks such as SED.

Shimada et al. (2020b) and Shimada et al. (2020a) adapted the MMDenseLSTM
architecture, originally proposed by Takahashi et al. (2018) for sound source
separation, to the SELD problem. This architecture consists of a series of
blocks made of convolutions and recurrent layers with residual connections.
Their system showed very good performance among the other participants to the
DCASE 2020 Challenge. Wang et al. (2020) used an ensemble learning approach in
which several variants of residual neural networks and recurrent layers were
trained to estimate the DoA, achieving the best performance of the DCASE 2020
Challenge.

Figure 5: The residual neural network architecture used by Yalta et al.
(2017). Three residual blocks are employed in this network, which are each
composed of two convolutional layers with $32$ $1\times 1$ filters with
another convolutional layer with $32$ $3\times 3$ filters in-between. For all
three residual blocks, the input is added to the output with a residual
connection, showed with a dashed arrow in this diagram. The authors show that
the use of residual connections not only reduces the learning cost, but also
improves the model performance. Note: Reprinted from (Yalta et al., 2017);
under Creative Commons Attribution-NoDerivatives 4.0 International License.

Guirguis et al. (2020) designed a neural network with a TCN in addition to
classical 2D convolutions and residual connections. Instead of using recurrent
layers as usually considered, the architecture was composed of TCN blocks that
were made of several residual blocks, including a 1D dilated convolutional
layer with an increasing dilated factor. The authors showed that replacing
recurrent layers with TCNs made the hardware implementation of the network
more efficient while slightly improving the SELD performance compared to the
baseline of Adavanne et al. (2019a).

Yasuda et al. (2020) exploited a CRNN with residual connections in an indirect
way for DoA estimation using an FOA pseudo-intensity vector input (see Section
5.5). A CRNN was first used to remove the reverberant part of the FOA pseudo-
intensity vector, after which another CRNN was used to estimate a TF mask,
which was applied to attenuate TF bins with a large amount of noise. The
source DoA was finally estimated directly from the dereverberated and denoised
pseudo-intensity vector.

###  4.6 Attention-based neural networks

An attention mechanism is a method that allows a neural network to put
emphasis on vectors of a temporal sequence that are more relevant for a given
task. Originally, attention was proposed by Bahdanau et al. (2016) to improve
sequence-to-sequence models such as RNNs for machine translation. The general
principle is to allocate a different weight to the vectors of the input
sequence when using a combination of these vectors for estimating a vector of
the output sequence. The model is trained to compute the optimal weights that
reflect both the link between vectors of the input sequence (self-attention)
and the relevance of the input vectors to explain each output vector
(attention at the decoder). This pioneering work has inspired the now popular
Transformer architecture proposed by Vaswani et al. (2017), which greatly
improved the machine translation performance. In the Transformer, RNNs are
removed, i.e. they are totally replaced by attention models.

Attention models are now used in an increasing number of DL applications,
including SSL. Phan et al. (2020a, b) submitted an attention-based neural
system for the DCASE 2020 Challenge. Their architecture was made of several
convolutional layers, followed by a BGRU, after which a self-attention layer
was used to infer the activity and the DoA of several distinct sound events at
each timestep. Schymura et al. (2020) added an attention mechanism after the
recurrent layers of a CRNN to output an estimation of the sound source
activity and its azimuth/elevation. Compared to the baseline of Adavanne et
al. (2019a), the addition of attention demonstrated a better use of temporal
information for SELD. An extension of the system of Chakrabarty and Habets
(2019b) based on attention mechanisms has been proposed by Mack et al. (2020).
Attention is employed to estimate binary masks to focus on frequency bins
where the target source is dominant. The first attention stage appears right
after the input layer (analogously to (Chakrabarty and Habets, 2019b), their
network uses phase spectrograms as inputs), while the second attention stage
takes place after new features have been extracted using convolutional layers.
Adavanne et al. (2021) used a self-attention layer after a GRU in order to
estimate the association matrix which matches predictions and references. This
solves the optimal assignment problem and resulted in large improvements in
terms of localization error.

Multi-head self-attention (MHSA), which is the parallel use of several
Transformer-type attention models (Vaswani et al., 2017), has also inspired
SSL methods. In the DCASE 2021 Challenge, Emmanuel et al. (2021) employed a
MHSA layer right after several convolution modules tailored to learn varying
spectral characteristics. Yalta et al. (2021) proposed using the whole encoder
part of the Transformer architecture, in addition to several convolutional
layers, to extract features from the input data. Wang et al. (2021) adapted
the Conformer architecture, originally designed by Gulati et al. (2020) for
automatic speech recognition, to SSL. This architecture is composed of a
feature extraction module based on ResNet and a MHSA module that learns local
and global context representations. The authors demonstrated the benefit of
using a specific data augmentation technique on this model. Zhang et al.
(2021) also employed this architecture in the DCASE 2021 Challenge. As briefly
mentioned in the previous subsection, Conformer blocks were also used in the
architecture proposed by Huang and Perez (2021), where they followed a sample-
level CNN with residual connections and squeeze-excitation. A Conformer block
was also used in the architecture proposed for SELD by Rho et al. (2021),
after convolutional and fully-connected layers and before BGRU layers. Cao et
al. (2021) positioned an 8-head attention layer after a series of
convolutional layers to track the source location predictions over time for
different sources (up to two sources in their experiments). Schymura et al.
(2021) used three 4-head self-attention encoders along the time axis after a
series of convolutional layers before estimating the activity and location of
several sound events (see Fig. 6). This neural architecture showed an
improvement over the DCASE Challenge baseline of Adavanne et al. (2019a). In
the same line, Xinghao et al. (2021) replaced the conventional convolutional
layers of the baseline with a combination of adaptive convolutional layers
(using dilated convolutions with different dilation factors) and attention
blocks. Another example of MHSA-based Transformer model for SSL can be found
in the work of Park et al. (2021a). In this work, a pretrained model is fine-
tuned with transfer learning. The output sequence corresponding to each
3s-sequence of input data is averaged to provide one DoA estimation.
Sudarsanam et al. (2021) enriched the CRNN baseline of Adavanne et al. (2019a)
with a set of several MHSA blocks followed by fully-connected layers. They
provided an analysis of the influence of the number and dimension of the MHSA
blocks (the optimal number was found to be 2) and the number of heads (optimal
was 8), as well as the effect of positional embedding, normalization layers
and residual connections. Grumiaux et al. (2021b) showed that replacing the
recurrent layers of a CRNN with self-attention encoders yielded a notable
reduction in the computation time. Moreover, the use of MHSA slightly improved
localization performance upon the baseline CRNN architecture of Perotin et al.
(2019b) for the considered multiple speaker localization task.

Finally, we can mention the use of cross-modal attention (CMA) models for SSL
by Lee et al. (2021a). A CMA model is the generalization of self-attention
with two data streams in place of one, which is used in the Transformer
decoder (Vaswani et al., 2017). Lee et al. (2021a) used two separate SED and
DoA estimation CNN blocks to separately produce SED and DoA embeddings (this
comes in contrast with most DCASE candidate systems where the first blocks are
shared between SED and DoA estimation.) Then these embeddings are merged,
first with a weighted linear combination and then with a second, more complex,
alignment process using two mirrored CMA models. Finally, the SED and DoA
outputs of the CMA modules are each sent to three parallel fully-connected
networks for final estimation (this is because in the DCASE 2021 Challenge
SELD Task, up to three sources can be simultaneously active).

In a general manner, it appears that attention modules, and MHSA in
particular, have a tendency to replace the recurrent units in the recent SSL
DNNs, following the “Attention is all you need” seminal line of Vaswani et al.
(2017). This is because compared to RNNs, attention modules can model longer-
term dependencies at a lower computational cost and can highly benefit from
parallel computations, especially at training time. This tendency is also
observed in other application domains, as we will discuss in Section 9.

Figure 6: The self-attention-based neural network architecture of Schymura et
al. (2021). The input is the multi-channel spectrogram shaped as a $K\times
L\times 2C$ tensor, with $K$ the number of frequency bins, $L$ the number of
frames, and $C$ the number of channels. A feature extraction is first done
with convolutional layers (not detailed in the figure) to produce
$\mathbf{z}_{k,n}$ to which is attached a positional encoding vector
$\mathbf{p}_{E}$. Then, a Transformer encoder computes a new representation of
shape $K\times D_{E}$, which is used to compute the source activity
$\gamma_{k,n}$ and the mean $\mathbf{\hat{x}}_{k,n}$ of the multivariate
Gaussian distributions representing the target sources’ location (the
corresponding covariance matrix $\mathbf{\hat{\Sigma}}_{k,n}$ is computed via
a parallel (simpler) mechanism.) Note: Reprinted from (Schymura et al., 2021);
copyright by the authors; reprinted with permission.

###  4.7 Encoder-decoder neural networks

An encoder-decoder network is an architecture made of two building blocks: an
encoder, which is fed by the input features and outputs a specific
representation of the input data, and a decoder, which transforms the new data
representation from the encoder into the desired output data. Architectures
following this principle have been largely explored in the DL literature due
to their capacity to provide compact data representations in an unsupervised
manner (Goodfellow et al., 2016).

####  4.7.1 Autoencoder

An autoencoder (AE) is an encoder-decoder neural network that is trained to
output a copy of its input. Often, the dimension of the encoder’s last layer
output is small compared to the dimension of the data. This layer is then
known as the bottleneck layer and it provides a compressed encoding of the
input data. Originally, AEs were made of feed-forward layers, but this term is
also contemporaneously used to designate AE networks with other types of
layers, such as convolutional or recurrent layers. To the best of our
knowledge, the first use of an AE for DoA estimation was reported by Zermini
et al. (2016). They used a simple AE to estimate TF masks for each possible
DoA, which were then used for source separation. An interesting AE-based
method was presented by Huang et al. (2020), in which an ensemble of AEs were
trained to reproduce the multichannel input signal at the output, with one AE
per candidate source position. Since the common latent information among the
different channels is the dry signal, each encoder approximately deconvolves
the signal from a given microphone. These dry signal estimates should be
similar provided that the source is indeed at the assumed position; hence, the
localization is performed by finding the AE with the most consistent latent
representation. However, it is not clear whether this model can generalize
well to unseen source positions and acoustic conditions.

Le Moing et al. (2020) presented an AE with a large number of convolutional
layers (and transposed convolutional layers, which are layers of the decoder
that process the inverse operation of the corresponding convolutional layer at
the encoder), which estimates the potential source activity of each subregion
in the $(x,y)$ plane divided in a grid, making it possible to locate multiple
sources. They evaluated several types of outputs (binary, Gaussian-based, and
binary followed by regression refinement), each of which showed promising
results on the simulated and real data. An extension of this work was
presented in (Le Moing et al., 2021), in which they proposed using adversarial
training (see Section 8) to improve network performance on real data, as well
as on microphone arrays unseen in the training set, in an unsupervised
training scheme. To do this, they introduced a novel explicit transformation
layer that helped the network to be invariant to the microphone array layout.
Another encoder-decoder architecture was proposed by He et al. (2021b), in
which a multichannel waveform was fed into a filter bank with learnable
parameters, after which a 1D convolutional encoder-decoder network processed
the filter bank output. The output of the last decoder was then fed separately
into two branches, one for SED and the other for DoA estimation.

An encoder-decoder structure with one encoder followed by two separate
decoders was proposed by Wu et al. (2021c). Signals recorded from several
microphone arrays were first transformed in the short-term Fourier transform
domain (see Section 5) and then stacked in a 4D-tensor (whose dimensions were
time, frequency, microphone array and microphone). This tensor was then sent
to the encoder block, which was made of a series of convolutional layers
followed by several residual blocks. The output of the encoder was then fed
into two separate decoders, the first of which was trained to output a
probability of source presence for each candidate $(x,y)$ region, while the
second was trained in the same way but with a range compensation to make the
network more robust. The same general encoder-decoder line was adopted in the
2D image mapping approach proposed by Wu et al. (2021b). Note that here, the
network is composed of convolutional layers at the encoder and transposed
convolutional layers at the decoder, which is typical for image mapping
applications in computer vision.

An indirect use of an AE was proposed by Vera-Diaz et al. (2020), who used
convolutional and transposed convolutional layers to estimate the TDoA from
GCC-based input features. The main idea was to rely on the encoder-decoder
capacity to reduce the dimension of the input data so that the bottleneck
representation forced the decoder to output a smoother version of the TDoA.
This technique was shown to outperform the classical GCC-PHAT method in the
reported experiments. This work was extended in the presence of two sources
(Vera-Diaz et al., 2021).

####  4.7.2 Variational autoencoder

A variational autoencoder (VAE) is a generative model that was originally
proposed by Kingma and Welling (2014) and Rezende et al. (2014) and is now
very popular in the DL community. A VAE can be seen as a probabilistic version
of an AE. Unlike a classical AE, a VAE learns a probability distribution of
the data at the output of the decoder and also models the probability
distribution of the so-called latent vector at the bottleneck layer, which
makes the VAE strongly connected to the concept of unsupervised representation
learning (Bengio et al., 2013). New data can thus be obtained with the decoder
by sampling these distributions.

To our knowledge, Bianco et al. (2020) were the first to apply a VAE for SSL.
Their VAE, made of convolutional layers, was trained to generate the phase of
inter-microphone RTFs (see Section 5.1.1), jointly with a classifier that
estimates the speaker’s DoA from the RTF phases. The interest of using a VAE
is that this generative model, originally designed for unsupervised training,
is here trained in a semi-supervised configuration using a large dataset of
unlabeled RTF data together with a limited set of labeled data (RTF values +
corresponding DoA labels). In such a limited labeled dataset configuration,
this model was shown to outperform an SRP-PHAT-based method as well as a
supervised CNN in reverberant scenarios. This semi-supervised (or weakly
supervised) approach is further discussed in Section 9.1. An extension of this
work has been further proposed in (Bianco et al., 2021), with refined network
architectures and more realistic acoustic scenarii.

####  4.7.3 U-Net architecture

A U-Net architecture is a particular fully-convolutional neural network
originally proposed by Ronneberger et al. (2015) for biomedical image
segmentation. In U-net, the input features are decomposed into successive
feature maps throughout the encoder layers and then recomposed into
“symmetrical” feature maps throughout the decoder layers, similarly to CNNs.
Having the same dimension for feature maps at the same level in the encoder
and decoder enables one to propagate information directly from an encoder
level to the corresponding level of the decoder via residual connections. This
leads to the typical U-shape schematization (see Fig. 7).

Figure 7: The U-Net network architecture of Chazan et al. (2019). The input
matrix $\mathcal{R}$ contains angular features extracted from the RTFs (see
Section 5.1.1) ($l$, $k$, and $M$ denote the time index, the frequency bin,
and the number of microphones, respectively). Several stages of encoders (in
blue) and decoders (in green) are used. At each encoder (or decoder) stage,
two or three convolutional layers with $3\times 3$ kernels are employed to
compute a new representation which is used as the input of the next encoder
(or decoder, respectively), except for the bottleneck stage from which the
output is fed as input into the upper-stage decoder. Residual connections are
used to concatenate one encoder output to the input of the same stage decoder,
to alleviate the loss information problem. The output of this system consists
of one TF mask $p_{l,k}(\theta)$ per considered DoA $\theta$. Note: Reprinted
from (Chazan et al., 2019); copyright by IEEE; reprinted with permission.

Regarding SSL and DoA estimation, several works have been inspired by the
original U-Net paper. Chazan et al. (2019) employed such an architecture to
estimate one TF mask per considered DoA (see Fig. 7), in which each TF bin was
associated with a single particular DoA. This spectral mask was finally
applied for source separation. This system was extended by Hammer et al.
(2021) to account for multiple moving speakers. Another joint localization and
separation system based on a U-Net architecture was proposed by Jenrungrot et
al. (2020). In this system, a U-Net was trained based on 1D convolutional
layers and GLUs. The input is the multichannel raw waveform accompanied by an
angular window that helps the network to perform separation on a particular
zone. If the output of the network on the window is empty, no source is
detected, otherwise, one or more sources are detected and the process is
repeated with a smaller angular window, until the angular window reaches $2$°.
This system shows interesting results on both synthetic and real reverberant
data containing up to eight speakers.

For the DCASE 2020 Challenge, a U-Net with several BGRU layers in-between the
convolutional blocks, was proposed for SELD by Patel et al. (2020). The last
transposed convolutional layer of this U-Net outputs a single-channel feature
map per sound event, corresponding to its activity and DoA for all frames.
This showed an improvement over the baseline of Adavanne et al. (2019a) in
terms of DoA error. Comanducci et al. (2020a) used a U-Net architecture in the
second part of their proposed neural network to estimate the source
coordinates $x$ and $y$. The first part, composed of convolutional layers,
learns to map GCC-PHAT features to the so-called ray space (where source
positions correspond to linear patterns, _cf._ (Bianchi et al., 2016)), which
is an intermediate representation used as the input of the U-Net architecture.

##  5 Input features

In this section, we provide an overview of the variety of input feature types
found in the DL-based SSL literature. Generally, the considered features can
be low-level signal representations such as waveforms or spectrograms, hand-
crafted features such as binaural features, or they can be borrowed from
traditional SP methods such as MUSIC or GCC-PHAT.

Overwhelmingly, the input features for the SSL neural networks are based on
some representation readily used in signal processing, often emphasizing
spatial and/or TF information embedded in the signal. This seems to yield good
results, despite the growing trend in other domains to learn the feature
representation directly from raw data. One interpretation may be that the
network architectures in SSL are usually of a relatively modest size, as
compared to end-to-end models used in some other domains, e.g., NLP. A few
publications have compared different types of input features for SSL, e.g.,
(Roden et al., 2015; Krause et al., 2020b).

It is also quite common to provide the network with concatenated features of
different nature (even if these carry redundant information), which usually
has positive impact on performance. This can be attributed to the flexibility
of the learning process, which seemingly adapts the network weights such that
the pertinent information is efficiently “routed” from such an input to the
upper layers of the network, where it is merged into an abstract, optimized
feature representation.

We organized this section into the following feature categories: inter-
channel, cross-correlation-based, spectrogram-based, Ambisonics, intensity-
based, and finally the direct use of the multichannel waveforms. Note that, as
stated above, different kinds of features are often combined at the input
layer of SSL neural networks.

###  5.1 Inter-channel features

####  5.1.1 Relative transfer function (RTF)

The RTF is a very general inter-channel feature that has been widely used for
conventional (non-deep) SSL and other spatial audio processing such as source
separation and beamforming (Gannot et al., 2017) and acoustic echo
cancellation (Valero and Habets, 2017), and is now considered for DL-based SSL
as well. The RTF is defined for a given sound source position and for a
microphone pair as the ratio $H(f)=A_{2}(f)/A_{1}(f)$ of the source-to-
microphone ATFs of the two microphones, $A_{2}(f)$ and $A_{1}(f)$ (here we are
working in the frequency or STFT domain and we recall that an ATF is the
discrete Fourier transform of the corresponding RIR). It is thus strongly
dependent on the source DoA (for a given recording set-up). In a multichannel
set-up with more than two microphones, we can define an RTF for each
microphone pair. Often, one microphone is used as a reference microphone, and
the ATFs of all other microphones are divided by the ATF of this reference
microphone.

As an ATF ratio, an RTF is thus a vector with an entry defined for each
frequency bin. If only one directional source is present in the recorded
signals and if the (diffuse) background noise is negligible, Eq. (2) shows
that an RTF estimate can be obtained for each STFT frame (indexed by $n$),
each frequency bin, and each microphone pair (indexed by $i$ and $k$) by
taking the ratio between the STFT transforms of the recorded waveforms of the
two considered channels, $X_{i}(f,n)$ and $X_{k}(f,n)$:

$\hat{H}_{i,k}(f)=\frac{X_{k}(f,n)}{X_{i}(f,n)}\approx\frac{A_{k}(f)S(f,n)}{A_{i}(f)S(f,n)}=\frac{A_{k}(f)}{A_{i}(f)}=H_{i,k}(f),$
(10)

where $S(f,n)$ is the STFT of the source signal. In the case where a
background/sensor noise is present, more sophisticated RTF estimation
procedures must be used, e.g., (Cohen, 2004; Markovich-Golan and Gannot, 2015;
Li et al., 2015). If multiple sources are present, things become more
complicated, but using the natural sparsity of speech/audio signals in the TF
domain, _i.e._ , only at most one source is assumed to be active in each TF
bin (Rickard, 2002), the same principle as for one active source can be
applied separately in each TF bin. Therefore, a multiple set of estimated RTFs
at different frequencies (and possibly at different time frames if the sources
are static or not moving too fast) can be used for multi-source localization.
The reader is referred to (Gannot et al., 2017) and references therein for
more information on the RTF estimation problem.

An RTF is a complex-valued vector. In practice, an equivalent real-valued pair
of vectors is often used. We can use either the real and imaginary parts or
the modulus and argument. Often, the log-squared value of the interchannel
power ratio is used, _i.e._ , the interchannel power ratio in dB, and the
argument of the RTF estimate ideally corresponds to the difference of the ATF
phases. Such RTF-based representations have been used in several DNN-based
systems for SSL. For example, Chazan et al. (2019), Hammer et al. (2021), and
Bianco et al. (2020, 2021) used as input features the arguments of the
measured RTFs obtained from all microphone pairs.

####  5.1.2 Binaural features

Binaural features have also been used extensively for SSL, in both
conventional and deep systems (Argentieri et al., 2015). These features
correspond to a specific two-channel recording set-up, one which attempts to
reproduce human hearing in the most realistic way possible. Toward this aim, a
dummy head/body with in-ear microphones is used to mimic the source-to-human-
ear propagation, and in particular the effects of the head and external ear
(pinnae), which are important for source localization by the human perception
system. In an anechoic binaural set-up environment, the (two-channel) source-
to-microphone impulse response is referred to as the binaural impulse response
(BIR). The frequency-domain representation of a BIR is the HRTF. Both BIR and
HRTF are functions of the source DoA. To take into account the room acoustics
in a real-world SSL application, BIRs are extended to binaural room impulse
responses (BRIRs), which combine head/body effects and room effects (in
particular reverberation, see further discussion on BRIR simulation in Section
7.1).

Several binaural features are derived from binaural recordings: The interaural
level difference corresponds to the short-term log-power magnitude of the
ratio between the two binaural channels in the STFT domain, $X_{2}(f,n)$ and
$X_{1}(f,n)$:

$ILD(f,n)=20\log_{10}\displaystyle\left\lvert\frac{X_{2}(f,n)}{X_{1}(f,n)}\right\rvert.$
(11)

The interaural phase difference is the argument of this ratio:

$IPD(f,n)=\angle\frac{X_{2}(f,n)}{X_{1}(f,n)},$ (12)

and the interaural time difference is the delay that maximizes the cross-
correlation between the two channels, similarly to the TDoA in (6). Just like
the RTF, these features are actually vectors with frequency-dependent entries.
In fact, the ILD and IPD are strongly related (not to say similar) to the log-
power and argument of the RTF, as shown by comparing (11) and (12) with (10),
the difference relying more on the set-up than on the features themselves. The
RTF can be seen as a more general (multichannel) concept, whereas binaural
features refer to the specific two-channel binaural setup. As for the RTF, the
ILD, IPD, and ITD implicitly encode the position of a source. Again, when
several sources are present, the sparsity of speech/audio signals in the TF
domain allows ILD/IPD/ITD values to provide information on the position of
several simultaneously active sources.

Youssef et al. (2013) used ILD and ITD vectors fed separately into specific
input branches of an MLP. Ma et al. (2015) and Yiwere and Rhee (2017)
concatenated the cross-correlation of the two binaural channels with the ILD
before feeding it into the input layer of their network. Ma et al. (2015)
justify this choice with two arguments. The first one is to avoid the noise-
sensitivity of the peak-picking operation for the computation of the ITD, the
second one is because of the systematic changes in the cross-correlation
function according to the source azimuth. Nguyen et al. (2018) used the IPD as
the argument of a unitary complex number that was decomposed into real and
imaginary parts. These parts were concatenated to the ILD for several
frequency bins and several time frames, leading to a 2D tensor that was then
fed into a CNN. Pang et al. (2019) also used a CNN to process ILD and IPD
features in the TF domain, but the ILD and IPD 2D-tensors were directly
concatenated at the input of the CNN. A system relying only on the IPD was
proposed by Pak and Shin (2019). An MLP was trained to output a clean version
of the noisy input IPD in order to better retrieve the DoA using a
conventional method. Sivasankaran et al. (2018) used as input features the
concatenation of the cosine and sine of the IPDs for several frequency bins
and time frames. This choice was based on a previous work that showed similar
performance for this type of input feature compared to classical phase maps,
but with a lower dimension. In an original way, Thuillier et al. (2018)
employed unusual binaural features. They used the ipsilateral and
contralateral spectra. These features were shown to be relevant for elevation
estimation using a CNN. We finally found other DNN-based systems that used
ILD, e.g., (Roden et al., 2015; Zermini et al., 2016), ITD, e.g., (Roden et
al., 2015), or IPD, e.g., (Shimada et al., 2020b, a; Zermini et al., 2016;
Subramanian et al., 2021b) in addition to other types of features.

###  5.2 Cross-correlation (CC)-based features

Another manner for extracting and exploiting inter-channel information that
depends on source location is to use features based on the cross-correlation
(CC) between the signals of different channels. In particular, as seen in
Section 3, a variant of CC known as GCC-PHAT is a common feature used in
classical localization methods (Knapp and Carter, 1976). It is less sensitive
to speech signal variations than standard CC, but it may be adversely affected
by noise and reverberation (Blandin et al., 2012). Therefore, it has been used
within the framework of neural networks, which was revealed to be robust to
this type of disturbance/artefact. In several systems, GCC-PHAT has been
computed for each microphone pair and several time delays, all concatenated to
form a 1D vector used as the input of an MLP, e.g., (Xiao et al., 2015;
Vesperini et al., 2016; He et al., 2018a). Other architectures include
convolutional layers to extract useful information from multi-frame GCC-PHAT
features, e.g., (He et al., 2018a; Vecchiotti et al., 2018, 2019b; Noh et al.,
2019; Lu, 2019; Maruri et al., 2019; Pratik et al., 2019; Song, 2020; Li et
al., 2018; Comanducci et al., 2020a).

Some SSL systems rely on the CPS, which we already mentioned in Section 3 and
which is linked to the CC by a Fourier transform operation (in practice,
short-term estimates of the CPS are obtained by multiplying the STFT of one
channel with the conjugate STFT of the other channel). Leung and Ren (2019)
and Xue et al. (2020) sent the CPS into a CRNN architecture to improve
localization performance over the baseline of Adavanne et al. (2019a) (see
Section 4). Grondin et al. (2019) also used the cross-spectrum for each
microphone pair in the convolutional block of their architecture, whereas GCC-
PHAT features were concatenated in a deeper layer. The CPS was also used by Ma
and Liu (2018) as an input feature. Acoustic imaging has traditionally shown
some interest in the CPS feature to predict localization and sound pressure
level of competing sources; coupled with different architectures, from the
simple MLP (Castellini et al., 2021) to the complex CNN DenseNet network (Xu
et al., 2021a), authors have shown that the use of DNN could outperform
traditional deconvolution methods, either in performance or computation time.

Traditional localization methods, such as MUSIC (Schmidt, 1986) or ESPRIT (Roy
and Kailath, 1989), have been widely examined in the literature (see Section
3). These methods are based on the eigen-decomposition of the CC matrix of a
multichannel recording. Several DNN-based SSL systems (Takeda and Komatani,
2016b, a, 2017; Takeda et al., 2018) have been inspired by these methods and
reuse such features as input for their neural networks. Nguyen et al. (2020a)
computed the spatial pseudo-spectrum based on the MUSIC algorithm and then
used it as input features for a CNN.

Power map methods, which were discussed in Section 3, have also been used to
derive input features for DNN-based SSL systems. Salvati et al. (2018)
proposed calculating the narrowband normalized steered response power for a
set of candidate TDoAs corresponding to an angular grid and feeding it into a
convolutional layer. This led to a localization performance improvement
compared to the traditional SRP-PHAT method. Such power maps were also used by
Diaz-Guerra et al. (2021a) as inputs of 3D convolutional layers. In acoustic
imaging, a SRP map is also a standard feature where finding the position and
the acoustic level is the main goal. Some recent works used a CNN (Gonçalves
Pinto et al., 2021) or a U-Net (Lee et al., 2021b) to produce clean
deconvolved maps, hence going beyond the intrisic resolution of the array.

###  5.3 Spectrogram-based features

Alternatively to inter-channel features or CC-based features which already
encode relative information between channels, another approach is to provide
an SSL system directly with “raw” multichannel information, _i.e._ , without
any pre-processing in the channel dimension.

This does not prevent some pre-processing in the other dimensions and, from an
historical perspective, we notice that many models in this line use spectral
or spectro-temporal features instead of raw waveforms (see next subsection) as
inputs. In practice, (multichannel) STFT spectrograms are typically used
(Vincent et al., 2018). These multichannel spectrograms are generally
organized as 3D tensors, with one dimension for time (or frames), one for
frequency (bins), and one for channel. The general spirit of DNN-based SSL
methods is that the network should be able to “see” by itself and
automatically extract and exploit the differences between TF spectrograms
along the channel dimension while exploiting the “sparsity” of TF signal
representation.

In several works, the individual spectral vectors from the different STFT
frames were provided independently to the neural model, meaning that the
network did not take into account their temporal correlation (and a
localization result is generally obtained independently for each frame). Thus,
in that case, the network input is a matrix of size $M\times K$, with $M$
being the number of microphones, and $K$ being the number of considered STFT
frequency bins. Hirvonen (2015) concatenated the log-spectra of eight channels
for each individual analysis frame and sent it into a CNN as a 2D matrix.
Chakrabarty and Habets (2019b, 2017a, 2017b, a) and Mack et al. (2020) used
the multichannel phase spectrogram as input features, disregarding the
magnitude information. This choice is motivated by the fact that it allows to
easily generate a training dataset from white noise signals. As an extension
of this work, phase maps were also exploited by Bohlender et al. (2021).

When several consecutive frames are considered, the STFT coefficients for
multiple timesteps and multiple frequency bins form a 2D matrix for each
recording channel. Usually, these spectrograms are stacked together in a third
dimension to form the 3D input tensor. Several systems considered only the
magnitude spectrograms, e.g., (Yalta et al., 2017; Wang et al., 2019; Patel et
al., 2020; Pertilä and Cakir, 2017), while others considered only the phase
spectrogram, e.g., (Zhang et al., 2019b; Subramanian et al., 2021b). When
considering both magnitude and phase, they can also be stacked in a third
dimension (as well as channels). This representation has been employed in many
DNN-based SSL systems, e.g., (He et al., 2021a; Guirguis et al., 2020; Krause
et al., 2020a; Lin and Wang, 2019; Maruri et al., 2019; Zhang et al., 2019a;
Kapka and Lewandowski, 2019; Krause and Kowalczyk, 2019; Schymura et al.,
2021).

Yang et al. (2021a) dedicated different input branches of their CRNN to
magnitude and phase features. Other authors have proposed to decompose the
complex-valued spectrograms into real and imaginary parts, e.g., (Hao et al.,
2020; He et al., 2018b; Le Moing et al., 2020; Küçük et al., 2019). Finally,
Leung and Ren (2019) tried several combinations of features computed from the
complex multi-channel spectrogram, including the magnitude and phase, the real
and imaginary parts and the CPS. They claim that providing this redundant
information could help the neural network for better localization.

While basic (STFT) spectrograms consider equally-spaced frequency bins, mel-
scale spectrograms and Bark-scale spectrograms are represented with a non-
linear sub-bands division, corresponding to a perceptual scale (low-frequency
sub-bands have a higher resolution than high-frequency sub-bands) (Peeters,
2004). Mel-spectrograms were preferred to STFT spectrograms in several SSL
neural networks, e.g., (Vecchiotti et al., 2018; Kong et al., 2019; Cao et
al., 2019a, b; Ranjan et al., 2019). The Bark scale was also explored for
spectrograms in the SSL system of Pratik et al. (2019).

###  5.4 Ambisonic signal representation

In the SSL literature, numerous systems utilize the Ambisonics format, _i.e._
, the spherical harmonics (SH) decomposition coefficients (Jarrett et al.,
2017), to represent the input signal. Ambisonics is a multichannel format that
is increasingly used due to its capability to represent the spatial properties
of a sound field, while being agnostic to the microphone array configuration
(Zotter and Frank, 2019).

The SH decomposition is done for the acoustic pressure measured on the surface
of a sphere $\mathbb{S}^{2}$, concentric with the microphone array. For a
fixed sound source in far field, the decomposition coefficient of order $\ell$
and degree $m\in[-\ell,\ell]$, in the STFT domain, is given as follows Jarrett
et al. (2017):

$B_{\ell,m}(f,n)=\int_{\Omega\in\mathbb{S}^{2}}X(f,n,\Omega)Y^{*}_{\ell,m}(\Omega)d\Omega,$
(13)

where $X(f,n,\Omega)$ and $Y_{\ell,m}(\Omega)$ are the acoustic pressure and
the SH function, at the direction $\Omega$, respectively. In practice, this
integral is approximated by a quadrature rule, since the number of microphones
consisting an array is finite. Such approximation implies that the pressure
$X(f,n,\Omega)$ is assumed to be an (almost) “order-limited” function on the
sphere Rafaely (2019), meaning that $B_{\ell>L,m}(f,n)=0$, for some maximal
order $L$ (that depends on the number of microphones in the array). Hence, for
FOA ($L=1$), the Ambisonics representation (13) counts only $4$ coefficients
(channels) per TF bin. Alternatively, the Higher-Order Ambisonics (HOA),
$L>1$, signals have more than four channels.

The plane wave, bearing an amplitude $S(f,n)$, and coming from a direction
$\Omega$, admits a simple SH representation
$B_{\ell,m}(f,n)=S(f,n)Y_{\ell,m}(\Omega)$ Rafaely (2019). Therefore, as
opposed to other types of microphone arrays, the Ambisonic channels are in
phase, since the spatial response of each channel $Y_{\ell,m}(\Omega)$ is TF-
independent.222In practice, the spatial response of Ambisonic microphones is
approximately frequency-independent only within certain bandwidth (dictated by
the HOA order), due to spatial aliasing in the high frequency range, and noise
amplification at lower frequencies Zotter and Frank (2019). Analogous to (3),
the multichannel Ambisonic spectrogram $\mathbf{B}(f,n)$, due to $J$ sources
and reverberation, is given by the multivariate expression:

$\mathbf{B}(f,n)=\sum\limits_{j=1}^{J}\sum\limits_{r=0}^{\infty}A_{jr}(f,n)S_{j}(f,n)\mathbf{Y}(\Omega_{jr})+\mathbf{N}(f,n),$
(14)

where $A_{jr}$ is the amplitude of the $r$th reflection of the source $S_{j}$
(with $r=0$ corresponding to the direct path), $\mathbf{Y}$ is the vector
whose entries are appropriate spherical harmonics $Y_{\ell,m}$ for all
considered Ambisonic orders, and $\mathbf{N}$ is the additive noise vector.
Note that the complex-valued amplitudes $A_{jr}$ account for the attenuation
and phase shift of a corresponding plane wave component.

The FOA spectrograms, decomposed into magnitude and phase components, have
been used by, e.g., Adavanne et al. (2019c); Guirguis et al. (2020); Adavanne
et al. (2018, 2019a); Kapka and Lewandowski (2019) and Krause and Kowalczyk
(2019). Varanasi et al. (2020) and Poschadel et al. (2021a, b) used third-
order Ambisonics spectrograms. Poschadel et al. (2021a, b) compared the
performance of a CRNN with HOA spectrograms from order $1$ to $4$, showing
that the higher the order, the better the localization accuracy of the network
(but still below the performance of the so-called FOA pseudo-intensity
features, which we will discuss in Section 5.5). They used the phase and
magnitude for both elevation and azimuth estimation. Another way of
representing the Ambisonics format was proposed by Comminiello et al. (2019).
Based on the FOA spectrograms, they proposed considering them as quaternion-
based input features, which proved to be a suitable representation in previous
works (Parcollet et al., 2018). To cope with this type of input feature, a
neural network was adapted from the one of Adavanne et al. (2019a), showing an
improvement over the baseline.

###  5.5 Intensity-based features

Sound intensity is an acoustic quantity defined as the product of sound
pressure and particle velocity (Jacobsen and Juhl, 2013; Rossing, 2007). In
the frequency or TF domain, sound intensity is a complex vector whose real
part (known as “active” intensity) is proportional to the gradient of the
phase of sound pressure, _i.e._ , it is orthogonal to the wavefront. This is a
useful property that has been extensively used for SSL, e.g., (Nehorai and
Paldi, 1994; Hickling et al., 1993; Jarrett et al., 2010; Tervo, 2009; Evers
et al., 2014; Kitić and Guérin, 2018; Pavlidi et al., 2015). The imaginary
part (“reactive” intensity) is related to oscillatory local energy transfers,
and its physical interpretation is less obvious (Maysenhölder, 1993). Hence,
it has been largely ignored by the SSL community, even though it is relevant
in room acoustics Nolan et al. (2019). While the pressure is directly
measurable by regular microphones, particle velocity requires specific
sensors, such as acoustic vector-sensors (Nehorai and Paldi, 1994; Jacobsen
and Juhl, 2013), e.g., the “Microflown” transducer de Bree (2003). Otherwise,
it has to be approximated using the acoustic pressure measurements. Under
certain conditions, particle velocity can be assumed to be proportional to the
spatial gradient of sound pressure (Rossing, 2007; Merimaa, 2006), which
allows for the estimation by, e.g., the finite difference method (Tervo, 2009)
or using the FOA channels discussed in the previous section (Zotter and Frank,
2019). The latter approximation is often called (FOA) complex _pseudo-
intensity_ vector Jarrett et al. (2010):

$\mathbf{I}(f,n)=B_{0,0}(f,n)\mathbf{B}_{\ell=1,m}(f,n)^{*},$ (15)

where $\mathbf{B}_{\ell=1,m}^{*}(f,n)$ is the vector of first-order SH
coefficients, excluding the zero-order $B_{0,0}(f,n)$. In free field
conditions, assuming the presence of a single source at the TF bin $(f,n)$,
the entries of $\Re\left(\mathbf{I}(f,n)\right)$ are the Cartesian coordinates
of a vector colinear with the DoA of the source (with $\Re$ denoting the real
part of a complex value).

The first use of an Ambisonics pseudo-intensity vector for DL-based SSL was
reported by Perotin et al. (2018b), showing superiority in performance
compared to the use of the raw Ambisonics waveforms and traditional
Ambisonics-based methods. Interestingly, the authors demonstrated that using
both active and reactive intensity improves SSL performance. Moreover, they
normalized the intensity vector of each frequency band by its energy, which
can be shown to yield features similar to RTFs in the spherical harmonics
domain (Daniel and Kitić, 2020; Jarrett et al., 2017). Yasuda et al. (2020)
proposed using two CRNNs to refine the input FOA pseudo-intensity vector. The
first CRNN is trained to estimate denoising and separation masks under the
assumption that there are two active sources and that the WDO hypothesis
holds. The second CRNN estimates another mask to remove the remaining unwanted
components (e.g., reverberation). The two networks, hence, produce an estimate
of the “clean” intensity vector for each active source (the NoS is estimated
by their system as well). The pseudo-intensity vector has consequently been
used in several other recent works, e.g., (Grumiaux et al., 2021b; Perotin et
al., 2019b; Grumiaux et al., 2021a; Nguyen et al., 2021a; Cao et al., 2019a;
Park et al., 2020; Song, 2020; Cao et al., 2021; Tang et al., 2019; Perotin et
al., 2019a).

Sound intensity was also explored by Liu et al. (2021) without the Ambisonics
representation. The authors computed the instantaneous complex sound intensity
using an average of the sound pressure across the four considered channels and
two orthogonal particle velocity components using the differences in sound
pressure for both microphone pairs. They kept only the real part of the
estimated sound intensity (active intensity) and applied a PHAT weighting to
improve the robustness against reverberation.

###  5.6 Waveforms

Since 2018, several authors have proposed directly providing their neural
network models with the raw multichannel recorded signal waveforms. This idea
relies on the DNN’s capability to find the best representation for SSL without
the need of hand-crafted features or pre-processing of any kind. This is in
line with the general trend of DL to go toward an end-to-end approach that is
observed in many other applications, including in speech/audio processing. Of
course, this goes together with the always increasing size of networks,
datasets and computational power.

To our knowledge, Suvorov et al. (2018) were the first to apply this idea.
They trained their neural network directly with the recorded eight-channel
waveforms, stacking many 1D convolutional layers to extract high-level
features for the final DoA classification. Vera-Diaz et al. (2018), Vecchiotti
et al. (2019a), Chytas and Potamianos (2019), Cao et al. (2020), and Pujol et
al. (2019, 2021) sent the raw multichannel waveforms into 2D convolutional
layers. Huang and Perez (2021) sent the raw multichannel waveforms (in
microphone format and FOA format) into a 1D CNN with residual connections and
squeeze-excitation blocks. Note that this model is used for SELD and the
authors motivate the use of raw waveform inputs by the fact that “SED and DOA
may have some common features that are better preserved in the raw audio
[wave]form.” Huang et al. (2020) sent the multichannel waveforms into an AE.
Jenrungrot et al. (2020) shifted the waveforms of each channel to make them
temporally aligned according to the TDoA before being injected into the input
layer of their network. In the same vein, Huang et al. (2018, 2019) proposed
time-shifting the multichannel signal by calculating the time delay between
the microphone position and the candidate source location, which requires
scanning for all candidate locations.

A potential disadvantage of waveform-based features is that the architectures
exploiting such data are often more complex, as one part of the network needs
to be dedicated to feature extraction. Moreover, some papers have reported
that learning the “optimal” feature representations from raw data becomes more
difficult when noise is present in the input signals (Wichern et al., 2019) or
may even harm generalization, in some cases (Sato et al., 2021). However, it
is interesting to mention that the visual inspection of the learned weights of
the input layers of some end-to-end (waveform-based) neural networks has
revealed that they resemble the filterbanks that are usually applied in the
pre-processing stage of SSL (see Section 5.3) and other various classical
speech/audio processing tasks (Sainath et al., 2017; Luo and Mesgarani, 2019).

###  5.7 Other types of features

Varzandeh et al. (2020) have proposed unusual types of features that do not
belong to one of the categories described above. Particularly, they have used
a periodicity degree feature together with GCC-PHAT features in a CNN. The
periodicity degree is computed for a given frame and period. It is equal to
the ratio between the harmonic power signal for the given period and the total
power signal. This conveys information about the harmonic content of the
source signal to the CNN.

##  6 Output strategies

In this section, we discuss the different strategies proposed in the
literature to obtain a final DoA estimate. We generally divide the strategies
into two categories: classification and regression. When the SSL network is
designed for the classification task, the source location search space is
generally divided into several zones, corresponding to different classes, and
the neural network outputs a probability value for each class. As for
regression, the goal is to directly estimate (continuous) source
position/direction values, which are usually either Cartesian coordinates
$(x,y,z)$, or spherical coordinates $(\theta,\phi,r)$ (although the source-
microphone distance $r$ is very rarely considered). However, the latter is an
important factor as it can affect the estimation accuracy (for instance, due
to the influence of the direct-to-reverberant ratio (DRR) Vincent et al.
(2018)). Therefore, in order to obtain a robust model, the training dataset
needs to be sufficiently diverse such that the network is exposed to sources
at different directions, but also at different source-microphone distances. In
the last subsection, we report a few non-direct methods in which the neural
network does not estimate the location of a source in its output layer.
Instead, it either helps another (conventional) algorithm to finally retrieve
the desired DoA, or the location estimate is a byproduct of some intermediate
network layer. A reader particularly interested in the comparison between the
classification and regression approaches may consult the papers of Tang et al.
(2019) and Perotin et al. (2019a).

###  6.1 DoA estimation via classification

Many systems treat DoA estimation as a classification problem, _i.e._ , each
class represents a certain zone in the considered search space. In other
words, space is divided into several subregions, usually of similar size, and
the neural network is trained to produce a probability of active source
presence for each subregion. Such a classification problem is often addressed
by using a feedforward layer as the last layer in the network, with as many
neurons as the number of considered subregions. Two activation functions are
generally associated with the final layer neurons: the softmax and sigmoid
functions. Softmax ensures that the sum of all neuron outputs is $1$, so it is
suitable for a single-source localization scenario. With a sigmoid, all neuron
outputs are within $[0,1]$ independently from each other, which is suitable
for multi-source localization. The last layer output is often referred to as
the spatial (pseudo)-spectrum, whose peaks correspond to a high probability of
source activity in the corresponding zone.

As already mentioned in Section 2.3, the final DoA estimate(s) is/are
generally extracted using a peak picking algorithm: If the number of sources
$J$ is known, the selection of the $J$ highest peaks gives the multi-source
DoA estimation; if the NoS is unknown, usually the peaks above a certain user-
defined threshold are selected, leading to a joint NoS and localization
estimations. Some preprocessing, such as spatial spectrum smoothing or angular
distance constraints, can be used for better DoA estimation. Hence, such a
classification strategy can be readily used for single-source and/or multi-
source localization, as the neural network is trained to estimate a
probability of source activity in each zone, regardless of the NoS.

####  6.1.1 Spherical coordinates

Regarding the quantization of the source location space, namely the
localization grid, different approaches have been proposed. Most early works
focused on estimating only the source’s azimuth $\theta$ relative to the
microphone array position, dividing the $360$° azimuth space into $N_{\theta}$
regions of equal size, leading to a grid quantization step of
$\frac{360}{N_{\theta}}$. Without being exhaustive, we found in the literature
many different values for $N_{\theta}$, e.g., $N_{\theta}=7$ (Roden et al.,
2015), $N_{\theta}=8$ (Hirvonen, 2015), $N_{\theta}=20$ (Suvorov et al.,
2018), $N_{\theta}=37$ (Vecchiotti et al., 2019a), $N_{\theta}=72$ (Ma et al.,
2015), and $N_{\theta}=360$ (Xiao et al., 2015). Some other works did not
consider the whole $360$° azimuth space. For example, Chazan et al. (2019)
focused on the region $[0,180]$ with $N_{\theta}=13$.

Estimating the elevation $\phi$ alone has not been frequently investigated in
the literature, probably because of the lack of interesting applications in
indoor scenarios. To the best of our knowledge, only one paper focused on
estimating the elevation alone (Thuillier et al., 2018). The authors divided
the whole elevation range into nine regions of equal size. The majority of
recent SSL neural networks are trained to estimate both source azimuth and
elevation, whenever the microphone array geometry makes it possible. To do
this, several options have been proposed in the literature. One can use two
separate output layers, each with the same number of neurons as the number of
subregions in the corresponding dimension. For example, the output layer of
the neural architecture proposed by Fahim et al. (2020) is divided into two
branches with fully connected layers, one for azimuth estimation ($N_{\theta}$
neurons), and the other for elevation estimation ($N_{\phi}$ neurons). One can
also have a single output layer where each neuron corresponds to a zone in the
unit sphere, _i.e._ , a unique pair $(\theta,\phi)$, e.g., (Grumiaux et al.,
2021b; Perotin et al., 2019b). Finally, one can directly design two separate
neural networks, with each estimating the azimuth or the elevation angle,
e.g., (Varanasi et al., 2020).

However, most of the neural networks following the classification strategy for
joint azimuth and elevation estimation are designed so that the output
corresponds to a 2D grid on the unit sphere. For example, Perotin et al.
(2018b, 2019b) and Grumiaux et al. (2021a) used a quasi-uniform spherical grid
with $429$ classes, each represented by a unique neuron in the output layer of
their network. Adavanne et al. (2018) sampled the unit sphere in the whole
azimuth axis but in the limited elevation range of $[-60\degree,60\degree]$,