# Selective-Memory Meta-Learning
with Environment Representations for
Sound Event Localization and Detection

Jinbo Hu, , Yin Cao, , Ming Wu, ,
Qiuqiang Kong, , Feiran Yang, ,
Mark D. Plumbley, , Jun Yang  Jinbo Hu, Ming Wu, and Jun Yang are with the Key
Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese
Academy of Sciences, Beijing 100190, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>jyang@mail.ioa.ac.cn). Jinbo Hu and Jun Yang are also
with the University of Chinese Academy of Sciences, Beijing 100049, China.
(Corresponding author: Jun Yang) Yin Cao is with the Department of Intelligent
Science, Xi’an Jiaotong Liverpool University, Suzhou 215123, China (e-mail:
yin.k.cao@gmail.com). Qiuqiang Kong is with the Chinese University of Hong
Kong, Hong Kong, China (e-mail: qqkong@ee.cuhk.edu.hk). Feiran Yang is with
the State Key Laboratory of Acoustics, Institute of Acoustics, Chinese Academy
of Sciences, Beijing 100190, China (feiran@mail.ioa.ac.cn). Mark D. Plumbley
is with the Centre for Vision, Speech and Signal Processing, University of
Surrey, Guildford GU2 7XH, U.K. (e-mail: m.plumbley@surrey.ac.uk). This work
was partly supported by the National Key Research and Development Project (NO.
2022YFB2602000), Grant ”XJTLU RDF-22-01-084”, and Engineering and Physical
Sciences Research Council (EPSRC) Grant EP/T019751/1. For the purpose of open
access, the authors have applied a Creative Commons Attribution (CC BY)
license to any Author Accepted Manuscript version arising.

###### Abstract

Environment shifts and conflicts present significant challenges for learning-
based sound event localization and detection (SELD) methods. SELD systems,
when trained in particular acoustic settings, often show restricted
generalization capabilities for diverse acoustic environments. Furthermore, it
is notably costly to obtain annotated samples for spatial sound events.
Deploying a SELD system in a new environment requires extensive time for re-
training and fine-tuning. To overcome these challenges, we propose
environment-adaptive Meta-SELD, designed for efficient adaptation to new
environments using minimal data. Our method specifically utilizes
computationally synthesized spatial data and employs Model-Agnostic Meta-
Learning (MAML) on a pre-trained, environment-independent model. The method
then utilizes fast adaptation to unseen real-world environments using limited
samples from the respective environments. Inspired by the Learning-to-Forget
approach, we introduce the concept of selective memory as a strategy for
resolving conflicts across environments. This approach involves selectively
memorizing target-environment-relevant information and adapting to the new
environments through the selective attenuation of model parameters. In
addition, we introduce environment representations to characterize different
acoustic settings, enhancing the adaptability of our attenuation approach to
various environments. We evaluate our proposed method on the development set
of the Sony-TAU Realistic Spatial Soundscapes 2023 (STARSS23) dataset and
computationally synthesized scenes. Experimental results demonstrate the
superior performance of the proposed method compared to conventional
supervised learning methods, particularly in localization.

###### Index Terms:

Sound event localization and detection, meta-learning, environment adaptation,
selective memory.

##  I Introduction

Sound event localization and detection (SELD) refers to detecting categories,
presence, and spatial locations of different sound sources. SELD was first
introduced in Task 3 of the Detection and Classification of Acoustics Scenes
and Events (DCASE) 2019 Challenge [1]. After three iterations of Task 3 of the
DCASE Challenge [1, 2, 3], the types of data have transformed from
computationally synthesized spatial recordings to real-scene recordings in
2022 and 2023 [4, 5]. Large-scale datasets of spatialized sound events were
released for these challenges to be used for training and evaluating learning-
based approaches.

###  I-A Learning-based SELD methods

SELD can be regarded as a multi-task learning problem. Adavanne et al.[6]
proposed SELDnet for a joint task of sound event detection (SED) and
regression-based direction-of-arrival (DOA) estimation. SELDnet cannot detect
homogeneous overlap, which refers to overlapping sound events of the same type
but from different locations. The Event-Independent Network V2 (EINV2), with a
track-wise output format and permutation invariant training, was proposed to
tackle the homogeneous overlap detection problem [7, 8, 9]. In contrast to the
use of two outputs of SED and DOA in SELDnet and EINV2, the Activity-coupled
Cartesian DOA (ACCDOA) approach merges the two subtasks into a single task[10,
11], where the Cartesian DOA vectors also contain the activity information of
sound events.

However, the performance of learning-based methods is usually degraded when
the training set and test set are mismatched. The training set cannot cover
all actual instances from different acoustic environments.

###  I-B Environment shifts and conflicts

The change in the data distribution between a training set and a test set is
known as the domain shift problem [12, 13]. In the practical deployment of
SELD systems, the differences among environments could potentially be very
significant. Unseen complex acoustic environments could result in a decline in
system performance, due to the distribution change in acoustic properties,
such as varying degrees of echo and reverberation, diverse types of ambient
noise, and directional interference. The change in the distribution of
acoustic properties among environments is referred to as the environment
shift. The issue is particularly salient when the system encounters acoustic
properties it has not been exposed to during training. On the other hand,
different acoustic environments may present conflicts. Optimal solutions for
diverse acoustic environments may display considerable variation, e.g. indoor
environments and outdoor environments. A single training configuration cannot
encompass all types of environments.

Fig. 1 illustrates the results of our previous system [14] submitted to Task 3
of the DCASE 2022 Challenge. STARSS22 [4] is a dataset of spatial recordings
of real scenes with spatiotemporal annotations of sound events. There are no
duplicated recording environments between the training and test sets in
STARSS22. The system was evaluated on the STARSS22 dataset and obtained second
in the team ranking. However, we found unsatisfactory generalization
performance for the Room 2 recordings in the dev-test-tau set of STARSS22 [14]
compared with other rooms. Experimental results show that class-dependent
localization error $\mathrm{LE}_{\mathrm{CD}}$ is much higher, and location-
dependent F-score $\mathrm{F}_{\leq 20^{\circ}}$ is much lower in Room 2, but
class-dependent localization recall $\mathrm{LR}_{\mathrm{CD}}$ is high. This
suggests that our system may have weak localizing performance in Room 2 due to
the environment shift or conflict.

Figure 1: Room-wise metric scores of our previous system [14] submitted to
Task 3 of the DCASE 2022 Challenge on the STARSS22 validation set. The
description of each metric is expounded in Section IV-B. The system obtained
the second rank in the team ranking.

###  I-C Data acquisition

One of the most effective methods for solving environment shifts and conflicts
is acquiring as much data as possible [12, 13, 15]. However, manually
collecting and annotating spatial sound event recordings is highly cost-
intensive. For instance, the STARSS22 dataset [4] employed sophisticated
setups involving a 32-channel spherical microphone array for recording,
wireless microphones for manually annotating types, onset, and offset of sound
events, a motion capture (mocap) system for extracting the tracked data, and a
$360^{\circ}$ camera for validating those annotations.

Another practical approach for acquiring data is computationally synthesizing
spatial sound event samples. Spatial sound event recordings are simulated by
convolving dry sound event samples with spatial room impulse responses
(SRIRs). The multi-channel simulation (MCS) framework [16] and the impulse
response simulation (IRS) framework [17] are both designed to simulate multi-
channel recordings, emphasizing augmentation from original data without the
reliance on external datasets. MCS and IRS involve the convolution of enhanced
source signals with extracted covariance matrices and computationally
simulated SRIRs, respectively. Noteworthy among existing external datasets
used for synthesizing spatial sound events include FSD50K [18], AudioSet [19],
and TAU-SRIR DB [20]. FSD50K and AudioSet are large-scale sound event
datasets, while TAU-SRIR DB is a real-recorded SRIR database tailored for the
DCASE Challenge. Given the insufficiency of publicly accessible SELD data,
particularly for specific microphone array types, e.g. the number of
microphones and geometry of the microphone array, the role of the SRIR
simulation technique becomes indispensable in data synthesis. Numerous RIR
simulation methodologies rely on geometric approaches [21, 22], where the
propagation of sound waves is modeled and manipulated in the form of a ray.
These techniques simulate the reflections and reverberation of sound waves.
Software packages like Pyroomacoustics [23], gpuRIR [24], and SMIR-Generator
[25] are representative tools for geometric-based SRIR simulations.

###  I-D Meta learning

Models trained on synthetic datasets typically demonstrate a degree of
generalization ability but could limit the robustness of the network on real-
world data due to the environment shift [15]. One line of research is to train
the model with realistic signals by transfer learning [26], pre-training a
model with large-scale synthetic datasets, and then fine-tuning the model with
limited real-recorded datasets [15].

In circumstances when environmental disparities occur between training sets
and test sets and collected sample availability is limited, few-shot learning
(FSL) [27] can offer a solution for adaptation to realistic and specific
environments.

The conventional supervised learning method refers to training a model on a
labeled dataset and then applying the trained model to predict unseen data.
The core issue of conventional supervised learning training methodologies in
the case of limited samples is that the empirical risk minimizer is unreliable
[27]. Nevertheless, by incorporating prior knowledge, FSL can effectively
generalize to the specific task, even with only a few samples [27]. Meta-
learning, which facilitates FSL, learns a general-purpose learning algorithm
that generalizes across tasks and ideally enables each new task to be learned
well from the task-distribution view [28]. Meta-learning has advanced FSL
significantly in computer vision [29, 30].

One of the most successful meta-learning algorithms is model-agnostic meta-
learning (MAML) [31]. MAML formulates prior knowledge as commonly initialized
parameters across tasks and then exploits a few samples of the target task to
adapt that task quickly. Due to its model-agnostic nature, MAML is compatible
with any model trained with gradient descent, making it applicable to various
learning problems, including classification, regression, and reinforcement
learning. In audio signal processing, the meta-learning method has recently
attracted interest in solving FSL problems. Based on MAML, several audio-
related researchers have investigated building systems to adapt their specific
tasks rapidly with only a few corresponding samples [32, 33, 34]. To the best
of our knowledge, the few-shot environment adaptation problems based on meta-
learning have not been thoroughly studied.

MAML can be designed to employ a set of trained initial parameters and a few
samples from a specific environment to cope with the environment shift problem
and train multiple models for each specific environment to somewhat mitigate
the environment conflict issue. However, forcibly sharing the initial
parameters can still lead to some conflicts and compromises among tasks [35].
Multimodal MAML (MMAML) [36] focuses on task-dependent initial parameters and
tries to learn task embeddings and transform the initial parameters with
affine parameters. Compared with Multimodal MAML, Learning-to-Forget (L2F)
[35] proposes layer-wise attenuation on the compromised initial parameters for
each task to reduce its influence.

###  I-E Our contributions

Figure 2: A diagram of the meta-training procedure for our proposed
environment-adaptive Meta-SELD. For simplicity, we only consider one gradient
update for the inner loop of the training procedure. $f_{\Theta}$,
$g_{\Omega}$, and $h_{\Phi}$ represent the backbone, environment extractor,
and attenuation network, respectively. $\Theta^{l}$, where $l=1\dots p$,
denotes the $l$-th layer of the total $p$-layer backbone $f_{\Theta}$.

In this work, we extend our previous work Meta-SELD [37] to environment-
adaptive Meta-SELD, investigating an adaptation to the environment shift
problem using meta-learning-based few-shot methods. Drawing inspiration from
L2F [35], we propose to selectively memorize components relevant to the target
environment and to learn the target environment by using environment
representations. Fig. 2 presents a flow diagram of environment-adaptive Meta-
SELD. In contrast to [38], which tackles unseen categories of sound events
problem in SELD, our proposed Meta-SELD focuses on adaptation to unknown
environments.

Figure 3: An illustration of Meta-SELD++ with and without selective memory.
Selective memory is proposed to tackle environment conflicts. It adds an
additional step for attenuation of initial parameters before fast adaptation
to the task $i$. The selective memory method provides a better solution
(closer to the optimal solution).

The method of fast adaptation to environments is mainly based on Model-
Agnostic Meta-Learning (MAML) [31] and an environment-independent (EI) model.
The EI model is pre-trained on a computationally synthesized dataset,
encompassing a wide range of acoustic environments. We then apply MAML to the
pre-trained EI model to create a meta-EI model, which enables fast adaptation
to an unseen environment using a few samples recorded in the target
environment. In addition, forcibly sharing common initial parameters across
environments allows the environment conflict issue to remain. Inspired by
Learning-to-Forget (L2F) [35], we adopt an attenuation network and propose
environment representations to selectively memorize target-environment-
relevant components and selectively forget contradicting information. An
illustration of our proposed environment-adaptive Meta-SELD is depicted in
Fig. 3. We evaluated the proposed method on the development set of the Sony-
TAu Realistic Spatial Soundscapes 2023 (STARSS23) [5] dataset and on
computationally synthesized scenes. Experimental results demonstrated the
effectiveness of environment-adaptive Meta-SELD.

Our main contributions can be summarized as follows:

1) Pre-training an environment-independent (EI) model on computationally
synthesized datasets to contain as many acoustic properties as possible.

2) Investigating an environment adaptation approach based on Model-Agnostic
Meta-Learning (MAML) and the pre-trained EI model.

3) Introducing a solution to selectively memorize prior information relevant
to the target environment to mitigate environment conflicts.

4) Proposing a technique to extract environment representations for selective
memory and designing comprehensive experiments to display environment
representations.

##  II Fast adaptation to the environment

###  II-A Pre-trained environment-independent models

We train a simple convolutional recurrent neural network (CRNN) on the
datasets synthesized using computationally generated SRIRs. Source code about
data synthesis of spatial sound events is released111https://github.com/Jinbo-
Hu/SELD-Data-Generator. This model is not training on samples in specific
environments and is consequently termed the environment-independent (EI)
model.

####  II-A1 Data synthesis

The spatial sound event recordings are simulated by convolving monophonic
sound event samples with SRIRs. The sound event samples are selected from
FSD50K and AudioSet, based on the similarity of the labels in those datasets
to target classes in STARSS23. These selected clips are then cleaned by the
pre-trained CNN14 [39] model. We utilize CNN14 to infer these clips and select
high-quality clips based on the output probability. The SRIRs are
computationally generated using geometric-based methods [21, 22]. The
computational generation method for SRIRs consists of two steps: microphone-
array RIRs simulation and Ambisonics format converter.

We use the image source method [21] to generate microphone-array RIRs with
four channels. This method replaces reflection on walls with virtual sources
playing the same sound as the source and builds an RIR from the corresponding
delays and attenuations. We use the python package pyroomacoustics [23] to
simulate room acoustics. As the microphones are mounted on an acoustically
hard spherical baffle in the official setup of STARSS23, the frequency
response of the $h$-th microphone with a wave number of $k$ on a rigid baffle
of radius $R$ for $l$-th image source is [40]:

$H_{hl}\left(k,\psi_{hl}\right)=\sum_{n=0}^{\infty}\mathrm{i}^{n}(2n+1)b_{n}(kR)P_{n}\left(\cos\psi_{hl}\right),$
(1)

where $\psi_{hl}$ denotes the angle between the DoA of the $l$-th sound source
and the orientation of the $h$-th microphone, $P_{n}$ denotes the Legendre
polynomial[41, 40, 42], $\mathrm{i}$ is the imaginary unit, and $b_{n}$ is the
mode strength term for a rigid baffle array given by [41]

$b_{n}(kR)=\frac{\mathrm{i}}{(kR)^{2}h_{n}^{(1)^{\prime}}(kR)},$ (2)

with $h_{n}^{(1)^{\prime}}$ denoting the derivate of the $n$-th-order
spherical Hankel function of the first kind[41].

The Ambisonics format conversion transforms the above-mentioned microphone-
array format signals to first-order Ambisonics (FOA) format signals. The
$n$-th-order and $m$-th-degree spherical harmonic function is defined with the
angle $\Psi=\\{\theta,\phi\\}$ as follows [41]:

$Y_{n}^{m}(\Psi)\equiv\sqrt{\frac{2n+1}{4\pi}\frac{(n-m)!}{(n+m)!}}P_{n}^{m}(\cos\theta)e^{im\phi},$
(3)

where $(\cdot)!$ represents the factorial operator, $\theta$ and $\phi$ are
elevation and azimuth, and $P_{n}^{m}(\cdot)$ is the associated Legendre
function. The spherical harmonic representation of the RIRs can be computed by
using the following encoding process[41, 17]:

$\mathbf{a}(k)=\mathbf{B}(k)^{-1}\mathbf{Y}^{\dagger}\mathbf{x}(k),$ (4)

where

$\mathbf{B}(k)=\left(\begin{array}[]{cccc}b_{0}&0&0&0\\\ 0&b_{1}&0&0\\\
0&0&b_{1}&0\\\ 0&0&0&b_{1}\\\ \end{array}\right),$ (5)

$\mathbf{x}(k)$ denotes simulated microphone-array RIR signals,
$(\cdot)^{\dagger}$ represents the Moore-Penrose pseudo inverse, and
$\mathbf{a}(k)$ denotes the resulting FOA format signal. $\mathbf{Y}$ denotes
first-order spherical harmony matrices with a four-channel microphone array as
follows:

$\mathbf{Y}=\left(\begin{array}[]{cccc}Y_{0}^{0}(\Psi_{1})&Y_{1}^{-1}(\Psi_{1})&Y_{1}^{0}(\Psi_{1})&Y_{1}^{1}(\Psi_{1})\\\
Y_{0}^{0}(\Psi_{2})&Y_{1}^{-1}(\Psi_{2})&Y_{1}^{0}(\Psi_{2})&Y_{1}^{1}(\Psi_{2})\\\
Y_{0}^{0}(\Psi_{3})&Y_{1}^{-1}(\Psi_{3})&Y_{1}^{0}(\Psi_{3})&Y_{1}^{1}(\Psi_{3})\\\
Y_{0}^{0}(\Psi_{4})&Y_{1}^{-1}(\Psi_{4})&Y_{1}^{0}(\Psi_{4})&Y_{1}^{1}(\Psi_{4})\\\
\end{array}\right),$ (6)

with $\Psi_{i}=\\{\theta_{i},\phi_{i}\\}$ being the direction of the $i$-th
microphone.

####  II-A2 Network architecture

Without loss of generality, in this study, we adopt a simple CRNN as our
backbone for our following experiments. The CRNN is similar to the baseline of
Task 3 of the DCASE 2022 Challenge[4] but with an ACCDOA representation [10].
As shown in Fig. 4, the backbone has four convolution blocks followed by a
one-layer bidirectional gated recurrent unit (BiGRU). The network takes
$C$-channel $T$-frame $F$-mel-bin spectrograms, the concatenation of log-mel
spectrograms and intensity vectors as input, and predicts active sound events
of $M$ classes with corresponding Cartesian DOA vectors for each time stamp.

Figure 4: The network architecture of the SELD network with the sub-network of
environment representation extraction. The environment representations are
extracted from output feature maps of each layer of the backbone.

###  II-B Meta-SELD

Given a model represented by a parameterized function $f_{\Theta}$ with
parameters $\Theta$, MAML [31] learns the initial parameters $\Theta_{0}$ from
general tasks $\mathcal{T}_{i}$ sampled from the meta-training set
$\mathcal{D}_{\mathtt{train}}$. The initial parameters $\Theta_{0}$ are
sensitive to task-specific fine-tuning [31] and expected to perform well on
unseen tasks from the meta-test set $\mathcal{D}_{\mathtt{test}}$ after a few
parameter updates with a few task-specific samples. The task in meta-learning
refers to a specific learning problem. Each task $\mathcal{T}_{i}$ consists of
a support set $\mathcal{S}_{i}$ of $K$ samples and a query set
$\mathcal{Q}_{i}$ of $Q$ samples, analogy to the training set and test set,
respectively. A new task is expected to be quickly learned with $K$ samples,
known as $K$-shot learning. The loss function of MAML is

$\mathcal{L}=\sum_{\mathcal{T}_{i}\sim
p(\mathcal{T})}\mathcal{L}_{\mathcal{T}_{i}}(f_{\Theta}),$ (7)

where $p(\mathcal{T})$, sampled from $\mathcal{D}_{\mathtt{train}}$, is a
distribution over tasks that we want our model to adapt to, and
$\mathcal{L}_{\mathcal{T}_{i}}$ indicates the task-specific loss. In contrast
to conventional supervised learning methods, the objective of which is to find
optimal parameters to minimize the loss function across all training samples,
MAML tries to find common generalized initial parameters across tasks and then
updates the initial parameters after several iterations of training on limited
data of new tasks.

There are two groups of parameters in the MAML algorithm: meta parameters and
adaptation parameters. In the meta-training phase, MAML starts with randomly
initialized meta parameters $\Theta$ and then adapts to a new specific task
$\mathcal{T}_{i}$ with several update iterations using the support set
$\mathcal{S}_{i}$. The meta parameters $\Theta$ become adaptation parameters
$\Theta_{i}^{\prime}$:

${\Theta}_{i}^{\prime}=\Theta-\alpha\nabla_{\Theta}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\Theta},\mathcal{S}_{i}\right),$
(8)

where $\alpha$ is the adaptation learning rate. For simplicity of notation, we
display only one iteration of the gradient update. After computing task-
specific loss on $\mathcal{Q}_{i}$ with $f_{{\Theta}_{i}^{\prime}}$ across a
batch of tasks, the meta parameters are updated as follows:

$\Theta\leftarrow\Theta-\beta\nabla_{\Theta}\sum_{\mathcal{T}_{i}}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{{\Theta}_{i}^{\prime}},\mathcal{Q}_{i}\right),$
(9)

where $\beta$ is the meta step size. After accumulating
$\mathcal{L}_{\mathcal{T}_{i}}$ for several tasks, $\Theta$ is updated by
gradient descent and will be used as the initial parameters for the subsequent
loops of meta-training steps.

Algorithm 1 Meta-training of proposed environment-adaptive Meta-SELD

0: Distribution over all environments $p(\mathcal{T})$, adaptation step size
$\alpha$, meta step size $\beta$

0: Parameters $\Theta=\\{\Theta^{l}\\}^{l=1\dots p}$ from pre-trained EI
model, where $l$ is the layer index and $p$ is the number of layers of the
backbone

1: Initialize meta parameters of backbone $f_{\Theta}$ with the pre-trained
parameters $\Theta$. Randomly initialize environment extractor $g_{\Omega}$
and attenuate network $h_{\Phi}$.

2: while not done do

3: Sample a batch of environments $\mathcal{T}_{i}\sim p(\mathcal{T})$

4: for each environment $\mathcal{T}_{i}$ do

5: Sample disjoint examples $(\mathcal{S}_{i},\mathcal{Q}_{i})$ from
$\mathcal{T}_{i}$

6: Compute representations of the $i$-th environment:
$\bm{e_{i}}=g_{\Omega}(f_{\Theta},\mathcal{S}_{i})$

7: Compute attenuation coefficients
$\bm{\lambda_{i}}=\\{\lambda_{i}^{l}\\}^{l=1\dots p}$ for each layer:
$\bm{\lambda_{i}}=h_{\Phi}(\bm{e_{i}})$

8: Compute attenuated initial parameters:
$\Theta_{i}^{l}=\lambda_{i}^{l}\cdot\Theta^{l}$

9: Let $\Theta_{i}^{\prime}=\\{\Theta_{i}^{l}\\}^{l=1\dots p}$

10: for gradient descent step $j:=0$ to $N-1$ do

11: Perform gradient descent to update adaptation parameters:
$\Theta_{i}^{\prime}\leftarrow\Theta_{i}^{\prime}-\alpha\nabla_{\Theta_{i}^{\prime}}\mathcal{L}_{\mathcal{T}_{i}}\left(\Theta_{i}^{\prime},\mathcal{S}_{i}\right)$

12: end for

13: Compute
$\mathcal{L}_{\mathcal{T}_{i}}(f_{\Theta_{i}^{\prime}},\mathcal{Q}_{i})$

14: end for

15: Perform gradient descent to update meta parameters:
$\Theta\leftarrow\Theta-\beta\nabla_{\Theta}\sum_{\mathcal{T}_{i}}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\Theta_{i}^{\prime}},\mathcal{Q}_{i}\right)$
$\Omega\leftarrow\Omega-\beta\nabla_{\Omega}\sum_{\mathcal{T}_{i}}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\Theta_{i}^{\prime}},\mathcal{Q}_{i}\right)$
$\Phi\leftarrow\Phi-\beta\nabla_{\Phi}\sum_{\mathcal{T}_{i}}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\Theta_{i}^{\prime}},\mathcal{Q}_{i}\right)$

16: end while

In the meta-testing phase, a specific unseen task
$\mathcal{T}_{j}^{\mathtt{test}}$ created using the meta-test set
$\mathcal{D}_{\mathtt{test}}$ is used. $\mathcal{T}_{j}^{\mathtt{test}}$
consists of a labeled support set $\mathcal{S}_{j}^{\mathtt{test}}$ of $K$
samples, and an unlabeled query set $\mathcal{Q}_{j}^{\mathtt{test}}$ of $Q$
samples. We update the model, initialized by well-trained parameter $\Theta$
in the meta-training phase, on $\mathcal{S}_{j}^{\mathtt{test}}$ to get
adaptation parameters ${\Theta_{j}}^{\prime}$ using the update procedure of
Eq. 8. The adaptation performance is evaluated on
$\mathcal{Q}_{j}^{\mathtt{test}}$ with $f_{\Theta_{j}^{\prime}}$.

We aim to adapt to an unseen environment with $K$ samples ($K$-shot learning).
The objective of MAML is to find optimal initial parameters across several
tasks, so we need to construct a set of tasks from the meta-training set
$\mathcal{D}_{\mathtt{train}}$. $\mathcal{D}_{\mathtt{train}}$ is split into
several tasks according to the different recording environments. Audio clips
recorded in different environments belong to different tasks. We first sample
a batch of environments and then sample $K+Q$ clips in each environment, where
$K$ clips from the support set $\mathcal{S}_{i}$ and $Q$ clips from the query
set $\mathcal{Q}_{i}$. The overall training procedure of MAML is summarized in
Algorithm 1. Step 11 in Algorithm 1 is the inner-loop update for adaptation
parameters, while Step 15 is the outer-loop update for meta parameters.

The meta-learning processes of SELD for testing and training are slightly
different in the data division. Fig. 5 shows the division of the meta-training
and the meta-test sets. Similar to the training, the meta-test set
$\mathcal{D}_{\mathtt{test}}$ is partitioned based on the recording
environments of each audio clip. For clips of each environment, we also chose
$K$ clips for meta-test support set $\mathcal{S}_{j}^{\mathtt{test}}$, and all
remaining clips for meta-test query set $\mathcal{Q}_{j}^{\mathtt{test}}$.
After $N$ iterations of update on $\mathcal{S}_{j}^{\mathtt{test}}$, the meta
parameters $\Theta$ are updated to ${\Theta_{j}^{\prime}}$. The final
performance is evaluated on $\mathcal{Q}_{j}^{\mathtt{test}}$ with
$f_{\Theta_{j}^{\prime}}$.

Figure 5: The data division for the meta-training and the meta-test sets
according to recording environments.

###  II-C Meta-SELD++

Instead of randomly initializing the meta parameters, we leverage the power of
the pre-trained environment-independent (EI) model to initialize the meta
parameters of MAML. In other words, we may utilize a well-trained EI model to
find better initial parameters and accelerate convergence for training the
meta-EI model.

Specifically, during meta-training, we initialize the meta parameters with
parameters of the pre-trained EI model and then create a meta-EI model trained
on synthetic datasets using collected SRIRs based on MAML. We denote this
method as Meta-SELD++, whose meta parameters are initialized with pre-trained
parameters. The training procedure will be expounded in Section IV-A.

Figure 6: A detailed implementation of the environment extractor.

##  III Environment-Adaptive Meta-SELD

###  III-A Selective memory

According to Eq. 7, MAML gives each task equal weight and tries to find
optimal initial parameters across tasks in an average sense so that it may
perform better or worse on a specific task than the conventional supervised
learning method. There may be conflicts when optimizing across a batch of
environments [37]. Inspired by L2F [35], which argues that forcibly sharing a
common initialization in MAML induces conflicts and thus leads to the
compromised location of the initialization, we selectively memorize parts
relevant to the target environment. More comprehensively, we employ an
environment-dependent layer-wise attenuation on the initialization, thereby
dynamically controlling the influence of prior knowledge for each environment.
The attenuation is generated by an attenuation network $h_{\Phi}$. Fig. 3
illustrates the optimization procedure of selective memory Meta-SELD. Compared
with Meta-SELD++, selective memory adds a step to attenuate the common initial
parameters for target environments before fast adaptation to corresponding
environments.

One general information input to $h_{\Phi}$ is gradients. At the beginning of
the inner loop of MAML, task-specific gradients
$\nabla_{\Theta}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\Theta},\mathcal{S}_{i}\right)$
on the support sets $\mathcal{S}_{i}$ of the $i$-th environment are computed
to generate layer-wise attenuation coefficients:

$\bm{\lambda_{i}}=h_{\Phi}\left(\nabla_{\Theta}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\Theta},\mathcal{S}_{i}\right)\right),$
(10)

where $h_{\Phi}$ is a 3-layer MLP network with parameters $\Phi$, with a
sigmoid in the final layer to facilitate attenuation. The layer-wise
attenuation coefficients $\bm{\lambda_{i}}=\\{\lambda_{i}^{l}\\}^{l=1\dots p}$
act on each layer of $\Theta=\\{\Theta^{l}\\}^{l=1\dots p}$:

$\Theta_{i}^{l}=\lambda_{i}^{l}\cdot\Theta^{l},$ (11)

where $l$ and $p$ indicate the layer index and the number of layers of the
backbone $f_{\Theta}$, respectively.

Task-specific gradients generate attenuation coefficients insensitive to
various environments and hence seem to be ineffective information to make
attenuation environment-adaptive, which will be presented and analyzed in
Section V-C. We adopt novel representations relevant to environments and
expect that these representations can effectively capture and represent the
acoustic properties of various environments. These representations are
acquired through an unsupervised learning approach, enabling the selective
memorization of target-environment-dependent prior knowledge to some extent.
We denote selective memory Meta-SELD with environment representations as
environment-adaptive Meta-SELD.

###  III-B Environment representations

Fig. 4 depicts the method of extracting environment representations.
Specifically, we employ the environment extractor to extract feature
embeddings from output feature maps of each backbone layer. Subsequently,
feature embeddings from each layer are concatenated to environment
representations. The detail of the environment extractor is shown in Fig. 6.
The feature embedding dimension $D=2048$ is used. The output feature maps of
each layer are averaged and weighted-averaged in the batch and time dimension.
We conjecture that these operators mitigate the influence of acoustic events
on the feature embeddings while preserving the environmental information. Fig.
2 illustrates the environment-adaptive Meta-SELD training procedure. Step 6 in
Algorithm 1 represents the extractor process of environment representations.

Our emphasis lies in evaluating the performance of Meta-SELD in unfamiliar
environments. We hypothesize that audio recordings captured in diverse spatial
locations within a given environment exhibit more similar acoustic properties
when compared to recordings from different environments, excluding extreme
cases where the speaker or microphone is close to reflections or each other.
Furthermore, these acoustic properties generally remain consistent across
recording moments and sound events. To test our hypothesis, we display
environment representations via the similarity map and t-SNE [43] in Section
V-F.

##  IV Experimental Setups

###  IV-A Datasets

We utilize the computationally simulated SRIRs to synthesize the datasets
containing 30,000 5-second clips with reverberation time (RT60) from 0.3
seconds to 0.5 seconds. Sound event examples from FSD50K and AudioSet are
cleaned by PANNs. We refer to the computationally simulated datasets as CSD.
For simple comparison and reproducibility, we adopt the official synthetic
datasets [44], which are synthesized using collected SRIRs for the baseline
training of Task 3 of the DCASE Challenge in 2022 and 2023 [4, 5]. The
official synthetic datasets are denoted as Base Dataset or Base in this work.
The Base Dataset contains 1200 1-minute audio clips and is synthesized using
real-scene SRIRs from TAU-SRIR DB [20], which are measured in 9 rooms at
Tampere University. There are 16 different recording rooms in total in the
development set of the STARSS23 dataset, including nine recording rooms in
dev-train-set and seven recording rooms in dev-test-set. The labels of the
STARSS23 evaluation set remain inaccessible, and hence all subsequent ablation
experiments are based solely on the analysis of the STARSS23 development set.
To further validate the effectiveness of our proposed method, we present the
performance of the computationally synthesized scenes in Section V-G.

The environment-adaptive Meta-SELD involves a three-stage pipeline, each stage
utilizing a distinct dataset, as illustrated in Fig. 7. We start with randomly
initializing the EI model and train the EI model on CSD. Subsequently,
selective memory meta-learning is applied to the EI model on the Base Dataset,
served as the meta-training set $\mathcal{D}_{\text{train}}$, to create the
meta-EI model. Finally, fast adaptation is performed to a specific environment
from STARSS23 using a few samples recorded in the corresponding environment.
All development sets of STARSS23 are used for meta-test set
$\mathcal{D}_{\text{test}}$ to evaluate the performance of the adaptation to
unknown environments.

Figure 7: A flowchart showing training procedures of our proposed method, from
a randomly initialized model to the final model to adapt to unknown
environments. The cylinders denote the dataset we used in different stages of
the training.

$\mathcal{D}_{\text{train}}$ and $\mathcal{D}_{\text{test}}$ are divided into
9 tasks and 16 tasks, respectively, corresponding to 9 rooms and 16 rooms. In
the meta-training phase, we first sample a batch of rooms and then sample a
batch of examples from each room. A batch of samples from an individual room
constructs a task, and a part of the samples are support samples while the
remaining samples are query samples. The batch sizes of rooms and samples are
set to 9 and 128, respectively. For each batch, the first 30 samples are
designated as support samples in this work. In the meta-test phase, samples
from each room, excluding the support samples from
$\mathcal{S}_{j}^{\mathtt{test}}$, comprise the final test (query) set
$\mathcal{Q}_{j}^{\mathtt{test}}$. We will evaluate the performance of
parameterized function $f_{\Theta_{j}^{\prime}}$ with adaptation parameters
$\Theta_{j}^{\prime}$ on $\mathcal{Q}_{j}^{\mathtt{test}}$ after iteration
updates on $\mathcal{S}_{j}^{\mathtt{test}}$ with the initial parameters
$\Theta$.

###  IV-B Evaluation metrics

We use a joint metric of localization and detection[45, 46] here: two
location-dependent detection metrics, F-score ($\mathrm{F}_{\leq 20^{\circ}}$)
and error rate ($\mathrm{ER}_{\leq 20^{\circ}}$), and two class-dependent
localization metrics, localization recall ($\mathrm{LR}_{\mathrm{CD}}$) and
localization error ($\mathrm{LE}_{\mathrm{CD}}$). $\mathrm{F}_{\leq
20^{\circ}}$ and $\mathrm{ER}_{\leq 20^{\circ}}$ consider true positives
predicted within a spatial threshold $20^{\circ}$ away from the ground truth.
$\mathrm{LE}_{\mathrm{CD}}$ and $\mathrm{LR}_{\mathrm{CD}}$ compute the mean
angular error and true positive rate in the case when the types of sound
events are predicted correctly, respectively.

We use an aggregated SELD metric for the method comparison and hyper-parameter
selection:

${\mathcal{E}_{\mathtt{SELD}}}=\frac{1}{4}\left[\mathrm{ER}_{\leq
20^{\circ}}+\left(1-\mathrm{F}_{\leq
20^{\circ}}\right)+\frac{\mathrm{LE}_{\mathrm{CD}}}{180^{\circ}}+\left(1-\mathrm{LR}_{\mathrm{CD}}\right)\right].$

(12)

A macro-average of $\mathrm{F}_{\leq 20^{\circ}}$,
$\mathrm{LR}_{\mathrm{CD}}$, $\mathrm{LE}_{\mathrm{CD}}$, and
$\mathcal{E}_{\mathtt{SELD}}$ across classes is used. A good system should
have small $\mathrm{ER}_{\leq 20^{\circ}}$, large $\mathrm{F}_{\leq
20^{\circ}}$, small $\mathrm{LE}_{\mathrm{CD}}$, large
$\mathrm{LR}_{\mathrm{CD}}$, and small ${\mathcal{E}_{\mathtt{SELD}}}$. Note
that different from [37], where room-wise metrics are micro-averaged in the
end, we compute the metrics in each room and then macro-average the metrics
across rooms.

###  IV-C Hyper-parameters

The sampling rate is 24 kHz. We extract 64-dimensional log mel spectrograms
from four-channel FOA signals with a Hanning window of 1024 points, and a hop
size of 320. Each audio clip is segmented to a fixed length of five seconds
with no overlap for training and inference.

AdamW [47] is used to update meta parameters, while SGD is used to update
adaptation parameters. The batch size is 128. For training the conventional
supervised learning models, the learning rate is set to 0.001 for the first 60
epochs out of 80 epochs and is then decreased by 10% every 10 epochs. In the
meta-training phase, we find that setting the momentum to 0.01 in the Batch
Normalization layer [48] has better performance. We set one epoch containing
nine meta batches, which encompass all rooms within the Base Dataset.
Subsequently, the gradients of one epoch are averaged to update meta
parameters in the outer loop step. For training Meta-SELD++, the learning rate
is 0.0003 for the first 300 epochs out of 500 epochs and is then decreased by
10% every 100 epochs. We only consider 5 iteration updates in the inner loop
of MAML. All networks are implemented using PyTorch.

##  V Experiments

###  V-A Effect of synthetic data

TABLE I: Results of the computationally synthesized datasets. Methods are evaluated on all development sets of STARSS23. Dataset | $\mathrm{ER}_{20^{\circ}}\downarrow$ | $\mathrm{F}_{20^{\circ}}\uparrow$ | $\mathrm{LE}_{\mathrm{CD}}\downarrow$ | $\mathrm{LR}_{\mathrm{CD}}\uparrow$ | $\mathcal{E}_{\mathtt{SELD}}\downarrow$
---|---|---|---|---|---
Base | 0.722 | 23.2% | $\mathbf{22.2^{\circ}}$ | 39.5% | 0.555
CSD | 0.746 | 20.9% | $25.6^{\circ}$ | 41.4% | 0.566
Base + CSD | 0.697 | 23.1% | $22.6^{\circ}$ | 44.3% | 0.537

We evaluate our synthetic data using the conventional supervised learning
method. The model is trained on the Base Dataset (Base) and CSD, and then
evaluated on all development sets of STARSS23.

Table I shows the results of the data synthesis method. The results
demonstrate that the model trained on the CSD can generalize to real-scene
datasets. Comparing Base with CSD, we observe the performance gap is mainly in
localization. One of the possible reasons is that there is some discrepancy
between computationally simulated SRIRs and measured SRIRs in real scenes. In
addition, adding CSD to the Base Dataset for training further improves the
performance.

###  V-B Effect of Meta-SELD

TABLE II: Description of utilized models. ($\cdot$) denotes the dataset for
training. $\mathcal{S}_{j}^{\text{test}}$ indicates the support set from the
$j$-th room of STARSS23.

Approach Initial parameters Datasets Conventional supervised learning methods
SELD ($\cdot$) - The combinations of Base, CSD, and
$\mathcal{S}^{\mathtt{test}}_{j}$ SELD ($\cdot$) w/ adapt. or Fine-tuned SELD
SELD (Base) or SELD (Base + CSD) $\mathcal{S}^{\mathtt{test}}_{j}$ Meta
learning methods Meta-SELD - Base Meta-SELD w/ adapt. Meta-SELD
$\mathcal{S}^{\mathtt{test}}_{j}$ Meta-SELD++ SELD (CSD) Base Meta-SELD++ w/
adapt. Meta-SELD++ $\mathcal{S}^{\mathtt{test}}_{j}$ Sel. Mem. Meta-SELD SELD
(CSD) Base Sel. Mem. Meta-SELD w/ adapt. Sel. Mem. Meta-SELD
$\mathcal{S}^{\mathtt{test}}_{j}$

To demonstrate the effectiveness of our proposed Meta-SELD, we compare the
Meta-SELD, the SELD method, and the fine-tuned SELD method. The differences
among these methods are described in Table II. Macro-averaged metrics for all
16 rooms of STARSS23 are shown in Table III. In SELD (Base +
$\mathcal{S}_{j}^{\text{test}}$), the support set
$\mathcal{S}_{j}^{\text{test}}$ from the $j$-th room of STARSS23 is added to
the synthetic datasets for training from scratch, and the query set
$\mathcal{Q}_{j}^{\text{test}}$ is used for evaluating that specific model.
This approach, however, requires training multiple models from scratch, one
for each specific room of STARSS23. In SELD (Base), we first train an EI model
on the Base Dataset and then fine-tune (adapt) on
$\mathcal{S}_{j}^{\text{test}}$. In Meta-SELD, we apply MAML to a SELD model
with random initialized meta-parameters. The SELD (Base) and Meta-SELD methods
without adaptation refer to no fine-tuning on $\mathcal{S}_{j}^{\text{test}}$.

The top block of Table III presents the method using the Base Dataset for
training or meta-training, while the bottom block presents the method using
the Base Dataset and computationally simulated datasets. In terms of the
average performance, we can see that after adaptation on
$\mathcal{S}_{j}^{\text{test}}$, both Meta-SELD and Meta-SELD++ exhibit
superior performance in comparison to the corresponding fine-tuned SELD
method. In a comparison between SELD (Base) and Meta-SELD, we observe a bigger
performance gap in Meta-SELD with and without adaptation, which means the meta
parameters of Meta-SELD are more suitable for adapting to a new environment.
However, the performance gap in Meta-SELD++ becomes smaller and meta
parameters from Meta-SELD++ outperform parameters from SELD (Base + CSD). We
conjecture the phenomenon results from conflicts while optimizing among
environments, especially obvious in Meta-SELD++. The SELD method, which adds
the support set $\mathcal{S}_{j}^{\text{test}}$ for training, performs the
best among these methods. This phenomenon may be attributed to the fact that
training multiple independent models from scratch could avoid compromise in
these environments.

TABLE III: The performance of meta-learning-based and conventional supervised-
learning-based methods. All of the methods are evaluated on
$\mathcal{Q}_{i}^{\text{test}}$ from STARSS23.

Method Adapt. $\mathrm{ER}_{20^{\circ}}\downarrow$
$\mathrm{F}_{20^{\circ}}\uparrow$ $\mathrm{LE}_{\mathrm{CD}}\downarrow$
$\mathrm{LR}_{\mathrm{CD}}\uparrow$ $\mathcal{E}_{\mathtt{SELD}}\downarrow$
SELD (Base + $\mathcal{S}_{i}^{\text{test}}$) - 0.642 32.8% $21.1^{\circ}$
54.5% 0.471 SELD (Base) ✗ 0.742 30.1% $23.9^{\circ}$ 53.6% 0.509 ✔ 0.690 31.6%
$22.8^{\circ}$ 53.7% 0.491 Meta-SELD ✗ 0.718 29.8% $24.0^{\circ}$ 53.5% 0.504
✔ 0.639 32.6% $21.3^{\circ}$ 52.8% 0.476 SELD (Base + CSD +
$\mathcal{S}_{i}^{\text{test}}$) - 0.653 33.6% $22.6^{\circ}$ 58.2% 0.465 SELD
(Base + CSD) ✗ 0.714 28.6% $25.2^{\circ}$ 58.8% 0.495 ✔ 0.686 30.1%
$24.7^{\circ}$ 59.2% 0.482 Meta-SELD++ ✗ 0.698 30.5% $23.6^{\circ}$ 59.6%
0.482 ✔ 0.657 31.4% $23.2^{\circ}$ 59.0% 0.470

###  V-C Effect of selective memory

####  V-C1 Inputs to selective memory

TABLE IV: Ablation studies on types of input in selective memory Meta-SELD.

Method Input $\mathrm{ER}_{20^{\circ}}\downarrow$
$\mathrm{F}_{20^{\circ}}\uparrow$ $\mathrm{LE}_{\mathrm{CD}}\downarrow$
$\mathrm{LR}_{\mathrm{CD}}\uparrow$ $\mathcal{E}_{\mathtt{SELD}}\downarrow$
Meta-SELD++ w/ adapt. - 0.657 31.4% $23.2^{\circ}$ 59.0% 0.470 \+ Selective
Memory None 0.635 33.4% $\mathbf{21.5^{\circ}}$ 60.2% 0.455 Gradients 0.635
34.2% $21.6^{\circ}$ 57.4% 0.460 Rep. (Ours) 0.629 34.1% $21.7^{\circ}$ 59.6%
0.453

Figure 8: Generated room-and-layer-wise attenuation factors $\lambda$ using
selective memory for each room of STARSS23.

Table IV shows the results of three types of input in the selective memory
methods, the task-specific gradients on the support set $\mathcal{S}_{i}$,
”None”, and the environment representations. The input ”None” in the selective
memory method refers to layer-wise learnable parameters as attenuation
coefficients instead of being generated by the attenuation network. We see
that applying selective memory to Meta-SELD++ obtains performance improvement,
particularly in localization. These methods are even more effective than the
SELD (Base + CSD + $\mathcal{S}_{i}^{\text{test}}$) in Table III. Among these
types of inputs, the environment representations perform better. This exhibits
the effectiveness of environment representations as the input to the selective
memory method.

####  V-C2 Attenuation factors

Fig. 8 illustrates the attenuation factor $\lambda$ of a few typical layers
for each room of STARSS23. However, observing room-and-layer-wise attenuation
factor $\lambda$ derived from the task-specific gradients, we note that
$\lambda$ varies over a small range from room to room. This suggests that
$\lambda$ can not be adaptive to the diverse acoustic environments of these
rooms. We analyze the inputs to the attenuation network, task-specific
gradients on $\mathcal{S}_{i}$, and find that most gradients exhibit
diminutive magnitudes. This phenomenon can be attributed to the pre-trained EI
model, which initializes the meta-parameters of Meta-SELD++, resulting in
minute gradient values. In contrast, a comparison of the gradients and
representations as input to the attenuation network indicates that environment
representations generate more environment-adaptive attenuation factors
$\lambda$. The changes in these attenuation factors are more conspicuous from
room to room.

For CNN-weight_1 in Fig. 8, which is the first layer of the backbone, we
observe that generated $\lambda$ of this layer is relatively large and changes
pretty little among rooms, compared with subsequent layers. The initial layers
encode environment-independent features, while deep layers prefer environment-
adaptive attenuation and encode environment-dependent features, which aligns
with the observation of [35].

####  V-C3 Representations of Environments

TABLE V: Ablation studies on approaches for extracting environment
representations.

Approach $\mathrm{ER}_{20^{\circ}}\downarrow$
$\mathrm{F}_{20^{\circ}}\uparrow$ $\mathrm{LE}_{\mathrm{CD}}\downarrow$
$\mathrm{LR}_{\mathrm{CD}}\uparrow$ $\mathcal{E}_{\mathtt{SELD}}\downarrow$
Last emb. (Mean) 0.660 33.1% $21.7^{\circ}$ 59.1% 0.465 Last emb. (Encode)
0.635 33.8% $\mathbf{21.2^{\circ}}$ 58.4% 0.458 Env. rep. (Ours) 0.629 34.1%
$21.7^{\circ}$ 59.6% 0.453

Figure 9: The effect of hyper-parameters of MAML. Each column represents one
metric. Top row: The number of inner-loop update steps. Bottom row: The number
of support samples.

We investigate various techniques for extracting environment representations:
feature maps from the last layer (before the linear layer) that are averaged
in the batch and time axes, feature embeddings derived from the last layer,
and environment representations constituted by concatenated feature embeddings
from all preceding layers. Table V demonstrates the effectiveness of the
environment extractor and preceding feature embeddings. Feature maps from the
last layer encoded by the environment extractor perform better compared to
being directly averaged. Moreover, preceding feature embeddings provide more
information and further performance improvement.

###  V-D Effect of adaptation setups

The most important hyper-parameters in the adaptation phase of MAML include
the number of inner-loop update steps and support samples. Intuitively, the
inner-loop optimization should be consistent during meta-training and meta-
testing. However, the large number of update steps and support samples leads
to excessive computation and memory burdens. We exploit previously well-
trained parameters and investigate the effect of adaptation setups in meta-
testing.

The top row of Fig. 9 shows the effect of the number of update steps on the
adaptation of SELD (Base + CSD), Meta-SELD++, and environment-adaptive Meta-
SELD. The environment-adaptive Meta-SELD achieves more efficient adaptation.
The benefits come from selectively memorizing necessary information, helping
the learner adapt to new environments more quickly.

We also investigate the effect of the number of shots (support samples). Note
that in this experimental setup, we select the first 50 samples of each room
as the support set, and all remaining samples are as the query set. The bottom
row of Fig. 9 shows that meta-learning-based methods exploit support samples
more effectively. When the number of support samples increases in the
environment-adaptive Meta-SELD method, the performance is consistently
improved, but the magnitude of performance improvement also appears to
decrease.

###  V-E Room-wise performance

Figure 10: Room-wise performance of all development sets of STARSS23. All
methods are evaluated on $\mathcal{Q}_{j}^{\text{test}}$. Note that the
average score of each metric computes the macro-average performance of all
these rooms.

Fig. 10 shows the room-wise metrics. Through a comparative analysis of
conventional supervised-learning-based and meta-learning-based methods on
identical datasets, we observe that meta-learning-based methods can reduce
$\mathcal{E}_{\mathtt{SELD}}$ effectively in rooms where conventional
supervised-learning-based methods exhibit high $\mathcal{E}_{\mathtt{SELD}}$,
such as Room 2, Room 4, and Room 22. This means when a model has an
unsatisfactory generalization to a specific room below the average
performance, using a few samples collected in the specific room for adapting
could improve the performance significantly. However, we also observe
performance degradation or insignificant improvement in
$\mathcal{E}_{\mathtt{SELD}}$, even though some new samples of unseen
environments are used for adapting in both conventional supervised-learning-
based and meta-learning-based methods, for example, in Room 7 and Room 15.
This phenomenon could arise from the fact that our methods have difficulty
extracting valid information for training from new samples. In Room 9, the
performance of SELD (Base) and SELD (Base + CSD) is improved after fine-tuning
using some new samples of the corresponding room, but the performance of Meta-
SELD and Meta-SELD++ after adaptation is degraded. The reason may be that
compromised initial meta parameters among those rooms fail to adapt to the
environment. The meta-learning-based methods find general initial parameters
that can be adapted to unknown environments in the sense of average, and
experimental results demonstrate that meta-learning-based methods outperform
the conventional supervised-learning-based methods in most rooms and perform
better on average.

In contrast to SELD (Base + CSD), the environment-adaptive Meta-SELD exhibits
comparable average performance in $\mathrm{LR}_{\mathtt{CD}}$, but lower
$\mathrm{LE}_{\mathtt{CD}}$. Consequently, performance improvement in
localization is the main factor for the reduction in
$\mathcal{E}_{\mathtt{SELD}}$. Additionally, the performance improvement in
Room 2 and Room 22 can also be attributed to analogous factors.

###  V-F Environment representations

To interpret the representation extracted by the sub-network in Fig. 4, we
study the relationship between the representation and the environment. We
hypothesize that different clips recorded at various spatial positions in the
same environment generally have more similar acoustic properties in contrast
to various environments, except in extreme cases. Therefore, we extract
representations from several disjoint batches of samples, and then we compute
the similarity between the representations and visualize these representations
via t-SNE [43]. Fig. 11 shows the visualization of extracted representations.

The SELD performance is greatly affected by noise and reverberation. To
dynamically control the acoustic properties of environments to investigate the
representation, we also simulate both reverberant rooms and noisy rooms. For
the reverberant simulations, rooms of the same size are simulated at intervals
of 0.3-second RT60, ranging from 0.4 to 2.5 seconds, therefore, there are 8
rooms with different RT60 in total. For the noisy simulations, we simulate 15
rooms of the same size, absorption coefficients, and reflection orders.
Subsequently, we add ambient noise from the NoiseX-92 [49] database into
synthetic recordings to generate SNR ranging from 10 dB to 15 dB. The
NoiseX-92 database contains 15 types of ambient noise. Hence each room has a
unique noise type. All sound event examples for synthesizing are randomly
sampled from FSD50K and AudioSet. Based on the previously trained model using
the environment-adaptive Meta-SELD method, we directly perform an inference
(meta-test) on these simulated datasets.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 11: Visualization of environment representations. The four plots (a-d)
of the top row display the cosine similarity maps between support
representations and query representations. The four plots (e-h) of the bottom
row display the representations via t-SNE, and points of the same color
indicate representations from the same room. Four datasets are shown: the Base
Dataset, the development set of STARSS23, computationally simulated
reverberant rooms, and computationally simulated noisy rooms.

####  V-F1 Similarity Maps

Representations from the same or similar rooms should have a high similarity.
In this work, the cosine distance of two representations is used to measure
their similarity. For a batch of 128 samples from the same room, we extract
support representations from the first 30 samples, corresponding to the number
of support samples for adaptation, and query representations from the last 98
samples. The cosine similarity is computed between support representations
from a room and query representations from the same or other rooms.

Fig. 11 (a) and (b) present the cosine similarity maps on the Base Dataset and
STARSS23. The experimental results show that all diagonal elements are the
maximum of their rows of the similarity matrix on the Base Dataset, while
$12/16$ of diagonal elements are the maximum of their rows on STARSS23. Fig.
11(c) and (d) show the cosine similarity maps on the simulated reverberant
rooms and simulated noisy rooms. We see that these maps are relatively
symmetrical. The diagonal elements of the similarity matrix are pretty large.
In addition, we observe environment representations have a high resolution in
rooms with low RT60, but a low resolution in rooms with high RT60. The
differences in the environment representations of these rooms with high RT60
are small. This may result from the small range RT60 of the training set.

####  V-F2 Visualization of representations via t-SNE

We sample 8 batches of clips from each dataset, with 32 clips per batch, and
then compute representations for each batch222For STARSS23, the number of
clips is less than 256 in a few rooms, therefore, we repeat the corresponding
batch of clips along the batch dimension.. Empirically, we find the learned
representations cluster meaningfully: representations from the same room tend
to be clustered, as shown in Fig. 11 (e-h). Especially in Fig. 11(h),
different recordings with the same noisy type have similar features, which
leads to better clustering performance.

These observations demonstrate the extracted representations are relevant to
the environments.

###  V-G Results on computationally synthesized scenes

TABLE VI: The performance of proposed methods on computationally synthesized
scenes.

Method Adapt. $\mathrm{ER}_{20^{\circ}}\downarrow$
$\mathrm{F}_{20^{\circ}}\uparrow$ $\mathrm{LE}_{\mathrm{CD}}\downarrow$
$\mathrm{LR}_{\mathrm{CD}}\uparrow$ $\mathcal{E}_{\mathtt{SELD}}\downarrow$
SELD (Base + CSD) ✗ 0.739 15.0% $30.0^{\circ}$ 30.8% 0.612 ✔ 0.733 15.4%
$28.9^{\circ}$ 30.6% 0.609 Meta-SELD++ ✗ 0.772 13.9% $32.1^{\circ}$ 32.0%
0.623 ✔ 0.745 15.5% $30.4^{\circ}$ 31.8% 0.610 Environment-Adaptive Meta-SELD
(Ours) ✔ 0.692 17.6% $\mathbf{25.3^{\circ}}$ 31.3% 0.586

We evaluate the method on computationally synthesized scenes to further
validate the effectiveness of environment-adaptive Meta-SELD. These scenes are
represented as semantically labeled 3D meshes from the 3D-FRONT dataset [50],
which contains 18,968 diversely furnished rooms in 6,813 professionally
designed houses. We computationally simulate SRIRs of 15 houses sampled from
the 3D-FRONT dataset, which is the same as the simulation of the Geometric-
Wave Acoustic (GWA) dataset [51]. We synthesize 256 5-second spatial
recordings for each acoustic environment with these simulated SRIRs and sound
events from FSD50K [18] and AudioSet [19]. Ambient noise from the NoiseX-92
[49] database is also mixed into spatial recordings to generate signal-to-
noise (SNR) ranging from 10 dB to 15 dB.

We leverage the previous methods trained on CSD and Base Dataset to perform an
inference (meta-test) on the simulated dataset. Table VI presents the results
on computationally synthesized scenes. Notably, we see weak generalizations to
these environments for both conventional supervised learning and meta-learning
methods, due to disparities in environmental conditions between the training
and test datasets. Additionally, the considerable variation in distribution
between meta-training sets and meta-testing sets puts the meta-learned prior
knowledge at high risk of losing effectiveness [29, 52, 53]. However, the
incorporation of selective memory into Meta-SELD++ manifests superior
performance in contrast to other methods, particularly improving the
localization performance.

##  VI Conclusion

This study presents environment-adaptive Meta-SELD designed for efficient
adaptation to specific acoustic environments using a limited number of samples
recorded in those settings. We apply Model-Agnostic Meta-Learning (MAML) to a
pre-trained environment-independent SELD model to obtain generalized initial
parameters for different environments. Subsequently, we introduce selective
memory and environment representations in Meta-SELD++ to alleviate conflicts
and the limitations of common initialization across different environments.
When evaluated on the development set of the STARSS23 datasets and
computationally synthesized scenes, our proposed environment-adaptive Meta-
SELD demonstrates superior performance compared to conventional supervised-
learning-based SELD methods. Furthermore, we investigate and exhibit
environment representations. Experimental results show that environment
representations effectively capture the nuances of diverse acoustic
environments. The potential applications of these environment representations
are extensive, promising significant advancements in enhancing acoustic scene
analysis in diverse settings.

## References

  * [1] S. Adavanne, A. Politis, and T. Virtanen, “A multi-room reverberant dataset for sound event localization and detection,” in _Proceedings of Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) Workshop_ , 2019, pp. 10–14.
  * [2] A. Politis, S. Adavanne, and T. Virtanen, “A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection,” in _Proceedings of Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) Workshop_ , 2020, pp. 165–169.
  * [3] A. Politis, S. Adavanne, D. Krause, A. Deleforge, P. Srivastava, and T. Virtanen, “A dataset of dynamic reverberant sound scenes with directional interferers for sound event localization and detection,” in _Proceedings of Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) Workshop_ , 2021, pp. 125–129.
  * [4] A. Politis, K. Shimada, P. Sudarsanam, S. Adavanne, D. Krause, Y. Koyama, N. Takahashi, S. Takahashi, Y. Mitsufuji, and T. Virtanen, “STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,” in _Proceedings of Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) Workshop_ , 2022, pp. 161–165.
  * [5] K. Shimada, A. Politis, P. Sudarsanam, D. Krause, K. Uchida, S. Adavanne, A. Hakala, Y. Koyama, N. Takahashi, S. Takahashi, T. Virtanen, and Y. Mitsufuji, “STARSS23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,” _arXiv preprint arXiv: 2306.09126_ , 2023.
  * [6] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 13, no. 1, pp. 34–48, 2019.
  * [7] Y. Cao, T. Iqbal, Q. Kong, Y. Zhong, W. Wang, and M. D. Plumbley, “Event-independent network for polyphonic sound event localization and detection,” in _Proceedings of Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) Workshop_ , 2020, pp. 11–15.
  * [8] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, “An improved event-independent network for polyphonic sound event localization and detection,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 885–889.
  * [9] J. Hu, Y. Cao, M. Wu, Q. Kong, F. Yang, M. D. Plumbley, and J. Yang, “A track-wise ensemble event independent network for polyphonic sound event localization and detection,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 9196–9200.
  * [10] K. Shimada, Y. Koyama, N. Takahashi, S. Takahashi, and Y. Mitsufuji, “ACCDOA: Activity-coupled cartesian direction of arrival representation for sound event localization and detection,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 915–919.
  * [11] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji, “Multi-ACCDOA: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 316–320.
  * [12] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” _Neurocomputing_ , vol. 312, pp. 135–153, 2018.
  * [13] W. M. Kouw and M. Loog, “A review of domain adaptation without target labels,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 3, pp. 766–785, 2019.
  * [14] J. Hu, Y. Cao, M. Wu, Q. Kong, F. Yang, M. D. Plumbley, and J. Yang, “Sound event localization and detection for real spatial sound scenes: Event-independent network and data augmentation chains,” in _Proceedings of Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) Workshop_ , 2022, pp. 46–50.
  * [15] P.-A. Grumiaux, S. Kitić, L. Girin, and A. Guérin, “A survey of sound source localization with deep learning methods,” _The Journal of the Acoustical Society of America_ , vol. 152, no. 1, pp. 107–151, 2022.
  * [16] Q. Wang, J. Du, H. Wu, J. Pan, F. Ma, and C. Lee, “A four-stage data augmentation approach to ResNet-Conformer based acoustic modeling for sound event localization and detection,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 1251–1264, 2023.
  * [17] Y. Koyama, K. Shigemi, M. Takahashi, K. Shimada, N. Takahashi, E. Tsunoo, S. Takahashi, and Y. Mitsufuji, “Spatial data augmentation with simulated room impulse responses for sound event localization and detection,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 8872–8876.
  * [18] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An open dataset of human-labeled sound events,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 829–852, 2022.
  * [19] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 776–780.
  * [20] A. Politis, S. Adavanne, and T. Virtanen, “TAU Spatial Room Impulse Response Database (TAU-SRIR DB),” Apr. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6408611
  * [21] J. B. Allen and D. A. Berkley, “Image method for efficiently simulating small-room acoustics,” _The Journal of the Acoustical Society of America_ , vol. 65, no. 4, pp. 943–950, 1979.
  * [22] Z. Tang, L. Chen, B. Wu, D. Yu, and D. Manocha, “Improving reverberant speech training using diffuse acoustic simulation,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6969–6973.
  * [23] R. Scheibler, E. Bezzam, and I. Dokmanić, “Pyroomacoustics: A python package for audio room simulation and array processing algorithms,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 351–355.
  * [24] D. Diaz-Guerra, A. Miguel, and J. R. Beltran, “gpuRIR: A python library for room impulse response simulation with GPU acceleration,” _Multimedia Tools and Applications_ , vol. 80, pp. 5653–5671, 2021.
  * [25] D. P. Jarrett, E. A. Habets, M. R. Thomas, and P. A. Naylor, “Rigid sphere room impulse response simulation: Algorithm and applications,” _The Journal of the Acoustical Society of America_ , vol. 132, no. 3, pp. 1462–1472, 2012.
  * [26] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” _Proceedings of the IEEE_ , vol. 109, no. 1, pp. 43–76, 2020.
  * [27] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM Computing Surveys_ , vol. 53, no. 3, pp. 1–34, 2020.
  * [28] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 9, pp. 5149–5169, 2021.
  * [29] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang, “A closer look at few-shot classification,” in _Proceedings of International Conference on Learning Representations (ICLR)_ , 2019.
  * [30] N. Gao, H. Ziesche, N. A. Vien, M. Volpp, and G. Neumann, “What matters for meta-learning vision regression tasks?” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 14 776–14 786.
  * [31] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceedings of International Conference on Machine Learning (ICML)_ , 2017, pp. 1126–1135.
  * [32] S.-F. Huang, C.-J. Lin, D.-R. Liu, Y.-C. Chen, and H.-y. Lee, “Meta-TTS: Meta-learning for few-shot speaker adaptive text-to-speech,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 1558–1571, 2022.
  * [33] M. Barhoush, A. Hallawa, A. Peine, L. Martin, and A. Schmeink, “Localization-driven speech enhancement in noisy multi-speaker hospital environments using deep learning and meta learning,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 670–683, 2023.
  * [34] W. Lin and M.-W. Mak, “Model-agnostic meta-learning for fast text-dependent speaker embedding adaptation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 1866–1876, 2023.
  * [35] S. Baik, S. Hong, and K. M. Lee, “Learning to forget for meta-learning,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 2379–2387.
  * [36] R. Vuorio, S.-H. Sun, H. Hu, and J. J. Lim, “Multimodal model-agnostic meta-learning via task-aware modulation,” in _Proceedings of Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 32, 2019.
  * [37] J. Hu, Y. Cao, M. Wu, Z. Yu, F. Yang, W. Wang, M. D. Plumbley, and J. Yang, “Meta-SELD: Meta-learning for fast adaptation to the new environment in sound event localization and detection,” in _Proceedings of Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) Workshop_ , 2023, pp. 51–55.
  * [38] K. Shimada, K. Uchida, Y. Koyama, T. Shibuya, S. Takahashi, Y. Mitsufuji, and T. Kawahara, “Zero- and few-shot sound event localization and detection,” _arXiv preprint arXiv:2309.09223_ , 2023.
  * [39] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [40] A. Politis, “Microphone array processing for parametric spatial audio techniques,” Ph.D. dissertation, Aalto University, 2016.
  * [41] B. Rafaely, _Fundamentals of Spherical Array Processing_. Springer, 2015.
  * [42] A. Politis and H. Gamper, “Comparing modeled and measurement-based spherical harmonic encoding filters for spherical microphone arrays,” in _Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , 2017, pp. 224–228.
  * [43] L. Van der Maaten and G. Hinton, “Visualizing data using t-SNE,” _Journal of Machine Learning Research_ , vol. 9, no. 86, pp. 2579–2605, 2008.
  * [44] A. Politis, “[DCASE2022 Task 3] Synthetic SELD mixtures for baseline training,” Apr. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6406873
  * [45] A. Mesaros, S. Adavanne, A. Politis, T. Heittola, and T. Virtanen, “Joint measurement of localization and detection of sound events,” in _Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , 2019, pp. 333–337.
  * [46] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, “Overview and evaluation of sound event localization and detection in DCASE 2019,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 684–698, 2021.
  * [47] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _Proceedings of International Conference on Learning Representations (ICLR)_ , 2018.
  * [48] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of International Conference on Learning Representations (ICLR)_ , 2015.
  * [49] A. Varga and H. J. Steeneken, “Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems,” _Speech Communication_ , vol. 12, no. 3, pp. 247–251, 1993.
  * [50] H. Fu, B. Cai, L. Gao, L.-X. Zhang, J. Wang, C. Li, Q. Zeng, C. Sun, R. Jia, B. Zhao _et al._ , “3D-FRONT: 3D furnished rooms with layouts and semantics,” in _Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 10 933–10 942.
  * [51] Z. Tang, R. Aralikatti, A. Ratnarajah, , and D. Manocha, “GWA: A large geometric-wave acoustic dataset for audio processing,” in _Proceedings of Special Interest Group on Computer Graphics and Interactive Techniques Conference (SIGGRAPH)_ , no. 36, 2022, pp. 1–9.
  * [52] A. Setlur, O. Li, and V. Smith, “Two sides of meta-learning evaluation: In vs. out of distribution,” in _Proceedings of Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 34, 2021, pp. 3770–3783.
  * [53] S. Chen, L.-K. Huang, J. R. Schwarz, Y. Du, and Y. Wei, “Secure out-of-distribution task generalization with energy-based models,” in _Proceedings of Advances in Neural Information Processing Systems (NeurIPS)_ , 2023.