:
166 2022 Holistic Evaluation of Audio Representations Editor’s name

# Learning Audio Representations with MLPs

Mashrur M. Morshed1<EMAIL_ADDRESS>
Ahmad Omar Ahsan1<EMAIL_ADDRESS>
Hasan Mahmud2<EMAIL_ADDRESS>
Md. Kamrul Hasan2<EMAIL_ADDRESS>
alt. email<EMAIL_ADDRESS>email<EMAIL_ADDRESS>1
Intelligent Machines Limited
2 Systems & Software Lab  Dept. of CSE  Islamic University of Technology

###### Abstract

In this paper, we propose an efficient MLP-based approach for learning audio
representations, namely timestamp and scene-level audio embeddings. We use an
encoder consisting of sequentially stacked gated MLP blocks, which accept 2D
MFCCs as inputs. In addition, we also provide a simple temporal interpolation-
based algorithm for computing scene-level embeddings from timestamp
embeddings. The audio representations generated by our method are evaluated
across a diverse set of benchmarks at the Holistic Evaluation of Audio
Representations (HEAR) challenge, hosted at the NeurIPS 2021 competition
track. We achieved first place on the Speech Commands (full), Speech Commands
(5 hours), and the Mridingham Tonic benchmarks. Furthermore, our approach is
also the most resource-efficient among all the submitted methods, in terms of
both the number of model parameters and the time required to compute
embeddings.

###### keywords:

Audio Representations, Embeddings, MLP, Interpolation, Representation Learning

##  1 Introduction

Multilayer Perceptrons (MLP) are densely connected (also called fully
connected) feed-forward neural networks, where all the neurons in an arbitrary
hidden layer are connected to every other neuron in the subsequent hidden
layer (Lippmann, 1987). MLPs are widely considered as classical deep neural
networks.

Over the past decade, researchers have opted for other neural network
variants, such as Convolutional Neural Networks (CNN) or Recurrent Neural
Networks (RNN), because of some shortcomings of classical MLPs. For instance,
CNNs have the very useful translation-invariance property, which make them
uniquely suitable for Computer Vision and spatial pattern recognition tasks.
Furthermore, because convolutional kernels are windowed across inputs, they
implement a form of weight sharing (Mouton et al., 2021). This makes them many
times more efficient in terms of computation and parameters over classical
MLPs, in addition to their better recognition performance. RNNs are similarly
preferred for tasks with sequential or time-based features, or tasks where it
is necessary to model temporal dependency.

However, MLPs have lately seen a resurgence in the field of deep learning,
particularly in vision. Recent research (Tolstikhin et al., 2021; Liu et al.,
2021b) propose efficient ways of using MLPs to learn spatial and channel
information in a way that makes them competitive with CNNs and other state-of-
the-art approaches (such as vision transformers (Dosovitskiy et al., 2021)).

In this paper, we show that MLPs can be used to learn useful audio
representations from Mel Filter Cepstrum Coefficients (MFCC), which are 2D
signals like images. While images are a 2D function of two spatial dimensions,
MFCCs are instead a function over time and mel-frequencies. The channel and
spatial operations of MLPs on images can be translated to frequency and
temporal operations on MFCCs, allowing us to learn high-level audio
representations which can be used on downstream tasks. We provide two types of
learned representations: timestamp-level embeddings and scene-level
embeddings.

The scene-level embeddings are generated from the timestamp-level embeddings,
using an iterative linear interpolation-based algorithm over the temporal
dimension. We perform a study of three scene-embedding algorithm approaches,
and show that in general, interpolation-based algorithms do perform better
than naive averaging approaches, and that the choice of interpolation
algorithms have a subtle but clear effect on downstream task performance.

Our method was evaluated at the Holistic Evaluation of Audio Representations
(HEAR) challenge (Turian et al., 2022), hosted at the NeurIPS 2021 competition
track. The challenge evaluated submissions across a highly diverse set of
nineteen tasks (derived from sixteen datasets). As a part of the challenge, we
provided a Python package following a common API developed by the organizers.
This means that our work is reproducible, easy to use in future research and
applications, and also extensively evaluated on diverse downstream audio
problems, which clearly highlights both the strengths and limitations of our
approach.

We can summarize our contributions as following:

  1. 1.

We propose an all-MLP approach for learning timestamp and scene-level
embeddings for audio, which shows promising results after extensive evaluation
on a diverse set of audio benchmarks. Our method is the most efficient among
all submissions in terms of both model parameters and embedding generation
time.

  2. 2.

We propose an iterative linear interpolation-based algorithm for generating
scene-level embeddings from timestamp-level embeddings.

  3. 3.

We perform several ablation experiments, with which we can improve some of our
existing results, and even achieve results better than the first place score
on certain tasks.

The rest of this paper is organized as follows: in Section 2 we provide a
brief background on some of the related concepts, as well as a review of
relevant related works. In Section 3 we describe our proposed method and
approach to the HEAR challenge. We observe our results on HEAR in Section 4
and analyze those results from various perspectives in Section 5. We then
describe three types of ablation experiments and their respective results in
Section 6, followed by our conclusion in Section 7.

##  2 Background

###  2.1 Low-Level Audio Representations

By low-level, we refer to representations which have not been learned through
any supervised or unsupervised training method. The lowest-level audio
representation is naturally the digital audio signal, directly obtained from
the raw analog audio or sound. Generally, digital audio can be represented as
a numeric array of length $t\times sr$, where $t$ is the time duration of the
audio and $sr$ is the sampling rate. This is a very high dimensional
representation, and thus requires additional processing in order to be used in
discriminative tasks. More useful low-level representations with lower
dimensionality can be obtained by applying handcrafted transforms, such as
log-Mel Spectrograms (Mermelstein, 1976), Mel Frequency Cepstrum Coefficients
(MFCC) (Davis and Mermelstein, 1980), Constant Q-Transform (CQT) (Schörkhuber
and Klapuri, 2010), Variable Q-Transform (VQT) (Schörkhuber et al., 2014), and
Gammatone Filterbanks (Patterson et al., 1987). High-level audio
representations (embeddings) are learned by training models which take the
low-level representations as inputs, and produces vectors or feature-maps. In
our work, our model learns audio embeddings by using MFCCs as inputs.

###  2.2 Learned Audio Representations

Throughout the past decade, numerous methods have been explored to learn audio
embeddings (which can be considered as high-level audio representations).

PANNs (Pretrained Audio Neural Networks) (Kong et al., 2020) proposed a
variety of audio CNN models with Mel Spectrogram inputs, pretrained at a large
scale on about 5000 hours of audio from the AudioSet dataset (Gemmeke et al.,
2017). In the original PANNs paper, the authors evaluated its performance
across six audio pattern recognition tasks, including audio tagging and sound
event detection. Gong et al. (2021a) proposed the AST (Audio Spectrogram
Transformer), which uses DEIT (Data-efficient Image Transformers) (Touvron et
al., 2021b) pretrained on ImageNet (Deng et al., 2009) and further pretrains
on AudioSet, showing good performance on several tasks. In a later work, (Gong
et al., 2021b) propose a self-supervised way to train AST, showing better
downstream task performance. SoundNet (Aytar et al., 2016) learns audio
representations from unlabelled videos by exploiting the synchronization
between audio and video.

Schneider et al. (2019), in their work Wav2Vec, explored an unsupervised pre-
training approach for speech recognition by learning representations from
large amounts of raw, unlabelled audio. In their later work Wav2Vec2.0
(Baevski et al., 2020), the authors explore a self-supervised method where
they use a single hour of labelled data and large amounts of unlabelled data
to outperform existing speech recognition methods.

Vygon and Mikhaylovskiy (2021) used a triplet loss-based embedding and a kNN
classifier to significantly improve the performance of convolutional networks
on Speech datasets. Wang and Oord (2021) use contrastive learning to jointly
learn audio representations from raw audio and spectrograms. CREPE (Kim et
al., 2018) proposes a CNN-based approach for learning pitch-sensitive
representations.

Ling et al. (2020) propose a representation learning approach where
reconstruction is done from past and future context frames; the learned deep
contextualized representations reduce the need for labelled data. TERA (Liu et
al., 2021a) (Transformer Encoder Representations from Alteration) is a self-
supervised speech pretraining method that trains transformer encoders on a
large amount of unlabelled speech data. TERA is evaluated on several
downstream tasks, such as phoneme classification, keyword spotting, speaker
recognition, and speech recognition. Lastly, HuBERT (Hidden-unit BERT) (Hsu et
al., 2021) uses a 1 billion parameter model and an offline clustering step to
match or improve on the performance of Wav2Vec2.0 on several speech tasks.

###  2.3 Timestamp and Scene-level Embeddings

#### Timestamp-level Embeddings

These embeddings either represent a particular time instance in the audio, or
a very short period centered around that particular time instance. Each
timestamp embedding is in practice a numeric vector of certain size, which we
refer to as the timestamp embedding size ($E_{T}$).

From a single audio clip, we can generate a variable number of timestamp
embeddings ($N_{T}$), depending on the duration of the input audio and the hop
length or stride of the model. $N_{T}$ is roughly equivalent to the length of
the audio divided by the hop length.

Timestamp-level embeddings for a single audio are thus of the shape
$N_{T}\times E_{T}$. As these embeddings represent specific, temporally
ordered instances in the audio, they can be used for tasks like Sound Event
Detection.

#### Scene-level Embeddings

On the other hand, scene-level embeddings represent the entire input audio.
These embeddings are numeric vectors which remain fixed in size even with
input audios of variable length. Naturally, there can only be a single scene-
level embedding for a single audio. The length of the scene embedding vector
is what we refer to as the scene embedding size ($E_{S}$). Since scene-level
embeddings represent the entire audio, they are mainly useful in various audio
classification tasks.

###  2.4 Transformers

Transformers (Vaswani et al., 2017) have been a highly disruptive innovation
in deep learning. They have shown remarkable performance in numerous Natural
Language Processing problems (Devlin et al., 2018; Liu et al., 2019) and can
currently be considered as the de facto approach to NLP (Wolf et al., 2019),
replacing recurrent neural networks.

With the advent of Vision Transformers (ViT) (Dosovitskiy et al., 2021), they
have also excelled in a large variety of Computer Vision problems (Khan et
al., 2021). A high-level reason for this is that transformers solve the
problem of long-term dependencies, a problem not limited to the NLP domain;
modelling long-term dependencies is also crucial to improving performance in
many vision problems.

Transformers are also gaining predominance in the audio field. There are
methods such as TERA (Liu et al., 2021a), Conformer (Gulati et al., 2020)
(convolution-augmented transformers, used in speech recognition), and then
ViT-like approaches such as the Keyword Transformer (KWT) (Berg et al., 2021)
and the Audio Spectrogram Transformer (AST) (Gong et al., 2021a). In recent
self-supervised audio representation learning methods, transformer-based
encoders have seen much use alongside convolutional or convolutional-recurrent
encoders (Liu et al., 2022).

###  2.5 Gated MLPs (gMLP)

A recent group of works, the MLP-Mixer (Tolstikhin et al., 2021), gated MLPs
(gMLP) (Liu et al., 2021b), ResMLP (Touvron et al., 2021a), as well as the
work of Melas-Kyriazi (2021), provide evidence that self-attention, a core
component in transformers and the ViT, may not be solely responsible for their
remarkable performance on vision tasks. These models are attention-free and
are solely comprised of MLPs, showing that the patch-embedding process they
share with ViTs may be the crucial component. We mainly focus on gMLP here, as
our work is closely related with theirs.

Like ViT, the gMLP is a patch-based model. It accepts patches of images as
inputs, which are first projected to a higher dimension through a patch
embedding layer. Then these patch embeddings are passed through a stack of
gMLP blocks, where each block is effectively a pair of channel-wise linear
projections, separated by a spatial linear projection.

Keyword-MLP (KW-MLP) (Morshed and Ahsan, 2021) is a much smaller gMLP-like
model used for Keyword Spotting on Speech Commands (Warden, 2018). Instead of
patches of an image, KW-MLP accepts a 2D MFCC without patching. The frequency
dimension of MFCC is projected to higher dimensions, followed by a variant of
the gMLP block which applies a pair of linear projections across the frequency
dimension, separated by a linear projection across the temporal dimension.
While the original gMLP model used on ImageNet has three variants with 6M, 20M
and 73M parameters, KW-MLP has only 0.4M parameters.

We use the KW-MLP as an encoder in our approach, due to its very efficient
size and some of its unique properties, which we will discuss in Section 3.2.

###  2.6 Holistic Evaluation of Audio Representations (HEAR)

From some of the existing works discussed above, it can be observed that audio
representation learning methods are generally evaluated on a limited set of
downstream tasks, usually containing speech recognition. This is a problem,
because the primary use of learned audio representations is to obtain better
performance in solving downstream tasks, particularly low-resource tasks for
which there aren’t much available data. Because of the lack of extensive
evaluation and the generally homogeneous type of downstream tasks existing
models are evaluated on, we do not have a good idea of what representation
would excel on a new, unknown task.

The HEAR challenge (Turian et al., 2022) addressed this very concern, by
evaluating methods on nineteen different tasks, derived from sixteen different
datasets. Submitted audio representations (timestamp and scene-level
embeddings) were used to train a very shallow fully connected network, without
finetuning the embedding model at all. This allowed empirical evaluation of
the generalization capability of various learned audio representations.

Figure 1: Overview of our approach. Each audio clip is divided into 1 s audio
segments, which are then converted to MFCCs and passed to the MLP-based
encoder.

##  3 Method

We approached the HEAR challenge by first training the KW-MLP model on the
Google Speech Commands V2-35 dataset (Warden, 2018). It is to be noted that
one of the open datasets in the HEAR challenge also featured Speech Commands,
but a different variant: the Speech Commands V2-12 dataset. We trained our
encoding model on the 35-class variant of the dataset with the aim of learning
more discriminative features.

After training, we detached the linear classification head from the tail end
of the model, and froze its weights. This frozen model, comprised solely of
MLP layers, is our encoder. A wrapper was added around the encoder to handle
audio processing, and subsequent generation of scene-level embeddings from
timestamp embeddings.

Lastly, we implemented the common API (neuralaudio.ai, 2021a) proposed by the
HEAR organizers, and ensured that our model wrapper was compatible with the
API standards. Afterwards, the HEAR organizers used the HEAR Evaluation Kit
(neuralaudio.ai, 2021b) to evaluate our submission by first generating
embeddings and then training shallow neural networks, on the nineteen tasks
(Turian et al., 2022). For the results we mention in our ablation studies
(Section 6), we perform the evaluation experiments ourselves using the same
open source evaluation kit.

###  3.1 Audio Preprocessing

As per the Common API established by the HEAR organizers , each model is
required to perform at one of 16000, 22050, 32000, 44100 or 48000 Hz sampling
rate. Our model accepts audio of 16000 Hz sampling rate, so any input audio
needs to first be resampled to that rate.

Next, the Keyword-MLP model which we use as our encoder or feature extractor
operates on MFCCs of 1 s audio. So after resampling, any arbitrary input audio
needs to be divided into 1 s long segments. For fractional durations, we
simply perform ceiling padding at the end of the audio to lengthen it to the
next s (e.g. for 4.5 s input audio, we pad it to 5 s).

If an input audio has the duration $b$ s (after appropriate padding) and a
sampling rate $sr$, then we obtain a tensor of shape $(b,sr)$, where $b$ is
the batch size (i.e. we have a batch of 1 s audios). We then compute the MFCC
for this batch of 1 s audio tensors in parallel with nnAudio (Cheuk et al.,
2020), an audio processing toolbox that uses PyTorch-based 1D CNNs as its
backend. This allows us to leverage GPU-parallelism when extracting features
on CUDA runtimes.

We compute MFCCs with a hop size of 10 ms (0.01 s), a window length of 30 ms
(0.03 s), and 40 MFCC bins. The encoder model subsequently accepts inputs of
shape (40, 98). All audio preprocessing settings are detailed in Section A.

###  3.2 KW-MLP Encoder

Keyword-MLP (KW-MLP) (Morshed and Ahsan, 2021) takes MFCCs of 1 s audio clips
as input. Let us assume $X\in{\mathbb{R}}^{F\times T}$ is an input MFCC, where
$F$ and $T$ are the mel frequency bins and time-steps respectively. As
mentioned earlier in Section 2.5, KW-MLP does not require patching its 2D
input, unlike ViT and gMLP. Technically, $X$ is to be divided into $T$ patches
of shape $F\times{1}$, getting $X_{0}\in{\mathbb{R}}^{T\times F}$(thus in
practice, patching is actually not required, and is equivalent to a
transpose).

Using a linear projection matrix $P_{0}\in{\mathbb{R}}^{F\times d}$, $X_{0}$
is projected to a higher dimension $d$, which is the embedding size of the
model (and also the timestamp embedding size). We obtain
$X_{E}\in{\mathbb{R}}^{T\times d}$ from this operation.

$X_{E}=X_{0}P_{0}$ (1)

$X_{E}$ can be considered as a frequency domain patch-embedding, as it is a
result of directly projecting the frequency dimension of the input MFCC.

Figure 2: Keyword-MLP architecture. It contains $L=12$ consecutive gated-MLP
blocks, and operates on patches from MFCCs of 1 s audio segments.

The obtained $X_{E}$ is passed through $L$ consecutive, identical gated-MLP
(gMLP) blocks. These blocks can be seen as a pair of projections across the
frequency embedding dimension, separated by a projection across the temporal
dimension. The block can also be formulated with the following set of
equations:

$\begin{split}Z&=\sigma(X_{in}U)\\\
\tilde{Z}=g(Z)&=g([Z_{r}Z_{g}])=Z_{r}\odot(Z_{g}^{T}G)^{T}\\\
X_{out}&=\tilde{Z}V\oplus X_{in}\end{split}$ (2)

Here, $\sigma$ represents the GELU activation function.
$U\in{\mathbb{R}}^{d\times{D}}$ and $V\in{\mathbb{R}}^{D/2\times{d}}$ are the
linear projection matrices which project from $d$ to the higher dimension $D$,
and $D/2$ back to $d$ respectively. $Z\in{\mathbb{R}}^{T\times{D}}$ is split
to $Z_{r}$ (residual) and $Z_{g}$ (gate) $\in{\mathbb{R}}^{T\times{D/2}}$
respectively, while $G\in{\mathbb{R}}^{T\times{T}}$ performs linear projection
on $Z_{g}$ across the temporal dimension. Each projection is effectively done
with a matrix multiplication, and in practice, can be implemented as a Dense
layer. The operation $(Z_{g}^{T}G)^{T}$, the temporal projection, requires two
additional transpose operations. This can be simplified as a pass through a
Conv1D(T, T, 1) layer.

(Note that $A^{T}$ signifies a transposed matrix, and $T$ in this context is
not the time dimension).

The overall architecture of the encoder can be observed in Figure 2. The
embedding size of KW-MLP $d$ (also the timestamp embedding size, $E_{T}$), is
64, while the embedding projection size $D$ is 256. We used $L=12$ blocks like
the one shown in equation 2 and in Figure 2, in our submission to the HEAR
challenge. For later ablation experiments (Section 6.2) we used fewer blocks
(such as 10 and 8). All other hyperparameters and settings are shown in
Section A. It is to be noted that the pretraining task of KW-MLP (training on
Speech Commands V2-35) can be done very quickly, as fast as about two hours on
Google Colab.

KW-MLPL denotes a KW-MLP model with $L$ blocks. The default model that we
simply refer to as KW-MLP is technically KW-MLP12.

###  3.3 Interpolation-based Scene Embeddings

Figure 3: After concatenation, timestamp embeddings are repeatedly downsampled
by linear interpolation till target size is achieved. Then scene embeddings
are obtained by flattening.

We use a hop size of 10 ms. Thus, for each 1 s audio segment, our model
outputs embeddings of size $(100,64)$, where 64 is the timestamp embedding
size, $E_{T}$ (Section 2.3). We concatenate these embeddings along the time
axis to obtain the output timestamp embeddings: for an $N$ s input audio, we
would have embeddings of shape $(N_{T},E_{T})=(100*N,64)$.

This variable-sized timestamp embedding further needs to be reduced to a
fixed-size scene-level representation. We do this by downsampling the
timestamp embedding across the time axis. The scene embedding size $E_{S}$
that we use is 1024, which is equivalent to $16\times 64$. So, we need to
reduce $N_{T}$ to $N_{S}$ (where $N_{S}=16$).

We use linear interpolation across the time axis to do this. However, instead
of directly downsampling from $N_{T}$ to $N_{S}$, we do interpolation multiple
times, each time reducing $N_{T}$ to $N_{T}/2$. The number of required
interpolations can be calculated by the following expression:

$N_{interp}=\left\lceil\log_{2}\left(\frac{N_{T}}{N_{S}}\right)\right\rceil$
(3)

The primary rationale behind iteratively interpolating $N_{interp}$ times is
to better conserve the embedding features when downsampling by a large factor.
PyTorch’s F.interpolate shows unsafe behaviour when directly downsampling by a
large factor, as can be seen from Figure 4. Alongside PyTorch, some common
Python image and array processing libraries exhibit similar behaviour (Zhang
and Lehtinen, 2020).

Figure 4: (left) 1 unit thick circle of diameter 1024 (middle) Directly
downsampled to 32x (right) Iteratively 2x downsampled to 32x

In Section 6.1, we perform an ablation study comparing this iterative
interpolation with regular interpolation and averaging.

##  4 Results

For the sake of reducing redundancy, we do not show full comparative results
for each of the nineteen tasks. We instead only show the result of our
approach and a comparative ranking in Table 1.

The full results comparing every team submission per task can be viewed at the
competition website (neuralaudio.ai, 2021c) and in Turian et al. (2022). To
re-clarify, our submitted Python package and overall submission is called
kwmlp, and our team name is ID56-SSL. KW-MLP is the name of the encoder model,
described in Section 3.2.

Table 1: Results of kwmlp on the nineteen tasks at HEAR 2021 (Turian et al., 2022) Task | Metric | Result | Ranking
---|---|---|---
Speech Commands (V2-12) Full | Accuracy | 0.978 | 1st
Speech Commands (V2-12) 5H | Accuracy | 0.976 | 1st
NSynth Pitch 5H | Pitch Acc | 0.440 | 11th
Chroma Acc | 0.480 | 11th
NSynth Pitch 50H | Pitch Acc | 0.605 | 14th
Chroma Acc | 0.648 | 14th
DCASE 2016 Task 2 | Event Onset FMS | 0.518 | 18th
Segment Error Rate | 0.527 | 21st
Beehive States | AUCROC | 0.760 | 3rd
Accuracy | 0.552 | 5th
Beijing Opera Percussion | Accuracy | 0.911 | 24th
CREMA-D | Accuracy | 0.424 | 25th
ESC-50 | Accuracy | 0.367 | 25th
FSD50K | mAP | 0.187 | 22nd
d’ | 1.511 | 21st
Gunshot Triangulation | Accuracy | 0.932 | 7th
GTZAN genre | Accuracy | 0.554 | 27th
GTZAN Music Speech | Accuracy | 0.889 | 27th
LibriCount | Accuracy | 0.451 | 27th
Maestro 5H | Note Onset FMS | 0.648 | 11th
Note Onset w/ Offset FMS | 0.0210 | 11th
Mridingham Stroke | Accuracy | 0.969 | 5th
Mridingham Tonic | Accuracy | 0.942 | 1st
Vocal Imitations | mAP | 0.056 | 26th
Accuracy | 0.049 | 26th
Vox Lingua Top 10 | Accuracy | 0.1812 | 24th

##  5 Analysis

In this section, we analyze the observed results from Table 1 from multiple
perspectives.

###  5.1 Tasks With Good Performance

kwmlp ranks first on both the complete Speech Commands as well as the shorter
and more challenging 5 hour variant. We do not consider this an unexpected
result, as the KW-MLP model that we use as an encoder was trained for keyword
spotting on Speech Commands V2-35. It is to be noted however that the tasks
aren’t quite the same; the HEAR challenge evaluated the V2-12 variant of the
dataset.

kwmlp also performs remarkably well on the Mridingham Stroke and Mridingham
Tonic tasks (Anantapadmanabhan et al., 2020). At the very least, we believe it
is a very interesting outcome that the embeddings learned by an extremely
small model, on a rather small dataset, can outperform much larger models
pretrained on larger audio corpuses on a secret task. This provides evidence
that audio representations learned by isotropic, all-MLP models require
further research.

In our ablation experiments (Section 6.3) we obtain improved results for
various tasks, including better than first place results on Gunshot
Triangulation (Cooper and Shaw, 2020) and Mridingham Stroke and Tonic
(Anantapadmanabhan et al., 2020).

###  5.2 Tasks With Poor Performance

Because of supervised pretraining on Speech Commands, we believe our model has
learned to be temporally shift invariant. As a result, performance on FSD50K
(Fonseca et al., 2022) and DCASE (Mesaros et al., 2018) suffers. In Section
5.6, we analyze this by visualizing the model weights. Our method is also
pitch and tone agnostic, thus suffering on tasks like GTZAN (Tzanetakis et
al., 2001), ESC-50 (Piczak, 2015), etc.

On language-based dataset such as Vox Lingua (Valk and Alumae, 2021), our
method did not perform well since our model was trained only English audio.
Again, training on Speech Commands also means not discriminating between
voices, which results in reduced performance on Vocal Imitations (Kim and
Pardo, 2018). Performance in these two tasks are generally sub-par among the
submissions, as most models are pretrained on English speech and may be
invariant to vocal identity.

###  5.3 Audio Duration

The task containing the longest audio clips in the HEAR challenge was Beehive
States (Nolasco et al., 2019), at ten minutes per clip. On the other side,
there are tasks with much shorter audio as well, such as Speech Commands,
containing 1 s audio clips. We can see from Table 1 that duration does not
have much impact on results.

###  5.4 Performance on Smallest Dataset

Learned representations are particularly important for approaching novel tasks
with scarce data. The task containing the fewest number of data instances is
Gunshot Triangulation (Cooper and Shaw, 2020), containing only 88 audio clips.
The original kwmlp performs reasonably well on this task, while in one of our
ablation experiments (Section 6.3, Table 4), we obtain results better than the
first place result in the competition.

###  5.5 Comparative Model Efficiency

Apart from accuracy, it is also important to consider the size and comparative
resource-efficiency of various models in the HEAR benchmark. For instance, for
the Beehive States task (Nolasco et al., 2019), only 11 submissions out of 28
could be evaluated, as the rest exceeded the GPU memory and time restrictions
established by the organizers. Beehive States is the task with the longest
audio clips among all HEAR tasks, at ten minutes long; thus we assume some of
the larger models scaled exponentially and exceeded resource limits.

kwmlp has a total of only 0.424 million parameters. For reference, the
immediate model after kwmlp in terms of size is YAMNet, at 3.8 million
parameters ($\sim 9\times$), while the largest models, combinations of HuBERT,
Wav2Vec2.0, and CREPE in a variety of ways (Wu et al., 2022), reached 1339
million parameters (over 3000$\times$ larger).

###  5.6 Temporal Shift Invariance

Figure 5: Visualization of the linear temporal projection matrix ($G$ from
equation 2) at blocks of different depths (1, 3, 7, 11).

We can observe an interesting phenonmenon if we visualize the weights of the
temporal linear projection (in Figure 5): that they resemble toeplitz
matrices.

This implies that the encoder model has learned a form of shift invariance in
its supervised pretraining task. To be precise, this is temporal shift
invariance, as these weight matrices are responsible for linear temporal
projection. This makes sense upon reviewing the nature of Speech Commands
classification, which is the task KW-MLP is trained on. In an arbitrary input
audio, the target speech keyword can appear at various positions, such as at
the start, the middle, or the end. The words can also have different time
durations, depending on the utterance style. Naturally, a model aiming to
solve the Speech Commands task must learn temporal shift invariance. As a
consequence, it becomes expected for our method to not perform well on
timestamp embedding tasks, which we can observe from the results on DCASE.

##  6 Ablation

For our ablation studies, we performed experiments on eight tasks with
relatively small datasets, which enabled quick experimentation and did not
require much compute resources.

###  6.1 Understanding the Impact of Scene Embedding Algorithms

For HEAR 2021, in kwmlp, we used iterative linear interpolation to generate
scene-level embeddings from timestamp-level embeddings. We were curious to
observe the effect of iterative interpolation as opposed to regular (single)
interpolation. We also made a comparison with a mean-based algorithm, which
makes $N_{S}$ (16) groups (in order) along the time dimension and takes the
average per group before flattening. The results of the three algorithms is
compared in Table 2.

Table 2: Impact of three different scene embedding algorithms Task | Metric | Group | Interpolation
---|---|---|---
Mean | Single | Iterative
Beijing Opera Percussion | Accuracy | 0.894 | 0.902 | 0.911
LibriCount | Accuracy | 0.46 | 0.422 | 0.451
Gunshot Triangulation | Accuracy | 0.911 | 0.917 | 0.932
ESC-50 | Accuracy | 0.334 | 0.325 | 0.367
Mridingham Stroke | Accuracy | 0.967 | 0.971 | 0.969
Mridingham Tonic | Accuracy | 0.945 | 0.942 | 0.942
NSynth Pitch 5h | Pitch Accuracy | 0.448 | 0.462 | 0.44
Chroma Accuracy | 0.498 | 0.516 | 0.48
Speech Commands 5H | Accuracy | 0.974 | 0.976 | 0.976

There is a difference of at most 3.3% between the mean and the iterative
interpolation algorithm, which is not much, but also not insignificant. It can
be seen that generally, the interpolation algorithms out-perform the mean-
based approach for generating scene embeddings. And between iterative
interpolation and single interpolation, iterative interpolation performs
slightly better.

###  6.2 Training a Shallower KW-MLP Encoder

Table 3: Impact of KW-MLP at different encoder depths Task | Metric | KW-MLP Depth
---|---|---
8 | 10 | 12
Beijing Opera Percussion | Accuracy | 0.915 | 0.915 | 0.911
LibriCount | Accuracy | 0.475 | 0.432 | 0.451
Gunshot Triangulation | Accuracy | 0.860 | 0.917 | 0.932
ESC-50 | Accuracy | 0.398 | 0.351 | 0.367
Mridingham Stroke | Accuracy | 0.970 | 0.973 | 0.969
Mridingham Tonic | Accuracy | 0.952 | 0.945 | 0.942
NSynth Pitch 5h | Pitch Accuracy | 0.462 | 0.470 | 0.44
Chroma Accuracy | 0.506 | 0.516 | 0.48
Speech Commands 5H | Accuracy | 0.969 | 0.978 | 0.976

In the HEAR challenge, we used a KW-MLP encoder with a depth of 12. We wanted
to observe whether downstream task performance was proportional to performance
on the pretraining task (Speech Commands V2-35) as well as the depth of the
encoder.

By varying the number of blocks, $L$, to 8 and 10, we trained two additional
encoder models on Speech Commands V2-35, KW-MLP8 and KW-MLP10. For reference,
KW-MLP12 had a top-1 accuracy of 97.56%, while the KW-MLP8 and KW-MLP10 we
later trained had 97.26% and 97.35% respectively.

We then used the evaluation kit provided by HEAR 2021 organizers to evaluate
these new encoders. Our comparative results with KW-MLP with varying number of
blocks can be observed at Table 3.

With fewer number of blocks, the model is forced to learn more compact
representations. Furthermore, the results on other tasks is not proportional
to the performance on the pre-training task. This suggests that the high-level
features of Speech Commands are detrimental to generalization. In the next
section, we attempt to take lower level representations from shallower depths
of KWMLP12.

###  6.3 Generating Embedding from Shallower Depth of KW-MLP

Figure 6: We can extract embeddings of the same size from shallower blocks of the model. Note the dotted lines represent subsequent blocks, i.e. between 4 and 8 there is 6 and 7. Table 4: Generating representations from shallower layers of KW-MLP12 Task | Metric | KW-MLP12 depth
---|---|---
4 | 8 | 12
Beijing Opera Percussion | Accuracy | 0.949 | 0.941 | 0.911
LibriCount | Accuracy | 0.485 | 0.516 | 0.451
Gunshot Triangulation | Accuracy | 0.976 | 0.943 | 0.932
ESC-50 | Accuracy | 0.446 | 0.485 | 0.367
Mridingham Stroke | Accuracy | 0.977 | 0.977 | 0.969
Mridingham Tonic | Accuracy | 0.965 | 0.963 | 0.942
NSynth Pitch 5h | Pitch Accuracy | 0.544 | 0.528 | 0.44
Chroma Accuracy | 0.576 | 0.562 | 0.48
Speech Commands 5H | Accuracy | 0.786 | 0.932 | 0.976

In deep neural networks, the initial couple of layers capture the low-level
features of the data, whereas the final group of layers capture the higher-
level features of data. For example, in images, the shallow layers capture the
edges, while in the last few layers, the model captures high-level features of
the image, such as patterns, objects, and shapes. This phenomenon is also
present in our method. We generated representations from encoder depth 4 and 8
from KW-MLP12, as shown in Figure 6.

From Table 4 we observe that in general, representations from shallower depths
improve our model’s performance, except Speech Commands. In Speech Commands
the encoder model at depth 12 outperforms encoder model at depth 4 and 8. This
is because the model is trained on a Speech Commands V2-35, and thus losing
the learned high-level speech representations hurts performance on Speech
Commands V2-12. On the other datasets, the embeddings from shallower layers
have significantly better performance, as the representations at those layers
are not biased on the Speech Commands dataset. We believe that the
intermediate embeddings at encoder depth 4 and 8 better captured the general
features of audio data compared to the embeddings at depth 12, which are too
strongly geared towards the Speech Commands dataset.

Comparing with the results at the leaderboard (neuralaudio.ai, 2021c), we
obtained better results than the 1st position ranking in Gunshot
Triangulation, Mridingham Stroke and Mridingham Tonic tasks.

##  7 Conclusion and Future Work

Our MLP-based audio representation learning is highly efficient, and performs
quite well in certain downstream tasks. However, it also has sub-par
performance on other tasks. One of the aims of the HEAR Challenge was to
evaluate whether it was possible for models to learn audio representations as
holistic as the human ear. From our results, as well as the results of the
other submissions, this seems to remain an open question, for now.

In order to enable fast pretraining and have an extremely efficient method,
kwmlp has compromised in certain aspects, such as the pretraining dataset or
task, and the model size. In future works, we would like to explore self-
supervised representation learning methods with similar all-MLP architectures
on larger and richer audio corpuses, such as LibriSpeech (Panayotov et al.,
2015), AudioSet (Gemmeke et al., 2017) or Synth1B1 (Turian et al., 2021),
among others. One fact the HEAR challenge has made apparent is that perhaps it
is difficult to learn truly generalized and holistic representations from
solely one type of audio data, such as speech.

We hope our work is helpful to future research on audio representation
learning, and opens up a new avenue of working with all-MLP models for audio
tasks.

## References

  * Anantapadmanabhan et al. (2020) Akshay Anantapadmanabhan, Ashwin Bellur, and Hema A. Murthy.  Mridangam stroke dataset, October 2020.  URL https://doi.org/10.5281/zenodo.4068196.
  * Aytar et al. (2016) Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  _Advances in neural information processing systems_ , 29, 2016.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33:12449–12460, 2020.
  * Berg et al. (2021) Axel Berg, Mark O’Connor, and Miguel Tairum Cruz.  Keyword transformer: A self-attention model for keyword spotting, 2021\.
  * Cheuk et al. (2020) Kin Wai Cheuk, Hans Anderson, Kat Agres, and Dorien Herremans.  nnaudio: An on-the-fly gpu audio to spectrogram conversion toolbox using 1d convolutional neural networks.  _IEEE Access_ , 8:161981–162003, 2020.
  * Cooper and Shaw (2020) Seth Cooper and Steven Shaw.  Gunshots recorded in an open field using iPod Touch devices, August 2020\.  URL https://doi.org/10.5061/dryad.wm37pvmkc.
  * Davis and Mermelstein (1980) Steven Davis and Paul Mermelstein.  Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences.  _IEEE transactions on acoustics, speech, and signal processing_ , 28(4):357–366, 1980.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * Fonseca et al. (2022) Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra.  Fsd50k: An open dataset of human-labeled sound events.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:829–852, 2022.  ISSN 2329-9304.  10.1109/taslp.2021.3133208.  URL http://dx.doi.org/10.1109/taslp.2021.3133208.
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 776–780. IEEE, 2017.
  * Gong et al. (2021a) Yuan Gong, Yu-An Chung, and James Glass.  Ast: Audio spectrogram transformer, 2021a.
  * Gong et al. (2021b) Yuan Gong, Cheng-I Jeff Lai, Yu-An Chung, and James Glass.  Ssast: Self-supervised audio spectrogram transformer.  _arXiv preprint arXiv:2110.09784_ , 2021b.
  * Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.  Conformer: Convolution-augmented transformer for speech recognition, 2020\.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Khan et al. (2021) Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  _ACM Computing Surveys (CSUR)_ , 2021.
  * Kim and Pardo (2018) Bongjun Kim and Bryan Pardo.  Vocal Imitation Set v1.1.3 : Thousands of vocal imitations of hundreds of sounds from the AudioSet ontology, August 2018.  URL https://doi.org/10.5281/zenodo.1340763.
  * Kim et al. (2018) Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello.  Crepe: A convolutional representation for pitch estimation.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 161–165. IEEE, 2018.
  * Kong et al. (2020) Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2880–2894, 2020.
  * Ling et al. (2020) Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff.  Deep contextualized acoustic representations for semi-supervised speech recognition.  In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6429–6433, 2020.  10.1109/ICASSP40776.2020.9053176.
  * Lippmann (1987) R. Lippmann.  An introduction to computing with neural nets.  _IEEE ASSP Magazine_ , 4(2):4–22, 1987.  10.1109/MASSP.1987.1165576.
  * Liu et al. (2021a) Andy T. Liu, Shang-Wen Li, and Hung-yi Lee.  Tera: Self-supervised learning of transformer encoder representation for speech.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:2351–2366, 2021a.  ISSN 2329-9304.  10.1109/taslp.2021.3095662.  URL http://dx.doi.org/10.1109/TASLP.2021.3095662.
  * Liu et al. (2021b) Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le.  Pay attention to mlps, 2021b.
  * Liu et al. (2022) Shuo Liu, Adria Mallol-Ragolta, Emilia Parada-Cabeleiro, Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, and Bjoern W Schuller.  Audio self-supervised learning: A survey.  _arXiv preprint arXiv:2203.01205_ , 2022.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Melas-Kyriazi (2021) Luke Melas-Kyriazi.  Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet, 2021.
  * Mermelstein (1976) Paul Mermelstein.  Distance measures for speech recognition, psychological and instrumental.  _Pattern recognition and artificial intelligence_ , 116:374–388, 1976.
  * Mesaros et al. (2018) A. Mesaros, T. Heittola, E. Benetos, P. Foster, M. Lagrange, T. Virtanen, and M. D. Plumbley.  Detection and classification of acoustic scenes and events: Outcome of the DCASE 2016 challenge.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 26(2):379–393, Feb 2018.  ISSN 2329-9290.  10.1109/TASLP.2017.2778423.
  * Morshed and Ahsan (2021) Mashrur M. Morshed and Ahmad Omar Ahsan.  Attention-free keyword spotting, 2021.
  * Mouton et al. (2021) Coenraad Mouton, Johannes C Myburgh, and Marelie H Davel.  Stride and translation invariance in cnns.  In _Southern African Conference for Artificial Intelligence Research_ , pages 267–281. Springer, 2021.
  * neuralaudio.ai (2021a) neuralaudio.ai.  Hear common api, 2021a.  URL https://neuralaudio.ai/hear2021-rules.html#common-api.  Online.
  * neuralaudio.ai (2021b) neuralaudio.ai.  Hear evaluation kit, 2021b.  URL https://github.com/neuralaudio/hear-eval-kit.  Online.
  * neuralaudio.ai (2021c) neuralaudio.ai.  Hear 2021 neurips challenge, 2021c.  URL https://neuralaudio.ai/hear2021-results.html.  Online.
  * Nolasco et al. (2019) Ines Nolasco, Alessandro Terenzi, Stefania Cecchi, Simone Orcioni, Helen L. Bear, and Emmanouil Benetos.  Audio-based identification of beehive states.  _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2019.  10.1109/icassp.2019.8682981.  URL http://dx.doi.org/10.1109/ICASSP.2019.8682981.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5206–5210. IEEE, 2015.
  * Patterson et al. (1987) Roy D Patterson, Ian Nimmo-Smith, John Holdsworth, and Peter Rice.  An efficient auditory filterbank based on the gammatone function.  In _a meeting of the IOC Speech Group on Auditory Modelling at RSRE_ , volume 2, 1987.
  * Piczak (2015) Karol J. Piczak.  ESC: Dataset for Environmental Sound Classification.  In _Proceedings of the 23rd Annual ACM Conference on Multimedia_ , pages 1015–1018. ACM Press, 2015.  ISBN 978-1-4503-3459-4.  10.1145/2733373.2806390.  URL http://dl.acm.org/citation.cfm?doid=2733373.2806390.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli.  wav2vec: Unsupervised pre-training for speech recognition.  _arXiv preprint arXiv:1904.05862_ , 2019.
  * Schörkhuber and Klapuri (2010) Christian Schörkhuber and Anssi Klapuri.  Constant-q transform toolbox for music processing.  In _7th sound and music computing conference, Barcelona, Spain_ , pages 3–64, 2010.
  * Schörkhuber et al. (2014) Christian Schörkhuber, Anssi Klapuri, Nicki Holighaus, and Monika Dörfler.  A matlab toolbox for efficient perfect reconstruction time-frequency transforms with log-frequency resolution.  In _Audio Engineering Society Conference: 53rd International Conference: Semantic Audio_. Audio Engineering Society, 2014.
  * Tolstikhin et al. (2021) Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  Mlp-mixer: An all-mlp architecture for vision, 2021.
  * Touvron et al. (2021a) Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al.  Resmlp: Feedforward networks for image classification with data-efficient training.  _arXiv preprint arXiv:2105.03404_ , 2021a.
  * Touvron et al. (2021b) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , pages 10347–10357. PMLR, 2021b.
  * Turian et al. (2021) Joseph Turian, Jordie Shier, George Tzanetakis, Kirk McNally, and Max Henry.  One billion audio sounds from gpu-enabled modular synthesis.  _arXiv preprint arXiv:2104.12922_ , 2021.
  * Turian et al. (2022) Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin, and Yonatan Bisk.  Hear 2021: Holistic evaluation of audio representations.  In Joseph Turian, Björn W. Schuller, Dorien Herremans, Katrin Kirchhoff, Paola Garcia Perera, and Philippe Esling, editors, _Proceedings of HEAR 2021: Holistic Evaluation of Audio Representations_ , volume 166 of _Proceedings of Machine Learning Research_. PMLR, 2022.
  * Tzanetakis et al. (2001) George Tzanetakis, Georg Essl, and Perry Cook.  Automatic musical genre classification of audio signals, 2001.  URL http://ismir2001.ismir.net/pdf/tzanetakis.pdf.
  * Valk and Alumae (2021) Jorgen Valk and Tanel Alumae.  Voxlingua107: A dataset for spoken language recognition.  _2021 IEEE Spoken Language Technology Workshop (SLT)_ , Jan 2021.  10.1109/slt48900.2021.9383459.  URL http://dx.doi.org/10.1109/SLT48900.2021.9383459.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * Vygon and Mikhaylovskiy (2021) Roman Vygon and Nikolay Mikhaylovskiy.  Learning efficient representations for keyword spotting with triplet loss.  _Lecture Notes in Computer Science_ , page 773–785, 2021.  ISSN 1611-3349.  10.1007/978-3-030-87802-3_69.  URL http://dx.doi.org/10.1007/978-3-030-87802-3_69.
  * Wang and Oord (2021) Luyu Wang and Aaron van den Oord.  Multi-format contrastive learning of audio representations.  _arXiv preprint arXiv:2103.06508_ , 2021.
  * Warden (2018) Pete Warden.  Speech commands: A dataset for limited-vocabulary speech recognition, 2018\.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Wu et al. (2022) Tung-Yu Wu, Chen-An Li, Tsu-Yuan Hsu, and Tzu-Han Lin.  The ability of self-supervised speech models for audio representations.  In Joseph Turian, Björn W. Schuller, Dorien Herremans, Katrin Kirchhoff, Paola Garcia Perera, and Philippe Esling, editors, _Proceedings of HEAR 2021: Holistic Evaluation of Audio Representations_ , volume 166 of _Proceedings of Machine Learning Research_. PMLR, 2022.  In submission to HEAR 2021.
  * Zhang and Lehtinen (2020) Richard Zhang and Jaakko Lehtinen.  Bilinear downsampling with f.interpolate in pytorch, 2020.  URL https://twitter.com/rzhang88/status/1258222917986312195.  Online.

##  Appendix A Settings and Hyperparameters for KW-MLP

Table 5: Overview of Hyper-Parameters and Settings Training |  | Augmentation
---|---|---
Epochs | 140 |  | # Time Masks | 2
Batch Size | 256 |  | Time Mask Width | [0, 25]
Optimizer | AdamW |  | # Freq Masks | 2
Learning Rate | 0.001 |  | Freq Mask Width | [0, 7]
Warmup Epochs | 10 |  |  |
Scheduling | Cosine |  |  |
Regularization |  | Audio Processing
Label Smoothing | 0.1 |  | Sampling Rate | 16000
Weight Decay | 0.1 |  | Window Length | 30 ms
Block Survival Prob. | 0.9 |  | Hop Length | 10 ms
|  |  | n_mfcc | 40
Model |  |  |
# Blocks, $L$ | 12/10/8 |  |  |
Input Shape | $40\times 98$ |  |  |
Patch Size | $40\times 1$ |  |  |
Dim, $d$ | 64 |  |  |
Dim Proj. | 256 |  |  |
# Classes | 35 |  |  |