# SLICER: Learning universal audio representations using low-resource self-
supervised pre-training

###### Abstract

We present a new Self-Supervised Learning (SSL) approach to pre-train encoders
on unlabeled audio data that reduces the need for large amounts of labeled
data for audio and speech classification. Our primary aim is to learn audio
representations that can generalize across a large variety of speech and non-
speech tasks in a low-resource un-labeled audio pre-training setting. Inspired
by the recent success of clustering and contrasting learning paradigms for
SSL-based speech representation learning, we propose SLICER (Symmetrical
Learning of Instance and Cluster-level Efficient Representations) which brings
together the best of both clustering and contrasting learning paradigms. We
use a symmetric loss between latent representations from student and teacher
encoders and simultaneously solve instance and cluster-level contrastive
learning tasks. We obtain cluster representations online by just projecting
the input spectrogram into an output subspace with dimensions equal to the
number of clusters. In addition, we propose a novel mel-spectrogram
augmentation procedure _k-mix_ , based on mixup [1], which does not require
labels and aids unsupervised representation learning for audio. Overall,
SLICER achieves state-of-the-art results on the LAPE Benchmark [2],
significantly outperforming DeLoRes-M and other prior approaches, which are
pre-trained on $10\times$ larger of unsupervised data. We will make all our
codes available on GitHub111We will release the code after paper acceptance.

Index Terms—  audio, speech, self-supervision

##  1 Introduction

SSL (self-supervised learning) is increasingly used to obtain good performance
for all modalities corresponding to speech [3], vision ,[4, 5] and text [6].
In practice, SSL-based speech representation learning has achieved state-of-
the-art results in a variety of speech tasks [7] like Automatic Speech
Recognition (ASR), Phoneme Recognition (PR), etc. However, current SSL-based
methods may fail to perform well on tasks that do not involve recognizing
phonetic, semantic, or syntactic information in speech [2], like acoustic
scene classification. We hypothesize that this might be due to models learning
an implicit language model through solving some form of Masked Acoustic
Modeling (MAM) task with individual speech frames (for eg. contrastive
learning [3] or clustering [8]). Learning SSL-based universal audio
representations (also known as general-purpose audio representation learning
in literature) still remains a relatively nascent area of research and differs
from speech representation learning by requiring to learn better global-scale
features.

Main Contributions: We present SLICER, a new SSL algorithm for learning
general-purpose audio representations from un-labeled audio that
simultaneously learns instance-level discriminative features and performs
online clustering in a one-stage and end-to-end manner without any extra
overhead like an offline clustering step. In general, offline clustering does
not scale well with large-scale datasets as they need to perform clustering on
the entire dataset at each epoch [9]. In contrast to prior methods, SLICER
learns a deep network that outputs a matrix where rows and columns correspond
to the instance and cluster representations, respectively (see Fig. 2). We
achieve this by projecting the input audio log-mel-spectrogram into an output
space that is equal to the number of desired clusters centroids. We build this
on top of the student and momentum-teacher learning paradigm, where one out of
two identical encoders is updated from the momentum of the other [2, 5].
Moreover, for instance-level learning, we use a symmetrical cross-contrastive
loss where each encoder calculates a separate loss by sampling negatives for
each of the positives from the output of the other. A pictorial representation
of our setup can be seen in Fig. 1.

Fig. 1: Illustration of SLICER. SLICER has 2 encoder networks, a student and a
momentum-teacher. An audio sample is first converted to its log-mel-
spectrogram representation before passing it through _k-mix_ and RRC
augmentations to generate 2 randomly augmented views of the audio sample. Then
both these samples are passed through both the encoder networks, and we
calculate a cross-contrastive loss between the student output for the first
view and the teacher output of the second view and vice-versa. Additionally,
the student solves a cluster-level contrastive loss by considering each column
in the encoder output as the probability distribution of an instance belonging
to each cluster.

##  2 Related Work

Recently, researchers have made considerable progress in devising new and
better SSL algorithms for learning general-purpose audio representations.
These algorithms solve contrastive [2, 10], clustering, [9] or reconstruction
tasks [11]. To the best of our knowledge, no prior work has successfully
solved both contrastive and clustering SSL tasks. Through SLICER, we bring the
best of both worlds in an efficient and non-trivial way by solving both
contrastive and clustering tasks simultaneously. Unlike MAM in speech
representation learning, most work in SSL for audio representation learning
tries to maximize agreement between differently augmented views of the same
audio sample [2, 12]. This means data augmentations play a critical role in
terms of the effectiveness of the SSL method. Though various methodologies and
libraries exist for augmenting raw audio in the time domain [13], there is
relatively less work on devising better and stronger augmentation schemes for
mel-spectrograms. [12] proposed the use of Random Resized Crop (RRC) and a
modified version of mixup, which works in an unsupervised setting without
labels. Recently they also proposed the Random Linear Fader, which randomly
applies a linear change in volume to the entire audio segment to simulate the
approaching or passing away of a sound source or fade-in/out. Our proposed _k-
mix_ builds on the unsupervised mixup setup proposed by [12].

##  3 Methodology

###  3.1 Problem Formulation

Let’s suppose X is an unlabeled dataset of size, $J$ where X =
$\\{x_{1},\cdots,x_{j},\cdots,x_{J}\\}$ . Here $J$ = 0.25 million, following
the exact pre-training setup in the LAPE Benchmark [2]. Also let
$\textbf{D}^{t}$ be the task-specific labeled dataset for task $t$ of size
$I$. $\textbf{D}^{t}$ =
$\\{(x^{t}_{1},y^{t}_{1}),\cdots,(x^{t}_{i},y^{t}_{i}),\cdots,(x^{t}_{I},y^{t}_{I})\\}$
and $y$ is the corresponding label for audio sample $x$. Our aim here is to
use X to pre-train our feature encoder using an SSL algorithm and then fine-
tune our model on $\textbf{D}^{t}$ with supervision, keeping the feature
encoder weights frozen.

###  3.2 SLICER: Instance-Level Contrastive Learning

In this section, we describe our instance-level cross-contrastive loss
formulation for SLICER. The basic idea of contrastive learning is to identify
the true latent representation among of set of distractors or negatives.
Following [2], instance-level contrastive learning aims at maximizing the
agreement between differently augmented views $x^{a}_{i}$ and $x^{b}_{i}$ of
the same audio sample $x_{i}\in X$, where $X$ is a batch of size $N$ indexed
by $\\{0,\cdots,i,\cdots,N-1\\}$. Formally put, we pass each sample $x_{i}$ in
a batch $X$ through a set of augmentations $A$ with some degree of randomness
to produce $x^{a}_{i}\in X^{a}$ and $x^{b}_{i}\in X^{b}$. Now each of $X^{a}$
and $X^{b}$ is passed through both the student $f(.)$ and teacher encoders
$h(.)$ to obtain embeddings
$f(X^{a}),f(X^{b}),h(X^{a}),h(X^{b})\in\mathbb{R}^{N\times C}$ and individual
instances form rows in the $\mathbb{R}^{N\times C}$ space. We denote these
instances as
$f(x^{a}_{i}),f(x^{b}_{i}),h(x^{a}_{i}),h(x^{b}_{i})\in\mathbb{R}^{C}$. As
mentioned earlier, the SLICER learning paradigm is comprised of two identical
student and teacher encoders, where the teacher is updated based on the
momentum-based moving average weights of the student. Finally, we calculate
the InfoNCE loss as follows:

$\mathcal{L}^{\text{i}}_{\text{InfoNCE}}(f,h)=-\log(\\\
\frac{\exp\left(f\left(x^{a}_{i}\right)\cdot
h\left(x^{b}_{i}\right)/\tau\right)}{\exp\left(f\left(x^{a}_{i}\right)\cdot
h\left(x^{b}_{i}\right)/\tau\right)+\sum_{i=0}^{K}\exp\left(f\left(x^{a}_{i}\right)\cdot
h\left(\tilde{x_{i}}\right)/\tau\right)})$ (1)

While ($f(x^{a}_{i})$,$h(x^{b}_{i})$) make a positive pair, there are $K$
negative pairs between $x^{a}_{i}$ and $\tilde{x_{i}}$ where each
$\tilde{x_{i}}$ is a randomized augmentation of a different audio sample
sampled from $h(x^{b})$. Finally we optimize a symmetric cross-contrastive
loss $\mathcal{L}^{\text{i}}_{\text{InfoNCE}}$ between the student and the
teacher $\mathcal{L}^{\text{i}}_{\text{InfoNCE}}$ =
$\mathcal{L}^{\text{i}}_{\text{InfoNCE}}(f,h)$ \+
$\mathcal{L}^{\text{i}}_{\text{InfoNCE}}(h,f)$ in addition to the cluster-
level contrastive loss mentioned in the next section.

###  3.3 SLICER: Cluster-Level Contrastive Learning

In this section, we describe our cluster-level contrastive loss formulation
for SLICER. As mentioned in the previous section, both our student and teacher
networks project the batch of spectrograms into an $N\times C$ dimensional
space. Here $C$ represents the number of cluster centroids indexed by
$\\{0,\cdots,c,\cdots,C-1\\}$ and $N$ represents the number of samples in the
batch indexed by $\\{0,\cdots,i,\cdots,N\\}$. Formally put, if
$f(X^{a})\in\mathbb{R}^{N\times C}$ is the output of the student for the first
augmentation scheme, $f(X^{a})_{n,c}$ can be interpreted as the probability of
sample $n$ being assigned to cluster $c$, which can also be interpreted as the
“soft label” for $n$. Thus, we solve the contrastive learning task on cluster
representations in the column space of $f(X^{a})$, where we regard each column
as the cluster distribution over each instance. The contrastive learning task
tries to solve a task like its instance-level counterpart, where we try to
make cluster representations invariant to augmentations applied over the batch
of samples. However, differences include the loss being non-symmetrical and
only calculated on the output representation space of the student. Thus,
considering $y^{a}_{c},y^{b}_{c}\in\mathbb{R}^{N}$ to be the column
representations for $f(X^{a}),f(X^{b})$, respectively, we can formulate our
contrastive loss as follows:

$\mathcal{L}^{\text{c}}=-\log(\frac{\exp\left(y^{a}_{c}\cdot
y^{b}_{c}/\tau\right)}{\exp\left(y^{a}_{c}\cdot
y^{b}_{c}/\tau\right)+\sum_{c=0}^{K}\exp\left(y^{a}_{c}\cdot\tilde{y_{c}}/\tau\right)})$
(2)

where $\tilde{y_{c}}$ is a random negative from the batch. Thus our final loss
to be optimized is $\mathcal{L}_{\text{InfoNCE}}$ =
$\mathcal{L}^{\text{i}}_{\text{InfoNCE}}$ \+
$\mathcal{L}^{\text{c}}_{\text{InfoNCE}}$.

##  4 Implementation Details

###  4.1 Data Augmentation

Contrastive learning has historically been known to depend on the quality of
augmentations, and for our setup, we use Random Resized Crop (RRC) and _k-mix_
as our set of augmentation functions $A$. In the next subsection, we describe
in detail our proposed _k-mix_ algorithm.

Table 1: Result comparison of various SSL methods with SLICER on the _linear
evaluation setup_ with frozen encoder. The best results for each task are
presented in bold. “$-$” signifies that the results were not reported for the
task in the original paper.

DT | COLA | BYOL-A | SimCLR | DECAR-V1 | DECAR-V2 | DeLoRes-S | MoCo | DeLoRes-M | SLICER
---|---|---|---|---|---|---|---|---|---
Speech |  |  |  |  |  |  |  |  |
SC-V1 | 71.7 | $-$ | 77.3 | 82.3 | 91.6 | 86.1 | $93.6$ | $94.0$ | 94.8
SC-V2(12) | $-$ | 91.0 | 77.2 | 83.0 | 90.6 | 85.4 | $93.2$ | $93.3$ | 94.2
SC-V2(35) | 62.4 | 92.2 | 66.0 | 73.6 | 87.2 | 80.0 | $89.3$ | $89.7$ | 90.4
LBS | 100.0 | $-$ | 89.0 | 91.0 | 92.5 | 90.0 | $95.5$ | $95.7$ | 95.7
VC | 29.9 | 40.1 | 28.9 | 25.6 | 33.0 | 31.2 | $42.5$ | $45.3$ | 49.4
IC | $-$ | $-$ | 59.8 | 63.2 | 65.2 | 60.7 | $65.1$ | $65.2$ | 66.4
VF | 71.3 | 90.2 | 69.2 | 74.1 | 78.2 | 76.5 | $87.3$ | $88.0$ | 89.9
Non-Speech |  |  |  |  |  |  |  |  |
NS | 63.4 | 74.1 | 61.3 | 70.7 | 69.8 | 66.3 | $74.7$ | $75.0$ | 76.3
BSD | 77.0 | $-$ | 85.2 | 87.7 | 88.5 | 86.7 | $89.0$ | $89.6$ | 90.0
TUT | $-$ | $-$ | 52.4 | 62.5 | 64.6 | 58.6 | $66.7$ | $65.7$ | 66.8
US8K | $-$ | 79.1 | 69.1 | 70.1 | 73.2 | 71.2 | $81.2$ | $82.7$ | 83.2
Average | $-$ | $-$ | 66.9 | 71.2 | 75.8 | 72.1 | $79.8$ | $80.4$ | 81.6

####  4.1.1 k-mix

Fig. 2: Illustration of the _k-mix_ augmentation algorithm for spectrograms.
_A_) We first quantize data using K-means. _B_) Based on the current sample
($x$) and its centroid ($C_{3}$), we sort the queue (Q) using C, and sample
uniformly from the first $r$ samples in sorted Q.

In this section, we describe our novel augmentation algorithm _k-mix_. _k-mix_
is inspired by the original mixup [1] proposed for images and its modification
for log-mel-spectrograms augmentation in the absence of labels proposed by
[12]. With a log-mel-spectrogram as an input, mixup mix past randomly selected
input audio in a small ratio [12]. As a result, added audio becomes a part of
the background sound in the mixed audio. Thus, the mixup augmentation
operation is defined as
$\tilde{x}_{i}=\log\left((1-\lambda)\exp\left(x_{i}\right)+\lambda\exp\left(x_{k}\right)\right)$
, where $x_{k}$ is a mixing counterpart and the mixing ratio $\lambda$ is
sampled from uniform distribution $U(0.0,\alpha)$ instead of from a beta
distribution in the original mixup. In addition, $\alpha$ is a mixing ratio
hyper-parameter that controls the degree of contrast between the resulting two
mixed outputs. $x_{k}$ is randomly chosen from a memory bank, which is a FIFO
queue storing past inputs; this is where k-mix tries to sample audio from the
memory bank, which would result in a stronger augmentation.

We acknowledge that the unlabeled audio dataset used for SSL might not be from
largely diverse sources, and thus we hypothesize that randomly sampling audio
from a FIFO queue for background noise might result in weak augmentations if
the audio is from the same hidden class or source. Thus, kmix tries to sample
audios from the queue, and these audio samples are further apart in the
Euclidean space, which can be identified in an unsupervised manner using
clustering. Formally put, we first train a simple k-means clustering algorithm
on about 10% of our unlabeled AudioSet dataset to obtain $k$ cluster
centroids, where $k_{i}\in R^{128}$ (spectrogram with frequency dimension 128
mean-pooled across time axis). The $k$ cluster centroids are then used to
calculate a $k\times k$ matrix $\mathbf{C}$ where $c_{i,j}\in C$ represents
the euclidean distance between the $i^{th}$ and $j^{th}$ cluster centroids.
Thus, as shown in Fig. 2, when a new audio sample $x$ is to be augmented, we
first find the closest cluster centroid among $k$, to $x$ and all other
samples in the FIFO queue. Then using $\mathbf{C}$, we sort the FIFO queue
$\mathbf{Q}$ in descending order of distance to the sample $x$ with respect to
the cluster centroids that $x$ and samples in $\mathbf{Q}$ belong to. Finally,
our sampler $\mathbf{S}$ randomly samples an audio segment from the first $r$
samples of the sorted FIFO queue to augment our incoming audio sample.

###  4.2 Experimental Setup

The primary focus of our work is to devise better SSL algorithms for audio
representation learning, and thus we do not focus on devising better
architectures or measuring the effect of SSL approaches across different
architectures. Thus following much of prior art [2, 12], for both our encoders
(student and teacher), we borrow the simple yet powerful architecture proposed
by [12]. For more details, we refer our readers to [2, 12]. For a fair
comparison, except COLA, which uses EfficientNet, in Table 1 (BYOL-A also uses
the same encoder as [12]), we reproduce results for SSL methodologies in
literature with our encoder if they were originally presented with a different
architecture. For _k-mix_ we find optimal values of $k$ = 128, $r$ = 128, and
length of $\mathbf{Q}$ = 2048. For SSL pre-training, we use an embedding
dimension $C$ = 256, a learning rate of $3e^{-4}$, a batch size of 1024, and
train it for 100 epochs.

###  4.3 Datasets

In our experiments, we use the exact same upstream and downstream training
setups proposed by LAPE [2]. For SSL-based pre-training, we use a balanced
subset of 10% of the complete AudioSet (0.2 million) and the FSD50K [14]. For
downstream tasks (DT), we evaluate our learned representations on LibriSpeech
(LBS) [15] and VoxCeleb (VC) [16] for speaker identification, Speech Commands
(SC) v1 and v2 [17] for keyword spotting, VoxForge (VF) [18] for language
identification, IEMOCAP (IC) [19] for speech emotion recognition, NSynth [20]
for , TUT Urban [21] and US8K [22] for acoustic event classification and
finally Bird Song Detection (BSD) [23].

##  5 Results and Result Analysis

As we see in Table 1, SLICER outperforms all other approaches in literature by
a significant margin. Results of COLA and BYOL-A were borrowed from their
original papers. SimCLR was proposed as the pre-training approach in [24]. We
attribute the gap in results from the original paper to the powerful encoder
proposed in the paper. However, as stated earlier, measuring the effect of
change in encoders is beyond the scope of this paper. MoCo inspired from [5],
can be viewed as SLICER without symmetric cross-contrastive instance-level
learning and cluster-level contrastive learning. Table 2 shows ablations on
various novel components in SLICER. Starting from MoCo proposed in [2], we get
0.4% average boost by first introducing symmetricity in a cross-contrastive
setting, followed 1.2% on adding cluster-level contrastive and finally another
0.4% with _k-mix_.

Table 2: Ablation on various components in SLICER. Method | Avg. Accuracy
---|---
MoCo | 79.8
\+ symmetric cross-contr. | 80.2
\+ cluster contr. (SLICER) | 81.2
\+ _k-mix_ | 81.6

##  6 Conclusion

In this paper, we propose SLICER, a novel methodology to learn general-purpose
audio representations from un-labeled data on low-resource data regimes.
SLICER significantly outperforms all other approaches in literature, sometimes
even systems trained on $10\times$ more unlabeled data than our setup. We also
propose _k-mix_ a new log-mel-spectrogram augmentation algorithm that improves
over the widely used [1].

## References

  * [1] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz,  “mixup: Beyond empirical risk minimization,”  arXiv preprint arXiv:1710.09412, 2017.
  * [2] Sreyan Ghosh, Ashish Seth, and S Umesh,  “Decorrelating feature spaces for learning general-purpose audio representations,”  IEEE Journal of Selected Topics in Signal Processing, pp. 1–13, 2022.
  * [3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  NeurIPS 2020, vol. 33, pp. 12449–12460.
  * [4] Grill et al.,  “Bootstrap your own latent-a new approach to self-supervised learning,”  NeurIPS 2020, vol. 33, pp. 21271–21284.
  * [5] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick,  “Momentum contrast for unsupervised visual representation learning,”  in IEEE CVPR 2020, pp. 9729–9738.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [7] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.,  “Superb: Speech processing universal performance benchmark,”  arXiv preprint arXiv:2105.01051, 2021.
  * [8] Hsu et al.,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM TASLP, vol. 29, pp. 3451–3460, 2021.
  * [9] Ghosh et al.,  “Deep clustering for general-purpose audio representations,”  arXiv preprint arXiv:2110.08895, 2021.
  * [10] Aaqib Saeed, David Grangier, and Neil Zeghidour,  “Contrastive learning of general-purpose audio representations,”  in IEEE ICASSP 2021, pp. 3875–3879.
  * [11] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  in IEEE ICASSP 2020, pp. 6419–6423.
  * [12] Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino,  “Byol for audio: Self-supervised learning for general-purpose audio representation,”  in IEEE IJCNN 2021, pp. 1–8.
  * [13] Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux,  “Data augmenting contrastive learning of speech representations in the time domain,”  in IEEE SLT 2021, pp. 215–222.
  * [14] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra,  “Fsd50k: an open dataset of human-labeled sound events,”  IEEE/ACM TASLP, vol. 30, pp. 829–852, 2021.
  * [15] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An asr corpus based on public domain audio books,”  in IEEE ICASSP 2015, pp. 5206–5210.
  * [16] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Voxceleb: A large-scale speaker identification dataset,”  ISCA Interspeech 2017.
  * [17] Pete Warden,  “Speech commands: A dataset for limited-vocabulary speech recognition,” 2018.
  * [18] Voxforge.org,  “Free speech… recognition (linux, windows and mac) - voxforge.org,” accessed 06/25/2014.
  * [19] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan,  “Iemocap: Interactive emotional dyadic motion capture database,”  LREC 2008, vol. 42, no. 4, pp. 335–359.
  * [20] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan,  “Neural audio synthesis of musical notes with wavenet autoencoders,”  in ICML 2017, pp. 1068–1077.
  * [21] Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen,  “A multi-device dataset for urban acoustic scene classification,” 2018\.
  * [22] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello,  “A dataset and taxonomy for urban sound research,”  in ACM MM 2014, 2014, p. 1041–1044.
  * [23] Dan Stowell, Michael D Wood, Hanna Pamuła, Yannis Stylianou, and Hervé Glotin,  “Automatic acoustic detection of birds through deep learning: the first bird audio detection challenge,”  Methods in Ecology and Evolution, vol. 10, no. 3, pp. 368–380, 2019\.
  * [24] Luyu Wang, Pauline Luc, Yan Wu, Adria Recasens, Lucas Smaira, Andrew Brock, Andrew Jaegle, Jean-Baptiste Alayrac, Sander Dieleman, Joao Carreira, et al.,  “Towards learning universal audio representations,”  in IEEE ICASSP 2022, pp. 4593–4597.