# Self-Supervised Speech Representation Learning:
A Review

Abdelrahman Mohamed*, Hung-yi Lee*, Lasse Borgholt*, Jakob D. Havtorn*, Joakim
Edin, Christian Igel
Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath,
Shinji Watanabe *equal contribution, order is random, remaining sorted
alphabetically.Abdelrahman Mohamed and Shang-Wen Li are with Meta (e-emails:
<EMAIL_ADDRESS>shangwel@fb.com).Hung-yi Lee is with the Department of Electrical
Engineering and the Department of Computer Science & Information Engineering
of National Taiwan University (e-mail: hungyilee@ntu.edu.tw).Lasse Borgholt is
with Corti AI and the Department of Computer Science, University of
Copenhagen, Denmark (e-mail: lb@corti.ai).Jakob D. Havtorn and Lars Maaløe are
with Corti AI and the Department of Applied Mathematics and Computer Science,
Technical University of Denmark, Denmark (e-mails<EMAIL_ADDRESS>lm@corti.ai).Joakim Edin is with Corti AI, Denmark (e-mail:
je@corti.ai).Christian Igel is with the Department of Computer Science,
University of Copenhagen, Denmark (e-mail: igel@di.ku.dk).Katrin Kirchhhoff is
with AWS AI Labs, Amazon, Seattle, 98121, USA (email:
katrinki@amazon.com).Karen Livescu is with the Toyota Technological Institute
at Chicago, Chicago, IL 60615 USA (e-mail: klivescu@ttic.edu).Shinji Watanabe
is with the Language Technologies Institute, Carnegie Mellon University,
Pittsburgh, PA 15213 USA (e-mail: shinjiw@ieee.org).

###### Abstract

Although supervised deep learning has revolutionized speech and audio
processing, it has necessitated the building of specialist models for
individual tasks and application scenarios. It is likewise difficult to apply
this to dialects and languages for which only limited labeled data is
available. Self-supervised representation learning methods promise a single
universal model that would benefit a wide variety of tasks and domains. Such
methods have shown success in natural language processing and computer vision
domains, achieving new levels of performance while reducing the number of
labels required for many downstream scenarios. Speech representation learning
is experiencing similar progress in three main categories: generative,
contrastive, and predictive methods. Other approaches rely on multi-modal data
for pre-training, mixing text or visual data streams with speech. Although
self-supervised speech representation is still a nascent research area, it is
closely related to acoustic word embedding and learning with zero lexical
resources, both of which have seen active research for many years. This review
presents approaches for self-supervised speech representation learning and
their connection to other research areas. Since many current methods focus
solely on automatic speech recognition as a downstream task, we review recent
efforts on benchmarking learned representations to extend the application
beyond speech recognition.

###### Index Terms:

Self-supervised learning, speech representations.

##  I Introduction

Figure 1: Framework for using self-supervised representation learning in
downstream applications

Over the past decade, deep learning approaches have revolutionized speech
processing through a giant leap in performance, enabling various real-world
applications. Supervised learning of deep neural networks has been the
cornerstone of this transformation, offering impressive gains for scenarios
rich in labeled data [1, 2, 3]. Paradoxically, this heavy reliance on
supervised learning has restricted progress in languages and domains that do
not attract the same level of labeling investment.

To overcome the need for labeled data, researchers have explored approaches
that use unpaired audio-only data to open up new industrial speech use-cases
and low-resource languages [4, 5, 6]. Inspired by how children learn their
first language through listening and interacting with family and surroundings,
scientists seek to use raw waveforms and spectral signals to learn speech
representations that capture low-level acoustic events, lexical knowledge, all
the way to syntactic and semantic information. These learned representations
are then used for target downstream applications requiring a minimal number of
labeled data [7, 8, 9]. Formally, representation learning refers to algorithms
for extracting latent features that capture the underlying explanatory factors
for the observed input [9].

Representation learning approaches are generally considered examples of
unsupervised learning, which refers to the family of machine learning methods
that discover naturally occurring patterns in training samples for which there
are no pre-assigned labels or scores [10]. The term “unsupervised” is used to
distinguish this family of methods from “supervised” approaches, which assign
a label to each training sample, and “semi-supervised” approaches, which
utilize a small number of training samples with labels to guide learning using
a larger volume of unlabeled samples. Examples of unsupervised learning
techniques include k-means clustering [11], mixture models [12], autoencoders
[13], and non-negative matrix factorization [14]. Self-supervised learning
(SSL) is a fast-growing subcategory of unsupervised learning approaches, which
are techniques that utilize information extracted from the input data itself
as the label to learn representations useful for downstream tasks.  For
example, unsupervised k-means clustering doesn’t adhere to this definition of
self-supervision since it iteratively minimizes the within-cluster variance
during learning. In this review, we focus on self-supervised learning
approaches.

Figure 1 outlines self-supervised representation learning in relation to
downstream applications. There are two stages in this framework. In the first
stage, we use SSL to pre-train a representation model, also called an upstream
model or a foundation model. In the second stage, downstream tasks use either
the learned representation from the frozen model, or fine-tune the entire pre-
trained model in a supervised phase [15]. Automatic speech recognition (ASR)
and speaker identification (SID) are examples of downstream applications in
Figure 1.

It is considered desirable for learned speech representations to be
disentangled, invariant, and hierarchical. Since spoken utterances contain
much richer information than the corresponding text transcriptions—e.g.,
speaker identity, style, emotion, surrounding noise, and communication channel
noise—it is important to learn representations that disentangle these factors
of variation. Furthermore, invariance of the learned features to changes in
background noise and in the communication channel ensures stability with
respect to downstream application scenarios. Learning feature hierarchies at
the acoustic, lexical, and semantic levels supports applications with
different requirements. For instance, whereas a speaker identification task
benefits from a low-level acoustic representation, a speech translation task
requires a more semantic representation of the input utterance.

Due to the popularity of SSL, reviews have been published about the technology
in general [16, 17, 18] as well as its application to natural language
processing (NLP) [19, 20, 21, 22] and computer vision (CV) [23]. Recently, a
brief overview with a general focus on speech representation learning was
published [24]. However, none of these overviews focus exclusively on SSL for
speech processing. Since the speech signal differs greatly from image and text
inputs, many theories and technologies have been developed to address the
unique challenges of speech. One review addresses speech representation
learning based on deep learning models [25], but does not address recent
developments in self-supervised learning. This motivates this overview of
speech SSL.

The structure of this paper is arranged as follows. Section II briefly reviews
the history of speech representation learning, and Section III reviews current
speech SSL models. Section IV surveys SSL datasets and benchmarks, and
discusses and compares results from different works. Section V analyzes
successful SSL approaches and offers insights into the importance of
technological innovations. Section VI reviews zero-resource downstream tasks
that utilize SSL. Finally, Section VII summarizes the paper and suggests
future research directions.

##  II Historical Context of Representation Learning

In this section we present the historical background of the current surge in
self-supervised representation learning methods in the context of two previous
waves of research work in the 1990s and 2000s. The discussed approaches go
beyond speech to describe the overall landscape of machine learning
development during the past few decades.

###  II-A Clustering and mixture models

Initial research in learning latent speech and audio representations involved
simple models in which the training data likelihood was optimized directly or
via the expectation–maximization (EM) algorithm.

Early work used simple clustering methods. For example, in work such as [26,
27], word patterns were clustered semi-automatically using techniques such as
k-means, after which isolated words were recognized by finding the training
cluster closest to the test data.

Through time, modeling techniques improved such that subword units were
represented by Gaussian mixture models (GMMs) [28], which facilitated the
modeling of more variability in the input data. GMMs were first built for
context-independent phonemes; state-clustering algorithms [29] then resulted
in GMMs for context-dependent phonemes. Each latent component of these mixture
models acted as a template of a prototypical speech frame, making it difficult
to handle large volumes of data with diverse characteristics. Furthermore,
dynamical models like hidden Markov models (HMMs) [30] allowed for the
processing of continuous speech rather than just isolated word recognition.
These generative GMM and HMM models were trained by maximizing the likelihood
of data given the model, which could be accomplished in either an unsupervised
or a supervised manner.

Another line of research focused on extracting speech features from generative
models. The main objective here was to render the knowledge learned by
generative models accessible to discriminative downstream classifiers, or to
map variable-length sequences to fixed-length representations. Feature vectors
were derived from the parameters of trained GMM models. In the case of Fisher
vectors, the features were the normalized gradients of the log-likelihood with
respect to the model parameters (mixture weights, means, and variances) of the
Gaussian mixtures. An extension of this approach (likelihood ratio score
space) used the derivative of the log-likelihood ratio of two models, e.g., a
background model and a foreground model. Examples of their use in speech
processing include speech recognition [31, 32] and speaker recognition [33].
Subsequent techniques in speaker and language verification [34, 35] similarly
extracted parameters (concatenated means) from trained background GMMs as
representations that were then combined with low-rank projections of
speaker/session- or language-specific vectors.

###  II-B Stacked neural models

More recently, representation learning has seen a shift of focus towards
neural models, which, compared to GMMs and HMMs, offer distributed
representations with more capacity to model diverse input signals into
efficient latent binary codes. Examples of early techniques include restricted
Boltzmann machines (RBM) [15], denoising autoencoders [36], noise contrastive
estimation (NCE) [37], sparse coding [38, 39, 40], and energy-based methods
[41]. Many of these techniques have also been applied to CV and NLP problems,
which provided inspiration for their application to speech.

Higher-capacity neural models were achieved by stacking several neural network
layers to build progressively higher-level concept representations. However,
these deeper networks also increased the training complexities. For example,
approximate training methods such as contrastive divergence [42] were a
practical technique to streamline RBM training. Furthermore, deep networks had
non-convex objective functions, which often resulted in long training times
compared to GMMs, which are trained using full batches instead of mini-batch
learning.

###  II-C Learning through pretext task optimization

A more recent trend is learning networks that map the input to desired
representations by solving a pretext task. Such studies have several
characteristics: (1) All layers are trained end-to-end to optimize a single
pretext task instead of relying on layer-wise pre-training (2) Past stacked
networks typically had only a few layers, but very deep networks with more
than ten layers are now common. (3) It is common to evaluate a representation
model on a wide range of tasks. For example, in NLP, a representation model is
usually assessed on GLUE, which comprises nine tasks [43], whereas in speech,
a representation model can be evaluated on SUPERB, which comprises ten tasks
[44], as described in detail in Section IV-E.

The cornerstone of this third wave is the design of a pretext task, which
allows the model to efficiently leverage knowledge from unlabeled data. The
pretext task should be challenging enough for the model to learn high-level
abstract representations and not be so easy as to encourage the exploitation
of low-level shortcuts. Early breakthroughs included end-to-end learning of
deep neural architectures via pretext tasks for restoring the true color of
black-and-white images [45], joint learning of latent representations and
their cluster assignments [46], and the prediction of the relative positions
of image patches [47]. Other popular approaches include variational
autoencoders (VAEs) [48, 49]. While typical autoencoders learn data
representations using unsupervised objectives by reconstructing the input
after passing it through an information bottleneck, VAEs estimate a neural
model of a probability density function (pdf) that approximates the unknown
“true” distribution of the observed data, for which we only have access to
independently identically distributed (iid) samples. It is also important to
mention dynamical VAEs [50], which is an extension of VAE for sequential data
such as speech.

In the SSL context, a pretext task related to autoencoding is to generate an
object from its partial information. Such tasks are widely used in NLP, for
example, using the previous tokens in a sentence to predict the next token
such as in ELMo [51], the GPT series [52], and Megatron [53], or predicting
the masked tokens in a sentence such as with the bidirectional encoder
representations from Transformers (BERT) series [54, 55]. Another common
pretext task in the third wave is contrastive learning [56], in which a model
learns to identify a target instance from a set of negative samples. This
approach has become especially popular in the CV context [57, 58, 59, 60]. In
this survey, we will mainly focus on techniques for pretext task optimization
for speech processing, and discuss these techniques in detail in Section III.

###  II-D Other related work

A closely related area of research that is not covered in this review is semi-
supervised pre-training methods such as pseudo-labeling (that is, self-
training). Pseudo-labeling (PL) relies on a supervised teacher model to label
a large volume of speech-only data, which is then used to augment the initial
labeled data to train a student model [4, 5, 6, 61]. PL has been successful
and widely adopted in the speech community since the 1990s. Other proposed
variations of PL include augmenting speech-only data with noise to improve
robustness, iterating over the PL process to improve teacher labeling quality,
and training student models with more parameters than their original teachers
to capture the complexities in vastly larger speech-only data [62, 63, 64].
Both SSL and PL leverage unlabeled speech-only data. One distinguishing factor
in PL is the utilization of supervised data for a specific task during model
pre-training, which limits the model’s focus to a single (or at best a few)
downstream tasks. SSL, in turn, is an attempt to learn task-agnostic
representations to benefit a wide range of tasks.

Transfer learning (TL) is another closely related area of research for pre-
training speech models. TL transfers knowledge captured by models trained on
one task to different but related tasks [65]. The past few decades have seen
active research on TL and its extension to multitask learning for more general
representations. Multilingual and cross-lingual supervised models have proven
superior in low-resource speech recognition tasks [66]. SSL can be regarded as
a type of TL because knowledge learned from pre-training is used for different
downstream tasks. This survey paper focuses on SSL, and not all TL
technologies for speech. One survey indeed addresses TL for speech processing
[67] but does not include current SSL technologies for speech.

##  III Speech Representation Learning Paradigms

Due to the characteristics of speech, SSL pretext tasks developed for CV and
NLP may not directly apply to speech. Below we summarize the characteristics
of speech as compared to CV and NLP.

  * •

Speech is a sequence. Unlike CV, in which an image usually has a fixed size
representation, it is natural to represent a speech utterance as a variable-
length sequence. Therefore, pretext tasks developed for CV cannot generally be
directly applied to speech.

  * •

Speech is a long sequence without segment boundaries. Both text and speech can
be represented as sequences. From this viewpoint, it is natural to apply
learning approaches developed for text directly to speech. In NLP, morpheme-
like tokens are widely used as sequence units in pre-training. The standard
BERT takes 512 morpheme-like tokens as input, usually covering a paragraph
including several sentences. However, speech signals consist of sound pressure
measurements with thousands of samples per second, resulting in sequences much
longer than those for text. Even spectral representations which reduce the
sequence length can have hundreds of frames per second. Processing such
sequences with typical neural network architectures like Transformers can
result in problems with running time and memory requirements. One could gather
consecutive frames to form shorter segments, but unlike text, there is no
obvious segmentation for unlabeled speech.

  * •

Speech is continuous. In NLP, it is common to use a pretext task that models a
categorical distribution of masked or future inputs. Since text is easily
broken down into individual tokens such as words, subwords, or characters, it
is straightforward to define a finite vocabulary for such tasks. However, this
idea does not apply to speech modeling because speech signals are continuous;
in this sense there is no such thing as a speech vocabulary.

  * •

Speech processing tasks are diverse. Building generalizable self-supervised
representation models for diverse speech processing tasks is challenging.
Speech contains rich, hierarchical information, and different speech tasks may
require mutually orthogonal information. For example, speech recognition
requires a model that extracts content information but ignores speaker
information; in contrast, speaker recognition requires a model that extracts
speaker information but removes content information. Therefore, it is
challenging to define a self-supervised model whose representations are
suitable for both speech recognition and speaker recognition. Analogous
considerations apply within CV and NLP.

In the sections below, we group modern SSL pretext tasks designed for speech
into three main categories: generative approaches, contrastive approaches and
predictive approaches. Figure 2 shows a timeline of the models covered in
these sections with each model colored according to our categorization. Table
I summarizes model pretext tasks along within the categories.

###  III-A Notation

To efficiently describe the different approaches, we use a simple notation.
Models are assumed to consist of functions $f(\cdot)$ and $g(\cdot)$, where
$f(\cdot)$ denotes the representation model to be used after pre-training and
$g(\cdot)$ is an auxiliary module needed only to support the pretext task. For
instance, in a classic autoencoder, $f(\cdot)$ would denote the encoder and
$g(\cdot)$ the decoder. For more complex models, these functions might consist
of several components indicated by sub-indices $f_{1}(\cdot)\dots
f_{N}(\cdot)$. As we will see, many self-supervised models use masking, which
replaces some parts of the input or a hidden representation by zeros or a
learned vector. We use $m(\cdot)$ to denote a function that applies such
masking to its input. Similar to $g(\cdot)$, this function is only used during
pre-training.

Given an acoustic input $X=\\{x_{1},x_{2},...,x_{T}\\}$, $f(\cdot)$ outputs a
representation $H=\\{h_{1},h_{2},...,h_{T}\\}$. The input $X$ may be either
the raw waveform samples or a sequence of spectral feature vectors. Both are
viable options in practice. For simplicity, we do not distinguish between the
two in our notation.

While $f(\cdot)$ always takes an acoustic input, the input to $g(\cdot)$ can
be either the acoustic signal or another learned representation. Most
importantly, $g(\cdot)$ produces an output that is used for the pretext task
but is not used by $f(\cdot)$ to produce the representation $H$. Hence,
$g(\cdot)$ can be discarded after pre-training. Finally, $f(\cdot)$ commonly
downsamples the temporal dimension, but again, this is not crucial to
understand the models, so consider only a single temporal scale
$t\in\\{1,\dots,T\\}$ for notational convenience.

We use $Q=\\{q_{1},q_{2},...,q_{T}\\}$ to denote representations that are
quantized via codebook learning. Alternatively, discrete representations may
take the form of one-hot vectors, or the equivalent integer IDs, which we
denote by $C=\\{c_{1},c_{2},...,c_{T}\\}$. We use a circumflex to denote that,
for instance, $\hat{x}_{t}$ is an approximation of $x_{t}$. Finally, we often
use a subscript when defining a loss, $\mathcal{L}_{i}$, to imply that the
total loss is computed as a sum over $i$, unless otherwise stated.

For some models, we will refer to $H$ as a _contextualized_ representation
which means that each $h_{t}$ is a function of some, linguistically speaking,
long sub-sequence of $X$ spanning at least several phonemes. Usually, $h_{t}$
depends on the entire input $X$ or all previous timesteps $X_{[1,t]}$. In
contrast, a _localized_ representation is one that only depends on a short
part of the input $X_{[t-u,t+u]}$, where $u\geq 0$. The distinction between
contextualized and localized may become fuzzy if $u$ is large, however, this
is rarely the case.

After pre-training, the representation model $f(\cdot)$ can be fine-tuned for
a downstream task directly or used to extract features which are fed to
another model, as visualized in Figure 1. It is not uncommon to use the output
representation $H$, but often representations from hidden layers of $f(\cdot)$
are better suited [68].

Figure 2: A selection of models listed according to first publication date on
arXiv or conference submission date when this clearly precedes the former. The
models are categorized as generative, contrastive, or predictive. In addition,
some models are characterized as embedding models or multi-modal models,
although most learn frame-level representations from speech only. Some models
use a mixture of generative and contrastive tasks. For instance, PASE and
PASE+ use a multi-task setup, but find that generative tasks are the most
important for downstream task performance [69].

###  III-B Generative approaches

####  III-B1 Motivation

In this category, the pretext task is to generate, or reconstruct, the input
data based on some limited view. This includes predicting future inputs from
past inputs, masked from unmasked, or the original from some other corrupted
view. “Generative” as used in this paper hence refers to models that target
the original input in their pretext task. Note that this differs from
generative models, which learn distributions that allow to sample new data.

####  III-B2 Approaches

##### Autoencoding

Since their introduction in the mid-1990s [13], autoencoders (AEs) have played
an essential role in learning distributed latent representations of sensory
data. As described above, AEs consist of an encoder and decoder; the pretext
task is to reconstruct the given input. The most common type of AE places an
information bottleneck on the latent representation by simply having fewer
hidden units available than input features. This forces the model to discard
low-level details and disccourages the learning of trivial solutions. Other
models add regularization to the latent space to further improve the quality
of the learned representations. For instance, denoising autoencoders (DAEs)
learn latent representations by reconstructing from input corrupted by noise
[36]. The Variational Autoencoder (VAE) is a probabilistic version of the AE
which defines the latent representation via a posterior distribution over
stochastic latent variables [48, 49]. VAEs have been applied to speech in
numerous works [70, 71, 72, 73, 74]. The vector-quantized variational
autoencoder (VQ-VAE) is another model in this category [75]; it extends the
original VAE [48] with a novel parameterization of the posterior distribution
for discrete latent representations. The VQ-VAE has been instrumental in
generative speech modelling and recent work on generative spoken language
modeling has successfully combined the idea of a discrete latent space with
self-supervised learning [76, 77, 78].

Specifically, in the VQ-VAE, the continuous representation vector $h_{t}$ at
the output of the encoder is quantized by mapping it to a codebook vector,
which is then used as the input to the decoder. This operation is non-
differentiable and the gradients of the loss with respect to the encoder
parameters must be obtained by approximation. In the VQ-VAE this is done using
the straight-through estimator [79], i.e., the gradients with respect to the
encoder output are taken to be equal to those with respect to the decoder
input (i.e., the quatization step is ignored). Given a learned codebook
$A\in\mathbb{R}^{K\times D}$, where $K$ is the codebook size and $D$ is the
dimensionality of each codebook vector $a_{k}$, the quantized representation
$q_{t}$ of $h_{t}$ is obtained as

$\displaystyle q_{t}=a_{k},\text{ where }k=\arg\min_{j}\left\lVert
h_{t}-a_{j}\right\rVert_{2}.$ (1)

The decoder $g(\cdot)$ is an autoregressive model that takes $q_{[1,t]}$ as
input to generate $x_{t}$ [80]. Codebook learning is facilitated by a two-term
auxiliary loss similar to classical vector quantization dictionary learning
[81, 82]. Gradients for the codebook vectors are given solely by a term that
moves codebook vectors $a_{k}$ closer to the non-quantized vectors $h_{t}$. A
so-called _commitment term_ is added to ensure that non-quantized vectors do
not grow unboundedly by enforcing the encoder to keep them close to a codebook
vector. This commitment term is optimized only by the encoder. The total VQ-
VAE loss for a single timestep is

(2)

where $\log p(x_{t}|q_{[1,t]})$ is a reconstruction likelihood term usually
using a categorical distribution, $\mathrm{sg}[x]=x$ is the so-called stop-
gradient operator which acts as the identity function during the forward pass
but is assumed to have partial derivatives all equal to zero during the
backward pass, $\alpha$ is a scalar hyperparameter, and we define
$\text{MSE}(h_{t},A)=\frac{1}{KD}\sum_{k=1}^{K}\sum_{i=1}^{D}\left(h_{t,i}-a_{k,i}\right)^{2}$.
The loss for a full sequence is the sum or mean over all $\mathcal{L}_{t}$.

These learned discrete representations have been shown to capture high-level
speech information closely related to phonemes, and are useful for
applications such as speaker conversion [83]. Vector quantization is not
exclusive to VQ-VAE but has seen widespread application within SSL for
regularization purposes and to define targets for the pretext task. We will
cover these applications below.

The Gumbel softmax [84] is another frequently used approach for obtaining a
discrete representation space, and has also been used for AEs [85]. In
addition to the approaches discussed above, several other works on speech
representation learning take inspiration from the AE framework [86, 87, 88,
89, 90, 91].

##### Autoregressive prediction

Autoregressive predictive coding (APC) [92, 93] takes inspiration from the
classic Linear Predictive Coding (LPC) approach for speech feature extraction
[94] and autoregressive language models (LM) for text, where the model learns
to predict future information from past. A function $f(\cdot)$ reads the input
sequence $X_{[1,t]}$ and outputs a representation sequence $H_{[1,t]}$. The
auxiliary module $g(\cdot)$ is a linear projection layer which takes the last
vector of $H_{[1,t]}$ as input to approximate $x_{t+c}$, where $c\geq 1$.
Thus, $c$ indicates how many timesteps the model predicts ahead. The modules
$f(\cdot)$ and $g(\cdot)$ are jointly learned to minimize the $L_{1}$ loss
between $x_{t+c}$ and its approximation $\hat{x}_{t+c}$. APC is formulated as

$\displaystyle H_{[1,t]}$ $\displaystyle=f(X_{[1,t]}),$ (3)
$\displaystyle\hat{x}_{t+c}$ $\displaystyle=g(h_{t}),$ (4)
$\displaystyle\mathcal{L}_{t}$
$\displaystyle=\lVert\hat{x}_{t+c}-x_{t+c}\rVert_{1}.$ (5)

In text-based autoregressive LMs, $c$ is set to $1$ to enable autoregressive
generation. However, due to the smoothness of the speech signal, neighboring
acoustic features are usually similar. Depending on the downstream task, we
are often interested in learning so-called _slow features_ that typically span
multiple input frames [95]. Even the smallest linguistic units of
speech—phonemes—span $0.07$ seconds on average in the English TIMIT dataset
[96], whereas spectrogram frames $\mathbf{x}_{t}$ are typically computed at
$0.01$ second intervals. Thus, simply predicting the next frame constitutes a
trivial pretext task for APC; the original work finds that $c=3$ performs
well. In [97], the APC objective is extended to multi-target training. The new
objective generates both past and future frames conditioned on previous
context. In VQ-APC [98], quantization is used with the APC objective, which
imposes an information bottleneck serving as a regularizer.

A drawback of APC is that it encodes information only from previous timesteps
and not the entire input. DeCoAR [99] combines the bidirectionality of the
popular NLP model ELMo [51] and the reconstruction objective of APC to
alleviate this issue and allow encoding information from the entire input. It
uses a forward LSTM $f_{1}(\cdot)$ to encode $X_{[1,t]}$ and a backward LSTM
$f_{2}(\cdot)$ to encode $X_{[t+k,T]}$, where $k>1$:

$\displaystyle H_{[1,t]}=f_{1}(X_{[1,t]}),$ (6) $\displaystyle
H^{\prime}_{[t+k,T]}=f_{2}(X_{[t+k,T]}),$ (7)
$\displaystyle\hat{X}_{[t+1,t+k-1]}=g(h_{t},h^{\prime}_{t+k}).$ (8)

The input feature vector used in the downstream tasks is the concatenation of
$h_{t}$ and $h^{\prime}_{t}$.

##### Masked Reconstruction

Masked reconstruction is largely inspired by the masked language model (MLM)
task from BERT [54]. During BERT pre-training, some tokens in the input
sentences are masked by randomly replacing them by a learned masking token or
another input token. The model learns to reconstruct the masked tokens from
the non-masked tokens. Recent work has explored similar pretext tasks for
speech representation learning. Similar to the DeCoAR model described above,
this allows a model to learn contextualized representations that encode
information from the entire input. While we here focus on the models that
reconstruct the masked input, it is important to note that masking has also
been used extensively for contrastive (Section III-C) and predictive (Section
III-D) models.

From a high-level perspective, the training phase of models using masked
reconstruction can be formulated as

$\displaystyle H$ $\displaystyle=f(m(X)),$ (9) $\displaystyle\hat{x}_{t}$
$\displaystyle=g(h_{t}),$ (10) $\displaystyle\mathcal{L}_{t}$
$\displaystyle=\lVert\hat{x}_{t}-x_{t}\rVert_{1}.$ (11)

The exact masking policy defined by $m(\cdot)$ differs from model to model and
will be discussed further below. The function $f(\cdot)$ is typically a
Transformer encoder [100, 101, 102], but recurrent neural networks have also
been used [103]. In general, the Transformer encoder architecture has been
adopted widely by self-supervised models for speech within all three surveyed
categories. The function $g(\cdot)$ is usually a linear projection or a
multilayer perceptron (MLP). Finally, the loss $\mathcal{L}_{t}$ is commonly
computed only for masked timesteps in order to discourage the model from
learning an identity mapping.

The masking policies used in NLP can be adapted to speech by considering a
speech segment equivalent to a token in a sentence; indeed, the masking
strategy of BERT has also been used for speech pre-training [100]. In the
standard BERT masking policy, each token is masked independently at random.
However, for speech, masking a single sample or spectrogram frame results in a
largely trivial reconstruction task since, as discussed in paragraph III-B2,
the smoothness of audio signals may encourage the model to learn to simply
interpolate neighboring frames. Therefore it is common to mask chunks of
consecutive frames [100, 104].

We can bring the pretext task closer to the NLP equivalent by using a masking
policy where the masked regions of the input correspond to linguistic units.
Instead of just masking a fixed number of consecutive frames, pMPC [105]
selects masked speech frames according to the phonetic segmentation in an
utterance. However, in order to obtain this segmentation, some labeled data is
of course needed.

Whereas most studies use masking along the temporal dimension of the input,
speech can also be masked along the frequency dimension when spectral input
features are used [103, 106]. Frequency masking has been shown to improve
representations used for speaker classification [106].

Some studies explore alternatives to masking the input directly. In non-
autoregressive predictive coding (NPC) [107], time masking is introduced
through masked convolution blocks. Taking inspiration from XLNet [108], it has
also been suggested that the input be reconstructed from a shuffled version
[109] to address the discrepancy between pre-training and fine-tuning of
masking-based approaches.

Regularization methods can further improve on masked reconstruction
approaches. DeCoAR 2.0 [110] uses vector quantization, which is shown to
improve the learned representations. Furthermore, two dropout regularization
methods—attention dropout and layer dropout—are introduced with the TERA model
[106, 111]. Both methods are variations on the original dropout method [112].

##### More Generative Approaches

Other than the autoregressive and masked reconstruction tasks discussed above,
various studies have explored the reconstruction of other targets derived from
the input. PASE and PASE+ [69, 113] use multiple targets, including the
waveform, log power spectrum, mel cepstral coefficients (MFCCs), and prosody
features. Models that learn acoustic embeddings of small speech segments have
targeted future and past spectrogram segments [114, 115, 116], phase
information [117], and the temporal gap between two segments [115, 116].

####  III-B3 Challenges

Although successful NLP models like BERT and GPT are based on generative
pretext tasks, the progress have not been translated directly to the speech
domain. A speech signal encodes more information than text, such as speaker
identity and prosodic features, which makes it harder to generate. However, in
order to generate all details of the input, the model must encode all
information in the speech signal. Hence, a model that learns to perfectly
reconstruct its input may not necessarily have learned to isolate the features
of interest and will encode redundant information for a given downstream task.

There are many choices involved in designing a generative pretext task. For
instance, masking strategy and the choice of input and target representation
(e.g., waveform samples or spectral features). These choices influence what
the model learns through the pretext task. However, there is little research
on the relationship between task design and the information encoded in the
learned representations.

TABLE I:  A summary of the approaches in the three categories of self-supervised learning. Column (a) lists the names of the models and related references, column (b) defines the model input, column (c) defines any corruption of the input or hidden representation, and column (d) defines the target of the pretext task; the pretext task itself is described by the overall model category and the main text. $X=\\{x_{1},x_{2},...,x_{T}\\}$ is the input sequence in which $x_{t}$ can be an acoustic feature vector (e.g., MFCC, filterbank, or spectrogram features) or a waveform sample. $X_{[t_{1}:t_{2}]}$ represents $\\{x_{t_{1}},x_{t_{1}+1},...,x_{t_{2}}\\}$. $X_{-[t_{1}:t_{2}]}$ represents $X$ in which the segment $X_{[t_{1}:t_{2}]}=\\{x_{t_{1}},x_{t_{1}+1},...,x_{t_{2}}\\}$ is masked. $x_{t}^{i}$ represents the $i$-th dimension of $x_{t}$. If $x_{t}$ is a frame in a spectrogram, then the $i$-th dimension corresponds to a specific frequency bin. $X^{-[f,f+j]}$ refers to a spectrogram $X$ which is masked along the frequency axis from the $f$-th to $(f+j)$-th bin. We indicate random temporal permutation of a sequence by indexing it with the set $\mathcal{P}_{t}\triangleq\textsc{permute}([0,t])$, where $\textsc{permute}(\cdot)$ returns a permutation of the given list. We indicate data augmentation (e.g., reverberation) by the function $\textsc{augment}(\cdot)$. Subscripts indicate different augmentations. $Z$ represents a localized latent representation sequence of $X$. $Z^{(l)}$ is $Z$ at the $l$-th layer of the model used to compute it. $\bar{H}$ is the contextualized sequence $H$ obtained from an exponential moving average (EMA) of the model undergoing training with no masking applied. $Q$ represents a sequence of quantized learned representations, and $C$ is a sequence of discrete cluster IDs. For contrastive models, we specify only positive targets.  Model (a) |  Input (b) |  Corruption (c) |  Target (d)
---|---|---|---
Generative models
Audio Word2vec [118], VQ-VAE [75] | $X$ | - | $X$
Speech2Vec [114], Audio2Vec [116] \- skip-gram | $X_{[t_{1},t_{2}]}$ | - |  $X_{[t_{0},t_{1}]}$,$X_{[t_{2},t_{3}]}$
Speech2Vec [114], Audio2Vec [116] \- cbow |  $X_{[t_{0},t_{1}]}$,$X_{[t_{2},t_{3}]}$ | - | $X_{[t_{1},t_{2}]}$
PASE [69], PASE+ [113]111PASE uses multiple pretext tasks, but the authors find that reconstruction is most important. | $X$ | - | Different modalities of $X$
APC [92, 98] | $X_{[1,t]}$ | - | $x_{t+c},\,c\geq 1$
Speech-XLNet [109] | $X_{\mathcal{P}_{t}}$ | $x_{i\sim\mathcal{P}^{c}_{t}}$
DeCoAR [99] | $X_{[1,t-1]},X_{[t+k+1,T]}$ | - | $X_{[t,t+k]}$
Mockingjay [100], Audio ALBERT [119], DeCoAR 2.0 [110] | $X_{-[t,t+k]}$ | $X_{[t,t+k]}$
TERA [106], BMR [103] | $X_{-[t,t+k]}^{-[f,f+j]}$ | $X$
pMPC [105] |  $X_{-[t,t+k^{\prime}]}$ ($X_{[t,t+k^{\prime}]}$ is a phoneme) | $X_{[t,t+k^{\prime}]}$
MPE [102] | $X$ | $Z_{-[t,t+k]}$ | $Z$
NPC [107] | $X$ | $Z_{-[t,t+k]}$ | $X$
Contrastive models
Unspeech [120] | $X_{[t_{1},t_{2}]}$ | - |  $X_{[t_{0},t_{1}]}$,$X_{[t_{2},t_{3}]}$
CPC [56], wav2vec [121], Modified CPC [122] | $X_{[1,t]}$ | - | $z_{t+c},\,c\geq 1$
Bidirectional CPC [123] |  $X_{[1,t]}$ or $X_{[t,T]}$ | - |  $z_{t+c}$ or $z_{t-c},\,c\geq 1$
vq-wav2vec [124] | $X_{[1,t]}$ | - | $q_{t+c},\,c\geq 1$
wav2vec 2.0 [125], wav2vec-C [126]222wav2vec-C adds reconstruction loss to wav2vec 2.0. | $X$ | $Z_{-[t,t+k]}$ | $Q_{[t,t+k]}$
w2v-BERT [127] | $X$ | $Z_{-[t,t+k]}$ |  $Q_{[t,t+k]}$ and $C_{[t,t+k]}$
Speech SimCLR [128]333Speech SimCLR targets the latent representation of an augmented version of $X$ using a differently augmented $X$, and vice-versa. |  $\textsc{augment}_{1}(X)$ and $\textsc{augment}_{2}(X)$ |  $\textsc{augment}_{2}(Z)$ and $\textsc{augment}_{1}(Z)$
Predictive models
Discrete BERT [124, 129] 444Discrete BERT obtains codes $C$ from vq-wav2vec. | $C_{-[t,t+k]}$ | $C_{[t,t+k]}$
HuBERT [130]555HuBERT is trained first using cluster IDs of the MFCCs as target and subsequently clusters IDs of the model representations from the last iteration., WavLM [131]666WavLM simulates noisy/overlapped speech as inputs. | $X$ | $Z_{-[t,t+k]}$ | $C_{[t,t+k]}$
data2vec [132] | $X$ | $Z_{-[t,t+k]}$ | $\sum_{l}\bar{H}^{(l)}_{[t,t+k]}$
BEST-RQ [133]777BEST-RQ obtains codes $C$ by quantizing acoustic features using a random projection quantizer. | $X_{-[t,t+k]}$ | $C_{[t,t+k]}$

###  III-C Contrastive approaches

####  III-C1 Motivation

As discussed above, speech contains many entangled features. Thus, learning to
reconstruct the raw speech signal might not be the best way to discover
contextualized latent factors of variations. Contrastive models learn
representations by distinguishing a target sample (positive) from distractor
samples (negatives) given an _anchor representation_. The pretext task is to
maximize latent space similarity between the anchor and positive samples while
minimizing the similarity between the anchor and negative samples. This
approach has been used extensively in the general ML community [134].

####  III-C2 Approaches

##### CPC

Contrastive Predictive Coding (CPC) [56] is a prominent example of a
contrastive model. CPC uses a convolutional module $f_{1}(\cdot)$ to produce
localized representations $z_{t}$ with a recurrent module $f_{2}(\cdot)$ on
top that outputs a contextualized representation $h_{t}$. An anchor
representation $\hat{z}_{t,k}$ is obtained via a linear projection
$g_{k}(\cdot)$ of $h_{t}$. The positives and negatives are sampled from the
localized representation $Z$. Hence, at a single timestep $t$, CPC forms
multiple anchor representations $\hat{z}_{t,k}$ for $k\in\\{1,\dots,K\\}$ and
associates with each one a single positive sample at the corresponding
timestep, $z_{t+k}$, $k$ steps in the future:

$\displaystyle z_{t}$ $\displaystyle=f_{1}(X_{[t-u,t+u]})\enspace,$ (12)
$\displaystyle H_{[1,t]}$ $\displaystyle=f_{2}(Z_{[1,t]})\enspace,$ (13)
$\displaystyle\hat{z}_{t,k}$ $\displaystyle=g_{k}(h_{t})\enspace.$ (14)

Each $z_{t}$ only encodes information from a limited receptive field, while
$f_{2}(\cdot)$ is limited to condition each $h_{t}$ on previous timesteps
$Z_{[1,t]}$. Without these restrictions, the model could collapse to a trivial
solution. $g_{k}$ is a unique transformation per offset $k$ (e.g., a linear
projection). The loss function measures the similarity between the anchor
representation $\hat{z}_{t,k}$ and the positive $z_{t+k}$ normalized by the
total similarity to the positive and negatives. The approach is similar to
previous work on Noise-Contrastive Estimation (NCE) [135]. Minimizing the loss
corresponds to maximizing a lower bound on the mutual information between
$h_{t}$ and $z_{t+k}$ (and in turn $x_{t+k-u:t+k+u}$) and is hence called
InfoNCE:

$\displaystyle\mathcal{L}_{t,k}$
$\displaystyle=-\log\left(\frac{\exp(\hat{z}_{t,k}^{\text{\tiny
T}}z_{t+k})}{\sum_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}i\in\mathcal{I}}}\exp(\hat{z}_{t,k}^{\text{\tiny
T}}z_{i})}\right)\enspace.$ (15)

Here, $\mathcal{I}$ is a random subset of $N$ indices which includes the
target index $t+k$ and $N-1$ negative samples drawn from a proposal
distribution, e.g., a uniform distribution over $\\{1,\dots,T\\}$. Including
the target index in $\mathcal{I}$ ensures that the loss is a proper
categorical cross-entropy and that minimizing it has the previously stated
relation to mutual information maximization. This corresponds to sampling
negatives from the same sequence and has been shown to give good performance
for phoneme classification [56]. The loss is indexed by $k$ to show that CPC
targets multiple offsets using different projection layers $g_{k}(\cdot)$. The
authors find $K=12$ to work well for phoneme classification.

The wav2vec model [121] extends the CPC approach and uses fully convolutional
parameterizations for the modules $f_{1}(\cdot)$ and $f_{2}(\cdot)$ with
receptive fields of $30\text{\,}\mathrm{m}\mathrm{s}$ and
$210\text{\,}\mathrm{m}\mathrm{s}$, respectively. While the CPC loss solves a
1-of-$N$ classification task per $(t,k)$, either assigning the anchor to the
positive class or (wrongly) to one of the $N-1$ negative classes, the wav2vec
loss considers a sequence of $N$ independent binary classifications. That is,
the anchor is compared independently to the positive and each negative, and
the loss is computed as a sum of the associated log-probabilities,

$\displaystyle\mathcal{L}_{t,k}$
$\displaystyle=-\log(\sigma(\hat{z}_{t,k}^{\text{\tiny
T}}z_{t+k}))+\sum_{i\in\mathcal{I}}\log(1-\sigma(\hat{z}_{t,k}^{\text{\tiny
T}}z_{i}))\enspace.$ (16)

Here, $\sigma(x)=1/(1+\exp(-x))$ is the sigmoid function,
$\sigma(\hat{z}_{t,k}^{\text{\tiny T}}z_{t+k})$ is the probability of the
anchor being the positive sample and $\sigma(\hat{z}_{t,k}^{\text{\tiny
T}}z_{i})$ is the probability of the anchor being the negative sample.
Evidently and contrary to CPC, $\mathcal{I}$ must not include the target index
$t+k$ as this would cancel out the positive term.

##### wav2vec 2.0

The wav2vec 2.0 model combines contrastive learning with masking. As the CPC
model, it uses the InfoNCE loss [56] to maximize the similarity between a
contextualized representation and a localized representation. However, instead
of using the $z_{t}$ directly as positive and negatives, it uses a
quantization module $g(\cdot)$ to obtain a discrete representation. This has
the practical implication that one can avoid sampling negatives from the same
category as the positive. The model takes as input a waveform and uses a
convolutional module $f_{1}(\cdot)$ followed by a Transformer encoder
$f_{2}(\cdot)$. Masking is applied to the output of the convolutional module:

$\displaystyle z_{t}$ $\displaystyle=f_{1}(X_{[t-u,t+u]})\enspace,$ (17)
$\displaystyle H$ $\displaystyle=f_{2}(m(Z))\enspace,$ (18) $\displaystyle
q_{t}$ $\displaystyle=g(z_{t})\enspace.$ (19)

The quantization module $g(\cdot)$ uses a Gumbel softmax [84] with a straight-
through estimator. Since the quality of the learned representations is
contingent on the quality of the quantization, wav2vec 2.0 combines two
techniques to learn high-quality codebooks. First, wav2vec 2.0 concatenates
quantized representations from multiple codebooks at each timestep, so-called
Product Quantization (PQ) [136]. Also, the primary training loss described
below is augmented with an auxiliary term designed to encourage equal use of
all codebook entries.

In wav2vec 2.0, anchors are taken to be $h_{t}$ at masked timesteps only, the
positive sample is chosen as the quantized vector, $q_{t}$, at the same
timestep, and negatives are sampled from other masked timesteps. The loss is

$\displaystyle\mathcal{L}_{t}$
$\displaystyle=-\log\left(\frac{\exp(S_{\text{c}}(h_{t},q_{t}))}{\sum_{i\in\mathcal{I}}\exp(S_{\text{c}}(h_{t},q_{i}))}\right)\enspace,$
(20)

where $S_{\text{c}}(\cdot)$ is the cosine similarity and $\mathcal{I}$
contains the target index $t$ and negative indices sampled from other masked
timesteps.

The wav2vec 2.0 approach was the first to reach single-digit word error rate
(WER) on LibriSpeech using only the low-resource Libri-light subsets for fine-
tuning a pre-trained model (see Section IV-B). It has subsequently inspired
many follow-up studies. The wav2vec-C [126] approach extends wav2vec 2.0 with
a consistency term in the loss that aims to reconstruct the input features
from the learned quantized representations, similar to VQ-VAE [137].

####  III-C3 Challenges

Although representations learned using contrastive approaches have proved
effective across a wide range of downstream applications, they face many
challenges when applied to speech data. One challenging aspect is that the
strategy used to define positive and negative samples can also indirectly
impose invariances on the learned representations. For example, sampling
negatives exclusively from the same utterance as the positive biases the
features towards speaker invariance, which may or may not be desired for
downstream applications. Another standing challenge is that since speech input
does not have explicit segmentation of acoustic units, the negative and
positive samples do not represent a whole unit of language but rather partial
or multiple units, depending on the span covered by each sample. Finally,
since speech input is smooth and lacks natural segmentation, it can be
difficult to define a contrastive sampling strategy that is guaranteed to
provide samples that always relate to the anchor as truly positives and
negatives in a sound way.

###  III-D Predictive approaches

####  III-D1 Motivation

Similar to the contrastive approaches discussed above, predictive approaches
are defined by using a learned target for the pretext task. However, unlike
the contrastive approaches, they do not employ a contrastive loss and instead
use a loss function such as squared error and cross-entropy. Whereas a
contrastive loss discourages the model from learning a trivial solution by the
use of negative samples, this must be circumvented differently for predictive
methods. For this reason, predictive methods compute the targets outside the
model’s computational graph; usually with a completely separate model. Thus,
the predictive setup is somewhat akin to teacher-student training. The first
predictive approaches were motivated by the success of BERT-like methods in
NLP [54] as well as the DeepCluster method in CV [138].

####  III-D2 Approaches

##### Discrete BERT

Applying BERT-type training directly to speech input is not possible due to
its continuous nature. The Discrete BERT approach [129] uses a pre-trained vq-
wav2vec model to derive a discrete vocabulary [124]. The vq-wav2vec model is
similar to wav2vec mentioned in Paragraph III-C2 but uses quantization to
learn discrete representations. Specifically, discrete units $c_{t}$ are first
extracted with the vq-wav2vec model $f_{1}(\cdot)$ and then used as inputs
_and_ targets in a standard BERT model $f_{2}(\cdot)$ with a softmax
normalized output layer $g(\cdot)$,

$\displaystyle c_{t}$ $\displaystyle=f_{1}(X_{[t-u,t+u]})\enspace,$ (21)
$\displaystyle H$ $\displaystyle=f_{2}(m(C))\enspace,$ (22)
$\displaystyle\hat{c}_{t}$ $\displaystyle=g(h_{t})\enspace.$ (23)

Similar to BERT, the model can then be trained with a categorical cross-
entropy loss,

$\displaystyle\mathcal{L}$ $\displaystyle=\sum_{t\in\mathcal{M}}-\log
p(c_{t}\mid X)\enspace,$ (24)

where $\mathcal{M}$ is the set of all masked timesteps. During training, only
the BERT model’s parameters are updated, while the vq-wav2vec model parameters
are frozen. Discrete BERT was the first model to demonstrate the effectiveness
of self-supervised speech representation learning by achieving a WER of 25% on
the standard test-other subset using a 10-minute fine-tuning set, setting the
direction for many approaches to follow.

##### HuBERT

Rather than relying on an advanced representation learning model for
discretizing continuous inputs, as Discrete BERT, the Hidden Unit BERT
(HuBERT) approach [130] uses quantized MFCC features as targets learned with
classic k-means. Thus, to compute the targets, the k-means model
$g_{1}(\cdot)$ assigns a cluster center to each timestep. Different from
Discrete BERT, HuBERT takes the raw waveform as input, rather than discrete
units. This helps to prevent loss of any relevant information due to input
quantization. HuBERT uses an architecture similar to that of wav2vec 2.0, with
a convolutional module $f_{1}(\cdot)$ and a Transformer encoder
$f_{2}(\cdot)$, as well as a softmax normalized output layer $g_{2}(\cdot)$:

$\displaystyle c_{t}$ $\displaystyle=g_{1}(X_{[t-w,t+w]})\enspace,$ (25)
$\displaystyle z_{t}$ $\displaystyle=f_{1}(X_{[t-u,t+u]})\enspace,$ (26)
$\displaystyle H$ $\displaystyle=f_{2}(m(Z))\enspace,$ (27)
$\displaystyle\hat{c}_{t}$ $\displaystyle=g_{2}(h_{t})\enspace,$ (28)

where $w$ defines the window size used to compute the MFCCs. The categorical
cross-entropy loss is computed on both masked, $\mathcal{L}_{m}$, and
unmasked, $\mathcal{L}_{u}$, timesteps:

$\displaystyle\mathcal{L}_{m}$ $\displaystyle=\sum_{t\in\mathcal{M}}-\log
p(c_{t}\mid X)\enspace,$ (29) $\displaystyle\mathcal{L}$
$\displaystyle=\beta\mathcal{L}_{m}+(1-\beta)\mathcal{L}_{u}\enspace.$ (30)

Again, $\mathcal{M}$ is the set of all masked timesteps, $\beta$ is a scalar
hyperparameter and $\mathcal{L}_{u}$ is computed as $\mathcal{L}_{m}$ but
summing over $t\notin\mathcal{M}$.

Intuitively, the HuBERT model is forced to learn both an acoustic and a
language model. First, the model needs to learn a meaningful continuous latent
representation for unmasked timesteps which are mapped to discrete units,
similar to a classical frame-based acoustic modeling problem. Second, similar
to other masked pre-training approaches, the model needs to capture long-range
temporal dependencies to make correct predictions for masked timesteps.

One crucial insight motivating this work is the importance of consistency of
the targets which enables the model to focus on modeling the sequential
structure of the input. Importantly though, for HuBERT, pre-training is a two-
step procedure. The first iteration is described above. Once completed, a
second iteration of pre-training follows. Here, representations from a hidden
layer of the model from the first iteration are clustered with k-means to
obtain new targets $c_{t}$.

For HuBERT, only two iterations are needed to match or outperform the previous
state-of-the-art results for low-resource speech recognition. And combining
the HuBERT approach with the wav2vec 2.0 approach, the w2v-BERT model has
managed to improve results even further [127].

##### WavLM

WavLM emphasizes spoken content modeling and speaker identity preservation
[131]. It is largely identical to HuBERT, but introduces two useful
extensions.

First, it extends the Transformer self-attention mechanism with a so-called
_gated relative position bias_. The bias is added prior to the softmax
normalization of the attention weights. For the attention weight at $i,j$, the
bias is computed based on the input to the Transformer layer at the current
time step $i$ and also incorporates a relative positional embedding for $i-j$.
The authors find that this extension improves performance on phoneme and
speech recognition tasks.

Second, it uses an utterance mixing strategy where signals from different
speakers are combined to augment the training data. Specifically, random
subsequences from other examples in the same batch are scaled and added to
each input example. Only the targets corresponding to the original example are
predicted during pre-training. Thus, the model learns to filter out the added
overlapping speech.

Most SSL methods are trained on data where each example only contains speech
from a single person; therefore, they can perform subpar on multispeaker tasks
like speaker separation and diarization.

The WavLM model achieved substantial improvements on the speech separation,
speaker verification and speaker diarization tasks in the SUPERB benchmark,
while also performing well on many other tasks compared to HuBERT and wav2vec
2.0.

##### data2vec

Motivated by the success of using an exponential moving average (EMA) teacher
for self-supervised visual representations [139, 140], the data2vec model
[132] computes targets $Y$ using an EMA of its own parameters. The targets are
constructed by averaging hidden representations of the top $k$ layers of the
EMA teacher network applied to unmasked inputs. Here, we denote this jointly
as $\bar{f}_{2}(\cdot)$.

The data2vec model was proposed for different data modalities, but for audio
it uses an architecture similar to wav2vec 2.0 and HuBERT with a convolutional
module $f_{1}(\cdot)$, a Transformer $f_{2}(\cdot)$ and masking applied to the
Transformer input.

$\displaystyle z_{t}$ $\displaystyle=f_{1}(X_{[t-u,t+u]})\enspace,$ (31)
$\displaystyle H$ $\displaystyle=f_{2}(m(Z))\enspace,$ (32) $\displaystyle Y$
$\displaystyle=\bar{f}_{2}(Z)\enspace.$ (33)

The teacher network $\bar{f}_{2}(\cdot)$ is a copy of the Transformer of the
student network but with the parameters at training step $i$,
$\theta_{\text{teacher},i}$, given by an EMA of the student parameters over
all previous training steps.

$\theta_{\text{teacher},i}=\begin{cases}\theta_{\text{student},0}&i=0\\\
\gamma\theta_{\text{student},i}+(1-\gamma)\theta_{\text{teacher},i-1}&i>0\end{cases}\enspace,$
(34)

where $\theta_{\text{student},i}$ are the parameters of the student network at
training step $i$, updated via gradient descent, and $\gamma$ is the EMA decay
rate.

The data2vec model uses a regression loss between target and prediction.
Specifically, to reduce sensitivity to outliers and prevent exploding
gradients, it uses the smoothed $L_{1}$ loss [141],

$\mathcal{L}_{t}=\begin{cases}\frac{1}{2}(y_{t}-h_{t})^{2}/\eta\text{,}&|y_{t}-h_{t}|\leq\eta\\\
|y_{t}-h_{t}|-\frac{1}{2}\eta\text{,}&\text{otherwise}\end{cases}\enspace,$
(35)

where the hyperparameter $\eta$ controls the transition from a squared loss to
an $L_{1}$ loss.

The data2vec approach was shown to work well for representation learning with
either speech, images or text data. It is the first approach to achieve
competitive results when trained on any one of the three modalities.

####  III-D3 Challenges

The iterative nature of pre-training for the HuBERT and wavLM could present a
practical inconvenience when working with large volumes of data. Another
challenge for these models centers around the quality of the initial
vocabulary generated from MFCC features. The data2vec approach improves over
other predictive models by allowing the targets to improve continuously via
the EMA teacher network; however, student-teacher approaches inflate the
existing computational challenges of very large models and may necessitate the
use of methods that decrease instantaneous memory utilization such as mixed
precision training, model parallelism and model sharding [142].

###  III-E Learning from multi-modal data

####  III-E1 Motivation

Multiple modalities are useful in many settings, where each modality provides
information that is complementary to other modalities. Multi-modal work
includes supervised settings, such as audio-visual ASR [143, 144] and person
identification [145] which have been studied for decades. In this section, we
focus only on unsupervised representation learning from multi-modal data.

One of the motivations for learning from multiple modalities is that it can
reduce the effect of noise, since noise in different modalities is likely to
be largely independent or uncorrelated. In addition, learning from speech data
with accompanying signals such as images or video can help learn
representations that encode more semantic information. Such “grounding”
signals can contain supplementary information that can be used by models to
infer the content of the speech. Human language learning provides a proof of
concept for this, as it is believed that infants benefit from the visual
modality when learning language [146]. Early computational models of multi-
modal language learning were motivated by (and tried to emulate) human
learning of language in the context of the visual surroundings [147].

####  III-E2 Approaches

We define two broad classes of approaches in this area. Specifically,
depending on what type of multi-modal data is involved we refer to “intrinsic”
or “extrinsic” modalities.

Intrinsic modalities are modalities produced directly by the speech source.
Examples of intrinsic modalities (besides the speech audio) include images or
video of the speaker’s face [148, 149], lip-movement [150], articulatory flesh
point measurements [151, 152], or simultaneous magnetic resonance imaging
(MRI) scans [153]. Typically, learning from multiple intrinsic modalities is
done so as to improve robustness to noise, since acoustic noise is likely to
be uncorrelated with the other modality(ies). This type of representation
learning is often referred to as “multi-view learning” because the multiple
intrinsic modalities can be seen as multiple views of the same content. Some
typical approaches in this category include

  * •

Multi-view autoencoders and variations [154, 155],

  * •

Multi-modal deep Boltzmann machines [156],

  * •

Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158,
159, 160, 161, 162, 163, 164, 165, 166],

  * •

Multi-view contrastive losses [167, 168],

  * •

More recently, audio-visual extensions of masked prediction methods [150,
169], specifically Audio-Visual HuBERT (AV-HuBERT) [150].

Extrinsic modalities are modalities that are not produced by the same source
but still provide context for each other. A typical example is an image and
its spoken caption: The image tells us what the speech is likely describing,
so a representation model that takes both modalities into account will
hopefully encode more of the meaning of the speech than a single-modality
model. There has recently been a surge of datasets collected for this purpose,
usually consisting of images and spoken captions, the audio and image frames
in a video, or video clips with their spoken descriptions. A recent review of
datasets, as well as methods, in this category is provided by Chrupała [170].

Typical approaches involve learning a neural representation model for each
modality, with a multi-modal contrastive loss that encourages paired examples
in the two modalities to have similar representations while unpaired examples
remain different [171, 172, 173, 174, 175, 176]. Other choices include
training with a masked margin softmax loss [177, 178] or a masked prediction
loss [179]. Such models are typically evaluated on cross-modal retrieval,
although some work has also used the models for other downstream tasks such as
the ZeroSpeech and SUPERB benchmark tasks [180]. Analyses of such models have
found that, despite the very high-level learning objective of matching speech
with a corresponding image (or other contextual modality), such models often
learn multiple levels of linguistic representations from the shallowest to the
deepest model layers [181, 182, 183]. They are also able to learn word-like
units [184, 185, 186] and can be used for cross-lingual retrieval, by
considering the visual signal as an “interlingua” [187, 188, 189]. In some
settings, even in the presence of some amount of textual supervision (i.e.,
the speech is transcribed), visual grounding still helps learn a better
representation for retrieval [190].

There has also been growing interest in learning joint speech and text
representations using paired and unpaired data. The SLAM approach [191] is an
example where speech and text are first represented using two separate pre-
trained encoders followed by a multi-modal encoder to build the joint
representations. The entire model is trained using a multi-task loss including
two supervised and two self-supervised tasks.

####  III-E3 Challenges

One of the challenges of using multi-modal approaches is that the multi-modal
data they rely on is often in shorter supply than single-modality data. In
addition, multi-modal data is typically drawn from specific domains, for
example domains involving descriptions of visual scenes. It is not clear how
well the learned speech representations apply to other speech domains that are
not necessarily describing or situated in a visual scene, and this question
requires further study.

###  III-F Acoustic Word Embeddings

Most of the representation learning techniques discussed in the preceding
sections are aimed at learning frame-level representations. For some purposes,
however, it may be useful to explicitly represent longer spans of speech audio
of arbitrary duration, such as phone, word, or phrase-level segments. For
example, searching within a corpus of recorded speech for segments that match
a given (written or spoken) query can be seen as finding segments whose
representations are most similar to that of the query [192, 193, 118, 194];
word embeddings can be defined by pooling representations of instances of a
given word [114]; unsupervised segmentation and spoken term discovery can be
seen as a problem of detecting and clustering segments [195, 196]; and even
ASR can be viewed as the problem of matching written word representations to
representations of audio spans [197, 198, 91].

Several lines of work have begun to address the problem of learning
representations of spans of speech, especially word segments, typically
referred to as acoustic word embeddings. Early work on unsupervised acoustic
word embeddings defined them as vectors of distances from the target segment
to a number of pre-defined “template” segments [199]. Later work used variants
of neural autoencoders [118, 200, 201, 202]. These are often evaluated on word
discrimination, that is the task of determining whether two word segments
correspond to the same word or not [203]. This task can be thought of as a
proxy for query-by-example search, since the basic operation in search is to
determine whether a segment in the search database matches a query segment,
and has been used for evaluation of both frame-level (e.g., [89]) and word-
level [199, 204] representations.

Since most work on acoustic word embeddings preceded the very recent wave of
new self-supervised frame-level representations, one question is whether word
(or more generally segment) embeddings could be derived more simply by pooling
self-supervised frame-level representations, as has been done for text span
embeddings by pooling over word embeddings [205, 206]. Some initial results
suggest that at least very simple pooling approaches like downsampling and
mean or max pooling are not successful [207, 202], but more work is needed to
reach conclusive results.

##  IV Benchmarks for Self-supervised Learning

The previous sections presented various methodologies by which to learn speech
representations from unlabeled corpora. This section surveys the datasets
available to learn and evaluate these representations. We also summarize
several studies and their results to demonstrate the usefulness of the learned
representations for various downstream tasks.

###  IV-A Datasets only for pre-training

Table II summarizes datasets used for pre-training SSL techniques in the
literature. These datasets are usually large but with limited or no labels.
Libri-light (LL) [208], one of these datasets, is derived from audiobooks that
are part of the LibriVox888https://librivox.org/  project. LL contains 60k
hours of spoken English audio tagged with SNR, speaker ID, and genre
descriptions. The speech examples in Audioset [209], which consists of over 2M
10-second YouTube video clips human-annotated with 632 audio events, have also
been used for pre-training. Audioset has 2.5k hours of audio of varying
quality, different languages, and sometimes multiple sound sources. AVSpeech
[210] is another large-scale audio-visual dataset used in SSL research,
comprising 4.7k hours of clips from a wide variety of languages. Each clip
contains a visible face and audible sound originating from a single speaker
without interfering background signals. The 3100-hour audio part of AVSpeech
has been used to learn audio-only representations [123]. The Fisher corpus
[211] collects over 2k hours of conversational telephone speech, 1k hours of
which is utilized for pre-training [104]. Industrial researchers have also
begun to build large-scale datasets for learning speech representations. For
instance, 10k hours of real-world far-field English voice commands for self-
supervised pre-training have been collected at Amazon [126].

In addition to these English and multilingual efforts, researchers have also
collected corpora for pre-training Chinese speech representations. Didi
Dictation and Didi Callcenter [101, 104] are two internal datasets containing
respectively 10k hours of read speech collected from mobile dictation
application and 10k hours of spontaneous phone calls between users and
customer service staff.

###  IV-B Datasets for both pre-training and evaluation

Several datasets that provide both speech and associated transcripts and
speaker labels have also been used to develop SSL techniques by enabling in-
domain pre-training and evaluation. Such datasets are also listed in Table II.
One of the most commonly used datasets in this category is LibriSpeech (LS)
[212], a labeled corpus containing 960 hours of read English speech, which is
also derived from an open-source audiobooks project.8 The corpus consists of
subsets train-clean-100, train-clean-360, train-other-500, dev-clean, dev-
other, test-clean, and test-other used for training, development, and testing,
respectively. Subsets tagged with other are more challenging utterances from
speakers that yield higher WER as measured with previously built models. LS is
used for unsupervised representation pre-training by ignoring its labels, and
can also be utilized to evaluate the performance of representation on ASR,
phoneme recognition (PR), phoneme classification (PC), and speaker
identification (SID) tasks. Wall Street Journal (WSJ) [213] is another widely
adopted, labeled corpus for pre-training. Its labels can evaluate performance
for ASR, PR, PC, and SID. The original WSJ corpus contains 400 hours of
English read speech data, and today its si284 (81 hours), dev93, eval92
subsets are the most-used partitions for unsupervised training, development,
and test, respectively. The si84 (15 hours) partition is also used for
training.

The speech community also utilizes multilingual corpora. These are often
large-scale, which facilitates pre-training, but are also partially labeled
for ASR evaluation (PC and PR can be enabled via phone-level forced
alignment). These corpora include Common Voice (CV) [214], Multilingual
LibriSpeech (MLS) [215], VoxPopuli (VP) [216], and BABEL (BBL) [217]. CV is an
open-source, multi-language, growing dataset of voices containing 11k hours of
audio from 76 languages as of the date this review was written (Common Voice
corpus 7.0). Researchers usually use part of this for pre-training (e.g., 7k
hours/60 languages in [218] and 430 hours/29 languages in [123]) or
evaluation. MLS derives content from read audiobooks of LibriVox and contains
data in eight European languages for a total of 50k hours of audio. VP
comprises a total of 400k hours of parliamentary speech from the European
parliament in 23 European languages. The entire dataset [218] or a 24k-hour
portion [219, 131] thereof has been used for pre-training. BBL consists of 1k
hours of conversational telephone speech in 17 African and Asian languages.

Several datasets, including GigaSpeech [220], TED-LIUM 3 (TED3) [221], TED-
LIUM 2 (TED2) [222], Switchboard (SWB) [223], TIMIT [96], and VoxLingua107
[224], are labeled and conventionally used for evaluation, while their audio
streams are also aggregated to build diversified and large-scale corpora for
unsupervised pre-training [123, 109, 218]. GigaSpeech is a multi-domain
English ASR corpus with 33k hours of audio collected from audiobooks,
podcasts, and YouTube. A subset of 10k audio is transcribed. TED2 comes with
118 hours of English speech extracted from TED conference talks and its
transcription for evaluating ASR. Its recordings are clear but with some
reverberation. TED3 is an extension of TED2 and comprises 450 hours of talks.
SWB is a 260-hour conversational speech recognition dataset containing two-
sided telephone conversations. The TIMIT corpus was designed to provide read
speech data and its word and phone-level transcriptions for acoustic-phonetic
studies. It contains recordings in American English. Compared to the previous
corpora labeled for ASR evaluation, VoxLingua107 consists of 6.6k hours of
audio in 107 languages and is annotated for language identification. Beyond
the original purpose of evaluation, these corpora are also used in pre-
training to improve the generalizability of learned representations.

For the purpose of pre-training and evaluating Mandarin speech
representations, the authors of [101, 104] also compiled Open Mandarin, an
open-source Mandarin dataset of 1.5k hours of speech from the Linguistic Data
Consortium (LDC) and OpenSLR.999https://openslr.org Open Mandarin consists of
the HKUST Mandarin Telephone Speech Corpus (HKUST, 200 hours of spontaneous
speech, of which 168 hours of audio is used for pre-training; the development
and test sets are excluded.) [225], AISHELL-1 [226] (178 hours of read
speech), aidatatang 200zh (200 hours, read speech) [227], MAGICDATA Mandarin
Chinese Read Speech Corpus (755 hours, read speech) [228], Free ST Chinese
Mandarin Corpus (ST-CMDS, 100 hours, read speech) [229], and Primewords
Chinese Corpus Set 1 (100 hours, read speech) [230]. Both HKUST and AISHELL-1
are labeled and are suitable for ASR evaluation.

###  IV-C Datasets for evaluation

Besides the aforementioned datasets, conventional speech processing benchmarks
are also used to evaluate self-supervised representations. Studies leverage
Hub5, DIRHA, and CHiME-5 to measure the efficacy of representations in ASR.
The Hub5 evaluation (LDC2002T43 and LDC2002S09, also referred to as the NIST
2000 Hub5 English evaluation set) contains 40 transcribed English telephone
conversations only for testing, where 20 are from conversations collected in
SWB studies but not released with the SWB dataset, and the rest are from
CallHome American English Speech (LDC97S42). DIRHA [231], short for Distant-
speech Interaction for Robust Home Applications, is a database composed of
utterances sampled from WSJ, speech of keywords and commands, and
phonetically-rich sentences. These utterances are read by UK and US English
speakers and recorded with microphone arrays. CHiME-5 [232] is a challenge
that aims to advance robust ASR and presents a dataset of natural
conversational speech collected under a dinner party scenario with microphone
arrays. A team at Amazon Alexa also recorded and transcribed a corpus of 1k
hours of audio for model training and evaluation [126].

Researchers also evaluate representations for sentiment analysis with the
INTERFACE [233] and MOSEI (CMU Multimodal Opinion Sentiment and Emotion
Intensity) [234] datasets. INTERFACE is an emotional speech database for
Slovenian, English, Spanish, and French, and contains six emotions: anger,
sadness, joy, fear, disgust, and surprise, plus neutral. MOSEI is comprised of
sentence-level sentiment annotations of 65 hours of YouTube videos using
emotion categories similar to INTERFACE, but replacing joy with happiness.

In addition, datasets employed to demonstrate the benefit of SSL
representations on various tasks include VCTK [235] and VoxCeleb1 [236] for
SID/ASV (automatic speaker verification) tasks, FSC (Fluent Speech Commands)
[237] for IC (intent classification), QUESST (QUESST 2014) [238] for QbE
(query by example), LS En-Fr [239] and CoVoST-2 [240] for ST (speech
translation), and ALFFA and OpenSLR-multi for multilingual ASR. The VCTK
corpus includes speech data with 109 English speakers of various accents, each
reading out about 400 sentences sampled from newspapers. VoxCeleb1 is an
audio-visual dataset comprised of short YouTube clips containing human speech.
It consists of 1251 unique speakers and 352 hours of audio. FSC contains
utterances of spoken English commands that one might use for a smart home or
virtual assistant, and is used to evaluate the performance of a spoken
language understanding system. The QUESST search dataset comprises spoken
documents and queries in 6 languages to measure the capability of models in
spotting spoken keywords from documents. LS En-Fr is a dataset augmenting
existing LS monolingual utterances with corresponding French translations to
train and evaluate English-French machine translators. CoVoST-2 is a
multilingual speech translation benchmark based on CV. It provides data for
translating from English into 15 languages and from 21 languages into English,
and has a total of 2.9k hours of speech. The ALFFA
project101010http://alffa.imag.fr collects speech of African languages to
promote the development of speech technologies in Africa, and [123] leverages
four African languages collected in the project for evaluation: Amharic [241],
Fongbe [242], Swahili [243], and Wolof [244]. In the same work [123], the
authors further select 21 phonetically diverse languages from OpenSLR to
evaluate the generalizability of SSL representations across languages. We
denote the collection as OpenSLR-multi below.

Last, [116] puts together five datasets (MUSAN [245], Bird Audio Detection
[246], Speech Commands [247], Spoken Language Identification [248], and TUT
Urban Acoustic Scenes 2018 [249]) plus an SID task built with the LS train-
clean-100 subset to evaluate the capability of representations on audio event
detection. [117] employs the NSynth dataset [250] on top of the six for
benchmarking. As many of the datasets are built for research in audio
processing, we here provide only a list of these datasets for reference.

TABLE II: Summary of datasets used in pre-training (denoted as PT) or evaluating (denoted as EV) SSL techniques in the literature. The languages and sizes of the datasets are provided in columns 3 and 4. Column 5 lists the tasks each dataset is used to evaluate. We use the following abbreviations: EN: English; Multi: multilingual; ZH: Chinese; Fr: French; ASR: automatic speech recognition; PR: phoneme recognition; PC: phoneme classification; SID: speaker identification; ASV: automatic speaker verification; Sentiment: sentiment analysis; ST: speech translation; QbE: query by example or spoken term detection; IC: intent classification; AED: audio event detection; and LID: language identification. We distinguish PR from PC based on whether the inference is made at the phone level sequentially or the frame level separately. SID and ASV both evaluate model capability in encoding speaker information; SID classifies one utterance into a pre-defined set of speaker labels, whereas ASV infers whether a given pair of utterances was uttered by the same speaker. Dataset | Purpose | Lang. | Size [hours] | Task | License
---|---|---|---|---|---
LibriLight (LL) | PT | EN | 60k | - | MIT License
AudioSet | PT | Multi | 2.5k | - | CC BY 4.0
AVSpeech | PT | Multi | 3.1k | - | CC BY 4.0
Fisher | PT | EN |  2k/1k [104] | - | Linguistic Data Consortium (LDC)
Alexa-10k | PT | EN | 10k | - | Not released
Didi Callcenter | PT | ZH | 10k | - | Not released
Didi Dictation | PT | ZH | 10k | - | Not released
LibriSpeech (LS) | PT/EV | EN | 960 | ASR/PR/PC/SID | CC BY 4.0
Wall Street Journal (WSJ) | PT/EV | EN | 81 | ASR/PR/PC/SID | Linguistic Data Consortium (LDC)
Common Voice (CV-dataset) | PT/EV | Multi |  11k/7k [218]/430 [123] | ASR/PR/PC | CC0
Multilingual LS (MLS) | PT/EV | Multi | 50k | ASR | CC BY 4.0
VoxPopuli (VP) | PT/EV | Multi |  400k [218]/24k [219, 131] | ASR | CC0
BABEL (BBL) | PT/EV | Multi | 1k | ASR | IARPA Babel Agreement
GigaSpeech | PT/EV | EN |  40k/10k [219, 131] | ASR | Apache-2.0 License
TED-LIUM 3 (TED3) | PT/EV | EN | 450 | ASR | CC BY-NC-ND 3.0
TED-LIUM 2 (TED2) | PT/EV | EN | 118 | ASR | CC BY-NC-ND 3.0
Switchboard (SWB) | PT/EV | EN | 260 | ASR | Linguistic Data Consortium (LDC)
TIMIT | PT/EV | EN | 4 | ASR/PR/PC | Linguistic Data Consortium (LDC)
VoxLingua107 | PT/EV | Multi | 6.6k | LID | CC BY 4.0
Open Mandarin | PT/EV | ZH | 1.5k | ASR |  CC BY-NC-ND 4.0, Apache License v.2.0, Linguistic Data Consortium (LDC)
HKUST | PT/EV | ZH | 168/200 | ASR | Linguistic Data Consortium (LDC)
AISHELL-1 | PT/EV | ZH | 178 | ASR | Apache License v.2.0
Hub5’00 | EV | EN | 13 | ASR | Linguistic Data Consortium (LDC)
DIRHA | EV | EN | 11 | ASR |  See link for detailsIV-C
CHiME-5 | EV | EN | 50 | ASR |  See link for detailsIV-C
Alexa-eval | EV | EN | 1k | ASR | Not released
INTERFACE | EV | Multi | 16 | Sentiment | No information
MOSEI | EV | EN | 65 | Sentiment |  See link for detailsIV-C
VCTK | EV | EN | 44 | SID/ASV | CC BY 4.0
VoxCeleb1 | EV | Multi | 352 | SID/ASV | CC BY 4.0
Fluent Speech Commands (FSC) | EV | EN | 14.7 | IC | CC BY-NC-ND 4.0
QUESST 2014 (QUESST) | EV | Multi | 23 | QbE | No information
LS En-Fr | EV | En-Fr | 236 | ST | CC BY 4.0
CoVoST-2 | EV | Multi | 2.9k | ST | CC0
ALFFA | EV | Multi | 5.2–18.3 | ASR-multi | MIT License
OpenSLR-multi | EV | Multi | 4.4–265.9 | ASR-multi |  CC BY-SA 3.0 US, CC BY-SA 4.0, CC BY 4.0, CC BY-NC-ND 4.0, Apache License v.2.0
AED datasets | EV | - | - | AED |  CC BY 4.0 (MUSAN, Speech Commands, NSynth, Bird Audio Detection), CC0 (Spoken Language Identification), Non-Commercial (TUT)
1111footnotetext: https://dirha.fbk.eu/node/1071212footnotetext: https://chimechallenge.github.io/chime6/download.html1313footnotetext: https://github.com/A2Zadeh/CMU-MultimodalSDK/blob/master/LICENSE.txt TABLE III: A summary of common experiment settings for various SSL evaluations (Part 1). Networks are usually pre-trained with SSL techniques, augmented with prediction heads, and fine-tuned (or trained) with labeled data in downstream tasks for benchmarking. The Pre-training corpus, Training (fine-tuning), and Test columns list the datasets used in each work, and the Task column lists the tasks performed in the corresponding papers. We follow the abbreviation introduced in Table II. The Transfer column indicates whether the SSL technique is evaluated by its capability for transfer learning, i.e., different datasets are utilized for pre-training and fine-tuning. The Fine-tuning labels used column summarizes the amount of labeled examples used in downstream fine-tuning. Work |  Pre-training corpus |  Task | Dataset |  Transfer |  Fine-tuning labels used
---|---|---|---|---|---
Training (fine-tuning) |  Test
CPC [56] |  LS 100 hrs |  PC |  LS 100 hrsIV-C |  LS 100 hrsIV-C |  - |  80IV-C hrs
SID |  LS 100 hrsIV-C |  LS 100 hrsIV-C |  - |  80IV-C hrs
PASE[69] |  LS 50 hrs [251] |  SID |  VCTKIV-C |  VCTKIV-C |  ✓ |  44 hrs
Sentiment |  INTERFACEIV-C |  INTERFACEIV-C |  ✓ |  3 hrs
|  |  PR |  TIMITIV-C |  TIMITIV-C |  ✓ |  4 hrs
|  |  ASR |  DIRHAIV-C |  DIRHAIV-C |  ✓ |  11 hrs
Audio2Vec [116] |  AudioSet |  AED |  6 AED datasetsIV-C |  6 AED datasetsIV-C |  ✓ |  See [116] for details
APC [92, 93] |  LS 360 hrs |  ASR |  WSJ si284IV-C |  WSJ dev93 |  ✓ |  72 hrs
ST |  LS En-FrIV-C |  LS En-FrIV-C |  - |  236 hrs
SID |  WSJ si284IV-C |  WSJ si284IV-C |  ✓ |  65IV-C hrs
wav2vec [121] |  LS 80/960 hrs, LS 960 hrs \+ WSJ si284 |  ASR |  WSJ si284 |  WSJ eval92 |  ✓ |  81 hrs
PR |  TIMITIV-C |  TIMITIV-C |  ✓ |  4 hr
PhasePredict [117] |  AudioSet |  AED |  7 AED datasetsIV-C |  7 AED datasetsIV-C |  ✓ |  See [117] for details
Bidir-CPC[123] |  LS 960 hrs, CPC-8kIV-C |  ASR |  WSJ si284, LS 960 hrs, TED3IV-C |  WSJ eval92, LS test-clean, LS test-other, TED3IV-C, SWBIV-C |  Different datasets for training and test |  81/960/450 hrs
ASR-multi |  ALFFAIV-C |  ALFFAIV-C |  ✓ |  4 languages, 5.2–18.3 hrs
ASR-multi |  OpenSLR-multiIV-C |  OpenSLR-multiIV-C |  ✓ |  21 languages, 4.4–265.9 hrs
MockingJay [100] |  LS 360 hrs |  PC |  LS 360 hrs |  LS test-clean |  - |  0.36IV-C/1.8IV-C/3.6IV-C/18IV-C/45IV-C/360 hrs
SID |  LS 100 hrsIV-C |  LS 100 hrsIV-C |  - |  90IV-C hrs
Sentiment |  MOSEIIV-C |  MOSEIIV-C |  ✓ |  65 hrs
CPC modified [122] |  LS 100 hrs |  PC |  LS 100 hrsIV-C |  LS 100 hrsIV-C |  - |  80IV-C hrs
LS 100 hrs, LS 960 hrs, LL 60k hrs |  PC |  CV-datasetIV-C |  CV-datasetIV-C |  ✓ |  1 hrs
vq-wav2vec [124] |  LS 960 hrs |  ASR |  WSJ si284 |  WSJ eval92 |  ✓ |  81 hrs
PR |  TIMITIV-C |  TIMITIV-C |  ✓ |  4 hrs
DeCoAR [99] |  LS 100/360/460/ 960 hrs, WSJ si284 |  ASR |  WSJ si284 |  WSJ eval92 |  - |  25IV-C/40IV-C/81 hrs
ASR |  LS 100/360/ 460/960 hrs |  LS test-clean, LS test-other |  - |  100/360/460/960 hrs
MT-APC[97] |  LS 360 hrs |  ASR |  WSJ si284IV-C |  WSJ dev93 |  ✓ |  72 hrs
ST |  LS En-FrIV-C |  LS En-FrIV-C |  - |  236 hrs
PASE+[113] |  LS 50 hrs [251] |  PR |  TIMITIV-C |  TIMITIV-C |  ✓ |  4 hrs
ASR |  DIRHAIV-C |  DIRHAIV-C |  ✓ |  11 hrs
ASR |  CHiME-5IV-C |  CHiME-5IV-C |  ✓ |  50 hrs
AALBERT [119] |  LS 360 hrs |  PC |  LS 100 hrsIV-C |  LS 100 hrsIV-C |  - |  80IV-C hrs
SID |  LS 360 hrsIV-C |  LS 360 hrsIV-C |  - |  288IV-C hrs
TABLE IV: A summary of common experiment settings for various SSL evaluations (Part 2). See the caption of Table III for a detailed description of all the abbreviations used in this table. Work |  Pre-training corpus |  Task | Dataset |  Transfer |  Fine-tuning labels used
---|---|---|---|---|---
Training (fine-tuning) |  Test
BMR [103] |  WSJ si284, LS 960 hrs |  ASR |  WSJ si284 |  WSJ eval92 |  - |  81 hrs
PR |  WSJ si84/si284 |  WSJ dev93 |  - |  15/81 hrs
vq-APC [98] |  LS 360 hrs |  PC |  WSJ si284IV-C |  WSJ dev93 |  ✓ |  81 hrs
SID |  WSJ si284IV-C |  WSJ si284IV-C |  ✓ |  65IV-C hrs
vq-wav2vec + DiscreteBERT [129] |  LS 960 hrs |  ASR |  LS 100 hrs |  LS test-clean, LS test-other |  - |  10 minsIV-C, 1IV-C/10IV-C/100 hrs
speech-XLNet [109] |  LS 960 hrs \+ WSJ si284 \+ TED2 |  PR |  TIMITIV-C |  TIMITIV-C |  ✓ |  4 hrs
ASR |  WSJ si284 |  WSJ eval92 |  - |  7IV-C/14IV-C/30IV-C/81 hrs
MPC [101, 104] |  Didi Callcenter, Didi Dictation, Open Mandarin |  ASR-zh |  HKUSTIV-C |  HKUSTIV-C |  ✓ |  168 hrs
ASR-zh |  AISHELL-1IV-C |  AISHELL-1IV-C |  ✓ |  178 hrs
SWB, Fisher 1k, LS 960 hrs |  ASR |  SWB |  Hub5’00 |  - |  260 hrs
MPE [102] |  WSJ si284, LS 960 hrs |  ASR |  WSJ si284 |  WSJ eval92 |  - |  25IV-C/40IV-C/81 hrs
ASR |  LS 100/360/960 hrs |  LS test-clean |  - |  100/360/960 hrs
ConvDMM [252] |  LS 50[251]/360/ 960 hrs |  PC/PR |  WSJ si284 |  WSJ eval92 |  ✓ |  5IV-C/50IV-C/100IV-C mins, 4IV-C/8IV-C/40IV-C hrs
wav2vec 2.0 [125] |  LS 960 hrs, LL 60k hrs |  ASR |  LS 960 hrs |  LS test-clean, LS test-other |  - |  10 minsIV-C, 1IV-C/10IV-C/100/960 hrs
PR |  TIMITIV-C |  TIMITIV-C |  ✓ |  4 hrs
NPC [107] |  LS 360 hrs |  PC |  WSJ si284IV-C |  WSJ dev93 |  ✓ |  81 hrs
SID |  WSJ si284IV-C |  WSJ si284IV-C |  ✓ |  65IV-C hrs
DeCoAR 2.0 [110] |  LS 960 hrs |  ASR |  LS 100 hrs |  LS test-clean, LS test-other |  - |  1IV-C/10IV-C/100 hrs
TERA [106] |  LS 100/360/ 960 hrs |  PC |  LS 100 hrsIV-C |  LS 100 hrsIV-C |  - |  80IV-C hrs
SID |  LS 100 hrsIV-C |  LS 100 hrsIV-C |  - |  80IV-C hrs
PR |  TIMITIV-C |  TIMITIV-C |  ✓ |  4 hrs
ASR |  LS 100 hrs |  LS test-clean |  - |  100 hrs
HuBERT [130] |  LS 960 hrs, LL 60k hrs |  ASR |  LS 960 hrs |  LS test-clean LS test-other |  - |  10 minsIV-C, 1IV-C/10IV-C/100/960 hrs
wav2vec-c [126] |  Alexa-10k |  ASR |  Alexa-eval |  Alexa-eval |  ✓ |  1k hrs
UniSpeech-SAT [219] |  LL 60k hrs \+ GigaSpeech-10k \+ VP-24k |  Multi |  SUPERB |  SUPERB |  ✓ |  See SUPERB [44] paper for details
WavLM [131] |  LL 60k hrs \+ GigaSpeech-10k \+ VP-24k |  Multi |  SUPERB |  SUPERB |  ✓ |  See [44] for details
XLS-R [218] |  VP-400k + MLS \+ CV-dataset-7k + VL \+ BBL |  ASR |  VP, MLS, CV-dataset, BBL, LS |  VP, MLS, CV-dataset, BBL, LS |  - |  See [218] for details
SID |  VoxCeleb1 |  VoxCeleb1 |  ✓
ST |  CoVoST-2 |  CoVoST-2 |  ✓
LID |  VL |  VL |  -
1414footnotetext:  Train/test split made available by [56] on Google drive
https://drive.google.com/drive/folders/1BhJ2umKH3whguxMwifaKtSra0TgAbtfb.1515footnotetext:
Utilizes official training or test split.1616footnotetext:  English utterances
used in experiments. The utterances correspond to approximately 3 hours for
training, 40 minutes for development, and 30 minutes for
testing.1717footnotetext:  The 6 AED datasets used in [116] are MUSAN [245],
Bird Audio Detection [246], Speech Commands [247], Spoken Language
Identification [248], TUT Urban Acoustic Scenes 2018 [249] plus an SID task
built with LS train-clean-100. In addition to the 6 datasets, [117] use the
NSynth dataset [250] for evaluation.1818footnotetext:  A collection of
AudioSet, AVSpeech, CV-dataset, LS, WSJ, TIMIT, Speech Accent Archive (SSA)
[253], TED3, and SWB. SSA is a growing annotated corpus of English speech with
various accents. Among the papers studied in this review, SSA is used in [123]
only for pre-training, and only 1 hour of audio is utilized. Thus, we exclude
it from our discussion in Section IV.1919footnotetext:  A subset of the
official training split is sampled, usually to mimic low-resource learning
conditions or to quickly evaluate for training and testing on the same split
but disjoint subsets.2020footnotetext:  Dataset split into training,
validation, and test subsets at a ratio of 8:1:1.2121footnotetext:  Dataset
split into training and validation subsets at a ratio of 9:1.2222footnotetext:
The dataset split into training and test subsets at a ratio of 9:1.

###  IV-D Experiment settings for evaluating SSL techniques

A common way to benchmark SSL techniques and show their efficacy is to fine-
tune a pre-trained SSL model for a supervised downstream task. Depending on
the corpora used in pre-training and fine-tuning, techniques can be
benchmarked in terms of their capability to transfer knowledge across datasets
(i.e., using pre-training corpora that differ from the fine-tuning ones),
their benefit when training with limited labeled examples (i.e., sampling a
subset of labeled examples for fine-tuning), or their improvement over a fully
supervised baseline (i.e., using the entire training split of downstream
datasets for fine-tuning). Tables III and IV summarize experiment settings
used in the SSL literature, including the pre-training corpora, downstream
tasks and datasets, and the amount of fine-tuning labels used, which indicates
the targeted benchmarking scenario as discussed above. Note that there are a
variety of ways to fine-tune pre-trained networks (e.g., fine-tune the entire
network, freeze certain layers during fine-tuning, and add various
architectures of prediction layers to pre-trained networks). We here omit
descriptions of these choices; readers can consult the original publications
for details.

As observed in Tables III and IV, LS and WSJ are the most commonly used pre-
training corpora. At the same time, we observe a growing industry investment
in pre-training with larger datasets, e.g., CPC-8k (8k hours) for Bidir-CPC
[123], LL (60k hours) for CPC modified [122], wav2vec 2.0 [125], and HuBERT
[130], Alexa internal datasets (10k hours) for wav2vec-c [126], Didi internal
datasets (10k hours) for MPC [101, 104], the combination of Gigaspeech,
VP-24k, and LL (94k hours in total) for UniSpeech-SAT [219] and WavLM [131],
and the combination of VP-400K, MLS, CV-dataset, VL and BBL (436k hours in
total) for XLS-R [218]. We expect this trend to continue with the growth in
available computing power. Most studies focuses on learning representations
for English, whereas Chinese [101, 104] and multilinguality [123, 218] are
also gaining attention. Compared to pre-training, datasets used for fine-
tuning are more diverse and cover downstream tasks as varied as ASR, PR, PC,
SID, AED, Sentiment, ST, and LID. For benchmarking training scenarios covering
full supervision as well as limited resources, the amount of labeled examples
used for fine-tuning also varies from several minutes up to 1k hours. Recent
benchmarks such as SUPERB [44] that consolidate multiple downstream tasks have
gained attention for evaluating SSL methodologies [219, 131]. The goal of such
benchmarks is to provide a holistic evaluation of the performance of learned
representations; we discuss these in detail in Section IV-E. With the
increasing popularity of SSL research, we expect future experiment settings to
proliferate and cover more languages, downstream tasks, and pre-training/fine-
tuning datasets.

Figure 3: SSL performance on ASR WER (vertical axis) evaluated with LS test-
clean split. Techniques are sorted based on the number of model parameters
along the horizontal axis. Markers in blue correspond to models initialized
with various SSL techniques and then fine-tuned using 960, 100, 10, 1 hour(s),
and 10 minutes respectively. The 960-hour training set is the aggregation of
train-clean-100, train-clean-360, and train-other-500 splits. The 100-, 10-,
1-hour, and 10-minute sets leverage train-clean-100 or its sampling, except
for Bidir-CPC, which samples 10% of the training examples from the entire
960-hour corpus. For simplicity, several SSL techniques are appended with
suffixes B, L, XL, or XXL indicating the Base, Large, X-Large, or XX-Large
variants specified in the original publication. We also compare with baselines
including the log mel filterbank (fBANK) and semi-supervised, self-training
approaches (iterative pseudo labeling (IPL) [63], slimIPL [254], noisy student
[62]). These approaches are visualized in black. Also, note that the current
state of the art—conformer XXL + noisy student [255]—is a combination of self-
training and SSL techniques. Given the diversity of the listed methods in
experiment settings (e.g., pre-training corpora and objectives, whether a
language model is used in decoding, whether model parameters are frozen in
fine-tuning), readers should be careful that the superiority of methods cannot
be decided only based on lower WER numbers.

###  IV-E Benchmark results and discussion

Given the diversity of datasets and downstream tasks used to evaluate SSL
techniques in the literature, it is infeasible to discuss all experiment
settings in this survey. Hence, due to their wide adoption for experiments
conducted by studies in both SSL and the speech community in general, we focus
first on ASR on the LS dataset to understand the efficacy of SSL. We examine
SSL techniques which report ASR results on the LS test-clean split, and
summarize the published WER in Figure 3. The ASR models were obtained first by
using unlabeled speech to pre-train a model with each SSL technique. The model
was then fine-tuned on labeled data by utilizing a supervised training
objective. Respectively, 960, 100, 10, 1 hour(s), and 10 minutes of labeled LS
training data were used for fine-tuning, as indicated in different panels of
Figure 3 (see the caption of Figure 3 for more details). Semi-supervised
methods such as self-training, where a model is first trained on labeled data
to annotate unlabeled speech, and then subsequently trained on combined golden
and self-annotated label-speech pairs, are gaining popularity in the speech
community and have yielded competitive results. For comparison, we also show
performance from such methods (iterative pseudo labeling (IPL) [63], slimIPL
[254], noisy student [62]), as well as the current state of the art—conformer
XXL + noisy student [255]—which augments SSL with various advanced techniques
including self-training. Furthermore, we illustrate in the figure the
performance of a baseline system [44] based on log mel filterbank (fBANK),
which is one of the most commonly used features designed by domain experts. As
observed in the figure, most SSL techniques outperform fBANK features, and
with the growing investment in model size, better performance is achieved. The
largest ones, such as wav2vec 2.0-L and HuBERT-L/XL, yield competitive results
when the entire 960-hour of labeled data is used in training/fine-tuning. The
benefit of SSL, especially models with more parameters like wav2vec 2.0 and
HuBERT, becomes more evident when the labeling resources become scarce.
Compared to popular semi-supervised methods such as IPL, slimIPL, and noisy
student using 100 hours of labels, wav2vec 2.0 and HuBERT achieve lower or
competitive WERs with 1 hour or even 10 minutes of labeled examples. The
results are highly favorable for low-resource use cases, for instance when
expanding systems to new domains or languages for which large amounts of
unlabeled audio are available, since collecting labels for new conditions is
often prohibitively slow or costly.

In addition to the ASR task, where the current state of the art is achieved by
a method combining SSL pre-training and self-training techniques [255], SSL
models are competitive in other tasks, including IC, SID, ASV, and QbE. We
summarize the performance of these models and previous non-SSL methods in
Table V. The results suggest that the benefit of SSL is generalizable among
tasks that require encoding information such as content, speaker, and
semantics. As SSL research gains more attention, we expect that SSL pre-
trained models will achieve state-of-the-art results on an increasing number
of tasks.

TABLE V: Tasks where the state of the art is models with SSL pre-training. Tasks | Dataset | non-SSL | SSL
---|---|---|---
ASR (WER $\downarrow$) | LS test-clean/other |  2.1/4.0 [63] |  1.4/2.6 [255]
IC (Acc $\uparrow$) | FSC |  98.8 [237] |  99.3[219]
SID (Acc $\uparrow$) | VoxCeleb1 |  94.8 [256] |  95.5 [131]
ASV (EER $\downarrow$) | VoxCeleb1 |  3.1 [257] |  2.4 [258]
QbE (MTWV $\uparrow$) | QUESST (EN) |  10.6 [259] |  11.2[219]

Despite the obvious trend of increasing performance as more parameters and SSL
pre-training data are being used, numbers in Figure 3 and Table V are less
comparable than might be expected. The task performance is obtained from the
original papers and is often achieved with different downstream fine-tuning
recipes, including various language models (used in the ASR system),
prediction heads (networks added to SSL for downstream inference), or choices
between fine-tuning the whole networks or freezing the SSL encoders. For
example, in the ASR task, HuBERT-L and wav2vec 2.0-L leverage Transformer as
their language model, while a 4-gram language model trained on LS is used in
DeCoAR 2.0. The lack of common and established mechanisms to evaluate SSL
techniques in downstream applications makes it difficult to compare techniques
fairly and understand their capabilities. To address this challenge, there are
increasing efforts to establish benchmarks with shared downstream tasks,
datasets, and downstream recipes. Such efforts include SUPERB [44],
LeBenchmark [260], ZeroSpeech [261], HEAR [262], NOSS [263], and HARES [264].

SUPERB [44] is a benchmarking platform that allows the SSL community to train,
evaluate, and compare speech representations on diverse downstream speech
processing tasks, from acoustic and speaker identity to paralinguistic and
semantics. SUPERB consolidates downstream recipes to focus on common and
straightforward settings (e.g., prediction head architectures, language
models, hyperparameter spaces) to facilitate generalizable and reproducible
benchmarking of SSL techniques. SUPERB also encourages researchers to innovate
for efficient use of model parameters and computation resources to democratize
SSL beyond race among Big Tech. LeBenchmark [260] shares a vision similar to
SUPERB and provides a reproducible framework for assessing SSL in French with
ASR, spoken language understanding, speech translation, and emotion
recognition. ZeroSpeech [261] (described in more detail in Section VI-B1)
challenges the scientific community to build speech and language understanding
systems using zero expert resources for millions of users of “low-resource”
languages. SSL techniques are also benchmarked with the ZeroSpeech challenge
[265, 266]. Apart from the speech community, researchers have also established
HEAR (holistic evaluation of audio representations) [262], NOSS (non-semantic
speech benchmark) [263], and HARES (holistic audio representation evaluation
suite) [264] to benchmark audio representations. These efforts promote the
creation of an audio embedding that is as holistic as the human ear in
interpreting speech, environmental sound, and music. Given the significant
need to understand and compare SSL techniques fairly and comprehensively, we
expect SSL benchmarking to remain an active research area.

##  V Analysis of Self-Supervised Representations

The previous sections have shown how self-supervised learning can result in
powerful representations that provide a robust starting point for several
downstream tasks. It is natural to ask if we can gain an even deeper
understanding of the nature of these representations, in order to further
optimize them or apply them to different problems. What is the information
encoded in these representations? How robust are they to distributional
shifts, and how dependent are they on the size of the training data? Do they
generalize across languages? What are the key ingredients for training
powerful representations: input data, network architecture, training
criterion, or all three? Can we predict their performance on downstream tasks
from their training behavior? This section tries to answer these questions by
summarizing several studies that analyze self-supervised representations.

###  V-A Information content

In [68] wav2vec 2.0 representations were analyzed with respect to their
acoustic-linguistic information content at different network layers. Three
different mechanisms were used for this purpose. The first of these is
canonical correlation analysis (CCA), which computes similarity scores between
two continuous vectors based on the maximum correlation of their linear
projections. These can be used to judge the similarity of embeddings at
different layers with each other, with standard acoustic representations such
as mel filterbank features, or word embeddings derived from text. The second
method clusters continuous representation vectors and computes the discrete
mutual information between cluster IDs and phone or word labels. The third
method involves probing tasks: representation vectors extracted from the
network are used to perform simple downstream tasks, in particular determining
whether two acoustic segments correspond to the same word, and a standard
benchmark of 11 word similarity tasks [267]. These are mostly used to gauge
the amount of lexical information present in the embeddings. Using this
battery of tests the authors compared pre-trained models of varying sizes as
well as models fine-tuned for ASR. They found that pre-trained models show an
autoencoder-style behavior, with early layers showing strong similarity with
input features, intermediate layers diverging more, and final layers reverting
back to higher similarity with input features and early layers. Generally, the
earlier layers in wav2vec 2.0 models encode acoustic information. The next set
of layers encodes phonetic class information, followed by word meaning
information, before reverting back to encoding phonetic/acoustic information.
Thus, extracting representations from the last layers for tasks that require
phonetic or word-related information may not be the best strategy. Indeed, the
authors of [268] show that a phone classifier trained on each of the 24 frozen
layers of a wav2vec 2.0 model showed the lowest phone error rates for layers
10–21 and higher error rates for the other layers. [68] further show that
fine-tuning the pre-trained model with a character-level CTC training
criterion changes the behavior of the last layers (especially the final two
layers), breaking the autoencoder-style behavior and focusing the information
encoded in the last layers on orthographic-phonetic and word information.

The peaking of class-relevant information in intermediate layers seems to be
common across different self-supervised learners and different modalities. In
an analysis of text-based Transformers trained with a masked language model
criterion [269] observed a similar compression plus reconstruction pattern.
Interestingly, similar network behavior was also recently described for self-
supervised learners in computer vision: using a contrastive self-supervised
learner (SimCLR) that optimizes for augmentation invariance, [270] show that
it is the intermediate representations that most closely approximate
information learned in a supervised way, i.e., they provide more class
information than the representations from final layers. This is similar to the
findings described above for wav2vec 2.0 without fine-tuning, where
intermediate layers provide more information about phone and word classes.

Self-supervised representations may encode other information besides phonetic
classes or words, for example, channel, language, speaker, and sentiment
information. It is shown that the per-utterance mean of CPC features captures
speaker information to a large extent[271]. Location of information pertaining
to speakers vs. language classes was analyzed in [272] for a 12-layer
BERTphone model. This model combines a self-supervised masked reconstruction
loss with a phone-based CTC loss to produce representations for speaker
recognition and language identification. By analyzing the weights of a linear
combination of layer representations for these two downstream tasks, it was
shown that language recognition draws on representation from higher layers
(peaking at layer 10) whereas speaker recognition benefited from layers at
positions 6, 9, and 12. This may indicate that language recognition relies
more on higher-level phonetic information whereas speaker recognition uses a
combination of acoustic and phonetic information. In a recent study [131] the
same technique was used to identify layer contributions for the downstream
SUPERB benchmark tasks in the WavLM model. For a smaller model (95M
parameters) it was again confirmed that lower layers encode speaker-related
information necessary for speaker diarization and verification whereas higher
layers encode phonetic and semantic information. Another study [273] used
explicit self-supervised loss at the intermediate layers rather than just the
output layer of a HuBERT model in order to enforce better learning of phonetic
information. The resulting model was indeed better at downstream tasks
requiring information about phonetic content, such as phone recognition, ASR,
and keyword spotting, but worse at speaker-related tasks like speaker
diarization and verification.

Most self-supervised learning approaches rely on a Transformer architecture
for the representation model. In [274] the attention patterns in generatively
trained Transformer representation models were analyzed. Self-attention heads
were grouped into three categories: diagonal, vertical, and global. It was
found that the diagonal head focuses on neighbors and is highly correlated
with phoneme boundaries, whereas the vertical head focuses on specific
phonemes in the utterance. Global heads were found to be redundant as removing
them resulted in faster inference time and higher performance.

###  V-B Training criterion

In [275], representations based on different training criteria (masked
predictive coding, contrastive predictive coding, and autoregressive
predictive coding) were compared and analyzed with respect to the correlation
between their training loss and performance on both phone discrimination and
speaker classification probing tasks. It was observed that the autoregressive
predictive coding loss showed the strongest correlation with downstream
performance on both tasks; however, models were not further analyzed
internally. An evaluation of the similarity of representations trained
according to the three criteria above (but with different architectures and
directionality of contextual information) also showed that it is the training
criterion that most influences the information encoded in the representations,
not the architecture of the learner or the directionality of the input.

A similar insight was obtained in [276], which compared vq-vae and vq-wav2vec
with respect to their ability to discover phonetic units. The vq-vae model
extracts continuous features from the audio signal; a quantizer then maps them
into a discrete space, and a decoder is trained to reconstruct the original
audio conditioned on the latent discrete representation and the past acoustic
observations. By contrast, vq-wav2vec predicts future latent discrete
representations based on contextualized embeddings of past discrete
representations, in a CPC-style way. The models were evaluated according to
their ability to discover phonetic units (as measured by phone recognition
error rate on TIMIT, and the ZeroSpeech ABX task (see Section VI for more
details)), and it was found that the predictive vq-wav2vec model fared better
than the autoencoder-like vq-vae model, most likely due to its superior
ability to model temporal dynamics.

###  V-C Effects of data and model size

How does the performance of self-supervised models change in relation to the
amount of training data, and in relation to the size (number of parameters) of
the model? Several studies have demonstrated better downstream performance
when using larger datasets [277, 123, 131]. For example, [123] compared
representations learned by a bidirectional CPC model from the standard 960
hour LS corpus and a corpus of 8,000 hours of diverse speech from multiple
sources.IV-C Not surprisingly, an ASR model trained on top of these
representations performed better when representations were learned from the
larger dataset. Although the precise relationship between data size and
performance has not been quantified, we can assume that it follows a law of
diminishing returns (or power law), similar to observations for most data-
intensive machine learning tasks. In addition to the size of the dataset, the
diversity of the data also seems to play a role, although this was not
quantified in this study. However, recent experiments with larger and more
diverse data collections [131] confirm this assumption, as do explicit
investigations of domain shift robustness (see Section V-D below).

The relation between model sizes and downstream performances have also been
investigated [278, 279]. Using the Mockingjay model [100], the authors in
[278] attempt to establish a relationship between model size and self-
supervised $L_{1}$ loss and demonstrate that it approximately follows a power
law. Model size and accuracy on downstream phone classification and speaker
recognition tasks are positively correlated but do not exactly follow a power
law; rather, the accuracy saturates as models increase in size, possibly due
to the lack of a corresponding expansion in training data size.

###  V-D Robustness and transferability

It is well known that traditional speech features like MFCCs lack robustness
against environmental effects such as additive noise, reverberation, accents,
etc., that cause differences in the distributions of speech features. Do pre-
trained representations offer greater robustness against distributional
shifts? One study [123] compared pre-trained representations from a CPC model
against MFCCs and found pre-trained representations to be more robust to
mismatches between training and test data. The training data consisted of
clean, read speech (LS) whereas test data consisted of the Switchboard corpus
and TED talks. The distributional shifts here may stem from both the acoustics
(microphone, room reverberation) as well as lexical effects related to topic
and style, as well as differences in speaker characteristics such as accent.
Similar problems were also investigated using HuBERT and wav2vec 2.0 models in
[280]. In [281] domain effects were studied in greater detail using datasets
from six different domains. In particular, the authors focused on the
usefulness of adding out-of-domain data to pre-training. The general
conclusions are that pre-training on more and diverse domains is preferable:
models pre-trained on more domains performed better than those pre-trained on
fewer when tested on held-out domains, regardless of which additional labeled
data was used for fine-tuning. Adding in-domain unlabeled data—if available—to
pre-training improves performance robustly; however, even out-of-domain
unlabeled data is helpful and closes 66–73% of the performance gap between the
ideal setting of in-domain labeled data and a competitive supervised out-of-
domain model.

In [277] the effectiveness of CPC-trained representations for phone
discrimination tasks was compared across several languages. It was found that
representations pre-trained only on English successfully enabled phone
discrimination in 10 other languages, rivaling supervised methods in accuracy
in low-data regimes (1h of labeled data per language). Thus, self-supervised
pre-training enables the model to learn contextualized speech features that
generalize across different languages. In [282], a wav2vec 2.0 model was
trained on data from multiple different languages and different corpora
(Babel, Common Voice, and multilingual LS) jointly, followed by fine-tuning
for each individual language. The largest model covers 53 languages in total
and consists of 56,000 hours of speech. Compared to monolingual pre-training,
even smaller models trained on only ten languages improve performance
substantially on a downstream character-based ASR task. Low-resource languages
with little labeled data improve the most under this training regime.
Multilingual representations also resulted in competitive performance (lower
character error rate than monolingual representations) for languages not
present in the training dataset, again showing that unsupervised pre-trained
representations can learn generic features of the speech signal that
generalize across different languages. The study also found that sharing data
from closely related languages is more beneficial than combining distant
languages. An analysis of language clusters in the shared discrete latent
representation space revealed that similar languages do indeed show a higher
degree of sharing of discrete tokens. Finally, one might ask whether the
interpretation of representations extracted from different layers of a self-
supervised models also generalizes to the multilingual setting. Experiments in
[268] on phone recognition in eight languages based on the different layers of
the multilingual wav2vec 2.0 XLSR-53 model indicate that this is indeed the
case: phone error rates showed the same pattern as in the monolingual
(English) scenario, with lower phone error rates for middle layers as opposed
to earlier/later layers.

##  VI From Representation Learning to Zero Resources

In the SSL framework, speech representations can be learned and used in
various downstream tasks to achieve competitive, robust, and transferable
performance, as shown in Sections IV to V. However, labeled data is still
required. For example, in ASR, utterances and their manual transcriptions are
needed to learn downstream models or fine-tune representation models. Can a
model learn without any labeled data? In Section VI-A, we show how to learn
ASR models without any paired audio and text and how SSL improves the
framework. In addition, many languages have no writing system. In Section
VI-B, the SSL representation is further used in scenarios where text data is
unavailable.

###  VI-A Unpaired text and audio

TABLE VI:  Unsupervised ASR. TIMIT numbers are phoneme error rates (PER),
while the numbers for LibriSpeech are word error rates (WER). SWC $=$ spoken
word classifier, ST $=$ speech translation. All speech and text are in English
if not specified.  The references in the table are sorted according to the
date of publication.

Reference | Speech representation | Speech segmentation | Text token/representation | Mapping approach | Refinement | Results
---|---|---|---|---|---|---
[283] | Audio word2vec [284] | Oracle | Phoneme | Adversarial Training [285] | - | TIMIT (PER): 63.6%
[286] | Speech2vec [114] | BES-GMM [196] | Word2Vec | Adversarial Training [287] | Self-training | SWC (Acc): 10.9%
[288] |  | Speech2vec
---
(English)
Oracle |  | Word2Vec
---
(French)
VecMap [289] |  | LM rescore,
---
sequence DAE
ST (BLEU): 10.8%
[290] | MFCC | GAS [291] | Phoneme | Empirical-ODM [292] | Self-training | TIMIT (PER): 41.6%
[293] | MFCC | GAS | Phoneme | Adversarial Training [285] | Self-training | TIMIT (PER): 33.1%
[268] | Wav2vec 2.0 [125] | k-means | Phoneme | Adversarial Training [285] | Self-training |  | TIMIT (PER): 18.6%,
---
LibriSpeech (WER): 5.9%
[294] |  | Universal Phone
---
Recogniser
- | Grapheme | Decipherment [295] | Self-training |  | GlobalPhone: 32.5% to just 1.9%
---
worse than supervised models
[296] | Wav2vec 2.0 [125] | - | Phoneme | Adversarial Training [285] | Self-training | LibriSpeech (WER): 6.3%
[296] | Wav2vec 2.0 [125] | - | Grapheme | Adversarial Training [285] | Self-training | LJSpeech (WER): 64.0%

####  VI-A1 Unsupervised ASR

If only unpaired speech and text are available, that is, the text is not a
manual transcription of speech, can the machine learn how to transcribe speech
into text? This scenario is called unsupervised ASR, and the framework is as
below. Given a set of unlabeled utterances
$\mathcal{S}=\\{S_{1},S_{2},...,S_{N}\\}$ and a set of sentences
$\mathcal{Y}=\\{Y_{1},Y_{2},...,Y_{M}\\}$,232323Note that the speech and text
are not paired, that is, $Y_{i}$ is not the transcription of $S_{i}$. a
mapping function $F$, which can take an utterance $S$ as input and generate
its transcription, is learned from data. Table VI summarizes recent work on
unsupervised ASR, including the speech representation used, the algorithm used
to learn the mapping without supervision, and the results. Below, we will
discuss these methods in more detail.

Adversarial training [297, 298, 285] is one common way to learn such a mapping
function. The framework includes a discriminator and a generator. The mapping
function $F$ plays the role of the generator, which takes speech utterances as
input and outputs text. The discriminator learns to distinguish real text from
the generated output; the generator learns to “fool” the discriminator. The
generator and the discriminator are trained in an iterative, interleaved way.
After the training, the generator serves as the speech recognition model.
There is a large amount of work using gradient penalty in the objective of
training discriminators [283, 293, 268, 296], which is inspired by Improved
Wasserstein Generative Adversarial Network (WGAN) [285].  Other ways to map
speech and text include via segmental empirical output distribution matching
(segmental empirical-ODM) [290] and decipherment algorithm [294].

Success in unsupervised neural machine translation (MT) [299, 287, 300] has
inspired innovative exploration of various unsupervised ASR algorithms. If
learning a translation model from unaligned sentences in two languages is
possible, considering speech and text as two different languages, learning the
mapping relationship from speech space to text space without an alignment
should likewise be possible. However, there are differences between
unsupervised MT and unsupervised ASR. In unsupervised MT, most discrete source
tokens can be mapped to specific target tokens representing the same meaning.
However, because speech has segmental structures, in unsupervised ASR, each
text token maps to a segment of consecutive acoustic features of variable
length in an utterance. The generator is supposed to learn the segmental
structure of an utterance because information like token boundaries is not
directly available. This makes unsupervised ASR more challenging than
unsupervised MT.

For unsupervised ASR to be feasible, the common idea is to make the speech and
text units close to each other. For the text side, word sequences can be
transformed into phoneme sequences if a lexicon is available. On the other
hand, we must first convert the speech signal into something close to
phonemes. To achieve that, most studies on unsupervised ASR use a phoneme
segmentation module before the generator to segment utterances into phoneme-
level segments [283, 293, 290]. A representation vector or a token then
represents each phoneme-level segment. It is easier for the generator to map
each segment-level representation or token to the correct phoneme when the
representation or token is highly correlated to the phonemes. Wav2vec-U [268]
selects the input feature from different layers of wave2vec 2.0 [125]. The
selection criterion is based on analysis of the phonetic information in each
layer.  If a universal phone recognizer trained from a diverse set of
languages is available, it is another way to transcribe speech into phone-
level tokens [294]. Another series of work is to transform a word into a word
embedding. [286, 288] use adversarial training to map the word-level speech
embedding space [114] to the word embedding space and achieve promising
performance on spoken word classification, speech translation, and spoken word
retrieval. Table VI summarizes the various ways to segment speech and
represent speech and text in each reference.

As shown in Table VI, most studies use self-training to refine the models. In
self-training, the generator serves as the first-version phoneme recognition
model. Inputting unpaired speech to the generator generates the corresponding
“pseudo transcription”. We then view the speech utterances and their pseudo
transcriptions as paired data which we use to train a model in a supervised
manner. Although the pseudo transcriptions have more errors than oracle
transcriptions, experiments show that training models on pseudo transcriptions
still significantly boosts performance compared to the first-version model.

Wav2vec-U [268] achieved state-of-the-art results at the time, which suggests
that representation learning is essential for the success of unsupervised ASR.
It achieved an 11.3% phoneme error rate on the TIMIT benchmark. On the LS
benchmark, wav2vec-U achieved a 5.9% WER on _test-other_ , rivaling some of
the best published systems trained on 960 hours of labeled data from only two
years earlier.  And wav2vec-U 2.0 [296] further removes the requirement of the
segmentation stage, so the unsupervised ASR model can be learned in an end-to-
end style. The robustness of wav2vec-U was further analyzed with respect to
domain-mismatch scenarios in which the domains of unpaired speech and text
were different [301]. Experimental results showed that domain mismatch leads
to inferior performance, but a representation model pre-trained on the
targeted speech domain extracts better representations and reduces this drop
in performance.

####  VI-A2 ASR-TTS

Here we describe an alternative approach by which to train an ASR and text-to-
speech (TTS) system based on unpaired text and audio. The ASR-TTS framework,
which combines the ASR and TTS systems in a cascaded manner, can be regarded
as an autoencoder, where the encoder $f$ corresponds to the ASR module and the
decoder $g$ corresponds to the TTS module. In this framework, we consider the
intermediate ASR output as a latent representation; the framework as a whole
can be regarded as a variant of self-supervised learning.242424However, to
make this complicated system work, we often require that data is paired.
Therefore, in practice, ASR-TTS and other methods described in this section
are categorized as semi-supervised learning.

The ASR-TTS framework can jointly optimize both ASR and TTS without using
paired data [302, 303, 304]. A speech chain [302, 305] is one successful way
to utilize audio-only and text-only data to train both end-to-end ASR/TTS
models. This approach first prepares pre-trained ASR model
$f_{\mathrm{asr}}(X)$ with acoustic input $X$ and pre-trained TTS model
$g_{\mathrm{tts}}(Y)$ with text input $Y$. By following the TTS system with an
ASR system, we generate new acoustic feature sequence $\hat{X}$, which must be
close to the original input $X$. Thus, we design a loss function
$\mathcal{L}_{\mathrm{asr}\rightarrow\mathrm{tts}}(X,\hat{X})$, where
$\hat{X}$ is generated by

$\hat{X}=g_{\mathrm{tts}}(f_{\mathrm{asr}}(X)).$ (36)

Thus, we train the ASR model (or both ASR and TTS models) using only the
acoustic input by minimizing
$\mathcal{L}_{\mathrm{asr}\rightarrow\mathrm{tts}}$. Note that this approach
does not require the supervised text data $Y$. As an analogy to the generative
approach in Section III-B, the intermediate ASR output $\hat{Y}$ can be
regarded as the latent representation $Z$.

The other cycle with the text-only data $Y$ is also accomplished by the
concatenated TTS-ASR systems:

$\hat{Y}=f_{\mathrm{asr}}(g_{\mathrm{tts}}(Y)).$ (37)

Similarly, this approach does not require the supervised audio data $X$, and
the intermediate TTS output $\hat{X}$ can be regarded as the latent
representation $Z$. Although this approach initially freezes either the ASR or
TTS model, extensions of this study [303, 306, 307] implement the joint
training of both ASR and TTS parameters using REINFORCE [308] and straight-
through estimators.

An emerging technique uses a well-trained TTS system to generate speech and
text data from text-only data. This technique is a sub-problem of the TTS-ASR
approach formulated in (37) in which we fix the TTS system part and estimate
only the ASR parameters. For example, a huge amount of text resources can be
obtained from the web and document archives without corresponding audio data.
The typical use case scenario of such a text resource for ASR is through the
language model. We combine the ASR and language model via a noisy channel
model [309], a weighted finite state transducer [310], or shallow fusion [311,
312]. However, the progress of TTS systems boosted by deep learning [80, 313]
has inspired another interesting and straightforward research direction:
artificially creating paired text and audio data $\\{\hat{X},Y\\}$ with only
text data $Y$ by generating the corresponding audio data $\hat{X}$ with TTS.
The most straightforward approach is to simply use multi-speaker TTS to
generate the waveform with various acoustic variations [314, 315, 316, 317,
318]. The other approaches are based on the generation of high-level (more
linguistic) features instead of generating the waveform, e.g., encoder
features [319] and phoneme features [320, 321]. This approach is similar to
the back-translation technique developed in neural machine translation [322].
One benefit of the above data generation approaches is that it can be used to
feed unseen word or context phrases to end-to-end ASR.

###  VI-B No text or lexicon

####  VI-B1 Zero-resource speech technologies and challenges

Zero-resource speech technologies, which seek to discover linguistic concepts
from audio only (no text nor lexicon), are one of the most active applications
of unsupervised/self-supervised speech processing. Zero-resource speech
technologies were initially studied for acoustic and linguistic unit discovery
from speech data without linguistic resources, e.g., transcriptions and other