# A Review of Deep Learning Techniques for Speech Processing

Ambuj Mehrish<EMAIL_ADDRESS>Singapore University of Technology
and DesignSingapore ,  Navonil Majumder  Singapore University of Technology
and DesignSingapore ,  Rishabh Bhardwaj  Singapore University of Technology
and DesignSingapore ,  Rada Mihalcea  University of MichiganUSA and  Soujanya
Poria  Singapore University of Technology and DesignSingapore

###### Abstract.

The field of speech processing has undergone a transformative shift with the
advent of deep learning. The use of multiple processing layers has enabled the
creation of models capable of extracting intricate features from speech data.
This development has paved the way for unparalleled advancements in speech
recognition, text-to-speech synthesis, automatic speech recognition, and
emotion recognition, propelling the performance of these tasks to
unprecedented heights. The power of deep learning techniques has opened up new
avenues for research and innovation in the field of speech processing, with
far-reaching implications for a range of industries and applications. This
review paper provides a comprehensive overview of the key deep learning models
and their applications in speech-processing tasks. We begin by tracing the
evolution of speech processing research, from early approaches, such as MFCC
and HMM, to more recent advances in deep learning architectures, such as CNNs,
RNNs, transformers, conformers, and diffusion models. We categorize the
approaches and compare their strengths and weaknesses for solving speech-
processing tasks. Furthermore, we extensively cover various speech-processing
tasks, datasets, and benchmarks used in the literature and describe how
different deep-learning networks have been utilized to tackle these tasks.
Additionally, we discuss the challenges and future directions of deep learning
in speech processing, including the need for more parameter-efficient,
interpretable models and the potential of deep learning for multimodal speech
processing. By examining the field’s evolution, comparing and contrasting
different approaches, and highlighting future directions and challenges, we
hope to inspire further research in this exciting and rapidly advancing field.

††copyright: none Figure 1. Evolution of speech processing models over the
years.

###### Contents

  1. 1 Introduction
  2. 2 Background
    1. 2.1 Speech Signals
    2. 2.2 Speech Features
    3. 2.3 Traditional models for speech processing
  3. 3 Deep Learning Architectures and Their Applications in Speech Processing Tasks
    1. 3.1 Recurrent Neural Networks (RNNs)
    2. 3.2 Convolutional Neural Networks
    3. 3.3 Transformers
    4. 3.4 Conformer
    5. 3.5 Sequence to Sequence Models
    6. 3.6 Reinforcement Learning
    7. 3.7 Graph Neural Network
    8. 3.8 Diffusion Probabilistic Model
  4. 4 Speech Representation Learning
    1. 4.1 Supervised Learning
    2. 4.2 Unsupervised learning
    3. 4.3 Semi-supervised Learning
    4. 4.4 Self-supervised representation learning (SSRL)
  5. 5 Speech Processing Tasks
    1. 5.1 Automatic speech recognition (ASR) & conversational multi-speaker AST
    2. 5.2 Neural Speech Synthesis
    3. 5.3 Speaker recognition
    4. 5.4 Speaker Diarization
    5. 5.5 Speech-to-speech translation
    6. 5.6 Speech enhancement
    7. 5.7 Audio Super Resolution
    8. 5.8 Voice Activity Detection (VAD)
    9. 5.9 Speech Quality Assessment
    10. 5.10 Speech Separation
    11. 5.11 Spoken Language Understanding
    12. 5.12 Audio/visual multimodal speech processing
  6. 6 Advanced Transfer Learning Techniques for Speech Processing
    1. 6.1 Domain Adaptation
    2. 6.2 Meta Learning
    3. 6.3 Parameter-Efficient Transfer Learning
  7. 7 Conclusion and Future Research Directions

##  1\. Introduction

Humans employ language as a means to effectively convey their emotions and
sentiments. Language encompasses a collection of words forming a vocabulary,
accompanied by grammar, which dictates the appropriate usage of these words.
It manifests in various forms, including written text, sign language, and
spoken communication. Speech, specifically, entails the utilization of
phonetic combinations of consonant and vowel sounds to articulate words from
the vocabulary. Phonetics, in turn, pertains to the production and perception
of sounds by individuals. Through speech, individuals are able to express
themselves and convey meaning in their chosen language.

Speech processing is a field dedicated to the study and application of methods
for analyzing and manipulating speech signals. It encompasses a range of
tasks, including automatic speech recognition (ASR) (Yu and Deng, 2016; Nassif
et al., 2019), speaker recognition (SR) (Bai and Zhang, 2021), and speech
synthesis or text-to-speech (Ning et al., 2019). In recent years, speech
processing has garnered increasing significance due to its diverse
applications in areas such as telecommunications, healthcare, and
entertainment. Notably, statistical modeling techniques, particularly Hidden
Markov Models (HMMs), have played a pivotal role in advancing the field (Gales
et al., 2008; Rabiner, 1989). These models have paved the way for significant
advancements and breakthroughs in speech processing research and development.

Over the past few years, the field of speech processing has been transformed
by introducing powerful tools, including deep learning. Figure 1 illustrates
the evolution of speech processing models over the years, the rapid
development of deep learning architecture for speech processing reflects the
growing complexity and diversity of the field. This technology has
revolutionized the analysis and processing of speech signals using deep neural
networks (DNNs), convolutional neural networks (CNNs), and recurrent neural
networks (RNNs). These architectures have proven highly effective in various
speech-processing applications, such as speech recognition, speaker
recognition, and speech synthesis. This study comprehensively overviews the
most critical and emerging deep-learning techniques and their potential
applications in various speech-processing tasks.

Deep learning has revolutionized speech processing by its ability to
automatically learn meaningful features from raw speech signals, eliminating
the need for manual feature engineering. This breakthrough has led to
significant advancements in speech processing performance, particularly in
challenging scenarios involving noise, as well as diverse accents and
dialects. By leveraging the power of deep neural networks, speech processing
systems can now adapt and generalize more effectively, resulting in improved
accuracy and robustness in various applications. The inherent capability of
deep learning to extract intricate patterns and representations from speech
data has opened up new possibilities for tackling real-world speech processing
challenges.

Deep learning architectures have emerged as powerful tools in speech
processing, offering remarkable improvements in various tasks. Pioneering
studies, such as (Hinton et al., 2012), have demonstrated the substantial
gains achieved by deep neural networks (DNNs) in speech recognition accuracy
compared to traditional HMM-based systems. Complementing this, research in
(Abdel-Hamid et al., 2014a) showcased the effectiveness of convolutional
neural networks (CNNs) for speech recognition. Moreover, recurrent neural
networks (RNNs) have proven their efficacy in both speech recognition and
synthesis, as highlighted in (Graves et al., 2013). Recent advancements in
deep learning have further enhanced speech processing systems, with attention
mechanisms (Chorowski et al., 2015) and transformers (Vaswani et al., 2017)
playing significant roles. Attention mechanisms enable the model to focus on
salient sections of the input signal, while transformers facilitate modeling
long-range dependencies within the signal. These developments have led to
substantial improvements in the performance and versatility of speech
processing systems, unlocking new possibilities for applications in diverse
domains.

Although deep learning has made remarkable progress in speech processing, it
still faces certain challenges that need to be addressed. These challenges
include the requirement for substantial amounts of labeled data, the
interpretability of the models, and their robustness to different
environmental conditions. To provide a comprehensive understanding of the
advancements in this domain, this paper presents an extensive overview of deep
learning architectures employed in speech-processing applications. Speech
processing encompasses the analysis, synthesis, and recognition of speech
signals, and the integration of deep learning techniques has led to
significant advancements in these areas. By examining the current state-of-
the-art approaches, this paper aims to shed light on the potential of deep
learning for tackling the existing challenges and further advancing speech
processing research.

The paper provides a comprehensive exploration of deep-learning architectures
in the field of speech processing. It begins by establishing the background,
encompassing the definition of speech signals, speech features, and
traditional non-neural models. Subsequently, the focus shifts towards an in-
depth examination of various deep-learning architectures specifically tailored
for speech processing, including RNNs, CNNs, Transformers, GNNs, and diffusion
models. Recognizing the significance of representation learning techniques in
this domain, the survey paper dedicates a dedicated section to their
exploration.

Moving forward, the paper delves into an extensive range of speech processing
tasks where deep learning has demonstrated substantial advancements. These
tasks encompass critical areas such as speech recognition, speech synthesis,
speaker recognition, speech-to-speech translation, and speech synthesis. By
thoroughly analyzing the fundamentals, model architectures, and specific tasks
within the field, the paper then progresses to discuss advanced transfer
learning techniques, including domain adaptation, meta-learning, and
parameter-efficient transfer learning.

Finally, in the conclusion, the paper reflects on the current state of the
field and identifies potential future directions. By considering emerging
trends and novel approaches, the paper aims to shed light on the evolving
landscape of deep learning in speech processing and provide insights into
promising avenues for further research and development.

##### Why this paper?

Deep learning has become a powerful tool in speech processing because it
automatically learns high-level representations of speech signals from raw
audio data. As a result, significant advancements have been made in various
speech-processing tasks, including speech recognition, speaker identification,
speech synthesis, and more. These tasks are essential in various applications,
such as human-computer interaction, speech-based search, and assistive
technology for people with speech impairments. For example, virtual assistants
like Siri and Alexa use speech recognition technology, while audiobooks and
in-car navigation systems rely on text-to-speech systems.

Given the wide range of applications and the rapidly evolving nature of deep
learning, a comprehensive review paper that surveys the current state-of-the-
art techniques and their applications in speech processing is necessary. Such
a paper can help researchers and practitioners stay up-to-date with the latest
developments and trends and provide insights into potential areas for future
research. However, to the best of our knowledge, no current work covers a
broad spectrum of speech-processing tasks.

A review paper on deep learning for speech processing can also be a valuable
resource for beginners interested in learning about the field. It can provide
an overview of the fundamental concepts and techniques used in deep learning
for speech processing and help them gain a deeper understanding of the field.
While some survey papers focus on specific speech-processing tasks such as
speech recognition, a broad survey would cover a wide range of other tasks
such as speaker recognition speech synthesis, and more. A broad survey would
highlight the commonalities and differences between these tasks and provide a
comprehensive view of the advancements made in the field.

##  2\. Background

Before moving on to deep neural architectures, we discuss basic terms used in
speech processing, low-level representations of speech signals, and
traditional models used in the field.

###  2.1. Speech Signals

Signal processing is a fundamental discipline that encompasses the study of
quantities that exhibit variations in space or time. In the realm of signal
processing, a quantity exhibiting spatial or temporal variations is commonly
referred to as a signal. Specifically, sound signals are defined as variations
in air pressure. Consequently, a speech signal is identified as a type of
sound signal, namely pressure variations, generated by humans to facilitate
spoken communication. Transducers play a vital role in converting these
signals from one form, such as air pressure, to another form, typically an
electrical signal.

In signal processing, a signal that repetitively manifests after a fixed
duration, known as a period, is classified as periodic. The reciprocal of this
period represents the frequency of the signal. The waveform of a periodic
signal defines its shape and concurrently determines its timbre, which
pertains to the subjective perception of sound quality by humans. To
facilitate the processing of speech, speech signals are commonly digitized.
This entails converting them into a series of numerical values by measuring
the signal’s amplitude at consistent time intervals. The sampling rate,
defined by the number of samples collected per second, determines the
granularity of this digitization process.

###  2.2. Speech Features

Speech features are numerical representations of speech signals that are used
for analysis, recognition, and synthesis. Broadly, speech signals can be
classified into two categories: time-domain features and frequency-domain
features.

Time-domain features are derived directly from the amplitude of the speech
signal over time. These are simple to compute and often used in real-time
speech-processing applications. Some common time-domain features include:

  * •

Energy: Energy is a quantitative measure of the amplitude characteristics of a
speech signal over time. It is computed by squaring each sample in the signal
and summing them within a specific time window. This captures the overall
strength and dynamics of the signal, revealing temporal variations in
intensity. The energy measure provides insights into segments with higher or
lower amplitudes, aiding in speech recognition, audio segmentation, and
speaker diarization. It also helps identify events and transitions indicative
of changes in vocal activity. By quantifying amplitude variations, energy
analysis contributes to a comprehensive understanding of speech signals and
their acoustic properties.

  * •

Zero-crossing rate: The zero-crossing rate indicates how frequently the speech
signal crosses the zero-axis within a defined time frame. It is computed by
counting the number of polarity changes in the signal during a specific
window.

  * •

Pitch: Pitch refers to the perceived tonal quality in a speaker’s voice, which
is determined by analyzing the fundamental frequency of the speech signal. The
fundamental frequency can be estimated through the application of pitch
detection algorithms (Rabiner et al., 1976) or by utilizing autocorrelation
techniques (Tan and Karnjanadecha, 2003).

  * •

Linear predictive coding (LPC):Linear Predictive Coding (LPC) is a powerful
technique that represents the speech signal as a linear combination of past
samples, employing an autoregressive model. The estimation of model parameters
is accomplished through methods like the Levinson-Durbin algorithm
(Castiglioni, 2005). The obtained coefficients serve as a valuable feature
representation for various speech-processing tasks.

Frequency-domain features are derived from the signal represented in the
frequency domain also known as its spectrum. A spectrum captures the
distribution of energy as a function of frequency. Spectrograms are two-
dimensional visual representations capturing the variations in a signal’s
spectrum over time. When compared against time-domain features, it is
generally more complex to compute frequency-domain features as they tend to
involve time-frequency transform operations such as Fourier transform.

  * •

Mel-spectrogram: A Mel spectrogram, also known as a Mel-frequency spectrogram
or Melspectrogram, is a representation of the short-term power spectrum of a
sound signal. It is widely used in audio signal processing and speech
recognition tasks. It is obtained by converting the power spectrum of a speech
signal into a mel-scale, which is a perceptual scale of pitches based on the
human auditory system’s response to different frequencies. The mel-scale
divides the frequency range into a set of mel-frequency bands, with higher
resolution in the lower frequencies and coarser resolution in the higher
frequencies. This scale is designed to mimic the non-linear frequency
perception of human hearing. To compute the Melspectrogram, the speech signal
is typically divided into short overlapping frames. For each frame, the Fast
Fourier Transform (FFT) is applied to obtain the power spectrum. The power
spectrum is then transformed into the mel-scale using a filterbank that
converts the power values at different frequencies to their corresponding mel-
frequency bands. Finally, the logarithm of the mel-scale power values is
computed, resulting in the Melspectrogram.

Melspectrogram provides a time-frequency representation of the audio signal,
where the time dimension corresponds to the frame index, and the frequency
dimension represents the mel-frequency bands. It captures both the spectral
content and temporal dynamics of the signal, making it useful for tasks such
as speech recognition, music analysis, and sound classification. By using the
Melspectrogram, the representation of the audio signal is transformed to a
more perceptually meaningful domain, which can enhance the performance of
various audio processing algorithms. It is particularly beneficial in
scenarios where capturing the spectral patterns and frequency content of the
signal is important for the analysis or classification task at hand.

  * •

Mel-frequency cepstral coefficients (MFCCs): Mel-frequency cepstral
coefficients (MFCCs) are a feature representation widely utilized in various
applications such as speech recognition, gesture recognition, speaker
identification, and cetacean auditory perception systems. MFCCs capture the
power spectrum of a sound over a short duration by utilizing a linear cosine
transformation of a logarithmically-scaled power spectrum on a non-linear mel
frequency scale. The MFCCs consist of a set of coefficients that collectively
form a Mel-frequency cepstrum 111https://en.wikipedia.org/wiki/Mel-
frequency_cepstrum. With just 12 parameters related to the amplitude of
frequencies, MFCCs provide an adequate number of frequency channels to analyze
audio, while still maintaining a compact representation. The main objectives
of MFCC extraction are to eliminate vocal fold excitation (F0) information
related to pitch, ensure the independence of the extracted features, align
with human perception of loudness and frequency, and capture the contextual
dynamics of phones. The process of extracting MFCC features involves A/D
conversion, pre-emphasis filtering, framing, windowing, Fourier transform, Mel
filter bank application, logarithmic operation, discrete cosine transform
(DCT), and liftering. By following these steps, MFCCs enable the extraction of
informative audio features while avoiding redundancy and preserving the
relevant characteristics of the sound signal.

Other types of speech features include formant frequencies, pitch contour,
cepstral coefficients, wavelet coefficients, and spectral envelope. These
features can be used for various speech-processing tasks, including speech
recognition, speaker identification, emotion recognition, and speech
synthesis.

In the field of speech processing, frequency-based representations such as Mel
spectrogram and MFCC are widely used since they are more robust to noise as
compared to temporal variations of the sound (Abdul and Al-Talabani, 2022).
Time-domain features can be useful when the task warrants this information
(such as pauses, emotions, phoneme duration, and speech segments). It is
noteworthy that the time-domain and frequency-domain features tend to capture
different sets of information and thus can be used in conjunction to solve a
task (Sondhi and Schroeter, 1987; Wang and Wang, 2020; Tang et al., 2021).

###  2.3. Traditional models for speech processing

Traditional speech representation learning algorithms based on shallow models
utilize basic non-parametric models for extracting features from speech
signals. The primary objective of these models is to extract significant
features from the speech signal through mathematical operations, such as
Fourier transforms, wavelet transforms, and linear predictive coding (LPC).
The extracted features serve as inputs to classification or regression models.
The shallow models aim to extract meaningful features from the speech signal,
enabling the classification or regression model to learn and make accurate
predictions.

  * •

Gaussian Mixture Models (GMMs): Gaussian Mixture Models (GMMs) are powerful
generative models employed to represent the probability distribution of a
speech feature vector. They achieve this by combining multiple Gaussian
distributions with different weights. GMMs have found widespread applications
in speaker identification (Kinnunen et al., 2005) and speech recognition tasks
(Reynolds, 2003). Specifically, in speaker identification, GMMs are utilized
to capture the distribution of speaker-specific features, enabling the
recognition of individuals based on their unique characteristics. Conversely,
in speech recognition, GMMs are employed to model the acoustic properties of
speech sounds, facilitating accurate recognition of spoken words and phrases.
GMMs play a crucial role in these domains, enabling robust and efficient
analysis of speech-related data.

  * •

Support Vector Machines (SVMs): Support Vector Machines (SVMs) are a widely
adopted class of supervised learning algorithms extensively utilized for
various speech classification tasks (Smith and Gales, 2001). They are
particularly effective in domains like speaker recognition (Hatch et al.,
2006; Solomonoff et al., 2004, 2005) and phoneme recognition (Campbell et al.,
2003). SVMs excel in their ability to identify optimal hyperplanes that
effectively separate different classes in the feature space. By leveraging
this optimal separation, SVMs enable accurate classification and recognition
of speech patterns. As a result, SVMs have become a fundamental tool in the
field of speech analysis and play a vital role in enhancing the performance of
speech-related classification tasks.

  * •

Hidden Markov Models (HMMs): Hidden Markov Models (HMMs) have gained
significant popularity as a powerful tool for performing various speech
recognition tasks, particularly ASR (Gales et al., 2008; Rabiner, 1989). In
ASR, HMMs are employed to model the probability distribution of speech sounds
by incorporating a sequential arrangement of hidden states along with
corresponding observations. The training of HMMs is commonly carried out using
the Baum-Welch algorithm, a variant of the Expectation Maximization algorithm,
which enables effective parameter estimation and model
optimization222Wikipedia: Baum-Welch algorithm:
http://en.wikipedia.org/wiki/Baum%e2%80%93Welch_algorithm.

By leveraging HMMs in speech recognition, it becomes possible to predict the
most likely sequence of speech sounds given an input speech signal. This
enables accurate and efficient recognition of spoken language, making HMMs a
crucial component in advancing speech recognition technology. Their
flexibility and ability to model temporal dependencies contribute to their
widespread use in ASR and various other speech-related applications, further
enhancing our understanding and utilization of spoken language.

  * •

The K-nearest neighbors (KNN) algorithm is a simple yet effective
classification approach utilized in a wide range of speech-related
applications, including speaker recognition (Sadjadi et al., 2014) and
language recognition. The core principle of KNN involves identifying the
K-nearest neighbors of a given input feature vector within the training data
and assigning it to the class that appears most frequently among those
neighbors. This algorithm has gained significant popularity due to its
practicality and intuitive nature, making it a reliable choice for classifying
speech data in numerous real-world scenarios. By leveraging the proximity-
based classification, KNN provides a straightforward yet powerful method for
accurately categorizing speech samples based on their similarities to the
training data. Its versatility and ease of implementation contribute to its
widespread adoption in various speech-related domains, facilitating
advancements in speaker recognition, language identification, and other
applications in the field of speech processing.

  * •

Decision trees: Decision trees are widely employed in speech classification
tasks as a class of supervised learning algorithms. Their operation involves
recursively partitioning the feature space into smaller regions, guided by the
values of the features. Within each partition, a decision rule is established
to assign the input feature vector to a specific class. The strength of
decision trees lies in their ability to capture complex decision boundaries by
hierarchically dividing the feature space. By analyzing the values of the
input features at each node, decision trees efficiently navigate the
classification process. This approach not only provides interpretability, but
also facilitates the identification of key features contributing to the
classification outcome. Through their recursive partitioning mechanism,
decision trees offer a flexible and versatile framework for speech
classification. They excel in scenarios where the decision rules are based on
discernible thresholds or ranges of feature values. The simplicity and
transparency of decision trees make them a valuable tool for understanding and
solving speech-related classification tasks.

To summarize, conventional speech representation learning algorithms based on
shallow models entail feature extraction from the speech signal, which is
subsequently used as input for classification or regression models. These
algorithms have found extensive applications in speech processing tasks like
speech recognition, speaker identification, and speech synthesis. However,
they have been progressively superseded by more advanced representation
learning algorithms, particularly deep neural networks, due to their enhanced
capabilities.

##  3\. Deep Learning Architectures and Their Applications in Speech
Processing Tasks

Deep learning architectures have revolutionized the field of speech processing
by demonstrating remarkable performance across various tasks. With their
ability to automatically learn hierarchical representations from raw speech
data, deep learning models have surpassed traditional approaches in areas such
as speech recognition, speaker identification, and speech synthesis. These
architectures have been instrumental in capturing intricate patterns,
uncovering latent features, and extracting valuable information from vast
amounts of speech data. In this section, we delve into the applications of
deep learning architectures in speech processing tasks, exploring their
potential, advancements, and the impact they have had on the field. By
examining the key components and techniques employed in these architectures,
we aim to provide insights into the current state-of-the-art in deep learning
for speech processing and shed light on the exciting prospects it holds for
future advancements in the field.

###  3.1. Recurrent Neural Networks (RNNs)

It is natural to consider Recurrent Neural Networks for various speech
processing tasks since the input speech signal is inherently a dynamic process
(Salehinejad et al., 2017). RNNs can model a given time-varying (sequential)
patterns that were otherwise hard to capture by standard feedforward neural
architectures. Initially, RNNs were used in conjunction with HMMs where the
sequential data is first modeled by HMMs while localized classification is
done by the neural network. However, such a hybrid model tends to inherit
limitations of HMMs, for instance, HMM requires task-specific knowledge and
independence constraints for observed states (Bourlard and Morgan, 1994). To
overcome the limitations inherited by the hybrid approach, end-to-end systems
completely based on RNNs became popular for sequence transduction tasks such
as speech recognition and text(Graves, 2012; Kawakami, 2008). Next, we discuss
RNN and it’s variants:

#### 3.1.1 RNN Models

##### Vanilla RNN

Give input sequence of T states $(x_{1},\ldots,x_{T})$ with
$x_{i}\in\mathbb{R}^{d}$, the output state at time $t$ can be computed as

(1) $h_{t}=\mathcal{H}(W_{hh}h_{t-1}+W_{xh}x_{t}+b_{h})$ (2)
$y_{t}=W_{hy}h_{t}+b_{y}$

where $W_{hh},W_{hx},W_{yh}$ are weight matrices and $b_{h},b_{y}$ are bias
vectors. $\mathcal{H}$ is non-linear activation functions such as Tanh, ReLU,
and Sigmoid. RNNs are made of high dimensional hidden states, notice $h_{t}$
in the above equation, which makes it possible for them to model sequences and
help overcome the limitation of feedforward neural networks. The state of the
hidden layer is conditioned on the current input and the previous state, which
makes the underlying operation recursive. Essentially, the hidden state
$h_{t-1}$ works as a memory of past inputs $\\{x_{k}\\}_{k=1}^{t-1}$ that
influence the current output $y_{t}$.

##### Bidirectional RNNs

For numerous tasks in speech processing, it is more effective to process the
whole utterance at once. For instance, in speech recognition, one-shot input
transcription can be more robust than transcribing based on the partial (i.e.
previous) context information (Graves et al., 2013). The vanilla RNN has a
limitation in such cases as they are unidirectional in nature, that is, output
$y_{t}$ is obtained from $\\{x_{k}\\}_{k=1}^{t}$, and thus, agnostic of what
comes after time $t$. Bidirectional RNNs (BRNNs) were proposed to overcome
such shortcomings of RNNs (Schuster and Paliwal, 1997). BRRNs encode both
future and past (input) context in separate hidden layers. The outputs of the
two RNNs are then combined at each time step, typically by concatenating them
together, to create a new, richer representation that includes both past and
future context.

(3) $\displaystyle\overrightarrow{h_{t}}$
$\displaystyle=\mathcal{H}(W_{\overrightarrow{hh}}\overrightarrow{h}_{t-1}+W_{\overrightarrow{xh}}x_{t}+b_{\overrightarrow{h}})$
(4) $\displaystyle\overleftarrow{h_{t}}$
$\displaystyle=\mathcal{H}(W_{\overleftarrow{hh}}\overleftarrow{h}_{t+1}+W_{\overleftarrow{xh}}x_{t}+b_{\overleftarrow{h}})$
(5) $\displaystyle y_{t}$
$\displaystyle=W_{\overrightarrow{hy}}\overrightarrow{h}_{t}+W_{\overleftarrow{hy}}\overleftarrow{h}_{t}+b_{y}$

where high dimensional hidden states $\overrightarrow{h}_{t-1}$ and
$\overleftarrow{h}_{t+1}$ are hidden states modeling the forward context from
$1,2,\ldots,t-1$ and backward context from $T,T-1,\ldots,t+1$, respectively.

##### Long Short-Term Memory

Vanilla RNNs are observed to face another limitation, that is, vanishing
gradients that do not allow them to learn from long-range context information.
To overcome this, a variant of RNN, named as LSTM, was specifically designed
to address the vanishing gradient problem and enable the network to
selectively retain (or forget) information over longer periods of time
(Hochreiter and Schmidhuber, 1997). This attribute is achieved by maintaining
separate purpose-built memory cells in the network: the long-term memory cell
$c_{t}$ and the short-term memory cell $h_{t}$. In Equation 2, LSTM redefines
the operator $\mathcal{H}$ in terms of forget gate $f_{t}$, input gate
$i_{t}$, and output gate $o_{t}$,

(6) $\displaystyle i_{t}$
$\displaystyle=\sigma(W_{xi}x_{t}+W_{hi}h_{t-1}+W_{ci}c_{t-1}+b_{i}),$ (7)
$\displaystyle f_{t}$
$\displaystyle=\sigma(W_{xf}x_{t}+W_{hf}h_{t-1}+W_{cf}c_{t-1}+b_{f}),$ (8)
$\displaystyle c_{t}$ $\displaystyle=f_{t}\odot
c_{t-1}+i_{t}\odot\tanh{(W_{xc}x_{t}+W_{hc}h_{t-1}+b_{c})},$ (9)
$\displaystyle o_{t}$
$\displaystyle=\sigma(W_{xo}x_{t}+W_{ho}h_{t-1}+W_{co}c_{t}+b_{o}),$ (10)
$\displaystyle h_{t}$ $\displaystyle=o_{t}\odot\tanh{(c_{t})},$

where $\sigma(x)=1/({1+e^{-x}})$ is a logistic sigmoid activation function.
$c_{t}$ is a fusion of the information from the previous state of the long-
term memory $c_{t-1}$, the previous state of short-term memory $h_{t-1}$, and
current input $x_{t}$. $W$ and $b$ are weight matrices and biases. $\odot$ is
the element-wise vector multiplication or Hadamard operator. Bidirectional
LSTMs (BLSTMs) can capture longer contexts in both forward and backward
directions (Graves, 2012).

##### Gated Recurrent Units

Gated Recurrent Units (GRU) aim to be a computationally-efficient approximate
of LSTM by using only two gates (vs three in LSTM) and a single memory cell
(vs two in LSTM). To control the flow of information over time, a GRU uses an
update gate $z_{t}$ to decide how much of the new input to be added to the
previous hidden state and a reset gate $r_{t}$ to decide how much of previous
hidden state information to be forgotten.

(11) $\displaystyle z_{t}$ $\displaystyle=\sigma(W_{xz}x_{t}+W_{hz}h_{t-1}),$
(12) $\displaystyle r_{t}$ $\displaystyle=\sigma(W_{xr}x_{t}+W_{hr}h_{t-1}),$
(13) $\displaystyle h_{t}$ $\displaystyle=(1-z_{t})\odot
h_{t-1}+z_{t}\odot\tanh(W_{xh}x_{t}+W_{rh}(r_{t}\odot h_{t-1})),$

where $\odot$ is element-wise multiplication between the two vectors (Hadamard
product).

RNNs and their variants are widely used in various deep learning applications
like speech recognition, synthesis, and natural language understanding.
Although seq2seq based on recurrent architectures such as LSTM/GRU has made
great strides in speech processing, they suffer from the drawback of slow
training speed due to internal recurrence. Another drawback of the RNN family
is their inability to leverage information from long distant time steps
accurately.

##### Connectionist Temporal Classification

Connectionist Temporal Classification (CTC) (Graves and Graves, 2012) is a
scoring and output function commonly used to train LSTM networks for sequence-
based problems with variable timing. CTC has been applied to several tasks,
including phoneme recognition, ASR, and other sequence-based problems. One of
the major benefits of CTC is its ability to handle unknown alignment between
input and output, simplifying the training process. When used in ASR (Deng et
al., 2021, 2022a; Miao et al., 2020a), CTC eliminates the need for manual data
labeling by assigning probability scores to the output given any input signal.
This is particularly advantageous for tasks such as speech recognition and
handwriting recognition, where the input and output can vary in size. CTC also
solves the problem of having to specify the position of a character in the
output, allowing for more efficient training of the neural network without
post-processing the output. Finally, the CTC decoder can transform the neural
network output into the final text without post-processing.

#### 3.1.2 Application

The utilization of RNNs in popular products such as Google’s voice search and
Apple’s Siri to process user input and predict the output has been well-
documented (He et al., 2017; Li et al., 2017). RNNs are frequently utilized in
speech recognition tasks, such as the prediction of phonetic segments from
audio signals (Papastratis, 2021). They excel in use cases where context plays
a vital role in outcome prediction and are distinct from CNNs as they utilize
feedback loops to process a data sequence that informs the final output
(Papastratis, 2021).

In recent times, there have been advancements in the architecture of RNNs,
which have been primarily focused on developing end-to-end (E2E) models (Li et
al., 2020a; Pan et al., 2022) for ASR. These E2E models have replaced
conventional hybrid models and have displayed substantial enhancements in
speech recognition (Li et al., 2020a, 2021b). However, a significant challenge
faced by E2E RNN models is the synchronization of the input speech sequence
with the output label sequence (Graves, 2012). To tackle this issue, a loss
function called CTC (Graves and Graves, 2012) is utilized for training RNN
models, allowing for the repetition of labels to construct paths of the same
length as the input speech sequence. An alternative method is to employ an
Attention-based Encoder-Decoder (AED) model based on RNN architecture, which
utilizes an attention mechanism to align the input speech sequence with the
output label sequence. However, AED models tend to perform poorly on lengthy
utterances.

The development of Bimodal Recurrent Neural Networks (BRNN) has led to
significant advancements in the field of Audiovisual Speech Activity Detection
(AV-SAD) (Tao and Busso, 2019). BRNNs have demonstrated immense potential in
improving the performance of speech recognition systems, particularly in noisy
environments, by combining information from various sources. By integrating
separate RNNs for each modality, BRNNs can capture temporal dependencies
within and across modalities. This leads to successful outcomes in speech-
based systems, where integrating audio and visual modalities is crucial for
accurate speech recognition. Compared to conventional audio-only systems,
BRNN-based AV-SAD systems display superior performance, particularly in
challenging acoustic conditions where audio-only systems might struggle.

To enhance the performance of continuous speech recognition, LSTM networks
have been utilized in hybrid architectures alongside CNNs (Passricha and
Aggarwal, 2019). The CNNs extract local features from speech frames that are
then processed by LSTMs over time (Passricha and Aggarwal, 2019). LSTMs have
also been employed for speech synthesis, where they have been shown to enhance
the quality of statistical parametric speech synthesis (Passricha and
Aggarwal, 2019).

Aside from their ASR and speech synthesis applications, LSTM networks have
been utilized for speech post-filtering. To improve the quality of synthesized
speech, researchers have proposed deep learning-based post-filters, with LSTMs
demonstrating superior performance over other post-filter types (Coto-Jiménez,
2019). Bidirectional LSTM (Bi-LSTM) is another variant of RNN that has been
widely used for speech synthesis (Fan et al., 2014). Several RNN-based
analysis/synthesis models such as WaveNet (Oord et al., 2016), SampleRNN
(Mehri et al., 2016), and Tacotron have been developed. These neural vocoder
models can generate high-quality synthesized speech from acoustic features
without requiring intermediate vocoding steps.

###  3.2. Convolutional Neural Networks

Convolutional neural networks (CNNs) are a specialized class of deep neural
architecture consisting of one or more pairs of alternating convolutional and
pooling layers. A convolution layer applies filters that process small local
parts of the input, where these filters are replicated along the whole input
space. A pooling layer converts convolution layer activations to low
resolution by taking the maximum filter activation within a specified window
and shifting across the activation map. CNNs are variants of fully connected
neural networks widely used for processing data with grid-like topology. For
example, time-series data (1D grid) with samples at regular intervals or
images (2D grid) with pixels constitute a grid-like structure.

As discussed in Section 2, the speech spectrogram retains more information
than hand-crafted features, including speaker characteristics such as vocal
tract length differences across speakers, distinct speaking styles causing
formant to undershoot or overshoot, etc. Also, explicitly expressed these
characteristics in the frequency domain. The spectrogram representation shows
very strong correlations in time and frequency. Due to these characteristics
of the spectrogram, it is a suitable input for a CNN processing pipeline that
requires preserving locality in both frequency and time axis. For speech
signals, modeling local correlations with CNNs will be beneficial. The CNNs
can also effectively extract the structural features from the spectrogram and
reduce the complexity of the model through weight sharing. This section will
discuss the architecture of 1D and 2D CNNs used in various speech-processing
tasks.

#### 3.2.1 CNN Model Variants

##### 2D CNN

Since spectrograms are two-dimensional visual representations, one can
leverage CNN architectures widely used for visual data processing (images and
videos) by performing convolutions in two dimensions. The mathematical
equation for a 2D convolutional layer can be represented as:

(14)
$y_{i,j}^{(k)}=\sigma\Big{(}\sum_{l=1}^{L}\sum_{m=1}^{M}x_{i+l-1,j+m-1}^{(l)}w_{l,m}^{(k)}+b^{(k)}\Big{)}$

Here, $x_{i,j}^{(l)}$ is the pixel value of the $l^{th}$ input channel at the
spatial location $(i,j)$, $w_{l,m}^{(k)}$ is the weight of the $m^{th}$ filter
at the $l^{th}$ channel producing the $k^{th}$ feature map, and $b^{(k)}$ is
the bias term for the $k^{th}$ feature map.

The output feature map $y_{i,j}^{(k)}$ is obtained by convolving the input
image with the filters and then applying an activation function $\sigma$ to
introduce non-linearity. The convolution operation involves sliding the filter
window over the input image, computing the dot product between the filter and
the input pixels at each location, and producing a single output pixel.

However, there are some drawbacks to using a 2D CNN for speech processing. One
of the main issues is that 2D convolutions are computationally expensive,
especially for large inputs. This is because 2D convolutions involve many
multiplications and additions, and the computational cost grows quickly with
the input size.

To address this issue, a 1D CNN can be designed to operate directly on the
speech signal without needing a spectrogram. 1D convolutions are much less
computationally expensive than 2D convolutions because they only operate on
one dimension of the input. This reduces the multiplications and additions
required, making the network faster and more efficient. In addition, 1D
feature maps require less memory during processing, which is especially
important for real-time applications. A neural network’s memory requirements
are proportional to its feature maps’ size. By using 1D convolutions, the size
of the feature maps can be significantly reduced, which can improve the
efficiency of the network and reduce the memory requirements.

##### 1D CNN

1D CNN is essentially a special case of 2D CNN where the height of the filter
is equal to the height the spectogram. Thus, the filter only slides along the
temporal dimension and the height of the resultant feature maps is one. As
such, 1D convolutions are computationally less expensive and memory efficient
(Kiranyaz et al., 2021), as compared to 2D CNNs. Several studies (Kiranyaz et
al., 2015; Karita et al., 2019b; Abdeljaber et al., 2017) have shown that 1D
CNNs are preferable to their 2D counterparts in certain applications. For
example, Alsabhan (2023) found that the performance of predicting emotions
with a 2D CNN model was lower compared to a 1D CNN model.

1D convolution is useful in speech processing for several reasons:

  * •

Since, speech signals are sequences of amplitudes sampled over time, 1D
convolution can be applied along temporal dimension to capture temporal
variations in the signal.

  * •

_Robustness to distortion and noise:_ Since, 1D convolution allows local
feature extraction, the resultant features are often resilient to global
distortions of the signal. For instance, a speaker might be interrupted in the
middle of an utterance. Local features would still produce robust
representations for those relevant spans, which is key to ASR, among many
speech processing task. On the other hand, speech signals are often
contaminated with noise, making extracting meaningful information difficult.
1D convolution followed by pooling layers can mitigate the impact of noise
(Hendrycks and Dietterich, 2019), improving speech recognition systems’
accuracy.

The basic building block of a 1D CNN is the convolutional layer, which applies
a set of filters to the input data. A convolutional layer employs a collection
of adjustable parameters called filters to carry out convolution operations on
the input data, resulting in a set of feature maps as the output, which
represent the activation of each filter at each position in the input data.
The size of the feature maps depends on the size of the input data, the size
of the filters, and the number of filters used. The activation function used
in a 1D CNN is typically a non-linear function, such as the rectified linear
unit (ReLU) function.

Given an input sequence $x$ of length $N$, a set of $K$ filters $W_{k}$ of
length $M$, and a bias term $b_{k}$, the output feature map $y_{k}$ of the
$k^{th}$ filter is given by

(15) $y_{k}[n]=\operatorname*{ReLU}(b_{k}+\sum_{m=0}^{M-1}W_{k}[m]*x[n-m])$

where $n$ ranges from $M-1$ to $N-1$, and $*$ denotes the convolution
operation. After the convolutional layer, the output tensor is typically
passed through a pooling layer, reducing the feature maps’ size by down-
sampling. The most commonly used pooling operation is the max-pooling, which
keeps the maximum value from a sliding window across each feature map.

CNNs often replace previously popular methods like HMMs and GMM-UBM in various
cases. Moreover, CNNs possess the ability to acquire features that remain
robust despite variations in speech signals resulting from diverse speakers,
accents, and background noise. This is made possible due to three key
properties of CNNs: locality, weight sharing, and pooling. The locality
property enhances resilience against non-white noise by enabling the
computation of effective features from cleaner portions of the spectrum.
Consequently, only a smaller subset of features is affected by the noise,
allowing higher network layers a better opportunity to handle the noise by
combining higher-level features computed for each frequency band. This
improvement over standard fully connected neural networks, which process all
input features in the lower layers, highlights the significance of locality.
As a result, locality reduces the number of network weights that must be
learned.

#### 3.2.2 Application

CNNs have proven to be versatile tools for a range of speech-processing tasks.
They have been successfully applied to speech recognition (Nassif et al.,
2019; Abdel-Hamid et al., 2014b), including in hybrid NN-HMM models for speech
recognition, and can be used for multi-class classification of words (Abdel-
Hamid et al., 2012). In addition, CNNs have been proposed for speaker
recognition in an emotional speech, with a constrained CNN model presented in
(Simić et al., 2022).

CNNs, both 1D and 2D, have emerged as the core building block for various
speech processing models, including acoustic models (Schneider et al., 2019;
Gulati et al., 2020; Kriman et al., 2020) in ASR systems. For instance, in
2021, researchers from Facebook AI proposed wav2vec2.0 (Schneider et al.,
2019), a hybrid ASR system based on CNNs for learning representations of raw
speech signals that were then fed into a transformer-based language model. The
system achieved state-of-the-art results on several benchmark datasets.

Similarly, Google’s VGGVox (Chung et al., 2018) used a CNN with VGG
architecture to learn speaker embeddings from Mel spectrograms, achieving
state-of-the-art results in speaker recognition. CNNs have also been widely
used in developing state-of-the-art speech enhancement and text-to-speech
architectures. For instance, the architecture proposed in (Li et al., 2021a;
Tzinis et al., 2022b) for Deep Noise Suppression (DNS) (Reddy et al., 2020)
challenge and Google’s Tacotron2 (Shen et al., 2018) are examples of models
that use CNNs as their core building blocks. In addition to traditional tasks
like ASR and speaker identification, CNNs have also been applied to non-
traditional speech processing tasks like emotion recognition (Kakuba et al.,
2022), Parkinson’s disease detection (Johri et al., 2019), language
identification (Singh et al., 2021) and sleep apnea detection (Simply et al.,
2019). In all these tasks, CNN extracted features from speech signals and fed
them into the task classification model.

#### 3.2.3 Temporal Convolution Neural Networks

Recurrent neural networks, including RNNs, LSTMs, and GRUs, have long been
popular for deep-learning sequence modeling tasks. They are especially favored
in the speech-processing domain. However, recent studies have revealed that
certain CNN architectures can achieve state-of-the-art accuracy in tasks such
as audio synthesis, word-level language modelling, and machine translation, as
reported in (Kalchbrenner et al., 2016, 2014; Dauphin et al., 2017). The
advantage of convolutional neural networks is that they enable faster training
by allowing parallel computation. They can avoid common issues associated with
recurrent models, such as the vanishing or exploding gradient problem or the
inability to retain long-term memory.

In a recent study by Bai et al. (2018), they proposed a generic Temporal
Convolutional Neural Network (TCNN) architecture that can be applied to
various speech-related tasks. This architecture combines the best practices of
modern CNNs and has demonstrated comparable performance to recurrent
architectures such as LSTMs and GRUs. The TCN approach could revolutionize
speech processing by providing an alternative to the widely used recurrent
neural network models.

#### 3.2.4 TCNN Model Variants

The architecture of TCNN is based upon two principles:(1) There is no
information “leakage” from future to past;(2) the architecture can map an
input sequence of any length to an output sequence of the same length, similar
to RNN. TCN consists of dilated, causal 1D fully-convolutional layers with the
same input and output lengths to satisfy the above conditions. In other words,
TCNN is simply a 1D fully-convolutional network (FCN) with casual convolutions
as shown in Figure 2.

Figure 2. TCNNs leverage causal and dilated convolutions to model temporal
dependencies in sequential data. Causal convolutions ensure that future
information is not used during training, while dilated convolutions increase
the receptive field without increasing computational complexity. This makes
TCNNs an effective and efficient solution for a wide range of tasks, including
speech recognition, action recognition, and music analysis.

  * •

_Causal Convolution(Oord et al., 2016):_ Causal convolution convolves the
input at a specific time point $t$ solely with the temporally-prior elements.

  * •

_Dilated Convolution(Yu and Koltun, 2015):_ By itself, causal convolution
filters have a limited range of perception, meaning they can only consider a
fixed number of elements in the past. Therefore, it is challenging to learn
any dependency between temporally distant elements for longer sequences.
Dilated convolution ameliorates this limitation by repeatedly applying
dilating filters to expand the range of perception, as shown in Figure 2. The
dilation is achieved by uniformly inserting zeros between the filter weights.

Consider a 1-D sequence $x\in\mathbf{R}^{n}$ and a filter:
$f:\\{0,...,k-1\\}\rightarrow\mathbf{R}$, the dilated convolution operation
$F_{d}$ on an element $y$ of the sequence is defined as

(16) $F_{d}(y)=(x*_{d}f)(s)=\sum_{i=0}^{k-1}f(i).x_{y-d.i},$

where $k$ is filter size, $d$ is dilation factor, and $y-d.i$ is the span
along the past. The dilation step introduces a fixed step between every two
adjacent filter taps. When $d=1$, a dilated convolution acts as a normal
convolution. Whereas, for larger dilation, the filter acts on a wide but non-
contiguous range of inputs. Therefore, dilation effectively expands the
receptive field of the convolutional networks.

#### 3.2.5 Application

Recent studies have shown that the TCNN architecture not only outperforms
traditional recurrent networks like LSTMs and GRUs in terms of accuracy but
also possesses a set of advantageous properties, including:

  * •

Parallelism is a key advantage of TCNN over RNNs. In RNNs, time-step
predictions depend on their predecessors’ completion, which limits parallel
computation. In contrast, TCNNs apply the same filter to each span in the
input, allowing parallel application thereof. This feature enables more
efficient processing of long input sequences compared to RNNs that process
sequentially.

  * •

The receptive field size can be modified in various ways to enhance the
performance of TCNNs. For example, incorporating additional dilated
convolutional layers, employing larger dilation factors, or augmenting the
filter size are all effective methods. Consequently, TCNNs offer superior
management of the model’s memory size and are highly adaptable to diverse
domains.

  * •

When dealing with lengthy input sequences, LSTM and GRU models tend to consume
a significant amount of memory to retain the intermediate outcomes for their
numerous cell gates. On the other hand, TCNNs utilize shared filters
throughout a layer, and the back-propagation route depends solely on the depth
of the network. This makes TCNNs a more memory-efficient alternative to LSTMs
and GRUs, especially in scenarios where memory constraints are a concern.

TCNNs can perform real-time speech enhancement in the time domain (Pandey and
Wang, 2019). They have much fewer trainable parameters than earlier models,
making them more efficient. TCNs have also been used for speech and music
detection in radio broadcasts (Hung et al., 2022; Lemaire and Holzapfel,
2019). They have been used for single channel speech enhancement (Lin et al.,
2021b; Richter et al., 2020) and are trained as filter banks to extract
features from waveform to improve the performance of ASR (Li et al., 2019b).

###  3.3. Transformers

While recurrence in RNNs (Section 3.1) is a boon for neural networks to model
sequential data, it is also a bane as the recurrence in time to update the
hidden state intrinsically precludes parallelization. Additionally, although
dedicated gated RNNs such as LSTM and GRU have helped to mitigate the
vanishing gradient problem to some extent, it can still be a challenge to
maintain long-term dependencies in RNNs.

Proposed by Vaswani et al. (2017), Transformer solved a critical shortcoming
of RNNs by allowing parallelization within the training sample, that is,
facilitating the processing of the entire input sequence at once. Since then,
the primary idea of using only the attention mechanism to construct an encoder
and decoder has served as the basic recipe for many state-of-the-art
architectures across the domains of machine learning. In this survey, we use
transformer to denote architectures that are inspired by Transformer (Devlin
et al., 2018; Brown et al., 2020; Radford et al., 2019, 2018; Han et al.,
2022). This section overviews the transformer’s fundamental design proposed by
Vaswani et al. (2017) and its adaptations for different speech-related
applications.

#### 3.3.1 Basic Architecture

Transformer architecture (Vaswani et al., 2017) comprises an attention-based
encoder and decoder, with each module consisting of a stack of identical
blocks. Each block in the encoder and decoder consists of two sub-layers: a
multi-head attention (MHA) mechanism and a position-wise fully connected
feedforward network as described in Figure 3. The MHA mechanism in the encoder
allows each input element to attend to every other element in the sequence,
enabling the model to capture long-range dependencies in the input sequence.
The decoder typically uses a combination of MHA and encoder-decoder attention
to attend to both the input sequence and the previously generated output
elements. The feedforward network in each block of the Transformer provides
non-linear transformations to the output of the attention mechanism. Next, we
discuss operations involved in transformer layers, that is, multi-head
attention and position-wise feedforward network:

##### Attention in Transformers

Attention mechanism, first proposed by Bahdanau et al. (2014), has
revolutionized sequence modeling and transduction models in various tasks of
NLP, speech, and computer vision (Galassi et al., 2020; Cho et al., 2015; Wang
and Tax, 2016; Chaudhari et al., 2021). Broadly, it allows the model to focus
on specific parts of the input or output sequence, without being limited by
the distance between the elements. We can describe the attention mechanism as
the mapping of a query vector and set of key-value vector pairs to an output.
Precisely, the output vector is computed as a weighted summation of value
vectors where the weight of a value vector is obtained by computing the
compatibility between the query vector and key vector. Let, each query $Q$ and
key $K$ are $d_{k}$ dimensional and value $V$ is $d_{v}$ dimensional. Specific
to the Transformer, the compatibility function between a query and each key is
computed as their dot product between scaled by $\sqrt{d_{k}}$. To obtain the
weights on values, the scaled dot product values are passed through a softmax
function:

Figure 3. Illustrations of attention (left) and multi-headed attention
(right).

(17)
$\operatorname*{Attention}(\textbf{Q},\textbf{K},\textbf{V})=\operatorname*{softmax}\left(\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{d_{k}}}\right)\textbf{V}$

Here multiple queries, keys, and value vectors, are packed together in matrix
form respectively denoted by $\textbf{Q}\in{\mathbb{R}^{N\times d_{k}}}$,
$\textbf{K}\in{\mathbb{R}^{M\times d_{k}}}$, and
$\textbf{V}\in{\mathbb{R}^{M\times d_{v}}}$. N and M represent the lengths of
queries and keys (or values). Scaling of dot product attention becomes
critical to tackling the issue of small gradients with the increase in $d_{k}$
(Vaswani et al., 2017).

Instead of performing single attention in each transformer block, multiple
attentions in lower-dimensional space have been observed to work better
(Vaswani et al., 2017). This observation gave rise to Multi-Head Attention:
For $h$ heads and dimension of tokens in the model $d_{m}$, the
$d_{m}$-dimensional query, key, and values are projected $h$ times to $d_{k}$,
$d_{k}$, and $d_{v}$ dimensions using learnable linear
projections333Projection weights are neither shared across heads nor query,
key, and values.. Each head performs attention operation as per Equation 17.
The $h$ $d_{v}$-dimensional are concatenated and projected back to $d_{m}$
using another projection matrix:

(18)
$\displaystyle\operatorname*{MultiHeadAttn}(\textbf{Q},\textbf{K},\textbf{V})$
$\displaystyle=\operatorname*{Concat}(\operatorname*{head}{}_{1},....\operatorname*{head}{}_{h})\textbf{W}^{O},$
(19) $\displaystyle\text{with }\operatorname*{head}{}_{i}$
$\displaystyle=\operatorname*{Attention}(\textbf{Q}\textbf{W}^{Q}_{i},\textbf{K}\textbf{W}^{K}_{i},\textbf{V}\textbf{W}^{V}_{i})$

Where $\textbf{W}^{Q},\textbf{W}^{K}\in{\mathbb{R}^{d_{model}\times
d_{k}}},\textbf{W}^{V}\in\mathbb{R}^{d_{model}\times
d_{v}},\textbf{W}^{O}\in{\mathbb{R}^{hd_{v}\times d_{model}}}$ are learnable
projection matrices. Intuitively, multiple attention heads allow for attending
to parts of the sequence differently (e.g., longer-term dependencies versus
shorter-term dependencies). Intuitively, multiple attention heads allow for
attending in different representational spaces jointly.

##### Position-wise FFN

The position-wise FNN consists of two dense layers. It is referred to
position-wise since the same two dense layers are used for each positioned
item in the sequence and are equivalent to applying two $1\times 1$
convolution layers.

##### Residual Connection and Normalization

Residual connection and Layer Normalization are employed around each module
for building a deep model. For example, each encoder block output can be
defined as follows:

(20)
$H^{{}^{\prime}}=\operatorname*{LayerNorm}(\operatorname*{SelfAttention}(X)+X)$
(21)
$H=\operatorname*{LayerNorm}(\operatorname*{FFN}(H^{{}^{\prime}})+H^{{}^{\prime}})$

$\operatorname*{SelfAttention}(.)$ denotes attention module with
$\textbf{Q}=\textbf{K}=\textbf{V}=\textbf{X}$, where X is the output of the
previous layer.

Transformer-based architecture turned out to be better than many other
architectures such as RNN, LSTM/GRU, etc. One of the major difficulties when
applying a Transformer to speech applications that it requires more complex
configurations (e.g., optimizer, network structure, data augmentation) than
the conventional RNN-based models. Speech signals are continuous-time signals
with much higher dimensionality than text data. This high dimensionality poses
significant computational challenges for the Transformer architecture,
originally designed for sequential text data. Speech signals also have
temporal dependencies, which means that the model needs to be able to process
and learn from the entire signal rather than just a sequence of inputs. Also,
speech signals are inherently variable and complex. The same sentence can be
spoken differently and even by the same person at different times. This
variability requires the model to be robust to differences in pitch, accent,
and speed of speech.

#### 3.3.2 Application

Recent advancements in NLP which lead to a paradigm shift in the field are
highly attributed to the foundation models that are primarily a part of the
transformers category, with self-attention being a key ingredient (Bommasani
et al., 2021). The recent models have demonstrated human-level performance in
several professional and academic benchmarks. For instance, GPT4 scored within
the top 10% of test takers on a simulated version of the Uniform Bar
Examination (OpenAI, 2023). While speech processing has not yet seen a shift
in paradigm as in NLP owing to the capabilities of foundational models, even
so, transformers have significantly contributed to advancement in the field
including but not limited to the following tasks: automatic speech
recognition, speech translation, speech synthesis, and speech enhancement,
most of which we discuss in detail in Section 5.

RNNs and Transformers are two widely adopted neural network architectures
employed in the domain of Natural Language Processing (NLP) and speech
processing. While RNNs process input words sequentially and preserve a hidden
state vector over time, Transformers analyze the entire sentence in parallel
and incorporate an internal attention mechanism. This unique feature makes
Transformers more efficient than RNNs (Karita et al., 2019a). Moreover,
Transformers employ an attention mechanism that evaluates the relevance of
other input tokens in encoding a specific token. This is particularly
advantageous in machine translation, as it allows the Transformer to
incorporate contextual information, thereby enhancing translation accuracy
(Karita et al., 2019a). To achieve this, Transformers combine word vector
embeddings and positional encodings, which are subsequently subjected to a
sequence of encoders and decoders. These fundamental differences between RNNs
and Transformers establish the latter as a promising option for various
natural language processing tasks (Karita et al., 2019a).

A comparative study on transformer vs. RNN (Karita et al., 2019a) in speech
applications found that transformer neural networks achieve state-of-the-art
performance in neural machine translation and other natural language
processing applications (Karita et al., 2019a). The study compared and
analysed transformer and conventional RNNs in a total of 15 ASR, one
multilingual ASR, one ST, and two TTS applications. The study found that
transformer neural networks outperformed RNNs in most applications tested.
Another survey of transformer-based models in speech processing found that
transformers have an advantage in comprehending speech, as they analyse the
entire sentence simultaneously, whereas RNNs process input words one by one.

Transformers have been successfully applied in end-to-end speech processing,
including automatic speech recognition (ASR), speech translation (ST), and
text-to-speech (TTS) (Li et al., 2019a). In 2018, the Speech-Transformer was
introduced as a no-recurrence sequence-to-sequence model for speech
recognition. To reduce the dimension difference between input and output
sequences, the model’s architecture was modified by adding convolutional
neural network (CNN) layers before feeding the features to the transformer. In
a later study (Nakatani, 2019), the authors proposed a method to improve the
performance of end-to-end speech recognition models based on transformers.
They integrated the connectionist temporal classification (CTC) with the
transformer-based model to achieve better accuracy and used language models to
incorporate additional context and mitigate recognition errors.

In addition to speech recognition, the transformer model has shown promising
results in TTS applications. The transformer based TTS model generates mel-
spectrograms, followed by a WaveNet vocoder to output the final audio results
(Li et al., 2019a). Several neural network-based TTS models, such as Tacotron
2, DeepVoice 3, and transformer TTS, have outperformed traditional
concatenative and statistical parametric approaches in terms of speech quality
(Li et al., 2019a; Shen et al., 2018; Ping et al., 2017).

One of the strengths of Transformer-based architectures for neural speech
synthesis is their high efficiency while considering the global context
(Gulati et al., 2020; Shi et al., 2020). The Transformer TTS model has shown
advantages in training and inference efficiency over RNN-based models such as
Tacotron 2 (Shen et al., 2018). The efficiency of the Transformer TTS network
can speed up the training about 4.25 times (Li et al., 2019a). Moreover,
Multi-Speech, a multi-speaker TTS model based on the Transformer (Li et al.,
2019a), has demonstrated the effectiveness of synthesizing a more robust and
better quality multi-speaker voice than naive Transformer-based TTS.

In contrast to the strengths of Transformer-based architectures in neural
speech synthesis, large language models based on Transformers such as BERT
(Devlin et al., 2018), GPT (Radford et al., 2018), XLNet (Yang et al., 2019a),
and T5 (Raffel et al., 2020) have limitations when it comes to speech
processing. One of the issues is that these models require discrete tokens as
input, necessitating using a tokenizer or a speech recognition system,
introducing errors and noise. Furthermore, pre-training on large-scale text
corpora can lead to domain mismatch problems when processing speech data. To
address these limitations, dedicated frameworks have been developed for
learning speech representations using transformers, including wav2vec
(Schneider et al., 2019), data2vec (Baevski et al., 2022), Whisper (Radford et
al., 2022), VALL-E (Wang et al., 2023a), Unispeech (Wang et al., 2021c),
SpeechT5 (Ao et al., 2021) etc. We discuss some of them as follows.

  * •

Speech representation learning frameworks, such as wav2vec, have enabled
significant advancements in speech processing tasks. One recent framework,
w2v-BERT (Wang et al., 2019d), combines contrastive learning and MLM to
achieve self-supervised speech pre-training on discrete tokens. Fine-tuning
wav2vec models with limited labeled data has also been demonstrated to achieve
state-of-the-art results in speech recognition tasks (Baevski et al., 2019b).
Moreover, XLS-R (Babu et al., 2021), another model based on wav2vec 2.0, has
shown state-of-the-art results in various tasks, domains, data regimes, and
languages, by leveraging multilingual data augmentation and contrastive
learning techniques on a large scale. These models learn universal speech
representations that can be transferred across languages and domains, thus
representing a significant advancement in speech representation learning.

  * •

Transformers have been increasingly popular in the development of frameworks
for learning representations from multi-modal data, such as speech, images,
and text. Among these frameworks, Data2vec (Baevski et al., 2022) is a self-
supervised training approach that aims to learn joint representations to
capture cross-modal correlations and transfer knowledge across modalities. It
has outperformed other unsupervised methods for learning multi-modal
representations in benchmark datasets. However, for tasks that require domain-
specific models, such as speech recognition or speaker identification, domain-
specific models may be more effective, particularly when dealing with data in
specific domains or languages. The self-supervised training approach of
Data2vec enables cost-effective and scalable learning of representations
without requiring labeled data, making it a promising framework for various
multi-modal learning applications.

  * •

The field of speech recognition has undergone a revolutionary change with the
advent of the Whisper model (Radford et al., 2022). This innovative solution
has proven to be highly versatile, providing exceptional accuracy for various
speech-related tasks, even in challenging environments. The Whisper model
achieves its outstanding performance through a minimalist approach to data
pre-processing and weak supervision, which allows it to deliver state-of-the-
art results in speech processing. The model is capable of performing
multilingual speech recognition, translation, and language identification,
thanks to its training on a diverse audio dataset. Its multitasking model can
cater to various speech-related tasks, such as transcription, voice
assistants, education, entertainment, and accessibility. One of the unique
features of Whisper is its minimalist approach to data pre-processing, which
eliminates the need for significant standardization and simplifies the speech
recognition pipeline. The resulting models generalize well to standard
benchmarks and deliver competitive performance without fine-tuning,
demonstrating the potential of advanced machine learning techniques in speech
processing.

  * •

Text-to-speech synthesis has been a topic of interest for many years, and
recent advancements have led to the development of new models such as VALL-E
(Wang et al., 2023a). VALL-E is a novel text-to-speech synthesis model that
has gained significant attention due to its unique approach to the task.
Unlike traditional TTS systems, VALL-E treats the task as a conditional
language modelling problem and leverages a large amount of semi-supervised
data to train a generalized TTS system. It can generate high-quality
personalized speech with a 3-second acoustic prompt from an unseen speaker and
provides diverse outputs with the same input text. VALL-E also preserves the
acoustic environment and the speaker’s emotions about the acoustic prompt,
without requiring additional structure engineering, pre-designed acoustic
features, or fine-tuning. Furthermore, VALL-E X (Zhang et al., 2023c) is an
extension of VALL-E that enables cross-lingual speech synthesis, representing
a significant advancement in TTS technology.

Figure 4. Timeline highlighting notable large Transformer models developed for
speech processing, along with their corresponding parameter sizes.

The timeline highlights the development of large transformer based models for
speech processing is shown in Figure 4. The size of the models has grown
exponentially, with significant breakthroughs achieved in speech recognition,
synthesis, and translation. These large models have set new performance
benchmarks in the field of speech processing, but also pose significant
computational and data requirements for training and inference.

###  3.4. Conformer

The Transformer architecture, which utilizes a self-attention mechanism, has
successfully replaced recurrent operations in previous architectures. Over the
past few years, various Transformer variants have been proposed (Gulati et
al., 2020). Architectures combining Transformers and CNNs have recently shown
promising results on speech-processing tasks (Wang et al., 2021b). To
efficiently model both local and global dependencies of an audio sequence,
several attempts have been made to combine CNNs and Transformers. One such
architecture proposed by the authors is the Conformer (Gulati et al., 2020), a
convolution-augmented transformer for speech recognition. Conformer
outperforms RNNs, previous Transformers, and CNN-based models, achieving
state-of-the-art performance in speech recognition. The Conformer model
consists of several building blocks, including convolutional layers, self-
attention layers, and feedforward layers. The architecture of the Conformer
model can be summarized as follows:

  * •

Input Layer: The Conformer model inputs a sequence of audio features, such as
MFCCs or Mel spectrograms.

  * •

Convolutional Layers: Local features are extracted from the audio signal by
processing the input sequence through convolutional layers.

  * •

Self-Attention Layers: The Conformer model incorporates self-attention layers
following the convolutional layers. Self-attention is a mechanism that enables
the model to focus on various sections of the input sequence while making
predictions. This is especially advantageous for speech recognition because it
facilitates capturing long-term dependencies in the audio signal.

  * •

Feedforward Layers: After the self-attention layers, the Conformer model
applies a sequence of feedforward layers intended to process the output of the
self-attention layers further and ready it for the ultimate prediction.

  * •

Output Layer: Finally, the output from the feedforward layers undergoes a
softmax activation function to generate the final prediction, typically
representing a sequence of character labels or phonemes.

The conformer model has emerged as a promising neural network architecture for
various speech-related research tasks, including but not limited to speech
recognition, speaker recognition, and language identification. In a recent
study by Gulati et al. (2020), the conformer model was demonstrated to
outperform previous state-of-the-art models, particularly in speech
recognition significantly. This highlights the potential of the conformer
model as a key tool for advancing speech-related research.

#### 3.4.1 Application

The Conformer model stands out among other speech recognition models due to
its ability to efficiently model both local and global dependencies of an
audio sequence. This is crucial for speech recognition, language translation,
and audio classification (ass, 2022; Gulati et al., 2020; nvi, 2022). The
model achieves this through self-attention and convolution modules, combining
the strengths of CNNs and Transformers. While CNNs capture local information
in audio sequences, the self-attention mechanism captures global dependencies
(nvi, 2022). The Conformer model has achieved remarkable performance in speech
recognition tasks, setting benchmarks on datasets such as LibriSpeech and
AISHELL-1.

Despite these successes, speech synthesis and recognition challenges persist,
including difficulties generating natural-sounding speech in non-English
languages and real-time speech generation. To address these limitations, Wang
et al. (Zhang et al., 2020c) proposed a novel approach that combines noisy
student training with SpecAugment and large Conformer models pre-trained on
the Libri-Light dataset using the wav2vec 2.0 pre-training method. This
approach achieved state-of-the-art word error rates on the LibriSpeech
dataset. Recently, Wang et al. (2022a) developed Conformer-LHUC, an extension
of the Conformer model that employs learning hidden unit contribution (LHUC)
for speaker adaptation. Conformer-LHUC has demonstrated exceptional
performance in elderly speech recognition and shows promise for the clinical
diagnosis and treatment of Alzheimer’s disease.

Several enhancements have been made to the Conformer-based model to address
high word error rates without a language model, as documented in (Liu and Wei,
2022). Wu (2022) proposed a deep sparse Conformer to improve its long-sequence
representation capabilities. Furthermore, Burchi and Timofte (2023) have
recently enhanced the noise robustness of the Efficient Conformer architecture
by processing both audio and visual modalities. In addition, models based on
Conformer, such as Transducers (Kim and Lee, 2021), have been adopted for
real-time speech recognition (Papastratis, 2021) due to their ability to
process audio data much more quickly than conventional recurrent neural
network (RNN) models.

###  3.5. Sequence to Sequence Models

The sequence-to-sequence (seq2seq) model in speech processing is popularly
used for ASR, ST, and TTS tasks. The general architecture of the seq2seq model
involves an encoder-decoder network that learns to map an input sequence to an
output sequence of varying lengths. In the case of ASR, the input sequence is
the speech signal, which is processed by the encoder network to produce a
fixed-length feature vector representation of the input signal. The decoder
network inputs this feature vector and produces the corresponding text
sequence. This can be achieved through a stack of RNNs (Prabhavalkar et al.,
2017), Transformer (Dong et al., 2018) or Conformer (Gulati et al., 2020) in
the encoder and decoder networks.

The sequence-to-sequence model has emerged as a potent tool in speech
translation. It can train end-to-end to efficiently map speech spectrograms in
one language to their corresponding spectrograms in another. The notable
advantage of this approach is eliminating the need for an intermediate text
representation, resulting in improved efficiency. Additionally, the Seq2seq
models have been successfully implemented in speech generation tasks, where
they reverse the ASR approach. In such applications, the input text sequence
serves as the input, with the encoder network creating a feature vector
representation of the input text. The decoder network then leverages this
representation to generate the desired speech signal.

Karita et al. (2019a) conducted an extensive study comparing the performance
of transformer and traditional RNN models on 15 different benchmarks for
Automatic Speech Recognition (ASR), including a multilingual ASR benchmark, a
Speech Translation (ST) benchmark, and two Text-to-Speech (TTS) benchmarks. In
addition, they proposed a shared Sequence-to-Sequence (S2S) architecture for
AST, TTS, and ST tasks, which is depicted in Figure 5.

Figure 5. Unified formulation for Sequence-to-Sequence architecture in speech
applications (Karita et al., 2019a). $X$ and $Y$ are source and target
sequences respectively.

  * •

Encoder

(22) $\displaystyle X_{0}$ $\displaystyle=\operatorname*{Encoder-PreNet}(X),$
$\displaystyle X_{e}$ $\displaystyle=\operatorname*{Encoder-Main}(X_{0})$

where $X$ is the sequence of speech features (e.g. Mel spectrogram) for AST
and ST and phoneme or character sequence for TTS.

  * •

Decoder

(23) $\displaystyle Y_{0}[1:t-1]$ $\displaystyle=\operatorname*{Decoder-
PreNet}(Y[1:t-1]),$ $\displaystyle Y_{d}[t]$
$\displaystyle=\operatorname*{Decoder-Main}(X_{e},Y_{0}[1:t-1]),$
$\displaystyle Y_{post}[1:t]$ $\displaystyle=\operatorname*{Decoder-
PostNet}(Y_{d}[1:t]),$

During the training stage, input to the decoder is ground truth target
sequence $Y[1:t-1]$. The Decoder-Main module is utilized to produce a
subsequent target frame. This is accomplished by utilizing the encoded
sequence $X_{e}$ and the prefix of the target prefix $Y_{0}[1:t-1]$. The
decoder is mostly unidirectional for sequence generation and often uses an
attention mechanism (Bahdanau et al., 2014) to produce the output.

Seq2seq models have been widely used in speech processing, initially based on
RNNs. However, RNNs face the challenge of processing long sequences, which can
lead to the loss of the initial context by the end of the sequence (Karita et
al., 2019a). To overcome this limitation, the transformer architecture has
emerged, leveraging self-attention mechanisms to handle sequential data. The
transformer has shown remarkable performance in tasks such as ASR, ST, and
speech synthesis. As a result, the use of RNN-based seq2seq models has
declined in favour of the transformer-based approach.

#### 3.5.1 Application

Seq2seq models have been used for speech processing tasks such as voice
conversion (Tanaka et al., 2019; Huang et al., 2019), speech synthesis (Wang
et al., 2017; Wang, 2019; Okamoto et al., 2019a, b; Huang et al., 2019), and
speech recognition. The field of ASR has seen significant progress, with
several advanced techniques emerging as popular options. These include the CTC
approach, which has been further developed and improved upon through recent
advancements (Graves and Jaitly, 2014), as well as attention-based approaches
that have also gained traction (Chorowski et al., 2015). The growing interest
in these techniques has increased the use of seq2seq models in the speech
community.

  * •

Attention-based Approaches: The attention mechanism is a crucial component of
sequence-to-sequence models, allowing them to effectively weigh input acoustic
features during decoding (Bahdanau et al., 2014; Luong et al., 2015).
Attention-based Seq2seq models utilize previously generated output tokens and
the complete input sequence to factorize the joint probability of the target
sequence into individual time steps. The attention mechanism is conditioned on
the current decoder states and runs over the encoder output representations to
incorporate information from the input sequence into the decoder output.
Incorporating attention mechanisms in Seq2Seq models has resulted in an
impressive performance in various speech processing tasks, such as speech
recognition (Nankaku et al., 2021; Prabhavalkar et al., 2017; Tüske et al.,
2019; Weng et al., 2018), text-to-speech (Shen et al., 2018; Yasuda et al.,
2019; Okamoto et al., 2020), and voice conversion (Tanaka et al., 2019; Huang
et al., 2019). These models have demonstrated competitiveness with traditional
state-of-the-art approaches. Additionally, attention-based Seq2Seq models have
been used for confidence estimation tasks in speech recognition, where
confidence scores generated by a speech recognizer can assess transcription
quality (Li et al., 2021c). Furthermore, these models have been explored for
few-shot learning, which has the potential to simplify the training and
deployment of speech recognition systems (Higy and Bell, 2018).

  * •

Connectionist Temporal Classification: While attention-based methods create a
soft alignment between input and target sequences, approaches that utilize CTC
loss aim to maximize log conditional likelihood by considering all possible
monotonic alignments between them. These CTC-based Seq2Seq models have
delivered competitive results across various ASR benchmarks (Higuchi et al.,
2022; Majumdar et al., 2021; Synnaeve et al., 2020; Gulati et al., 2020) and
have been extended to other speech-processing tasks such as voice conversion
(Zhang et al., 2019c, 2021a; Liu et al., 2021a), speech synthesis (Zhang et
al., 2019c) etc. Recent studies have concentrated on enhancing the performance
of Seq2Seq models by combining CTC with attention-based mechanisms, resulting
in promising outcomes. This combination remains a subject of active
investigation in the speech-processing domain.

###  3.6. Reinforcement Learning

Reinforcement learning (RL) is a machine learning paradigm that trains an
agent to perform discrete actions in an environment and receive rewards or
punishments based on its interactions. The agent aims to learn a policy that
maximizes its long-term reward. In recent years, RL has become increasingly
popular and has been applied to various domains, including robotics, game
playing, and natural language processing. RL has been utilized in speech
recognition, speaker diarization, and speech enhancement tasks in the speech
field. One of the significant benefits of using RL for speech tasks is its
ability to learn directly from raw audio data, eliminating the need for hand-
engineered features. This can result in better performance compared to
traditional methods that rely on feature extraction. By capturing intricate
patterns and relationships in the audio data, RL-based speech systems have the
potential to enhance accuracy and robustness.

#### 3.6.1 Basic Models

The utilization of deep reinforcement learning (DRL) in speech processing
involves the environment (a set of states $S$), agent, actions ($A$), and
reward ($r$). The semantics of these components depends on the task at hand.
For instance, in ASR tasks, the environment can be composed of speech
features, the action can be the choices of phonemes, and the reward could be
the correctness of those phonemes given the input. Audio signals are one-
dimensional time-series signals that undergo pre-processing and feature
extraction procedures. Pre-processing steps include noise suppression, silence
removal, and channel equalization, improving audio signal quality and creating
robust and efficient audio-based systems. Previous research has demonstrated
that pre-processing improves the performance of deep learning-based audio
systems (Latif et al., 2020).

Feature extraction is typically performed after pre-processing to convert the
audio signal into meaningful and informative features while reducing their
number. MFCCs and spectrograms are popular feature extraction choices in
speech-based systems (Latif et al., 2020). These features are then given to
the DRL agent to perform various tasks depending on the application. For
instance, consider the scenario where a human speaks to a DRL-trained machine,
where the machine must act based on features derived from audio signals.

  * •

Value-based DRL: Given the state of the environment ($s$), a value function
$Q:S\times A\rightarrow\mathbb{R}$ is learned to estimate overall future
reward $Q(s,a)$ should an action $a$ be taken. This value function is
parameterized with deep networks like CNN, Transformers, etc.

  * •

Policy-based DRL: As opposed to value-based RL, policy-based RL methods learns
a policy function $\pi:S\rightarrow A$ that chooses the best possible action
($a$) based on reward.

  * •

Model-based DRL: Unlike the previous two approaches, model-based RL learns the
dynamics of the environment in terms of the state transition probabilities,
i.e., a function $M:S\times A\times S\rightarrow\mathbb{R}$. Given such a
model, policy, or value functions are optimized.

#### 3.6.2 Application

In speech-related research, deep reinforcement learning can be used for
several purposes, including:

##### Speech recognition and Emotion modeling

Deep reinforcement learning (DRL) can be used to train speech recognition
systems (Kala and Shinozaki, 2018; Rajapakshe et al., 2020; Tjandra et al.,
2018; Chung et al., 2020c, d) to transcribe speech accurately. In this case,
the system receives an audio input and outputs a text sequence corresponding
to the spoken words. The environmental states might be learned from the input
audio features. The actions might be the generated phonemes. The reward could
be the similarity between the generated and gold phonemes, quantified in edit
distance. Several works have also achieved promising results for non-native
speech recognition (Radzikowski et al., 2019)

DRL pre-training has shown promise in reducing training time and enhancing
performance in various Human-Computer Interaction (HCI) applications,
including speech recognition (Rajapakshe et al., 2020). Recently, researchers
have suggested using a reinforcement learning algorithm to develop a Speech
Enhancement (SE) system that effectively improves ASR systems. However, ASR
systems are often complicated and composed of non-differentiable units, such
as acoustic and language models. Therefore, the ASR system’s recognition
outcomes should be employed to establish the objective function for optimizing
the SE model. Other than ASR, SE, some studies have also focused on SER using
DRL algorithms (Lakomkin et al., 2018; Rajapakshe et al., 2022; Kansizoglou et
al., 2019)

##### Speaker identification

Similarly, for speaker identification tasks, the actions can be the speaker’s
choices, and a binary reward can be the correctness of choice.

##### Speech synthesis and coding

Likewise, the states can be the input text, the actions can be the generated
audio, and the reward could be the similarity between the gold and generated
mel-spectrogram.

Deep reinforcement learning has several advantages over traditional machine
learning techniques. It can learn from raw data without needing hand-
engineered features, making it more flexible and adaptable. It can also learn
from feedback, making it more robust and able to handle noisy environments.

However, deep reinforcement learning also has some challenges that must be
addressed. It requires a lot of data to train and can be computationally
expensive. It also requires careful selection of the reward function to ensure
that the system learns the desired behavior.

###  3.7. Graph Neural Network

Over the past few years, the field of Graph Neural Networks (GNNs) has
witnessed a remarkable expansion as a widely adopted approach for analysing
and learning from data on graphs. GNNs have demonstrated their potential in
various domains, including computer science, physics, mathematics, chemistry,
and biology, by delivering successful outcomes. Furthermore, in recent times,
the speech-processing domain has also witnessed the growth of GNNs.

Figure 6. A standard experimental pipeline for GCNs, which embeds the graph
node and embeds the graph node edge features, performs several GNN layers to
compute convolutional features, and finally predicts a task-specific MLP
layer.

#### 3.7.1 Basic Models

Speech processing involves analysing and processing audio signals, and GNNs
can be useful in this context when we represent the audio data as a graph. In
this answer, we will explain the architecture of GNNs for speech processing.
The standard GNN pipeline is shown in Figure 6, according to the application
the GNN layer can consist of Graph Convolutional Layers (Zhang et al., 2019d),
Graph Attention Layers (Velickovic et al., 2017), or Graph Transformer (Yun et
al., 2019).

##### Graph Representation of Speech Data

The first step in using GNNs for speech processing is representing the speech
data as a graph. One way to do this is to represent the speech signal as a
sequence of frames, each representing a short audio signal segment. We can
then represent each frame as a node in the graph, with edges connecting
adjacent frames.

##### Graph Convolutional Layers

Once the speech data is represented as a graph, we can use graph convolutional
layers to learn representations of the graph nodes. Graph convolutional layers
are similar to traditional ones, but instead of operating on a grid-like
structure, they operate on graphs. These layers learn to aggregate information
from neighboring nodes to update the features of each node.

##### Graph Attention Layers

Graph attention layers can be combined with graph convolutional layers to give
more importance to certain nodes in the graph. Graph attention layers learn to
assign weights to neighbor nodes based on their features, which can help
capture important patterns in speech data. Several works have used graph
attention layers for neural speech synthesis (Liu et al., 2021c) or speaker
verification (Jung et al., 2021a) and diarization (Kwon et al., 2022).

##### Recurrent Layers

Recurrent layers can be used in GNNs for speech processing to capture temporal
dependencies between adjacent frames in the audio signal. Recurrent layers
allow the network to maintain an internal state that carries information from
previous time steps, which can be useful for modeling the dynamics of speech
signals.

##### Output Layers

The output layer of a GNN for speech processing can be a classification layer
that predicts a label for the speech data (e.g., phoneme or word) or a
regression layer that predicts a continuous value (e.g., pitch or loudness).
The output layer can be a traditional fully connected layer or a graph pooling
layer that aggregates information from all the nodes in the graph.

#### 3.7.2 Application

The advantages of using GNNs for speech processing tasks include their ability
to represent the dependencies and interrelationships between various entities,
which is suitable for speech processing tasks such as speaker diarization
(Singh et al., 2023; Wang et al., 2020e; Singh and Ganapathy, 2021), speaker
verification (Jung et al., 2021b; Shim et al., 2022), speech synthesis (Sun et
al., 2020; Liu et al., 2021c; Sun et al., 2021), or speech separation (Wang et
al., 2023c; von Neumann et al., 2021), which require the analysis of complex
data representations. GNNs retain a state representing information from their
neighborhood with arbitrary depth, unlike standard neural networks. GNNs can
be used to model the relationship between phonemes and words. GNNs can learn
to recognize words in spoken language by treating the phoneme sequence as a
graph. GNNs can also be used to model the relationship between different
acoustic features, such as pitch, duration, and amplitude, in speech signals,
improving speech recognition accuracy.

GNNs have shown promising results in multichannel speech enhancement, where
they are used for extracting clean speech from noisy mixtures captured by
multiple microphones (Tzirakis et al., 2021). The authors of a recent study
(Nguyen et al., 2022c) propose a novel approach to multichannel speech
enhancement by combining Graph Convolutional Networks (GCNs) with spatial
filtering techniques such as the Minimum Variance Distortionless Response
(MVDR) beamformer. The algorithm aims to extract speech and noise from noisy
signals by computing the Power Spectral Density (PSD) matrices of the noise
and the speech signal of interest and then obtaining optimal weights for the
beam former using a frequency-time mask. The proposed method combines the MVDR
beam former with a super-Gaussian joint maximum a posteriori (SGJMAP) based SE
gain function and a GCN-based separation network. The SGJMAP-based SE gain
function is used to enhance the speech signals, while the GCN-based separation
network is used to separate the speech from the noise further.

###  3.8. Diffusion Probabilistic Model

Diffusion probabilistic models, inspired by non-equilibrium thermodynamics (Ho
et al., 2020; Sohl-Dickstein et al., 2015), have proven to be highly effective
for generating high-quality images and audio. These models create a Markov
chain of diffusion steps ($x_{t}\sim q(x_{t}|x_{t-1})$) from the original data
($x_{0}$) to the latent variable $x_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$
by gradually adding pre-scheduled noise to the data. The reverse diffusion
process then reconstructs the desired data samples ($x_{0}$) from the noise
$x_{T}$, as shown in Figure 7. Unlike VAE or flow models, diffusion models
keep the dimensionality of the latent variables fixed. While mostly used for
image and audio synthesis, diffusion models have potential applications in
speech-processing tasks, such as speech synthesis and enhancement. This
section offers a comprehensive overview of the fundamental principles of
diffusion models and explores their potential uses in the speech domain.

Figure 7. The Diffusion Probabilistic Model is a generative model that
progressively transforms a noise distribution into the target data
distribution through a series of diffusion steps, where the noise level
decreases as the process continues. The model is trained by maximizing the
likelihood of the data distribution and can be used for tasks such as speech
synthesis, enhancement, and denoising.

##### Forward diffusion process

Given a clean speech data $x_{0}\sim q_{data}(x_{0})$,

(24) $q(x_{1},...,x_{T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1}).$

At every time step $t$,
$q(x_{t}|x_{t-1}):=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I})$
where $\\{\beta_{t}\in(0,1)\\}_{t=1}^{T}$. As the forward process progresses,
the data sample $x_{0}$ losses its distinguishable features, and as
$T\to\infty$, $x_{T}$ approaches a standard Gaussian distribution.

##### Reverse diffusion process

The reverse diffusion process is defined by a Markov chain from
$x_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to $x_{0}$ and parameterized by
$\theta$:

(25)
$p_{\theta}(x_{0},...,x_{T-1}|x_{T})=\prod_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t})$

where $x_{T}\sim\mathcal{N}(0,I)$ and the transition probability
$p_{\theta}(x_{t-1}|x_{t})$ is learnt through noise-estimation. This process
eliminates the Gaussian noise added in the forward diffusion process.

#### 3.8.1 Application

Diffusion models have emerged as a leading approach for generating high-
quality speech in recent years (Chen et al., 2020c; Kong et al., 2020b; Popov
et al., 2021a, b; Jeong et al., 2021; Huang et al., 2022a). These non-
autoregressive models transform white noise signals into structured waveforms
via a Markov chain with a fixed number of steps. One such model, FastDiff, has
achieved impressive results in high-quality speech synthesis (Huang et al.,
2022a). By leveraging a stack of time-aware diffusion processes, FastDiff can
generate high-quality speech samples 58 times faster than real-time on a V100
GPU, making it practical for speech synthesis deployment for the first time.
It also outperforms other competing methods in end-to-end text-to-speech
synthesis. Another powerful diffusion probabilistic model proposed for audio
synthesis is DiffWave (Kong et al., 2020b). It is non-autoregressive and
generates high-fidelity audio for different waveform generation tasks, such as
neural vocoding conditioned on mel spectrogram, class-conditional generation,
and unconditional generation. DiffWave delivers speech quality on par with the
strong WaveNet vocoder (Oord et al., 2016) while synthesizing audio much
faster.

Diffusion models have shown great promise in speech processing, particularly
in speech enhancement (Lu et al., 2021; Serrà et al., 2022; Qiu et al., 2022;
Lu et al., 2022a). Recent advances in diffusion probabilistic models have led
to the development of a new speech enhancement algorithm that incorporates the
characteristics of the noisy speech signal into the diffusion and reverses
processes (Lu et al., 2022b). This new algorithm is a generalized form of the
probabilistic diffusion model, known as the conditional diffusion
probabilistic model. During its reverse process, it can adapt to non-Gaussian
real noises in the estimated speech signal. In addition, Qiu et al. (2022)
propose SRTNet, a novel method for speech enhancement that uses the diffusion
model as a module for stochastic refinement. The proposed method comprises a
joint network of deterministic and stochastic modules, forming the “enhance-
and-refine” paradigm. The paper also includes a theoretical demonstration of
the proposed method’s feasibility and presents experimental results to support
its effectiveness.

##  4\. Speech Representation Learning

The process of speech representation learning is essential for extracting
pertinent and practical characteristics from speech signals, which can be
utilized for various downstream tasks such as speaker identification, speech
recognition, and emotion recognition. While traditional methods for
engineering features have been extensively used, recent advancements in deep-
learning-based techniques utilizing supervised or unsupervised learning have
shown remarkable potential in this field. Nonetheless, a novel approach
founded on self-supervised representation learning has surfaced, aiming to
unveil the inherent structure of speech data and acquire representations that
capture the underlying structure of the data. This approach surpasses
traditional feature engineering methods and can significantly increase the
accuracy to a considerable extent and effectiveness of downstream tasks. The
primary objective of this new paradigm is to uncover informative and
meaningful features from speech signals and outperform existing approaches.
Therefore, this approach is considered a promising direction for future
research in speech representation learning.

This section provides a comprehensive overview of the evolution of speech
representation learning with neural networks. We will examine various
techniques and architectures developed over the years, including the emergence
of unsupervised representation learning methods like autoencoders, generative
adversarial networks (GANs), and self-supervised representation learning
frameworks. We will also examine the difficulties and constraints associated
with these techniques, such as data scarcity, domain adaptation, and the
interpretability of learned representations. Through a comprehensive analysis
of the advantages and limitations of different representation learning
approaches, we aim to provide insights into how to harness their power to
improve the accuracy and robustness of speech processing systems.

###  4.1. Supervised Learning

In supervised representation learning, the model is trained using annotated
datasets to learn a mapping between input data and output labels. The set of
parameters that define the mapping function is optimized during training to
minimize the difference between the predicted and true output labels in the
training data. The goal of supervised representation learning is to enable the
model to learn a useful representation or features of the input data that can
be used to accurately predict the output label for new, unseen data. For
instance, supervised representation learning in speech processing using CNNs
learn speech features from spectrograms. CNNs can identify patterns in
spectrograms relevant to speech recognition, such as those corresponding to
different phonemes or words. Unlike CNNs, which typically require spectrogram
input, RNNs can directly take in the raw speech signals as input and learn to
extract features or representations that are relevant for speech recognition
or other speech-processing tasks. Learning speaker representations typically
involves minimizing a loss function. Chung et al. (2020b) compares their
effectiveness for speaker recognition tasks, we distill it in Table 1 to
present an overview of commonly used loss functions. Additionally, a new
angular variant of the prototypical loss is introduced in their work. Results
from extensive experimental validation on the VoxCeleb1 test set indicate that
the GE2E and prototypical networks outperform other models in terms of
performance.

#### 4.1.1 Deep speaker representations

Speaker representation is a critical aspect of speech processing, allowing
machines to analyze and process various parts of a speaker’s voice, including
pitch, intonation, accent, and speaking style. In recent years, deep neural
networks (DNNs) have shown great promise in learning robust features for
speaker recognition. This section reviews deep learning-based techniques for
speaker representation learning that have demonstrated significant
improvements over traditional methods.

These deep speaker representations can be applied to a range of speaker-
recognition tasks beyond verification and identification, including
diarization (Wang et al., 2018a; Zhang et al., 2019e; Larcher et al., 2021),
voice conversion (Wu and Lee, 2020a; Lin et al., 2021a; Chou et al., 2019),
multi-speaker TTS (Saito et al., 2021; Paul et al., 2021; Xue et al., 2022),
speaker adaptation (Chorowski et al., 2019) etc. To provide a comprehensive
overview, we analyzed deep embeddings from the perspectives of input raw (Jung
et al., 2019; Ravanelli and Bengio, 2018) or mel-spectogram (Snyder et al.,
2018), network architecture (Lin and Mak, 2020; Desplanques et al., 2020),
temporal pooling strategies (Monteiro et al., 2019), and loss functions (Snell
et al., 2017; Chung et al., 2020b; Wang et al., 2018f). In the following
subsection, we introduce two representative deep embeddings: $d$-vector
(Variani et al., 2014a) and $x$-vector (Snyder et al., 2017, 2018). These
embeddings have been widely adopted recently and have demonstrated state-of-
the-art performance in various speaker-recognition tasks. By understanding the
strengths and weaknesses of different deep learning-based techniques for
speaker-representation learning, we can better leverage their power to improve
the accuracy and robustness of speaker-recognition systems.

  * •

d-vector technique, proposed by Variani et al. (2014) (Variani et al., 2014a),
serves as a frame-level speaker embedding method, as illustrated in Figure 8.
In this approach, during the training phase, each frame within a training
utterance is labeled with the speaker’s true identity. This transforms the
training process into a classification task, where a maxout Deep Neural
Network (DNN) classifies the frames based on the speaker’s identity. The DNN
employs softmax as the output layer to minimize the cross-entropy loss between
the ground-truth frame labels and the network’s output. During the testing
phase, the $d$-vector technique extracts the output activation of each frame
from the last hidden layer of the DNN, serving as the deep embedding feature
for that frame. To generate a compact representation called the $d$-vector,
the technique computes the average of the deep embedding features from all
frames within an utterance. The underlying hypothesis is that the compact
representation space developed using a development set can effectively
generalize to unseen speakers during the testing phase (Variani et al.,
2014a).

Figure 8. $d$-vector model architecture.

  * •

x-vector (Snyder et al., 2017, 2018) is a segment-level speaker embedding and
an advancement over the $d$-vector method as it incorporates additional
modeling of temporal information and phonetic information in speech signals,
resulting in improved performance compared to the $d$-vector. $x$-vector
employs an aggregation process to move from frame-by-frame speaker labeling to
utterance-level speaker labeling as highlighted in Figure 9. The network
structure of the $x$-vector is depicted in a figure, which consists of time-
delay layers for extracting frame-level speech embeddings, a statistical
pooling layer for concatenating mean and standard deviation of embeddings as a
segment-level feature, and a standard feedforward network for classifying the
segment-level feature to its speaker. $x$-vector is the segment-level speaker
embedding generated from the feedforward network’s second-to-last hidden
layer. The authors in (Yang and Liu, 2022; Rouvier et al., 2021) have also
discovered the significance of data augmentation in enhancing the performance
of the $x$-vector.

Figure 9. $x$-vector model architecture. $x_{1}$,$x_{2}$,….,$x_{T}$ are the
spectral features such as Mel spectrograms of the speech utterance.

Table 1. The table summarizes various loss functions used in training the
speaker recognition models including their formulation (Chung et al., 2020b).

Loss Function | Objective Type | Description
---|---|---
Softmax | Classification | $L_{S}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp{W^{T}_{y_{i}}x_{i}+b_{y_{i}}}}{\sum_{j=1}^{C}\exp{W^{T}_{j}x_{i}+b_{y_{i}}}}$
AM-Softmax (CosFace) (Wang et al., 2018f) | Classification | $L_{C}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp{s(\cos{\theta_{y_{i},i}}-m)}}{\exp{s(\cos{\theta_{y_{i},i}}-m)}+\sum_{j\neq y_{i}}\exp{s(\cos{\theta_{j,i}})}}$
AAM-Softmax (ArcFace) (Deng et al., 2019) | Classification | $L_{A}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp{s(\cos{\theta_{y_{i},i}}-m)}}{\exp{s(\cos{\theta_{y_{i},i}}+m)}+\sum_{j\neq y_{i}}\exp{s(\cos{\theta_{j,i}})}}$
Triplet (Schroff et al., 2015) | Metric learning (Zhang and Koishida, 2017) | $L_{T}=\frac{1}{N}\sum_{j=1}^{N}\max(0,||x_{j,0}-x_{j,1}||^{2}_{2},||x_{j,0}-x_{k\neq j,1}||^{2}_{2}+m)$
Prototypical (Snell et al., 2017) | Metric learning (Snell et al., 2017) | $L_{P}=-\frac{1}{N}\sum_{j=1}^{N}\log\frac{\exp{\textbf{S}_{j,j}}}{\sum_{k=1}^{N}\exp{\textbf{S}_{j,k}}}$
Generalized end-to-end (GE2E) (Wan et al., 2018) | Metric learning (Wang et al., 2019a) | $L_{G}=-\frac{1}{N}\sum_{j,i}\log\frac{\exp{\textbf{S}_{j,i,j}}}{\sum_{k=1}^{N}\exp{\textbf{S}_{j,i,k}}}$
Angular Prototypical | Metric learning | $L_{AP}=-\frac{1}{N}\sum_{j,i}\log\frac{\exp{\textbf{S}_{j,i,j}}}{\sum_{k=1}^{N}\exp{\textbf{S}_{j,i,k}}}$

###  4.2. Unsupervised learning

Unsupervised representation learning for speech processing has gained
significant emphasis over the past few years. Similar to visual modality in CV
and text modality in NLP, speech i.e. audio modality introduces unique
challenges. Unsupervised speech representation learning is concerned with
learning useful speech representations without using annotated data. Usually,
the model is first pre-trained on the task where plenty of data is available.
The model is then fined tuned or used to extract input representations for a
small model, specifically targeting tasks with limited data.

One approach to addressing the unique challenges of unsupervised speech
representation learning is to use probabilistic latent variable models (PLVM),
which assume an unknown generative process produces the data and enables the
learning of rich structural representations and reasoning about observed and
unobserved factors of variation in complex datasets such as speech within a
probabilistic framework. PLVM specified a joint distribution $p(x,z)$ over
unobserved stochastic latent variable z and observed variables x. By
factorizing the joint distribution into modular components, it becomes
possible to learn rich structural representations and reason about observed
and unobserved factors of variation in complex datasets such as speech within
a probabilistic framework.

Figure 10. Overview of difference between probabilistic latent variable models
and self-supervised learning. In latent variable models learn the functions
$f(.)$ and $g(.)$ learn the parameters of distribution $p$ and $q$. The latent
variable $z$ is used for representing learning.

The likelihood of a PLVM given a data x can be written as

(26) $p(x)=\int p(x|z)p(z)dz.$

Probabilistic latent variable models provide a powerful way to learn a
representation that captures the underlying relationships between observed and
unobserved variables, without requiring explicit supervision or labels. These
models involve unobserved latent variables that must be inferred from the
observed data, typically using probabilistic inference techniques such as
Markov Chain Monte Carlo (MCMC) methods. In the context of representation
learning, Variational autoencoders (VAE) are commonly used with latent
variable models for various speech processing tasks, leveraging the power of
probabilistic modeling to capture complex patterns in speech data.

###  4.3. Semi-supervised Learning

Semi-supervised learning can be viewed as a process of optimizing a model
using both labeled and unlabeled data. The set of labeled data points, denoted
by $X_{L}$, contains $N_{L}$ items, where each item is represented as
$(x_{i},y_{i})$ with $y_{i}$ being the label of $x_{i}$. On the other hand,
the set of unlabeled data points, denoted by $X_{U}$, consists of $N_{U}$
items, represented as $x_{N_{L}+1},x_{N_{L}+2},...,x_{N_{L}+N_{U}}$.

In semi-supervised learning, the objective is to train a model $f_{\theta}$
with parameters $\theta$ that can minimize the expected loss over the entire
dataset. The loss function $L(y,f_{\theta}(x))$ is used to quantify the
deviation between the model’s prediction $f_{\theta}(x)$ and the ground truth
label $y$. The expected loss can be mathematically expressed as:

(27) $L(y,f_{\theta}(x))=E_{(x,y)\sim p_{data}(x,y)}[L(y,f_{\theta}(x))]$

where $p_{data}(x,y)$ is the underlying data distribution.In semi-supervised
learning, the loss function is typically decomposed into two parts: a
supervised loss term that is only defined on the labeled data, and an
unsupervised loss term that is defined on both labeled and unlabelled data.
The supervised loss term is calculated as follows:

(28) $\mathcal{L}_{sup}=\frac{1}{N_{L}}\sum_{(x,y)\in
X_{L}}L(y,f_{\theta}(x))$

The unsupervised loss term leverages the unlabelled data to encourage the
model to learn meaningful representations that capture the underlying
structure of the data. One common approach is to use a regularization term
that encourages the model to produce similar outputs for similar input data.
This can be achieved by minimizing the distance between the output of the
model for two similar input data points. One such regularization term is the
entropy minimization term, which can be expressed as:

(29) $\mathcal{L}_{unsup}=\frac{1}{N_{U}}\sum_{(x_{i})\in
X_{U}}\sum_{j=1}^{|y|}p_{\theta}(y_{j},x_{i})\log p_{\theta}(y_{j},x_{i})$

where $p_{\theta}(y_{j}|x_{i})$ is the predicted probability of the $j$-th
label for the unlabelled data point $x_{i}$. Finally the overall objective
function for semi-supervised learning can be expressed as
$\mathcal{L}=\mathcal{L}_{sup}+\alpha\mathcal{L}_{unsup}$, $\alpha$ is a
hyperparameter that controls the weight of the unsupervised loss term. The
goal is to find the optimal parameters $\theta$ that minimize this objective
function. Semi-supervised learning involves learning a model from both
labelled and unlabelled data by minimizing a combination of supervised and
unsupervised loss terms. By leveraging the additional unlabelled data, semi-
supervised learning can improve the generalization and performance of the
model in downstream tasks.

Semi-supervised learning techniques are increasingly being employed to enhance
the performance of DNNs across a range of downstream tasks in speech
processing, including ASR, TTS, etc. The primary objective of such approaches
is to leverage large unlabelled datasets to augment the performance of
supervised tasks that rely on labelled datasets. The recent advancements in
speech recognition have led to a growing interest in the integration of semi-
supervised learning methods to improve the performance of ASR and TTS systems
(Zhang et al., 2020c; Baskar et al., 2019; Zhang et al., 2022b; Kahn et al.,
2020; Xu et al., 2021a; Chung et al., 2020d). This approach is particularly
beneficial in scenarios where labelled data is scarce or expensive to acquire.
In fact, for many languages around the globe, labelled data for training ASR
models are often inadequate, making it challenging to achieve optimal results.
Thus, using a semi-supervised learning model trained on abundant resource data
can offer a viable solution that can be readily extended to low-resource
languages.

Semi-supervised learning has emerged as a valuable tool for addressing the
challenges of insufficient annotations and poor generalization (Hady and
Schwenker, 2013). Research in various domains, including image quality
assessment (Liu et al., 2019), has demonstrated that leveraging both labelled
and unlabelled data through semi-supervised learning can lead to improved
performance and generalization. In the domain of speech quality assessment,
several studies (Serrà et al., 2021) have exploited the generalization
capabilities of semi-supervised learning to enhance performance.

Moreover, semi-supervised learning has gained significant attention in other
areas of speech processing, such as end-to-end speech translation (Pino et
al., 2020). By leveraging large amounts of unlabelled data, semi-supervised
learning approaches have demonstrated promising results in improving the
performance and robustness of speech translation models. This highlights the
potential of semi-supervised learning to address the limitations of
traditional supervised learning approaches in a variety of speech processing
tasks.

###  4.4. Self-supervised representation learning (SSRL)

Self-supervised representation learning (SSRL) is a machine learning approach
that focuses on achieving robust and in-depth feature learning while
minimizing reliance on extensively annotated datasets, thus reducing the
annotation bottleneck (Ericsson et al., 2022; Lee et al., 2022b). SSRL
comprises various techniques that allow models to be trained without needing
human-annotated labels (Ericsson et al., 2022; Lee et al., 2022b). One of the
key advantages of SSRL is its ability to operate on unlabelled datasets, which
reduces the need for large annotated datasets (Ericsson et al., 2022; Lee et
al., 2022b). In recent years, self-supervised learning has progressed rapidly,
with some methods approaching or surpassing the efficacy of fully supervised
learning methods. Self-supervised learning methods typically involve pretext
tasks that generate pseudo labels for discriminative model training without
actual labeling. The difference between self-supervised representation
learning and unsupervised representation is highlighted in Figure 10. In
contrast to unsupervised representation learning, SSRL techniques are designed
to generate these pseudo labels for model training. The ability of SSRL to
achieve robust and in-depth feature learning without relying heavily on
annotated datasets holds great promise for the continued development of
machine learning techniques.

SSRL differs from supervised learning mainly in terms of its data
requirements. While supervised learning relies on labeled data, where the
model learns from input-output pairs, SSL generates its own labels from the
input data, eliminating the need for labeled data (Lee et al., 2022b). The SSL
approach trains the model to predict a portion of the input data, which is
then utilized as a label for the task at hand (Lee et al., 2022b). Although
SSRL is an unsupervised learning technique, it seeks to tackle tasks commonly
associated with supervised learning without relying on labeled data (Lee et
al., 2022b).

Figure 11. Generative approaches to self-supervised learning.

#### 4.4.1 Generative Models

This method involves instructing a model to produce samples resembling the
input data without explicitly learning the labels, creating valuable
representations applicable to other tasks. The detailed architecture for
generative models with three different variants is shown in Figure 11. The
earliest self-supervised method, predicting masked inputs using surrounding
data, originated from the text field in 2013 with word2vec. The continuous bag
of words (CBOW) concept of word2vec predicts a central word based on its
neighbors, resembling ELMo and BERT’s masked language modeling (MLM). These
non-autoregressive generative approaches differ in their use of advanced
structures, such as bidirectional LSTM (for ELMo) and transformer (for BERT),
with recent models producing contextual embeddings. In the context of the
speech, Mockingjay (Liu et al., 2020) applied masking to all feature
dimensions in the speech domain, whereas TERA (Liu et al., 2021b) applied to
mask only to a particular subset of feature dimensions. The summary of
generative self-supervised approaches along with the data used for training
the models are outlined in Table 2. We further discuss different generative
approaches as highlighted in Figure 11 as follows:

Table 2.  Summary of generative self-supervised approaches and proposed models
for speech processing with associated metrics and training Data. ASR:
Automatic Speech Recognition, PR: Phoneme Recognition. PC: Phoneme
Classification, SR: Speaker Recognition, LS: LibriSpeech.

Model | Reference | Task (Metric) |  Pre-Training Dataset (hours) | Dataset
---|---|---|---|---
Training | Test
Mockingjay | (Liu et al., 2020) | PC | LS (360h) | LS (360h) | LS (test-clean)
SR | LS (360h) | LS (100h) | LS (100h)
PASE | (Pascual et al., 2019) | ASR | LS (50 hr) | DIRHA | DIRHA
PASE+ | (Ravanelli et al., 2020) | ASR | LS (50 hr) |  | DIRHA
---
CHiME-5
| DIRHA
---
CHiME-5
DeCoAR | (Ling and Liu, 2020) | ASR |  | LS (100h, 360h, 460 h, 960h)
---
WSJ si284
| LS (100h, 360h, 460 h, 960h)
---
WSJ si284
| LS (test-clean)
---
LS (test-other)

  * •

Auto-encoding Models: Auto-encoding Models have garnered significant attention
in the domain of self-supervised learning, particularly Autoencoders (AEs) and
Variational Autoencoders (VAEs). AEs consist of an encoder and a decoder that
work together to reconstruct input while disregarding less important details,
prioritizing the extraction of meaningful features. VAEs, a probabilistic
variant of AEs, have found wide-ranging applications in the field of speech
modeling. Furthermore, the vector-quantized variational autoencoder (VQ-VAE)
(Van Den Oord et al., 2017) has been developed as an extended generative
model. The VQ-VAE introduces parameterization of the posterior distribution to
represent discrete latent representations. Remarkably, the VQ-VAE has
demonstrated notable success in generative spoken language modeling. By
combining a discrete latent space with self-supervised learning, its
performance is further improved.

  * •

Autoregressive models: Autoregressive generative self-supervised learning uses
autoregressive prediction coding technique (Chung et al., 2019) to model the
probability distribution of a sequence of data points. This approach aims to
predict the next data point in a sequence based on the previous data points.
Autoregressive models typically use RNNs or a transformer architecture as a
basic model.

The authors in paper (Oord et al., 2016) introduce a generative model for raw
audio called WaveNet, based on PixelCNN (Van den Oord et al., 2016). To
enhance the model’s ability to handle long-range temporal dependencies, the
authors incorporate dilated causal convolutions (Oord et al., 2016). They also
utilize Gated Residual blocks and skip connections to improve the model’s
expressivity.

  * •

Masked Reconstruction: The concept of masked reconstruction is influenced by
the masked language model (MLM) task proposed in BERT (Devlin et al., 2018).
This task involves masking specific tokens in input sentences with learned
masking tokens or other input tokens, and training the model to reconstruct
these masked tokens from the non-masked ones. Recent research has explored
similar pretext tasks for speech representation learning that help models
develop contextualized representations capturing information from the entire
input, like the DeCoAR model (Ling and Liu, 2020). This approach assists the
model in comprehending input data better, leading to more precise and
informative representations.

#### 4.4.2 Contrastive Models

The technique involves training a model to differentiate between similar and
dissimilar pairs of data samples, which helps the model acquire valuable
representations that can be utilized for various tasks, as shown on Figure 12.
The fundamental principle of contrastive learning is to generate positive and
negative pairs of training samples based on the comprehension of the data. The
model must learn a function that assigns high similarity scores to two
positive samples and low similarity scores to two negative samples. Therefore,
generating appropriate samples is crucial for ensuring that the model
comprehends the fundamental features and structures of the data. Table 3
outlines popular contrastive self-supervised models used for different speech-
processing tasks. We discuss Wav2Vec 2.0 since it has achieved state-of-the-
art results in different downstream tasks.

Figure 12. Contrastive Self-supervised learning: Contrastive Predictive
Coding.

  * •

Wav2Vec 2.0 (Baevski et al., 2020) is a framework for self-supervised learning
of speech representations that is one of the current state-of-the-art models
for ASR (Baevski et al., 2020). The training of the model occurs in two
stages. Initially, the model operates in a self-supervised mode during the
first phase, where it uses unlabelled data and aims to achieve the best speech
representation possible. The second phase is fine-tuning a particular dataset
for a specific purpose. Wav2Vec 2.0 takes advantage of self-supervised
training and uses convolutional layers to extract features from raw audio.

Table 3.  Summary of contrastive self-supervised approaches and proposed
models for speech processing with associated metrics and training Data. ASR:
Automatic Speech Recognition, PR: Phoneme Recognition. PC: Phoneme
Classification, SR: Speaker Recognition, LS: LibriSpeech, LL: LibriLight, WSJ:
Wall Street Journal.

Model | Reference | Task |  Pre-Training Dataset (hours) | Dataset
---|---|---|---|---
Training | Test
CPC | (Oord et al., 2018a) | PC | LS (100h) | LS (100h) | LS (100h)
SR | LS (100h) | LS (100h) | LS (100h)
Modified CPC | (Riviere et al., 2020) | PC |  | LS (100h, 360h)
---
Zerospeech2017(45h)
CV-Dataset | CV-Dataset
Bidirectional CPC | (Kawakami et al., 2020) | ASR |  | WSJ (80h)
---
LS (960h)
TIMIT (5h)
SSA (1h)
TED3 (440h)
SwithBoard (310h)
| WSJ (80h)
---
LS (960h)
TIMIT (5h)
SSA (1h)
TED3 (440h)
SwithBoard (310h)
| WSJ (test92, test93)
---
LS (test-clean, test-other)
TED3 (dev, test)
SwithBoard (eval2000)
ASR-Multi |  | Audio Set (2500h)
---
AVSpeech (3100h)
CV-Dataset (430h
| Audio Set (2500h)
---
AVSpeech (3100h)
CV-Dataset (430h)
| OpenSLR
---
ALFFA
wav2vec | (Schneider et al., 2019) | ASR |  | LS 80/860h
---
LS 960h + WSJ (si284)
WSJ (si284) | WSJ (eval92)
PR | TIMIT | TIMIT | TIMIT
wav2vec 2.0 | (Baevski et al., 2020) | ASR |  | LS (960h)
---
LL (60000h)
LS (960h) |  | LS (test-clean)
---
LS (test-other)
PR |  | LS (960h)
---
LL (60000h)
TIMIT | TIMIT
vq-wav2vec 2.0 | (Baevski et al., 2019b) | ASR | LS (960h) | WSJ (si284) | WSJ (eval92)
PR | LS (960h) | TIMIT | TIMIT
wav2vec-C | (Sadhu et al., 2021) | ASR | Alexa-10k | Alexa-eval | Alexa-eval
w2v-BERT | (Chung et al., 2021) | ASR | LL (60000h) | LS (960h) |  | LS (test)
---
LS (test-other)
LS (dev)
LS (dev-other)
Speech SimCLR | (Jiang et al., 2020) | ASR |  | LS (960h)
---
WJS (si284)
TED2
WJS (si284) | WJS (si284)
PR |  | LS (960h)
---
WJS (si284)
TED2
TIMIT | TIMIT
UnSpeech | (Milde and Biemann, 2018) | ASR-Mult |  | LL (60000h)
---
GigaSpeech (10000h)
VP (24000h)
SUPERB | SUPERB

In the speech field, researchers have explored different approaches to avoid
overfitting, including augmentation techniques like Speech SimCLR (Jiang et
al., 2020) and the use of positive and negative pairs through methods like
Contrastive Predictive Coding (CPC) (Ooster and Meyer (2019)), Wav2vec (v1,