Slot Filling: Slot Filling (SF) is a widely used technique in Speech Language
Understanding (SLU) that enables the extraction of important information, such
as names, dates, and locations, from a user’s speech. The process involves
identifying the specific pieces of information that are relevant to the user’s
request and placing them into pre-defined slots. For instance, if a user asks
for the weather in a particular city, the system will identify the city name
and fill it into the appropriate slot, thereby providing an accurate and
relevant response.

#### 5.11.2 Dataset

  * •

Keyword Spotting Datasets:

    * –

Coucke et al. (2019): This dataset is a speech command recognition dataset
that consists of 105,000 spoken commands in English, with each command being
one of 35 keywords. The dataset is designed to be highly varied and
challenging, with a diverse set of speakers and background noise conditions.

    * –

Leroy et al. (2019): This dataset is a federated learning-based keyword
spotting dataset, it is composed of data from multiple sources that are
trained together without sharing the raw data. The dataset consists of audio
recordings from multiple devices and environments, with the goal of improving
the robustness of KS across different devices and settings

    * –

Auto-KWS (Wang et al., 2021a): This dataset is automatically generated using
TTS approach. The dataset consists of 1000 keywords spoken by 100 different
synthetic voices, with variations in accent, gender, and age.

    * –

Speech Commands (Warden, 2018): This data is a large-scale dataset for KS task
that consists of over $100,000$ spoken commands in English, with each command
belonging to 35 different keywords. The dataset is specifically designed to be
highly varied and challenging, with a diverse set of speakers and background
noises. It is commonly used as a benchmark dataset for KS research.

  * •

Intent Classification and Slot Filling

    * –

ATIS (Hemphill et al., 1990): The Airline Travel Information System (ATIS)
dataset is a collection of spoken queries and responses related to airline
travel, such as flight reservations, flight status, and airport information.
The dataset is annotated with both intent labels (e.g. “flight booking”,
“flight status inquiry”) and slot labels (e.g. depart city, arrival city,
date). The ATIS dataset has been used extensively as a benchmark for natural
language understanding models.

    * –

SNIPS (Coucke et al., 2018): SNIPS is a dataset of voice commands designed for
building a natural language understanding system. It consists of thousands of
examples of spoken requests, each annotated with the intent of the request
(e.g. “play music”, “set an alarm”, etc.). The dataset is widely used for
training IC and SF models.

    * –

Fluent Speech Commands (Lugosch et al., 2019): It is a dataset of voice
commands for controlling smart home devices, such as lights, thermostats, and
locks. The dataset consists of over 1,5000 spoken commands, each labeled with
the intended devices and action (e.g. “turn on the living room lights”, “set
the thermostat to 72 degrees”). The dataset is designed to have variations in
speaker accent, background noise, and device placement.

    * –

MIT-Restaurant and MIT-Movie (Liu et al., 2013): These are two datasets
created by researchers at MIT for training natural language understanding
models from restaurant and movie information requests. The dataset contains
spoken and text-based queries, each labeled with the intent of the request
(e.g. “find a nearby Italian restaurant”,” get information about the movie
Inception”) and relevant slot information (e.g. restaurant type, movie name,
etc). The datasets are widely used for benchmarking natural language
understanding models.

#### 5.11.3 Models

  * •

Keyword Spotting: The state-of-the-art techniques for keyword spotting in
speech involve deep learning models, such as CNNs (Rostami et al., 2022) and
transformers (Berg et al., 2021). Wav2Keyword is one of the popular model
based on Wav2Vec2.0 architecture (Seo et al., 2021) and have achieved SOTA
results on Speech Commands data V1 and V21. Another model that achieves SOTA
classification accuracy on the Google Speech commands dataset is Keyword
Transformer (KWT) (Seo et al., 2021). KWT uses a transformer model and
achieves $98.6\%$ and $97.7\%$ accuracy on the 12 and 35-word tasks,
respectively. KWT also has low latency and can be used on mobile devices.

  * •

The DIET architecture, as introduced in (Bunk et al., 2020), is a transformer-
based multitask model that addresses intent classification and entity
recognition simultaneously. DIET allows for the seamless integration of
various pre-trained embeddings such as BERT, GloVe, and ConveRT. Results from
experiments show that DIET outperforms fine-tuned BERT and has the added
benefit of being six times faster to train.

  * •

Chang et al. (2022) investigated the effectiveness of prompt tuning on the
GSLM architecture and showcased its competitiveness on various SLU tasks, such
as KS, IC, and SF. Impressively, this approach achieves comparable results
with fewer trainable parameters than full fine-tuning. Despite being a popular
and effective technique in numerous NLP tasks, prompt tuning has not received
much attention in the speech community. Additionally, other researchers have
pursued a different path by utilizing pre-trained wav2vec2.0 and different
adapters (Li et al., 2023) to attain state-of-the-art outcomes.

Table 12. Comprehensive performance analysis of various models for Keyword
Spotting (KS) and Slot Filling (SF) tasks, evaluated on two benchmark
datasets: Google Speech Commands for KS and ATIS for SF.

Keyword Spotting on Google Speech Commands (Accuracy % $\uparrow$) | Slot Filling on ATIS (F1 $\uparrow$)
---|---
Model | Reference | Google Speech Commands V1 12 | Google Speech Commands V2 12 | Google Speech Commands V2 35 | Model | Reference | ATIS
TripletLoss-res15 | (Vygon and Mikhaylovskiy, 2021) | 98.56 | 98.37 | 97.0 | CTRAN | (Rafiepour and Sartakhti, 2023) | 0.9846
Wav2KWS | (Seo et al., 2021) | 97.9 | 98.5 | 97.8 | Bi-model with a decoder | (Wang et al., 2018d) | 0.9689
KWT-3 | (Berg et al., 2021) | 97.49 ±0.15 | 98.56 ±0.07 | 97.69 ±0.09 | Joint BERT | (Chen et al., 2019b) | 0.961
KWT-1 | (Berg et al., 2021) | 97.27 ±0.08 | 98.43±0.08 | 97.74 ±0.03 | Joint BERT + CRF | (Chen et al., 2019b) | 0.96
KWT-2 | (Berg et al., 2021) | 97.26±0.18 | 98.08±0.10 | 96.95±0.14 | SF-ID | (Niu et al., 2019) | 0.958
Attention RNN | (Rybakov et al., 2020) | 95.6 | 96.9 | 93.9 | Capsule-NLU | (Zhang et al., 2018a) | 0.952

Despite the remarkable progress made in the field of SLU, accurately
comprehending human speech in real-life situations continues to pose
significant challenges. These challenges are amplified by the presence of
diverse accents, dialects, and linguistic variations. In a notable study,
Vanzo et al. (Vanzo et al., 2016) emphasize the significance of SLU in
facilitating effective human-robot interaction, particularly within the
context of house service robots. The authors delve into the specific obstacles
encountered in this domain, which encompass handling noisy and unstructured
speech, accommodating various accents and speech variations, and deciphering
complex commands involving multiple actions. To overcome these obstacles,
ongoing research endeavors are dedicated to developing innovative solutions
that enhance the precision and efficacy of SLU systems. By addressing these
challenges, the aim is to enable more robust and accurate speech comprehension
in diverse real-life scenarios.

Recent studies, including the comprehensive analysis of the performance of
different models and techniques for Keyword Spotting (KS) and Slot Filling
(SF) tasks on Google Speech Commands and ATIS benchmark datasets (Table 12),
have furnished valuable insights into the strengths and limitations of such
approaches in SLU. Capitalizing on these findings and leveraging the latest
advances in deep learning and speech recognition could help us continue to
expand the frontiers of spoken language understanding and drive further
innovation in this domain.

###  5.12. Audio/visual multimodal speech processing

The process of speech perception in humans is intricate and involves multiple
sensory modalities, including auditory and visual cues. The generation of
speech sounds involves articulators such as the tongue, lips, and teeth, whose
movements are critical for producing different speech sounds and visible to
others. The importance of visual cues becomes more pronounced for individuals
with hearing impairments who depend on lip-reading to comprehend spoken
language, while individuals with normal hearing can also benefit from visual
cues in noisy environments.

When investigating language comprehension and communication, it is essential
to consider both auditory and visual information, as studies have demonstrated
that visual information can assist in distinguishing between acoustically
similar sounds that differ in articulatory characteristics. A comprehensive
understanding of the interaction between these sensory modalities can lead to
the development of assistive technologies for individuals with hearing
impairments and enhance communication strategies in challenging listening
environments.

#### 5.12.1 Task Description

The tasks under audiovisual multimodal processing can be subdivided into the
following categories.

  * •

Lip-reading: Lip-reading is a remarkable ability that allows us to comprehend
spoken language from silent videos. However, it is a challenging task even for
humans. Recent advancements in deep learning technology have enabled the
development of neural network-based lip-reading models to accomplish this task
with high accuracy. These models take silent facial videos as input and
produce the corresponding speech audio or characters as output. The potential
applications of automatic lip-reading models are vast and diverse, including
enabling videoconferencing in noisy environments, using surveillance videos as
long-range listening devices, and facilitating conversations in noisy social
settings. Developing these models could significantly improve our daily lives.

  * •

Audiovisual speech separation: Recent years have witnessed a growing interest
in audiovisual speech separation, driven by the remarkable human capacity to
selectively focus on a specific sound source amidst background noise, commonly
known as the ”cocktail party effect.” This phenomenon poses a significant
challenge in computer speech recognition, prompting the development of
automatic speech separation techniques aimed at isolating individual speech
sources from complex audio signals. In a noteworthy study by Ephrat et al.
(2018) Ephrat et al. (2018), the authors proposed that audiovisual speech
separation surpasses audio-only approaches by leveraging visual cues from a
speaker’s face to resolve ambiguity in speech signals. By integrating visual
information, the model’s ability to disentangle overlapping speech signals is
enhanced. The implications of automatic speech separation extend across
diverse applications, including assistive technologies for individuals with
hearing impairments and head-mounted devices designed to facilitate effective
communication in noisy meeting scenarios.

  * •

Talking face generation: Generating a realistic talking face of a target
character, synchronized with a given speech and ensuring smooth transitions
between facial images, is the objective of talking face generation. This task
has garnered substantial interest and poses a significant challenge due to the
dynamic nature of facial movements, which depend on both visual information
(input face image) and acoustic information (input speech audio) to achieve
accurate lip-speech synchronization. Despite its challenges, talking face
generation holds immense potential for various applications, including
teleconferencing, creating virtual characters with specific facial
expressions, and enhancing speech comprehension. In recent years, significant
advancements have been made in the field of talking face generation, as
evidenced by notable studies (Song et al., 2018; Zhou et al., 2019b; Chen et
al., 2019a; Eskimez et al., 2020, 2021).

#### 5.12.2 Datasets

Several datasets are widely used for audiovisual multimodal research,
including VoxCeleb, TCD-TIMID (Harte and Gillen, 2015) , etc. We briefly
discuss some of them in the following section.

  * •

_TCD-TIMID(Harte and Gillen, 2015):_ This is an extensive and diverse
audiovisual dataset that encompasses both audio and video recordings of 600
distinct sentences spoken by 60 participants. The dataset features a wide
range of speakers with different genders, accents, and backgrounds, making it
highly suitable for talker-independent speech recognition research. The audio
recordings are of exceptional quality, captured using high-fidelity
microphones with a sampling rate of 48kHz. Meanwhile, the video footage is of
720p resolution and includes depth information for every frame

  * •

_LipReading in the Wild (LRW)(Chung and Zisserman, 2017):_ The LRW is a
comprehensive audiovisual dataset that encompasses 500 distinct words spoken
by more than 1000 speakers. This dataset has been segmented into distinct
training, evaluation, and test sets to facilitate efficient research.
Additionally, the LRW-1000 dataset (Yang et al., 2019c) represents a subset of
LRW, featuring a 1000-word vocabulary. Researchers can benefit from pre-
trained weights included with this dataset, simplifying the evaluation
process. Overall, these datasets are highly regarded in the scientific
community for their size and versatility in supporting research related to
speech recognition and natural language processing

  * •

_LRS2 and LRS3
101010https://www.robots.ox.ac.uk/ṽgg/data/lip_reading/lrs2.html_: The LRS2
and LRS3 datasets are additional examples of audiovisual speech recognition
datasets that have been gathered from videos captured in real-world settings.
Each of these datasets has its own distinct train/test split and includes
cropped face tracks as well as corresponding audio clips sourced from British
television. Both datasets are considered to be of significant value to
researchers in the field of speech recognition, particularly those focused on
audiovisual analysis.

  * •

_GRID(Cooke et al., 2006):_ This dataset comprises high-fidelity audio and
video recordings of more than 1000 sentences spoken by 34 distinct speakers,
including 18 males and 16 females. The sentences were gathered using the
prompt ”put red at G9 now” and are widely employed in research related to
audio-visual speech separation and talking face synthesis. The dataset is
considered to be of exceptional quality and is highly sought after in the
scientific community.

#### 5.12.3 Models

In recent years, there has been a remarkable surge in the development of
algorithms tailored for multimodal tasks. Specifically, significant attention
has been devoted to the advancement of neural networks for Text-to-Speech
(TTS) applications (Ren et al., 2019, 2020, 2021; Kim et al., 2021b). The
integration of visual and auditory modalities through multimodal processing
has played a pivotal role in enhancing various tasks relevant to our daily
lives. Lip-reading, for instance, has witnessed notable progress in recent
years, whether accompanied by audio or not. Son et al. have made a significant
contribution to this field with their hybrid model (Son Chung et al., 2017).
Combining convolutional neural networks (CNN), long short-term memory (LSTM)
networks, and an attention mechanism, their model captures correlations
between lip videos and audio, enabling accurate character generation.
Additionally, the authors introduce a new dataset called LRS, which
facilitates the development of lip-reading models.

Another noteworthy model, LiRA (Ma et al., 2021), focuses on self-supervised
learning for lip-reading. It leverages lip image sequences and audio waveforms
to derive high-level representations during the pre-training stage, achieving
word-level and sentence-level lip-reading capabilities. In the realm of
capturing human emotions expressed through acoustic signals, Ephrat et al.
(Ephrat et al., 2017) propose an innovative model that frames the task as an
acoustic regression problem instead of a visual-to-text modeling approach.
Their work emphasizes the advantages of this perspective. Furthermore,
Vid2Speech (Ephrat and Peleg, 2017), a CNN-based model, takes facial image
sequences as input and generates corresponding speech audio waveforms. It
employs a two-tower CNN model that processes facial grayscale images while
calculating optical flow between frames. Additionally, other models such as
those based on mutual information maximization (Zhao et al., 2020b) and
spatiotemporal fusion (Zhang et al., 2019a) have been proposed for the lip-
reading task, further expanding the methodologies explored in this domain.

In an early attempt to develop algorithms for audiovisual speech separation,
the authors of (Ephrat et al., 2018) proposed a CNN-based architecture that
encodes facial images and speech spectrograms to compute a complex mask for
speech separation. Additionally, they introduced the AVspeech dataset in this
work. AV-CVAE (Nguyen et al., 2021) utilizes a conditional VAE to detect the
lip movements of the speaker and predict separated speech. In a deviation from
speech signals, (Montesinos et al., 2021) focuses on audiovisual singing
separation and employs a two-stream CNN architecture, Y-Net (Mehta et al.,
2018), to process audio and video separately. This work introduces a large
dataset of solo singing videos for audiovisual singing separation. The
VisualSpeech (Gao and Grauman, 2021) architecture takes a face image sequence
and mixed audio of lip movement as input and predicts a complex mask. It also
proposes a cross-modal embedding space to facilitate the correlation of audio
and visual modalities. Finally, FaceFilter (Chung et al., 2020a) uses still
images as visual information, and other methods for the audiovisual speech
separation task are proposed in (Afouras et al., 2018; Michelsanti et al.,
2021; Gabbay et al., 2017).

The rise of Deepfake videos on the internet has led to a surge in demand for
creating realistic talking faces for various applications, such as video
production, marketing, and entertainment. Previously, the conventional
approach involved manipulating 3D meshes to create specific faces, which was
time-consuming and limited to certain identities. However, recent advancements
in deep generative models have made significant progress. For example, DAVS
(Zhou et al., 2019b) introduced an end-to-end trainable deep neural network
capable of learning a joint audiovisual representation, which uses adversarial
training to disentangle the latent space. Another architecture proposed by
ATVGnet (Chen et al., 2019a) consists of an audio transformation network (AT-
net) and a visual generation network (VG-net) for processing acoustic and
visual information, respectively. This method introduced a regression-based
discriminator, a dynamically adjustable pixel-wise loss, and an attention
mechanism. In (Zhu et al., 2018), a novel framework for talking face
generation was presented, which discovers audiovisual coherence through an
asymmetrical mutual information estimator. Furthermore, the authors in
(Eskimez et al., 2020) proposed an end-to-end approach based on generative
adversarial networks that use noisy speech for talking face generation. In
addition, alternative methods based on conditional recurrent adversarial
networks and speech-driven talking face generation were introduced in (Song et
al., 2018; Eskimez et al., 2021).

##  6\. Advanced Transfer Learning Techniques for Speech Processing

###  6.1. Domain Adaptation

#### 6.1.1 Task Description

Domain adaptation is a field that deals with adapting a model trained on a
labeled dataset from a source domain to a target domain, where the source
domain differs from the target domain. The goal of domain adaptation is to
reduce the performance gap between the source and target domains by minimizing
the difference between their distributions. In speech processing, domain
adaptation has various applications such as speech recognition (Bousquet and
Rouvier, 2019; Nidadavolu et al., 2019; Lee et al., 2019; Chowdhury et al.,
2022; Hu et al., 2022a), speaker verification (Xia et al., 2019; Chen et al.,
2021f; Wang et al., 2019b; Zhang et al., 2023b; Himawan et al., 2019), and
speech synthesis (Xin et al., 2020; Yue et al., 2022). This section explores
the use of domain adaptation in these tasks by reviewing recent literature on
the subject. Specifically, we discuss the techniques used in domain
adaptation, their effectiveness, and the challenges that arise when applying
them to speech processing.

#### 6.1.2 Models

Various techniques have been proposed to adapt a deep learning model for
speech processing tasks. An example of a technique is reconstruction-based
domain adaptation, which leverages an additional reconstruction task to
generate a communal representation for all the domains. The Deep
Reconstruction Classification Network (DRCN) (Ghifary et al., 2016) is an
illustration of such an approach, as it endeavors to address both tasks
concurrently: (i) classification of the source data and (ii) reconstruction of
the input data. Another technique used in domain adaptation is the domain-
adversarial neural network architecture, which aims to learn domain-invariant
features using a gradient reversal layer (C S et al., 2021; Zhang et al.,
2022e; Wang et al., 2018c).

Different domain adaptation techniques are successfully applied to different
speech processing tasks, such as speaker recognition (Li et al., 2022e; Hu et
al., 2022a; Bousquet and Rouvier, 2019; Nidadavolu et al., 2019) and
verification (Chen et al., 2021f, 2020b; Zhang et al., 2023b; Li et al.,
2022c; Zhu and Chen, 2022), where the goal is to verify the identity of a
speaker using their voice. One approach for domain adaptation in speaker
verification is to use adversarial domain training to learn speaker-
independent features insensitive to variations in the recording environment
(Chen et al., 2020b).

Domain adaptation has also been applied to speech recognition (Mani et al.,
2020; Hwang et al., 2022; Yue et al., 2022; Sukhadia and Umesh, 2023) to
improve speech recognition accuracy in a target domain. One recent approach
for domain adaptation in ASR is prompt-tuning (Dingliwal et al., 2021), which
involves fine-tuning the ASR system on a small amount of data from the new
domain. Another approach is to use adapter modules for transducer-based speech
recognition systems (Majumdar et al., 2023; Sathyendra et al., 2022), which
can balance the recognition accuracy of general speech and improve recognition
on adaptation domains. The Machine Speech Chain integrates both end-to-end
(E2E) ASR and neural text-to-speech (TTS) into one circle (Yue et al., 2022).
This integration can be used for domain adaptation by fine-tuning the E2E ASR
on a small amount of data from the new domain and then using the TTS to
generate synthetic speech in the new domain for further training.

In addition to domain adaptation techniques used in speech recognition, there
has been growing interest in adapting text-to-speech (TTS) models to specific
speakers or domains. This research direction is critical, especially in low-
resource settings where collecting sufficient training data can be
challenging. Several recent works have proposed different approaches for
speaker and domain adaptation in TTS, such as AdaSpeech (Chen et al., 2021e;
Yan et al., 2021; Wu et al., 2022).

###  6.2. Meta Learning

#### 6.2.1 Task Description

Meta-learning is a branch of machine learning that focuses on improving the
learning algorithms used for tasks such as parameter initialization,
optimization strategies, network architecture, and distance metrics. This
approach has been demonstrated to facilitate faster fine-tuning, better
performance convergence, and the ability to train models from scratch, which
is especially advantageous for speech-processing tasks. Meta-learning
techniques have been employed in various speech-processing tasks, such as low-
resource ASR (Hsu et al., 2020; Indurthi et al., 2020), SV (Zhang et al.,
2021c), TTS (Huang et al., 2022b) and domain generalization for speaker
recognition (Kang et al., 2020).

Meta-learning has the potential to improve speech processing tasks by learning
better learning algorithms that can adapt to new tasks and data more
efficiently. Meta-learning can also reduce the cost of model training and
fine-tuning, which is particularly useful for low-resource speech processing
tasks. Further investigation is required to delve into the full potential of
meta-learning in speech processing and to develop more effective meta-learning
algorithms for different speech-processing tasks.

#### 6.2.2 Models

In low-resource ASR, meta-learning is used to quickly adapt unseen target
languages by formulating ASR for different languages as different tasks and
meta-learning the initialization parameters from many pretraining languages
(Hsu et al., 2020; Singh et al., 2022). The proposed approach, MetaASR (Hsu et
al., 2020), significantly outperforms the state-of-the-art multitask
pretraining approach on all target languages with different combinations of
pretraining languages. In speaker verification, meta-learning is used to
improve the meta-learning training for SV by introducing two methods to
improve the backbone embedding network (Chen et al., 2021b). The proposed
methods can obtain consistent improvements over the existing meta-learning
training framework (Kye et al., 2020).

Meta-learning has proven to be a promising approach in various speech-related
tasks, including low-resource ASR and speaker verification. In addition to
these tasks, meta-learning has also been applied to few-shot speaker adaptive
TTS and language-agnostic TTS, demonstrating its potential to improve
performance across different speech technologies. Meta-TTS (Huang et al.,
2022b) is an example of a meta-learning model used for a few-shot speaker
adaptive TTS. It can synthesize high-speaker-similarity speech from a few
enrolment samples with fewer adaptation steps. Similarly, a language-agnostic
meta-learning approach is proposed in (Lux and Vu, 2022) for low-resource TTS.

###  6.3. Parameter-Efficient Transfer Learning

Transfer learning has played a significant role in the recent progress of
speech processing. Fine-tuning pre-trained large models, such as those trained
on LibriSpeech (Panayotov et al., 2015) or Common Voice (Ardila et al., 2019),
has been widely used for transfer learning in speech processing. However,
fine-tuning all parameters for each downstream task can be computationally
expensive. To overcome this challenge, researchers have been exploring
parameter-efficient transfer learning techniques that optimize only a fraction
of the model parameters, aiming to improve training efficiency. This article
investigates these parameter-efficient transfer learning techniques in speech
processing, evaluates their effectiveness in improving training efficiency
without sacrificing performance, and discusses the challenges and
opportunities associated with these techniques, highlighting their potential
to advance the field of speech processing.

#### 6.3.1 Adapters

In recent years, retrofitting adapter modules with a few parameters to pre-
trained models has emerged as an effective approach in speech processing. This
involves optimizing the adapter modules while keeping the pre-trained
parameters frozen for downstream tasks. Recent studies (Li et al., 2023; Liu
et al., 2021) (Li et al., 2023; Yang et al., 2021a) have shown that adapters
often outperform fine-tuning while using only a fraction of the total
parameters. Different adapter architectures are available, such as bottleneck
adapters (Houlsby et al., 2019)(Houlsby et al., 2019b), tiny attention
adapters (Zhao et al., 2022)(Zhao et al., 2022), prefix-tuning adapters (Li
and Liang, 2021)(Li and Liang, 2021), and LoRA adapters (Hu et al., 2022)(Hu
et al., 2022b), among others Next, we will review the different approaches for
parameter-efficient transfer learning. The different approaches are
illustrated in Figure 17 and Figure 18

Figure 17. Transformer architecture and Adapter, Prefix Tuning, and LoRA.
Figure 18. The architecture of 1D convolution layer-based lightweight adapter.
$k$ is the kernel size of 1D convolution. $*$ denotes depth-wise convolution.

Adapter Tuning. Adapters are a type of neural module that can be retrofitted
onto a pre-trained language model, with significantly fewer parameters than
the original model. One such type is the bottleneck or standard adapter
(Houlsby et al., 2019; Pfeiffer et al., 2020) (Houlsby et al., 2019a; Pfeiffer
et al., 2020). The adapter takes an input vector $h\in\mathbf{R}^{d}$ and
down-projects it to a lower-dimensional space with dimensionality $m$ (where
$m<d$), applies a non-linear function $g(\cdot)$, and then up-projects the
result back to the original $d$-dimensional space. Finally, the output is
obtained by adding a residual connection.

(30) $\bm{h}\leftarrow\bm{h}+g(\bm{h}\bm{W}_{\text{down}})\bm{W}_{\text{up}}$

where matrices $\bm{W}_{\text{down}}$ and $\bm{W}_{\text{up}}$ are used as
down and up projection matrices, respectively, with $\bm{W}{\text{down}}$
having dimensions $\mathbb{R}^{d\times m}$ and $\bm{W}_{\text{up}}$ having
dimensions $\mathbb{R}^{m\times d}$. Previous studies have empirically shown
that a two-layer feedforward neural network with a bottleneck is effective. In
this work, we follow the experimental settings outlined in (Pfeiffer et al.,
2020) for the adapter, which is inserted after the feedforward layer of every
transformer module, as depicted in Figure 17.

Prefix tuning. Recent studies have suggested modifying the attention module of
the Transformer model to improve its performance in natural language
processing tasks. This approach involves adding learnable vectors to the pre-
trained multi-head attention keys and values at every layer, as depicted in
Figure 17. Specifically, two sets of learnable prefix vectors, $\bm{P_{K}}$
and $\bm{P_{V}}$, are concatenated with the original key and value matrices
$\bm{K}$ and $\bm{V}$, while the query matrix $\bm{Q}$ remains unchanged. The
resulting matrices are then used for multi-head attention, where each head of
the attention mechanism is computed as follows:

(31)
$\text{head}_{i}=\text{Attn}(\bm{Q}\bm{W}_{Q}^{(i)},[\bm{P}_{K}^{(i)},\bm{K}\bm{W}_{Q}^{(i)}],[\bm{P}_{V}^{(i)},\bm{V}\bm{W}_{Q}^{(i)}])$

where Attn($\cdot$) is scaled dot-product attention given by:

(32)
$\text{Attn}(\bm{Q},\bm{K},\bm{V})=\text{softmax}(\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d_{k}}})\bm{V}$

The attention heads in each layer are modified by prefix tuning, with only the
prefix vectors $\bm{P}{K}$ and $\bm{P}{V}$ being updated during training. This
approach provides greater control over the transmission of acoustic
information between layers and effectively activates the pre-trained model’s
knowledge.

LoRA. LoRA is a novel approach proposed by Hu et al. (2021) (Hu et al., 2021),
which aims to approximate weight updates in the Transformer by injecting
trainable low-rank matrices into its layers. In this method, a pre-trained
weight matrix $W\in\mathbb{R}^{d\times k}$ is updated by a low-rank
decomposition
$\bm{W}+\Delta\bm{W}=\bm{W}+\bm{W}{\text{down}}\bm{W}{\text{up}}$, where
$\bm{W}{\text{down}}\in\mathbb{R}^{d\times r}$,
$\bm{W}{\text{up}}\in\mathbb{R}^{r\times k}$ are tunable parameters and $r$
represents the rank of the decomposition matrices, with $r<d$. Specifically,
for a given input $\bm{x}$ to the linear projection in the multi-headed
attention layer, LoRA modifies the projection output $\bm{h}$ as follows:

(33)
$\bm{h}\leftarrow\bm{h}+s\cdot\bm{x}\bm{W}_{\text{down}}\bm{W}_{\text{up}}$

In this work, LoRA is integrated into four locations of the multi-head
attention layer, as illustrated in Figure 17. Thanks to its lightweight
nature, the pre-trained model can accommodate many small modules for different
tasks, allowing for efficient task switching by replacing the modules.
Additionally, LoRA incurs no inference latency and achieves a convergence rate
that is comparable to that of training the original model, unlike fully fine-
tuned models (Hu et al., 2021).

Convolutional Adapter. CNNs have become increasingly popular in the field of
speech processing due to their ability to learn task-specific information and
combine channel-wise information within local receptive fields. To further
improve the efficiency of CNNs for speech processing tasks, Li et al. (2023)
(Li et al., 2023) proposed a lightweight adapter, called the ConvAdapter,
which uses three 1D convolutional layers, layer normalization, and a squeeze-
and-excite module (Zhang et al., 2017) (Hu et al., 2017), as shown in Figure
18. By utilizing depth-wise convolution, which requires fewer parameters and
is more computationally efficient, the authors were able to achieve better
performance while using fewer resources. In this approach, the ConvAdapter is
added to the same location as the Bottleneck Adapter (Figure 17).

Table 13. The study evaluated various parameter-efficient training methods on
pre-trained Word2Vec 2.0, including full fine-tuning, on the SURE benchmark.
The fraction of trainable parameters were represented by percentages, with the
number of KS task’s trainable parameters given. Results are reported using
weighted-f1 as the metric (w-f1) on MELD, with the best performance in bold
and the second best underlined. To avoid data imbalance, the researchers opted
for using weighted-f1 as the metric. The study cites Li et al. (2023) (Li et
al., 2023) as a reference.

Method | #Parameters | SER (acc % / w-f1) $\uparrow{}$ | SR (acc %) $\uparrow{}$ | ASR (wer) $\downarrow$ | KS (acc %) $\uparrow$
---|---|---|---|---|---
ESD | MELD | ESD | VCTK | ESD | FLEURS | LS | Speech Command
Fine Tuning | 315,703,947 | 96.53 | 42.93 | 99.00 | 92.36 | 0.2295 | 0.135 | 0.0903 | 99.08
Adapter | 25,467,915 (8.08%) | 94.07 | 41.58 | 98.87 | 96.32 | 0.2290 | 0.214 | 0.2425 | 99.19
Prefix Tuning | 1,739,787 (0.55%) | 90.00 | 44.21 | 99.73 | 98.49 | 0.2255 | 0.166 | 0.1022 | 98.86
LoRA | 3,804,171 (1.20%) | 90.00 | 47.05 | 99.00 | 97.61 | 0.2428 | 0.149 | 0.1014 | 98.28
ConvAdapter | 2,952,539 (0.94%) | 91.87 | 46.30 | 99.60 | 97.61 | 0.2456 | 0.2062 | 0.2958 | 98.99

Table 14. Results on SURE benchmark for full fine-tuning and other parameter-
efficient training methods on pre-trained Wav2Vec 2.0 for IC and PR tasks on
FS: Fluent Speech (Lugosch et al., 2019) and LS: LibriSpeech (Panayotov et
al., 2015) datasets, respectively.

Method | IC | PR | SF
---|---|---|---
#Parameters | FS | #Parameters | LS | #Parameters | SNIPS
ACC% $\uparrow$ | PER $\downarrow$ | F1 % $\uparrow$ | CER $\downarrow$
Fine-Tuning | 315707288 | 99.60 | 311304394 | 0.0577 | 311375119 | 93.89 | 0.1411
Adapter | 25471256 (8.06%) | 99.39 | 25278538 (8.01%) | 0.1571 | 25349263 (8.14%) | 92.60 | 0.1666
Prefix Tuning | 1743128 (0.55%) | 93.43 | 1550410 (0.49%) | 0.1598 | 1621135 (0.50%) | 62.32 | 0.6041
LoRA | 3807512 (1.20%) | 99.68 | 3614794 (1.16%) | 0.1053 | 3685519 (1.18%) | 90.61 | 0.2016
ConvAdapter | 3672344 (1.16%) | 95.60 | 3479626 (1.11%) | 0.1532 | 3550351 (1.14%) | 59.27 | 0.6405

Table 15. Results on the SURE benchmark for the TTS task. MCD and WER are the
metrics used to compare fine-tuning and other parameter-efficient approaches.

Method | Parameters (%) | LTS | L2ARCTIC
---|---|---|---
MCD $\downarrow$ | WER $\downarrow$ | MCD $\downarrow$ | WER $\downarrow$
Fine-tuning | 35802977 | 6.2038 | 0.2655 | 6.71469 | 0.2141
Adapter | 659200 | 6.1634 | 0.3143 | 6.544 | 0.2504
Prefix | 153600 | 6.2523 | 0.3334 | 7.4264 | 0.3244
LoRA | 81920 | 6.8319 | 0.3786 | 7.0698 | 0.3291
Convadapter | 108800 | 6.9202 | 0.3365 | 6.9712 | 0.3227

Table 13, Table 14, and Table 15 present the results of various speech
processing tasks in the SURE benchmark. The findings demonstrate that the
adapter-based methods perform comparably well in fine-tuning. However, there
is no significant advantage of any particular adapter type over others for
these benchmark tasks and datasets.

#### 6.3.2 Knowledge Distillation (KD)

Knowledge distillation involves training a smaller model to mimic the behavior
of a larger and more complex model. This can be done by training the smaller
model to predict the outputs of the larger model or, by using, the larger
model’s hidden representations as input to the smaller model. Knowledge
distillation is effective in reducing the computational cost of training and
inference.

Cho et al. (2020) conducted knowledge distillation (KD) by directly applying
it to the downstream task. One way to improve this approach is to use KD as
pre-training for various downstream tasks, thus allowing for knowledge reuse.
A noteworthy result achieved by Denisov and Vu (2020) was using KD in
pretraining. However, they achieved this by initializing an utterance encoder
with a trained ASR model’s backbone, followed by a trained NLU backbone.
Knowledge distillation can be applied directly into a wav2vec 2.0 encoder
without ASR training and a trained NLU module to enhance this method. Kim et
al. (2021a) implemented a more complex architecture, utilizing KD in both the
pretraining and fine-tuning stages.

#### 6.3.3 Model Compression

Researchers have also explored various architectural modifications to existing
models to make them more parameter-efficient. One such approach is _pruning_
(Frantar and Alistarh, 2023; Wang et al., 2019c), where motivated by lottery-
ticket hypothesis (LTH) (Frankle and Carbin, 2018), the task-irrelevant
parameters are masked based on some threshold defined by importance score,
such as some parameter norm. Another form of compression could be _low-rank
factorization_ (Hsu et al., 2022a), where the parameter matrices are
factorized into lower-rank matrices with much fewer parameters. Finally,
_quantization_ is a popular approach to reduce the model size and improve
energy efficiency with a minimal performance penalty. It involves transforming
32-bit floating point model weights into integers with fewer bit-counts (Yao
et al., 2020)—8-bit, 4-bit, 2-bit, and even 1-bit—through scaling and
shifting. At the same time, the quantization of the activation is also handled
based on the input.

Lai et al. (2021) iteratively prune and subsequently fine-tune wav2vec2.0 on
downstream tasks to obtained improved results over fine-tuned wav2vec2.0.
Winata et al. (2020) employ low-rank transformers to excise the model size by
half and increase the inference speed by 1.35 times. Peng et al. (2021) employ
KD and quantization to make wav2vec2.0 twice as fast, twice as energy
efficient, and 4.8 times smaller at the cost of a 7% increase in WER. Without
the KD step, the model is 3.6 times smaller with mere 0.1% WER degradation.

##  7\. Conclusion and Future Research Directions

The rapid advancements in deep learning techniques have revolutionized speech
processing tasks, enabling significant progress in speech recognition, speaker
recognition, and speech synthesis. This paper provides a comprehensive review
of the latest developments in deep learning techniques for speech-processing
tasks. We begin by examining the early developments in speech processing,
including representation learning and HMM-based modeling, before presenting a
concise summary of fundamental deep learning techniques and their applications
in speech processing. Furthermore, we discuss key speech-processing tasks,
highlight the datasets used in these tasks, and present the latest and most
relevant research works utilizing deep learning techniques.

We envisage several lines of development in speech processing:

  1. (1)

_Large Speech Models:_ In addition to the advancements made with wav2vec2.0,
further progress in the field of ASR and TTS models involves the development
of larger and more comprehensive models, along with the utilization of larger
datasets. By leveraging these resources, it becomes possible to create TTS
models that exhibit enhanced naturalness and human-like prosody. One promising
approach to achieve this is through the application of adversarial training,
where a discriminator is employed to distinguish between machine-generated
speech and reference speech. This adversarial framework facilitates the
generation of TTS models that closely resemble human speech, providing a
significant step forward in achieving more realistic and high-quality
synthesized speech. By exploring these avenues, researchers aim to push the
boundaries of speech synthesis technology, ultimately enhancing the overall
performance and realism of TTS systems.

  2. (2)

_Multilingual Models:_ Self-supervised learning has emerged as a
transformative approach in the field of speech recognition, particularly for
low-resource languages characterized by scarce or unavailable labeled
datasets. The recent development of the XLS-R model, a state-of-the-art self-
supervised speech recognition model, represents a significant milestone in
this domain. With a remarkable scale of over 2 billion parameters, the XLS-R
model has been trained on a diverse dataset spanning 128 languages, surpassing
its predecessor in terms of language coverage. The notable advantage of
scaling up larger multilingual models like XLS-R lies in the substantial
performance improvements they offer. As a result, these models are poised to
outperform single-language models and hold immense promise for the future of
speech recognition. By harnessing the power of self-supervised learning and
leveraging multilingual datasets, the XLS-R model showcases the potential for
addressing the challenges posed by low-resource languages and advancing the
field of speech recognition to new heights.

  3. (3)

_Multimodal Speech Models:_ Traditional speech and text models have typically
operated within a single modality, focusing solely on either speech or text
inputs and outputs. However, as the scale of generative models continues to
grow exponentially, the integration of multiple modalities becomes a natural
progression. This trend is evident in the latest developments, such as the
unveiling of groundbreaking language models like GPT-4 (OpenAI, 2023) and
Kosmos-I (Huang et al., 2023), which demonstrate the ability to process both
images and text jointly. These pioneering multimodal models pave the way for
the emergence of large-scale architectures that can seamlessly handle speech
and other modalities in a unified manner. The convergence of multiple
modalities within a single model opens up new avenues for comprehensive
understanding and generation of multimodal content, and it is highly
anticipated that we will witness the rapid development of large multimodal
models tailored for speech and beyond in the near future.

  4. (4)

_In-Context Learning:_ Utilizing mixed-modality models opens up possibilities
for the development of in-context learning approaches for a wide range of
speech-related tasks. This paradigm allows the tasks to be explicitly defined
within the input, along with accompanying examples. Remarkable progress has
already been demonstrated in large language models (LLMs), including notable
works such as InstructGPT (Ouyang et al., 2022), FLAN-T5 (Chung et al., 2022),
and LLaMA (Touvron et al., 2023). These models showcase the efficacy of in-
context learning, where the integration of context-driven information empowers
the models to excel in various speech tasks. By leveraging mixed-modality
models and incorporating contextual cues, researchers are advancing the
boundaries of speech processing capabilities, paving the way for more
versatile and context-aware speech systems.

  5. (5)

_Controllable Speech Generation:_ An intriguing application stemming from the
aforementioned concept is controllable text-to-speech (TTS), which allows for
fine-grained control over various attributes of the synthesized speech.
Attributes such as tone, accent, age, gender, and more can be precisely
controlled through in-context text guidance. This controllability in TTS opens
up exciting possibilities for personalization and customization, enabling
users to tailor the synthesized speech to their specific requirements. By
leveraging advanced models and techniques, researchers are making significant
strides in developing controllable TTS systems that provide users with a
powerful and flexible speech synthesis experience.

  6. (6)

_Parameter-efficient Learning:_ With the increasing scale of LLMs and speech
models, it becomes imperative to adapt these models with minimal parameter
updates. This necessitates the development of specialized adapters that can
efficiently update these emerging mixed-modality large models. Additionally,
model compression techniques have proven to be practical solutions in
addressing the challenges posed by these large models. Recent research (Lai et
al., 2021; Winata et al., 2020; Peng et al., 2021) has demonstrated the
effectiveness of model compression, highlighting the sparsity that exists
within these models, particularly for specific tasks. By employing model
compression techniques, researchers can reduce the computational requirements
and memory footprint of these models while preserving their performance,
making them more practical and accessible for real-world applications.

  7. (7)

_Explainability:_ Explainability remains elusive to these large networks as
they grow. Researchers are steadfast in explaining these networks’ functioning
and learning dynamics. Recently, much work has been done to learn the fine-
tuning and in-context learning dynamics of these large models for text under
the neural-tangent-kernel (NTK) asymptotic framework (Malladi et al., 2022).
Such exploration is yet to be done in the speech domain. More yet,
explainability could be built-in as inductive bias in architecture. To this
end, brain-inspired architectures (Millet et al., 2022) are being developed,
which may shed more light on this aspect of large models.

  8. (8)

_Neuroscience-inspired Architectures:_ In recent years, there has been
significant research exploring the parallels between speech-processing
architectures and the intricate workings of the human brain (Millet et al.,
2022). These studies have unveiled compelling evidence of a strong correlation
between the layers of speech models and the functional hierarchy observed in
the human brain. This intriguing finding has served as a catalyst for the
development of neuroscience-inspired speech models that demonstrate comparable
performance to state-of-the-art (SOTA) models (Millet et al., 2022). By
drawing inspiration from the underlying principles of neural processing in the
human brain, these innovative speech models aim to enhance our understanding
of speech perception and production while pushing the boundaries of
performance in the field of speech processing.

  9. (9)

_Text-to-Audio Models for Text-to-Speech:_ Lately, transformer and diffusion-
based text-to-audio (TTA) model development is turning into an exciting area
of research. Until recently, most of these models (Liu et al., 2023a; Kreuk et
al., 2022; Yang et al., 2022b; Ghosal et al., 2023; Wang et al., 2023b)
overlooked speech in favour of general audio. In the future, however, the
models will likely strive to be equally performant in both audio and speech.
To that end, current TTS methods will likely be an integral part of those
models. Recently, Suno-AI (2023) have aimed at striking a good balance between
general audio and speech, although their implementation is not public, nor
have they provided any detailed paper.

## Acknowledgement

This research is supported by the Ministry of Education, Singapore, under its
AcRF Tier-2 grant (Project no. T2MOE2008, and Grantor reference no.
MOE-T2EP20220-0017), and A*STAR under its RIE 2020 AME programmatic grant
(project reference no. RGAST2003. Any opinions, findings and conclusions or
recommendations expressed in this material are those of the author(s) and do
not reflect the views of the Ministry of Education, Singapore.

## References

  * (1)
  * ass (2022) 2022\.  Conformer-1.  _AssemblyAI_ (2022).  https://www.assemblyai.com/blog/conformer-1/
  * nvi (2022) 2022\.  Speech Recognition With Conformer.  _Nvidia_ (2022).  https://docs.nvidia.com/tao/tao-toolkit/text/asr/speech_recognition_with_conformer.html
  * Abdel-Hamid et al. (2014a) Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. 2014a.  Convolutional neural networks for speech recognition.  _IEEE/ACM Transactions on audio, speech, and language processing_ 22, 10 (2014), 1533–1545.
  * Abdel-Hamid et al. (2014b) Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. 2014b.  Convolutional Neural Networks for Speech Recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 22, 10 (2014), 1533–1545.  https://doi.org/10.1109/TASLP.2014.2339736
  * Abdel-Hamid et al. (2012) Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, and Gerald Penn. 2012.  Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition. In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 4277–4280.  https://doi.org/10.1109/ICASSP.2012.6288864
  * Abdeljaber et al. (2017) Osama Abdeljaber, Onur Avci, Serkan Kiranyaz, Moncef Gabbouj, and Daniel J Inman. 2017.  Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks.  _Journal of Sound and Vibration_ 388 (2017), 154–170.
  * Abdul and Al-Talabani (2022) Zrar Kh. Abdul and Abdulbasit K. Al-Talabani. 2022\.  Mel Frequency Cepstral Coefficient and its Applications: A Review.  _IEEE Access_ 10 (2022), 122136–122158.  https://doi.org/10.1109/ACCESS.2022.3223444
  * Abdulatif et al. (2022) Sherif Abdulatif, Ruizhe Cao, and Bin Yang. 2022.  CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement.  _arXiv preprint arXiv:2209.11112_ (2022).
  * Achanta et al. (2021) Sivanand Achanta, Albert Antony, Ladan Golipour, Jiangchuan Li, Tuomo Raitio, Ramya Rasipuram, Francesco Rossi, Jennifer Shi, Jaimin Upadhyay, David Winarsky, et al. 2021\.  On-device neural speech synthesis. In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 1155–1161.
  * Afouras et al. (2018) Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018\.  The conversation: Deep audio-visual speech enhancement.  _arXiv preprint arXiv:1804.04121_ (2018).
  * Aggarwal et al. (2020) Vatsal Aggarwal, Marius Cotescu, Nishant Prateek, Jaime Lorenzo-Trueba, and Roberto Barra-Chicote. 2020\.  Using Vaes and Normalizing Flows for One-Shot Text-To-Speech Synthesis of Expressive Speech. In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 6179–6183.  https://doi.org/10.1109/ICASSP40776.2020.9053678
  * Alsabhan (2023) Waleed Alsabhan. 2023\.  Human–Computer Interaction with a Real-Time Speech Emotion Recognition with Ensembling Techniques 1D Convolution Neural Network and Attention.  _Sensors_ 23, 3 (2023), 1386.
  * Amodei et al. (2016) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016\.  Deep speech 2: End-to-end speech recognition in english and mandarin. In _International conference on machine learning_. PMLR, 173–182.
  * Anguera et al. (2012) Xavier Anguera, Simon Bozonnet, Nicholas Evans, Corinne Fredouille, Gerald Friedland, and Oriol Vinyals. 2012.  Speaker diarization: A review of recent research.  _IEEE Transactions on audio, speech, and language processing_ 20, 2 (2012), 356–370.
  * Ansari et al. (2020) Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, et al. 2020\.  Findings of the IWSLT 2020 evaluation campaign. In _Proceedings of the 17th International Conference on Spoken Language Translation_. 1–34.
  * Ao et al. (2021) Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. 2021\.  Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing.  _arXiv preprint arXiv:2110.07205_ (2021).
  * Ardila et al. (2019) Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019\.  Common voice: A massively-multilingual speech corpus.  _arXiv preprint arXiv:1912.06670_ (2019).
  * Arık et al. (2017) Sercan Ö Arık, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. 2017\.  Deep voice: Real-time neural text-to-speech. In _International conference on machine learning_. PMLR, 195–204.
  * Audhkhasi et al. (2019) Kartik Audhkhasi, George Saon, Zoltán Tüske, Brian Kingsbury, and Michael Picheny. 2019\.  Forget a Bit to Learn Better: Soft Forgetting for CTC-Based Automatic Speech Recognition.. In _Interspeech_. 2618–2622.
  * Babu et al. (2021) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021\.  XLS-R: Self-supervised cross-lingual speech representation learning at scale.  _arXiv preprint arXiv:2111.09296_ (2021).
  * Badlani et al. (2022a) Rohan Badlani, Adrian Łańcucki, Kevin J Shih, Rafael Valle, Wei Ping, and Bryan Catanzaro. 2022a.  One TTS alignment to rule them all. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6092–6096.
  * Badlani et al. (2022b) Rohan Badlani, Adrian Łańcucki, Kevin J. Shih, Rafael Valle, Wei Ping, and Bryan Catanzaro. 2022b.  One TTS Alignment to Rule Them All. In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 6092–6096.  https://doi.org/10.1109/ICASSP43922.2022.9747707
  * Baevski et al. (2019a) Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. 2019a.  Effectiveness of self-supervised pre-training for speech recognition.  _arXiv preprint arXiv:1911.03912_ (2019).
  * Baevski et al. (2022) Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022\.  Data2vec: A general framework for self-supervised learning in speech, vision and language. In _International Conference on Machine Learning_. PMLR, 1298–1312.
  * Baevski et al. (2019b) Alexei Baevski, Steffen Schneider, and Michael Auli. 2019b.  vq-wav2vec: Self-supervised learning of discrete speech representations.  _arXiv preprint arXiv:1910.05453_ (2019).
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Bahar et al. (2019) Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019\.  A comparative study on end-to-end speech to text translation. In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 792–799.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Bai et al. (2022) He Bai, Renjie Zheng, Junkun Chen, Mingbo Ma, Xintong Li, and Liang Huang. 2022\.  A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing. In _International Conference on Machine Learning_. PMLR, 1399–1411.
  * Bai et al. (2018) Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018\.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  _arXiv preprint arXiv:1803.01271_ (2018).
  * Bai and Zhang (2021) Zhongxin Bai and Xiao-Lei Zhang. 2021.  Speaker recognition based on deep learning: An overview.  _Neural Networks_ 140 (2021), 65–99.
  * Bak et al. (2022) Taejun Bak, Junmo Lee, Hanbin Bae, Jinhyeok Yang, Jae-Sung Bae, and Young-Sun Joo. 2022\.  Avocodo: Generative adversarial network for artifact-free vocoder.  _arXiv preprint arXiv:2206.13404_ (2022).
  * Barker et al. (2018) Jon Barker, Shinji Watanabe, Emmanuel Vincent, and Jan Trmal. 2018.  The fifth’CHiME’speech separation and recognition challenge: dataset, task and baselines.  _arXiv preprint arXiv:1803.10609_ (2018).
  * Baskar et al. (2019) Murali Karthick Baskar, Shinji Watanabe, Ramon Astudillo, Takaaki Hori, Lukáš Burget, and Jan Černockỳ. 2019\.  Semi-supervised sequence-to-sequence ASR using unpaired speech and text.  _arXiv preprint arXiv:1905.01152_ (2019).
  * Battenberg et al. (2020) Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. 2020.  Location-relative attention mechanisms for robust long-form speech synthesis. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6194–6198.
  * Beerends et al. (2013) John G Beerends, Christian Schmidmer, Jens Berger, Matthias Obermann, Raphael Ullmann, Joachim Pomy, and Michael Keyhl. 2013.  Perceptual objective listening quality assessment (polqa), the third generation itu-t standard for end-to-end speech quality measurement part i—temporal alignment.  _Journal of the Audio Engineering Society_ 61, 6 (2013), 366–384.
  * Berg et al. (2021) Axel Berg, Mark O’Connor, and Miguel Tairum Cruz. 2021.  Keyword transformer: A self-attention model for keyword spotting.  _arXiv preprint arXiv:2104.00769_ (2021).
  * Bińkowski et al. (2019a) Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. 2019a.  High fidelity speech synthesis with adversarial networks.  _arXiv preprint arXiv:1909.11646_ (2019).
  * Bińkowski et al. (2019b) Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. 2019b.  High Fidelity Speech Synthesis with Adversarial Networks. In _International Conference on Learning Representations_.
  * Birnbaum et al. (2019) Sawyer Birnbaum, Volodymyr Kuleshov, Zayd Enam, Pang Wei W Koh, and Stefano Ermon. 2019.  Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Boll (1979) Steven Boll. 1979\.  Suppression of acoustic noise in speech using spectral subtraction.  _IEEE Transactions on acoustics, speech, and signal processing_ 27, 2 (1979), 113–120.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Bourlard and Morgan (1994) Herve A Bourlard and Nelson Morgan. 1994.  _Connectionist speech recognition: a hybrid approach_. Vol. 247.  Springer Science & Business Media.
  * Bousquet and Rouvier (2019) Pierre-Michel Bousquet and Mickael Rouvier. 2019.  On robustness of unsupervised domain adaptation for speaker recognition. In _Interspeech_.
  * Bredin and Laurent (2021) Hervé Bredin and Antoine Laurent. 2021.  End-to-end speaker segmentation for overlap-aware resegmentation.  _arXiv preprint arXiv:2104.04045_ (2021).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Bullock et al. (2020) Latané Bullock, Hervé Bredin, and Leibny Paola Garcia-Perera. 2020.  Overlap-aware diarization: Resegmentation using neural end-to-end overlapped speech detection. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7114–7118.
  * Bunk et al. (2020) Tanja Bunk, Daksh Varshneya, Vladimir Vlasov, and Alan Nichol. 2020.  Diet: Lightweight language understanding for dialogue systems.  _arXiv preprint arXiv:2004.09936_ (2020).
  * Burchi and Timofte (2023) Maxime Burchi and Radu Timofte. 2023.  Audio-Visual Efficient Conformer for Robust Speech Recognition. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2258–2267.
  * Butryna et al. (2020) Alena Butryna, Shan-Hui Cathy Chu, Isin Demirsahin, Alexander Gutkin, Linne Ha, Fei He, Martin Jansche, Cibu Johny, Anna Katanova, Oddur Kjartansson, et al. 2020\.  Google crowdsourced speech corpora and related open-source resources for low-resource languages and dialects: an overview.  _arXiv preprint arXiv:2010.06778_ (2020).
  * C S et al. (2021) Anoop C S, Prathosh A P, and A G Ramakrishnan. 2021.  Unsupervised Domain Adaptation Schemes for Building ASR in Low-Resource Languages. In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. 342–349.  https://doi.org/10.1109/ASRU51503.2021.9688269
  * Campbell et al. (2003) William Campbell, Joseph Campbell, Douglas Reynolds, Douglas Jones, and Timothy Leek. 2003\.  Phonetic speaker recognition with support vector machines.  _Advances in neural information processing systems_ 16 (2003).
  * Carletta et al. (2006) Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2006\.  The AMI meeting corpus: A pre-announcement. In _Machine Learning for Multimodal Interaction: Second International Workshop, MLMI 2005, Edinburgh, UK, July 11-13, 2005, Revised Selected Papers 2_. Springer, 28–39.
  * Castiglioni (2005) Paolo Castiglioni. 2005\.  Levinson-durbin algorithm.  _Encyclopedia of Biostatistics_ 4 (2005).
  * Catellier and Voran (2020) Andrew A Catellier and Stephen D Voran. 2020.  Wawenets: A no-reference convolutional waveform-based approach to estimating narrowband and wideband speech quality. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 331–335.
  * Cattoni et al. (2021) Roldano Cattoni, Mattia Antonino Di Gangi, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021.  MuST-C: A multilingual corpus for end-to-end speech translation.  _Computer Speech & Language_ 66 (2021), 101155.
  * Cauchi et al. (2019) Benjamin Cauchi, Kai Siedenburg, Joao F Santos, Tiago H Falk, Simon Doclo, and Stefan Goetze. 2019\.  Non-intrusive speech quality prediction using modulation energies and lstm-network.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 27, 7 (2019), 1151–1163.
  * Chan et al. (2021) William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi. 2021\.  Speechstew: Simply mix all available speech recognition data to train one large neural network.  _arXiv preprint arXiv:2104.02133_ (2021).
  * Chang et al. (2022) Kai-Wei Chang et al. 2022\.  An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks.  _arXiv preprint arXiv:2203.16773_ (2022).
  * Chaudhari et al. (2021) Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath. 2021.  An attentive survey of attention models.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 12, 5 (2021), 1–32.
  * Chen et al. (2021a) Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. 2021a.  Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio.  _arXiv preprint arXiv:2106.06909_ (2021).
  * Chen et al. (2020a) Junkun Chen, Mingbo Ma, Renjie Zheng, and Liang Huang. 2020a.  Mam: Masked acoustic modeling for end-to-end speech-to-text translation.  _arXiv preprint arXiv:2010.11445_ (2020).
  * Chen et al. (2021c) Junkun Chen, Mingbo Ma, Renjie Zheng, and Liang Huang. 2021c.  SpecRec: An Alternative Solution for Improving End-to-End Speech-to-Text Translation via Spectrogram Reconstruction.. In _Interspeech_. 2232–2236.
  * Chen et al. (2021d) Jiawei Chen, Xu Tan, Yichong Leng, Jin Xu, Guihua Wen, Tao Qin, and Tie-Yan Liu. 2021d.  Speech-t: Transducer for text to speech and beyond.  _Advances in Neural Information Processing Systems_ 34 (2021), 6621–6633.
  * Chen et al. (2019a) Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2019a.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 7832–7841.
  * Chen et al. (2021e) Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, and Tie-Yan Liu. 2021e.  Adaspeech: Adaptive text to speech for custom voice.  _arXiv preprint arXiv:2103.00993_ (2021).
  * Chen et al. (2020c) Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. 2020c.  WaveGrad: Estimating gradients for waveform generation.  _arXiv preprint arXiv:2009.00713_ (2020).
  * Chen et al. (2020d) Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. 2020d.  WaveGrad: Estimating Gradients for Waveform Generation. In _International Conference on Learning Representations_.
  * Chen et al. (2021g) Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. 2021g.  Wavegrad 2: Iterative refinement for text-to-speech synthesis.  _arXiv preprint arXiv:2106.09660_ (2021).
  * Chen et al. (2019b) Qian Chen, Zhu Zhuo, and Wen Wang. 2019b.  Bert for joint intent classification and slot filling.  _arXiv preprint arXiv:1902.10909_ (2019).
  * Chen et al. (2022b) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022b.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ 16, 6 (2022), 1505–1518.
  * Chen et al. (2022c) Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, et al. 2022c.  Unispeech-sat: Universal speech representation learning with speaker aware pre-training. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6152–6156.
  * Chen et al. (2021b) Yafeng Chen, Wu Guo, and Bin Gu. 2021b.  Improved meta-learning training for speaker verification.  _arXiv preprint arXiv:2103.15421_ (2021).
  * Chen et al. (2022a) Zehua Chen, Xu Tan, Ke Wang, Shifeng Pan, Danilo Mandic, Lei He, and Sheng Zhao. 2022a.  Infergrad: Improving Diffusion Models for Vocoder by Considering Inference in Training. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 8432–8436.
  * Chen et al. (2020b) Zhengyang Chen, Shuai Wang, and Yanmin Qian. 2020b.  Adversarial Domain Adaptation for Speaker Verification Using Partially Shared Network.. In _Interspeech_. 3017–3021.
  * Chen et al. (2021f) Zhengyang Chen, Shuai Wang, and Yanmin Qian. 2021f.  Self-supervised learning based domain adaptation for robust speaker verification. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5834–5838.
  * Chen et al. (2015) Zhuo Chen, Shinji Watanabe, Hakan Erdogan, and John R Hershey. 2015.  Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks. In _Sixteenth Annual Conference of the International Speech Communication Association_.
  * Chiu et al. (2022) Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. 2022.  Self-supervised learning with random-projection quantizer for speech recognition. In _International Conference on Machine Learning_. PMLR, 3915–3924.
  * Chiu and Raffel (2017) Chung-Cheng Chiu and Colin Raffel. 2017.  Monotonic chunkwise attention.  _arXiv preprint arXiv:1712.05382_ (2017).
  * Cho et al. (2015) Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. 2015\.  Describing multimedia content using attention-based encoder-decoder networks.  _IEEE Transactions on Multimedia_ 17, 11 (2015), 1875–1886.
  * Cho et al. (2020) Won Ik Cho, Donghyun Kwak, J. Yoon, and Nam Soo Kim. 2020\.  Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillation. In _Interspeech_.
  * Choi et al. (2021) Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. 2021\.  Neural analysis and synthesis: Reconstructing speech from self-supervised representations.  _Advances in Neural Information Processing Systems_ 34 (2021), 16251–16265.
  * Choi et al. (2022) Hyeong-Seok Choi, Jinhyeok Yang, Juheon Lee, and Hyeongju Kim. 2022\.  NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis.  _arXiv preprint arXiv:2211.09407_ (2022).
  * Chorowski et al. (2019) Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron Van Den Oord. 2019.  Unsupervised speech representation learning using wavenet autoencoders.  _IEEE/ACM transactions on audio, speech, and language processing_ 27, 12 (2019), 2041–2053.
  * Chorowski et al. (2015) Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015\.  Attention-based models for speech recognition.  _Advances in neural information processing systems_ 28 (2015).
  * Chou et al. (2019) Ju-chieh Chou, Cheng-chieh Yeh, and Hung-yi Lee. 2019.  One-shot voice conversion by separating speaker and content representations with instance normalization.  _arXiv preprint arXiv:1904.05742_ (2019).
  * Chowdhury et al. (2022) Anurag Chowdhury, Austin Cozzo, and Arun Ross. 2022.  Domain Adaptation for Speaker Recognition in Singing and Spoken Voice. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7192–7196.
  * Chung et al. (2020c) Hoon Chung, Hyeong-Bae Jeon, and Jeon Gue Park. 2020c.  Semi-supervised training for sequence-to-sequence speech recognition using reinforcement learning. In _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–6.
  * Chung et al. (2020d) Hoon Chung, Hyeong-Bae Jeon, and Jeon Gue Park. 2020d.  Semi-supervised Training for Sequence-to-Sequence Speech Recognition Using Reinforcement Learning. In _2020 International Joint Conference on Neural Networks (IJCNN)_. 1–6.  https://doi.org/10.1109/IJCNN48605.2020.9207023
  * Chung et al. (2022) Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022\.  Scaling Instruction-Finetuned Language Models.  _ArXiv_ abs/2210.11416 (2022).
  * Chung et al. (2020b) Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and Icksang Han. 2020b.  In defence of metric learning for speaker recognition.  _arXiv preprint arXiv:2003.11982_ (2020).
  * Chung et al. (2018) J. S. Chung, A. Nagrani, and A. Zisserman. 2018.  VoxCeleb2: Deep Speaker Recognition. In _INTERSPEECH_.
  * Chung and Zisserman (2017) Joon Son Chung and Andrew Zisserman. 2017.  Lip reading in the wild. In _Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13_. Springer, 87–103.
  * Chung et al. (2020a) Soo-Whan Chung, Soyeon Choe, Joon Son Chung, and Hong-Goo Kang. 2020a.  Facefilter: Audio-visual speech separation using still images.  _arXiv preprint arXiv:2005.07074_ (2020).
  * Chung et al. (2019) Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019\.  An unsupervised autoregressive model for speech representation learning.  _arXiv preprint arXiv:1904.03240_ (2019).
  * Chung et al. (2021) Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021.  W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 244–250.
  * Cooke et al. (2006) Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. 2006\.  An audio-visual corpus for speech perception and automatic speech recognition.  _The Journal of the Acoustical Society of America_ 120, 5 (2006), 2421–2424.
  * Coria et al. (2021) Juan M Coria, Hervé Bredin, Sahar Ghannay, and Sophie Rosset. 2021.  Overlap-aware low-latency online speaker diarization based on end-to-end local segmentation. In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 1139–1146.
  * Coto-Jiménez (2019) Marvin Coto-Jiménez. 2019\.  Improving post-filtering of artificial speech using pre-trained LSTM neural networks.  _Biomimetics_ 4, 2 (2019), 39.
  * Coucke et al. (2019) Alice Coucke, Mohammed Chlieh, Thibault Gisselbrecht, David Leroy, Mathieu Poumeyrol, and Thibaut Lavril. 2019.  Efficient keyword spotting using dilated convolutions and gating. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6351–6355.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018\.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _arXiv preprint arXiv:1805.10190_ (2018).
  * Dauphin et al. (2017) Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017.  Language modeling with gated convolutional networks. In _International conference on machine learning_. PMLR, 933–941.
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019\.  Arcface: Additive angular margin loss for deep face recognition. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4690–4699.
  * Deng et al. (2021) Keqi Deng, Songjun Cao, Yike Zhang, and Long Ma. 2021\.  Improving Hybrid CTC/Attention End-to-End Speech Recognition with Pretrained Acoustic and Language Models. In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. 76–82.  https://doi.org/10.1109/ASRU51503.2021.9688009
  * Deng et al. (2022a) Keqi Deng, Songjun Cao, Yike Zhang, Long Ma, Gaofeng Cheng, Ji Xu, and Pengyuan Zhang. 2022a.  Improving CTC-Based Speech Recognition Via Knowledge Transferring from Pre-Trained Language Models. In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 8517–8521.  https://doi.org/10.1109/ICASSP43922.2022.9747887
  * Deng et al. (2022b) Keqi Deng, Songjun Cao, Yike Zhang, Long Ma, Gaofeng Cheng, Ji Xu, and Pengyuan Zhang. 2022b.  Improving CTC-based speech recognition via knowledge transferring from pre-trained language models. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 8517–8521.
  * Denisov and Vu (2020) Pavel Denisov and Ngoc Thang Vu. 2020.  Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning. In _Interspeech_.
  * Desplanques et al. (2020) Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. 2020\.  Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.  _arXiv preprint arXiv:2005.07143_ (2020).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Di Gangi et al. (2019) Mattia A Di Gangi, Matteo Negri, and Marco Turchi. 2019\.  Adapting transformer to end-to-end spoken language translation.  In _Proceedings of INTERSPEECH 2019_. International Speech Communication Association (ISCA), 1133–1137.
  * Diez et al. (2020) Mireia Diez, Lukáš Burget, Federico Landini, Shuai Wang, and Honza Černockỳ. 2020.  Optimizing Bayesian HMM based x-vector clustering for the second DIHARD speech diarization challenge. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6519–6523.
  * Dingliwal et al. (2021) Saket Dingliwal, Ashish Shenoy, Sravan Bodapati, Ankur Gandhe, Ravi Teja Gadde, and Katrin Kirchhoff. 2021\.  Prompt-tuning in ASR systems for efficient domain-adaptation.  _arXiv preprint arXiv:2110.06502_ (2021).
  * Donahue et al. (2018) Chris Donahue, Julian McAuley, and Miller Puckette. 2018\.  Adversarial audio synthesis.  _arXiv preprint arXiv:1802.04208_ (2018).
  * Donahue et al. (2020a) Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. 2020a.  End-to-end Adversarial Text-to-Speech. In _International Conference on Learning Representations_.
  * Donahue et al. (2020b) Jeff Donahue, Sander Dieleman, Mikołaj Bińkowski, Erich Elsen, and Karen Simonyan. 2020b.  End-to-end adversarial text-to-speech.  _arXiv preprint arXiv:2006.03575_ (2020).
  * Dong et al. (2018) Linhao Dong, Shuang Xu, and Bo Xu. 2018.  Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 5884–5888.  https://doi.org/10.1109/ICASSP.2018.8462506
  * Dong et al. (2020) Peiyan Dong, Siyue Wang, Wei Niu, Chengming Zhang, Sheng Lin, Zhengang Li, Yifan Gong, Bin Ren, Xue Lin, and Dingwen Tao. 2020.  Rtmobile: Beyond real-time mobile acceleration of rnns for speech recognition. In _2020 57th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 1–6.
  * Dong and Williamson (2020a) Xuan Dong and Donald S Williamson. 2020a.  An attention enhanced multi-task model for objective speech assessment in real-world environments. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 911–915.
  * Dong and Williamson (2020b) Xuan Dong and Donald S Williamson. 2020b.  A pyramid recurrent network for predicting crowdsourced speech-quality ratings of real-world signals.  _arXiv preprint arXiv:2007.15797_ (2020).
  * Dovrat et al. (2021) Shaked Dovrat, Eliya Nachmani, and Lior Wolf. 2021\.  Many-speakers single channel speech separation with optimal permutation training.  _arXiv preprint arXiv:2104.08955_ (2021).
  * Drexler and Glass (2019) Jennifer Drexler and James Glass. 2019.  Explicit alignment of text and speech encodings for attention-based end-to-end speech recognition. In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 913–919.
  * Du and Yu (2022) Chenpeng Du and Kai Yu. 2022.  Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 30 (2022), 190–201.  https://doi.org/10.1109/TASLP.2021.3133205
  * Du et al. (2014) Jun Du, Qing Wang, Tian Gao, Yong Xu, Li-Rong Dai, and Chin-Hui Lee. 2014\.  Robust speech recognition with speech enhanced deep neural networks. In _Fifteenth annual conference of the international speech communication association_.
  * Duarte et al. (2021) Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021.  How2sign: a large-scale multimodal dataset for continuous american sign language. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2735–2744.
  * Edu et al. (2020) Jide S Edu, Jose M Such, and Guillermo Suarez-Tangil. 2020\.  Smart home personal assistants: a security and privacy review.  _ACM Computing Surveys (CSUR)_ 53, 6 (2020), 1–36.
  * Elias et al. (2021) Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Ron J Weiss, and Yonghui Wu. 2021.  Parallel tacotron: Non-autoregressive and controllable tts. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5709–5713.
  * Elneima and Bińkowski (2022) Ashraf Elneima and Mikołaj Bińkowski. 2022.  Adversarial Text-to-Speech for low-resource languages. In _Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)_. 76–84.
  * Ephraim (1992) Yariv Ephraim. 1992\.  A Bayesian estimation approach for speech enhancement using hidden Markov models.  _IEEE Transactions on Signal Processing_ 40, 4 (1992), 725–735.
  * Ephrat et al. (2017) Ariel Ephrat, Tavi Halperin, and Shmuel Peleg. 2017\.  Improved speech reconstruction from silent video. In _Proceedings of the IEEE International Conference on Computer Vision Workshops_. 455–462.
  * Ephrat et al. (2018) Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. 2018.  Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.  _arXiv preprint arXiv:1804.03619_ (2018).
  * Ephrat and Peleg (2017) Ariel Ephrat and Shmuel Peleg. 2017.  Vid2speech: speech reconstruction from silent video. In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5095–5099.
  * Ericsson et al. (2022) Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. 2022.  Self-supervised representation learning: Introduction, advances, and challenges.  _IEEE Signal Processing Magazine_ 39, 3 (2022), 42–62.
  * Eskimez et al. (2020) Sefik Emre Eskimez, Ross K Maddox, Chenliang Xu, and Zhiyao Duan. 2020.  End-to-end generation of talking faces from noisy speech. In _ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 1948–1952.
  * Eskimez et al. (2021) Sefik Emre Eskimez, You Zhang, and Zhiyao Duan. 2021\.  Speech driven talking face generation from a single image and an emotion condition.  _IEEE Transactions on Multimedia_ 24 (2021), 3480–3490.
  * Fan et al. (2020) Yue Fan, JW Kang, LT Li, KC Li, HL Chen, ST Cheng, PY Zhang, ZY Zhou, YQ Cai, and Dong Wang. 2020\.  Cn-celeb: a challenging chinese speaker recognition dataset. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7604–7608.
  * Fan et al. (2014) Yuchen Fan, Yao Qian, Feng-Long Xie, and Frank K Soong. 2014\.  TTS synthesis with bidirectional LSTM based recurrent neural networks. In _Fifteenth annual conference of the international speech communication association_.
  * Fazel et al. (2021) Amin Fazel, Wei Yang, Yulan Liu, Roberto Barra-Chicote, Yixiong Meng, Roland Maas, and Jasha Droppo. 2021.  Synthasr: Unlocking synthetic data for speech recognition.  _arXiv preprint arXiv:2106.07803_ (2021).
  * Fonseca et al. (2021) Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. 2021.  Fsd50k: an open dataset of human-labeled sound events.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 30 (2021), 829–852.
  * Franco-Galván et al. (2019) Carlos Franco-Galván, Abel Herrera-Camacho, and Boris Escalante-Ramírez. 2019.  Application of different statistical tests for validation of synthesized speech parameterized by cepstral coefficients and lsp.  _Computación y Sistemas_ 23, 2 (2019), 461–467.
  * Frankle and Carbin (2018) Jonathan Frankle and Michael Carbin. 2018.  The Lottery Ticket Hypothesis: Training Pruned Neural Networks.  _CoRR_ abs/1803.03635 (2018).  arXiv:1803.03635 http://arxiv.org/abs/1803.03635
  * Frantar and Alistarh (2023) Elias Frantar and Dan Alistarh. 2023.  SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot.  _ArXiv_ abs/2301.00774 (2023).
  * Fu et al. (2019) Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. 2019\.  Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In _International Conference on Machine Learning_. PMLR, 2031–2041.
  * Fu et al. (2016) Szu-Wei Fu, Yu Tsao, and Xugang Lu. 2016.  SNR-Aware Convolutional Neural Network Modeling for Speech Enhancement.. In _Interspeech_. 3768–3772.
  * Fu et al. (2021) Yihui Fu, Luyao Cheng, Shubo Lv, Yukai Jv, Yuxiang Kong, Zhuo Chen, Yanxin Hu, Lei Xie, Jian Wu, Hui Bu, et al. 2021\.  Aishell-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario.  _arXiv preprint arXiv:2104.03603_ (2021).
  * Fujita et al. (2019) Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Yawen Xue, Kenji Nagamatsu, and Shinji Watanabe. 2019\.  End-to-end neural speaker diarization with self-attention. In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 296–303.
  * Gabbay et al. (2017) Aviv Gabbay, Asaph Shamir, and Shmuel Peleg. 2017\.  Visual speech enhancement.  _arXiv preprint arXiv:1711.08789_ (2017).
  * Gabryś et al. (2022) Adam Gabryś, Goeric Huybrechts, Manuel Sam Ribeiro, Chung-Ming Chien, Julian Roth, Giulia Comini, Roberto Barra-Chicote, Bartek Perz, and Jaime Lorenzo-Trueba. 2022\.  Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing module. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7902–7906.
  * Galassi et al. (2020) Andrea Galassi, Marco Lippi, and Paolo Torroni. 2020\.  Attention in natural language processing.  _IEEE transactions on neural networks and learning systems_ 32, 10 (2020), 4291–4308.
  * Gales et al. (2008) Mark Gales, Steve Young, et al. 2008\.  The application of hidden Markov models in speech recognition.  _Foundations and Trends® in Signal Processing_ 1, 3 (2008), 195–304.
  * Gao et al. (2023) Chenyang Gao, Yue Gu, Francesco Caliva, and Yuzong Liu. 2023\.  Self-supervised speech representation learning for keyword-spotting with light-weight transformers.  _arXiv preprint arXiv:2303.04255_ (2023).
  * Gao and Grauman (2021) Ruohan Gao and Kristen Grauman. 2021.  Visualvoice: Audio-visual speech separation with cross-modal consistency. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 15490–15500.
  * Garcia-Romero et al. (2020) Daniel Garcia-Romero, Alan McCree, David Snyder, and Gregory Sell. 2020.  JHU-HLTCOE system for the VoxSRC speaker recognition challenge. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7559–7563.
  * Garofolo (1993) John S Garofolo. 1993\.  Timit acoustic phonetic continuous speech corpus.  _Linguistic Data Consortium, 1993_ (1993).
  * Ghifary et al. (2016) Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. 2016\.  Deep reconstruction-classification networks for unsupervised domain adaptation. In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_. Springer, 597–613.
  * Ghosal et al. (2023) Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. 2023.  Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model.  arXiv:2304.13731 [eess.AS]
  * Gibiansky et al. (2017) Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017\.  Deep voice 2: Multi-speaker neural text-to-speech.  _Advances in neural information processing systems_ 30 (2017).
  * Giri et al. (2019) Ritwik Giri, Umut Isik, and Arvindh Krishnaswamy. 2019\.  Attention wave-u-net for speech enhancement. In _2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 249–253.
  * Graves (2012) Alex Graves. 2012\.  Sequence transduction with recurrent neural networks.  _arXiv preprint arXiv:1211.3711_ (2012).
  * Graves and Graves (2012) Alex Graves and Alex Graves. 2012.  Connectionist temporal classification.  _Supervised sequence labelling with recurrent neural networks_ (2012), 61–93.
  * Graves and Jaitly (2014) Alex Graves and Navdeep Jaitly. 2014.  Towards end-to-end speech recognition with recurrent neural networks. In _International conference on machine learning_. PMLR, 1764–1772.
  * Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013\.  Speech recognition with deep recurrent neural networks. In _2013 IEEE international conference on acoustics, speech and signal processing_. Ieee, 6645–6649.
  * Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020\.  Conformer: Convolution-augmented transformer for speech recognition.  _arXiv preprint arXiv:2005.08100_ (2020).
  * Guo et al. (2022) Haohan Guo, Fenglong Xie, Frank K Soong, Xixin Wu, and Helen Meng. 2022.  A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS.  _arXiv preprint arXiv:2209.10887_ (2022).
  * Habibie et al. (2022) Ikhsanul Habibie, Mohamed Elgharib, Kripasindhu Sarkar, Ahsan Abdullah, Simbarashe Nyatsanga, Michael Neff, and Christian Theobalt. 2022\.  A motion matching-based framework for controllable gesture synthesis from speech. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–9.
  * Hady and Schwenker (2013) Mohamed Farouk Abdel Hady and Friedhelm Schwenker. 2013.  Semi-supervised learning.  _Handbook on Neural Information Processing_ (2013), 215–239.
  * Han et al. (2021) Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021\.  Learning shared semantic space for speech-to-text translation.  _arXiv preprint arXiv:2105.03095_ (2021).
  * Han et al. (2022) Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. 2022\.  A survey on vision transformer.  _IEEE transactions on pattern analysis and machine intelligence_ 45, 1 (2022), 87–110.
  * Han and Lee (2022) Seungu Han and Junhyeok Lee. 2022.  NU-Wave 2: A general neural audio upsampling model for various sampling rates.  _arXiv preprint arXiv:2206.08545_ (2022).
  * Han et al. (2020) Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. 2020.  Contextnet: Improving convolutional neural networks for automatic speech recognition with global context.  _arXiv preprint arXiv:2005.03191_ (2020).
  * Hanifa et al. (2021) Rafizah Mohd Hanifa, Khalid Isa, and Shamsul Mohamad. 2021\.  A review on speaker recognition: Technology and challenges.  _Computers & Electrical Engineering_ 90 (2021), 107005.
  * Hansen (1997) Peter SK Hansen. 1997\.  _Signal subspace methods for speech enhancement_.  Ph. D. Dissertation. Citeseer.
  * Hao et al. (2021) Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li. 2021\.  Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6633–6637.
  * Harte and Gillen (2015) Naomi Harte and Eoin Gillen. 2015.  TCD-TIMIT: An audio-visual corpus of continuous speech.  _IEEE Transactions on Multimedia_ 17, 5 (2015), 603–615.
  * Hatch et al. (2006) Andrew O Hatch, Sachin Kajarekar, and Andreas Stolcke. 2006\.  Within-class covariance normalization for SVM-based speaker recognition. In _Ninth international conference on spoken language processing_.
  * Haykin and Chen (2005) Simon Haykin and Zhe Chen. 2005.  The cocktail party problem.  _Neural computation_ 17, 9 (2005), 1875–1902.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He et al. (2017) Yanzhang He, Rohit Prabhavalkar, Kanishka Rao, Wei Li, Anton Bakhtin, and Ian McGraw. 2017\.  Streaming small-footprint keyword spotting using sequence-to-sequence models. In _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 474–481.
  * He et al. (2019) Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. 2019\.  Streaming end-to-end speech recognition for mobile devices. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6381–6385.
  * Hemphill et al. (1990) Charles T Hemphill, John J Godfrey, and George R Doddington. 1990\.  The ATIS spoken language systems pilot corpus. In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990_.
  * Hendrycks and Dietterich (2019) Dan Hendrycks and Thomas Dietterich. 2019.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=HJz6tiCqYm
  * Hershey et al. (2016) John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. 2016.  Deep clustering: Discriminative embeddings for segmentation and separation. In _2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 31–35.
  * Higuchi et al. (2022) Yosuke Higuchi, Brian Yan, Siddhant Arora, Tetsuji Ogawa, Tetsunori Kobayashi, and Shinji Watanabe. 2022.  BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model.  _arXiv preprint arXiv:2210.16663_ (2022).
  * Higy and Bell (2018) Bertrand Higy and Peter Bell. 2018.  Few-shot learning with attention-based sequence-to-sequence models.  _arXiv preprint arXiv:1811.03519_ (2018).
  * Himawan et al. (2019) Ivan Himawan, Fernando Villavicencio, Sridha Sridharan, and Clinton Fookes. 2019.  Deep domain adaptation for anti-spoofing in speaker verification systems.  _Computer Speech & Language_ 58 (2019), 377–402.
  * Hinton et al. (2012) Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. 2012\.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  _IEEE Signal processing magazine_ 29, 6 (2012), 82–97.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hou et al. (2018) Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, and Hsin-Min Wang. 2018\.  Audio-visual speech enhancement using multimodal deep convolutional neural networks.  _IEEE Transactions on Emerging Topics in Computational Intelligence_ 2, 2 (2018), 117–128.
  * Houlsby et al. (2019a) Neil Houlsby, , et al. 2019a.  Parameter-efficient transfer learning for NLP. In _International Conference on Machine Learning_. 2790–2799.
  * Houlsby et al. (2019b) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019b.  Parameter-Efficient Transfer Learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 97)_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 2790–2799.  https://proceedings.mlr.press/v97/houlsby19a.html
  * Hsu et al. (2017) Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang. 2017.  Voice conversion from unaligned corpora using variational autoencoding wasserstein generative adversarial networks.  _arXiv preprint arXiv:1704.00849_ (2017).
  * Hsu et al. (2020) Jui-Yang Hsu, Yuan-Jui Chen, and Hung-yi Lee. 2020.  Meta learning for end-to-end low-resource speech recognition. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7844–7848.
  * Hsu et al. (2021) Wei-Ning Hsu et al. 2021\.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 (2021), 3451–3460.
  * Hsu et al. (2022b) Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. 2022b.  ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement.  _arXiv preprint arXiv:2212.11377_ (2022).
  * Hsu et al. (2019) Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Yu-An Chung, Yuxuan Wang, Yonghui Wu, and James Glass. 2019.  Disentangling correlated speaker and noise for speech synthesis via data augmentation and adversarial factorization. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5901–5905.
  * Hsu et al. (2018) Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, et al. 2018\.  Hierarchical Generative Modeling for Controllable Speech Synthesis. In _International Conference on Learning Representations_.
  * Hsu et al. (2022a) Yen-Chang Hsu, Ting Hua, Sung-En Chang, Qiang Lou, Yilin Shen, and Hongxia Jin. 2022a.  Language model compression with weighted low-rank factorization.  _ArXiv_ abs/2207.00112 (2022).
  * Hu et al. (2021) Edward J Hu et al. 2021\.  LoRA: Low-Rank Adaptation of Large Language Models. In _International Conference on Learning Representations_.
  * Hu et al. (2022b) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022b.  LoRA: Low-Rank Adaptation of Large Language Models. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=nZeVKeeFYf9
  * Hu et al. (2022a) Hang-Rui Hu, Yan Song, Ying Liu, Li-Rong Dai, Ian McLoughlin, and Lin Liu. 2022a.  Domain Robust Deep Embedding Learning for Speaker Recognition. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7182–7186.
  * Hu et al. (2017) Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. 2017.  Squeeze-and-Excitation Networks.  https://doi.org/10.48550/ARXIV.1709.01507
  * Hu et al. (2020) Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. 2020.  DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement.  _arXiv preprint arXiv:2008.00264_ (2020).
  * Hu and Loizou (2007) Yi Hu and Philipos C Loizou. 2007.  Evaluation of objective quality measures for speech enhancement.  _IEEE Transactions on audio, speech, and language processing_ 16, 1 (2007), 229–238.
  * Huang et al. (2022a) Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. 2022a.  Fastdiff: A fast conditional diffusion model for high-quality speech synthesis.  _arXiv preprint arXiv:2204.09934_ (2022).
  * Huang et al. (2022c) Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. 2022c.  Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech.  _Advances in Neural Information Processing Systems_ 35 (2022), 10970–10983.
  * Huang et al. (2022d) Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. 2022d.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In _Proceedings of the 30th ACM International Conference on Multimedia_. 2595–2605.
  * Huang et al. (2023) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023.  Language Is Not All You Need: Aligning Perception with Language Models.  _ArXiv_ abs/2302.14045 (2023).
  * Huang et al. (2022b) Sung-Feng Huang, Chyi-Jiunn Lin, Da-Rong Liu, Yi-Chen Chen, and Hung-yi Lee. 2022b.  Meta-TTS: Meta-learning for few-shot speaker adaptive text-to-speech.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 30 (2022), 1558–1571.
  * Huang et al. (2021) Wen-Chin Huang, Tomoki Hayashi, Xinjian Li, Shinji Watanabe, and Tomoki Toda. 2021.  On prosody modeling for ASR+ TTS based voice conversion. In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 642–649.
  * Huang et al. (2019) Wen-Chin Huang, Tomoki Hayashi, Yi-Chiao Wu, Hirokazu Kameoka, and Tomoki Toda. 2019.  Voice transformer network: Sequence-to-sequence voice conversion using transformer with text-to-speech pretraining.  _arXiv preprint arXiv:1912.06813_ (2019).
  * Huang et al. (2020) Zhiying Huang, Hao Li, and Ming Lei. 2020.  Devicetts: A small-footprint, fast, stable network for on-device text-to-speech.  _arXiv preprint arXiv:2010.15311_ (2020).
  * Hung et al. (2022) Yun-Ning Hung, Chih-Wei Wu, Iroro Orife, Aaron Hipple, William Wolcott, and Alexander Lerch. 2022\.  A large TV dataset for speech and music activity detection.  _EURASIP Journal on Audio, Speech, and Music Processing_ 2022, 1 (2022), 21.
  * Hwang et al. (2022) Dongseong Hwang, Ananya Misra, Zhouyuan Huo, Nikhil Siddhartha, Shefali Garg, David Qiu, Khe Chai Sim, Trevor Strohman, Françoise Beaufays, and Yanzhang He. 2022.  Large-scale asr domain adaptation using self-and semi-supervised learning. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6627–6631.
  * Inaguma et al. (2020) Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Enrique Yalta Soplin, Tomoki Hayashi, and Shinji Watanabe. 2020\.  ESPnet-ST: All-in-one speech translation toolkit.  _arXiv preprint arXiv:2004.10234_ (2020).
  * Indurthi et al. (2020) Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020.  End-end speech-to-text translation with modality agnostic meta-learning. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7904–7908.
  * Isik et al. (2020) Umut Isik, Ritwik Giri, Neerad Phansalkar, Jean-Marc Valin, Karim Helwani, and Arvindh Krishnaswamy. 2020.  Poconet: Better speech enhancement with frequency-positional embeddings, semi-supervised conversational data, and biased loss.  _arXiv preprint arXiv:2008.04470_ (2020).
  * Ito and Johnson (2017) Keith Ito and Linda Johnson. 2017.  The LJ Speech Dataset.  https://keithito.com/LJ-Speech-Dataset/.
  * Jeong et al. (2021) Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. 2021.  Diff-tts: A denoising diffusion model for text-to-speech.  _arXiv preprint arXiv:2104.01409_ (2021).
  * Jia et al. (2022) Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022.  Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In _International Conference on Machine Learning_. PMLR, 10120–10134.
  * Jiang et al. (2020) Dongwei Jiang, Wubo Li, Miao Cao, Wei Zou, and Xiangang Li. 2020.  Speech simclr: Combining contrastive and reconstruction objective for self-supervised speech representation learning.  _arXiv preprint arXiv:2010.13991_ (2020).
  * Jiao et al. (2021) Yunlong Jiao, Adam Gabryś, Georgi Tinchev, Bartosz Putrycz, Daniel Korzekwa, and Viacheslav Klimkov. 2021.  Universal neural vocoding with parallel wavenet. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6044–6048.
  * Jin et al. (2009) Wen Jin, Xin Liu, Michael S Scordilis, and Lu Han. 2009\.  Speech enhancement using harmonic emphasis and adaptive comb filtering.  _IEEE transactions on audio, speech, and language processing_ 18, 2 (2009), 356–368.
  * Jo et al. (2021) Yong Rae Jo, Young Ki Moon, Won Ik Cho, and Geun Sik Jo. 2021\.  Self-attentive vad: Context-aware detection of voice from noise. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6808–6812.
  * Johri et al. (2019) Anubhav Johri, Ashish Tripathi, et al. 2019\.  Parkinson disease detection using deep neural networks. In _2019 twelfth international conference on contemporary computing (IC3)_. IEEE, 1–4.
  * Ju et al. (2022) Yooncheol Ju, Ilhwan Kim, Hongsun Yang, Ji-Hoon Kim, Byeongyeol Kim, Soumi Maiti, and Shinji Watanabe. 2022.  TriniTTS: Pitch-controllable End-to-end TTS without External Aligner. In _Proc. Interspeech_. 16–20.
  * Jung et al. (2019) Jee-weon Jung, Hee-Soo Heo, Ju-ho Kim, Hye-jin Shim, and Ha-Jin Yu. 2019.  Rawnet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification.  _arXiv preprint arXiv:1904.08104_ (2019).
  * Jung et al. (2021a) Jee-weon Jung, Hee-Soo Heo, Ha-Jin Yu, and Joon Son Chung. 2021a.  Graph attention networks for speaker verification. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6149–6153.
  * Jung et al. (2021b) Jee-weon Jung, Hee-Soo Heo, Ha-Jin Yu, and Joon Son Chung. 2021b.  Graph Attention Networks for Speaker Verification. In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 6149–6153.  https://doi.org/10.1109/ICASSP39728.2021.9414057
  * Kahn et al. (2020) Jacob Kahn, Ann Lee, and Awni Hannun. 2020.  Self-training for end-to-end speech recognition. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7084–7088.
  * Kakuba et al. (2022) Samuel Kakuba, Alwin Poulose, and Dong Seog Han. 2022\.  Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features.  _IEEE Access_ 10 (2022), 125538–125551.
  * Kala and Shinozaki (2018) Taku Kala and Takahiro Shinozaki. 2018.  Reinforcement learning of speech recognition system based on policy gradient and hypothesis selection. In _2018 ieee international conference on acoustics, speech and signal processing (icassp)_. IEEE, 5759–5763.
  * Kalchbrenner et al. (2018) Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018.  Efficient neural audio synthesis. In _International Conference on Machine Learning_. PMLR, 2410–2419.
  * Kalchbrenner et al. (2016) Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016.  Neural machine translation in linear time.  _arXiv preprint arXiv:1610.10099_ (2016).
  * Kalchbrenner et al. (2014) Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014\.  A convolutional neural network for modelling sentences.  _arXiv preprint arXiv:1404.2188_ (2014).
  * Kameoka et al. (2019) Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. 2019.  ACVAE-VC: Non-parallel voice conversion with auxiliary classifier variational autoencoder.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 27, 9 (2019), 1432–1443.
  * Kanda et al. (2022a) Naoyuki Kanda, Jian Wu, Yu Wu, Xiong Xiao, Zhong Meng, Xiaofei Wang, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Takuya Yoshioka. 2022a.  Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings.  _arXiv preprint arXiv:2203.16685_ (2022).
  * Kanda et al. (2022b) Naoyuki Kanda, Xiong Xiao, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, and Takuya Yoshioka. 2022b.  Transcribe-to-diarize: Neural speaker diarization for unlimited number of speakers using end-to-end speaker-attributed asr. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 8082–8086.
  * Kaneko et al. (2019) Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. 2019.  Cyclegan-vc2: Improved cyclegan-based non-parallel voice conversion. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6820–6824.
  * Kaneko et al. (2020) Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. 2020.  Cyclegan-vc3: Examining and improving cyclegan-vcs for mel-spectrogram conversion.  _arXiv preprint arXiv:2010.11672_ (2020).
  * Kaneko et al. (2021) Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. 2021.  Maskcyclegan-vc: Learning non-parallel voice conversion with filling in frames. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5919–5923.
  * Kaneko et al. (2022) Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki. 2022.  iSTFTNet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time Fourier transform. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6207–6211.
  * Kang et al. (2020) Jiawen Kang, Ruiqi Liu, Lantian Li, Yunqi Cai, Dong Wang, and Thomas Fang Zheng. 2020\.  Domain-invariant speaker vector projection by model-agnostic meta-learning.  _arXiv preprint arXiv:2005.11900_ (2020).
  * Kansizoglou et al. (2019) Ioannis Kansizoglou, Loukas Bampis, and Antonios Gasteratos. 2019.  An active learning paradigm for online audio-visual emotion recognition.  _IEEE Transactions on Affective Computing_ 13, 2 (2019), 756–768.
  * Karita et al. (2019a) Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al. 2019a.  A comparative study on transformer vs rnn in speech applications. In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 449–456.
  * Karita et al. (2019b) Shigeki Karita, Nelson Yalta, Shinji Watanabe, Marc Delcroix, Atsunori Ogawa, and Tomohiro Nakatani. 2019b.  Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration. In _INTERSPEECH_.
  * Kawakami (2008) Kazuya Kawakami. 2008\.  _Supervised sequence labelling with recurrent neural networks_.  Ph. D. Dissertation. Technical University of Munich.
  * Kawakami et al. (2020) Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, and Aaron van den Oord. 2020.  Learning robust and multilingual speech representations.  _arXiv preprint arXiv:2001.11128_ (2020).
  * Kenter et al. (2019) Tom Kenter, Vincent Wan, Chun-An Chan, Rob Clark, and Jakub Vit. 2019.  CHiVE: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network. In _International Conference on Machine Learning_. PMLR, 3331–3340.
  * Kim et al. (2022b) Heeseung Kim, Sungwon Kim, and Sungroh Yoon. 2022b.  Guided-tts: A diffusion model for text-to-speech via classifier guidance. In _International Conference on Machine Learning_. PMLR, 11119–11133.
  * Kim et al. (2020) Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020\.  Glow-tts: A generative flow for text-to-speech via monotonic alignment search.  _Advances in Neural Information Processing Systems_ 33 (2020), 8067–8077.
  * Kim et al. (2021b) Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021b.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In _International Conference on Machine Learning_. PMLR, 5530–5540.
  * Kim and Lee (2021) Juntae Kim and Jeehye Lee. 2021.  Generalizing RNN-transducer to out-domain audio via sparse self-attention layers.  _arXiv preprint arXiv:2108.10752_ (2021).
  * Kim et al. (2022d) Jaechang Kim, Yunjoo Lee, Seunghoon Hong, and Jungseul Ok. 2022d.  Learning continuous representation of audio for arbitrary scale super resolution. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 3703–3707.
  * Kim et al. (2021c) Ji-Hoon Kim, Sang-Hoon Lee, Ji-Hyun Lee, and Seong-Whan Lee. 2021c.  Fre-gan: Adversarial frequency-consistent audio synthesis.  _arXiv preprint arXiv:2106.02297_ (2021).
  * Kim et al. (2022a) Sehoon Kim, Amir Gholami, Albert Shaw, Nicholas Lee, Karttikeya Mangalam, Jitendra Malik, Michael W Mahoney, and Kurt Keutzer. 2022a.  Squeezeformer: An efficient transformer for automatic speech recognition.  _arXiv preprint arXiv:2206.00888_ (2022).
  * Kim et al. (2021a) Seongbin Kim, Gyuwan Kim, Seongjin Shin, and Sangmin Lee. 2021a.  Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding. In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 7463–7467.  https://doi.org/10.1109/ICASSP39728.2021.9414619
  * Kim et al. (2022c) Sungwon Kim, Heeseung Kim, and Sungroh Yoon. 2022c.  Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data.  _arXiv preprint arXiv:2205.15370_ (2022).
  * Kim et al. (2018) Sungwon Kim, Sang-gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. 2018.  FloWaveNet: A generative flow for raw audio.  _arXiv preprint arXiv:1811.02155_ (2018).
  * Kinnunen et al. (2005) Tomi Kinnunen, Evgeny Karpov, and Pasi Franti. 2005\.  Real-time speaker identification and verification.  _IEEE Transactions on Audio, Speech, and Language Processing_ 14, 1 (2005), 277–288.
  * Kinoshita et al. (2020) Keisuke Kinoshita, Tsubasa Ochiai, Marc Delcroix, and Tomohiro Nakatani. 2020.  Improving noise robust automatic speech recognition with single-channel time-domain enhancement network. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7009–7013.
  * Kiranyaz et al. (2021) Serkan Kiranyaz, Onur Avci, Osama Abdeljaber, Turker Ince, Moncef Gabbouj, and Daniel J. Inman. 2021\.  1D convolutional neural networks and applications: A survey.  _Mechanical Systems and Signal Processing_ 151 (2021), 107398.  https://doi.org/10.1016/j.ymssp.2020.107398
  * Kiranyaz et al. (2015) Serkan Kiranyaz, Turker Ince, Ridha Hamila, and Moncef Gabbouj. 2015.  Convolutional neural networks for patient-specific ECG classification. In _2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2608–2611.
  * Koizumi et al. (2020) Yuma Koizumi, Kohei Yatabe, Marc Delcroix, Yoshiki Masuyama, and Daiki Takeuchi. 2020.  Speech enhancement using self-adaptation and multi-head self-attention. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 181–185.
  * Koizumi et al. (2022) Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. 2022.  SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping.  _arXiv preprint arXiv:2203.16749_ (2022).
  * Kolbæk et al. (2017) Morten Kolbæk, Dong Yu, Zheng-Hua Tan, and Jesper Jensen. 2017\.  Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 25, 10 (2017), 1901–1913.
  * Koluguri et al. (2022) Nithin Rao Koluguri, Taejin Park, and Boris Ginsburg. 2022\.  TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 8102–8106.
  * Kominek and Black (2004) John Kominek and Alan W Black. 2004.  The CMU Arctic speech databases. In _Fifth ISCA workshop on speech synthesis_.
  * Kong et al. (2020a) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020a.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ 33 (2020), 17022–17033.
  * Kong et al. (2020b) Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. 2020b.  Diffwave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ (2020).
  * Koval and Krynov (2020) Sergey Koval and Sergey Krynov. 2020.  Practice of usage of spectral analysis for forensic speaker identification. In _RLA2C 1998-Speaker Recognition and its Commercial and Forensic Applications_. 136–140.
  * Koyama et al. (2020) Yuichiro Koyama, Tyler Vuong, Stefan Uhlich, and Bhiksha Raj. 2020.  Exploring the best loss function for DNN-based low-latency speech enhancement with temporal convolutional networks.  _arXiv preprint arXiv:2005.11611_ (2020).
  * Kreuk et al. (2022) Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D’efossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022.  AudioGen: Textually Guided Audio Generation.  _ArXiv_ abs/2209.15352 (2022).
  * Kriman et al. (2020) Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. 2020.  Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6124–6128.
  * Kulkarni et al. (2020) Ajinkya Kulkarni, Vincent Colotte, and Denis Jouvet. 2020\.  Transfer learning of the expressivity using FLOW metric learning in multispeaker text-to-speech synthesis. In _INTERSPEECH 2020_.
  * Kumar et al. (2019) Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. 2019\.  Melgan: Generative adversarial networks for conditional waveform synthesis.  _Advances in neural information processing systems_ 32 (2019).
  * Kwon et al. (2019) Ohsung Kwon, Inseon Jang, ChungHyun Ahn, and Hong-Goo Kang. 2019\.  An Effective Style Token Weight Control Technique for End-to-End Emotional Speech Synthesis.  _IEEE Signal Processing Letters_ 26, 9 (2019), 1383–1387.  https://doi.org/10.1109/LSP.2019.2931673
  * Kwon et al. (2022) Youngki Kwon, Hee-Soo Heo, Jee-weon Jung, You Jin Kim, Bong-Jin Lee, and Joon Son Chung. 2022\.  Multi-scale speaker embedding-based graph attention networks for speaker diarisation. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 8367–8371.
  * Kwon et al. (2021) Youngki Kwon, Jee-weon Jung, Hee-Soo Heo, You Jin Kim, Bong-Jin Lee, and Joon Son Chung. 2021\.  Adapting speaker embeddings for speaker diarisation.  _arXiv preprint arXiv:2104.02879_ (2021).
  * Kye et al. (2020) Seong Min Kye, Youngmoon Jung, Hae Beom Lee, Sung Ju Hwang, and Hoirin Kim. 2020.  Meta-learning for short utterance speaker recognition with imbalance length pairs.  _arXiv preprint arXiv:2004.02863_ (2020).
  * Lai et al. (2021) Cheng-I Jeff Lai, Yang Zhang, Alexander H. Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David D. Cox, and James R. Glass. 2021\.  PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition.  _CoRR_ abs/2106.05933 (2021).  arXiv:2106.05933 https://arxiv.org/abs/2106.05933
  * Lakhotia et al. (2021) Kushal Lakhotia et al. 2021\.  On generative spoken language modeling from raw audio.  _Transactions of the Association for Computational Linguistics_ 9 (2021).
  * Lakomkin et al. (2018) Egor Lakomkin, Mohammad Ali Zamani, Cornelius Weber, Sven Magg, and Stefan Wermter. 2018\.  Emorl: continuous acoustic emotion classification using deep reinforcement learning. In _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 4445–4450.
  * Lam et al. (2021) Max WY Lam, Jun Wang, Dan Su, and Dong Yu. 2021\.  Sandglasset: A light multi-granularity self-attentive network for time-domain speech separation. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5759–5763.
  * Łańcucki (2021) Adrian Łańcucki. 2021\.  Fastpitch: Parallel text-to-speech with pitch prediction. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6588–6592.
  * Landini et al. (2022) Federico Landini, Ján Profant, Mireia Diez, and Lukáš Burget. 2022.  Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: theory, implementation and analysis on standard tasks.  _Computer Speech & Language_ 71 (2022), 101254.
  * Larcher et al. (2012) Anthony Larcher, Kong Aik Lee, Bin Ma, and Haizhou Li. 2012\.  The RSR2015: Database for text-dependent speaker verification using multiple pass-phrases. In _Annual Conference of the International Speech Communication Association (Interspeech)_.
  * Larcher et al. (2021) Anthony Larcher, Ambuj Mehrish, Marie Tahon, Sylvain Meignier, Jean Carrive, David Doukhan, Olivier Galibert, and Nicholas Evans. 2021\.  Speaker embeddings for diarization of broadcast data in the allies challenge. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5799–5803.
  * Latif et al. (2020) Siddique Latif, Junaid Qadir, Adnan Qayyum, Muhammad Usama, and Shahzad Younis. 2020.  Speech technology for healthcare: Opportunities, challenges, and state of the art.  _IEEE Reviews in Biomedical Engineering_ 14 (2020), 342–356.
  * Lee et al. (2022b) Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, and Katrin Kirchhoff. 2022b.  Self-supervised Representation Learning for Speech Processing. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts_. 8–13.
  * Lee and Han (2021) Junhyeok Lee and Seungu Han. 2021.  Nu-wave: A diffusion probabilistic model for neural audio upsampling.  _arXiv preprint arXiv:2104.02321_ (2021).
  * Lee et al. (2015) Kong Aik Lee, Anthony Larcher, Guangsen Wang, Patrick Kenny, Niko Brümmer, David Van Leeuwen, Hagai Aronowitz, Marcel Kockmann, Carlos Vaquero, Bin Ma, et al. 2015\.  The RedDots data collection for speaker recognition. In _Interspeech 2015_.
  * Lee et al. (2019) Kong Aik Lee, Qiongqiong Wang, and Takafumi Koshinaka. 2019\.  The CORAL+ algorithm for unsupervised domain adaptation of PLDA. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5821–5825.
  * Lee et al. (2021a) Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. 2021a.  PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior. In _International Conference on Learning Representations_.
  * Lee et al. (2020) Sang-gil Lee, Sungwon Kim, and Sungroh Yoon. 2020.  Nanoflow: Scalable normalizing flows with sublinear parameter complexity.  _Advances in Neural Information Processing Systems_ 33 (2020), 14058–14067.
  * Lee et al. (2022a) Sang-Hoon Lee, Seung-Bin Kim, Ji-Hyun Lee, Eunwoo Song, Min-Jae Hwang, and Seong-Whan Lee. 2022a.  HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis.  _Advances in Neural Information Processing Systems_ 35 (2022), 16624–16636.
  * Lee et al. (2021b) Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. 2021b.  Bidirectional variational inference for non-autoregressive text-to-speech. In _International Conference on Learning Representations_.
  * Lemaire and Holzapfel (2019) Quentin Lemaire and Andre Holzapfel. 2019.  Temporal convolutional networks for speech and music detection in radio broadcast. In _20th International Society for Music Information Retrieval Conference, ISMIR 2019, 4-8 November 2019_. International Society for Music Information Retrieval.
  * Lemercier et al. (2022) Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. 2022.  StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation.  _arXiv preprint arXiv:2212.11851_ (2022).
  * Leng et al. (2022) Yichong Leng, Zehua Chen, Junliang Guo, Haohe Liu, Jiawei Chen, Xu Tan, Danilo Mandic, Lei He, Xiang-Yang Li, Tao Qin, et al. 2022\.  BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis.  _arXiv preprint arXiv:2205.14807_ (2022).
  * Leroy et al. (2019) David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. 2019\.  Federated learning for keyword spotting. In _ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 6341–6345.
  * Levkovitch et al. (2022) Alon Levkovitch, Eliya Nachmani, and Lior Wolf. 2022\.  Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models.  _arXiv preprint arXiv:2206.02246_ (2022).
  * Li et al. (2020a) Bo Li, Shuo-yiin Chang, Tara N Sainath, Ruoming Pang, Yanzhang He, Trevor Strohman, and Yonghui Wu. 2020a.  Towards fast and accurate streaming end-to-end ASR. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6069–6073.
  * Li et al. (2021b) Bo Li, Anmol Gulati, Jiahui Yu, Tara N Sainath, Chung-Cheng Chiu, Arun Narayanan, Shuo-Yiin Chang, Ruoming Pang, Yanzhang He, James Qin, et al. 2021b.  A better and faster end-to-end model for streaming asr. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5634–5638.
  * Li et al. (2017) Bo Li, Tara N Sainath, Arun Narayanan, Joe Caroselli, Michiel Bacchiani, Ananya Misra, Izhak Shafran, Hasim Sak, Golan Pundak, Kean K Chin, et al. 2017\.  Acoustic Modeling for Google Home.. In _Interspeech_. 399–403.
  * Li et al. (2022a) Jinyu Li et al. 2022a.  Recent advances in end-to-end automatic speech recognition.  _APSIPA Transactions on Signal and Information Processing_ 11, 1 (2022).
  * Li et al. (2022c) Jingyu Li, Wei Liu, and Tan Lee. 2022c.  EDITnet: A Lightweight Network for Unsupervised Domain Adaptation in Speaker Verification.  _arXiv preprint arXiv:2206.07548_ (2022).
  * Li et al. (2019b) Jingdong Li, Hui Zhang, Xueliang Zhang, and Changliang Li. 2019b.  Single channel speech enhancement using temporal convolutional recurrent neural networks. In _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 896–900.
  * Li et al. (2022d) Kai Li, Runxuan Yang, and Xiaolin Hu. 2022d.  An efficient encoder-decoder architecture with top-down attention for speech separation.  _arXiv preprint arXiv:2209.15200_ (2022).