# A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Zhaofeng Shi
<EMAIL_ADDRESS>
University of Electronic Science and Technology of China

###### Abstract

With the development of deep learning and artificial intelligence, audio
synthesis has a pivotal role in the area of machine learning and shows strong
applicability in the industry. Meanwhile, significant efforts have been
dedicated by researchers to handle multimodal tasks at present such as audio-
visual multimodal processing. In this paper, we conduct a survey on audio
synthesis and audio-visual multimodal processing, which helps understand
current research and future trends. This review focuses on text to
speech(TTS), music generation and some tasks that combine visual and acoustic
information. The corresponding technical methods are comprehensively
classified and introduced, and their future development trends are prospected.
This survey can provide some guidance for researchers who are interested in
the areas like audio synthesis and audio-visual multimodal processing.

##  1 introduction

Audio synthesis, which aims to synthesis various form of natural and
intelligible sound such as speech, music, has a wide range of application
scenario in human society and industry. Initially, researchers took advantages
of pure signal processing methods to find some convenient representations for
audio, which can be easily modelled and transform to temporal audio. For
example, short-time Fourier Transform(STFT) is an efficient way to convert
audio into frequency domain and Griffin-Lim [31] is a kind of pure signal
processing algorithm that is able to decode STFT sequence to temporal
waveform. Methods similar to Griffin-Lim are WORLD [62], etc. In recent years,
with the rapid development of deep learning technology, researchers have begun
to build deep neural networks for audio synthesis and other multimodal tasks
in order to simplify the pipeline and improve the performance of the model.
Numerous neural network models have emerged so far for the tasks such as text
to speech(TTS) and music generation. There are a lot of models for TTS that
have been reported, such as Parallel WaveGAN [103], MelGAN [45],
FastSpeech2/2s [80], EATs [21], VITS [40]. Simultaneously, there are many
models for music generation like song from PI [12], C-RNN-GAN [60], MidiNet
[104], MuseGAN [23] and Jukebox [18]. These models bring great convenience to
human production and life, and they provide key reference for future research.

Vision is a physiological word. Humans and animals visually perceive the size,
brightness, color, etc. of external objects, and obtain information that is
essential for survival. Vision is the most important sense for human beings.
Over these years, deep learning has been widely explored in various image
processing and computer vision tasks such as image dehazing/deraining,
objective detection and image segmentation, which contribute to the
development of social productivity. Image dehazing/deraining means given a
blurred image with haze/rain, algorithms are used to remove the haze/rain in
the image to make it clear. [95, 46, 100, 47, 99, 54, 98, 96, 48] proposed
neural network-based models for image dehazing/deraining respectively.
Objective detection means finding out all the targets of interest in the image
and determining their locations and classes. In [19, 74, 75, 93, 53, 51, 52,
73, 11, 10, 50], several neural network-based models with high performance for
objective detection can be found. Image segmentation means dividing an image
into a number of specific regions with unique properties and presenting
objects of interest. The work on image segmentation includes [108, 32, 85,
106, 105, 58, 107, 101, 82, 55, 84].

Modality means the form or source of every kind of information such as visual,
auditory and tactile. Information from different modalities is often closely
related in our life. For example, when talking, we need to combine the other’s
facial expression and speech to determine his emotions. With the development
of deep learning, multimodal machine learning has become a research hotspot.
In this review, we focus on audio-visual multimodal processing, which has a
large range of applications. Many well-performing models for audio-visual
multimodal processing has been proposed in the past few years. For instance,
Vid2Speech [24] reported by Ephrat et al. is able to handle lipreading task.
Acapella [61] is a model for singing voice separation. VisualSpeech [29] is a
versatile model, which achieves state of the art performance in speech
separation and speech enhancement tasks. More models for audio-visual
multimodal processing will be comprehensively summarized in section 4.

In this review, we mainly review the research work on three aspects, TTS,
music generation and audio-visual multimodal processing based on deep learning
technology. In section 2, section 3 and section 4, we introduce the background
and technical methods corresponding to the three tasks respectively. In
section 5, we present several of the most commonly used datasets in studies of
different areas respectively. Finally, in section 6, we present conclusions
based on the description and discussion in the above. The contributions of
this review include:

  * •

We introduce and summarize the background and technical methods of TTS, music
generation and multimodal audio-visual processing tasks.

  * •

This review provides some guidance for researchers who are interested in this
direction.

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=left, font=, rectangle, draw=hiddendraw,
rounded corners, align=left, minimum width=2.5em, inner xsep=4pt, inner
ysep=0pt, , [Survey [Sec. 2: Text to speech [Sec. 2.1: Two-stage methods [Sec.
2.1.1: Acoustic models ] [Sec. 2.1.2: Vocoders ] ] [Sec. 2.2: End-to-End
methods ] ] [sec. 3: Music generation [Sec. 3.1: Symbolic music generation ]
[Sec. 3.2: Audio music generation ] ] [sec. 4: Audio-visual multimodal
processing [sec. 4.1: Lipreading ] [sec. 4.2: Audio-visual speech separation ]
[sec. 4.3: Talking face generation ] [sec. 4.4: Sound generation from video ]
] [sec. 5: Datasets [sec. 5.1: Datasets for TTS ] [sec. 5.2: Datasets for
music generation ] [sec. 5.3: Datasets for audio-visual multimodal processing
] ] ]

Figure 1: Organization of this paper.

##  2 Text to Speech

Text to speech, also known as speech synthesis, which aims to generate natural
and intelligible speech waveform under the condition of the input text [88].
With the continuous development of artificial intelligence technology, TTS has
gradually become a hot research topic and it has been widely adopted in the
field of audiobook services, automotive navigation services, automated
answering services, etc. However, the requirements of the naturalness and
accuracy of the speech generated by the TTS system are also increasing as
well. A well-designed neural network is needed in order to meet these
requirements.

Modelling the raw audio waveform is a challenging problem because the temporal
resolution of the audio data is extremely high. The sampling rate of audio is
at least 16000Hz. And another challenge is that there are long-term and short-
term dependencies in the audio waveform, which are difficult to model. In
addition to this, there are other difficulties for the TTS tasks such as the
alignment problem and the one-to-many problem. The alignment problem means
that the output speech audio content should be temporal aligned with the input
text or phoneme sequences. At present, the mainstream solution to this problem
is to adopt Monotonic Alignment Search(MAS) [39]. As for the one-to-many
problem, it means that the text input can be spoken in multiple ways with
different variations such as duration, pitch, and there are many kinds of
neural network structures have been designed to deal with this problem.

The frameworks of TTS tasks are mainly based on two-stage methods and end-to-
end methods respectively. Most previous work construct two-stage pipelines
apart from text preprocessing such as text normalization and conversion from
text to phoneme. The first stage is to produce intermediate representations
such as linguistic features [66] or acoustic features such as mel-spectrograms
[83] under the condition of the input sequence consist of phonemes,
characters, etc. The model of the second stage is also known as vocoder, which
generates raw audio waveforms conditioned on the input intermediate
representations. Numerous models of each part of the two-stage pipelines have
been developed independently. Taking advantage of the two-stage method, we can
get high-fidelity audio. Nonetheless, two-stage pipelines remain problematic
because the models need fine-tuning and the errors of the previous module may
affect the next module causing error accumulation. To solve this problem, more
and more end-to-end models have been developed by researchers. End-to-end
models output raw audio waveforms directly without any intermediate
representations. Because it can parallel generate audio in a short time and
alleviate the problem of error accumulation without spending a lot of time
tuning parameters, the end-to-end method has become the current research
direction in TTS field. The details of these two methods will be introduced
below.

###  2.1 Two-stage methods

As mentioned above, the first stage of the two-stage method is to convert the
input text sequence into the intermediate representation and the second stage
generates raw audio waveform. In the first stage, a linguistic model or an
acoustic model is deployed to generate linguistic features or acoustic
features as intermediate features. The model in the second stage, which is
also called vocoder, converts the intermediate representations into the raw
audio waveform. To generate high-quality speech audio, these two models must
operate well together. The models corresponding to two-stage methods will be
introduced and summarized below.

####  2.1.1 Acoustic models

Nowadays, acoustic features are usually used as the intermediate features in
TTS tasks. As a result, we focus on the research work on acoustic models in
this section. Acoustic features are generated by acoustic models from
linguistic features or directly from characters or phonemes [88]. The aim of
acoustic models is to generate acoustic features that are further converted
into raw audio waveform using vocoders in two-stage methods of TTS tasks. The
acoustic models that have been reported in recent years will be
comprehensively summarized below.

Tacotron and Tacotron 2 [94, 83] are autoregressive neural network-based
models that take raw text characters as input and output mel-spectrograms.
Other autoregressive models similar to them, which take mel-spectrograms as
output, including DeepVoice 3, Transformer TTS, Flowtron, MelNet [68, 49, 90,
91], etc. It is worth mentioning that MelNet is an autoregressive model that
takes both time and frequency into consideration. After the knowledge
distillation [36] was proposed by Geoffrey et al., more and more researchers
have tried to take advantage of knowledge distillation to train acoustic
models like ParaNet [67] and Fastspeech [79]. They are both non-autoregressive
models. In addition, Flow-TTS [59], Glow-TTS [39], etc. are flow-based non-
autoregressive models and Fastspeech 2 [80] is a non-autoregressive model
based on Transformer [92].

####  2.1.2 Vocoders

Vocoder is the model that converts the intermediate representations produced
in the first stage into the raw audio waveform. In recent years, many neural
network-based vocoders have been designed to achieve high-fidelity speech
audio synthesis. We will summarize the research work related to vocoders
below, and classify these models into two categories: autoregressive models
and non-autoregressive models.

##### Autoregressive models

WaveNet [66] is the first neural network-based vocoder, which takes linguistic
features as input and generated raw audio waveform autoregressively. WaveNet
is fully convolutional. By using dilated causal convolutions, WaveNet can
achieve a large receptive field, which plays an important role in addressing
the issue of modelling the high temporal resolution audio waveform. SampleRNN
[57] is another autoregressive model based on RNN. It is an unconditional end-
to-end neural audio generative model, which uses a hierarchy of RNN module,
each operating at a different time resolution to deal with the challenge of
modelling the raw audio waveform at different temporal resolution. DeepVoice
[2] and LPCNet [89] are also autoregressive models, like the two models
mentioned above. However, for autoregressive models, each output sample point
is conditioned on the previously generated sample points. As a result,
inference with these models is time-consuming because audio sample points must
be generated sequentially. In response to these problems, researchers designed
non-autoregressive models that can generate audio in parallel.

##### Non-autogressive models

Non-autoregressive models can generate raw audio samples in parallel to
quickly generate speech audio to satisfy real-time requirements. There are
many technical methods that have been proven to be applicable to non-
autoregressive models. WaveGlow, FloWaveNet and WaveFlow [72, 41, 70] are
flow-based non-autoregressive models. Parallel WaveNet [65] is also a flow-
based model, which combines knowledge distillation [36] to allow the inverse
autoregressive flow(IAF) [43] network as a student network and the pretrained
WaveNet as a teacher network. ClariNet [69] is also a vocoder that employs
knowledge distillation [36]. However, the training process with distillation-
based methods remain problems. It requires not only a robust teacher model,
but also a methodology to optimize the complicated distillation process. Much
recent progress in Generative Adversarial Networks(GANs) [30] has been
expedited and WaveGAN [20] is the first GAN-based vocoder, which modifies the
structure of DCGAN [76] to realize audio synthesis. In addition, Parallel
WaveGAN [103] is a GAN-based model using multi resolution short time Fourier
Transform(STFT) to capture the time-frequency distribution of the natural
speech audio. MelGAN [45] is another non-autoregressive feed-forward vocoder
based on GAN, which inverts the input mel-spectrogram into raw speech audio.
Other GAN-based vocoders include GAN-TTS [5], HiFi-GAN [44], etc.

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=left, font=, rectangle, draw=hiddendraw,
rounded corners, align=left, minimum width=2.5em, inner xsep=4pt, inner
ysep=0pt, , where level=1text width=6.9em,font=, where level=2text
width=9em,font=, where level=3font=,yshift=0.25pt, [TTS [Two-stage [Acoustic
models [Tacotron [94],Tacotron 2 [83]
DeepVoice 3 [68],TransformerTTS [49]
Flowtron [90],MelNet [91]
ParaNet [67],Fastspeech [79]
FlowTTS [59],GlowTTS [39]
Fastspeech2 [80] ,leaf,text width=15em ] ] [Vocoders [Autoregressive
WaveNet [66],SampleRNN [57]
DeepVoice [2],LPCNet [89] ,leaf,text width=15em ] [Non-autoregressive
WaveGlow [72],FloWaveNet [41]
WaveFlow [70],Parallel WaveNet [65]
ClariNet [69],WaveGAN [20]
Parallel WaveGAN [103],MelGAN [45]
GAN-TTS [5],HiFi-GAN [44] ,leaf,text width=15em ] ] ] [End-to-End [Char2Wav
[87],Fastspeech 2s [80]
EATs [21],VITS [40] ,leaf,text width=15em ] ] ]

Figure 2: A taxonomy of TTS.

###  2.2 End-to-End methods

End-to-end TTS models can directly convert the input text or phoneme sequence
into speech audio waveform without any intermediate representations and this
kind of framework admits many straightforward extensions. Fully end-to-end
models can not only reduce the time cost of training and reference, but also
avoid error propagation in cascade models. Moreover, it also requires less
human annotations in the datasets. However, these advantages come with
challenges. The first thing is the huge difference between modalities of text
and waveform. For instance, the temporal resolution of text and waveform is
very different: there are 5 words per second on average during speech, the
length of corresponding phoneme sequence is just around 20. But the sample
rate of speech audio is usually as high as 16000Hz or even higher. Extremely
large difference in temporal resolution makes it difficult to model text-audio
correlations. And the requirement of memory is high because of high temporal
resolution of audio. Besides, the lack of aligned intermediate representations
makes the alignment between text and audio more difficult. In response to
these challenges, researchers have designed a variety of end-to-end TTS
networks, which achieve excellent performance.

Char2Wav [87] is the first end-to-end model based on RNN and various end-to-
end solutions emerged after it was reported. Fastspeech 2s [80] is a GAN-based
non-autoregressive model that uses the feed-forward Transformer [92] block and
1D-convolution as the basic structure for the encoder and decoder. In
addition, the variance adaptor was designed to deal with the one-to-many
problem and facilitate the alignment of text and speech. EATs [21] is a GAN-
based non-autoregressive end-to-end TTS model like Fastspeech 2s. A well-
designed aligner is the key part of the whole framework to obtain the aligned
representation from the input phoneme sequence and decode it into raw audio
waveform. Dynamic time warping is applied to solve the one-to-many problem.
VITS [40] is a TTS model based on flow and Variational Autoencoder(VAE) [42],
which adopts Monotonic Alignment Search(MAS) [39] and stochastic duration
predictor to address the alignment and one-to-many problems respectively and
achieves state-of-the-art performance.

Although there have been many two-stage TTS models with excellent performance,
end-to-end models has become a hot research topic in recent years because of
their efficiency and accuracy. Hence, high-efficiency end-to-end models will
become a trend of future research in TTS area.

##  3 Music generation

In recent years, researchers have begun to try to take advantage of artificial
intelligence generating realistic and aesthetic pieces such as images, text
and videos. Music is an art that reflects the emotions of human beings in real
life. The use of computer technology for algorithmic composition dates back to
the last century. With the development of deep learning technology, generating
music by deep neural networks has become a trending research topic. Using deep
learning technology to generate music is meaningful. In life, people without
any music theory knowledge can also use the technology to generate music to
relax themselves. In the commercial field, large-scale tailer-made music
generation can be realized with low time and economic costs through the
technology. In addition, neural network-based music generation technology has
broad prospects in education, medical and other fields. Hence, it is a
promising research topic.

Different from image generation and text generation, music generation has its
unique challenges. Firstly, music is an art of time [23]. Music is
hierarchical, and there are usually long-term and short-term dependencies
within a piece of music. Therefore, the hierarchical structure and temporal
dependencies in music must be well modeled. Secondly, Music usually consists
of multiple tracks and instruments. It is necessary to organically compose
different tracks and instruments to generate natural, pleasant music. Thirdly,
there are chords and melodies in music, which consist of a series of notes
arranged in a specific order. The formation of chords and melodies plays a
pivotal role in music generation. In summary, the three points mentioned above
should be considered carefully when designing the structure of neural networks
for music generation.

Based on the current research status of neural network-based music generation,
music generation approaches can be divided into two classes: symbolic music
generation and audio music generation. Symbolic music generation means that
music is symbolically generated in the form of a sequence of notes, which
specifies information such as pitch, duration and instrument. These notes
further compose melodies and chords and the generated music is stored in midi
format finally. Taking advantage of this approach, music can be modeled in the
low-dimensional space, which can significantly reduce the difficulty of music
generation task. Moreover, since music is composed of a series of musical
events, no harsh noise is introduced. As for audio music generation, it means
that music is generated at the audio sample point level. The sample rate of
music audio waveform is usually 44100Hz. Therefore, the key bottleneck of
audio music generation is that modeling long-term and short-term dependencies
at such a high temporal resolution. This approach has some advantages.
Firstly, directly generating raw audio waveform of music can avoid a large
number of complicated human annotations of datasets. Secondly, audio music
generation can generate unprecedented sounds and even voice from singer
instead of being limited to several specific musical instruments. We will
introduce research work related to these two approaches below, respectively.

###  3.1 Symbolic music generation

Symbolic music generation means that the model generates a series of musical
events such as note, instrument, pitch and duration and composes them into
music in midi format. Nowadays, some neural network-based symbolic music
generative models have been proposed, which can be divided into two classes
according to the network structure: RNN-based models and CNN-based models. We
will present several typical models in each of these two classes,
respectively.

##### RNN-based models

Many excellent generative models of symbolic-domain music have been proposed
up to now. DeepBach [33] is an RNN-based model, which is capable of generating
highly natural Bach-style chorales. Song from PI [12] is a four-layer
hierarchical RNN model to generate symbolic pop music, with different layers
corresponding to Drum, Chord, Press and Key, respectively. C-RNN-GAN [60] is
the first symbolic-domain music generative model that uses GAN [30]. In the
generator and discriminator of this model, the recurrent network used is the
long short-term memory(LSTM) [37] network in order to model long-term
dependency in symbolic music. HRNN [97] is a three-layer hierarchical RNN
model, which models music from high-level to low-level by outputting bar
profiles, beat profiles and notes in different hierarchical layers. MusicVAE
[81] is a model based on VAE [42]. This work introduced a novel sequential
autoencoder and a hierarchical recurrent decoder to address the difficulty of
modeling sequences with long-term dependency. It is not difficult to see that
most of RNN-based symbolic music generative models design a kind of
hierarchical structure to model the long-term and short-term structures in
music.

##### CNN-based models

The currently proposed CNN-based symbolic music generative models usually take
advantage of GAN. These frameworks adopted convolutional neural network as the
structure of discriminator and generator of GAN network. MidiNet [104] is the
first GAN-based model with CNN as the core structure. MidiNet ingeniously adds
two conditional inputs: 1-D condition and 2-D condition. In detail, 1-D
condition is used to condition chords of the generation, while 2-D condition
is the previous generated bar in order to condition the current bar. The
introduction of these two conditions enhances the naturalness and continuity
of the generated music. MuseGAN [23] is a GAN-based model for multi-track
sequence generation. It proposed three GAN models: Jamming model, Composer
model and Hybrid model. In addition, two models for modeling temporal
relationships were also proposed. MuseGAN combines the aforementioned models
and achieve high performance. Binary MuseGAN [22] improved upon MuseGAN, which
proposed to append to the generator an additional refiner, instead of using
some post-processing approaches such as hard thresholding or Bernoulli
sampling to achieve binary output. The proposed refiner has been proven to be
beneficial to improve the performance of generated music. Compared to RNN-
based models, the computational cost of CNN-based models is usually lower. As
a result, CNN-based symbolic-domain music generative models can be trained
more conveniently.

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=left, font=, rectangle, draw=hiddendraw,
rounded corners, align=left, minimum width=2.5em, inner xsep=4pt, inner
ysep=0pt, , where level=1text width=6em,font=, where level=2text
width=6.9em,font=, where level=3font=,yshift=0.25pt, [Music generation
[Symbolic [RNN-based [DeepBach [33],MusicVAE [81]
C-RNN-GAN [60],HRNN [97]
Song from PI [12] ,leaf,text width=12em ] ] [CNN-based [MidiNet [104],MuseGAN
[23]
MuseGAN [22] ,leaf,text width=12em ] ] ] [Audio [WaveNet [66],SampleRNN [57]
WaveGAN [20],MelGAN [45]
MelNet [91],Jukebox [18] ,leaf,text width=12em ] ] ]

Figure 3: A taxonomy of music generation.

###  3.2 Audio music generation

Different from symbolic music generation, approaches of audio music generation
directly generate music at the raw audio sample point level, which allows for
the generation of more diverse music and even vocals. So far, core structures
of models for audio music generation include but are not limited to CNN and
GAN. In the following, several neural network-based models for audio music
generation will be introduced.

Note that several models used for TTS task are also capable of generating
music in the raw audio domain, such as WaveNet, WaveGAN, MelGAN, MelNet and
SampleRNN [66, 20, 45, 91, 57]. WaveNet [66] is a versatile model that is
competent for tasks such as multi-speaker speech generation, TTS, music
generation and speech recognition. In terms of music generation, WaveNet
adopts dilated casual convolutional layers to enlarge receptive field, which
is crucial to obtain samples that sounded music. SampleRNN [57] can generate
Beethoven-style piano sonatas at raw audio sample point level thanks to its
hierarchy RNN structure. WaveGAN [20] improve the structure of DCGAN [76] to
generate the sound of some instruments such as piano and drum. MelGAN [45] is
a GAN-based model that realizes music generation by decoding mel-spectrograms.
MelNet [91] is also a generative model for music in the frequency domain,
which is based on RNN.

Jukebox [18] is a model dedicated to audio music generation, which is based on
VQ-VAE [78]. Jukebox is so powerful that it can even synthesize the singer’s
vocals. To tackle the long context of raw audio, researchers use three
separate VQ-VAE models with different timescales and make autoregressive
Transformers as the key component of the encoders and decoders. Hence, Jukebox
can generate high-fidelity and diverse songs in the raw audio domain with
coherence up to multiple minutes. But the price is that it requires too much
computational cost. Training a jukebox model requires hundreds of GPUs trained
in parallel for several weeks. Similarly, reference also takes a lot of time
and GPUs.

Although there is less research work on music generation in the raw audio
domain, this area has attracted more and more attention from researchers.
Audio music generative models not only does not require prior musical theory
and a large number of human annotations, but also can generate more diverse
music and even model singer’s vocal. However, audio noise and scratchiness
that accompanies the music generated by these models is something we cannot
ignore. In the future, we have to work more on the reduction of noise in audio
music generative models and their computational cost.

##  4 Audio-visual multimodal processing

Audio-visual multimodal processing means to accurately extract the low-
dimensional features of acoustic and visual information, and then modelling
the correlations between the two features to achieve some tasks such as
lipreading. For computers, solving such problem is still a huge challenge due
to the heterogeneous between audio and video. The audio and visual modalities
are very different from each other in many aspects, for example, dimension and
temporal resolution. In this review, we focus on four tasks related to audio-
visual multimodal processing: lipreading, audio-visual speech separation,
talking face generation and sound generation from video. They will be
comprehensively introduced and summarized below.

###  4.1 Lipreading

Lipreading is an impressive skill to recognize what is being said from silent
videos, and is a notoriously difficult task for humans to perform. Researchers
take advantage of deep learning technology to train models to handle the
lipreading task in recent years. Specifically, neural network-based lipreading
models take silent facial(lip) videos as input, and output corresponding
speech audio or characters. Several applications come to mind for automatic
lipreading model: Enabling videoconferencing within a noisy environment; using
surveillance video as a long-range listening device; facilitating conversation
at a noisy party between people [24]. As a result, it is meaningful to develop
automatic lipreading models, which bring convenience to our daily life.
Several neural network-based lipreading models with high performance will be
introduced below.

WLAS [15] is a lipreading model that can transcribe videos of lip motion to
characters, with or without the audio. CNN, LSTM [37], and Attention mechanism
are adopted to model the correlations between the lip video and audio to
generate the characters. The LRS dataset was also proposed by this work. LiRA
[56] is a self-supervised model, which projects the lip image sequence and
audio waveform into the high-level representation respectively and the loss
between them is calculated in the pretrain stage, then fine-tunes the model to
achieve word-level and sentence-level lip reading. Ephrat et al. [25] proposed
that modeling the task as an acoustic regression problem has many advantages
over visual-to-text modeling. For example, human emotions can be expressed in
acoustic signal. Vid2Speech [24] is a CNN-based model that takes facial image
sequences as input and outputs the corresponding speech raw audio waveform.
[25] proposed a two-tower CNN-based model. One tower inputs facial grayscale
images and the other inputs optical flow computed between facial image frames.
Furthermore, the model adopts both mel-scale spectrograms and linear-scale
spectrograms as the representation of audio waveform. [3, 15] also proposed
models for lipreading task.

###  4.2 Audio-visual speech separation

Humans are capable of focusing their auditory attention on a single sound
source within a noisy environment, which is also known as the “cocktail party
effect”. In recent years, the cocktail problem has become a typical problem in
the field of computer speech recognition. Separating an input audio signal
into its individual speech source is the definition of automatic speech
separation. Ephrat et al. [26] proposed that audio-visual speech separation
can achieve higher performance than audio-only speech separation because
viewing a speaker’s face enhances the model’s capacity to resolve perceived
ambiguity. Automatic speech separation has many applications such as assistive
technology for the hearing impaired and head-mounted assistive devices for
noisy meeting scenarios. In this section, we focus on audio-visual speech
separation and quantity of related work will be introduced below.

[26] is the first attempt at speaker-independent audio-visual speech
separation, which proposed a CNN-based model to encode the input facial images
and speech audio spectrogram and output a complex mask for speech separation.
In addition, the AVspeech dataset was also proposed in this work. AV-CVAE [64]
is an audio-visual speech separation model based on VAE [42], which detects
the lip movements of the speaker and predicts the individual separated speech
audio. Acapella [61] is for audio-visual singing separation. The architecture
is a two-stream CNN, which is called Y-Net for processing audio and video
respectively. The Y-Net takes complex spectrogram and lip-region frames as
input and outputs complex masks. In addition, a large dataset of solo singing
videos was also proposed. VisualSpeech [29] takes a face image, an image
sequence of lip movement and mixed audio as input, and outputs a complex mask.
It also innovatively proposed the cross-modal embedding space to further
correlate the information of audio and visual modalities. Facefilter [16] uses
still images as visual information and [1, 27] also proposed methods for
audio-visual speech separation task.

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=left, font=, rectangle, draw=hiddendraw,
rounded corners, align=left, minimum width=2.5em, inner xsep=4pt, inner
ysep=0pt, , where level=1text width=9.7em,font=, [AV multimodal processing
[Lipreading [WLAS [15],LiRA [56]
Vid2Speech [24],Lipnet [3]
Ephrat et al. [25]
Chung et al. [15] ,leaf,text width=10.2em ] ] [AV speech separation [AV-CVAE
[64]
VisualSpeech [29]
Acapella [61],Facefilter [16]
Ephrat et al. [26]
Afouras et al. [1]
Gabbay et al. [27] ,leaf,text width=10.2em ] ] [Talking face generation [DAVS
[110],ATVGnet [8]
Chung et al. [14]
Zhu et al. [112]
Song et al. [86]
Chen et al. [7] ,leaf,text width=10.2em ] ] [Sound generation [I2S [6],REGNET
[9]
Foley Music [28]
Zhou et al. [111] ,leaf,text width=10.2em ] ] ]

Figure 4: A taxonomy of audio-visual multimodal processing.

###  4.3 Talking face generation

Talking face generation means given a target face image and a clip of
arbitrary speech audio, generating a natural talking face of the target
character saying the given speech with lip synchronization. Meanwhile, the
smooth transition of facial images should be guaranteed. It is a fresh and
interesting task that has been an active topic. Generating talking face is
challenging because the continuously changing facial region depends not only
on visual information(given face image), but also on acoustic
information(given speech audio), as well as achieving lip-speech
synchronization. Meanwhile, it has many potential applications such as
teleconferencing, generating virtual characters with specific facial movement
and enhancing speech comprehension. A lot of work related to talking face
generation task will be introduced below.

In previous efforts, researches performed 3D modeling of specific faces, then
manipulating 3D meshes to generate the talking face. However, this method
strongly relies on 3D modeling, which is time consuming, and cannot be
extended to arbitrary identities. To our best knowledge, [14] is the first
work to use deep generative model for talking face generation. DAVS [110] is
an end-to-end trainable deep neural network for talking face generation. The
network firstly learns the joint audio-visual representation, then uses
strategy of adversarial training to realize the latent space disentangling.
[8] proposed ATVGnet and the architecture of which can be divided into two
parts: audio transformation network(AT-net) and visual generation network(VG-
net) to process the acoustic and visual information respectively. In addition,
a regression-based discriminator and a novel dynamically adjustable pixel-wise
loss with an attention mechanism were also proposed. [112] proposed a novel
talking face generative framework, which discovers audio-visual coherence via
asymmetrical mutual information estimator. [86, 7] also proposed methods for
talking face generation, respectively.

###  4.4 Sound generation from video

The visual and auditory senses are arguably the two most important channels
for humans to perceive the surrounding environment, and they are often closely
associated with each other. From long-term observation of the natural
environment, people can learn the association between audio and vision. In
recent years, researchers have attempt leveraging learnable network to achieve
this. Automatically generating sound from video has many application
scenarios. For instance, combining videos with automatically generated sound
to enhance the environment of virtual reality; adding background sound effects
to videos automatically to avoid a lot of manual work. Several related studies
will be introduced below.

[6] is the first attempt to solve the audio-visual multimodal generation
problem taking advantage of deep generative neural network. Two models named
I2S and S2I based on CNN and GAN were proposed. These two models achieve
sound-to-image and image-to-sound generation tasks, respectively. Moreover,
the corresponding generation result of five representations of raw audio
waveform were compared in the experiment part of this paper. In [111], a
SampleRNN [57]-style model was adopted as the sound generator, while three
variants of the video encoder were proposed and compared. The VEGAS dataset
containing 28109 cleaned videos spanning in 10 categories was also released.
Foley Music [28] is a Graph-Transformer based framework that learns to
generate MIDI music from videos. The human pose features were extracted to
capture the body movement, which was achieved by detecting the key points of
body and hand. A Transformer-style decoder was also adopted. REGNET was
proposed by [9], which introduced an information bottleneck to generate
aligned sound from the given video.

In addition to the tasks mentioned above, leveraging the link between acoustic
and visual information, multimodal learning methods explore a large range of
interesting tasks such as face reconstruction from audio, emotion recognition,
speaker identification. In the future, audio-visual multimodal processing will
receive more attention from researchers within the area of deep learning.

##  5 Datasets

In this section, several datasets that are widely adopted in TTS, music
generation and audio-visual multimodal processing areas will be presented
respectively.

###  5.1 Datasets for TTS

##### LJ speech

The LJ speech dataset [38] is a public speech dataset consists of 13100 short
audio clips from a single speaker reading passages from 7 non-fiction books
with a total length of approximately 24 hours. Clips vary in length from 1 to
10 seconds. The audio format is 16-bit PCM and the sample rate of audio is
22kHz.

##### VCTK

The VCTK dataset [102] includes speech data uttered by 110 English speakers
with different accents. Each speaker reads about 400 sentences from a
newspaper, the rainbow passage and an elicitation paragraph used for the
speech accent archive. The audio format is 16-bit PCM with a sample rate of
44kHz.

##### LibriTTS corpus

The LibriTTS dataset [109] is a multi-speaker English corpus, which consists
approximately 585 hours of read English speech with a sample rate of 24kHz.
The speech is split at sentence breaks and both normalize and original texts
are included. Moreover, utterances with significant background noise are
excluded and contextual information can be extracted.

##### Blizzard

The Blizzard [71] is a dataset for speech synthesis task, which consists of
315 hours English reading voice by a single female actor. The audio format is
16-bit PCM with a sample rate of 16kHz.

###  5.2 Datasets for music generation

##### Lakh MIDI Dataset

The Lakh MIDI Dataset [77] is a collection of 176581 unique midi-format files.
45129 files in this dataset have been matched and aligned to entries in the
Million Song Dataset [4]. The goal of the Lakh MIDI dataset is to facilitate
large-scale music information retrieval, both symbolic and audio content-
based.

##### Lakh Pianoroll Dataset

The Lakh Pianoroll Dataset [23] consists of 174154 multitrack pianorolls,
which are derived from the Lakh MIDI Dataset.

##### MAESTRO

The MAESTRO [35] is a dataset consists of 200 hours of paired audio and MIDI
recordings from ten years of International Piano-e-Competition. Audio and MIDI
files are aligned with about 3ms accuracy and sliced to individual pieces.
Uncompressed audio can achieve CD-quality or higher and the audio format is
16-bit PCM with a sample rate of 44.1kHz-48kHz.

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=left, font=, rectangle, draw=hiddendraw,
rounded corners, align=left, minimum width=2.5em, inner xsep=4pt, inner
ysep=0pt, , where level=1text width=10.5em,font=, [Datasets [Text to speech
[LJ speech [109],VCTK [102]
LibriTTS [109],Blizzard [71] ,leaf,text width=10.2em ] ] [Music generation
[Lakh MIDI [77]
MAESTRO [35]
Lakh Pianoroll [23] ,leaf,text width=10em ] ] [Audio-visual multimodal [GRID
[17],LRW [13]
TCD-TIMID [34],LRS [15]
VoxCeleb [63] ,leaf,text width=10em ] ] ]

Figure 5: A taxonomy of datasets.

###  5.3 Datasets for audio-visual multimodal processing

##### GRID

The GRID audio-visual sentence corpus [17] is a large dataset of audio and
facial video, which includes 1000 sentences spoken by 34 talkers. Talkers
include 18 men and 16 women. Each sentence consists of a six-word sequence and
the total vocabulary of this dataset is 51.

##### LRW

The LRW dataset [13] consists of about 1000 utterances containing 500
different words spoken by hundreds of different speakers. All videos in the
dataset are 29 frames(1.16 second) in length and the word occurs in the middle
of the video. The dataset is divided into training set, validation set and
test set.

##### TCD-TIMID

The TCD-TIMID dataset [34] is widely used in audio-video speech processing
research, which consists of high-quality audio-video utterances from around 60
speakers. Every speaker uttering 98 sentences.

##### LRS

The LRS dataset [15] is a dataset for visual speech recognition, which
consists of over 100000 natural sentences from British television.

##### VoxCeleb

The VoxCeleb dataset [63] is a large scale text-independent speaker
identification dataset, which contains over 100000 utterances for 1251
celebrities. The videos are extracted from YouTube website. Speakers include
various races, accents, ages and occupations, with a balanced gender
distribution. The audio format is 16bit-PCM with a sample rate of 16 kHz.

##  6 Conclusion

In this review, we paid attention to the work on audio synthesis and audio-
visual multimodal processing. The text-to-speech(TTS) and music generation
tasks, which plays a crucial role in the maintenance of the audio synthesis
field, were comprehensively summarized respectively. In the TTS task, two-
stage and end-to-end methods were distinguished and introduced. As for the
music generation task, symbolic domain and raw audio domain generative models
were presented respectively. In the field of audio-visual multimodal
processing, we focused on four typical tasks: lipreading, audio-visual speech
separation, talking face generation and sound generation from video. The
frameworks related to these tasks were introduced. Finally, several widely
adopted datasets were also presented. Overall, this review provides
considerable guidance to relevant researchers.

## References

  * Afouras et al. [2018] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.  The conversation: Deep audio-visual speech enhancement.  _arXiv preprint arXiv:1804.04121_ , 2018.
  * Arık et al. [2017] Sercan Ö Arık, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al.  Deep voice: Real-time neural text-to-speech.  In _International Conference on Machine Learning_ , pages 195–204. PMLR, 2017.
  * Assael et al. [2016] Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas.  Lipnet: End-to-end sentence-level lipreading.  _arXiv preprint arXiv:1611.01599_ , 2016.
  * Bertin-Mahieux et al. [2011] Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere.  The million song dataset.  2011\.
  * Bińkowski et al. [2019] Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan.  High fidelity speech synthesis with adversarial networks.  In _International Conference on Learning Representations_ , 2019.
  * Chen et al. [2017] Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu.  Deep cross-modal audio-visual generation.  In _Proceedings of the on Thematic Workshops of ACM Multimedia 2017_ , pages 349–357, 2017.
  * Chen et al. [2018] Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu.  Lip movements generation at a glance.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 520–535, 2018.
  * Chen et al. [2019] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7832–7841, 2019.
  * Chen et al. [2020a] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan.  Generating visually aligned sound from videos.  _IEEE Transactions on Image Processing_ , 29:8292–8302, 2020a.
  * Chen et al. [2020b] Xiaoyu Chen, Hongliang Li, Qingbo Wu, King Ngi Ngan, and Linfeng Xu.  High-quality r-cnn object detection using multi-path detection calibration network.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 31(2):715–727, 2020b.
  * Chen et al. [2021] Xiaoyu Chen, Hongliang Li, Qingbo Wu, Fanman Meng, and Heqian Qiu.  Bal-r2cnn: High quality recurrent object detection with balance optimization.  _IEEE Transactions on Multimedia_ , 2021.
  * Chu et al. [2016] Hang Chu, Raquel Urtasun, and Sanja Fidler.  Song from pi: A musically plausible network for pop music generation.  _arXiv preprint arXiv:1611.03477_ , 2016.
  * Chung and Zisserman [2016] Joon Son Chung and Andrew Zisserman.  Lip reading in the wild.  In _Asian conference on computer vision_ , pages 87–103. Springer, 2016.
  * Chung et al. [2017a] Joon Son Chung, Amir Jamaludin, and Andrew Zisserman.  You said that?  _arXiv preprint arXiv:1705.02966_ , 2017a.
  * Chung et al. [2017b] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman.  Lip reading sentences in the wild.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3444–3453. IEEE, 2017b.
  * Chung et al. [2020] Soo-Whan Chung, Soyeon Choe, Joon Son Chung, and Hong-Goo Kang.  Facefilter: Audio-visual speech separation using still images.  _arXiv preprint arXiv:2005.07074_ , 2020.
  * Cooke et al. [2006] Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao.  An audio-visual corpus for speech perception and automatic speech recognition.  _The Journal of the Acoustical Society of America_ , 120(5):2421–2424, 2006.
  * Dhariwal et al. [2020] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.  Jukebox: A generative model for music.  _arXiv preprint arXiv:2005.00341_ , 2020.
  * Ding et al. [2020] Jisheng Ding, Linfeng Xu, Jiangtao Guo, and Shengxuan Dai.  Human detection in dense scene of classrooms.  In _2020 IEEE International Conference on Image Processing (ICIP)_ , pages 618–622. IEEE, 2020.
  * Donahue et al. [2018] Chris Donahue, Julian McAuley, and Miller Puckette.  Adversarial audio synthesis.  In _International Conference on Learning Representations_ , 2018.
  * Donahue et al. [2020] Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan.  End-to-end adversarial text-to-speech.  In _International Conference on Learning Representations_ , 2020.
  * Dong and Yang [2018] Hao-Wen Dong and Yi-Hsuan Yang.  Convolutional generative adversarial networks with binary neurons for polyphonic music generation.  _arXiv preprint arXiv:1804.09399_ , 2018.
  * Dong et al. [2018] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang.  Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018\.
  * Ephrat and Peleg [2017] Ariel Ephrat and Shmuel Peleg.  Vid2speech: speech reconstruction from silent video.  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5095–5099. IEEE, 2017.
  * Ephrat et al. [2017] Ariel Ephrat, Tavi Halperin, and Shmuel Peleg.  Improved speech reconstruction from silent video.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , pages 455–462, 2017.
  * Ephrat et al. [2018] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.  _arXiv preprint arXiv:1804.03619_ , 2018.
  * Gabbay et al. [2017] Aviv Gabbay, Asaph Shamir, and Shmuel Peleg.  Visual speech enhancement.  _arXiv preprint arXiv:1711.08789_ , 2017.
  * Gan et al. [2020] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba.  Foley music: Learning to generate music from videos.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pages 758–775. Springer, 2020.
  * Gao and Grauman [2021] Ruohan Gao and Kristen Grauman.  Visualvoice: Audio-visual speech separation with cross-modal consistency.  _arXiv preprint arXiv:2101.03149_ , 2021.
  * Goodfellow et al. [2014] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  _Advances in Neural Information Processing Systems_ , 3:2672–2680, 2014.
  * Griffin and Lim [1984] Daniel Griffin and Jae Lim.  Signal estimation from modified short-time fourier transform.  _IEEE Transactions on acoustics, speech, and signal processing_ , 32(2):236–243, 1984.
  * Guo et al. [2020] Jiangtao Guo, Linfeng Xu, Jisheng Ding, Bin He, Shengxuan Dai, and Fangyu Liu.  A deep supervised edge optimization algorithm for salt body segmentation.  _IEEE Geoscience and Remote Sensing Letters_ , 2020.
  * Hadjeres et al. [2017] Gaëtan Hadjeres, François Pachet, and Frank Nielsen.  Deepbach: a steerable model for bach chorales generation.  In _International Conference on Machine Learning_ , pages 1362–1371. PMLR, 2017.
  * Harte and Gillen [2015] Naomi Harte and Eoin Gillen.  Tcd-timit: An audio-visual corpus of continuous speech.  _IEEE Transactions on Multimedia_ , 17(5):603–615, 2015.
  * Hawthorne et al. [2018] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck.  Enabling factorized piano music modeling and generation with the maestro dataset.  _arXiv preprint arXiv:1810.12247_ , 2018.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Ito and Johnson [2017] Keith Ito and Linda Johnson.  The lj speech dataset, 2017.
  * Kim et al. [2020] Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon.  Glow-tts: A generative flow for text-to-speech via monotonic alignment search.  _arXiv preprint arXiv:2005.11129_ , 2020.
  * Kim et al. [2021] Jaehyeon Kim, Jungil Kong, and Juhee Son.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  _arXiv preprint arXiv:2106.06103_ , 2021.
  * Kim et al. [2018] Sungwon Kim, Sang-gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon.  Flowavenet: A generative flow for raw audio.  _arXiv preprint arXiv:1811.02155_ , 2018.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma et al. [2016] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved variational inference with inverse autoregressive flow.  _Advances in neural information processing systems_ , 29:4743–4751, 2016.
  * Kong et al. [2020] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Kumar et al. [2019] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron Courville.  Melgan: Generative adversarial networks for conditional waveform synthesis.  _arXiv preprint arXiv:1910.06711_ , 2019.
  * Li et al. [2020a] Hui Li, Qingbo Wu, King Ngi Ngan, Hongliang Li, and Fanman Meng.  Region adaptive two-shot network for single image dehazing.  In _2020 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6. IEEE, 2020a.
  * Li et al. [2020b] Hui Li, Qingbo Wu, Haoran Wei, King Ngi Ngan, Hongliang Li, Fanman Meng, and Linfeng Xu.  Haze-robust image understanding via context-aware deep feature refinement.  In _2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)_ , pages 1–6. IEEE, 2020b.
  * Li et al. [2021] Hui Li, Qingbo Wu, King Ngi Ngan, Hongliang Li, and Fanman Meng.  Single image dehazing via region adaptive two-shot network.  _IEEE MultiMedia_ , 2021.
  * Li et al. [2019a] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu.  Neural speech synthesis with transformer network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6706–6713, 2019a.
  * Li et al. [2018] Wei Li, Qingbo Wu, Linfeng Xu, and Chao Shang.  Incremental learning of single-stage detectors with mining memory neurons.  In _2018 IEEE 4th International Conference on Computer and Communications (ICCC)_ , pages 1981–1985. IEEE, 2018.
  * Li et al. [2019b] Wei Li, Hongliang Li, Qingbo Wu, Xiaoyu Chen, and King Ngi Ngan.  Simultaneously detecting and counting dense vehicles from drone images.  _IEEE Transactions on Industrial Electronics_ , 66(12):9651–9662, 2019b.
  * Li et al. [2019c] Wei Li, Hongliang Li, Qingbo Wu, Fanman Meng, Linfeng Xu, and King Ngi Ngan.  Headnet: An end-to-end adaptive relational network for head detection.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 30(2):482–494, 2019c.
  * Li et al. [2020c] Wei Li, Zhenting Wang, Xiao Wu, Ji Zhang, Qiang Peng, and Hongliang Li.  Codan: Counting-driven attention network for vehicle detection in congested scenes.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 73–82, 2020c.
  * Luo et al. [2021] Hao Luo, Hanxiao Luo, Qingbo Wu, King Ngi Ngan, Hongliang Li, Fanman Meng, and Linfeng Xu.  Single image deraining via multi-scale gated feature enhancement network.  In _Digital TV and Wireless Multimedia Communication: 17th International Forum, IFTC 2020, Shanghai, China, December 2, 2020: Revised Selected Papers_ , volume 1390, page 73. Springer Nature, 2021.
  * Luo et al. [2018] Kunming Luo, Fanman Meng, Qingbo Wu, and Hongliang Li.  Weakly supervised semantic segmentation by multiple group cosegmentation.  In _2018 IEEE Visual Communications and Image Processing (VCIP)_ , pages 1–4. IEEE, 2018.
  * Ma et al. [2021] Pingchuan Ma, Rodrigo Mira, Stavros Petridis, Björn W Schuller, and Maja Pantic.  Lira: Learning visual speech representations from audio through self-supervision.  _arXiv preprint arXiv:2106.09171_ , 2021.
  * Mehri et al. [2016] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio.  Samplernn: An unconditional end-to-end neural audio generation model.  _arXiv preprint arXiv:1612.07837_ , 2016.
  * Meng et al. [2019] Fanman Meng, Lili Guo, Qingbo Wu, and Hongliang Li.  A new deep segmentation quality assessment network for refining bounding box based segmentation.  _IEEE Access_ , 7:59514–59523, 2019.
  * Miao et al. [2020] Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao.  Flow-tts: A non-autoregressive network for text to speech based on flow.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7209–7213. IEEE, 2020.
  * Mogren [2016] Olof Mogren.  C-rnn-gan: Continuous recurrent neural networks with adversarial training.  _arXiv preprint arXiv:1611.09904_ , 2016.
  * Montesinos et al. [2021] Juan F Montesinos, Venkatesh S Kadandale, and Gloria Haro.  A cappella: Audio-visual singing voice separation.  _arXiv preprint arXiv:2104.09946_ , 2021.
  * Morise et al. [2016] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa.  World: a vocoder-based high-quality speech synthesis system for real-time applications.  _IEICE TRANSACTIONS on Information and Systems_ , 99(7):1877–1884, 2016.
  * Nagrani et al. [2017] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  Voxceleb: a large-scale speaker identification dataset.  _arXiv preprint arXiv:1706.08612_ , 2017.
  * Nguyen et al. [2020] Viet-Nhat Nguyen, Mostafa Sadeghi, Elisa Ricci, and Xavier Alameda-Pineda.  Deep variational generative models for audio-visual speech separation.  _arXiv preprint arXiv:2008.07191_ , 2020.
  * Oord et al. [2018] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al.  Parallel wavenet: Fast high-fidelity speech synthesis.  In _International conference on machine learning_ , pages 3918–3926. PMLR, 2018.
  * Oord et al. [2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Peng et al. [2020] Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao.  Non-autoregressive neural text-to-speech.  In _International conference on machine learning_ , pages 7586–7598. PMLR, 2020.
  * Ping et al. [2017] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan Ömer Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller.  Deep voice 3: 2000-speaker neural text-to-speech.  2017\.
  * Ping et al. [2018] Wei Ping, Kainan Peng, and Jitong Chen.  Clarinet: Parallel wave generation in end-to-end text-to-speech.  _arXiv preprint arXiv:1807.07281_ , 2018.
  * Ping et al. [2020] Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song.  Waveflow: A compact flow-based model for raw audio.  In _International Conference on Machine Learning_ , pages 7706–7716. PMLR, 2020.
  * Prahallad et al. [2013] Kishore Prahallad, Anandaswarup Vadapalli, Naresh Elluru, Gautam Mantena, Bhargav Pulugundla, Peri Bhaskararao, Hema A Murthy, Simon King, Vasilis Karaiskos, and Alan W Black.  The blizzard challenge 2013–indian language task.  In _Blizzard challenge workshop_ , volume 2013, 2013.
  * Prenger et al. [2019] Ryan Prenger, Rafael Valle, and Bryan Catanzaro.  Waveglow: A flow-based generative network for speech synthesis.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 3617–3621. IEEE, 2019.
  * Qiu et al. [2019] Heqian Qiu, Hongliang Li, Qingbo Wu, Fanman Meng, King Ngi Ngan, and Hengcan Shi.  A2rmnet: Adaptively aspect ratio multi-scale network for object detection in remote sensing images.  _Remote Sensing_ , 11(13):1594, 2019.
  * Qiu et al. [2020a] Heqian Qiu, Hongliang Li, Qingbo Wu, Fanman Meng, Linfeng Xu, King Ngi Ngan, and Hengcan Shi.  Hierarchical context features embedding for object detection.  _IEEE Transactions on Multimedia_ , 22(12):3039–3050, 2020a.
  * Qiu et al. [2020b] Heqian Qiu, Hongliang Li, Qingbo Wu, and Hengcan Shi.  Offset bin classification network for accurate object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13188–13197, 2020b.
  * Radford et al. [2015] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_ , 2015.
  * Raffel [2016] Colin Raffel.  _Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching_.  Columbia University, 2016.
  * Razavi et al. [2019] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In _Advances in neural information processing systems_ , pages 14866–14876, 2019.
  * Ren et al. [2019] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech: Fast, robust and controllable text to speech.  _arXiv preprint arXiv:1905.09263_ , 2019.
  * Ren et al. [2020] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_ , 2020.
  * Roberts et al. [2018] Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck.  A hierarchical latent vector model for learning long-term structure in music.  In _International conference on machine learning_ , pages 4364–4373. PMLR, 2018.
  * Shang et al. [2019] Chao Shang, Qingbo Wu, Fanman Meng, and Linfeng Xu.  Instance segmentation by learning deep feature in embedding space.  In _2019 IEEE International Conference on Image Processing (ICIP)_ , pages 2444–2448. IEEE, 2019.
  * Shen et al. [2018] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4779–4783. IEEE, 2018.
  * Shi et al. [2018] Hengcan Shi, Hongliang Li, Fanman Meng, and Qingbo Wu.  Key-word-aware network for referring expression image segmentation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 38–54, 2018.
  * Shi et al. [2020] Hengcan Shi, Hongliang Li, Qingbo Wu, and King Ngi Ngan.  Query reconstruction network for referring expression image segmentation.  _IEEE Transactions on Multimedia_ , 23:995–1007, 2020.
  * Song et al. [2018] Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, and Hairong Qi.  Talking face generation by conditional recurrent adversarial network.  _arXiv preprint arXiv:1804.04786_ , 2018.
  * Sotelo et al. [2017] Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio.  Char2wav: End-to-end speech synthesis.  2017\.
  * Tan et al. [2021] Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu.  A survey on neural speech synthesis.  _arXiv preprint arXiv:2106.15561_ , 2021.
  * Valin and Skoglund [2019] Jean-Marc Valin and Jan Skoglund.  Lpcnet: Improving neural speech synthesis through linear prediction.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5891–5895. IEEE, 2019.
  * Valle et al. [2020] Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro.  Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis.  _arXiv preprint arXiv:2005.05957_ , 2020.
  * Vasquez and Lewis [2019] Sean Vasquez and Mike Lewis.  Melnet: A generative model for audio in the frequency domain.  _arXiv preprint arXiv:1906.01083_ , 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Wang et al. [2020] Lanxiao Wang, Chao Shang, Heqian Qiu, Taijin Zhao, Benliu Qiu, and Hongliang Li.  Multi-stage tag guidance network in video caption.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 4610–4614, 2020.
  * Wang et al. [2017] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al.  Tacotron: Towards end-to-end speech synthesis.  _Proc. Interspeech 2017_ , pages 4006–4010, 2017.
  * Wei et al. [2020] Haoran Wei, Qingbo Wu, Hui Li, King Ngi Ngan, Hongliang Li, and Fanman Meng.  Single image dehazing via artificial multiple shots and multidimensional context.  In _2020 IEEE International Conference on Image Processing (ICIP)_ , pages 1023–1027. IEEE, 2020.
  * Wei et al. [2021] Haoran Wei, Qingbo Wu, Hui Li, King Ngi Ngan, Hongliang Li, Fanman Meng, and Linfeng Xu.  Non-homogeneous haze removal via artificial scene prior and bidimensional graph reasoning.  _arXiv preprint arXiv:2104.01888_ , 2021.
  * Wu et al. [2019a] Jian Wu, Changran Hu, Yulong Wang, Xiaolin Hu, and Jun Zhu.  A hierarchical recurrent neural network for symbolic melody generation.  _IEEE transactions on cybernetics_ , 50(6):2749–2757, 2019a.
  * Wu et al. [2019b] Qingbo Wu, Lei Wang, King N Ngan, Hongliang Li, and Fanman Meng.  Beyond synthetic data: A blind deraining quality assessment metric towards authentic rain image.  In _2019 IEEE International Conference on Image Processing (ICIP)_ , pages 2364–2368. IEEE, 2019b.
  * Wu et al. [2020a] Qingbo Wu, Li Chen, King Ngi Ngan, Hongliang Li, Fanman Meng, and Linfeng Xu.  A unified single image de-raining model via region adaptive coupled network.  In _2020 IEEE International Conference on Visual Communications and Image Processing (VCIP)_ , pages 1–4. IEEE, 2020a.
  * Wu et al. [2020b] Qingbo Wu, Lei Wang, King Ngi Ngan, Hongliang Li, Fanman Meng, and Linfeng Xu.  Subjective and objective de-raining quality assessment towards authentic rain image.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 30(11):3883–3897, 2020b.
  * Xu et al. [2019] Xiaolong Xu, Fanman Meng, Hongliang Li, Qingbo Wu, Yuwei Yang, and Shuai Chen.  Bounding box based annotation generation for semantic segmentation by boundary detection.  In _2019 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS)_ , pages 1–2. IEEE, 2019.
  * Yamagishi et al. [2019] Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al.  Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).  2019\.
  * Yamamoto et al. [2020] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim.  Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6199–6203. IEEE, 2020.
  * Yang et al. [2017] Li-Chia Yang, Szu-Yu Chou, and Yi-Hsuan Yang.  Midinet: A convolutional generative adversarial network for symbolic-domain music generation.  _arXiv preprint arXiv:1703.10847_ , 2017.
  * Yang et al. [2020a] Longrong Yang, Hongliang Li, Qingbo Wu, Fanman Meng, and King Ngi Ngan.  Mono is enough: Instance segmentation from single annotated sample.  In _2020 IEEE International Conference on Visual Communications and Image Processing (VCIP)_ , pages 120–123. IEEE, 2020a.
  * Yang et al. [2020b] Longrong Yang, Fanman Meng, Hongliang Li, Qingbo Wu, and Qishang Cheng.  Learning with noisy class labels for instance segmentation.  In _European Conference on Computer Vision_ , pages 38–53. Springer, 2020b.
  * Yang et al. [2019] Yuwei Yang, Fanman Meng, Hongliang Li, King N Ngan, and Qingbo Wu.  A new few-shot segmentation network based on class representation.  In _2019 IEEE Visual Communications and Image Processing (VCIP)_ , pages 1–4. IEEE, 2019.
  * Yang et al. [2020c] Yuwei Yang, Fanman Meng, Hongliang Li, Qingbo Wu, Xiaolong Xu, and Shuai Chen.  A new local transformation module for few-shot segmentation.  In _International Conference on Multimedia Modeling_ , pages 76–87. Springer, 2020c.
  * Zen et al. [2019] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu.  Libritts: A corpus derived from librispeech for text-to-speech.  _arXiv preprint arXiv:1904.02882_ , 2019.
  * Zhou et al. [2019] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang.  Talking face generation by adversarially disentangled audio-visual representation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 9299–9306, 2019.
  * Zhou et al. [2018] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg.  Visual to sound: Generating natural sound for videos in the wild.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3550–3558, 2018.
  * Zhu et al. [2018] Hao Zhu, Huaibo Huang, Yi Li, Aihua Zheng, and Ran He.  Arbitrary talking face generation via attentional audio-visual coherence learning.  _arXiv preprint arXiv:1812.06589_ , 2018.