# A Review on Methods and Applications in Multimodal Deep Learning

Jabeen Summaira<EMAIL_ADDRESS>,  Xi Li<EMAIL_ADDRESS>College of
Computer Science, Zhejiang University, ChinaP.O. Box W-99Hangzhou310027 ,
Amin Muhammad Shoib  School of Software Engineering, East China Normal
University3663 North Zhongshan Road.ShanghaiChina<EMAIL_ADDRESS>and  Jabbar Abdul<EMAIL_ADDRESS>College of Computer Science, Zhejiang
University, ChinaP.O. Box W-99Hangzhou310027

###### Abstract.

Deep Learning has implemented a wide range of applications and has become
increasingly popular in recent years. The goal of multimodal deep learning
(MMDL) is to create models that can process and link information using various
modalities. Despite the extensive development made for unimodal learning, it
still cannot cover all the aspects of human learning. Multimodal learning
helps to understand and analyze better when various senses are engaged in the
processing of information. This paper focuses on multiple types of modalities,
i.e., image, video, text, audio, body gestures, facial expressions, and
physiological signals. Detailed analysis of the baseline approaches and an in-
depth study of recent advancements during the last five years (2017 to 2021)
in multimodal deep learning applications has been provided. A fine-grained
taxonomy of various multimodal deep learning methods is proposed, elaborating
on different applications in more depth. Lastly, main issues are highlighted
separately for each domain, along with their possible future research
directions.

Deep Learning, Multimedia, Multimodal learning, datasets, Neural Networks,
Survey

††ccs: Computing methodologies Machine learning††ccs: Information systems
Multimedia and multimodal retrieval

##  1\. Introduction:

Multimodal learning proposes that we are able to remember and understand more
when engaging multiple senses during the learning process. MMDL technically
contains different aspects and challenges like representation, translation,
alignment, fusion, co-learning when learning from two or more modalities
(Cukurova et al., 2020; Hong et al., 2020). This information from multiple
sources is contextually related and occasionally provides the additional
necessary information to one another, revealing features that would not be
viewable when working with individual modalities. MMDL models combine
heterogeneous data from multiple sources, allowing for more appropriate
predictions (Sengupta et al., 2020). Extracting and presenting relevant
information from multimodal data remains an inspirational motive for MMDL
research. Merging various modalities to optimize effectiveness is still an
appealing challenge. Furthermore, the accuracy and flexibility of multimodal
systems are not optimum due to the insufficiency of labeled data.

The recent advances and trends of MMDL are from Audio-visual speech
recognition (AVSR) (Yuhas et al., 1989), multimedia content indexing and
retrieval (Atrey et al., 2010; Snoek and Worring, 2005), understanding human
multimodal behaviors during social interaction, multimodal emotion recognition
(Chen et al., 2021b; Boateng, 2020), image and video captioning (Bai and An,
2018; Lei and Huang, 2021), Visual Question-Answering (VQA) (Long et al.,
2021), multimedia retrieval (Souza et al., 2021) to health analysis (Yazdavar
et al., 2020) and so on. In this article, we analyzed the latest MMDL models
to propose typical models and techniques for advancing the field forward.
Various modalities, i.e., image, video, text, audio, body gestures, facial
expressions, and physiological signals, are focused. The main goal of MMDL is
to construct a model that can process information from different modalities
and relate it. MMDL methods and applications are categorized into multiple
groups, and the impact of feature extractor, deep learning architecture,
datasets, and evaluation metrics are analyzed for each group. Moreover, key
features of each model are also discussed to highlight the main contribution
of the work model.

###  1.1. Contribution and Relevance to other surveys:

Recently , numerous surveys are already published relating to the topic of
multimodal learning. A summarized list of these review articles is presented
and analyzed in Table 1. A summary of modalities used and applications
discussed is shown in the table. Our contributions to this article are
described in section 1.1.1. Most of the baseline surveys only focus on models
using Image, Video, Text, and Audio modalities, while our survey paper is
focused on some additional models and groups of applications using body
gestures, facial expressions, and physiological signals. Comparative analysis
on experimental results based on feature extractors, architectures, standard
datasets, and evaluation metrics are also presented to elaborate the
performance of various models.

####  1.1.1. Our contributions:

All these literature surveys provide a review on only a specific domain of
multimodal learning. Some authors only discussed methods and applications of
representation learning and some on fusion learning. Others discussed
representation and fusion learning methods together using few modalities like
vision or text. As mentioned earlier, Mogadala et al. (Mogadala et al., 2019)
explained different models of MMDL using image, video and text modalities.
However, we have taken one step further; along with these modalities, this
paper provides a technical review of various models using audio, body
gestures, facial expressions and physiological signals modalities. Our primary
focus is distinctive in that we seek to survey the literature from up-to-date
deep learning concepts using more modalities. The main contributions of our
article are listed below:

Table 1. Analysis of baseline literature surveys, Where I=Image, T=Text, A=Audio, V=Video, BG=Body Gesture, FE=Facial Expression, PS=Physiological Signals, Med=DNA & MRI |  |  | Modalities |
---|---|---|---|---
Paper |  Year |  Pub. | I | T | A | V | BG | FE | PS | Med |  Applications
C. Zhang et al. (Zhang et al., 2020b) |  2020 |  IEEE | ✓ | ✓ |  | ✓ |  |  |  |  |  Image Description, Image-to-Text generation, VQA
Y. Bisk et al. (Bisk et al., 2020) |  2020 |  arXiv | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  Not Available
J. Gao et al. (Gao et al., 2020) |  2020 |  NC(MIT) | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  Not Available
A. Mogadala et al. (Mogadala et al., 2019) |  2019 |  arXiv | ✓ | ✓ |  | ✓ |  |  |  |  |  Caption Generation, Storytelling, QA. Dialogue, Reasoning, Referring Expression, Entailment, Visual Generations, Navigation, Machine Translation
SF Zhang et al. (Zhang et al., 2019b) |  2019 |  IEEE | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  Multimodal matching, Multimodal classification, Multimodal interaction, Multimedia content indexing, Multimedia Content retrieval, Multimedia description
W Guo et al. (Guo et al., 2019b) |  2019 |  IEEE | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  Video Classification, Event Detection, VQA, Text-to-image synthesis, Transfer Loading
T Baltrušaitis et al. (Baltrušaitis et al., 2018) |  2019 |  IEEE | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  Speech recognition, Media Description, Multimedia Retrieval
Y Li et al. (Li et al., 2018) |  2019 |  IEEE | ✓ | ✓ | ✓ | ✓ |  |  |  |  |  Cross Media Retrieval, NLP, Video Analysis, Recommended System
D Ramachandram et al. (Ramachandram and Taylor, 2017) |  2017 |  IEEE | ✓ | ✓ | ✓ | ✓ |  |  |  | ✓ |  Human Activity Recognition, Medical Applications, Autonomous Systems
Ours |  - |  - | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  Multimodal Image Description, Multimodal Video Description, Multimodal VQA, Multimodal Speech Synthesis, Multimodal Emotion Recognition, Multimodal Event Detection

  * •

We propose a novel fine-grained taxonomy of various MMDL applications, which
elaborates different groups of applications in more depth.

  * •

We provide an explicit overview of various models using the modalities of
images, videos, text, audio, body gestures, facial expressions, and
physiological signals.

  * •

We analyze and summarize the recent baseline MMDL models with the perspective
of architecture, multimedia, dataset and evaluation metric used along with
model features, which provide substantial guidance for future research
inquiries.

  * •

We provide a comparative analysis of MMDL models on benchmark datasets and
evaluation metrics. Comparative analysis unique representation helps
researchers to get directions in future research to improve domain performance
using specific or combination of feature extractors and architectures.

  * •

We highlights the main challenges of MMDL applications, open research
problems, and their possible future research directions.

###  1.2. Structure of Survey:

The rest of the article is structured as follows. Some background of MMDL is
discussed in section 2. Section 3 presents various MMDL methods and
applications. Comparative analysis on experimental results of MMDL application
is discussed in section 4. In section 5, we provided a brief discussion and
propose future research directions in this active area. Finally, section 6
concludes the survey paper. Due to page limits details of related MMDL
architecture, and Dataset and evaluation metrics are provided in Section
1111Section 1 in supplementary meterials contains the complete details about
architectures used in MMDL methods., and 2222Section 2 in supplementary
materials contains the details of datasets & evaluation metrics used in MMDL
methods. of supplementary material. Structure of survey article is shown in
Figure 1.

Figure 1. Structure of Survey article.

##  2\. Background:

A lot of ML approaches have been developed over the past decade to deal with
multimodal data. Multimodal machine learning leads to a wide range of
applications: from Audio-visual speech recognition, multimedia content
indexing and retrieval, understanding human multimodal behaviors, emotion
recognition, multimodal affect recognition, image and video captioning, VQA,
multimedia retrieval, to health analysis and so on. A list of abbreviations is
presented in the Appendix A.

Multimodal applications came into existence in the 1970s and were categorized
into four eras (Morency, [n. d.]). This section discusses the core challenges
of multimodal learning, and the description of four eras is explained further.
In this paper, we concentrate mainly on the deep learning era for multimodal
applications.

###  2.1. Challenges of multimodal learning:

The research domain of MMDL presents some specific challenges for researchers
due to heterogeneous data. Learning from multiple sources allows recording
correspondences among various modalities and an in-depth comprehension of
natural phenomena. In multimodal learning, extracting and combining features
from multiple sources contributes to making larger-scale predictions. Data
from various sources are contextually linked and occasionally give
supplementary information to one another, revealing patterns that would not be
noticeable when functioning with individual modalities separately. Such
systems combine disparate, heterogeneous data from multiple sensors, allowing
for more accurate predictions. Five main technical challenges of multimodal
deep learning are summarized in this section, which is listed below.

  * •

Multimodal Representation: The process of representing information from
several media in a tensor or vector form is known as multimodal
representation. As information from several media frequently contains both
redundant and complementary data, the goal is to represent data in a
meaningful and efficient manner. Dealing with missing data, various levels of
noise, and data combinations from several media are all challenges linked with
multimodal representation. Joint and Coordinated representations are used to
deal with these representation challenges.

  * •

Multimodal Translation: The problem of translating or mapping information from
one modality to another is addressed by multimodal translation. Consider the
tasks like producing an image from a caption or producing a caption for an
image. Due to data heterogeneity, it is not possible to create one perfect
description of an image. One of the most challenging aspects of multimodal
translation is assessing translation quality. The results quality for speech
synthesis and video or image description is highly subjective, and there is
frequently no single correct translation.

  * •

Multimodal Alignment: The process of developing correspondence among
information from two or more media for the same event is known as multimodal
alignment. A framework must measure similarities among different media and
cope with long-term dependencies to align them. Other challenges in the
alignment task include the formation of better similarity metrics, the lack of
annotated datasets, and various correct alignments.

  * •

Multomodal Fusion: The process of combining data from two or more media to
perform regression or classification is known as multimodal fusion. Sentiment
analysis is one of the commonly known examples of multimodal fusion; three
modalities (Visual, acoustic, and language) combine to predict sentiment.

  * •

Multimodal Co-learning: The process of transmitting information/knowledge
among modalities is addressed by multimodal co-learning. Transmitting
information/knowledge from a resource-rich modality is very effective for
creating a model in a limited resource modality with noisy inputs, a lack of
annotated data, and unreliable labels. Parallel, non-parallel, and hybrid co-
learning approaches are used to transfer knowledge between modalities.

Figure 2. Taxonomy diagram of Multimodal Deep Learning Applications. Where
MMDL=Multimodal Deep Learning, EDID=Encoder-Decoder based Image Description,
SCID=Semantic Concept-based Image Description, AID=Attention-based Image
Description, DRL=Deep Reinforcement Learning, MMJEM=Multimodal Joint-Embedding
Models, MMAM=Multimodal Attention-based Models, MMEKM=Multimodal External
Knowledge Bases Models, DLTTS=Deep Learning Text To Speech, MMER=Multimodal
Event Recognition, and MMED=Multimodal Emotion Detection.

###  2.2. Eras of multimodal learning:

Prior research on multimodal learning can be categorized into four eras,
listed below.

  * •

Behavioral Era (BE): The BE starts from the 1970s to the late 1980s. During
this era, A Lazarus (Lazarus, 1973) proposed Multimodal behavior therapy based
on seven different personality dimensions using inter-related modalities. M.
Mulligan & M. Shaw (Mulligan and Shaw, 1980) presents multimodal signal
detection, wherein the signal detection task, the pooling of information is
examined from other modalities, i.e., auditory and visual stimuli. L.E.
Bahrick (Bahrick, 1983) found that infants detected bimodal temporal structure
specifying object elasticity and rigidity and temporal synchrony between
sights and sound of object. D. Hoffman-Plotkin (Hoffman-Plotkin and Twentyman,
1984) proposed a “Multimodal assessment of behavioral and cognitive deficits
in abused and neglected pre-schoolers”.

  * •

Computational Era (CE): The CE starts from the late 1980s to 2000. During this
era, E.D. Petajan (Petajan, 1985) proposed an “Automatic Lipreading to Enhance
Speech Recognition.” B.P. Yuhas et al. (Yuhas et al., 1989) increase the
performance of automatic speech recognition system even in noisy environments
by using neural networks (NN). The inspiration for these approaches was McGurk
effect (McGurk and MacDonald, 1976) – an interaction between vision and
hearing during understanding of speech. Hidden Markov Model (HMM) (Juang and
Rabiner, 1991) has become so popular in speech recognition because of the
inherent statistical framework, the ease and accessibility of training
algorithms to estimate model parameters from finite speech data sets,
flexibility of the resulting recognition system, in which the size, type or
architecture of the models can be easily changed to suit specific words,
sounds.

  * •

Interaction Era (IE): The IE starts from 2000 to 2010. During this era the
understanding of human multimodal behaviors is achieved during social
interactions. One of the first milestones is the AMI Meeting Corpus (Carletta
et al., 2005) (a multimodal dataset) with over 100 hours of meeting
recordings, all thoroughly annotated and transcribed. AMI seeks to create a
repository of meetings and to evaluate conversations. CHIL project motive
(Waibel et al., 2009) uses ML methods to extract nonverbal human behaviors
automatically. Under CHIL, different people tracking systems were built using
audio, video or both modalities. CALO Meeting Assistant (CALO-MA) (Tur et al.,
2008) real-time meeting recognition system takes the speech models. CALO-MA
architecture includes real-time and offline speech transcription, action item
recognition, question-answer pair identification, decision extraction, and
summarization. Social Signal Processing (SSP) (Pantic et al., 2011) aims at
understanding and modelling social interactions and offering similar
capabilities to computers in human-computer interaction scenarios.

  * •

Deep Learning Era (DLE): The DLE starts in 2010 to date. This deep learning
era is the main focus of our review paper. We comprehensively discuss
different groups of methods and applications proposed during this era in
section 3, and datasets and evaluation metrics in section 2 of supplementary
material proposed during this era.

##  3\. MMDL Methods and Applications:

Various methods and applications are designed using multimodal deep learning
techniques. In this article, these methods and applications are grouped with
relevance and dominance across multiple research areas. The taxonomy diagram
of these applications is presented in the Figure 2.

###  3.1. Multimodal Image Description (MMID):

Image Description is mostly used to generate a textual description of visual
contents provided through input image. During the deep learning era, two
different fields are merged to perform image descriptions, i.e., CV and NLP.
In this process, two main kinds of modalities are used, i.e., image and text.
The image description’s general structure diagram is shown in Figure 3. Image
description frameworks are categorized into Retrieval-based, Template-based,
and DL-based image descriptions. Retrieval and Template based image
descriptions are one of the earliest techniques for describing visual contents
from images. In this article, DL-based image description techniques are
explained in detail, which are further categorized into encoder-decoder-based,
semantic concept-based, and attention-based image descriptions. Different
image description approaches are analyzed comparatively according to
architectures, multimedia, publication year, datasets, and evaluation metrics
in Table 2.

Figure 3. General structure diagram of Image Description Model.

####  3.1.1. Encoder-Decoder based Image Description (EDID):

EDID plays a vital role in image captioning tasks using DL architectures. CNN
architectures are used mainly as encoder parts to extract and encode data from
images, and RNN architectures are used as decoder part to decode and generate
captions. J. Wu & H.hu (Wu and Hu, 2017) proposed a Cascade Recurrent Neural
Network (CRNN) for image description. For the learning of visual language
interactions, a cascade network is adopted by CRNN from the forward and
backward directions. M. Chen et al. (Chen et al., 2017) proposed referenced
based LSTM model for image description task. In this model, training images
are used as reference for proposed framework, to minimize the misrecognitions
for the description task. W. Jiang et al. (Jiang et al., 2018) proposed a
recurrent fusion network for the task of image captioning on the basis of
encoder-decoder. In this network, CNN architecture are used to extract
information from input image and RNN architecture is used to generate the
description in the form of text. L. Guo et al. (Guo et al., 2019a) proposed a
multi-style image captioning framework using CNN, GAN, LSTM and GRU
architectures. In this framework, five different captioning styles are
introduced for the image: romantic, negative, positive, factual and humorous
styles. X. He et al. (He et al., 2019) proposed an image caption generation
framework with the guidance of Part of Speech (PoS). In the word generation
part, PoS tags are fed into LSTM as guidance to generate more effective image
captions. Y. Feng et al. (Feng et al., 2019) proposed an unsupervised image
captioning framework. In this model first attempt is made to do captioning of
the image without any labeled image-sentence pairs. J. Ji et al. (Ji et al.,
2021) introduced a globally enhanced transformation network for encoder and
decoder. In this network, the encoder is used to extract global features from
the image and the global adaptive controller at the decoder side is used for
controlled image description.

####  3.1.2. Semantic Concept-based Image Description (SCID):

A collection of semantic concepts extracted from the image are selectively
addressed by SCID approaches. These concepts are extracted at the encoding
stage along with other features of an image, then merged into hidden states of
language models, and output is used to generate descriptions of images based
on semantic concepts. W. Wang et al. (Wang et al., 2018b) proposed an
attribute-based image caption generation framework. Visual features are
extracted by using salient semantic attributes and are pass as input to LSTM’s
encoder. Z. Zhang et al. (Zhang et al., 2018) proposed a semantic guided
visual attention mechanism based image captioning model. Fully Convolutional
Network (FCN) is primarily intended for semantic segmentation, especially for
dense pixel level feature extractions and semantic labeling in the form of
spatial grid. P. Cao et al. (Cao et al., 2019) proposed a semantic-based model
for image description. In this model, semantic attention-based guidance is
used for LSTM architecture to produce a description of an image. L. Cheng et
al. (Cheng et al., 2020) proposed a multi-stage visual semantic attention
mechanism based image description model. In this approach, top-down and
bottom-up attention modules are combined to control the visual and semantic
level information for producing fine-grained descriptions of the image. L.
Chen et al. (Chen et al., 2021a) proposed a model to improve the accuracy of
image captioning by introducing Verb-specific Semantic Roles (VSR). This model
targets the activity and entities roles involved in that particular action to
extract and generate the most specific information from image.

Table 2. Comparative analysis of Image Description models. Where I=Image,
T=Text, EM = Evaluation Metrics, B=BLEU, C=CIDEr, M=METEOR, R=ROUGE, S=SPICE

Model Year Architecture Media Dataset EM Model Features CRNN (Wu and Hu, 2017)
2017 CNN/VGG16-InceptionV3, SGRU I,T MS-COCO B,C,M * Cascaded network for
learning visual language interactions. * Two embedding layers for dense word
expressions are designed. * SGRU is designed for image to word mapping R-LSTM
(Chen et al., 2017) 2017 CNN/VGG16, RNN/LSTM I,T MS-COCO B,C, M,$R_{L}$ *
Trained images are used as reference for model * Words are weighted according
to relevance score during training * Consensus score ranks the reference
information for caption generation RFNet (Jiang et al., 2018) 2018
CNN/ResNet/DenseNet/ InceptionV3-V4,RNN/LSTM I,T MS-COCO B,C,M, S,$R_{L}$ *
Use 5 feature extraction techniques * Multiple encoders are used for feature
extraction * Comprehensive information/thought vectors are produced by fusing
encoder information EDID MSCap (Guo et al., 2019a) 2019 Deep CNN, GAN,
RNN/LSTM, GRU I,T FlickrStyle10K, SentiCap,COCO B,C,M, PPLX * A unified multi-
style caption generation model is introduced * Real human-like and stylized
(Factual, Humorous, Romantic, Positive, Negative)captions are generated He et
al (He et al., 2019) 2019 CNN/VGG16, RNN/LSTM I,T Flickr30k, MS-COCO B,C,M *
Use PoS sentence tags as guidance to LSTM word generator * PoS tags are
controlled by switches to embed only specific information Feng et al. (Feng et
al., 2019) 2019 CNN/InceptionV4, RNN/LSTM I,T MS-COCO B,C,M, S,R * ”Visual
Concept detector” provides guidance for caption generation * Visual feature
and generated captions are projected to a common latent space * Training is
done without any labeled (image, sentence) pairs GET (Ji et al., 2021) 2021
Faster-RCNN/ResNet, RNN/LSTM I,T MS-COCO B,C,M, S,$R_{L}$ * Global gated
adaptive controller is used to fuse relavent information at decoder * Inter
and Intra layer representations are used to merge local/global information
Wang et al. (Wang et al., 2018b) 2018 CNN/VGG16, RNN/LSTM I,T MS-COCO B,C,M *
Feature extraction using semantic attributes enhance the caption generation
accuracy * Multiple instance learning transfer rules are designed at fc7 layer
of VGG-16 to extract more semantic features FCN-LSTM (Zhang et al., 2018) 2018
FCN/VGG16, RNN/LSTM I,T MSCOCO, COCO-Stuff B,C,M * Semantic-guided attention
mechanism enhance the caption generation accuracy * Model works well for large
areas like sky, beach, etc. by assigning additional semantic context SCID Bag-
LSTM (Cao et al., 2019) 2019 CNN/VGG16, RNN/BLSTM I,T Flickr8k, MS-COCO B,C,M
* Feedback propagation extracts the text-related image features * Semantic
attention-based guidance leverage the information dynamically Stack-VS (Cheng
et al., 2020) 2020 Faster-RCNN, RNN/LSTM I,T MS-COCO B,S, M,C,R * Multi-Stage
Bottom-Up and Top-Down stacked attention generates rich captions * Two LSTM
layers at decoder re-optimize the semantic attention weights VSR (Chen et al.,
2021a) 2021 Faster-RCNN, SSP, RNN/LSTM I,T Flickr30K, MS-COCO B,C,M, S,$R_{L}$
*Verb-specific Semantic Roles controls the image caption generation process
*Human-like semantic structures ranks the verb & semantic roles using SSP GLA
(Li et al., 2017) 2017 CNN/VGG16-Faster RCNN, RNN/LSTM I,T Flickr8K/30K, MS-
COCO B,C,M, $R_{L}$ * Object-level features are extracted along with image
level features * Important features are extracted by focusing on specific
regions of image using global-local attentions Up-Down (Anderson et al., 2018)
2018 Faster RCNN/ResNet101, RNN/LSTM, GRU I,T MS-COCO B,C,M, S,R * Top-down
attention mechanism ranks extracted features from image * LSTM one layer is
designed as top-down attention model and other layer as language model
NICVATP2L (Liu et al., 2020a) 2020 CNN/InceptionV4, RNN/LSTM I,T Flickr8k-CN,
AIC-ICC B,C,M, R * Topic modeling introduced to increase the accuracy of
caption generation * Visual regions and topic features are merged at decoding
stage to guide two-layers LSTM for image captioning AID NICNDA (Liu et al.,
2020b) 2020 CNN/InceptionV4, RNN/LSTM I,T AIC-ICC B,C,M, R * Visual attention
at decoder increases the image understanding * Information integrity is
boosted by textual attention * Language model and textual attention are work
together to fine-grain description Wang et al. (Wang et al., 2020) 2020
CNN/InceptionV4, RNN/LSTM I,T Flickr8K, Flickr8k-CN B, R, C, M * Attention
mechanism based on vector similarity is introduced * Semantic words and image
features correlation express the appropriate attention MAGAN (Wei et al.,
2020b) 2020 GAN, RNN/LSTM I,T MS-COCO B,C,M, S,R * Local and non-local
evidences are used to rich the feature representations * Multi-attention
discriminator and generator increased the image caption accuracy MGAN (Jiang
et al., 2021) 2021 Faster-RCNN, RNN/LSTM I,T MS-COCO B,C,M, $R_{L}$,S * Self
Gate and Attention Weight Gate are used with existing self-attention mechanism
to extract intra-object relationships * Pre-layernorm transformer is designed
for features enhancement.

####  3.1.3. Attention-based Image Description (AID):

AID plays a vital role because it helps the image description process by
focusing on distinct regions of the image according to their context. In
recent years, various techniques have been proposed to better describe an
image by applying an attention mechanism. Some of these attention mechanism
based image descriptions techniques are; L. Li et al. (Li et al., 2017)
proposed a new framework for describing images by using local and global
attention mechanisms. Selective object-level features are combined with image-
level features according to the context using local and global attention
mechanisms. P. Anderson et al. (Anderson et al., 2018) proposed bottom-up and
top-down attention based framework for image description to encourage deeper
image understanding and reasoning. M. Liu et al. proposed a dual attention
mechanism-based framework to describe an image for Chinese (Liu et al., 2020a)
and English (Liu et al., 2020b) languages. The textual attention mechanism is
used to improve the data credibility, and the visual attention mechanism is
used to a deep understanding of image features. B. Wang et al. (Wang et al.,
2020) proposed an E2E-DL approach for image description using a semantic
attention mechanism. In this approach, features are extracted from specific
image regions using an attention mechanism for producing corresponding
descriptions. Y. Wei et al. (Wei et al., 2020b) proposed an image description
framework by using multi attention mechanism to extract local and non-local
feature representations. W. Jiang et al. (Jiang et al., 2021) proposed a
multi-gate expansion of self-attention mechanism. In this network, attention
mechanism is expanded by adding self-gated module and attention weight gate
module to eliminate the irrelevant information from description.

Figure 4. General structure diagram of Video Description Model. (a) is general
structure diagram of CNN-RNN architecture video description and (b) is General
structure diagram of RNN-RNN architecture combination for video description.

###  3.2. Multimodal Video Description (MMVD):

Like image Description, video description is used to generate a textual
description of visual contents provided through input video. Here, DL
approaches for the description of visual contents from videos are discussed in
detail. Advancements in this field open up many opportunities in various
application domains. During this process, mainly two types of modalities are
used, i.e., video stream and text. The general structure diagram of the video
description is shown in Figures 4 (a) & (b). In this research, the video
description approach is categorized based on the following architectural
combinations for visual feature extraction and text generation. These
approaches are comparatively analyzed according to architectures, multimedia,
datasets, and evaluation metrics in Table 3.

####  3.2.1. CNN-RNN Architectures:

Most broadly used architecture combination in the domain of video description
is CNN-RNN. Figure 4(a) presents the general view of the video description
process by using CNN-RNN architectures. Where at the visual extraction
(encoder) stage variants of CNN architectures are used and at the sentence
generation (decoder) stage variants of RNN architectures are used. During deep
learning era, several authors proposed techniques for describing videos that
are based on this encoder, decoder combination. R Krishna et al. (Krishna et
al., 2017) proposed a video description technique using action/event detection
by applying a dense captioning mechanism. This is the first framework to
detect and describe several events, but it didn’t significantly improve video
captioning. B. Wang et al. (Wang et al., 2018c) proposed a reconstruction
network for video description using an encoder-decoder reconstructor
architecture, which utilizes both forward flow (from video to sentence) and
backward flow (from sentence to video). W. Pei et al. (Pei et al., 2019)
proposed an attention mechanism based encoder-decoder framework for video
description. An additional memory based decoder is used to enhance the quality
of video description. N Aafaq et al. (Aafaq et al., 2019) proposed video
captioning and capitalized on Spatio-temporal dynamics of videos to extract
high-level semantics using 2D and 3D CNNs hierarchically, and GRU is used for
the text generation part. S. Liu et al. (Liu et al., 2020c) proposed SibNet; a
sibling convolutional network for video description. Two architectures are
used simultaneously to encode video, i.e., the content-branch to encode visual
features and the semantic-branch to encode semantic features. J. Perez-Martin
et al. (Perez-Martin et al., 2021) improves the visual captioning quality by
implementing visual syntactic embedding. A PoS tagging structure is used to
extract the syntactic representations from video and guide the decoder with
these temporal compositions to achieve accurate descriptions.

Table 3. Comparative analysis of Video description models. where, DRL=Deep
Reinforcement Learning, V=Video, T=Text, EM = Evaluation Metrics, B=BLEU,
C=CIDEr, M=METEOR, R=ROUGE

Model Year Architecture Media Dataset EM Model Features Krishna et al.
(Krishna et al., 2017) 2017 CNN/C3D, RNN/LSTM V,T ActivityNet Captions B,C,M *
Dense event detection and description are introduced * All the events are
identified in one pass then forward to language model for rich description
RecNet (Wang et al., 2018c) 2018 CNN/InceptionV4, RNN/LSTM V,T MSVD, MSR-VTT,
B,C,M $R_{L}$ * Both video to sentence and sentence to video flows are
monitered * Video to sentence flow generates the description with video
semantic guidance CNN to MARN (Pei et al., 2019) 2019
CNN/ResNet101/ResNeXt101, RNN/GRU V,T MSVD, MSR-VTT B,C,M, $R_{L}$ * A memory
structure explores the correlation between words and its visual features
during training * Description quality is improved by this correlation *
Modeling of adjacent words are done explicitly RNN GRU-MP/EVE (Aafaq et al.,
2019) 2019 CNN/IRV2, 3DCNN/C3D, RNN/GRU V,T MSVD, MSR-VTT B,C,M, R * Temporal
instances in visual features are extracted by hierarchically applying Short
Fourier Transform * Semantic features are extracted during object detection to
enhance visual representation SibNet (Liu et al., 2020c) 2020
CNN/GoogLeNet/Inception, RNN/LSTM V,T MSVD, MSR-VTT B,C,M, R * Two branch
(context and semantic) modules simultaneously encode visual features * Both
branches features are merged with soft-attention for rich and concise
descriptions SemSynAN (Perez-Martin et al., 2021) 2021 2D-CNN/3D-CNN, RNN/LSTM
V,T MSVD, MSR-VTT B,C,M, $R_{L}$ * Syntactic representation extracts the rich
visual information * SemSynAN guide language model by merging syntactic,
visual and semantic representations to enhance description accuracy SSVC
(Rahman et al., 2020) 2020 CNN/VGG16-LSTM, RNN/LSTM V,T MSVD B * Content
generation technique is modified using stacked attention and spatial hard pull
* Objects in videos are prioritized layer-by-layer using stacked attention.
RNN to V2C (Fang et al., 2020) 2020 CNN/ResNet-LSTM, RNN/LSTM V,T V2C, MSR-VTT
B,M, $R_{L}$ * Commonsense captions are generated for events detected from
videos * V2C-transfer mechanism generates the commonsense enriched captions
RNN GPaS (Zhang et al., 2020a) 2021 GCN, RNN/LSTM, RNN/LSTM V,T ActivityNet
Captions B,C,M $R_{L}$ * Partition and Summarization strategy enhances the
video description * Partition splits event proposal into video segments and
summarization module generates description for each segment HRL (Wang et al.,
2018a) 2018 CNN/ResNet, RNN/BLSTM/LSTM, HRL V,T MSR-VTT, Charades B,C,M,
$R_{L}$ * Manager and Worker modules design sub-goals and compulsory actions
to accomplish these goals * Attention mechanism focus widely on temporal
dynamics of visual features PickNet (Chen et al., 2018) 2018 RNN/LSTM, RNN/GRU
V,T MSVD, MSR-VTT B,C,M, $R_{L}$ * Pick selective informative frames from
video to extract visual features * Video description can be done with compact
frames * Computation cost of image description is reduced E2E. (Li and Gong,
2019) 2019 CNN/Inception-Resnet-V2, RNN/LSTM V,T MSVD, MSR-VTT B,C,M, $R_{L}$
* During training, attributes extracted from videos are used to maximize the
reward * High-level features information is extracted using five duplicates
networks of CNN at encoder DRL SDVC (Mun et al., 2019) 2019 SST, C3D,
RNN/GRU/LSTM V,T ActivityNet Captions B,C,M * From videos, temporal
dependencies are extracted explicitly * Linguistic and visual features context
are extracted from previous events are core part of video description process
RecNet(RL) (Zhang et al., 2019a) 2019 CNN/InceptionV4, RNN/LSTM V,T
ActivityNet, MSVD, MSR-VTT B,C,M, $R_{L}$ * A similar approach to RecNet for
video description task with addition of RL technique to improve description *
Two kinds of reconstructors extract local and global representations from
video DPRN (Xu et al., 2020) 2020 MDP, RNN/LSTM V,T MSVD, MSR-VTT B,C,M,
$R_{L}$ * Word denoising and grammar checking networks refine the generated
descriptions * Long term reward enhances the description by considering global
representations into account Wei et al. (Wei et al., 2020a) 2020 3D
CNN/ResNet, RNN/LSTM V,T MSVD, MSR-VTT, Charades B,C,M * Adaptive moving
window extracts better event representations * Additionally, temporal
attention enhance description process by focusing on temporal frame features
while words captioning

####  3.2.2. RNN-RNN Architectures:

During the DL era, RNN-RNN is also a popular architectural combination because
many authors contribute a lot by proposing various methods using this
combination. Authors extract the visual content of the video by using RNN
architectures instead of CNN. Figure 4(b) presents the general view of the
video description process by using RNN-RNN architectures. Both visual
extraction (encoder) and sentence generation (decoder) stage variants of RNN
architectures are used. M. Rahman Et al. (Rahman et al., 2020) proposed a
video captioning framework that modifies the generated context using spatial
hard pull and stacked attention mechanisms. This approach illustrates that
mounting an attention layer for a multi-layer encoder will result in a more
semantically correct description. Z. Fang et al. (Fang et al., 2020) proposed
a framework to generate commonsense captions of the input video. Commonsense
description seeks to identify and describe the latent aspects of video like
effects, attributes, and intentions. A new dataset, ”Video-to-Commonsense
(V2C),” is also proposed for this framework. Z. Zhang et al. (Zhang et al.,
2020a) proposed an encoder decoder based framework for dense video captioning.
A graph based summarization and partition modules are used to enhance the word
relation ship between context and event fount in video.

####  3.2.3. Deep Reinforcement Learning (DRL) architectures:

DRL is a learning mechanism where machines can learn intelligently from
actions like human beings can learn from their experiences. In it, an agent is
penalized or rewarded based on actions that bring the model closer to the
target outcome. The general structure diagram of the video description DRL is
shown in Figure 5 (a). The main contributions of authors using DRL
architectures are; X. Wang et al. (Wang et al., 2018a) proposed a hierarchical
based reinforcement learning (HRL) model for describing a video. In this
framework, a high-level manager module designed sub-goals and low-level worker
module recognize actions to fulfill these goals. Y. Chen et al. (Chen et al.,
2018) proposed a framework based on RL for choosing informative frames from an
input video. Fewer frames are required to generate video description in this
approach. L. Li & B. Gong (Li and Gong, 2019) proposed an E2E multitask RL
framework for video description. The proposed method combines RL with
attribute prediction during the training process, which results in improved
video description generation. J. Mun et al. (Mun et al., 2019) proposed a
framework where an event sequence generation network is used to monitor the
series of events for generated captions from the video. W. Zhang et al. (Zhang
et al., 2019a) proposed a reconstruction network for a description of visual
contents, which operates on both forward flow (from video to sentence) and
backward flow (from sentence to video). W. Xu et al. (Xu et al., 2020)
proposed a polishing network that utilizes the RL technique to refine the
generated captions. This framework consists of word denoising and grammar
checking networks for fine-tuning generated sentences. R. Wei et al. (Wei et
al., 2020a) proposed a framework for better exploration of RL events to
generate more accurate and detailed video captions.

Figure 5. (a) General structure diagram of Video Description Deep
Reinforcement Learning Architectures, and (b) General structure diagram of VQA
System

###  3.3. Multimodal Visual Question Answering (MMVQA):

VQA is an emerging technique that has piqued the interest of both the CV and
NLP groups. It is a field of research about creating an AI System capable of
answering natural language questions. Extracted features from input
image/video and question are processed and combined to answer the question
about the image as presented in Figure 5 (b). VQA is more complex as compared
to other vision and language functions like text-to-image retrieval, video
captioning, image captioning, etc. because; (1) Questions asked in VQA are not
specific or predetermined. (2) Visual information in VQA is at a high degree
of dimensionality. Usually, VQA required a more thorough and detailed
understanding of an image/video. (3) VQA solves several CV sub-tasks. Many
authors contribute to the field of VQA by using various DL techniques. These
methods are grouped and presented into three groups, i.e., multimodal joint-
embedding models, multimodal attention-based models, and multimodal external
knowledge-based models. Various methods of these models are comparatively
analyzed in Table 4.

####  3.3.1. Multimodal Joint-Embedding Models (MMJEM):

MMJEM join and learn representations of multiple modalities in a common
feature space. This rationale is improved further in VQA by performing more
reasoning over modalities than image/video description. H. Ben-Younes et al.
(Ben-Younes et al., 2017) proposed a MUTAN framework for VQA. A tensor-based
tucker decomposition model is used with a low-rank matrix constraint to
parameterize the bi-linear relations between visual and text interpretations.
MT Desta et al. (Desta et al., 2018) proposed a framework that merges the
visual features and language with abstract reasoning. High-level abstract
facts extracted from an image optimize the reasoning process. R. Cadene et al.
(Cadene et al., 2019) proposed an E2E reasoning network for VQA. This
research’s main contribution is introducing the MuRel cell, which produces an
interaction between question and corresponding image regions. B. Patro et al.
(Patro et al., 2020) proposed a joint answer and textual explanation
generation model. A collaborative correlated (encoder, generator, and
correlated) module is used to ensure that answer and its generated explanation
is correct and coherent. S. Lobry et al. (Lobry et al., 2020) proposed a VQA
framework for remote sensing data, which can be useful for land cover
classification tasks. Z. Fang et al. (Fang et al., 2020) proposed an open-
ended VQA framework for videos using commonsense reasoning in the language
part, where questions are asked about effects, intents, and attributes.

####  3.3.2. Multimodal Attention-based Models (MMAM):

During the encoding stage, a general encoder-decoder can feed some noisy and
unnecessary information at the prediction phase. MMAM are designed to improve
the general baseline models to overcome this problem. The attention
mechanism’s main objective is to use local features of image/video and allow
the system to assign priorities to the extracted features from different
regions. This concept is also used in VQA to enhance model’s performance by
focusing on a specific part of an image according to the asked question.

P. Wang et al. (Wang et al., 2017b) proposed a VQA framework based on a co-
attention mechanism. In this framework, the co-attention mechanism can process
facts, images, and questions with higher order. Z. Yu et al. (Yu et al., 2017)
proposed a factorized bi-linear pooling method with co-attention learning for
the VQA task. Bilinear pooling methods outperform the conventional linear
approaches, but the practical applicability is limited due to their high
computational complexities and high dimensional representations. P. Anderson
et al. (Anderson et al., 2018) proposed bottom-up and top-down attention based
framework for VQA. This attention mechanism enables the model to calculate
feature based on objects or salient image regions. Z. Yu et al. (Yu et al.,
2019) proposed a deep modular co-attention network for the VQA task. Each
modular co-attention layer consists of question guided self-attention
mechanism of images using the modular composition of both image and question
attention units. L. Li et al. (Li et al., 2019) proposed a relation aware
graph attention mechanism for VQA. This framework encodes the extracted visual
features of an image into a graph, and to learn question-adaptive relationship
representations, a graph attention mechanism modeled inter-object relations.
W. Guo et al. (Guo et al., 2020) proposed a re-attention based mechanism for
VQA. The attention module correlates the pair of object-words and produces the
attention maps for question and image with each other’s guidance.

Table 4. Comparative analysis of Visual Question Answering models. where
MMJEM=Multimodal Joint-Embedding Models, MMAM=Multimodal Attention-based
Models, MMEKM=Multimodal External Knowledge bases Models, I=Image, V=Video,
T=Text, EM=Evaluation Metrics, Acc=Accuracy, B=BLEU, C=CIDEr, M=METEOR,
R=ROUGE, S=SPICE

Model Year Architecture Media Dataset EM Model Features MUTAN (Ben-Younes et
al., 2017) 2017 CNN/ResNet152, RNN/GRU I,T VQA Acc * Bilinear interactions
between textual and visual features are effectively parameterize by tucker
framework * Matrix-based module is designed to explicitly constrain the
interaction rank Desta et al. (Desta et al., 2018) 2018 Faster RCNN/Resnet101,
RNN/LSTM, RNN/GRU I,T CLEVR Acc * Reasoning efficiency is increased by
extracted abstract facts from text and video inputs * Visual features
extraction relies on information linked with objects MuReL (Cadene et al.,
2019) 2019 Faster RCNN, GRU I,T VQAv2.0,TDIUC, VQA-CPv2 Acc * MuReL cell
represents correlation between image and question regions * Cell features are
incorporated into whole model to refine question and visual representations
MMJEM CCM (Patro et al., 2020) 2020 CNN, RNN/LSTM I,T VQA-X B,S,C, M,$R_{L}$ *
A collaborative correlated module verifies the coherency of explanations and
answers * Attention maps used in these correlated networks are closely related
with human generated maps RSVQA (Lobry et al., 2020) 2020 CNN/ResNet152, RNN
I,T OpenStreetMap (LR,HR) Acc * RSVQA system extracts features from remote
sensing data * High-level features are associated with relational dependencies
among visible objects V2C (Fang et al., 2020) 2020 CNN/ResNet-LSTM, RNN/LSTM
V,T V2C-QA TopK * Open ended commonsense reasoning is used to enhance the VQA
quality * This reasoning part also helps to improve the video captioning task
Wang et al. (Wang et al., 2017b) 2017 CNN/VGG19-ResNet100, RNN/LSTM I,T Visual
Genome QA, VQA-real WUPS, Acc * Different combination of CV algorithms learns
adaptively generate the reasoning part * Existing Co-Attention model is
upgraded to process images, facts and question jointly MFB (Yu et al., 2017)
2017 CNN/ResNet152, RNN/LSTM I,T VQA MS-COCO Acc * MFB framework fuse the
textual and visual features from question and images jointly * And co-
attention structure learns the question and image attentions Up-Down (Anderson
et al., 2018) 2018 Faster R-CNN, RNN/GRU I,T Visual Genome, VQA v2.0 Acc *
Top-down attention mechanism ranks extracted features from image and used
question features as context * Multimodal embedding of image and question
increase the VQA model accuracy MMAM MCAN (Yu et al., 2019) 2019 Faster
R-CNN/ResNet101, RNN/LSTM I,T VQA v2.0 Acc * ”Modular Co-Attention (MCA)”
layer use composition of images and questions self-attention * MCA layer
gradually refines the question and image representations ReGAT (Li et al.,
2019) 2019 Faster R-CNN/ResNet101, RNN/LSTM I,T VQA v2.0, VQA-CP v2 Acc * Both
implicit and explicit relations are used to get appropriate image and question
representations * ReGAT captures/records the inter-object relations to reveal
fine-grained features from image Xi et al. (Xi et al., 2020) 2020 CNN,
RNN/LSTM I,T DQAUAR, COCO-QA Acc, WUPS * Appearance model is extended by word
vector similarity to enhance VQA accuracy * Appearance features form objects
are used instead of image features Guo et al. (Guo et al., 2020) 2020 Faster
R-CNN, RNN/LSTM I,T VQA v2.0 Acc * Merge image and question features are
associated together to find object similarities * Then according to generated
answer, model re-attend the associated object and reconstruct attention maps
to improve answers FVQA (Wang et al., 2018d) 2018 Faster R-CNN/VGG16, RNN/LSTM
I,T MS-COCO, FVQA TopK,Acc, WUPS * The dataset designed for VQA task based on
facts * These facts are gathered from external sources (DBPedia, ConceptNet,
Web Child) to produce answers Narasimhan et al. (Narasimhan and Schwing, 2018)
2018 CNN/Faster RCNN/VGG16, RNN/LSTM I,T FVQA Acc, TopK * Parametric mapping
of image-questions pairs and facts are gathered into a common embedding space
* Most aligned facts are used to produce answer of question MMEKM OK-VQA
(Marino et al., 2019) 2019 CNN/ResNet, RNN/GRU I,T OK-VQA Recall, TopK * OK-
VQA is a diverse, large and also a difficult database as compared to existing
databases * Ok-VQA works well where model requires external information
instead of only visual features GRUC (Yu et al., 2020) 2020 Faster-RCNN,
RNN/LSTM I,T FVQA,OKVQA, Visual7W+KB Acc, TopK * Image features are extracted
by multiple knowledge graph from three (semantic, factual and visual) views *
Recurrent reasoning model extract the compulsory representation from multi
model space AQuA (Basu et al., 2020) 2020 YOLO, SRE I,T CLEVR Acc * Answer set
programming is introduced to effectively understand the query about image * To
deal with complex queries, semantic features are extracted and used to enhance
scenario understanding

####  3.3.3. Multimodal External Knowledge Bases Models (MMEKM):

Traditional multimodal joint embedding and attention-based models only learn
from the information that is present in training sets. Existing datasets do
not cover all events/activities of the real world. Therefore, MMEKM is vital
to coping with real-world scenarios. Performance of VQA task is more
increasing by linking knowledge bases (KB) databases to VQA task. Freebase
(Bollacker et al., 2008), DBPedia (Auer et al., 2007), WordNet (Miller, 1995),
ConceptNet (Liu and Singh, 2004), and WebChild (Tandon et al., 2014) are
extensively used KB. A robust VQA framework requires access to broad
information content from KB. It has been effectively integrated into the VQA
task by embedding the various entities and relations.

During the DL era, various external KB methods are proposed for VQA tasks. P.
Wang et al. (Wang et al., 2018d) proposed another framework for the VQA task
names ”Fact-based VQA (FVQA)” that uses data-driven approaches and LSTM
architecture to map image/question queries. FVQA framework used DBPedia,
ConceptNet, and WebChild KB. M. Narasimhan & AG. Schwing (Narasimhan and
Schwing, 2018) proposed a framework for the VQA task using external knowledge
resources that contain a set of facts. This framework can answer both fact-
based and visual-based questions. K. Marino et al. (Marino et al., 2019)
proposed an outside knowledge dataset for VQA, which contains more than 14,000
questions. This dataset contains several categories like sports, science and
technology, history, etc. This dataset requires external resources to answer,
instead of only understanding the question and image features. K. Basu et al.
(Basu et al., 2020) proposed a commonsense based VQA framework. In this
framework, the image’s visual contents are extracted and understood by the
YOLO framework and represented in the answer set program. Semantic relations
features and additional commonsense knowledge answer the complex questions for
natural language reasoning. J. Yu et al. (Yu et al., 2020) proposed a
framework in which visual contents of an image is extracted and processed in
multiple perspectives of knowledge graph like semantic, visual, and factual
perspectives.

Figure 6. General structure diagram of Deep Learning Text To Speech framework
using DNN architecture.

###  3.4. Multimodal Speech synthesis (MMSS):

The most important aspect of human behavior is communication (write/speak).
Humans can communicate using natural language by text and speech, representing
the written and vocalized form of natural language, respectively. The latest
research in language and speech processing helps systems talk like a human
being. Speech synthesis is the complicated process of generating natural
language spoken by a machine. Natural language text modality is converted into
its respective spoken waveform modality in real-time by the Text To Speech
(TTS) system. Various applications are introduced in the real world using
speech synthesis, like human-computer interactive systems, screen readers,
telecommunication and multimedia applications, talking toys games, etc. The
main research objective of TTS systems these days is to produce a sound like a
human’s. Therefore, various aspects are used for the evaluation of the TTS
system’s quality, such as naturalness (quality with the perspective of
generated speech timing structure, rendering emotions and pronunciation),
intelligibility (in a sentence, quality of each word being produced),
synthetic speech preferences (choice of a listener in term of voice and signal
quality for better TTS system) and human perception factors like
comprehensibility (understanding quality of received messages). Articulatory
TTS, Concatenative TTS, Formant TTS, Parametric TTS, and Deep Learning TTS are
the main categories of the speech synthesis process. This section discusses
recent research trends and advancements of Deep Learning TTS.

Table 5. Comparatively analysis of speech synthesis (DLTTS) models, where
T=Text, A=Audio, V=Video, EM=Evaluation Metrics, Acc=Accuracy

Model Year Architecture Media Dataset EM Model Features TACOTRON (Wang et al.,
2017a) 2017 Bidirectional-GRU/RNN T,A North American English MOS * TACOTRON
synthesizes speech directly from corresponding characters embedding * Frame-
based technique increases the speed of inference procedure DeepVoice (Arik et
al., 2017) 2017 GRU/RNN T,A Internal English speech database, Blizzard MOS *
Five modules (Segmentation, grapheme-phoneme, phoneme duration prediction,
frequency prediction, and audio synthesis) synthesize the speech from text *
Inference speed of DeepVoice is faster than baseline methods DeepVoice2
(Gibiansky et al., 2017) 2017 GRU/RNN T,A Internal English speech database MOS
* Improvement over DeepVoice by introducing higher performance building blocks
and multi-speaker speech synthesis * Processing vocoder improves the quality
of generated speech DeepVoice3 (Ping et al., 2017) 2017 Griffin-Lim, WaveNet
vocoder T,A VCTK, Internal English speech database, LibriSpeech MOS *
DeepVoice3 is upgraded with the addition of attention mechanism to improve
speech quality * The training and scaling speed is also increases Parallel
WaveNet (Oord et al., 2018) 2018 CNN A North American English, Japanese MOS *
To increase the speech synthesis processing speed, probability density
distillation model is used * Parallel WaveNet is much faster than WaveNet and
produce multiple voices Arik et al. (Arik et al., 2018) 2018 Griffin-Lim
vocoder T,A LibriSpeech, VCTK Acc,EER, MOS * This model clone the person’s
voice with fewer audio samples * Speaker encoding and adaptation models infer
new speaker embedding and fine tune these embeddings VoiceLoop (Taigman et
al., 2018) 2018 GMM T,A Lj speech, Blizzard, VCTK MOS,MCD TopK * Shifting
buffer working memory estimates the attention and synthesize the audio *
VoiceLoop deal un-constrained audio samples without phoneme alignment
TACOTRON2 (Shen et al., 2018) 2018 STFT, BiLSTM T,A Internal US English MOS *
Feature prediction model maps characters to mel-scale spectograms * A modified
WaveNet model synthesize these spectograms to wave form AV-ASR (Tao and Busso,
2020) 2020 CNN, RNN/LSTM A,V Manual data collection (CRSS-4ENGLISH-14 corpus)
CER, F-Score * AV-ASR combines the modalities fusion feature extraction,
classification task and information modeling to improve synthesis process *
Multitask learning improves the framework performance Parallel TACOTRON (Elias
et al., 2020) 2021 GLU, VAE, LSTM T,A Proprietary speech MOS * Model is highly
parallelizable during inference and training process * Auto-encoder improves
naturalness by one-to-many mapping

####  3.4.1. Deep Learning TTS (DLTTS):

In DLTTS frameworks, DNN architectures model the relationship between text and
their acoustic realizations. The main advantage of DLTTS is the development of
its extensive features without human prepossessing. Also, the naturalness and
intelligibility of speech are improved using these systems. Text to speech
synthesis process is explained in the general structure diagram of Deep
Learning Text To Speech frameworks using DNN architectures, shown in the
Figure 6. Comparative analysis of these approaches is shown in Table 5.

Y. Wang et al. (Wang et al., 2017a) proposed ”Tacotron,” a sequence 2 sequence
TTS framework that synthesizes speech from text and audio pairs. Encoder
embeds the text that extracts its sequential representations. The attention-
based decoder process these representations, and after that, post-processing
architecture generates the synthesized waveforms. In an another research,
”Deep Voice” model using DNN architecture are proposed by SO Arik et al. (Arik
et al., 2017) synthesizes audio from characters. This model consists of five
significant blocks for the production of synthesized speech from text. The
computational speed is increased compared to existing baseline models because
the model can train without human involvement. A. Gibiansky et al. (Gibiansky
et al., 2017) proposed a Deep Voice-2 architecture. This framework is designed
to improve existing state-of-the-art methods, i.e., Tacotron and Deep Voice-1,
by extending multi-speaker TTS through low-dimension trainable speaker
embedding. In third version of Deep Voice, W. Ping et al. (Ping et al., 2017)
proposed a neural TTS system based on the fully convolutional model with an
attention mechanism. This model performs parallel computations by adapting
Griffin-Lim spectrogram inversion, WORLD, and WaveNet vocoder speech
synthesis. ”Parallel WaveNet” an advanced version of WaveNet proposed by A.
Oord et al. (Oord et al., 2018) using the probability density distribution
method to train networks. A teacher and a student WaveNet are used parallelly
in this model. SO Arik et al. (Arik et al., 2018) proposed a neural voice
cloning system that learns human voice from fewer samples. For that purpose,
two techniques, i.e., speaker adaptation and encoding, are used together. Y.
Taigman et al. (Taigman et al., 2018) proposed a VoiceLoop framework for a TTS
system. This model can deal with un-constrained voice samples without the need
for linguistic characteristics or aligned phonemes. This framework transformed
the text into speech from voices using a short-shifting memory buffer. J. Shen
et al. (Shen et al., 2018) proposed ”Tacotron2. It is neural TTS architecture
used to synthesize speech directly from the text. A recurrent based sequence
to sequence feature prediction network can map characters to spectrogram, and
then these spectrogram are used to synthesize waveforms by using a modified
version of WaveNet vocoder. F. Tao and C. Busso (Tao and Busso, 2020) proposed
an speech recognition system using multitask learning mechanism. The proposed
design takes into account the temporal dynamics across and within modalities,
resulting in an enticing and feasible fusion method. Parallel Tacotron is
another brilliant invention in recent times for neural TTS approach proposed
by I. Elias et al. (Elias et al., 2020). During inference and training
processes, this approach is highly parallelizable to achieve optimum synthesis
on modern hardware. One to many mapping nature of VAE enhance the performance
of TTS and also improves its naturalness.

Figure 7. General structure diagram of Multimodal Emotion Recognition using
Facial, Textual, and Speech Cues (Mittal et al., 2020).

###  3.5. Other MMDL Applications:

####  3.5.1. Multimodal Emotion Recognition (MMER):

Emotions of human is one of the approach for expressing feelings. MMER is
enormously vital for enhancing the interaction experience between humans and
computers. The task of ML is to empower computers to learn and identify new
inputs from training datasets, hence it can be used efficiently to detect,
process, respond, understand and recognize emotions of human through computers
training. Thus, the primary purpose of affective computing is to provide
machines/systems emotional intelligence ability. It has several research areas
such as learning, health, education, communication, gaming, personalized user
interface, virtual reality and information retrieval. Multimodal emotion
recognition framework can be developed on an AI/ML based prototypes designed
to extract and process emotion information from various modalities like
speech, text, image, video, facial expression, body gesture, body posture, and
physiological signals. The general structure diagram of multiplicative
multimodal emotion recognition using facial, textual, and speech cues are
shown in Figure 7.

During the DL era, various authors contribute to emotion recognition by using
different architecture and multiple modalities. Like, Y. Huang et al. (Huang
et al., 2017) proposed a fusion method for emotion recognition using two
modalities, i.e., facial expression and electroencephalogram (EEG) signals. A
NN classifier detects happiness, neutral, sadness, and fear states of
emotions. D. Nguyen et al. (Nguyen et al., 2017) proposed another approach for
emotion recognition by using audio and video streams. In this method, a
combination of C3D and DBN architectures is used to model Spatio-temporal
information and representation of video audio streams. D. Nguyen et al.
(Nguyen et al., 2018) proposed another approach for emotion recognition by
using audio and video streams. In this method, a combination of C3D, MCB and
DBN architectures is used to model Spatio-temporal information and
representation of video audio streams. S. Tripathi et al. (Tripathi and Beigi,
2018) proposed a multimodal emotion recognition framework using various
modalities like, text, speech, face expression, and hands movement on the
IEMOCAP dataset. The fusion of modalities is performed only at the final layer
to improve emotion recognition performance. D. Hazarika et al. (Hazarika et
al., 2018b) proposed an emotion recognition framework from video
conversations. This method can generate self and interpersonal affective
summaries from conversations by using contextual information extracted from
videos. They also proposed another framework for detecting emotions using the
attention mechanism (Hazarika et al., 2018a). In this framework, audio and
textual modalities are used for the detection of emotions. M. Jaiswal et al.
(Jaiswal et al., 2019) analyzes the change of emotional expressions under
various stress levels of an individual. The performance of this task is
affected by the degree of lexical or acoustic features. L. Chong et al. (Chong
et al., 2019) proposed a new online-chatting system called ”EmoChat” to
automatically recognize the user’s emotions and attach identified emotion
automatically and the sent message within a short period. During the chatting
user can know each other’s emotions in this network. M. Li et al. (Li et al.,
2020) proposed a multi-step deep system for reliable detection of emotions by
using collected data that contains invalid data as well. Neural networks are
used to filter this invalid data from videos and physiological signals by
using continuity and semantic compatibility. H. Lai et al. (Lai et al., 2020)
proposed a model for emotion recognition in interactive conversations. In this
model, different RNN architectures with variable contextual window sizes
differentiate various aspects of contexts in conversations to improve the
accuracy. RH Huan et al. (Huan et al., 2020) proposed a model using attention
mechanism. In this method, for better use of visual, textual and audio
features for video emotion detection, bidirectional GRU is cascaded with an
attention mechanism. The attention function is used to work with various
contextual states of multiple modalities in real-time. Y Cimtay et al. (Cimtay
et al., 2020) proposed a hybrid fusion method for emotion recognition using
three modalities, i.e., facial expression, galvanic skin response (GSR) and
EEG signals. This model can identify the actual emotion state when it is
prominent or concealed because of natural deceptive face behavior.

In this section, recent existing methods for emotion recognition and analysis
using multimodal DL are presented. Comparative analysis of these techniques is
shown in Table 6. This literature review on emotion detection is based on
text, audio, video, physiological signals, facial expressions and body
gestures modalities. It is clearly shown from an empirical study that
automatic emotion analysis is feasible and can be very beneficial in
increasing the accuracy of the system response and enabling the subject’s
emotional state to be anticipated more rapidly. Cognitive assessment and their
physical response are also analyzed along with primary emotional states.

Table 6. Comparative analysis of MMER and MMED models, Where PS=Physiological
Signals, FE=Facial Expression, BE=Body Gesture I=Image, V=Video, A=Audio,
T=Text, EM=Evaluation Metrics, WAcc=Weighted Accuracy, P=Precision, R=Recall

Model Year Architecture Media Dataset EM Model Features Huang et al. (Huang et
al., 2017) 2017 SVM PS,FE Manual offline data collection WAcc * Fusion between
peripheral and brain signals are used for emotion recognition * Three
intensities (weak, moderate, and strong) are detected against each emotion A-V
C3D+DBN (Nguyen et al., 2017) 2017 C3D, DBN V,A eNTERFACE Audio-visual WAcc *
Three dimensional CNN model spatio-temporal representations from audio and
video modality * Four combination of C3D and DBN models increases the accuracy
of framework A-V C3D+MCB +DBN (Nguyen et al., 2018) 2018 C3D, MCB, DBN V,A
FABO,eNTERFACE Audio-visual WAcc * A-V C3D+DBN framework is further improved
with the addition of bi-linear pooling * Feature-level fusion technique
increases the accuracy of baseline method Tripathi et al. (Tripathi and Beigi,
2018) 2018 MFCC, RNN/LSTM A,T, FE,BE IEMOCAP WAcc * Multiple modalities (text,
speech and emotions captured from hand movement, face expressions) are used
for emotion recognition task * Each modality is targeted individually and
finally fuse together to get accurate representations ICON (Hazarika et al.,
2018b) 2018 CNN, 3D-CNN, RNN/GRU T,A,V IEMOCAP, SEMAINE WAcc,MAE, F1-Score *
Multimodal representations are extracted from all utterance-videos *
Contextual summaries for conversation are used for emotion recognition
Hazarika et al. (Hazarika et al., 2018a) 2018 CNN, OpenSMILE, MFCC T,A IEMOCAP
WAcc,R, F1-Score * Text and audio modalities are assigned proper score using
attention mechanism * Prioritize features from these modalities are fuse
together for accurate emotion recognition MMER Jaiswal et al. (Jaiswal et al.,
2019) 2019 GRU PS, T MuSE, IEMOCAP, MSP-Improv WAcc Adversial networks are
designed to decorrelate stress representations from emotion features * Model
shows stress representation affect the quality of emotion recognition EmoChat
(Chong et al., 2019) 2019 TextCNN, Mini-Xception, HMM T, FE FER2013 WAcc *
Emotions of user are identified online from text conversation * Joint
processing of text and Facial expressions predicts the emotions in real-time
MSD (Li et al., 2020) 2020 C3D, DBN, SVM-RBF V, PS RECOLA WAcc,R, F1-Score *
MSD process and filter the collected records containing invalid data * MSD
emotion recognition performance improves automatically by extracting invalid
data DCWS-RNNs (Lai et al., 2020) 2020 CNN, 3D-CNN, RNN/GRU T,A,V IEMOCAP,
AVEC Acc, F1-Score * Four RNNs with different sizes context window is used to
convert context into memories * Attention-based multiple maps merged these
memories for emotion recognition $Bi-GRU_{init}$ (Huan et al., 2020) 2020
RNN/GRU V,T,A CMU-MOSI, POM F1-Score,P R,Acc,MAE * Multimodal contextual
information is recorded in real time by attention fusion model * This model
improves emotion recognition accuracy in time context Cimtay et al. (Cimtay et
al., 2020) 2020 CNN/InceptionResnetV2 FE, PS LUMED-2, DEAP WAcc * Seven
emotion states are dealt with from face, EEG, and GSR modalities * Model
predict dominant and hidden emotion states with good accuracy $M^{2}DN$ (Gao
et al., 2017) 2017 CNN, HMM I,T Brand-Social-Net R,P, F1-Score * $M^{2}DN$ is
capable of handling weakly labeled microblog data * The Social tracking
mechanism extracts the event information from these microblogs Huang et al.
(Huang et al., 2018) 2018 CRBM, CDBN, SVM V UCSD, Avenue AUC,EER, EDR,ACC *
Unsupervised DL model detects anomaly event from the given crowd scenes
*Energy, low-level visual and motion map features increase the accuracy of
event detection MMED Koutras et al. (Koutras et al., 2018) 2018 CNN, C3D A,V
COGNIMUSE AUC * C3D-CNN approach detects salient events from the visual
stream, while the 2D-CNN approach detects salient events from the audio stream
SMDR (Yang et al., 2019) 2019 CNN/VGGNet-AlexNet I, T MED, SED NMI, F1-Score *
SMDR model is used to detect real-life events from heterogeneous data * Class-
wise residual models in SMDR are designed to detect events with higher
accuracy

####  3.5.2. Multimodal Event Detection (MMED):

Due to the popularity of media sharing on the internet, users can easily share
their events, activities, and ideas anytime. The aim of multimodal event
detection (MMED) systems is to find actions and events from multiple
modalities like images, videos, audio, text, etc. According to statistics,
Million of tweets are posted per day, and similarly, YouTube users post more
than 30,000 hours of videos per hour. Hence, in many CV applications,
automatic event and action detection mechanisms are required from this large
volume of user-generated videos. Finding events and actions from this
extensive collection of data is a complex and challenging task. It has various
applications in the real-world like disease surveillance, governance,
commerce, etc and also helps internet users to understand and captures
happenings around the world. Summary of various MMED methods are presented in
Table 6.

Researchers have contributed a lot during the DL era and proposed many methods
for event and action detection using multiple modalities. Like, Y. Gao et al.
(Gao et al., 2017) proposed a method for event classification via social
tracking and deep learning in microblogs. Images and text media are fed to a
multi-instance deep network to classify events in this framework. S. Huang et
al. (Huang et al., 2018) proposed an unsupervised method for the detection of
anomaly events from crowded scenes using DL architectures. In this model,
visual, motion map and energy features are extracted from video frames. A
multimodal fusion network utilized these features for the detection of anomaly
events by using the SVM model. In another research to detect salient events
from videos, P. Koutras et al. (Koutras et al., 2018) employed CNN
architecture using audio and video modalities. In this framework, a CNN
architecture based on C3D nets is used to detect events from a visual stream,
and a 2D-CNN architecture is used to detect events from the audio stream.
Experimental results show the improvement of the proposed method performance
over the baseline methods for salient event detection. Z. Yang et al. (Yang et
al., 2019) proposed a framework for event detection from multiple data domains
like social and news media to detect real-world events. A unified multi-view
data representation is built for image and text modalities from social and
news domains in this framework. Class wise residual units are formulated to
identify the multimedia events.

##  4\. Experimental results comparisons on benchmark Datasets and Evaluation
Metrics:

In this section, we provide a summary of the performance and experimental
results of several models that have been reported. The following subsections
show the results of all the models presented in section 3.

###  4.1. Multimodal Image Description Results:

Multimodal image description models are categorized into three parts, i.e.,
encoder-decoder-based image description, semantic concept-based image
description, and attention-based image description. We can discuss the
performance of these models on the MS-COCO widely used dataset. Comparative
analysis of the MMID model’s experimental results on benchmark evaluation
metrics is shown in Table 7. Visual feature extractor plays an important role
in the evaluation of the model’s performance. Like for CRNN (Wu and Hu, 2017)
model, experimental results for BLEU, CIDEr, and METEOR metrics are improved
using InceptionV3 visual extractor instead of VGG16. GET (Ji et al., 2021)
model achieves the best results on the MS-COCO dataset using all the
evaluation metrics as compared to all EDID models. This model gets the best
experimental results on encoder-decoder-based image description models and
achieves the best results on standard evaluation metrics on semantic concept-
based image description and attention-based image description models. Inter
and intra layer representations are used to merge local and global features,
and the global gated adaptive controller fuses the relevant information at the
decoder to enhance the model’s performance. The experimental results showed
the dominance of GET as compared to other models of image captioning.
Experimental results on all listed MMID models are compiled and listed
separately for each standard evaluation metric as presented in Figure 8. Top
row shows values (percentage) of image description models on BLEU (B_1,2,3,4)
metric and bottom row shows values (percentage) on CIDEr (C), METEOR (M),
ROUGE (R), SPICE (S) metrics. GET (Ji et al., 2021) model results are better
than all comparative image description models on B_1, B_4, M, R metrics except
results of VSR model on C and S metrics.

Table 7. Comparative analysis of MMID model results on MSCOCO dataset using benchmark evaluation metrics, Where B=BLEU, C=CIDEr, M=METEOR, R=ROUGE, S=SPICE, ”-” indicates no result |  Model |  Feature Extractor | B_1 | B_2 | B_3 | B_4 | C |  M | R | S
---|---|---|---|---|---|---|---|---|---|---
| CRNN (Wu and Hu, 2017) |  VGG16 | 69.1 | 51.4 | 37.6 | 27.5 | 86.7 |  23.2 | - | -
InceptionV3 | 69.5 | 51.8 | 38.0 | 27.7 | 89.4 |  23.5 | - | -
R-LSTM (Chen et al., 2017) |  VGG16 | 76.1 | 59.6 | 45.0 | 33.7 | 102.9 |  25.7 | 55.0 | -
EDID |  RFNet (Jiang et al., 2018) |  ResNet, DenseNet, Inception V3&V4, Inception-ResNet-V2 | 80.4 | 64.7 | 50.0 | 37.9 | 125.7 |  28.3 | 58.3 | 21.7
|  He et al. (He et al., 2019) |  VGG16 | 71.1 | 53.5 | 38.8 | 27.9 | 88.2 |  23.9 | - | -
|  Feng et al. (Feng et al., 2019) |  InceptionV4 | 58.9 | 40.3 | 27.0 | 18.6 | 54.9 |  17.9 | 43.1 | 11.1
|  GET (Ji et al., 2021) |  ResNet101 | 81.5 | - | - | 39.5 | 131.6 |  29.3 | 58.9 | 22.8
|  Wang et al. (Wang et al., 2018b) |  VGG16 | 70.0 | 53.0 | 39.0 | 30.0 | 90.0 |  25.0 | - | -
FCN-LSTM (Zhang et al., 2018) |  VGG16 | 71.2 | 51.4 | 36.8 | 26.5 | 88.2 |  24.7 | - | -
SCID |  Bag-LSTM (Cao et al., 2019) |  VG16 | 71.9 | 54.5 | 40.5 | 30.2 | 99.8 |  25.3 | - | -
|  Stack-VS (Cheng et al., 2020) |  Faster-RCNN | 79.4 | 63.6 | 49.0 | 37.2 | 122.6 |  27.9 | 57.7 | 21.6
|  VSR (Chen et al., 2021a) |  ResNet101 | - | - | - | 16.0 | 162.8 |  23.2 | 47.1 | 35.7
|  GLA (Li et al., 2017) |  Faster-RCNN, VGG16 | 72.5 | 55.6 | 41.7 | 31.2 | 96.4 |  24.9 | 53.3 | -
Up-Down (Anderson et al., 2018) |  Faster-RCNN, ResNet101 | 77.2 | - | - | 36.2 | 113.5 |  27.0 | 56.4 | 20.3
AID |  MAGAN (Wei et al., 2020b) |  ResNet | 77.9 | 61.7 | 46.4 | 34.8 | 114.5 |  27.1 | 56.4 | 20.4
|  MGAN (Jiang et al., 2021) |  Faster-RCNN | 78.4 | 62.8 | 48.9 | 37.5 | 118.8 |  28.2 | 57.8 | 21.6

Figure 8. Experimental results of MMID models on MACOCO dataset, where BLEU
(B), CIDEr (C), METEOR (M), ROUGE (R), SPICE (S), and No Result (NR)

###  4.2. Multimodal Video Description Results:

In this article, multimodal video description models are categorized into CNN-
to-RNN encoder-decoder group, RNN-to-RNN encoder-decoder group, and deep
reinforcement learning group of applications. Most of these model’s
performances are evaluated on MSVD and MSR-VTT datasets using BLEU (B), CIDEr
(C), METEOR (M), and ROUGE (R) evaluation metrics as shown in Table 8.
SemSynAN (Perez-Martin et al., 2021) a CNN-to-RNN encoder-decoder based model,
achieved the best experimental results on B, C, M, and R metrics as compared
to all other video description models. This model outperforms on both MSR-VTT
and MSVD datasets. In the SymSynAN model, syntactic representations extract
rich visual information from input video frames. Both 2D-CNN and 3D-CNN visual
feature extractors are used to extract rich visual features from video and are
globally represented. Model guide language model by merging syntactic, visual,
and semantic representations to enhance video description accuracy at the
decoder stage. Experimental results of all listed MMVD models are compiled and
listed separately on common MSR-VTT and MSVD datasets as presented in Figure
9. The top row shows values (percentage) of MMVD model’s on B, C, M, and R
metrics on the MSR-VTT dataset, while the bottom row shows values (percentage)
of MMVD model’s on B, C, M, and R metrics on MSVD dataset. SemSynAN (Perez-
Martin et al., 2021) model clearly shows better results than listed
comparative video description models.

Table 8. Comparative analysis of Multimodal Video description model results on MSR-VTT and MSVD using benchmark evaluation metrics, Where B=BLEU, C=CIDEr, M=METEOR, R=ROUGE, ”-” indicates no result |  | MSR-VTT | MSVD
---|---|---|---
Model |  Feature Extractor |  B_4 |  M |  R |  C |  B_4 |  M |  R |  C
RecNet (Wang et al., 2018c) |  InceptionV4 |  39.1 |  26.6 |  59.3 |  42.7 |  52.3 |  34.1 |  69.8 |  80.3
MARN (Pei et al., 2019) |  ResNet101, ResNeXt101 |  40.4 |  28.1 |  60.7 |  48.1 |  48.6 |  35.1 |  71.9 |  92.2
GRU-MP/EVE (Aafaq et al., 2019) |  Inception-ResNet-V2 |  38.3 |  28.4 |  60.7 |  48.1 |  47.9 |  35.0 |  71.5 |  78.1
SibNet (Liu et al., 2020c) |  GoogLeNet, Inception |  40.9 |  27.5 |  60.2 |  51.9 |  54.2 |  34.8 |  71.7 |  88.2
SemSynAN (Perez-Martin et al., 2021) |  ResNet152, ECO |  46.4 |  30.4 |  64.7 |  51.9 |  64.4 |  41.9 |  79.5 |  111.5
HRL (Wang et al., 2018a) |  ResNet152 |  41.3 |  28.7 |  61.7 |  48.0 |  - |  - |  - |  -
PickNet (Chen et al., 2018) |  ResNet152 |  41.3 |  27.7 |  59.8 |  44.4 |  52.3 |  33.3 |  69.6 |  76.5
E2E (Li and Gong, 2019) |  Inception-ResNet-V2 |  40.4 |  27.0 |  61.0 |  48.3 |  50.3 |  34.1 |  70.8 |  87.5
RecNet(RL) (Zhang et al., 2019a) |  InceptionV4 |  39.2 |  27.5 |  60.3 |  48.7 |  52.3 |  34.1 |  69.8 |  80.3
DPRN (Xu et al., 2020) |  ResNeXt |  39.5 |  27.7 |  61.0 |  49.2 |  57.3 |  34.3 |  72.0 |  78.3
Wei et al. (Wei et al., 2020a) |  ResNet152 |  38.5 |  26.9 |  - |  43.7 |  46.8 |  34.4 |  - |  85.7

Figure 9. Experimental results of Video Description models on MSR-VTT and MSVD
dataset, where BLEU (B), METEOR (M), ROUGE (R), CIDEr (C), and No Result (NR)

###  4.3. Multimodal Visual Question Answering Results:

VQA is a system where a machine provides the answer in the form of words or
short phrases by combining CV and NLP domains. In this review article, MMVQA
models are categorized into MMJEM, MMAM, and MMEKM. Table 9 shows the
performance of VQA models by presenting findings of experimental results
conducted on the VQA and VQA family dataset on two test splits termed ”test-
std and test-dev” to determine the accuracy. VQA is a pretty big dataset
comprised of 265,016 pictures and open-ended image-related questions. A strong
grasp of language, vision, and common sense knowledge is required to answer
these questions. MCAN (Yu et al., 2019) achieves the best overall accuracy on
test-dev and test-std as compared to other comparative VQA models. MCAN is a
multimodal attention-based network, where the MCA layer uses image features
and questions self-attention for the reasoning part. This MCA layer refines
the question and image representations to improve the overall accuracy of the
model.

Table 9. Comparative analysis of Multimodal VQA model results on VQA dataset using benchmark evaluation metrics, where ”-” indicates no result |  | test-dev | test-std
---|---|---|---
Model | Feature Extractor | Y/N | Num | Others | All | Y/N | Num | Others | All
MUTAN (Ben-Younes et al., 2017) | ResNet152 | 82.88 | 44.54 | 56.50 | 66.01 | - | - | - | 66.38
MuReL (Cadene et al., 2019) | ResNet152 | 84.77 | 49.81 | 57.85 | 68.03 | - | - | - | 68.41
Wang et al. (Wang et al., 2017b) | VGG-Open ended | 81.2 | 37.7 | 50.5 | 61.7 | 81.3 | 36.7 | 50.9 | 61.9
VGG-Multiple Choice | 81.3 | 39.9 | 60.5 | 66.8 | 81.4 | 39.0 | 60.8 | 67.0
ResNet-Open ended | 81.5 | 38.4 | 53.0 | 63.1 | 81.4 | 38.2 | 53.2 | 63.3
ResNet-Multiple Choice | 81.5 | 40.0 | 62.2 | 67.7 | 81.4 | 39.8 | 62.3 | 67.8
MFB (Yu et al., 2017) | ResNet152 | 84.1 | 39.1 | 58.4 | 66.9 | 84.2 | 38.1 | 57.8 | 66.6
Up-Down (Anderson et al., 2018) | Faster RCNN, ResNet101 | 81.2 | 44.21 | 56.05 | 65.32 | - | - | - | 65.67
MCAN (Yu et al., 2019) | ResNet101 | 86.82 | 53.26 | 60.72 | 70.63 | - | - | - | 70.90
ReGAT (Li et al., 2019) | ResNet101 | 86.08 | 54.42 | 60.33 | 70.27 | - | - | - | 70.58
Guo et al. (Guo et al., 2020) | Faster RCNN | 87.0 | 53.06 | 60.19 | 70.43 | 86.97 | 52.62 | 60.72 | 70.72

###  4.4. Multimodal Speech synthesis results:

Comparative analysis of MM speech synthesis model’s results on U.S. English
speech database is presented in Table 10. For evaluation of speech synthesis,
experimental results Mean Opinion Score (MOS) and Mel-Cepstral Distortion
(MCD) are the most commonly used subjective and objective evaluation metrics.
Subjective assessment approaches are often better suited for assessing speech
synthesis models, but they demand substantial resources and confront problems
in terms of findings validity, reliability, and reproduction of results. In
contrast, the quantitative assessment of the Text To Speech model is performed
by the objective evaluation technique. The discrepancies between produced and
real speech data are commonly utilized to assess the model. These assessment
measures can only represent the model’s data parsing capacity to a limited
level and cannot indicate the produced speech’s quality. The MOS is the most
widely employed subjective assessment technique, which evaluates naturalness
by inviting participants to rate the synthetic speech. MOS higher scores
indicate better speech quality. MCD is the most commonly used objective
assessment method to determine the difference between actual and reproduced
samples. TACOTRON2 (Shen et al., 2018) DLTTS model performs better as compared
to other speech synthesis models. Parallel TACOTRON (Elias et al., 2020) is
also the closest model in terms of experimental results, but its relevancy is
much more than TACOTRON2.

##  5\. Discussion and Future Direction:

Issues and limitations of various methods and applications mentioned in
section 3 are highlighted below. Their’s possible future research directions
are also proposed separately.

###  5.1. Multimodal Image Description:

In recent years, the image description domain improves a lot, but there is
still some space available for improvement in some areas. Some methods did not
detect prominent attributes, objects and, to some extent does not generate
related or multiple captions. Here, some issues and their possible future
directions are mentioned below to make some advancement in the image
description.

  * •

The produced captions’ accuracy depends mostly on grammatically correct
captions, which depends on a powerful language generation model. Image
captioning approaches strongly depend on the quality and quantity of the
training data. Therefore, another key challenge in the future will be to
create large-scale image description datasets with accurate and detailed text
descriptions in an appropriate manner.

  * •

Exiting models demonstrate their output on datasets comprising a collection of
images from the same domain. Therefore, implement and test image description
models on open-source datasets will be another significant research
perspective in the future.

  * •

Too much work has been done on supervised learning approaches, and training
requires a vast volume of labeled data. Consequently, reinforcement and
unsupervised learning approaches have less amount of work done and have a lot
of space for improvement in the future.

  * •

More advanced attention-based image captioning mechanism or captioning based
on regions/multi-region is also a research direction in the future.

Table 10. Comparative analysis of MM Speech synthesis results on U.S. English speech database using Mean Opinion Score (MOS). Model | Architecture | MOS
---|---|---
TACOTRON_Griffin-Lim (Wang et al., 2017a) | Bidirectional GRU/RNN |  2.57 $\pm$ 0.28
TACOTRON_WaveNet |  4.17 $\pm$ 0.18
DeepVoice (Arik et al., 2017) | GRU/RNN |  2.05 $\pm$ 0.24
DeepVoice2 (Gibiansky et al., 2017) | GRU/RNN |  2.96 $\pm$ 0.38
DeepVoice3_Griffin-Lim (Ping et al., 2017) | Griffin-Lim, WaveNet Vocoder |  3.62 $\pm$ 0.31
DeepVoice3_WORLD |  3.63 $\pm$ 0.27
DeepVoice3_WaveNet |  3.78 $\pm$ 0.30
Parallel WaveNet (Oord et al., 2018) | CNN |  4.21 $\pm$ 0.081
TACOTRON2 (Shen et al., 2018) | SIFT, BiLSTM |  4.526 $\pm$ 0.066
Parallel TACOTRON_Transformer | GLU, VAE, LSTM |  4.36 $\pm$ 0.04
Parallel TACOTRON_LConv (Elias et al., 2020) |  4.40 $\pm$ 0.04

###  5.2. Multimodal Video Description:

The performance of the video description process is enhanced with the
advancement of DL techniques. Even though the present video description
method’s performance is already far underneath the human description process,
this accuracy rate is decreasing steadily, and still, there is sufficient
space available for enhancement in this area. Here, some issues and their
possible future directions are mentioned below to make some advancement in the
video description.

  * •

It is expected in the future that humans would be capable of interacting with
robots as humans can interact with each other. Video dialogue is a promising
area to cope with this circumstance similarly to audio dialogue (like, Hello
Google, Siri, ECHO, and Alexa). In visual Dialogue, the system needs to be
trained to communicate with humans or robots in a conversation/dialogue manner
by ignoring conversational statements’ correctness and wrongness.

  * •

In CV, most research has focused on descriptions of visual contents without
extraction of audio/speech features from the video. Existing approaches
extract features from visual frames or clips for a description of the video.
Therefore, extraction of these audio features may be more beneficial for the
improvement of this description process. Audio features, like the sound of
water splash, cat, dog, car, guitar, etc., provide occasional/eventual
information when no visual clue is available for their existence in the video.

  * •

Existing approaches have performed end-to-end training, resultantly more and
more data is utilized to improve the accuracy of the model. But extensive
dataset still does not cover the occurrences of real-world events. To improve
system performance in the future, learning from data itself is a better choice
and achieves its optimum computational power.

  * •

In the video description process, mostly humans describe the visual content
based on extensive prior knowledge. They do not all the time rely solely on
visual contents; some additional background knowledge is applied by the domain
expertise as well. Augmentation of video description approaches with some
existing external knowledge would be an attractive and promising technique in
this research area. This technique shows some better performance in VQA. Most
likely, this would dominate some accuracy improvement in this domain as well.

  * •

Video description can be used in combination with machine translation for auto
video subtitling. This combination is very beneficial for entertainment and
other areas as well. So this combination needs to be focused on making the
video subtitling process easier and cost-effective in the future.

###  5.3. Multimodal Visual Question Answering:

In recent years, the VQA task has gained tremendous attention and accelerated
the development process by the contribution of various authors in this
specific domain. VQA is especially enticing because it comprises a complete AI
task that considers free-form, open-ended answers to questions by extracting
features from an image/video and the question asked. The accuracy of VQA
research still goes beyond answering the visual-based questions as compared to
human equivalent accuracy. Some issues and their possible future directions
for the VQA task are mentioned below.

  * •

So far, the image/video feature extraction part is almost fixed to a model
like, most of the time ImageNet Challenge model is used. In this model, image
features are extracted by the split of frames into uniform boxes that works
well for object detection, but VQA needs features by tracking all objects
using semantic segmentation. Therefore, some more visual feature extraction
mechanisms for VQA tasks need to be explored in the future.

  * •

Goal-oriented datasets for VQA research need to be designed in feature to
support real-time applications like instructing users to play the game,
helping visually impaired peoples, support for data extraction from colossal
data pool and robot interaction, etc. So far, VizWiz (Gurari et al., 2018) and
VQA-MED (Abacha et al., 2019)are two publicly available goal-oriented datasets
for VQA research. Therefore, in the future, more goal-oriented datasets need
to be built for mentioned above applications.

  * •

Most baseline VQA methods have been tested using traditional accuracy measure
that is adequate for the multiple-choice format. In the future, the assessment
or evaluation method for open-ended frameworks for VQA tasks is examined to
improve these models’ accuracy.

###  5.4. Multimodal Speech synthesis:

The DLTTS models use distributed representations and complete context
information to substitute the clustering phase of the decision tree in HMM
models. Therefore, to produce a better quality of speech synthesis process
compared to the traditional method, several hidden layers are used to map
context features to high dimensional acoustic features by DLTTS methods. More
hidden layers inevitably raise the number of system parameters to achieve
better performance. As a result, space and time complexity for system training
is also increased. Therefore for network training, a large amount of
computational resources and corpora is required. Besides these achievements,
there is still room available for DLTTS models in terms of quality improvement
in intelligibility and speech’s naturalness. Therefore, some issues and their
feature research directions are discussed below.

  * •

DLTTS approaches usually require a huge amount of high-quality (text, speech)
pairs for training, which is time-consuming and an expensive process.
Therefore in the future, data efficiency for E2E DLTTS models training is
improved by the publicly availability of unpaired text and speech recordings
on a large scale.

  * •

Little progress is made in front-end text analysis to extract valuable context
or linguistic features to minimize the gap between the text-speech synthesis
process. Therefore it is a good direction in the future to utilize specific
context or linguistic information for E2E DLTTS systems.

  * •

Parallelization will be an essential aspect for DLTTS systems to improve
system efficiency because most DNN architecture needs many calculations. Some
frameworks proposed in recent years also use parallel networks for training or
inference and achieve some good results, but there is still room available to
achieve optimum results in the future.

  * •

As for DLTTS application concerns, the use of speech synthesis for other real-
world applications like voice conversion or translation, cross-lingual speech
conversion, audio-video speech synthesis, etc., are good future research
directions.

###  5.5. Multimodal Emotion Recognition:

Analysis of automatic emotions requires some advanced modeling and recognition
techniques along with AI systems. For more advanced emotion recognition
systems future research based on AI based automated systems constitutes more
scientific progress in this area. Some future directions of listed issues of
multimodal emotion recognition are mentioned below.

  * •

Existing baseline approaches are successful, but further experience,
knowledge, and tools regarding the analysis and measurement of automatic non-
invasive emotions are required.

  * •

Humans use more modalities to recognize emotions and are compatible with
signal processing; machines are expected to behave similarly. But the
performance of automated systems is limited with a restricted set of data.
Therefore, to overcome this limitation, new multimodal recordings with a more
representative collection of subjects are needed to be considered in the
future.

  * •

The preprocessing complexity of physiological signals in emotion detection is
a big challenge. In physiological signals, detection of emotion states through
electrocardiogram, electromyography, and skin temperature is still emerging.
Hence, to determine the potency of these techniques a detailed research can be
carried out in the future.

###  5.6. Multimodal Event Detection:

Due to the exponential increase in web data, multimodal event detection has
attracted significant research attention in recent years. MMED seeks to
determine a collection of real-world events in a large set of social media
data. Subspace learning is an efficient approach to handle the issue of
heterogeneity for the learning of features from multimodal data. On the other
hand, E2E learning models are more versatile because they are structured to
obtain heterogeneous data correlations directly. MMED from social data is
still challenging and needs to be improved in some open issues in the future.

  * •

The ”curse of dimensionality” issue is raised with the concatenation of
various feature vectors of multiple modalities. Some considerable advancement
has been made in curse of dimensionality issue, but existing techniques still
require some improvement in terms of achieving better learning accuracy for
social platforms. Like, GAN or some RNN architectures’ extensions are valuable
to improve feature learning accuracy of multimodal social data.

  * •

Textual data also coincide with audio and video media. Therefore, MMED is
improved if this textual information is considered jointly with these media.

  * •

Current research primarily focuses on the identification of events from a
single social platform. A comprehensive understanding of social data events is
obtained by synthesizing the information from multiple social platforms.
Hence, event detection methods can be implemented simultaneously to examine
social data from multiple platforms using a transfer learning strategy.

##  6\. Conclusion:

In this survey, we discussed the recent advancements and trends in MMDL.
Various DL methods are categorized into different MMDL application groups and
explained thoroughly using multiple media. This article focuses on an up-to-
date review of numerous applications using various modalities like image,
audio, video, text, body gesture, facial expression, and physiological signals
compared to previous related surveys. A novel fine-grained taxonomy of various
MMDL methods is proposed. Additionally, a brief discussion on architectures,
datasets, and evaluation metrics used in these MMDL methods and applications
is provided. A detailed comparative analysis is provided for each group of
applications by discussing the model’s architectures, media, datasets,
evaluation metrics, features. Experimental results comparisons for MMDL
applications are provided on benchmark datasets and evaluation metrics.
Finally, open research problems are also mentioned separately for each group
of applications, and possible research directions for the future are listed in
detail. We expected that our proposed taxonomy and research directions would
promote future studies in multimodal deep learning and helps in a better
understanding of unresolved issues in this particular area.

## References

  * (1)
  * Aafaq et al. (2019) Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian. 2019.  Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 12487–12496.
  * Abacha et al. (2019) Asma Ben Abacha, Sadid A Hasan, Vivek Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller. 2019.  VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019.. In _CLEF (Working Notes)_.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6077–6086.
  * Arik et al. (2018) Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. 2018.  Neural voice cloning with a few samples. In _Advances in Neural Information Processing Systems_. 10019–10029.
  * Arik et al. (2017) Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. 2017\.  Deep voice: Real-time neural text-to-speech.  _arXiv preprint arXiv:1702.07825_ (2017).
  * Atrey et al. (2010) Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. 2010.  Multimodal fusion for multimedia analysis: a survey.  _Multimedia systems_ 16, 6 (2010), 345–379.
  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007\.  Dbpedia: A nucleus for a web of open data.  In _The semantic web_. Springer, 722–735.
  * Bahrick (1983) Lorraine E Bahrick. 1983\.  Infants’ perception of substance and temporal synchrony in multimodal events.  _Infant Behavior and Development_ 6, 4 (1983), 429–451.
  * Bai and An (2018) Shuang Bai and Shan An. 2018.  A survey on automatic image caption generation.  _Neurocomputing_ 311 (2018), 291–304.
  * Baltrušaitis et al. (2018) Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 2 (2018), 423–443.
  * Basu et al. (2020) Kinjal Basu, Farhad Shakerin, and Gopal Gupta. 2020\.  Aqua: Asp-based visual question answering. In _International Symposium on Practical Aspects of Declarative Languages_. Springer, 57–72.
  * Ben-Younes et al. (2017) Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. 2017.  Mutan: Multimodal tucker fusion for visual question answering. In _Proceedings of the IEEE international conference on computer vision_. 2612–2620.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020\.  Experience grounds language.  _arXiv preprint arXiv:2004.10151_ (2020).
  * Boateng (2020) George Boateng. 2020\.  Towards Real-Time Multimodal Emotion Recognition among Couples. In _Proceedings of the 2020 International Conference on Multimodal Interaction_. 748–753.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge. In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_. 1247–1250.
  * Cadene et al. (2019) Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nicolas Thome. 2019.  Murel: Multimodal relational reasoning for visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1989–1998.
  * Cao et al. (2019) Pengfei Cao, Zhongyi Yang, Liang Sun, Yanchun Liang, Mary Qu Yang, and Renchu Guan. 2019\.  Image Captioning with Bidirectional Semantic Attention-Based Guiding of Long Short-Term Memory.  _Neural Processing Letters_ 50, 1 (2019), 103–119.
  * Carletta et al. (2005) Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005\.  The AMI meeting corpus: A pre-announcement. In _International workshop on machine learning for multimodal interaction_. Springer, 28–39.
  * Chen et al. (2021b) Jing Chen, Chenhui Wang, Kejun Wang, Chaoqun Yin, Cong Zhao, Tao Xu, Xinyi Zhang, Ziqiang Huang, Meichen Liu, and Tao Yang. 2021b.  HEU Emotion: a large-scale database for multimodal emotion recognition in the wild.  _Neural Computing and Applications_ (2021), 1–17.
  * Chen et al. (2021a) Long Chen, Zhihong Jiang, Jun Xiao, and Wei Liu. 2021a.  Human-like controllable image captioning with verb-specific semantic roles. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16846–16856.
  * Chen et al. (2017) Minghai Chen, Guiguang Ding, Sicheng Zhao, Hui Chen, Qiang Liu, and Jungong Han. 2017\.  Reference based LSTM for image captioning. In _Thirty-first AAAI conference on artificial intelligence_.
  * Chen et al. (2018) Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. 2018\.  Less is more: Picking informative frames for video captioning. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 358–373.
  * Cheng et al. (2020) Ling Cheng, Wei Wei, Xianling Mao, Yong Liu, and Chunyan Miao. 2020.  Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation.  _IEEE Access_ (2020).
  * Chong et al. (2019) Luyao Chong, Meng Jin, and Yuan He. 2019.  EmoChat: Bringing multimodal emotion detection to mobile conversation. In _2019 5th International Conference on Big Data Computing and Communications (BIGCOM)_. IEEE, 213–221.
  * Cimtay et al. (2020) Yucel Cimtay, Erhan Ekmekcioglu, and Seyma Caglar-Ozhan. 2020\.  Cross-subject multimodal emotion recognition based on hybrid fusion.  _IEEE Access_ 8 (2020), 168865–168878.
  * Cukurova et al. (2020) Mutlu Cukurova, Michail Giannakos, and Roberto Martinez-Maldonado. 2020\.  The promise and challenges of multimodal learning analytics.  , 1441–1449 pages.
  * Desta et al. (2018) Mikyas T Desta, Larry Chen, and Tomasz Kornuta. 2018\.  Object-based reasoning in VQA. In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 1814–1823.
  * Elias et al. (2020) Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Ron Weiss, and Yonghui Wu. 2020.  Parallel Tacotron: Non-Autoregressive and Controllable TTS.  _arXiv preprint arXiv:2010.11439_ (2020).
  * Fang et al. (2020) Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020.  Video2commonsense: Generating commonsense descriptions to enrich video captioning.  _arXiv preprint arXiv:2003.05162_ (2020).
  * Feng et al. (2019) Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. 2019\.  Unsupervised image captioning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4125–4134.
  * Gao et al. (2020) Jing Gao, Peng Li, Zhikui Chen, and Jianing Zhang. 2020\.  A survey on deep learning for multimodal data fusion.  _Neural Computation_ 32, 5 (2020), 829–864.
  * Gao et al. (2017) Yue Gao, Hanwang Zhang, Xibin Zhao, and Shuicheng Yan. 2017\.  Event classification in microblogs via social tracking.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 8, 3 (2017), 1–14.
  * Gibiansky et al. (2017) Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017\.  Deep voice 2: Multi-speaker neural text-to-speech. In _Advances in neural information processing systems_. 2962–2970.
  * Guo et al. (2019a) Longteng Guo, Jing Liu, Peng Yao, Jiangwei Li, and Hanqing Lu. 2019a.  Mscap: Multi-style image captioning with unpaired stylized text. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4204–4213.
  * Guo et al. (2019b) Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019b.  Deep multimodal representation learning: A survey.  _IEEE Access_ 7 (2019), 63373–63394.
  * Guo et al. (2020) Wenya Guo, Ying Zhang, Xiaoping Wu, Jufeng Yang, Xiangrui Cai, and Xiaojie Yuan. 2020\.  Re-Attention for Visual Question Answering. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 91–98.
  * Gurari et al. (2018) Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018\.  Vizwiz grand challenge: Answering visual questions from blind people. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3608–3617.
  * Hazarika et al. (2018a) Devamanyu Hazarika, Sruthi Gorantla, Soujanya Poria, and Roger Zimmermann. 2018a.  Self-attentive feature-level fusion for multimodal emotion detection. In _2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_. IEEE, 196–201.
  * Hazarika et al. (2018b) Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018b.  Icon: Interactive conversational memory network for multimodal emotion detection. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 2594–2604.
  * He et al. (2019) Xinwei He, Baoguang Shi, Xiang Bai, Gui-Song Xia, Zhaoxiang Zhang, and Weisheng Dong. 2019\.  Image caption generation with part of speech guidance.  _Pattern Recognition Letters_ 119 (2019), 229–237.
  * Hoffman-Plotkin and Twentyman (1984) Debbie Hoffman-Plotkin and Craig T Twentyman. 1984.  A multimodal assessment of behavioral and cognitive deficits in abused and neglected preschoolers.  _Child development_ (1984), 794–802.
  * Hong et al. (2020) Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, Qian Du, and Bing Zhang. 2020.  More diverse means better: Multimodal deep learning meets remote-sensing imagery classification.  _IEEE Transactions on Geoscience and Remote Sensing_ 59, 5 (2020), 4340–4354.
  * Huan et al. (2020) Ruo-Hong Huan, Jia Shu, Sheng-Lin Bao, Rong-Hua Liang, Peng Chen, and Kai-Kai Chi. 2020\.  Video multimodal emotion recognition based on Bi-GRU and attention fusion.  _Multimedia Tools and Applications_ (2020), 1–28.
  * Huang et al. (2018) Shaonian Huang, Dongjun Huang, and Xinmin Zhou. 2018\.  Learning multimodal deep representations for crowd anomaly event detection.  _Mathematical Problems in Engineering_ 2018 (2018).
  * Huang et al. (2017) Yongrui Huang, Jianhao Yang, Pengkai Liao, and Jiahui Pan. 2017\.  Fusion of facial expressions and EEG for multimodal emotion recognition.  _Computational intelligence and neuroscience_ 2017 (2017).
  * Jaiswal et al. (2019) Mimansa Jaiswal, Zakaria Aldeneh, and Emily Mower Provost. 2019\.  Controlling for confounders in multimodal emotion classification via adversarial learning. In _2019 International Conference on Multimodal Interaction_. 174–184.
  * Ji et al. (2021) Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, and Rongrong Ji. 2021\.  Improving image captioning by leveraging intra-and inter-layer global representation in transformer network. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 1655–1663.
  * Jiang et al. (2021) Weitao Jiang, Xiying Li, Haifeng Hu, Qiang Lu, and Bohong Liu. 2021.  Multi-Gate Attention Network for Image Captioning.  _IEEE Access_ 9 (2021), 69700–69709.
  * Jiang et al. (2018) Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. 2018.  Recurrent fusion network for image captioning. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 499–515.
  * Juang and Rabiner (1991) Biing Hwang Juang and Laurence R Rabiner. 1991.  Hidden Markov models for speech recognition.  _Technometrics_ 33, 3 (1991), 251–272.
  * Koutras et al. (2018) Petros Koutras, Athanasia Zlatinsi, and Petros Maragos. 2018\.  Exploring cnn-based architectures for multimodal salient event detection in videos. In _2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP)_. IEEE, 1–5.
  * Krishna et al. (2017) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017.  Dense-captioning events in videos. In _Proceedings of the IEEE international conference on computer vision_. 706–715.
  * Lai et al. (2020) Helang Lai, Hongying Chen, and Shuangyan Wu. 2020\.  Different Contextual Window Sizes Based RNNs for Multimodal Emotion Detection in Interactive Conversations.  _IEEE Access_ 8 (2020), 119516–119526.
  * Lazarus (1973) Arnold A Lazarus. 1973\.  Multimodal behavior therapy: Treating the” BASIC ID.”.  _Journal of Nervous and Mental Disease_ (1973).
  * Lei and Huang (2021) Z Lei and Y Huang. 2021\.  Video Captioning Based on Channel Soft Attention and Semantic Reconstructor. Future Internet 2021, 13, 55.
  * Li et al. (2019) Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019\.  Relation-aware graph attention network for visual question answering. In _Proceedings of the IEEE International Conference on Computer Vision_. 10313–10322.
  * Li and Gong (2019) Lijun Li and Boqing Gong. 2019.  End-to-end video captioning with multitask reinforcement learning. In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 339–348.
  * Li et al. (2017) Linghui Li, Sheng Tang, Yongdong Zhang, Lixi Deng, and Qi Tian. 2017.  GLA: Global–local attention for image description.  _IEEE Transactions on Multimedia_ 20, 3 (2017), 726–737.
  * Li et al. (2020) Minjia Li, Lun Xie, Zeping Lv, Juan Li, and Zhiliang Wang. 2020.  Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things.  _IEEE Access_ 8 (2020), 187208–187221.
  * Li et al. (2018) Yingming Li, Ming Yang, and Zhongfei Zhang. 2018.  A survey of multi-view representation learning.  _IEEE transactions on knowledge and data engineering_ 31, 10 (2018), 1863–1883.
  * Liu and Singh (2004) Hugo Liu and Push Singh. 2004.  ConceptNet—a practical commonsense reasoning tool-kit.  _BT technology journal_ 22, 4 (2004), 211–226.
  * Liu et al. (2020a) Maofu Liu, Huijun Hu, Lingjun Li, Yan Yu, and Weili Guan. 2020a.  Chinese Image Caption Generation via Visual Attention and Topic Modeling.  _IEEE Transactions on Cybernetics_ (2020).
  * Liu et al. (2020b) Maofu Liu, Lingjun Li, Huijun Hu, Weili Guan, and Jing Tian. 2020b.  Image caption generation with dual attention mechanism.  _Information Processing & Management_ 57, 2 (2020), 102178\.
  * Liu et al. (2020c) Sheng Liu, Zhou Ren, and Junsong Yuan. 2020c.  Sibnet: Sibling convolutional encoder for video captioning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2020).
  * Lobry et al. (2020) Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia. 2020\.  RSVQA: Visual Question Answering for Remote Sensing Data.  _IEEE Transactions on Geoscience and Remote Sensing_ (2020).
  * Long et al. (2021) Yu Long, Pengjie Tang, Hanli Wang, and Jian Yu. 2021\.  Improving reasoning with contrastive visual information for visual question answering.  _Electronics Letters_ 57, 20 (2021), 758–760.
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019.  Ok-vqa: A visual question answering benchmark requiring external knowledge. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3195–3204.
  * McGurk and MacDonald (1976) Harry McGurk and John MacDonald. 1976.  Hearing lips and seeing voices.  _Nature_ 264, 5588 (1976), 746–748.
  * Miller (1995) George A Miller. 1995\.  WordNet: a lexical database for English.  _Commun. ACM_ 38, 11 (1995), 39–41.
  * Mittal et al. (2020) Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020\.  M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues.. In _AAAI_. 1359–1367.
  * Mogadala et al. (2019) Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. 2019\.  Trends in integration of vision and language research: A survey of tasks, datasets, and methods.  _arXiv preprint arXiv:1907.09358_ (2019).
  * Morency ([n. d.]) Louis-Philippe Morency. [n. d.].  Multimodal Machine Learning (or Deep Learning for Multimodal Systems).  https://www.microsoft.com/en-us/research/wp-content/uploads/2017/07/Integrative_AI_Louis_Philippe_Morency.pdf
  * Mulligan and Shaw (1980) Robert M Mulligan and Marilyn L Shaw. 1980.  Multimodal signal detection: Independent decisions vs. integration.  _Perception & Psychophysics_ 28, 5 (1980), 471–478.
  * Mun et al. (2019) Jonghwan Mun, Linjie Yang, Zhou Ren, Ning Xu, and Bohyung Han. 2019.  Streamlined dense video captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6588–6597.
  * Narasimhan and Schwing (2018) Medhini Narasimhan and Alexander G Schwing. 2018.  Straight to the facts: Learning knowledge base retrieval for factual visual question answering. In _Proceedings of the European conference on computer vision (ECCV)_. 451–468.
  * Nguyen et al. (2018) Dung Nguyen, Kien Nguyen, Sridha Sridharan, David Dean, and Clinton Fookes. 2018.  Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition.  _Computer Vision and Image Understanding_ 174 (2018), 33–42.
  * Nguyen et al. (2017) Dung Nguyen, Kien Nguyen, Sridha Sridharan, Afsane Ghasemi, David Dean, and Clinton Fookes. 2017\.  Deep spatio-temporal features for multimodal emotion recognition. In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 1215–1223.
  * Oord et al. (2018) Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. 2018\.  Parallel wavenet: Fast high-fidelity speech synthesis. In _International conference on machine learning_. PMLR, 3918–3926.
  * Pantic et al. (2011) Maja Pantic, Roderick Cowie, Francesca D’Errico, Dirk Heylen, Marc Mehu, Catherine Pelachaud, Isabella Poggi, Marc Schroeder, and Alessandro Vinciarelli. 2011.  Social signal processing: the research agenda.  In _Visual analysis of humans_. Springer, 511–538.
  * Patro et al. (2020) Badri Patro, Shivansh Patel, and Vinay Namboodiri. 2020\.  Robust Explanations for Visual Question Answering. In _The IEEE Winter Conference on Applications of Computer Vision_. 1577–1586.
  * Pei et al. (2019) Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. 2019\.  Memory-attended recurrent network for video captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8347–8356.
  * Perez-Martin et al. (2021) Jesus Perez-Martin, Benjamin Bustos, and Jorge Pérez. 2021\.  Improving video captioning with temporal composition of a visual-syntactic embedding. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 3039–3049.
  * Petajan (1985) Eric David Petajan. 1985\.  Automatic Lipreading to Enhance Speech Recognition (Speech Reading).  (1985).
  * Ping et al. (2017) Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. 2017\.  Deep voice 3: Scaling text-to-speech with convolutional sequence learning.  _arXiv preprint arXiv:1710.07654_ (2017).
  * Rahman et al. (2020) Md Rahman, Thasin Abedin, Khondokar SS Prottoy, Ayana Moshruba, Fazlul Hasan Siddiqui, et al. 2020\.  Semantically Sensible Video Captioning (SSVC).  _arXiv preprint arXiv:2009.07335_ (2020).
  * Ramachandram and Taylor (2017) Dhanesh Ramachandram and Graham W Taylor. 2017.  Deep multimodal learning: A survey on recent advances and trends.  _IEEE Signal Processing Magazine_ 34, 6 (2017), 96–108.
  * Sengupta et al. (2020) Saptarshi Sengupta, Sanchita Basak, Pallabi Saikia, Sayak Paul, Vasilios Tsalavoutis, Frederick Atiah, Vadlamani Ravi, and Alan Peters. 2020.  A review of deep learning with special emphasis on architectures, applications and recent trends.  _Knowledge-Based Systems_ (2020), 105596.
  * Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018\.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 4779–4783.
  * Snoek and Worring (2005) Cees GM Snoek and Marcel Worring. 2005.  Multimodal video indexing: A review of the state-of-the-art.  _Multimedia tools and applications_ 25, 1 (2005), 5–35.
  * Souza et al. (2021) Rafael Souza, Andre Fernandes, Thiago SFX Teixeira, George Teodoro, and Renato Ferreira. 2021\.  Online multimedia retrieval on CPU–GPU platforms with adaptive work partition.  _J. Parallel and Distrib. Comput._ 148 (2021), 31–45.
  * Taigman et al. (2018) Yaniv Taigman, Lior Wolf, A. Polyak, and Eliya Nachmani. 2018\.  VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop.  _arXiv: Learning_ (2018).
  * Tandon et al. (2014) Niket Tandon, Gerard De Melo, Fabian Suchanek, and Gerhard Weikum. 2014.  Webchild: Harvesting and organizing commonsense knowledge from the web. In _Proceedings of the 7th ACM international conference on Web search and data mining_. 523–532.
  * Tao and Busso (2020) Fei Tao and Carlos Busso. 2020.  End-to-end audiovisual speech recognition system with multitask learning.  _IEEE Transactions on Multimedia_ 23 (2020), 1–11.
  * Tripathi and Beigi (2018) Samarth Tripathi and Homayoon Beigi. 2018.  Multi-modal emotion recognition on IEMOCAP with neural networks.  _arXiv preprint arXiv:1804.05788_ (2018).
  * Tur et al. (2008) Gokhan Tur, Andreas Stolcke, Lynn Voss, John Dowding, Benoît Favre, Raquel Fernández, Matthew Frampton, Michael Frandsen, Clint Frederickson, Martin Graciarena, et al. 2008\.  The CALO meeting speech recognition and understanding system. In _2008 IEEE Spoken Language Technology Workshop_. IEEE, 69–72.
  * Waibel et al. (2009) Alex Waibel, Hartwig Steusloff, Rainer Stiefelhagen, and Kym Watson. 2009.  Computers in the human interaction loop.  In _Computers in the Human Interaction Loop_. Springer, 3–6.
  * Wang et al. (2018c) Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. 2018c.  Reconstruction network for video captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 7622–7631.
  * Wang et al. (2020) Bin Wang, Cungang Wang, Qian Zhang, Ying Su, Yang Wang, and Yanyan Xu. 2020\.  Cross-Lingual Image Caption Generation Based on Visual Attention Model.  _IEEE Access_ 8 (2020), 104543–104554.
  * Wang et al. (2018d) Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2018d.  Fvqa: Fact-based visual question answering.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 10 (2018), 2413–2427.
  * Wang et al. (2017b) Peng Wang, Qi Wu, Chunhua Shen, and Anton van den Hengel. 2017b.  The vqa-machine: Learning how to use existing vision algorithms to answer new questions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1173–1182.
  * Wang et al. (2018b) Wei Wang, Yuxuan Ding, and Chunna Tian. 2018b.  A novel semantic attribute-based feature for image caption generation. In _2018 IEEE International conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 3081–3085.
  * Wang et al. (2018a) Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. 2018a.  Video captioning via hierarchical reinforcement learning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4213–4222.
  * Wang et al. (2017a) Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017a.  Tacotron: Towards end-to-end speech synthesis.  _arXiv preprint arXiv:1703.10135_ (2017).
  * Wei et al. (2020a) Ran Wei, Li Mi, Yaosi Hu, and Zhenzhong Chen. 2020a.  Exploiting the local temporal information for video captioning.  _Journal of Visual Communication and Image Representation_ 67 (2020), 102751\.
  * Wei et al. (2020b) Yiwei Wei, Leiquan Wang, Haiwen Cao, Mingwen Shao, and Chunlei Wu. 2020b.  Multi-Attention Generative Adversarial Network for image captioning.  _Neurocomputing_ 387 (2020), 91–99.
  * Wu and Hu (2017) Jie Wu and Haifeng Hu. 2017\.  Cascade recurrent neural network for image caption generation.  _Electronics Letters_ 53, 25 (2017), 1642–1643.
  * Xi et al. (2020) Yuling Xi, Yanning Zhang, Songtao Ding, and Shaohua Wan. 2020\.  Visual question answering model based on visual relationship detection.  _Signal Processing: Image Communication_ 80 (2020), 115648.
  * Xu et al. (2020) Wanru Xu, Jian Yu, Zhenjiang Miao, Lili Wan, Yi Tian, and Qiang Ji. 2020\.  Deep Reinforcement Polishing Network for Video Captioning.  _IEEE Transactions on Multimedia_ (2020).
  * Yang et al. (2019) Zhenguo Yang, Qing Li, Wenyin Liu, and Jianming Lv. 2019\.  Shared multi-view data representation for multi-domain event detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 42, 5 (2019), 1243–1256.
  * Yazdavar et al. (2020) Amir Hossein Yazdavar, Mohammad Saeid Mahdavinejad, Goonmeet Bajaj, William Romine, Amit Sheth, Amir Hassan Monadjemi, Krishnaprasad Thirunarayan, John M Meddar, Annie Myers, Jyotishman Pathak, et al. 2020\.  Multimodal mental health analysis in social media.  _Plos one_ 15, 4 (2020), e0226248.
  * Yu et al. (2020) Jing Yu, Zihao Zhu, Yujing Wang, Weifeng Zhang, Yue Hu, and Jianlong Tan. 2020\.  Cross-modal knowledge reasoning for knowledge-based visual question answering.  _Pattern Recognition_ 108 (2020), 107563.
  * Yu et al. (2019) Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019.  Deep modular co-attention networks for visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6281–6290.
  * Yu et al. (2017) Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. 2017\.  Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In _Proceedings of the IEEE international conference on computer vision_. 1821–1830.
  * Yuhas et al. (1989) Ben P Yuhas, Moise H Goldstein, and Terrence J Sejnowski. 1989\.  Integration of acoustic and visual speech signals using neural networks.  _IEEE Communications Magazine_ 27, 11 (1989), 65–71.
  * Zhang et al. (2020b) Chao Zhang, Zichao Yang, Xiaodong He, and Li Deng. 2020b.  Multimodal intelligence: Representation learning, information fusion, and applications.  _IEEE Journal of Selected Topics in Signal Processing_ (2020).
  * Zhang et al. (2019b) Su-Fang Zhang, Jun-Hai Zhai, Bo-Jun Xie, Yan Zhan, and Xin Wang. 2019b.  Multimodal Representation Learning: Advances, Trends and Challenges. In _2019 International Conference on Machine Learning and Cybernetics (ICMLC)_. IEEE, 1–6.
  * Zhang et al. (2019a) Wei Zhang, Bairui Wang, Lin Ma, and Wei Liu. 2019a.  Reconstruct and represent video contents for captioning via reinforcement learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2019).
  * Zhang et al. (2018) Zongjian Zhang, Qiang Wu, Yang Wang, and Fang Chen. 2018\.  High-quality image captioning with fine-grained and semantic-guided visual attention.  _IEEE Transactions on Multimedia_ 21, 7 (2018), 1681–1693.
  * Zhang et al. (2020a) Zhiwang Zhang, Dong Xu, Wanli Ouyang, and Luping Zhou. 2020a.  Dense video captioning using graph-based sentence summarization.  _IEEE Transactions on Multimedia_ 23 (2020), 1799–1810.