# Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-
Modal Retrieval

Donghuo Zeng  KDDI Research, Inc.2-1-15 OharaFujiminoSaitama356-8502Japan ,
Jianming Wu  KDDI Research, Inc.2-1-15 OharaFujiminoSaitama356-8502Japan ,
Gen Hattori  KDDI Research, Inc.2-1-15 OharaFujiminoSaitama356-8502Japan ,  Yi
Yu  National Institute of Informatics, SOKENDAI2-1-2 HitotsubashiChiyoda-
kuTokyo100-0003Japan and  Rong Xu

(2018)

###### Abstract.

Learning common subspace is prevalent way to solve the problem of data from
different modalities having inconsistent distributions and representations
that cannot be directly compared when achieved in cross-modal retrieval.
Previous cross-modal retrieval methods focus on projecting the data between
modalities into a common latent subspace by learning the correlation between
them to bridge the modality gap. However, due to the rich semantic information
in the video, the heterogeneous nature of audio-visual data leads to
heterogeneous gaps in the distribution of modalities, which are difficult to
be bridged into a common subspace by the previous cross-modality learning
methods, which may lead to the loss of important semantic content in the video
when the modality gap is eliminated by supervised learning, e.g., optimizing
pairwise correlation, while the semantics of the categories may undermine the
properties of the original features In this work, we aim to learn effective
audio-visual representations to support audio-visual cross-modal retrieval
(AVCMR). We propose a novel model that maps audio-visual modalities into two
distinct shared latent subspaces: explicit and implicit shared spaces. In
particular, the explicit shared space is used to optimize pairwise
correlations, where learned representations across modalities capture the
commonalities of audio-visual pairs and reduce the modality gap. The implicit
shared space is used to preserve the distinctive features between modalities
by maintaining the discrimination of audio/video patterns from different
semantic categories. Finally, the fusion of the features learned from the two
latent subspaces is used for the similarity computation of the AVCMR task. The
comprehensive experimental results on two audio-visual datasets demonstrate
that our proposed model for using two different latent subspaces for audio-
visual cross-modal learning is effective and significantly outperforms the
state-of-the-art cross-modal models that learn features from a single
subspace.

Explicit common subspace, implicit common subspace, CCA layer, audio-visual
cross-modal learning

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††journal: JACM††journalvolume: 37††journalnumber:
4††article: 111††publicationmonth: 8††ccs: Computer systems organization
Embedded systems††ccs: Computer systems organization Redundancy††ccs: Computer
systems organization Robotics††ccs: Networks Network reliability

##  1\. Introduction

Figure 1. Audio and visual are two main tracks from annotated video, the
relation between audio and visual is captioned by representations learning
method via projecting the audio-visual features into two distinct subspaces:
explicit and implicit common subspaces to learn new representations,
respectively. At last, the fusion features of these learned representations
are utilized for audio-visual cross-modal similarity calculation.

With the development of the Internet and the gradual improvement of the
functions of mobile devices, users are progressively becoming self-publishing
consumers in recent years, short video applications such as TikTok, Instagram,
Snapchat have been users’ favourite entertainment tools, some people almost
cannot live without short videos. In fact, videos have become the next
generation of artificial intelligence, the rich semantics provide us with
challenges and opportunities to extract useful information and build
interaction between different modalities. Cross-modal learning methods have
found success at the intersection of computer vision and natural language
processing. However, as recently published papers (Zhen et al., 2019; Roy et
al., 2020) show, there is still room for improvement in some common image-text
benchmark datasets, such as the Wikipedia dataset (Pereira et al., 2013) and
the NUS-WIDE-10k dataset (Chua et al., 2009).

Moreover, there are few research works and available datasets on audio-visual
cross-modal learning, which should be taken into account. In recent years,
some works have applied audio-visual joint embedding learning to some tasks,
such as Multimodal Sentiment Analysis (Hazarika et al., 2020), audio-visual
separation (Lu et al., 2018), Visual2sound generation (Zhou et al., 2018), and
audio-visual cross-modal retrieval (Zeng et al., 2020). The AVCMR task faces
the same challenge as other cross-modal learning methods, since the data from
different modalities cannot be directly compared. In addition, audio-visual
cross-modal learning is more challenge and potential than other combination of
cross-modal learning such as image-text because the research situation of
audio-visual cross-modal learning is still in its infancy.

A general approach to bridge the heterogeneous gap of two different modalities
is by representation learning method (Zhu et al., 2019; Xu et al., 2020; Wang
et al., 2017; Zeng and Oyama, 2019), where a common subspace is learned to
obtain the same dimension of the representation in all modalities by
optimizing the correlation between them, so that the similarity of the new
representations can be measured directly. In the past, several cross-modal
learning methods have been reported that use different methods to project the
cross-modal data into a common subspace. Traditional methods such as CCA-
variant (Hardoon et al., 2004; Rasiwasia et al., 2014; Zeng et al., 2020)
learning linear/non-linear projections by optimizing pairwise correlation to
generate a pair of new cross-modal features, some CCA-variant (Hardoon et al.,
2004; Zeng et al., 2020) have success in incorporating deep Learning methods
to learn high-level semantic representation. For this reason, we adopt the
idea of combining pairwise correlation learning and deep learning methods in
the explicit common subspace.

Audio and visual are different modalities that are extracted from the same
video, moreover, they share the same motive and goal of semantics that allows
us to learn the correlation between them by cross-modal learning methods. To
ensure the correlation do not deviate from the original semantic information
such as human annotation. Unlike the YouTube-8M (Abu-El-Haija et al., 2016),
VGG-SOUND (Chen et al., 2020), and Kinetics (Kay et al., 2017) dataset are
labeled by computer vision techniques, in our work, we use the audio-visual
datasets including VEGAS (Zhou et al., 2018) and AVE (Tian et al., 2018) with
audio-visual human double-check labels. Compared with the previous works (Wang
et al., 2016a, 2017), the category information is only used to learn
individual features for each modality or to learn from the inter-modalities,
which may cause the semantic information to be lost during the joint subspace
learning. In this case, we learn the semantic information in an additional
shared subspace without explicit learning of a shared representation in the
implicit shared subspace. To avoid that the two learned joint subspaces learn
from different directions, which leads to the inability to merge the learned
representation. The implicit shared subspace is the supplement of explicit
shared subspace, which is unsupervised learning is based on the concept of
mutual information (Borga, 1998). Our proposed model applies a constraint loss
to control the consistency of the two shared subspaces. Finally, the end
representation is fused by the features mapped from two subspaces.

The main contributions of can be summarized as follows: We propose a simple
and flexible audiovisual cross-learning model that extracts a comprehensive
and unmixed representation across modalities by learning from the explicit
shared space and the implicit shared space. After merging the output
representations from two subspaces, the computation of cross-modality
similarity is promoted. Experiments on the VEGAS and AVE datasets show that
learned representations with simple fusion can outperform state-of-the-art
models.

The rest of this paper is organized as follows. Section 2 describes related
work in common spaces of cross-modal learning. Section 3 explains our approach
and evaluates it experimental explanation in Section 4. Section 5 presents the
results and conducts an analysis. Section 6 concludes this paper.

##  2\. Related Work

This section predominantly discusses the related works of common subspace
learning in two aspects and analysis the difference of our proposed model
compared with these related works.

Since the heterogeneous gap between modalities limits the similarly
computation of parings, existing cross-modal learning methods (Wang et al.,
2016b; Müller et al., 2018; Kaur et al., 2021) aim to learn a common space to
generate modality gap-free representations so that eventually the transformed
data can be compared using distance metrics, such as cosine similarity
distance and Euclidean distance. Many models (Ayyavaraiah and Venkateswarlu,
2019) have been proposed to learn such a shared space, and it is difficult to
summarize these approaches with well-defined categories. In previous work on
the survey of cross-modal retrieval (Peng et al., 2017a; Priyanka et al.,
2013; Ayyavaraiah and Venkateswarlu, 2018; Peng et al., 2017b), the definition
of categories depends on whether the learning space is a real-valued space or
a binary/Hamming space. In this paper, we divided the related works in two
lines by the cross-modal similarity calculation of the model is with explicit
common representation or not.

###  2.1. Explicit Common Subspace Representations Learning

Explicit common subspace representations are obtained by learning pairwise
correlations to calculate the cross-modal similarity.

####  2.1.1. Canonical Correlation Analysis (CCA) variant methods

CCA (Hardoon et al., 2004) and CCA-variant (Lai and Fyfe, 2000; Andrew et al.,
2013; Rasiwasia et al., 2014; Ranjan et al., 2015; Yu et al., 2018; Zeng et
al., 2020; Dorfer et al., 2018) methods are typical methods to learn explicit
common subspace for generating transferred features for each modality. The CCA
(Hardoon et al., 2004) is to find linear transformation for each modality to
optimize the correlation of the transferred data in the common space. The KCCA
(Lai and Fyfe, 2000) extends CCA with a kernel trick by projecting the data
into high dimension space to generate non-linear representations. The DCCA
(Andrew et al., 2013) extend CCA with deep neural network skills to learn
complex nonlinear transformations of cross-modal data so that the generated
representations are highly linearly correlated. The Cluster-CCA (Rasiwasia et
al., 2014) is a cluster-based CCA method that is to caption the possible
corresponds within cluster to learn the linearly correlation of the
transferred data. Compared with existing CCA-variant methods, The Multi-label
CCA (Ranjan et al., 2015) is other way to extend CCA method, which learns
common space to extract high-level semantic representations by the multi-label
information instead of learning explicit pairwise correlation. The C-DCCA (Yu
et al., 2018) is a significant extension of DCCA model, which learns
correlation of data from two modalities for specific single label annotation.
The TNN-C-CCA (Zeng et al., 2020) applies audio-visual learning based task
specific loss function to improve the Cluster-CCA method, which is not only
take the pairings correlation into account, but also consider the non-pairings
correlation, by exploiting triplet loss to update the weights of triplet
neural networks for the constructed triplet inputs. The NLCCA method (Hsieh,
2000) is to extend CCA methods via using three feedforward neural networks,
the first neural network consists of two branches that is to maximize the
correlation between the two output representations. The other two neural
networks use to project the canonical variables into the two original set of
data. To enhance the correlation of pairings in the of CCA mechanism, the
$CCAL-L_{rank}$ (Dorfer et al., 2018) employ the pairwise ranking losses to
reform the performance of CCA methods, by learning better embedding with the
combination of CCA layer with present objectives for embedding space learning.
In the application of the $CCAL-L_{rank}$, task specific loss functions can be
set on the output of final layer. The ICCA (Shao et al., 2017) method is
toward improving CCA for cross-modal learning tasks, by solve the essential
problems of CCA methods including the weakness of intra-modal semantic
consistency ignored, the troublesome non-linear correlation learning, and
similarity measure indirectly optimized. In addition, CCA can be applied in
improving the adversarial learning method such as GAN, this work (Harada et
al., 2019) argues that the input and generated data is not clear and the
generated data is out of control in GAN, by using CCA to control the generated
samples with optimizing the canonical correlation between input and the
generated data during the training at the same time.

####  2.1.2. Deep Neural Networks (DNN) Based CMR methods

With the success of DNN models, which allows to extend to the multimodal
learning by supplying scalable nonlinear transformations for valid cross-modal
representation learning (Ngiam et al., 2011; Cao et al., 2016; Yang et al.,
2017; Carvalho et al., 2018; Zhang et al., 2016; Karpathy et al., 2014; Li et
al., 2020; Menon et al., 2015; Jiang et al., 2015; Socher et al., 2014; Frome
et al., 2013; Geigle et al., 2021). Deep learning has already succeed in the
representation learning for single modality, such as text, image, and audio,
bridging the heterogeneous gap across modalities is still a big challenge for
deep learning methods. Multimodal Deep Learning (Ngiam et al., 2011)
introduced a representation learning for multimodal tasks, in particular, this
cross-modal representation learning approach demonstrate if multiple
modalities data used to learn can get more powerful features. The DVSH (Cao et
al., 2016) is a hybrid visual-semantic fusion network that use to capture the
intrinsic image-text correspondences for learning joint embedding space of
these two modalities, in particular, the inputs of fusion network are two
modality-specific hashing networks to generate perfect hashing codes. This
work (Carvalho et al., 2018) propose a unified framework that takes both the
pairwise similarity for leveraging retrieval and class-based features in a
common space, they applied double triplet loss to take the the high-level
semantic and the fine-grained features into account. DeViSE (Frome et al.,
2013) is to solve the problem of large numbers of object categories that is
hard to recognise without enough training data, by introducing a combination
model encompassing visual and semantic embedding learning with image data and
its unstructured text information. The cooperative and joint methods for
promoting the CMR (Geigle et al., 2021) proposed a novel framework for the
limitation of current CMR methods less scalable and huge retrieval cost, by
combining a twin networks to encode all data for efficient initial retrieval
and a cross-encoder structure for retrieved same unit of data. The $C^{2}MLR$
(Jiang et al., 2015) model is to discover better representations and optimize
pairwise rank function by enhancing both local alignment and global alignment,
in detail, local alignment is about the alignment within the visual object and
textual words, and global alignment is about the visual high level and
semantic high level alignment. Similar to this work, the Deep Fragment
(Karpathy et al., 2014) present a model, unlike other methods that projected
cross-modal representations into a common space, consider the finer level
information, visual objects, and sentence fragment. Except the rank function
during mapping the data into common space, the work also added a new fragment
alignment function to learn the alignment between these fragments. The SDPP
(Li et al., 2020) extract the correlation in a common space by additionally
eliminating the modality-specific information which is not related to the
cross-modal retrieval task, with Hilbert Schmidt Independence Criterion (HSIC)
method. A Pairwise Classification Approach (Menon et al., 2015) for CMR is to
retrieve the best matching in the retrieval candidate list when given a query
from other modality based on the pairwise classification, which is not only
can learn the correlation on positive links between two entities, but also can
caption the unlabeled links in the cross-modal retrieval datasets. The DT-RNNs
(Socher et al., 2014) designs dependency trees structure to project the
sentences into a space to retrieve related images with these sentences, DT-
RNNs learn the action and agents information in the sentence that can learn
better representation from the logic of word sequence and the language
expression. The PRDH (Yang et al., 2017) work argued that existing cross-modal
hashing method directly using manual features and overlook the internal
correlation that hurt the performance of the CMR task. The PRDH is an end-to-
end deep learning method that can capture the internal correlation between
cross-modal data. The PL-ranking (Zhang et al., 2016) is a low-rank
optimization model that consider the top samples and improve the precision of
the top samples in the rank list for a given query when learning
representation in the low-dimension shared spaces. PL-ranking used pairwise
ranking loss for the top of rankings and considered the classification
information by a applying a list-wise constraint, in the end, the correlation
between data features and label embedding is enhanced by a low-rank
regularization.

####  2.1.3. Generative Adversarial Networks (GAN) Based CMR Methods

Currently, generative adversarial networks (GAN) (Roy et al., 2020; Chaudhuri
et al., 2020; Xu et al., 5555; Zhu et al., 2019; Shang et al., 2019) prove a
large potential room can improve the performance of cross-modal learning. The
multi-stream encoder-decoder model (Chaudhuri et al., 2020) not only ensures
enhancing the correlation between the RGB and sketch image, but also guarantee
the semantic feature is discriminative in encoded feature space. In other
word, the model can preserve the semantic space while eliminating the modality
dependency during the training process. The ZSCRGAN (Roy et al., 2020) is to
address the problem of previously unseen classes in query on training set by
presenting a GAN-based zero-shot learning model for text to image retrieval.
When given a text as the query, the model can get relevant images with zero-
shot learning by using an Expectation-Maximization. The DLA-CMR model (Shang
et al., 2019) applied dictionary learning algorithm to reconstruct
discriminative representations, and adversarial learning exploit statistical
characteristics of each modality. The model maintains the specificity of
training/testing set, and ensures the weight of main features increases while
other features decreases. The JFSE model (Xu et al., 5555) investigated the
problem of adversarial learning in existing GAN-based cross-modal methods into
two kinds and introduced Joint Feature Synthesis and Embedding method that
combined the multimodal feature and representation learning in common space to
remedy the investigation problem by developing a conditional wassertein GAN
modules for the cross-modal input data. The R2GAN model (Zhu et al., 2019) is
short for a new version of GAN, named Recipe Retrieval Generative Adversarial
Network, which is to study the feasibility of the retrieval problem that
generates image from text. The R2GAN model consists of two discriminative
models and one generative model where achieved with a two kinds of rank loss
in both spaces.

####  2.1.4. Semantic Supervised Learning Based CMR Methods

With the growth of data scale, the accuracy of label annotation gets hard, to
overcome the situation, it requires us to extract more latent useful features.
One common solution (Xu et al., 2020; Xie et al., 2020; Xu et al., 2020) for
this case is to utilize the label feature as an additional feature to aid
cross-modal learning.

The TANSS model (Xu et al., 2020) is motivated by the zero-shot cross-modal
retrieval that there are some categories in the test set that have not
appeared in the training set. The TANSS model is via designing three
subnetworks to address the zero-shot problem in CMR tasks. In detail, the
first two subnetworks are to learn the internal information inside the data
for each set while keeping the instances across modalities sharing the same
semantic information. The third subnetwork is a self-supervised learning
method that is used to transfer the existing labels to unseen labels in the
test. In the SCCMR (Xu et al., 2020), they explained the shortcoming of the
existing supervised cross-modal retrieval including overlooking the abundant
semantic information without annotations during bridge the heterogeneous gap.
The SCCMR model is to globally optimize the combination of label prediction
and the projection matrices, while exploiting the manifold graph structure to
mine the latent semantic information across modalities.

In the cross-modal hashing methods, semantic inconsistency problem after
bridging the heterogeneous gap is also a big challenge for CMR task. The
(CPAH) model (Xie et al., 2020) tried to completely preserve the semantic
consistency during the correlation learning between two sets by introducing a
consistency refined mechanism to depart the different modalities data into two
different latent spaces, then using an adversarial learning to ensure the
representation from shared space full of semantic consistency and generate
compact and discriminative representations in the end.

###  2.2. Implicit Common Subspace Representations Learning

Accept existing heterogeneous gap problem in cross-modal learning, different
modalities have imbalanced and supplementary relation, with different
information when drawing the same semantics. Learning explicit common subspace
representations will easily overlook the modality-specific information. In
this case, many works (Peng et al., 2018a; Wu et al., 2020; Yanagi et al.,
2020; Wang et al., 2015; Liu et al., 2018; Xiong et al., 2020; Peng et al.,
2018b; Nie et al., 2021; Cao et al., 2019; Zhen et al., 2020; Zhang et al.,
2018a, 2021) construct the independent individual semantic space for each
modality.

In particular, the work (Peng et al., 2018a) proposed an end-to-end framework
that generates the modality-specific cross-modal similarity without learning
explicit common subspace and in each semantic subspace for modality, the model
can preserve the modality-specific characteristics. $MS^{2}GAN$ (Wu et al.,
2020) present a model that can learn modality-specific representation and
modality-shared representation for each modality by training with a
adversarial learning between generative and discriminative model. Generative
model is to generate the semantic labels of representations, caption the
inter-correlation and intra-correlation, and also differentiate the two
different kinds of representations. Discriminative model is used to
distinguish the modality of representations. The MsMFH (Xiong et al., 2020)
learns the modality-specific semantic representation for each modality and
then applying correlation information to align the representations. Cross-
modal hashing is a prevalent way to achieve cross-modal retrieval application,
however, gaining the storage- and time-saving advantage may result in
performance degradation because the finite-length hash code for similarity
matrices cannot convey the correlation across modalities. The MLCAH model (Ma
et al., 2020) converts the multi-level correlation information to hash codes
by learning global and local semantic information with multi-level information
and label information. A single common space for learning optimal
representation of visual and text features is not enough, this kind of methods
will lead to loss key information of data, this work (Yanagi et al., 2020)
introduced a robust method to learn effective representations space for
embeddings of visual and language by adopting two different types of
generative models. Some existing methods learn implicit common space for
cross-modal gap bridging that highly depend on the low-level features but
overlook the internal characteristic in original data across modalities. This
work (Wang et al., 2015) presents a new model to learn modality-specific
representations by using two CNN structures to map the raw data. Hashing-based
cross-modal retrieval methods also ignore the modality-specific features
during bridging the heterogeneous gap. The MsPH (Liu et al., 2018) model is
able to maintain the local structure and internal correlation. To address the
problem of manual labeling uncertainty, this model adopted a label enhancement
mechanism. The heterogeneous gap comes from the inconsistent and complementary
correlation across modalities, different amounts of the same level semantic
information. Simply projecting them into a common space will lose their
exclusive modality-specific information. This work (Peng et al., 2018b)
introduced a novel cross-modal similarity measurement by considering the
modality-specific characteristics, which training through a end-to-end
architecture to generate modalities-specific features, RNNs model and
attention model are used to exploit the modality-specific for semantic space.
Hashing code in cross-modal retrieval with modality-specific features will
ignore the semantic relevance between two sets of modality-specific features.
The DMFH model (Nie et al., 2021) is a hashing based method that utilizes the
multi-scale fusion models in each subnetwork to caption the semantic
relevance. The DNN-based method (Cao et al., 2019) is a hybrid representation
learning, which is used to learn shared space to generate embedding for each
modality. This model employs stacked restricted Boltzmann machines to extract
modality-specific features, a complicated network with joint autoencoders are
applied, and the final representation for each modality is obtained by a
stacked cross-modal autoencoder. To avoid the need for a large amount of
annotation data for achieving expected performance, transferring the valuable
information from effective annotation data to extend the dataset is
considered. The DMTL (Zhen et al., 2020) model transfers the previous labels
to improve the retrieval system of new unseen labels by applying a pseudolabel
algorithm. The CSGH model (Zhang et al., 2018a) is to learn the diversity of
cross-modal features in the common spaces by capturing the individual
modality-specific transformation. In detail, this model learns separate common
subspaces for each modality by modality-specific transformation matrices then
mapping these subspaces into a Hamming space with a common transformation
matrix. In addition, the model captures the manifold structure with Laplacian
regularization and the cross-modal relation with a correlation constraint. The
unified hash codes from a shared space cannot fully obtain the characteristic
of modality because of the internal heterogeneous difference. The LFMH model
(Zhang et al., 2021) is a novel cross-modal hashing approach that learns
modality-specific subspace by matrix factorization. Finally, the model can
transform the cross-modal raw data into modality-specific features

Different from the above methods, our work is inspired by the domain adaption
network (Bousmalis et al., 2016), we learn two different common subspaces. To
alleviate the loss of key information in only one common space learning, we
learn additional implicit common space, where is for domain adaptation and
weakly maintains the semantic representation.

##  3\. APPROACH

Figure 2. The model consists of three parts: 1)Feature Extraction; 2)Modality
Representations; 3)Feature Fusion. The model takes the global audio and visual
representations extracted by Vggish and Inception model respectively, and
projects each modality into two different latent subspaces: Explicit and
Implicit common spaces. Then, these hidden representations are utilized to
fuse features by linear CCA-layer to calculate cross-modal similarity.

###  3.1. Problem Formulation

Assume that there is a collection of $n$ videos, extracted audio and visual
features from each video, denoted as $\Phi=\\{\phi_{i}\\}_{i=1}^{n}$,
$\phi_{i}=(a_{i},v_{i})$, where $a_{i}\in R^{d_{a}}$ is an audio feature and
$v_{i}\in R^{d^{v}}$ is a visual feature, here $d_{a}$=128 and $d_{v}$=1024
are the dimension of input features. Each instance $\phi_{i}$ labeled by a
semantic vector $Y_{i}=[y_{i1},y_{i2},...,y_{ic}]$, where $c$ is the number of
semantic categories. If the video $i$ belong to $y_{ij}$ $(1\leqslant
j\leqslant c$), $y_{ij}$=1, otherwise $y_{ij}$=0. Finally, the above
modalities and categories can be represented by feature matrix,
$A=\\{a_{i},...,a_{n}\\}\in R^{d_{a}\times 128},V=\\{v_{i},...,v_{n}\\}\in
R^{d_{v}\times 1024}$.

Since the audio and visual representations $A$ and $V$ have inconsistent
distributions and different dimensions, they are unable to compare with each
other for the AVCMR task. It requires us to learn common space, explicit
common space $S^{ex}$, and implicit common space $S^{im}$. In the first space,
audio and visual representations are projected as $S_{ex}^{a}$ and
$S_{ex}^{v}$ by $f(a_{i})$, $g(v_{i})$, respectively. In the second space,
audio and visual representations are projected as $S_{im}^{a}$ and
$S_{im}^{v}$ by $\psi(a_{i})$, $\tau(v_{i})$, respectively. Finally, the
output of audio and visual representations is the fusion of
$S_{ex}^{a}$&$S_{im}^{a}$ and $S_{ex}^{v}$&$S_{im}^{v}$, respectively.

###  3.2. Proposed method

The function of can be divided into three main parts: Feature Extraction
(Section 4.3), Modality Representations and Modality Fusion (Section 3.3). The
general framework of the is shown in Figure 2, and the configuration of is
presented in Table 1.

Table 1. Configuration of our proposed model. Log mel-spectrogram audio inputs | 96x64
---|---
Input of audio/visual branch | 128-D/1024-D
Output of Ex-subspace | 10-D
Output of Im-subspace | 10-D
Hidden units for audio | [1024, 1024, 10]
Hidden units for visual | [1024, 2048, 10]
Output of CCA-layer | 10-D

###  3.3. Modality Representations and Fusion

Global-level Representations. For audio and visual modality are extracted by
pre-trained models (Section 4.3). The output of the features is frame-level
features. Convert frame-level feature to global-level feature is as follows.

(1) $\displaystyle x_{g}=(m(x_{1}),m(x_{2}),...,m(x_{l})),$ $\displaystyle
m(x_{i})=\frac{1}{|d_{x}|}\sum_{j=1}^{d_{x}}(x_{ij})$

where $d_{x}$ is the dimension of $x_{i}$, $l$ is the number of frames.

Explicit and Implicit Common Space. We now project global-level
representations across modalities into two distinct common subspaces. Based on
whether learning one-to-one pair correlation, we classify the common space
into explicit- and implicit-subspaces. Firstly, the explicit common space is
to learn the commonalities and correlation for each audio-visual pair extract
from the same video. Secondly, the implicit common space captures the unique
characteristics of that modality by minimizing the difference between label
and feature.

####  3.3.1. $L_{corr}$-Correlation loss

. We apply correlation loss to reduces the cross-modal discrepancy, which
makes the common audio-visual representation to be aligned close in the
explicit common space. Technically, our correlation loss is CCA-variant loss
as follows:

(2) $\displaystyle
L_{corr}=\frac{1}{n}\sum_{i=1}^{n}\frac{W^{T}_{a}\Sigma_{av}W_{v}}{\sqrt{W^{T}_{a}\Sigma_{aa}W_{a}\cdot
W^{T}_{v}\Sigma_{vv}W_{v}}}$

where $W_{a}$, and $W_{v}$ are the transformation weight $\Sigma_{aa}$ and
$\Sigma_{vv}$ are the covariance matrices of $A$ and $V$, respectively and
$\Sigma_{av}$ is the cross covariance.

####  3.3.2. $L_{dis}$-discriminative loss

. Preserve the discrimination of generated representation in the shared
subspace is very important for AVCMR task. However, if we only learn the one-
to-one pair correlation and overlook the category information during the
training, the learned representation will far from our expectations. Here, we
set a trick that is to make the dimension of output vector is the same as the
label vector, such that the output vector is not only the generated
representations but also it’s the predicted label of the sample. We propose
the following objective loss function to keep the discrimination of modality
representations in the label space.

(3) $\displaystyle
L_{dis}=\frac{1}{n}||\psi(a_{i})-y_{a_{i}}||_{F}+\frac{1}{n}||\tau(v_{i})-y_{v_{i}}||_{F}$

where $||\cdot||_{f}$ represents the Frobenius-Norm.

####  3.3.3. $L_{cons}$-constraint loss

. Loss $L_{corr}$ and $L_{dis}$ capture different aspects of the input.
However, the end-to-end model training may cause redundancy of NN structures.
In this case, we involve a constrain loss to enforce the two NN structures
with the same input learned in the different direction followed by (Bousmalis
et al., 2016; Liu et al., 2017; Ruder and Plank, 2018). Let $Q_{ex}^{m}$ and
$Q_{im}^{m}$ be the matrix, we apply the orthogonality constraint for each
modality pair as:

(4) $\displaystyle||(Q_{ex}^{m})^{T}Q_{im}^{m}||_{F}^{2}$

where $||\cdot||_{f}^{2}$ is the squared Frobenius-Norm. In addition, we add
the constraint in the implicit common space between audio and visual
representation learning. The final constraint loss is calculated as:

(5) $\displaystyle
L_{cons}=||(Q_{ex}^{m})^{T}Q_{im}^{m}||_{F}^{2}+||(Q_{im}^{m1})^{T}Q_{im}^{m2}||_{F}^{2}$

Incorporating above 3 loss equation (2, 3 and 5 and assign the weight for each
loss based on the output loss value. The overall loss function of our model is
as:

(6) $\displaystyle L=L_{corr}+\alpha L_{dis}+\beta L_{cons}$

Here, $\alpha$, $\beta$ are the interaction weights that decide the
contribution of each component to the loss $L$, we will discuss these
hyperparameter in Section 5.2.4. Finally, the final loss function $L$ can be
optimized using a SGD algorithm.

Feature fusion. Once we get the output representation from two subspaces, it
requires us to fuse them into a joint embedding. We apply a simple fusion
mechanism by adding a linear-CCA layer and the input of the layers is the
concatenation of representations from two subspaces for each modality. Our
system will output 10-D representation for each modality for cross-modal
similar calculation.

##  4\. EXPERIMENTS

###  4.1. Datasets

(a)

(b)

Figure 3. The number of videos with different categories in train set and
testing set. (a) VEGAS dataset, (b) AVE dataset.

We evaluate our model on two different video datasets: VEGAS (Zhou et al.,
2018) dataset and AVE-dataset (Tian et al., 2018). As for VEGAS dataset, we
follow the work (Zeng et al., 2020) to extract each video with audio feature
and visual feature by using advanced pre-trained models and then apply the
global feature by frame-axis average as the final representations, more
details about the two audio-visual dataset are as follows.

####  4.1.1. VEGAS dataset

The VEGAS dataset is a subset of Google Audioset and employ Amazon Mechanical
Turks to clean the data. This dataset encompasses 10 human voices vehicle
sounds, and nature sounds categories (chainsaw, rail transport, helicopter,
baby cry or infant cry, drum, snoring, printer, fireworks, water flowing, and
dog.) that are labeled by audio and visual double tracks check. Moreover, The
length of a video ranges from 2 to 10 seconds and the average length overall
is around 7 seconds. This dataset contains 28,103 videos and each video is
labeled by a single label. We divided all the videos into train and test set
by 80% and 20% and keep each category is balance appeared in the train and
test sets, seen in Figure 3(a). Finally, we apply 5,621 videos for testing and
22,482 videos for training in our experiments.

####  4.1.2. AVE dataset

Audio-Visual Event (AVE) dataset contains 4,143 videos covering 28 event
categories, each video is temporally annotated with audio-visual event
boundaries. The lengths of video are range from 2 seconds to 10 seconds. In
our experiment, we remove the music-related categories and keep 15 categories
(Clock, motorcycle, train horn, bark, cat, bus, rodents/rats, toilet flush,
acoustic guitar, frying (food), chainsaw, horse, helicopter, infant cry,
truck). We select 189 videos as testing set, 1766 videos as train set, and
keep category balance in each set, seen in Figure 3(b).

###  4.2. Evaluation Metric

To evaluate our proposed model, we apply Mean Average Precision (MAP) as the
main computational metric for audio-visual cross-modal retrieval on both VEGAS
dataset and AVE dataset. In addition, to further leverage the performance of
our model, we follow the work (Zeng et al., 2020) to take Time Cost as another
metric to evaluate real-time started from training model to test the model on
the VEGAS dataset. Besides, we explore Precision-Scope Curves (PSC) and
Average Precision (AP) per category to leverage our architecture in order to
further investigate our proposed model. During the testing, the system
produces a rank list in one modality for each query in another modality, we
regard the candidates in the rank list with the same category as query is
correct.

###  4.3. Feature Extraction

To make a fair comparison, we apply the same audio-visual features as input of
all models. Because all the compared methods are required learning common
spaces, so the input audio-visual representations are global features.

####  4.3.1. Audio Features

We utilize the 111https://librosa.org/Librosa Library to extract Mel
spectrogram feature as the input of Vggish model (Hershey et al., 2017), with
default parameters: hop size=512, nftt=2,048 and etc. The pre-trained VGG-like
model is trained on AudioSet benchmark for audio classification task. Finally,
audio representation is a 128-D global feature.

####  4.3.2. Visual Features

We extract frame-level visual features by using the public Inception V3
222https://github.com/fchollet/deep-learning-models model (Abu-El-Haija et
al., 2016). which is trained on the ImageNet dataset. Here, we decode each
video at one frame-per-second, so the number of frames is the equal to the
number of seconds, then we feed the features into Inception V3 network and use
ReLu activation in the last hidden layer. The frame-level features of
Inception V3 model is 2048-D per frame and discard the motion information.
Afterwards, we exploit Principal component analysis (PCA) to reduce the
dimension to 1024, so the final frame-level visual features is 1024*$len$,
where $len$ is the number of frames. In our experiment, we apply global 1024-D
feature obtained by frame-axis average as visual representations.

###  4.4. Baselines

We comprehensively compare our model with various baselines, including Random
case, CCA-variant methods and other state-of-the-art methods, the detail is as
follows. As for the Random case, instead of getting the rank list for each
query by the cross-modal similarity, it is by random.

####  4.4.1. CCA-variant Models

Multitudinous models have been proposed for cross-modal learning, CCA-variant
as traditional methods for learning the correlation in two variables across
modalities should be taken into account, so that to conduct a comprehensive
comparative study against our model. Prevalent CCA-variant baselines include:

  * •

CCA (Hardoon et al., 2004) (Canonical Correlation Analysis) is to find basis
vectors for two variable sets to ensure the correlation between the
projections of the two variables on basis vectors are mutually maximised. An
alternative description is that CCA is to find linear transformations for two
multivariate sets to optimize the correlation between them, which is the
unsupervised learning and strongly associate to mutual information.

  * •

KCCA (Lai and Fyfe, 2000) (Kernel Canonical Correlation Analysis) differ from
CCA in finding the non-linear correlation by the kernel method, such as the
known method ”kernel trick”, which is to improve the computational power of
the linear CCA and extract more useful information of the input data by
projecting the data into a high-dimensional space.

  * •

DCCA (Andrew et al., 2013) (Deep Canonical Correlation Analysis) is to obtain
the nonlinear transformations of two data sets by deep learning and CCA-linear
layer, which can be regard as a nonlinear extension of the linear CCA,
different from CCA, DCCA is a parametric model and did not require the inner
product. In addition, DCCA can be loosely viewed as learning a kernel for KCCA
and offer a flexible nonlinear alternative to KCCA.

  * •

C-CCA (Rasiwasia et al., 2014) (Cluster Canonical Correlation Analysis) is to
cluster data across modalities into several classes and tries to enhance the
intra-cluster correlation. Unlike the pairwise learning between data points
from difference modalities, C-CCA is to divide the cross-modal data points
into several clusters and learn new representations to maximize the
correlation between two data modalities while set the data points from
different classes apart in the common space.

  * •

C-DCCA (Yu et al., 2018) (Cluster Deep Canonical Correlation Analysis)is a
combination of Cluster-CCA and DCCA, which segregate the non-linear
representation from deep learning method into several related clusters to
optimize the correlation, where cross-modal data are non-linear projected into
a common space while the data points from the same cluster have high
correlation.

  * •

TNN-C-CCA (Zeng et al., 2020) (Triplet Neural Network & Cluster Canonical
Correlation Analysis) can be viewed as an extension of C-CCA and C-DCCA, which
considers the category information from both similar and dissimilar
correlation. TNN-C-CCA presents an audio-visual specific loss function to
improve the Cluster-CCA and is the present state-of-the-art method on VEGAS
dataset, where the output of cluster CCA are mapped into a latent shared
subspace through a triplet neural network while the data points from different
clusters are highly correlated and the data points from different clusters are
little correlated.

####  4.4.2. Advanced methods

Numerous models of cross-modal learning are presented in recent years, we list
some advanced models that publish in multimedia domain of top conferences as
baselines.

  * •

UGACH (Zhang et al., 2018b) (Unsupervised Generative Adversarial Cross-modal
Hashing approach) utilizes Generative Adversarial Network to exploit the
potential manifold structure of data across modalities by unsupervised
learning. Given a data point from on modality, the generative mechanism will
fit the manifold structure of the distribution and select the correlated data
points from another modality to feint the discriminative model.

  * •

AGAH (Gu et al., 2019) (Adversary Guided Asymmetric Hashing) applies a multi-
label attention model with adversarial learning to enhance discrimination of
cross-modal representations, which can generate hashing codes to fully keep
the multi-label information of all the data points.

  * •

UCAL (He et al., 2017) and ACMR (Wang et al., 2017) (Unsupervised cross-modal
retrieval and Adversarial Cross-Modal Retrieval) apply unsupervised and
supervised adversarial learning for cross-modal representation learning. A
feature projector of ACMR to learn modality-invariant representation while
modality classifier try to distinguish the modality of generated
representation.

  * •

DSCMR (Zhen et al., 2019) (Deep Supervised Cross-modal Retrieval) is advanced
in learning discriminative features in label space and feature common space by
supervised learning that minimizes the discrimination loss in representation
learning space and label learning space while bridge the modality gap by using
a weight sharing strategy. In our experiment, DSCMR model achieves the best
performance on AVE dataset.

##  5\. RESULTS AND ANALYSIS

###  5.1. Comparison with Existing Methods

The comparative results for AVCMR task on VEGAS dataset and AVE dataset are
presented in Table 2, it shows that our model achieves the best performances
and significantly outperforms all CCA-variant methods including the state-of-
the-art method TNN-C-CCA, and others advanced methods on both audio-visual
datasets. In particular, our model gains 4.6% of MAP in $audio\rightarrow
visual$ retrieval, 4.1% of MAP in $visual\rightarrow audio$ retrieval, 4.3% of
MAP in average MAP, over TNN-C-CCA on VEGAS dataset. Also, our model
outperforms the GOAT model DSCMR on AVE dataset by the same 2.3% of MAP in
$audio\rightarrow visual$ and $visual\rightarrow audio$ retrieval.

Except for the evaluation with MAP, we apply precision-scope curves as an
additional assessment. Figure 4 shows the curves of and of CCA, C-CCA, DCCA,
C-DCCA, UCAL, ACMR, DSCMR, TNN-C-CCA. The precision-scope curves and the MAP
are consistent when audio or visual as query and our model have significant
gains.

The supervised model is not completely learning representation with label in
each common space. In the explicit common space, our model projects modalities
into this subspace to optimize the correlation of pair to pair. In Table 4,
the variant model only with explicit common space is unsupervised learning.
Compared with the other unsupervised learning methods CCA, KCCA, UGACH and
UCAL, variant model also can achieve the best performance.

(a)

(b)

Figure 4. Precision-scope curves on the VEGAS dataset for (a) audio2visual and visual2audio retrieval experiments, with K ranges from 10 to 1000. Table 2. The MAP of Audio-Visual Cross-Modal retrieval for Our Proposed Model and Some Previous State-of-the-art Models on VEGAS Dataset and AVE Dataset Models | VEGAS Dataset | AVE Dataset
---|---|---
A$\rightarrow$V | V$\rightarrow$A | Avg. | A$\rightarrow$V | V$\rightarrow$A | Avg.
Random | 0.110 | 0.109 | 0.109 | 0.127 | 0.124 | 0.126
CCA (Hardoon et al., 2004) | 0.332 | 0.327 | 0.330 | 0.190 | 0.189 | 0.190
KCCA (Lai and Fyfe, 2000) | 0.288 | 0.273 | 0.281 | 0.133 | 0.135 | 0.134
DCCA (Andrew et al., 2013) | 0.478 | 0.457 | 0.468 | 0.221 | 0.223 | 0.222
C-CCA (Rasiwasia et al., 2014) | 0.711 | 0.704 | 0.708 | 0.228 | 0.226 | 0.227
C-DCCA (Zeng et al., 2018) | 0.722 | 0.716 | 0.719 | 0.230 | 0.227 | 0.229
UGACH (Zhang et al., 2018b) | 0.182 | 0.179 | 0.181 | 0.165 | 0.159 | 0.162
AGAH (Gu et al., 2019) | 0.578 | 0.568 | 0.573 | 0.200 | 0.196 | 0.198
UCAL (He et al., 2017) | 0.446 | 0.436 | 0.441 | 0.153 | 0.150 | 0.152
ACMR (Wang et al., 2017) | 0.465 | 0.442 | 0.454 | 0.162 | 0.159 | 0.161
DSCMR (Zhen et al., 2019) | 0.732 | 0.721 | 0.727 | 0.314 | 0.256 | 0.285
TNN-C-CCA (Zeng et al., 2020) | 0.751 | 0.738 | 0.745 | 0.253 | 0.258 | 0.256
Our model | 0.797 | 0.779 | 0.788 | 0.337 | 0.279 | 0.308

###  5.2. Ablation Study

####  5.2.1. Role of Time Cost

Table 1 shows the fused representation of our model is set as 10 and the same
as other models. To fair comparison of time cost, we only consider the time
consuming during the model training. Table 3 presents the Time-consuming
(minutes) of model training on VEGAS dataset. Our model only takes around 8
minutes to achieve the model training, which only falls behind the C-CCA and
DCCA. Our model saves time significant over state-of-the-art TNN-C-CCA that
takes 192.0 minutes to train the final model.

Table 3. Time Cost of our model and Some Baselines model training on VEGAS dataset. Models | Time(Minutes) | Models | Time(Minutes)
---|---|---|---
Random | 0.04 | CCA | 1.08
DCCA | 5.15 | KCCA | 2.03
ACMR | 176.7 | TNN-C-CCA | 192.00
C-CCA | 1.24 | C-DCCA | 18.56
DSCMR | 22.05 | Explicit-space | 6.15
Implicit-space | 6.77 | Our model | 8.36

####  5.2.2. Role of Common Space

The two subspaces of our model are achieved by the three-terms combined
objective function. Each term is consistent with the function of each
subspace. To further investigate our model, we design three variations of our
model. Our model-invariant (Explicit-space and Implicit-space) share the same
neural network (NN) structures with our model and apply the combinations
(correlation loss with constraint loss, discriminative loss with constraint
loss) respectively. invariant model (Ex-Im-share) means the same modality
shares the same NN structures for two subspaces and utilizes the same loss
function with our model. From the results in Table 4, our model combines the
three terms will stimulate the advantage of representation learning in two
common subspaces and improve the performance of AVCMR task.

Our model jointly optimize the correlation loss and discriminative loss in the
objective function, To further exploit the effect of multi-subspaces in our
model, we sample the correlation loss and the discriminative loss values from
epoch ranges from 1 to 1000, seen in Figure 5. The loss and other two loss
combination decrease monotonously and smoothly converge. The loss curves are
consistent with the expectation of two subspaces combination.

(a) Figure 5. The objective function value of our model with the change of training epochs [1, 1000], on the VEGAS dataset. Table 4. The MAP of Audio-Visual Cross-Modal retrieval for Our Proposed Model and Some Baselines on VEGAS Dataset and AVE Dataset Models | VEGAS Dataset | AVE Dataset
---|---|---
A$\rightarrow$V | V$\rightarrow$A | Avg. | A$\rightarrow$V | V$\rightarrow$A | Avg.
Random | 0.110 | 0.109 | 0.109 | 0.127 | 0.124 | 0.126
Explicit-space | 0.687 | 0.692 | 0.690 | 0.232 | 0.244 | 0.238
Implicit-space | 0.614 | 0.622 | 0.618 | 0.222 | 0.263 | 0.242
Ex-Im-share | 0.781 | 0.773 | 0.777 | 0.278 | 0.243 | 0.261
Our model | 0.797 | 0.779 | 0.788 | 0.337 | 0.279 | 0.308

####  5.2.3. Role of Category

In Figure 6, we show the average precision (AP) per category of our model and
its variant models on VEGAS dataset. In particular, the Best AP perform by our
model, the Mean AP is the mean of our model and three variant mentions in
subsection 5.2.2. We divide the dataset into train and testing set and keep
the category balance, the figure can roughly show the higher AP indicates the
easier to retrieve. We can see the Dog barking and the firework get the
highest AP with our model and the overall variant methods when audio as query.
Take look at the Dog barking, audio2visual can get 84.6% with our model but
visual2audio only 79.4%, almost 5% difference.

(a)

(b)

Figure 6. Summary of the Audio-Visual Retrieval Results by Class. For each
class two values are shown: the maximum AP obtained by our model(max) and the
Median AP over all Our Baselines (mean).

####  5.2.4. Role of Model Parameters.

Alpha ($\alpha$) and Beta ($\beta$) is a couple of hyper-parameters in loss
function of our model. As $\alpha$ and $\beta$ distribute the contribution of
the difference between two subspace and the modality-specific characteristic
remains, we try to analyze the effect of them on cross-modal representation
generation during training on two datasets (VEGAS and AVE). Here, we sample
$\alpha$, $\beta$ from 0, 0.001, 0.01, 0.1, 1, note that $\alpha$=0, our model
is the (explicit-space) model. The figure shows that when $\alpha$, $\beta$
are set as 0.1, 0.01 or 0.001 can surpass the state-of-the-art TNN-C-CCA
model, within the couple, when $\alpha$=0.01, $\beta$=0.001, our model can get
the best performance on both datasets, seen in Figure 7.

(a) Audio-Visual (VEGAS)

(b) Visual-Audio (VEGAS)

(c) Average (VEGAS)

(d) Audio-Visual (AVE)

(e) Visual-Audio (AVE)

(f) Average (AVE)

Figure 7. The MAP Figures of with the Changing of Parameters: Alpha and Beta.
The Planes Represent the MAP of the State-of-the-art model TNN-C-CCA, the
Green Circles Denote the Situation (Fixed Alpha and Beta) that the MAP of our
Model is lower than TNN-C-CCA Model, the Blue Circles Show the Higher and the
Red Circles Imply the Best Case.

(a) Original Audio Samples

(b) Original Visual Samples

(c) Original Audio-Visual Samples

(d) Audio Representations

(e) Visual Representations

(f) Audio-Visual Representations

Figure 8. t-SNE visualization for the test data in the VEGAS Dataset. It
includes original audio/visual sample and audio/visual representations.

####  5.2.5. Visualisation of the Learned Representation

To investigate the effectiveness of representations fusion from two subspaces
of our model, we visualize the change of representations distribution from
testing set during our model training on the VEGAS dataset by t-SNE tool. A
comparison between the first row and second row in Figure 8, which indicates
our model separate data points into semantic clusters and some of them are
segregated from other clusters with clear boundary, but there are still some
clusters will intersect with each other and hard to segregate. The segregated
cluster will get easier to retrieve the related documents and the MAP will be
higher than those intersected clusters. The AVCMR in this work is category-
based retrieval, the ideal generated representation of the proposed model is
all the clusters are segregated and the distance of boundary is enough for the
system to distinguish the relevant candidates.

##  6\. CONCLUSION

In this paper, we presented a comprehensive audio-visual representation
learning model that projects the modalities into explicit common space and
implicit common space, the learned representations from these two subspaces
via fusing skill to calculation the cross-modal similarity. Compared with
previous methods that learning representations on a common space, our model
learns mutual information and semantic information in two distinct subspaces,
we find our model to be effective to improve the performance and significant
gains over GOAT models in of audio-visual cross-modal retrieval task. The
result analysis indicates the modality-specific characteristic utilized in the
implicit common space, which can be an additional remedy for losing
information during bridging the modality gap. Overall, we emphasize the
importance of multi-common spaces during representation learning for audio-
visual learning. The comprehensive experiments demonstrate our idea is
effective.

We plan to propose more powerful fusion skills and try to apply attention
mechanisms to consider the local features in implicit space in the future.

###### Acknowledgements.

This work was supported by KDDI research, Inc. Project.

## References

  * (1)
  * Abu-El-Haija et al. (2016) Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. 2016\.  YouTube-8M: A Large-Scale Video Classification Benchmark.  arXiv:1609.08675 [cs.CV]
  * Andrew et al. (2013) Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013.  Deep Canonical Correlation Analysis. In _Proceedings of the 30th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 28)_ , Sanjoy Dasgupta and David McAllester (Eds.). PMLR, Atlanta, Georgia, USA, pp.1247–1255.  https://proceedings.mlr.press/v28/andrew13.html
  * Ayyavaraiah and Venkateswarlu (2018) Monelli Ayyavaraiah and Bondu Venkateswarlu. 2018.  Joint graph regularization based semantic analysis for cross-media retrieval: a systematic review.  _International Journal of Engineering & Technology_ 7, 2.7 (2018), pp.257–261.
  * Ayyavaraiah and Venkateswarlu (2019) Monelli Ayyavaraiah and Bondu Venkateswarlu. 2019.  Cross media feature retrieval and optimization: A contemporary review of research scope, challenges and objectives. In _International Conference On Computational Vision and Bio Inspired Computing_. Springer, Springer International Publishing, Coimbatore, India, pp.1125–1136.
  * Borga (1998) Magnus Borga. 1998\.  _Learning multidimensional signal processing_.  Ph.D. Dissertation. Linköping University Electronic Press.
  * Bousmalis et al. (2016) Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016\.  Domain Separation Networks.  arXiv:1608.06019 [cs.CV]
  * Cao et al. (2019) Wenming Cao, Qiubin Lin, Zhihai He, and Zhiquan He. 2019\.  Hybrid representation learning for cross-modal retrieval.  _Neurocomputing_ 345 (2019), pp.45–57.  https://doi.org/10.1016/j.neucom.2018.10.082
  * Cao et al. (2016) Yue Cao, Mingsheng Long, Jianmin Wang, Qiang Yang, and Philip S. Yu. 2016.  Deep Visual-Semantic Hashing for Cross-Modal Retrieval. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (Eds.). ACM, San Francisco, CA, USA, pp.1445–1454.  https://doi.org/10.1145/2939672.2939812
  * Carvalho et al. (2018) Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, and Matthieu Cord. 2018\.  Cross-modal retrieval in the cooking context: Learning semantic text-image embeddings. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. Association for Computing Machinery, New York, NY, USA, pp.35–44.
  * Chaudhuri et al. (2020) Ushasi Chaudhuri, Biplab Banerjee, Avik Bhattacharya, and Mihai Datcu. 2020.  A Simplified Framework for Zero-shot Cross-Modal Sketch Data Retrieval. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. Computer Vision Foundation / IEEE, Seattle, WA, USA, pp.182–183.
  * Chen et al. (2020) Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020\.  Vggsound: A large-scale audio-visual dataset. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, IEEE, Barcelona, Spain, pp.721–725.
  * Chua et al. (2009) Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009\.  Nus-wide: a real-world web image database from national university of singapore. In _Proceedings of the ACM international conference on image and video retrieval_. Association for Computing Machinery, New York, NY, USA, pp.1–9.
  * Dorfer et al. (2018) Matthias Dorfer, Jan Schlüter, Andreu Vall, Filip Korzeniowski, and Gerhard Widmer. 2018\.  End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss.  _International Journal of Multimedia Information Retrieval_ 7, 2 (2018), pp.117–128.
  * Frome et al. (2013) Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomás Mikolov. 2013.  DeViSE: A Deep Visual-Semantic Embedding Model. In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013\. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_ , Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (Eds.). ACM, Lake Tahoe, Nevada, United States, pp.2121–2129.  https://proceedings.neurips.cc/paper/2013/hash/7cce53cf90577442771720a370c3c723-Abstract.html
  * Geigle et al. (2021) Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulić, and Iryna Gurevych. 2021.  Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval.  _arXiv preprint_ abs/2103.11920 (2021).  arXiv:2103.11920 http://arxiv.org/abs/2103.11920
  * Gu et al. (2019) Wen Gu, Xiaoyan Gu, Jingzi Gu, Bo Li, Zhi Xiong, and Weiping Wang. 2019\.  Adversary Guided Asymmetric Hashing for Cross-Modal Retrieval. In _Proceedings of the 2019 on International Conference on Multimedia Retrieval, ICMR 2019, Ottawa, ON, Canada, June 10-13, 2019._ Association for Computing Machinery, New York, NY, USA, pp.159–167.  https://doi.org/10.1145/3323873.3325045
  * Harada et al. (2019) Shota Harada, Hideaki Hayashi, and Seiichi Uchida. 2019\.  Biosignal generation and latent variable analysis with recurrent generative adversarial networks.  _IEEE Access_ 7 (2019), pp.144292–144302.
  * Hardoon et al. (2004) David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. 2004.  Canonical Correlation Analysis: An Overview with Application to Learning Methods.  _Neural Computation._ Vol.16, no.12 (2004), pp.2639–2664.  https://doi.org/10.1162/0899766042321814
  * Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020\.  MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis. In _Proceedings of the 28th ACM International Conference on Multimedia_. Association for Computing Machinery, New York, NY, USA, pp.1122–1131.
  * He et al. (2017) Li He, Xing Xu, Huimin Lu, Yang Yang, Fumin Shen, and Heng Tao Shen. 2017\.  Unsupervised cross-modal retrieval through adversarial learning. In _2017 IEEE International Conference on Multimedia and Expo, ICME 2017, Hong Kong, China, July 10-14, 2017_. IEEE Computer Society, Hong Kong, China, pp.1153–1158.  https://doi.org/10.1109/ICME.2017.8019549
  * Hershey et al. (2017) Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017\.  CNN architectures for large-scale audio classification. In _2017 ieee international conference on acoustics, speech and signal processing (icassp)_. IEEE, IEEE, New Orleans, LA, USA, pp.131–135.
  * Hsieh (2000) William W Hsieh. 2000\.  Nonlinear canonical correlation analysis by neural networks.  _Neural Networks_ 13, 10 (2000), pp.1095–1105.
  * Jiang et al. (2015) Xinyang Jiang, Fei Wu, Xi Li, Zhou Zhao, Weiming Lu, Siliang Tang, and Yueting Zhuang. 2015.  Deep Compositional Cross-modal Learning to Rank via Local-Global Alignment. In _Proceedings of the 23rd Annual ACM Conference on Multimedia Conference, MM ’15, Brisbane, Australia, October 26 - 30, 2015_ , Xiaofang Zhou, Alan F. Smeaton, Qi Tian, Dick C. A. Bulterman, Heng Tao Shen, Ketan Mayer-Patel, and Shuicheng Yan (Eds.). ACM, Brisbane, Australia, pp.69–78.  https://doi.org/10.1145/2733373.2806240
  * Karpathy et al. (2014) Andrej Karpathy, Armand Joulin, and Li Fei-Fei. 2014\.  Deep Fragment Embeddings for Bidirectional Image Sentence Mapping. In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (Eds.). ACM, Montreal, Quebec, Canada, pp.1889–1897.  https://proceedings.neurips.cc/paper/2014/hash/84d2004bf28a2095230e8e14993d398d-Abstract.html
  * Kaur et al. (2021) Parminder Kaur, Husanbir Singh Pannu, and Avleen Kaur Malhi. 2021.  Comparative analysis on cross-modal information retrieval: A review.  _Computer Science Review_ 39 (2021), pp.100336.
  * Kay et al. (2017) Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017.  The Kinetics Human Action Video Dataset.  _CoRR_ abs/1705.06950 (2017).  arXiv:1705.06950 http://arxiv.org/abs/1705.06950
  * Lai and Fyfe (2000) Pei Ling Lai and Colin Fyfe. 2000.  Kernel and Nonlinear Canonical Correlation Analysis.  _Int. J. Neural Syst._ Vol.10, no.5 (2000), pp.365–377.  https://doi.org/10.1142/S012906570000034X
  * Li et al. (2020) Jinxing Li, Mu Li, Guangming Lu, Bob Zhang, Hongpeng Yin, and David Zhang. 2020\.  Similarity and diversity induced paired projection for cross-modal retrieval.  _Inf. Sci._ 539 (2020), pp.215–228.  https://doi.org/10.1016/j.ins.2020.06.032
  * Liu et al. (2017) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017.  Adversarial Multi-task Learning for Text Classification. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, Vancouver, Canada, pp.1–10.  https://doi.org/10.18653/v1/P17-1001
  * Liu et al. (2018) Xingbo Liu, Xiushan Nie, Haoliang Sun, Chaoran Cui, and Yilong Yin. 2018.  Modality-specific structure preserving hashing for cross-modal retrieval. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, Calgary, AB, Canada, pp.1678–1682.  https://doi.org/10.1109/ICASSP.2018.8462454
  * Lu et al. (2018) Rui Lu, Zhiyao Duan, and Changshui Zhang. 2018.  Listen and look: Audio–visual matching assisted speech source separation.  _IEEE Signal Processing Letters_ 25, 9 (2018), pp.1315–1319.
  * Ma et al. (2020) Xinhong Ma, Tianzhu Zhang, and Changsheng Xu. 2020\.  Multi-level correlation adversarial hashing for cross-modal retrieval.  _IEEE Transactions on Multimedia_ 22, 12 (2020), pp.3101–3114.
  * Menon et al. (2015) Aditya Krishna Menon, Didi Surian, and Sanjay Chawla. 2015\.  Cross-Modal Retrieval: A Pairwise Classification Approach. In _Proceedings of the 2015 SIAM International Conference on Data Mining, Vancouver, BC, Canada, April 30 - May 2, 2015_ , Suresh Venkatasubramanian and Jieping Ye (Eds.). SIAM, Vancouver, BC, Canada, pp.199–207.  https://doi.org/10.1137/1.9781611974010.23
  * Müller et al. (2018) Meinard Müller, Andreas Arzt, Stefan Balke, Matthias Dorfer, and Gerhard Widmer. 2018\.  Cross-modal music retrieval and applications: An overview of key methodologies.  _IEEE Signal Processing Magazine_ 36, 1 (2018), pp.52–62.
  * Ngiam et al. (2011) Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011\.  Multimodal Deep Learning. In _Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011_. Omnipress, Madison, WI, USA, pp.689–696.
  * Nie et al. (2021) Xiushan Nie, Bowei Wang, Jiajia Li, Fanchang Hao, Muwei Jian, and Yilong Yin. 2021\.  Deep Multiscale Fusion Hashing for Cross-Modal Retrieval.  _IEEE Trans. Circuits Syst. Video Technol._ 31, 1 (2021), pp.401–410.  https://doi.org/10.1109/TCSVT.2020.2974877
  * Peng et al. (2017a) Yuxin Peng, Xin Huang, and Yunzhen Zhao. 2017a.  An overview of cross-media retrieval: Concepts, methodologies, benchmarks, and challenges.  _IEEE Transactions on circuits and systems for video technology_ 28, 9 (2017), pp.2372–2385.
  * Peng et al. (2018a) Yuxin Peng, Jinwei Qi, and Yuxin Yuan. 2018a.  Modality-specific cross-modal similarity measurement with recurrent attention network.  _IEEE Transactions on Image Processing_ 27, 11 (2018), pp.5585–5599.
  * Peng et al. (2018b) Yuxin Peng, Jinwei Qi, and Yuxin Yuan. 2018b.  Modality-Specific Cross-Modal Similarity Measurement With Recurrent Attention Network.  _IEEE Trans. Image Process._ 27, 11 (2018), pp.5585–5599.  https://doi.org/10.1109/TIP.2018.2852503
  * Peng et al. (2017b) Yu-xin Peng, Wen-wu Zhu, Yao Zhao, Chang-sheng Xu, Qing-ming Huang, Han-qing Lu, Qing-hua Zheng, Tie-jun Huang, and Wen Gao. 2017b.  Cross-media analysis and reasoning: advances and directions.  _Frontiers of Information Technology & Electronic Engineering_ 18, 1 (2017), pp.44–57.
  * Pereira et al. (2013) Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Nikhil Rasiwasia, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. 2013\.  On the role of correlation and abstraction in cross-modal multimedia retrieval.  _IEEE transactions on pattern analysis and machine intelligence_ 36, 3 (2013), pp.521–535.
  * Priyanka et al. (2013) M Priyanka, B Sunita Devi, SM Riyazoddin, and M Janga Reddy. 2013.  Analysis of cross-media web information fusion for text and image association-a survey paper.  _Global Journal of Computer Science and Technology_ 13, 1 (2013), pp.9–15.
  * Ranjan et al. (2015) Viresh Ranjan, Nikhil Rasiwasia, and CV Jawahar. 2015\.  Multi-label cross-modal retrieval. In _Proceedings of the IEEE International Conference on Computer Vision_. IEEE Computer Society, Santiago, Chile, pp.4094–4102.
  * Rasiwasia et al. (2014) Nikhil Rasiwasia, Dhruv Mahajan, Vijay Mahadevan, and Gaurav Aggarwal. 2014.  Cluster Canonical Correlation Analysis. In _Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, Reykjavik, Iceland, April 22-25, 2014_. JMLR.org, Reykjavik, Iceland, pp.823–831.  https://doi.org/10.1201/b18358-8
  * Roy et al. (2020) Anurag Roy, Vinay Kumar Verma, Kripabandhu Ghosh, and Saptarshi Ghosh. 2020.  ZSCRGAN: A GAN-based Expectation Maximization Model for Zero-Shot Retrieval of Images from Textual Descriptions. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. ACM, Virtual Event, Ireland, pp.1315–1324.
  * Ruder and Plank (2018) Sebastian Ruder and Barbara Plank. 2018.  Strong Baselines for Neural Semi-Supervised Learning under Domain Shift. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, Melbourne, Australia, pp.1044–1054.  https://doi.org/10.18653/v1/P18-1096
  * Shang et al. (2019) Fei Shang, Huaxiang Zhang, Lei Zhu, and Jiande Sun. 2019\.  Adversarial cross-modal retrieval based on dictionary learning.  _Neurocomputing_ 355 (2019), pp.93–104.  https://doi.org/10.1016/j.neucom.2019.04.041
  * Shao et al. (2017) Jie Shao, Zhicheng Zhao, Fei Su, and Ting Yue. 2017\.  Towards Improving Canonical Correlation Analysis for Cross-modal Retrieval. In _Proceedings of the on Thematic Workshops of ACM Multimedia 2017, Mountain View, CA, USA, October 23 - 27, 2017_ , Wanmin Wu, Jianchao Yang, Qi Tian, and Roger Zimmermann (Eds.). ACM, Mountain View, CA, USA, pp.332–339.  https://doi.org/10.1145/3126686.3126726
  * Socher et al. (2014) Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. 2014\.  Grounded compositional semantics for finding and describing images with sentences.  _Transactions of the Association for Computational Linguistics_ 2 (2014), pp.207–218.
  * Tian et al. (2018) Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. 2018.  Audio-Visual Event Localization in Unconstrained Videos. In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II_ _(Lecture Notes in Computer Science, Vol. 11206)_ , Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer, Munich, Germany, pp.252–268.  https://doi.org/10.1007/978-3-030-01216-8_16
  * Wang et al. (2017) Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. 2017.  Adversarial Cross-Modal Retrieval. In _Proceedings of the 25th ACM International Conference on Multimedia_ (Mountain View, California, USA) _(MM ’17)_. Association for Computing Machinery, New York, NY, USA, pp.154–162.  https://doi.org/10.1145/3123266.3123326
  * Wang et al. (2015) Jian Wang, Yonghao He, Cuicui Kang, Shiming Xiang, and Chunhong Pan. 2015.  Image-Text Cross-Modal Retrieval via Modality-Specific Feature Learning. In _Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, June 23-26, 2015_ , Alexander G. Hauptmann, Chong-Wah Ngo, Xiangyang Xue, Yu-Gang Jiang, Cees Snoek, and Nuno Vasconcelos (Eds.). ACM, Shanghai, China, pp.347–354.  https://doi.org/10.1145/2671188.2749341
  * Wang et al. (2016b) Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, and Liang Wang. 2016b.  A Comprehensive Survey on Cross-modal Retrieval.  _CoRR_ abs/1607.06215 (2016), arXiv:1607.06215.  arXiv:1607.06215 http://arxiv.org/abs/1607.06215
  * Wang et al. (2016a) Wei Wang, Xiaoyan Yang, Beng Chin Ooi, Dongxiang Zhang, and Yueting Zhuang. 2016a.  Effective deep learning-based multi-modal retrieval.  _The VLDB Journal_ 25, 1 (2016), pp.79–101.
  * Wu et al. (2020) Fei Wu, Xiao-Yuan Jing, Zhiyong Wu, Yimu Ji, Xiwei Dong, Xiaokai Luo, Qinghua Huang, and Ruchuan Wang. 2020\.  Modality-specific and shared generative adversarial network for cross-modal retrieval.  _Pattern Recognition_ 104 (2020), pp.107335.
  * Xie et al. (2020) De Xie, Cheng Deng, Chao Li, Xianglong Liu, and Dacheng Tao. 2020.  Multi-task consistency-preserving adversarial hashing for cross-modal retrieval.  _IEEE Transactions on Image Processing_ 29 (2020), pp.3626–3637.
  * Xiong et al. (2020) Haixia Xiong, Weihua Ou, Zengxian Yan, Jianping Gou, Quan Zhou, and Anzhi Wang. 2020\.  Modality-specific matrix factorization hashing for cross-modal retrieval.  _Journal of Ambient Intelligence and Humanized Computing_ 1 (2020), pp.1–15.  https://doi.org/10.1007/s12652-020-02177-7
  * Xu et al. (2020) Gongwen Xu, Xiaomei Li, and Zhijun Zhang. 2020.  Semantic Consistency Cross-Modal Retrieval With Semi-Supervised Graph Regularization.  _IEEE Access_ 8 (2020), pp.14278–14288.  https://doi.org/10.1109/ACCESS.2020.2966220
  * Xu et al. (5555) X. Xu, K. Lin, Y. Yang, A. Hanjalic, and H. Shen. 5555.  Joint Feature Synthesis and Embedding: Adversarial Cross-modal Retrieval Revisited.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 1, 01 (dec 5555), pp.1–1.  https://doi.org/10.1109/TPAMI.2020.3045530
  * Xu et al. (2020) X. Xu, H. Lu, J. Song, Y. Yang, H. T. Shen, and X. Li. 2020\.  Ternary Adversarial Networks With Self-Supervision for Zero-Shot Cross-Modal Retrieval.  _IEEE Transactions on Cybernetics_ 50, 6 (2020), pp.2400–2413.  https://doi.org/10.1109/TCYB.2019.2928180
  * Yanagi et al. (2020) Rintaro Yanagi, Ren Togo, Takahiro Ogawa, and Miki Haseyama. 2020\.  Enhancing cross-modal retrieval based on modality-specific and embedding spaces.  _IEEE Access_ 8 (2020), pp.96777–96786.
  * Yang et al. (2017) Erkun Yang, Cheng Deng, Wei Liu, Xianglong Liu, Dacheng Tao, and Xinbo Gao. 2017\.  Pairwise relationship guided deep hashing for cross-modal retrieval. In _proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 31. AAAI Press, San Francisco, California, USA, pp.1618–1625.  http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14326
  * Yu et al. (2018) Yi Yu, Suhua Tang, Kiyoharu Aizawa, and Akiko Aizawa. 2018\.  Category-based deep CCA for fine-grained venue discovery from multimodal data.  _IEEE transactions on neural networks and learning systems_ 30, 4 (2018), pp.1250–1258.
  * Zeng and Oyama (2019) Donghuo Zeng and Keizo Oyama. 2019.  Learning Joint Embedding for Cross-Modal Retrieval. In _2019 International Conference on Data Mining Workshops (ICDMW)_. IEEE, IEEE, Bejing, China., pp.1070–1071.
  * Zeng et al. (2018) Donghuo Zeng, Yi Yu, and Keizo Oyama. 2018.  Audio-Visual Embedding for Cross-Modal Music Video Retrieval through Supervised Deep CCA. In _2018 IEEE International Symposium on Multimedia, ISM 2018, Taichung, Taiwan, December 10-12, 2018_. IEEE Computer Society, Taichung, Taiwan, pp.143–150.  https://doi.org/10.1109/ISM.2018.00-21
  * Zeng et al. (2020) Donghuo Zeng, Yi Yu, and Keizo Oyama. 2020.  Deep triplet neural networks with cluster-cca for audio-visual cross-modal retrieval.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 16, 3 (2020), pp.1–23.
  * Zhang et al. (2021) Donglin Zhang, Xiao-Jun Wu, and Jun Yu. 2021.  Label consistent flexible matrix factorization hashing for efficient cross-modal retrieval.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 17, 3 (2021), pp.1–18.
  * Zhang et al. (2018b) Jian Zhang, Yuxin Peng, and Mingkuan Yuan. 2018b.  Unsupervised Generative Adversarial Cross-Modal Hashing. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). AAAI Press, New Orleans, Louisiana, USA, pp.539–546.  https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16746
  * Zhang et al. (2016) Liang Zhang, Bingpeng Ma, Guorong Li, Qingming Huang, and Qi Tian. 2016.  PL-Ranking: A Novel Ranking Method for Cross-Modal Retrieval. In _Proceedings of the 24th ACM International Conference on Multimedia_ (Amsterdam, The Netherlands) _(MM ’16)_. Association for Computing Machinery, New York, NY, USA, pp.1355–1364.  https://doi.org/10.1145/2964284.2964336
  * Zhang et al. (2018a) Xiang Zhang, Guohua Dong, Yimo Du, Chengkun Wu, Zhigang Luo, and Canqun Yang. 2018a.  Collaborative subspace graph hashing for cross-modal retrieval. In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_. ACM, Yokohama, Japan, pp.213–221.  https://doi.org/10.1145/3206025.3206042
  * Zhen et al. (2020) Liangli Zhen, Peng Hu, Xi Peng, Rick Siow Mong Goh, and Joey Tianyi Zhou. 2020.  Deep multimodal transfer learning for cross-modal retrieval.  , 13 pages.
  * Zhen et al. (2019) Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. 2019\.  Deep Supervised Cross-Modal Retrieval. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, Long Beach, CA, USA, pp.10394–10403.  https://doi.org/10.1109/CVPR.2019.01064
  * Zhou et al. (2018) Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L. Berg. 2018.  Visual to Sound: Generating Natural Sound for Videos in the Wild. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_. Computer Vision Foundation / IEEE Computer Society, Salt Lake City, UT, USA, pp.3550–3558.  https://doi.org/10.1109/CVPR.2018.00374
  * Zhu et al. (2019) Bin Zhu, Chong-Wah Ngo, Jingjing Chen, and Yanbin Hao. 2019\.  R2GAN: Cross-Modal Recipe Retrieval With Generative Adversarial Network. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, Long Beach, CA, USA, pp.11477–11486.  https://doi.org/10.1109/CVPR.2019.01174


# How to make the toss fair in cricket?

Haris Aziz
Latest version here
UNSW Sydney, Australia. Email<EMAIL_ADDRESS>
(27th Nov, 2021)

###### Abstract

In the sport of cricket, the side that wins the toss and has the first choice
to bat or bowl can have an unfair or a critical advantage. The issue has been
discussed by International Cricket Council committees, as well as several
cricket experts. In this article, I outline a method to make the toss fair in
cricket. The method is based on ideas from the academic fields of game theory
and fair division.

##  1 Introduction

Do you remember when your heart sank when your team lost the toss of a crucial
match on a tricky pitch? Or do you recall the annoying moment when the rival
fans downplayed your team’s victory by pointing out the toss outcome?

There is no doubt that the outcome of a cricket match can be affected by the
toss of a coin (Monga, 2018). On a fresh lush pitch, the test team being sent
out to bat first may already be at a disadvantage as the juicy pitch provides
ideal conditions for seam and swing. On a dry dusty pitch, the team that bats
last on the fifth day faces a significant disadvantage as the pitch will have
greatly deteriorated over the course of the match.

The winner of the toss can also get an undue advantage in T20 matches
especially in dewy conditions in the evenings. The trend has been noticed in
various T20 tournaments including PSL (Rasool, 2021). During the ICC T20 World
Cup 2021, 12 out of 13 matches in Dubai were won by the chasing team. Simon
Burnton from The Guardian wrote that “ _every evening match has seen 22 highly
skilled sportspeople spend several hours straining to see if they can have a
greater impact on the result than the momentary flight of a small metal disc
before the action began, and in general they have failed_ ” (Burnton, 2021).
When huge stakes are at play (such as the final of an ICC World Cup or the ICC
Test Championship), it seems highly unreasonable for the outcome of the match
to be affected by the outcome of the toss. When two evenly matched teams play,
you want an exciting and evenly matched game. Chopra (2013) pointedly asked:

> _“Would football tolerate a toss of a coin making one side play with only 10
> players? In swimming or Formula One could chance decide pole position? Would
> every boxing match begin with the tossing of a coin which would permit one
> of the two boxers a chance to land a couple of hard rights as the bout
> began? I don’t think so.”_

##  2 The Toss, Propose and Choose Method

The issue of the unreasonable impact of the toss has been discussed for years
(Sood and Willis, 2016). It is important enough that the ICC has considered
searching for fairer solutions (Cricket365, 2018). So how can we ensure that
the toss continues to be meaningful but doesn’t have an unfair effect on the
outcome of the match? I have a proposal to keep the toss in cricket but make
it fair. I propose a minimal adaptation to the rules of cricket. I call the
proposal the Toss, Propose and Choose method.

The Toss, Propose and Choose method.  TOSS: The toss takes place as per
tradition. [Let us call the captain who wins the toss the ‘Lucky Captain’ and
the captain who loses the toss the ‘Unlucky Captain’]. PROPOSE: The Unlucky
Captain judges whether bowling or batting first is disadvantageous and
estimates the marginal impact of bowling versus batting first in terms of
runs. Say the estimated number of runs is $b$. The Unlucky Captain proposes
the following two options to the Lucky Captain: Option 1. Take the
advantageous turn but give $b$ bonus runs to the other team Option 2. Take the
disadvantageous turn but get $b$ bonus runs. CHOOSE: The Lucky Captain (who
won the toss) then chooses either option 1 or 2.

Figure 1 gives a graphical illustration of the Toss, Propose and Choose method

Figure 1: A graphical illustration of the Toss, Propose and Choose method.

######  Example 1.

Let us illustrate the proposed method with the help of an example. Suppose
Australia is playing New Zealand in the ICC Test Championship Final. We do not
wish for a championship spanning multiple years to be eventually heavily
influenced by the toss outcome. The coin is tossed and Australia’s captain
wins. At this point, the New Zealand captain is asked to give two options to
the Australian captain. New Zealand factors in the playing conditions and team
lineups, and feels that they will gain a net 50 run advantage if they bowl
first. In view of this, the Australian captain gets two options: bowl or bat
first but with the team batting first getting 50 additional runs in the extras
column.

Now let us logically explain why the proposed method is fair to both parties.
The Unlucky Captain (who loses the toss) gets a full opportunity to ensure
that the possible options (1 and 2) weigh up equally, so irrespective of which
option is chosen, s/he does not feel unlucky. More precisely, the Unlucky
Captain has no envy and does not wish to switch between the two options. The
Lucky Captain gets the final say in choosing the preferred option among 1 and
2. Hence, the Lucky Captain also feels that their team is not disadvantaged.
Voila! The method I am advocating has solid foundations as it can be viewed as
a suitable adaptation of the classical Divide and Choose method for fairly
dividing a divisible resource (Brams and Taylor, 1996). Just as Divide and
Choose, my method does leave scope for the New Zealand Captain to be
strategic. For example, if Australia’s captain habitually prefers bowling
first, the New Zealand captain can increase the bonus runs from 50 to 70.
However, even under strategic behaviour, the outcome remains fair (envy-free).

Let me give another perspective of why the proposed method is fair. Suppose
bowling first versus batting are compared on a balanced weight scale by the
unlucky captain. The unlucky captain is concerned that the lucky captain is
going to select the heavier side. The unlucky captain has the chance to put
sufficient weight / bonus runs on the lighter side to ensure both sides have
the same weight. In that case the Unlucky Captain does not need to worry which
side the lucky captain will choose.

A typical complaint against the proposal could be why complicate a simple
toss. However, cricket has already been open to more complicated methods such
as Duckworth-Lewis-Stern to make other situations fair. If the Unlucky Captain
truly believes that it does not matter who bats first, then s/he can stick to
the default of zero bonus runs. Otherwise, the captain can do the following
thought experiment. “Would I prefer to bowl or bat? Would I be indifferent if
I added 10 runs to the lesser preferred option? Or 20 runs?” At some point,
one would be indifferent between the two options because of the bonus runs
added to the lesser preferred option. Cricket legend Sunil Gavaskar opined
that that issue of the unfair toss “i _s something for the ICC Cricket
Committee to get their heads around and make sure that there is a level
playing field for both teams_ ” (Nanda, 2021). Ian Chappel echoed similar
thoughts (hindustantimes.com, 2021). The main objective of the Toss, Propose
and Choose method is to level the playing field and eliminate the potentially
significant headstart the winner of the coin toss gets.

##  3 Comparison with Other Proposals

Apart from my proposal, a few other ideas have been floated to avoid the undue
impact of a coin. A frequently proposed solution is for the teams to alternate
batting first over the course of a series. The solution regains some fairness
over time, especially if the series has enough matches. However, it doesn’t
resolve the issue of an unfair toss for the deciding or final match or for
tournaments. Another proposal is for the weaker, or the touring team, to
decide. However, the proposal may give too much advantage to the slightly
weaker team and may incentivize teams to have lower rankings. Secondly, the
match (say the ICC World Cup) may be on neutral ground, in which case both
teams may be touring.

One elegant proposal to achieve fairness is to get rid of the toss entirely
and introduce an auction (Franklin, 2018; Sood and Willis, 2021). It leads to
certain issues such as ensuring simultaneous bids in a transparent yet TV-
friendly manner; the decision always going to the more aggressive bidder (that
helps teams that are one-trick ponies such as always chasing); the higher
bidder being victim of the winner’s curse (a phenomenon that is encountered in
real estate auctions); and the need for tie-breaking under identical bids.
Finally, bypassing the toss completely eliminates an element that many might
see as intrinsic to traditional cricketing sensibilities. In this sense, the
Toss, Propose and Choose method is a minimal and transparent adaptation of the
existing toss to regain fairness and only requires a handicap elicitation from
one of the captains. It makes it easy to understand why both sides are free of
envy of each other. If the Unlucky Captain is indifferent between bowling and
batting first, the method coincides with the traditional toss. In 2018, some
of the fair alternatives were considered but the ICC decided to keep the toss
(Wisden, 2018). The decision was in line with the MCC clause 13.5 (“” _the
captain of the side winning the toss shall decide whether to bat or to field
and shall notify the opposing captain and the umpires of this decision_ ”
(MCC, n.d.)). My proposal is to keep the toss but to make it fair.

##  4 Final Words

So is the cricket community ready to consider the Toss, Propose and Choose
method that is designed for a high-stakes match? Ironically, I feel that a
suitable place to experiment is in low-stakes junior matches to see how the
cricketing community feels about it. I won’t complain if the proposal also
gets taken up at the next ICC Test Championship Final! I suspect that the
captain who loses the toss won’t either.

## Acknowledgments

The author thanks Robbie Boland for valuable comments.

## References

  * (1)
  * Brams and Taylor (1996) Brams, S. J. and A. D. Taylor, Fair Division: From Cake-Cutting to Dispute Resolution, Cambridge University Press, 1996.
  * Burnton (2021) Burnton, Simon, “T20 World Cup was flawed, fun and must be viewed through dewy highs,” https://www.theguardian.com/sport/2021/nov/15/t20-world-cup-flawed-fun-dew-cricket Nov 2021.
  * Chopra (2013) Chopra, Samir, “The fair and the unfair: the mystery of the toss,” https://www.espncricinfo.com/story/samir-chopra-the-fair-and-the-unfair-the-mystery-of-the-toss-620121 Feb 2013.
  * Cricket365 (2018) Cricket365, “ICC cricket committee opts against scrapping the toss,” https://www.cricket365.com/top-story/icc-cricket-committee-opts-against-scrapping-the-toss May 2018.
  * Franklin (2018) Franklin, David, “Pushing the (sealed) envelope: An alternative to the coin toss,” https://wisden.com/stories/long-room/a-good-toss-to-lose-david-franklin-pushes-crickets-sealed-envelope April 2018.
  * hindustantimes.com (2021) hindustantimes.com, “‘Win the toss, win the game’: Former Australia captain Ian Chappell points out ‘major flaws’ of T20 World Cup,” https://www.hindustantimes.com/cricket/former-australia-captain-ian-chappell-points-major-flaws-of-t20-world-cup-101637476324095.html Nov 2021.
  * MCC (n.d.) MCC, “The Laws,” https://www.lords.org/mcc/the-laws-of-cricket/innings.
  * Monga (2018) Monga, Sidharth, “Toss advantage $>$ home advantage?,” https://www.espncricinfo.com/story/toss-advantage-home-advantage-1168330 Dec 2018.
  * Nanda (2021) Nanda, Rashmi, “T20 World Cup 2021: ICC needs to ensure a level playing field and look into unfair toss advantage — Gavaskar,” https://circleofcricket.com/category/T20_World_Cup/71120/t20-world-cup-2021-icc-needs-to-ensure-a-level-playing-field-and-look-into-unfair-toss-advantage-gavaskar Nov 2021.
  * Rasool (2021) Rasool, Danyal, “Win the toss and opt to chase — the only certainty in PSL,” https://www.espncricinfo.com/story/psl-2021-win-the-toss-and-opt-to-chase-the-only-certainty-in-psl-1265581 June 2021.
  * Sood and Willis (2016) Sood, Gaurav and Derek Willis, “Fairly Random: The Impact of Winning the Toss on the Probability of Winning,” arXiv: Applications, 2016.
  * Sood and Willis (2021) Sood, Gaurav and Derek Willis, “Why replacing the toss with an auction is the fair thing to do,” https://www.espncricinfo.com/story/why-replacing-the-toss-with-an-auction-is-the-fair-thing-to-do-1277744 Sep 2021.
  * Wisden (2018) Wisden, “ICC Cricket Committee decides against scrapping the toss,” https://wisden.com/stories/archive/news/icc-cricket-committee-toss-coin May 2018.