#  IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level
Grounding of Images

Varuna Krishna1 S Suryavardan 1 Shreyash Mishra1
Sathyanarayanan Ramamoorthy2 Parth Patwa3 Megha Chakraborty4
Aman Chadha†5,6 Amitava Das4 Amit Sheth4
1IIIT Sri City, India 2CMU, USA 3UCLA, USA
4University of South Carolina, USA 5Amazon AI, USA 6Stanford University, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Word embeddings, i.e., semantically meaningful vector representation of words,
are largely influenced by the distributional hypothesis "You shall know a word
by the company it keeps" Harris (1954), whereas modern prediction-based neural
network embeddings rely on design choices and hyperparameter optimization.
Word embeddings like Word2Vec, GloVe etc. well capture the contextuality and
real-world analogies but contemporary convolution-based image embeddings such
as VGGNet, AlexNet, etc. do not capture contextual knowledge. The popular
king-queen analogy does not hold true for most commonly used vision
embeddings.

In this paper, we introduce a pre-trained joint embedding (JE), named
IMAGINATOR, trained on 21K distinct image objects level from 1M image+text
pairs. JE is a way to encode multimodal data into a vector space where the
text modality serves as the grounding key, which the complementary modality
(in this case, the image) is anchored with. IMAGINATOR encapsulates three
individual representations: (i) object-object co-location, (ii) word-object
co-location, and (iii) word-object correlation. These three ways capture
complementary aspects of the two modalities which are further combined to
obtain the final JEs.

Generated JEs are intrinsically evaluated to assess how well they capture the
contextuality and real-world analogies. We also evaluate pre-trained
IMAGINATOR JEs on three downstream tasks: (i) image captioning, (ii)
Image2Tweet, and (iii) text-based image retrieval. IMAGINATOR establishes a
new standard on the aforementioned downstream tasks by outperforming the
current SoTA on all the selected tasks. IMAGINATOR will be made publicly
available. The codes are available at https://github.com/varunakk/IMAGINATOR

22footnotetext: Work does not relate to position at Amazon.

##  1 Joint Modality and Contextuality

Word embeddings are learned representations such that words with similar
meanings are represented similarly. Distribution-based compositional word
embeddings like Word2vec Mikolov et al. (2013a) and GloVe Pennington et al.
(2014) are popular in modern NLP. These are used to extract the notion of
relatedness across different words, and capture the overall semantic meaning
of a text. Consider the king-queen Mikolov et al. (2013b) word vector analogy
(figure 1), which shows how good these word embeddings are at capturing
syntactic and semantic regularities in language.

The notion of contextual similarity (i.e., words occurring together) is used
in learning the representations, because of which vector arithmetic like King
- Man + Woman = Queen are possible. See figure 1 Mikolov et al. (2013b).
Deriving an analogous representation using images is a challenging task since
the concept of relatedness among images is not well-defined. Motivated by this
argument, we propose creating joint embeddings (JEs) that can represent real-
world analogies, which can aid in solving several multimodal tasks owing to
their distributional semantics.

Figure 1: CNN-based image embeddings are unable to capture contextuality like
existing word embeddings. The king-queen vs. man-woman analogy has been
popularized by Mikolov et al. (2013b), whereas drawing a similar analogy in
image vector space is rather difficult. We argue joint embedding is the
alternative.

##  2 Contemporary Joint Embedding Methods

Canonical Correlation Analysis (CCA) based methods use similarities to project
two inputs onto a vector space. CLIP Radford et al. (2021) utilizes
contrastive pre-training and encodes aligned image and text embeddings with
the help of text and visual modality encoders. Stanford’s Joint Embedding
Kolluru (2019) uses VGG-19 Simonyan and Zisserman (2014) and GLoVe Pennington
et al. (2014) to generate the image and text encodings using a triplet loss.
Chen et al. (2020) proposed UNITER, trained on a large dataset, which uses an
image and text encoder and a transformer to generate the final embeddings. Jia
et al. (2021) use a noisy dataset of 1 billion (image, alt-text) pairs and
propose a dual architecture for aligning and generating the visual and textual
embeddings. This architecture uses contrastive loss for learning. Tan and
Bansal (2019) proposed a framework to create a relation between visual and
language modalities. This architecture consists of three encoders, one object
relation encoder, a language encoder and a cross-modal encoder. Compared to
the aforementioned prior works, illustrated in appendix figure 8, the unique
differentiating factor with IMAGINATOR is that we focus on the word-level
grounding Gunti et al. (2022) of images while prior works perform embedding
generation at the sentence level. Our belief is that this will help us learn
rich relational features, i.e., features that are rich encapsulations of words
and the corresponding objects they represent via images.

##  3 IMAGINATOR - Learning Joint Embeddings

Off-the-shelf word embeddings like Word2vec Mikolov et al. (2013a), GloVe
Pennington et al. (2014) and the embeddings generated by BERT Devlin et al.
(2018), GPT Radford et al. (2021) are used for tacking several downstream NLP
tasks. The motivation behind creating IMAGINATOR is to have similar pre-
trained embeddings for vision-language tasks. Researchers can download pre-
trained JEs and utilize them for any vision-language task they have in hand.
Existing techniques have only explored JEs from the sentence-level
perspective, which makes it less flexible to re-purpose them for other tasks,
but most importantly, demands a lot more data for the model to understand and
derive meaningful relationships. We thus operate at the word level rather than
sentence-level, to help improve the "sharpness" of the data, with the hope
that this would, in turn, help synthesize higher relational features that can
offer optimal performance on downstream tasks. To that end, we make some
simple assumptions and posit arguments on their choice as better alternatives.

###  3.1 Object vs. Word - a Unit Hypothesis

The smallest meaningful unit of text is a word, which we assume signifies a
visual object embedded in an image. Albeit, the common trend is to train end-
to-end network on sentence-level, but system may not be able to learn fine
grained contextual relations like king-queen analogy. This design choice also
aligns with our motivation to generate general-purpose JEs suited for a wide
variety of downstream tasks (refer section 5).

####  3.1.1 Number of Objects

The number of objects in available datasets like Flickr30k Young et al. (2014)
and COCO Lin et al. (2014) is limited only to a few hundred. However, if we
are interested to learn real-world analogies like king-queen analogy, we
require far more real-life objects to be detected by the system. Detic Zhou et
al. (2022), a recent object detection technique, provides 21K object classes
and thus, seems the most pertinent. Results shown in table 6 indicate that an
increment in the number of objects leads to a corresponding increase the
accuracy.

Figure 2:  Architecture for creating text embeddings and $v_{oo}$ and
$v_{wo}$: the rows and columns in the co-location matrix are the words from
the text or objects detected from the images from dataset. Each cell of this
matrix represents the occurrence count of each row-column pair in the dataset.
The two final vectors are generated using PPMI and eigenvalue weighting over
the vectors from co-location matrices. (Bottom) Architecture for learning
$v_{wor}$: (left) the averaged VGG19 representation of a particular object
across the whole dataset is passed; (right) word2vec representation of the
word (i.e., the name of the visual object; for e.g., horse in this case).

Based on the unit hypothesis, we capture three aspects of the input data while
generating joint embeddings: Object-object co-location: $v_{oo}$, Word-object
co-location: $v_{wo}$, Word-object correlation: $v_{wor}$.

###  3.2 Learning $v_{oo}$ and $v_{wo}$

Figure 2 offers a visual summary of the process of generating object-object
co-location embeddings $v_{oo}$ and word-object co-location embeddings
$v_{wo}$. $v_{oo}$ and $v_{wo}$ are learned using an object co-location
matrix, where objects refer to the entities detected using an object detection
model. Object co-location matrix is a matrix where the rows and columns
correspond to objects detected in our images and each cell represents the co-
occurrences of the respective two objects. We then take the rows and apply
dimensionality reduction techniques like SVD along with Eigenvalue weighting.
The vector obtained is then used as the embedding. This yields object-object
co-location, which encodes how frequently a detected object co-appears with
other detected objects in the dataset. On the other hand, word-object co-
location is built using the objects from object detection on images and the
words from the associated text given in the datasets. This might seem similar
to object-object co-location at first glance, but a major difference is that
the value in each cell represents the number of image captions having the
corresponding object and word pair. With this co-location matrix, we get
information on how frequently every object co-appears with other words in the
dataset.

###  3.3 Learning $v_{wor}$

Figure 2 illustrates the process of generating the word-object correlation
embeddings $v_{wor}$. $v_{wor}$ is learnt using a different approach when
compared with the other two embeddings. Co-location can be defined using the
co-occurrence of two entities but correlation calls for a deeper understanding
of the two entities. Therefore, we get joint embeddings for word-object
correlation using word and object vectors.

(a)

(b)

(c)

(d)

Figure 3:  Similar images and the vector space distance between them. Word
pairs taken from Flickr30k datsset. The IMAGINATOR JE vector space captures
real-world analogies well.

We generate object embeddings by passing all detected crops of the object from
the dataset to VGG19 Simonyan and Zisserman (2014). An average of these
embeddings across all instances give us the final embedding for the object
encoded as a mean representation. The word embeddings are acquired by creating
a word-word co-location matrix for the text in the dataset, similar to the
aforementioned co-location matrices, where each cell represents the number of
co-occurrences of the corresponding word pair.

To obtain the final joint embedding from these two vectors, we project the
object embedding in the word embedding space instead of projecting both
embeddings in a common space Kolluru (2019); Radford et al. (2021). The
motivation behind this is to maintain the contextuality captured in word
embeddings and thus enforce the object embeddings to learn the correlations.
Once they learn a correlated vector space, we get the JEs from a weighted
average of the projected word and object embedding. We perform experiments to
compare several projection methods (such as CCA Thompson (2000), Kernel CCA
Hardoon et al. (2004), Deep CCA Andrew et al. (2013) etc.) and loss functions
(InfoNCE Oord et al. (2018), contrastive loss Hadsell et al. (2006), and
triplet loss Schroff et al. (2015)). Emperically, we find that orthogonal
projection and triplet loss give the best JE results. We believe CCA overfits
on our data while orthogonal projection Artetxe et al. (2018) uses the
features based on the dataset size. Please refer to table 6 in Appendix for
more on these experiments and their results.

##  4 Lessons Learnt from NLP

Levy et al. (2015) argue that the performance gains of neural network based
word embeddings are due to certain system design choices and hyperparameter
optimizations, rather than the embedding algorithms themselves. Furthermore,
they show that these modifications can be transferred to traditional
distributional models, yielding similar gains. In contrast to prior reports,
they show mostly local or insignificant performance differences between the
methods, with no global advantage to any single approach over the others.
Therefore, we remain grounded to count-based distributional semantics methods.
Raw counts or normalized counts are not useful, rather we choose alternatives
like PMI and SVD.

###  4.1 PPMI and Context Distribution Smoothing

The PPMI (Positive Pointwise Mutual Information) between a word and its
context is well known to be an effective association measure in the word
similarity literature. Levy et al. (2015) show that the skip-gram with
negative-sampling training method (SGNS) is implicitly factorizing a word-
context matrix whose cell values are essentially shifted PMI. Following their
analysis, we present two variations of the PMI (and implicitly PPMI)
association metric, which we adopt from SGNS. In this section, $w$ and $c$
represent the word and context matrix.

Shifted PMI. The shift caused by $1<k$ (the number of negative samples in the
optimization $(w,c)$: $PMI(w,c)-log(k)$) can be applied to distributional
methods through shifted PPMI Levy and Goldberg (2014):

The $k$ here, firstly, estimates negative sample distribution and secondly,
acts as a prior on the probability of an occurrence of $(w,c)$ in the corpus
vs. a negative sample. Shifted PPMI captures the latter, i.e, the prior aspect
of $k$.

$SPPMI(w,c)=max(PMI(w,c)-log(k),\>0)$ (1)

Context Distribution Smoothing (CDS). Word2Vec Mikolov et al. (2013a) samples
negative samples according to a smoothed unigram distribution. This smoothing
variation has an analog when calculating PMI directly:

$PMI_{\alpha}(w,c)=log\frac{\hat{P}(w,c)}{\hat{P}(w).\hat{P}_{\alpha}(c)}$ (2)
$PMI_{\alpha}(c)=\frac{\\#(c)^{\alpha}}{\Sigma_{c}\\#(c)^{\alpha}}$ (3)

By enlarging the probability of sampling a rare context (since
$\hat{P}_{\alpha}(c)>\hat{P}(c)$ when $c$ is infrequent), CDS reduces the PMI
of $(w,c)$ for a rare context $c$ – thus removing PMI’s bias towards rare
words.

###  4.2 SVD and Eigenvalue Weighting

Word and context vectors derived using SVD of co-location matrices can be
represented by:

$W^{SVD}=U_{d}\cdot\Sigma_{d}\;\;\;C^{SVD}=V_{d}$ (4)

However, in this case, $C^{SVD}$ is orthonormal while $W^{SVD}$ is not.
Factorization achieved by SGN is much more symmetric and a similar symmetry
can be derived using the following factorization:

$W=U_{d}\cdot\sqrt{\Sigma_{d}}\;\;\;C=V_{d}\cdot\sqrt{\Sigma_{d}}$ (5)

Levy et al. (2015) states that while it is not theoretically clear why a
symmetric approach performs better for semantic tasks, it works empirically.

For our vector-deriving implementation, we use this as a dimensionality
reduction technique. It is similar to SVD but instead of the usual
representation: $W=U.\Sigma_{d}$ and $C=V_{d}$, eigenvalue weighting uses
$W=U.\Sigma_{d}^{0.5}$ and $C=V_{d}$. To summarize, after creating the co-
location matrix, we derive vectors by initially applying SPPMI with CGS. This
is followed by the SVD of the matrices with eigenvalue weighting.

###  4.3 Merging $v_{oo}$, $v_{wo}$, and $v_{wor}$

The three vectors can be merged using approaches such as concatenation,
averaging or autoencoding. Autoencoder is a pertinent research topic where
merging of a number of vectors is learnt automatically by a trained model.
This approach considers learning the embeddings by considering complementary
information from it’s source embeddings. In the interest of simplifying this
aspect of our design, for our experiments, we use weighted average to combine
the embeddings. The weights are decided empirically. The best weights we find
are 10, 10, and 80 for $v_{oo}$, $v_{wo}$, and $v_{wor}$ respectively.

Figure 4: Architecture of Image captioning using IMAGINATOR. We use VGG19 for
embedding and then we use IMAGINATOR to project into the joint embedding
space. We pick K nearest objects/words and pass to seq2seq model to generate
caption.

##  5 Intrinsic Evaluation of IMAGINATOR

To be able to make vector arithmetic like King - Man + Woman = Queen in a
generated word vector space is well known as the intrinsic evaluation
paradigm. Contemporary image embeddings are devoid of contextuality, whereas
text embeddings are much more meaningful, as shown in figure 1. With joint
embeddings, we aim to add a contextual component to improve the semantic
richness of the joint embeddings vector space. We use two kinds of intrinsic
evaluation setup to evaluate IMAGINATOR: (i) word contextuality, and (ii)
image similarity.

Dataset | GloVe | CLiP | SJE | Ours
---|---|---|---|---
WS353 Finkelstein et al. (2001) | 2.65 | 0.35 | 0.19 | 0.14
MTurk Halawi et al. (2012) | 1.99 | 0.26 | 0.28 | 0.20
RG65 Rubenstein et al. (1965) | 0.75 | 0.38 | 0.20 | 0.13
RW Pilehvar et al. (2018) | 0.96 | 0.19 | 0.17 | 0.25
SimLex999 Hill et al. (2015) | 2.31 | 0.18 | 0.22 | 0.12
MEN Bruni et al. (2014) | 0.51 | 0.22 | 0.13 | 0.11
Google Analogy Mikolov et al. (2013a) | 2.09 | 0.18 | 0.12 | 0.15
MSR Analogy Mikolov et al. (2013b) | 0.63 | 0.30 | 0.09 | 0.22
SemEval2012 Jurgens et al. (2012) | 1.2 | 0.21 | 0.32 | 0.26
BLESS Baroni and Lenci (2011) | 2.77 | 0.22 | 0.19 | 0.11
Average | 1.5 | 0.25 | 0.19 | 0.16

Table 1: Results (average euclidean distance) for intrinsic evaluation of our
JEs based on notable word contextually datasets. Lower is better. Our model
outperforms the baselines on most of the datasets and has the lowest overall
average distance.

###  5.1 Word Contextuality

We use all the $10$ datasets mentioned in Jastrzebski et al. (2017) to
evaluate the generated word embeddings intrinsically. Intrinsic means that
only basic arithmetic functions are performed on the embeddings and no other
models are trained. The datasets cover three tasks: (i) word similarity, (ii)
word analogy, and (iii) word categorization. First, the word embeddings for
given pair of similar words from the datasets are computed. Then, we use
average euclidean distance to derive the final results ( as shown in table 1)
for embeddings from GloVe Pennington et al. (2014), CLIP Radford et al.
(2021), and IMAGINATOR. We can see that IMAGINATOR performs better than the
baselines.

###  5.2 Image Similarity

Analogy-making on images is relatively challenging. Our hypothesis is that
vectors of the same/similar objects must be nearby in the IMAGINATOR vector
space. We evaluate IMAGINATOR intrinsically on image similarity task using
objects five datasets - Caltech 101 Li et al. (2022a), Flickr 30k, MS COCO,
Google CC Sharma et al. (2018), Visual Genome Krishna et al. (2017). We
extract the list of similar objects from the the datasets, obtain features
from the VGG19 and then orthogonally Artetxe et al. (2018) project those
objects to the IMAGINATOR vector space. We then calculate the pairwise-
euclidean distance between such vectors and average them for the entire
dataset. Table 2 shows the object similarity performance of SJE, CLIP and
IMAGINATOR on a a variety of datasets. Our baseline comprehensively
outperfroms the baselines on all the datasets. Figure 3 shows some examples of
the relation between projected JEs of these objects. From the examples we can
see that IMAGINATOR captures the nuances of the images.

Dataset | SJE | CLIP | IMAGINATOR
---|---|---|---
Caltech 101 | 1.9 | 1.5 | 0.13
Flickr 30K | 0.8 | 0.4 | 0.06
MS COCO | 0.9 | 1.3 | 0.2
Google CC | 0.2 | 0.4 | 0.08
Visual Genome | 1.1 | 1.4 | 0.1
Average | 0.98 | 1.00 | 0.11

Table 2: Average pairwise euclidean distance between similar objects from each
dataset. Lower is better.

##  6 IMAGINATOR for Downstream Tasks

(a) IMAGINATOR: A kitchen with a sink, stove, oven, and beers. Gold Caption: A
commercial stainless kitchen with a pot of food cooking. OSCAR: a kitchen with
a lot of pots and pans in it.

(b)  IMAGINATOR: A vocalist, drummer, and a guitarist sings a tune. Gold
Caption: A musical band are by their instruments most likely playing a song.
OSCAR: A man on a stage with a guitar and a keyboard.

Figure 5: Examples of some image captioning outputs generated by IMAGINATOR
along with the original caption and the caption generated by OSCAR Li et al.
(2020). IMAGINATOR gives richer and more detailed captions than OSCAR. For
more examples please refer the appendix.

The downstream vision-language (VL) tasks chosen to test our pre-trained JEs
are: (i) image captioning, (ii) Image2Tweet Jha et al. (2021), and (iii) text-
based image retrieval.

###  6.1 Image Captioning

Image captioning is a common multimodal task which involves the generation of
a textual description for an image. Describing the contents of an image
requires visual understanding at an object level. We use JEs from IMAGINATOR
to generate captions on datasets such as Flickr30k Young et al. (2014) and
COCO Lin et al. (2014).

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 6: Image retrieved for the query - "several climbers climbing rock
together" \- it is evident that ALBEF Li et al. (2021) wrongly emphasized on
"rock together", whereas XVLM Zeng et al. (2021) is unable to comprehend
plurality in the query here, while BERTIMAGINATOR can do the job well.

For an input image, we start by obtaining an image embedding using VGG19
Simonyan and Zisserman (2014), which is then orthogonally projected in
IMAGINATOR embedding space. We use the JE of the image to find $k$ nearest
objects in the vector space. For our experiments we used $k=10$, giving us 10
objects associated with the input image. These objects are then passed to a
sequence-to-sequence module, namely, the T5 transformer Bhatia (2021), which
generates the final caption. We use a pre-trained T5 model, fine-tuned on
Flickr30k and COCO. Figure 4 describes the captioning pipeline while Figure 5
shows some output examples.

Table 3 shows the quantitative results of baseline models and IMAGINATOR. We
can see that our model outperforms all the baselines in terms of BLEU score
and BERTScore Zhang et al. (2020) on both the datasets.

method | Flickr30k | COCO
---|---|---
| BLEU | BERTScore | BLEU | BERTScore
UVLP | 30.1 | - | - | -
OSCAR | - | - | 41.7 | -
SJE | 30.5 | 0.78 | 35.6 | 0.8
CLIP | 31.3 | 0.83 | 36.3 | 0.85
BLIP | - | - | 40.4 | -
IMAGINATOR | 33.2 | 0.87 | 43.1 | 0.88

Table 3: Results of different models on the image captioning task. Higher is
better. Unavailable scores are left blank.

###  6.2 Image2Tweet

Image2Tweet Jha et al. (2021) is a task which is a notch above traditional
image captioning in terms of complexity. Given an input image, the task
involves generating a tweet like a human news reporter. Figure 22 shows some
examples from the dataset.

The tweet is generated using a method similar to image captioning. The joint
embedding of the input image is used to find the $k$ nearest neighbouring
embeddings in the projections space. These neighbours are then used to
generate the tweet using a sequence-to-sequence model.

Figure 7: $BERT_{IMAGINATOR}$ \- Training approach for Image Retrieval.
Training happens in batches and cosine similarity between corresponding image-
sentence pair is maximised while for other pairs it is minimized.

The results are based on the CIDEr metric (refer table 4). We found that using
other datasets for training SoTA models yielded abysmal results, indicating
that Image2Tweet is a fairly complex problem. However, IMAGINATOR performs
reasonably well on the task and surpasses comparable contemporary SoTA
captioning methods, namely UVLP Zhou et al. (2020) and OSCAR Li et al. (2020).

Method | CIDEr
---|---
Baseline of Image2tweet Jha et al. (2021) | 0.0003
UVLP Zhou et al. (2020) [SoTA on Flickr] | 0.003
OSCAR Li et al. (2020) [SoTA on COCO] | 0.004
CLIP Radford et al. (2021) | 0.006
Stanford joint embedding Kolluru (2019) | 0.007
5 ensemble Luo et al. (2018) | 0.0090
IMAGINATOR + $k$ nearest objects + T5 | 0.0095

Table 4: Results (CIDEr score) of various multi-modal models on the
Image2Tweet task. Higher is better. Our model outperforms other all models.

###  6.3 Text-based Image Retrieval

The fundamental question that we are seeking an answer to is whether using
IMAGINATOR word-object level embeddings we can achieve compositionally and
achieve a vector representation for sentence-image level. For example, by
passing on word vectors in a sequence to a language model we can obtain a
sentence-level vector representation. To verify the compositionality of joint
modality embeddings, we test our approach on the task of text-based image
retrieval on the Flickr30K dataset Young et al. (2014). The main challenge of
this task is to find out the appropriate content in the visual space while the
input is in the text space. Another reason for introducing compositionality is
that each word is usually associated with multiple images. Hence, there is a
need for us to learn a single image representation for a given text. Though we
explore contrastive methods in section Contrastive Learning, to solve the
above-mentioned challenges, we introduce an approach using BERT and evaluate
it on text-based image retrieval.

Method | R@1 | R@5 | R@10
---|---|---|---
ALBEF Li et al. (2021) | 85.6 | 97.5 | 98.9
XVLM Zeng et al. (2021) | 86.1 | 97.3 | 98.7
BLIP Li et al. (2022b) | 87.6 | 97.7 | 99.0
BERTIMAGINATOR | 89.48 | 98.1 | 99.2

Table 5: Results on image retrieval: Recall@{1, 5, 10} Score for the Flickr30K
dataset. Higher is better.

####  6.3.1 Compositionality of Joint Embeddings - $BERT_{IMAGINATOR}$

BERT is arguably the most successful modelling architectures in NLP. It
accepts token embeddings as input and produces contextualized embeddings as
output. In contrast, we propose $BERT_{IMAGINATOR}$, which is trained to take
image+text as input and output a compositional vector representation for both
modalities.

We utilize BERT Devlin et al. (2018) and CLIP Radford et al. (2021) as our
backbones to generate JEs. Instead of feeding the BERT model tokenized words
obtained via a tokenizer, we use IMAGINATOR (refer section 3.3) word-object
embeddings as input to the model. We process necessary tokenization, position
encoding, and segment embeddings accordingly, per the BERT architecture.

We utilize CLIP Radford et al. (2021) for generating another JE using an
image-sentence pair by obtaining the image and text embeddings from CLIP
encoders and concatenating them. We refer to this as the sentence JE. Both
these embeddings, viz., the sentence JE and projected $BERT_{IMAGINATOR}$, are
projected to a common space using orthogonal projection Artetxe et al. (2018),
on which we compute our loss. Figure 7 visually depicts our training process
while table 5 shows BERTIMAGINATOR outperforming SoTA information retrieval
(IR) baselines, namely ALBEF Li et al. (2021) and XLVM Zeng et al. (2021) on
Recall@{1, 5, 10}. Some output examples are shown in figure 6.

##  7 Conclusion and Futurework

We proposed a new pre-trained joint embedding IMAGINATOR. Our major
contribution is on adopting count-based methods for joint modality, echoing
the philosophy from Levy et al. (2015). We present an in-depth intrinsic
evaluation along with a new architecture $BERT_{IMAGINATOR}$. IMAGINATOR
outperformed SoTA on three tasks: (i) image captioning, (ii) Image2Tweet, and
(iii) text-based image retrieval. In the future, we would like to explore
other multimodal tasks such as VQA.

## Discussion and Limitations

While IMAGINATOR pushes the boundaries of the state-of-the-art in tasks that
involve language and vision joint modelling, there are some limitations.

### Object Detection - Limited Number of Classes

IMAGINATOR utilizes the atomic units of multimodal data – individual words for
text representation and individual objects for image representation.
Typically, the number of unique words (i.e., the vocabulary) is quite large in
a given text relative to the number of objects in images. As such, IMAGINATOR
being a joint learning technique is bottlenecked by the capabilities of
existing object detection techniques since they only typically deal with a
limited repertoire of objects. To enhance the richness and expressivity of
JEs, object detection models that can identify the wide gamut of objects in
the world would be critical.

### Contrastive Learning

Contrastive learning is a task-independent technique that focuses in learning
the similarity and differences between samples in a dataset. The objective
here is to learn an embedding space where similar inputs, say samples
belonging to the same class, are embedded as similar representations while
samples from dissimilar classes are separated in the embedding space.
IMAGINATOR performs well in several tasks, despite out our object
representation being a simple average of image embeddings. However,
contrastive learning might be able to learn even better vectors that capture
the relations between images and their objects.

### Vision Transformer and Positional Encoding

A Vision Transformer (ViT) is a transformer that is targeted at vision
processing tasks, such as object recognition and is much more robust than
CNNs. It divides an image into fixed-size patches, embeds each of them, and
includes a positional embedding along with the patch embedding as an input to
the transformer encoder. In our case, if we could draw meaningful cross-modal
connections between sections of text and the corresponding parts of images, a
significant performance uptick can be potentially reached. This can be
implemented using the various positional encoding schemes in ViT.

## References

  * Andrew et al. (2013) Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013.  Deep canonical correlation analysis.  In _International conference on machine learning_ , pages 1247–1255. PMLR.
  * Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018.  Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Baroni and Lenci (2011) Marco Baroni and Alessandro Lenci. 2011.  How we BLESSed distributional semantic evaluation.  In _Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics_ , pages 1–10, Edinburgh, UK. Association for Computational Linguistics.
  * Bhatia (2021) Gagan Bhatia. 2021.  keytotext.
  * Bruni et al. (2014) Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014.  Multimodal distributional semantics.  _J. Artif. Int. Res._ , 49(1):1–47.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , pages 104–120. Springer.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Finkelstein et al. (2001) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001.  Placing search in context: the concept revisited.  In _Proceedings of the Tenth International World Wide Web Conference, WWW 10, Hong Kong, China, May 1-5, 2001_ , pages 406–414. ACM.
  * Firth (1957) John Rupert Firth. 1957.  _"A synopsis of linguistic theory 1930-1955."_.  Oxford University Press.
  * Gunti et al. (2022) Nethra Gunti, Sathyanarayanan Ramamoorthy, Parth Patwa, and Amitava Das. 2022.  Memotion analysis through the lens of joint embedding (student abstract).  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(11):12959–12960.
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 2, pages 1735–1742. IEEE.
  * Halawi et al. (2012) Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012.  Large-scale learning of word relatedness with constraints.  In _Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’12, page 1406–1414, New York, NY, USA. Association for Computing Machinery.
  * Hardoon et al. (2004) David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004.  Canonical correlation analysis: An overview with application to learning methods.  _Neural computation_ , 16(12):2639–2664.
  * Harris (1954) Zellig Harris. 1954.  Distributional Structure.  _Word_ , 10:146–162.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  SimLex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Jastrzebski et al. (2017) Stanislaw Jastrzebski, Damian Lesniak, and Wojciech Marian Czarnecki. 2017.  How to evaluate word embeddings? on importance of data efficiency and simple supervised tasks.  _CoRR_ , abs/1702.02170.
  * Jha et al. (2021) Rishabh Jha, Varshith Kaki, Varuna Kolla, Shubham Bhagat, Parth Patwa, Amitava Das, and Santanu Pal. 2021.  Image2tweet: Datasets in Hindi and English for generating tweets from images.  In _Proceedings of the 18th International Conference on Natural Language Processing (ICON)_ , pages 670–676, National Institute of Technology Silchar, Silchar, India. NLP Association of India (NLPAI).
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR.
  * Jurgens et al. (2012) David Jurgens, Saif Mohammad, Peter Turney, and Keith Holyoak. 2012.  SemEval-2012 task 2: Measuring degrees of relational similarity.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 356–364, Montréal, Canada. Association for Computational Linguistics.
  * Kolluru (2019) Sethu Hareesh Kolluru. 2019.  A neural architecture to learn image-text joint embedding.  In _Stanford_.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_ , 123.
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Neural Word Embedding as Implicit Matrix Factorization.  In _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Levy et al. (2015) Omer Levy, Yoav Goldberg, and Ido Dagan. 2015.  Improving distributional similarity with lessons learned from word embeddings.  _Transactions of the association for computational linguistics_ , 3:211–225.
  * Li et al. (2022a) Fei-Fei Li, Marco Andreeto, Marc’Aurelio Ranzato, and Pietro Perona. 2022a.  Caltech 101.
  * Li et al. (2022b) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR.
  * Li et al. (2021) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Lu et al. (2018) Di Lu, Spencer Whitehead, Lifu Huang, Heng Ji, and Shih-Fu Chang. 2018.  Entity-aware image caption generation.  _arXiv preprint arXiv:1804.07889_.
  * Luo et al. (2018) Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. 2018.  Discriminability objective for training descriptive captions.  _arXiv preprint arXiv:1803.04376_.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  In _ICLR_.
  * Mikolov et al. (2013b) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b.  Linguistic regularities in continuous space word representations.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 746–751, Atlanta, Georgia. Association for Computational Linguistics.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Pilehvar et al. (2018) Mohammad Taher Pilehvar, Dimitri Kartsaklis, Victor Prokhorov, and Nigel Collier. 2018.  Card-660: Cambridge rare word dataset - a reliable benchmark for infrequent word representation models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1391–1401, Brussels, Belgium. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_.
  * Rubenstein et al. (1965) Herbert Rubenstein, John B. Goodenough, and John B. Goodenough. 1965.  Contextual correlates of synonymy.  _Commun. ACM_ , 8(10):627–633.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 815–823.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_.
  * Thompson (2000) Bruce Thompson. 2000.  Canonical correlation analysis.
  * Wang et al. (2022) Jianfeng Wang et al. 2022.  Recent advances in vision-and-language pre-training (tutorial).
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78.
  * Zeng et al. (2021) Yan Zeng, Xinsong Zhang, and Hang Li. 2021.  Multi-grained vision language pre-training: Aligning texts with visual concepts.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020.  Unified vision-language pre-training for image captioning and vqa.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 13041–13049.
  * Zhou et al. (2022) Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. 2022.  Detecting twenty-thousand classes using image-level supervision.  _arXiv preprint arXiv:2201.02605_.

## Frequently Asked Questions (FAQs)

  1. 1.

Doesn’t averaging individual object embeddings (or even word embeddings)
result in a noisy object embedding?

Ans. Yes, averaging individual embeddings is a limitation of this work and a
future avenue of exploration. On the other hand, concatenation is
computationally expensive. However, empirically, we found that averaging gave
better results than concatenation. We would like to explore autoencoding and
contrastive learning in the future as mitigation methods.

  2. 2.

Why does orthogonal projection work better than CCA-based methods?

Ans. Orthogonal projection is a discriminative method that attempts to find
out the discriminative projection for two vector spaces aligned to a unified
dimension. On the other hand, CCA tries to learn relations among two vector
spaces. While orthogonal projection offers competitive performance with a
limited number of classes, CCA is undoubtedly more powerful when the number of
classes is higher. In our case, since we only have 21K objects, orthogonal
projection yielded better results.

  3. 3.

Instead of directly learning a caption generation model based on the learned
joint embedding, this paper projects VGG-19 embeddings orthogonally in the
learned joint embedding space, using it to find the $k$ nearest objects in the
vector space, and then passes these objects through T5 for caption generation.
What is the motivation behind this approach?

Ans. Image object detection is a separate task altogether, and we are not
trying to solve that problem here. Given an image, we first get its VGG-19
embedding and then project it to IMAGINATOR space since VGG-19 and IMAGINATOR
have disparate embedding spaces and need to be aligned. A by-product of this
approach is that it also helps affirm that IMAGINATOR performs well, otherwise
it might raise doubts that the performance gain is happening due to T5
efficiency rather than IMAGINATOR. Lastly, we would like to draw the attention
of readers that the proposed captioning architecture is very simple and still
outperforms SoTA.

  4. 4.

Did you consider experimenting with ResNet or Fast-RCNN?

Ans. We performed experiments using ResNet, but the results were poor. One
plausible explanation is the fact that higher embedding dimensions lead to a
performance drop.

  5. 5.

Why was the Detic the baseline architecture of choice for IMAGINATOR?

Ans. The presumption of this work is to leverage the legacy of the NLP-centric
count-based vectorification methods for joint modality. Therefore, maximizing
the number of objects will give us a denser matrix to calculate the so-called
co-location. In the future, we plan to seek methods that can detect more than
21K objects, and strongly believe that will have a positive effect on the
learned joint embedding space.

## Appendix

### What is the value-addition of this work given Joint Embeddings have been
explored in various ways for the past few years?

Learning joint embeddings has been a topic that has received immense interest
from the multimodal AI community over the past decade Chen et al. (2020); Jia
et al. (2021); Tan and Bansal (2019). A concise survey on this topic has been
presented in Wang et al. (2022), which offers an extensive treatment of both
early (i.e., input-level), mid (i.e., feature-level), and late (i.e.,
decision-level) fusion methods, depicted visually in figure 8. Learning joint
embeddings using early-fusion methods (like the one we adopted in our work)
essentially enables identifying cross-correlations between various modalities
(such as text, images, video, audio, spatial/point-cloud information, etc.)
early on in the learning process. As such, the resultant vector
representations typically lead to top-notch performance in most downstream
tasks. On the other hand, a vast majority of work focuses on feature fusion
where modalities are first individually processed and then projected to a
common vector space to draw correlations using variety of projection methods
like CCA Thompson (2000), Deep CCA Andrew et al. (2013), etc.

As mentioned in Section 2, IMAGINATOR’s novelty is associated with the word-
level grounding of objects using traditional count-based approaches, an NLP
tradition that was prevalent before the neural era. This is a significant
detour from recent work in learning joint embeddings that uses deep learning-
based techniques, which suffer from a lack of control or ease of
interpretation owing to their inherent black-box nature. As such, this design
decision has allowed us to learn rich features that are co-location-based
representations of visual objects that are grounded in words which represent
the object’s moniker(s). The co-location-based contextual word vectorization
is primarily influenced by the distributional hypothesis "You shall know a
word by the company it keeps" Firth (1957). Intrigued by how such a co-
location-based method can aid visual contextual learning, we sought to testify
its utility in learning joint emebddings. However, the ability of count-based
joint embedding techniques can be severely limited due to the insufficient
number of objects detected, which led to us overcoming this issue by using
statistical correlational methods inspired by NLP (Levy et al. (2015). We plan
to further scale this technique by first enabling detection of additional
(>20K) visual objects, hypothesizing that this learning paradigm can lead to
even richer representations.

Figure 8: Notable recent work related to vision-language pre-training. Taken
from Wang et al. (2022).

### Rolodex of additional experiments carried out for the optimal generation
of $v_{oo}$, $v_{wo}$, and $v_{wor}$

Table 6 expands on section 3.2 and 3.3 and shows a comparison of different
embedding methods, dataset combinations, number of objects, loss functions,
and projection methods with performance on the captioning task. We consider
normalized count, PMI, PPMI, and Factorized PPMI for vector building and SVD
and Eigenvalue factorization for dimensionality reduction. For projection, we
consider Orthogonal Projection, CCA, regularized CCA, and Deep CCA for our
experiments. From Table 6, we can see that with an increase in the number of
objects, the captioning score also correspondingly increases.

Embedding method Dataset No. of objects Loss function Projection method
Performance Flickr30K COCO Normalized count Flickr30K 1000 Triplet loss
Orthogonal 32.1 29.3 Flickr30K + COCO 1080 Triplet loss Orthogonal 32.4 33.7
PMI Flickr30K + COCO 17000 Triplet Loss - 33.4 34.8 PPMI Flickr30K + COCO
17000 Triplet Loss - 33.9 35.4 Factorized PPMI Flickr30K + COCO 17000 Triplet
Loss - 34.1 37.9 Factorized PPMI + SVD Flickr30K + COCO 17000 Triplet Loss -
32.4 36.2 Factorized PPMI + Eigen Value Factorization Flickr30K 1000 Triplet
Loss CCA 30.2 32.2 Regularized CCA 30.9 33.9 Deep CCA 30.5 33.2 Flickr30K +
COCO 1080 Triplet loss Orthogonal 31.9 35.7 Flickr30K + COCO 17000 Triplet
loss Regularized CCA 33.2 40.1 Flickr30K + COCO 17000 Triplet loss Deep CCA
33.8 38.1

Table 6: Results on image captioning datasets (Flickr30K and COCO) for
different embedding methods, datasets, loss functions, and projection methods.

### Intrinsic evaluation of IMAGINATOR

The goal behind intrinsic evaluation is to understand how well the embeddings
adhere to the contextuality constraint. Building upon section 5, we consider
standard relational terms - king, queen, boy, woman and performed an intrinsic
evaluation on them to identify the relationships between these terms. We
project the joint embeddings of the image and check the Euclidean distance
among them; the lower the distance between similar terms, the better the
contextuality. Figure 9 shows additional intrinsic evaluation examples.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 9: Examples for intrinsic evaluation of our JEs based on image
similarity.

### Additional examples of image captioning - IMAGINATOR vs. Gold Caption vs.
OSCAR (SoTA)

(a)  IMAGINATOR: A bedroom with bedspreads, pillows, and a nightstand.
Gold Caption: the - bedroom stone cottage can sleep people.
OSCAR: A bedroom with a bed, dresser, and nightstand.

(b)  IMAGINATOR: A group of people are singing and clapping while a group of
musicians are performing.
Gold Caption: A band is playing in front of an audience and the singer is
wearing an orange shirt.
OSCAR: A man holding a baseball bat in front of a crowd.

(c)  IMAGINATOR: Photograph of a tall tower with steeples.
Gold Caption: sandcastle beach on bright sky.
OSCAR: A castle made of sand with a clock tower in the background.

(d)  IMAGINATOR: A photo of a console.
Gold Caption: The player staring intently at a computer screen.
OSCAR: A man sitting in front of a flat screen TV.

(e)  IMAGINATOR: A group of people playing ping-pong together.
Gold Caption: Young girls line up across each other and a ping-pong table in a
gymnasium while a few boys plan on a table further back.
OSCAR: A group of children playing a game of ping pong.

(f)  IMAGINATOR: "girls" and "boys" at a venue.
Gold Caption: party in the park under cherry blossoms.
OSCAR: A group of people sitting around a park with pink flowers.

(g)  IMAGINATOR: A young woman and man rowing a boat.
Gold Caption: A man and woman are on a gray and white rowboat.
OSCAR: Group of people on a small boat in the water.

(h)  IMAGINATOR: A kitchen with cabinets, cabinets, and a dishwasher.
Gold Caption: A kitchen with wooden cabinets and black appliances
OSCAR: A kitchen with a sink, dishwasher, stove and refrigerator.

(i)  IMAGINATOR: Chefs cooking with a stover and other cookware in a
laboratory.
Gold Caption: Two chefs in a restaurant kitchen preparing food .
OSCAR: Two men in a commercial kitchen preparing food.

Figure 10: Examples of some image captioning outputs generated by IMAGINATOR
along with the original caption and the caption generated by OSCAR Li et al.
(2020) for each respective image

## Examples of image retrieval - IMAGINATOR vs. SoTA: (i) ALBEF Li et al.
(2021), and (ii) XVLM Zeng et al. (2021)

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 11: Image retrieved for the query: "Two little children, one boy and
one girl laughing".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 12: Image retrieved for the query: "A dog is running in the sand".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 13: Image retrieved for the query: "Bride and groom walking side by
side".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 14: Image retrieved for the query: "Redhead woman in pig- tails and
glasses sewing on a sewing machine".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 15: Image retrieved for the query: "Smiling boy in white shirt and blue
jeans in front of rock wall with man in overalls behind him".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 16: Image retrieved for the query: "Two Asian or Spanish people, a
woman and a man, sitting together in front of a glass window as cars pass".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 17: Image retrieved for the query: "A little boy plays with a Nintendo
GameCube controller inside a McDonald’s".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 18: Image retrieved for the query: "A blonde woman wearing glasses and
a gray sweatshirt is cutting something with scissors".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 19: Image retrieved for the query: "A person wearing skis looking at
framed pictures set up in the snow".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 20: Image retrieved for the query: "A very young girl playing with a
bubble-blowing wand , holding a bottle of bubble solution and walking through
a park or field".

(a)  rank 1 image ALBEF Li et al. (2021)

(b)  rank 1 image XVLM Zeng et al. (2021)

(c)  rank 1 image BERTIMAGINATOR

Figure 21: Image retrieved for the query: "A woman in an outdoor marketplace,
wearing a large cone-shaped hat, standing behind two large baskets containing
loaves of bread".

### Image2Tweet examples - Gold vs. 5 ensemble SoTA Lu et al. (2018) vs.
$BERT_{IMAGINATOR}$

Image2tweet is a particularly hard problem to solve. It can involve social
engineering, web information scraping, face recognition, etc. The results in
table 4 show the current status of the problem and it needs substantial
research work to develop a solution. Figure 22 shows some Image2Tweet
examples.

(d)  Gold Caption: Should you wear a mask to protect yourself from
#coronavirus? #Coronavirus #COVID19
5 ensemble Luo et al. (2018): a group of surgeons prepare for surgery.
IMAGINATOR: people wearing masks during the pandemic.

(e)  Gold Caption: Donald Trump’s India visit will be beneficial for both the
countries.
5 ensemble Luo et al. (2018): politician shakes hands with politician during a
bilateral meeting.
IMAGINATOR: Two men are handshaking with an Indian flag in the background.

(f)  Gold Caption: I am here to play cricket not gimmick - @PrithviShaw to
press.
5 ensemble Luo et al. (2018): cricket player during a press conference.
IMAGINATOR: A man in a press conference.

(g)  Gold Caption: JEE (Main) begins today - students are following protocols
- queue, social distancing, masks.
5 ensemble Luo et al. (2018): students wearing face masks during a protest.
IMAGINATOR: young girls wearing masks in a queue.

(h)  Gold Caption: Country needs so many doctors than politicians - pandemic
realization.
5 ensemble Luo et al. (2018): person, left, and person, right, are both
members of the team.
IMAGINATOR: Two doctors with face shields.

(i)  Gold Caption: 5G tech is picking up pace and expectations are high, but
rollout is still years away in India.
5 ensemble Luo et al. (2018): the logo on a background of a blue sky with
clouds.
IMAGINATOR: 5G logo.

(j)  Gold Caption: SC refuses to entertain plea against Madras HC order on
Patanjali’s use of ’Coronil’.
5 ensemble Luo et al. (2018): a gothic buildiing.
IMAGINATOR: supreme court of India building.

(k)  Gold Caption: No rugby for world champion as South Africa maintains ban.
5 ensemble Luo et al. (2018): rugby player looks dejected after defeat
IMAGINATOR: A scene of a rugby match with three players visible.

(l)  Gold Caption: I love India, but Indians don’t like me.
5 ensemble Luo et al. (2018): politician addresses a crowd of supporters.
IMAGINATOR: An angry politician delivering a speech.

(m)  Gold Caption: Indian prime minister addressing to the nation in his own
man ki baat.
5 ensemble Luo et al. (2018): politician making a speech at a function.
IMAGINATOR: Modi is delivering a speech on camera.

(n)  Gold Caption: Kamala Harris bringing energy, dollars and more to Joe
Biden’s campaign.
5 ensemble Luo et al. (2018): politician gives a speech during the second day.
IMAGINATOR: Harris making promises.

(o)  Gold Caption: US Presidential election: Hillary-Tulsi spat scorches
Democratic Party.
5 ensemble Luo et al. (2018): Two politicians are debating.
IMAGINATOR: Hillary Clinton and another woman in white dress.

Figure 22: Examples of Image2Tweet task - gold vs IMAGINATOR vs 5 ensemble
SoTA Li et al. (2020)


# Impact of quantum noise on the training of quantum Generative Adversarial
Networks

Kerstin Borras1,2 Su Yeon Chang3,4 Lena Funcke5 Michele Grossi3 Tobias
Hartung6,7 Karl Jansen1 Dirk Kruecker1 Stefan Kühn6 Florian Rehm3,2 Cenk
Tüysüz1,8 and Sofia Vallecorsa3 1 Deutsches Elektronen-Synchrotron DESY,
Platanenallee 6, 15738 Zeuthen, Germany  2 RWTH Aachen University,
Templergraben 55, 52062 Aachen, Germany  3 CERN, 1211 Geneva 23, Switzerland
4 Institute of Physics, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015
Lausanne, Switzerland  5 Center for Theoretical Physics, Co-Design Center for
Quantum Advantage, and NSF AI Institute for Artificial Intelligence and
Fundamental Interactions, Massachusetts Institute of Technology, 77
Massachusetts Avenue, Cambridge, MA 02139, USA  6 Computation-Based Science
and Technology Research Center, The Cyprus Institute, 20 Kavafi Street, 2121
Nicosia, Cyprus  7 Department of Mathematical Sciences, 4 West, University of
Bath, Claverton Down, Bath BA2 7AY, UK  8 Institüt für Physik, Humboldt-
Universität zu Berlin, Newtonstr. 15, 12489 Berlin, Germany
<EMAIL_ADDRESS>
Preprint number: MIT-CTP/5400

###### Abstract

Current noisy intermediate-scale quantum devices suffer from various sources
of intrinsic quantum noise. Overcoming the effects of noise is a major
challenge, for which different error mitigation and error correction
techniques have been proposed. In this paper, we conduct a first study of the
performance of quantum Generative Adversarial Networks (qGANs) in the presence
of different types of quantum noise, focusing on a simplified use case in
high-energy physics. In particular, we explore the effects of readout and two-
qubit gate errors on the qGAN training process. Simulating a noisy quantum
device classically with IBM’s Qiskit framework, we examine the threshold of
error rates up to which a reliable training is possible. In addition, we
investigate the importance of various hyperparameters for the training process
in the presence of different error rates, and we explore the impact of readout
error mitigation on the results.

##  1 Introduction

In classical deep learning, an extensive amount of studies have proven that
noise plays a crucial role for the training of neural networks. Artificial
noise injection is an efficient regularization method to speed up convergence
and to improve the stability of the training process [1, 2].

Meanwhile, quantum computing, which is a completely new paradigm of
computation, is characterized by statistical uncertainty from its
probabilistic nature. Furthermore, on current and near-term quantum hardware
one encounters the challenge to overcome the noise due to gate errors, readout
errors, and interactions with the environment [3]. The presence of this
intrinsic quantum noise suggests the possibility of replacing the artificial
noise in the context of classical machine learning with the noise of the
quantum hardware.

In this paper, we study the impact of quantum noise on Quantum Machine
Learning (QML), more specifically on qGANs, which are the quantum analog of
classical GANs. Focusing on a simplified high-energy physics (HEP) use case,
we first explore the impact of readout noise on the qGAN model by classically
simulating a noisy quantum device with IBM’s Qiskit software development kit
[4]. In particular, we study the importance of several training
hyperparameters at various levels of noise. We also apply different readout
mitigation techniques and study their impact on the performance of qGAN
training. Finally, we assess the performance of qGAN training in the presence
of the full noise model, including two-qubit gate errors. This work provides
broad insights into the impact of noise on QML, which is particularly relevant
for applying current noisy intermediate-scale quantum (NISQ) devices [3] to
QML.

##  2 Quantum Generative Adversarial Networks

First, we briefly review the qGAN model introduced in Ref. [5], which is a
hybrid quantum-classical algorithm illustrated in Fig. 1(a). This model
consists of a classical discriminator and a quantum generator applied to $n$
qubits. The generator is a variational quantum circuit, which consists of
alternating layers of single-qubit rotation gates and two-qubit entangling
gates. In addition, an initial layer of Hadamard gates provides an equal-
weight superposition of all computational basis states. By comparing the qGAN
performance for different gate choices, we found optimal performance for
$R_{Y}(\theta_{i})$ rotation gates and controlled-$Z$ entangling gates.

(a)

(b)

Figure 1: (a) Schematic diagram of the qGAN model, (b) exemplary results of
the trained qGAN model applied to a simplified HEP problem using a statevector
simulator.

The quantum generator is parameterized by the angles $\theta_{i}$ of the
$R_{Y}(\theta_{i})$ gates, while the classical discriminator is parameterized
by the neural network parameters $\phi_{i}$. During training, the generator
and discriminator are optimized alternately. After training, the generator
reproduces a probability distribution $Q(x)$ in the computational basis
$\ket{x}\in\\{\ket{0},...,\ket{2^{n}-1}\\}$, which approximates the target
distribution $P_{\rm target}$. The input for the discriminator contains both
real data, which are continuous scalars following the distribution $P_{\rm
target}$, and fake data, which are discrete integers between $0$ and
$2^{n}-1$, chosen by an affine mapping between the computationalbasis states
produced by the generator and an equidistant grid over the real data.

In this paper, we use the qGAN model to reproduce electron energy profiles in
HEP calorimeters. To simplify the problem, we sum up the energy distribution
along the longitudinal profile, which contains the most essential information
to classify the incoming particle. We average over the training set, binning
the resulting probability distribution into $2^{n}=N$ pixels. Figure 1(b)
displays the result of an exemplary qGAN training with $n=3$ qubits using a
statevector simulator, in the absence of statistical fluctuations and quantum
noise.

##  3 Quantum Noise Study

In order to study the impact of quantum noise on the training process, we use
a qGAN model with a quantum generator consisting of $n=3$ qubits and $2$
learning layers. Readout errors dominate over two-qubit gate errors for
shallow circuits; thus, we choose to focus on the former. We assume that
readout errors, i.e., misidentifying a measurement outcome as $1$ given it was
$0$ and vice versa, are uncorrelated between different qubits and that the
bit-flip probability $p$ for misidentifying the measurement outcome is the
same for each qubit. For the following simulations, we evaluate the
performance of the qGAN model using the relative entropy

$D_{\rm KL}(P||Q)=\sum_{x=0}^{N-1}P(x)\log\frac{P(x)}{Q(x)},$ (1)

also called the Kullback-Leibler (KL) divergence. Here, $Q(x)$ is the output
distribution of the quantum generator and $P(x)$ is the discretized version of
the target distribution $P_{\rm target}$.

###  3.1 Hyperparmeter Scan

In this section, we study different values of the bit-flip probability,
$p=\\{0.01,0.05,0.1\\}$, as well as different subsets of hyperparameters: the
generator learning rate $lr_{g}$, the discriminator learning rate $lr_{d}$,
and the the exponential decay rate $\gamma$ for the learning layers. In Fig.
2, we plot our results for the relative entropy using the best (orange) and
worst (blue) choices of hyperparameters, as well as an average over all runs
(green). For an appropriate choice of hyperparameters, the qGAN model
converges for all choices of $p$. However, the relative entropy increases with
a higher noise level, which cannot be fully mitigated even when choosing
optimal hyperparameters.

(a) $p=0.01$

(b) $p=0.05$

(c) $p=0.1$

Figure 2: Relative entropy defined in Eq. (1) as a function of training
epochs, shown for different hyperparameter choices and for different bit-flip
probabilities $p$.  Figure 3: Importance of hyperparameters for different
strengths of the readout error.

To examine which of the three hyperparameters has the strongest impact on the
training for different noise levels, we display the so-called importance value
of the hyperparameters for the objective value in Fig. 3, which we calculated
based on the fANOVA hyperparameter importance evaluation algorithm [6]. We
find that the generator learning rate $lr_{g}$ has the highest impact on the
training, which demonstrates the difficulty of training the quantum generator.
Thus, as expected, the impact of $lr_{g}$ grows as the bit-flip probability
increases.

###  3.2 Instability of qGAN Training

Instabilities, which manifest as large fluctuations of the relative entropy
when choosing different initial training parameters, are one of the major
challenges that have to be overcome in GAN training, both in the classical and
the quantum case. In this section, we investigate the instability of qGAN
training in presence of readout noise, using the optimal hyperparameters found
in Section 3.1. The initial training parameters $\theta_{i}$ of the quantum
generator circuit are sampled randomly from a normal distribution with a mean
of 0.1 and a standard deviation of 0.2.

(a) $p=0.01$

(b) $p=0.05$

(c) $p=0.1$

Figure 4: Relative entropy defined in Eq. (1) as a function of training
epochs, averaged over different numbers of repetitions $n_{\rm rep}$, shown
for different bit-flip probabilities $p$.

Figure 4 summarizes the results of the qGAN training for different numbers of
repetitions, $n_{\rm rep}=20$ (blue) and $n_{\rm rep}=100$ (orange), and for
different readout-noise levels. Interestingly, for all values of $p$, both the
average relative entropy and the standard deviation do not substantially
change as we increase the statistics by increasing $n_{\rm rep}$. Thus,
although the model converges in a stable manner on the ensemble of
simulations, individual unstable runs occur with a constant probability. These
results demonstrate that training instabilities occur independently from the
readout-error level and cannot be alleviated by quantum noise or increasing
statistics.

###  3.3 Error Mitigation

In the following, we study two different methods to mitigate readout errors:
conventional bit-flip (BF) mitigation [7] and independent bit-flip (IBF)
mitigation [8]. These methods rely on a calibration matrix ($M$), which maps
the measured probability distribution to the expected probability distribution
($P_{\rm noisy}=MP_{\rm ideal}$). In this way, the inverse of $M$ can be used
to obtain the expected outcome from measured probabilities. The matrix $M$ for
a single qubit reads

$M=\begin{bmatrix}1-p_{01}&p_{10}\\\ p_{01}&1-p_{10}\end{bmatrix},$ (2)

where $p_{01}$ ($p_{10}$) is the bit-flip probability of misidentifying 0 as 1
(1 as 0) during readout.

For the conventional BF method [7], the computational resources required to
obtain the inverse of $M$ grow exponentially with respect to the number of
qubits ($M$ is a $2^{n}\times 2^{n}$ matrix). In contrast, the IBF method [8]
assumes that the single-qubit matrix $M$ in Eq. (2) can be extended to
multiple qubits with a tensor product over all qubits ($M=M_{1}\otimes
M_{2}\otimes\cdots\otimes M_{N}$). Thus, this method assumes uncorrelated
multi-qubit readout errors, which is an assumption that has recently been
verified experimentally [9]. This way, computing the inverse of $M$ can be
done by computing the inverse of $n$ many $2\times 2$ matrices using
polynomial resources.

Correct estimation of the bit-flip probabilities is essential for successful
error mitigation. Therefore, one needs to execute calibration circuits on the
quantum device to obtain these values. The precise estimation of these values
might be limited by their time dependence and by the finite sampling size
(shot noise), e.g., $\mathcal{O}(1/\epsilon^{2})$ shots are required for
$\epsilon$ precision.

(a)

(b)

Figure 5: (a) Final relative entropy after qGAN training in the presence of
readout noise, as a function of the bit-flip probability $p=p_{01}=p_{10}$,
averaged over $n_{\rm rep}=20$ repetitions, shown with and without error
mitigation. (b) Relative entropy as a function of training epochs, shown with
different types of noise, with and without readout-error mitigation.

In Fig. 5(a), we examine the impact of readout errors and error mitigation on
the final relative entropy after qGAN training, comparing the cases of no
noise (black), readout noise (orange), IBF mitigation (blue), and BF
mitigation (green). When focusing on the average value of the relative
entropy, the results for training with a small readout error ($p<0.06$) seem
to outperform the results without noise, which aligns with our initial
suggestion of enhancing the qGAN training with quantum noise. However, within
the statistical uncertainty, the noisy results for $p<0.06$ are compatible
with both the noise-free and the error-mitigated results. For larger levels of
readout error ($p\geq 0.06$), we observe that the noisy results become worse
compared to the noise-free and error-mitigated cases. This implies that error
mitigation plays a crucial role for qGAN training in the presence of the large
readout errors on current NISQ devices.

When comparing the two different error mitigation methods in Fig. 5(a), we
find that the resulting values for the relative entropy agree within the
statistical uncertainty, as expected. For small values of $p$, we find a small
deviation between the mean values for the relative entropy, which is most
likely caused by using exact probabilities to estimate the calibration matrix
$M$ in the case of the IBF method and using a finite number of samples (3000
shots) in the case of the BF method. In the latter case, the finite number of
samples results in a precision error of $\epsilon\sim\mathcal{O}(0.02)$ for
estimating $p=p_{01}=p_{10}$, in agreement with our results in Fig. 5(a).

Finally, we study the impact of both readout and two-qubit gate errors on the
qGAN training. In Fig. 5(b), we compare the results for the relative entropy
(orange) to the results for the no-noise case (black), the readout-error-only
case (blue), and the case of BF-mitigated readout errors and unmitigated gate
errors (green). For this comparison, we consider error rates of 2.5% for the
readout error and 1.5% for the gate error. As shown in Fig. 5(b), new
optimized hyperparameters are found to reduce the number of training epochs to
only 300 epochs until reaching convergence. Furthermore, a new training schema
is adapted, by updating the quantum generator once and the classical
discriminator 10 times at each step. Similar to our findings in Fig. 5(a), the
inclusion of quantum noise does not improve the results for the relative
entropy, as the results for the noise-free, noisy, and error-mitigated cases
agree within the statistical uncertainty. Interestingly, the average values of
the relative entropy quickly converge in the noise-free and readout-error-only
cases, but do not seem to converge when including two-qubit gate errors. These
findings reveal the critical effect of two-qubit gate errors on qGAN training
with current NISQ devices.

##  4 Conclusion

In this paper, we studied the impact of quantum noise on qGAN training,
focusing on a simplified use case in high-energy physics. We first
investigated different levels of readout noise and demonstrated that the qGAN
model can be successfully trained even in the presence of this noise, although
the common intrinsic instability of qGAN training cannot be overcome. We then
examined the impact of different hyperparameters on the training and found
that the learning rate of the quantum generator has the strongest impact,
which becomes more pronounced with increasing readout-error rates. We also
demonstrated that readout-error probabilities smaller than $p\sim 6\%$ do not
significantly impact the qGAN training, while slightly larger readout errors
still yield training convergence but substantially worse training results. To
enhance these results, we studied the impact of two different readout-error
mitigation methods, which both improved the qGAN training to a performance
similar to the noise-free case. For two-qubit gate errors, we found that their
effect on qGAN training is even more pronounced, preventing the convergence of
training already for small error rates of $p\sim 1.5\%$. These results
demonstrate that the mitigation of readout and gate errors plays a crucial
role for qGAN training on current NISQ devices.

In future work, we will extend our current study to qGAN training on real
quantum hardware. Moreover, we will investigate mitigation techniques to
tackle the common issue of training instabilities for qGAN models. Finally, we
are planning to study the impact of quantum noise and error mitigation methods
on other QML models beyond qGANs.

This project is supported by CERN’s Quantum Technology Initiative. L.F. is
supported by the U.S. Department of Energy, Office of Science, National
Quantum Information Science Research Centers, Co-design Center for Quantum
Advantage (C2QA) under contract number DE-SC0012704, by the DOE QuantiSED
Consortium under subcontract number 675352, by the National Science Foundation
under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial
Intelligence and Fundamental Interactions, http://iaifi.org/), and by the U.S.
Department of Energy, Office of Science, Office of Nuclear Physics under grant
contract numbers DE-SC0011090 and DE-SC0021006. S.K. acknowledges financial
support from the Cyprus Research and Innovation Foundation under project
“Future-proofing Scientific Applications for the Supercomputers of Tomorrow
(FAST)”, contract no. COMPLEMENTARY/0916/0048.

## References

## References

  * [1] C. M. Bishop.  Training with Noise is Equivalent to Tikhonov Regularization.  Neural Comput., 7:108, 1995.
  * [2] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.  Analyzing and Improving the Image Quality of StyleGAN.  CVPR 2020, 2020.
  * [3] J. Preskill.  Quantum Computing in the NISQ era and beyond.  Quantum, 2:79, 2018.
  * [4] H. Abraham et al.  Qiskit: An open-source framework for quantum computing.  Zenodo, 2019.
  * [5] C. Zoufal, A. Lucchi, and S. Woerner.  Quantum Generative Adversarial Networks for learning and loading random distributions.  npj Quantum Inf., 5:103, 2019.
  * [6] F. Hutter, H. Hoos, and K. Leyton-Brown.  An Efficient Approach for Assessing Hyperparameter Importance.  In Proceedings of the 31st International Conference on Machine Learning, Bejing, China, 2014.
  * [7] Qiskit documentation and source code.  Accessed: 09-02-2022.
  * [8] L. Funcke, T. Hartung, K. Jansen, S. Kühn, P. Stornati, and X. Wang.  Measurement Error Mitigation in Quantum Computers Through Classical Bit-Flip Correction.  2020\.  arXiv:2007.03663.
  * [9] C. Alexandrou, L. Funcke, T. Hartung, K. Jansen, S. Kühn, G. Polykratis, P. Stornati, and X. Wang.  Using classical bit-flip correction for error mitigation including 2-qubit correlations.  2021\.  arXiv:2111.08551.