# Composing Object Relations and Attributes for Image-Text Matching

Khoi Pham1 Chuong Huynh1 Ser-Nam Lim2 Abhinav Shrivastava1
1University of Maryland, College Park 2University of Central Florida

###### Abstract

We study the visual semantic embedding problem for image-text matching. Most
existing work utilizes a tailored cross-attention mechanism to perform local
alignment across the two image and text modalities. This is computationally
expensive, even though it is more powerful than the unimodal dual-encoder
approach. This work introduces a dual-encoder image-text matching model,
leveraging a scene graph to represent captions with nodes for objects and
attributes interconnected by relational edges. Utilizing a graph attention
network, our model efficiently encodes object-attribute and object-object
semantic relations, resulting in a robust and fast-performing system.
Representing caption as a scene graph offers the ability to utilize the strong
relational inductive bias of graph neural networks to learn object-attribute
and object-object relations effectively. To train the model, we propose losses
that align the image and caption both at the holistic level (image-caption)
and the local level (image-object entity), which we show is key to the success
of the model. Our model is termed Composition model for Object Relations and
Attributes, CORA. Experimental results on two prominent image-text retrieval
benchmarks, Flickr30K and MS-COCO, demonstrate that CORA outperforms existing
state-of-the-art computationally expensive cross-attention methods regarding
recall score while achieving fast computation speed of the dual encoder. Our
code is available at https://github.com/vkhoi/cora_cvpr24

##  1 Introduction

Figure 1: Illustration of CORA. CORA has a dual-encoder architecture,
consisting of one encoder that embeds the input image and one encoder that
embeds the text caption scene graph into a joint embedding space. (Best viewed
in color and zoomed in.)

Image-text matching is a fundamental computer vision problem that aims to
measure the semantic correspondence between an image and a text. Such
correspondence can be used for image retrieval given a text description, or
text retrieval provided an image query, both of which are important in various
computer vision applications (_e.g_., weakly supervised problems [18, 19]).
The problem is inherently challenging due to the ambiguous nature of the image
and text modalities [46, 6]. For example, an image can depict a complicated
situation that a multitude of different captions can describe, whereas a
single caption is too abstract and can semantically apply to multiple images.
Various studies have been proposed and can be categorized into two main
directions: (1) the unimodal dual encoder and (2) the cross-attention
approach.

In the dual-encoder framework, two modality-independent encoders embed the
image and text caption separately into a joint embedding space. In this space,
a similarity function such as a dot product can measure the image-text
similarity. This strategy is also referred to as the global alignment
approach, as the goal is to holistically represent an image (or text) as a
single embedding. Due to their simplicity and low computational cost (_e.g_.,
retrieving an image given a text query can be done via a vector-matrix
multiplication with the cached embeddings), such methods are more widely
adopted for real-world retrieval databases.

The second approach, cross-attention network, constitutes the majority of
recent work. Instead of embedding each modality separately, cross-modality
attention is adopted to locally align fine-grained visual cues of an image
(image regions) with textual cues of a caption (word tokens), from which the
overall correspondence score is aggregated. While this approach outperforms
dual encoder in terms of power, it presents a substantial computational
challenge. Upon receiving a text (or image) query, every image vs. text query
pair must be processed through the cross-attention model to determine their
similarity scores. This requirement renders the method impractical for
retrieval systems managing large databases due to its extensive computational
demands. This work focuses on the dual-encoder approach and shows that our
dual-encoder proposal even outperforms the SOTA cross-attention networks.

Existing approaches use a text sequence model (_e.g_., GRU [7], LSTM [15]) to
encode the text caption. A text usually contains an extensive range of
semantic information, such as object categories, attributes of objects, and
relations between objects. Attributes describe appearance of objects [36, 22,
38, 44, 39], while relations describe how objects interact with one another
[56]. Forcing a text sequence model to learn to parse a caption into different
levels of semantics is challenging, especially in the low data regime. For
example, by design, a sequence model that simply processes a caption from left
to right (GRU, LSTM) may find it challenging to determine which attributes
belong to an object and which objects participate in a relation. Numerous
works have shown that Transformer-based text sequence models (BERT [8]) can
produce good structural parsing of a sentence [14], however, these models must
be trained on large amounts of data. Nevertheless, it has been shown in [3]
that even the CLIP text encoder [42] in Stable Diffusion [43, 40] still
exhibits incorrect object-attribute binding (_i.e_., pair an attribute with
the wrong object in the sentence) despite having been trained on large
datasets. Therefore, it becomes desirable to have a text embedding model that
can capture the semantic relations between concepts accurately.

In this work, instead of a sequence model, we propose representing a caption
as a scene graph of object and attribute nodes connected by relation edges. An
example of a scene graph is illustrated in Fig. 1, where we show that semantic
structures such as object-attribute and object-object pairings are already
organized. To this end, we propose our Composition model for Object Relations
and Attributes, CORA, a dual-encoder model for image-text matching. On the
image side, we re-use GPO [4] which is a SOTA pooling operator for image-text
matching to embed the image as a vector. On the text side, we propose to use a
graph attention network [48, 2] with strong relational inductive bias to
produce a holistic scene graph embedding for the caption. Scene graph-based
approaches have been previously explored in [51, 25, 28, 30] for image-text
matching, but they all employ expensive cross-attention. In addition to the
margin-based triplet ranking loss [10] adopted by prior work, we propose a
contrastive loss to guide CORA in making alignment at both the holistic image-
caption level and the local image-object entity level. The proposed loss helps
make training more stable and result in better downstream retrieval accuracy,
as well as additionally acquires CORA with the image-object entity retrieval
capability.

Our model is evaluated on two image-text retrieval benchmarks, Flickr30K and
MS-COCO, where it outperforms SOTA dual-encoder and expensive cross-attention
methods. Our paper makes the following contributions:

  * •

We propose CORA, a dual encoder for image-text matching that uses a graph
attention network instead of a sequence model to produce scene graph embedding
for a caption.

  * •

We propose using contrastive loss that trains the model to make global
alignment (image-caption) and local alignment (image-object entity), resulting
in more stable training, better retrieval accuracy, and image-object retrieval
capability.

  * •

Our model CORA achieves SOTA retrieval performance on Flickr30K and MS-COCO,
two prominent benchmarks for image-text retrieval.

##  2 Related Work

Dual-encoder. This approach is dominant in earlier works [11, 21, 50, 10, 24]
in image-text matching. The image and text captions are independently embedded
in a joint metric space where matching image-caption pairs are located close
to each other. Existing work in this paradigm often improves the joint
embedding space by introducing new losses [10, 6], proposing new architecture
for each modality encoder [24, 52, 54], or learning better pooling methods [4,
26]. For example, VSE++ [10] proposes a triplet loss with hard negative mining
which has been adopted by all following image-text matching work. VSRN [24],
DSRAN [52], SAEM [54] implement graph convolution and self-attention to
improve the encoder architecture. GPO [4] achieves competitive results by
designing a new pooling operator that can learn from data. Recently, MV-VSE
[26] and SDE [20] propose using multiple embeddings per sample data, and HREM
[12] presents a dual-encoder model that can be trained with a cross-modality
matching loss for enhancing the embedding quality.

Cross-attention. In contrast to embedding the image and text independently,
this approach considers the fine-grained local correspondence between image
features and text tokens before computing the similarity. SCAN [23] is the
first representative work that introduces this idea of using cross-attention
between the two modalities to find their alignments. CAAN [58] later improves
the idea by employing an additional intra-modal interaction step after the
cross-modal interaction. SGARF [9] proposes to learn jointly from both the
global and local alignment to highlight important image regions. Recently,
NAAF [57] encourages the dissimilarity degrees between mismatched pairs of
image region and word to boost the similarity matching, and CHAN [35] proposes
a new cross-modal alignment method that can neglect the redundant
misalignments.

Graph-based image-text matching. Among both dual-encoder and cross-attention
methods, some have utilized scene graphs as part of their pipeline for more
accurate image-text alignment [51, 25, 28, 30]. Frameworks based on this
approach leverage the capacity of Graph Convolutional Networks (GCN) to
capture the spatial and semantic relationships between visual regions and
textual tokens. For example, SGM [51], GCN+DIST [25], GraDual [30] utilize
off-the-shelf visual scene graph generator [56] to extract scene graph from
images, then perform cross-modal alignment between the visual and textual
graph. GSMN [28], on the other hand, uses a fully connected graph for the
visual regions but additionally uses the regions’ polar coordinates to encode
their spatial relationships.

In our work, we build upon the scene graph representation of the caption to
develop the text encoder for our dual-encoder model. Our model focuses on
explicitly learning to compose objects with their attributes and all objects
in the scene through their relationships to produce a single embedding vector
for the text rich in semantic information. To the best of our knowledge, there
has yet to be any previous dual-encoder work on explicitly capturing the
object, attribute, and relation semantics through scene graphs for image-text
matching. Our method is different from previous graph-based approaches in that
we do not use external visual scene graph generator, which is prone to wrong
prediction, and we carefully design a 2-step graph encoding approach trained
with a contrastive loss to align at both the global image-text and local
image-object level. Our network outperforms SOTA methods without the heavy
cross-attention module.

##  3 Method

This section describes our Composition model for Object Relations and
Attributes. We first describe the overall framework in Sec. 3.1, then present
in Sec. 3.2 how we perform visual embedding on the input image, how we parse
the text caption into a scene graph and extract text features for each node in
the graph. In Sec. 3.3, we describe how we can embed this scene graph into the
joint embedding space with the image using the graph attention network.
Finally, training objectives are detailed in Sec. 3.4.

###  3.1 Overall Framework

We begin by describing the overall framework of CORA, which is illustrated in
Fig. 2. The model consists of two encoders: a visual encoder $f^{\mathcal{V}}$
that takes in an input image $\mathbf{x}$ and produces the image embedding
vector $v=f^{\mathcal{V}}(\mathbf{x})\in\mathbb{R}^{D}$; and a text encoder
$f^{\mathcal{T}}$ that takes in the text caption $\mathbf{y}$ and produces its
embedding $t=f^{\mathcal{T}}(\mathbf{y})\in\mathbb{R}^{D}$ in the joint
$D$-dimensional embedding space. Instead of embedding the text caption
directly, we first parse it into a scene graph using a parser
$\phi^{\text{SG}}$, then apply a graph attention network $f^{\mathcal{G}}$ to
embed this scene graph. Our text embedding formulation therefore can be
rewritten as $t=f^{\mathcal{G}}(\phi^{\text{SG}}(\mathbf{y}))$.

Figure 2: Overview of CORA. a) CORA consists of (1) an image encoder that
detects and extracts the salient regions’ features from the input image,
contextualizes them through a multi-head self-attention, then aggregates them
into a single image embedding through the GPO [4] pooling operator, (2) a text
encoder that first parses the input text into a scene graph where all semantic
information is readily organized, then two graph attention networks Object-
Attribute GAT and Object-Object GAT are used to encode this graph into the
same joint space with the image. The red arrow denotes the edge of the active
role, while the yellow arrow is for the passive role in the relation (refer to
Sec. 3.3.2). b) The semantic concept encoder that uses GRU or BERT to encode
each semantic concept in the graph corresponding to the object, attribute
nodes and relation edges.

The similarity score between the image and the text caption is defined as the
cosine similarity between their embeddings $v$ and $t$:

$\text{sim}\mathbf{(x,y)}=\frac{v^{\text{T}}t}{\|v\|\|t\|}.$ (1)

The dual-encoder is efficient for image-text retrieval. In the context of
image retrieval, all image embeddings can be computed and cached in advance.
When a text query arrives, it only needs to be embedded with
$f^{\mathcal{G}}(\phi^{\text{SG}}(.))$, then a simple vector-matrix
multiplication is sufficient to retrieve all nearest neighbor images of the
query.

###  3.2 Feature Extraction

Visual feature extractor. Given an input image $\mathbf{x}$, we follow
convention from prior work to use the pre-trained bottom-up detection model
BUTD [1]. With this model, the top-36 most confident salient regions in
$\mathbf{x}$ are detected, along with their visual features
$\\{x_{k}\in\mathbb{R}^{2048}\\}_{k=1}^{N_{\mathcal{V}}},N_{\mathcal{V}}=36$.
The detection model used here is a Faster R-CNN with ResNet-101 backbone [13],
pre-trained on Visual Genome [22]. We also transform the region features with
an FC layer so that they have the same dimensions as the joint embedding
space: $x_{k}\in\mathbb{R}^{D}$. Furthermore, we also apply multi-head self-
attention to contextualize the region features against one another. Then, in
order to perform feature aggregation on this set to obtain a holistic
representation for the input image
$v=f^{\mathcal{V}}(\mathbf{x})\in\mathbb{R}^{D}$, we implement
$f^{\mathcal{V}}$ using GPO [4] which is a SOTA pooling operator for image-
text matching. Essentially, GPO learns to generate the best pooling
coefficient for every visual region, which is better than naively applying
mean pooling over the visual feature set.

Scene graph parser. Formally, we implement a textual scene graph parser that
can construct a graph $G=(V,E)$ given a text caption $\mathbf{y}$, where
$V=O\cup A$ denotes the set of object nodes $O$ and attribute nodes $A$, and
$E=E_{\text{OA}}\cup E_{\text{OO}}$ represents the set of object-attribute
edges $E_{\text{OA}}$ and object-object relation edges $E_{\text{OO}}$.
Example of a scene graph is illustrated in Fig. 1. We implement a scene graph
parser based on [45, 53], using the syntactical dependency parser from the
spaCy library [16]. We develop rules to extract object nouns (_e.g_.,
construction worker), adjective and verb attributes (_e.g_., salmon-colored,
sitting), verb relations (_e.g_., person-jump over-fence, dog-wear-costume),
and preposition relations (_e.g_., flag-above-building). Existing scene graph
parsers [45, 53] are developed upon inferior language toolkits, thus often
misdetect concepts (_e.g_., those consisting of multiple word tokens are not
detected). The implementation of our parser is made publicly available.

Semantic concept encoder. We denote the set of object nodes $O=\\{o_{i}\\}$,
attribute nodes $A=\\{a_{i}\\}$, and object-object relation edges
$E_{\text{OO}}=\\{r_{ij}\\}$. These concepts are still in text format that
need to be encoded into vector representation. As these concepts often consist
of multiple word tokens (_e.g_., pair of shoes, jump over), we use a text
sequence model as a phrase encoder to encode all semantic concepts. To
demonstrate the generalizability of our method across different language
features, we implement this semantic concept encoder using Bi-GRU [7] and BERT
[8]. For Bi-GRU, given an $L$-word semantic concept, we use the GloVe [37]
word embedding of each word to obtain a sequence of $L$ $300$-dimensional
vectors. Next, we employ a Bi-GRU and take the final hidden states as the
representation for the concept $c\in\mathbb{R}^{300}$. For BERT, we use the
average of the output hidden states of all tokens at the last layer to
represent the concept $c\in\mathbb{R}^{768}$. For both types of features, we
then use an FC layer to transform the concept embedding to have the same
dimension $D$ as the joint embedding space. These concept embeddings are used
to initialize the node features for $\\{o_{i}\\}$ and $\\{a_{i}\\}$ and the
edge features for $\\{r_{ij}\\}$ in the scene graph.

###  3.3 Scene Graph Embedding

After obtaining the graph structure from the parser and the initialized
features for all nodes and edges in the graph, we continue to elaborate on our
scene graph embedding method as follows. The core idea of our method is that
the scene semantics should be composed at two levels in a bottom-up manner,
where we use a separate graph attention network (GAT) [48, 2] for each level.
At the bottom level, a GAT models the relations between an object and its
associated attributes. At the top level, another GAT is used to model the
relations between solely the objects, compose them together and produce the
final scene embedding.

GAT Preliminaries. GAT is among the most popular graph neural network methods,
with SOTA results in graph representation learning. We follow the
implementation of GATv2 [2], which is an improved version of the original GAT
[48]. We provide a brief description of GATv2 here. Given a directed graph
$G=(V,E)$, containing nodes $V=\\{1,...,N\\}$ and $E\subseteq V\times V$ where
$(j,i)\in E$ denotes an edge from node $j$ to $i$. For each node $i$, we also
have its initial representation denoted as $h_{i}\in\mathbb{R}^{d}$. In a
message passing step, to update features for node $i$, we first compute the
importance value of neighbor node $j$ w.r.t. $i$ as following

$e(h_{i},h_{j})=a^{\text{T}}\text{LeakyReLU}(W\cdot[h_{i}\|h_{j}]),$ (2)

where $\|$ denotes vector concatenation, $W\in\mathbb{R}^{d\times
2d},a\in\mathbb{R}^{d\times 1}$. Followed by softmax, normalized attention
coefficients of all neighbors $j\in\mathcal{N}_{i}$ can be obtained:
$\alpha_{i,j}=\text{softmax}(e(h_{i},h_{j}))$. Then, new representation
$h_{i}$ for node $i$ is aggregated by

$h^{\prime}_{i}=\text{ReLU}(\sum_{j\in\mathcal{N}_{i}}\alpha_{i,j}Wh_{j}).$
(3)

Formally, the output of one GAT layer on a graph $G$ is

$\\{h^{\prime}_{i}\\}=\text{GAT}(\\{h_{i}\\},G).$ (4)

####  3.3.1 Object-Attribute GAT

At the bottom level, we care about how the semantic representation of an
object is modified by its connected attributes in the graph. These attributes
are modifiers that alter the visual appearance of the object. Because an
attribute of one object should in no way alter the appearance of another
object, in this step, we apply GAT only on the subgraph
$G_{\text{OA}}=(V,E_{\text{OA}})$ consists of only edges between the object
and attribute nodes.

We denote $\\{h_{i}\\}_{i=1}^{|V|},h_{i}\in\mathbb{R}^{D}$ as the initial
representations for all nodes in the graph. These representations are
initialized from the aforementioned semantic concept embedding step. We train
a graph attention network, which we name $\text{GAT}_{\text{Obj-Att}}$ to
perform message passing in graph $G_{\text{OA}}$. The updated representation
of all nodes is therefore

$\\{h^{\prime}_{i}\\}=\text{GAT}_{\text{Obj-Att}}(\\{h_{i}\\},G_{\text{OA}}).$
(5)

At the output, we are only interested in the updated representation of the set
of object nodes. Since these objects have been composed with their
corresponding attributes, we name them as entities and denote them as
$\\{e_{i}\\}_{i=1}^{|O|}$, which will be used in one of our proposed losses.

####  3.3.2 Object-Object Relation GAT

At the top level, after acquiring the entity embeddings
$\\{e_{i}\\}_{i=1}^{|O|}$ for all object nodes, we continue to apply another
GAT, which we name $\text{GAT}_{\text{Obj-Obj}}$ on the subgraph
$G_{\text{OO}}=(O,E_{\text{OO}})$ consisting of only object nodes and edges
between them. Because these object nodes are connected with object-object
relation edges $\\{r_{ij}\\}$, our first step before applying GAT is to
contextualize the entity embeddings with their corresponding edges.

Edge features. Consider a directed relation edge $r_{ij}$. In this relation,
node $i$ plays the subject (active) role while node $j$ plays the object
(passive) role. For example, in the relation man-hold-cup, man is the subject
while cup is the object. To obtain the edge features for this relation, we
concatenate its semantic encoding $r_{ij}$ with the embedding of the entity
that plays the passive role $e_{j}$ as follows:
$r^{\prime}_{ij}=[r_{ij}\|e_{j}]$. While existing work [34] often concatenates
$r_{ij}$ with both the subject and object entity, in our work, we find that it
is empirically better to characterize a relation with only the passive object
entity. This is intuitively reasonable since the meaning of a relation such as
hold-cup, use-computer does not depend on what kind of subject is involved.

Edge-contextualized entity features. Consider object node $i$, we define
$\text{Active}(i)=\\{j|r_{ij}\in E_{\text{OO}}\\}$ consisting all nodes that
node $i$ has a subject (active) relation with. Vice-versa, we define
$\text{Passive}(i)=\\{j|r_{ji}\in E_{\text{OO}}\\}$ which is all nodes that
node $i$ has an object (passive) relation. We contextualize the embedding of
entity $i$ with its edges as

$e^{\prime}_{i}=e_{i}+\frac{\sum_{j\in\text{Active}(i)}W_{A}r^{\prime}_{ij}}{|\text{Active}(i)|}+\frac{\sum_{j\in\text{Passive}(i)}W_{P}r^{\prime}_{ji}}{|\text{Passive}(i)|},$
(6)

where $W_{A}$ and $W_{P}$ are two learnable matrices mapping edge features to
have the same dimension with entity embeddings.

Scene graph embedding. With $\\{e^{\prime}_{i}\\}_{i=1}^{|O|}$ as the initial
representation for all object nodes. We train a $\text{GAT}_{\text{Obj-Obj}}$
on graph $G_{\text{OO}}$. The updated representation for all nodes is

$\\{\hat{e}_{i}\\}=\text{GAT}_{\text{Obj-
Obj}}(\\{e^{\prime}_{i}\\},G_{\text{OO}}).$ (7)

In order to pool the whole graph into one single embedding vector, we also use
GPO [4] similar to our visual feature extraction step. We take the output
representation that is pooled from GPO as the scene embedding $t$ to represent
the original input text caption in the joint embedding space.

###  3.4 Training Objectives

Let $B=\\{(v_{i},t_{i},\\{e_{ik}\\}_{k=1}^{|O_{i}|})\\}_{i=1}^{N}$ be the
training batch of output image embedding $v_{i}$ of the $i$-th image, output
text embedding $t_{i}$ of the $i$-th text caption from $\text{GAT}_{\text{Obj-
Obj}}$, and set of output entity embeddings $\\{e_{ik}\\}_{k=1}^{|O_{i}|}$ of
the $i$-th text caption from $\text{GAT}_{\text{Obj-Att}}$. It is reminded
that these entities $\\{e_{ik}\\}$ are embeddings of the object nodes in the
scene graph of $t_{i}$. We train our model CORA with the following losses. For
brevity, we denote $s(v,t)=v^{\text{T}}t/(\|v\|\|t\|)$ to be the cosine
similarity between $v$ and $t$.

Triplet loss with hardest negatives. Following prior work in image-text
retrieval [10, 4], we also adopt the hinge-based triplet loss with hardest
negative mining,

$\displaystyle\mathcal{L}_{\text{HARD}}=\sum_{i}\max_{j}[\alpha+s(v_{i},t_{j})-s(v_{i},t_{i})]_{+}$
(8) $\displaystyle+\max_{j}[\alpha+s(v_{j},t_{i})-s(v_{i},t_{i})]_{+}.$ (9)

Essentially, for every matching image-caption $v_{i}$ and $t_{i}$ in the
training batch, this loss looks for the negative caption $t_{j}$ that is
closest to $v_{i}$, and the negative image $v_{j}$ that is closest to $t_{i}$
in the embedding space. $t_{j}$ and $v_{j}$ are the hardest negatives in the
training batch and help provide a strong discriminative learning signal to the
model.

Contrastive loss. As observed by previous work [4], the hardest triplet loss
above results in unstable learning during early training epochs. We find that
applying a contrastive loss that encourages the model to align the output
representations of all matching image, text, and object entity together
results in more stable training and better final results. Because the entity
embeddings $\\{e_{ik}\\}_{k=1}^{|O_{i}|}$ are also involved in the equation
here, our model CORA is also trained to perform image retrieval given an
object entity (_e.g_., image searching for straw hat). The loss is formulated
as follows

$\displaystyle\mathcal{L}_{\text{CON}}=-\sum_{i}\sum_{u}\log\frac{\exp{(s(v_{i},u)})}{\sum_{u^{\prime}\in\mathcal{N}_{i}}\exp{(s(v_{i},u^{\prime}))}}$
(10)
$\displaystyle-\sum_{i}\sum_{u}\log\frac{\exp{(s(v_{i},u)})}{\sum_{v^{\prime}\in\mathcal{N}_{u}}\exp{(s(v^{\prime},u))}},$
(11)

where $u\in\\{t_{i}\\}\cup\\{e_{ik}\\}_{k=1}^{|O_{i}|}$ is the semantic
embedding of either the text or an object entity corresponding to image $i$,
$\mathcal{N}_{i}$ is the negative set of semantic concepts that do not
correspond to image $i$, and similarly $\mathcal{N}_{u}$ is the negative set
of images that do not contain semantic concept $u$.

Specificity loss. The contrastive loss above aligns the embeddings of image,
text, and entity together in the joint space. In addition, we would like to
impose some structure in this space such that the similarity between an image
$v_{i}$ and text $t_{i}$ should be larger than between $v_{i}$ and all
entities $\\{e_{ik}\\}$. The reason is that a caption always depicts more
semantic information than an entity alone, hence $t_{i}$ should be more
specific w.r.t. $v_{i}$ and exhibits a larger similarity score. The loss takes
the form of a hinge-based triplet loss

$\mathcal{L}_{\text{SPEC}}=\sum_{i}\sum_{k}[\alpha+s(v_{i},e_{ik})-s(v_{i},t_{i})]_{+}.$
(12)

Our overall loss is therefore a weighted sum of all losses:

$\mathcal{L}=\mathcal{L}_{\text{HARD}}+\lambda_{\text{CON}}\mathcal{L}_{\text{CON}}+\lambda_{\text{SPEC}}\mathcal{L}_{\text{SPEC}}.$
(13)

##  4 Experiments

We describe our experiments to validate the effectiveness of CORA. We describe
the datasets in Sec 4.1 and analyze the results in Sec 4.2. To validate design
choices, we present ablations in Sec 4.3. We refer to supplementary for
implementation, qualitative results, and inference time analysis.

###  4.1 Dataset and Evaluation Metrics

Datasets. We perform experiments on two standard benchmarks, Flickr30K [41]
and MS-COCO [27], on the image-to-text retrieval (I2T) and text-to-image
retrieval (T2I) tasks. In both datasets, every image is annotated with five
text descriptions. As in prior work [4], we follow the splits convention on
both datasets. Flickr30K contains 31K images, of which 29K images are for
training, 1K for validation, and 1K for testing. MS-COCO provides 123,287
images and is split into 113,287 images for training, 5000 images for
validation, and 5000 images for testing.

Metrics. We report the commonly used Recall@K (R@K), where K
$\in\\{1,5,10\\}$. This metric computes the percentage of queries where the
correct match appears in the top-K retrievals. To summarize performance, we
report RSUM which is the sum of R@K at all values of K $\in\\{1,5,10\\}$ on
I2T and T2I tasks. For MS-COCO, by convention, the results are reported in two
settings: 5K setting, and 1K setting where the results are averaged over five
1K data folds.

###  4.2 Quantitative Results

Table 1: Our framework achieves the best and second-best on the Flickr30K
dataset with two different encoders. Without the CA - “cross-attention”, our
method still has competitive results to other baselines. $\dagger$ denotes
methods that use ensembling of multiple models, and we highlight the highest
and second-highest RSUM for each section.

Method | Venue | CA | Image $\rightarrow$ Text | Text $\rightarrow$ Image | RSUM
---|---|---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Faster R-CNN + Bi-GRU
SCAN† [23] | ECCV’18 | ✓ | 67.4 | 90.3 | 95.8 | 48.6 | 77.7 | 85.2 | 465.0
VSRN [24] | ICCV’19 |  | 71.3 | 90.6 | 96.0 | 54.7 | 81.8 | 88.2 | 482.6
SGM [51] | WACV’20 | ✓ | 71.8 | 91.7 | 95.5 | 53.5 | 79.6 | 86.5 | 478.6
GCN+DIST [25] | CVPR’20 | ✓ | 70.8 | 92.7 | 96.0 | 60.9 | 86.1 | 91.0 | 497.5
GSMN† [28] | CVPR’20 | ✓ | 76.4 | 94.3 | 97.3 | 57.4 | 82.3 | 89.0 | 496.8
CAAN [58] | CVPR’20 | ✓ | 70.1 | 91.6 | 97.2 | 52.8 | 79.0 | 87.9 | 478.6
VSE∞ [4] | CVPR’21 |  | 76.5 | 94.2 | 97.7 | 56.4 | 83.4 | 89.9 | 498.1
VSRN++ [24] | PAMI’22 |  | 79.2 | 94.6 | 97.5 | 60.6 | 85.6 | 91.4 | 508.9
SGARF† [9] | AAAI’23 | ✓ | 77.8 | 94.1 | 97.4 | 58.5 | 83.0 | 88.8 | 499.6
CODER [49] | ECCV’22 | ✓ | 79.4 | 94.9 | 97.7 | 59.0 | 85.2 | 91.0 | 507.2
MV-VSE† [26] | IJCAI’22 |  | 79.0 | 94.9 | 97.7 | 59.1 | 84.6 | 90.6 | 505.8
GraDual† [30] | WACV’23 | ✓ | 78.3 | 96.0 | 98.0 | 60.4 | 86.7 | 92.0 | 511.4
CHAN [35] | CVPR’23 | ✓ | 79.7 | 94.5 | 97.3 | 60.2 | 85.3 | 90.7 | 507.7
NAAF† [57] | CVPR’23 | ✓ | 81.9 | 96.1 | 98.3 | 61.0 | 85.3 | 90.6 | 513.2
SDE† [20] | CVPR’23 |  | 80.9 | 94.7 | 97.6 | 59.4 | 85.6 | 91.1 | 509.3
HREM† [12] | CVPR’23 |  | 81.4 | 96.5 | 98.5 | 60.9 | 85.6 | 91.3 | 514.2
Ours |  |  | 82.2 | 95.6 | 97.7 | 61.8 | 86.5 | 92.0 | 515.8
Ours† |  |  | 82.3 | 96.1 | 98.0 | 63.0 | 87.4 | 92.8 | 519.6
Faster R-CNN + BERT |
VSE∞ [4] | CVPR’21 |  | 81.7 | 95.4 | 97.6 | 61.4 | 85.9 | 91.5 | 513.5
CODER [49] | ECCV’22 | ✓ | 83.2 | 96.5 | 98.0 | 63.1 | 87.1 | 93.0 | 520.9
MV-VSE† [26] | IJCAI’22 |  | 82.1 | 95.8 | 97.9 | 63.1 | 86.7 | 92.3 | 517.5
CHAN [35] | CVPR’23 | ✓ | 80.6 | 96.1 | 97.8 | 63.9 | 87.5 | 92.6 | 518.5
HREM† [12] | CVPR’23 |  | 84.0 | 96.1 | 98.6 | 64.4 | 88.0 | 93.1 | 524.2
Ours |  |  | 83.7 | 96.6 | 98.3 | 62.3 | 87.1 | 92.6 | 520.1
Ours† |  |  | 83.4 | 95.9 | 98.6 | 64.1 | 88.1 | 93.1 | 523.3

Table 2:  Our method yields competitive results on the MS-COCO dataset. Our
performance is competitive in all test schema with previous works, especially
on the simple Bi-GRU architecture. $\dagger$ denotes methods that use
ensembling of multiple models. Bold and underline highlight the best and
second-best performance.

Method |  Venue | Cross- | MS-COCO 5-fold 1K Test | MS-COCO 5K Test
---|---|---|---|---
Image $\rightarrow$ Text |  Text $\rightarrow$ Image | RSUM |  Image $\rightarrow$ Text |  Text $\rightarrow$ Image | RSUM
Attention | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Faster R-CNN + Bi-GRU |
SCAN† [23] | ECCV’18 | ✓ | 72.7 | 94.8 | 98.4 | 58.8 | 88.4 | 94.8 | 507.9 | 50.4 | 82.2 | 90.0 | 38.6 | 69.3 | 80.4 | 410.9
VSRN [24] | ICCV’19 |  | 76.2 | 94.8 | 98.2 | 62.8 | 89.7 | 95.1 | 516.8 | 53.0 | 81.1 | 89.4 | 40.5 | 70.6 | 81.1 | 415.7
SGM [51] | WACV’20 | ✓ | 73.4 | 93.8 | 97.8 | 57.5 | 87.3 | 94.3 | 504.1 | 50.0 | 79.3 | 87.9 | 35.3 | 64.9 | 76.5 | 393.9
CAAN [58] | CVPR’20 | ✓ | 75.5 | 95.4 | 98.5 | 61.3 | 89.7 | 95.2 | 515.6 | 52.5 | 83.3 | 90.9 | 41.2 | 70.3 | 82.9 | 421.1
VSE∞ [4] | CVPR’21 |  | 78.5 | 96.0 | 98.7 | 61.7 | 90.3 | 95.6 | 520.8 | 56.6 | 83.6 | 91.4 | 39.3 | 69.9 | 81.1 | 421.9
SGARF† [9] | AAAI’23 | ✓ | 79.6 | 96.2 | 98.5 | 63.2 | 90.7 | 96.1 | 524.3 | 57.8 | - | 91.6 | 41.9 | - | 81.3 | -
CODER [49] | ECCV’22 | ✓ | 78.9 | 95.6 | 98.6 | 62.5 | 90.3 | 95.7 | 521.6 | 58.5 | 84.3 | 91.5 | 40.9 | 70.8 | 81.4 | 427.4
MV-VSE† [26] | IJCAI’22 |  | 78.7 | 95.7 | 98.7 | 62.7 | 90.4 | 95.7 | 521.9 | 56.7 | 84.1 | 91.4 | 40.3 | 70.6 | 81.6 | 424.6
GraDual† [30] | WACV’23 | ✓ | 77.0 | 96.4 | 98.6 | 65.3 | 91.9 | 96.4 | 525.6 | - | - | - | - | - | - | -
CHAN [35] | CVPR’23 | ✓ | 79.7 | 96.7 | 98.7 | 63.8 | 90.4 | 95.8 | 525.1 | 60.2 | 85.9 | 92.4 | 41.7 | 71.5 | 81.7 | 433.4
NAAF† [57] | CVPR’23 | ✓ | 80.5 | 96.5 | 98.8 | 64.1 | 90.7 | 96.5 | 527.2 | 58.9 | 85.2 | 92.0 | 42.5 | 70.9 | 81.4 | 430.9
SDE† [20] | CVPR’23 |  | 80.6 | 96.3 | 98.8 | 64.7 | 91.4 | 96.2 | 528.0 | 60.4 | 86.2 | 92.4 | 42.6 | 73.1 | 83.1 | 437.8
HREM† [12] | CVPR’23 |  | 81.2 | 96.5 | 98.9 | 63.7 | 90.7 | 96.0 | 527.0 | 60.6 | 86.4 | 92.5 | 41.3 | 71.9 | 82.4 | 435.1
Ours |  |  | 80.9 | 96.3 | 98.8 | 64.9 | 91.3 | 96.4 | 528.5 | 61.4 | 85.6 | 92.4 | 43.3 | 72.9 | 83.3 | 438.9
Ours† |  |  | 81.7 | 96.7 | 99.0 | 66.0 | 92.0 | 96.7 | 532.1 | 63.0 | 86.8 | 92.7 | 44.2 | 73.9 | 84.0 | 444.6
Faster R-CNN + BERT |
VSE∞ [4] | CVPR’21 |  | 79.7 | 96.4 | 98.9 | 64.8 | 91.4 | 96.3 | 527.5 | 58.3 | 85.3 | 92.3 | 42.4 | 72.7 | 83.2 | 434.2
CODER [49] | ECCV’22 | ✓ | 82.1 | 96.6 | 98.8 | 65.5 | 91.5 | 96.2 | 530.7 | 62.6 | 86.6 | 93.1 | 42.5 | 73.1 | 83.3 | 441.2
MV-VSE† [26] | IJCAI’22 |  | 80.4 | 96.6 | 99.0 | 64.9 | 91.2 | 96.0 | 528.1 | 59.1 | 86.3 | 92.5 | 42.5 | 72.8 | 83.1 | 436.3
CHAN [35] | CVPR’23 | ✓ | 81.4 | 96.9 | 98.9 | 66.5 | 92.1 | 96.7 | 532.5 | 59.8 | 87.2 | 93.3 | 44.9 | 74.5 | 84.2 | 443.9
HREM† [12] | CVPR’23 |  | 82.9 | 96.9 | 99.0 | 67.1 | 92.0 | 96.6 | 534.5 | 64.0 | 88.5 | 93.7 | 45.4 | 75.1 | 84.3 | 451.0
Ours |  |  | 82.4 | 96.8 | 98.8 | 66.2 | 91.9 | 96.6 | 532.7 | 62.4 | 86.8 | 92.6 | 44.2 | 73.6 | 83.9 | 443.6
Ours† |  |  | 82.8 | 97.3 | 99.0 | 67.3 | 92.4 | 96.9 | 535.6 | 64.3 | 87.5 | 93.6 | 45.4 | 74.7 | 84.6 | 450.1

We summarize our results compared with SOTA methods on Flickr30K and MS-COCO
in Tab. 1 and Tab. 2. The methods are denoted with whether they are cross-
attention or dual-encoder approaches, and are divided into groups depending on
the textual backbone used (Bi-GRU vs. BERT). Following previous work [57, 20,
12], we also report the ensemble results which are obtained by averaging the
similarities from two checkpoints trained with different seeds.

Comparisons with state-of-the-art methods. When using Bi-GRU as the semantic
concept encoder, our method CORA outperforms all state-of-the-art methods by
an impressive margin. CORA achieves $+5.4$ RSUM absolute improvement over HREM
on Flickr30K, and $+13.7$ RSUM over NAAF on MS-COCO 5K. Note that NAAF is
among the SOTA cross-attention methods (CHAN, GraDual, CODER, SGARF) which are
more computationally expensive but having more learning capacity advantage
over dual encoders, however CORA is still able to surpass them. The non-
ensemble version of CORA also outperforms all non-ensemble methods while even
exceeding the ensemble ones (SDE, GraDual).

When using BERT for encoding semantic concepts, CORA achieves second best RSUM
score on Flickr30K and MS-COCO and is only inferior to the recent SOTA HREM.
HREM is also a dual encoder, but is trained with a cross-modality mechanism
(which is later discarded at inference) to enhance each modality embedding for
matching. The same idea of HREM can be applied to CORA to boost the
performance even further, but is out of the scope of our work. Switching from
Bi-GRU to using BERT, our method enjoys a smaller RSUM improvement compared to
other work ($+5.5$ RSUM for CORA vs. $+15.9$ for HREM and $+10.5$ for CHAN on
MS-COCO 5K). This is due to BERT being more suitable for encoding long text
while CORA is using BERT to encode short phrases (_e.g_., construction worker,
sitting). A text encoder that is more suitable for encoding short phrases is
therefore more desirable and we leave this as future work. Similarly, when
using BERT, CORA surpasses the performance of SOTA cross-attention methods
CODER and CHAN. This shows the ability to generalize across different feature
extractors of CORA.

Comparisons with scene graph-based approaches. CORA outperforms all scene-
graph based methods, which includes SGM [51], GCN+DIST [25], GSMN [28], and
GraDual [30]. These methods all employ an additional off-the-shelf visual
scene graph generator [56] (except GSMN) to produce a scene graph for an input
image, and use cross-attention to exchange information between the textual and
the visual graph, but still achieve inferior results to CORA. This further
shows that CORA is very effective at encoding scene graphs. These methods all
embed a whole scene graph holistically (unlike CORA which separates it into
two object-attribute and object-object steps), are trained with a holistic
loss to align image and text (unlike CORA that has loss terms to additionally
align image and local object entity), and use visual scene graph generator
[56] which is susceptible to making wrong predictions and has been reported to
misdetect rare object relationships [47].

###  4.3 Ablation Studies

We perform a series of ablation studies to explore the impact of our graph
attention network design and how the losses affect the final performance. All
experiments in this section use Bi-GRU for the semantic encoder and are
performed on the Flickr30K dataset. The results are reported in Tab. 3.

Table 3:  Ablation studies for the number of layers in GAT, the graph
structure whether encoding scene graph jointly or in 2 separate steps is
beneficial, and the impact of losses. Bold and underline highlight the best
and second-best performance.

|  |  |  |  |  | Image $\rightarrow$ Text | Text $\rightarrow$ Image | RSUM
---|---|---|---|---|---|---|---|---
|  |  |  |  |  | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Number of GAT layers |  |  |  |  |  |  |
$n_{\text{Obj-Att}}$ | $n_{\text{Obj-Obj}}$ |  |  |  |  |  |  |
1 | 1 | 79.8 | 95.3 | 97.1 | 60.5 | 85.4 | 91.0 | 509.1
1 | 2 | 82.2 | 95.6 | 97.7 | 61.8 | 86.5 | 92.0 | 515.8
1 | 3 | 81.6 | 95.8 | 97.6 | 61.3 | 86.2 | 91.9 | 514.4
2 | 1 | 79.9 | 95.1 | 97.5 | 60.8 | 85.5 | 91.3 | 510.1
2 | 2 | 81.7 | 95.3 | 96.8 | 61.6 | 87.0 | 92.1 | 514.5
2 | 3 | 80.2 | 95.3 | 96.5 | 60.9 | 85.9 | 91.8 | 510.6
Graph structure |  |  |  |  |  |  |
Joint | Obj-Att&Obj-Obj | FC |  |  |  |  |  |  |
✓ |  |  |  |  | 78.9 | 93.5 | 96.3 | 59.6 | 85.8 | 90.4 | 504.8
✓ |  |  | ✓ | 77.6 | 93.0 | 96.0 | 59.4 | 85.1 | 90.5 | 501.6
|  | ✓ |  |  | 82.2 | 95.6 | 97.7 | 61.8 | 86.5 | 92.0 | 515.8
|  | ✓ | ✓ | 81.2 | 95.1 | 96.9 | 61.3 | 86.9 | 91.8 | 513.2
Losses |  |  |  |  |  |  |
$\mathcal{L}_{\text{HARD}}$ | $\mathcal{L}_{\text{CON}}$ | $\mathcal{L}_{\text{SPEC}}$ |  |  |  |  |  |  |
✓ |  |  |  |  | 71.6 | 92.2 | 95.9 | 53.9 | 83.4 | 89.9 | 486.9
|  | ✓ |  |  | 75.0 | 94.0 | 96.4 | 57.9 | 84.4 | 90.1 | 497.8
✓ | ✓ |  |  | 78.9 | 95.2 | 97.3 | 59.1 | 85.8 | 91.0 | 507.3
✓ | ✓ | ✓ | 82.2 | 95.6 | 97.7 | 61.8 | 86.5 | 92.0 | 515.8

Number of layers in GAT. The experiments show that having $1$ layer for
$\text{GAT}_{\text{Obj-Att}}$ and 2 layers for $\text{GAT}_{\text{Obj-Obj}}$
achieves the best accuracy. For the object-attribute graph, 1 layer is
sufficient to propagate the attribute information to their corresponding
object node. For the object-object relation graph, using only 1 layer is not
enough to aggregate information from the whole graph, while increasing to 3
layers starts to give diminishing returns.

Graph structure. We study whether our 2-step scene graph encoding step is
beneficial to the final performance. We refer to Joint as the model that uses
a single GAT on the whole graph at once, FC as the variant that uses fully
connected graph instead of the structure parsed from the scene graph parser,
and Obj-Att & Obj-Obj as our proposed 2-step scene graph encoding model. Note
that having a separate object-attribute encoding step allows our model to
produce individual entity embeddings (see Sec. 3.3.1) that are later used in
the contrastive loss to align an image with each of its objects independently.
This is not possible with the Joint model because after the GAT step, all
object nodes are already contextualized. The results indeed show that our
proposed 2-step encoding step is superior, and using the graph structure
parsed from the scene graph parser is better than connecting all nodes
together in a fully connected manner.

Losses. Similar to prior work, we first explore using the triplet loss with
hardest negatives $\mathcal{L}_{\text{HARD}}$ and find that using it alone is
insufficient, since it can only be used on the image-text level and not at the
image-object entity level. Using solely the contrastive loss
$\mathcal{L}_{\text{CON}}$ gives an accuracy boost, but is still far from
optimal. By combining $\mathcal{L}_{\text{HARD}}$ and
$\mathcal{L}_{\text{CON}}$, we achieve a significant accuracy improvement.
Since the contrastive loss treats the graph embedding and entity embedding
equally, by imposing the structure that a whole text caption should depict
more information than an entity alone through the loss
$\mathcal{L}_{\text{SPEC}}$, CORA achieves SOTA results.

###  4.4 Qualitative Results & Text-to-Entity Retrieval

Fig. 3 illustrates how CORA can perform image-to-text and image-to-object
entity retrieval. More qualitative results can be found in the supplementary.

In addition, we also experiment with using the image-entity score for re-
ranking the image-text matching. We employ the idea that if an object entity
in the text is not closely matched with the image, then the image-text
matching score should be lower. Formally, we utilize the following formula

$\displaystyle\hat{s}(v_{i},t_{i})=\beta\cdot
s(v_{i},t_{i})+(1-\beta)\cdot\min_{k}s(v_{i},e_{ik}),$ (14)

where $\beta\in[0,1]$ is a hyperparameter that we select on the validation
set. The results on MS-COCO are reported in Tab. 4, where we achieve a slight
accuracy improvement with this simple strategy. One potential future direction
is to explore smarter mechanism to combine the image-text and image-object
entity embedding alignment score.

Table 4:  Reranking results on MS-COCO 5K after ensembling with image-entity
score.

| Image $\rightarrow$ Text | Text $\rightarrow$ Image | RSUM
---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Faster R-CNN + Bi-GRU
CORA | 63.0 | 86.8 | 92.7 | 44.2 | 73.9 | 84.0 | 444.6
CORA + reranking | 63.2 | 86.8 | 92.7 | 44.3 | 74.1 | 84.0 | 445.1
Faster R-CNN + BERT
CORA | 64.3 | 87.5 | 93.6 | 45.4 | 74.7 | 84.6 | 450.1
CORA + reranking | 64.2 | 87.6 | 93.8 | 45.5 | 74.8 | 84.7 | 450.6

Figure 3: Qualitative result demonstrates how CORA can perform image-to-text
and image-to-entity retrieval. Green denotes correct retrieval while  red
denotes incorrect ones.

##  5 Conclusion

Limitation. Despite achieving new SOTA results, CORA still faces some
limitations. CORA is strongly dependent on the scene graph quality from the
parser. If the parser fails to extract a scene graph from the input text, CORA
also fails to encode the text. This happens seldomly in MS-COCO, where there
are captions that are just exclamatory sentences uttered by the annotator,
_e.g_., “I am so happy to see this view”, “There are so many things to see
here.” On the other hand, text sequence model is still able to capture the
nuances of these text descriptions.

In this paper, we propose a dual-encoder model CORA for image-text matching
that is based on scene graph. CORA achieves new SOTA results, outperforms all
SOTA computationally expensive cross-attention methods. We show a promising
future direction for image-text matching that, by representing a caption as a
scene graph of object and attribute nodes connected by relation edges, we can
utilize the strong relational inductive bias of graph neural network to
compose objects, relations, and their attributes into a scene graph embedding
that is effective for image-text retrieval.

Acknowledgements. This project was partially funded by NSF CAREER Award
(#2238769) to AS.

## References

  * Anderson et al. [2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In _CVPR_ , 2018.
  * Brody et al. [2021] Shaked Brody, Uri Alon, and Eran Yahav.  How attentive are graph attention networks?  _arXiv preprint arXiv:2105.14491_ , 2021.
  * Chefer et al. [2023] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  _arXiv preprint arXiv:2301.13826_ , 2023.
  * Chen et al. [2021] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang.  Learning the best pooling strategy for visual semantic embedding.  In _CVPR_ , 2021.
  * Chen et al. [2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _ECCV_. Springer, 2020.
  * Chun et al. [2021] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus.  Probabilistic embeddings for cross-modal retrieval.  In _CVPR_ , 2021.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ , 2014.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Diao et al. [2021] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu.  Similarity reasoning and filtration for image-text matching.  In _AAAI_ , 2021.
  * Faghri et al. [2017] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler.  Vse++: Improving visual-semantic embeddings with hard negatives.  _arXiv preprint arXiv:1707.05612_ , 2017.
  * Frome et al. [2013] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  Devise: A deep visual-semantic embedding model.  _NeurIPS_ , 26, 2013.
  * Fu et al. [2023] Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang.  Learning semantic relationship among instances for image-text matching.  In _CVPR_ , 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Hewitt and Manning [2019] John Hewitt and Christopher D Manning.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, 2019.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Honnibal et al. [2020] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd.  spaCy: Industrial-strength Natural Language Processing in Python.  2020\.
  * Huang et al. [2019] Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou.  Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks.  _arXiv preprint arXiv:1909.00964_ , 2019.
  * Huynh et al. [2023] Chuong Huynh, Yuqian Zhou, Zhe Lin, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi, and Abhinav Shrivastava.  Simpson: Simplifying photo cleanup with single-click distracting object segmentation network.  In _CVPR_ , 2023.
  * Huynh et al. [2024] Chuong Huynh, Seoung Wug Oh, , Abhinav Shrivastava, and Joon-Young Lee.  Maggie: Masked guided gradual human instance matting.  In _CVPR_ , 2024.
  * Kim et al. [2023] Dongwon Kim, Namyup Kim, and Suha Kwak.  Improving cross-modal retrieval with set of diverse embeddings.  In _CVPR_ , 2023.
  * Kiros et al. [2014] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  _arXiv preprint arXiv:1411.2539_ , 2014.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _IJCV_ , 2017.
  * Lee et al. [2018] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He.  Stacked cross attention for image-text matching.  In _ECCV_ , 2018.
  * Li et al. [2019] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu.  Visual semantic reasoning for image-text matching.  In _ICCV_ , pages 4654–4662, 2019.
  * Li et al. [2020] Yongzhi Li, Duo Zhang, and Yadong Mu.  Visual-semantic matching by exploring high-order attention and distraction.  In _CVPR_ , 2020.
  * Li et al. [2022] Zheng Li, Caili Guo, Zerun Feng, Jenq-Neng Hwang, and Xijun Xue.  Multi-view visual semantic embedding.  In _IJCAI_ , page 7, 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. [2020] Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Bin Wang, and Yongdong Zhang.  Graph structured network for image-text matching.  In _CVPR_ , 2020.
  * Liu et al. [2021] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  _arXiv preprint arXiv:2110.07602_ , 2021.
  * Long et al. [2022] Siqu Long, Soyeon Caren Han, Xiaojun Wan, and Josiah Poon.  Gradual: Graph-based dual-modal representation for image-text matching.  In _WACV_ , 2022.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _NeurIPS_ , 32, 2019.
  * Mahajan et al. [2018] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten.  Exploring the limits of weakly supervised pretraining.  In _ECCV_ , pages 181–196, 2018.
  * Nguyen et al. [2021] Kien Nguyen, Subarna Tripathi, Bang Du, Tanaya Guha, and Truong Q Nguyen.  In defense of scene graphs for image captioning.  In _ICCV_ , 2021.
  * Pan et al. [2023] Zhengxin Pan, Fangyu Wu, and Bailing Zhang.  Fine-grained image-text matching by cross-modal hard aligning network.  In _CVPR_ , 2023.
  * Patterson and Hays [2016] Genevieve Patterson and James Hays.  Coco attributes: Attributes for people, animals, and objects.  In _ECCV_ , pages 85–100. Springer, 2016.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014.
  * Pham et al. [2021] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava.  Learning to predict visual attributes in the wild.  In _CVPR_ , pages 13018–13028, 2021.
  * Pham et al. [2022] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava.  Improving closed and open-vocabulary attribute prediction using transformers.  In _ECCV_ , 2022.
  * Phung et al. [2024] Quynh Phung, Songwei Ge, and Jia-Bin Huang.  Grounded text-to-image synthesis with attention refocusing.  In _CVPR_ , 2024.
  * Plummer et al. [2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _ICCV_ , 2015.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Saini et al. [2022] Nirat Saini, Khoi Pham, and Abhinav Shrivastava.  Disentangling visual embeddings for attributes and objects.  In _CVPR_ , pages 13658–13667, 2022.
  * Schuster et al. [2015] Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning.  Generating semantically precise scene graphs from textual descriptions for improved image retrieval.  In _Workshop on Vision and Language (VL15)_ , Lisbon, Portugal, 2015. Association for Computational Linguistics.
  * Song and Soleymani [2019] Yale Song and Mohammad Soleymani.  Polysemous visual-semantic embedding for cross-modal retrieval.  In _CVPR_ , pages 1979–1988, 2019.
  * Tang et al. [2020] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang.  Unbiased scene graph generation from biased training.  In _CVPR_ , 2020.
  * Veličković et al. [2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ , 2017.
  * Wang et al. [2022] Haoran Wang, Dongliang He, Wenhao Wu, Boyang Xia, Min Yang, Fu Li, Yunlong Yu, Zhong Ji, Errui Ding, and Jingdong Wang.  Coder: Coupled diversity-sensitive momentum contrastive learning for image-text retrieval.  In _ECCV_ , 2022.
  * Wang et al. [2016] Liwei Wang, Yin Li, and Svetlana Lazebnik.  Learning deep structure-preserving image-text embeddings.  In _CVPR_ , 2016.
  * Wang et al. [2020] Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, and Xilin Chen.  Cross-modal scene graph matching for relationship-aware image-text retrieval.  In _WACV_ , 2020.
  * Wen et al. [2020] Keyu Wen, Xiaodong Gu, and Qingrong Cheng.  Learning dual semantic relations with graph attention for image-text matching.  _IEEE transactions on circuits and systems for video technology_ , 31(7):2866–2879, 2020.
  * Wu et al. [2019a] Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma.  Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations.  In _CVPR_ , 2019a.
  * Wu et al. [2019b] Yiling Wu, Shuhui Wang, Guoli Song, and Qingming Huang.  Learning fragment self-attention embeddings for image-text matching.  In _ACM MM_ , 2019b.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _CVPR_ , pages 1492–1500, 2017.
  * Zellers et al. [2018] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi.  Neural motifs: Scene graph parsing with global context.  In _CVPR_ , 2018.
  * Zhang et al. [2022] Kun Zhang, Zhendong Mao, Quan Wang, and Yongdong Zhang.  Negative-aware attention framework for image-text matching.  In _CVPR_ , 2022.
  * Zhang et al. [2020] Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z Li.  Context-aware attention network for image-text retrieval.  In _CVPR_ , 2020.

Supplementary Material

##  6 Implementation Details

###  6.1 Visual features

We use the pre-extracted Faster R-CNN 2048-dimensional region features from
BUTD [23, 1], which is the standard convention from prior work in the image-
text matching literature [4, 12]. To transform them to have the same
dimensions $D$ as the joint embedding space, we implement a 2-layer MLP with
residual connection. The region features are then pooled using the GPO [4]
pooling operator into $\mathbb{R}^{D}$.

###  6.2 Textual features

Bi-GRU. The dimension of the word embedding is set to 300 for both experiments
where we initialize the word embedding from GloVe or from scratch (refer to
Sec. 8 for experiment of CORA without using GloVe). The GRU has 1 layer and
its hidden dimension is also 300.

BERT. Similar to prior work, we use the bert-base-uncased architecture and
pre-trained weights for the BERT semantic concept encoder. As mentioned in the
main paper, using BERT to encode short phrases (_e.g_., construction worker,
sitting) does not take advantage of the full capability of BERT. BERT has
never seen short text during its pre-training stage [8], and with its ability
to capture long-range dependencies, BERT is more suitable for encoding long
sentences. As a result, direct fine-tuning BERT for CORA leads to slightly
lower results.

In our work, instead of fine-tuning the whole BERT model (with 110M params),
we employ the prefix tuning technique P-Tuning v2 [29] in order to repurpose
the pre-trained BERT model into encoding short phrases. With this technique,
at every BERT encoding layer, a sequence of learnable $N$ token embeddings
$\mathbb{R}^{N\times 768}$ is added as prefix into the textual prompt.
Intuitively, these tokens provide learnable context that assist BERT into
learning the task at hand, which is encoding short phrases. The number of
trainable params with P-Tuning v2 is only $2N\times L\times 768$ (where $L=12$
is the number of BERT encoding layers, and $N=24$ is the number of prefix
tokens). In our experiment, we find fine-tuning the last BERT layer along with
P-Tuning gives slightly better results. In overall, the number of trainable
params of our BERT component is only 7M, which is much smaller than 110M
params of the whole BERT model.

For both types of features (Bi-GRU and BERT), we implement an FC layer to
transform the semantic encoded output into $\mathbb{R}^{D}$ before using them
to initialize the node and edge features of the GATs.

###  6.3 Training and hyperparameter details

We use the AdamW optimizer [31] to train our model for 50 epochs. The learning
rate is initialized at 5e-4, then decayed to 5e-5 after 15 epochs. The
learning rate for the pre-trained components (_i.e_., GloVe and BERT) is
scaled by 0.1 w.r.t. the base learning rate. We set the batch size to 128 when
training on Flickr30K, and 256 when training on MS-COCO. The margin $\alpha$
in the triplet loss is set to 0.4, while the cosine similarity in the
contrastive loss is scaled by a temperature of 0.01 similar to CLIP [42].
Following [4], we perform size augmentation to randomly drop 35% region
features. For data augmentation on the text, we perform subsampling on the
scene graph by randomly dropping 10% of the nodes and edges and randomly
masking 10% of the word tokens. We set $\lambda_{\text{CON}}=0.25$ and
$\lambda_{\text{SPEC}}=3.0$.

##  7 Inference Time

Figure 4: Inference time comparison. We compare the text-to-image retrieval
inference time between our method CORA against two SOTA cross-attention
methods SGRAF [9] and NAAF [57] (lower is better). The inference time is
calculated with different number of images in the database. CORA with its
dual-encoder architecture is much faster and scalable than cross-attention
approaches.

We illustrate in Fig. 4 the inference time comparison between our method CORA
against SOTA cross-attention methods SGRAF [9] and NAAF [57] with different
number of images in the database (ranging from a very small to a very large
number of images).

To conduct this experiment, for all methods, we first forward all images
through the image encoder of each respective method in order to cache all
image embeddings. Then, for CORA, when a text query arrives, it takes 0.04s to
parse it into a scene graph, 0.014s to compute its scene graph embedding, then
0.01s to perform the vector-matrix multiplication with the image embeddings to
find nearest neighbor results, which in total accounts to around 0.06s per
query for all number of images from 10 to $10^{5}$. On the other hand, for
cross-attention approaches SGRAF and NAAF, when a text query arrives, these
methods have to pair the text query with every image embedding in the
database, then forward each pair through the cross-attention module in order
to calculate their similarity. Fig. 4 shows that the inference time for SGRAF
and NAAF scale up linearly w.r.t. the number of images in the database
(_e.g_., SGRAF takes 46s with $10^{4}$ images, and 470s with $10^{5}$ images),
which is due to the iterative pairing of the input text with each image. Our
model CORA enjoys the benefit of being fast and scalable of the dual-encoder
architecture, while still achieving better retrieval results than SOTA cross-
attention approaches (_e.g_., SGRAF and NAAF).

##  8 More Ablation Studies

Initialize from GloVe vs. from scratch. When using Bi-GRU, we follow all
recent studies [30, 49, 57, 35, 20, 12] to initialize the word embeddings
using GloVe [37]. To fairly compare against other methods prior to these work,
we also report our results when using Bi-GRU with word embeddings initialized
from scratch in Tabs. 5 and 6 for the Flickr30K and MS-COCO dataset
respectively. The results show that even when initializing the word embeddings
from scratch, our method CORA still outperforms all previous work with and
without cross-attention.

Table 5: Our framework achieves the best results on the Flickr30K dataset when
initializing the word embeddings fom scratch for the Bi-GRU semantic encoder.
Without the CA - “cross-attention”, our method still has competitive results
to other baselines. $\dagger$ denotes methods that use ensembling of multiple
models, and we highlight the highest and second-highest RSUM.

Method | Venue | CA |  Image $\rightarrow$ Text |  Text $\rightarrow$ Image | RSUM
---|---|---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Faster R-CNN + Bi-GRU
SCAN† [23] | ECCV’18 | ✓ | 67.4 | 90.3 | 95.8 | 48.6 | 77.7 | 85.2 | 465.0
VSRN [24] | ICCV’19 |  | 71.3 | 90.6 | 96.0 | 54.7 | 81.8 | 88.2 | 482.6
SGM [51] | WACV’20 | ✓ | 71.8 | 91.7 | 95.5 | 53.5 | 79.6 | 86.5 | 478.6
GCN+DIST [25] | CVPR’20 | ✓ | 70.8 | 92.7 | 96.0 | 60.9 | 86.1 | 91.0 | 497.5
GSMN† [28] | CVPR’20 | ✓ | 76.4 | 94.3 | 97.3 | 57.4 | 82.3 | 89.0 | 496.8
CAAN [58] | CVPR’20 | ✓ | 70.1 | 91.6 | 97.2 | 52.8 | 79.0 | 87.9 | 478.6
VSE∞ [4] | CVPR’21 |  | 76.5 | 94.2 | 97.7 | 56.4 | 83.4 | 89.9 | 498.1
SGARF† [9] | AAAI’21 | ✓ | 77.8 | 94.1 | 97.4 | 58.5 | 83.0 | 88.8 | 499.6
MV-VSE† [26] | IJCAI’22 |  | 79.0 | 94.9 | 97.7 | 59.1 | 84.6 | 90.6 | 505.8
Ours |  |  | 80.1 | 95.5 | 97.7 | 60.6 | 85.6 | 91.1 | 510.5
Ours† |  |  | 81.7 | 95.5 | 98.1 | 62.0 | 86.6 | 91.8 | 515.7

Table 6: Our framework achieves the best results on the MS-COCO dataset when
initializing the word embeddings fom scratch for the Bi-GRU semantic encoder.
Without the CA - “cross-attention”, our method still has competitive results
to other baselines. $\dagger$ denotes methods that use ensembling of multiple
models, and we highlight the highest and second-highest RSUM.

Method |  Venue | Cross- | MS-COCO 5-fold 1K Test | MS-COCO 5K Test
---|---|---|---|---
Image $\rightarrow$ Text |  Text $\rightarrow$ Image | RSUM |  Image $\rightarrow$ Text |  Text $\rightarrow$ Image | RSUM
Attention | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Faster R-CNN + Bi-GRU |
SCAN† [23] | ECCV’18 | ✓ | 72.7 | 94.8 | 98.4 | 58.8 | 88.4 | 94.8 | 507.9 | 50.4 | 82.2 | 90.0 | 38.6 | 69.3 | 80.4 | 410.9
VSRN [24] | ICCV’19 |  | 76.2 | 94.8 | 98.2 | 62.8 | 89.7 | 95.1 | 516.8 | 53.0 | 81.1 | 89.4 | 40.5 | 70.6 | 81.1 | 415.7
SGM [51] | WACV’20 | ✓ | 73.4 | 93.8 | 97.8 | 57.5 | 87.3 | 94.3 | 504.1 | 50.0 | 79.3 | 87.9 | 35.3 | 64.9 | 76.5 | 393.9
CAAN [58] | CVPR’20 | ✓ | 75.5 | 95.4 | 98.5 | 61.3 | 89.7 | 95.2 | 515.6 | 52.5 | 83.3 | 90.9 | 41.2 | 70.3 | 82.9 | 421.1
VSE∞ [4] | CVPR’21 |  | 78.5 | 96.0 | 98.7 | 61.7 | 90.3 | 95.6 | 520.8 | 56.6 | 83.6 | 91.4 | 39.3 | 69.9 | 81.1 | 421.9
SGARF† [9] | AAAI’21 | ✓ | 79.6 | 96.2 | 98.5 | 63.2 | 90.7 | 96.1 | 524.3 | 57.8 | - | 91.6 | 41.9 | - | 81.3 | -
MV-VSE† [26] | IJCAI’22 |  | 78.7 | 95.7 | 98.7 | 62.7 | 90.4 | 95.7 | 521.9 | 56.7 | 84.1 | 91.4 | 40.3 | 70.6 | 81.6 | 424.6
Ours |  |  | 80.5 | 96.0 | 98.6 | 62.9 | 90.6 | 96.0 | 524.6 | 60.4 | 85.1 | 91.7 | 40.5 | 70.6 | 81.2 | 429.5
Ours† |  |  | 81.2 | 96.2 | 98.7 | 63.4 | 90.9 | 96.2 | 526.6 | 60.9 | 85.6 | 92.0 | 40.8 | 71.0 | 81.8 | 432.1

Table 7:  Ablation studies to compare between fine-tuning the whole BERT model
versus using P-Tuning v2 [29] to encode the short phrases of semantic
concepts. The models are evaluated on the MS-COCO 1K Test set. Gray denotes
our best model.

Method | Image-to-text | Text-to-image | RSUM
---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
BERT | 82.0 | 96.5 | 98.8 | 64.5 | 91.1 | 96.2 | 529.1
BERT P-Tuning, N = 8 | 81.7 | 96.5 | 99.0 | 64.5 | 91.1 | 96.1 | 528.9
BERT P-Tuning, N = 16 | 81.4 | 96.9 | 98.8 | 65.0 | 91.2 | 96.3 | 529.6
BERT P-Tuning, N = 24 | 81.9 | 96.6 | 98.9 | 65.0 | 91.2 | 96.4 | 530.0
BERT P-Tuning, N = 32 | 82.2 | 96.7 | 98.7 | 64.8 | 91.3 | 96.2 | 529.9

BERT P-Tuning v2. We compare between direct fine-tuning the whole BERT model
against using P-Tuning v2 [29] to encode short phrases of semantic concepts.
The results are displayed in Tab. 7. Note that this model is ablated without
having multi-head self-attention in the visual encoder.

##  9 More Analysis

Table 8: Analysis on using the larger visual backbone ResNeXT-101 [55]. We
plug ResNeXT-101 into our CORA model with BERT as the semantic concept
encoder.

Method | Image-to-text | Text-to-image | RSUM
---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
CORA-BERT | 84.4 | 96.7 | 98.7 | 62.3 | 87.5 | 92.5 | 522.1
CORA-BERT + ResNeXT-101 | 90.9 | 99.1 | 99.8 | 76.8 | 95.1 | 97.7 | 559.4
VSE∞ \+ ResNeXT-101 [4] | 88.7 | 98.9 | 99.8 | 76.1 | 94.5 | 97.1 | 555.1
SDE + ResNeXT-101 [16] | 90.6 | 99.0 | 99.6 | 75.9 | 94.7 | 97.3 | 557.1

With larger visual backbone. We select ResNeXT-101 [55] pretrained on the
Instagram dataset [33] as the larger visual extractor than the Faster R-CNN
model used in our main experiments. This visual backbone is also reported in
VSE∞ [4] and SDE [20]. The results of this experiment on the Flickr30K test
set are displayed in Tab. 8, where it shows we obtain a large increase over
region features and others.

Simulate parsing errors. As discussed in the conclusions, our CORA model is
strongly dependent on the scene graph quality from the parser. To study this
dependence, we simulate errors by performing the followings onto the parsed
graphs: drop word tokens from nodes and edges, move attribute node to wrong
object node, and move edge to wrong object pair. We randomly perform these
onto 10%, 20%, 30% F30K captions and achieve 513.2, 512.2, 509.8 RSUM
(original performance is 515.8). We observe that moving the edge affects
performance more than moving the attribute.

Why consider CORA. CORA is a promising graph method that can supplement what
CLIP (& other text encoders) may struggle against, _i.e_., sentences with many
semantics that are mixed among objects (discussed in Sec. 1). To show example,
we select 100 sentences in Flickr30K with the highest number of attributes and
relations, then evaluate image retrieval on them. We obtain results
respectively for CLIP, HREM, CORA as 239.3, 240.0, 241.5 RSUM. This shows the
large model CLIP is even slightly inferior to CORA on sentences that are rich
in semantics.

Table 9: Compare with pre-trained image-text models.

Method | RSUM
---|---
ViLBERT - NeurIPS’19 - Data: CC3M | 502.7
UNITER - ECCV’20 - Data: CC3M, SBU | 510.9
UNITER - ECCV’20 - Data: CC3M, SBU, VG, COCO | 542.8
Unicoder-VL - AAAI’20 - Data: CC3M, SBU | 538.8
CORA-BERT - Data: CC1M | 530.1
CLIP zero-shot \- ICML’21 - Data: CLIP 400M | 540.6

Compare with pretrained image-text models. We scale up CORA on larger data and
compare with prior SOTA image-text models in Tab. 9. We pretrain CORA on 1M
image-text pairs in Conceptual Captions. All of the models in the table are
finally fine-tuned on Flickr30K. Compared with CORA-BERT (refer to Tab. 1),
CORA pre-trained gets a +6.8 score. Despite smaller data and not using cross-
attention, CORA is better than ViLBERT [32], UNITER [5], and can potentially
reach Unicoder [17] with more data. However, it is inferior to CLIP zero-shot
[42]. This shows the promising ability to scale up CORA further, _e.g_. by
using ViT instead of regions, CLIP-text instead of BERT, and more data.

##  10 Qualitative Results

Image-to-text and image-to-entity retrieval. We illustrate some examples of
successful and failed results when performing image-to-text retrieval using
our CORA model with Faster R-CNN + BERT trained on the MS-COCO dataset in Fig.
5 and Fig. 6. Because CORA also has the ability to retrieve object entities,
we also include image-to-entity retrieval results in the figures. The image-
to-entity retrieval results also help display some of the biases of the model.
One interesting application of image-to-entiy retrieval is for auto image
tagging.

Among the examples in Fig. 5, the wrong matching texts and entities are
understandable because they are still very semantically aligned with the input
image. We explain each case below:

  1. 1.

In the top image, all retrieved captions are correct. Among the retrieved
entities, there are a few incorrect results which show that the model has not
learned very accurately the visual appearance of receipt, hairbrush, calendar.
Images of toddler holding a hairbrush is common in the training set, which
must have made the model steered towards aligning hairbrush with something
that a toddler is holding.

  2. 2.

In the middle image, most matching captions are correctly retrieved except one
that is incorrect due to object counting. Counting the correct number of
objects is indeed a challenge for image-text matching model. The model also
mistakenly recognizes the kite as a plane.

  3. 3.

In the bottom image, the 1st caption is incorrect, but the model still ranks
it at the top due to multiple semantic information in the text are still
correct w.r.t. the image (_e.g_., young boy, living room, cat). All other
captions are correctly retrieved. The image-to-entity retrievals show the
concepts that the model does not grasp well.

Figure 5: Successful image-to-text and image-to-entity retrieval on MS-COCO.
In image-to-text retrieval, green denotes matching text according to the
ground truth of MS-COCO, while red denotes incorrect matching. In image-to-
entity retrieval, green and red denote correct and incorrect matching,
respectively, as judged subjectively by us.

We continue to explain the failure cases in Fig. 6 as following:

  1. 1.

In the top image, all of the retrieved captions are incorrect matchings as
determined by the ground truth data. However, we notice that the 2nd, 3rd
captions still correctly describe the image to a certain extent. This is a
weakness of the benchmark.

  2. 2.

In the middle image, the model must have wrongly associated skin with bikini,
hence why it retrieves captions with bikini at rank 4 and 5. In the entity
retrieval results, interestingly, we notice the model returns dental procedure
and dental work. We figure that the model must have aligned the action of
mouth opening with dental, hence why these two entities are retrieved in this
case.

  3. 3.

In the bottom image, this is again an example of where the retrieved captions
correctly describe the image, but because the ground truth data specify
otherwise, they are considered incorrect by the benchmark.

Figure 6: Failure cases of image-to-text and image-to-entity retrieval on
Flickr30K. In image-to-text retrieval, green denotes matching text according
to the ground truth of Flickr30K, while red denotes incorrect matching. In
image-to-entity retrieval, green and red denote correct and incorrect
matching, respectively, as judged subjectively by us.

Text-to-image retrieval results. We illustrate some text-to-image retrieval
results in Fig. 7. In both examples, our model is able to retrieve the correct
image at rank 1. The images from rank 2 to rank 5 all exhibit visual traits
that match partially with the input text.

Figure 7: Text-to-image retrieval on MS-COCO. For every text, we show the
top-5 retrieved images on MS-COCO. The image with the green tick mark is the
correct matching according to ground truth in the dataset.