# Graph Neural Networks in Vision-Language Image Understanding: A Survey

Henry Senior, Gregory Slabaugh, Shanxin Yuan, and Luca Rossi H. Senior, G.
Slabaugh, and S. Yuan are with the Digital Environment Research Institute,
Queen Mary University of London, United Kingdon. e-mail<EMAIL_ADDRESS>Rossi is with The Hong Kong Polytechnic University, Hong Kong.Manuscript
received April 19, 2005; revised August 26, 2015.

###### Abstract

2D image understanding is a complex problem within Computer Vision, but it
holds the key to providing human level scene comprehension. It goes further
than identifying the objects in an image, and instead it attempts to
understand the scene. Solutions to this problem form the underpinning of a
range of tasks, including image captioning, Visual Question Answering (VQA),
and image retrieval. Graphs provide a natural way to represent the relational
arrangement between objects in an image, and thus in recent years Graph Neural
Networks (GNNs) have become a standard component of many 2D image
understanding pipelines, becoming a core architectural component especially in
the VQA group of tasks. In this survey, we review this rapidly evolving field
and we provide a taxonomy of graph types used in 2D image understanding
approaches, a comprehensive list of the GNN models used in this domain, and a
roadmap of future potential developments. To the best of our knowledge, this
is the first comprehensive survey that covers image captioning, visual
question answering, and image retrieval techniques that focus on using GNNs as
the main part of their architecture.

###### Index Terms:

Graph Neural Networks, Image Captioning, Visual Question Answering, Image
Retrieval

##  I Introduction

Recent years have seen an explosion of research into Graph Neural Networks
(GNNs), with a flurry of new architectures being presented in top-tier machine
learning conferences and journals every year [1, 2, 3, 4, 5, 6]. The ability
of GNNs to learn in non-Euclidean domains makes them powerful tools to analyse
data where structure plays an important role, from chemoinformatics [7] to
network analysis [8]. Indeed, these models can also be applied to problems not
traditionally associated with graphs such as 3D object detection in LiDAR
point clouds [9] and shape analysis [10].

GNN-based approaches have gained increasing popularity for solving 2D image
understanding vision-language tasks, similar to other domains [11, 12, 13, 14,
15]. Whilst advances in this domain are discussed in [16], it is a wide
ranging survey. Our work focuses specifically on vision-language and therefore
covers these topics more extensively.

We view 2D image understanding as the high level challenge of making a
computer understand a two-dimensional image to a level equal to or greater
than a human. Models that enable this should be able to reason about the image
in order to describe it (image captioning), explain aspects of it (Visual
Question Answering (VQA), or find similar images (image retrieval). These are
all tasks that humans can do with relative ease, however, they are incredibly
difficult for deep learning models and require a large amount of data. These
tasks also fall under the category of vision-language problems, as they
require the model to have an understanding of both the image pixels and a
language (typically English) in which the models can express their
understanding. Whilst there is a plethora of techniques that have been applied
to these problems [17, 18, 19, 20, 21, 22, 23], this survey focuses on graph-
based approaches. There are a range of graphs that are applicable, but the
most widely used and understood is the semantic scene graph [24, 25]. This
graph is constructed of nodes representing visual objects and edges
representing the semantic relationships between them. The semantic graph as
well as further graph types are discussed in Section II-C.

Alongside a taxonomy of the graph types used across 2D image understanding
tasks, this paper contributes a much needed overview of these approaches.
Covering the three main tasks, we also include an overview of popular GNN
techniques as well as insights on the direction of future GNN work. In the
discussion section of this paper we argue that the increasingly popular
Transformer architecture [26] is actually a special case GNN [27]. We expand
upon this argument to suggest that GNNs should not be overlooked as they may
offer better inductive biases for a range of tasks.

Our main contributions are: 1) a taxonomy of the graph types used in 2D image
understanding tasks; 2) a comprehensive survey of GNN-based approaches to
common 2D image understanding tasks; 3) a roadmap of potential future
developments for the community to explore.

The remainder of this paper is organised as follows: Section II gives a
taxonomy of the tasks discussed and their corresponding datasets, as well as
an overview of the different graph types used throughout. Section III gives an
overview of the common GNN architectures used. It also briefly mentions
current and future research directions for GNNs and signposts appropriate
surveys. The main body of the paper is formed of Sections IV, V, and VI, which
detail GNN-based approaches to image captioning, VQA, and image retrieval,
respectively. We then conclude the paper with a three part discussion, with
Section VII-A covering the advantages that GNNs still offer despite the rapid
adoption of the Transformer architecture. This is followed by Section VII-B
which links the emerging field of latent diffusion and image generation to
image captioning. Finally, Section VII-C concludes the paper and provides
potential directions for future work.

##  II Background and Definitions

In this section, we outline the background required to view this survey in
context. We first briefly define a generic graph before outlining the taxonomy
of the field. Finally, we give an overview of the various graph types.

###  II-A 2D vision-language Tasks Taxonomies

Figure 1: 2D vision-language task taxonomy.

This paper follows the taxonomies of [28, 29, 30, 31] and joins them together
for a more complete overview of 2D vision-language tasks (see Figure 1). This
section gives a brief overview of the existing taxonomies and highlights the
sections of them this survey focuses on. It also highlights the main datasets
used for various tasks discussed in the paper, these are summarised in Table
I.

TABLE I: A table showing a summary of common datasets

Dataset | Main Task | Features
---|---|---
Visual Genome [32] | Mutli-Task | 108,000 images; 5.4 million region descriptions; 1.7 million question-answer pairs; scene graphs
COCO [33] | Image Captioning | 330,000 images with 5 human generated reference captions for training and validation sets
Flickr30K [34] | Image Captioning | 31,000 images each with 5 human generated reference captions
VQA 2.0 [35] | VQA | 265,000 images; Average of 5.4 questions per image each with 10 ground truth answers
FVQA [36] | VQA | 2,190 images; 5,826 questions; Knowledge base of 4,216 facts
OKVQA [37] | VQA | 14,000 questions, each with five ground truth answers, with knowledge extracted from wikipedia
TextVQA [38] | VQA | 28,000 images; 45,000 questions each with 10 ground truth answers
Text-KVQA [39] | VQA | 257,000 images; 1.3 million QA pairs; Inclusion of a knowledge base

Whilst individual vision-language tasks have their own unique datasets, they
are unified by the Visual Genome [32], an expansive dataset that provides
ground truths for a range of vision-language tasks. As the most generic
dataset, it has $33,877$ object categories and $68,111$ attribute categories.
At the time of its publication this was the largest and most dense dataset
containing image descriptions, objects, attributes, relationships, and
question answer pairs. Additionally, the Visual Genome also contains region
graphs, scene graphs, and question-answer pairs. This results in it being a
very wide ranging dataset with lots of applications in visual cognition tasks
such as scene graph generation [40] and VQA [41].

For image captioning, we follow [28] who identify three main approaches: 1)
retrieval-based captioning, 2) template-based captioning, and 3) deep
learning-based captioning. Retrieval-based captioning is built on the
assumption that for every image, a caption exists, and needs to be retrieved
from a bank of existing captions. It was the foundation of early image
captioning approaches [17] and yielded good results without the need for deep
learning. However, not all images may have appropriate captions. If the
captions are generic, they will only be able to describe aspects of an image
and may omit its most important feature. In contrast, template-based
captioning [42] uses a pre-defined caption format and uses object detection to
fill in the blanks. This approach is good for generating consistent captions,
but can result in captions that are unnatural and clearly generated by a
machine. Contemporary approaches to the task of image captioning are based on
deep learning models. Early work focused on a CNN encoder feeding an RNN-based
decoder [43], however more recent deep learning approaches have developed to
incorporate a wide variety of techniques including GNNs [25, 44] and
Transformers [45, 46]. In this survey, we focus specifically on deep learning
approaches to image captioning, and focus on graph-based approaches. Deep
learning approaches are typically trained on the COCO [33] or Flickr30k [34]
which contain a set of images accompanied by five human generated captions.

Taxonomies of VQA are usually defined through the lens of the datasets used by
the various tasks [29, 30]. Here we focus on 1) the standard VQA task of
answering a question about an image, 2) the fact-based VQA (FVQA) task of
answering questions that require external knowledge to answer, and 3) text-
VQA, the task of answering questions that require the model to read text in
the scene and combine it with visual data. Each of the various VQA tasks have
their own set of speicalised datasets. The original VQA dataset [35], and the
subsequently updated VQA 2.0 [47] dataset address the original task of
answering questions based on the visual information in the image. The FVQA
dataset [36] is built using images from ImageNet [48] and COCO [33] alongside
facts from DBPedia [49], ConceptNet [50], and WebChild [51]. The images have
three forms of visual concepts extracted from them using a range of models.
These visual concepts include objects (items identified in the image), scene
(scene level features such as room label), and actions. Question-answer pairs
were generated by human annotators who selected a visual concept and an
accompanying fact triplet which they used to generate a question. Finally, the
text-KVQA dataset [39] was built by compiling images from a Kaggle movie
poster challenge111https://www.kaggle.com/datasets/neha1703/movie-genre-from-
its-poster, [52], and Google Image search results from combining brand names
with postfixes such as “store” or “building.” This collection of images was
then given to human annotators who removed images that did not contain text of
brand names. The result is a dataset of 257K images with three groupings:
book, movie, and scene. Accompanying these images are 1.3 million question-
answer pairs. Each image grouping gets its own triplet-based knowledge base
from a relevant source: WikiData [53], IMBd, and [52] respectively.

Image retrieval spans multiple tasks, all of which make use of deep learning
in contemporary approaches. We follow the taxonomy of Alexander et al. [31]
and address the following sub tasks: text-based image retrieval, content-based
image retrieval, sketch-based retrieval, semantic-based retrieval, and
annotation-based retrieval. The number of datasets used for image retrieval
are vast and the community has not solidified around a single dataset in the
way image captioning has around COCO [33]. This presents a challenge when
making accurate comparisons between systems as the challenge presented by
different datasets varies complicating direct comparisons across datasets.
Whilst image retrieval specific datasets exist [54], there are papers [55, 56,
57] that make use of image captioning datasets [33, 34], showing the wide
range of varied datasets that exist for image retrieval.

###  II-B Fundamental Graph Theoretical Concepts

Undirected Graph. We define an undirected graph $G$ to be a tuple of sets $V$
and $E$, i.e., $G=(V,E)$. The set $V$ contains $n$ vertices (sometimes
referred to as nodes) that are connected by the edges in the set $E$, i.e., if
$v\in V$ and $u\in V$ are connected by an edge then $e_{v,u}\in E$. For an
undirected graph we have that $e_{v,u}=e_{u,v}$.

Directed Graph. A directed graph is a graph where the existence of $e_{v,u}$
does not imply the existence of $e_{u,v}$ as well. Let $A$ be the $n\times n$
binary adjacency matrix such that $A_{v,u}=1$ if $e_{v,u}\in E$. Then it
follows that $A$ is asymmetric (symmetric) for directed (undirected) graphs.
More in general, $A$ can be a real-valued matrix, where the value of $A_{v,u}$
can be interpreted as the strength of the connection between $v$ and $u$.

Neighbourhood. The neighbourhood $\mathcal{N}(v)$ of a vertex $v\in V$ is the
subset of nodes in $V$ that are connected to $v$. The neighbour $u$ can be
either directly connected to $v$, i.e., $(v,u)\in E$, or it can be indirectly
connected by traversing $r$ edges from $v$ to $u$. Note that some definitions
include $v$ itself as part of the neighbourhood.

Complete Graph. A complete graph is one (directed or undirected) where for
each vertex, there is an edge connecting it to every other vertex in the set
$V$. A complete graph is therefore a graph with the maximum number of edges
for a given number of nodes.

Multipartite Graph. A multipartite graph (also known as $K$-partite graph) is
a graph where the nodes can be separated into $K$ different sets. For scene
understanding tasks, this allows for a graph representation where one set of
nodes represent objects and another represents relationship between objects.

Multimodal Graph. A multimodal graph is one with nodes that have features from
different modalities. This approach is commonly used in VQA where the image
and text modalities are mixed. Multimodal graphs enable visual features to co-
exist in a graph with word embeddings.

###  II-C Common Graph Types in 2D vision-language Tasks

This section organises the various graph types used across all three tasks
discussed in the survey. Some graphs, such as the semantic and spatial graphs,
are used across all tasks [25, 41, 56], while others are more domain specific,
like the knowledge graph [58, 39]. Figure 2 shows a sample image from the COCO
dataset [33] together with various types of graphs that can be used to
describe it. This section, alongside the figure, is organised so that graph
that represent a single image and graphs that represent portions the dataset
are grouped together.

Semantic Graph. Sometimes referred to as a scene graph, a semantic graph
(shown in Figure 2c) is a one that encapsulates the semantic relationships
between visual objects within a scene. Across the literature, the terms
‘semantic graph’ and ‘scene graph’ are used somewhat interchangeably,
depending on the paper. However, in this survey we use the term ‘semantic
graph’ because there are many ways to describe a visual scene as a graph,
whereas the ‘semantic graph’ label is more precise about what the graph
represents. Semantic graphs come in different flavours. One approach is to
define a directed graph with nodes representing visual objects extracted by an
object detector such as Faster-RCNN [59] and edges representing semantic
relationships between them. This is the approach of Yao et al. [25], where,
using a dataset such as Visual Genome [32], a model predicts the semantic
relationships to form edges in the graph. Alternatively, the semantic graph
can be seen as a multipartite graph [60, 61, 44, 62] (shown in Figure 2d),
where attribute nodes describe the object nodes they are linked to. They also
change the way relationships are represented by using nodes rather than edge
features. This yields a semantic graph with three node types: visual object,
object attribute, and inter-object relationship. This definition follows that
of the ‘scene graph’ defined by Johnson et al. [24]. Finally, another form of
semantic graph exists, the textual semantic graph[44, 63] (shown in figure
2f). Unlike visual semantic graphs, textual ones are not generated from the
image itself but rather its caption. Specifically, the caption is parsed
through the Stanford Dependency Parser [64], a widely used [65, 66]
probabilistic sentence parser. Given a caption, the parser will return its
grammatical structure, identifying components such nouns, verbs, and
adjectives and marking the relationship between them. This is then modified
from a tree into a graph, following the techniques outlined in [67].

Spatial Graph. Yao et al. [25] define a spatial graph (Figure 2g) as one
representing the spatial relationship between objects. Visual objects detected
by an object detector form nodes, and the edges between the nodes represent
one of 11 pre-defined spatial relationships that may occur between the two
objects. These include inside (labelled ‘1’), cover (labelled ‘2’), overlap
(labelled ‘3’), and eight positional relationships (labelled ‘4’-‘11’) based
on the angle between the centroid of the two objects. These graphs are
directional but will not always be complete as there are cases where two
objects have a weak spatial relationship and are therefore not connected by an
edge in the spatial graph. Guo et al. [61] define a graph of a similar nature
known as a geometry graph. It is defined as an undirected graph that encodes
relative spatial positions between objects with an overlap and relative
distance that meet certain thresholds.

Hierarchical Spatial. These graphs build on from the spatial graph but the
relationships between nodes focus on the hierarchical nature of the spatial
relationship between the detected objects within an image. Yao et al. [68]
propose to use a tree (i.e., a graph where each pair of nodes is connected by
a single path) to define a hierarchical image representation. An image
($\mathcal{I}$) is first divided into regions using Faster-RCNN [59]
($\mathcal{R}=\\{r_{i}\\}^{K}_{i=1}$) with each region being further divided
into instance segmentations ($\mathcal{M}=\\{m_{i}\\}^{K}_{i=1}$). This gives
a three-layer tree structure
($\mathcal{T}=(\mathcal{I},\mathcal{R},\mathcal{M},\mathcal{E}_{tree})$, where
$\mathcal{E}_{tree}$ is the set of connecting edges) to represent the image,
as shown in Figure 2e. He et al. [46] use a hierarchical spatial graph, with
relationships representing ‘parent’, ‘child’, and ‘neighbour’ relationships
depending on the intersection over union of the bounding boxes.

Similarity Graph. The similarity graph (Figure 2h) proposed by Kan et al.[69]
(referred to as a semantic graph by the authors) is generated by computing the
dot product between two visual features extracted by Faster-RCNN [59]. The dot
products are then used to form the values of an adjacency matrix $A$ as the
operation captures the similarity between two vectors, the higher the dot
product, the closer the two vectors are. Faster-RCNN extracts a set of $n$
visual features, where each feature $x(v)$ is associated to a node $v$ and the
value of the edge between two nodes $v$ and $u$ is given by
$A_{u,v}=\sigma\left(x(v)^{T}Mx(u)\right)$, where $\sigma(\cdot)$ is a non-
linear function and $M$ is a learnt weight matrix. The authors of [69] suggest
that generating the graph this way allows for relationships between objects to
be discovered in a data-driven manner, rather than relying on a model trained
on a dataset such as the Visual Genome [32].

Image Graphs/$K$-Nearest Neighbour Graph. In their 2021 image captioning work,
Dong et al. [70] construct an image graph by converting images into a latent
feature space by averaging the object vectors output by feeding the image into
Faster-RCNN [59]. The $K$ closest images from the training data or search
space in terms of $l_{2}$ distance are then turned into an undirected complete
graph, shown in Figure 2i. This is a similar approach used by Liu et al. [71]
with their $K$-nearest neighbour graph.

Topic Graph. Proposed by Kan et al. [69], the topic graph is an undirected
graph of nodes representing topics extracted by GPU-DMM [72]. Topics are
latent features representing shared knowledge across the entire caption set.
Modelling them as a graph, as shown in Figure 2j, with edges computed by
taking the dot product of the two nodes, allows the modelling of knowledge
represented in the captions.

Region Adjacency Graph. Defined in [73], a Region Adjacency Graph uses a
superpixel segmentation. Superpixels form the nodes of the graph and edges are
added to connect adjacent region pairs. Edges are then weighted to represent
how compatible the two adjacent regions are.

Knowledge Graph.  A knowledge graph, or fact graph, is a graph-based
representation of information. Whilst there is no agreed structure of these
graphs [74], they typically take the form of triplets. They are used in a wide
variety of tasks to provide the information needed to ”reason”. Hence,
knowledge graphs enable the FVQA task.

(a) COCO training image 391895 (Cropped)

(b) Image with object detection labels

(c) A semantic graph

(d) A multipartite semantic graph

(e) An image tree representation

(f) A textual semantic graph representation of the caption “A man riding on
the back of a motorcycle”

(g) A spatial graph representation

(h) A similarity graph representation

(i) An image graph representation

(j) A topic graph representation

Figure 2: A visual comparison of the various graph types used across vision-
language tasks. Best viewed in colour.

##  III An Overview of Graph Neural Networks

Over the past years a large number of GNN architectures have been introduced
in the literature. Wu et al. [75] proposed a taxonomy containing four distinct
groups: recurrent GNNs, convolutional GNNs, autoencoder GNNs, and spatial-
temporal GNNs. The applications discussed in this paper mostly utilise
convolutional GNNs, for a comprehensive overview of other architectures
readers are directed to [75]. GNNs, especially traditional architectures such
as Graph Convolutional Network, have a deep grounding in relational inductive
biases [27]. They are built on the assumption of homophily, i.e. that
connected nodes are similar.

###  III-A Graph Convolutional Networks (GCNs)

One common convolutional GNN architecture is the Message Passing Neural
Networks (MPNNs) proposed by Gilmer et al. Although this architecture has been
shown to be limited [76], it forms a good abstraction of GNNs.

Gilmer et al. describe MPNNs as being comprised of a message function, update
function, and readout function. These functions will vary depending on the
application of the network, but are learnable, differentiable, and permutation
invariant. The message and update functions will run for a number of time
steps $T$, passing messages between connected nodes of the graph. These are
used to update the hidden feature vectors of the nodes, which are then used to
update the node feature vector, which in turn is used in the readout function.

The messages are defined as

$\bar{m}^{(t+1)}_{v}=\sum_{u\in\mathcal{N}(v)}M_{t}(\bar{h}^{(t)}_{v},\bar{h}^{(t)}_{u},\bar{e}_{v,u})\,,$
(1)

where a message for a node at the next time step $\bar{m}^{(t+1)}_{v}$ is
given by combining its current hidden state $\bar{h}^{(t)}_{v}$ with that of
its neighbour $\bar{h}^{(t)}_{u}$ and any edge feature $\bar{e}_{v,u}$ in a
multilayer perceptron (MLP) $M_{t}(\cdot)$. Given that a message is an
aggregation of all the connected nodes, the summation acts over the nodes
connected to the node $u\in\mathcal{N}(v)$, i.e., the neighbourhood of $v$.

These messages are then used to update the hidden vectors by combining the
node current state with the message in an MLP $U_{t}$.

$\bar{h}^{(t+1)}_{v}=U_{t}(\bar{h}^{t}_{v},\bar{m}^{(t+1)}_{v})$ (2)

Once the message passing phase has run for $T$ time steps, a readout phase is
then conducted using a readout function, $R(\cdot)$. This stage makes use of
an MLP that considers the updated feature vectors of nodes on the graph to
produce a prediction and is defined as:

$\hat{y}=R(\\{\bar{h}^{T}_{v}|\bar{v}\in G\\})$ (3)

In order to make the GCN architecture scale to large graphs, the GraphSAGE
[77] architecture changes the message function. Rather than taking messages
from the entire neighbourhood of a node, a random sample is used. This reduces
the number of messages that require processing, resulting in an architecture
that works well on large graphs.

###  III-B Gated Graph Neural Networks

The core idea behind the Gated Graph Neural Network (GGNN) [78] is to replace
the update function from the message passing architecture (Equation 2) with a
Gated Recurrent Unit (GRU) [79]. The GRU is a recurrent neural network with a
update and reset gates that controls which data can flow through the network
(and be retained) and which data cannot (and therefore be forgotten).

$\bar{h}^{(t+1)}_{v}=GRU(\bar{h}^{(t)}_{v},\sum_{w\in\mathcal{N}(v)}\mathbf{W}\bar{h}^{(t)}_{w})\,.$
(4)

The GGNN also replaces the message function from Equation 1 with a learnable
weight matrix. Using the GRU alongside back-propagation through time enables
the GGNN to operate on series data. However, due to the recurrent nature of
the architecture, it can become unfeasible in terms of memory to run the GGNN
on large graphs.

###  III-C Graph Attention Networks (GATs)

Following on from the multi-head attention mechanism of the popular
Transformer architecture [26], Graph Attention Networks (GATs) [80] extend the
common GCN to include this attention attribute. Using an attention function,
typically modelled by an MLP, the architecture calculates an attention
weighting between two nodes. This process is repeated $K$ times using $K$
attention heads in parallel. The attention scores are then averaged to give
the final weights.

The self-attention is computed by a function $a(h^{t}_{v},h^{t}_{w})$
(typically an MLP) that attends to a node and one of its neighbours. Once
every node pairing in the graph has their attention computed, the scores are
passed through a softmax function to give a normalised attention coefficient.
This process is then extended to multi-head attention by repeating the process
across $K$ different attention heads, each with different initialisation
weights. The final node representation is achieved by concatenating or
averaging (represented as $\|$) the $K$ attention heads together.

$\bar{h}_{v}^{(t+1)}=\Bigg{\|}^{K}_{k=1}\sigma(\sum_{w\in\mathcal{N}(v)}\alpha_{v,w}^{(k)}\mathbf{W}^{(k)}\bar{h}_{w})$
(5)

###  III-D Graph Memory Networks

Recent years have seen the development of Graph Memory Networks, which can
conceptually be thought of as models with an internal and external memory.
When there are multiple graphs overlapping the same spatial information, as in
[81], the use of some form of external memory can allow for an aggregation of
node updates and the graph undergoes message passing. This essentially allows
for features from multiple graphs to be combined in some way that goes beyond
a more simplistic pooling operation. In the case of Khademi [81], two graphs
are constructed across the same image but may have different nodes. These
graphs are updated using a GGNN. An external spatial memory is constructed to
aggregate information from across the graphs as they are updated, using a
neural network with an attention mechanism. The final state of the spatial
memory is used to perform the final task.

###  III-E Modern Graph Neural Network Architectures

In recent years, the limits of message passing GNNs have become increasingly
evident, from their tendency to oversmooth the input features as the depth of
the network increases [82] to their unsatisfactory performance in heterophilic
settings [83], i.e., when neighbouring nodes in the input graphs are
dissimilar. Furthermore, the expressive power of GNNs based on the message
passing mechanism has been shown to be bounded by that of the well-known
Weisfeiler-Lehman isomorphism test [76], meaning that there are inherent
limits to their ability to generate different representations for structurally
different input graphs.

Motivated by the desire to overcome these issues, researchers have now started
looking at alternative models that move away from standard message passing
architectures. Efforts in this direction include, among many others, higher-
order message passing architectures [84], cell complexes networks [85],
networks based on diffusion processes [86, 2, 83]. To the best of our
knowledge, the application of these architectures to the 2D image
understanding tasks discussed in this paper has not been explored yet. As
such, we refer the readers to the referenced papers for detailed information
on the respective architectures.

##  IV Image Captioning

Image captioning is the challenging task of producing a natural language
description of an image. Outside of being an interesting technical challenge,
it presents an opportunity to develop accessibility technologies for severely
sight impaired (formally ‘blind’) and sight impaired users (formally ‘visually
impaired’ 222The UK Department of Health and Social Care adopted the more
inclusive phrasing around 2017). Additionally, it has applications in problems
ranging from image indexing [87] to surveillance [69]. There are three forms
of image captioning techniques: 1) retrieval-based captioning, where a caption
is retrieved from a set of existing captions, 2) template-based captioning,
where a pre-existing template is filled in using information extracted from
the image, 3) and Deep Learning-based image captioning, where a neural network
is tasked with generating a caption from an input image. We propose to refine
this taxonomy to differentiate between GNN-based approaches and more
traditional Deep Learning powered image captioning. The following section
details the GNN-based approaches to image captioning, of which there have been
a number of in recent years. Figure 3 illustrates the structure of a generic
GNN-based image captioning architecture.

Figure 3: An abstract overview of GNN-based image captioning architectures
discussed in this section. Most architectures extract image features and use
them to construct at least one graph to represent the image. Some papers [70,
69] build higher level graphs at an image level rather than an object level. A
GNN is then applied to these graphs and the resulting features are fed into a
language generator that creates an appropriate caption for the image.
Traditionally this was an LSTM, but more recently the trend is to use
Transformers [26]. Best viewed in colour.

GNN-based approaches to image captioning all follow the traditional Encoder-
Decoder-based approach common in Deep Learning image captioning techniques.
Images first undergo object detection, the output of which is used to create
an encoding. These encodings are then decoded, traditionally with a long
short-term memory network (LSTM), into a caption. Through incorporating GNNs,
researchers have been able to enhance the encoded image representation by
incorporating spatial and semantic information into the embeddings.

As the task of image captioning has developed over time, so have the
evaluation metrics used to assess the performance of proposed architectures.
Originally, image captioning relied heavily on machine translation evaluation
techniques such as BLEU [88], ROUGE [89], and METEOR [90] as no image
captioning specific metric existed. However, this changed with the
introduction of both CIDEr [91] and SPICE [67]. The performance metrics are
detailed in Table II.

TABLE II: A table detailing the different image captioning performance metrics

Metric | Original Field | Based On |  Description
---|---|---|---
BLEU | Machine translation | Precision |  Based on a modified n-gram precision where the reference word is exhausted after a matching candidate word is identified. BLEU favours captions that are a similar in length to the reference caption.
ROUGE | Machine text summarisation | Recall |  Built with four components: an n-gram recall between the candidate and reference set, a comparison of the longest common sub-sequence, a comparison of the weighted longest common sub-sequence, and finally, the skip-bigram co-occurrence statistic.
METEOR | Machine translation | $F_{mean}$ |  Uses the harmonic mean of the precision and recall between candidate caption and reference captions.
CIDEr | Image Captioning | n-grams |  The metric is based on a number of intuitions. Firstly, that if an n-gram is not present in the reference captions it should not appear in the candidate caption. Secondly, that it should encode how often n-grams present in the candidate captions are present. And finally, n-grams that occur across all the reference captions should be assigned a lower weighting as they will be things like articles and have little to no important information.
SPICE | Image Captioning | Textual Semantic Tree |  Reference and candidate captions are converted into textual scene graphs to compare the semantic makeup of the captions.

The first architecture to use a GNN to improve image captioning was by Yao et
al. [25]. In their work, they propose the use of a GCN to improve the feature
embeddings of objects in an image. They first start by applying a Faster R-CNN
object detector [59] to the image in order to extract feature vectors
representing objects. These feature vectors are then used to create two
graphs: a bidirectional spatial graph encoding spatial relationships between
objects and a directed semantic graph which encodes the semantic relationships
between objects. A GCN is then applied to both graphs before the enhanced
features of the graphs undergo mean pooling. They are then decoded by an LSTM
into a caption. As the whole graphs are used to inform the caption generation,
it may lead to scenarios where dense graphs lead to redundant or low value
information being included in the caption.

Zhong et al. [60] focus solely on a semantic scene graph and address the
problem of which nodes and edges to include in the final caption. This is
challenging for scenes containing a lot of detected objects as the semantic
scene graphs can become relatively large. The problem is addressed by
decomposing the semantic graph into various subgraphs that cover various parts
of the image. They are then scored using a function trained to determine how
closely the subgraph resembles the ground truth caption. This enables the
selection of subgraphs from the main scene graph that will go on to generate
useful captions. The starting semantic graph is generated by MotifNet [92] (a
common off-the-shelf semantic graph generator). Zhong et al. [60] make use of
a GCN to aggregate neighbourhood information of the proposed sub-graph. Unlike
Yao et al., the authors of [60] use only a semantic graph. They focus on the
link between the language and semantic graph and do not make use of spatial
information.

Another work that makes use of the semantic graph is that of Song et al. [93].
They investigate how both implicit and explicit features can be utilised to
generate accurate and high quality image captions. The authors define implicit
features as representing global interactions between objects and explicit
features as those defined on a semantic graph. For the latter, rather than
using multiple graphs, [93] only uses a single semantic graph. However, rather
than predicting the graph directly via MotifNet [92] as in other works [60],
its construction starts with a spatial graph. After object detection, a fully
connected directed graph is generated between the objects (with nodes being
represented by the object feature vector). The edges of this graph are then
whittled away in a two step process. Firstly, edges between objects that have
zero overlap (measured as intersection over union) and an $l_{2}$ distance
less than the longest side of either objects bounding box are removed. The
remaining edges are used to determine which object pairs have their
relationship detected by MotifNet [92]. Those relationships with a high enough
probability are kept whilst the others are removed. This results in a semantic
graph that indirectly contains spatial information, going beyond the semantic
graph of [60]. The final graph is then processed by a GGNN, the output of
which is a representation of the explicit features. The implicit features are
generated by a Transformer encoder [26]. The entire image, alongside the
regions within the detected object bounding boxes are encoded. These features
are then used alongside those of the explicit features as input to an LSTM
language decoder that is used to generate the final caption. The work
demonstrates the successes possible when using GNNs alongside Transformers,
using their different inductive biases to best model different interactions
(see Table III). However, both the implicit and explicit relationships remain
local to a single image. Further work could consider how often certain
relationships occur over the entire dataset.

Guo et al. [61] took a very similar approach to Yao et al. [25] with their
work, utilising a dual graph architecture containing a semantic and spatial
graph. However, they make the observation that images can be represented by a
collection of Visual Semantic Unit (VSU) vectors, which represent an object,
its attributes, and its relationships. These VSUs are combined into a semantic
graph that models relationships as nodes rather than edge features and adds
attribute nodes conencted to objects, thus making it multipartite. Doing so
gives the graph a closer resemblance to the captions it will go on to generate
as objects map to nouns, relationships to verbs and prepositions, and finally
attributes to adjectives. The authors argue that this approach allows the
model to explicitly learn relationships and model them directly. As argued in
[61], a scene graph of an image has a close mapping to the image caption.
Nodes representing objects map directly to nouns, edge features (in the case
of [25]) or nodes (in the case of [61]) that encode relationships map clearly
to prepositions, and nodes representing attributes map to adjectives. This
strong relationship between the graph structure generated by the encoder and
the final sentence outputted by the decoder further supports the use of the
image-graph-sentence architecture used by many image captioning systems.

Zhou et al. [62] use an LSTM alongside a Faster-RCNN [59] based image feature
extractor, with the addition of a visual self-attention mechanism. The authors
make use of a multipartite semantic scene graph, following the style of [24,
61]. Specifically, they propose to use three GCNs to create context aware
feature vectors for each of the object, attribute, and relationship nodes. The
resulting context aware nodes undergo fusion with the self attention maps,
enabling the model to control the granularity of captions. Finally, the
authors test two methods of training an LSTM-based language generator, the
first being a traditional supervised approach with cross entropy loss, the
second being a reinforcement learning-based approach that uses CIDEr [91] as
the reward function. By utilising context dependent GCNs in their
architecture, to specifically account for the object, attribute, and
relationship nodes, SASG is able to achieve competitive results when compared
with similar models, as shown in Table III.

SGAE (Scene Graph Auto-Encoder) is another paper to make use of a multipartite
semantic graph. In the paper, Yang et al. [44] take a caption and convert it
into a multipartite textual semantic graph using a similar process to that of
the SPICE metric [67] (detailed further in Table II). The nodes of the graph
are converted to word embeddings which are then converted into feature
embeddings by way of a GCN, with each node type being given its own GCN with
independent parameters. These feature embeddings are then combined with a
dictionary to enable them to be re-encoded before they are used to generate a
sentence. The dictionary weights are updated via back-propagating the cross
entropy loss from the sentence regeneration. By including a dictionary, the
authors are able to learn inductive biases from the captions. This allows
generated captions to go from “man on motorcycle” to “man riding motorcycle”.
When given an image, SGAE generates a multipartite visual semantic graph,
similar to [24, 61], using Faster-RCNN [59] and MotifNet [92]. These visual
features are then combined with their word embeddings through a multi-modal
GCN and then re-encoded using the previously learnt dictionary. These features
are then used to generate the final sentence.

Rather than utilising multiple graphs, Wang et al. [94] instead use a single
fully connected spatial graph with an attention mechanism to learn the
relationships between different regions. This graph is formed of nodes that
represent the spatial information of regions within the image. Once formed, it
is passed through a GGNN [78] to learn the weights associated with the edges.
Once learnt, these edge weights correspond to the probability of a
relationship existing between the two nodes.

The work of Yao et al. [68], following on from their GCN-LSTM [25], presents
an image encoder that makes use of a novel HIerarchy Parsing (HIP)
architecture. Rather than encoding the image in a traditional scene graph
structure like most contemporary image captioning papers [25, 60, 70], Yao et
al. [68] take the novel approach of using a tree structure (discussed in
Section II-C), exploiting the hierarchical nature of objects in images. Unlike
their previous work which focused on the semantic and spatial relationships,
this work is about the hierarchical structure within an image. This
hierarchical relationship can be viewed as a combination of both semantic and
spatial information - therefore merging the two graphs used previously. The
feature vectors representing the vertices on the tree are then improved
through the use of Tree-LSTM [95]. As trees are a special case graph, the
authors also demonstrate that their previous work GCN-LSTM [25] can be used to
to create enriched embeddings from the tree before decoding it with an LSTM.
They demonstrate that the inclusion of the hierarchy passing improves scores
on all benchmarks when compared with GCN-LSTM [25], which does not use
hierarchical relationships.

The work of He et al. [46] build on the idea of a hierarchical spatial
relationships proposed by Yao et al.[68]. However, rather than use a tree to
represent these relationships, they use a graph with three relationship types:
parent, neighbour, and child. They then propose a modification to the popular
Transformer layer to better adapt it to the task of image processing. After
detecting objects using Faster-RCNN [59], a hierarchical spatial relationship
graph is constructed. Three adjacency matrices are then built from this graph
to model the three relationship types ($\Omega_{p},\Omega_{n},\Omega_{c}$
respectively). The authors modify the Transformer layer so that rather compute
self-attention across the whole spatial graph, there is a sub-layer for each
relationship type. Each sub-layer processes the query $Q$ with its own key
$K_{i}$ and value $V_{i}$ with the modified attention mechanism:

$Attention(Q,K_{i},V_{i})=\Omega_{i}\odot
Softmax\left(\frac{QK_{i}^{T}}{\sqrt{d}}\right)V_{i}$ (6)

Where $\odot$ is the Hadamard product and $i$ refers to the relationship type
$i\in\\{parent,neighbour,child\\}$. Using the Hadamard product essentially
zeroes out the attention between regions whose relationship is not being
processed by that sub-layer. The resulting encodings are decoded by an LSTM to
produce captions.

Like [46], the $\mathcal{M}2$ meshed memory Transformer proposed by Cornia et
al. [45] also makes use of the increasingly popular Transformer architecture
[26]. Unlike other papers [25, 68, 44, 46] which make use of some predefined
structure on extracted image features (spatial graph, semantic graph, etc),
$\mathcal{M}2$ uses stacks of self-attention layers across the set of all the
image regions. The standard key and values from the Transformer are edited to
include the concatenation of learnable persistent memory vectors. These allow
the architecture to encode a-priori knowledge such as ‘eggs’ and ‘toast’ make
up the concept ‘breakfast’. When decoding the output of the encoder, a stack
of self-attention layers is also used. Each decoder layer is connected via a
gated cross attention mechanism to each of the encoder layers, giving way to
the “meshed” concept of the paper. The output of the decoder block is used to
generate the final output caption.

The authors of [69] propose using a novel similarity (referred to as a
semantic in the paper) and topic graphs. Built on dot product similarity, the
graphs are produced without the requirement of graph extraction models such as
MotifNet [92]. Rather, a set of vertices
$V=\\{v_{i}\in\mathbb{R}^{d_{obj}}\\}^{n_{obj}}_{i=1}$ are extracted as ResNet
features from a Faster-RCNN object detector [59]. Edges in the adjacency
matrix are then populated using the dot product between the feature vectors in
$V$ with $a_{ij}=\sigma(v_{i}^{T}Mv_{j})$. Once both graphs have been
constructed, a GCN is applied to both in order to enrich the nodes with local
context. A graph self-attention mechanism is then applied to ensure nodes are
not just accounting for their immediate neighbours. The improved graphs are
then decoded via an LSTM to generate captions.

Following [25], Dong et al. [70] use a spatial graph to show a directed
relationship between detected objects within the input image. Locally, object
features are extracted by a CNN to associate a vector to each vertex of the
spatial graph. This process is completed for each image in the dataset. In
addition to this graph, the authors introduce an image level graph.
Specifically, each image is represented by a feature vector that is the
average of its associated set of object feature vectors. The image graph for a
corresponding image is formed as a fully connected undirected graph of the $K$
images whose $l_{2}$ distance is the closest to the input image. Both the
local spatial graph and the more global image level graph are processed by
GCNs to create richer embeddings that can be used for caption generation. This
approach is shown to work extremely well, with Dual-GCN achieving
outperforming comparable models in the BLEU, METEOR, and ROGUE metrics (see
Table III).

TABLE III: A table showing the model details and various benchmark results of
selected image captioning models trained on the COCO [33] dataset using the
Karpathy split [43]. Bold: Best score.

Model | Graph Types | Architecture(s) | Language Generator | BLEU-1 | BLEU-4 | METEOR | ROGUE | CIDEr | SPICE
---|---|---|---|---|---|---|---|---|---
GCN-LSTM [25] | Spatial, Semantic | GCN | LSTM | 80.9 | 38.3 | 28.6 | 58.5 | 128.7 | 22.1
HIP [68] | Hierarchical Spatial (Tree) | GCN | LSTM | - | 39.1 | 28.9 | 59.2 | 130.6 | 22.3
SGAE [44] | Multipartite Textual, Multipartite Semantic | GCN, multi-modal GNN | LSTM | 80.8 | 38.4 | 28.4 | 58.6 | 127.8 | 22.1
VSUA [61] | Multipartite Semantic, Spatial | GCN | LSTM | - | 38.4 | 28.5 | 58.4 | 128.6 | 22.0
ARL [94] | Spatial | GGNN | LSTM | 75.9 | 35.8 | 27.8 | 56.4 | 111.3 | -
SUB-GC [60] | Semantic | GCN | LSTM | 76.8 | 36.2 | 27.7 | 56.6 | 115.3 | 20.7
$\mathcal{M}^{2}$ [45] | - | Transformer | Transformer | 80.8 | 39.1 | 29.2 | 58.6 | 131.2 | 22.6
Image Transformer [46] | Hierarchical Spatial | Transformer | LSTM | 80.8 | 39.5 | 29.1 | 59.0 | 130.8 | 22.8
Topic [69] | Similarity | GCN | LSTM | - | 39.2 | 29.1 | 59.0 | 129.5 | 22.6
Dual-GCN [70] | Spatial, Image | (Dual) GCN | Transformer | 82.2 | 39.7 | 29.7 | 59.7 | 129.2 | -
EIVRN [93] | Spatial, Semantic | GGNN | Transformer | - | 39.4 | 29.3 | 59.1 | 131.9 | 22.8
SASG [62] | Birpartite Semantic | GCN | LSTM | 81.8 | 38.9 | 29.2 | 59.4 | 128.9 | 25.0

##  V Visual Question Answering

VQA is the challenging task of designing and implementing models that are able
to answer natural language questions about a given image. These answers can
range from simple yes/no to more natural, longer form answers. Questions can
also vary in complexity. As the field has developed, more specific VQA tasks
have emerged. The first to emerge was FVQA, sometimes known as Knowledge
Visual Question Answering (KVQA), where external knowledge sources are
required to answer the questions. Another task that has emerged is Textual
VQA, where the models must understand the text within the scene in order to
generate answers. All three tasks have their own datasets [35, 32, 38, 36, 39]
and have an active community developing solutions [35, 65, 81].

###  V-A VQA

Originally proposed in [35], VQA has developed beyond simple ‘yes’ or ‘no’
answers to richer natural language answers. A common thread of work is to
leverage the multi-modal aspect of VQA and utilise both visual features from
the input image and textual features from the question [65, 81, 66].

One of the first works in VQA to make use of GNNs was that of Teney et al.
[65]. Their work is based on the clip art focused dataset [35]. Their model
takes a visual scene graph as input alongside a question. The question is then
parsed into a textual scene graph using the Stanford Dependency Parser [64].
These scene graphs are then processed independently using a GGNN [78] modified
to incorporate an attention mechanism. The original feature vectors are then
combined using an attention mechanism that reflects how relevant two nodes
from the scene graphs are to one another.

Khademi [81] takes a multimodal approach to VQA by using dense region captions
alongside extracted visual features. Given a query and input image, the model
will first extract visual regions using a Faster-RCNN object detector and
generated a set of features using ResNet and encoding the bounding box
information into these features. An off-the-shelf dense region captioning
model is also used to create a set of captions and associated bounding boxes.
The captions and bounding box information are encoded using a GRU. Each set of
features is turned into a graph (visual and textual respectively) with
outgoing and incoming edges existing between features if the Euclidean
distance between the centre of the normalised bounding boxes is less than
$\gamma=0.5$. Both graphs are processed by a GGNN with updated features being
used to update an external spatial memory unit - thus making the network a
Graph Memory Network (described in Section III-D). After propagating the node
features, the final state of the external spatial memory network is turned
into a complete graph using each location as a node. This final graph is
processed by a GGNN to produce the final answer. The multimodal approach
presented in this paper is shown to be highly effective when compared to
similar VQA methods. This approach is shown to work extremely well in
benchmarks, with the proposed MN-GMN architecture [81] performing favourably
with comparable models (Table IV).

MORN [66] is another work that focuses on capturing the complex multi-modal
relationships between the question and image. Like many recent works in Deep
Learning, it adopts the Transformer [26] architecture. Built with three main
components, the model first creates a visual graph of the image starting from
a fully connected graph of detected objects and a GCN is used to aggregate the
visual features. The second part of the model creates a textual scene graph
from the input question. Both graphs are merged together by the final
component of the model, a relational multi-modal Transformer, which is used to
align the representations.

Sharma et al. [96] follow the vision-language multi-modal approach but diverge
from the use of a textual semantic graph and instead opt to use word
embeddings. The authors utilise a novel GGNN-based architecture that processes
an undirected complete graph of nodes representing visual features. Nodes are
weighted with the probability that a relationship occurs between them. In line
with other VQA work [81], the question is capped to $14$ words, with each one
being converted into GloVe embeddings [97]. Questions with fewer than $14$
words are padded with zero-vectors. A question embedding is then generated
using a GRU applied to the word embeddings. An LSTM-based attention mechanism
considers both the question vector and the visual representations making up
the nodes of the scene graph. This module considers previously attended areas
when exploring new visual features. Finally, an LSTM-based language generator
is used to generate the final answer. Another work to forgo using a textual
scene graph, Zhang et al. [41] make use of word vectors to embed information
about the image into a semantic graph. Using a GNN, they are able to create
enriched feature vectors representing the nodes, edges, and an image feature
vector representing the global state. They include the question into the image
feature by averaging the word vectors, which enables the GNN to reason about
the image. Whilst both [96] and [41] yield good results, by only using word or
sentence level embeddings and not using a textual scene graph, they fail to
model relationships in the textual domain. This therefore removes the ability
for the models to reason in that domain alone.

Both Li et al. [98] and Nuthalapati et al. [99] take a different route to the
established multi-modal approach and instead use different forms of visual
information. Li et al. [98] take inspiration from [25] and make use of both
semantic and spatial graphs to represent the image. In addition to these
explicit graphs, they also introduce an implicit graph, i.e., a fully
connected graph between the detected objects with edge weights set by a GAT.
The relation-aware visual features are then combined with the question vector
using multi-modal fusion. The fused output is then used to predict an answer
via an MLP.

Nuthalapati et al. [99] use a dual scene graph approach, using both visual and
semantic graphs. These graphs are merged into a single graph embedding using a
novel GAT architecture [80] that is able to attend to edges as well as nodes.
The graphs are enriched with negative entities that appear in the question but
not the graph. Pruning then takes place to remove nodes and edges that are $K$
hops away from features mentioned in the question. A decoder is then used to
produce an answer to the inputted question.

TABLE IV: A table showing the model details and VQA [35] Test-Dev results of
selected VQA models. Bold: Best score.

Model | Graphs Used | Architecture | Overall | Y/N | Number | Other | Test-Std
---|---|---|---|---|---|---|---
GraphVQA [65] (Abstract Scenes only) | Visual and Textual Semantic | GCN | 70.42 | 81.26 | 76.47 | 56.28 | -
ReGAT [98] | Semantic, Spatial | GAT | 70.27 | 86.08 | 54.42 | 60.33 | 70.59
MN-GMN [81] | Visual and Textual Semantic | Graph Memory Network | 73.2 | 88.2 | 56 | 64.2 | 73.5
Sharma et al [96] | Semantic | GGNN | 67.96 | 84.12 | 46.12 | 58.13 | 67.98
MORN [66] | Visual and Textual Semantic | GCN | 71.21 | 87.15 | 55.22 | 61.19 | 71.53

###  V-B Knowledge/Fact-Based VQA

Knowledge or Fact-Based VQA is the challenging task of making use of external
knowledge given in knowledge graphs such as WikiData [53] to answer questions
about an image. The major challenge of this task is to create a model that can
make use of all three mediums (image, question, and fact) to generate an
appropriate answer. The MUCKO [100] architectural diagram shown in Figure 4
(reused with permission), is shown as a representative example of models that
approach FVQA.

Figure 4: The MUCKO architecture [100] (reused with permission). Best viewed
in colour.

In [101], the authors present a novel GCN-based architecture for FVQA.
Alongside the question and answer sets, a knowledge base of facts is also
included, $KB=\\{f_{1},f_{2},...,f_{|KB|}\\}$. Each fact $f=(x,r,y)$ is formed
of a visual concept grounded in the image ($x$), an attribute or phrase ($y$),
and a relation linking the two $r$. Relationships exist in a predefined set of
13 different ways a concept and attribute can be related. Their work first
reduces the search space to the 100 facts most likely to contain the correct
answer by using GloVe embeddings [97] of words in the question and facts
before further reducing it to the most relevant facts $f_{rel}$. These most
relevant facts are turned into a graph where all the visual concepts and
attributes from $f_{rel}$ form the nodes. An edge joins two nodes if they are
related by a fact in $f_{rel}$. A GCN is then used to ‘reason’ over the graph
to predict the final answer. Using a message passing architecture, the authors
are able to update the feature representations of the nodes which can then be
fed into an MLP which predicts a binary label corresponding to whether or not
the entity contains the answer.

Zhu et al. [100] use a multi-modal graph approach to representing images with
a visual, semantic, and knowledge graph. After graph construction, GCNs are
applied to each modality to create richer feature embeddings. These embeddings
are then processed in a cross-modal manner. Visual-Fact aggregation and
Semantic-Fact aggregation operations produce complimentary information which
is then used with a Fact-Fact convolutional layer. This final layer takes into
account all three modalities and produces an answer that considers the global
context. The authors continue their work in [58] by changing the cross-modal
mechanism for a novel GRUC (Graph-based Read, Update, and Control) mechanism.
The GRUC operates in a parallel pipeline. One pipeline starts with a concept
from the knowledge graph and recurrently incorporates knowledge from the
visual graph. Another starts with the same knowledge graph concept but
incorporates semantic knowledge. At the end of the recurrent operations, the
outputs of the two pipelines are fused together with the question and original
fact node. This fused feature is then used to predict the final answer. The
change made to the cross-modal attention mechanism yields significant
improvements in the F-VQA benchmark when compared with MUCKO [100].

Liu et al. [102] also adopt a multi-modal approach, but use only the semantic
and knowledge modalities. They propose a dual process system for FVQA that is
based on the Dual-Process Theory from Cognitive Science [103]. Their approach
utilises a BERT encoder to represent the input question and a Faster-RCNN [59]
based feature extractor to represent the image features. The first of the two
systems, based on the Transformer architecture [26], joins these two
representations into a single multi-modal representation. The second system
then develops a semantic graph by turning dense region captions into textual
scene graphs (using SPICE), as well as a knowledge graph generated using the
question input. A message passing GNN is then used to identify the important
nodes and aggregate information between them using an attention weighting. A
joint representation for each knowledge graph node is then learned by
combining the whole semantic graph with the node with relation to an attention
weighting. This joint representation is then used to predict the final answer.

Moving away from the multi-modal approach, SGEITL [104] makes a semantic graph
of the image and then follows Yang et al. [40] and introduces skip edges to
the graph, essentially making it a complete graph. This graph then goes
through a multi-hop graph Transformer, which masks the attention between nodes
based on their distance, ensuring that only close by nodes are attended to.
Through their work, they demonstrate that structural information is useful
when approaching the complex VQA task.

With their TRiG model, Gao et al. [105] advocate taking an alternative
approach to FVQA and rather than generating the answer in some multi-modal
space, they propose to use the textual space. They argue that this prevents
further fusion with additional outside knowledge, and that as most of this
data are in textual form, it makes sense to work in that domain. TRiG
therefore has three components. It first converts the image into a caption
using an off-the-shelf image captioning tool. The model then finds the top $K$
relevant facts from a knowledge base of Wikipedia articles before using a T5
backboned Transformer [106] to fuse and decode the $<$question, visual
context, knowledge$>$ triplet into an answer.

TABLE V: A table showing the model details and results of selected models
trained and tested against the OK-VQA [37] and F-VQA [36] datasets. Bold: Best
score.

Model | Graphs Used | GNN Architecture | OK-VQA Top-1 Results | OK-VQA Top-3 Results | F-VQA Top-1 Results | F-VQA Top-3 Results
---|---|---|---|---|---|---
Out of the box [101] | Knowledge | GCN | - | - | 69.35 | 80.25
Mucko [100] | Visual, Semantic, Knowledge | GCN | - | - | 73.06 | 85.94
GRUC [58] | Visual, Semantic, Knowledge | GCN | 29.87 | 32.65 | 79.63 | 91.20
Dual Process [102] | Semantic, Knowledge | GCN | 29.43 | 32.83 | 63.57 | 76.47

###  V-C Text VQA

TextVQA is the sub-task of VQA where the answers require the model to be able
to read text that appears in images. Typically this involves tasks like
reading brand names from buildings or the title of book covers. This
information can then be combined with an external knowledge base, enabling the
models to answer questions such as “Is the shop an American brand?” by reading
the shop name and searching it in a knowledge base.

Gao et al. [107] focus on the in-image text and how it can be better leveraged
to improve VQA. They use a novel multi-modal graph made up of fully connected
visual, semantic, and numeric subgraphs. Each subgraph represents a unique
modality that can be found in an image: visual entities (represented by image
feature extractors), semantic meaning of discovered text (initially discovered
by OCR), along with numeric values and their semantic meaning. The paper
proposed a model that aggregates information across modalities together using
a relevance score. Once the three modalities have been aggregated, an
attention mechanism is deployed to help predict the final answer. The focus on
different modalities proves a useful approach, with the model performing
favourably in benchmarks (see Table VI).

Another work that makes use of multi-modal graphs is Liang et al. [108]. Their
work uses both image features and scene text features (extracted by OCR) to
generate a spatial relationship graph similar to that of [25]. The graph
undergoes multi-head attention before being processed by a GNN that makes use
of the attention weights. Multi-modal fusion is then used to join the node
features with the question embedding and positional features. The output of
this fusion operation is then used to predict a final answer.

TABLE VI: A table showing the model details and TextVQA-Val Accuracy results
of selected Text-VQA models. Bold: Best score.

Model | Graphs Used | GNN Architecture | TextVQA-Val Accuracy (%) | TextVQA-Test Accuracy (%)
---|---|---|---|---
MM-GNN [107] | Visual, Textual Semantic, Numeric | Multi-modal GNN | 31.44 | 31.10
MCG [108] | Spatial | Multi-modal (contextual) GNN | 29.40 | 29.61

##  VI Image Retrieval

Image retrieval is the task of finding images from a database given some
query. These queries can take many forms, including a similar image, a natural
language query, or even a sketch. A common approach is to represent the
database images as being in some space, where similar images are those with a
minimal distance to the query. When this space is represented using graphs,
GNNs become valuable for sharing features and acquiring more global context
for the features.

Johnson et al. [24] show that a scene graph can be used as the input of the
image retrieval system. By allowing end users to create a scene graph where
nodes represent objects, attributes, and relationships, they are able to
return appropriate images via a scene graph grounding process. This involves
matching each scene graph object node with a bounding box predicted by an
object detector, and is represented probabilistically using a conditional
random field (CRF). The advantage of using scene graphs as search queries over
natural language is that they scale well in terms of complexity. Once a basic
scene graph has been constructed, it is straightforward for it to be extended
and made more complex by adding additional nodes. Another advantage is that it
reduces the operations required to map the search query to the image.

Following on from [24], Yoon et al. propose IRSGS (Image Retrieval with Scene
Graph Similarity) [56], which makes use of a semantic graph, referred to as a
scene graph in the paper. Given a query image, the model will generate a
semantic graph and compare its similarity with graphs of images in the
database. This graph comparison is achieved by taking the inner product of
graph embeddings generated by a GNN (either GCN [109] or GIN[110]). One key
contribution of the paper is the concept of Surrogate Relevance, which is the
similarity between the captions of the images being compared. Surrogate
Relevance is calculated using the inner product between Sentence-BERT
embeddings of the captions. This measure is used as the training signal of the
model to hone the feature embeddings generated by the GNN. The graph-to-graph
comparison behind the model allows this work to better scale to large image
databases when compared to [24]. The use of Surrogate Relevance allows the
work to be potentially expanded to match against user queries if they are in
the style of the captions used to power the relevance measure.

Using a $K$-nearest neighbour graph of images represented as feature
embeddings, Liu et al. [71] propose using a GCN alongside a novel loss
function based on image similarity. The feature embeddings are enhanced to
account for a global context across the whole image database using a GCN.
Similarity between images is calculated by taking the inner product of the
feature embeddings. The higher the similarity, the better the retrieval
candidate. The author’s novel loss function is designed to move similar images
closer together in the embedding space and dissimilar images further apart.
Compared with [56], by using the inner product, the similarity measure is far
more deterministic. However, unlike [56], it cannot be expanded to work
alongside text-based image retrieval with a user query.

Zhang et al. [111] also use a $K$-nearest neighbour graph, but focus on
improving the re-ranking process in content based image retrieval. A GNN is
applied to aggregate features created from a modified adjacency matrix. Using
a GNN allows the re-ranking process to de-emphasise nodes with a low
confidence score.

Rather than use a pure $K$-nearest neighbour graph, the DGCQ model [112] is
based on vector quantisation, a process from Information Theory for reducing
the cardinality of a vector space. It can essentially be thought of as a many-
to-one clustering technique where vectors in one space $x\in\mathbb{R}^{d}$
are mapped to a set of code words ($c_{i}$) that make up a code book
$q(x)\in\mathcal{C}=\\{c_{i};i\in\mathcal{I}\\}$. Where
$\mathcal{I}=1...(k-1)$. By using vector quantisation, the model learns code
words that can be combined with image features to form landmark graph. This
graph is based on the similarity graph except it also has nodes learned
through the quantisation process. Once the landmark graph has been
constructed, a GCN is use to propagate features with the objective of moving
similar images closer together in the feature space. The use of vector
quantisation allows for the landmark graph to exist in a lower dimensional
space, reducing computation when computing which images from the graph to
return as candidates.

The authors of [57] move to adopt a multi-modal approach. They use GraphSAGE
[77] to effectively learn multi-modal node embeddings containing visual and
conceptual information from the connections in the graph. The distance between
connected nodes are reduced, whilst the distance between disconnected nodes is
increased. By using graph nodes that represent images as well as nodes
representing metadata tags, their model is able to provide content-based image
retrieval as well as tag prediction. At inference time, images shown to the
model can be attached to the graph through their $K$ nearest images, attached
to relevant tags, or both. Unlike previous works [71, 56, 24], Misraa et al.
[57] make use of multi-modal embeddings in the graph nodes.

Schuster et al. [63] continue the work of Johnson et al. [24], by creating a
natural language parser that converts a query into a scene graph that can be
processed by their work. This allows them to go beyond content-based image
retrieval and move into text-based image retrieval. Their parser works by
creating a dependency tree using the Stanford Dependency Parser [64] and then
modifying the tree. They first execute a quantification modifier that ensures
nouns are the head of the phrase. This is followed by pronoun resolution to
make the relationship between two objects more explicit. Finally, plural nouns
are processed. This involves copying noun instances when numeric modifiers are
given. This textual scene graph is then mapped to images following [24].

Cui et al. [55] also tackle text-based image retrieval. They present work that
makes use of a GCN to provide cross-modal reasoning on visual and textual
information. Input features are split into channels which form a complete
graph and undergo graph convolution. Once the textual and visual features are
projected into a common space, they have their distances measured using the
cosine similarity. These similarity scores are then stored in a matrix
representing the similarities between visual and textual inputs.

Zhang et al. [113] tackle the challenging task of Composing Text and Image to
Image Retrieval, where given a reference image and modification query the
image retrieval system must find an image similar to the reference that
contains the modifications outlined in the query. The principle challenge of
this emerging task is its cross-modality nature. The authors tackle this
challenge by first generating a spatial graph of the reference image and a
textual feature of the modification query. These features are then
concatenated before the graph is processed by a GAT whose attention mechanism
has been altered to account for the directionality of the graph and the
spatial data it encodes. A collection of GRUs that form a Global Semantic
Reasoning (GSR) unit are then used to create the final embedding for the
reference image. The same process is used on the target image but without the
concatenation of the textual feature. A cross-modal loss function and
adversarial loss function are combined to ensure that the features outputted
by the GSR of the same category are moved closer together.

Chaudhuri et al. [73] adopt a Siamese-based network architecture where two
similar inputs go into two separate networks that share weights. This network
architecture typically uses contrastive loss or triplet loss to ensure the
outputs of these networks are similar. The authors use a novel Siamese-GCN on
a region adjacency graph that is formed by connecting adjacent segmented
regions and weighting the edge accounting for the distance and angle between
centroids of the regions. They apply their technique to high resolution remote
sensing images for content-based image retrieval. By using a Siamese-GCN with
contrastive loss, the authors are able to learn an embedding that brings
similar images together and forces dissimilar images apart. This work is then
followed up by the authors in [114], where they add a range of attention
mechanisms. They implement both node-level and edge-level attention mechanisms
(in a similar style to GAT [80]). These attention mechanisms are then
incorporated into the Siamese-GCN to yield improvements over their previous
work.

Another work to incorporate a siamese network design was Zhang et al. [115].
They use a three part network design to perform zero-shot sketch-based image
retrieval with a Siamese-based encoding network which creates features of the
image and associated sketch using ResNet50. These features are the
concatenated together to create node features. The similarity between nodes is
calculated using a metric function modelled by an MLP, and this operation is
used to populate the adjacency matrix of a similarity graph. A GCN is then
applied to the similarity graph to create fusion embeddings of sketch-image
pairs. Rather than use an MLP to reconstruct the semantic information from the
GCN embeddings, the authors chose to use a Conditional Variational Autoencoder
[116]. Doing so enables the model to generate semantic information for
sketches of unseen classes, aiding the zero-shot component of the model.

##  VII Discussion and Conclusion

In this section, we draw upon the views of Battaglia et al. [27], and discuss
how the popular Transformer [26] can be viewed through the lens of GNNs. We
then discuss how its dependence on consistent structure may pose challenges
should image generation techniques be applied to create new training data for
image captioning. The section concludes with a final summary of the paper and
an overview of the challenges and future research directions that lie ahead
for graph-based 2D image understanding.

###  VII-A Why GNNs When We Have Transformers?

Recent years have seen the rapid rise in popularity of the Transformer
architecture [26]. Originally proposed in the Natural Language Processing
domain, it was quickly applied as a generalised encoder in computer vision
tasks [46] Further work then expanded the architecture so that it can process
images directly [117, 118], allowing it to operate as a backbone for common
vision tasks. The wide range of applications the architecture can be applied
to has led to it dominating much of deep learning in recent years.

There has been some effort by the community to unify the attention-based
approach with GNNs. Battaglia et al. [27] proposes a more generic Graph
Network which both Transformers and GNNs fall into. They present a viewpoint
where Transformers can be viewed as a neural architecture operating on a
complete graph.

Viewing GNNs and Transformers as Graph Networks shows that they share a number
of similarities. Both architectures take a set of values and decide how much
different values should be considered when transforming them to update the
values, with GNNs ignoring nodes that are not connected and Transformers
scaling the importance of an input. It is worth noting that if the graph being
processed by a GNN is a complete graph, the graph network will allow all nodes
to have their messages propagated to one being updated. Therefore, it is
possible to view the Transformer as a special case GNN operating on a complete
graph. While GNNs use the read module to take advantage of an underlying
structure, the Transformer learns one based on the task.

By applying a Transformer to a task, a graph structure is being learnt from
scratch. Meanwhile, there are plenty of graph structures that appear naturally
within vision-language tasks. This multitude of graph types allow for
different structures to be taken for the image, from the semantic structure of
an image to the hierarchical structure of the image with regards to the entire
training set. Graphs appear naturally in the language component of the tasks
as well, with sentence dependency trees being closely aligned to semantic
scene graphs (when the scene graph is made multipartite as in the case of
[61]). When clear graph representations of data exist, they should be utilised
rather than ignored, rather than learning a graph structure using a more
general purpose architecture. Utilising existing graph structures enables a
Graph Network with the appropriate inductive biases to be deployed. It also
results in fewer computations as messages are not being passed between all
possible node connections.

When it is possible to utilise multiple graphs, it is advantageous to do so
when compared to using a single graph. As shown with image captioning (Table
III), architectures that only use a single graph type perform sub-optimally
compared to their multigraph counterparts. ARL [94], Sub-GC [60], and Topic
[69] all use a single graph (spatial, semantic, similarity respectively) and
all three suffer in benchmarks. Whilst Topic performs well in BLEU, METEOR,
and ROGUE, when evaluated using metrics designed specifically for image
captioning (SPICE and CIDEr) its performance falters against comparable
models. This theme of multigraph approaches performing more favourably is also
found across the VQA, FVQA, and text-VQA tasks, with multigraph approaches
outperforming their single graph counterparts.

###  VII-B Latent Diffusion and the Future of Image Captioning

Currently, image captioning techniques are constrained by their training data.
As popular as COCO is within the Computer Vision community for its wide
ranging scenes and generalisability to the real world, it has its
shortcomings. Captioning systems trained on it alone will never understand
particular art styles, or objects outside of the 80 categories covered by the
COCO dataset. The advent of image generation techniques such as DALLE$\cdot$2
[119] present an opportunity for image captioning systems to go well beyond an
80 category limit and start understanding various stylistic elements of
images. Work in this area is in its infancy [120, 121], but previous non-
generative unsupervised approaches to image captioning are very promising
[18].

We speculate that latent diffusion-based captioning may be a promising avenue
of research. However, for this approach to work effectively, image generation
techniques will need to develop further. Currently DALLE$\cdot$2 [119] and
similar systems do not understand structure as deeply as would be required for
them to be able to replace the training data of a captioning system. As
impressive as they are, they can struggle to assemble images correctly when
the prompt asks for something that is unlikely in real life. When asked to
generate an image of “A monkey riding on the back of a polar bear”,
DALLE$\cdot$2 [119] can sometimes struggle to understand the requested spatial
relation between the two animals, resulting in the sample result shown in
Figure 5.

Figure 5: One of the images generated by OpenAI’s DALLE$\cdot$2 [119] given
the prompt ‘A monkey riding on the back of a polar bear’. Note the inverted
relationship in the generated image. Best viewed in colour.

Discovering examples of incorrect relationships in images is not just a case
of dreaming up relationships between objects that are unlikely to exist in
training data. Conwell and Ullman [122] conducted a participant study where
they asked $169$ people to select generated images that they felt well matched
a given prompt. They found that across the generated images in their study,
only 22% matched the original prompt. The authors conclude that “current image
generation models do not yet have a grasp of even basic relations involving
simple objects and agents” [122]. Whilst latent diffusion methods may play a
role in the future of image captioning, they have a long way to go
understanding structure before this is possible. In order for Graph Networks
[27] to be applicable to diffusion generated training data, the structure
within the image and the caption/prompt will need to be consistent. Supervised
learning approaches require large amounts of very clean training data in order
to work well, so Graph Networks [27] may struggle if the underlying structure
in the image data is not as expected.

###  VII-C Final Notes

Vision-language tasks such as image captioning and VQA pose significant
opportunities for accessibility technology to be developed for those with
sight impairment or severe sight impairment. Having widespread automatic alt-
text generation on websites and applications enabling queries about images
shared online, there is substantial impact that research in these fields can
have. However, models trained on current datasets are prone to the biases of
sighted humans. The questions asked in VQA datasets, and the captions given in
image captioning datasets do not necessarily cater to the needs of possible
end users of this technology. A lot is said in the field of the technology
being applied to aid those with various levels of sight impairment, but little
action is actually taken. Whilst the release of trained models is promising,
making these models available outside of the research community would be
beneficial. Another direction the community could take towards using this
research to aid those with forms of sight impairment would be to curate a
dataset of images with questions posed by those we seek to help, i.e., people
with sight impairment. This dataset could also include captions that focus on
aspects of an image deemed to be important to those with sight impairment. The
inclusion of these captions would yield models that generate captions that
prioritise the information required by someone with sight impairment rather
than trying to mirror the style of captions generated by sighted humans as is
the case with models trained on existing image captioning datasets COCO [33]
or Flickr30k [34].

The state of the art (SOTA) in vision-language tasks is currently dominated by
large Transformer-based models developed by industrial labs [123, 124, 125].
This makes comparing these models to those discussed in this paper difficult
given the model size and compute power used for training. However, there are a
few take home points.

In the case of image captioning, the Transformer-based model $\mathcal{M}2$ is
outperformed by GNN-based architectures, namely Dual-GCN [70]. This leads the
authors to posit that there is a strong inductive bias in using imposed graph
structures rather than allowing all relationships between detected objects to
be processed using self-attention. The use of a global context graph (taking
into account the whole dataset) alongside a local context graph (image level
relationships) by Dual-GCN [70] is shown to work extremely well and this dual
graph approach could be the seed for future works.

It could be that given the scale of the models currently achieving SOTA that
there are some emergent properties that develop in these models when they
achieve such as scale. Future work should consider scaling graph-based
architectures, such as those discussed in this survey, to the scale of the
large models being produced by industry labs.

For FVQA and image retrieval, the graph-based approaches have stronger
inductive biases for the reasoning stages of the tasks. Both tasks require the
processing of graph data (in the case of a knowledge graph in FVQ or some
graph representation of the search space in image retrieval). It is well
documented that Transformers do not perform well on sparse graphs (such as
knowledge graphs) or large graphs (such as those used in image retrieval).

The adoption of GNN-based image captioning techniques has proven promising.
Given that this approach is relatively new, there is ample opportunity for
further research to be carried out in this field. As shown in Section IV the
majority of image captioning techniques make use of either GCN or GGNN
architectures. As GNNs develop and newer more expressive techniques are
approached, the community should move to adopt these over traditional message
passing style networks. Models such as GAT [80] may provide advantages over
the techniques being used as they incorporate self-attention mechanisms into
the architecture, a technique proven to yield impressive results given the
popularity of the Transformer.

All the GNNs being used in the vision-language tasks discussed in the survey
are built on the concept of homophily, i.e., similar nodes are connected by an
edge. This is not always the case though given that a semantic graph connects
dissimilar objects that are semantically related. Some of the graphs detailed
are homophilic (e.g., image graph), but many others are not. This leads us to
speculate that there are ample research opportunities for applying GNN
architectures that respect the amount of homophily or heterophily of the graph
being processed.

Another direction of research would be investigating combinations of different
graph representations (both at the image level and dataset level) to identify
combinations that work well together. Using different graph representations
will allow for better utilisation of both local and global features.

The incorporation of outside knowledge into image captioning could provide an
interesting research direction. It is often pointed out that image captioning
is a useful accessibility technology for those with sight impairment. However,
this assumes the user is an adult with a developed understanding of the world.
Image captioning systems may struggle to be applied in a paediatric
accessibility setting. Having the model explain the world in greater detail
may be of use.

Another potential future research direction would be the unification of the
three tasks discussed in this paper. Developing a single unified model that
could perform competently in all three would hail an important breakthrough.
In order to perform this, a model would have to have a common intermediary
space for which it could map between the text and image spaces. We posit that
this space would most likely be graph-based due to their expressive nature.
However, a textual representation may also be performant as Gao et al. [105]
showed reasoning in the text space improved performance over graph-based
reasoning in VQA.

In summary, vision-language tasks such as those discussed in this paper are
set to have a fruitful future, with many opportunities for various graph
structures to be exploited.

## Acknowledgment

The authors would like to thank colleagues in DERI for their thoughtful and
insightful feedback as we developed some of the ideas presented in this paper.

## References

  * [1] B. P. Chamberlain, S. Shirobokov, E. Rossi, F. Frasca, T. Markovich, N. Hammerla, M. M. Bronstein, and M. Hansmire, “Graph neural networks for link prediction with subgraph sketching,” _arXiv preprint_ , 2022.
  * [2] F. Barbero, C. Bodnar, H. S. de Ocáriz Borde, M. Bronstein, P. Veličković, and P. Liò, “Sheaf neural networks with connection laplacians,” in _Topological, Algebraic and Geometric Learning Workshops 2022_. PMLR, 2022, pp. 28–36.
  * [3] F. Frasca, B. Bevilacqua, M. M. Bronstein, and H. Maron, “Understanding and extending subgraph gnns by rethinking their symmetries,” _arXiv preprint_ , 2022.
  * [4] L. Peng, R. Hu, F. Kong, J. Gan, Y. Mo, X. Shi, and X. Zhu, “Reverse graph learning for graph neural network,” _IEEE trans. on neural networks and learning systems_ , pp. 1–12, 2022.
  * [5] X. Ai, C. Sun, Z. Zhang, and E. R. Hancock, “Two-level graph neural network,” _IEEE trans. on neural networks and learning systems_ , pp. 1–14, 2022.
  * [6] R. Yang, W. Dai, C. Li, J. Zou, and H. Xiong, “Ncgnn: Node-level capsule graph neural network for semisupervised classification,” _IEEE trans. on neural networks and learning systems_ , pp. 1–15, 2022.
  * [7] Z. Wang, M. Liu, Y. Luo, Z. Xu, Y. Xie, L. Wang, L. Cai, Q. Qi, Z. Yuan, T. Yang _et al._ , “Advanced graph and sequence neural networks for molecular property prediction and drug discovery,” _Bioinformatics_ , vol. 38, no. 9, pp. 2579–2586, 2022.
  * [8] S. J. Clipman, S. H. Mehta, S. Mohapatra, A. K. Srikrishnan, K. J. Zook, P. Duggal, S. Saravanan, P. Nandagopal, M. S. Kumar, G. M. Lucas _et al._ , “Deep learning and social network analysis elucidate drivers of hiv transmission in a high-incidence cohort of people who inject drugs,” _Science Advances_ , vol. 8, no. 42, p. eabf0158, 2022.
  * [9] W. Shi and R. Rajkumar, “Point-gnn: Graph neural network for 3d object detection in a point cloud,” in _CVPR_ , 2020, pp. 1711–1719.
  * [10] L. Cosmo, G. Minello, M. Bronstein, E. Rodolà, L. Rossi, and A. Torsello, “3d shape analysis through a quantum lens: the average mixing kernel signature,” _IJCV_ , pp. 1–20, 2022.
  * [11] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” _AI Open_ , vol. 1, pp. 57–81, 2020.
  * [12] H.-C. Yi, Z.-H. You, D.-S. Huang, and C. K. Kwoh, “Graph representation learning in bioinformatics: trends, methods and applications,” _Briefings in Bioinformatics_ , vol. 23, no. 1, p. bbab340, 2022.
  * [13] J. J. Thomas, T. H. N. Tran, G. P. Lechuga, and B. Belaton, “Convolutional graph neural networks: a review and applications of graph autoencoder in chemoinformatics,” _Deep learning techniques and optimization strategies in big data analytics_ , pp. 107–123, 2020.
  * [14] W. Liu, Y. Zhang, J. Wang, Y. He, J. Caverlee, P. P. K. Chan, D. S. Yeung, and P.-A. Heng, “Item relationship graph neural networks for e-commerce,” _IEEE trans. on neural networks and learning systems_ , vol. 33, no. 9, pp. 4785–4799, 2022.
  * [15] W. Lin and B. Li, “Status-aware signed heterogeneous network embedding with graph neural networks,” _IEEE trans. on neural networks and learning systems_ , pp. 1–13, 2022.
  * [16] C. Chen, Y. Wu, Q. Dai, H.-Y. Zhou, M. Xu, S. Yang, X. Han, and Y. Yu, “A survey on graph neural networks and graph transformers in computer vision: A task-oriented perspective,” _arXiv preprint_ , 2022.
  * [17] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, “Every picture tells a story: Generating sentences from images,” in _ECCV_. Springer, 2010, pp. 15–29.
  * [18] I. Laina, C. Rupprecht, and N. Navab, “Towards unsupervised image captioning with shared multimodal embeddings,” in _ICCV_ , 2019, pp. 7414–7424.
  * [19] G. Li, L. Zhu, P. Liu, and Y. Yang, “Entangled transformer for image captioning,” in _ICCV_ , 2019, pp. 8928–8937.
  * [20] Z. Shao, J. Han, D. Marnerides, and K. Debattista, “Region-object relation-aware dense captioning via transformer,” _IEEE trans. on neural networks and learning systems_ , pp. 1–12, 2022.
  * [21] Y. Liu, X. Zhang, F. Huang, L. Cheng, and Z. Li, “Adversarial learning with multi-modal attention for visual question answering,” _IEEE trans. on neural networks and learning systems_ , vol. 32, no. 9, pp. 3894–3908, 2021.
  * [22] Y. Gu, K. Vyas, M. Shen, J. Yang, and G.-Z. Yang, “Deep graph-based multimodal feature embedding for endomicroscopy image retrieval,” _IEEE trans. on neural networks and learning systems_ , vol. 32, no. 2, pp. 481–492, 2021.
  * [23] P. Staszewski, M. Jaworski, J. Cao, and L. Rutkowski, “A new approach to descriptors generation for image retrieval by analyzing activations of deep neural network layers,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 12, pp. 7913–7920, 2022.
  * [24] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” in _CVPR_ , 2015, pp. 3668–3678.
  * [25] T. Yao, Y. Pan, Y. Li, and T. Mei, “Exploring visual relationship for image captioning,” in _ECCV_ , 2018, pp. 684–699.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _NeurIPS_ , vol. 30, 2017.
  * [27] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner _et al._ , “Relational inductive biases, deep learning, and graph networks,” _arXiv preprint_ , 2018.
  * [28] M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga, “A comprehensive survey of deep learning for image captioning,” _ACM Computing Surveys (CsUR)_ , vol. 51, no. 6, pp. 1–36, 2019.
  * [29] Y. Zou and Q. Xie, “A survey on vqa: Datasets and approaches,” in _ITCA_. IEEE, 2020, pp. 289–297.
  * [30] A. A. Yusuf, F. Chong, and M. Xianling, “An analysis of graph convolutional networks and recent datasets for visual question answering,” _Artificial Intelligence Review_ , pp. 1–24, 2022.
  * [31] M. Alexander and S. Gunasekaran, “A survey on image retrieval methods,” _Preprint_ , 2014. [Online]. Available: https://web-archive.southampton.ac.uk/cogprints.org/9815/1/SurveyonImageRetrievalMethods.pdf
  * [32] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma _et al._ , “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” _IJCV_ , vol. 123, no. 1, pp. 32–73, 2017.
  * [33] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _ECCV_. Springer, 2014, pp. 740–755.
  * [34] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” _Transactions of the ACL_ , vol. 2, pp. 67–78, 2014.
  * [35] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in _ICCV_ , 2015, pp. 2425–2433.
  * [36] P. Wang, Q. Wu, C. Shen, A. Dick, and A. Van Den Hengel, “Fvqa: Fact-based visual question answering,” _PAMI_ , vol. 40, no. 10, pp. 2413–2427, 2017\.
  * [37] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “Ok-vqa: A visual question answering benchmark requiring external knowledge,” in _CVPR_ , 2019\.
  * [38] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” in _CVPR_ , 2019, pp. 8317–8326.
  * [39] A. K. Singh, A. Mishra, S. Shekhar, and A. Chakraborty, “From strings to things: Knowledge-enabled vqa model that can read and reason,” in _ICCV_ , 2019, pp. 4602–4612.
  * [40] J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh, “Graph r-cnn for scene graph generation,” in _ECCV_ , September 2018.
  * [41] C. Zhang, W.-L. Chao, and D. Xuan, “An empirical study on leveraging scene graphs for visual question answering,” in _BMVC_ , K. Sidorov and Y. Hicks, Eds. BMVA Press, September 2019, pp. 151.1–151.14. [Online]. Available: https://dx.doi.org/10.5244/C.33.151
  * [42] S. Wu, J. Wieland, O. Farivar, and J. Schiller, “Automatic alt-text: Computer-generated image descriptions for blind users on a social network service,” in _ACM Conference on Computer Supported Cooperative Work and Social Computing_ , 2017, pp. 1180–1192.
  * [43] A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in _CVPR_ , 2015, pp. 3128–3137.
  * [44] X. Yang, K. Tang, H. Zhang, and J. Cai, “Auto-encoding scene graphs for image captioning,” in _CVPR_ , 2019, pp. 10 685–10 694.
  * [45] M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara, “Meshed-memory transformer for image captioning,” in _CVPR_ , 2020, pp. 10 578–10 587.
  * [46] S. He, W. Liao, H. R. Tavakoli, M. Yang, B. Rosenhahn, and N. Pugeault, “Image captioning through image transformer,” in _ACCV_ , 2020.
  * [47] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering,” in _CVPR_ , 2017.
  * [48] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_. Ieee, 2009, pp. 248–255.
  * [49] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives, “Dbpedia: A nucleus for a web of open data,” in _The semantic web_. Springer, 2007, pp. 722–735.
  * [50] H. Liu and P. Singh, “Conceptnet—a practical commonsense reasoning tool-kit,” _BT technology journal_ , vol. 22, no. 4, pp. 211–226, 2004.
  * [51] N. Tandon, G. Melo, and G. Weikum, “Acquiring comparative commonsense knowledge from the web,” in _AAAI_ , vol. 28, no. 1, 2014.
  * [52] B. K. Iwana, S. T. R. Rizvi, S. Ahmed, A. Dengel, and S. Uchida, “Judging a book by its cover,” _arXiv preprint_ , 2016.
  * [53] D. Vrandečić and M. Krötzsch, “Wikidata: a free collaborative knowledgebase,” _Communications of the ACM_ , vol. 57, no. 10, pp. 78–85, 2014.
  * [54] X. Han, Z. Wu, P. X. Huang, X. Zhang, M. Zhu, Y. Li, Y. Zhao, and L. S. Davis, “Automatic spatially-aware fashion concept discovery,” in _ICCV_ , 2017, pp. 1463–1471.
  * [55] Z. Cui, Y. Hu, Y. Sun, J. Gao, and B. Yin, “Cross-modal alignment with graph reasoning for image-text retrieval,” _Multimedia Tools and Applications_ , pp. 1–18, 2022.
  * [56] S. Yoon, W. Y. Kang, S. Jeon, S. Lee, C. Han, J. Park, and E.-S. Kim, “Image-to-image retrieval by learning similarity between scene graphs,” in _AAAI_ , vol. 35, no. 12, 2021, pp. 10 718–10 726.
  * [57] A. K. Misraa, A. Kale, P. Aggarwal, and A. Aminian, “Multi-modal retrieval using graph neural networks,” _arXiv preprint_ , 2020.
  * [58] J. Yu, Z. Zhu, Y. Wang, W. Zhang, Y. Hu, and J. Tan, “Cross-modal knowledge reasoning for knowledge-based visual question answering,” _Pattern Recognition_ , vol. 108, p. 107563, 2020.
  * [59] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _NeurIPS_ , vol. 28, 2015\.
  * [60] Y. Zhong, L. Wang, J. Chen, D. Yu, and Y. Li, “Comprehensive image captioning via scene graph decomposition,” in _ECCV_. Springer, 2020, pp. 211–229.
  * [61] L. Guo, J. Liu, J. Tang, J. Li, W. Luo, and H. Lu, “Aligning linguistic words and visual semantic units for image captioning,” in _ACM International Conference on Multimedia_ , 2019, pp. 765–773.
  * [62] D. Zhou, J. Yang, C. Zhang, and Y. Tang, “Joint scence network and attention-guided for image captioning,” in _ICDM_. IEEE, 2021, pp. 1535–1540.
  * [63] S. Schuster, R. Krishna, A. Chang, L. Fei-Fei, and C. D. Manning, “Generating semantically precise scene graphs from textual descriptions for improved image retrieval,” in _Proceedings of the fourth workshop on vision and language_ , 2015, pp. 70–80.
  * [64] M.-C. De Marneffe and C. D. Manning, “The stanford typed dependencies representation,” in _Coling 2008: proceedings of the workshop on cross-framework and cross-domain parser evaluation_ , 2008, pp. 1–8.
  * [65] D. Teney, L. Liu, and A. van Den Hengel, “Graph-structured representations for visual question answering,” in _CVPR_ , 2017, pp. 1–9.
  * [66] H. Pan and J. Huang, “Multimodal high-order relational network for vision-and-language tasks,” _Neurocomputing_ , vol. 492, pp. 62–75, 2022\.
  * [67] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in _ECCV_ , 2016, pp. 382–398.
  * [68] T. Yao, Y. Pan, Y. Li, and T. Mei, “Hierarchy parsing for image captioning,” in _ICCV_ , 2019, pp. 2621–2629.
  * [69] J. Kan, K. Hu, Z. Wang, Q. Wu, M. Hagenbuchner, and A. C. Tsoi, “Topic-guided local-global graph neural network for image captioning,” in _ICME_. IEEE, 2021, pp. 1–6.
  * [70] X. Dong, C. Long, W. Xu, and C. Xiao, “Dual graph convolutional networks with transformer and curriculum learning for image captioning,” in _ICME_ , 2021, pp. 2615–2624.
  * [71] C. Liu, G. Yu, M. Volkovs, C. Chang, H. Rai, J. Ma, and S. K. Gorti, “Guided similarity separation for image retrieval,” _NeurIPS_ , vol. 32, 2019.
  * [72] C. Li, H. Wang, Z. Zhang, A. Sun, and Z. Ma, “Topic modeling for short texts with auxiliary word embeddings,” in _ACM SIGIR conference on Research and Development in Information Retrieval_ , 2016, pp. 165–174.
  * [73] U. Chaudhuri, B. Banerjee, and A. Bhattacharya, “Siamese graph convolutional network for content based remote sensing image retrieval,” _Computer vision and image understanding_ , vol. 184, pp. 22–30, 2019.
  * [74] A. Hogan, E. Blomqvist, M. Cochez, C. d’Amato, G. d. Melo, C. Gutierrez, S. Kirrane, J. E. L. Gayo, R. Navigli, S. Neumaier _et al._ , “Knowledge graphs,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 4, pp. 1–37, 2021\.
  * [75] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” _IEEE trans. on neural networks and learning systems_ , vol. 32, no. 1, pp. 4–24, 2020.
  * [76] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe, “Weisfeiler and leman go neural: Higher-order graph neural networks,” in _AAAI_ , vol. 33, no. 01, 2019, pp. 4602–4609.
  * [77] W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in _NeurIPS_ , 2017, pp. 1025–1035.
  * [78] Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” _CoRR_ , vol. abs/1511.05493, 2015.
  * [79] K. Cho, B. van Merrienboer, Çaglar Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder–decoder for statistical machine translation,” in _Conference on Empirical Methods in Natural Language Processing_ , 2014.
  * [80] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph Attention Networks,” _ICLR_ , 2018, accepted as poster. [Online]. Available: https://openreview.net/forum?id=rJXMpikCZ
  * [81] M. Khademi, “Multimodal neural graph memory networks for visual question answering,” in _Proceedings of the 58th Annual Meeting of the ACL_ , 2020, pp. 7177–7188.
  * [82] D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun, “Measuring and relieving the over-smoothing problem for graph neural networks from the topological view,” in _AAAI_ , vol. 34, no. 04, 2020, pp. 3438–3445.
  * [83] C. Bodnar, F. D. Giovanni, B. P. Chamberlain, P. Liò, and M. M. Bronstein, “Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs,” in _NeurIPS_ , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. [Online]. Available: https://openreview.net/forum?id=vbPsD-BhOZ
  * [84] S. Abu-El-Haija, B. Perozzi, A. Kapoor, N. Alipourfard, K. Lerman, H. Harutyunyan, G. Ver Steeg, and A. Galstyan, “Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing,” in _ICML_. PMLR, 2019, pp. 21–29.
  * [85] C. Bodnar, F. Frasca, N. Otter, Y. Wang, P. Lio, G. F. Montufar, and M. Bronstein, “Weisfeiler and lehman go cellular: Cw networks,” _NeurIPS_ , vol. 34, pp. 2625–2640, 2021.
  * [86] B. Chamberlain, J. Rowbottom, M. I. Gorinova, M. Bronstein, S. Webb, and E. Rossi, “Grand: Graph neural diffusion,” in _ICML_. PMLR, 2021, pp. 1407–1418.
  * [87] D. S. Lakshminarasimhan Srinivasan and A. Amutha, “Image captioning-a deep learning approach,” _International Journal of Applied Engineering Research_ , vol. 13, no. 9, pp. 7239–7242, 2018.
  * [88] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the ACL_ , 2002, pp. 311–318.
  * [89] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [90] S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in _ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 2005, pp. 65–72.
  * [91] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _CVPR_ , 2015, pp. 4566–4575.
  * [92] F. Monti, K. Otness, and M. M. Bronstein, “Motifnet: a motif-based graph convolutional network for directed graphs,” in _2018 IEEE Data Science Workshop (DSW)_. IEEE, 2018, pp. 225–228.
  * [93] Z. Song and X. Zhou, “Exploring explicit and implicit visual relationships for image captioning,” in _ICME_. IEEE, 2021, pp. 1–6.
  * [94] J. Wang, W. Wang, L. Wang, Z. Wang, D. D. Feng, and T. Tan, “Learning visual relationship and context-aware attention for image captioning,” _Pattern Recognition_ , vol. 98, p. 107075, 2020.
  * [95] K. S. Tai, R. Socher, and C. D. Manning, “Improved semantic representations from tree-structured long short-term memory networks,” _arXiv preprint_ , 2015.
  * [96] H. Sharma and A. S. Jalal, “Visual question answering model based on graph neural network and contextual attention,” _Image and Vision Computing_ , vol. 110, p. 104165, 2021.
  * [97] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _EMNLP_ , 2014, pp. 1532–1543.
  * [98] L. Li, Z. Gan, Y. Cheng, and J. Liu, “Relation-aware graph attention network for visual question answering,” in _ICCV_ , 2019, pp. 10 313–10 322.
  * [99] S. V. Nuthalapati, R. Chandradevan, E. Giunchiglia, B. Li, M. Kayser, T. Lukasiewicz, and C. Yang, “Lightweight visual question answering using scene graphs,” in _ACM International Conference on Information & Knowledge Management_, 2021, pp. 3353–3357.
  * [100] Z. Zhu, J. Yu, Y. Sun, Y. Hu, Y. Wang, and Q. Wu, “Mucko: Multi-layer cross-modal knowledge reasoning for fact-based visual question answering,” in _IJCAI_ , 2020.
  * [101] M. Narasimhan, S. Lazebnik, and A. Schwing, “Out of the box: Reasoning with graph convolution nets for factual visual question answering,” _NeurIPS_ , vol. 31, 2018.
  * [102] L. Liu, M. Wang, X. He, L. Qing, and H. Chen, “Fact-based visual question answering via dual-process system,” _Knowledge-Based Systems_ , vol. 237, p. 107650, 2022.
  * [103] K. E. Stanovich and R. F. West, “24. individual differences in reasoning: Implications for the rationality debate?” _Behavioural and Brain Science_ , vol. 23, no. 5, pp. 665–726, 2000.
  * [104] Z. Wang, H. You, L. H. Li, A. Zareian, S. Park, Y. Liang, K.-W. Chang, and S.-F. Chang, “Sgeitl: Scene graph enhanced image-text learning for visual commonsense reasoning,” in _AAAI_ , vol. 36, no. 5, 2022, pp. 5914–5922.
  * [105] F. Gao, Q. Ping, G. Thattai, A. Reganti, Y. N. Wu, and P. Natarajan, “A thousand words are worth more than a picture: Natural language-centric outside-knowledge visual question answering,” _arXiv preprint_ , 2022.
  * [106] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu _et al._ , “Exploring the limits of transfer learning with a unified text-to-text transformer.” _JMLR_ , vol. 21, no. 140, pp. 1–67, 2020.
  * [107] D. Gao, K. Li, R. Wang, S. Shan, and X. Chen, “Multi-modal graph neural network for joint reasoning on vision and scene text,” in _CVPR_ , 2020, pp. 12 746–12 756.
  * [108] Y. Liang, X. Wang, X. Duan, and W. Zhu, “Multi-modal contextual graph neural network for text visual question answering,” in _ICPR_ , 2021, pp. 3491–3498.
  * [109] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _ICLR_ , 2017. [Online]. Available: https://openreview.net/forum?id=SJU4ayYgl
  * [110] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” _arXiv preprint_ , 2018.
  * [111] X. Zhang, M. Jiang, Z. Zheng, X. Tan, E. Ding, and Y. Yang, “Understanding image retrieval re-ranking: a graph neural network perspective,” _arXiv preprint_ , 2020.
  * [112] M. Wang, W. Zhou, Q. Tian, and H. Li, “Deep graph convolutional quantization networks for image retrieval,” _IEEE trans. on Multimedia_ , 2022.
  * [113] F. Zhang, M. Xu, Q. Mao, and C. Xu, “Joint attribute manipulation and modality alignment learning for composing text and image to image retrieval,” in _ACM International Conference on Multimedia_ , 2020, pp. 3367–3376.
  * [114] U. Chaudhuri, B. Banerjee, A. Bhattacharya, and M. Datcu, “Attention-driven graph convolution network for remote sensing image retrieval,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 19, pp. 1–5, 2021.
  * [115] Z. Zhang, Y. Zhang, R. Feng, T. Zhang, and W. Fan, “Zero-shot sketch-based image retrieval via graph convolution network,” in _AAAI_ , vol. 34, no. 07, 2020, pp. 12 943–12 950.
  * [116] B. Zhang, D. Xiong, J. Su, H. Duan, and M. Zhang, “Variational neural machine translation,” in _Conference on Empirical Methods in Natural Language Processing_. Austin, Texas: ACL, Nov. 2016, pp. 521–530. [Online]. Available: https://aclanthology.org/D16-1050
  * [117] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint_ , 2020.
  * [118] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , 2021, pp. 10 012–10 022.
  * [119] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint_ , 2022\.
  * [120] S. Xu, “Clip-diffusion-lm: Apply diffusion model on image captioning,” _arXiv preprint_ , 2022.
  * [121] H. Li, J. Gu, R. Koner, S. Sharifzadeh, and V. Tresp, “Do dall-e and flamingo understand each other?” _arXiv preprint_ , 2022.
  * [122] C. Conwell and T. Ullman, “Testing relational understanding in text-guided image generation,” _arXiv preprint_ , 2022.
  * [123] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer _et al._ , “Pali: A jointly-scaled multilingual language-image model,” _arXiv preprint_ , 2022\.
  * [124] Y. Zeng, X. Zhang, H. Li, J. Wang, J. Zhang, and W. Zhou, “X2-vlm: All-in-one pre-trained model for vision-language tasks,” _arXiv preprint_ , 2022.
  * [125] C. Li, H. Xu, J. Tian, W. Wang, M. Yan, B. Bi, J. Ye, H. Chen, G. Xu, Z. Cao _et al._ , “mplug: Effective and efficient vision-language learning by cross-modal skip-connections,” _arXiv preprint_ , 2022.

Henry Senior is a PhD student at Queen Mary, University of London in the
Digital Environment Research Institute (DERI). His research focuses on the
development and application of Graph Neural Networks to Vision-Language
problems such as image captioning. Before joining DERI, Henry completed a BSc
(Hons) in Computer Science with Professional Experience at the University of
Salford and an MSc in Advanced Computer Science at the University of
Liverpool.
---
Gregory Slabaugh is Professor of Computer Vision and AI and Director of the
Digital Environment Research Institute (DERI) at Queen Mary University of
London. Previously, he was Chief Scientist in Computer Vision (EU) for Huawei
Technologies R&D. His research interests include computational photography,
medical image computing, and applications of deep learning.
---
Shanxin Yuan is a Lecturer in the Digital Environment Research Institute
(DERI) at Queen Mary University of London. He received his PhD degree from
Imperial College London, where he researched on 3D hand pose estimation. His
research interests are machine learning and computer vision, particularly 3D
digital humans and computational photography. He regularly reviews for major
computer vision conferences (CVPR, ICCV, ECCV, and NeurIPS) and related
journals (TPAMI, IJCV and TIP).
---
Luca Rossi received his PhD degree in computer science from Ca’ Foscari
University of Venice, Italy, in 2013. He is currently an assistant professor
with the Hong Kong Polytechnic University, having held various positions with
Queen Mary University of London, U.K., the University of Birmingham, U.K.,
Aston University, U.K., and the Southern University of Science and Technology,
China. He has published more than 50 papers in international journals and
conferences. His research interests include the areas of pattern recognition,
data mining, and network science. He is currently a member of the editorial
board of the journal Pattern Recognition.
---