# Upgrading the Newsroom:
An Automated Image Selection System for News Articles

Fangyu Liu<EMAIL_ADDRESS>Language Technology Lab (LTL), University of
CambridgeCambridgeCambridgeshireUnited Kingdom ,  Rémi Lebret
<EMAIL_ADDRESS>Distributed Information System Lab (LSIR),
EPFLLausanneSwitzerland ,  Didier Orel<EMAIL_ADDRESS>TamediaSwitzerland ,  Philippe Sordet<EMAIL_ADDRESS>TamediaSwitzerland and  Karl Aberer<EMAIL_ADDRESS>Distributed
Information System Lab (LSIR), EPFLLausanneSwitzerland

###### Abstract.

We propose an automated image selection system to assist photo editors in
selecting suitable images for news articles. The system fuses multiple textual
sources extracted from news articles and accepts multilingual inputs. It is
equipped with char-level word embeddings to help both modeling morphologically
rich languages, e.g. German, and transferring knowledge across nearby
languages. The text encoder adopts a hierarchical self-attention mechanism to
attend more to both key words within a piece of text and informative
components of a news article. We extensively experiment our system on a large-
scale text-image database containing multimodal multilingual news articles
collected from Swiss local news media websites. The system is compared with
multiple baselines with ablation studies and is shown to beat existing text-
image retrieval methods in a weakly-supervised learning setting. Besides, we
also offer insights on the advantage of using multiple textual sources and
multilingual data.

multimodal retrieval, multimodal machine learning, neural networks, deep
learning, natural language processing, news image article analysis, news media

††copyright: none††copyright: acmcopyright††journal: TOMM††journalyear:
2020††journalvolume: 1††journalnumber: 1††article: 1††publicationmonth:
1††price: 15.00††doi: 10.1145/3396520

##  1\. Introduction

Motivation. Photo editors play a key role in the traditional workflow of the
journalism industry. They are usually in charge of a huge image database that
belongs to the journal and capable of picking suitable images for news in a
very short time. With the development of information infrastructure and the
internet, the collection and distribution of image data also have been much
easier. Relying on human brains to memorize the database and find matches is
becoming unsustainable. While the flood of data is hard for the human to
process, it provides abundant examples that enable the training of data-hungry
machine learning algorithms in a self-supervised manner.

Here, we propose the task of News Image Selection. With tons of (online) news
samples composed of the headline, lead, an image with a caption and the
article body, we build a system that learns from these samples, takes in only
texts then ranks image candidates for the news. The task resembles _query
based visual search_ task like Google Image Search. Entering the deep learning
era, novel models have been proposed for query based visual search. This line
of works adopts a two-branch architecture where each branch is an encoder for
image or text. Image and text are both encoded into vectors to be compared
with some similarity metric in a joint space. By jointly modeling vision and
language, the class of models are usually referred to as Visual Semantic
Embeddings (VSE). However, things are different in several ways inside the
Newsroom. First, instead of one line of text (in the form of phrases or
keywords) as we input to Google Image Search, news articles contain multiple
text sources with different features and characteristics. Unlike phrases or
keywords, these text inputs are considerably longer, and most of them (image
caption, lead and article body) are composed of complete sentences with clear
structures. Also, they are written for readers, containing (subjective)
discussions or arguments, not necessarily describing an image. Second,
previous VSEs are usually trained on academic datasets like MS-COCO (Lin et
al., 2014) and Flickr30k (Plummer et al., 2015), which are human annotated,
containing clean and accurate labels. We will show that the state-of-the-art
model on (Lin et al., 2014; Plummer et al., 2015) fails on News Image
Selection. We also will discuss why and rationalize how training on a noisy
real-world dataset has significantly changed how we should train and evaluate
the models.

Figure 1. A screenshot of the news image selection system’s interface. Windows
on the left-hand side hold the news article inputs with options to choose the
desired entity(ies), the number of images expected and model choice while
right-hand side lists the retrieved images in ranked order. This demo can be
accessed at https://modemos.epfl.ch/article.
Notice that the demo image dataset contains only limited published images. In
the real world industrial setting, we extend this to a much larger full
archival image database.

With the above analysis considered, we propose an image selection system for
news that fuses multiple textual sources with a hierarchical attention
mechanism. First, we conduct word-level attention within each textual source
to select out the informative keywords. As regardless of the structure and
layout of the article and sentences, the event or entity (in the form of
keywords) the news talks about should be what to emphasize. Then, we apply
component-level attention to decide how much to rely on each textual source.
As in application, certain textual source would potentially be missing. Also,
even if all modalities are complete, they contain different level of
information that varies from news to news. Besides hierarchical attention, we
also introduce the use of pre-trained subword embeddings and novel training
techniques for multiple text sources. In the Experiment section, we will both
qualitatively and quantitatively show how these improvements have made our
image selection system to perform better than baselines.

We mainly focus on enhancing text encoder in this work as the _de facto_ image
encoders - deep Convolutional Neural Networks (CNN) pre-trained on ImageNet
(Deng et al., 2009) yield high-quality image encodings and are robust image
feature extractors in almost every vision-related task including Image
Retrieval. Comparatively, text encoders in the text-image retrieval literature
are less impressive and also not being paid enough attention to. Simple models
like randomly initialized standard word embeddings plus single layer Recurrent
Neural Network (RNN) or CNN were used since the start of this line of work
(Kiros et al., 2015) (2014) and are still being used today (Faghri et al.,
2018; Liu et al., 2020) (2018 & 2020). Our work would be one of the first ones
to use n-gram subword embeddings and self-attention mechanism in the context
of Image Retrieval.

Figure 2. Architecture of the multi-textual-source model. On the left branch,
the four textual sources are input parallelly into word embedding. Embeddings
of each source go into its corresponding text encoder. A feature fuser further
combines encodings from the four textual sources into one single text
encoding. On the right branch, a ResNet152 pre-trained on ImageNet encodes
input image into a vector. The vector goes through a linear transformation and
is then regarded as image encoding. Cosine similarity is used for computing
rankings among text encodings and image encodings.

In addition, there is another crucial aspect of Newsrooms - they (potentially)
need to work multilingually, as we do here in Switzerland. It is also
reflected in our database - it contains both German and French news articles.
Simple searching methods based on keywords matching on metadata (which have
been used in Newsrooms nowadays) clearly fail in the multilingual setting as
only images attached to the specific language will be retrieved (not to
mention that the quality of tagging in metadata is heterogeneous). We thus
propose a language-agnostic model using aligned multilingual word embeddings.
Being trained in both German and French, it outperforms models trained on
monolingual data in both German and French text-image retrieval. Also, we
explore the potential of transferring knowledge across language using our
multilingual model. Starting with model weights trained on language A, we can
achieve much better retrieval performance on language B by substituting only
the word embedding and finetuning.

Contributions. In this paper, we present a novel system for multimodal
multilingual bidirectional retrieval with image and text as modalities. It is
equipped with advanced language encoders inspired by latest advancements in
machine translation (Bahdanau et al., 2015; Vaswani et al., 2017; Lample et
al., 2018a) and char-level word embeddings (Bojanowski et al., 2017),
considering subword information and utilizing attention mechanisms in
different levels. The system benefits from fusing multiple textual sources and
learns from multilingual training data, demonstrating outstanding performance
in the News Image Selection task. We experiment with our system on a database
collected from Swiss online news websites, larger than any existing public
text-image dataset. We compare our model with various baselines on the
database with ablation studies, showing that the proposed model is more
suitable for News Image Selection than existing text-image retrieval methods.

Our main contributions are as follows:

  * •

Proposing the task of News Image Selection - a large-scale, multimodal,
multilingual and weakly supervised machine learning task;

  * •

An attention-based image retrieval system fusing different sources of text
inputs and accepting different languages;

  * •

A recipe for training multimodal multilingual models and transferring
knowledge among languages, achieving better results in all languages on the
task of text-image retrieval.

The rest of the paper is organized as follows: in Section 2, we discuss
related work; Section 3 describes our multimodal multilingual news database;
in Section 4, we describe our multimodal bidirectional retrieval approach; in
Section 5, we introduce the ingredients for adapting the retrieval model to
online news articles; in Section 6, we discuss our experiments and performance
results; then we reflect the limitation of this work in Section 7 and conclude
in Section 8.

##  2\. Related Works

###  2.1. (News) Image Recommendation and Analysis

The News Image Retrieval task has not been formally defined and well-studied
before. There are, however, similar problems related to web (news) images
being investigated in web content analysis community.

Project DBpedia (Auer et al., 2007) aims to extract structured information
from Wikipedia. Khalid and Noah (2011) utilize this structured information to
retrieve news images. Chatfield et al. (2015) study visually searching large-
scale video datasets for semantic entities specified by a text query. Feng and
Lapata (2010b, a, 2013) propose neural methods for automatically captioning
news images. Lin et al. (2018) build a system to retrieve relevant
visualizations that already exist on the web, given a news article. However,
(Auer et al., 2007; Khalid and Noah, 2011) are very out of date; (Chatfield et
al., 2015) mainly focuses on short-query based searching which is more similar
to Google-Image-Search-style visual search; (Feng and Lapata, 2010b, a,
2013)’s task is fundamentally different as it seeks a caption for an image but
not the other way around; (Lin et al., 2018) is limited to retrieve only
visualizations which already have context themselves.

Ramisa et al. (2018) propose a news text-image dataset called BreakingNews
collected from English news media websites like _BBC News_ , _The Guardian_
and _The Washington Post_. It contains around 100k news articles. Along with
the dataset, Ramisa (2017) propose several tasks regarding multimodal news
analysis like source detection, text illustration, and image caption
generation etc.. However, they do not define any retrieval task, which would
be what we are proposing. Also, the dataset we use is bilingual (German &
French) and significantly larger ($>$500k samples) than BreakingNews
($\approx$100k samples). We formally introduce our dataset in details in
Section 3.

###  2.2. Text-Image Retrieval and Visual Semantic Embeddings

_Siamese Network_ (Bromley et al., 1994) introduces the idea of using two
identical networks to find a similarity or a relationship between two
comparable things. It provides a basic framework for combining or comparing
different modalities. Besides (Bromley et al., 1994) which is designed for
discriminating signatures, later works use similar structures for face
verification (Chopra et al., 2005), dimension reduction (Hadsell et al.,
2006), measuring semantic similarity (Huang et al., 2013), etc.

Entering the era of deep learning, works emerged using this two-branch
structure to connect both language and vision. Frome et al. (2013) bring up
the idea called _Visual-Semantic Embedding (VSE)_ to embed pairs of (text,
image) data and compare them in a joint space. Later works extend _VSE_ for
the task of text-image retrieval (Hodosh et al., 2013; Kiros et al., 2015;
Gong et al., 2014; Ma et al., 2015; Vendrov et al., 2016; Hubert Tsai et al.,
2017; Wang et al., 2018b, a; Lee et al., 2018; Faghri et al., 2018; Liu and
Ye, 2019; Wu et al., 2019b; Li et al., 2019; Liu et al., 2020), which is
similar to our work when using a single text source. Notice that text-image
retrieval is different from generating novel captions for images, e.g. (Lebret
et al., 2015; Karpathy and Fei-Fei, 2015; Feng et al., 2019), but to retrieve
existing descriptive texts or images in a database. Another line of highly
related work is called Story Visualization (Ravi et al., 2018; Chen et al.,
2019b), which seeks to retrieve a sequence of images that describes a piece of
text (composed of multiple sentences). Our News Image Selection task aims to
recommend atomic images for news articles instead of retrieving sequential
visualizations.

The text-image retrieval literature mainly validates models on small-scale
well-labeled standard text-image datasets. We argue that the News Image
Selection task differs from standard text-image retrieval not only in terms of
its multi-sources of textual inputs (and potentially multilingualism) but more
importantly, also it is put in the context of weakly supervised learning.
Faghri et al. (2018) propose a hardest negative mining method used by almost
all state-of-the-art systems across several academic datasets (Engilberge et
al., 2018; Lee et al., 2018; Wu et al., 2019a; Li et al., 2019). However, it
performs poorly on our data. In Section 6.1.1, we will discuss in details that
why it fails. And we would discuss in details how this weakly-supervised-
learning setup has made a major difference when it comes to creating and
evaluating models. We also conduct extensive experiments comparing our
proposed model with previous works.

##  3\. Dataset

###  3.1. Dataset Description

Our dataset consists of 528,474 articles published between 2007-11-19 and
2018-05-08 on _20 Minuten_ , _Der Bund_ , _Tages Anzeiger_ , _Berner Zeitung_
, _Zürichsee Zeitung_ , _Der Landbote_ , _Thuner Tagblatt_ , etc. (German);
_20 Minutes_ , _Le Matin_ , _24 Heures_ , _Tribune de Gèneve_ , etc. (French)
which are popular news websites in Switzerland. As seen in Figure 5, all news
articles contain at least one image along with the four textual modalities,
listed below by their level of abstraction from the highest to the lowest:

  * •

_Headline_ : text above a newspaper article, indicating its topic.

  * •

_Lead_ : sentence capturing the attention of the reader and summing up the
focus of the story.

  * •

_Caption_ : basic information needed to understand the image and its relevance
to the news111One may raise the concern that a caption is only identified
after an image is selected and should not be used as part of a query. But
actually, captions rarely directly describe their corresponding images and may
as well be written before selecting an image..

  * •

_Body_ : multiple paragraphs with quotes, details, and elaboration.

In the rest of the paper, we call a tuple of (image, caption, body, headline,
lead) one sample where the first entry – image – means digital photos and the
remaining four entries are the textual elements. There are four national
languages in Switzerland: German, French, Italian, and Romansh. The two
dominant languages being German and French222In 2015, the population of
Switzerland was 63.0% native speakers of German, 22.7% French, 8.4% Italian;
and 0.6% Romansh., our dataset contains 350,204 German samples and 178,270
French samples.

###  3.2. A Weakly-supervised Learning Task

We view the News Image Selection task as a weakly-supervised learning problem
for two reasons.

First, from the perspective of optimization, we have a relatively loose metric
for _good_ retrievals. In standard text-image retrieval, for a specific text,
the _good_ image in a database is usually unique and models aim to rank that
image as high as possible. However, we do not rigorously require the exact
positive image being ranked in the top. Instead, we expect a group of related
(_good_) images being ranked in the top area. As long as images in the top are
meaningful, we actually do not care if the original corresponded image is
among them or not.

Secondly, in terms of training data, we only have a weakly-labeled dataset.
The data is weakly-labeled for reasons of three folds: 1) There is a
discrepancy between the annotations and true semantic text-image links. For
one text, more than one image could have been considered descriptive in the
database. Contrarily, more than one text might be considered descriptive for
one image. Though we are assuming the one-to-one correspondence between the
two modalities to hold for training & testing, the _true_ semantic links
should be more heterogeneous in reality. 2) (text, image) pairs have very
different levels of semantic correspondences. Some texts might be descriptive
in a high level but would require extra knowledge to see their connections and
some might even be very non-descriptive to each other as the selection of
images is subjective to every photo editor. 3) In the multi-textual-source
setting, one or more textual sources are sometimes missing, leaving the
grounding signal to be weak and inconsistent.

Methodology-wise, the weakly supervised learning setting has affected our
choice of learning objective and training strategy: 1) We opt for a loss
function that resists inaccurate/false labels well, which is different from
what has being used for most state-of-the-art models on MS-COCO. We justify
our choice with quantitative evidence and extensively discuss the intuition in
Section 6.1.1. 2) We introduce a _random_drop_ regularization technique to
accommodate missing textual source(s) during training, preventing the model
from overly relying on specific textual input(s). We explain more of its
effectiveness in Section 6.2.

##  4\. A Multimodal Bidirectional Retrieval System

In this section, we describe the details of our retrieval system. Given a
single text source or multiple text sources as an input, we define our goal as
retrieving the best matching images, or vice versa. We introduce our system
that achieves this goal in three main modules:

  1. (1)

a bidirectional retrieval framework (Section 4.1);

  2. (2)

an image encoder which represents the image input in a $d$-dimensional vector
(Section 4.2);

  3. (3)

a text encoder which represents the text input(s) in a $d$-dimensional vector
(Section 4.3).

###  4.1. Bidirectional Retrieval Framework

Figure 3. This figure demonstrates a single-textual-source model’s workflow in
details. For a (text, image) pair, we input the text into the left branch and
image to the right branch to get encodings for both modalities. With one pair
of text and image encodings, we can measure their similarity using cosine.
Having obtained encodings for all the text and image samples, any combination
of (text, image) can be measured and for any of text or image. We can then
give a ranking of its closest counterpart in the opposite modality.

Model. Our bidirectional333It is called “bidirectional” as it can conduct both
text-to-image and image-to-text retrieval. retrieval framework consists of a
text encoder and an image encoder. For a text-image pair $(t,i)$, the text
encoder acts as a function $f$ which encodes $t$ into a vector representation
$f(t;\theta_{t})\in\mathbb{R}^{d}$; similarly, the image encoder encodes $i$
into another vector representation $g(i;\theta_{i})\in\mathbb{R}^{d}$, where
$\theta_{t},\theta_{i}$ are parameters of the two encoders and $d$ is
dimension of the vector representations. To ensure that the encoded
representations lie in a unit hyperspace, we further normalize them by
$l^{2}$-norm. The text and image representations $\mathbf{x}_{t}$ and
$\mathbf{x}_{i}$ would respectively be:

$\mathbf{x}_{t}=\frac{f(t;\theta_{t})}{\|f(t;\theta_{t})\|_{2}}\in\mathbb{R}^{d},\mathbf{x}_{i}=\frac{g(i;\theta_{i})}{\|g(i;\theta_{i})\|_{2}}\in\mathbb{R}^{d}.$

The encoded and normalized text and image representations are of the same size
and live in a joint space, convenient for us to compare their similarities. We
define a simple inner product and obtain a scalar similarity score from it:

$s(i,t)=\langle\mathbf{x}_{i},\mathbf{x}_{t}\rangle:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}.$

Training. Let $\theta$ denote all the model parameters that receive gradients.
Let $I,T$ be all the images and texts samples respectively. We optimize a
margin based triplet ranking loss:

(1) $\begin{split}\mathcal{L_{\textsc{Sum}}}=\sum_{i\in I}\sum_{\bar{t}\in
T\backslash\\{t\\}}[\alpha-s(i,t)+s(i,\bar{t})]_{+}+\sum_{t\in
T}\sum_{\bar{i}\in
I\backslash\\{i\\}}[\alpha-s(t,i)+s(t,\bar{i})]_{+},\end{split}$

where $[\cdot]_{+}=\max\\{0,\cdot\\}$; $\alpha$ is a preset margin; $t$ is the
descriptive text for image $i$ and vice versa; $\bar{t}$ denotes non-
descriptive texts for $i$ while $\bar{i}$ denotes non-descriptive images for
$t$.

###  4.2. Image encoder

We use a Convolutional Neural Network (CNN) pre-trained on ImageNet (Deng et
al., 2009) as a feature extractor for images. Many off-the-shelf image
classification neural nets like ResNet (He et al., 2016), VGG (Simonyan and
Zisserman, 2015) and DenseNet (Huang et al., 2017) would be suitable for this
need. For empirical reasons, we adopt ResNet152 as the image encoder
architecture. We take the output of the layer before logits (which would be a
vector of size $d^{\prime}$) and send it into to a linear layer to be mapped
to its required size $d$. So, the image encoder is composed of two parts: 1) a
chopped pre-trained ResNet152 (frozen) and 2) a linear transformation
(trainable). We denote the parameters of the chopped pre-trained ResNet152 and
the linear transformation as $\theta_{i_{1}},\theta_{i_{2}}$ respectively
where $\theta_{i_{2}}\in\mathbb{R}^{d^{\prime}\times d}$ will simply be
applied as a matrix multiplication. The image encoder as a whole is thus
parameterized by $\theta_{i}=\\{\theta_{i_{1}},\theta_{i_{2}}\\}$. Since
$\theta_{i_{1}}$ is fixed, trainable parameters in Equation 1 would result in
$\theta=\\{\theta_{t},\theta_{i_{2}}\\}$. The trainable and frozen parts of
the model are also shown in Figure 3. Denoting
$g^{\prime}\in\mathbb{R}^{d^{\prime}}$ the image representation extracted from
a pre-trained CNN with ImageNet, the image encoder $g$ can be written as:
$g(i;\theta_{i})=g^{\prime}(i;\theta_{i_{1}})\cdot\theta_{i_{2}}$.

###  4.3. Text encoder

Text encoder consists of a word embedding layer, a Multi-Head Attention layer,
and a Point-Wise CNN. It resembles the Transformer architecture by Vaswani et
al. (2017) and utilizes the attention mechanism (Bahdanau et al., 2015) to
softly align keywords in texts with latent visual objects.

We first walk through the pipeline of the text encoder as suggested in Figure
3:

  1. (1)

Assuming that the raw text $t$ has been pre-processed and tokenized, a word
embedding layer maps every token in the sequence into a word embedding of size
$w$. Given the sequence of $l$ tokens, we get a word embedding matrix of size
$(l,w)$.

  2. (2)

The Multi-Head Attention takes the word embedding matrix in and produces a
feature matrix of the same size.444Multi-Head Attention also outputs $h$
attention maps of size $(l,l)$ supposing that it is a $h$-head attention
layer. We preserve the attention maps for the purpose of visualization.
Details can be found in Figure 8.

  3. (3)

The feature matrix is given as input to a Point-Wise CNN which produces an
encoding of size $(l,d)$, where $d$ would be the preset text encoding feature
size.

  4. (4)

A word-wise Global Max Pooling is then applied to obtain the final text
representation $f(t;\theta_{t})\in\mathbb{R}^{d}$.

Appropriate word embeddings and an efficient attention mechanism are the two
critical factors in the success of our task. When humans try to extract
valuable information from news articles, they look for _entities_ only, rather
than trying to understand complete sentences or structure of the article. This
is where an attention mechanism plays its role: capturing _entities_. However,
in regular (pre-trained) word embeddings, _entities_ are more likely to be
out-of-vocab than other words as they are usually context (language and
location) specific. Without appropriate numerical representations of words,
the attention mechanism would also fail. So, we enrich word embeddings with
subword information to better encode entities even if they are out-of-vocab as
a full word. We now explain each part of the text encoder in detail.

####  4.3.1. fastText n-gram word & subword embedding

We adopt the fastText Language Model pre-trained word and n-gram subword
embeddings (Bojanowski et al., 2017). fastText pre-trained word embeddings are
considered as state-of-the-art word vectors. Also, n-gram subword embedding
helps a lot for dealing with out-of-vocab words.

The general idea is that a word would be represented as the average of the
vector representations of itself and its n-gram subwords. For example,
considering the word newsroom. A regular word embedding layer looks it up in
its vocabulary and return its vector representation if it’s in-vocab or the
vector for <unk> if it’s out-of-vocab. But with our word & subword embedding
layer, it would be represented as the average of all word vectors of the set

{$\underbrace{\color[rgb]{0,0.44921875,0.8046875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.44921875,0.8046875}\text{<newsroom>}}_{\text{original
word}}$,
$\underbrace{\text{{\color[rgb]{0.86328125,0.078125,0.234375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.078125,0.234375}<new},{\color[rgb]{0.86328125,0.078125,0.234375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.078125,0.234375}news},{\color[rgb]{0.86328125,0.078125,0.234375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.078125,0.234375}ewsr},{\color[rgb]{0.86328125,0.078125,0.234375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.078125,0.234375}wsro},{\color[rgb]{0.86328125,0.078125,0.234375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.078125,0.234375}sroo},{\color[rgb]{0.86328125,0.078125,0.234375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.078125,0.234375}room},{\color[rgb]{0.86328125,0.078125,0.234375}\definecolor[named]{pgfstrokecolor}{rgb}{0.86328125,0.078125,0.234375}oom>}}}_{3-\text{grams}}$,
$\underbrace{\text{{\color[rgb]{0.33203125,0.41796875,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.33203125,0.41796875,0.18359375}<news},{\color[rgb]{0.33203125,0.41796875,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.33203125,0.41796875,0.18359375}newsr},{\color[rgb]{0.33203125,0.41796875,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.33203125,0.41796875,0.18359375}ewsro},{\color[rgb]{0.33203125,0.41796875,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.33203125,0.41796875,0.18359375}wsroo},{\color[rgb]{0.33203125,0.41796875,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.33203125,0.41796875,0.18359375}sroom},{\color[rgb]{0.33203125,0.41796875,0.18359375}\definecolor[named]{pgfstrokecolor}{rgb}{0.33203125,0.41796875,0.18359375}room>}}}_{4-\text{grams}}$,
$\underbrace{\text{{\color[rgb]{0.58203125,0,0.828125}\definecolor[named]{pgfstrokecolor}{rgb}{0.58203125,0,0.828125}
<newsr},{\color[rgb]{0.58203125,0,0.828125}\definecolor[named]{pgfstrokecolor}{rgb}{0.58203125,0,0.828125}newsro},{\color[rgb]{0.58203125,0,0.828125}\definecolor[named]{pgfstrokecolor}{rgb}{0.58203125,0,0.828125}ewsroo},{\color[rgb]{0.58203125,0,0.828125}\definecolor[named]{pgfstrokecolor}{rgb}{0.58203125,0,0.828125}wsroom},{\color[rgb]{0.58203125,0,0.828125}\definecolor[named]{pgfstrokecolor}{rgb}{0.58203125,0,0.828125}sroom>}}}_{5-\text{grams}}$}.555We
use 3 to 5-grams in our model as it empirically works the best for German.

The word has been given a more continuous representation in which morphology
of words would be encoded. This is a simple but extremely powerful technique
in our context as it would tolerate typos, as well as compound words, and
could even help to transfer knowledge across nearby languages. We would
discuss more these benefits of subword information in the Experiments section
6.1.3.

####  4.3.2. Multi-Head Attention

Multi-Head Attention has first been introduced to improve Neural Machine
Translation (NMT) systems (Vaswani et al., 2017). While recent NMT systems are
mainly based on RNN (Johnson et al., 2017) or CNN (Kalchbrenner et al., 2016;
Gehring et al., 2017) architectures, the original paper presents a pure
attention-based architecture called _Transformer_ achieving state-of-the-art
performance on English-to-German and English-to-French translation tasks. As
our task does not require outputting a corresponding sequence (translation)
with order and grammar, we removed two modules from the original architecture:
1) the positional encoding which memorizes the input sequence order; 2) the
masks which prevent seeing future words.

Our Multi-Head Attention layer consists of $h$ parallel heads where every head
is acting as a Self-Attention layer.

##### Self-Attention

A Self-Attention layer helps the encoder look at other words in the input
sentence as it encodes a specific word. It creates three vectors from each of
the encoder’s input vectors (in this case, the embedding of each word). So for
each word, it creates a Query vector, a Key vector, and a Value vector. These
vectors are computed by multiplying the word embedding matrix
$E\in\mathbb{R}^{l\times w}$ by three matrices
$W^{Q},W^{K}\in\mathbb{R}^{w\times d_{k}},W^{V}\in\mathbb{R}^{w\times d_{v}}$
that we trained during the training process. Explicitly, it uses the following
equation:

(2)
$\text{SelfAttention}(E)=\text{Softmax}\Big{(}\frac{(EW^{Q})(EW^{K})^{T}}{\sqrt{d_{k}}}\Big{)}(EW^{V}),$

where $d_{k},d_{v}\in\mathbb{N}^{+}$ are hyper-parameters (we use
$d_{k}=d_{v}=64$ for our model). The output is of size $l\times d_{v}$.

For simplicity, we further denote $Q=EW^{Q}\in\mathbb{R}^{l\times
d_{k}},K=EW^{K}\in\mathbb{R}^{l\times d_{k}},V=EW^{V}\in\mathbb{R}^{l\times
d_{v}}$ where $Q,K,V$ are called _query_ , _key_ and _value_ respectively.
Notice that with _query_ -_key_ matrix $QK^{T}\in\mathbb{R}^{l\times l}$ we
could already do a weighted sum on the input. We further scale it with
$\sqrt{d_{k}}$ and send it to Softmax function for obtaining weight maps with
row sum and column sum that equals to $1$:

(3) $M=\text{Softmax}\Big{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Big{)}.$

We call $M\in\mathbb{R}^{l\times l}$ an attention map as it decides
dependencies of the output vectors on the input vectors. The map is preserved
to be used for visualization like the toy example in Figure 4. We discuss more
how attention maps can be used to interpret model’s behavior in Experiments
section 6.1.2.

Figure 4. A screenshot of visualizing attention scores in our web application.
(Best seen in colors, attention decreases in the order of red, orange, yellow
and black.)

##### Multi-Head

For a $h$-Head Attention (we used $h=6$), it concatenates outputs of $h$
independent Self-Attention layers and applies a linear transformation as
suggested below:

$\begin{split}\text{MultiHead}(E)=[\text{head}_{1},\cdots,\text{head}_{h}]W^{O}+E\\\
\text{where }\text{head}_{i}=\text{SelfAttention}_{i}(E)\end{split}$

where $W^{O}\in\mathbb{R}^{hd_{v}\times d_{\text{model}}}$ are the weights of
a linear transformation to be trained (we use $d_{\text{model}}=w$ for our
model) ; $[\cdot]$ denotes the concatenation. It is worth mentioning that
every $\text{head}_{i}$ is computed from its exclusive
$\text{SelfAttention}_{i}$ function, which means that there are $h$
SelfAttention modules working in parallel with their own independent weights.
The Multi-Head Attention layer also adds its input directly to the final
output which forms a residual connection.

####  4.3.3. Point-Wise CNN

The output of the Multi-Head Attention is sent into a Point-Wise CNN where
kernels are all of size $1$:

$\begin{split}\text{PointWiseCNN}(A)=\text{ReLU}(AW_{1}+b_{1})W_{2}+b_{2},\end{split}$

where $A\in\mathbb{R}^{l\times d_{\text{model}}}$ is the output of the Multi-
Head Attention layer; $W_{1}\in\mathbb{R}^{d_{\text{model}}\times
d_{\text{hidden}}},W_{2}\in\mathbb{R}^{d_{\text{hidden}}\times d}$ are weights
to be trained ($d_{\text{hidden}}$ is a hyper-parameter, $d$ is the vector
dimension of the final text encoding, we use $d_{\text{hidden}}=2048,d=1024)$;
$b_{1},b_{2}$ are biases. We use a rectified linear unit (ReLU) (Glorot et
al., 2011) as activation function.

####  4.3.4. Global Max Pooling

After the Point-Wise CNN layer, we obtain a matrix of size ($l\times d$). We
then apply a Global Max Pooling on the last dimension of this matrix to get
the final representation of the text input $t$,
$\mathbf{x}_{t}\in\mathbb{R}^{d}$.

##  5\. Special Ingredients for News Articles

Different from standard Image Retrieval, News Image Selection task has some
special ingredients: it incorporates multiple text modalities (image caption,
article body, headline, lead) and sometimes required to accept multiple input
languages.

In the following two subsections, we explore the use of special ingredients in
our news articles recipe. We aim to build models that, first, utilize more
context (textual modalities) to do better searching; second, learns from
multiple languages and accepts multilingual inputs.

###  5.1. The More the Better - Textual Sources Fusion

The dataset described in Section 3 provides more than one text source, we make
use of them. To fuse these text sources, we use more attention. Inside each
text source, we’ve already had Multi-Head Attentions to conduct word-level
attention, as described in Section 4.3.2. But viewing from a higher level,
different components of a news would contain different amount of information
varying from news to news. Sometimes it’s the article that tells more about
the content of the image and sometimes it might be the lead, etc. We thus
propose a _Feature Fuser_ that combines fully-connected layers with another
component-level attention deciding the attention to put on each textual source
as suggested in Figure 5.

Figure 5. For each text source input, the output of the text encoder is
stacked together and provided as input to the Self-Attention layer, which
decides how much attention is being paid to each textual source. Then the
output is split into vectors and concatenated to be passed into two linear
layers with a ReLU activation in between.

Our _Feature Fuser_ achieves this goal through the following steps:

  1. (1)

obtains a $d$-dimensional feature vector for each of the $4$ different text
sources;

  2. (2)

stacks these features together into one feature matrix of size $4\times d$;

  3. (3)

applies a Self-Attention (or Single-Head Attention) on it using the exact same
equation for word-level attention in Equation 2;

  4. (4)

reshapes the resulting feature map of size $4\times d$ into $4d$-dimensional
global feature vector by concatenating across its first dimension;

  5. (5)

feeds the global feature vector into two linear layers with a ReLU activation
in between666Notice that the first linear transformation is of size $(4d,4d)$
while the second is $(4d,d)$..

The final $d$-dimensional vector is regarded as the text representation of all
four text sources and is used to compute similarities with image encoding.

###  5.2. Rebuilding a (very small) Tower of Babel - Multilingual Model

In this section, we attempt to overstride language barriers in text-image
retrieval, demonstrating the potential of building language agnostic models
and transferring knowledge across languages. We first briefly discuss how our
multilingual setup is different from previous ones in Section 5.2.1. Then we
introduce two ways of conducting transfer learning in Section 5.2.2 and 5.2.3.

####  5.2.1. Transfer Learning without Parallel Corpus

In human brains, semantic concepts regardless of language are likely to be
grounded similarly in the perceptual system (Pecher and Zwaan, 2005). Recent
works in the natural language understanding community have pointed out that by
grounding multilingual data visually, a model can yield improved bilingual
text representations and achieve better results in visual-language tasks
including text-image retrieval (Rajendran et al., 2016; Gella et al., 2017;
Rotman et al., 2018).

However, these proposed methods require parallel text corpus from different
languages along with visual anchors, i.e., a dataset composed of (image, text
of lang. A, text of lang. B) samples, where text of lang. A and lang. B
describes a common image. Although there are indeed human-labeled academic
datasets like this as mentioned in Section 3, they rarely exist in real world
settings.

In our setup, without parallel corpus, to still benefit from multilingual
data, we choose to start with already aligned word embeddings. Instead of
relying on multilingual data to align word embeddings, the model can focus on
discovering connections of language and visual representations.

####  5.2.2. One for All

Wouldn’t it be nice to have one model that understands all languages? In our
multilingual context, it is very ideal to have a language agnostic model as
people may search with different languages on a common image database. We
achieve this by utilizing fastText _MUSE_ word embeddings (Lample et al.,
2018a; Lample et al., 2018b; Joulin et al., 2018). In _MUSE_ , word embeddings
of different languages are aligned to have alike shapes and structures in
hyperspace, which means for tokens of similar meanings in different languages,
_MUSE_ outputs similar numerical representations. This fact bridges data from
different languages.

In our image retrieval context, words with identical meanings in different
languages would be grounded by similar images as vision is universal. In terms
of the text encoder, it means: 1) from above, it receives uniform numerical
signals of word vectors that do not depend on language; 2) down below, these
language agnostic signals would always be grounded by images with universal
semantic meanings.

Figure 6. _One_ model _for All_ languages. Though the model is only trained on
German and French data, it is capable of taking other languages sources with
its corresponding _MUSE_ word embeddings.

Based on the above we propose the model in Figure 6 which can take text inputs
from different languages at the same time within one epoch in training. When
the input language is changed, the word embedding switches to the
corresponding language. To prevent the aligned hyperspaces from being broken,
we freeze the word embeddings of all training languages.

####  5.2.3. All for One

Deep neural nets are data-hungry models. Imagine the scenario when we want to
build a model for language B, but would like to also benefit from data in
language A.

We propose a recipe for transferring knowledge across languages: 1) Initialize
word embeddings with fastText _MUSE_ weights for language A. Freeze word
embeddings. Train model on language A; 2) Substitute the _MUSE_ word
embeddings with language B’s. Add vocabularies that are missing in _MUSE_ word
embeddings. Preserve weights in other parts of the model. Train the model on
language B, fine-tuning the word embeddings. This process can be visualized in
Figure 7.

Figure 7. Utilize data in _All_ languages _for_ training _One_ model.

Low-resource languages can benefit the most from knowledge transfer. With this
recipe, we can pre-train our model on large datasets of high-resource
languages then fine-tune it on our own language of interest.

##  6\. Experiments

Table 1. Comparison between our model with ablation studies (line 1.6-1.11) and baseline models (line 1.1-1.5) in the single-textual-source setting. Recall@10 is reported here. “-” means unable to converge. # | Models |  | image to text |  | text to image
---|---|---|---|---|---
| caption | body | headline | lead |  | caption | body | headline | lead
1.1 |  Vse++ (Faghri et al., 2018) |  | 39.7 | 25.2 | 17.8 | 27.3 |  | 41.2 | 26.7 | 17.1 | 26.7
1.2 |  Vse (Kiros et al., 2015) |  | 42.4 | 32.6 | 22.2 | 29.2 |  | 42.0 | 32.8 | 21.8 | 28.6
1.3 |  Order (Vendrov et al., 2016) |  | 38.9 | 22.1 | - | - |  | 39.3 | 22.6 | - | -
1.4 |  Chain (Wehrmann and Barros, 2018) |  | 22.2 | 10.3 | 16.6 | 14.5 |  | 20.7 | 11.7 | 16.9 | 14.4
1.5 |  Hal (Liu et al., 2020) |  | 49.8 | 32.2 | 23.3 | 33.1 |  | 49.9 | 32.3 | 22.6 | 33.5
1.6 |  Vse \+ _wiki_ |  | 48.3 | 35.8 | 23.0 | 31.7 |  | 48.1 | 36.4 | 22.7 | 32.5
1.7 |  Vse \+ _cc-wiki_ |  | 45.9 | 37.1 | 22.1 | 33.6 |  | 46.2 | 37.7 | 22.3 | 32.4
1.8 |  Vse \+ _wiki-subword_ |  | 52.8 | 34.5 | 24.9 | 32.1 |  | 51.4 | 34.0 | 23.9 | 31.2
1.9 | Ours (w/o attention) |  | 46.2 | 33.8 | 22.7 | 32.2 |  | 46.5 | 34.7 | 22.7 | 32.4
1.10 |  Ours (w/o _wiki-subword_) |  | 45.8 | 28.2 | 20.2 | 26.5 |  | 45.2 | 28.3 | 18.8 | 24.8
1.11 | Ours (full model) |  | 54.2 | 37.2 | 24.2 | 35.4 |  | 52.0 | 36.8 | 22.9 | 32.2

In this section we present experimental results in the order of single textual
source; multiple textual sources; multilingual model. Before diving into them,
we first introduce some general of model configuration, training & evaluation
schemes.

Implementation, Training and data configurations. We implemented our system
using PyTorch and referenced open source projects like (Faghri et al., 2018;
Gehring et al., 2017; Rush, 2018). The models are trained on two NVIDIA TITAN
X Graphics Cards. All single-textual-source models are trained for $30$ epochs
starting with a learning rate of $0.001$ decaying $5$ times for every $10$
epochs. Training details of multi-textual-source models could be found in
Section 6.2. All models are optimized using the Adam optimizer (Kingma and Ba,
2015). The full dataset is split into train set and validation set. For data
of each language, we randomly picked 1.2k samples as the validation set for
that language. All rest are considered as training samples. Quantitative
results in all tables come from evaluations conducted on these validation
sets.

Word Embedding Configurations. We used four kinds of pre-trained word vectors
in experiments:

  * •

fastText Wikipedia word embedding (_wiki_) (Bojanowski et al., 2017);

  * •

fastText Wikipedia word & subword embedding (_wiki-subword_) (Bojanowski et
al., 2017);

  * •

fastText Common Crawl & Wikipedia word embedding (_cc-wiki_) (Grave et al.,
2018);

  * •

fastText Multilingual Unsupervised and Supervised Embeddings (_MUSE_) (Lample
et al., 2018a; Lample et al., 2018b; Joulin et al., 2018).

_wiki-subword_ , _wiki_ , and _MUSE_ are all pre-trained on (German or French,
depending on the language of training model) Wikipedia while _cc-wiki_ is pre-
trained on both Common Crawl and Wikipedia. _wiki-subword_ and _wiki_ are
essentially the same word embedding except that _wiki-subword_ also considers
subword information. Though _cc-wiki_ is pre-trained on more data, it contains
a smaller vocabulary than _wiki_ and _wiki-subword_. Results in Table 1 (_line
1.3, 1.4_) suggests that their performances are comparable while _wiki_ does
better on image caption, which is the textual source we care about the most as
it has the richest information for retrieving images. We thus conduct all
other experiments on _wiki_ and _wiki-subword_. _MUSE_ is only used in the
multilingual setting. We further introduce _MUSE_ in Section 6.3.

Metrics. We evaluate models by the commonly used Recall at $K$ (i.e. R@$K$)
metric in Information Retrieval:

$\text{R}@K=\frac{\text{\\# of queries that the groundtruth item is ranked in
top $K$}}{\text{total \\# of queries}}.$

And we use this metric for both text-to-image and image-to-text retrieval. In
single-textual-source setting, we report only Recall at top 10 retrievals
(R@10). In our News Image Selection task (and also weakly supervised setting),
we do not expect the model ranks the one single positive image as the high as
possible, but to rank all related images in the top range. For such reason, we
consider R@10 as our most relevant metric rather than R@1 or R@5. When fusing
multiple textual sources, the best model achieves nearly $70\%$ for R@10, we
thus include R@1, R@5 and also Median rank (Med r, the lower the better) to
evaluate model from multiple angles.

###  6.1. Models with One Single Textual Source

In this section, we experiment single-textual-source models on German data,
including all textual sources: caption, body, headline, and lead. We
demonstrate experiment results in three perspectives: _Quantitative results_ ,
_Multi-Head Attention as a query system_ and _Subword information matters_.

####  6.1.1. Quantitative results

We compare our model with five baseline models: 1) Vse (Kiros et al., 2015);
Vse++ (Faghri et al., 2018); 3) Order (Vendrov et al., 2016); 4) Chain
(Wehrmann and Barros, 2018); 5) Hal (Liu et al., 2020). According to prior
works (Faghri et al., 2018; Wehrmann and Barros, 2018; Liu et al., 2020) and
our own experiments, an ImageNet pre-trained ResNet152 yields the best image
encodings in our context. To be consistent, we adopt ResNet152 as image
encoder to compare all text encoder architectures. We conduct grid search to
tune the hyperparameters used in loss functions for each baseline. The
presented results are produced using hyperparameters with the best
performance.

Overall, our model (_line 1.11_) demonstrates improvements comparing to all
baseline models under all textual sources and in both text-to-image and image-
to-text task. Among baseline models, the recently proposed Hal performs the
best across almost all textual sources. It uses a novel loss function and
produce state-of-the-art results on MS-COCO (without using object bounding box
labels). We analyze Hal in more details in the coming section of comparing
loss functions. Following Hal, Vse remains a very strong baseline despite
having succinct model architecture, achieving higher scores than Vse++ and
Order. In our industrial setting, models that are sensitive to noise are very
hard to train and can result in poor performance or even failure of
convergence (e.g. when training Order on headline & lead data, line 1.3). We
analyze more on this in the later section of comparing loss functions. Chain
fails significantly, producing the worst results under all textual sources. We
suspect it is due to Chain uses a char-level text encoder and does not
explicitly have a word embedding module. The benefit from using word
embeddings over char embeddings is particularly prominent when the vocabulary
size is large such as in our dataset (¿100k) as opposed to MS-COCO (10K).

Ablation studies. We claim that our model’s outstanding performance mainly
benefit from two modules: 1) subword embeddings and 2) multi-head attention.
_line 1.9_ in Table 1 represents our model without the multi-head attention
module, in which case the text encoding is produced with subword embeddings
followed by a point-wise CNN. The model’s performance drops significantly
comparing to the full model (_line 1.11_) and is also worse than Vse equipped
with subword embeddings (_line 1.8_). _line 1.10_ lists results of our model
without the subword embeddings. The model produces even lower scores. Only
when both modules are combined, our model achieves significantly better
results than baselines.

Why Attention? Here we try to interpret why the transformer-based attentional
encoder performs better than RNNs like LSTM (Hochreiter and Schmidhuber, 1997)
and GRU (Cho et al., 2014) in our setting. RNNs force models to take in tokens
sequentially and encode the information recurrently. However, in our context,
the main objective is spotting the key entities in texts rather than modelling
their semantics. As a results, modeling texts in a recurrent manner might
limit the model’s flexibility regarding to how they encode relations among
different input tokens. Imagine the behavior patterns of human beings. When
trying to find matches of text and images, it is likely that we look at both
text and image as one piece. We do not try to capture the exact semantic
meaning derived from a long and complete sentence but paying attention only to
keywords. We further compare these keywords with images and decide if it is a
match. Attention mechanism also yields more interpretable models. As Self-
Attention is designed based on _key_ , _value_ , _query_ and produces
attention maps from them, we know exactly what is happening to each word
vector and how they are being used to produce features. We discuss this more
with examples in Section 6.1.3.

Comparing loss functions. Faghri et al. (2018) claimed state-of-the-art text-
image retrieval results on MS-COCO (Lin et al., 2014). However, it performs
worse than its prior work (Kiros et al., 2015) on our data. The two models
mainly differ on the loss function where (Kiros et al., 2015) adopts a triplet
ranking loss that rewards positive pairs and punishes all negative pairs
within a mini-batch (same as we do in Equation 1), while (Faghri et al.,
2018), similarly, also rewards positive pairs, but punishes only the _hardest_
negative as shown in Equation 4.

(4) $\begin{split}\mathcal{L_{\textsc{Max}}}=\sum_{i\in I}\max_{\bar{t}\in
T\backslash\\{t\\}}[\alpha-s(i,t)+s(i,\bar{t})]_{+}+\sum_{t\in
T}\max_{\bar{i}\in
I\backslash\\{i\\}}[\alpha-s(t,i)+s(t,\bar{i})]_{+}\end{split}$

In MS-COCO, the descriptions (texts) of an image are usually rich and
accurate. The hardest negative within a mini-batch is likely to have a clear
difference with the positive. We call them _true_ negatives. Put this in
another way, the MS-COCO is providing a clear and strong supervision to the
model. When it comes to our text-image news dataset, the correspondences among
text-image pairs are much weaker. This leads to the fact that, in the context
of our weakly supervised data, the hardest negative is likely to contain very
little negative information or even contains very positive signals (if it
happens to be another descriptive counterpart) - the hard negatives are
actually _pseudo_ hard negatives. As only the hardest sample is selected, if
it happens to be _pseudo_ , the gradients of the whole mini-batch would be
misleading. In contrast, Equation 1 can cancel out false gradients by other
correct negatives within the mini-batch, preventing model being harmed by
wrong labels. Also, we can tell a bit from the single-textual-source model
results in Table 1. Image caption is the strongest supervision (or most
descriptive text) among the four textual sources. It is also the one where the
two loss functions’ performances have the least difference. It agrees to our
interpretation that with a stronger supervision, it is less likely to
encounter _pseudo_ hard negatives. Despite _pseudo_ negatives, noise is also a
possible source of error. (Wu et al., 2017) provides a theoretical analysis of
this problem: when negative examples are very hard, the direction of the
gradient is likely to be dominated by noise, leading to failure of
optimization.

Liu et al. (2020) recently propose a novel learning objective Hal that makes
use of hard samples and avoid _pseudo_ hard negatives at the same time. As
suggested in Table 1, Hal improves R@10 on caption data by $7-8\%$ without
changing any encoder architectures but using the following loss function:

(5)
$\begin{split}\mathcal{L_{\textsc{Hal}}}=\frac{1}{N}\sum_{i=1}^{N}\Big{(}\frac{1}{\alpha}\log(1+\sum_{m\not=i}e^{\alpha(S_{mi}-\epsilon)})+\frac{1}{\alpha}\log(1+\sum_{n\not=i}e^{\alpha(S_{in}-\epsilon)})-\log(1+\beta
S_{ii})\Big{)},\end{split}$

with hyperparameters $N=256,\alpha=20,\beta=30,\epsilon=0.2$. Due to limit of
time and space, we leave the experiments combining Hal with our model for
future works.

We analyzed the choice of learning objective very carefully as it reveals a
gap worth noticing between the datasets used in academia and real-world data.
Benchmarks like MS-COCO (Lin et al., 2014), Flickr8k (Hodosh et al., 2013) and
Flickr30k (Plummer et al., 2015) are a little bit _too perfect_ and may
inadvertently harm the generality of models if not used carefully. When models
were designed based on certain attributes of the toy datasets, they have
potentially followed strong hypothesis which would lead them into overfitting
these sets and won’t incorporate the real-world data who is usually noisier
and weakly labeled.

####  6.1.2. Multi-Head Attention as a query system

Figure 8. From right to left: compute the mean of feature maps and compute the
mean of each row. Attention maps are obtained from $M$ computed by Equation 3.

We argue that the Multi-Head Attention mechanism is acting as a query system
to filter out unimportant words (information) and preserve the important ones.
We validate this by visualizing Multi-Head Attention layer’s attention maps.
We do this by averaging rows of averaged Multi-Head attention maps.

A toy example of explaining how to compute attention scores can be found in
Figure 8. The columns denote features, which are the outputs of the Multi-Head
Attention layer; while the rows denote embeddings of words, which are the
inputs of the Multi-Head Attention layer. For an $h$-Head Attention, it
produces $h$ attention maps of size $(l,l)$ where $l$ is the length of the
input sequence. We average these $h$ maps to obtain a single attention map as
suggested in the figure. For an output feature $i\in\\{1,\cdots,l\\}$,
githuthe $i$-th column of attention map specifies weights assigned to each
input word. In other words, the $i$-th column can be interpreted as the
importance of each word to feature $i$. And accordingly, the $j$-th row
($j\in\\{1,\cdots,l\\}$) can be interpreted as how important word $j$ is to
all the output features. The more a word contributes to (all) output features,
the higher the weights would be on that row. Then it makes sense to average
rows. Words being paid more attention would have higher average scores.

Figure 9. The first row is searching with full text. The key components
_Basistunnel_ (base tunnel) and _arbeiter_ (workers) are both shown in most of
the top 5 retrievals. The second row is searching without the token in red.
_arbeiter_ are still presented however the _Basistunnel_ is gone and scenes
become very random. The third row is searching without all highlighted words.
The results become completely irrelevant. The fourth row, on the contrary, is
searching with only the highlighted words. Retrievals are comparable to
searching with full text. (Best seen in colors.)

Inside the text encoder, Multi-head Attention and Point-Wise CNN are the only
two learnable parts. As all kernels of Point-Wise CNN are of size $1$, they do
not perform any _weighted sum_ operation and treat all vectors in the input
matrix equally. So, Multi-Head Attention is mainly responsible for querying
out useless information as it is doing a global _weighted sum_ and the network
has sufficient freedom to drop worthless information with simply changing
$W^{Q}$ and $W^{K}$ in query-key weight matrix $QK^{T}=(EW^{Q})(EW^{K})^{T}$
(see Equation 2).

As we argue that attention is important for our task, we do an extensive
analysis on how it works in the context of retrieving relevant news images. We
play with the following toy example:

  * •

_Der Bauriese will sich als experte für anspruchsvolle infrastrukturprojekte
etablieren: arbeiter im Gottard-Basistunnel. (20. November 2014)_ 777 _The
construction giant wants to establish itself as an expert for sophisticated
infrastructure projects: workers in the Gottard Base Tunnel. (November 20,
2014)_ (English translation)

Searching this text with our model trained only with captions, the
visualization of attention scores received are presented in the first row of
Figure 9 where attention decreases in the order of red, orange, yellow and
black. We compare the qualitative results of searching with: i) full text; ii)
the token with the highest score crossed out888When crossing out tokens, a
meaningless out-of-vocab placeholder would be used to replace the original
ones. We used a random emoji to do the job.; iii) all highlighted tokens
crossed out; iv) only the highlighted words. All the results can be found in
Figure 9.

We can clearly see that searching with the highlighted words is comparable to
searching with full text. On the contrary, the removal of highlighted words
decreases the quality of retrieved images.

####  6.1.3. Subword information matters (a lot)

In our context of retrieving images using short texts (like headline, lead,
caption), there might only be a few informative words within one piece of
text. These words are usually entities (as news is about entities). If these
entity words are missed, the retrieval task will almost certainly fail. We
argue that our word&subword embedding layer handles entity names by 1)
tolerating typos; 2) recognizing compound and conjunction words and 3)
transferring knowledge across close languages. Please refer to Appendix for
specific examples.

Table 2. Comparing different multi-textual-source feature fusion methods. # | Fusion Method | image to text | text to image
---|---|---|---
R@1 | R@5 | R@10 | Med r | R@1 | R@5 | R@10 | Med r
2.1 | Global Max Pooling | 21.4 | 50.0 | 64.3 | 5.0 | 19.6 | 47.4 | 62.3 | 6.0
2.2 | Global Max Pooling (w/o random drop) | 22.0 | 50.3 | 65.2 | 5.0 | 17.9 | 47.6 | 63.5 | 6.0
2.3 | Element-wise Adding | 21.2 | 50.6 | 64.4 | 5.0 | 20.8 | 49.4 | 63.1 | 6.0
2.4 | Element-wise Adding (w/o random drop) | 21.8 | 50.5 | 65.6 | 5.0 | 17.2 | 46.3 | 62.0 | 6.0
2.5 | Neural Net Fuser | 24.7 | 53.2 | 68.7 | 5.0 | 17.8 | 47.8 | 63.0 | 6.0
2.6 | Neural Net Fuser (w/o random drop) | 25.6 | 54.0 | 67.8 | 5.0 | 20.2 | 49.4 | 62.7 | 6.0
2.7 | Attention Fuser (from scratch) | 14.9 | 39.4 | 54.3 | 9.0 | 12.1 | 33.2 | 47.9 | 12.0
2.8 | Attention Fuser (w/o random drop) | 22.8 | 51.7 | 66.6 | 5.0 | 19.3 | 47.2 | 62.8 | 6.0
2.9 | Attention Fuser | 24.3 | 54.8 | 70.6 | 5.0 | 18.2 | 46.8 | 62.7 | 6.0

###  6.2. Fusion of Multiple Textual Sources

Preserving everything we had for single textual source, in this section we
further experiment methods for fusing information obtained from different
textual inputs. We start by introducing two techniques used during training -
_Divide-and-Conquer_ and _random drop_. Then we compare different fusion
policies including both pre-defined and learnable ones.

Divide-and-Conquer: Starting with Single Source. We find that the multi-
textual-source model can greatly benefit from starting with weights trained on
single textual source. Specifically, for each encoder in the _fourway_ model,
we load their corresponding single-textual-source encoder’s weights. Since the
four single models share a common word embedding in the _fourway_ model, we
initialize the word embedding layer with caption model’s word embeddings
weights (which is the one with the best single-textual-source performance). To
avoid these pre-trained weights receiving random gradients at the starting
phase of joint training, we freeze them for $1$ epoch during which we only
train the _Feature Fuser_. Then all parts of text encoder are unfrozen and
being trained as a whole for another $9$ epochs. We use a learning rate of
$10^{-4}$ for the first $5$ epochs and $2\times 10^{-5}$ for the second $5$
epochs. We empirically find it very important to start with what we already
have as suggested in Table 2. R@10s are improved by $\approx\textbf{15\%}$!
Training four text encoders together from scratch might be too hard a task to
optimize.

Random Drop. To tackle possible missing sources of text inputs in real-world
applications, we use a very simple _random_drop_ mechanism and it unexpectedly
improves R@10s of models with learnable fusion policy.

Briefly, texts from each sources have a probability of being dropped and
substituted with empty string during the training phase. For every sample, one
source of texts is randomly chosen to be preserved (to make sure at least one
meaningful source of input) while for the other three sources, each one has
$0.7$ probability of being kept and $0.3$ probability of being dropped. We
switch back to _complete_inputs_ mode in the validation phase. According to
the validation results in Table 2, models with learnable fusion policy
(_Neural Net Fuser_ and _Attention Fuser_) using _random_drop_ is doing even
better than using _complete_inputs_ in most indicators.

We infer that _random_drop_ is acting as a regularizer in our model. The deep
learning community has developed robust and effective regularization
techniques like dropout (Srivastava et al., 2014) which randomly suppresses
neurons of the network by simply erasing their outputs; or ShakeDrop (Yamada
et al., 2018), which even reverses signs of them. Also, in computer vision, it
is very common to apply random crop on original images to alleviate model from
overfitting. Very recently, (DeVries and Taylor, 2017; Zhong et al., 2017)
propose _random erasing data augmentation_ which is to randomly erase patches
of input images (and fill them with zeros or random values). These are also
regarded as regularization methods. _random_drop_ is following the same idea
of randomly hiding part of the input to prevent the model from relying on some
specific feature (or textual source, in our context).

Other Fusion Methods. We also test other fusion methods in the literature such
as Global Max Pooling, Element-wise Adding, Neural Nets Fusion. We use similar
manner loading pre-trained single-textual-source models, then train them
jointly with random drop. Global Max Pooing is taking the max value of the
four outputs for every element; Element-wise Adding means adding features from
every textual source directly, which should be working similarly as Global
Average Pooling; Neural Nets Fusion comes from (Zahavy et al., 2018) where
they find stacking two blocks of (fully-connected layer + ReLU activation) is
the optimal structure for their multimodal feature fusion task. We compare
these results with our Attention Fuser in Table 2. We notice that Element-wise
Adding performs better in image-to-text task and Attention Fuser is better for
text-to-image.

###  6.3. Multilingual Model

We adopt the recently released Multilingual Unsupervised or Supervised word
Embeddings (_MUSE_). Comparing to fastText _wiki_ which we used in previous
experiments, _MUSE_ is rather new and has a relatively small vocabulary. Since
it is being adjusted for alignment, the quality of its numerical
representations may also be compromised. However, it is promising as a solid
step towards erasing the language barrier in NLP and machine learning.
Experiments show that by using _MUSE_ to bridge multilingual data, our
multilingual model can already beat the best single-textual-source model
without subword information (previous best: $48.1\%$, _MUSE_ best: $48.5\%$).
It is, however, less powerful than the best model with subword embeddings. We
expect better multilingual word embeddings considering subword information
emerge in the near future.

We compare a single-textual-source bidirectional model being trained on
monolingual data and trained on bilingual data (both on image captions),
following the process described in Section 5.2.2. As suggested in Table 3, the
language-agnostic model performs better than the monolingual model in both
German and French.

Table 3. Quantitative results of one-for-all. R@10 is reported here. # |  training language | image to text | text to image
---|---|---|---
de | fr | de | fr
3.1 | fr | - | 39.4 | - | 38.2
3.2 | de | 41.1 | - | 41.4 | -
3.3 | de + fr |  43.1 (+2.0) |  42.0 (+2.6) |  43.3 (+1.9) |  41.7 (+3.5)

Notice that the German single-textual-source model (with regular word
embeddings) results in Table 1, Table 3 and Table 4 are all different (_line
1.10, 3.2, 4.2_). In terms of R@10 for text-to-image, they are _line 1.10_ :
$45.8\%$, _line 3.2_ : $41.1\%$ and _line 4.2_ : $42.1\%$ respectively. This
is due to the three experiments differ in vocabulary and choice of pre-trained
vectors. All three are using only regular full word embeddings pre-trained on
Wikipedia and without subword information, however, _1.10_ is using _wiki_
(Bojanowski et al., 2017) and the word embedding is finetuned while training
while _3.2_ and _4.2_ are using _MUSE_ (Lample et al., 2018a). Though also
pre-trained on Wikipedia, _MUSE_ is further processed and aligned with word
vectors of other languages. And more importantly, to better align with other
languages, _MUSE_ only contains 200k word vectors while _wiki_ has over 2275k,
which is more than 10 times larger. Using _MUSE_ leads to many more out-of-
vocab words who might be the main cause for the drop in performance. Though
both using _MUSE_ , _line 3.2_ and _line 4.2_ differs in vocabulary and
training. To be consistent with the _one-for-all_ experiment setting, _line
3.2_ uses a frozen word embedding layer to prevent the aligned multilingual
space from vanishing while _line 4.2_ finetunes its word embedding. As _line
3.2_ ’s word embeddings are frozen, all out-of-_MUSE_ -vocab words are
considered as unknown while in _line 4.2_ learns out-of-_MUSE_ -vocab word
vectors on the fly. This also explains the difference in French baseline
results in Table 3 and Table 4 (_line 3.1, 4.1_).

Also, in the _all-for-one_ setting as described in Section 5.2.3, where
maintaining aligned word embedding is no longer required, the model trained on
both German and French has improved $7.7\%$ in German and $4.1\%$ in French
for R@10 scores in text-to-image retrieval as suggested in Table 4.

Table 4. Quantitative results of all-for-one. R@10 is reported here. # | training language | image to text | text to image
---|---|---|---
4.1 | fr, full vocab | 41.0 | 39.9
4.2 | de, full vocab | 42.1 | 42.2
4.3 |  de (+ fr), _MUSE_ vocab | 46.4 (+4.3) | 44.1 (+1.9)
4.4 | de (+ fr), full vocab |  49.8 (+7.7) |  48.5 (+6.3)
4.5 |  fr (+ de), _MUSE_ vocab | 44.9 (+3.9) | 43.2 (+3.3)
4.6 | fr (+ de), full vocab |  45.1 (+4.1) |  43.4 (+3.5)

Notice that it is also very important to add out-of-_MUSE_ -vocab vocabularies
into word embedding. As these words might contain lots of language-specific
knowledge. It is especially important for languages with very large
vocabularies (like German). As suggested in Table 4, R@10 only improves
$4.3\%$ if not adding full vocabulary while there is $7.7\%$ improvement with
full vocabulary.

##  7\. Limitation

In this section, we analyze the major limitations of our model and how we
could fix some of them.

_What’s the Real Ground-truth?_ What should be considered a _correct_ image
for a news article is subjective, and the standards are heterogeneous. The
ambiguity of the real article-image relationship has caused inconvenience in
both training and evaluating models. As described in the weakly-supervised
learning setting, the model learns from article-image pairs with very
different levels of correspondences during training, and some samples might
even be contradicting. During the evaluation, the quantitative metrics only
roughly measure the correspondence of articles and images. However, it remains
unclear how exactly judgments should be made towards _correct_ retrievals.

_Photo Illustrations and Factual Photos._ Our model works the best on
retrieving _photo illustrations_ 999Photo Illustrations, or Photo Prétexte in
French, is a notion from Journalism that refers to non-factual photos that
visually illustrate contents of news articles. They usually come from one of
the image banks to which most media subscribe or each media’s own image
collection. It is a controversial topic in the news industry on how photo
illustrations should be used, but they are only getting more and more popular
on the web today as “an article without an image has no chance of generating
traffic” (fot, [n. d.]). rather than factual photos with specific named
entities in it. Photo illustrations mainly appear in news articles relating to
general but random topics such as car accident, traffic jam, snowstorm,
pregnancy, university lab, school, etc.. The retrieval quality for named
entities, however, heavily relies on the frequency that named entity appears
in the (training) database. For instance, it may perform well on really famous
people, eg. “Donald Trump”, but may miss less notable ones. There are also
classes of entities having only subtle visual differences. For instance, when
the query is “Lake Geneva”, the model also retrieves many other lakes as they
look alike. One possible improvement can be creating more sophisticated visual
encoders. For instance, finetuning image encoders, adding object detection
modules and transfer learning from a specific domain could be plausible
options.

_A Fix - Named Entity Filtering. _ News images are usually taken and stored in
a database with _metadata_. The metadata is either given by the photographer
or editors and usually contain the main named entity presented in the image.
Traditionally, news media use keyword matching to search for related images
(text-to-text search). As long as the metadata is accurate, retrieving the
correct named entities is guaranteed. We can partially fix the model’s
weakness in retrieving named entity photos by matching the retrievals’
metadata with named entities detected in the news articles. Specifically, we
use (Honnibal and Montani, 2017) to detect four sorts of named entities - PER,
LOC, ORG and MISC that are presented in the news article. Then a user may
select the desired entity(ies) and all photo candidates without that entity in
its metadata would be filtered out. Notice that the combination of our model
and metadata search is more powerful than metadata search only as it puts
entities in context (sentiment, gestures, companions, environments, etc.). We
provide some more specific examples in the Appendices.

##  8\. Conclusion and Future Work

Our work proposes the News Image Selection task in the context of weakly-
supervised learning and large-scale database. We introduce an automated image
selection system to assist photo editors in a multilingual multimodal setting.
Our system incorporates latest advancement in neural machine translation and
deep learning community like subword embedding and self-attention mechanism.
We also propose novel techniques for multimodal and multilingual learning that
contributes to the natural language understanding and machine learning
literature. We also extensively discuss problematic issues in News Image
Selection, explaining how it has made state-of-the-art text-image retrieval
method failed on our dataset. And we analyze the limitations of our proposed
method and show how combining traditional metadata search can help mitigate
some of the problems. Besides News Image Selection, the system can potentially
be used for other text-image recommendation scenarios. As this work mainly
focuses on enhancing text encoder, we plan to improve image encoder as well in
future works, e.g. incorporating a pre-trained object detector (Ren et al.,
2015) to enhance image encodings. We also plan to explore improve the textual
encodings by exploring more variations of recently emerged pre-trained/pre-
aligned multimodal/multilingual embeddings (Devlin et al., 2019; Chen et al.,
2019a; Su et al., 2020).

##  9\. Acknowledgement

We thank Dr Hamza Harkous for helpful discussions and feedbacks. We thank
anonymous reviewers for their comments based on which we were able to enhance
our work. We gratefully acknowledge the support of NVIDIA Corporation with the
donation of the Titan Xp GPU used for this research.

## References

  * (1)
  * fot ([n. d.]) [n. d.].  Les photos ”prétexte” ont envahi la presse.  _france inter_ ([n. d.]).  https://www.franceinter.fr/emissions/l-edito-m/l-edito-m-01-novembre-2018
  * eas ([n. d.]) [n. d.].  Mein nächster Easyjet-Flug ging erst in 5 Tagen.  _20 minuten_ ([n. d.]).  https://www.20min.ch/finance/news/story/-Der-Flug-mit-der-Airline-war-eine-Katastrophe--16483014
  * der ([n. d.]) [n. d.].  Neues Quartierzentrum im Südwesten von Thun geplant.  _Der Bund_ ([n. d.]).  https://www.derbund.ch/bern/region/neues-quartierzentrum-im-suedwesten-von-thun-geplant/story/15516432
  * Auer et al. (2007) S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives. 2007\.  Dbpedia: A nucleus for a web of open data.  In _The semantic web_. Springer, 722–735.
  * Bahdanau et al. (2015) D. Bahdanau, K. Cho, and Y. Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  _International Conference on Learning Representations (ICLR)_ (2015).
  * Bojanowski et al. (2017) P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. 2017\.  Enriching Word Vectors with Subword Information.  _Transactions of the Association for Computational Linguistics (TACL)_ 5 (2017), 135–146.
  * Bromley et al. (1994) J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah. 1994.  Signature verification using a” siamese” time delay neural network. In _Advances in neural information processing systems (NIPS)_. 737–744.
  * Chatfield et al. (2015) K. Chatfield, R. Arandjelović, O. M. Parkhi, and A. Zisserman. 2015.  On-the-fly learning for visual search of large-scale image and video datasets.  _International Journal of Multimedia Information Retrieval_ (2015).
  * Chen et al. (2019a) S. Chen, Q. Jin, and A. Hauptmann. 2019a.  Unsupervised Bilingual Lexicon Induction from Mono-Lingual Multimodal Data. In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , Vol. 33. 8207–8214.
  * Chen et al. (2019b) S. Chen, B. Liu, J. Fu, R. Song, Q. Jin, P. Lin, X. Qi, C. Wang, and J. Zhou. 2019b.  Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences. In _Proceedings of the 27th ACM International Conference on Multimedia_. 2236–2244.
  * Cho et al. (2014) K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. 2014.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1724–1734.
  * Chopra et al. (2005) S. Chopra, R. Hadsell, and Y. LeCun. 2005.  Learning a similarity metric discriminatively, with application to face verification. In _IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , Vol. 1. IEEE, 539–546.
  * Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_. IEEE, 248–255.
  * Devlin et al. (2019) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (NAACL-HLT)_. 4171–4186.
  * DeVries and Taylor (2017) T. DeVries and G. W. Taylor. 2017.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_ (2017).
  * Engilberge et al. (2018) M. Engilberge, L. Chevallier, P. Pérez, and M. Cord. 2018.  Finding beans in burgers: Deep semantic-visual embedding with localization. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 3984–3993.
  * Faghri et al. (2018) F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler. 2018\.  VSE++: Improving Visual-Semantic Embeddings with Hard Negatives.  _Proceedings of the British Machine Vision Conference (BMVC)_ (2018).  https://github.com/fartashf/vsepp
  * Feng and Lapata (2010a) Y. Feng and M. Lapata. 2010a.  How many words is a picture worth? automatic caption generation for news images. In _Proceedings of the 48th annual meeting of the Association for Computational Linguistics (ACL)_. Association for Computational Linguistics, 1239–1249.
  * Feng and Lapata (2010b) Y. Feng and M. Lapata. 2010b.  Topic models for image annotation and text illustration. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)_. Association for Computational Linguistics, 831–839.
  * Feng and Lapata (2013) Y. Feng and M. Lapata. 2013\.  Automatic caption generation for news images.  _IEEE transactions on pattern analysis and machine intelligence (TPAMI)_ 35, 4 (2013), 797–812.
  * Feng et al. (2019) Y. Feng, L. Ma, W. Liu, and J. Luo. 2019\.  Unsupervised image captioning. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_. 4125–4134.
  * Frome et al. (2013) A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, and T. Mikolov. 2013.  Devise: A deep visual-semantic embedding model. In _Advances in neural information processing systems (NIPS)_. 2121–2129.
  * Gehring et al. (2017) J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. 2017.  Convolutional Sequence to Sequence Learning. In _International Conference on Machine Learning (ICML)_. 1243–1252.
  * Gella et al. (2017) S. Gella, R. Sennrich, F. Keller, and M. Lapata. 2017\.  Image Pivoting for Learning Multilingual Multimodal Representations. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 2839–2845.
  * Glorot et al. (2011) X. Glorot, A. Bordes, and Y. Bengio. 2011.  Deep sparse rectifier neural networks. In _Proceedings of the fourteenth international conference on artificial intelligence and statistics (AISTAT)_. 315–323.
  * Gong et al. (2014) Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik. 2014.  Improving image-sentence embeddings using large weakly annotated photo collections. In _European Conference on Computer Vision (ECCV)_. Springer, 529–545.
  * Grave et al. (2018) E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov. 2018.  Learning Word Vectors for 157 Languages. In _Proceedings of the International Conference on Language Resources and Evaluation (LREC)_.
  * Hadsell et al. (2006) R. Hadsell, S. Chopra, and Y. LeCun. 2006.  Dimensionality reduction by learning an invariant mapping. In _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_. 770–778.
  * Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hodosh et al. (2013) M. Hodosh, P. Young, and J. Hockenmaier. 2013.  Framing image description as a ranking task: Data, models and evaluation metrics.  _Journal of Artificial Intelligence Research_ 47 (2013), 853–899.
  * Honnibal and Montani (2017) M. Honnibal and I. Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  _To appear_ (2017).
  * Huang et al. (2017) G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. 2017.  Densely Connected Convolutional Networks.. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , Vol. 1. 3\.
  * Huang et al. (2013) P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck. 2013.  Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. In _ACM International Conference on Information and Knowledge Management (CIKM)_.
  * Hubert Tsai et al. (2017) Y.-H. Hubert Tsai, L.-K. Huang, and R. Salakhutdinov. 2017\.  Learning Robust Visual-Semantic Embeddings. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_. 3571–3580.
  * Johnson et al. (2017) M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, et al. 2017\.  Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation.  _Transactions of the Association of Computational Linguistics (TACL)_ 5, 1 (2017), 339–351.
  * Joulin et al. (2018) A. Joulin, P. Bojanowski, T. Mikolov, H. Jégou, and E. Grave. 2018.  Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Kalchbrenner et al. (2016) N. Kalchbrenner, L. Espeholt, K. Simonyan, A. v. d. Oord, A. Graves, and K. Kavukcuoglu. 2016\.  Neural machine translation in linear time.  _arXiv preprint arXiv:1610.10099_ (2016).
  * Karpathy and Fei-Fei (2015) A. Karpathy and L. Fei-Fei. 2015.  Deep Visual-Semantic Alignments for Generating Image Descriptions. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Khalid and Noah (2011) Y. Khalid and S. Noah. 2011\.  A framework for integrating DBpedia in a multi-modality ontology news image retrieval system. In _2011 International Conference on Semantic Technology and Information Retrieval_. IEEE, 144–149.
  * Kingma and Ba (2015) D. P. Kingma and J. Ba. 2015.  Adam: A method for stochastic optimization.  _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_ (2015).
  * Kiros et al. (2015) R. Kiros, R. Salakhutdinov, and R. S. Zemel. 2015\.  Unifying visual-semantic embeddings with multimodal neural language models.  _Transactions of the Association for Computational Linguistics (TACL)_ (2015).
  * Lample et al. (2018a) G. Lample, A. Conneau, L. Denoyer, and M.-A. Ranzato. 2018a.  Unsupervised Machine Translation Using Monolingual Corpora Only.  _International Conference on Learning Representations (ICLR)_.
  * Lample et al. (2018b) G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou. 2018b.  Word translation without parallel data. In _International Conference on Learning Representations (ICLR)_.  https://openreview.net/forum?id=H196sainb
  * Lebret et al. (2015) R. Lebret, P. O. Pinheiro, and R. Collobert. 2015\.  Phrase-based image captioning. In _Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37 (ICML)_. JMLR. org, 2085–2094.
  * Lee et al. (2018) K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He. 2018.  Stacked cross attention for image-text matching. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 201–216.
  * Li et al. (2019) K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu. 2019.  Visual semantic reasoning for image-text matching. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_. 4654–4662.
  * Lin et al. (2018) A. Y. Lin, J. Ford, E. Adar, and B. Hecht. 2018\.  VizByWiki: Mining Data Visualizations from the Web to Enrich News Articles. In _Proceedings of the 2018 World Wide Web Conference on World Wide Web (WWW)_. International World Wide Web Conferences Steering Committee, 873–882.
  * Lin et al. (2014) T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and L. Zitnick. 2014.  Microsoft coco: Common objects in context. In _European conference on computer vision (ECCV)_. Springer, 740–755.
  * Liu and Ye (2019) F. Liu and R. Ye. 2019\.  A Strong and Robust Baseline for Text-Image Matching. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_. 169–176.
  * Liu et al. (2020) F. Liu, R. Ye, X. Wang, and S. Li. 2020\.  HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs. In _Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI)_.
  * Ma et al. (2015) L. Ma, Z. Lu, L. Shang, and H. Li. 2015\.  Multimodal convolutional neural networks for matching image and sentence. In _Proceedings of the IEEE international conference on computer vision (ICCV)_. 2623–2631.
  * Pecher and Zwaan (2005) D. Pecher and R. A. Zwaan. 2005.  _Grounding cognition: The role of perception and action in memory, language, and thinking_.  Cambridge University Press.
  * Plummer et al. (2015) B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. 2015\.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In _Proceedings of the IEEE international conference on computer vision (ICCV)_. 2641–2649.
  * Rajendran et al. (2016) J. Rajendran, M. M. Khapra, S. Chandar, and B. Ravindran. 2016\.  Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning. In _Proceedings of NAACL-HLT_. 171–181.
  * Ramisa (2017) A. Ramisa. 2017\.  Multimodal news article analysis. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI)_. 5136–5140.
  * Ramisa et al. (2018) A. Ramisa, F. Yan, F. Moreno-Noguer, and K. Mikolajczyk. 2018\.  Breakingnews: Article annotation by image and text processing.  _IEEE transactions on pattern analysis and machine intelligence (TPAMI)_ 40, 5 (2018), 1072–1085.
  * Ravi et al. (2018) H. Ravi, L. Wang, C. Muniz, L. Sigal, D. Metaxas, and M. Kapadia. 2018\.  Show me a story: Towards coherent neural story illustration. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 7613–7621.
  * Ren et al. (2015) S. Ren, K. He, R. Girshick, and J. Sun. 2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks. In _Advances in neural information processing systems (NIPS)_. 91–99.
  * Rotman et al. (2018) G. Rotman, I. Vulić, and R. Reichart. 2018.  Bridging languages through images with deep partial canonical correlation analysis. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL)_ , Vol. 1. 910–921.
  * Rush (2018) A. Rush. 2018.  The Annotated Transformer. In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_. 52–60.
  * Simonyan and Zisserman (2015) K. Simonyan and A. Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _International Conference on Learning Representations (ICLR)_.
  * Srivastava et al. (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research (JMLR)_ 15, 1 (2014), 1929–1958.
  * Su et al. (2020) W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai. 2020\.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In _International Conference on Learning Representations (ICLR)_.  https://openreview.net/forum?id=SygXPaEYvH
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017\.  Attention is all you need. In _Advances in Neural Information Processing Systems (NIPS)_. 5998–6008.
  * Vendrov et al. (2016) I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun. 2016\.  Order-embeddings of images and language.  _International Conference on Learning Representations (ICLR)_ (2016).
  * Wang et al. (2018b) L. Wang, Y. Li, J. Huang, and S. Lazebnik. 2018b.  Learning two-branch neural networks for image-text matching tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ (2018).
  * Wang et al. (2018a) S. Wang, Y. Chen, J. Zhuo, Q. Huang, and Q. Tian. 2018a.  Joint global and co-attentive representation learning for image-sentence retrieval. In _Proceedings of the 26th ACM international conference on Multimedia_. 1398–1406.
  * Wehrmann and Barros (2018) J. Wehrmann and R. C. Barros. 2018.  Bidirectional Retrieval Made Simple. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wu et al. (2017) C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Krähenbühl. 2017.  Sampling matters in deep embedding learning. In _Proc. IEEE International Conference on Computer Vision (ICCV)_.
  * Wu et al. (2019a) H. Wu, J. Mao, Y. Zhang, Y. Jiang, L. Li, W. Sun, and W.-Y. Ma. 2019a.  Unified Visual-Semantic Embeddings: Bridging Vision and Language with Structured Meaning Representations.  _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2019).
  * Wu et al. (2019b) Y. Wu, S. Wang, G. Song, and Q. Huang. 2019b.  Learning Fragment Self-Attention Embeddings for Image-Text Matching. In _Proceedings of the 27th ACM International Conference on Multimedia_. 2088–2096.
  * Yamada et al. (2018) Y. Yamada, M. Iwamura, and K. Kise. 2018.  ShakeDrop regularization.  _International Conference on Learning Representations workshop track (ICLR workshop)_ (2018).  https://openreview.net/forum?id=S1NHaMW0b
  * Zahavy et al. (2018) T. Zahavy, A. Magnani, A. Krishnan, and S. Mannor. 2018\.  Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce.  _The Thirtieth AAAI Conference on Innovative Applications of Artificial Intelligence_ (2018).
  * Zhong et al. (2017) Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang. 2017.  Random erasing data augmentation.  _arXiv preprint arXiv:1708.04896_ (2017).

##  10\. Appendices

###  10.1. Impact of modeling word on the char level

Opting for a subword embedding layer enables three features for our system: 1)
tolerating typos; 2) recognizing compound and conjunction words and 3)
transferring knowledge across close languages. In the following we show real
examples and retrieval results for demonstration.

Figure 10. Top 3 retrieval results of the original sentence (first row) and
with typos (second row). The two rows are of the same quality. (Best seen in
colors.)

Tolerate Typos. Subword embedding can tolerate typos rather than regarding
everything rare as out-of-vocab words. This is extremely important in real-
world applications.

  * •

_Roger Federer verwertet nach knapp zwei Stunden seinen ersten Matchball gegen
Benoit Paire._101010 _Roger Federer exploits his first match ball against
Benoit Paire in less than two hours._ (English translation)

  * •

_RogerFedererrr verwertet nach knapp zwei Stunden seinen ersten
Matc$!@>\$$hball gegen Benoit Paire._

The first is the original sentence and the second is with typos where most
informative words _Roger Federer_ and _Matchball_ are wrongly spelled and
unlikely to be in the vocabulary. However, subwords of the typos would still
contain enough information to retrieve the correct results as suggested in
Figure 10.

Compound and Conjunction Words. Let’s look at two examples. The first example
is for compound words. In this news (der, [n. d.]) which talks about building
new apartments in the city of Bern, the below three words are the only
informative ones in image caption:

  * •

_Miet wohnungen_ (rental apartments)

  * •

_Stockwerkeigentums wohnungen_ (condominiums)

  * •

_Alters wohnungen_ (senior apartments)

The subword _wohnungen_ means apartment in English, the compound forms are
however not likely to be in the vocabulary. With subword information,
searching with this kind of text would still work.

The second example is for conjunction words. Again, the following word would
be the only informative word in the headline of this news (eas, [n. d.])
discussing low-cost airlines.

  * •

_easyjet -flug_ (easyjet flight)

_easyjet -flug_ is likely to be identified as unknown words in regular word
embeddings. With n-gram subword embeddings, subwords _easyjet_ and _flug_
would both be kept as part of the final representation of _easyjet -flug_.

Transferring knowledge across nearby languages. Names of entities are usually
very similar in terms of morphology in nearby languages.

Figure 11. The three rows are the top 3 retrieval results of searching in
three different languages respectively. The highlighted words from Multi-Head
Attention also suggests that _President_ and _cabinet_ received the highest
attention scores in both three languages. (Best seen in colors.)

Let’s look at an example of three different _Presidents_ :

  * •

_Der Prësident trifft sich mit seinem Kabinett._ (German)

  * •

_Le Président rencontre son cabinet._ (French)

  * •

_The President is meeting with his cabinet._ (English)

The two informative words _President_ and _cabinet_ have many n-grams overlaps
among all three languages. We train the caption model only on German and test
it also on French and English. Though the model has never seen French and
English before, the qualitative results still look relevant as shown in Figure
11.

###  10.2. Combine our system with metadata matching

We show two examples of combining our model with metadata matching. It aims to
demonstrate how the model performs when searching for non-illustrative
“factual” photos.

####  10.2.1. Le Président

Figure 12. Top 9 retrieval results of the query “ _Le Président rencontre son
cabinet._ ” as title. The left is without selecting any named entity; the
right is searching with named entity “Macron”.

Figure 12 shows the top 9 retrievals with the French query “ _Le Président
rencontre son cabinet._ ”111111English translation: _The President is meeting
with his cabinet._ as a title. The left figure is searching without selecting
any named entity. The result seems to be any politician(s) meeting or speaking
as it’s unclear who the “Président” refers to. The right figure is searching
with named entity “macron”, then 7 of the 9 retrievals contain the French
President Emmanuel Macron meeting people or speaking.

####  10.2.2. Der Zürichsee

Figure 13. Top 9 retrievals of the query “ _Mehr Besucher am Zürichsee im
Sommer._ ” as title. The left is without selecting any named entity; the right
is searching with named entity “Zürichsee”.

Figure 13 shows the top 9 retrievals with the German query “ _Mehr Besucher am
Zürichsee im Sommer._ ”121212English translation: _More visitors coming to
Lake Zurich in summer._ as a caption. The left figure is searching without
selecting any named entities. The retrievals appear to be people gathering
around any water, not necessaily a lake or Lake Zurich. The right figure is
searching with named entity “Zürichsee”131313English translation: Lake
Zurich.. Retrievals are still crowds and boats on/around waters but now we
know that the water is guaranteed to be Lake Zurich.

###  10.3. Naive Named Entity Filtering

One may wonder what happens when directly searching with named entities on
metadata (without our model ranking the candidates). Here are the results in
Figure 14. Only 2 of the 9 retrievals in the left figure has President Macron
and the right figure looks completely random. It again verifies our claim that
the metadata can be very heterogeneous and some might even be misleading.

Figure 14. Top 9 retrievals of searching with named entities only. Left is the
result of “macron” and right is the results of “Zürichsee”.