# Show, Interpret and Tell: Entity-aware Contextualised Image Captioning in
Wikipedia

Khanh Nguyen, Ali Furkan Biten, Andres Mafla, Lluis Gomez, Dimosthenis
Karatzas

###### Abstract

Humans exploit prior knowledge to describe images, and are able to adapt their
explanation to specific contextual information, even to the extent of
inventing plausible explanations when contextual information and images do not
match. In this work, we propose the novel task of captioning Wikipedia images
by integrating contextual knowledge. Specifically, we produce models that
jointly reason over Wikipedia articles, Wikimedia images and their associated
descriptions to produce contextualized captions. Particularly, a similar
Wikimedia image can be used to illustrate different articles, and the produced
caption needs to be adapted to a specific context, therefore allowing us to
explore the limits of a model to adjust captions to different contextual
information. A particular challenging task in this domain is dealing with out-
of-dictionary words and Named Entities. To address this, we propose a pre-
training objective, Masked Named Entity Modeling (MNEM), and show that this
pretext task yields an improvement compared to baseline models. Furthermore,
we verify that a model pre-trained with the MNEM objective in Wikipedia
generalizes well to a News Captioning dataset. Additionally, we define two
different test splits according to the difficulty of the captioning task. We
offer insights on the role and the importance of each modality and highlight
the limitations of our model. The code, models and data splits are publicly
available at Uponacceptance.

## Introduction

Scene understanding involves composing a story that explains the perceptual
observations. This ability, to draw from previous experience to explain what
is happening, is termed image interpretation (Lake et al. 2017). Image
interpretation is one of the hallmarks of linguistic intelligence (Gardner
2011) where contextual information is drawn upon to compose an explanation of
the depicted events (Terman 1916). We identify Wikipedia as an excellent
probing ground to further the advance in image interpretation and integrate
contextual information into captioning models.

Figure 1: The role of contextual information in Wikipedia Captioning. The same
image can have completely different captions depending on contextual
information given in the form of article and section. On the top side, the
corresponding page is a description of a location, while the bottom one talks
about Banking in Switzerland.

Wikipedia is an invaluable source of world knowledge information. Each page
consists of different information sources comprised of title, sections, images
and their captions. Each author of Wikipedia writes an article and illustrates
it with images that are selected from Wikimedia. Each image has a
corresponding description that includes very detailed information, including
Named Entities (NEs). In order to write a good caption for the image while
remaining relevant to the article, the writers need to take into account the
section they wrote, the image they selected and its corresponding description
to produce a contextualized caption, a task referred to as Wikipedia
captioning. In this work, we propose Wikipedia Captioning as a real use case
scenario to alleviate the work load of Wikipedia writers while allowing them
to better illustrate their articles. Wikipedia captioning offers the unique
opportunity to potentially aid thousands of writers directly and millions of
readers indirectly on a daily basis.

To grasp the complexity of Wikipedia captioning, an example of a
contextualized caption can be observed in Figure 1. The importance of the
context is paramount given the possibility of having several matching and
distinct captions for a single image. Naturally, this introduces numerous
challenges. On one hand, the context (description and section) needs to be
encoded and information selectively drawn from it while being guided by the
visual content. On the other hand, explicit contextual information, typically
found in the form of NEs such as proper names, prices, locations, dates, etc.,
needs to be properly injected in the generated captions in natural language.
This explicit contextual-dependant information is typically out-of-dictionary
or at best it is underrepresented in the statistics of the dictionary used. To
better prepare a model that deals with Named Entities, we propose a pre-
training strategy in which we extend Masked Language Modeling (MLM) (Devlin et
al. 2018) to Masked Named Entity Modeling (MNEM). We show that MNEM is more
effective at selecting the correct NE given an article while reducing the
language prior produced by NEs.

Wikipedia Captioning is closely related to News Captioning (Biten et al. 2019;
Feng and Lapata 2013; Ramisa et al. 2018). In the case of News Captioning, a
single source of context is available in which the news article is roughly
equivalent to a Wikipedia article, and no separate (article independent)
description for the image is available. Yet in Wikipedia Captioning, images
are associated to specific sections of the article, plus specific article-
independent descriptions for each image are also given. The two tasks would be
equivalent if we ignore the descriptions and we equate a Wikipedia section
with the whole news article. Given the similarities between the two tasks, we
examine whether the performance can be improved by transferring from Wikipedia
to News captioning. We discover that the knowledge acquired by the model on
Wikipedia captioning is transferable to News captioning, achieving better
results over the baseline. Moreover, we demonstrate that MNEM is effective as
well on News captioning, outperforming a base model in terms of automatic
image captioning evaluation metrics.

Furthermore, we uncover several key insights on the Wikipedia captioning task
and on the capabilities of the proposed methodology. Firstly, since such
models are prone to take shortcuts (Geirhos et al. 2020), we discover a bypass
strategy that our model is taking, a copying process of NEs from descriptions
to captions. Additionally, humans have the ability to come up with a caption
for any image given a specific context (article, section, description, etc.).
Contrary to open-world story generation (Martin et al. 2018) which is to come
up with stories for any topic without prior knowledge, context-based story-
telling is to generate captions with context acting as a prior knowledge for
any given image. Trying to emulate this specific skill, we examine if our
model can generate a contextualized caption given mismatched but potentially
relevant images. Despite our model’s state-of-the-art performance according to
captioning metrics and injecting the most relevant NEs, we reveal that current
models are still far off from endowing this skill. To summarize, the key
contributions of our work are:

  * •

We introduce the Wikipedia Captioning as a real use case scenario to benefit
writers and readers on a daily basis.

  * •

We propose an architecture-agnostic pretext task, namely Masked Named Entity
Masking, that can be applied to any contextualized captioning model. We
empirically show that when it is used, a gain in performance is achieved on
Wikipedia and News captioning domains.

  * •

We exhibit that pre-training models on Wikipedia captioning boosts performance
on News captioning, transferring the knowledge from one domain to another.

  * •

We extensively study the limitations of our models and discuss a setting that
shows we are far off on mimicking the human ability of context-based story-
telling.

## Related Work

Classic Image Captioning. Early image captioning methods (Farhadi et al. 2010;
Ordonez, Kulkarni, and Berg 2011) focus on hand crafted features with
templates. The advent of deep learning with large scale datasets (Lin et al.
2014), in contrast, steers the trend into end-to-end encoder-decoder networks.
Centralized by attention mechanisms (Bahdanau, Cho, and Bengio 2014; Devlin et
al. 2018), architectures, ranging from LSTM-based (Vinyals et al. 2015; You et
al. 2016; Anderson et al. 2017; Lu et al. 2017) to Transformers (Pan et al.
2020; Luo et al. 2021; Ji et al. 2021; Chen et al. 2021) achieve remarkable
progress in captioning metrics. Advanced models (Dai et al. 2017; Shetty et
al. 2017) employ GANs to favor human-like caption outputs. Works applying
Reinforcement Learning (Rennie et al. 2016; Liu et al. 2017) successfully
optimize models on non-differential metrics. Nevertheless, such models are
restricted by design to the mere description of what is shown, thus unable to
attempt any interpretation.

News Articles Captioning. Recent advances in News image captioning (Feng and
Lapata 2013) come in both architectures and datasets. Typically, proposed
models follow the standard encoder-decoder paradigm while large-scale datasets
such as Breaking News (Ramisa et al. 2018), GoodNews (Biten et al. 2019),
VisualNews (Liu et al. 2020) etc. have been contributed. To deal with NEs,
(Biten et al. 2019; Liu et al. 2020) propose two-stage template-based methods
that train models to capture NE-tag distribution followed by post-generation
NE insertion. Closely related to our work, (Tran, Mathews, and Xie 2020)
employs a transformer (Vaswani et al. 2017) with byte-pair-encoding (BPE)
(Sennrich, Haddow, and Birch 2015) to directly decode NEs. Lastly, (Yang et
al. 2021) enhances caption quality with template guidance and a pre-trained NE
Embedding module based on Wikipedia Knowledge graph (Yamada et al. 2018) to
better encode NEs. Given the importance of NEs in this task, we introduce the
Named Entity Masking pretext task for accurate NE prediction and coverage.

Self-supervised Pre-training Methods. Self-supervised approaches in NLP
(Devlin et al. 2018; Radford et al. 2019; Roberts and Raffel 2020) have
greatly inspired many works in vision-and-language (VL) (Li et al. 2020; Kim,
Son, and Kim 2021; Zhang et al. 2021) that follow the pre-training and fine-
tuning paradigm, elevating the performance in a wide variety of benchmarks.
While existing pre-training objectives in VL either focus on representation
learning or multi-modal alignments, our proposed method MNEM aims to learn
context-entity alignment via the MLM protocol. Compared to other entity-
centric methods (Sun et al. 2020; Lin et al. 2021), we are the first to
explore MNEM in a multi-modal scheme, with careful adaptation to suit the
image-text distribution found in Wikipedia.

## Method

In this section, we first define the Wikipedia captioning task formally and
then present our approach which utilizes well-designed transformer-based
models to solve the task. We further discuss how we extend these models by
employing a novel pre-training objective which we refer to as Masked Named
Entity Modeling (MNEM).

Figure 2: Overview of the Transformer-based models and our MNEM pre-training
task. Left: Transformers take multi-modal input features (image, section,
description) processed by single-modal encoders; Right: During pre-training
(right-top) we replace the NEs in the ground-truth caption by the MASK token
and train our models to reconstruct the caption.

### Problem Definition

We refer to $I=\\{i_{1},i_{2},...\\}$, $C=\\{c_{1},c_{2},...\\}$ as the image
set and its corresponding captions set where each caption contains words
defined as $c_{k_{j}}$ or $c_{k}=\\{c_{k_{0}},c_{k_{1}},...\\}$. Moreover,
$D=\\{d_{1},d_{2},...\\}$ and $S=\\{s_{1},s_{2},...\\}$ refer to description
and section sets, respectively. Given a Wikipedia article, a description
$d_{k}$ is tied to each specific image $i_{k}$ for which it provides Named
Entities (when and where the photo was taken, who is depicted in the photo,
author, etc.). Each Wikipedia article is divided in sections $s_{k}$ to
enhance readability. To emulate the process that the Wikipedia writers go
through during Wikipedia captioning, we define the dataset with quadruples
where $(i_{k},c_{k},s_{k},d_{k})$ refers to the $k^{th}$ datum. We define the
task of generating a caption that is contextualized as
$\mathcal{P}(c_{k}|i_{k},s_{k},d_{k})$.

### Transformer-based Models

The overview of our proposed method is depicted in Figure 2, which follows the
commonly used encoder-decoder paradigm. In particular, we investigate the
capacity of Transformer-based models, either trained from scratch or
initialized with pre-trained weights acquired from self-supervised training.
For the latter, we explore Pre-trained language models (PLMs) due to its
compelling performances on numerous open-ended text generation tasks. However,
as only pre-trained on textual modality, properly adapting PLMs to solve VL
tasks remains a challenge. Therefore, we propose a simple strategy to enable
PLMs to reason on multi-modal context when generating captions.

Transform and Tell (TaT) (Tran, Mathews, and Xie 2020).  Proposed to address
the News Image Captioning task in an end-to-end manner, TaT consists of a set
of frozen encoding modules followed by a trainable Transformer-decoder. The
decoder thus generates captions conditioned on a concatenation of multi-modal
input features extracted from the image and the article. Perceiving the
similarities between the two tasks, we apply TaT to Wikipedia captioning and
modify it by stripping away some of the building blocks of the main model.
Particularly, we remove the MTCNN (Zhang et al. 2016) for face detection,
FaceNet (Schroff, Kalenichenko, and Philbin 2015) for face embedding, YOLOv3
(Redmon and Farhadi 2018) for object representation and the location
embedding. Removing these blocks then speeds up the training process by
decreasing complexity, while the remaining modules could be easily adapted as
required.

The reduced encoder now comprises two pre-trained modules for image and text
to produce a set of representations. More specifically, given an image
$i_{k}$, we extract image features using a ResNet-152 model pre-trained on
ImageNet to obtain ($f_{i_{k}}$). We use RoBERTa (Liu et al. 2019) to encode
text sequences into semantic representations capturing a bi-directional
context, yielding two sets of text features for sections and descriptions
($f_{s_{k}},f_{d_{k}}$) respectively. The auto-regressive decoder consists of
the same BPE embedding matrix introduced in RoBERTa, followed by a stack of
$N$ identical transformer blocks. Making use of a shared BPE embedding layer
implicitly grants the model the ability of generating rare words. For each
transformer block, we use a Dynamic Convolution (DC) (Wu et al. 2019) layer
for past-token conditioning and a Multi-Head Attention (MHA) (Vaswani et al.
2017) layer for multi-modal conditioning. This architecture design serves as
an efficient alternative to the traditional self-attention mechanism, relaxing
memory requirements which are quadratic with the input sequence length. At
time step $t$, previously-generated token representations $Z_{\leq
t}=\\{z_{0},z_{1},...,z_{t}\\}$ are efficiently enriched by attending to its
leftward context thanks to the DC. After going through the decoder stack, the
final representation $z^{\prime}_{t}\in Z^{\prime}_{\leq t}$ is fed to the
vocabulary layer to predict the next token at $t+1$.

GPT-2 (Radford et al. 2019) is a transformer decoder variant which performs
text generation in auto-regressive manner. We want to leverage the linguistic
knowledge stored in the model’s pre-trained weights to emulate its success in
generating captions, while preserving the Wikipedia style. We therefore decide
to fine-tune the entire network, including the learned embedding matrix. We
start by transforming the description and section sequences into continuous
embeddings ($f_{s_{k}}$, $f_{d_{k}}$) using the internal embedding layer of
GPT-2. Concurrently, we represent the image as a set of visual features
($f_{i_{k}}$) extracted from ResNet-152, followed by a MLP layer with Layer
Normalization to map images into the word embedding space. We form this set of
representations ($f_{i_{k}}$, $f_{s_{k}}$, $f_{d_{k}}$) as prefix and pass it
to the decoder (referred as $\text{GPT-2}_{\textit{BASE}}\texttt{++}$) whose
output $z^{\prime}_{t}\in Z^{\prime}_{\leq t}$ is used as a logit vector over
the vocabulary distribution to predict the caption token at $t+1$.

T5 (Roberts and Raffel 2020). Built on top of the vanilla Transformer encoder-
decoder, T5 is another language model pre-trained on C4. Particularly, T5
presents a new self-supervised training objective called Replace Corrupted
Spans for efficient computation conforming to its encoder-decoder design. To
apply the off-the-shelf version of T5 to Wikipedia captioning, we extend its
text-only encoder to multi-modal encoder, referred as
$\text{T5}_{\textit{BASE}}\texttt{++}$. We incorporate image features
$f_{i_{k}}$ as additional input to the description and section embeddings
($f_{s_{k}}$, $f_{d_{k}}$), with each set of embeddings computed the same way
as in the previous section. The decoder then attends to previous caption
tokens $Z_{\leq t}$ via self-attention and the contextualized joint
representations from the encoder via cross-attention to predict the next
token.

### Masked Named Entity Modeling

Referring to the right NEs is among the most important but challenging tasks
of Wikipedia captioning to assure the veracity of information included in a
caption. MLM, as proposed in BERT (Devlin et al. 2018), has shown remarkable
success, building richer representations by exploiting the distributional
hypothesis present in language to reconstruct masked out words. To extend the
applicability of MLM to our captioning task and mitigate the rare NEs
insertion issue, we apply the masking strategy explicitly on NEs to learn more
expressive representations for them. More formally, let
$\mathcal{W}=\\{w_{1},w_{2},...,w_{n}\\}$ be the set of all byte-pair tokens.
Now, let $\mathcal{M}_{l}=\\{j,j+1,...,j+k\\}$ be the $l^{th}$ mask span where
$j$ is the starting index to mask such that
$\max(\mathcal{M}_{l})<\min(\mathcal{M}_{l+1})$. Then,
$\\{w_{j},...,w_{j+k}\\}$ are replaced with the special mask token [MASK] to
form the corrupted caption
$\tilde{\mathcal{W}}=\\{\tilde{w}_{1},\tilde{w}_{2},...,\tilde{w}_{n}\\}$. We
use cross-entropy to reconstruct the original caption $\mathcal{W}$, with no
restriction to masked tokens compared to BERT, in order to retain the
generation ability:

$\begin{split}\tilde{w_{i}}&=\ [MASK],\text{where}\ i\in\mathcal{M}_{l}\\\
\mathcal{L}&=\sum_{i=0}^{N}{-\log{p(w_{i}|\tilde{\mathcal{W}}_{0<i},f_{i_{k}},f_{s_{k}},f_{d_{k}})}}\end{split}\vspace{-3cm}$
(1)

Inspired from T5, we introduce a small modification to MNEM to specifically
pre-train encoder-decoder architectures. We first substitute each entity span
$\mathcal{M}_{l}$ with a unique mask token to create $\tilde{\mathcal{W}}$
then append it to the encoder input sequence. Consequently, instead of
reconstructing the caption, we task the decoder to predict all the masked name
entities, which are concatenated in original order as the target sequence.
This conforms the training procedure performed on T5 while explicitly tells
the model to focus on name entities within context.

Figure 2 depicts our proposed design with masked NEs language modeling during
the pre-training stage. We extract NEs contained in captions and randomly mask
the NEs with a probability of 0.8 while adopting a whole word masking
strategy. We only mask NEs that fall into the categories of: [Person,
ORGanization, GeoPolitical Entity] as they appear to be dominant types in
Wikipedia captions. Moreover, changing every NEs might be too disrupting of
the semantic structure in captions which tend to be short.

By masking the NEs within the captions, the model can learn to precisely
capture the contextual information relating to the NEs to produce useful
representations. Also, our proposed method aims to mitigate another issue,
which is further studied below. We found that when there is a high overlap
between the captions and context, the model learns to directly copy NEs to
generate captions, making it ineffective to deal with the diversity and
semantic richness of Wikipedia Captioning. By limiting the input caption with
non-NEs words in training phase, we decrease the influence of the language
bias, allowing a better utilization of the entire context given as input.

## Experiments

In our experimentation for Wikipedia Captioning, we utilize the WIT
(Srinivasan et al. 2021) dataset. All the details regarding the dataset
statistics, pre-processing and implementation as well as an in-depth
explanation of our baselines can be found in the Appendix.

Table 1: Performance of proposed models on Wikipedia Captioning. Seq2seq
performs better than TaT without image while MNEM is effective to align the
image and text modality. Metric scores are reported as percentage. $\dagger$
indicates that MNEM pre-training is performed. $\Delta$ indicates the
improvement with MNEM compared to respective models. Bold numbers are the best
performance of each metric.

Model B M R C S Pr Re No Image Seq2seq 6.04 11.53 31.94 137.38 20.25 26.42
14.05 Seq2seq+Attn. 15.86 18.82 43.69 242.87 31.45 44.10 27.60 Seq2seq+Ptr.
21.04 21.27 45.22 255.87 32.64 40.81 30.67 Seq2seq+Ptr.+Cvrg. 21.57 21.56
45.50 258.84 33.00 41.23 31.37 TaT 18.74 19.65 43.74 229.77 29.47 32.09 27.12
$\text{GPT-2}_{\textit{BASE}}$ 19.18 20.68 45.68 254.94 32.8 42.28 30.38
$\text{T5}_{\textit{BASE}}$ 25.70 24.17 48.8 275.31 34.31 39.11 35.09
$\text{TaT}^{\dagger}$ 19.93 20.42 44.57 235.9 30.03 32.94 28.47
$\text{GPT-2}_{\textit{BASE}}^{\dagger}$ 19.64 21 46.32 260.4 33.34 43.31 30.9
$\text{T5}_{\textit{BASE}}^{\dagger}$ 24.60 23.65 48.96 278.68 34.69 40.51
34.26 $\Delta$ TaT 1.19$\uparrow$ 0.77$\uparrow$ 0.83$\uparrow$ 6.13$\uparrow$
0.56$\uparrow$ 0.85$\uparrow$ 1.35$\uparrow$ $\Delta$
$\text{GPT-2}_{\textit{BASE}}$ 0.46$\uparrow$ 0.32$\uparrow$ 0.64$\uparrow$
5.46$\uparrow$ 0.54$\uparrow$ 1.03$\uparrow$ 0.52$\uparrow$ $\Delta$
$\text{T5}_{\textit{BASE}}$ 1.1$\downarrow$ 0.52$\downarrow$ 0.16$\uparrow$
2.08$\uparrow$ 0.38$\uparrow$ 1.4$\uparrow$ 0.83$\downarrow$ Image Seq2seq
6.10 11.26 31.20 136.98 19.83 25.97 13.88 Seq2seq+Attn. 14.71 18.06 42.79
235.44 30.70 43.49 26.60 Seq2seq+Ptr. 20.46 20.89 44.76 252.79 32.12 40.37
29.97 Seq2seq+Ptr.+Cvrg. 20.22 20.72 44.45 249.67 31.77 40.20 29.68 TaT 24.47
23.15 48.07 267.37 33.33 32.10 34.64 $\text{GPT-2}_{\textit{BASE}}\texttt{++}$
19.4 20.72 45.97 256.25 32.79 41.28 30.06
$\text{T5}_{\textit{BASE}}\texttt{++}$ 23.83 23.33 48.8 276.6 34.59 40.7 33.72
$\text{TaT}^{\dagger}$ 25.31 23.64 48.72 272.46 34.36 33.34 35.56
$\text{GPT-2}_{\textit{BASE}}^{\dagger}\texttt{++}$ 20.65 21.54 47.3 266.29
34.12 42.86 31.48 $\text{T5}_{\textit{BASE}}^{\dagger}\texttt{++}$ 25.89 24.39
49.28 279.09 34.64 39.65 35.73 $\Delta$ TaT 0.84$\uparrow$ 0.49$\uparrow$
0.65$\uparrow$ 5.09$\uparrow$ 0.73$\uparrow$ 1.24$\uparrow$ 0.92$\uparrow$
$\Delta$ $\text{GPT-2}_{\textit{BASE}}\texttt{++}$ 1.25$\uparrow$
0.82$\uparrow$ 1.33$\uparrow$ 10.04$\uparrow$ 1.33$\uparrow$ 1.58$\uparrow$
1.12$\uparrow$ $\Delta$ $\text{T5}_{\textit{BASE}}\texttt{++}$ 2.56$\uparrow$
1.06$\uparrow$ 0.48$\uparrow$ 3.78$\uparrow$ 0.05$\uparrow$ 1.05$\downarrow$
2.01$\uparrow$

### Wikipedia Captioning

Table 1 showcases the results for Wikipedia Captioning task in BLEU-4
(Papineni et al. 2002), METEOR (Banerjee and Lavie 2005), ROUGE-L (Lin 2004),
CIDEr (Vedantam, Lawrence Zitnick, and Parikh 2015), and SPICE (Anderson et
al. 2016), as well as precision and recall of the NE insertion as defined by
(Biten et al. 2019). The high performing baselines especially without image
imply strong language priors learned by the models, outperforming transformer
language models like TaT or GPT-2. The observed drawbacks of such methods
indicate two things: (1) the baselines tend to rely on copying NEs excessively
coming from the description of the images only and (2) the models prefer to
generate few NEs. These hypotheses are confirmed by the high Precision and low
Recall, suggesting the models have found a shortcut to generate few NEs by
only copying them from the description. We examine this behaviour more
thoroughly in the analysis section. Moreover, the addition of images as an
input clearly disrupts the performance. According to our examination, we find
out that the attention weights of seq2seq models are significantly lower for
the image modality than the context one. In other words, seq2seq tries to
disregard the images while putting more emphasis on the context, failing to
find the correspondence between two modalities.

Yet, we see opposite behaviour on the efficacy of images on transformer
models, showcasing the superiority of transformers over LSTM encoder-decoder.
The addition of images significantly improves the performance on all the
captioning metrics. Interestingly, images have a direct impact on the Recall
for TaT, improving the results from 27% to 34% while not affecting the
Precision score. Improving the Recall score discerns a crucial lesson that
aligning the image and text modality is imperative to generate higher number
of correct NEs. However, this effect is not present in GPT-2 and T5, which is
expected, as these PLMs naturally gravitate towards linguistic features, thus
require more well-designed adaptation to balance the visual and textual
information in VL tasks. MNEM aims to connect two modalities by means of
predicting NEs, leading to better NEs selection given multi-modal alignment.
It is clearly observed that in all transformer models, MNEM exceeds the
performance significantly on captioning metrics (+10 and +5 points in CIDEr
for GPT-2 and TaT respectively). MNEM also provides cues to select NEs
properly, gaining on both Precision and Recall with or without image. Finally,
T5 shows a consistent superiority among all models, with MNEM it achieves
state-of-the-art in Wikipedia Captioning task.

Table 2: Performance of proposed models on GoodNews. Ft means fine-tuning the
models with the captioning objective in the corresponding dataset. Note that
for captioning metrics, we directly use the results that are reported in the
original papers.

WIT GoodNews B M R C Pr Re MNEM Ft MNEM Ft TaT $\checkmark$ 6 10.1 21.2 53.1
21.3 19.7 TaT(Full) $\checkmark$ 6.05 10.3 21.4 53.8 21.85 19.93
VisualNews(Full) $\checkmark$ 6.1 8.3 21.6 55.4 – – JoGANIC $\checkmark$ 6.34
10.78 21.65 59.19 – – JoGANIC(Full) $\checkmark$ 6.83 11.25 23.05 61.22 – –
$\text{T5}_{\textit{BASE}}\texttt{++}$ $\checkmark$ 6.34 10.05 21.8 59.68
23.61 20.08 $\checkmark$ $\checkmark$ 6.56 10.21 21.71 60.17 23.28 20.43
$\checkmark$ $\checkmark$ 7.17 10.8 22.6 62.76 23.11 20.92 $\checkmark$
$\checkmark$ $\checkmark$ 7.19 10.84 22.89 63.89 23.37 21.48 $\checkmark$
$\checkmark$ 5.89 9.38 19.88 51.49 20.36 18.85 $\checkmark$ $\checkmark$
$\checkmark$ 6.63 10.19 21.16 57.03 21.51 20.43 $\checkmark$ $\checkmark$
$\checkmark$ 6.66 10.28 21.41 58.47 21.84 20.59 $\checkmark$ $\checkmark$
$\checkmark$ $\checkmark$ 6.75 10.38 21.53 59.07 21.89 20.55

### Transfer Learning from Wikipedia to News

Given the similar challenges that the two tasks face, we examine the
possibility of transfer learning from Wikipedia to News captioning, i.e. fine-
tuning models on GoodNews with weights initialized from pre-training on
Wikipedia. To imitate the News captioning setting, we utilize the section
content as the only context, disregarding the description. We select the
$\text{T5}_{\textit{BASE}}\texttt{++}$ model for the experiments given that it
is the best performing model in Wikipedia Captioning. Table 2 shows the
results of the transferability from Wikipedia to News with MNEM pre-training.
At first, simply fine-tuning $\text{T5}_{\textit{BASE}}\texttt{++}$ on
GoodNews can match or surpass previous transformer models (+6 points in CIDEr
compared to TaT), demonstrating PLMs impressive ability to generate captions.
Pre-training on Wikipedia further improves the performance over JoGANIC, which
is the state of the art method with Wikipedia pre-training, (+0.36 points in
BLEU-4 and +2.67 in CIDEr) while not including specific template guidance.
This illustrates the advantage of proposing this Wikipedia Captioning task, in
which the acquired knowledge is useful for the News domain. Moreover,
incorporating MNEM pre-training on GoodNews constantly results in additive
gains, demonstrating its effectiveness in this task. However, the improvement
is not transferable as we additionally perform MNEM on Wikipedia (rows 9–12).
This extra pre-training stage guides the model to focus on Wikipedia NEs,
which come from a different distribution compared to GoodNews, thus degrading
the final model performance.

### Ablation Study on MNEM

To further probe into the efficacy of MNEM, we compare it against the well
established MLM introduced by BERT. The MLM strategy selects 15% of the words
in a sentence in which it replaces WordPiece tokens (Wu et al. 2016) 80% of
the time with a special [MASK] token, 10% of the time with a random word and
10% of the time without changing the word. We introduce another variant of MLM
as a comparison namely Full Masking where given 15% of the words in a
sentence, we always mask a word without employing other cases. We perform the
pre-training stage with these masking strategies on transformer models and
report results on Wikipedia Captioning in Table 3.

Table 3: Ablation studies on different masking strategies. Our proposed method
MNEM outperforms MLM and Full masking strategy on every context and on every
metrics. All the numbers are obtained in Wiki test set.

Model Context Masking B M R C S Pr. Re.
$\text{GPT-2}_{\textit{BASE}}\texttt{++}$ Wiki. Full 20.44 21.47 47.3 266.4
34.06 43.24 31.4 MLM 20.34 21.38 47.3 266. 34.03 43.4 31.4 MNEM 20.65 21.54
47.3 266.29 34.12 43.12 31.55 $\text{T5}_{\textit{BASE}}\texttt{++}$ Wiki.
Full 2502 24 49.04 276.58 34.45 38.87 35.08 MLM 24.57 2371 48.88 275.22 34.37
38.75 34.94 MNEM 25.89 24.39 49.28 279.09 34.64 39.65 35.73 TaT Wiki. Full
24.01 22.84 47.49 260.87 32.79 31.83 34.35 MLM 23.81 22.74 47.21 258.25 32.65
31.47 34.12 MNEM 25.31 23.64 48.72 272.46 34.06 33.34 35.56 Sec. Full 8.96
11.98 32.13 147.43 19.71 20.02 15.13 MLM 8.71 11.40 32.07 147.40 19.59 19.81
14.46 MNEM 9.54 12.52 33.03 153.87 20.37 20.75 16.01 Desc. Full 20.95 20.39
42.10 215.42 27.97 28.80 26.55 MLM 20.92 20.31 41.86 213.69 27.73 28.41 26.16
MNEM 22.36 21.29 43.14 224.38 28.87 29.36 27.90

We note that our newly defined Full Masking strategy outperforms the well
established MLM, challenging the hyper-parameters defined by MLM. Our
experiments demonstrate that masking hyper-parameters are rather task
dependent which need to be treated selectively. Overall, MNEM yields a
significant boost over its counterparts in TaT (+12 points in CIDEr), while
this behaviour is less apparent in T5 and GPT-2. This is justified since TaT
is specifically designed for News Image Captioning, highlighting the
effectiveness of MNEM in a fine-grained multi-modal scheme. In contrast, T5
and GPT-2 essentially are language models that have been pretrained on text
thus require more careful adaptations in architecture for further
improvements.

To examine how the context affects the masking strategies, we carry out
experiments with TaT, with the context defined in three distinct ways:
description, section and the original Wiki context. We observe that MNEM again
outperforms other methods on every context type. Notably, the difference
between MNEM and the other two masking strategies is much higher with the
description compared to section used as a context. This is somehow expected
given that a section tends to be longer and it is harder to generate
contextualized captions. However, we observe that MNEM is especially effective
when section and description are combined (Wiki as context). MNEM performs 2%
better in Recall in Wiki context while improving 1% more in Recall compared to
MLM or Full Masking in description or section as a context, this behaviour is
more apparent in CIDEr. This demonstrates that MNEM is not only better on
encoding the contextual information but especially better at combining
description and section information. We believe the potential of MNEM is
unlocked further because it can amalgamate the two contexts better by
discovering strong correlations between description and section, resulting in
better representations for NEs as the numbers clearly suggest.

### Analysis of Wikipedia Captioning

In this section, we investigate the performance of our models in Wikipedia
Captioning and provide an in-depth analysis in both quantitative and
qualitative results. Furthermore, we examine if our models have the ability to
generate a caption for “irrelevant” images given a context. This evaluates how
close our model performs in terms of generating captions for any given image
while staying faithful to the section.

Table 4: Performance on Easy and Hard subsets. They are defined based on the
Jaccard similarity between the context and caption. Overlap scores measure the
similarity compared to the context, reported in average Jaccard score. We
refer the (ground truth caption, context) overlap as GT ol., Gen. ol. is the
(generated caption, context) overlap and Gen. len. is the average length of
generated captions. $\dagger$ indicates that MNEM pre-training is performed.

Model Test Set GT ol. Gen. ol. Gen. len. B C Pr Re TaT Easy 0.81 0.92 7.65
29.93 313.99 39.49 37.2 Hard 0.35 0.75 6.03 8.48 133.68 19.06 16.06 Full 0.71
0.88 6.46 24.47 267.37 32.10 34.64 $\text{TaT}^{\dagger}$ Easy 0.81 0.92 7.89
30.76 319.01 39.71 38.20 Hard 0.35 0.73 6.16 8.91 137.81 19.39 16.53 Full 0.71
0.87 7.08 25.31 272.46 33.34 35.56 $\text{GPT-2}_{\textit{B}}\texttt{++}$ Easy
0.81 0.93 5.35 24.74 304.71 46.37 34.24 Hard 0.35 0.81 4.2 6.38 128.15 24.28
15.84 Full 0.71 0.90 5.06 19.4 256.25 41.28 30.36
$\text{GPT-2}_{\textit{B}}^{\dagger}\texttt{++}$ Easy 0.81 0.93 5.49 25.63
312.16 47.69 35.87 Hard 0.35 0.8 4.38 7.01 133.64 24.97 16.92 Full 0.71 0.89
5.2 20.65 266.29 42.86 31.48 $\text{T5}_{\textit{B}}\texttt{++}$ Easy 0.81
0.93 6.35 29.55 325.32 45.37 38.37 Hard 0.35 0.8 4.92 7.46 130.79 23.7 18.29
Full 0.71 0.9 5.98 23.83 276.6 40.7 33.72
$\text{T5}_{\textit{B}}^{\dagger}\texttt{++}$ Easy 0.81 0.91 7.16 31.96 330.79
43.9 40.9 Hard 0.35 0.79 5.39 7.95 129.84 22.63 18.57 Full 0.71 0.88 6.7 25.89
279.09 39.65 35.73

Paul Allman Siple December 18 1908 – November 25 1968 was an American
Antarctic explorer and geographer who took part in six Antarctic expeditions
including the two Byrd expeditions of 1928–1930 and 1933–1935 representing the
Boy Scouts of America as … ranked list of retrieved images Siple in 1930 Paul
Allman Siple, c. 1930 Paul Allman Siple Siple in 1930 Paul Allman Siple on the
Byrd Antarctic Paul Allman Siple Siple in the 1930s Siple’s Eagle Scout badge
Cover of Siple’s first book Siple ’s grave at the National Museum of the
United States Siple’s signature Grave at Arlington National Cemetery
---
The Bela Vista Park Portuguese Parque da Bela Vista is one of the largest open
areas within the city limits of Lisbon. Portugal Bela Vista Beautiful View is
the name of its adjacent neighborhood. The park comprises an area of 85,000 m²
and is often used as a venue for large concerts … ranked list of retrieved
images The park in winter Aerial view of Bela Vista Park Bela Vista Park Bela
Vista Park Aerial view of Bela Vista Park Entrance to the park Bela Vista Park
from space Location of Bela Vista Park in Lisbon Aerial view of Bela Vista
Park Aerial view of the park Aerial view of Bela Vista Park Aerial view of
Bela Vista Park
Figure 3: Qualitative samples of generated captions with TAT-MNEM (top,
green), GPT2-MNEM (middle, orange) and T5-MNEM (bottom, purple) for unpaired
images given a similar Wikipedia article section. The models usually prefer to
generate short, repetitive and template-based sentences. However, in some
cases, they can integrate relevant visual details into the caption.

#### Easy vs Hard Samples.

Previously, we show in Table 1 that seq2seq models exploit a shortcut of
copying the NEs from the context, reflected in the high precision at inserting
NEs. Hence, to better understand our model’s limitations, we devise a
procedure to identify the captions that can be produced simply by utilizing
the context. To this end, we employ the Jaccard similarity (Jaccard 1912) at
the word level where given two sets $\mathcal{C},\mathcal{S}$ for caption and
context respectively composed of word tokens, we calculate
$J(\mathcal{C},\mathcal{S})=\frac{|\mathcal{C}\cap\mathcal{S}|}{|\mathcal{C}\cup\mathcal{S}|}$.
Accordingly, we assume the image has a diminishing impact on the caption when
the caption and the context have a Jaccard score of higher than 0.5. Thus, we
refer to these samples as easy samples (hard otherwise) since the caption can
be generated simply by copying from the context and ignoring the image.

Table 4 presents the scores in terms of Easy and Hard subset, as well as the
average Jaccard score between the caption output and the context. We observe
that the average GT overlap score between Easy and Hard subset (GT. ol.) is
quite disparate, yet we do not see a similar distribution learned by our
models in generated overlap score (see Gen. ol. in Table 4). The models
instead have higher overlap scores in both Easy and Hard samples (see Gen.
ol.) compared to GT overlap. This highlights the first limitation of our model
where the copying shortcut is exploited and learnt through the Easy subset
while being carried over to the Hard samples. This behaviour is also apparent
in the captioning metrics as well as Precision and Recall by the excessive gap
between Easy and Hard subset regardless of the models. Nevertheless, the
benefits of MNEM are observable, as this method outperforms our base models in
both subsets regarding captioning metrics. Improvements in Recall scores,
along with increases in length, especially indicate better diversity and NE
insertion in generated captions which can be found in all the models. However,
MNEM still can not effectively deal with the distribution shift that exists
within Wikipedia. In other words, all of the models suffer from relying too
much on the contextual information.

#### Context-based Story Telling.

Humans have an innate capability of coming up with a well sounding explanation
for an image given any context. In order to assess this skill in current
captioning pipelines, we employed a retrieval system to sort images according
to the degree of relevance given a section. Our goal is to evaluate our
models’ ability to adapt their produced image descriptions based on the
context utilized, i.e. exploit the given context that comes in the form of
Named Entities. In our experiments, we trained two fully-connected layers on
top of the visual and textual representations of CLIP (Radford et al. 2021) to
obtain ranked images according to the semantic relatedness in a decreasing
order. In Figure 3, we provide qualitative samples for the studied models with
and without MNEM. We ask the reader to refer to the Appendix for more samples.
It can be seen that models can generate context-based captions even with
irrelevant images to a certain degree. For example, even though the images in
1st and 2nd row, 3rd column are irrelevant to the input section, MNEM models
can “hallucinate” better sounding captions that can improve upon the
combination of modalities. Also, MNEM models, tend to include more relevant
information in the form of Named Entities, thus integrating relevant incoming
information from the section. However, this is not always the case, we can
also note that models tend to generate short sentences that are repeated over
many images, thus missing some salient details coming from the visual input.
All in all, even though improved captions are generated with MNEM, this innate
human skill lies far compared to current approaches that favor template-based,
short and repetitive sentences, thus opening ways for future work.

## Conclusions

In this work, we have proposed the novel task of Wikipedia Captioning. We
introduce the importance of the Wikipedia captioning due to the direct impact
of it on several applications, as well as advancing visio-linguistic tasks. We
show extensive experimentation to compare our approach with several baselines.
By incorporating a novel pretext task, namely Masked Named Entity Modelling
(MNEM), we model the distribution of NEs as an objective to be optimized. We
show that by integrating this approach, we reach top performance on Wikipedia
captioning. We also show that a model pre-trained on Wikipedia and optimized
with MNEM generalizes well to the News Captioning task. We found that
baselines tend to learn a shortcut to the captioning task and directly copy
the image description to generate a caption. To properly evaluate this effect,
we divide the Wikipedia Captioning test set according to the Jaccard
similarity between the context and ground-truth captions. We show that there
is still a big gap among these two test splits, which opens the path for
future research. Finally, we show qualitative examples regarding the human
capability of generating captions adapted to any context. We show limitations
that current models have and provide insights of the benefits of our approach.

## References

  * Anderson et al. (2016) Anderson, P.; Fernando, B.; Johnson, M.; and Gould, S. 2016.  Spice: Semantic propositional image caption evaluation.  In _European Conference on Computer Vision_.
  * Anderson et al. (2017) Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2017.  Bottom-up and top-down attention for image captioning and VQA.  _arXiv preprint arXiv:1707.07998_.
  * Bahdanau, Cho, and Bengio (2014) Bahdanau, D.; Cho, K.; and Bengio, Y. 2014.  Neural Machine Translation by Jointly Learning to Align and Translate.  1–15.
  * Banerjee and Lavie (2005) Banerjee, S.; and Lavie, A. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 65–72.
  * Biten et al. (2019) Biten, A. F.; Gomez, L.; Rusinol, M.; and Karatzas, D. 2019.  Good news, everyone! context driven entity-aware captioning for news images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12466–12475.
  * Chen et al. (2021) Chen, H.; Wang, Y.; Yang, X.; and Li, J. 2021.  Captioning transformer with scene graph guiding.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , 2538–2542. IEEE.
  * Dai et al. (2017) Dai, B.; Fidler, S.; Urtasun, R.; and Lin, D. 2017.  Towards diverse and natural image descriptions via a conditional gan.  _arXiv preprint arXiv:1703.06029_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Farhadi et al. (2010) Farhadi, A.; Hejrati, M.; Sadeghi, M. A.; Young, P.; Rashtchian, C.; Hockenmaier, J.; and Forsyth, D. 2010.  Every picture tells a story: Generating sentences from images.  In _European conference on computer vision_ , 15–29. Springer.
  * Feng and Lapata (2013) Feng, Y.; and Lapata, M. 2013.  Automatic caption generation for news images.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(4): 797–812.
  * Gardner (2011) Gardner, H. E. 2011.  _Frames of mind: The theory of multiple intelligences_.  Hachette Uk.
  * Geirhos et al. (2020) Geirhos, R.; Jacobsen, J.-H.; Michaelis, C.; Zemel, R.; Brendel, W.; Bethge, M.; and Wichmann, F. A. 2020.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11): 665–673.
  * Honnibal and Montani (2017) Honnibal, M.; and Montani, I. 2017.  SpaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.
  * Jaccard (1912) Jaccard, P. 1912.  The distribution of the flora in the alpine zone. 1.  _New phytologist_ , 11(2): 37–50.
  * Ji et al. (2021) Ji, J.; Luo, Y.; Sun, X.; Chen, F.; Luo, G.; Wu, Y.; Gao, Y.; and Ji, R. 2021.  Improving image captioning by leveraging intra-and inter-layer global representation in transformer network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 1655–1663.
  * Kim, Son, and Kim (2021) Kim, W.; Son, B.; and Kim, I. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , 5583–5594. PMLR.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Klein et al. (2017) Klein, G.; Kim, Y.; Deng, Y.; Senellart, J.; and Rush, A. M. 2017.  Opennmt: Open-source toolkit for neural machine translation.  _arXiv preprint arXiv:1701.02810_.
  * Kudo and Richardson (2018) Kudo, T.; and Richardson, J. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 66–71. Brussels, Belgium: Association for Computational Linguistics.
  * Lake et al. (2017) Lake, B. M.; Ullman, T. D.; Tenenbaum, J. B.; and Gershman, S. J. 2017.  Building machines that learn and think like people.  _Behavioral and Brain Sciences_ , 40.
  * Li et al. (2020) Li, X.; Yin, X.; Li, C.; Zhang, P.; Hu, X.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , 121–137. Springer.
  * Lin et al. (2021) Lin, C.; Miller, T.; Dligach, D.; Bethard, S.; and Savova, G. 2021.  EntityBERT: Entity-centric masking strategy for model pretraining for the clinical domain.  Association for Computational Linguistics (ACL).
  * Lin (2004) Lin, C.-Y. 2004.  Rouge: A package for automatic evaluation of summaries.  In _ACL Workshop on Text Summarization_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft COCO: Common objects in context.  In _European Conference on Computer Vision_.
  * Liu et al. (2020) Liu, F.; Wang, Y.; Wang, T.; and Ordonez, V. 2020.  Visual News: Benchmark and Challenges in News Image Captioning.  _arXiv preprint arXiv:2010.03743_.
  * Liu et al. (2017) Liu, S.; Zhu, Z.; Ye, N.; Guadarrama, S.; and Murphy, K. 2017.  Improved image captioning via policy gradient optimization of spider.  In _Proceedings of the IEEE international conference on computer vision_ , 873–881.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Lu et al. (2017) Lu, J.; Xiong, C.; Parikh, D.; and Socher, R. 2017.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning.  In _Conference on Computer Vision and Pattern Recognition_.
  * Luo et al. (2021) Luo, Y.; Ji, J.; Sun, X.; Cao, L.; Wu, Y.; Huang, F.; Lin, C.-W.; and Ji, R. 2021\.  Dual-level collaborative transformer for image captioning.  _arXiv preprint arXiv:2101.06462_.
  * Martin et al. (2018) Martin, L.; Ammanabrolu, P.; Wang, X.; Hancock, W.; Singh, S.; Harrison, B.; and Riedl, M. 2018.  Event representations for automated story generation with deep neural nets.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Ordonez, Kulkarni, and Berg (2011) Ordonez, V.; Kulkarni, G.; and Berg, T. L. 2011.  Im2text: Describing images using 1 million captioned photographs.  In _Advances in Neural Information Processing Systems_ , 1143–1151.
  * Pan et al. (2020) Pan, Y.; Yao, T.; Li, Y.; and Mei, T. 2020.  X-linear attention networks for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10971–10980.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W. 2002.  BLEU: a method for automatic evaluation of machine translation.  _Annual Meeting on Association for Computational Linguistics_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , 8748–8763. PMLR.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8): 9.
  * Ramisa et al. (2018) Ramisa, A.; Yan, F.; Moreno-Noguer, F.; and Mikolajczyk, K. 2018.  Breakingnews: Article annotation by image and text processing.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(5): 1072–1085.
  * Redmon and Farhadi (2018) Redmon, J.; and Farhadi, A. 2018.  Yolov3: An incremental improvement.  _arXiv preprint arXiv:1804.02767_.
  * Rennie et al. (2016) Rennie, S. J.; Marcheret, E.; Mroueh, Y.; Ross, J.; and Goel, V. 2016.  Self-critical Sequence Training for Image Captioning.
  * Roberts and Raffel (2020) Roberts, A.; and Raffel, C. 2020.  Exploring transfer learning with t5: the text-to-text transfer transformer.  _Accessed on_ , 23–07.
  * Schroff, Kalenichenko, and Philbin (2015) Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 815–823.
  * See, Liu, and Manning (2017) See, A.; Liu, P. J.; and Manning, C. D. 2017.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_.
  * Sennrich, Haddow, and Birch (2015) Sennrich, R.; Haddow, B.; and Birch, A. 2015.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_.
  * Shetty et al. (2017) Shetty, R.; Rohrbach, M.; Anne Hendricks, L.; Fritz, M.; and Schiele, B. 2017.  Speaking the same language: Matching machine to human captions by adversarial training.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4135–4144.
  * Srinivasan et al. (2021) Srinivasan, K.; Raman, K.; Chen, J.; Bendersky, M.; and Najork, M. 2021.  Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2443–2449.
  * Sun et al. (2020) Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Tian, H.; Wu, H.; and Wang, H. 2020.  Ernie 2.0: A continual pre-training framework for language understanding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 8968–8975.
  * Sutskever, Vinyals, and Le (2014) Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014.  Sequence to Sequence Learning with Neural Networks.  _CoRR_ , abs/1409.3: 1–9.
  * Terman (1916) Terman, L. M. 1916.  _The measurement of intelligence: An explanation of and a complete guide for the use of the Stanford revision and extension of the Binet-Simon intelligence scale_.  Houghton Mifflin.
  * Tran, Mathews, and Xie (2020) Tran, A.; Mathews, A.; and Xie, L. 2020.  Transform and tell: Entity-aware news image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13035–13045.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vedantam, Lawrence Zitnick, and Parikh (2015) Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015.  Cider: Consensus-based image description evaluation.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Vinyals, Fortunato, and Jaitly (2015) Vinyals, O.; Fortunato, M.; and Jaitly, N. 2015.  Pointer networks.  _Advances in neural information processing systems_ , 28.
  * Vinyals et al. (2015) Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015.  Show and tell: A neural image caption generator.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 38–45. Online: Association for Computational Linguistics.
  * Wu et al. (2019) Wu, F.; Fan, A.; Baevski, A.; Dauphin, Y. N.; and Auli, M. 2019.  Pay less attention with lightweight and dynamic convolutions.  _arXiv preprint arXiv:1901.10430_.
  * Wu et al. (2016) Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K.; et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Xu et al. (2015) Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R.; and Bengio, Y. 2015.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.  In _International Conference on Machine Learning_.
  * Yamada et al. (2018) Yamada, I.; Asai, A.; Shindo, H.; Takeda, H.; and Takefuji, Y. 2018.  Wikipedia2Vec: an optimized tool for learning embeddings of words and entities from Wikipedia.  _arXiv preprint arXiv:1812.06280_.
  * Yang et al. (2021) Yang, X.; Karaman, S.; Tetreault, J.; and Jaimes, A. 2021.  Journalistic Guidelines Aware News Image Captioning.  _arXiv preprint arXiv:2109.02865_.
  * You et al. (2016) You, Q.; Jin, H.; Wang, Z.; Fang, C.; and Luo, J. 2016.  Image Captioning with Semantic Attention.  In _IEEE conference on computer vision and pattern recognition_.
  * Zhang et al. (2016) Zhang, K.; Zhang, Z.; Li, Z.; and Qiao, Y. 2016.  Joint face detection and alignment using multitask cascaded convolutional networks.  _IEEE signal processing letters_ , 23(10): 1499–1503.
  * Zhang et al. (2021) Zhang, P.; Li, X.; Hu, X.; Yang, J.; Zhang, L.; Wang, L.; Choi, Y.; and Gao, J. 2021\.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5579–5588.

##  Appendix A Technical Appendix

### Dataset

The WIT111https://github.com/google-research-datasets/wit dataset contains
over 37.6 million data points based on a Wikipedia Crawl. Originally, this
dataset is proposed for the image-text retrieval task with special emphasis on
multilingual text. We discard all the data that comes from non-English
Wikipedia pages and get a subset of around 5.4M samples. In addition, we
perform extra pre-processing steps by removing short text, rare punctuation
and also filter all images in uncommon format and resize into 256x256. For
each data point including an image, we take the caption_reference_description
as the ground truth caption, caption_attribution_description as the
description and context_section_description as the associated section.
Depending on the context, we accordingly de-duplicate the dataset to assure
all the triplets (image, context, caption) unique. All the filter steps reduce
the training data to a final dataset of total over 2.6M samples. We create a
split by randomly partitioning the training data to 2.6M training, 8k
validation, 20K for testing, as the official split is not publicly available
at the time of this work.

For News Image Captioning experiments, we train the model on the GoodNews
dataset using the 421K training, 18K validation, and 23K test split, provided
by (Biten et al. 2019). Similar to WIT, each data sample in this dataset is a
triplet consisting of an article as the whole context, an image, and a
caption.

### Baselines

We define several baselines, which are composed of an LSTM encoder-decoder, to
serve as comparison to our proposed method. Particularly, we frame Wikipedia
captioning as a machine translation problem in which we treat (image, section,
description) sequence as a source language and the caption as a target
language. We thus consider machine translation sequence-to-sequence (Seq2seq)
model and its incremental variants. For this experimentation, input image
features extracted from a ResNet-152 are reused whereas a SentencePiece
tokenizer (Kudo and Richardson 2018) is trained for text tokenization.

Seq2seq. The source of image captioning models (Vinyals et al. 2015; Xu et al.
2015; Anderson et al. 2017) can be associated to the success of machine
translation sequence-to-sequence (seq2seq) models (Bahdanau, Cho, and Bengio
2014; Sutskever, Vinyals, and Le 2014). Thus, for a robust baseline we treat
the Wikipedia captioning as a machine translation task in which we treat
section with description as a source language and the caption as a target
language.

Seq2seq with Attention. The next advance in image captioning was the
introduction of attention module (Xu et al. 2015) which was also inspired from
machine translation field (Bahdanau, Cho, and Bengio 2014). We use the
attention block in the decoder to attend to the encoded features at each time
step.

Seq2seq with Pointer. The introduction of pointer mechanism (Vinyals,
Fortunato, and Jaitly 2015) allowed the seq2seq models to have a supervisory
signal on attention weights. It was mostly used for out-of-dictionary words to
directly copy it from the encoder.

Seq2seq with Pointer and Coverage. The coverage idea was introduced by (See,
Liu, and Manning 2017) to overcome the repetition problem that seq2seq models
suffering. Conceptually, they penalize the attention weights that repeatedly
attends to the same locations.

Experiments quantitatively show the robustness of the baselines in this task,
despite of its simplicity, with competitive performances compared to large
transformer counterparts including state-of-the-art methods in News Image
Captioning. This not only explains the baseline choice, but also reveals one
of the most challenging aspects of Wikipedia Captioning: a strong linguistic
bias intrinsically embedded in the Wikipedia data, misleading a model to
adversely generalize over a specific dataset subset. Hence, a mechanism for
effectively reasoning over multi-modal Wikipedia input is essential to escape
this sub-optimal convergence.

### Implementation Details

We base our implementation on the code provided by
TaT222https://github.com/alasdairtran/transform-and-tell and Huggingface
Transformers (Wolf et al. 2020).

Transformer models We use the same configuration in terms of architecture as
proposed in its original papers. We utilize the pre-trained tokenizer
available in each model to decompose text inputs. In our pre-training stage,
we use the Name Entity Recognition (NER) model available from SpaCy (Honnibal
and Montani 2017) library. We mask 80% of NEs at token level with [MASK] token
which is added to the model’s built-in vocabulary if not included. From NER
output, we only mask NEs with labels of: [PERSON, ORG, GPE] for Wikipedia and
additional ones of [NORP, LOC, FAC] in case of GoodNews. In all experiments,
we use a batch size of 16, and train the model for 3 epochs on WIT, and 8
epochs on GoodNews. For parameter optimizations, we use AdamW (Loshchilov and
Hutter 2017) with $\beta^{1}=0.9$ and $\beta^{2}=0.98$, both in pre-training
and fine-tuning. In pre-training stage, we use learning rate of $10^{-4}$
linearly warmed up during the first 5% training steps, while at fine-tuning,
we set the initial learning rate to $10^{-5}$ with the same weight schedule.
We use COCO evaluation toolkit (Lin et al. 2014) for the evaluation of our
models. Baselines Models are trained with the OpenNMT (Klein et al. 2017)
library, in which all of our models are composed of an LSTM encoder-decoder.
In all of our models we follow the same settings. We utilize batch size of 32
with Adam (Kingma and Ba 2014) optimizer, accumulating the gradients for 3
iterations before forwarding the loss. We set the learning rate to $5\times
10^{-4}$ and employ gradient clipping set to 5. We use a bi-directional LSTM
for encoder and a LSTM for decoder with hidden and embedding size is set to
512 and employ dropout with 0.1 probability. All of our models are trained for
100K iterations with inputs tokenized with SentencePiece tokenizer.

Table 5: Comparison between different transformer-based methods in News Image
Captioning. ✓means consisting of the component while X means not. The
components with (f) are frozen during training. *: excludes the data used to
pre-train ResNet-152 and RoBERTa.

Model Method Model Architecture Template #Trainable Params Pre-training Data*
Image Encoder Text Encoder Text Decoder TaT End-to-end ResNet-152(f)
RoBERTa(f) Transformer X 200M X VisualNews Two-stage ResNet-152(f) Transformer
Transformer ✓ 93M X JoGANIC End-to-end ResNet-152(f) RoBERTa(f) Transformer ✓
205M Wikipedia KB Ours End-to-end ResNet-152(f) X Transformer X 224M WIT-2M

### WiT Statistics

We supply statistics on the WIT data split mentioned in the main paper. As can
be appreciated from Table 6, around 34% of the words in the captions are Named
Entities (NEs) and description follows a very similar distribution in terms of
NEs as well. We also provide the average length of each domain. As observed,
captions are shorter in length which explains the behaviour of our models
generating short sentences. However, we do see that the average length of
sections is significantly higher compared to descriptions.

Table 6: Dataset statistics. The split and statistics of the Wiki training
data.

Train Val Test Total Number of Samples 2613732 8000 20000 2641732 Avg. Len.
Caption (Words) 8.87 8.71 8.91 8.88 Avg. Len. Description (Words) 22.06 22.15
22.01 22.06 Avg. Len. Section (Words) 193.71 193.6 193.89 193.70 #NEs in
Captions (Word) 34.36% 34.73% 34.94% 34.36% #NEs in Descriptions (Word) 32.14%
32.23% 32.57% 32.14% #NEs in Sections (Word) 22.06% 22.15% 22.12% 22.06%
#Captions with NEs 84.99% 84.28% 84.1% 84.98% #Descriptions with NEs 92.1%
91.98% 91.6% 92.1% #Sections with NEs 98.58% 98.71% 98.56% 98.58%

As mentioned in the main paper, we divide the test set into two subsets,
namely Easy and Hard. These subsets are defined according to Jaccard
similarity (i.e. Intersection over Union). This setting discloses a shortcut
in which our models exploit plain text similarities existing in the Easy set.
Here, we present a distribution of the Jaccard similarity according to three
types of context in Figure 4. Clearly, description has a higher overlap with
captions than sections, and its combination even provides stronger
correlation. Hence, it is expected that models exploit a copying mechanism
directly from the context, especially descriptions rather than sections.
Furthermore, we see that there are more samples that have Jaccard similarity
higher than 0.5, leading the models to “overfit” to the Easy set distribution
instead of focusing on the Hard set.

Figure 4: Overlap between the caption and description (left); caption and
section (right), measured by Jaccard similarity with a small modification. For
each training example, we split each of these text into a sequence of words,
then compute the intersection by adding up the length of all common words,
then normalize it by the caption length. A score closer to 1 means a higher
degree of intersection, i.e a larger chunk of the caption appears in the
description. In this figure, each value $x$ in the horizontal axis indicates
the interval $[x-1,x)$ except the last column, which is $[0.9,1]$

### WIT vs GoodNews

We provide a comparison between WIT and GoodNews in Table 7. The first
advantage of Wikipedia over News captioning is the amount of data available to
train our models. Also, Wikipedia has a lower caption length compared to
GoodNews (8.88 to 18.21). This could prompt us to interpret Wikipedia
Captioning as a simpler task in terms of grammar and generation-wise
correctness compared to GoodNews. Nevertheless, we observe that in terms of
NEs, Wikipedia contains a higher percentage of NEs at word level (35% in Wiki
to 19.5% in GoodNews). Hence, in Wikipedia captioning, the difficulty lies in
selecting, injecting or generating more relevant NEs on average than News
captioning while another difficulty is the highly contextualised nature of the
task (same image used in different articles).

Moreover, Figure 4 shows how different NE categories are distributed within
both datasets. Aside from the magnitude, we observe a similarity between these
two distributions which motivates us to use Wikipedia as a probing domain for
News captioning. Among all, people’s names (PERSON), geopolitical entities
(GPE) and organizations (ORG) are the most prevalent NE categories. As a
result, we perform MNEM pre-training focusing on these NEs while regarding to
the rest as context.

Figure 5: Entity categories and its cardinality in WIT and GoodNews. Among
most common types in both datasets are people’s names, organizations and
geopolitical entities.

Table 7: Comparison between WIT and GoodNews datasets.

GoodNews Wiki Number of Samples 466000 5411978 Unique Images – 2082504 Unique
(Image+Desc) – 2550904 Unique (Image+Sec) – 2640462 Average Caption Length
(Words) 18.21 8.88 Named Entities (Word) 19.59% 35.31% Named Entities
(Sentence) 95.51% 96.29% Nouns 46.70% 36.58% Verbs 11.22% 8.03% Adjectives 5%
6.31%

### More on effects of different types of context

Given the effect of the Easy subset on the copy behavior as figured out in the
main paper, we investigate the influence of three forms of context on this
phenomenon: description, section and Wiki. We conduct experiments on TaT with
MNEM as this model shows to be effective in dealing with multi-modal inputs.
Table 8 presents the results corresponding to each type of contextual
information. When description is used as context, both models favor copying
strategy (see Gen. ol.), demonstrating that description has a greater impact
on this behavior. On the other hand, when section is utilized as context, the
models have lower generated overlap score in both subsets. This is somehow
expected given the larger amount of information at a fine-grained level in
sections. Moreover, captions present a compact version of sections summarizing
the whole context instead of just picking out a part of the text to describe
the image.

Table 8: Performance on Easy and Hard subsets. Wiki. indicates the context
combining both description and section. $\dagger$ indicates that MNEM pre-
training is performed.

Model Context Test Set GT ol. Gen. ol. Gen. len. B C Pr Re TaT Desc. Easy 0.78
0.86 8.66 33.40 336.10 40.32 37.83 Desc. Hard 0.19 0.64 6.65 7.06 95.27 16.55
13.57 Desc. Overall 0.50 0.74 7.72 20.94 212.31 27.71 26.33 Sec. Easy 0.71
0.78 5.66 14.65 208.09 25.92 22.08 Sec. Hard 0.25 0.61 5.49 4.40 76.23 12.50
8.50 Sec. Overall 0.51 0.70 5.87 9.20 150.20 20.02 15.86 Wiki. Easy 0.81 0.92
7.65 29.93 313.99 39.49 37.2 Wiki. Hard 0.35 0.75 6.03 8.48 133.68 19.06 16.06
Wiki. Full 0.71 0.88 6.46 24.47 267.37 32.10 34.64 $\text{TaT}^{\dagger}$
Desc. Easy 0.78 0.85 8.73 33.59 338.8 40.76 38.10 Desc. Hard 0.19 0.62 6.72
7.28 98.33 16.88 13.84 Desc. Overall 0.50 0.74 7.77 22.36 224.38 29.36 27.90
Sec. Easy 0.71 0.78 5.66 15.04 213.11 26.60 22.76 Sec. Hard 0.25 0.60 6.22
4.79 78.18 13.13 8.94 Sec. Overall 0.51 0.70 5.44 9.54 153.87 20.75 16.01
Wiki. Easy 0.81 0.92 7.89 30.76 319.01 39.71 38.20 Wiki. Hard 0.35 0.73 6.16
8.91 137.81 19.39 16.53 Wiki. Full 0.71 0.87 7.08 25.31 272.46 33.34 35.56

### Methods in News Image Captioning

Table 5 provides a high-level summary of transformer models for News
Captioning. Overall, these methods still rely on prominent pre-trained
encoders to independently process multi-modal inputs while leave the burden of
both reasoning and generating on a decoder. However, our proposed method aims
to explore the potential of rich semantic and linguistic knowledge in PLMs,
effectively freeing the model from the dependency on the external text encoder
and templates for handling NEs. In addition, our pre-training strategy in
Wikipedia Captioning sufficiently equips the model with Wikipedia knowledge,
which is useful to perform similar tasks in different domains.

Table 9: Performance of proposed models on GoodNews. Ft means fine-tuning the
models with the captioning objective in the corresponding dataset. Note that
for captioning metrics, we directly use the results that are reported in the
original papers.

Model GoodNews B M R C Pr Re MNEM Ft TaT $\checkmark$ 6 10.1 21.2 53.1 21.3
19.7 $\checkmark$ $\checkmark$ 6.23 10.08 22.47 55.14 19.8 18.72
$\text{T5}_{\textit{BASE}}$ $\checkmark$ 6.06 9.56 20 52.1 19.6 18.06
$\checkmark$ $\checkmark$ 6.07 9.61 20.16 53.11 19.98 18.66

### More on GoodNews

We conduct additional experiments on GoodNews with results reported in Table
9. As observed, MNEM pre-training improves the performance on both TaT and T5,
showing the efficacy of NE-centric strategy in News domain. On the other hand,
there is a big gap in performance between $\text{T5}_{\textit{BASE}}$ compared
to $\text{T5}_{\textit{BASE}}\texttt{++}$ (see Table 2 in the main paper),
suggesting that our model can effectively reason on multi-modal inputs rather
than text-only inputs.

### Qualitative Samples on Wikipedia Captioning

In this section, we show qualitative samples on the Wikipedia captioning to
showcase our models capabilities. As can be seen, all of the models tend to
directly copy descriptions into the generated captions. Moreover, we show that
T5 and TaT pre-trained with MNEM usually generate NEs with more variety and
higher precision.

Description
View of Tourane Airfield (Da Nang), Indochina, looking north, circa in 1954.
Original caption: ”New metal mat area on northeast side of runway. Two of
three F8F aircraft observed departing on napalm mission. Aircraft returned in
35 minutes.”. Section
XF8F-1 Prototype aircraft, two built. F8F-1 Bearcat Single-seat fighter
aircraft, equipped with folding wings, a retractable tailwheel, self-sealing
fuel tanks, a very small dorsal fin, powered by a 2,100hp (1,566kW) Pratt and
Whitney R-2800-34W Pratt & Whitney R-2800-34W Double Wasp radial piston
engine, armed with four 0.50 in (12.7 mm) machine guns, 658 built. … Ground-
Truth: French Bearcats at Tourane Air Base, circa 1954.TaT: F8F-1 in 1954
TaT+MNEM: F8F-1 in 1954 GPT-2: Tourane Airfield (Da Nang), Indochina,
1954GPT-2+MNEM: Tourane Airfield, Indochina, 1954 T5: Tourane Airfield (Da
Nang), Indochina, looking north, circa 1954. T5+MNEM: Two of three F8F
aircraft observed departing on napalm mission at Tourane Airfield (Da Nang),
Indochina, c. 1954
---
Description
Portrait of George Townshend, Lord Ferrers, in uniform of the 15th Kings light
Dragoons, standing full length. Section
George Townshend, 2nd Marquess Townshend, PC, FRS (18 April 1753 – 27 July
1811), known as The Lord Ferrers of Chartley from 1770 to 1784 and as The Earl
of Leicester from 1784 to 1807, was a British peer and politician. Ground-
Truth: Lord Townshend by Sir Joshua Reynolds.TaT George Townshend in uniform
of the 15th Kings light Dragoons standing full length TaT+MNEM: George
Townshend, Lord Ferrers, in uniform of the 15th Kings light Dragoons standing
full length GPT-2: George Townshend, Lord Ferrers, in uniform of the 15th
Kings light Dragoons, standing full lengthGPT-2+MNEM: The Earl of Leicester
T5: George Townshend, 2nd Marquess Townshend T5+MNEM: Lord Ferrers, in uniform
of the 15th Kings Light Dragoons, standing full length
---

Description
Español: Soldado no identificado de la División Azul española en el Frente
Oriental. Section
Devised in June 1943, Nurnberg was purely a defensive operation in the
Pyrenees along both sides of the Spanish-French border in the event of Allied
landings in the Iberian peninsula, which were to repel an Allied advance from
Spain into France. Ground-Truth: A Spanish volunteer of the Blue DivisionTaT A
Spanish soldier in the Pyrenees TaT+MNEM: Spanish Soldier of the Division Azul
in the Pyrenees GPT-2: Nurnberg in 1943 GPT-2+MNEM: Luis Nurnberg T5: Azul
division in the Frente Oriental. T5+MNEM: Nurnberg in the Spanish-French
border.
---