11institutetext: Spoken Language Systems (LSV)
Saarland Informatics Campus, Saarland University
Saarbrücken, Germany
11email<EMAIL_ADDRESS>
# Fusion Models for Improved Visual Captioning

Marimuthu Kalimuthu corresponding author Aditya Mogadala  Marius Mosbach
Dietrich Klakow

###### Abstract

Visual captioning aims to generate textual descriptions given images or
videos. Traditionally, image captioning models are trained on human annotated
datasets such as Flickr30k and MS-COCO, which are limited in size and
diversity. This limitation hinders the generalization capabilities of these
models while also rendering them liable to making mistakes. Language models
can, however, be trained on vast amounts of freely available unlabelled data
and have recently emerged as successful language encoders [10] and coherent
text generators [4]. Meanwhile, several unimodal and multimodal fusion
techniques have been proven to work well for natural language generation [11]
and automatic speech recognition [30]. Building on these recent developments,
and with the aim of improving the quality of generated captions, the
contribution of our work in this paper is two-fold: First, we propose a
generic multimodal model fusion framework for caption generation as well as
emendation where we utilize different fusion strategies to integrate a
pretrained Auxiliary Language Model (AuxLM) within the traditional encoder-
decoder visual captioning frameworks. Next, we employ the same fusion
strategies to integrate a pretrained Masked Language Model (MLM), namely BERT,
with a visual captioning model, viz. Show, Attend, and Tell, for emending both
syntactic and semantic errors in captions. Our caption emendation experiments
on three benchmark image captioning datasets, viz. Flickr8k, Flickr30k, and
MSCOCO, show improvements over the baseline, indicating the usefulness of our
proposed multimodal fusion strategies. Further, we perform a preliminary
qualitative analysis on the emended captions and identify error categories
based on the type of corrections.

###### Keywords:

Visual Captioning Multimodal/Model Fusion Auxiliary Language Model AuxLM
Emendation Natural Language Generation.

##  1 Introduction

The field of deep learning has seen tremendous progress ever since the
breakthrough results of AlexNet [16] on the ImageNet Large Scale Visual
Recognition Challenge (ILSVRC). Significant algorithmic improvements have
since been achieved, both for language [10, 31] and visual scene understanding
[6, 13]. Further, there has been numerous efforts for the joint understanding
of language and visual modalities in terms of defining novel tasks, creating
benchmark datasets, and proposing new methods to tackle the challenges arising
out of the multimodal nature of data [21]. One specific problem that has
piqued the interest of researchers and practitioners in Computer Vision (CV)
and Natural Language Processing (NLP) is Visual Captioning. Considerable
progress has been made since the introduction of Show and Tell [33], which is
an end-to-end model for image captioning. Despite these encouraging
developments, the visual captioning models are still brittle, unreliable,
prone to making unexplainable mistakes [27], and, most importantly, their
performance is nowhere close to human-level understanding. However, although
still a black-box, the recent unprecedented progress on language modeling has
proven the potential of these models to generate semantically fluent and
coherent text [4, 26] and encode powerful representations of language using
bidirectional context [10]. Building on these developments, we propose to
incorporate external language models into visual captioning frameworks to aid
and improve their capabilities both for description generation and emendation.
Although the proposed architecture (See Figure 1) could be used for both
caption generation and correction, in this paper, we only focus on the task of
emending captions. However, we describe the changes needed to the architecture
to achieve caption generation. Broadly, our architecture consists of four
major components, viz. a CNN encoder, an LSTM decoder, a pretrained AuxLM, and
a Fusion module (See Figure 1). To the best of our knowledge, our architecture
is novel since the current caption editing approaches [28, 29] in the
literature do not leverage AuxLMs.

The rest of the paper is organized as follows. In Section 2, we briefly review
relevant works on caption editing. In Section 3, we provide background on
fusion strategies that have been empirically shown to work well for neural
machine translation, automatic speech recognition, and story generation and
introduce our fusion architecture. Following this, we describe our
experimental setup including the implementation details of our caption editing
model in Section 4. We then present our quantitative results along with a
short qualitative analysis of the emended captions in Section 5. Finally, we
conclude our paper in Section 6 discussing some future research directions.

##  2 Related Work

Recently, few approaches have been proposed for editing image captions [28,
29]. Although these methods have been shown to produce improved quantitative
results, they have some limitations such as completely relying on labelled
data, which is limited in size and diversity, for training. In addition, it is
cost-intensive to obtain caption annotations and, sometimes, it is not even
possible due to privacy concerns. This is the case, for instance, in the
medical domain. Furthermore, these approaches implicitly learn a language
model on the decoder-side with limited textual data. Moreover, they do not
make use of external language models, i.e., AuxLMs, which can be trained on
freely available unlabelled data, are more powerful since they learn rich
language representations [10, 26], and recently have been trained on enormous
amounts of data with billions of parameters [4, 26]. To the best of our
knowledge, there have been no previous works that incorporate AuxLMs into the
caption model either for description generation or to correct errors. To
address the above stated limitations and leverage the advantages of AuxLMs as
powerful language learners, we propose a generic framework that can accomplish
both caption generation and editing tasks depending on the type of AuxLM used.

##  3 Fusion Techniques and Variations

Unimodal and multimodal model fusion has been explored extensively in the
context of ASR [7, 30], Neural Machine Translation (NMT) [12], and
hierarchical story generation [11]. However, to the best of our knowledge,
there have been no similar works for visual captioning. In this section, we
review these fusion methods and relate them to our goal of achieving image
caption generation and emendation by using additional language models, i.e.,
AuxLMs.

####  3.0.1 Deep Fusion.

Gulcehre et al. [12] explored Deep Fusion for NMT in which a translation model
trained on parallel corpora and an AuxLM trained on monolingual, target
language data are combined using a trainable single layer neural network
model. Typically, the AuxLM is an autoregressive LSTM-based recurrent neural
network [20] while the translation model follows a typical sequence-to-
sequence (Seq2Seq) [8] encoder-decoder architecture. To achieve Deep Fusion, a
gate is learned from the hidden state of AuxLM and then concatenated with the
hidden state of the translation model.

The drawback of this approach is that both the AuxLM and the translation model
are trained separately and kept frozen while the fusion layer is trained.
Hence, they never get a chance to communicate and adapt their parameters
during training. This forces these models to learn redundant representations.

####  3.0.2 Cold Fusion.

To address the limitation of Deep Fusion, Sriram et al. [30] introduced the
so-called Cold Fusion strategy for ASR. The mechanism uses a pretrained AuxLM
while training the Seq2Seq model. The AuxLM is still kept frozen as in the
case of Deep Fusion, however the parameters of the fusion layer and the
Seq2Seq decoder are trained together, leveraging the already-learned language
representations of AuxLM. For more details see Section 3.2.2.

####  3.0.3 Hierarchical Fusion.

In a similar vein, Fan et al. [11] has explored a sophisticated fusion
strategy for textual story generation in a hierarchical fashion where they
combine a pretrained convolutional Seq2Seq model with a trainable
convolutional Seq2Seq model. More specifically, the hidden states of a
pretrained Seq2Seq model are fused with the hidden states of a trainable
Seq2Seq model using a slightly modified version of the Cold Fusion [30]
approach. We call this scheme as Hierarchical Fusion hereafter.

Considering the advantages of the fusion techniques of Sriram et al. [30] and
Fan et al. [11] over Deep Fusion, we adapt them in our fusion model framework.

###  3.1 Auxiliary Language Model

Depending on the task, i.e., caption generation or editing, two classes of
language models can be used as AuxLMs. For caption generation, since we will
not have access to the right-side context during test time, an autoregressive
(i.e., causal) language model [3, 4, 19, 26] that predicts the next token
using only the left-side context is qualified to be used as the AuxLM. For a
caption editing task, however, a bidirectionally contextualized language
model, for example BERT [10], would be better suited as AuxLM since they
better encode sequences than their unidirectional counterparts. Also,
intuitively, since our task is to edit captions, we already have a noisy
version of captions (either from the baseline or from other off-the-shelf
caption models). If any errors are detected, we want to correct them, but
otherwise leave them unmodified. In our experiments, we use a pretrained BERT
model (uncased) that has been trained on English Wikipedia and the BooksCorpus
using a combination of Masked Language Modeling (MLM) and next-sentence
prediction objectives. Although it is possible to fine-tune the pretrained
AuxLMs [22] on the target domain (the image captions in our case) before
integrating it in the fusion model, we did not perform this step in our
experiments. However, the fine tuning process would likely prove to be
beneficial if we intend to adapt the AuxLMs to more focused domains such as
medical [25] or remote sensing [18] and we leave this pursuit for future work.

###  3.2 Fusion Strategies and Architecture

Inspired by the success of multimodal fusion strategies for ASR [7, 30] and
unimodal fusion for NLG [11, 12], we slightly modify these fusion schemes and
utilize them for the task of emending image captions. By doing so, we leverage
the rich language representations of AuxLMs to achieve sentence-level
coherency and grammatical correctness in the emended captions.

Figure 1: Architecture of our proposed fusion model. The encoded image is fed
to the LSTM decoder only at the first time step. BERT MLM [10] has been
pretrained to predict the masked token at current time step whereas the LSTM
decoder is trained to predict the next token given all previous tokens. The
Fusion Module can be any instance of the fusion schemes discussed in Section
3.2.

Figure 1 depicts the general architecture of the proposed fusion model. It
contains four major components, namely a ConvNet encoder, an LSTM decoder, a
BERT encoder (pretrained MLM), and a Fusion module. The architecture is
flexible and can be used for two tasks, i.e., visual captioning and caption
emendation, depending on the type of AuxLM chosen. For caption emendation, a
text encoder such as BERT is used as an AuxLM. The LSTM decoder processes the
sequence from left-to-right whereas the BERT model utilizes the entire
sequence due to its inherent nature of encoding contextualized representation.

For the visual captioning task, the AuxLM must be an autoregressive model
since we do not have access to the whole sequence at inference time. One
possibility could be to replace the BERT MLM component with an LSTM-based
AuxLM or the recently proposed (and more powerful) Transformer-based [31]
autoregressive language model GPT-3 [4].

Further, for both captioning and emendation tasks, the Fusion Module component
is flexible enough to support any sophisticated fusion method, which provides
a framework and an opportunity for future works to come up with improved
fusion schemes. In addition, the architecture could be employed in a domain-
adapted way for visual captioning by integrating an AuxLM that has been
trained in the domain of interest. There has been growing interest in recent
years to automatically generate descriptions for medical
images111https://www.imageclef.org/2020/medical/caption such as radiology
outputs [14, 25]. The domain adapted AuxLM can be useful particularly in
settings where labelled image caption data is scarce. However, unlabelled
textual data is usually abundant. In such scenarios, an AuxLM can be trained
on the target domain data and integrated into our fusion model framework for
generating target domain specific image descriptions.

We now introduce the notations of hidden states used in our fusion model.
Following Sriram et al. [30], we represent the final layer hidden states of
pretrained BERT and trainable LSTM decoder as $h^{MLM}$ and $h^{LSTM}$
respectively.

####  3.2.1 Simple Fusion (SF).

One of the simplest possible fusion mechanisms is through the concatenation of
the hidden states of pretrained AuxLM and trainable visual captioning model,
followed by a single projection layer with some non-linear activation function
($\sigma$), such as ReLU.

$\displaystyle\textbf{g}_{t}$
$\displaystyle=\sigma\left(\textbf{W}[\textbf{h}_{t}^{LSTM};\textbf{h}_{t}^{MLM}]+\textbf{b}\right)$
(1a)

The output of above non-linear transformation ($\textbf{g}_{t}$) can then be
passed through a single linear layer with dropout to obtain prediction scores
over the vocabulary.

####  3.2.2 Cold Fusion (CF).

A more sophisticated fusion can be achieved by introducing gates, thereby
allowing the captioning model and AuxLM to moderate the information flow
between them during the training phase. We slightly modify the cold fusion
approach of Sriram et al. [30] in our fusion model which is as follows:

$\displaystyle\textbf{h}_{t}^{LM}$
$\displaystyle=\sigma\left(\textbf{W}[\textbf{h}_{t}^{MLM}]+\textbf{b}\right)$
(2a) $\displaystyle\textbf{g}_{t}$
$\displaystyle=\sigma\left(\textbf{W}[\textbf{h}_{t}^{LSTM};\textbf{h}_{t}^{LM}]+\textbf{b}\right)$
(2b) $\displaystyle\textbf{h}_{t}^{CF}$
$\displaystyle=[\textbf{h}_{t}^{LSTM};\left(\textbf{g}_{t}\circ\textbf{h}_{t}^{LM}\right)]$
(2c) $\displaystyle\textbf{r}_{t}^{CF}$
$\displaystyle=\sigma\left(\textbf{W}[\textbf{h}_{t}^{CF}]+\textbf{b}\right)$
(2d)

As with simple fusion, the representation $\textbf{r}_{t}^{CF}$ is followed by
a single linear layer with dropout (not shown here) to obtain prediction
scores over the vocabulary.

####  3.2.3 Hierarchical Fusion (HF).

In the context of text generation, an advanced fusion mechanism based on Cold
Fusion has been introduced by Fan et al. [11] for the open-ended and creative
task of story generation. We adopt their way of model fusion with minor
modifications, in the spirit of keeping the model simple. More specifically,
after learning two separate gates followed by a concatenation, we only use a
single linear layer with GLU activations [9] instead of 5. Further, to capture
the rich sequence representation for caption editing, we use an MLM as AuxLM
instead of a convolutional Seq2Seq model. We refer to Fan et al. [11] for full
details of their fusion mechanism.

$\displaystyle\textbf{h}_{t}^{C}$
$\displaystyle=\left[\textbf{h}_{t}^{MLM};\textbf{h}_{t}^{LSTM}\right]$ (3a)
$\displaystyle\textbf{g}_{t}^{left}$
$\displaystyle=\sigma\left(\textbf{W}[\textbf{h}_{t}^{C}]+\textbf{b}\right)\circ\textbf{h}_{t}^{C}$
(3b) $\displaystyle\textbf{g}_{t}^{right}$
$\displaystyle=\textbf{h}_{t}^{C}\circ\left(\sigma\left(\textbf{W}[\textbf{h}_{t}^{C}]+\textbf{b}\right)\right)$
(3c) $\displaystyle\textbf{g}_{t}^{C}$
$\displaystyle={GLU}\left([\textbf{g}_{t}^{left};\textbf{g}_{t}^{right}]\right)$
(3d) $\displaystyle\textbf{g}_{t}^{lp}$
$\displaystyle=\textbf{W}[\textbf{g}_{t}^{C}]+\textbf{b}$ (3e)
$\displaystyle\textbf{g}_{t}^{f}$
$\displaystyle={GLU}\left(\textbf{g}_{t}^{lp}\right)$ (3f)

Again, the result of the final GLU (i.e., $\textbf{g}_{t}^{f}$) is passed
through a single linear layer with dropout to obtain prediction scores over
the image caption vocabulary.

In all our fusion methods (i.e., SF, CF, and HF), ; represents concatenation,
$\circ$ stands for hadamard product, and $\sigma$ indicates a non-linear
activation function, for which we use ReLU. The gating parameters W and b,
which are part of the Fusion Module (see Figure 1), are learned while training
the LSTM decoder of the fusion model whereas all the parameters of BERT MLM
are kept frozen.

##  4 Experiments

We train one baseline and three fusion models on the three commonly used image
captioning datasets: Flickr8k, Flickr30k, and MS-COCO. Descriptions and
implementation details are given in the following sections.

###  4.1 Baseline

This will be the model without any AuxLM component in its architecture. Any
off-the-shelf visual captioning model satisfying this condition can be used as
a baseline. The only requirement is that it should be possible to generate
captions given the test set images from the dataset in question. In our
experiments, we use Show, Attend, and Tell [35] where we replace the original
VGGNet with ImageNet pretrained ResNet-101 [13] which encodes the images to a
feature map of size 14 x 14 x 2048. For the decoder, we use the standard LSTM
with two hidden layers. After training, we use a beam size of 5 for generating
captions on the test sets of respective datasets.

###  4.2 Fusion Model Training

For each of the datasets, we train three caption models with different
initializations for all the fusion techniques (i.e., SF, CF, and HF) proposed
in Section 3.2.

####  4.2.1 Implementation Details.

First, we lowercase the captions and tokenize them using
WordPiece222https://github.com/google-research/bert tokenization [34] in the
same way the BERT MLM model was trained. This consistency in tokenization is
important for successful training since the captioning model relies on AuxLM
for the hidden state representations at all time steps throughout the training
and testing phases. Tokens appearing less than 5 times are replaced with a
special $<$unk$>$ token yielding a vocabulary size of 25k. We implement our
fusion models in PyTorch [24].

####  4.2.2 Decoder and Fusion Module Training.

The images are rescaled to a fixed size of 256 x 256 and encoded using
ResNet101 [13] pretrained on ImageNet and kept frozen throughout training. As
with the baseline model, we use an LSTM with 2 hidden layers and set the
embedding and decoder dimensions to 1024. The LSTM decoder takes the token at
current time step along with previous history and predicts the next token. The
BERT MLM model, however, consumes the entire sequence with the token at the
next time step being masked using a special [MASK] token and it predicts this
masked token (See Figure 1). The hidden state representations of both LSTM
decoder and BERT are then passed to the Fusion Module, which can be any one of
the fusion mechanisms discussed in Section 3.2, to predict the next token
(seen from the perspective of the LSTM decoder).

We minimize the Cross Entropy loss using the Adam optimizer [15] with a
learning rate of $5e^{-4}$ and a batch size of 128. Initially, the model is
scheduled to be trained for 7 epochs. However, the learning rate is halved or
an early stopping is triggered if the validation BLEU did not improve for 2
and 4 consecutive epochs respectively.

####  4.2.3 Caption Emendation.

After the fusion model is trained, it can be used in inference mode to correct
errors in the captions. While decoding, at time-step t, the LSTM predicts
$\textit{(t+1)}^{th}$ token based on left-side history whereas BERT MLM
predicts the [MASK]ed $\textit{(t+1)}^{th}$ token based on the bidirectional
context. This process continues until a special $<$eos$>$ symbol is predicted.
As with the baseline model, we again use the same beam size 5 during our
evaluations.

##  5 Results

We evaluate our models using both quantitative and qualitative approaches. In
the following, we present each of them separately.

###  5.1 Quantitative Analysis

To evaluate our proposed models i.e., baseline and the fusion approaches, we
use the standard metrics used for image captioning such as BLEU-{1-4} [23],
METEOR [2], ROUGE-L [17], CIDEr [32], and SPICE [1]. Table 1 presents the
average scores over three runs on the test sets of “Karpathy
split”333https://cs.stanford.edu/people/karpathy/deepimagesent on the
respective datasets.

Table 1:  Results of proposed fusion methods on three benchmark image captioning datasets. BL-Baseline, SF-Simple fusion, CF - Cold fusion, HF - HNSG fusion, B-n - BLEU, M - METEOR, R-L - ROUGE-L, C - CIDEr, and S - SPICE. |  | Automatic Evaluation Measures
---|---|---
Dataset | Model | B-1 | B-2 | B-3 | B-4 | M | R-L | C | S
| BL | 62.8 | 44.9 | 31.4 | 21.3 | 20.6 | 47.1 | 55.1 | 14.3
| SF | 64.6 | 46.6 | 32.8 | 22.8 | 21.2 | 47.8 | 56.9 | 14.8
Flickr8k | CF | 64.5 | 46.7 | 32.7 | 22.8 | 21.3 | 47.6 | 56.5 | 14.6
| HF | 64.1 | 45.8 | 32 | 21.8 | 20.9 | 47 | 55.5 | 14.4
| BL | 63.3 | 44.4 | 30.9 | 21.6 | 19.2 | 44.5 | 45.1 | 13.2
| SF | 64.7 | 45.6 | 32.0 | 22.4 | 19.7 | 44.9 | 46.7 | 13.6
Flickr30k | CF | 64.5 | 45.7 | 31.8 | 22.1 | 19.8 | 45 | 46.3 | 13.7
| HF | 64.6 | 45.4 | 31.7 | 22 | 19.4 | 45 | 46.2 | 13.3
| BL | 70.1 | 52.8 | 38.4 | 28 | 24.5 | 51.8 | 91.5 | 17.5
| SF | 70.8 | 53.7 | 40.4 | 30.2 | 25.1 | 52.6 | 94.6 | 17.8
MSCOCO | CF | 71 | 53.9 | 40.7 | 30.5 | 25.3 | 52.9 | 95 | 17.9
| HF | 70.9 | 53.8 | 40.6 | 30.5 | 25 | 52.7 | 94.8 | 17.8

It can be observed from Table 1 that all our fusion models outperform the
baseline model. However, when we compare performance of fusion models with one
another we comprehend that there is no considerable difference. To be
specific, on the MS-COCO dataset the Cold Fusion strategy outperforms other
fusion techniques in all metrics while there is no clear winner for both
Flickr8k and Flickr30k. Nevertheless we observe for the Flickr8k and Flickr30k
datasets that the Simple Fusion model is a preferable option, since it gives
the largest BLEU-4 score. This can be attributed to our optimization criterion
since all our models are optimized for BLEU-4 while training. This leads to
the increase of the BLEU-4 score; especially for the Simple Fusion model
trained on Flickr8k and Flickr30k which are much smaller datasets in
comparison to MS-COCO.

###  5.2 Qualitative Analysis

In Figure 2, we present the token emendation distributions of the fusion
techniques on all three datasets. When comparing the edits made by different
fusion techniques, the distribution is similar. To understand edit
distributions among datasets, we define token edit range as the range between
smallest possible token edits, which is 1, to largest possible token edits,
which is the maximum length of captions. We observe that the token edit range
(1-3 for MS-COCO) is smaller than Flickr8k (1-5) and Flickr30k (1-4) even
though MS-COCO is about 14x and 4x larger than Flickr8k and Flickr30k
respectively. This indicates the challenging nature of the Flickr caption
datasets where the baseline model makes more mistakes, for which case our
fusion model editing has been more helpful.

(a) Flickr8k

(b) Flickr30k

(c) MSCOCO

Figure 2: Distribution of (token) corrections made by fusion models on
Flickr8k, Flickr30k, and MS-COCO. X-axis represents how many tokens have been
changed by the fusion model while the Y-axis shows the frequencies.

Owing to the criticism of the BLEU metric to correlate poorly with human
judgements [5], we perform a preliminary study on the emendations of our
fusion models to better understand the quality of emended captions. We
identify several types of emendations and group them broadly into the
following five categories based on whether the langauge or image-related
attributes have been changed in the caption.

  1. 1.

Gender: Modification of gender to correctly describe image.

  2. 2.

Color: Modification of color to correctly describe image.

  3. 3.

Specificity: Emendations to achieve specific captions instead of generic ones.

  4. 4.

Syntactic: Emendation to achieve syntactic correctness.

  5. 5.

Semantic: Emendations to correctly describe the scene.

It should however be noted that this classification has been done with a
preliminary study and a comprehensive human evaluation is needed to arrive at
a more fine-grained classification. For illustrations, see Appendix 0.A.

##  6 Conclusion

In this paper, we have proposed a generic multimodal model fusion framework
that can be utilized for both caption generation and editing tasks depending
on the type of AuxLM that is integrated in the fusion model. We have
implemented a caption editing model by integrating a pretrained BERT model and
showed improved results over the baseline model on three image captioning
benchmark datasets. Further, we conducted a preliminary qualitative analysis
on the emended captions and identified a litany of categories based on the
image or language-related attributes modified in the captions. For the future
work, we plan to focus on three aspects. First, we will focus on utilizing the
proposed fusion model for the caption generation task using a state-of-the-art
autoregressive language model. Second, we aspire to employ our fusion model
for automatic description generation of medical images while training a
domain-adapted AuxLM. Third, we plan to conduct a human evaluation on the
emended captions and come up with a fine-grained classification of errors
corrected by our fusion model.

## Acknowledgements

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) – project-id 232722074 – SFB 1102. We extend our thanks
to Matthew Kuhn for painstakingly proofing the whole manuscript.

##  Appendix 0.A Appendix

Here we provide examples of (token) corrections made by the fusion models and
categorize the edits into one of the following five categories: (i) Gender
(ii) Color (iii) Specificity (iv) Syntactic (v) Semantic, based on the nature
of change.

The above classification has been provided only for the purpose of preliminary
illustration. For a thorough understanding of the trends in caption
emendations and to draw conclusions, a detailed study using human evaluation
should be performed on all three datasets. We leave this aspiration for future
work. In the following examples, we color the incorrect tokens in red, the
correct replacements in green, and the equally valid tokens in brown.

####  0.A.0.1 Semantic correction.

This section presents examples where the fusion models have corrected few
tokens in the baseline captions so as to make them semantically valid with
respect to the image. Edits to achieve semantic correctness may include
emendation of visual attributes such as colors, objects, object size, etc.

baseline: a man wearing a black hat and red hat stands in front of a brick
wall simple fusion: a man in a black jacket and black hat stands in front of a
brick wall

Figure 3: An example illustrating the correction of semantic errors in the
captions by our simple fusion model.

baseline: a man standing next to a sheep in a field cold fusion: a man
standing next to cows in a field

Figure 4: Another example to show correction of semantic errors with cold
fusion.

####  0.A.0.2 Gender alteration.

This section provides an illustration of the case where the fusion models
corrected the wrong gender of captions from the baseline model.

baseline: a woman riding a wave on top of a surfboard cold fusion: a man
riding a wave on top of a surfboard

Figure 5: An example of cold fusion approach achieving gender correction.

####  0.A.0.3 Specificity.

This deals with emendations of fusion models where the corrected captions end
up describing the images more precisely than the baseline captions.

baseline: a person in a helmet is riding a wave hierarchical fusion: a man
wearing a harness is riding a wave

Figure 6: An example to show achievement of specificity with hierarchical
fusion.

####  0.A.0.4 Syntactic correction.

In this section, we show an example to demonstrate the case where syntactic
errors such as token repetitions in the baseline captions are correctly
emended by the fusion models.

baseline: a white bowl filled with bananas and bananas simple fusion: a white
bowl filled with bananas and nuts

Figure 7: Replacement of repetitive tokens with a correct alternative.

####  0.A.0.5 Color correction.

In this part, we show an example to illustrate the case where the fusion
models emended color attributes in the captions of the baseline model.

baseline: a vase filled with pink flowers on a table cold fusion: a vase
filled with purple flowers on a table

Figure 8: An example of cold fusion achieving emendation of color attribute.

## References

  * [1] Anderson, P., Fernando, B., Johnson, M., Gould, S.: SPICE: Semantic propositional image caption evaluation. In: ECCV, 2016. vol. 9909, pp. 382–398 (2016)
  * [2] Banerjee, S., Lavie, A.: METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72. Association for Computational Linguistics (2005)
  * [3] Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. Journal of Machine Learning Research 3, 1137–1155 (2003)
  * [4] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. CoRR abs/2005.14165 (2020), https://arxiv.org/abs/2005.14165
  * [5] Callison-Burch, C., Osborne, M., Koehn, P.: Re-evaluating the role of Bleu in machine translation research. In: 11th Conference EACL. Trento, Italy (2006)
  * [6] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. CoRR abs/2005.12872 (2020)
  * [7] Cho, J., Watanabe, S., Hori, T., Baskar, M.K., Inaguma, H., Villalba, J., Dehak, N.: Language model integration based on memory control for sequence to sequence speech recognition. In: ICASSP, Brighton, United Kingdom. pp. 6191–6195 (2019)
  * [8] Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of EMNLP. pp. 1724–1734 (2014)
  * [9] Dauphin, Y.N., Fan, A., Auli, M., Grangier, D.: Language modeling with gated convolutional networks. In: Proceedings of the 34th ICML. pp. 933–941 (2017)
  * [10] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL 2019, Minneapolis. pp. 4171–4186 (2019), https://doi.org/10.18653/v1/n19-1423
  * [11] Fan, A., Lewis, M., Dauphin, Y.: Hierarchical neural story generation. In: Proceedings of the 56th Annual Meeting of ACL 2018, Melbourne. pp. 889–898 (2018)
  * [12] Gülçehre, Ç., Firat, O., Xu, K., Cho, K., Barrault, L., Lin, H., Bougares, F., Schwenk, H., Bengio, Y.: On using monolingual corpora in neural machine translation. CoRR abs/1503.03535 (2015), http://arxiv.org/abs/1503.03535
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on CVPR 2016. pp. 770–778 (2016)
  * [14] Kalimuthu, M., Nunnari, F., Sonntag, D.: A competitive deep neural network approach for the imageclefmed caption 2020 task. In: Working Notes of CLEF 2020, Thessaloniki. CEUR Workshop Proceedings, vol. 2696. CEUR-WS.org (2020)
  * [15] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR, San Diego (2015)
  * [16] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS 2012. Lake Tahoe. pp. 1106–1114 (2012)
  * [17] Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. Assoc. for Comp. Linguistics (2004)
  * [18] Lu, X., Wang, B., Zheng, X., Li, X.: Exploring models and data for remote sensing image caption generation. Trans. on Geoscience and Remote Sensing (2017)
  * [19] Merity, S., Keskar, N.S., Socher, R.: Regularizing and optimizing LSTM language models. In: Proc. of ICLR 2018, Vancouver, Conference Track Proceedings (2018)
  * [20] Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., Khudanpur, S.: Recurrent neural network based language model. In: INTERSPEECH. pp. 1045–1048 (2010)
  * [21] Mogadala, A., Kalimuthu, M., Klakow, D.: Trends in integration of vision and language research: A survey of tasks, datasets, and methods. CoRR abs/1907.09358 (2019), http://arxiv.org/abs/1907.09358, (Accepted at the Journal of AI Research)
  * [22] Mosbach, M., Andriushchenko, M., Klakow, D.: On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. CoRR abs/2006.04884 (2020), https://arxiv.org/abs/2006.04884
  * [23] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: A method for automatic evaluation of machine translation. In: Proc. of ACL. pp. 311–318 (2002)
  * [24] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An imperative style, high-performance deep learning library. In: NeurIPS 2019. pp. 8026–8037 (2019)
  * [25] Pelka, O., Friedrich, C.M., Garcıa Seco de Herrera, A., Müller, H.: Overview of the imageclefmed 2020 concept prediction task: medical image understanding. In: CLEF2020 Working Notes, CEUR Workshop Proceedings. vol. 2696 (2020)
  * [26] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [27] Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T., Saenko, K.: Object hallucination in image captioning. In: Proc. of EMNLP Brussels. pp. 4035–4045 (2018)
  * [28] Sammani, F., Elsayed, M.: Look and modify: Modification networks for image captioning. In: 30th BMVC 2019, Cardiff, UK. p. 75. BMVA Press (2019)
  * [29] Sammani, F., Melas-Kyriazi, L.: Show, edit and tell: A framework for editing image captions. In: Proc. of CVPR 2020, Seattle. pp. 4808–4816. IEEE (2020)
  * [30] Sriram, A., Jun, H., Satheesh, S., Coates, A.: Cold fusion: Training seq2seq models together with language models. In: Proc. Interspeech 2018. pp. 387–391 (2018)
  * [31] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NIPS 2017. pp. 5998–6008 (2017)
  * [32] Vedantam, R., Lawrence Zitnick, C., Parikh, D.: CIDEr: Consensus-based image description evaluation. In: Proc. of CVPR 2015. pp. 4566–4575. IEEE CS (2015)
  * [33] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR. pp. 3156–3164. IEEE Computer Society (2015)
  * [34] Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR abs/1609.08144 (2016), http://arxiv.org/abs/1609.08144
  * [35] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning. pp. 2048–2057 (2015)