# Language Models Can See:
Plugging Visual Controls in Text Generation

Yixuan Su♠, Tian Lan♢, Yahui Liu♣,22footnotemark: 2 Fangyu
Liu♠,22footnotemark: 2
Dani Yogatama♡ Yan Wang♢ Lingpeng Kong▷ Nigel Collier♠,
♠University of Cambridge ♢Tencent AI Lab ♣University of Trento
♡DeepMind ▷The University of Hong Kong
Project Lead<EMAIL_ADDRESS>AuthorsCorresponding Author,
<EMAIL_ADDRESS>
###### Abstract

Generative language models (LMs) such as GPT-2/3 can be prompted to generate
text with remarkable quality. While they are designed for text-prompted
generation, it remains an open question how the generation process could be
guided by modalities beyond text such as images. In this work, we propose a
training-free framework, called MAGIC (iMAge-Guided text generatIon with
CLIP), for plugging in visual controls in the generation process and enabling
LMs to perform multimodal tasks (e.g., image captioning) in a zero-shot
manner. MAGIC is a simple yet efficient plug-and-play framework, which
directly combines an off-the-shelf LM (i.e., GPT-2) and an image-text matching
model (i.e., CLIP) for image-grounded text generation. During decoding, MAGIC
influences the generation of the LM by introducing a CLIP-induced score,
called magic score, which regularizes the generated result to be semantically
related to a given image while being coherent to the previously generated
context. Notably, the proposed decoding scheme does not involve any gradient
update operation, therefore being computationally efficient. On the
challenging task of zero-shot image captioning, MAGIC outperforms the state-
of-the-art method by notable margins with a nearly 27 times decoding speedup.
MAGIC is a flexible framework and is theoretically compatible with any text
generation tasks that incorporate image grounding. In the experiments, we
showcase that it is also capable of performing visually grounded story
generation given both an image and a text prompt. Our code, models and other
related resources are publicly released at https://github.com/yxuansu/MAGIC.

###  1 Introduction

Since the introduction of GPT-2 [60], generative language models (LMs), which
are pre-trained on enormous amount of unstructured text, have produced
unmatched performances on a wide range of NLP tasks [4, 11]. Given a textual
prompt, LMs can continuously generate texts with the next-token prediction
decoding scheme. Although controlling the outputs of LMs have become possible
by inserting textual prompts, it is still unknown how the decoding process
could be guided by information beyond texts, such as images.

Recently, multimodal representation learning of text and images have been
rejuvenated by pre-trained image-text joint embedding models, such as CLIP
[59] and ALIGN [28]. They leverage large-scale nosiy image-text pairs with
weak correspondence for contrastive embedding learning and the learned joint
model achieves impressive zero-shot performance competitive to supervised
models on tasks such as image classification and image-text retrieval.
However, they are still under-explored for image-grounded text
generation.111Note that while such noisy weak image-text pair supervision is
sufficient for learning embeddings, they could not be directly used to train
image captioning model due to the data’s extreme level of noise [78].

How can we combine the best of both the pre-trained LMs and image-text
embedding models for visually grounded text generation? Existing supervised
methods combine multimodal encoders by further training them on human-
annotated paired image-text data [51, 6]. Differently, weakly supervised
approaches [2, 19, 37] rely on pre-trained object detectors to identify visual
concepts and create pseudo image-text pairs. Instead of training on annotated
image-text pairs, they directly train on the pseudo data. However, such
methods are usually limited by the object detectors that are trained with a
fixed set of labels. The closest to our proposal is ZeroCap [78] which is an
unsupervised image captioning method by combining frozen CLIP and GPT-2. One
of the advantages of ZeroCap is it performs _ex post facto_ in the activation
space without re-training or fine-tuning the CLIP and GPT-2 models. However,
ZeroCap relies on gradient update and optimization over the context cache,
which significantly slows down the inference and hindering its use in real-
world scenarios.

In this paper, we propose to solve this challenging task in a completely new
perspective by designing a novel text decoding scheme, called MAGIC (iMAge-
Guided text generatIon with CLIP). During inference, MAGIC does not rely on
any additional training or parameters and utilizes explicit “control knobs” to
select desired outputs following the guidance of both the GPT-2 and CLIP
models. Different from the standard decoding process of GPT-2, we insert a
CLIP-induced term, called magic score, in the next token search to encourage
the predicted result to demonstrate information that is close to a given
image. Our experiments show that such a framework enables zero-shot image
captioning and also visually grounded story generation under a simple plug-
and-play principle.

To verify the qualitative and quantitative performance of the proposed MAGIC
method, we conduct comprehensive experiments on two commonly used benchmarks
(Section section 4): MS-COCO [42] and Flickr30k [58]. To our surprise, MAGIC
achieves state-of-the-art (SOTA) performance across different evaluation
metrics, especially outperforming all unsupervised and weakly supervised
baselines notably. Moreover, since MAGIC involves no gradient update, the
inference speed accelerates upon previous zero-shot image captioning SOTA by
around 27 times. Beyond image captioning, we also test our approach on
visually grounded story generation (Section section 5). In this task, given an
image and a text prompt, MAGIC can generate high-quality stories that
outperform strong baseline methods on both human and automatic evaluations.

In summary, we make the following contributions:

  * •

To the best of our knowledge, we are the first to propose a zero-shot method,
called MAGIC, to utilize explicit “control knobs” to efficiently select
desired outputs following the guidance of both the pre-trained GPT-2 and CLIP
models;

  * •

We empirically show that MAGIC is extremely effective on zero-shot image
captioning, achieving SOTA across different benchmarks;

  * •

We demonstrate that MAGIC could be used in creative ways: it can perform
complex multimodal generation tasks such as visually grounded story generation
and reaches near-human performances on a wide range of evaluation metrics.

###  2 Background

In this section, we briefly introduce previous work related to our research.

####  2.1 Image Captioning

Our work is closely related to the literature of image captioning, where the
goal is to describe images with meaningful and syntactically correct
sentences. Although this topic has been extensively explored in the past few
years, it is still far from being considered as a solved task. Given the
training strategies (e.g., the type of training data), we can roughly classify
the previous methods into two categories: (1) Supervised and (2) Weakly-/Un-
Supervised approaches. The former heavily depends on manually labelled image-
text datasets. In contrast, the latter tries to create pseudo image-text pairs
(i.e., weakly supervised) or even avoid using any paired image-text data
(i.e., unsupervised).

Supervised Approaches. With the development of deep learning, most of the
existing models use one CNN to encode the input image and one RNN to generate
the corresponding sentence describing the image [47, 81]. These models are
trained to maximize the probability of generating the ground-truth captions
conditioned on the input image. After that, the main focus of following
methods is to model the interaction between visual and textual cues via
attention mechanism to get more faithful and richer captions [85, 46, 3, 91,
27]. Meanwhile, some controllable image captioning methods [48, 21, 8, 70, 7]
propose to generate diverse descriptions by feeding different control signals
(e.g., label and text), which require additional annotations for training.
Beyond describing the whole image scene, dense captioning methods [30, 5, 35,
88, 89] aim to describe the visual objects in a sub-region of the input image.
Recently, vision-language pre-training methods [94, 40, 51, 26], benefiting
from the rich visual-textual representation of pre-trained models on large-
scale datasets, are tendencies for vision-language generation by re-training
or fine-tuning the model parameters on downstream tasks. Although these
methods have achieved impressive results, a certain amount of paired image-
text data is indispensable during training.

Weakly-/Un-Supervised Approaches. Till now, there has been several attempts to
reduce the reliance on paired image-text data for the training of image
captioning model. In weakly-supervised approaches, employing pseudo-captions,
i.e., sentences that contain the object labels detected from the given images,
has been a common choice [2, 19, 37]. However, a weakly supervised cross-modal
alignment between image and text is needed. Besides, pseudo-captions tend to
contain irrelevant words for the given images [25]. Therefore, it requires
carefully designed constraints or learning schema to alleviate the noise. What
is more, such methods require a pre-trained object detector bounded by a fixed
set of labels to provide visual concepts. They are thus ineffective for any
out-of-domain concepts and scenes.

How can we get rid of creating pseudo-captions and perform image captioning in
a truly unsupervised manner? Recently, CLIP [59] has emerged as a successful
vision-language alignment model by training on 400M noisy web-collected image-
sentence pairs. It has shown impressive zero-shot capabilities on various
vision-language tasks and can open new avenues for answering the former
question. ZeroCap [78] is the most related to our work. It is built on a pre-
trained CLIP model together with the GPT-2 language model [60]. Different from
previous work, ZeroCap is truly zero-shot, where the optimization is performed
“ex post facto” in the activation space without re-training or fine-tuning the
model parameters. In ZeroCap, the whole context cache (i.e., all the $K$ and
$V$ in the self-attention modules [79, 17]) is updated with the guidance of
CLIP and GPT-2 for every prediction step. As a result, the computational
overhead of such optimization steps will increase drastically as the size of
language model goes up. One key difference of our proposal with respect to
ZeroCap is that MAGIC utilizes explicit “control knobs” to select desired
outputs corresponding to the given image. Since our procedure does not involve
any gradient updating or optimization, it significantly speeds up the decoding
process by around 27 times (Section section 4.1).

####  2.2 Plug and Play Generative Models

Lagre-scale pre-trained generative models have revolutionized the field of
natural language processing [60, 4] and computer vision [59, 63, 62, 33, 34,
32] in the past few years. Various previous work [54, 53, 14, 69] have
revealed that there are rich meaningful semantics in the features learned by
such models. This shows a promising pathway to synthesize the desired outputs
by interpreting the existing generative models in a “plug and play” manner. We
can then directly enjoy the powerful capabilities of these off-the-shelf big
models (without any re-training or fine-tuning) and focus on the lightweight
task-specific optimization.

For instance, in the image generation field, DGN-AM [54] can generate images
conditioned on a class by finding a hidden code that clearly activates a
neuron in another classifier. Then, PPGN [53] improves the diversity and
quality of the synthesized images by incorporating approximate Metropolis-
adjusted Langevin (MALA) algorithm [65, 64]. Shen _et al._ [69] propose to
directly travel in the latent space of pre-trained unconditional GANs to
manipulate the attributes of the input image. Patashnik _et al._ [56] use CLIP
to connect the text prompt and images to search the latent codes of StyleGAN
by gradient descent optimization, which finally results in the manipulation of
the visual attributes in the input image. Similarly, in the text generation
field, PPLM [14] extends the previous PPGN to text generation tasks (i.e.,
editing topic and sentiment), where the image generative models is replaced
with a GPT-2 language model. Most recently, ZeroCap [78] shows image
captioning task can be tackled by directly combining the existing CLIP and
GPT-2 models. In general, most of these mentioned “plug and play” methods
require iteratively shifting the hidden code (or latent code, or context
cache) with gradient descent optimization.

Different from previous work, our proposed approach extends the “plug and
play” paradigm by optimizing the decoding strategy of generative LMs. Since
MAGIC does not involve any gradient update in the inference, it is
computationally efficient (e.g., $\sim$27 times faster than ZeroCap). Notably,
although GPT-2 could generate synthetic text samples of unprecedented quality,
it may not be natural on some task-specific text [51, 68, 90]. Following this
observation, we continue fine-tuning the GPT-2 model on the task-specific text
corpus in an unsupervised manner to adapt it to the textual domain of the end
task [37, 25]. The computational consumption of such adaptation is negligible
(e.g., less than 2 hours with 1 NVIDIA 1080Ti GPU on MS-COCO). During
decoding, the fine-tuned GPT-2 and CLIP models are fixed.

###  3 Methodology

####  3.1 Unsupervised Language Modelling

Following previous studies [37, 25], we first learn an unsupervised language
model on the text corpus of the end task to adapt to its textual domain.
Typically, given a variable-length text sequence $\boldsymbol{x}$, the maximum
likelihood estimation (MLE) objective is used to train the language model
$\theta$ as

$\mathcal{L}_{\textup{MLE}}=-\frac{1}{|\boldsymbol{x}|}\sum_{i=1}^{|\boldsymbol{x}|}\log
p_{\theta}(x_{i}|\boldsymbol{x}_{<i}).$ (1)

Recently, Su _et al._ [72] propose to incorporate contrastive objective into
the training of the language model to calibrate the model’s representation
space and obtain better language model perplexity. Given the text sequence
$\boldsymbol{x}$, the contratsive objective $\mathcal{L}_{\textup{CL}}$ is
defined as

$\mathcal{L}_{\textup{CL}}=\frac{1}{|\boldsymbol{x}|\times(|\boldsymbol{x}|-1)}\sum_{i=1}^{|\boldsymbol{x}|}\sum_{j=1,j\neq
i}^{|\boldsymbol{x}|}\max\\{0,\rho-s(h_{x_{i}},h_{x_{i}})+s(h_{x_{i}},h_{x_{j}})\\},$
(2)

where $\rho$ is a pre-defined margin that regularizes the distribution of the
model’s representation space. The $h_{x_{i}}$ is the representation of token
$x_{i}$ and the similarity function $s$ computes the cosine similarity between
token representations as
$s(h_{x_{i}},h_{x_{j}})=h_{x_{i}}^{\top}h_{x_{j}}/(\|h_{x_{i}}\|\cdot\|h_{x_{j}}\|)$.

The overall learning objective $\mathcal{L}$ of the language model is then
defined as

$\mathcal{L}=\mathcal{L}_{\textup{MLE}}+\mathcal{L}_{\textup{CL}}.$ (3)

####  3.2 MAGIC Search

We propose a new decoding scheme, MAGIC Search, which aims to steer the
decoding process of the language model towards a desired visual direction.
Formally, given a text prefix $\boldsymbol{x}_{<t}$ and an image
$\mathcal{I}$, the selection of the output token $x_{t}$ at time step $t$
follows

$\displaystyle\begin{split}x_{t}=\operatorname*{arg\,max}_{v\in
V^{(k)}}\bigg{\\{}(1-\alpha)\times&\underbrace{p_{\theta}(v|\boldsymbol{x}_{<t})}_{\textup{model
confidence}}-\\\ \alpha\times\underbrace{(\max\\{s(h_{v},h_{x_{j}}):1\leq
j\leq t-1\\})}_{\textup{degeneration
penalty}}&\>+\>\beta\times\underbrace{f(v|\mathcal{I},\boldsymbol{x}_{<t},V^{(k)})}_{\textup{magic
score}}\bigg{\\}},\\\ \end{split}$ (4)

where $V^{(k)}$ is the set of top-$k$ predictions from the model’s probability
distribution $p_{\theta}(\cdot|\boldsymbol{x}_{<t})$ and $s$ is described in
Section section 3.1. $h_{v}$ is the representation of the candidate token $v$
which is computed by the model given the concatenation of
$\boldsymbol{x}_{<t}$ and $v$. Inspired by Su _et al._ [72], we incorporate
the model confidence and degeneration penalty into Eq. (4) to let the model
decode the likely output while avoiding the model degeneration problem.

Meanwhile, we introduce a novel scoring criterion, magic score, to plug in
visual controls into the decoding process. Given the candidate $v$, the prefix
$\boldsymbol{x}_{<t}$, and the image $\mathcal{I}$, the magic score is defined
as the distribution of image-text similarity over the candidate set $V^{(k)}$.
We build our image-text similarity measurement with a pre-trained CLIP model
and the magic score is then defined as

$f(v|\mathcal{I},\boldsymbol{x}_{<t},V^{(k)})=\frac{e^{\textup{CLIP}(\mathcal{I},[\boldsymbol{x}_{<t}:v])}}{\sum_{z\in
V^{(k)}}e^{\textup{CLIP}(\mathcal{I},[\boldsymbol{x}_{<t}:z])}}=\frac{e^{h_{\mathcal{I}}^{\top}h_{[\boldsymbol{x}_{<t}:v]}}}{\sum_{z\in
V^{(k)}}e^{h_{\mathcal{I}}^{\top}h_{[\boldsymbol{x}_{<t}:z]}}},$ (5)

where $h_{\mathcal{I}}$ is the image embedding of $\mathcal{I}$ produced by
the CLIP image encoder. The $h_{[\boldsymbol{x}_{<t}:v]}$ is the text
embedding of the sequence $[\boldsymbol{x}_{<t}:v]$ produced by the CLIP text
encoder and $[:]$ denotes the concatenation operation. Intuitively, the magic
score encourages the language model to generate text that is semantically
related to the image content and the strength of the visual control is
regulated by the hyper-parameter $\beta$ in Eq. (4). When $\beta=0$, the
visual control is disabled and MAGIC Search degenerates to the vanilla
contrastive search [72].

It should be emphasized that MAGIC Search allows us to directly plug visual
controls into the decoding process of the language model, without the need of
extra supervised training [14] or gradient update on additional features [14,
78]. This property makes our method much more computationally efficient than
previous approaches as demonstrated in our experiments (Section section 4.1).

###  4 Zero-Shot Image Captioning

We first evaluate our approach on the task of zero-shot image captioning.

Evaluation Benchmarks. We conduct experiments on two widely used benchmarks:
MS-COCO [42] and Flickr30k [58]. For both datasets, we set up the training,
validation, and test splits according to the protocols provided by Karpathy
_et al._ [31].

Implementation Details. As described in Section section 3.1, for each
benchmark, we fine-tune the GPT-2 model on the training text corpus for 3
epochs and the contrastive loss margin $\rho$ in Eq. (3) is set as 0.5. We
optimize the model with the Adam optimizer [36] and a learning rate of 2e-5.
Notably, this fine-tuning procedure is computationally negligible, i.e., less
than 2 hours with 1 NVIDIA 1080Ti GPU. During decoding, the generation of the
language model starts with a special start-of-sequence (i.e., [sos]) token.
For MS-COCO, we set the $k$, $\alpha$, and $\beta$ in MAGIC Search (i.e., Eq.
(4)) as 45, 0.1, and 2.0 based on the model’s performance on the validation
set. As for Flickr30k, these values are set as 25, 0.1, and 2.0,
respectively.222In Appendix B, we provide detailed ablation studies on the
effect of different hyper-parameter setups.

Baselines. We include several zero-shot methods as our baselines. (1) We
compare the generated results of the language model by starting from the
start-of-sequence (i.e., [sos]) token with different decoding methods,
including top-$k$ sampling [18] with $k=40$ and nucleus sampling [24] with
$p=0.95$. Moreover, we include contrastive search [72] using the same $k$ and
$\alpha$ as in MAGIC Search to see the direct effect of the proposed magic
score (Eq. (4)).333Note that, our proposed MAGIC Search is equivalent to the
contrastive search when $\beta$ in Eq. (4) equals to 0. Note that, these
methods do not take into account the image input, therefore can be used to
assess the performance lower-bound of the language model. (2) We also compare
with a CLIP-based method, called CLIPRe. Given an image, it retrieves the most
related caption from the training text corpus based on the image-text
similarity as measured by CLIP. (3) Lastly, we compare with the current state-
of-the-art approach, ZeroCap [78], which performs CLIP-guided gradient update
on the language model features during the decoding process. For a fair
comparison, we use the same language model for ZeroCap as in our approach.

Evaluation Metrics. Following the common practice in the literature, we
perform evaluation using BLEU-1 (B@1), BLEU-4 (B@4) [55], METER (M) [15],
ROUGE-L (R-L) [41], CIDEr [80], and SPICE [1]. In addition, we compare the
relative decoding speed of our approach against other generation-based
baselines. Here, the decoding speed is measured from the average inference
time per image instance.444The decoding speed of different methods are
measured on the same hardware platform with a batch size of 1.

####  4.1 Results

Table 1 shows the results on zero-shot image captioning. For a comprehensive
comparison, we also include the results of several representative (1)
supervised methods: BUTD [3], GVD [93], UniVLP [94], ClipCap [51], Oscar [40],
and LEMON [26]; and (2) weakly supervised methods: UIC [19], IC-SME [37],
S2S-SS and S2S-GCC [25].

From the results of Top-$k$, Nucleus, and Contrastive, we see that solely
using the unsupervised language model without conditioning on image inputs can
hardly generate meaningful captions.555For stochastic sampling methods (i.e.,
Top-$k$ and Nucleus), we report the results averaged over 3 runs with
different random seeds. We refer to Appendix C for more details on the
numerical results. On the other hand, the results of CLIPRe show that the
ability of measuring image-text similarity enables CLIP to retrieve captions
that better correlate with the test image from the training text corpus.
However, the performance of CLIPRe still lags behind the current SOTA method,
ZeroCap, by a large margin due to the data discrepancy between the training
and test sets. Lastly, we observe that, on both benchmarks, MAGIC achieves the
best performance on 11 out of 13 metrics, demonstrating the clear advantages
of our proposed approach. Notably, while outperforming ZeroCap on 12 out of 13
metrics, MAGIC achieves a nearly 27$\times$ decoding speedup. This is because,
during the decoding process, MAGIC does not involve any computationally
inefficient operations like gradient updates [14, 78], which further validates
the practical usage of our approach.

Model | MS-COCO | Flickr30k | Speed
---|---|---|---
B@1 | B@4 | M | R-L | CIDEr | SPICE | B@1 | B@4 | M | R-L | CIDEr | SPICE
| Supervised Approach |
BUTD | 77.2 | 36.2 | 27.0 | 56.4 | 113.5 | 20.3 | - | 27.3 | 21.7 | - | 56.6 | 16.0 | -
GVD | - | - | - | - | - | - | 66.9 | 27.3 | 22.5 | - | 62.3 | 16.5 | -
UniVLP | - | 36.5 | 28.4 | - | 116.9 | 21.2 | - | 30.1 | 23.0 | - | 67.4 | 17.0 | -
ClipCap | - | 33.5 | 27.5 | - | 113.1 | 21.1 | - | - | - | - | - | - | -
Oscar | - | 36.5 | 30.3 | - | 123.7 | 23.1 | - | - | - | - | - | - | -
LEMON | - | 40.3 | 30.2 | - | 133.3 | 23.3 | - | - | - | - | - | - | -
| Weakly Supervised Approach |
UIC | 41.0 | 5.6 | 12.4 | 28.7 | 28.6 | 8.1 | - | - | - | - | - | - | -
IC-SME | - | 6.5 | 12.9 | 35.1 | 22.7 | - | - | 7.9 | 13.0 | 32.8 | 9.9 | - | -
S2S-SS | 49.5 | 6.3 | 14.0 | 34.5 | 31.9 | 8.6 | - | - | - | - | - | - | -
S2S-GCC | 50.4 | 7.6 | 13.5 | 37.3 | 31.8 | 8.4 | - | - | - | - | - | - | -
| Unsupervised Approach |
Top-$k$ | 33.6 | 2.4 | 8.3 | 25.6 | 3.8 | 1.7 | 34.0 | 2.9 | 9.0 | 24.4 | 3.3 | 2.7 |  69.9$\times$
Nucleus | 32.6 | 2.3 | 7.8 | 24.8 | 3.1 | 1.4 | 32.6 | 2.4 | 8.1 | 23.4 | 2.5 | 2.4 | 72.5$\times$
Contrastive | 39.5 | 3.0 | 10.8 | 30.8 | 7.7 | 2.9 | 37.6 | 4.3 | 9.8 | 25.7 | 8.9 | 4.6 |  50.4$\times$
CLIPRe | 39.5 | 4.9 | 11.4 | 29.0 | 13.6 | 5.3 | 38.5 | 5.2 | 11.6 | 27.6 | 10.0 | 5.7 | -
ZeroCap | 49.8 | 7.0 | 15.4 | 31.8 | 34.5 | 9.2 | 44.7 | 5.4 | 11.8 | 27.3 | 16.8 | 6.2 |  1.0$\times$
MAGIC | 56.8 | 12.9 | 17.4 | 39.9 | 49.3 | 11.3 | 44.5 | 6.4 | 13.1 | 31.6 | 20.4 | 7.1 |  26.6$\times$
|  |  |  |  |  |  |  |  |  |  |  |  |

Table 1: Image Captioning Results on MS-COCO and Flickr30k.

Model |  MS-COCO $\Longrightarrow$ Flickr30k |  Flickr30k $\Longrightarrow$ MS-COCO
---|---|---
B@1 | B@4 | M | R-L | CIDEr | SPICE | B@1 | B@4 | M | R-L | CIDEr | SPICE
Top-$k$ | 34.9 | 2.4 | 7.5 | 24.2 | 2.3 | 1.7 | 30.0 | 1.8 | 8.5 | 23.6 | 2.5 | 1.7
Nucleus | 33.4 | 1.7 | 7.0 | 23.3 | 1.8 | 1.3 | 29.1 | 1.6 | 8.0 | 22.9 | 2.1 | 1.6
Contrastive | 40.3 | 5.3 | 10.7 | 30.5 | 5.1 | 3.4 | 33.8 | 3.2 | 10.2 | 25.5 | 4.2 | 3.7
CLIPRe | 38.7 | 4.4 | 9.6 | 27.2 | 5.9 | 4.2 | 31.1 | 3.0 | 9.9 | 22.8 | 8.5 | 3.9
MAGIC | 46.4 | 6.2 | 12.2 | 31.3 | 17.5 | 5.9 | 41.4 | 5.2 | 12.5 | 30.7 | 18.3 | 5.7
|  |  |  |  |  |  |  |  |  |  |  |

Table 2: Cross-Domain Evaluation. X $\Longrightarrow$ Y means source domain
$\Longrightarrow$ target domain.

####  4.2 Cross-Domain Experiment

To test the generalization ability of our approach, we conduct a cross-domain
experiment. Specifically, we apply the unsupervised language model fine-tuned
on the training text corpus of the source domain (e.g., MS-COCO) to perform
inference on the test set of the target domain (e.g., Flickr30k). We compare
MAGIC with several zero-shot methods, including Top-$k$, Nucleus, Contrastive,
and CLIPRe.666Due to its extremely high computational overhead, we do not
include ZeroCap in this experiment. For CLIPRe, given a test image from the
target domain, it retrieves the most related caption from the training text
corpus of the source domain.

Table 2 shows the results on cross-domain evaluation, where we observe
performance drops in all methods as compared with the in-domain evaluation
results shown in Table 1.777For stochastic sampling methods (i.e., Top-$k$ and
Nucleus), we report the results averaged over 3 runs with different random
seeds. We refer to Appendix C for more detailed numerical results.
Nonetheless, MAGIC still performs the best among all compared methods,
demonstrating its clear advantages in terms of robustness and generalization
ability.

Figure 1: Examples of zero-shot image captioning. (Best viewed by zooming in.)

####  4.3 Qualitative Evaluation

Figure 1 shows visual comparisons between our approach and other two strong
zero-shot baselines along with the reference caption.888More examples of zero-
shot image captioning are provided in Appendix D. The results demonstrate that
MAGIC can generate fluent captions while being more effective at grounding on
the given image. For example, in Figure 1(a), the result of CLIPRe only
contains the object “building” that is partially related to the image. As for
ZeroCap, it erroneously elaborates the object “school bus” which is not
displayed in the image. On the other hand, MAGIC correctly describes the
“street sign” object in the image as well as the building in the background.
Next, we take Figure 1(d) as an example, in which the result of CLIPRe is
clearly irrelevant to the image. As for ZeroCap, while it produces objects
like “boatboard” and “small boat” that are related to the image, the generated
result is not fluent and ungrammatical. In contrast, MAGIC is able to describe
the correct objects such as “yellow boat” and “beach” as well as their
positional relationship (i.e., lined up on) while maintaining the fluency and
grammaticality of the generated text.

###  5 Story Generation

To verify the versatility and extensibility of MAGIC, we test it on a popular
text generation task, i.e., story generation. In this task, given a story
title (i.e., text prompt), the language model is asked to generate an
interesting and coherent story that is related to the story title.

Evaluation Benchmark. We evaluate our approach on the widely used ROCStories
[52] dataset. In this dataset, each story title is accompanied with a five-
sentence commonsense story written by human. We create the training,
validation, and test sets following the official split.

Model and Baselines. We use a publicly available GPT-based language model [72]
which is fine-tuned on the ROCStories
benchmark.999https://huggingface.co/cambridgeltl/simctg_rocstories As MAGIC
Search is a language model decoding scheme, we compare it with a range of
strong text decoding methods, including (1) Greedy search; (2) Beam search
with beam width of $10$; (3) Top-$k$ sampling [18] with $k=40$; (4) Nucleus
sampling [24] with $p=0.95$; (5) Typical sampling [49] with $\tau=0.2$; and
(6) Contrastive search [72] with $k=5$ and $\alpha=0.6$. The hyperparameters
of different methods are selected based on their optimal MAUVE [57] (detailed
in Section section 5.1) performance on the validation set.

Implementation Details of MAGIC. To perform MAGIC Search, given the story
title, we first retrieve the image (from an image index) that is most related
to the story title as measured by CLIP. We construct the image index with the
public ConceptualCaptions [67] dataset that contains over 3.3M images from the
web. In practice, we pre-compute the image representations with CLIP and build
the image index with FAISS [29], therefore supporting a fast “story title-
image” retrieval with sub-linear time complexity. Then, by visually grounding
on the retrieved image, we generate the story from the story title using MAGIC
Search ($k=5$, $\alpha=0.6$, and $\beta=0.15$).101010The hyper-parameters are
selected based on the model’s optimal MAUVE performance on the validation set.

Method | Automatic Evaluation | Human Evaluation
---|---|---
rep-2$\downarrow$ |  rep-3$\downarrow$ |  rep-4$\downarrow$ |  div.$\uparrow$ |  coh.$\uparrow$ |  MAUVE$\uparrow$ |  CLIPScore$\uparrow$ |  coh.$\uparrow$ |  flu.$\uparrow$ |  inform.$\uparrow$ |  si-rel.$\uparrow$
Agreement | - | - | - | - | - | - | - | 0.68 | 0.57 | 0.66 | 0.73
Greedy | 22.27 | 15.42 | 12.36 | 0.58 | 0.473 | 0.53 | 0.23 | 2.67 | 3.20 | 3.10 | 2.03
Beam | 26.76 | 21.79 | 18.85 | 0.47 | 0.478 | 0.46 | 0.25 | 2.71 | 3.23 | 3.15 | 2.05
Top-$k$ | 3.38 | 0.76 | 0.23 | 0.95 | 0.458 | 0.86 | 0.21 | 2.52 | 3.69 | 3.62 | 1.96
Nucleus | 2.92 | 0.60 | 0.18 | 0.96 | 0.452 | 0.88 | 0.21 | 2.48 | 3.68 | 3.71 | 1.92
Typical | 2.52 | 0.46 | 0.12 | 0.97 | 0.450 | 0.84 | 0.19 | 2.32 | 3.70 | 3.76 | 1.75
Contrastive | 2.49 | 0.38 | 0.09 | 0.97 | 0.488 | 0.89 | 0.28 | 2.86 | 3.72 | 3.76 | 2.07
MAGIC | 2.51 | 0.38 | 0.09 | 0.97 | 0.514 | 0.91 | 0.36 | 3.20★ | 3.76 | 3.85 | 2.40★
Human | 2.21 | 0.37 | 0.09 | 0.97 | 0.542 | 1.00 | 0.40 | 3.77 | 4.11 | 4.22 | 2.59
|  |  |  |  |  |  |  |  |  |  |

Table 3: Evaluation results on story generation. $\uparrow$ means higher is
better and $\downarrow$ means lower is better. The best result is bold and the
second best is underlined. In human evaluation, ${\bigstar}$ results
significantly outperforms the results of other compared methods (Sign Test
with p-value ¡ 0.05).

####  5.1 Automatic Evaluation

Following previous studies [83, 49, 72], we first evaluate the quality of the
generated results from different methods using automatic evaluation metrics,
including (1) $n$-gram repetition (rep-$n$); (2) generation diversity (div.);
(3) semantic coherence (coh.) between the generated story and the story title;
and (4) MAUVE [57] score that measures the token distribution closeness
between the generated text and the human-written text. In addition, to verify
that MAGIC is able to generate stories that are semantically related to the
given images, we employ CLIPScore [23] to measure the semantic similarity
between the generated story and the image retrieved by the story title.

Table 3 shows the automatic results, from which we observe that MAGIC performs
the best on most of the evaluation metrics.111111For stochastic methods (i.e.,
Top-$k$, Nucleus, and Typical sampling), we report the numbers averaged over 3
runs with different random seeds. We refer to Appendix E for more details. The
results of rep-$n$, diversity, and MAUVE score demonstrate that MAGIC
generates the most diverse stories while being closest to human-written
stories in terms of token distribution [57]. Moreover, on the coherence (coh.)
metric, MAGIC notably outperforms other baselines. We conjecture that the
image retrieved by the story title contains rich visual concepts and features,
therefore providing more grounding information. As a result, by leveraging
these visual knowledge, MAGIC can generate stories are more semantically
coherent to the story titles. Lastly, on the CLIPScore metric, MAGIC surpasses
other methods by large margins, suggesting it generates stories that are more
related to the images. In conclusion, the generated text of MAGIC is
effectively guided by both the text prompt (i.e., story title) as well as the
image, while other methods can only leverage the information from the text
prompt.

####  5.2 Human Evaluation

We also conduct a human evaluation with the help of graders proficient in
English from a third-party grading platform. We sample 200 instances from the
test set. All generated results, plus the reference, are randomly shuffled and
evaluated by five graders, resulting in 8,000 annotated samples in total. The
evaluation follows a 5-point Likert scale (1, 2, 3, 4, or 5) for each of the
following features:121212We refer to Appendix F for the detailed human
evaluation guidelines.

  * •

Coherence (coh.): Whether the generated story is semantically consistent with
the title.

  * •

Fluency (flu.): Whether the generated story is fluent and easy to understand.

  * •

Informativeness (inform.): Whether the generated story is diverse and
interesting.

  * •

Story-Image Relevance (si-rel.): Whether the generated story is related to the
image that is retrieved by the story title.

Table 3 presents the human evaluation results, with the first row showing
strong inter-annotator agreements as measured by Fleiss′ kappa coefficient
[20]. Firstly, on the fluency (flu.) and informativeness (inform.) metrics,
MAGIC performs better than other methods. This indicates that the introduction
of visual guidance helps the model to generate more interesting content while
maintaining the grammaticality and fluency of the generated story. Moreover,
on the coherence metric, the performance gain of MAGIC over other baselines is
significant (Sign Test with p-value ¡ 0.05), showing it better maintains the
consistency between the generated story and the story title. This conclusion
is also validated by the results of coherence score in the automatic
evaluation (Section section 5.1). Lastly, on the story-image relevance (si-
rel.) metric, MAGIC also outperforms other methods significantly,
demonstrating its ability in generating text by visually grounding on the
given image.

Figure 2: Examples of story generation. MAGIC can generate text (highlighted
in blue) that is related to the visual concepts displayed in the image. (Best
viewed in color and by zooming in.)

####  5.3 Qualitative Evaluation

In Figure 2, we compare our approach with the strongest baseline (i.e.,
contrastive search), where the image retrieved by the story title is shown on
the left-hand side.131313We refer to Appendix G for more examples of story
generation. We see that MAGIC can generate text (highlighted in blue)
conditioned on the visual concepts of the image. In the first example, MAGIC
elaborates details of three types of ice cream. Such details are more
interesting as well as more related to the story title (i.e., ice cream
tasting) as compared with the story generated by contrastive search. In the
second example, the result of contrastive search is clearly off-the-topic. In
contrast, by conditioning on the image, the story generated by MAGIC stays on
the topic and describes intriguing details about a sand volleyball game: (i)
the game was hosted on the beach; (ii) it lasted for two hours; and (iii) the
friend won the game. In summary, by leveraging visual guidance from the image,
MAGIC can generate semantically coherent story with interesting content.

###  6 Conclusion and Future Work

In this work, we present MAGIC, a novel decoding scheme that plugs visual
controls into the generation of a language model. MAGIC is a training-free
framework that enables the LM to address challenging multimodal tasks in a
zero-shot manner without sacrificing the decoding speed. To verify the
versatility and extensibility of MAGIC, we comprehensively evaluate our
approach on two image-grounded text generation tasks: (i) image captioning and
(ii) visually grounded story generation. Experimental results demonstrate that
our approach notably outperforms previous state-of-the-art methods in both
automatic and human evaluations.

Future Work. While our focus in this study is zero-shot image grounded text
generation using a language model, we would like to note that MAGIC Search is
a model architecture agnostic decoding scheme. In other words, it can
naturally fit into any existing multimodal generative model which takes both
the image and text as input. However, it is out of the scope of this paper and
we will leave it to future work.

Moreover, in theory, MAGIC is a generic framework that can be extended to
modalities beyond text and image. Controls in any form, of any modalities, can
be plugged into the language model as long as a certain similarity metric can
be found to measure the relevance between the control and the generated text.
In future work, we would like to explore the possibility of adapting MAGIC to
other modalities beyond images (e.g., audios and videos) therefore enabling
the language model to generate text grounded on multimodal intelligence.

### Acknowledgments

The first author would like to thank Jialu Xu and Qianchu Liu for the
insightful discussions and support.

### References

  * [1] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In European Conference on Computer Vision (ECCV), 2016.
  * [2] Peter Anderson, Stephen Gould, and Mark Johnson.  Partially-supervised image captioning.  Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * [3] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [5] Moitreya Chatterjee and Alexander G Schwing.  Diverse and coherent paragraph generation from images.  In European Conference on Computer Vision (ECCV), 2018.
  * [6] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny.  Visualgpt: Data-efficient adaptation of pretrained language models for image captioning.  arXiv preprint arXiv:2102.10407, 2021.
  * [7] Long Chen, Zhihong Jiang, Jun Xiao, and Wei Liu.  Human-like controllable image captioning with verb-specific semantic roles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [8] Tianlang Chen, Zhongping Zhang, Quanzeng You, Chen Fang, Zhaowen Wang, Hailin Jin, and Jiebo Luo.  “factual”or“emotional”: Stylized image captioning with adaptive learning and attention.  In European Conference on Computer Vision (ECCV), 2018.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning (ICML), 2020.
  * [10] Sumit Chopra, Raia Hadsell, and Yann LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2005.
  * [11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [12] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In 8th International Conference on Learning Representations, (ICLR), 2020.
  * [13] Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J Passonneau, and Rui Zhang.  Container: Few-shot named entity recognition via contrastive learning.  arXiv preprint arXiv:2109.07589, 2021.
  * [14] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.  Plug and play language models: A simple approach to controlled text generation.  In International Conference on Learning Representations (ICLR), 2020\.
  * [15] Michael Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the Workshop on Statistical Machine Translation, 2014.
  * [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2019.
  * [17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021\.
  * [18] Angela Fan, Mike Lewis, and Yann N. Dauphin.  Hierarchical neural story generation.  In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
  * [19] Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo.  Unsupervised image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [20] J.L. Fleiss et al.  Measuring nominal scale agreement among many raters.  Psychological Bulletin, 76(5):378–382, 1971.
  * [21] Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng.  Stylenet: Generating attractive visual captions with styles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [22] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  SimCSE: Simple contrastive learning of sentence embeddings.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  * [23] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  arXiv preprint arXiv:2104.08718, 2021.
  * [24] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  In International Conference on Learning Representations (ICLR), 2020\.
  * [25] Ukyo Honda, Y. Ushiku, Atsushi Hashimoto, Taro Watanabe, and Yuji Matsumoto.  Removing word-level spurious alignment between images and pseudo-captions in unsupervised image captioning.  In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021.
  * [26] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Scaling up vision-language pre-training for image captioning.  arXiv preprint arXiv:2111.12233, 2021.
  * [27] Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu.  Unifying multimodal transformer for bi-directional image and text generation.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 1138–1147, 2021.
  * [28] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning (ICML), 2021.
  * [29] Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with gpus.  IEEE Transactions on Big Data, 7(3):535–547, 2019.
  * [30] Justin Johnson, Andrej Karpathy, and Li Fei-Fei.  Densecap: Fully convolutional localization networks for dense captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [31] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [32] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [33] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [34] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [35] Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon.  Dense relational captioning: Triple-stream networks for relationship-based captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [36] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, International Conference on Learning Representations (ICLR), 2015.
  * [37] Iro Laina, C. Rupprecht, and Nassir Navab.  Towards unsupervised image captioning with shared multimodal embeddings.  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [38] Tian Lan, Deng Cai, Yan Wang, Yixuan Su, Xian-Ling Mao, and Heyan Huang.  Exploring dense retrieval for dialogue response selection.  arXiv preprint arXiv:2110.06612, 2021.
  * [39] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
  * [40] Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision (ECCV), 2020.
  * [41] Chin-Yew Lin and Franz Josef Och.  Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2004.
  * [42] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European Conference on Computer Vision (ECCV), 2014.
  * [43] Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier.  Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  * [44] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * [45] Yixin Liu and Pengfei Liu.  SimCLS: A simple framework for contrastive learning of abstractive summarization.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL), 2021.
  * [46] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [47] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L Yuille.  Explain images with multimodal recurrent neural networks.  arXiv preprint arXiv:1410.1090, 2014.
  * [48] Alexander Mathews, Lexing Xie, and Xuming He.  Senticap: Generating image descriptions with sentiments.  In Proceedings of the AAAI conference on artificial intelligence (AAAI), 2016.
  * [49] Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell.  Typical decoding for natural language generation.  arXiv preprint arXiv:2202.00666, 2022.
  * [50] Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, and Nigel Collier.  Rewire-then-probe: A contrastive recipe for probing biomedical knowledge of pre-trained language models.  CoRR, abs/2110.08173, 2021.
  * [51] Ron Mokady, Amir Hertz, and Amit H. Bermano.  Clipcap: Clip prefix for image captioning.  ArXiv, abs/2111.09734, 2021.
  * [52] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2016.
  * [53] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.  Plug & play generative networks: Conditional iterative generation of images in latent space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR, 2017.
  * [54] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  Advances in Neural Information Processing Systems (NeurIPS), 2016\.
  * [55] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2002.
  * [56] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [57] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [58] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2015.
  * [59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning (ICML), 2021.
  * [60] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [61] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21:1–67, 2020.
  * [62] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [63] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning (ICLR), 2021.
  * [64] Gareth O Roberts and Jeffrey S Rosenthal.  Optimal scaling of discrete approximations to langevin diffusions.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(1):255–268, 1998.
  * [65] Gareth O Roberts and Richard L Tweedie.  Exponential convergence of langevin distributions and their discrete approximations.  Bernoulli, pages 341–363, 1996.
  * [66] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine.  Time-contrastive networks: Self-supervised learning from video.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
  * [67] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
  * [68] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer.  How much can clip benefit vision-and-language tasks?  arXiv preprint arXiv:2107.06383, 2021.
  * [69] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [70] Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston.  Engaging image captioning via personality.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [71] Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier.  Non-autoregressive text generation with pre-trained language models.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL), 2021.
  * [72] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier.  A contrastive framework for neural text generation.  CoRR, abs/2202.06417, 2022.
  * [73] Yixuan Su, Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Collier.  Tacl: Improving BERT pre-training with token-aware contrastive learning.  CoRR, abs/2111.04198, 2021.
  * [74] Yixuan Su, Zaiqiao Meng, Simon Baker, and Nigel Collier.  Few-shot table-to-text generation with prototype memory.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP, 2021.
  * [75] Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang.  Multi-task pre-training for plug-and-play task-oriented dialogue system.  CoRR, abs/2109.14739, 2021.
  * [76] Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier.  Plan-then-generate: Controlled data-to-text generation via planning.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP, 2021.
  * [77] Yixuan Su, Yan Wang, Deng Cai, Simon Baker, Anna Korhonen, and Nigel Collier.  PROTOTYPE-TO-STYLE: dialogue generation with style-aware editing on retrieval memory.  IEEE ACM Trans. Audio Speech Lang. Process., 29:2152–2161, 2021\.
  * [78] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf.  Zero-shot image-to-text generation for visual-semantic arithmetic.  arXiv preprint arXiv:2111.14447, 2021.
  * [79] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [80] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [81] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [82] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
  * [83] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston.  Neural text generation with unlikelihood training.  arXiv preprint arXiv:1908.04319, 2019.
  * [84] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  * [85] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International Conference on Machine Learning (ICML), 2015.
  * [86] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao.  Taco: Token-aware cascade contrastive learning for video-text alignment.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [87] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [88] Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, and Jing Shao.  Context and attribute grounded dense captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [89] Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan.  Dense regression network for video grounding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [90] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [91] Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji.  Rstnet: Captioning with adaptive attention on visual and non-visual words.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [92] Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng.  Dialoglm: Pre-trained model for long dialogue understanding and summarization.  CoRR, abs/2109.02492, 2021.
  * [93] Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach.  Grounded video description.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [94] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.

## Appendix

###  Appendix A Related Work

In this section, we describe other researches that are related to our work.

Pre-trained Language Models. Since the rising of GPT-2 [60] and BERT [16], the
research community has witnessed remarkable progress in the field of language
model pre-training on a large amount of free text. Such advancements have led
to significant progresses in a wide range of natural language understanding
(NLU) tasks [44, 87, 12, 38] and text generation tasks tasks [60, 39, 61, 71,
76, 75, 92, 74, 77].

Contrastive Learning. Generally, contrastive learning methods distinguish
observed data points from fictitious negative samples. They have been widely
applied to various computer vision areas, including image [10] and video [82,
66]. Recently, Chen _et al._ [9] propose a simple framework for contrastive
learning of visual representations (SimCLR) based on multi-class N-pair loss.
Radford _et al._ [59] and Jia _et al._ [28] apply the contrastive learning
approach for language-image pretraining. Xu _et al._ [84] and Yang _et al._
[86] propose a contrastive pre-training approach for video-text alignment.

In the field of NLP, numerous approaches have been proposed to learn better
token-level [73] and sentence-level [22, 43] representations using contrastive
learning. Beyond representation learning, contrastive learning has also been
applied to other NLP applications such as name entity recognition (NER) [13]
and document summarisation [45], knowledge probing for pre-trained language
models [50], and open-ended text generation [72].

###  Appendix B Ablation Study on Hyperparameters of MAGIC Search

Figure 3: Ablation Study on MS-COCO: (a) Effect of $k$ on MAGIC Search; (b)
Effect of $\alpha$ on MAGIC Search; (c) Effect of $\beta$ on MAGIC Search.
(Best viewed in color and by zooming in.)

In this section, we provide further analysis on the effect of hyperparameters
in MAGIC Search. To this end, we conduct ablation study experiments on MS-
COCO. Recall from Section section 4 that we set $k$, $\alpha$, and $\beta$ in
Eq. (4) for MS-COCO as 45, 0.1, 2.0, respectively. To isolate the effect of
each hyperparameter, in every experiment, we only vary the value of one
hyperparamter while keeping others constant.

Effect of $k$. Figure 3(a) shows the performances (i.e., CIDEr and ROUGE-L) of
MAGIC Search by varing $k$ from 5 to 55. We observe that, when $k$ is too
small (i.e., $k\leq 40$), the performances are not optimal. The reason is
that, a small $k$ leads to a too constrained search space, therefore MAGIC
Search cannot find the optimal text sequence that best describes the given
image. On the other hand, when $k$ is too large (i.e., $k\geq 50$), the search
space becomes too large therefore introducing extra noise that causes the
performance drop in CIDEr and ROUGE-L. In our experiments, the optimal setup
for $k$ in MAGIC Search is 45.

Effect of $\alpha$. Figure 3(b) demonstrates the performances on MS-COCO by
varying $\alpha$ from 0.0 to 1.0. We see that, when $\alpha$ is small (i.e.,
$\alpha\leq 0.2$), the performances are relatively the same. On the other
hand, a large $\alpha$ (i.e., $\alpha\geq 0.3$) causes notable drop in the
performances. This is due to the fact that a large $\alpha$ forces the
language model to generate the text continuation that is less semantically
similar to the previously generated context [72], therefore affecting the
performances of MAGIC Search. In our experiments, the optimal setup for
$\alpha$ is 0.1.

Effect of $\beta$. Lastly, Figure 3(c) illustrates the effect of $\beta$ (from
0.0 to 4.0) on MAGIC Search. Recall from Eq. (4) that, when $\beta=0.0$, the
visual control (i.e., magic score in Eq. (4)) is disabled, therefore MAGIC
Search degenerates to the vanilla contrastive search [72]. From the results,
we see that, by increasing $\beta$ from 0.0 to 0.5, a significant performance
improvement in CIDEr is obtained. Such performance gain clearly demonstrates
that the magic score in MAGIC Search is the key factor that enables the
language model to generate text grounded on the given image. When $\beta$ is
within the range of $[0.5,2.0]$, the performances of MAGIC Search are
relatively the same, indicating the robustness of our approach in terms of the
choice of $\beta$. On the other hand, we also see that a large $\beta$ (i.e.,
$\beta\geq 2.5$) causes the performances to drop, suggesting that the
importance of different terms (i.e., model confidence, degeneration penalty,
and magic score in Eq. (4)) in MAGIC Search should be properly balanced. In
our experiments, the optimal setup for $\beta$ is 2.0.

###  Appendix C Detailed Results of Zero-Shot Image Captioning

In-domain Result
---
Method | run | MS-COCO | Flickr30k
B@1 | B@4 | METEOR | R-L | CIDEr | SPICE | B@1 | B@4 | METEOR | R-L | CIDEr | SPICE
Top-$k$ | run-1 | 33.8 | 2.4 | 8.4 | 25.7 | 3.9 | 1.8 | 34.1 | 3.1 | 9.0 | 24.4 | 3.3 | 2.8
run-2 | 33.7 | 2.5 | 8.4 | 25.6 | 4.0 | 1.7 | 34.4 | 2.8 | 9.1 | 24.8 | 3.3 | 2.7
run-3 | 33.4 | 2.2 | 8.2 | 25.6 | 3.6 | 1.6 | 33.4 | 2.9 | 8.9 | 23.9 | 3.2 | 2.7
average | 33.6 | 2.4 | 8.3 | 25.6 | 3.8 | 1.7 | 34.0 | 2.9 | 9.0 | 24.4 | 3.3 | 2.7
std | 0.2 | 0.1 | 0.1 | 0.0 | 0.2 | 0.1 | 0.4 | 0.1 | 0.1 | 0.4 | 0.0 | 0.0
Nucleus | run-1 | 32.6 | 2.3 | 7.8 | 24.8 | 3.2 | 1.5 | 32.5 | 2.5 | 8.4 | 23.5 | 2.7 | 2.4
run-2 | 32.5 | 2.3 | 7.8 | 24.8 | 3.1 | 1.4 | 32.6 | 2.4 | 8.1 | 23.2 | 2.6 | 2.5
run-3 | 32.7 | 2.2 | 7.9 | 24.9 | 3.0 | 1.3 | 32.6 | 2.3 | 7.9 | 23.4 | 2.3 | 2.3
average | 32.6 | 2.3 | 7.8 | 24.8 | 3.1 | 1.4 | 32.6 | 2.4 | 8.1 | 23.4 | 2.5 | 2.4
std | 0.1 | 0.0 | 0.0 | 0.0 | 0.1 | 0.1 | 0.0 | 0.1 | 0.2 | 0.1 | 0.2 | 0.1
Cross-domain Result
Method | run |  MS-COCO $\Longrightarrow$ Flickr30k |  Flickr30k $\Longrightarrow$ MS-COCO
B@1 | B@4 | METEOR | R-L | CIDEr | SPICE | B@1 | B@4 | METEOR | R-L | CIDEr | SPICE
Top-$k$ | run-1 | 34.6 | 2.1 | 7.3 | 24.0 | 2.2 | 1.7 | 29.9 | 1.7 | 8.4 | 23.6 | 2.4 | 1.7
run-2 | 35.2 | 2.5 | 7.5 | 24.2 | 2.3 | 1.7 | 30.0 | 1.8 | 8.5 | 23.6 | 2.6 | 1.7
run-3 | 35.0 | 2.6 | 7.6 | 24.5 | 2.5 | 1.8 | 30.0 | 1.8 | 8.5 | 23.6 | 2.6 | 1.7
average | 34.9 | 2.4 | 7.5 | 24.2 | 2.3 | 1.7 | 30.0 | 1.8 | 8.5 | 23.6 | 2.5 | 1.7
std | 0.2 | 0.2 | 0.1 | 0.2 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.1 | 0.0
Nucleus | run-1 | 33.3 | 1.8 | 7.0 | 23.3 | 1.6 | 1.4 | 29.0 | 1.6 | 8.0 | 22.9 | 2.2 | 1.6
run-2 | 33.4 | 1.8 | 7.1 | 23.3 | 2.0 | 1.4 | 29.1 | 1.6 | 7.9 | 22.8 | 2.1 | 1.6
run-3 | 33.5 | 1.5 | 6.9 | 23.4 | 1.9 | 1.2 | 29.1 | 1.6 | 8.0 | 22.9 | 2.0 | 1.5
average | 33.4 | 1.7 | 7.0 | 23.3 | 1.8 | 1.3 | 29.1 | 1.6 | 8.0 | 22.9 | 2.1 | 1.6
std | 0.1 | 0.1 | 0.1 | 0.0 | 0.2 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.1 | 0.0
|  |  |  |  |  |  |  |  |  |  |  |  |

Table 4: Complete numerical results of stochastic methods on zero-shot image
captioning. The average and std rows show the mean and standard deviation of
results from three different runs.

In Table 4, we show the detailed numerical results of stochastic baselines
(i.e., Top-$k$ and Nucleus) on the task of zero-shot image captioning. The
upper part of Table 4 presents the results for in-domian experiments (Section
section 4.1) and the lower part of Table 4 presents the results for cross-
domain experiments (Section section 4.2). For each method, we report the
results of three different runs with different random seeds along with the
mean and standard deviation of different runs.

###  Appendix D More Visual Examples of Zero-Shot Image Captioning

Figure 4: More examples of zero-shot image captioning. (Best viewed by zooming
in.)

Figure 4 presents more visual comparisons between our approach against other
two strong zero-shot baselines (i.e., CLIPRe and ZeroCap) along with the
reference caption.

###  Appendix E Detailed Results of Story Generation

Method | run |  rep-2$\downarrow$ |  rep-3$\downarrow$ |  rep-4$\downarrow$ |  diversity$\uparrow$ |  coherence$\uparrow$ |  MAUVE$\uparrow$ |  CLIPScore$\uparrow$
---|---|---|---|---|---|---|---|---
Top-$k$ | run-1 | 3.42 | 0.75 | 0.22 | 0.95 | 0.460 | 0.85 | 0.21
run-2 | 3.27 | 0.77 | 0.25 | 0.95 | 0.455 | 0.87 | 0.21
run-3 | 3.46 | 0.75 | 0.21 | 0.95 | 0.458 | 0.86 | 0.21
average | 3.38 | 0.76 | 0.23 | 0.95 | 0.458 | 0.86 | 0.21
std | 0.08 | 0.01 | 0.02 | 0.00 | 0.002 | 0.01 | 0.00
Nucleus | run-1 | 2.88 | 0.57 | 0.16 | 0.96 | 0.448 | 0.88 | 0.21
run-2 | 2.88 | 0.61 | 0.18 | 0.96 | 0.454 | 0.87 | 0.22
run-3 | 2.99 | 0.62 | 0.20 | 0.96 | 0.454 | 0.88 | 0.21
average | 2.92 | 0.60 | 0.18 | 0.96 | 0.452 | 0.88 | 0.21
std | 0.05 | 0.02 | 0.02 | 0.00 | 0.003 | 0.00 | 0.00
Typical | run-1 | 2.44 | 0.42 | 0.11 | 0.97 | 0.454 | 0.83 | 0.18
run-2 | 2.56 | 0.48 | 0.12 | 0.97 | 0.448 | 0.84 | 0.19
run-3 | 2.55 | 0.48 | 0.14 | 0.97 | 0.448 | 0.85 | 0.19
average | 2.52 | 0.46 | 0.12 | 0.97 | 0.450 | 0.84 | 0.19
std | 0.05 | 0.03 | 0.01 | 0.00 | 0.002 | 0.01 | 0.00
|  |  |  |  |  |  |  |

Table 5: Complete numerical results of stochastic methods on story generation.
The average and std rows show the mean and standard deviation of results from
three different runs.

Table 5, we present the detailed numerical results of stochastic baselines
(i.e., Top-$k$, Nucleus, and Typical) on the task of story generation. For
each method, we report the results of three different runs with different
random seeds along with the mean and standard deviation of different runs.

###  Appendix F Human Evaluation Guidelines

Given the story title and the image, please evaluate the system’s result with
respect to the following features: (1) Coherence; (2) Fluency; (3)
Informativeness; and (4) Story-Image Relevance. In the following, we provide
some guidelines regarding how to judge the quality of the system’s result in
terms of different features.

####  F.1 Coherence

This metric measures whether the system’s result is semantically and factually
consistent with the story title. The definitions of different scores are:

  * •

[5]: The system’s result is perfectly in line with the semantic meaning
defined by the story title. And all its content is factually supported by or
can be logically inferred from the title.

  * •

[4]: The system’s result is very related to the story title but with some
minor errors that does not affect its overall relevance with respect to the
story title.

  * •

[3]: The system’s result is, to some extent, relevant to the story title with
some errors that display minor semantic inconsistency or contradiction.

  * •

[2]: At the first glance, the system’s result seems to be related to the story
title. But with careful inspection, the semantic inconsistency can be easily
spotted.

  * •

[1]: The system’s result is obviously off-the-topic or it is semantically
contradicted to the content contained in the story title.

####  F.2 Fluency

This metric measures the fluency of the system’s result. The definitions of
different scores are:

  * •

[5]: The system’s result is human-like, grammatically correct, and very easy
to understand.

  * •

[4]: Choose this score when you are hesitant between the score 3 and score 5.

  * •

[3]: The system’s result contains minor errors but they do not affect your
understanding.

  * •

[2]: Choose this score when you are hesitant between the score 1 and score 3.

  * •

[1]: The system’s result does not make sense and it is unreadable.

####  F.3 Informativeness

This metric measures the diversity, informativeness, and interestingness of
the system’s result. The definitions of different scores are:

  * •

[5]: The system’s result is very informative and contains novel content. In
addition, it displays a high level of diversity and it is enjoyable to read.

  * •

[4]: Choose this score when you are hesitant between the score 3 and score 5.

  * •

[3]: The system’s result contains some new information and it displays a
certain level of diversity.

  * •

[2]: Choose this score when you are hesitant between the score 1 and score 3.

  * •

[1]: The system’s result is dull, repetitive, and does not have new
information.

####  F.4 Story-Image Relevance

This metric measures the relevance between the system’s result with respect to
the given image. The definitions of different scores are:

  * •

[5]: The system’s result is very related to the given image and contains a
diverse set of concepts that are visually displayed in the image.

  * •

[4]: Choose this score when you are hesitant between the score 3 and score 5.

  * •

[3]: The system’s result is partially related to the image and some of its
content can be found in the image.

  * •

[2]: Choose this score when you are hesitant between the score 1 and score 3.

  * •

[1]: The system’s result is completely irrelevant to the given image.

###  Appendix G More Examples of Story Generation

Figure 5: More examples of story generation. MAGIC can generate text
(highlighted in blue) that is related to the visual concepts displayed in the
image. (Best viewed in color and by zooming in.)

Figure 5 presents more examples generated by contrastive search along with the
examples generated by MAGIC Search grounded on the retrieved image. The image
retrieved by the story title is shown on the left-hand side of Table 5.