# Zero-Shot Video Captioning with Evolving Pseudo-Tokens

Yoad Tewel Yoav Shalev Roy Nadler Idan Schwartz Lior Wolf
School of Computer Science, Tel Aviv University

###### Abstract

We introduce a zero-shot video captioning method that employs two frozen
networks: the GPT-2 language model and the CLIP image-text matching model. The
matching score is used to steer the language model toward generating a
sentence that has a high average matching score to a subset of the video
frames. Unlike zero-shot image captioning methods, our work considers the
entire sentence at once. This is achieved by optimizing, during the generation
process, part of the prompt from scratch, by modifying the representation of
all other tokens in the prompt, and by repeating the process iteratively,
gradually improving the specificity and comprehensiveness of the generated
sentence. Our experiments show that the generated captions are coherent and
display a broad range of real-world knowledge. Our code is available at:
https://github.com/YoadTew/zero-shot-video-to-text.

##  1 Introduction

Image captioning is becoming increasingly accurate and can successfully tackle
more complex benchmarks than ever before. However, the progress in video
captioning is slower due to both methodological reasons and dataset
construction challenges. First, the video captioning task itself has more
possible definitions than image captioning. For example, do we want a complete
description of the events in the video or a general description of it? Second,
even considerably more limited tasks, such as action recognition in pre-
trimmed videos, are still technologically challenging. Third, the descriptions
attached to web videos are often not an accurate depiction of the content and
events in the video, making the construction of web-scale datasets more
challenging.

These challenges mean that strategies used in related tasks are less suitable
for the task of video captioning. (i) One cannot train on large and noisy
datasets, since the amount of noise would be too high. (ii) Learning on a
carefully curated dataset would be too restrictive in terms of the obtained
coverage, and too limited in the use of language (due to the nature of cost-
effective human annotation). (iii) A divide-and-conquer approach that captions
single frames would still require a sophisticated way of combining all
information and text. (iv) Relying on pretrained action recognition models
would lead to inaccurate results that are very much limited in the scope of
captured actions.

In light of the challenges faced by model-learning approaches, the method we
introduce is a zero-shot method. It uses the information stored in two pre-
trained and frozen networks to perform the video captioning task. One model is
an autoregressive language model that can generate natural and mostly logical
sentences The second model is an image-text matching model that is used to
steer the language model, via a contrastive loss, toward sentences that match
a set of input frames.

Figure 1: We present a novel way of optimizing a sentence generation process
to match a set of images by using pseudo-tokens and iteratively generating
sentences. The method exhibits real-world knowledge, which enables it to
construct a narrative from images and describe videos. This narrative evolves
through the iterations and tends to become increasingly specific.

We make extensive use of the fact that the autoregressive process is
conditioned on its prompt. Our method’s generative process employs a prompt
that contains three parts: (i) pseudo-tokens that are vectors in the latent
space of the language models, (ii) a random prompt such as “Image of” that
provides context for the captioning task, but also varies (“Photo of”, “Video
of”, etc.) as a form of inference-time augmentation, and (iii) the previously
generated tokens.

The first part differs from the other two in that it does not originate from
actual language tokens. However, all three parts are manipulated, in memory,
by optimizing them to increase the image-matching score of the next word being
selected. This is done while regularizing in such a way that the language
model does not drastically alter the likelihood of the next word.

Since the sentence generation process is autoregressive, the initial words of
the generated sentence employed psuedo-tokens that were not optimized based on
the signal obtained from the entire sentence. We, therefore, repeat the
process and start the autoregressive process with the pseudo-tokens that were
obtained at the end of the previous generation iteration. As we show in Fig.
1, this leads to increasingly concrete prompts. The process is repeated
sixteen times, and the sentence that maximizes the image-text matching score
is selected.

In our implementation, we use the GPT-2 language model [36], due to its
availability, and the CLIP image-text matching model [35], which is often used
in zero-shot learning. We experiment with both video captioning and describing
image sets. Our results show a clear advantage over the state-of-the-art video
captioning methods and over recent zero-shot image captioning methods. Our
code is attached as supplementary.

##  2 Related Work

Image captioning is a fundamental vision and language task. Early methods
applied RNNs [27, 16]. Attention was added to identify relevant salient
objects [55, 38]. Graph neural networks and transformers helped model spatial
and semantic interactions [57, 15, 56].

Other video-based tasks include action recognition [43], paragraph captioning
[58], and video object segmentation [33]. Some approaches to video analysis
use different experts, including speech, audio, motion, OCR, appearance, and
face detection [25, 1]. This work considers video captioning by generating a
single sentence that adequately describes a set of frames. Despite the lack of
temporal information in a set, we find that the language model generates a
logical order of events.

In various contributions, sparse sampling along with better spatial reasoning
have proved sufficient for handling reasoning tasks, such as video dialogs
[39] and video retrieval [17, 61, 37, 18]. An attention module that selects
the relevant frames can reduce the temporal dimension [2, 9]. In our work, we
also employ sparse sampling that utilizes distances in the CLIP embedding
space to construct a set of the most relevant frames.

Significant improvements have been achieved by using large-scale unsupervised
vision-language data sets with millions of image and text pairs [60, 8, 21]
and million of videos [29, 62, 26]. The unsupervised data is used in a pre-
training phase. Fine-tuning for a particular task is done in the final stage,
using smaller datasets annotated by humans. Although this process has been
shown to boost performance, it remains highly dependent on the human
annotations. Our experiments show that captions learned based on human
annotations are dull. Moreover, the same repetitive patterns are produced by
significantly different baseline methods [34, 6].

As a way to reduce dependence on supervision, CLIP has been using web-based
learning to study language-image relationships [35]. CLIP’s strength lies in
the amount of data it collects. CLIP is trained in a contrastive manner, using
400M image-text pairs from the web, unlike supervised methods, which typically
use thousands of samples. The method learns a bimodal distance metric between
an input image and a text sentence, which we refer to as the CLIP score.
Matching videos to text also benefited from a contrastive approach [53, 19].
However, video data can be challenging to collect. In our case, we used CLIP
to guide a language generator in a zero-shot manner.

Contrastive learning is effective for zero-shot capabilities, achieving good
results for several tasks, including image classification and action
recognition. However, generative tasks based directly on the CLIP score are
rare, since the score requires seeing both text and image. Instead, multiple
contributions rely on CLIP’s image and text encodings, which are known to
improve performance in vision+language tasks [40], especially in image
captioning [30] and video captioning [46]. However, fine-tuning distorts the
latent semantics of CLIP’s encoder [47]. MAGIC employs CLIP scores to shift
PLM logits towards image correspondence [44]. Despite this, they fine-tunes
the PLM on the text corpus of MS-COCO captions, so robustness is still
compromised. Alternatively, CLIP can be used as part of a loss term to guide
generative processes to match language and text, for example, as a loss term
for 3D mesh generation [28] or text-guided image generation [32, 5].

Recently, it was suggested to use CLIP loss to guide a Pretrained Language
Model (PLM) for image captioning [47]. Compared with ours, their method
optimizes each generated token individually, aiming to obtain the token
closest to the given image. In contrast, our method optimizes pseudo-tokens
through iteratively generating sentences, aiming to steer the generation
process of the entire sentence. Although their process is effective in
describing one visual cue, it is challenged by the more difficult task of
describing multiple images coherently. We demonstrate that the ability to
manipulate an entire sentence without committing to a single generation path
has beneficial effects. Optimizing an entire sentence also means not requiring
sequence generation strategies such as beam search.

The literature on tuning prior knowledge within large-scale PLMs, such as
GPT-2 [36], is growing rapidly. There are several on-going directions: (i)
Fine-tuning, e.g., by using GANs [63] or RL [59]; (ii) Disentangling the
latent representations [41]; (iii) Training a controllable LM with fixed
control codes [14]; (iv) Trainable decoding [12]; (v) Decoding steering [7];
and (vi) Prompt engineering [42], as well as prompt learning [50, 22, 10, 24].
In this work, we present a novel PLM decoding approach that combines steering
and prompt tuning by generating sentences iteratively and applying prompt
tuning.

Figure 2: Illustration of our method for guiding a PLM to generate the word
‘landing’. Optimization takes place during the autoregression inference, by
tuning pseudo-tokens ($\hat{C}_{\Psi}$). Two signals steer the pseudo-tokens’
representations, visual correspondence ($\mathcal{L}_{\text{vision}}$) and
language fluency ($\mathcal{L}_{\text{language}}$).

##  3 Method

Our goal is to create a sentence $S=\\{t_{1},\ldots,t_{M}\\}$ of length $M$
that describes a set of video frames $\mathcal{F}=\\{F_{1},\ldots,F_{N}\\}$,
where $N$ is the number of frames. When $N=1$, the problem corresponds to
traditional image captioning.

Two components are at the core of our solution. The first is a pre-trained
language model (PLM) that generates sentences, for which we use GPT-2. The
second, CLIP, is a pre-trained model that computes the distance between a
frame $F$ and a sentence $S$, and guides the PLM during inference.

Guiding a PLM with CLIP has recently shown promising results for image
captioning [47, 45]. These approaches use CLIP to optimize the next token to
fit the image; we call this technique token-level optimization. However, the
application of these approaches to videos is limited, especially in the case
of low homogeneity between frames. In this case, maintaining language fluency
is difficult, since: (i) a single token has to describe a set of non-
homogeneous frames, and (ii) the generation commits to a single direction,
restricting the flexibility of the process. By contrast, rather than
optimizing tokens, our method performs a sentence-level optimization. To
achieve this, the inference starts with randomly initialized pseudo-tokens.
These tokens do not need to be actual words in the dictionary, but rather
hidden states of words, that can be optimized using gradient descent.
Description of visual content is driven using prefix-tokens, such as ‘Video
showing’. The next step consists of generating multiple sentences and
continuously optimizing the pseudo-tokens. This is accomplished by calculating
two types of losses: (i) $\mathcal{L}_{\text{vision}}$, which is the sum of
the distance between all frames in $\mathcal{F}$ and the generated sentence,
and (ii) $\mathcal{L}_{\text{language}}$ which takes into account language
characteristics by considering the PLM token distribution. With no additional
supervision or training, we benefit from the extensive knowledge embedded in
CLIP and GPT-2. Our autoregressive process in depicted in Fig. 2.

|  | Supervised Metrics | Unsupervised Metrics
---|---|---|---
Dataset | Method | B@4 | M | C | R | $\text{CLIP-S}^{\text{Ref}}$ | CLIP-S | BLIP-S | Retrieval | PP
MSR-VTT | VNS-GRU [6] | 0.453 | 0.299 | 0.530 | 0.634 | 0.739 | 0.626 | 0.623 | 0.446 | 118.81
SemSynAN [34] | 0.464 | 0.304 | 0.519 | 0.647 | 0.733 | 0.619 | 0.608 | 0.437 | 155.01
Zero-Shot Methods |
ZeroCap* [47] | 0.023 | 0.129 | 0.058 | 0.304 | 0.739 | 0.710 | 0.575 | 0.442 | 54.71
MAGIC* [44] | 0.055 | 0.133 | 0.074 | 0.354 | 0.628 | 0.566 | 0.434 | 0.392 | 30.48
| Ours | 0.030 | 0.146 | 0.113 | 0.277 | 0.785 | 0.775 | 0.675 | 0.504 | 18.35
MSVD | VNS-GRU [6] | 0.665 | 0.421 | 1.215 | 0.797 | 0.780 | 0.673 | 0.646 | 0.557 | 418.72
SemSynAN [34] | 0.644 | 0.419 | 1.115 | 0.795 | 0.767 | 0.660 | 0.633 | 0.546 | 242.46
Zero-Shot Methods |
ZeroCap* [47] | 0.029 | 0.163 | 0.096 | 0.354 | 0.762 | 0.765 | 0.642 | 0.500 | 28.44
| MAGIC* [44] | 0.066 | 0.161 | 0.140 | 0.401 | 0.670 | 0.623 | 0.497 | 0.469 | 29.84
| Ours | 0.030 | 0.178 | 0.174 | 0.314 | 0.805 | 0.822 | 0.743 | 0.569 | 18.94

Table 1: Quantitative results for video captioning. We separate the results
into two categories: (i) supervised metrics that require human references, B@4
= BLEU-4, M = METEOR, C = CIDEr, S = SPICE, and $\text{CLIP-S}^{\text{Ref}}$.
(ii) Unsupervised metrics that use a pre-trained model, CLIP-S = CLIP-based
image-text similarity, BLIP-S = BLIP-based image-text similarity [20],
Retrieval = VideoCLIP-based video-text similarity [53], and PP = caption
perplexity computed with BERT [8]. (*) denotes that the model is adapted from
image captioning to video captioning.

#### PLM Guidance with Prompt Learning

PLMs are trained on vast web knowledge to optimize a sum of conditionals,
i.e., $\max_{\theta}p(S)=\sum_{i=1}^{L}p_{\theta}(w_{i}|w_{1:i-1}),$ where
$\theta$ are trainable weights. The likelihood of each sub-sentence depends on
its context. Thus, one can solve various tasks by altering the input context.
For instance, to answer the question “what is the capital of Britain?” one
could plug into the PLM the prompt “The capital of Britain is.” The PLM then
finds the most likely next token (“London”) to optimize the conditional
probability.

Prompt engineering entails finding the most suitable prompt for a given task.
In our case, the task is to generate a sentence that maximizes similarity to a
set of frames $\mathcal{F}$. Similarity is measured in terms of CLIP’s
distance metric between image and text. Any image imposes its own set of
constraints, and the prompt needs to account for all of them. The prompt must
be flexible enough, which is ensured by optimizing pseudo-tokens, i.e.,
instead of finding real tokens for each video, we tune representative
embeddings of tokens.

The GPT-2 PLM is built with $L$ layers of Transformers, each composed of key
and value embeddings, to model interactions between tokens [48]. The context
of previous tokens can be cached by keeping their key and value
representations. We denote the cache with
$C_{i}=[K_{j}^{l},V_{j}^{l}]_{j<i,l\leq L}$, where $i$ is the number of
tokens, and $K_{j}^{l},V_{j}^{l}$ are the key and value embeddings of the
$l$-th Transformer layer of the $j$-th token.

Our method starts the autoregression process with a randomly initialized
cached pseudo-prompt context, i.e.,
$\hat{C}_{\Psi}=[\hat{K}_{j}^{l},\hat{V}_{j}^{l}]_{0<j<\Psi,l\leq L}$, which
represents $\Psi$ pseudo-tokens.

We further include $\bar{C}_{p}=[\bar{K}_{j}^{l},\bar{V}_{j}^{l}]_{0<j<p,l\leq
L}$, where $p$ is the prefix length. The prefix-tokens serve to direct the
task towards captioning a set of images. The prefix-tokens are sampled as one
of the prompts in the set $\mathcal{P}=${“Image of”, “Picture of”, “Photo of”,
“Video of”,“Image shows”, “Picture shows”, “Photo shows”, “Video shows”,
“Image showing”, “Picture showing”, “Photo showing”, “Video showing”}.

Overall, the autoregressive process takes the form

$\displaystyle
p_{i+1}(\hat{C}_{\Psi})=\text{PLM}(t_{i},[\hat{C}_{\Psi},\bar{C}_{p},C_{i}]),$
(1)

where $p_{i+1}$ is the distribution of the next token.

#### Loss:

At each step in the auto-regression process, we aggregate our loss, which will
be used for optimization only after generating a complete sentence. Our first
loss term encourages the generated text to correspond to the set of images.

Let $S_{k}$ be the sentence generated up until this stage, ending with the
token $k$. The visual-semantic loss calculates the cross-entropy
($\operatorname{CE}$) between the optimized PLM distribution and the CLIP
potential similarity distribution $\theta_{\text{CLIP}}$:

$\displaystyle\mathcal{L}_{\text{vision}}(\hat{C}_{\Psi})=\operatorname{CE}\left(p_{i+1}(\hat{C}_{\Psi}),\theta_{\text{CLIP}}\right),$
(2)

where
$\theta_{\text{CLIP}}(k)\propto\sum_{F\in\mathcal{F}}\operatorname{CLIP}\left(F,S_{k}\right)$
is the sum of CLIP’s matching scores of $S_{k}$ with all the frames in
$\mathcal{F}$. We compute the score for the top 100 tokens according to the
original PLM distribution, with the rest set to zero.

While the PLM is trained on natural text, the model in which the free-form
context $C_{\Psi}$ is added (Eq. 1) can shift to very different distributions
during optimization. In order to maintain fluent language, we define a
language-related loss term,

$\displaystyle\mathcal{L}_{\text{language}}(\hat{C}_{\Psi})=\operatorname{CE}\left(p_{i+1}(\hat{C}_{\Psi}),\operatorname{PLM}(t_{i},[\bar{C}_{p},C_{i}])\right),$

which is the cross-entropy loss of the optimized PLM, as defined in Eq. 1,
with the unmodified PLM distribution.

In order to have the generated text describe the set of images using fluent
language, we solve the following optimization problem:

$\displaystyle\min_{\hat{C}_{\Psi}}\mathcal{L}(\hat{C}_{\Psi})=\min_{\hat{C}_{\Psi}}\mathcal{L}_{\text{vision}}(\hat{C}_{\Psi})+\lambda\mathcal{L}_{\text{language}}(\hat{C}_{\Psi})$

where hyper-parameter $\lambda$ calibrates the trade-off between relevance to
the video and language fluency. The optimization process occurs during
autoregression inference, generating sentences iteratively. We detail this
process next.

#### Evolving Pseudo-Tokens Optimization

The optimization occurs during the generation of the entire sentence,
increasing image correspondence by applying iterations. Notably, we do not use
annotations, nor are any parameters fine-tuned in a separate phase. For each
generated token, we perform one optimization step

$\displaystyle\hat{C}_{\Psi}\longleftarrow\hat{C}_{\Psi}+\alpha\frac{\nabla_{C_{\Psi}}\mathcal{L}(\hat{C}_{\Psi})}{\|\mathcal{L}(\hat{C}_{\Psi})\|^{2}},$
(3)

and continue the generation process. We stop the process and start a new
sentence upon reaching a dot token, keeping the optimized pseudo-tokens. This
optimization process is autoregressive in its nature, and, therefore, the
first words are generated with preliminary context. Though early sentences
usually have good language, each successive sentence increases the ability to
ground objects.

As a result, the process shifts from a general discussion to a more concrete
explanation, see Sec. 4. Throughout our experiments, we employ 16 generation
iterations. Each time a new sentence is generated, we pick a prefix-token from
the set $\mathcal{P}$ at random, which acts as data augmentation, see Sec. B.1
for an analysis of using a random prefix.

#### CLIP-based Frame Sampling for Video Captioning

Any set of images, including video frames, can be described with our method.
Adapting our model for video captioning begins by sampling three frames every
second. In order to avoid repetition and to capture diverse frames, we further
subsample the frames with a CLIP-based strategy.

We begin with the first frame as an anchor. The next frame (a third of a
second later, due to the initial sampling) is included in the set if its
distance from the anchor, in the space defined by the CLIP image encoder,
exceeds a certain threshold $\lambda_{\text{frame}}$. The newly selected frame
becomes the new anchor and the selection process continues in the same manner.
See appendix Fig. 17 for qualitative results. We use
$\lambda_{\text{frame}}$=$0.9$ in all our experiments.

Figure 3: Examples of our video captions with two types of baselines: (i) the
supervised methods SemSynAN and VNS-GRU; and (ii) the zero-shot methods
ZeroCap and MAGIC. Notably, our method grounds objects from different frames
and exhibits real-world knowledge. The 1st and 2nd rows provide examples of
real-world knowledge. Figure 4: Evolution of video captions. We show the
sentence with the highest CLIP score at different generation iterations.
Grounded words are highlighted.

##  4 Results

For all experiments, the following settings are used (see appendices for
parameter sensitivity and ablations): We set $\lambda$ to $0.8$. During
sentence generation, we pick one of the top-3 tokens at random. To avoid long
repetitive sentences, the number of generated tokens per sentence was limited
to 20. To avoid generating irrelevant entities, such as names, we reduce by 1
the logits of tokens with uppercase letters. Using a single Titan X GPU, each
sentence takes 5 seconds to generate. All 16 sentence-generating iterations
take approximately a minute and a half.

Two video datasets are used: MSR-VTT [54] and MSVD [51]. MSR-VTT is a large-
scale dataset with about 50 hours of videos divided into 10,000 videos with 20
descriptions each. It includes a variety of categories, such as video games
and TV shows. The test set consists of 2,990 videos. MSVD contains 1,970 short
video clips, 670 of which are dedicated for testing. All experiments are
carried out on the test set.

We use two types of metrics: (i) Supervised metrics that measure text
correspondence to human references: BLEU [31], METEOR [4], CIDEr [49], SPICE
[3]. Lastly, $\text{CLIP-S}^{\text{Ref}}$ [13] measures semantic similarity by
utilizing CLIP’s textual encoder. (ii) Unsupervised metrics that are computed
without referring to the human annotation. Relatedness to the visual cue is
measured by averaging CLIP or BLIP image similarity scores to the generated
sentence across the frames. Relatedness to the video is measured by the
VideoCLIP [53] video-to-text distance metric (“Retrieval” in the results
table). Language quality is estimated using the perplexity score of the
generated caption, employing BERT [8].

| Supervised Metrics | Unsupervised Metrics
---|---|---
Method | B@4 | M | C | S | $\text{CLIP-S}^{\text{Ref}}$ | CLIP-S | PP
VinVL [60] | 0.41 | 0.311 | 1.409 | 0.252 | 0.83 | 0.780 | 24.16
| Zero-Shot Methods |
ZeroCap [47] | 0.029 | 0.12 | 0.131 | 0.055 | 0.778 | 0.870 | 25.737
MAGIC [44] | 0.129 | 0.174 | 0.493 | 0.113 | 0.763 | 0.737 | 37.126
Ours | 0.022 | 0.127 | 0.172 | 0.073 | 0.798 | 0.885 | 19.049
Table 2: Quantitative results for image captioning methods. We evaluate supervised metrics that measure text correspondence to human references and unsupervised metrics that are computed without referring to the human annotation. Method | Image | Video
---|---|---
MAGIC [44] | 1.65 | 1.77
ZeroCap [47] | 2.52 | 2.30
Ours | 4.01 | 4.14
Table 3: Mean Opinion Score (MOS, scale of 1–5) for caption quality using
real-world images and videos.

#### Quantitative analysis

In Tab. 1, we compare our approach with supervised state-of-the-art baselines
for video captioning. We also compare it with zero-shot video captioning
baselines we created by modifying CLIP-based zero-shot image captioning
methods: ZeroCap [47], a zero-shot method for image captioning, which also
optimizes the generated sentence during inference. We adapt their method from
image captioning to video captioning by replacing their single image CLIP loss
with ours (i.e., a sum of CLIP losses for each frame). MAGIC [44], another
zero-shot method for image captioning, which uses MAGIC scores, i.e., a CLIP-
based measure of how closely a sentence ending with a given token matches an
image, to skew the next-token distribution of a pre-trained language model to
match a given image. To adapt their model to videos, we aggregate the CLIP
score of all sampled frames to calculate the magic score before applying a
softmax.

As expected, the supervised models VNS-GRU [6] and SemSynAN [34] perform
significantly better on supervised metrics, based on correspondence to human
references. However, when considering semantic relatedness to annotations
(i.e., CLIPScoreRef), our method is better (0.785 vs. 0.739 and 0.733). We
next look at unsupervised metrics. BLIP-Score suggests that our text is more
relevant to the frames (0.675 vs. 0.623 and 0.608). Furthermore, when
considering the entire video temporally, our method has the lowest VideoCLIP
text-to-video distance (0.504 vs. 0.446 and 0.437). To understand the source
of the weakness of the supervised methods, we measure the novelty of the
generated sentences. Aggregated over the entire MSR-VTT test set, our method
has a vocabulary size of 4,372. In contrast, SemSynAN and VNS-GRU use only 359
and 435 words, respectively, with roughly 40% of the generated sentences
existing in the training set.

Focusing next on zero-shot methods, Zero-Cap has higher CLIP-related scores
than the supervised method. However, in all other metrics, it falls short. One
of the main limitations of this technique is that the generated token is
optimized until convergence in a greedy manner, without the option of altering
past tokens. In contrast, our approach optimizes the generation of an entire
sentence, which is beneficial for coherent generation from noisy signals
resulting from multiple visual cues.

Figure 5: Counterfactual examples, which reorder the events in generated
sentences. The analysis shows that temporal knowledge is embedded in the
language and in visual cues.

MAGIC samples a next token at each generation step from a distribution that
combines both CLIP potentials and the distribution computed using a language
model. To ensure that the resulting captions lie in the text domain of image
captions, they fine-tune the language model on the text corpus of MS-COCO
captions. While MAGIC is comparable to our method with regards to the
supervised metrics, it falls short in all other metrics. We hypothesize that
MAGIC’s per token greedy decoding harms the ability to handle the noisy signal
of multiple visual cues found in a video.

#### Qualitative Analysis:

In Fig. 3, we show examples of our video captions. First, we evaluate captions
created using supervised methods. Despite having significantly different
architectures, the supervised methods produce very similar captions. Although
supervised methods may be on topic, their grounding capabilities are limited
to relatively abstract objects. For example, the supervised method recognizes
that the video is about a car or phone in the first row. However, it misses
the true intent of the video.

Our model displays real-world knowledge. For instance, it can detect brands.
The video in the first row, on the left, shows Samsung’s advertisement (i.e.,
Samsung Galaxy) about credit card payments, which is recognized by our method.
The video on the right shows an Audi navigation commercial. Our model
identifies the Audi brand and the video’s purpose. The video in the 2nd row,
on the left, demonstrates TV animation. Our method recognizes the character
Ash from Pokemon. Next, we demonstrate that our model can recognize events
from various frames. In the 2nd row, right side, our model describes the
entire scene, both the audience and the comedian being interviewed.

We also compare our method to two other zero-shot methods, ZeroCap and MAGIC.
We find that captions generated by other methods are often broad, failing in
videos that require real-world knowledge or aggregating information from
multiple frames. For instance, both methods miss the Samsung brand.
Furthermore, they often identifies a related but wrong entity (e.g., Pickachu
in the Pokemon video). Often, they miss events from different frames, e.g.,
they do not describe the comedian or the interviewer in the right video on the
2nd row.

In Fig. 4 we evaluate how sentences evolve during the inference process. We
first note that, quantitatively, the CLIP score increases between generation
iterations, e.g., on the left, at the fourth sentence iteration, the clip
score is 0.58, while at the 16th iteration, the score is 0.89. This
improvement can also be seen qualitatively. In the left video, we see that the
sentence grounds the truck, which is visible in all frames, after four
iterations. The model grounded both the truck and trailer in its eighth
iteration. Only after the 16th iteration does the model recognize it as a Lego
truck. We note an interesting failure case in which, after 16 iterations, the
model incorrectly identifies the type of video as a trailer. The video on the
right shows similar behavior. In the 16th sentence iteration, the CLIP score
increased from 0.70 to 0.88. In the fourth iteration, the video was grounded
to more abstract objects (e.g., soldier, battle, alien), while the eighth
iteration identified the characters from Halo. As a final step, the model
figures out that the video is an animated cartoon in the 16th iteration.

Tab. 2 compares our method to state-of-the-art image captioning approaches,
ZeroCap, and MAGIC. These approaches optimize at the token level resulting in
a drop in the PP metric. Our sentence-level optimizations generate more fluent
captions compared to token-level optimizations. We also assess supervised
metrics aimed at language correspondence against human references. MAGIC
excels on the supervised metrics. As the method fine-tunes the PLM on the
human references, it allows the language to relate to their style. When
evaluating captions using CLIP-based scores, performance drops. Our next step
is to explore the reason by using a small sample of images outside the COCO
dataset. A significant advantage of zero-shot captioning methods is that they
can describe images not included in the training set. In Tab. 3, we assess the
caption quality of zero-shot methods when presented with images or videos
requiring real-world knowledge. We asked 20 annotators to consider three
properties: language fluency, human-likeness, and grounding, and rank each
caption between 1 (lowest score) and 5 (highest score). The test included 10
images randomly sampled from the web, and 10 videos randomly sampled from the
MSR-VTT [54] test set. Our approach is significantly better for videos (4.14
vs. 2.30) and images (4.01 vs. 2.52). In Fig. 13, we show different samples
from the human evaluation. Generally, our method’s captions provide a coherent
description of the given entity. MAGIC, on the other hand, does not perform
well. The proposed method may be hindered by the fine-tuning of the language
model, which still interferes with the method’s ability to discuss real-world
entities.

Furthermore, we stress-test our model by captioning sets of random images,
which explores its ability to create a coherent text that describes unrelated
images, see Sec. B.2 for further discussion.

##  5 Discussion and Limitations

Despite showing video as the main application in most of our experiments, our
method is invariant to the order of frames in the sequences. This is a result
of using a zero-shot strategy, in which the underlying models are trained per
frame. Examining the results, the generated captions display a logical order.
This is not surprising, since ordering frames is typically not an extremely
challenging visual understanding task, and is also used as a self-supervised
task [52]. Evidently, the information that exists in both the Language Model
and the CLIP network is sufficient for generating sentences that adhere to the
natural order of events.

To further examine this, we created counter-factual sentences in which the
order of events is altered, and measured the PLM and CLIP scores. As can be
seen in the examples of Fig. 5, changing the order leads to a drop in both
scores. For example, judges giving a score before a contestant sings has a
higher perplexity, or a player’s pose can be used to identify them as someone
who throws a ball and not someone who catches it.

A limitation of large-scale models is that they can sometimes generate sexist,
or otherwise toxic language. Before viewing the examples (see appendix Fig.
18), readers are advised that they contain harsh language. CLIP and GPT-2 may
be at fault because they use web-based, uncurated data [11]. It is advisable
to be aware of these weaknesses before deploying our method or any other
method that uses these models.

##  6 Conclusions

We present a method for creating natural-sounding captions from a video. The
method is a zero-shot method based on two frozen networks: a language model
and an image-text matching model. The method is based on learning, for each
input, a sequence of vectors that serve as pseudo-tokens that drive the
generation process. An autoregressive process generates the caption while
updating the pseudo-tokens. Once the caption is generated, the process
repeats, using the learned pseudo-tokens as the starting point, leading to
increasingly concrete and well-grounded captions. Our experiments show that
our model generates novel captions that ground objects from multiple images
into one coherent narrative.

## Acknowledgments

This project has received funding from the European Research Council (ERC)
under the European Unions Horizon 2020 research, innovation programme (grant
ERC CoG 725974).

## References

  * [1] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  Advances in Neural Information Processing Systems, 34, 2021.
  * [2] A. Ali, I. Schwartz, T. Hazan, and L. Wolf.  Video and text matching with conditioned embeddings.  In WACV, 2022.
  * [3] P. Anderson, B. Fernando, M. Johnson, and S. Gould.  Spice: Semantic propositional image caption evaluation.  In ECCV, 2016.
  * [4] S. Banerjee and A. Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In ACL, 2005.
  * [5] H. Chefer, S. Benaim, R. Paiss, and L. Wolf.  Image-based clip-guided essence transfer.  ECCV, 2022.
  * [6] H. Chen, J. Li, and X. Hu.  Delving deeper into the decoder for video captioning.  ECAI, 2021.
  * [7] S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu.  Plug and play language models: A simple approach to controlled text generation.  In ICLR, 2020.
  * [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  NAACL, 2019.
  * [9] V. Gabeur, C. Sun, K. Alahari, and C. Schmid.  Multi-modal transformer for video retrieval.  ECCV, 2020.
  * [10] T. Gao, A. Fisch, and D. Chen.  Making pre-trained language models better few-shot learners.  arXiv preprint arXiv:2012.15723, 2020.
  * [11] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  EMNLP, 2020.
  * [12] J. Gu, K. Cho, and V. O. Li.  Trainable greedy decoding for neural machine translation.  EMNLP, 2017.
  * [13] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi.  Clipscore: A reference-free evaluation metric for image captioning.  EMNLP, 2021.
  * [14] N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher.  Ctrl: A conditional transformer language model for controllable generation.  arXiv preprint arXiv:1909.05858, 2019.
  * [15] T. N. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks.  ICLR, 2017.
  * [16] B. Klein, G. Lev, G. Sadeh, and L. Wolf.  Fisher vectors derived from hybrid gaussian-laplacian mixture models for image annotation.  arXiv preprint arXiv:1411.7399, 2014.
  * [17] J. Lei, T. L. Berg, and M. Bansal.  Revealing single frame bias for video-and-language learning.  arXiv preprint arXiv:2206.03428, 2022.
  * [18] J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In CVPR, 2021.
  * [19] D. Li, J. Li, H. Li, J. C. Niebles, and S. C. Hoi.  Align and prompt: Video-and-language pre-training with entity prompts.  arXiv preprint arXiv:2112.09583, 2021.
  * [20] J. Li, D. Li, C. Xiong, and S. Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022.
  * [21] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In ECCV, 2020.
  * [22] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  ACL, 2021.
  * [23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [24] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang.  Gpt understands, too.  arXiv preprint arXiv:2103.10385, 2021.
  * [25] Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman.  Use what you have: Video retrieval using representations from collaborative experts.  BMVC, 2019.
  * [26] H. Luo, L. Ji, B. Shi, H. Huang, N. Duan, T. Li, J. Li, T. Bharti, and M. Zhou.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  arXiv preprint arXiv:2002.06353, 2020.
  * [27] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille.  Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN).  CoRR, abs/1412.6632, 2014.
  * [28] O. Michel, R. Bar-On, R. Liu, S. Benaim, and R. Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  arXiv preprint arXiv:2112.03221, 2021.
  * [29] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV, 2019.
  * [30] R. Mokady, A. Hertz, and A. H. Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [31] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu.  Bleu: a method for automatic evaluation of machine translation.  In ACL, 2001.
  * [32] O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In ICCV, 2021.
  * [33] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In CVPR, 2016.
  * [34] J. Perez-Martin, B. Bustos, and J. Perez.  Improving video captioning with temporal composition of a visual-syntactic embedding.  In WACV, 2021.
  * [35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  ICML, 2021.
  * [36] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [37] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap.  A simple neural network module for relational reasoning.  In Advances in neural information processing systems, pages 4967–4976, 2017.
  * [38] I. Schwartz, A. G. Schwing, and T. Hazan.  High-order attention models for visual question answering.  NIPS, 2017.
  * [39] I. Schwartz, A. G. Schwing, and T. Hazan.  A simple baseline for audio-visual scene-aware dialog.  In CVPR, 2019.
  * [40] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, and K. Keutzer.  How much can clip benefit vision-and-language tasks?  arXiv preprint arXiv:2107.06383, 2021.
  * [41] T. Shen, T. Lei, R. Barzilay, and T. Jaakkola.  Style transfer from non-parallel text by cross-alignment.  NIPS, 2017.
  * [42] T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  EMNLP, 2020.
  * [43] K. Simonyan and A. Zisserman.  Two-Stream Convolutional Networks for Action Recognition in Videos.  In NIPS, 2014.
  * [44] Y. Su, T. Lan, Y. Liu, F. Liu, D. Yogatama, Y. Wang, L. Kong, and N. Collier.  Language Models Can See: Plugging Visual Controls in Text Generation.  arXiv e-prints, page arXiv:2205.02655, May 2022.
  * [45] Y. Su, T. Lan, Y. Liu, F. Liu, D. Yogatama, Y. Wang, L. Kong, and N. Collier.  Language models can see: Plugging visual controls in text generation.  arXiv preprint arXiv:2205.02655, 2022.
  * [46] M. Tang, Z. Wang, Z. Liu, F. Rao, D. Li, and X. Li.  Clip4caption: Clip for video caption.  In MM, 2021.
  * [47] Y. Tewel, Y. Shalev, I. Schwartz, and L. Wolf.  Zero-shot image-to-text generation for visual-semantic arithmetic.  CVPR, 2022.
  * [48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [49] R. Vedantam, C. L. Zitnick, and D. Parikh.  Cider: Consensus-based image description evaluation.  In CVPR, 2014.
  * [50] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh.  Universal adversarial triggers for attacking and analyzing NLP.  EMNLP, 2019.
  * [51] Z. Wu, T. Yao, Y. Fu, and Y.-G. Jiang.  Deep Learning for Video Classification and Captioning.  arXiv e-prints, page arXiv:1609.06782, Sept. 2016.
  * [52] D. Xu, J. Xiao, Z. Zhao, J. Shao, D. Xie, and Y. Zhuang.  Self-supervised spatiotemporal learning via video clip order prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10334–10343, 2019.
  * [53] H. Xu, G. Ghosh, P.-Y. Huang, D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  2021\.
  * [54] J. Xu, T. Mei, T. Yao, and Y. Rui.  Msr-vtt: A large video description dataset for bridging video and language.  CVPR, 2016.
  * [55] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057, 2015.
  * [56] X. Yang, K. Tang, H. Zhang, and J. Cai.  Auto-encoding scene graphs for image captioning.  In CVPR, 2019.
  * [57] T. Yao, Y. Pan, Y. Li, and T. Mei.  Exploring visual relationship for image captioning.  In ECCV, 2018.
  * [58] H. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu.  Video paragraph captioning using hierarchical recurrent neural networks.  In CVPR, 2016.
  * [59] L. Yu, W. Zhang, J. Wang, and Y. Yu.  Seqgan: Sequence generative adversarial nets with policy gradient. arxiv e-prints, page.  AAAI, 2017.
  * [60] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao.  Vinvl: Revisiting visual representations in vision-language models.  In CVPR, 2021.
  * [61] B. Zhou, A. Andonian, and A. Torralba.  Temporal relational reasoning in videos.  arXiv preprint arXiv:1711.08496, 2017.
  * [62] L. Zhu and Y. Yang.  Actbert: Learning global-local video-text representations.  In CVPR, 2020.
  * [63] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving.  Fine-tuning language models from human preferences.  arXiv preprint arXiv:1909.08593, 2019.

##  Appendix A Appendix

In this appendix we provide additional results and ablations studies.

|  | Supervised Metrics | Unsupervised Metrics
---|---|---|---
Method | Prefix | B@4 | M | C | R | S | $\text{CLIP-S}^{\text{Ref}}$ | CLIP-S | BLIP-S | PP
ZeroCap [47] | None | 0.021 | 0.1 | 0.139 | 0.207 | 0.051 | 0.760 | 0.821 | 0.604 | 109.959
ZeroCap [47] | ‘A’ | 0.026 | 0.116 | 0.145 | 0.276 | 0.054 | 0.771 | 0.845 | 0.611 | 33.661
ZeroCap [47] | ‘Image of a’ | 0.029 | 0.12 | 0.131 | 0.268 | 0.055 | 0.778 | 0.870 | 0.605 | 25.737
Ours | None | 0.024 | 0.127 | 0.200 | 0.239 | 0.071 | 0.791 | 0.852 | 0.652 | 20.412
Ours | Random | 0.022 | 0.127 | 0.172 | 0.228 | 0.073 | 0.798 | 0.885 | 0.651 | 19.049

Table 4: Quantitative results for image captioning on the MS-COCO test set
using different prefixes.

##  Appendix B Quantitative Results

###  B.1 Image Captioning

Tab. 4 shows results for single-image captioning, evaluated on the MS-COCO
test-set [23]. We compare our method with another zero-shot method, ZeroCap.
Unlike the baseline, our method is trained with a perturbed prefix instead of
‘Image of a’. As a result, our method is more robust to prefix changes.
Notably, the perplexity score is significantly higher when the prefix is
removed from the baseline sentence (109.959 vs. 25.737). Furthermore, compared
to the baseline’s 0.870 CLIP score, our method has a higher CLIP score of
0.885. CLIPScoreRef is also improved (0.778 vs. 0.798), which means that our
caption matches human references better. In particular, we optimize complete
sentences, resulting in a significant improvement in language fluency (19.049
vs. 25.737).

###  B.2 Image Set Captioning

In order to stress-test our method, we consider the task of captioning random
image sets. The goal is to describe a set of images with one coherent
sentence. We use the MS-COCO test-set [23] for images. The number of images
varies between one and four, one being the conventional image captioning task.

It is expected that the more heterogeneous the image set, the harder it is to
generate a coherent caption. To quantify this, we measure the homogeneity of a
set by the average CLIP score between the human captions of each image and the
rest of the images in the set. Intuitively, this tells us whether the images
depict similar concepts or not. In addition to comparing sets by their size,
we also differentiate them based on their level of homogeneity. The results
are shown in Fig. 6. The graphs reveal that the CLIP score increases with
homogeneity, which is expected, since a single sentence can describe
homogeneous images better. Our approach has a higher CLIP score at all levels
of homogeneity and for all set sizes.

In Fig. 7, an experiment similar to the one above, based on BERT-based
perplexity, is conducted to measure language quality. We find that our method
produces much better sentences. A particularly interesting case is that of
homogeneous pairs (i.e., homogeneity level of 0.8). Only in this case, which
involves two very similar images, does ZeroCap perform as well as we do (logPP
of 3.0 vs. 3.5). This highlights the ability of our approach to generate
coherent sentences that describe a set of images across various challenges.

(a) Two images

(b) Three images

(c) Four images

Figure 6: CLIP-score for different sets of images varying by size and set
homogeneity.

(a) Two images

(b) Three images

(c) Four images

Figure 7: BERT-based perplexity score for different sets of images varying by
size and set homogeneity.

##  Appendix C Ablation Study

To assess different hyperparameters, we use the MSVD [51] validation set,
which consists of 100 videos. We examine two properties: (i) Video
correspondence, which we examine with the Retrieval score, and (ii) language
fluency, which we analyze with the BERT perplexity score. Additionally, we
report CLIP Score and BLIP score, which measure image correspondence with the
selected frames.

In Fig. 8, we study different values for $\lambda$, which controls the trade-
off between CLIP loss ($\mathcal{L}_{\text{CLIP}}$) and language fluency loss
($\mathcal{L}_{\text{PLM}}$). Increasing the value of $\lambda$ decreases the
Retrieval score. Our results show that $\lambda=0.8$ provides a good trade-off
between image correspondence and language fluency (i.e., low perplexity).

In Fig. 9, we ablate the learning rate (i.e., $\alpha$). Since optimization
occurs during inference, the number of iterations is fixed, so a higher
learning rate ensures convergence. In our experiments, we use $\alpha=0.1$,
which has the lowest perplexity, and the highest Retrieval score. Note that
the graphs might be misleading due to the wide range of values. The method is
relatively stable to this parameter.

In Fig. 10, we study different prompts. In our method, we perturbed a prompt
for each generated sentence to increase robustness to different scenarios
(e.g., image set captioning and videos). Note that while the option of no
prefix at all results in good performance, we find it less focused for the
task of visual captioning.

In Fig. 11, we assess our CLIP-based sampling method. Our method employs
CLIP’s visual encoder to compute image similarity. The Retrieval score
increases, as can be expected, with the CLIP image similarity. The perplexity
score is relatively stable, but there is a trade-off between the two.

In Fig. 12, we demonstrate how the CLIP score progresses during the generation
process. We report the following statistics: (i) Mean is the average CLIP
score at the given iteration across the set (ii) Max is the maximum CLIP score
at the given iteration across the set (iii) Best Mean is the best mean score
up to this iteration. The challenge of fitting to multiple visual cues can
cause performance instability during optimization. Thus, we suggest selecting
the sentence with the highest CLIP score from all the generated sentences.

##  Appendix D Qualitative Analysis

In Fig. 13 we demonstrate the ability of our model to caption images while
incorporating real-world knowledge. We provide comparison with two baseline
zero-shot image captioning models: ZeroCap [47] and MAGIC [44]. The baseline
models fail to describe the images with fluent natural language, nor do they
provide real-world information. Our model recognizes Bill Gates, the Great
Wall of China and the map of Italy, and incorporates that knowledge in rich
descriptive captions.

In Fig. 15 we illustrate the progression of captions during the optimization
process. The left image set shows two very different pictures of a toy bear
and a baseball game. Earlier captions discuss the crowd and the dinner
separately. The eighth iteration improves grounding, and the method recognizes
the baseball game. A coherent narrative is built in the 16th iteration. It is
described as a table of a pitcher at a dinner party. There are baseball cards
on the table, and bears serve as a metaphor for phrasing a quote. For the
right image set, after four iterations our method generates a caption that
includes the word Pyongyang as the location and the word ’wildlife’. At the
eighth iteration, the caption identifies the animal as a bird. As a result of
detecting Pyongyang as the location, the bird is described as being from the
DPRK. A reference is also made to the flowers.

In Fig. 15 we illustrate captions generated for sets of various sizes. We are
able to identify and describe the content of two images even if there is no
significant correlation between them. A stop sign and surfing images are
translated to “Surfing stops…”, while pictures of a toilet and ladies in
formal attire are captioned with “The toilets at the wedding reception.”.
Also, pictures of a sheep and a birthday cake are captioned with “Sheep’s
birthday…”.

When three images can be described with a coherent story, the model can do so.
As an example, for a set of images of a bus, a hotel bed, and a beach, our
method generates the caption: “Photo of bus driver sleeping on the beach from
the hotel.”. This caption grounds all the images while still creating a
plausible narrative. In addition, even when real-world knowledge is necessary,
e.g., a picture of Obama, the caption relates to it.

Our method was able to produce a coherent narrative even when dealing with a
complex case of four images. It describes a narrative of an image of birds
taken while cycling in Melbourne. We note that ZeroCap’s sentences tends to
create an irrational context, e.g., “Captive Obama…,” which is perhaps the
result of token-based optimization rather than sentence-based optimization.

In Fig. 16, we demonstrate our method’s zero-shot image captioning
capabilities. We compare with ZeroCap. We find that ZeroCap’s captions are
more direct, whereas the narrative of our captions is more natural. For
example, in the first row, on the left, the captions describe the girls and
indicate that it is their summer vacation, whereas ZeroCap mentions what
appears in the image. These results might come from the way we construct
sentences. By letting the PLM construct sentences, we improve language
fluency. ZeroCap, on the other hand, alternates each token to correspond to
the image, which might hinder the language.

In Fig. 17, we illustrate the CLIP-based mechanism we use to pick novel and
diverse frames. As a result of using CLIP image similarity, the method is able
to find frames with a very different content, e.g. where the environment or
objects change. We highlight the selected frames with a red border. For
example, the first row contains a frame depicting a pitcher, followed by a
frame showing the catcher. Frames following this one are ignored until the
ball is hit. In the following video, only four frames are selected, filtering
out many repetitions. The strategy also works with animations, as shown in the
third video.

In Fig. 19, we show more examples for the full generation process for videos.
We present the frames selected by our CLIP-based sampling method for each
video. Additionally, we report BERT-based perplexity score and CLIP score. The
low perplexity score indicates that early sentences have good language, but
subsequent sentences improve the CLIP score significantly. Our method can
ground objects and generate coherent sentences in various contexts.

In Fig. 20, we illustrate the evolution of sentences, using two images.
Interestingly, the method uses stories to weave the photos into a coherent
story. For instance, the image of prison and a bedroom photo results in a
caption about a prisoner’s bedroom.

In Fig. 21, three images are employed, and the generation process is
displayed. Often, creating a coherent sentence from three images is too
challenging. Therefore, in those cases, it is better to choose the sentence
based on the perplexity indicator rather than using the CLIP score. Thus, the
language will be fluent as it describes a story line without describing
everything in every image. Fig 22 shows the same phenomenon when there are
four images.

Figure 8: Ablation study for the hyper-parameter $\lambda$.

Figure 9: Ablation study for the learning rate, i.e., $\alpha$.

Figure 10: Ablation study for different prompts.

Figure 11: Ablation study for the CLIP-based frame selection method. We ablate
different threshold values used to pick significant frames.

(a) Two images

(b) Three images

(c) Four images

Figure 12: CLIP Score progress over the generation process. Figure 13:
Examples of our image captions on examples that require real-world knowledge,
with two zero-shot image captioning baselines.

Figure 14: Evolution of image pair captions. We show the sentence with the
highest CLIP score at different generation iterations.

Figure 15: Examples of our image set captioning, for different set sizes. We
compare our method with ZeroCap, another zero-shot method. Figure 16: Examples
of image captions. Figure 17: Illustration of our CLIP-based sampling
strategy. The picked frames are outlined in red. Figure 18: Example of harsh
language being generated by our model. This illustrates a limitation of web-
scale models. Figure 19: The evolution of captions for videos. (below and for
multiple pages) Figure 20: The evolution of captions for two images in an
image set. (below and for multiple pages) Figure 21: The evolution of captions
for three images in an image set.(below and for multiple pages) Figure 22: The
evolution of captions for four images in an image set. (below and for multiple
pages)