# Large-scale Vision-Language Models Learn Super Images
for Efficient and High-Performance Partially Relevant Video Retrieval

Taichi Nishimura, Shota Nakada, Masayoshi Kondo
LY Corporation

###### Abstract

In this paper, we propose an efficient and high-performance method for
partially relevant video retrieval (PRVR), which aims to retrieve untrimmed
long videos that contain at least one relevant moment to the input text query.
In terms of both efficiency and performance, the overlooked bottleneck of
previous studies is the visual encoding of dense frames. This guides
researchers to choose lightweight visual backbones, yielding sub-optimal
retrieval performance due to their limited capabilities of learned visual
representations. However, it is undesirable to simply replace them with high-
performance large-scale vision-and-language models (VLMs) due to their low
efficiency. To address these issues, instead of dense frames, we focus on
super images, which are created by rearranging the video frames in a $N\times
N$ grid layout. This reduces the number of visual encodings to
$\frac{1}{N^{2}}$ and compensates for the low efficiency of large-scale VLMs,
allowing us to adopt them as powerful encoders. Surprisingly, we discover that
with a simple query-image attention trick, VLMs generalize well to super
images effectively and demonstrate promising zero-shot performance against
SOTA methods efficiently. In addition, we propose a fine-tuning approach by
incorporating a few trainable modules into the VLM backbones. The experimental
results demonstrate that our approaches efficiently achieve the best
performance on ActivityNet Captions and TVR.

##  1 Introduction

With the explosive increase of videos on the web, text-to-video retrieval
(T2VR), a task of searching relevant videos to the input text query, has
garnered significant attention in computer vision and natural language
processing [6, 8, 16, 19, 27, 35, 40]. Current T2VR assumes that videos are
pre-trimmed with short duration and that the input text queries are single
sentences fully relevant to the videos. However, in practice, videos are
generally untrimmed with long duration and contain multiple moments [22, 46,
45, 4]. For this reason, the input query typically pertains to specific
moments within the videos, rather than encompassing their entire content. To
fit T2VR to the real-world scenario, partially relevant video retrieval (PRVR)
was proposed [10], which aims at retrieving videos that contain at least one
relevant moment with the query (Figure 1 (a)). For practical use for PRVR, it
is essential to achieve high computational efficiency and retrieval
performance. To address this goal, previous studies [10, 11] proposed a method
to link a text query with the dense frame representations.

Figure 1: (a) Overview of the PRVR task. (b) The key concept of our approach.
Super images are created by rearranging the frames in a $N\times N$ grid
layout, enabling the model to reduce the number of visual encodings. With a
simple query-image attention trick, large-scale VLMs generalize well to super
images and show promising performance against the SOTA methods.

In terms of both efficiency and performance, the important but overlooked
bottleneck of previous methods is the visual encoding of dense frames. From an
efficiency perspective, dense frames require a model to encode the increased
number of frames, leading to a rise in computation costs especially for long
videos. To relieve them, previous methods chose lightweight backbones such as
ResNet152 [17] and CLIP-B/32 [33]. This backbone choice may yield performance
issues due to its limited capabilities of learned visual representations
compared with large-scale vision-and-language models (VLMs), such as CLIP-L/14
and EVA CLIP [14]. A recent study demonstrated that VLMs follow a scaling law
[21], wherein increased model sizes correlate with enhanced performance on
downstream tasks, thus larger VLMs are expected to improve the retrieval
performance on PRVR. However, simply replacing the backbone with such VLMs is
undesirable because they suffer from low efficiency, consuming much higher
GFLOPs. For example, the EVA02-L-14 [14] visual backbone consumes 162.0 GFLOPs
to encode a single image, which is considerably higher than the lightweight
visual backbones of ResNet152 ($=$11.6 GFLOPs) and CLIP-B/32 ($=$8.8 GFLOPs).
Therefore, a dilemma arises in balancing computational efficiency and high
performance when leveraging large-scale VLMs, and one potential solution is to
reduce the number of visual encodings. The most famous approach of such less
is more ideas is sparse sampling [24, 25, 39], but is not suitable for PRVR
because it requires a model to focus on local moments in the long videos, and
sparse sampling miss them.

In this study, we overcome these difficulties and propose an efficient and
high-performance PRVR method that employs large-scale VLMs (Figure 1 (b)).
Instead of encoding dense frames or relying on sparse sampling, we focus on
super images [12], which are created by rearranging the video frames in a
$N\times N$ grid layout. This reduces the number of visual encodings to
$\frac{1}{N^{2}}$ and compensates for the low efficiency of large-scale VLMs,
allowing us to adopt them as a powerful image-text encoder. Then, two natural
questions arise: (1) how much do large-scale VLMs generalize well to super
images for PRVR? and if yes, (2) how can we achieve an efficient and best-
performance method by combining them and super images? In terms of (1),
although large-scale VLMs generalize to various kinds of images, such as
natural images, computer screens, cartoons, and red circles in images [20, 42,
34], no previous research reported whether they can handle super images. In
terms of (2), previous studies validated the high efficiency and good
performance of using super images in uni-modal vision tasks, such as action
recognition and deep fake detection [12, 44, 39], but it remains unclear
whether the super images are effective for the cross-modal domain.

To answer these research questions, we investigate two scenarios for the PRVR
task: zero-shot and fine-tuning settings. Surprisingly, with a simple query-
image attention trick, large-scale VLMs can represent super images effectively
and demonstrate promising zero-shot performance against SOTA methods that use
lightweight visual backbones on public datasets, ActivityNet Captions [22] and
TVR [23]. Moreover, the fine-tuning approach enhances the capabilities of VLMs
and achieves the best performance by introducing minimal trainable modules.
The insights gained from varying the grid size, input image resolution, and
backbone VLMs are two-fold: (1) the grid size and image resolution are trade-
off parameters between retrieval performance and computation costs, and (2)
the choice of VLMs significantly influences the overall performance.

##  2 Related work

###  2.1 Text-to-video retrieval

T2VR [6, 8, 16, 19, 27, 35, 40, 9] aims at searching relevant videos with the
input text query from a collection of pre-trimmed videos with short duration.
Traditional T2VR approaches encode videos and queries, convert them into joint
embedding spaces, and compute the cross-modal similarity. Recently, as image-
text VLMs [33, 14] have strong capabilities of zero-shot text-to-image
retrieval performance, researchers have made tremendous efforts in training
video-text VLMs [31, 2, 43, 24]. They reported that pre-trained video-text
VLMs also achieved outstanding performance on the zero-shot T2VR tasks.

T2VR assumes that videos are pre-trimmed and the queries are fully relevant to
them. However, in practice, videos are untrimmed with multiple moments [22,
46, 45, 4], and the queries are partially relevant to them. To fit the task
for the practical scenario, PRVR was proposed [10], which aims at searching
videos partially relevant to the input query. Previous studies tackled this
task by employing fine-grained matching between the text queries and frame-
level dense representations. Dong _et al_. [10] proposed multi-scale
similarity learning (MS-SL), which constructs the multiple clips from the
encoded frame-level representations as candidate moments and computes the
cross-modal similarity between the clips and text queries. In [11], inspired
by VLM capabilities, they proposed DL-DKD, where student branches learn the
cross-modal similarity distribution between the frames and input query from
the teacher branch of CLIP-B/32, a lightweight VLM backbone. Note that they
also reported that training a model only with CLIP-B/32 features based on [3]
does not reach the best performance because CLIP-B/32 fails to capture the
matching between the query and untrimmed videos that contain mixed relevant
and irrelevant moments to the query.

In previous studies, large-scale VLMs are not tested on PRVR because of their
high computation costs to extract frame-level representations. This study
addresses this problem by using super images to reduce the number of visual
encodings and enable us to adopt large-scale VLMs.

###  2.2 Summarized video representations

To mitigate computational costs, researchers have been pursuing effective
video representations for downstream tasks. Summarized video representations
[7, 5, 32, 37, 36, 12, 44] is one of the research directions that focuses on
extracting compressed video features without compromising the essential
information for downstream tasks. The mainstream is to represent videos as
single images by summarizing RGB and motion in a video. Representative
approaches include dynamic image network [5], informative frame synthesis
[32], adaptive weighted spatio-temporal distillation [37], and adversarial
video distillation [36].

Instead of representing videos as single images, super images for action
recognition (SIFAR) [12] proposed super images created by rearranging video
frames. They reported that pre-trained image encoders (e.g., Swin Transformer
[29]) fine-tuned on super image representations can achieve higher performance
with smaller GFLOPs than the SOTA methods. The idea of super images was
exported into deep fake detection [44], yet not validated on cross-modal
domains. This work is the first attempt to utilize super images for PRVR,
demonstrating the effectiveness of super images on the cross-modal domains.

##  3 Approach

In this section, we describe our zero-shot and fine-tuning approaches. We
first introduce background on super images in Section 3.1 and then describe
the zero-shot and fine-tuning approaches in Section 3.2 and 3.3. We emphasize
that the novelty of our work is to combine large VLMs and super images for
efficient and high-performance PRVR, and the network architecture itself is
not novel. In Section 4.2.1 and 4.2.2, we discover that zero-shot approaches
efficiently achieve promising performance against the SOTA methods trained on
each dataset and fine-tuned versions significantly outperform them.

###  3.1 Preliminary: super images

Figure 2: Examples of square grid layout used in [12]. This study uses
$2\times 2$, $3\times 3$, and $4\times 4$ square layouts in the experiments.
Figure 3: A Comparison of the original video frames and the created super
images in different layouts: $2\times 2$, $3\times 3$, and $4\times 4$.

Super images were first proposed in [12] for action recognition. Given
sequential video frames, they are ordered by a pre-defined layout to form a
super image. Figure 2 shows a square grid layout that SIFAR used. They
reported that among various layouts, the square layout achieved the best
performance across different datasets because it may encourage the models to
learn temporal relationships between frames. As with SIFAR, this work uses the
same layout.

To create super images, SIFAR first chooses the number of sampling frames $M$,
samples frames uniformly, and forms a super image by placing them in order
onto a grid of size $(N-1)\times N$ when $M<(N-1)\times N$ or $N\times N$ when
$M\geq(N-1)\times N$ where $N=\lceil\sqrt{M}\rceil$. Empty images are padded
at the end if the grid is not full. Based on this idea, for example, 8 frames
and 16 frames are fit into $3\times 3$ and $4\times 4$ layouts, respectively.

SIFAR sparsely samples frames by choosing $M$ beforehand and creates single
super images that capture the global content of videos. However, it is not
suitable for our task because PRVR requires models to capture local moments in
the videos, and single super images via sparse sampling likely miss them. We
thus set frame per second (FPS), sample frames, and create sequential super
images by fitting them into the pre-defined grid layout. Figure 3 shows a
comparison of the original video frames and created sequential super images.
We can observe that when increasing a grid size $N$, the number of super
images decreases quadratically and objects in each grid become smaller. In
Section 4.2.1, we delve into this matter, exploring the optimal grid size that
enables models to achieve high performance efficiently.

###  3.2 Zero-shot approach

Figure 4: Overview of the zero-shot QASIR, which first encodes super images
and a text query into vectors
$(\mbox{\boldmath{$Z$}}_{v},\mbox{\boldmath{$z$}}_{t})$, computes a query-
attentive visual vector $\mbox{\boldmath{$\hat{z}$}}_{v}$, and finally
calculates the cosine similarity
$\cos(\mbox{\boldmath{$\hat{z}$}}_{v},\mbox{\boldmath{$\hat{z}$}}_{t})$ as a
similarity score.

In this setting, we tackle the task of retrieving super images from the
corresponding partially relevant query without fine-tuning each dataset. Let
$(\mbox{\boldmath{$V$}},\mbox{\boldmath{$t$}})$ be positive pairs in the test
set $\mathcal{D}_{test}$, where
$\mbox{\boldmath{$V$}}=(\mbox{\boldmath{$v$}}^{1},\mbox{\boldmath{$v$}}^{2},\ldots,\mbox{\boldmath{$v$}}^{k},\ldots,\mbox{\boldmath{$v$}}^{K})$
and $t$ represents super images and text query, respectively ($K$ is the
number of super images). Given $t$, our task is to retrieve corresponding $V$
from a collection of videos, yet partial images correspond to the query on
PRVR, rather than matching the whole content of $V$ and $t$. Therefore,
globally aggregating $V$ does not perform well.

To address this problem, this study proposes a simple yet effective trick,
namely query-attentive super image retrieval (QASIR), which aggregates $V$ by
attending $t$ to compute similarity with an attention mechanism [1] (Figure
4). Specifically, the vision and text branches of VLMs convert super images
$V$ and a query $t$ into their latent representations
$\mbox{\boldmath{$Z$}}_{v}\in\mathbb{R}^{K\times d}$ and
$\mbox{\boldmath{$z$}}_{t}\in\mathbb{R}^{d}$ as:

$\displaystyle\mbox{\boldmath{$Z$}}_{v}$ $\displaystyle=$
$\displaystyle(\mbox{\boldmath{$z$}}_{v}^{1},\mbox{\boldmath{$z$}}_{v}^{2},\ldots,\mbox{\boldmath{$z$}}_{v}^{k},\ldots,\mbox{\boldmath{$z$}}_{v}^{K}),$
(1) $\displaystyle\mbox{\boldmath{$z$}}_{v}^{k}$ $\displaystyle=$
$\displaystyle\mathrm{VLM_{v}}(\mbox{\boldmath{$v$}}^{k}),$ (2)
$\displaystyle\mbox{\boldmath{$z$}}_{t}$ $\displaystyle=$
$\displaystyle\mathrm{VLM_{t}}(\mbox{\boldmath{$t$}}),$ (3)

where $\mathrm{VLM}_{v}(\cdot),\mathrm{VLM}_{t}(\cdot)$ are the vision and
text branches of VLMs and $d$ represents the dimension of the latent vectors.
Then, the super image representations are aggregated into the query-attentive
visual vectors $\hat{\mbox{\boldmath{$z$}}}_{v}$ as follows:

$\displaystyle\alpha_{k}$ $\displaystyle=$
$\displaystyle\frac{\exp{(\mbox{\boldmath{$z$}}_{v}^{k}\cdot\mbox{\boldmath{$z$}}_{t})}}{\sum_{i}\exp{(\mbox{\boldmath{$z$}}_{v}^{i}\cdot\mbox{\boldmath{$z$}}_{t})}},$
(4) $\displaystyle\hat{\mbox{\boldmath{$z$}}}_{v}$ $\displaystyle=$
$\displaystyle\sum_{k}\alpha_{k}\mbox{\boldmath{$z$}}_{v}^{k}.$ (5)

The similarity scores are computed by cosine similarity between
$\hat{\mbox{\boldmath{$z$}}}_{v}$ and $\mbox{\boldmath{$z$}}_{t}$. Note that
no additional training parameters are introduced in this approach.

###  3.3 Fine-tuning approach

Figure 5: Overview of how to fine-tune QASIR. Given the textual query
$\mbox{\boldmath{$\hat{z}$}}_{t}$, the model computes
$\mbox{\boldmath{$\hat{z}$}}_{t}$-weighted super image vectors for the
positive $\mbox{\boldmath{$\hat{z}$}}_{v}^{+}$ and negative
$\mbox{\boldmath{$\hat{z}$}}_{v}^{-}$ pairs. Then, their cosine similarity
$\cos(\mbox{\boldmath{$\hat{z}$}}_{v}^{+},\mbox{\boldmath{$\hat{z}$}}_{t})$
and
$\cos(\mbox{\boldmath{$\hat{z}$}}_{v}^{-},\mbox{\boldmath{$\hat{z}$}}_{t})$ is
used for loss calculation.

Based on the QASIR, we propose a fine-tuning method to boost its retrieval
performance without harming the computational efficiency (Figure 5). We add
two modules to QASIR: feature adapters and temporal encoder.

The feature adapters are based on CLIP adapter [15], which freezes the
original VLM networks and trains additional small networks, e.g., multi-
layered (typically two or three) perceptrons (MLPs) with activation functions.
The researchers reported that compared with training all of the layers in
VLMs, adaptation-based methods can achieve promising performance with minimum
training costs, preventing the VLMs from catastrophic forgetting [18].

For simplicity, we focus on describing the vision branch. Instead of utilizing
the output vector $\mbox{\boldmath{$z$}}_{k}^{v}$ from VLMs, the vision
adapter computes $\bar{\mbox{\boldmath{$z$}}}_{v}^{k}$ by interpolating
$\mbox{\boldmath{$z$}}_{v}^{k}$ and the output vectors from MLPs as:

$\displaystyle\bar{\mbox{\boldmath{$z$}}}_{v}^{k}$ $\displaystyle=$
$\displaystyle\beta_{v}\mbox{\boldmath{$z$}}_{v}^{k}+(1-\beta_{v})\mathrm{MLP}_{v}(\mbox{\boldmath{$z$}}_{v}^{k}),$
(6)

where $\mathrm{MLP}_{v}(\cdot)$ represents the three layer MLPs with ReLU
functions and $\beta_{v}$ is a hyper parameter. Similarly, we prepare the text
branch, dubbed text adapter, which acquires
$\bar{\mbox{\boldmath{$z$}}}_{t}=\beta_{t}\mbox{\boldmath{$z$}}_{t}+(1-\beta_{t})\mathrm{MLP}_{t}(\mbox{\boldmath{$z$}}_{t})$,
where $\mathrm{MLP}_{t}(\cdot)$ is another MLPs that is same network structure
to $\mathrm{MLP}_{v}(\cdot)$. The architecture of MLPs is written in the
supplementary materials.

The temporal encoder is introduced to represent the temporal information of
super images. Modeling the temporal dependency is essential for video
understanding tasks [41], thus we add a single-layer Transformer [38] into the
vision side. Based on the super image vectors
$\bar{\mbox{\boldmath{$z$}}}_{v}^{k}$ from the vision CLIP adapter, we add
positional encoding (PE) and input them into the Transformer as:

$\displaystyle\dot{\mbox{\boldmath{$z$}}}_{v}^{k}=\mathrm{Transformer}(\bar{\mbox{\boldmath{$z$}}}_{v}^{k}+\mathrm{PE}(k)),$
(7)

where $\mathrm{Transformer(\cdot)}$ and $\mathrm{PE}(\cdot)$ represents the
Transformer layer and PE. Then, we acquire the query-attentive visual vectors
$\hat{\mbox{\boldmath{$z$}}}_{v}^{k}$ by replacing
$\mbox{\boldmath{$z$}}_{v}^{k}$ with $\dot{\mbox{\boldmath{$z$}}}_{v}^{k}$ in
Eq (4) and Eq (5). The similarity scores are computed between
$\hat{\mbox{\boldmath{$z$}}}_{v}^{k}$ and $\bar{\mbox{\boldmath{$z$}}}_{t}$.
Loss calculation. To fine-tune the model, as with CLIP [33], we use a
symmetric cross-entropy loss over the similarity scores. From an anchor query
vector $\hat{\mbox{\boldmath{$z$}}}_{t}^{+}$, let the positive and negative
super image vectors be
$\hat{\mbox{\boldmath{$z$}}}_{v}^{+},\hat{\mbox{\boldmath{$z$}}}_{v}^{-}$.
Conversely, from an anchor super image vector
$\hat{\mbox{\boldmath{$z$}}}_{v}^{+}$, let the positive and negative query
vectors be
$\hat{\mbox{\boldmath{$z$}}}_{t}^{+},\hat{\mbox{\boldmath{$z$}}}_{t}^{-}$. The
total loss of $\mathcal{L}$ is computed as follows:

$\displaystyle{\footnotesize\mathcal{L}}\\!\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\\!\\!{\footnotesize\mathcal{L}_{q}+\mathcal{L}_{i}},$
$\displaystyle{\footnotesize\mathcal{L}_{q}}\\!\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\\!\\!{\footnotesize\frac{1}{|\mathcal{B}|}\\!\\!\\!\\!\\!\sum_{(\mbox{\boldmath{$z$}}_{v}^{+},\mbox{\boldmath{$z$}}_{t}^{+})\in\mathcal{B}}\\!\\!\\!\\!\\!\\!\\!\\!\log{\left\\{\frac{\cos{(\mbox{\boldmath{$z$}}_{v}^{+},\mbox{\boldmath{$z$}}_{t}^{+})}}{\cos{(\mbox{\boldmath{$z$}}_{v}^{+},\mbox{\boldmath{$z$}}_{t}^{+})}+\sum_{\mbox{\boldmath{$z$}}_{v}^{-}\in\mathcal{N}}\cos{(\mbox{\boldmath{$z$}}_{v}^{-},\mbox{\boldmath{$z$}}_{t}^{+})}}\right\\}}},$
$\displaystyle{\footnotesize\mathcal{L}_{i}}\\!\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\\!\\!{\footnotesize\frac{1}{|\mathcal{B}|}\\!\\!\\!\\!\\!\sum_{(\mbox{\boldmath{$z$}}_{v}^{+},\mbox{\boldmath{$z$}}_{t}^{+})\in\mathcal{B}}\\!\\!\\!\\!\\!\\!\\!\\!\log{\left\\{\frac{\cos{(\mbox{\boldmath{$z$}}_{v}^{+},\mbox{\boldmath{$z$}}_{t}^{+})}}{\cos{(\mbox{\boldmath{$z$}}_{v}^{+},\mbox{\boldmath{$z$}}_{t}^{+})}+\sum_{\mbox{\boldmath{$z$}}_{t}^{-}\in\mathcal{N}}\cos{(\mbox{\boldmath{$z$}}_{v}^{+},\mbox{\boldmath{$z$}}_{t}^{-})}}\right\\}}},$

where $\mathcal{L}_{q},\mathcal{L}_{i}$ represents query-to-image/image-to-
query losses and $\mathcal{B},\mathcal{N}$ indicates the samples from mini-
batch and negatives, respectively.

##  4 Experiments

| Model name | Resolution | Vision backbone | Text backbone | Parameters (M)
---|---|---|---|---|---
| MS-SL | 224 | I3D (+ResNet152) | RoBERTa† | 433.3
SOTA | DL-DKD | 224 | I3D (+ResNet152, OpenAI CLIP-B/32∗) | RoBERTa† (+OpenAI CLIP-B/32∗) | 439.0
| QASIR-L14 | 224 | OpenAI CLIP-L/14 | 427.6 (-5.7)
| EVA-QASIR-L14 | 224 | EVA-02-CLIP-L/14 | 427.8 (-11.2)
| QASIR-L14-336 | 336 | OpenAI CLIP-L/14-336 | 427.9 (-11.1)
Zero-shot | EVA-QASIR-L14-336 | 336 | EVA-02-CLIP-L/14-336 | 428.1 (-10.9)
| QASIR-L14 | 224 | OpenAI CLIP-L/14 | 431.0 (-8.0)
Fine-tuned | QASIR-L14-336 | 336 | OpenAI CLIP-L/14-336 | 431.3 (-7.7)

Table 1: A comparison of models. Note that when training the SOTA models on
TVR, they utilize RoBERTa$\dagger$ fine-tuned on the queries and subtitle
sentences of TVR. ResNet152 was additionally used only for TVR, not for
ActivityNet Captions. DL-DKD uses CLIP-B/32∗ to transfer vision-and-language
knowledge into visual and textual backbone in the training phase. Red values
represent the decreased number of parameters (M) from DL-DKD.

###  4.1 Experimental settings

####  4.1.1 Datasets

We evaluate our methods on two PRVR datasets:

  * •

ActivityNet Captions [22] collected YouTube videos and annotated them with
event and sentence pairs. The average length of videos is 117.6 seconds, which
is longer than TVR ($=$76.2 seconds). Following [10, 11], 10,009 videos with
37,421 events are used for training and 4,917 videos with 17,505 events are
used for testing.

  * •

TV show Retrieval (TVR) [23] is a multimodal dataset, containing 21.8K TV show
videos with five sentences that describe a specific event in the video. As
with [10, 11], 17,435 videos with 87,175 events are used for training and
2,179 videos with 10,895 events are used for testing.

####  4.1.2 Evaluation metrics

Performance metrics. As with previous PRVR studies [10, 11], we use Recall@$K$
($K=1,5,10,100$), which computes the percentage of queries that correctly
retrieve positive videos in the top $K$ of the ranking list. We also report
the sum of all recalls (sumR) for overall comparison.
Computation cost metrics. To evaluate the computational costs of the models,
we report the total number of parameters (#parameters) for memory consumption
and video-text GFLOPs for throughput, which computes the total number of
floating point operations from visual/textual backbone encodings to video-text
similarity calculation. The accurate definition of video-text GFLOPs is
written in the supplementary material.

####  4.1.3 Comparative components

SOTA models with lightweight visual backbones. We compare our approaches with
two SOTA methods that reported the best scores on the PRVR task: MS-SL [10]
and DL-DKD [11]. They use the same pre-extracted features on both visual and
textual sides. On the visual side, I3D features extracted at 1 FPS and
concatenated features of I3D and ResNet152 at 15 FPS are utilized for
ActivityNet Captions and TVR, respectively. For the textual side, they use
RoBERTa [28] for query feature extraction for both datasets. For TVR, RoBERTa
was additionally fine-tuned on the queries and subtitle sentences of TVR. Note
that DL-DKD uses CLIP-B/32, a relatively lightweight VLM backbone only for the
training phase to transfer CLIP’s knowledge into the visual/textual features.
They reported that using only CLIP-B/32 features does not perform well on PRVR
and a hybrid approach of CLIP and CNN-based features can achieve the best
retrieval performance.
Grid sizes, image resolution, and VLM backbones. To investigate how the grid
sizes, image resolution, and VLM backbones affect the performance on PRVR, we
prepare for four zero-shot and two fine-tuned models listed in Table 1. A
comparison of different large-scale VLMs (OpenAI CLIP and EVA-02-CLIP) tests
whether the choice of VLM affects the PRVR performance. In addition, a
comparison of the input resolution of 224/336 and grid sizes of $1\times 1$,
$2\times 2$, $3\times 3$, and $4\times 4$ tests whether the visibility of
objects in super images affects the performance. Note that $1\times 1$ is
equivalent to the frame-level representations and means not using super
images.

####  4.1.4 Implementation details

We set 0.5 FPS for the ActivityNet Captions and 3 FPS for TVR to obtain frames
from the videos. Note that frames extracted at 3 FPS are officially
distributed by the TVR authors, but raw videos are not due to copyright
issues. For both vision and text adapters, we set the
$\beta_{v}=\beta_{t}=0.2$ as with [15] and hidden size to be 192. We fine-tune
the model using the AdamW optimizer [30] with an initial learning rate of
$0.0001$. We set the batch size to 64, continue training at most 50 epochs,
and use the weights that achieve the best performance on sumR.

###  4.2 Results

####  4.2.1 Zero-shot evaluation

| ActivityNet Captions | TVR
---|---|---
| #Frames | GFLOPs | R@1 | R@5 | R@10 | R@100 | sumR | #Frames | GFLOPs | R@1 | R@5 | R@10 | R@100 | sumR
MS-SL [10] | 118.2 | 3.4$\times$103 | 7.1 | 22.5 | 34.7 | 75.8 | 140.1 | 5677.2 | 7.3$\times$104 | 13.5 | 32.1 | 43.4 | 83.4 | 172.4
DL-DKD [11] | 118.2 | 3.4$\times$103 | 8.0 | 25.0 | 37.5 | 77.1 | 147.6 | 5677.2 | 7.3$\times$104 | 14.4 | 34.9 | 45.8 | 84.9 | 179.9
QASIR-L14
1$\times$1 | 60.3 | 9.8$\times$103 | 14.2 | 31.7 | 42.7 | 77.0 | 165.6 | 229.4 | 3.7$\times$104 | 16.4 | 32.5 | 40.7 | 74.0 | 163.6
2$\times$2 | 15.5 | 2.5$\times$103 | 13.7 | 31.3 | 41.9 | 77.1 | 164.0 | 57.7 | 9.2$\times$103 | 14.1 | 29.6 | 37.4 | 72.5 | 154.0
3$\times$3 | 7.1 | 1.1$\times$103 | 12.4 | 29.4 | 39.5 | 75.0 | 156.4 | 23.9 | 4.1$\times$103 | 11.8 | 25.4 | 33.2 | 68.9 | 139.3
4$\times$4 | 4.2 | 7.0$\times$102 | 10.7 | 26.0 | 36.1 | 70.8 | 143.8 | 14.8 | 2.3$\times$103 | 8.4 | 19.0 | 26.3 | 61.6 | 115.4
EVA-QASIR-L14
1$\times$1 | 60.3 | 9.8$\times$103 | 17.3 | 36.6 | 48.0 | 79.2 | 181.0 | 229.4 | 3.7$\times$104 | 11.3 | 23.8 | 30.9 | 64.7 | 130.7
2$\times$2 | 15.5 | 2.5$\times$103 | 16.8 | 36.1 | 47.4 | 79.8 | 180.1 | 57.7 | 9.2$\times$103 | 9.2 | 20.5 | 27.1 | 61.6 | 118.4
3$\times$3 | 7.1 | 1.1$\times$103 | 15.0 | 33.8 | 44.9 | 77.9 | 171.7 | 23.9 | 4.1$\times$103 | 8.3 | 18.2 | 24.8 | 59.4 | 110.7
4$\times$4 | 4.2 | 7.0$\times$102 | 12.4 | 29.4 | 39.7 | 73.6 | 155.1 | 14.8 | 2.3$\times$103 | 5.3 | 13.4 | 18.8 | 51.2 | 88.8
QASIR-L14-336
1$\times$1 | 60.3 | 2.3$\times$104 | 14.6 | 32.8 | 43.8 | 77.9 | 169.1 | 229.4 | 8.7$\times$104 | 16.8 | 34.1 | 42.5 | 75.4 | 168.8
2$\times$2 | 15.5 | 5.9$\times$103 | 15.7 | 33.9 | 45.3 | 78.7 | 173.6 | 57.7 | 2.1$\times$104 | 17.2 | 33.3 | 42.1 | 76.1 | 168.7
3$\times$3 | 7.1 | 2.7$\times$103 | 14.8 | 32.3 | 44.0 | 77.5 | 169.6 | 23.9 | 9.8$\times$103 | 15.2 | 30.9 | 39.3 | 73.7 | 159.1
4$\times$4 | 4.2 | 1.6$\times$103 | 13.2 | 30.8 | 41.3 | 75.4 | 160.6 | 14.8 | 5.6$\times$103 | 12.4 | 26.6 | 34.5 | 69.9 | 143.4
EVA-QASIR-L14-336
1$\times$1 | 60.3 | 2.3$\times$104 | 18.6 | 38.3 | 49.2 | 80.7 | 186.8 | 229.4 | 8.7$\times$104 | 13.8 | 27.0 | 35.3 | 68.1 | 144.3
2$\times$2 | 15.5 | 5.9$\times$103 | 19.0 | 38.9 | 50.2 | 81.4 | 189.6 | 57.7 | 2.1$\times$104 | 12.8 | 25.7 | 33.1 | 67.4 | 138.9
3$\times$3 | 7.1 | 2.7$\times$103 | 17.5 | 36.7 | 48.4 | 80.0 | 182.6 | 23.9 | 9.8$\times$103 | 11.1 | 23.2 | 30.5 | 65.3 | 130.0
4$\times$4 | 4.2 | 1.6$\times$103 | 15.3 | 33.8 | 45.1 | 77.9 | 172.1 | 14.8 | 5.6$\times$103 | 8.9 | 19.5 | 26.5 | 61.5 | 116.4

Table 2: Zero-shot performance on the ActivityNet Captions and TVR dataset.
Bold values represent the highest scores among each VLM and underlined values
in video-text GFLOPs indicate that they are smaller than the SOTA methods.

| ActivityNet Captions | TVR
---|---|---
Fully-trained | GFLOPs | R@1 | R@5 | R@10 | R@100 | sumR | GFLOPs | R@1 | R@5 | R@10 | R@100 | sumR
MS-SL [10] | 3.4$\times$103 | 7.1 | 22.5 | 34.7 | 75.8 | 140.1 | 7.3$\times$104 | 13.5 | 32.1 | 43.4 | 83.4 | 172.4
DL-DKD [11] | 3.4$\times$103 | 8.0 | 25.0 | 37.5 | 77.1 | 147.6 | 7.3$\times$104 | 14.4 | 34.9 | 45.8 | 84.9 | 179.9
QASIR-L14
$1\times 1$ | 9.8$\times$103 |  18.9 (+10.9) |  41.0 (+16.0) |  53.2 (+15.7) |  84.8 (+7.7) |  197.9 (+50.3) | 3.7$\times$104 |  23.6 (+9.2) |  47.0 (+12.1) |  56.5 (+10.7) |  89.0 (+4.1) |  216.1 (+36.2)
$2\times 2$ | 2.5$\times$103 | 18.2 (+10.2) | 39.6 (+14.6) | 52.2 (+14.7) |  84.8 (+7.7) | 194.9 (+47.3) | 9.2$\times$103 | 23.0 (+8.6) | 45.4 (+10.5) | 56.3 (+10.5) | 88.9 (+4.0) | 213.6 (+33.7)
$3\times 3$ | 1.1$\times$103 | 16.5 (+8.5) | 37.5 (+12.5) | 49.9 (+12.4) | 83.2 (+6.1) | 187.1 (+39.5) | 4.1$\times$103 | 21.0 (+6.6) | 42.5 (+7.6) | 54.0 (+8.2) | 88.4 (+3.5) | 205.8 (+25.9)
$4\times 4$ | 7.0$\times$102 | 14.1 (+6.1) | 33.8 (+8.8) | 45.6 (+8.1) | 80.4 (+3.3) | 174.0 (+26.4) | 2.3$\times$103 | 16.3 (+1.9) | 35.7 (+0.8) | 46.4 (+0.6) | 84.6 (+0.3) | 182.9 (+3.0)
QASIR-L14-336
$1\times 1$ | 2.3$\times$104 |  19.7 (+11.7) | 41.4 (+16.4) |  53.9 (+16.4) | 85.1 (+8.0) |  200.1 (+52.5) | 8.7$\times$104 |  26.9 (+12.5) |  50.6 (+15.7) | 60.7 (+14.9) |  91.4 (+6.5) | 229.6 (+49.7)
$2\times 2$ | 5.9$\times$103 | 19.3 (+11.3) |  41.5 (+16.5) | 53.8 (+16.3) |  85.3 (+8.2) | 200.0 (+52.4) | 2.1$\times$104 | 26.7 (+12.3) |  50.6 (+15.7) |  61.4 (+15.6) |  91.4 (+6.5) |  230.1 (+50.2)
$3\times 3$ | 2.7$\times$103 | 18.7 (+10.7) | 40.6 (+15.6) | 52.7 (+15.2) | 84.4 (+7.3) | 196.4 (+48.8) | 9.8$\times$103 | 24.2 (+9.8) | 47.2 (+12.3) | 58.3 (+12.5) | 89.9 (+5.0) | 219.6 (+39.7)
$4\times 4$ | 1.6$\times$103 | 16.8 (+8.8) | 38.1 (+13.1) | 50.0 (+12.5) | 83.4 (+6.3) | 188.2 (+40.6) | 5.6$\times$103 | 21.6 (+7.2) | 43.3 (+8.4) | 54.3 (+8.5) | 88.7 (+3.8) | 208.0 (+28.1)

Table 3: Fine-tuning performance for QASIR-L14 and QASIR-L14-336 on the
ActivityNet Captions and TVR datasets. Bold and underlined rules are the same
as Table 2. Red scores represent the increased points from DL-DKD.

Do large-scale VLMs generalize well to super images? Table 2 shows the zero-
shot performance on the benchmark datasets. Surprisingly, the zero-shot QASIR
with super images achieves promising performance against SOTA methods, which
adopt lightweight visual backbones. In addition, half $2\times 2$ models
achieves comparable performance to $1\times 1$ ($\pm$3.0 sumR). These results
indicate that large-scale VLMs can generalize to super images.
Balance between the retrieval performance and computation costs. QASIR is an
efficient approach because its total #parameters are smaller than SOTA methods
and some models with increased grid sizes achieve better performance and
smaller video-text GFLOPs than them. It is worth noting that $2\times 2$ with
large-scale VLMs can reduce the computation costs to $\frac{1}{4}$ from
$1\times 1$ while minimizing the retrieval performance decrease. Therefore,
combining super images and large-scale VLMs strikes a commendable balance
between enhanced performance and computational costs.
Performance difference across datasets. QASIR consistently achieves higher
performance than the SOTA methods on ActivityNet Captions but falls short in
TVR. We notice that most nouns in queries are general objects (e.g., “man” and
“dog”) in ActivityNet Captions while they are character names (e.g.,
“Sheldon”) in TVR. Although the training data for OpenAI CLIP is private and
we cannot confirm this, one possible reason for the gap is that VLMs do not
know these TV show characters and fail to link the character names and their
visual representations.
Performance change when varying the grid size, input resolution, and VLM
backbones. When changing the grid size, input resolution, and backbone VLMs,
two insights emerge. First, the grid size and input resolution are key trade-
off parameters between the performance and computation costs. As the input
resolution increases and the grid size decreases, objects in each super image
are more visible, yielding improved retrieval performance. However, this
requires increased computation costs. Thus, it is essential to carefully
select the optimal grid size and input resolution based on use cases. Second,
the choice of VLMs has a substantial impact on overall performance. When
comparing QASIR-L14-336 and EVA-QASIR-L14-336, the EVA-based model outperforms
the CLIP-based one in ActivityNet captions; it falls short in TVR. EVA CLIP
averagely performs better than OpenAI CLIP on image downstream tasks but is
not consistently better in all of the tasks [13]. A careful selection of VLMs
is important depending on the datasets.
Is query attention the best?

Aggregation | R@1 | R@5 | R@10 | R@100 | sumR
---|---|---|---|---|---
Mean | 10.1 | 25.0 | 35.9 | 72.2 | 144.1
Max | 7.6 | 20.0 | 28.9 | 63.8 | 120.2
Query attention | 15.7 | 33.9 | 50.2 | 81.4 | 173.6

Table 4: Zero-shot $2\times 2$ QASIR-L14-336 performance when changing the
aggregation methods on mean, max, and query attention on the ActivityNet
Captions dataset.

To show the effectiveness of the query-image attention, we report the
performance change when replacing the aggregation methods with other global
pooling. Table 4 shows that the query attention significantly outperforms
other methods, indicating that attending local segments in super images with
reference to the input query is essential for PRVR.

| R@1 | R@5 | R@10 | R@100 | sumR
---|---|---|---|---|---
Sparse sampling | 0.02 | 0.08 | 31.0 | 75.7 | 106.8
Super images | 15.7 | 33.9 | 50.2 | 81.4 | 173.6

Table 5: A comparison of sparse sampling and $2\times 2$ super images. The
model and dataset are QASIR-L14-336 and ActivityNet Captions.

Sparse sampling or super images for PRVR? Table 5 shows a comparison of sparse
sampling and super images. For a fair comparison, we equalize the computation
costs by uniformly sampling $\frac{1}{4}$ frames from $1\times 1$ and compare
the performance with $2\times 2$. The results indicate that super images
outperform sparse sampling.

####  4.2.2 Fine-tuning evaluation

Figure 6: Examples of retrieved videos from different models on the
ActivityNet Captions dataset: MS-SL, zero-shot QASIR-L14-336, and its fine-
tuned version. The red and blue rectangles represent objects corresponding to
the words in the input query. The green frame borders indicate that the videos
are successfully retrieved from the input query. Other examples are displayed
in the supplementary materials.

Table 3 shows the fine-tuning performance on the benchmark datasets.
Naturally, the fine-tuned models outperform the zero-shot versions and SOTA
methods, indicating that our fine-tuning approach is appropriate for large-
scale VLMs to boost their performance. In line with the zero-shot experiments,
the fine-tuned $1\times 1$ achieve the best performance with huge computation
costs, but $2\times 2$ reduces them while maintaining a comparable performance
to $1\times 1$. In addition, if faster processing is required at the expense
of a slight decrease in performance, the fine-tuning approach enables us to
increase grid sizes, such as $3\times 3$ and $4\times 4$. Even with these
larger grid sizes, they consistently surpass the SOTA methods.

####  4.2.3 Moment-to-video performance

Figure 7: Moment-to-video (M/V) performance on ActivityNet Captions and TVR.
$2\times 2$ and $3\times 3$ QASIR-L14-336 are used.

As with previous PRVR studies [10, 11], we report grouped sumR based on the
moment-to-video ratio (M/V), which is computed as the moment’s length ratio in
the entire video duration. The smaller M/V indicates that the videos have less
relevant moments with irrelevant content, thereby retrieving them from the
corresponding queries is more challenging. As with [11], we compute the sumR
scores for three M/V settings, where the moments are short ($r\in(0,0,2]$),
middle ($r\in(0.2,0.4]$), and long ($r\in(0.4,1.0]$).

Figure 7 shows M/V results on ActivityNet Captions and TVR, where we reported
the scores of SOTA, zero-shot $2\times 2$ and $3\times 3$ QASIR-L14-336, and
their fine-tuned versions. The results demonstrate that the fine-tuned models
consistently outperform the SOTA methods in all settings. When comparing QASIR
models across different M/V ratios, we observe improved performance as the
ratio increases. This suggests that super images excel at retrieving videos
containing middle and long moments, rather than short moments. Our future work
is to improve the performance in the short moments as much as middle and long
moments.

####  4.2.4 Ablation study

| $\Delta_{flp}$ | $\Delta_{par}$ | R@1 | R@5 | R@10 | R@100 | sumR
---|---|---|---|---|---|---|---
Zero-shot | 0 | 0 | 15.9 | 34.8 | 45.8 | 78.9 | 175.4
VA | 1.1$\times$10-4 | 0.13 | 16.0 | 36.2 | 48.2 | 82.8 | 183.0
TA | 7.6$\times$10-6 | 0.13 | 18.1 | 39.6 | 52.2 | 85.0 | 194.9
T | 2.2$\times$10-3 | 3.15 | 18.4 | 40.1 | 52.2 | 84.7 | 195.4
VA+TA | 1.2$\times$10-4 | 0.26 | 19.1 | 41.2 | 53.4 | 84.8 | 197.6
VA+TA+T | 2.3$\times$10-3 | 3.41 | 19.3 | 41.5 | 53.8 | 85.3 | 200.0

Table 6: Ablation study on the fine-tuned $2\times 2$ QASIR-L14-336 on the
ActivityNet Captions dataset. VA, TA, T, $\Delta_{flp}$, $\Delta_{par}$
represents the vision adapter, text adapter, temporal encoder, and the
difference of GFLOPs and parameters (M) from the zero-shot model,
respectively.

Table 6 shows an ablation study on the fine-tuned QASIR-L14-336. The results
indicate that all components contribute to the retrieval performance. When
comparing the vision adapter, text adapter, and temporal encoder, two insights
emerge. First, the temporal encoder yields the highest performance increase
(+20 sumR) among them, but it also incurs the highest increase in
computational costs. Second, the text adapter strikes a commendable balance
between enhanced performance and computational costs. It achieves comparable
performance to the temporal encoder, with only a slight decrease of -0.6 in
sumR, while minimizing additional computational costs to a mere
7.6$\times$10-6 GFLOPs.

####  4.2.5 Qualitative results

Figure 6 shows two examples of the retrieved videos from different models: MS-
SL, zero-shot QASIR-L14-336, and its fine-tuned version. Although MS-SL
correctly recognizes objects in the videos, e.g., “backpack Sprayer” in (a),
it misses fine-grained word-level information, such as “A woman,” “the lower
part of a gate” in (a) and “young man with headphones” in (b). With a powerful
VLM ability, the QASIR-based models can capture such concepts and retrieve
videos that match the corresponding query. In addition, by fine-tuning the
model, we find that the model acquires a more fine-grained concept to search
for correct videos. In (b), the zero-shot QASIR retrieves the video of a man
pushing the grass cutter but discards other information. The fine-tuned QASIR
can retrieve the correct videos that fully match the corresponding text query.

##  5 Conclusion

In this study, we proposed an efficient and high-performance PRVR method by
combining super images and large-scale VLMs. The super images are created by
rearranging the frames onto the $N\times N$ grid layout and can reduce the
number of visual encodings to $\frac{1}{N^{2}}$. This compensates for the low
efficiency of large-scale VLMs, allowing us to adopt them as powerful
encoders. From our experiments, QASIR achieved promising performance in the
zero-shot setting and outperformed SOTA methods using lightweight backbones in
the fine-tuning setting while reducing the computation costs. In addition, we
obtained insights by varying model settings for optimal balance between the
retrieval performance and computation costs.

## Appendix

##  A MLP architecture of the feature adapters

Figure 8 shows the detailed architecture of MLPs in the vision adapter and
text adapter. As with [15], their architecture is an auto encoder, consisting
of two layers of linear with the ReLU activation function. Given visual and
textual representations $\mbox{\boldmath{$z$}}_{v}^{k}$ and
$\mbox{\boldmath{$z$}}_{t}$, the adapters compute their intermediate
representations $\mbox{\boldmath{$\bar{z}$}}_{v}^{k}$ and
$\mbox{\boldmath{$\bar{z}$}}_{t}$ by interpolating input vectors and outputs
of MLPs.

Figure 8: MLP architecture used in the vision and text adapter. They consist
of two layers of linear with ReLU activation functions.

##  B Accurate definition of video-text GFLOPs

Figure 9: Word clouds for ActivityNet Captions and TVR. The size of each word
is propotional to the frequency in the dataset.

The current PRVR methods [10, 11], including ours, contain three phases to
compute video-text similarity: (1) visual and textual backbones extract
features, (2) visual and textual encoders map them into a joint embedding
space, and (3) matching module computes their similarity. The video-text
GFLOPs computes the total number of floating point operations of these three
processes.

We formulate it using mathematical notations. Let $y$ be a query sentence and
$\mbox{\boldmath{$X$}}=(\mbox{\boldmath{$x$}}_{1},\ldots,\mbox{\boldmath{$x$}}_{l},\ldots,\mbox{\boldmath{$x$}}_{L})$
be a video containing $L$ frames where $\mbox{\boldmath{$x$}}_{l}$ is a $l$-th
frame (image) in the test set $\mathcal{D}_{test}$. Given
$(\mbox{\boldmath{$X$}},\mbox{\boldmath{$y$}})$, visual and textual backbones
$B_{x},B_{y}$ convert them into intermediate representations
$\mbox{\boldmath{$z$}}_{x},\mbox{\boldmath{$z$}}_{y}$. The visual and textual
encoders $E_{x},E_{y}$ further map them into a joint embedding space, yielding
their embedded representations
$\mbox{\boldmath{$\hat{z}$}}_{x},\mbox{\boldmath{$\hat{z}$}}_{y}$. Finally,
based on them, the matching module $M$ computes their similarity. The video-
text GFLOPs $G_{vt}$ computes the average of the total GFLOPs of all of these
processes in video-text pairs in the test set, and is defined as:

$\displaystyle G_{vt}(\mbox{\boldmath{$X$}},\mbox{\boldmath{$Y$}})$
$\displaystyle=$ $\displaystyle\frac{1}{|\mathcal{D}_{test}|}$
$\displaystyle\sum_{(\mbox{\boldmath{$X$}},\mbox{\boldmath{$y$}})\in\mathcal{D}_{test}}\biggl{\\{}\underbrace{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\sum_{l=1}^{L}G(B_{x}(\mbox{\boldmath{$x$}}_{l}))}+G(B_{y}(\mbox{\boldmath{$y$}})}_{\text{backbone
GFLOPs}})$
$\displaystyle+\underbrace{G(E_{x}(\mbox{\boldmath{$z$}}_{x}))+G(E_{y}(\mbox{\boldmath{$z$}}_{y}))}_{\text{encoder
GFLOPs}}$
$\displaystyle+\underbrace{G(M(\mbox{\boldmath{$\hat{z}$}}_{x},\mbox{\boldmath{$\hat{z}$}}_{y}))}_{\text{matching
module GFLOPs}}\biggr{\\}},$ (8)

where $|D_{test}|$ represents the number of total pairs of a video and query
in the test set and $G(\cdot)$ computes the GFLOPs of each process. Note that
the red-term GFLOPs of image backbone encoding is much higher than others
especially for long untrimmed videos because increasing frames $L$ yields
increased GFLOPs. The super images address this problem by reducing the number
of such image backbone encodings.

| TVR
---|---
TV show title | #Videos | R@1 | R@5 | R@10 | R@100 | sumR
The Big Bang Theory | 2,155 | 22.6 | 42.6 | 53.1 | 84.4 | 202.7
How I Met Your Mother | 745 | 24.8 | 43.4 | 54.6 | 87.2 | 210.1
Friends | 2,800 | 19.5 | 36.8 | 46.4 | 81.0 | 183.8
Grey’s Anatomy | 520 | 15.9 | 32.1 | 40.2 | 78.1 | 166.3
House | 2,310 | 11.9 | 24.3 | 31.6 | 66.0 | 133.9
Castle | 2,365 | 12.4 | 26.6 | 33.8 | 68.7 | 141.5
Total | 10,835 | 17.2 | 33.3 | 42.1 | 76.1 | 168.7

Table 7: Zero-shot retrieval performance on each TV show. We use the $2\times
2$ QASIR-L14-336 as our base model.

##  C Additional results

###  C.1 Performance gap analysis across ActivityNet Captions and TVR

In Section 4.2.1 in the main paper, we assume that the performance difference
across the datasets arises due to the noun gap. To demonstrate this, we show
word clouds111To generate word clouds, we use this library:
https://amueller.github.io/word_cloud/ generated from query sentences in the
test set on each dataset (Figure 9). As the word clouds suggest, general
objects, such as “man,” and “woman,” appear frequently in the ActivityNet
Captions [22], while the character names, such as “Beckett,” “Leonard,” and
“Sheldon” appear frequently in TVR [23]. Therefore, if VLMs do not know the
link between these names and their visual representations, they cannot
retrieve the corresponding videos from input queries successfully. This is the
reason for decreased zero-shot performance in TVR and increased fine-tuning
performance by enabling models to understand their linking and outperform
previous SOTA methods.
Performance across TV shows. Then, a question arises, How much do VLMs know
the TV shows that TVR covers? If VLMs do not see them completely, VLMs perform
poorly because of disjoint between characters and their visual
representations. To investigate them, we aggregate the zero-shot performance
on each TV show. Table 7 shows the result, presenting a large performance gap
across TV shows. There is a 76.2 sumR gap between the highest (How I Met Your
Mother) and lowest (House). Therefore, we infer that VLMs possess knowledge of
these TV shows to some extent, but their understanding is influenced by biases
that stem from their training dataset. Because the VLM backbone is OpenAI
CLIP-L/14-336 [33] pre-trained on private data, further analysis of the
training data to determine its inclusion of images from these TV shows is
infeasible.

###  C.2 Performance change w.r.t. query length

Figure 10: Performance aggregated on query length for three groups, short
$(S\leq 10)$, medium $(10<S\leq 15)$, and long $(S>15)$, where $S$ represents
query length, the number of words in the query.

The queries in the benchmark datasets have different lengths. We assume that
the query length $S$ affects the retrieval performance because shorter ones
are less informative than the longer ones to distinguish videos. To verify
this, we investigate the sumR aggregated on query length for three groups,
short ($S\leq 10$), medium ($10<S\leq 15$), and long ($15<S$). Figure 10
presents the performance change w.r.t query length. As we expected, all of the
models decrease their performance on short queries, rather than longer
queries. To address this issue, one of the promising ideas is to enhance
queries from human feedback. Interactive video retrieval was recently
introduced [26], which asks questions to humans to retrieve videos accurately.
We think that it is an interesting future direction to incorporate our idea of
combining super images and large-scale VLMs into interactive video retrieval.

###  C.3 Retrieval examples

Figure 11: Additional examples of retrieved videos from different models. We
provide two successful cases (a) and (b), and two failure cases (c) and (d).
red, blue, and magenta rectangles represent objects corresponding to the words
in the input query. The green frame borders indicate that the videos are
successfully retrieved from the input query.

Figure 11 shows additional retrieval examples. (a) and (b) are successful
cases in TVR. Because we display two ActivityNet Captions examples in the main
paper, we display two TVR examples in this supplementary material. (c) and (d)
are failure cases of QASIR. Here, we first discuss the successful cases in
TVR, then discuss failure cases and the limitations of our approach.
Successful cases in TVR. As described in Section C.1, if the VLMs know the
linking between characters and their visual representations, the zero-shot
model can retrieve videos correctly. (a) belongs to The Big Bang Theory, and
as shown in Table 7, the zero-shot model knows it and achieves strong
retrieval performance. Therefore, in (a), it correctly recognizes Sheldon and
Leonard and understands their relationship with Sheldon using a fake steering
wheel and Leonard standing behind him. MS-SL [10] fails to capture this
relationship, retrieving a video that Sheldon is standing behind Leonard. The
fine-tuning enables the model to understand videos that the zero-shot models
cannot capture. For example, in (b), the zero-shot model recognizes a pregnant
woman but the character is not Erica. The fine-tuned version can recognize
Erica correctly and understand the situation, where Erica is pregnant and
sitting at the table.
Failure cases and limitations. Although the fine-tuned model achieves higher
retrieval performance than MS-SL, it sometimes fails to retrieve videos. As
described in moment-to-video (M/V) retrieval performance in Section 4.2.3 in
the main paper, our approach tends to miss shorter ($r<0.2$) moments. The M/V
ratios of (c) and (d) are 0.07 and 0.09. Owing to dense frame representations,
MS-SL can capture such very short moments correctly but our approach misses
them. In addition to the temporal encoder in Section 3.3 in the main paper,
spatial attention on grids may be one of the approaches to this problem, which
is our future work.

## References

  * Bahdanau et al. [2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In _International Conference on Learning Representations_ , 2015.
  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1728–1738, 2021.
  * Bain et al. [2022] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  A clip-hitchhiker’s guide to long video retrieval.  _arXiv preprint arXiv:2205.08508_ , 2022.
  * Bao et al. [2021] Peijun Bao, Qian Zheng, and Yadong Mu.  Dense events grounding in video.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 920–928, 2021.
  * Bilen et al. [2016] Hakan Bilen, Basura Fernando, Efstratios Gavves, Andrea Vedaldi, and Stephen Gould.  Dynamic image networks for action recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3034–3042, 2016.
  * Chen et al. [2020] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu.  Fine-grained video-text retrieval with hierarchical graph reasoning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10638–10647, 2020.
  * Davis and Bobick [1997] James Davis and Aaron Bobick.  The representation and recognition of action using temporal templates.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2736–2744, 1997.
  * Dong et al. [2018] Jianfeng Dong, Xirong Li, and Cees GM Snoek.  Predicting visual features from text for image and video caption retrieval.  _IEEE Transactions on Multimedia_ , 20(12):3377–3388, 2018.
  * Dong et al. [2021] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang.  Dual encoding for video retrieval by text.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Dong et al. [2022] Jianfeng Dong, Xianke Chen, Minsong Zhang, Xun Yang, Shujie Chen, Xirong Li, and Xun Wang.  Partially relevant video retrieval.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 246–257, 2022.
  * Dong et al. [2023] Jianfeng Dong, Minsong Zhang, Zheng Zhang, Xianke Chen, Daizong Liu, Xiaoye Qu, Xun Wang, and Baolong Liu.  Dual learning with dynamic knowledge distillation for partially relevant video retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11302–11312, 2023.
  * Fan et al. [2022] Quanfu Fan, Chun-Fu Chen, and Rameswar Panda.  Can an image classifier suffice for action recognition?  In _International Conference on Learning Representations_ , 2022.
  * Fang et al. [2023a] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva-02: A visual representation for neon genesis.  _arXiv preprint arXiv:2303.11331_ , 2023a.
  * Fang et al. [2023b] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19358–19369, 2023b.
  * Gao et al. [2023] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _International Journal of Computer Vision_ , 2023.
  * Han et al. [2021] Ning Han, Jingjing Chen, Guangyi Xiao, Hao Zhang, Yawen Zeng, and Hao Chen.  Fine-grained cross-modal alignment network for text-video retrieval.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 3826–3834, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 2790–2799, 2019.
  * Hu et al. [2022] Fan Hu, Aozhu Chen, Ziyue Wang, Fangming Zhou, Jianfeng Dong, and Xirong Li.  Lightweight attentional feature fusion: A new baseline for text-to-video retrieval.  In _Proceedings of the 17th European Conference on Computer Vision_ , 2022.
  * Huang et al. [2023] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei.  Language is not all you need: Aligning perception with language models.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Krishna et al. [2017] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 706–715, 2017.
  * Lei et al. [2020] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal.  TVR: A large-scale dataset for video-subtitle moment retrieval.  In _European Conference on Computer Vision_ , pages 447–463, 2020.
  * Lei et al. [2021] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7331–7341, 2021.
  * Lei et al. [2023] Jie Lei, Tamara Berg, and Mohit Bansal.  Revealing single frame bias for video-and-language learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_ , pages 487–507, 2023.
  * Liang and Albanie [2023] Kaiqu Liang and Samuel Albanie.  Simple baselines for interactive video retrieval with questions and answers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11091–11101, 2023.
  * Liu et al. [2021a] Hongying Liu, Ruyi Luo, Fanhua Shang, Mantang Niu, and Yuanyuan Liu.  Progressive semantic matching for video-text retrieval.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 5083–5091, 2021a.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Liu et al. [2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10012–10022, 2021b.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.
  * Miech et al. [2020] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9879–9889, 2020.
  * Qiu et al. [2021] Zhaofan Qiu, Ting Yao, Yan Shu, Chong-Wah Ngo, and Tao Mei.  Condensing a sequence to one informative frame for video recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 16311–16320, 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 8748–8763, 2021.
  * Shtedritski et al. [2023] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi.  What does clip know about a red circle? visual prompt engineering for vlms.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11987–11997, 2023.
  * Song and Soleymani [2019] Yale Song and Mohammad Soleymani.  Polysemous visual-semantic embedding for cross-modal retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1979–1988, 2019.
  * Tavakolian et al. [2019a] Mohammad Tavakolian, Mohammad Sabokrou, and Abdenour Hadid.  Avd: Adversarial video distillation.  _arXiv preprint arXiv:1907.05640_ , 2019a.
  * Tavakolian et al. [2019b] Mohammad Tavakolian, Hamed R. Tavakoli, and Abdenour Hadid.  Awsd: Adaptive weighted spatiotemporal distillation for video representation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8020–8029, 2019b.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008, 2017.
  * Wang et al. [2023] Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  All in one: Exploring unified video-language pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6598–6608, 2023.
  * Wang et al. [2020] Wei Wang, Junyu Gao, Xiaoshan Yang, and Changsheng Xu.  Learning coarse-to-fine graph neural networks for video-text retrieval.  _IEEE Transactions on Multimedia_ , 2020.
  * Wu and Krahenbuhl [2021] Chao-Yuan Wu and Philipp Krahenbuhl.  Towards long-form video understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1884–1894, 2021.
  * Wu et al. [2023] Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing Qin.  An early evaluation of gpt-4v(ision).  _arXiv preprint arXiv:2310.16534_ , 2023.
  * Xu et al. [2021] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  VideoCLIP: Contrastive pre-training for zero-shot video-text understanding.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 6787–6800, 2021.
  * Xu et al. [2023] Yuting Xu, Jian Liang, Gengyun Jia, Ziming Yang, Yanhao Zhang, and Ran He.  TALL: Thumbnail layout for deepfake video detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22658–22668, 2023.
  * Zhang et al. [2023] Gengyuan Zhang, Jisen Ren, Jindong Gu, and Volker Tresp.  Multi-event video-text retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22113–22123, 2023.
  * Zhou et al. [2018] Luowei Zhou, Chenliang Xu, and Jason J. Corso.  Towards automatic learning of procedures from web instructional videos.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 7590–7598, 2018.