# CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

Huaishao Luo1 , Lei Ji2, Ming Zhong3, Yang Chen3, Wen Lei3, Nan Duan2, Tianrui
Li1
1Southwest Jiaotong University, Chengdu, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
2Microsoft Research Asia, Beijing, China
3Microsoft STCA, Beijing, China
<EMAIL_ADDRESS>This work was done during
the first author’s internship in MSR Asia

###### Abstract

Video-text retrieval plays an essential role in multi-modal research and has
been widely used in many real-world web applications. The CLIP (Contrastive
Language-Image Pre-training), an image-language pre-training model, has
demonstrated the power of visual concepts learning from web collected image-
text datasets. In this paper, we propose a CLIP4Clip model to transfer the
knowledge of the CLIP model to video-language retrieval in an end-to-end
manner. Several questions are investigated via empirical studies: 1) Whether
image feature is enough for video-text retrieval? 2) How a post-pretraining on
a large-scale video-text dataset based on the CLIP affect the performance? 3)
What is the practical mechanism to model temporal dependency between video
frames? And 4) The Hyper-parameters sensitivity of the model on video-text
retrieval task. Extensive experimental results present that the CLIP4Clip
model transferred from the CLIP can achieve SOTA results on various video-text
retrieval datasets, including MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo.
We release our code at https://github.com/ArrowLuo/CLIP4Clip.

##  1 Introduction

With the increasing of videos uploaded online every day, video-text retrieval
is becoming an emerging requirement for people to find relevant videos
efficiently. Beyond the actual web application, video-text retrieval is a
fundamental research task for multi-modal visual and language understanding.
We can distinguish the previous works directly by their input: raw video
(pixel-level) or video feature (feature-level).

Usually, the pretrain models Zhu and Yang (2020); Luo et al. (2020); Li et al.
(2020); Gabeur et al. (2020); Patrick et al. (2021); Rouditchenko et al.
(2020) are feature-level because they are trained on some large-scale video-
text datasets, e.g., Howto100M Miech et al. (2019). The input is the cached
video features generated via off-the-shelf frozen video feature extractors. If
the input is the raw video, it makes the pretrain very slow or infeasible.
Nevertheless, benefitting from the large-scale dataset, the pretrain models
show a significant performance gain for video-text retrieval.

The pixel-level approach trains the model with raw video as input directly
Torabi et al. (2016); Kiros et al. (2014); Yu et al. (2016a); Kaufman et al.
(2017); Yu et al. (2017, 2018). Early literature almost belongs to this
approach. This approach learns the video feature extractor jointly with the
paired text. On the contrary, the feature-level approach highly depends on a
suitable feature extractor. It can not propagate the learning back to the
fixed video encoder.

Some recent works begin to pretrain the model with the pixel-level approach,
making the pretrain model learn from the raw video. The big challenge is how
to reduce the high computational overload of dense video input. Typically,
ClipBERT Lei et al. (2021) employs a sparse sampling strategy to make the end-
to-end pretrain possible. Concretely, the model only sparsely samples a single
or a few short clips from a video at each training step. The performance shows
that the end-to-end training can benefit the low-level feature extraction. A
few sparsely sampled clips are enough to solve the video-text retrieval task.
Frozen Bain et al. (2021) treats an image as a single-frame video and designs
a curriculum learning schedule to train the model on both image and video
datasets. The results show that the curriculum learning schedule learns
increasingly from image to multi frames can increase efficiency. Our target is
not to pretrain a new model on video-text retrieval. We mainly investigate how
to transfer the knowledge from the image-text pretrained model CLIP Radford et
al. (2021) to video-text retrieval in this paper.

We exploit the pre-trained CLIP and propose a model named CLIP4Clip (CLIP For
video Clip retrieval) to solve video-text retrieval. Concretely, the CLIP4Clip
is constructed on top of the CLIP and designs a similarity calculator to
investigate three similarity calculation approaches: parameter-free type,
sequential type, and tight type. Like our work, the concurrent work from
Portillo-Quintero et al. (2021) is also built on the CLIP for video-text
retrieval. The difference is that their work directly leveraged the CLIP for
zero-shot prediction without considering different similarity calculation
mechanisms. However, we design some similarity calculation approaches to
improve the performance and train the model in an end-to-end manner. The
contributions of our work are: 1) we investigate three mechanisms of
similarity calculation based on the pretrained CLIP; 2) we further post pre-
train the CLIP on a noisy large-scale video-language dataset to learn a better
retrieval space. The extensive experiments show our model achieves the new
SOTA results on MSR-VTT Xu et al. (2016), MSVC Chen and Dolan (2011), LSMDC
Rohrbach et al. (2015), ActivityNet Krishna et al. (2017a), and DiDeMo
Hendricks et al. (2017) datasets.

Besides, we can conclude the following insights from our extensive
experiments:

1) One single image is far from enough for video encoding for video-text
retrieval.

2) Post-pretraining at a large-scale video-text dataset on the CLIP4Clip model
is required and can improve the performance, especially for zero-shot
prediction by a large margin.

3) With the powerful pre-trained CLIP, it is better not to introduce new
parameters and adopt a mean pooling mechanism on video frames for small
datasets. At the same time, it is better to introduce more parameters, e.g.,
the self-attention layer, to learn the temporal dependency for large datasets.

4) We carefully study the hyper-parameters and report the best setting.

##  2 Related Works

#### Video Encoder Backbone

Previous works mainly focus on 2D/3D spatial-temporal convolution for video
representation Tran et al. (2015); Xie et al. (2018); Feichtenhofer et al.
(2019). Recently ViT Dosovitskiy et al. (2021), a transformer-based image
encoder, has attracted much attention. The transformer based video encoder is
still in its early stage for action classification Bertasius et al. (2021);
Arnab et al. (2021). We mainly investigate the effective transformer based
video backbone for multimodal video-text retrieval.

(a)

(b)

Figure 1: The framework of CLIP4Clip, which comprises three components,
including two single-modal encoders and a similarity calculator. The model
takes a video-text pair as input. For the input video, we first sample the
input video into ordinal frames (images). Next, these image frames are
reshaped into a sequence of flattened 2D patches. These patches are mapped to
the 1D sequence of embeddings with a linear patch embedding layer and input to
the image encoder for representation as in ViT Dosovitskiy et al. (2021).
Finally, the similarity calculator predicts the similarity score between the
text representation and representation sequence of these frames. We
investigate three types of similarity calculators in this work, including
parameter-free, sequential, and tight types. $\otimes$ means cosine
similarity. We initial the two single-modal encoders with CLIP (ViT-B/32)
Radford et al. (2021).

#### Visual Representation Learning from Text Supervision

Visual representation learning is a challenging task and has been widely
studied with supervised or self-supervised methods. Considering semantic
supervision from large-scale unlabeled data, learning visual representation
from text representation Radford et al. (2021); Miech et al. (2020); Lei et
al. (2021) is an emerging research topic with the benefit of large-scale
visual and linguistic pairs collected from the Internet. The prominent success
of the CLIP (Contrastive Language-Image Pre-training) Radford et al. (2021)
has demonstrated its capability of learning SOTA image representations from
linguistic supervision with pre-training on large-scale image and text pairs.
The pre-trained model can learn fine-grained visual concepts for images and
transfer the knowledge for the retrieval task. Typically, MIL-NCE Miech et al.
(2020) mainly investigated to leverage noisy large-scale Howto100M Miech et
al. (2019) instructional videos to learn a better video encoder in an end-to-
end manner. Furthermore, ClipBERT Lei et al. (2021) proposed an efficient end-
to-end approach through sparse sampling and presented that the pretrained by
image-language dataset facilitated a better initialization of video-text
retrieval. Different from ClipBERT, we adopt CLIP with transformer based
visual backbone and extend this image-language pre-trained model to video-
language pre-training for video-text retrieval. Considering the temporal
sequence of video, a 2D/3D linear embedding and a similarity calculator
attached to the visual transformer are used to capture temporal sequence
features.

#### Video-Text Retrieval

Early works on video-text retrieval Torabi et al. (2016); Kiros et al. (2014);
Yu et al. (2016a); Kaufman et al. (2017); Yu et al. (2017, 2018)designed
intensive fusion mechanisms for cross-modal learning. Recently, the pre-
trained models Zhu and Yang (2020); Amrani et al. (2021); Luo et al. (2020);
Li et al. (2020); Miech et al. (2020); Gabeur et al. (2020); Patrick et al.
(2021); Lei et al. (2021); Dzabraev et al. (2021); Liu et al. (2021) have
dominated the leaderboard of the video-text retrieval with noticeable results
on zero-shot retrieval and fine-tuned retrieval. Concurrent to our work,
Portillo-Quintero et al. (2021) applied CLIP for zero-shot prediction, and
Bain et al. (2021) proposed a transformer-based video backbone. We propose to
directly transfer the powerful knowledge from the pre-trained CLIP and
continue pre-train the designed video-based CLIP4Clip on a large-scale video-
language dataset. Empirical studies present the effectiveness of the CLIP4Clip
model.

##  3 Framework

Given a set of videos (or video clips) $\mathcal{V}$ and a set of captions
$\mathcal{T}$, our target is to learn a function $s(v_{i},t_{j})$ to calculate
the similarity between the video (or video clip) $v_{i}\in\mathcal{V}$ and the
caption $t_{j}\in\mathcal{T}$. The goal is to rank all the videos (or video
clips) given the query caption according to their similarity score in the
text-to-video retrieval or rank all the captions given the query video (or
video clip) in the task of video-to-text retrieval. The objective of the
$s(v_{i},t_{j})$ is to calculate a high similarity for relevant video-text
pairs and a low similarity score for irrelevant ones.

The video (or video clip) $v_{i}\in\mathcal{V}$ is represented as a sequence
of frames (images) in this paper. Formally, the video (or video clip) $v_{i}$
is composed of $|v_{i}|$ sampled frames such that
$v_{i}=\\{v^{1}_{i},v^{2}_{i},\dots,v^{|v_{i}|}_{i}\\}$. Our model is an end-
to-end manner (E2E) trained on pixels directly via taking the frames as input.
Figure 1 demonstrates our framework, which mainly contains a text encoder, a
video encoder, and a similarity calculator. We introduce each part in detail
in this section.

###  3.1 Video Encoder

To get the video representation, we first extract the frames from the video
clip and then encode them via a video encoder to obtain a sequence of
features. In this paper, we adopt the ViT-B/32 Dosovitskiy et al. (2021) with
12 layers and the patch size 32 as our video encoder. Concretely, we use the
pre-trained CLIP (ViT-B/32) Radford et al. (2021) as our backbone and mainly
consider transferring the image representation to video representation. The
pre-trained CLIP (ViT-B/32) is effective for the video-text retrieval task in
this paper.

(a)

(b)

Figure 2: Different views of Linear Projection of Flattened Patches in Video
Encoder. Dotted boxes with color are kernels.

The ViT Dosovitskiy et al. (2021) first extracts non-overlapping image
patches, then performs a linear projection to project them into 1D tokens, and
exploits the transformer architecture to model the interaction between each
patch of the input image to get the final representation. Following the ViT
and CLIP, we use the output from the [class] token as the image
representation. For the input frame sequence of video
$v_{i}=\\{v^{1}_{i},v^{2}_{i},\dots,v^{|v_{i}|}_{i}\\}$, the generated
representation can denote as
$\mathbf{Z}_{i}=\\{\mathbf{z}^{1}_{i},\mathbf{z}^{2}_{i},\dots,\mathbf{z}^{|v_{i}|}_{i}\\}$.

We investigate two types of linear projections in the Linear Projection of
Flattened Patches module shown in Figure 1(a) named 2D linear and 3D linear
separately. a) The linear projection of flattened patches of ViT is regarded
as 2D linear, which embeds each 2D frame patch independently. Such a 2D linear
ignores the temporal information among frames. b) Therefore, we investigate a
3D linear projection, similar to Arnab et al. (2021), to enhance temporal
feature extraction. The comparison between 2D and 3D is shown in Figure 2. The
3D linear embeds patches across time. Concretely, the 3D linear uses a 3D
convolution with kernel $[t\times h\times w]$ as the linear instead of the
kernel of $[h\times w]$ in 2D linear, where $t,h$, and $w$ are temporal,
height, and width dimensions, respectively.

###  3.2 Text Encoder

We directly apply the text encoder from the CLIP to generate the caption
representation. The text encoder is a Transformer Vaswani et al. (2017) with
the architecture modifications described in Radford et al. (2019). It is a
12-layer 512-wide model with 8 attention heads. Following CLIP, the
activations from the highest layer of the transformer at the [EOS] token are
treated as the feature representation of the caption. For the caption
$t_{j}\in\mathcal{T}$, we denote the representation as $\mathbf{w}_{j}$.

###  3.3 Similarity Calculator

After extracting the video representation
$\mathbf{Z}_{i}=\\{\mathbf{z}^{1}_{i},\mathbf{z}^{2}_{i},\dots,\mathbf{z}^{|v_{i}|}_{i}\\}$
and caption representation $\mathbf{w}_{j}$, the key point comes to the
similarity calculation. Since our model is built based upon a pre-trained
image-text model, we should carefully add new learnable weights in the
similarity calculator module. It is hard to learn without weight
initialization and may hurt the performance of the pre-trained model training
with backpropagation. Therefore, we categorize the mechanisms of the
similarity calculator into three categories depending on whether the module
introduces new parameters to learn. The _parameter-free_ approach, i.e.,
meaning pooling, fuses the video representation without new parameters.
Additionally, two other approaches introduce new weights to learn including a
_sequential_ type and a _tight_ type methods with different sizes of new
weights. Figure 1(b) illustrates the detailed structure of the three
mechanisms. The parameter-free type and sequential type similarity calculators
belong to the loose type that adopts two separate branches for video and text
representation independently to calculate cosine similarity. While the tight
type similarity calculator uses the transformer model for multi-modal
interaction and further calculates the similarity via a linear projection,
both of which consist of new weights to learn.

#### Parameter-free type

According to the CLIP, the frames representation $\mathbf{Z}_{i}$ and the
caption representation $\mathbf{w}_{j}$ have been layer normalized and
linearly projected into a multi-modal embedding space through the large-scale
pretraining on the image-text pairs. The natural idea is to employ a
parameter-free type to calculate similarity directly with the image/frame from
the video perspective. The parameter-free type first uses the mean pooling to
aggregate the feature of all frames to obtain an ‘average frame’,
$\hat{\mathbf{z}}_{i}=\texttt{mean-
pooling}(\mathbf{z}^{1}_{i},\mathbf{z}^{2}_{i},\dots,\mathbf{z}^{|v_{i}|}_{i})$.
Then, the similarity function $s(v_{i},t_{j})$ is defined as the cosine
similarity,

$\displaystyle
s(v_{i},t_{j})=\frac{{\mathbf{w}_{j}}^{\top}\hat{\mathbf{z}}_{i}}{\lVert\mathbf{w}_{j}\rVert\lVert\hat{\mathbf{z}}_{i}\rVert}.$
(1)

#### Sequential type

The mean-pooling operation ignores the sequential information between frames.
In this way, we explore two methods to model the sequential feature for
Sequential type similarity calculator. One is LSTM Hochreiter and Schmidhuber
(1997); Gers et al. (2002), and the other one is Transformer encoder Vaswani
et al. (2017) with position embedding $\mathbf{P}$, Both of which are
effective models for sequence features. We formulate them as
$\tilde{\mathbf{Z}}_{i}=\texttt{LSTM}(\mathbf{Z}_{i})$ and
$\tilde{\mathbf{Z}}_{i}=\texttt{Transformer-Enc}(\mathbf{Z}_{i}+\mathbf{P})$,
respectively. Through the encoding, the $\tilde{\mathbf{Z}}_{i}$ already
embeds the temporal information. The subsequent operations are the same as the
parameter-free type similarity calculator, and the similarity function is also
the Eq. (1), and $\hat{\mathbf{z}}_{i}=\texttt{mean-
pooling}(\tilde{\mathbf{Z}}_{i})$.

#### Tight type

Different from above parameter-free type and sequential type, the tight type
uses a Transformer Encoder Vaswani et al. (2017) for multimodal interaction
between video and caption similar to Luo et al. (2020), and predict the
similarity through a linear layer, which introduces the most uninitialized
weights. First, the Transformer Encoder takes the concatenated caption
representation $\mathbf{w}_{j}$ and frames’ representation
$\mathbf{Z}_{i}=\\{\mathbf{z}^{1}_{i},\mathbf{z}^{2}_{i},\dots,\mathbf{z}^{|v_{i}|}_{i}\\}$
as the fused feature $\mathbf{U}_{i}$ formulated as,

$\displaystyle\mathbf{U}_{i}$
$\displaystyle=[\mathbf{w}_{j},\mathbf{z}^{1}_{i},\mathbf{z}^{2}_{i},\dots,\mathbf{z}^{|v_{i}|}_{i}],$
(2) $\displaystyle\tilde{\mathbf{U}}_{i}$ $\displaystyle=\texttt{Transformer-
Enc}(\mathbf{U}_{i}+\mathbf{P}+\mathbf{T}).$ (3)

where $[,]$ denotes concatenate operation. $\mathbf{P}$ is the position
embedding, and $\mathbf{T}$ is type embedding similar to Segment Embeddings in
BERT Devlin et al. (2019). The $\mathbf{T}$ contains two types of embedding,
one is for caption and the other is for video frames. Next, we calculate
similarity score with two linear projection layers plus an activation upon the
first token output of the last layer $\tilde{\mathbf{U}}_{i}[0,:]$.
Concretely, the similarity function
$s(v_{i},t_{j})=\texttt{FC}\big{(}\texttt{ReLU}\big{(}\texttt{FC}(\tilde{\mathbf{U}}_{i}[0,:])\big{)}\big{)}$,
where FC is the linear projection, and the ReLU means ReLU activation function
Agarap (2018).

###  3.4 Training Strategy

#### Loss Function

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
C+LSTM+SAa | M | ✓ | 4.2 | 12.9 | 19.9 | 55 | -
VSEb | M | ✓ | 3.8 | 12.7 | 17.1 | 66 | -
SNUVLc | M | ✓ | 3.5 | 15.9 | 23.8 | 44 | -
Kaufman et al.d | M | ✓ | 4.7 | 16.6 | 24.1 | 41 | -
CT-SANe | M | ✓ | 4.4 | 16.6 | 22.3 | 35 | -
JSFusionf | M | ✓ | 10.2 | 31.2 | 43.2 | 13 | -
HowTo100Mg | H+M | ✓ | 14.9 | 40.2 | 52.8 | 9 | -
ActBERTh | H+M |  | 8.6 | 23.4 | 33.1 | 36 | -
NoiseEi | H+M |  | 17.4 | 41.6 | 53.6 | 8 | -
UniVLj | H+M |  | 21.2 | 49.6 | 63.1 | 6 | -
HEROk | H+M |  | 16.8 | 43.4 | 57.7 | - | -
ClipBERTl | C+G+M | ✓ | 22.0 | 46.8 | 59.9 | 6 | -
(Ours)-meanP | W+M | ✓ | 42.1 | 71.9 | 81.4 | 2 | 15.7
(Ours)-seqLSTM | W+M | ✓ | 41.7 | 68.8 | 78.7 | 2 | 16.6
(Ours)-seqTransf | W+M | ✓ | 42.0 | 68.6 | 78.7 | 2 | 16.2
(Ours)-tightTransf | W+M | ✓ | 37.8 | 68.4 | 78.4 | 2 | 17.2

(a) Training on Training-7K

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
MIL-NCEm | H | ✓ | 9.9 | 24.0 | 32.4 | 29.5 | -
CLIP-straightn | W | ✓ | 31.2 | 53.7 | 64.2 | 4 | -

(b) Zero-shot

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
CEo | M |  | 20.9 | 48.8 | 62.4 | 6 | 28.2
MMTp | H+M |  | 26.6 | 57.1 | 69.6 | 4 | 24.0
AVLnetq | H+M |  | 27.1 | 55.6 | 66.6 | 4 | -
SSBr | H+M |  | 30.1 | 58.5 | 69.3 | 3 | -
MDMMTs | MD+M |  | 38.9 | 69.0 | 79.7 | 2 | 16.5
Frozent | CW+M | ✓ | 31.0 | 59.5 | 70.5 | 3 | -
HiTu | H+M |  | 30.7 | 60.9 | 73.2 | 2.6 | -
TT-CE+v | M |  | 29.6 | 61.6 | 74.2 | 3 | -
(Ours)-meanP | W+M | ✓ | 43.1 | 70.4 | 80.8 | 2 | 16.2
(Ours)-seqLSTM | W+M | ✓ | 42.5 | 70.8 | 80.7 | 2 | 16.7
(Ours)-seqTransf | W+M | ✓ | 44.5 | 71.4 | 81.6 | 2 | 15.3
(Ours)-tightTransf | W+M | ✓ | 40.2 | 71.5 | 80.5 | 2 | 13.4

(c) Training on Training-9K

Table 1: Results of text-to-video retrieval on MSR-VTT dataset. Table (a) and
(c) present the results on different splits of the dataset. ‘Training-7K’
follows the data splits from Miech et al. (2019) and ‘Training-9K’ follows the
data splits from Gabeur et al. (2020). They have the same test set but
different training sets. For each table, the column ‘TrainD’ shows the
datasets used for pre-training and training, where M, H, W, C, G denote MSR-
VTT, HowTo100M Miech et al. (2019), WIT Radford et al. (2021), COCO Captions
Chen et al. (2015), and Visual Genome Captions Krishna et al. (2017b).
Besides, MD used in MDMMTDzabraev et al. (2021) denotes a combined multidomain
dataset including MSR-VTT, LSMDC, HowTo100M, etc., and CW means CC3M Sharma et
al. (2018) plus WebVid-2M Bain et al. (2021). The column ‘E2E’ with ✓means
training from raw video in an end-to-end manner. The baseline methods are
aC+LSTM+SA Torabi et al. (2016), bVSE Kiros et al. (2014), cSNUVL Yu et al.
(2016b), dKaufman et al. Kaufman et al. (2017), eCT-SAN Yu et al. (2017),
fJSFusion Yu et al. (2018), gHowTo100M Miech et al. (2019), hActBERT Zhu and
Yang (2020), iNoiseE Amrani et al. (2021), jUniVL Luo et al. (2020), kHERO Li
et al. (2020), lClipBERT Lei et al. (2021), mMIL-NCE Miech et al. (2020),
nCLIP-straight Portillo-Quintero et al. (2021), oCE Liu et al. (2019), pMMT
Gabeur et al. (2020), qAVLnet Rouditchenko et al. (2020), rSSB Patrick et al.
(2021), sMDMMT Dzabraev et al. (2021), tFrozen Bain et al. (2021), uHiT Liu et
al. (2021), vTT-CE+ Croitoru et al. (2021).

Given a batch of $B$ (video, text) or (video clip, text) pairs, the model
needs to generate and optimize $B\times B$ similarities. We use a symmetric
cross entropy loss over these similarity scores to train the model’s
parameters,

$\displaystyle\mathcal{L}_{v2t}$
$\displaystyle=-\frac{1}{B}\sum_{i}^{B}{\log\frac{\exp(s(v_{i},t_{i}))}{\sum_{j=1}^{B}{\exp(s(v_{i},t_{j})}}},$
(4) $\displaystyle\mathcal{L}_{t2v}$
$\displaystyle=-\frac{1}{B}\sum_{i}^{B}{\log\frac{\exp(s(v_{i},t_{i}))}{\sum_{j=1}^{B}{\exp(s(v_{j},t_{i})}}},$
(5) $\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{v2t}+\mathcal{L}_{t2v}.$ (6)

The loss $\mathcal{L}$ is the sum of video-to-text loss $\mathcal{L}_{v2t}$
and text-to-video loss $\mathcal{L}_{t2v}$.

#### Frame Sampling

Since our model is trained on pixels directly via taking the frames as input,
it is an important strategy to extract frames. An effective sampling strategy
is required to consider the balance between the information richness and the
computational complexity (especially memory cost). To consider the sequential
information in the video (or video clip), we adopt a uniform frame sampling
strategy instead of a random sparse sampling strategy used in Lei et al.
(2021). The sampling rate is 1 frame per second. Besides, we also investigate
different frame lengths and different extraction positions in our experiments.

#### Pre-training

Although the CLIP is effective for learning the visual concepts of images, it
is essential to learn temporal features from video. To further transfer the
knowledge to video, our CLIP4Clip model is post-pretrained on Howto100M
dataset Miech et al. (2019). Pretraining on the video-text dataset is
extremely challenging due to efficiency consideration. We conduct a
preliminary exploration and use the ‘Food and Entertaining’ category, around
380k videos, as the post-pretraining dataset (called _Howto100M-380k_ in the
rest of the paper). We adopt the MIL-NCE loss Miech et al. (2020) to optimize
the CLIP in our parameter-free type. The optimizer is Adam Kingma and Ba
(2015), with a learning rate 1e-8. The token length is 32, the frame length is
12, and the batch size is 48. The training is processed on 8 NVIDIA Tesla V100
GPUs. We run 5 epochs and it takes about 2 weeks. In this paper, the post-
pretraining test can be regarded as a preliminary study on this direction for
future work.

##  4 Experiments

We first describe the datasets and implementation details before presenting
state-of-the-art results on five datasets. We then ablate various settings of
our model. Finally, we discuss some aspects of promising direction.

###  4.1 Datasets

We validate our model on five datasets: MSR-VTT, MSVC, LSMDC, ActivityNet, and
DiDeMo.

MSR-VTT Xu et al. (2016) is a dataset composed of 10,000 videos, each with a
length that ranges from 10 to 32 seconds and 200,000 captions. We use two
types of data splits, ‘Training-7K’ and ‘Training-9K’, to compare with
baselines. The ‘Training-7K’ follows the data splits from Miech et al. (2019)
and the ‘Training-9K’ follows the data splits from Gabeur et al. (2020). The
test data in both splits is ‘test 1k-A’, which contains 1,000 clip-text pairs
following JSFusion Yu et al. (2018). We use ‘Training-9K’ as the default
setting if there is no extra annotation.

MSVD Chen and Dolan (2011) contains 1,970 videos, each with a length that
ranges from one to 62 seconds. Train, validation and, test splits contain
1,200, 100, and 670 videos, respectively. Each video has approximately 40
associated sentences in English.

LSMDC Rohrbach et al. (2015) is comprised 118,081 videos, each with a length
that ranges from two to 30 seconds. The videos were extracted from 202 movies.
The validation set contains 7,408 videos, and the test set 1,000 videos from
movies independent from the training and validation splits.

ActivityNet Krishna et al. (2017a) consists of 20,000 YouTube videos. We
follow the setting from Zhang et al. (2018); Gabeur et al. (2020) to
concatenate all the descriptions of a video to form a paragraph and evaluate
the model with video-paragraph retrieval on the ‘val1’ split.

DiDeMo Hendricks et al. (2017) contains 10,000 videos annotated with 40,000
sentences. We evaluate video-paragraph retrieval following Liu et al. (2019);
Lei et al. (2021); Bain et al. (2021), where all sentence descriptions for a
video are concatenated into a single query.

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
Multi Cuesa | M | ✓ | 20.3 | 47.8 | 61.1 | 6 | -
CEb | M |  | 19.8 | 49.0 | 63.8 | 6 | -
SSBc | H+M |  | 28.4 | 60.0 | 72.9 | 4 | -
NoiseEd | H+M |  | 20.3 | 49.0 | 63.3 | 6 | -
CLIP-straighte | W | ✓ | 37.0 | 64.1 | 73.8 | 3 | -
Frozenf | CW+M | ✓ | 33.7 | 64.7 | 76.3 | 3 | -
TT-CE+g | M |  | 25.4 | 56.9 | 71.3 | 4 | -
(Ours)-meanP | W+M | ✓ | 46.2 | 76.1 | 84.6 | 2 | 10.0
(Ours)-seqLSTM | W+M | ✓ | 46.2 | 75.3 | 84.5 | 2 | 10.2
(Ours)-seqTransf | W+M | ✓ | 45.2 | 75.5 | 84.3 | 2 | 10.3
(Ours)-tightTransf | W+M | ✓ | 40.0 | 71.5 | 82.1 | 2 | 13.3

Table 2: Results of text-to-video retrieval on MSVD dataset. In the column
‘TrainD’, M, H, and W denote training on MSVD, HowTo100M Miech et al. (2019),
and WIT Radford et al. (2021), and CW means CC3M Sharma et al. (2018) plus
WebVid-2M Bain et al. (2021). The column ‘E2E’ with ✓means training from raw
video in an end-to-end manner. The baseline methods are aMulti Cues Mithun et
al. (2018), bCE Liu et al. (2019), cSSB Patrick et al. (2021), dNoiseE Amrani
et al. (2021), eCLIP-straight Portillo-Quintero et al. (2021), fFrozen Bain et
al. (2021), gTT-CE+ Croitoru et al. (2021).

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
CT-SANa | L | ✓ | 5.1 | 16.3 | 25.2 | 46.0 | -
JSFusionb | L | ✓ | 9.1 | 21.2 | 34.1 | 36.0 | -
CEc | L |  | 11.2 | 26.9 | 34.8 | 25.3 | 96.8
MMTd | H+L |  | 12.9 | 29.9 | 40.1 | 19.3 | 75.0
NoiseEe | H+L |  | 6.4 | 19.8 | 28.4 | 39.0 | -
CLIP-straightf | L | ✓ | 11.3 | 22.7 | 29.2 | 56.5 | -
MDMMTg | MD+L |  | 18.8 | 38.5 | 47.9 | 12.3 | 58.0
Frozenh | CW+L | ✓ | 15.0 | 30.8 | 39.8 | 20.0 | -
HiTi | H+L |  | 14.0 | 31.2 | 41.6 | 18.5 | -
TT-CE+j | L |  | 17.2 | 36.5 | 46.3 | 13.7 | -
(Ours)-meanP | W+L | ✓ | 20.7 | 38.9 | 47.2 | 13.0 | 65.3
(Ours)-seqLSTM | W+L | ✓ | 21.6 | 41.8 | 49.8 | 11.0 | 58.0
(Ours)-seqTransf | W+L | ✓ | 22.6 | 41.0 | 49.1 | 11.0 | 61.0
(Ours)-tightTransf | W+L | ✓ | 18.9 | 37.8 | 46.7 | 13.0 | 61.6

Table 3: Results of text-to-video retrieval on LSMDC dataset. In the column
‘TrainD’, L, H, and W denote training on LSMDC, HowTo100M Miech et al. (2019),
and WIT Radford et al. (2021), MD used in Dzabraev et al. (2021) denotes a
combined multidomain dataset containing MSR-VTT, LSMDC, HowTo100M, etc., and
CW means CC3M Sharma et al. (2018) plus WebVid-2M Bain et al. (2021). The
column ‘E2E’ with ✓means training from raw video in an end-to-end manner. The
baseline methods are aCT-SAN Yu et al. (2017), bJSFusion Yu et al. (2018), cCE
Liu et al. (2019), dMMT Gabeur et al. (2020), eNoiseE Amrani et al. (2021),
fCLIP-straight Portillo-Quintero et al. (2021), gMDMMT Dzabraev et al. (2021),
hFrozen Bain et al. (2021), iHiT Liu et al. (2021), jTT-CE+ Croitoru et al.
(2021).

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@50$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
FSEa | A |  | 18.2 | 44.8 | 89.1 | 7.0 | -
CEb | A |  | 18.2 | 47.7 | 91.4 | 6.0 | 23.1
HSEa | A |  | 20.5 | 49.3 | - | - | -
MMTc | H+A |  | 28.7 | 61.4 | 94.5 | 3.3 | 16.0
SSBd | H+A |  | 29.2 | 61.6 | 94.7 | 3.0 | -
HiTe | H+A |  | 29.6 | 60.7 | 95.6 | 3.0 | -
ClipBERTf | C+G+A | ✓ | 21.3 | 49.0 | - | 6.0 | -
TT-CE+g | A |  | 23.5 | 57.2 | 96.1 | 4.0 | -
(Ours)-meanP | W+A | ✓ | 40.5 | 72.4 | 98.1 | 2.0 | 7.4
(Ours)-seqLSTM | W+A | ✓ | 40.1 | 72.2 | 98.1 | 2.0 | 7.3
(Ours)-seqTransf | W+A | ✓ | 40.5 | 72.4 | 98.2 | 2.0 | 7.5
(Ours)-tightTransf | W+A | ✓ | 19.5 | 47.6 | 93.1 | 6.0 | 17.3

Table 4: Results of text-to-video retrieval on ActivityNet dataset. In the
column ‘TrainD’, A, H, W, C, and G denote training on ActivityNet, HowTo100M
Miech et al. (2019), WIT Radford et al. (2021), COCO Captions Chen et al.
(2015), and Visual Genome Captions Krishna et al. (2017b). The column ‘E2E’
with ✓means training from raw video in an end-to-end manner. The baseline
methods are aFSE,HSE Zhang et al. (2018), bCE Liu et al. (2019), cMMT Gabeur
et al. (2020), dSSB Patrick et al. (2021), eHiT Liu et al. (2021), fClipBERT
Lei et al. (2021), gTT-CE+ Croitoru et al. (2021).

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
S2VTa | D | ✓ | 11.9 | 33.6 | - | 13.0 | -
FSEb | D |  | 13.9 | 36.0 | - | 11.0 | -
CEc | D |  | 16.1 | 41.1 | - | 8.3 | 43.7
ClipBERTd† | C+G+D | ✓ | 20.4 | 48.0 | 60.8 | 6.0 | -
Frozene† | CW+D | ✓ | 34.6 | 65.0 | 74.7 | 3.0 | -
TT-CE+f | D |  | 21.6 | 48.6 | 62.9 | 6.0 | -
(Ours)-meanP | W+D | ✓ | 43.4 | 70.2 | 80.6 | 2.0 | 17.5
(Ours)-seqLSTM | W+D | ✓ | 43.4 | 69.9 | 80.2 | 2.0 | 17.5
(Ours)-seqTransf | W+D | ✓ | 42.8 | 68.5 | 79.2 | 2.0 | 18.9
(Ours)-tightTransf | W+D | ✓ | 25.8 | 52.8 | 66.3 | 5.0 | 27.3

Table 5: Results of text-to-video retrieval on DiDeMo dataset. In the column
‘TrainD’, D, H, W, C, and G denote training on DiDeMo, HowTo100M Miech et al.
(2019), WIT Radford et al. (2021), COCO Captions Chen et al. (2015), and
Visual Genome Captions Krishna et al. (2017b), CW means CC3M Sharma et al.
(2018) plus WebVid-2M Bain et al. (2021). The column ‘E2E’ with ✓means
training from raw video in an end-to-end manner. $\dagger$ means that the
candidate video is concatenated using ground truth proposals. The baseline
methods are aS2VT Venugopalan et al. (2015), bFSE Zhang et al. (2018), cCE Liu
et al. (2019), dClipBERT Lei et al. (2021), eFrozen Bain et al. (2021), fTT-
CE+ Croitoru et al. (2021).

We use standard retrieval metrics: recall at rank $K$ (R@$K$, higher is
better), median rank (MdR, lower is better), and mean rank (MnR, lower is
better) to evaluate the performance of our model. R@$K$ (Recall at $K$)
calculates the percentage of test samples for which the correct result is
found in the top-$K$ retrieved points to the query sample. We report results
for R@1, R@5, and R@10 (or R@50 for the ActivityNet). Median Rank calculates
the median of the ground-truth results in the ranking. Similarly, Mean Rank
calculates the mean rank of all correct results.

(a)

(b)

(c)

(d)

Figure 3: Retrieval results on different batch sizes, frame length, freeze
layer, and learning rate. Batch size: freeze layer is 6. Learning rate: batch
size is 128, freeze layer is 6, frame length is 12. Freeze layer: Fz-[NO.]
means freeze layers below [NO.]-th layer (inclusive), Fz-Linear means only
freeze the linear layer at the bottom, No-Fz is training without freeze, batch
size is 128, frame length is 12, learning rate is 5e-8. Frame length: batch
size is 128, freeze layer is 6, learning rate is 5e-8.

###  4.2 Experimental Details

We initial the text encoder and video encoder with CLIP (ViT-B/32) Radford et
al. (2021) in this paper. The practical question is how to initialize the
parameters in the similarity calculator, e.g., parameters in sequential type.
Our solution is to reuse similar parameters from the CLIP (ViT-B/32).
Concretely, for the position embedding in sequential type and tight type, we
initialize them by repeating the position embedding from CLIP’s text encoder.
Similarly, the transformer encoder is initialized by the corresponding layers’
weight of the pretrained CLIP’s image encoder. The rest of the parameters,
e.g., LSTM and linear projection, are initialized randomly. The temporal
dimension $t$, height dimension $h$, and width dimension $w$ of 3D linear and
2D linear in Section 3.1 are set to 3, 32, 32, respectively. For 3D linear, we
set stride and padding with 1 at the temporal dimension. We initialize the 3D
linear following Arnab et al. (2021) with a ‘central frame initialization’
strategy from the pretrained 2D linear of CLIP. Concretely, we use
$[\mathbf{0},\mathbf{E}_{2D},\mathbf{0}]$ from the 2D weight $\mathbf{E}_{2D}$
of CLIP’ image encoder.

We finetune the model with the Adam optimizer Kingma and Ba (2015). For the
learning rate, we decay it using a cosine schedule Loshchilov and Hutter
(2017) following the CLIP Radford et al. (2021). If no otherwise specified,
the initial learning rate is 1e-7 for text encoder and video encoder
(including linear projection) and 1e-4 for new modules, e.g., LSTM, the
caption token length is 32, the frame length is 12, the batch size is 128, and
running 5 epochs. The layer of LSTM is 1, and the layer of Transformer Encoder
in both sequential type and tight type is 4 in our experiments. All finetune
experiments are carried out on 4 NVIDIA Tesla V100 GPUs. Note that the
ActivityNet and the DiDeMo are regarded as video-paragraph retrieval, so we
set the caption token length and the frame length 64. The experiments on them
are carried out on 16 NVIDIA Tesla V100 GPUs.

###  4.3 Comparison to the State of the Art

We compare all types of similarity calculator based on the pretrained CLIP
against the state-of-the-art (SOTA): ‘-meanP’, ‘-seqLSTM’, ‘-seqTransf’, and
‘-tightTransf’ are short for parameter-free type (i.e., mean pooling),
sequential type of LSTM, Transformer Encoder, and tight type mentioned in
Section 3.3. Table 1-5 present the text-to-video retrieval results of our
model on MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo. The baselines of each
dataset are listed in the caption of each table for clarification. We achieve
the SOTA results on all five datasets by a large margin compared with all
baselines. We find that the growth of retrieval performance benefits from the
pretrained CLIP via our results and the concurrent CLIP-straight Portillo-
Quintero et al. (2021). Besides, the improvement from our end-to-end finetune
proves the potential of the image-text pretrain model on video-text retrieval.

For the MSR-VTT dataset, the model with the parameter-free type (-meanP)
achieves the best results for the ‘Training-7k’ data split, while the model
with the sequential type (-seqTransf) outperforms other methods for the
‘Training-9k’ data split. We think that it is hard to learn extra parameters
beyond the pre-trained parameters given a small dataset. With a large dataset,
it is capable of learning the extra parameters. For the LSMDC dataset, the
model with the sequential type performs better than the other two types. The
two sequential types, -seqLSTM and -seqTransf, achieve comparable results. For
the MSVD dataset, the performance of the parameter-free type is the best. We
notice that the MSVD training data is smaller than the MSR-VTT and MSVD
datasets at least 2 times, and consider the reason is that the extra
parameters need extra large dataset to keep the advance from pretrained
weight. The performance of video-paragraph retrieval on the ActivityNet and
DiDeMo further proves the advantage of the parameter-free type when utilizing
the pretrained model. Among five datasets, almost all the results of the tight
type (-tightTransf) are the worst among all calculators. We think that the
tight type is still hard to learn the cross-modality interaction without
enough dataset.

Frame Selection | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---
MSR-VTT
Head | 42.3 | 70.8 | 80.8 | 2 | 15.7
Tail | 40.5 | 67.9 | 77.4 | 2 | 18.8
Uniform | 42.6 | 70.4 | 80.1 | 2 | 16.3
MSVD
Head | 45.7 | 75.1 | 84.1 | 2 | 10.2
Tail | 45.6 | 75.3 | 84.3 | 2 | 10.2
Uniform | 46.0 | 75.3 | 84.5 | 2 | 10.1
LSMDC
Head | 20.3 | 39.2 | 46.8 | 13 | 63.3
Tail | 20.7 | 38.2 | 46.4 | 14 | 63.6
Uniform | 20.7 | 39.0 | 47.1 | 13 | 63.4

Table 6: Study on sampling strategy. ‘Head’, ‘Tail’ and ‘Uniform’ are three
sample strategies to select frames from a video. Batch size is 128, freeze
layer is 0, frame length is 12, and learning rate is 5e-8.

###  4.4 Hyperparameters and Learning Strategy

We run extensive experiments on studying the hyperparameters and learning
strategies to search for the best settings. Figure 3 shows the resulting
chart. With increasing of _Batch size_ in 3(a), the performance increases and
it achieves comparable result for batch size 128 and 256. In our experiment,
we set the batch size to 128. About the study on _frame length_ in 3(d), we
can see a significant increase between 1 and 6 frames which shows that it is
required for video to actually model with a sequence of multiple frames
instead of one single frame. We sampled 12 frames for our experiment, which is
both efficient and effective. We also study whether we should _freeze_ the
parameters of each layer pre-trained by CLIP. From the Figure 3(c), it is
better to fine-tune all the transformer encoder layers at a small learning
rate and keep the bottom linear layer. About the _Learning rate_ shown in
3(b), the best learning rate is 1e-7, which can not be too large or too
small.A large learning rate hurts the performance. Even more, it can not
leverage the advantage of pre-trained weights.

Pretrain | P-PT | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---
MSR-VTT
ZS |  | 30.6 | 54.4 | 64.3 | 4 | 41.8
ZS | ✓ | 32.0 | 57.0 | 66.9 | 4 | 34.0
FT |  | 43.1 | 70.4 | 80.8 | 2 | 16.2
FT | ✓ | 43.5 | 70.7 | 80.5 | 2 | 16.3
MSVD
ZS |  | 36.2 | 63.8 | 73.5 | 3 | 20.4
ZS | ✓ | 38.5 | 66.9 | 76.8 | 2 | 17.8
FT |  | 46.2 | 76.1 | 84.6 | 2 | 10.0
FT | ✓ | 46.6 | 76.1 | 84.8 | 2 | 9.9
LSMDC
ZS |  | 13.6 | 27.9 | 35.5 | 32 | 134.5
ZS | ✓ | 15.1 | 28.5 | 36.4 | 28 | 117.0
FT |  | 20.7 | 38.9 | 47.2 | 13 | 65.3
FT | ✓ | 21.7 | 39.5 | 49.1 | 11 | 61.2

Table 7: Test on post-pretraining (P-PT) on (Ours)-meanP model with
HowTo100M-380k dataset. ZS: zero-shot, FT: fine-tuning.

2D/3D | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---
MSR-VTT
2D | 43.1 | 70.4 | 80.8 | 2 | 16.2
3D | 41.6 | 69.9 | 79.5 | 2 | 17.3
MSVD
2D | 46.2 | 76.1 | 84.6 | 2 | 10.0
3D | 44.0 | 73.6 | 83.0 | 2 | 11.3
LSMDC
2D | 20.7 | 38.9 | 47.2 | 13 | 65.3
3D | 20.8 | 40.6 | 49.3 | 11 | 61.0

Table 8: Test 2D and 3D patch linear on (Ours)-meanP.

###  4.5 Post-pretraining on Video Dataset

Our model is built on the pre-trained CLIP, which is an image pre-training
model. To solve this data type (image v.s. video) variance, we conduct a
preliminary exploration on the post-pretraining of our model on Howto100M-380k
video dataset, and report the results for both zero-shot and fine-tune. From
the table 7, we can see that the performance increases for both zero-shot and
fine-tuning setting. The increase of zero-shot is much larger, which shows
that post-pretraining with the same data type (video) can learn general
knowledge and directly transfer to the task. In addition, the fine-tuning on
the post-pretrained model also improves the performance on both LSMDC and MSVD
datasets and achieves approximate results for the MSR-VTT dataset. In future
work, we will explore the capability of pre-training with an even larger
dataset.

###  4.6 Sampling strategy

We conduct three different sampling strategies for video. ‘Head’ is to sample
the first frames at the beginning of the video, ‘Tail’ is to select the last
frames at the end of the video, and ‘Uniform’ is to sample the whole frames of
the video uniformly. The experimental results show that ‘Uniform’ is
relatively a good choice, and ‘Head’ is comparable. The ‘Tail’ sampling
strategy is unlikely to be used.

###  4.7 2D/3D Patch Linear

We conduct the comparison on 2D and 3D linear mentioned in Section 3.1. Table
8 presents the performance of them. Against our expectation that 3D patch
linear can extract temporal information among frames and generate better
discriminant features and performance, the 3D linear generate worse results
than 2D linear on both MSR-VTT and MSVD. We suppose that the CLIP is trained
for 2D linear instead of 3D linear, and the discrepant initialization on 3D
linear makes it hard to learn the temporal information. We will pretrain on a
large video-text dataset to unleash its potential in future work.

##  5 Conclusion

In this paper, we use the pretrained CLIP as our backbone to solve the video
clip retrieval task from frame-level input. We employ parameter-free type,
sequential type, and tight type similarity calculator to obtain the final
results. The experimental results demonstrate the effectiveness of our model
and achieve the SOTA results on MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo.
Besides, we give serval insights from our empirical studies: 1) image feature
can also promote the video-text retrieval, 2) post-pretrain on even
outstanding image-text pretrained CLIP can further improve the performance on
video-text retrieval, 3) 3D patch linear projection and sequential type
similarity are promising approaches on the retrieval task, and 4) The CLIP
used on video-text retrieval is learning-rate sensitivity.

## References

  * Agarap (2018) Abien Fred Agarap. 2018.  Deep learning using rectified linear units (relu).  _arXiv preprint arXiv:1803.08375_.
  * Amrani et al. (2021) Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. 2021.  Noise estimation using density estimation for self-supervised multimodal learning.  In _AAAI_.
  * Arnab et al. (2021) Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021.  Vivit: A video vision transformer.  _arXiv preprint arXiv:2103.15691_.
  * Bain et al. (2021) Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  _arXiv preprint arXiv:2104.00650_.
  * Bertasius et al. (2021) Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021.  Is space-time attention all you need for video understanding?  _arXiv preprint arXiv:2102.05095_.
  * Chen and Dolan (2011) David Chen and William Dolan. 2011.  Collecting highly parallel data for paraphrase evaluation.  In _ACL-HLT_ , pages 190–200.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Croitoru et al. (2021) Ioana Croitoru, Simion-Vlad Bogolin, Yang Liu, Samuel Albanie, Marius Leordeanu, Hailin Jin, and Andrew Zisserman. 2021.  Teachtext: Crossmodal generalized distillation for text-video retrieval.  _arXiv preprint arXiv:2104.08271_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , pages 4171–4186.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_.
  * Dzabraev et al. (2021) Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko. 2021\.  Mdmmt: Multidomain multimodal transformer for video retrieval.  _arXiv preprint arXiv:2103.10699_.
  * Feichtenhofer et al. (2019) Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019.  Slowfast networks for video recognition.  In _ICCV_ , pages 6202–6211.
  * Gabeur et al. (2020) Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020.  Multi-modal transformer for video retrieval.  In _ECCV_ , volume 5.
  * Gers et al. (2002) Felix A Gers, Nicol N Schraudolph, and Jürgen Schmidhuber. 2002.  Learning precise timing with lstm recurrent networks.  _Journal of machine learning research_ , 3(Aug):115–143.
  * Hendricks et al. (2017) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017.  Localizing moments in video with natural language.  In _ICCV_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Kaufman et al. (2017) Dotan Kaufman, Gil Levi, Tal Hassner, and Lior Wolf. 2017.  Temporal tessellation: A unified approach for video analysis.  In _ICCV_ , pages 94–104.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _ICLR_.
  * Kiros et al. (2014) Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2014.  Unifying visual-semantic embeddings with multimodal neural language models.  _arXiv preprint arXiv:1411.2539_.
  * Krishna et al. (2017a) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017a.  Dense-captioning events in videos.  In _ICCV_ , pages 706–715.
  * Krishna et al. (2017b) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017b.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _Int. J. Comput. Vis._ , 123(1):32–73.
  * Lei et al. (2021) Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. 2021.  Less is more: Clipbert for video-and-language learningvia sparse sampling.  In _CVPR_.
  * Li et al. (2020) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020\.  HERO: Hierarchical encoder for video+ language omni-representation pre-training.  _arXiv preprint arXiv:2005.00200_.
  * Liu et al. (2021) Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, and Zhongyuan Wang. 2021.  Hit: Hierarchical transformer with momentum contrast for video-text retrieval.  _arXiv preprint arXiv:2103.15049_.
  * Liu et al. (2019) Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. 2019.  Use what you have: Video retrieval using representations from collaborative experts.  _arXiv preprint arXiv:1907.13487_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  SGDR: stochastic gradient descent with warm restarts.  In _ICLR_.
  * Luo et al. (2020) Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020.  UniVL: A unified video and language pre-training model for multimodal understanding and generation.  _arXiv preprint arXiv:2002.06353_.
  * Miech et al. (2020) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020.  End-to-End Learning of Visual Representations from Uncurated Instructional Videos.  In _CVPR_.
  * Miech et al. (2019) Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  _ICCV_.
  * Mithun et al. (2018) Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. 2018.  Learning joint embedding with multimodal cues for cross-modal video-text retrieval.  In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_ , pages 19–27.
  * Patrick et al. (2021) Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F. Henriques, and Andrea Vedaldi. 2021.  Support-set bottlenecks for video-text representation learning.  In _ICLR_.
  * Portillo-Quintero et al. (2021) Jesús Andrés Portillo-Quintero, José Carlos Ortiz-Bayliss, and Hugo Terashima-Marín. 2021.  A straightforward framework for video retrieval using clip.  _arXiv preprint arXiv:2102.12443_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Rohrbach et al. (2015) Anna Rohrbach, Marcus Rohrbach, and Bernt Schiele. 2015.  The long-short story of movie description.  In _GCPR_ , volume 9358, pages 209–221. Springer.
  * Rouditchenko et al. (2020) Andrew Rouditchenko, Angie Boggust, David Harwath, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, et al. 2020.  Avlnet: Learning audio-visual language representations from instructional videos.  _arXiv preprint arXiv:2006.09199_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _ACL_.
  * Torabi et al. (2016) Atousa Torabi, Niket Tandon, and Leonid Sigal. 2016.  Learning language-visual embedding for movie understanding with natural-language.  _arXiv preprint arXiv:1609.08124_.
  * Tran et al. (2015) Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015\.  Learning spatiotemporal features with 3d convolutional networks.  In _ICCV_ , pages 4489–4497.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_ , pages 5998–6008.
  * Venugopalan et al. (2015) Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, and Kate Saenko. 2015.  Translating videos to natural language using deep recurrent neural networks.  In _NAACL-HLT_ , pages 1494–1504.
  * Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In _ECCV_ , pages 318–335.
  * Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016.  Msr-vtt: A large video description dataset for bridging video and language.  In _CVPR_ , pages 5288–5296.
  * Yu et al. (2016a) Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. 2016a.  Video paragraph captioning using hierarchical recurrent neural networks.  In _CVPR_ , pages 4584–4593.
  * Yu et al. (2018) Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018.  A joint sequence fusion model for video question answering and retrieval.  In _ECCV_ , pages 487–503.
  * Yu et al. (2016b) Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2016b.  Video captioning and retrieval models with semantic attention.  In _ECCVLSMDC2016 Workshop_.
  * Yu et al. (2017) Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2017.  End-to-end concept word detection for video captioning, retrieval, and question answering.  In _CVPR_ , pages 3261–3269.
  * Zhang et al. (2018) Bowen Zhang, Hexiang Hu, and Fei Sha. 2018.  Cross-modal and hierarchical modeling of video and text.  In _ECCV_ , pages 385–401.
  * Zhu and Yang (2020) Linchao Zhu and Yi Yang. 2020.  Actbert: Learning global-local video-text representations.  In _CVPR_.

##  Appendix A Video-to-Text Retrieval

Table A1-A3 present the video-to-text retrieval results of CLIP4Clip on MSR-
VTT, LSMDC, MSVD, ActivityNet, and DiDeMo.

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
Zero-shot
CLIP-straighta | W | ✓ | 27.2 | 51.7 | 62.6 | 5 | -
Training-7K
HowTo100Mb | H+M | ✓ | 16.8 | 41.7 | 55.1 | 8 | -
Training-9K
CEc | M |  | 20.6 | 50.3 | 64.0 | 5.3 | -
MMTd | H+M |  | 27.0 | 57.5 | 69.7 | 3.7 | -
AVLnete | H+M |  | 28.5 | 54.6 | 65.2 | 4 | -
SSBf | H+M |  | 28.5 | 58.6 | 71.6 | 3 | -
HiTg | H+M |  | 32.1 | 62.7 | 74.1 | 3 | -
TT-CE+h | M |  | 32.1 | 62.7 | 75.0 | 3 | -
1-8[3pt/4pt] (Ours)-meanP | W+M | ✓ | 43.1 | 70.5 | 81.2 | 2 | 12.4
(Ours)-seqLSTM | W+M | ✓ | 42.8 | 71.0 | 80.4 | 2 | 12.3
(Ours)-seqTransf | W+M | ✓ | 42.7 | 70.9 | 80.6 | 2 | 11.6
(Ours)-tightTransf | W+M | ✓ | 40.6 | 69.5 | 79.5 | 2 | 13.6

Table A1: Results of video-to-text retrieval on MSR-VTT dataset. ‘Training-7K’
follows the data splits from Miech et al. (2019) and ‘Training-9K’ follows the
data splits from Gabeur et al. (2020). They have the same test set but
different training set. The column ‘TrainD’ shows the datasets used for pre-
training and training, where M, H, W denote MSR-VTT, HowTo100M Miech et al.
(2019) and WIT Radford et al. (2021). The column ‘E2E’ with ✓means training
from raw video in an end-to-end manner. The baseline methods are aCLIP-
straight Portillo-Quintero et al. (2021), bHowTo100M Miech et al. (2019), cCE
Liu et al. (2019), dMMT Gabeur et al. (2020), eAVLnet Rouditchenko et al.
(2020), fSSB Patrick et al. (2021), gHiT Liu et al. (2021), hTT-CE+ Croitoru
et al. (2021).

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
CLIP-straighta | W | ✓ | 59.9 | 85.2 | 90.7 | 1 | -
TT-CE+b | W |  | 27.1 | 55.3 | 67.1 | 4 | -
(Ours)-meanP | W+M | ✓ | 56.6 | 79.7 | 84.3 | 1 | 7.6
(Ours)-seqLSTM | W+M | ✓ | 52.5 | 74.0 | 78.1 | 1 | 14.7
(Ours)-seqTransf | W+M | ✓ | 62.0 | 87.3 | 92.6 | 1 | 4.3
(Ours)-tightTransf | W+M | ✓ | 54.3 | 85.3 | 91.0 | 1 | 6.0

Table A2: Results of video-to-text retrieval on MSVD dataset. In the column
‘TrainD’, M and W denote training on MSVD and WIT Radford et al. (2021), and
CW means CC3M Sharma et al. (2018) plus WebVid-2M Bain et al. (2021). The
column ‘E2E’ with ✓means training from raw video in an end-to-end manner. The
baseline method is aCLIP-straight Portillo-Quintero et al. (2021), bTT-CE+
Croitoru et al. (2021).

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
JSFusiona | L | ✓ | 12.3 | 28.6 | 38.9 | 20 | -
CLIP-straightb | L | ✓ | 6.8 | 16.4 | 22.1 | 73 | -
TT-CE+c | L |  | 17.5 | 36.0 | 45.0 | 14.3 | -
(Ours)-meanP | W+L | ✓ | 20.6 | 39.4 | 47.5 | 13 | 56.7
(Ours)-seqLSTM | W+L | ✓ | 20.9 | 40.7 | 49.1 | 11 | 53.9
(Ours)-seqTransf | W+L | ✓ | 20.8 | 39.0 | 48.6 | 12 | 54.2
(Ours)-tightTransf | W+L | ✓ | 17.4 | 36.7 | 45.0 | 15 | 65.3

Table A3: Results of video-to-text retrieval on LSMDC dataset. In the column
‘TrainD’, L and W denote training on LSMDC and WIT Radford et al. (2021), MD
used in Dzabraev et al. (2021) denotes a combined multidomain dataset
containing MSR-VTT, LSMDC, HowTo100M, etc., and CW means CC3M Sharma et al.
(2018) plus WebVid-2M Bain et al. (2021). The column ‘E2E’ with ✓means
training from raw video in an end-to-end manner. The baseline methods are
aJSFusion Yu et al. (2018), bCLIP-straight Portillo-Quintero et al. (2021),
cTT-CE+ Croitoru et al. (2021).

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
FSEa | A |  | 16.7 | 43.1 | - | 7.0 | -
CEb | A |  | 17.7 | 46.6 | - | 6.0 | 24.4
HSEa | A |  | 18.7 | 48.1 | - | - | -
MMTc | H+A |  | 28.9 | 61.1 | - | 4.0 | 17.1
SSBd | H+A |  | 28.7 | 60.8 | - | 2.0 | -
TT-CE+e | A |  | 23.0 | 56.1 | - | 4.0 | -
(Ours)-meanP | W+A | ✓ | 42.5 | 74.1 | 85.8 | 2.0 | 6.6
(Ours)-seqLSTM | W+A | ✓ | 42.6 | 73.4 | 85.6 | 2.0 | 6.7
(Ours)-seqTransf | W+A | ✓ | 41.4 | 73.7 | 85.3 | 2.0 | 6.7
(Ours)-tightTransf | W+A | ✓ | 18.9 | 49.6 | 65.8 | 6.0 | 16.3

Table A4: Results of video-to-text retrieval on ActivityNet dataset. In the
column ‘TrainD’, A, H, and W denote training on ActivityNet, HowTo100M Miech
et al. (2019), and WIT Radford et al. (2021). The column ‘E2E’ with ✓means
training from raw video in an end-to-end manner. The baseline methods are
aFSE,HSE Zhang et al. (2018), bCE Liu et al. (2019), cMMT Gabeur et al.
(2020), dSSB Patrick et al. (2021), eTT-CE+ Croitoru et al. (2021).

Methods | TrainD | E2E | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MdR$\downarrow$ | MnR$\downarrow$
---|---|---|---|---|---|---|---
S2VTa | D | ✓ | 13.2 | 33.6 | - | 15.0 | -
FSEb | D |  | 13.1 | 33.9 | - | 12.0 | -
CEc | D |  | 15.6 | 40.9 | - | 8.2 | 42.4
TT-CE+d | D |  | 21.1 | 47.3 | 61.1 | 6.3 | -
(Ours)-meanP | W+D | ✓ | 42.5 | 70.6 | 80.2 | 2.0 | 11.6
(Ours)-seqLSTM | W+D | ✓ | 42.4 | 69.2 | 79.2 | 2.0 | 11.8
(Ours)-seqTransf | W+D | ✓ | 41.4 | 68.2 | 79.1 | 2.0 | 12.4
(Ours)-tightTransf | W+D | ✓ | 21.5 | 51.1 | 64.8 | 5.0 | 22.4

Table A5: Results of video-to-text retrieval on DiDeMo dataset. In the column
‘TrainD’, D and W denote training on DiDeMo and WIT Radford et al. (2021). The
column ‘E2E’ with ✓means training from raw video in an end-to-end manner.
$\dagger$ means that the candidate video is concatenated using ground truth
proposals. The baseline methods are aS2VT Venugopalan et al. (2015), bFSE
Zhang et al. (2018), cCE Liu et al. (2019), dClipBERT Lei et al. (2021),
eFrozen Bain et al. (2021), dTT-CE+ Croitoru et al. (2021).