# Advancing High-Resolution Video-Language Representation
with Large-Scale Video Transcriptions

Hongwei Xue, Tiankai Hang11footnotemark: 1, Yanhong Zeng11footnotemark: 1,
Yuchong Sun11footnotemark: 1,
Bei Liu, Huan Yang, Jianlong Fu, Baining Guo
Microsoft Research Asia
<EMAIL_ADDRESS>Equal contribution in alphabetical order. This work was performed when Hongwei
Xue, Tiankai Hang, Yanhong Zeng and Yuchong Sun were visiting Microsoft
Research Asia as research interns. Corresponding authors: Bei Liu, Huan Yang,
Jianlong Fu.

###### Abstract

We study joint video and language (VL) pre-training to enable cross-modality
learning and benefit plentiful downstream VL tasks. Existing works either
extract low-quality video features or learn limited text embedding, while
neglecting that high-resolution videos and diversified semantics can
significantly improve cross-modality learning. In this paper, we propose a
novel High-resolution and Diversified VIdeo-LAnguage pre-training model (HD-
VILA) for many visual tasks. In particular, we collect a large dataset with
two distinct properties: 1) the first high-resolution dataset including 371.5k
hours of 720p videos, and 2) the most diversified dataset covering 15 popular
YouTube categories. To enable VL pre-training, we jointly optimize the HD-VILA
model by a hybrid Transformer that learns rich spatiotemporal features, and a
multimodal Transformer that enforces interactions of the learned video
features with diversified texts. Our pre-training model achieves new state-of-
the-art results in 10 VL understanding tasks and 2 more novel text-to-visual
generation tasks. For example, we outperform SOTA models with relative
increases of 38.5% R@1 in zero-shot MSR-VTT text-to-video retrieval task, and
53.6% in high-resolution dataset LSMDC. The learned VL embedding is also
effective in generating visually pleasing and semantically relevant results in
text-to-visual manipulation and super-resolution tasks.

##  1 Introduction

Recent years have witnessed an increasing number of videos with the popularity
of appealing video websites and mobile apps (e.g., YouTube, TikTok). As the
rapid development of smartphone cameras, device storage, and 5G networks,
high-quality video creation, and diverse content sharing like travel, sports,
and music become a new fashion. Therefore, the capability of video analytic
and joint high-level understanding with language plays a key role in many
video applications, such as video search [3, 39], video recommendation [6],
and video editing [49, 38]. To facilitate video understanding, we study joint
video and language (VL) pre-training, which is a new paradigm in both natural
language processing [9] and computer vision [53, 7, 18].

Existing video-language understanding models are highly limited in the scale
and scope of video-language datasets. Early datasets (e.g., MSR-VTT [54],
DiDeMo [2], EPIC-KITCHENS [8]) consist of videos and textual descriptions that
are manually annotated by humans. The heavy and expensive annotation cost
limits the scale of data. Moreover, datasets with only descriptive sentences
are limited in complexity and variability that largely hinders generalization
power. Recently, several datasets [3, 37] are proposed by transcriptions along
with videos using ASR (automatic speech recognition), so that the data scale
can be greatly enlarged. One of the most representative works is HowTo100M
[37] that consists of million-scale instructional videos. However, there are
still large gaps between these video datasets and real-scenario videos in
terms of video quality and semantic diversity.

Figure 1: Examples of video clips and ASR generated transcriptions in the
proposed HD-VILA-100M dataset. We present six samples (four frames for each),
with diverse video categories covering HowTo & Style, People & Blog, Sports,
Travel & Event, Pets & Animals, Film & Animation. Relevant words from auto-
generated video transcriptions are manually highlighted in red. [Best viewed
in Color]

To solve the above limitations, we propose the HD-VILA-100M dataset (i.e.,
High-resolution and Diversified VIdeo and LAnguage) that consists of a wide
range of video categories and will benefit a plenty of VL tasks, such as text-
to-video retrieval [39] and video QA [26]. This dataset has the following key
properties: (1) Large: we have collected one of the largest video-language
datasets, which consists of 100M video clip and sentence pairs from 3 million
videos with 371.5K hours in total ($2.8\times$ video hour and $8\times$
average sentence length than HowTo100M [37]). (2) High resolution: all the
videos are 720p which is much higher quality than existing datasets that are
mostly 240p or 360p. (3) Diverse and balanced: we cover a wide range of topics
from the YouTube, with 15 popular categories (e.g., sports, music, autos).
Meanwhile, we ensure a balanced video clip number in each category to ease
underfit problem.

To enable video-language pre-training, effective video representation plays a
key role. Due to computational limitations (e.g., memory), previous works
either 1) adopt simple frame-based encoders, and turn to end-to-end visual
encoding and multimodal fusion [26], or 2) choose advanced spatiotemporal
encoders [5, 50], while having to do visual encoding and multimodal fusion
step-by-step. Few works can learn joint spatiotemporal video representation in
end-to-end video-language pre-training.

In this paper, we propose to utilize hybrid image sequence that consists of
few high-resolution (HR) frames and more low-resolution (LR) neighbor frames
for multiple video learning tasks. Such a design enables end-to-end training
with high-resolution spatiotemporal video representation. To achieve this
goal, we will answer the following questions regarding model designs: (1)
Which HR and LR frames should be sampled? (2) How to learn spatiotemporal
features with the hybrid image sequences? To tackle these two problems, we
first randomly sample HR frames from a video clip to ensure the robustness of
learned video features. LR frames are uniformly sampled from the surrounding
of HR frames considering that middle frames contain similar spatial
information and are critical to temporal feature learning. Second, we propose
to encode HR and LR frames separately while mapping the HR feature to a joint
embedding space with LR features by a hybrid Transformer. Such design ensures
the spatiotemporal representation of videos covers both HR and LR frames in a
learnable way. The learned spatiotemporal feature is further combined with
detailed spatial features, followed by a multimodal Transformer that learns to
optimize video and language embedding in an end-to-end manner.

Our contributions are summarized as follows: 1) We use automatic video
transcriptions to build to-date the largest high-resolution and diversified
video and language datasets; 2) We propose a novel pre-training framework to
learn spatiotemporal information for video representation from hybrid image
sequences that consists of HR and LR frames; 3) Extensive experiments verify
the effectiveness of the learned cross-modality embedding in 10 video
understanding and 2 more text-to-visual generation tasks.

Dataset | Domain | #Video clips | #Sentence | Avg len(sec) | Sent len | Duration(h) | Resolution
---|---|---|---|---|---|---|---
MSR-VTT [54] | open | 10K | 200K | 15.0 | 9.3 | 40 | 240p
DideMo [2] | Flickr | 27K | 41K | 6.9 | 8.0 | 87 | -
LSMDC [41] | movie | 118K | 118K | 4.8 | 7.0 | 158 | 1080p
YouCook II [61] | cooking | 14K | 14K | 19.6 | 8.8 | 176 | -
ActivityNet Caption [24] | action | 100K | 100K | 36.0 | 13.5 | 849 | -
WebVid-2M [3] | open | 2.5M | 2.5M | 18.0 | 12.0 | 13K | 360p
HowTo100M [37] | instructional | 136M | 136M | 3.6 | 4.0 | 134.5K | 240p
HD-VILA-100M (Ours) | open | 100M | 100M | 13.4 | 32.5 | 371.5K | 720p
Table 1: Statistics of HD-VILA-100M and its comparison with existing video-
language datasets.

##  2 Related Work

#### Video Representation

Video representation are typically designed with 2D/3D CNNs [45, 16, 47, 5,
50] or Transformers [4]. Pioneering works of VL pre-training [44, 43, 34, 39,
62] adopt pre-extracted video features (e.g., S3D [59], I3D [5]) for video
representation. While in image-language pre-training, researchers find that
end-to-end training will decrease the domain gap of visual representation and
improve the generalization for image-text tasks [18]. While for video
representation, it is too heavy to make the video-based encoder (e.g., S3D,
I3D, ResNet [16], SlowFast [12]) trainable. Thus, some works [26, 57] utilize
the image-based encoder (e.g., ResNet [16], ViT [10]) with a sparse sampling
mechanism to make the visual encoder trainable. In this paper, we explore how
to make a video encoder trainable in consideration of both spatial and
temporal features.

#### Video-Language Pre-Training

Vision and language pre-training has attracted extensive attention in very
recent years. Aligned with the success of image-language pre-training [18, 30,
7], video-language pre-training is showing more and more promising potentials
[44, 34, 28, 62, 46, 26, 39]. Among them, some works concentrate on specific
type of downstream tasks such as video-text retrieval [3, 53] and video
question answering [57]. In this paper, we explore to pre-train a generalized
model on diversified and large-scale data to adapt to different video-language
tasks. Video-language pre-training tasks can be mainly categorized into two
types: reconstructive, contrastive. Reconstructive methods [44, 62, 28, 46]
usually adopt an early fusion architecture and aim to reconstruct a masked
part in the visual or textual domain. Typical pre-training tasks are masked
language modeling (MLM), masked frame modeling (MFM), frame order modeling
(FOM). Contrastive methods [43, 35, 15, 57] are inspired by contrastive
learning and target to learn video-text matching. In this paper, we combine
these two types of objectives for the final target.

##  3 Dataset

To facilitate the multimodal representation learning, we collect HD-VILA-100M,
a large-scale, high-resolution, and diversified video-language dataset. In
this section, we will introduce how we collect the videos and the way we
select clips and process their corresponding texts. We have also an overview
of the statistics of the whole dataset. HD-VILA-100M will be released in the
future.

###  3.1 Video Collection

We choose YouTube as the video resource since it covers diverse categories of
videos uploaded by different users, ranging from documentary films by
professional TV channels to everyday vlogs by ordinary users. To cover more
topics, we start from several official topics of YouTube videos. To ensure the
high quality of videos as well as better alignment of video and transcription,
we search on the YouTube website and a video analysis website
111https://socialblade.com/youtube/ to find popular YouTube channels, such as
BBC Earth, National Geography, etc. Videos in these channels and videos
appeared in YouTube-8M [1] and YT-Temporal-180M [57] make up a list of 14
million videos. We only keep videos with subtitles and 720p resolution. We
then limit the time length of each category to 30K hours to avoid long tail.
Finally, we obtain 3.1M videos in total with high-quality and distributed in
15 categories in balance (as in Figure 2).

###  3.2 Video Clip Selection and Text Processing

To effectively generate video-text pairs, we use transcriptions along with the
videos as the language in HD-VILA-100M. Different from traditional video-
language datasets [54, 2] that use manual annotations of descriptive sentences
for videos, transcriptions are available in large quantities and involve
richer information than descriptions. However, many subtitles in YouTube
videos are generated by ASR and are usually fragmentary and lacking
punctuation. To overcome this problem, we utilize an off-the-shelf tool
222https://github.com/ottokart/punctuator2 to split the subtitles for complete
sentences. Then we make video clips by aligning the sentences to corresponding
clips via Dynamic Time Warping using the timestamp of the original subtitles.
After processing, each pair in the HD-VILA-100M consists of a video clip about
13.4 seconds on average and a sentence with 32.5 words on average.

Figure 2: The distribution of categories in HD-VILA-100M dataset: (a) video,
(b) video clip. [Best viewed in Color]

###  3.3 Data Statistics

The detailed data statistics of HD-VILA-100M are listed in Table 1. Compared
with other video-language datasets, HD-VILA-100M is the largest video-language
dataset in terms of duration and word number. More videos indicate richer
visual information contained in HD-VILA-100M and longer sentences mean that
the language includes more detailed and richer semantics. Compared with
HowTo100M [37] which only includes instructional videos, HD-VILA-100M is
derived from a wide range of domains and videos of each category is relatively
balanced as shown in Figure 2. This merit can improve the generalization power
of the pre-trained model. Moreover, all the videos in HD-VILA-100M are in 720p
and the high quality ensures detailed information for video representation
learning. In summary, HD-VILA-100M represents the largest, high-resolution,
and diversified dataset for video and language learning.

##  4 Approach

Figure 3: The framework of HD-VILA. Yellow and green colors indicate HR- and
LR-related input, operation and output, respectively. Hybrid Transformer
learns spatiotemporal representation from HR and LR features. [Best viewed in
Color]

Figure 3 shows the overall framework of High-resolution and Diversified VIdeo-
LAnguage (HD-VILA) model that consists of three parts: (a) hybrid video
encoder, (b) language encoder, and (c) multi-modal joint learning.

###  4.1 Hybrid Video Encoder

Since the video clips in our dataset are long-range with 13.4 seconds on
average, we adopt the strategy of sparsely sampling a sequence of segments
from a video clip and then aggregating their predictions similar to ClipBERT
[26]. As explained in Section 1, for each segment $s_{i}$, we randomly takes
one HR frame at $t$-th timestep
$\boldsymbol{X}^{s_{i}}_{t}\in\mathbb{R}^{3\times H\times W}$ and $2N$
surrounding LR frames
$\\{\boldsymbol{X}^{s_{i}}_{t+kr}\in\mathbb{R}^{3\times\frac{H}{4}\times\frac{W}{4}},k\in(-N,\ldots,-1,1,\ldots,N)\\}$
to build a hybrid image sequence, where $r$ is LR frame sampling rate.

In Figure 3, the hybrid video encoder includes three parts: an HR encoder for
HR frame, an LR encoder for LR neighbor frames and a Hybrid Transformer
$T_{\mathrm{hy}}$ that learns spatiotemporal features by self-attention. HR
encoder consists of a 4-stage ResNet $F_{\mathrm{hr}}$ and an adapter
$D_{\mathrm{hr}}$. LR encoder is a 3-stage ResNet $F_{\mathrm{lr}}$ to encode
LR frames. Note that $F_{\mathrm{hr}}$ and $F_{\mathrm{lr}}$ are learnable to
ensure both HR and LR frames can be encoded in the same space before feeding
into Hybrid Transformer. We extract hybrid spatiotemporal feature
$\mathcal{V}_{\mathrm{hy}}$ of segment $s_{i}$ as the output of
$T_{\mathrm{hy}}$. In addition, we use the HR frame feature extracted by stage
3 of $F_{\mathrm{hr}}$ (denoting as $F^{3}_{\mathrm{hr}}$) as HR input of
$T_{\mathrm{hy}}$:

$\mathcal{V}_{\mathrm{hy}}=T_{\mathrm{hy}}(F_{\mathrm{lr}}(\boldsymbol{X}^{s_{i}}_{t-Nr}),...,\phi(F^{3}_{\mathrm{hr}}(\boldsymbol{X}^{s_{i}}_{t})),...),$
(1)

where $\phi$ is an interpolate operation to align feature size. In
$T_{\mathrm{hy}}$, we adopt Divided Space-Time Attention to encode
spatiotemporal information similar to [4]. We extract detailed spatial feature
$\mathcal{V}_{\mathrm{hr}}$ of segment $s_{i}$ as the output of
$E_{\mathrm{hr}}$ by:

$\mathcal{V}_{\mathrm{hr}}=D_{\mathrm{hr}}(F_{\mathrm{hr}}(\boldsymbol{X}^{s_{i}}_{t})),$
(2)

To adapt the output of the HR encoder to the hybrid spatiotemporal feature
$\mathcal{V}_{\mathrm{hy}}$, $D_{\mathrm{hr}}$ consists of a convolution layer
to adjust the output feature channel, as well as a $2\times 2$ max-pooling
layer for down-sampling. The segment features is the fusion of
$\mathcal{V}_{\mathrm{hr}}$ and $\mathcal{V}_{\mathrm{hy}}$ by a linear layer:

$\mathcal{V}=\boldsymbol{Linear}([\mathcal{V}_{\mathrm{hr}},\mathcal{V}_{\mathrm{hy}}]).$
(3)

###  4.2 Language Encoder and Multi-Modality Joint Embedding Learning

For both language encoder and multi-modality joint embedding learning, we use
self-attention to model the relationship of both uni-modality and multi-
modality by Transformer. More specifically, we adopt a 24-layer,
1024-dimensional Transformer, mirroring the BERT-large and we initialize it
with pre-trained BERT-large parameters. The first 12 layers are used as the
language-only Transformer and the last 12 layers are utilized as multi-modal
Transformer. Language-only Transformer extracts language representation which
is concatenated with video features of a segment as the input of multi-modal
Transformer. We add learnable 1D and 2D position embedding to language and
vision tokens, respectively. Such a modal-independent design has two
advantages. Firstly, it enables to provide powerful embedding for a single-
modal input in downstream tasks. For example, the vision-aware language-only
embedding could be used for language-guided video generation tasks. Secondly,
the two-stream architecture improves the calculation efficiency of similarity
between video and language to linear complexity in some specific downstream
tasks, such as video-language retrieval.

Method | Acc
---|---
ST-VQA [19] | 30.9
Co-Memory [14] | 32.0
AMU [51] | 32.5
Heterogeneous Mem [11] | 33.0
HCRN [25] | 35.6
ClipBERT [26] | 37.4
Ours | 39.2
(a)

Method | Acc
---|---
CT-SAN [56] | 66.4
MLB [23] | 76.1
JSFusion [55] | 83.4
ActBERT PT [62] | 85.7
ClipBERT [26] | 88.2
VideoClip [53] | 92.1
Ours | 96.3
(b)

Method | Action | Trans | Frame
---|---|---|---
ST-VQA [19] | 60.8 | 67.1 | 49.3
Co-Memory [14] | 68.2 | 74.3 | 51.5
PSAC [29] | 70.4 | 76.9 | 55.7
HCRN [25] | 75.0 | 81.4 | 55.9
QueST [20] | 75.9 | 81.0 | 59.7
ClipBERT [26] | 82.8 | 87.8 | 60.3
Ours | 84.0 | 89.0 | 60.5
(c)

Table 2:  Comparison of HD-VILA with state-of-the-art methods on video
question answering tasks. (a) Results of ST-VQA and Co-Memory are implemented
by [11]. (b) Results of CT-SAN and MLB are implemented by [55].

###  4.3 Pre-Training Tasks

We adopt two pre-training tasks in HD-VILA: video-language matching to enhance
cross-modal matching and masked language modeling (MLM) to encourage the
mapping between visual and language tokens in fine-grained level. In
particular, since the matching between video and language is somewhat weak
compared with the video description dataset, we apply contrastive video-
language matching to take advantage of large data.

#### Contrastive Video-Language Matching

To align the feature space of video and language, we use a contrastive loss to
maximize the similarity of a video clip and a sentence. Specifically, we treat
matched pairs in a batch as positives, and all other pairwise combinations as
negatives:

$\displaystyle\mathcal{L}_{v2t}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp\left(v_{i}^{\top}t_{i}/\tau\right)}{\sum_{j=1}^{B}\exp\left(v_{i}^{\top}t_{j}/\tau\right)}$
(4) $\displaystyle\mathcal{L}_{t2v}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp\left(t_{i}^{\top}v_{i}/\tau\right)}{\sum_{j=1}^{B}\exp\left(t_{i}^{\top}v_{j}/\tau\right)},$

where $v_{i}$ and $t_{j}$ are the normalized embeddings of $i$-th video and
$j$-th sentence in a batch of size $B$ and $\tau$ is the temperature. Video
and sentence features are computed by our hybrid video encoder and language
encoder. The mean of segment embeddings is used as the video-level embedding.

#### Masked Language Modeling

We adopt Masked Language Modeling (MLM) to better build the mapping between
visual and language domain. MLM aims to predict the ground-truth labels of
masked text tokens from the contextualized tokens:

$\mathcal{L}_{\mathrm{MLM}}=-\mathbb{E}_{(\mathcal{W},\mathcal{V})}\log
p\left(w_{i}\mid\mathcal{W}_{\backslash i},\mathcal{V}\right),$ (5)

where $\mathcal{W}$ denotes the text embedding token set, $\mathcal{V}$
denotes the visual token set, and $w_{i}$ denotes the masked token.
$(\mathcal{W},\mathcal{V})$ is sampled from the distribution of text-video
pairs. We adopt the same masking strategy as in BERT and use an MLP as the MLM
head to output logits over vocabulary, which is then computed as the negative
log-likelihood loss for the masked token. We aggregate the logits of different
segments to derive a consensus, so that MLM is able to be calculated in video-
level as we adopt in the approach.

##  5 Experiments

In this section, we will conduct extensive experiments to evaluate the
proposed HD-VILA pre-training model. To specify our model training, we first
introduce implementation details, and then report the state-of-the-art
performances in plentiful downstream video-language understanding and visual
generation tasks, followed by ablation studies to verify each design module.

###  5.1 Pre-training Details

Inspired by the idea of “align before fuse”[27], we adopt a two-stage fashion
for pre-training on HD-VILA-100M dataset. In the first stage, we perform a
contrastive video-language matching task on the entire dataset to learn cross-
modality alignment. In the second stage, pre-training tasks are performed by
MLM to facilitate understanding tasks. For video encoder, we use ResNet-50 for
$F_{hr}$ and $F_{lr}$, and a four-layer transformer with 16 heads and 1024
hidden size for $T_{hy}$. We empirically divide a video clip into two segments
and sample seven frames for each. In this setting, the two segments can cover
about 6s video content, which are adequate to model the video clips in our
dataset. Besides, we randomly crop $640\times 1024$ areas for the middle high-
resolution frames, and select aligned $160\times 256$ areas for low-resolution
neighboring frames. The size of resultant feature map before feeding into the
multimodal Transformer is $10\times 16$. For language processing, we follow
BERT [9] to adopt the WordPiece tokenizer to split a sentence into word tokens
with a max length of 50.

In pre-training, we use AdamW optimizer [33] with an initial learning rate of
5e-5 and a fixed weight decay of 1e-3. We also employ a linear decay learning
rate schedule with a warm-up strategy. We train our model by using 64 NVIDIA
Tesla V100 GPUs. The batch size is set to 512 and the contrastive similarity
is calculated on gathered features from all GPUs. The model is trained with
five epochs for stage one and five epochs for stage two, till the evaluation
metrics in our validation sets saturate. In downstream tasks, we keep the same
model configuration if not otherwise specified.

Method |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$ |  MedR $\downarrow$
---|---|---|---|---
HowTo100M [37] | 14.9 | 40.2 | 52.8 | 9.0
CE [31] | 20.9 | 48.8 | 62.4 | 6.0
DECEMBERT [46] | 17.5 | 44.3 | 58.6 | 9.0
HERO [28] | 16.8 | 43.4 | 57.7 | -
ClipBERT [26] | 22.0 | 46.8 | 59.9 | 6.0
VLM [52] | 28.1 | 55.5 | 67.4 | 4.0
MMT [13] | 26.6 | 57.1 | 69.6 | 4.0
Support Set [39] | 30.1 | 58.5 | 69.3 | 3.0
VideoCLIP [53] | 32.2 | 62.6 | 75.0 | -
Ours | 35.0 | 65.2 | 77.2 | 3.0
Zero-shot |  |  |  |
HT MIL-NCE [35] | 9.9 | 24.0 | 32.4 | 29.5
Support Set [39] | 8.7 | 23.0 | 31.1 | 31.0
VideoCLIP [53] | 10.4 | 22.2 | 30.0 | -
Ours | 14.4 | 31.6 | 41.6 | 17.5
Table 3: Comparison of text-to-video retrieval in MSR-VTT [54]. We gray out
some lines to highlight fair comparisons with traditional retrieval models and
general pre-training models. This mark is also applicable to Table 5, 6.

###  5.2 Video Question and Answering

#### Datasets

(a) MSRVTT-QA [51] is created based on video and captions in MSR-VTT,
containing 10K videos and 243K open-ended questions. Given a question in a
complete sentence, the model selects an answer from a pre-defined set. (b)
MSRVTT multiple-choice test [55] is a multiple-choice task with videos as
queries, and captions as answers. Each video contains five candidate captions,
with only one positive match. The benchmark has 2,990 questions for the
multiple-choice test. (c) TGIF-QA [19] contains 165K QA pairs on 72K GIF
videos. We experiment with three TGIF-QA tasks: Action is defined as a
multiple-choice task to identify an action that has been repeated in a video.
Transition aims to identify the state before or after another state. FrameQA
is about open-ended questions about the given video. The task objective is
identical to MSRVTT-QA.

#### Implementation Details

For TGIF Action and Transition, we respectively concatenate five candidate
answers with the question into five sequences. On top of the [CLS] token of
the question, we train a two-layer MLP to predict the confidence of the five
candidates with cross-entropy loss. For MSRVTT-QA and TGIF Frame, we encode
the answers in a one-hot fashion, and train 2-layer MLP classifier over all
answer candidates with a cross-entropy loss on-top of the [CLS] token of the
question. For MSRVTT Multiple-choice, we directly choose the answer with the
highest similarity. We choose one segment containing seven frames for TGIF and
MSR-VTT. For MSR-VTT, we resize HR frame of each segment to 720p and LR frames
to 180p. Due to the various resolution of videos in TGIF, we set the max edge
of LR frames to 192 and HR frame to 768, then all pad to square frames. We set
max epoch to 80 for TGIF and 20 for MSRVTT-QA with 5e-5 learning rate with a
multi-step decay. We report the results of model with best performance on
validation set. We set the max batch size to fine-tune on 8 V100 32G GPUs.

#### Results

The results of HD-VILA on video QA are shown in Table 2. We can find that our
model outperforms existing methods on five tasks in all the three datasets,
with 1.2, 1.2 and 0.2 absolute improvements on Action, Trans and Frame tasks
with TGIF-QA dataset. The limited gain of Frame is because Frame focuses on
one frame while hindering the advantage of our hybrid image sequence. On
MSRVTT-QA and MSRVTT multiple-choice tests, we achieve 4.8% and 4.6% relative
improvement over SOTA methods. Among all the compared methods, ClipBERT [26]
and ActBERT [62] are two pre-training models. We can see that pre-training
with more data will marginally improve the performance. Compared with ClipBERT
which is pre-trained on an image-language dataset, videos provide richer
information. Note that the language used in ClipBERT for pre-training is more
closer to the downstream dataset in both content and length while the language
in HD-VILA-100M has a domain gap with TGIF and MSR-VTT languages. This further
indicates the generalization of the video representation learned by our HD-
VILA.

Method |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$ |  MedR $\downarrow$
---|---|---|---|---
HERO [28] | 2.1 | - | 11.4 | -
S2VT [48] | 11.9 | 33.6 | - | 13.0
FSE [58] | 13.9 | 36.0 | - | 11.0
CE [31] | 16.1 | 41.1 | - | 8.3
ClipBERT [26] | 20.4 | 48.0 | 60.8 | 6.0
Ours | 26.0 | 54.8 | 69.0 | 4.0
Table 4: Comparison of text-to-video retrieval on DiDeMo [17].  Method |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$ |  MedR $\downarrow$
---|---|---|---|---
JSFusion [55] | 9.1 | 21.2 | 34.1 | 36.0
MEE [36] | 9.3 | 25.1 | 33.4 | 27.0
CE [31] | 11.2 | 26.9 | 34.8 | 25.3
MMT [13] | 12.9 | 29.9 | 40.1 | 19.3
Ours | 17.2 | 32.9 | 43.0 | 16.0
Table 5: Comparison of text-to-video retrieval on LSMDC [41]. Method |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$ |  MedR $\downarrow$
---|---|---|---|---
FSE [58] | 18.2 | 44.8 | 89.1 | 7.0
CE [31] | 18.2 | 47.7 | 91.4 | 6.0
HSE [58] | 20.5 | 49.3 | - | -
ClipBERT [26] | 21.3 | 49.0 | - | 6.0
MMT [13] | 28.7 | 61.4 | 94.5 | 3.3
Support Set [39] | 29.2 | 61.6 | 94.7 | 3.0
Ours | 27.4 | 56.0 | 93.1 | 4.0
Table 6: Comparison of text-to-video retrieval on ActivityNet [24].

###  5.3 Video-Text Retrieval

#### Datasets

We conduct video-text retrieval experiments on four datasets. (a) MSR-VTT [54]
contains 10K YouTube videos with 200K descriptions. We follow previous works
[55, 31], training models on 9K videos, and reporting results on the 1K-A test
set. (b) DiDeMo [2] consists of 10K Flickr videos annotated with 40K
sentences. We follow [31, 58] to evaluate paragraph-to-video retrieval, where
all descriptions for a video are concatenated to form a single query. (c)
LSMDC [41] consists of 118,081 video clips sourced from 202 movies. Each video
has a caption. The validation set contains 7,408 clips and evaluation is
conducted on a test set of 1,000 videos from movies disjoint from the train
and val sets. (d) ActivityNet Captions [24] contains 20K YouTube videos
annotated with 100K sentences. We follow the paragraph-to-video retrieval
protocols [58, 31] training on 10K videos and reporting results on the val1
set with 4.9K videos.

Figure 4: Text-guided manipulation compared with StyleCLIP [38] and TediGAN
[49]. Our model is able to handle complex descriptions and edit the inputs
according to the target attributes (highlighted in red) better. All the inputs
are of $1024\times 1024$ size.

#### Implementation Details

Due to the various resolution for videos in downstream datasets, we resize HR
frame of each segment to 720p and LR frames to 180p. We adopt stage one model
and the same training methods and objective for fine-tuning. We adjust the
number of sampled segments and frames according to the average time of videos
for each dataset to cover about half of the video. For evaluation, we double
the number of segments. We train 100 epochs at most with 5e-6 learning rate
with a multi-step decay. We report the results of model with best performance
on validation set. If no validation set, we report the results of the last
model. To conduct zero-shot evaluation on low-resolution MSR-VTT videos, we
crop a $224\times 320$ patch for each frame and up-sample the middle frames by
$4$ times. We report the result of the last saved model.

Figure 5: Text-guided super-resolution compared with pSp [40] and SR3 [42].
Our model is able to reconstruct more accurate target attributes with
descriptions (e.g., eyeglasses in the third case). All inputs are upsampled
from $16\times 16$ to $1024\times 1024$.

#### Results

Table 3, 4, 5, 6 show the text-to-video results of HD-VILA on four datasets.
For MSR-VTT, we outperform the previous works that are pre-trained on
HowTo100M [37] by large margins. Compared with VideoCLIP [53], we have 38.5%
relatively gains of R@1 in zero-shot settings, which shows the generalization
ability of our learned video feature. Our fine-tuned model outperforms all the
baseline models. In LSMDC, we further obtain much larger relative gains with
53.6% under fair comparison. The movie videos are significantly different from
How-to style videos. Thus the models pre-trained on HowTo100M are difficult to
be adapted to the movie domain. Additional improvement comes from our high-
resolution pre-training since LSMDC also contains high-resolution videos. On
DiDeMo and ActivityNet, the pre-trained model HD-VILA also achieves better
performance. The videos in these two datasets are diversified in both scale
and category, and are much longer. In this setting, models usually need more
capacity for temporal understanding. Since our videos in HD-VILA-100M are
longer and the transcripts have more detailed information, our pre-trained
model HD-VILA performs better on long video retrieval. Note that there are
also pre-training models that are specifically designed for video-text
retrieval task by improving noise contrastive learning like SupportSet [39],
or use more features other than vision and motion like MMT [13]. To make fair
comparison, we gray them out in tables, and just list their results for
reference. Compared with these approaches, we can even outperform them in MSR-
VTT and LSMDC.

###  5.4 Text-to-Visual Generation

Recent studies like StyleCLIP[38] and TediGAN [49] propose to leverage cross-
modal pre-training power to facilitate language-guided generation tasks, and
have obtained some promising results. As shown in their work, the quality of
visual generation results can reflect the quality of cross-modality embedding.
Hence, in this section, we will specify how our pre-trained model can achieve
this task, and verify our learned embedding by showing higher-quality
visualized results compared with SOTA models.

#### Datasets

To conduct this research, we introduce the first Face-Description-Video
Dateset (FDVD). The dataset consists of 613 high-resolution ($1024\times
1024$) videos, resulting in 74,803 frames of human faces. The videos are
collected from Ryerson audio-visual dataset [32] and post-processed following
Karras et al. [21]. We generate ten different text descriptions for each video
following previous works [49]. To increase the diversity of human faces, we
also leverage Multi-modal CelebA-HQ [49, 21] for training.

#### Implementation Details

We follow previous works [49, 38] to leverage a well pre-trained StyleGAN [22]
as our generator, due to its superior performance. In practice, we learn
several linear layers to map the vision and text embedding in HD-VILA to the
latent codes $w^{+}$ used in StyleGAN. Then, images can be generated by the
latent codes. To ensure the visual quality, identity preservation, and
matching with descriptions of the generated results, we carefully choose a
mean-square-error, a LPIPS loss [60], an identity loss [40] and a text-vision
matching loss [42] for optimization following previous works [38, 49, 40].

#### Text-to-Visual Manipulation

We compare our model with the recent state-of-the-art text-guided manipulation
models, StyleCLIP [38] and TediGAN [49] in Figure 4. The results show that our
model is able to edit the target attributes of inputs according to text
descriptions. For example, in the first case in Figure 4, our model turns the
hair to wavy hair and also wears lipstick on the lips, where StyleCLIP and
TediGAN fail to wear lipstick on the face.

Type | Size |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$ |  MedR $\downarrow$
---|---|---|---|---|---
HowTo | 720p | 3.3 | 8.2 | 13.5 | 113.0
Ours | 360p | 3.9 | 11.0 | 18.3 | 67.0
Ours | 720p | 5.5 | 13.1 | 20.5 | 58.0
Table 7: Ablation study on pre-training data. We report results of zero-shot MSR-VTT retrieval. #HR | #LR |  R@1 $\uparrow$ |  R@5 $\uparrow$ |  R@10 $\uparrow$ |  MedR $\downarrow$
---|---|---|---|---|---
1 | 0 | 14.3 | 34.6 | 49.3 | 11.0
0 | 10 | 25.5 | 54.6 | 67.8 | 5.0
1 | 6 | 31.2 | 59.6 | 71.0 | 4.0
1 | 10 | 31.8 | 60.1 | 71.8 | 3.0
1 | 14 | 30.1 | 58.4 | 70.4 | 3.0
Table 8: Ablation study on frame selection (trained on 8 GPUs). We report
results of MSR-VTT retrieval, where #HR/#LR are the numbers of high/low-
resolution frames.

#### Text-to-Visual Super-Resolution

We further compare our model with SOTA super-resolution methods SR3 [42] and
pSp [40]. We generate $1024\times 1024$ images from their $16\times 16$ LR
counterparts. Note that this task is extremely challenging due to such low-
resolution inputs. As shown in the second case of Figure 5, SR3 [42] and pSp
[40] can not reconstruct high-quality faces by only using visual information.
Compared with them, our model is able to accurately reconstruct the lipstick
and the straight hair with the help of text description, thanks to the pre-
trained models.

###  5.5 Ablation Studies

In this section, we conduct ablation studies to further verify the
effectiveness of the new HD-VILA-100M dataset, and the proposed hybrid video
encoder. (1) Diversity of HD-VILA-100M. We sample two video subsets from HD-
VILA-100M with two million clip-text pairs for each. One subset only includes
“HowTo” type, while the other consists of diversified and balanced categories
sampled from the full dataset. As shown in Table 7, compared with the “HowTo”
dataset with limited semantics, our diversified pre-training dataset
(indicated as “Ours-720p”) helps to achieve higher performance in the MSR-VTT
retrieval task, with relative 66.7% R@1 gains. We choose MSR-VTT zero-shot
retrieval task for this ablation study, as it is the most widely-used
evaluation task in video-language pre-training. (2) High-resolution of HD-
VILA-100M. We downsample “Ours-720p” subset into lower resolutions
(“Ours-360p”), and observed a significant drop with 29.1% relative decreases
of R@1. Such evaluations demonstrate the superiority of the diversified
categories and higher resolution of the proposed dataset. (3) Numbers of
high/low-resolution frames. As the number of high/low-resolution frames used
for video modeling often plays a key role in video pre-training, we adjust
frame numbers and fine-tune the pre-training model in different settings. As
shown in Table 8, high-resolution frames lead to significant increases
compared with the setting only using low-resolution inputs. In particular, the
setting of 1-HR & 10-LR achieves the best performance, compared with 0-HR &
10-LR (“0” indicates that one branch is removed), and 1-HR & 0-LR, which
demonstrates the rationality of jointly modeling spatial and temporal features
in our approach.

##  6 Conclusion

In this paper, we propose to learn high-resolution and diversified video-
language multi-modal representation by pre-training on large-scale video-
language pairs. To empower pre-training, we introduce a new dataset HD-
VILA-100M which is the largest high-resolution and diversified video-language
dataset. To more efficiently employ the richer information in videos, we
propose a novel pre-training model HD-VILA that learns spatiotemporal
information using HR and LR frames as a hybrid image sequence with a hybrid
Transformer. We carefully select 12 video-language understanding and text-to-
visual generation tasks to show the capability of HD-VILA-100M dataset and the
effectiveness of our pre-trained model.

#### Limitation and Social Impact

The proposed video-language dataset and pre-training model shows the capacity
and generalization of learned VL representation which could benefit many
applications of CV and NLP. This is also a result of much computation resource
and how to reduce the model size and computing effort becomes more essential
for future research. In addition, the usage of user generated data might bring
the risk of bias. We tackle this problem by balancing various video
categories, yet the videos might contain biased content. Moreover, how to
avoid malicious usage of visual generation technique for conscious attack is
also critical. However, these concerns are general to the entire fields and
are not amplified by this work.

## References

  * [1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.  Youtube-8m: A large-scale video classification benchmark.  arXiv preprint arXiv:1609.08675, 2016.
  * [2] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In ICCV, pages 5803–5812, 2017.
  * [3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In ICCV, 2021.
  * [4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, July 2021.
  * [5] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, pages 6299–6308, 2017.
  * [6] Bisheng Chen, Jingdong Wang, Qinghua Huang, and Tao Mei.  Personalized video recommendation through tripartite graph propagation.  In ACM MM, pages 1133–1136, 2012.
  * [7] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In ECCV, pages 104–120, 2020.
  * [8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Scaling egocentric vision: The epic-kitchens dataset.  In ECCV, pages 720–736, 2018.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In NAACL, pages 4171–4186, 2019.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2020.
  * [11] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang.  Heterogeneous memory enhanced multimodal attention model for video question answering.  In CVPR, pages 1999–2007, 2019.
  * [12] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In CVPR, pages 6202–6211, 2019.
  * [13] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal transformer for video retrieval.  In ECCV, pages 214–229, 2020.
  * [14] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia.  Motion-appearance co-memory networks for video question answering.  In CVPR, pages 6576–6585, 2018.
  * [15] Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox.  Coot: Cooperative hierarchical transformer for video-text representation learning.  2020\.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [17] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell.  Localizing moments in video with natural language.  pages 5804–5813, 2017.
  * [18] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu.  Seeing out of the box: End-to-end pre-training for vision-language representation learning.  In CVPR, pages 12976–12985, 2021.
  * [19] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In CVPR, pages 2758–2766, 2017.
  * [20] Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao.  Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering.  In AAAI, pages 11101–11108, 2020.
  * [21] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In ICLR, 2018.
  * [22] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, pages 4401–4410, 2019.
  * [23] Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang.  Hadamard product for low-rank bilinear pooling.  In ICLR, 2016.
  * [24] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  pages 706–715, 2017.
  * [25] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran.  Hierarchical conditional relation networks for video question answering.  In CVPR, pages 9972–9981, 2020.
  * [26] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In CVPR, pages 7331–7341, 2021.
  * [27] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In NeurIPS, 2021.
  * [28] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  In EMNLP, pages 2046–2065, 2020.
  * [29] Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan.  Beyond rnns: Positional self-attention with co-attention for video question answering.  In AAAI, pages 8658–8665, 2019.
  * [30] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In ECCV, pages 121–137, 2020.
  * [31] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman.  Use what you have: Video retrieval using representations from collaborative experts.  pages 73.1–73.14, 2019.
  * [32] Steven R Livingstone and Frank A Russo.  The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.  PloS one, page e0196391, 2018.
  * [33] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  ICLR, 2019.
  * [34] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  arXiv preprint arXiv:2002.06353, 2020.
  * [35] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In CVPR, pages 9879–9889, 2020.
  * [36] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learning a text-video embedding from incomplete and heterogeneous data.  arXiv preprint arXiv:1804.02516, 2018.
  * [37] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV, pages 2630–2640, 2019.
  * [38] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In ICCV, pages 2085–2094, 2021.
  * [39] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F. Henriques, and Andrea Vedaldi.  Support-set bottlenecks for video-text representation learning.  In ICLR, 2021.
  * [40] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In CVPR, pages 2287–2296, 2021.
  * [41] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Joseph Pal, H. Larochelle, Aaron C. Courville, and Bernt Schiele.  Movie description.  IJCV, pages 94–120, 2016.
  * [42] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  arXiv preprint arXiv:2104.07636, 2021.
  * [43] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid.  Learning video representations using contrastive bidirectional transformer.  arXiv preprint arXiv:1906.05743, 2019.
  * [44] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In ICCV, pages 7464–7473, 2019.
  * [45] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, pages 1–9, 2015.
  * [46] Zineng Tang, Jie Lei, and Mohit Bansal.  Decembert: Learning from noisy instructional videos via dense captions and entropy minimization.  In NAACL, pages 2415–2426, 2021.
  * [47] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, pages 4489–4497, 2015.
  * [48] Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J Mooney, and Kate Saenko.  Translating videos to natural language using deep recurrent neural networks.  In HLT-NAACL, 2015.
  * [49] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In CVPR, pages 2256–2265, 2021.
  * [50] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In ECCV, pages 305–321, 2018.
  * [51] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In ACM MM, page 1645–1653, 2017.
  * [52] Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer.  Vlm: Task-agnostic video-language model pre-training for video understanding.  arXiv preprint arXiv:2105.09996, 2021.
  * [53] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  In EMNLP, pages 6787–6800, 2021.
  * [54] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In CVPR, pages 5288–5296, 2016.
  * [55] Youngjae Yu, Jongseok Kim, and Gunhee Kim.  A joint sequence fusion model for video question answering and retrieval.  In ECCV, pages 471–487, 2018.
  * [56] Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim.  End-to-end concept word detection for video captioning, retrieval, and question answering.  In CVPR, pages 3261–3269, 2017.
  * [57] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  arXiv preprint arXiv:2106.02636, 2021.
  * [58] Bowen Zhang, Hexiang Hu, and Fei Sha.  Cross-modal and hierarchical modeling of video and text.  In ECCV, pages 374–390, 2018.
  * [59] Da Zhang, Xiyang Dai, Xin Wang, and Yuan-Fang Wang.  S3d: Single shot multi-span detector via fully 3d convolutional network.  In BMVC, 2018.
  * [60] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, pages 586–595, 2018.
  * [61] Luowei Zhou, Chenliang Xu, and Jason J. Corso.  Towards automatic learning of procedures from web instructional videos.  In AAAI, 2018.
  * [62] Linchao Zhu and Yi Yang.  Actbert: Learning global-local video-text representations.  In CVPR, pages 8746–8755, 2020.