#  HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training

Qinghao Ye Guohai Xu Ming Yan Haiyang Xu
Qi Qian Ji Zhang Fei Huang

DAMO Academy, Alibaba Group
Corresponding Author.

###### Abstract

Video-language pre-training has advanced the performance of various downstream
video-language tasks. However, most previous methods directly inherit or adapt
typical image-language pre-training paradigms to video-language pre-training,
thus not fully exploiting the unique characteristic of video, i.e., temporal.
In this paper, we propose a Hierarchical Temporal-Aware video-language pre-
training framework, HiTeA, with two novel pre-training tasks for modeling
cross-modal alignment between moments and texts as well as the temporal
relations of video-text pairs. Specifically, we propose a cross-modal moment
exploration task to explore moments in videos, which results in detailed video
moment representation. Besides, the inherent temporal relations are captured
by aligning video-text pairs as a whole in different time resolutions with
multi-modal temporal relation exploration task. Furthermore, we introduce the
shuffling test to evaluate the temporal reliance of datasets and video-
language pre-training models. We achieve state-of-the-art results on 15 well-
established video-language understanding and generation tasks, especially on
temporal-oriented datasets (_e.g_., SSv2-Template and SSv2-Label) with 8.6%
and 11.1% improvement respectively. HiTeA also demonstrates strong
generalization ability when directly transferred to downstream tasks in a
zero-shot manner. Models and demo will be available on ModelScope.

##  1 Introduction

Vision and language are two primary signals that constitute the real-world
perception of humanity. With the success of image-language pre-training [8,
30, 59, 27], video-language pre-training [44, 31, 28, 32] has recently
received increasing attention. Large-scale video-language pre-training helps
the model to learn effective multi-modal representation, which has shown
significant improvement on a variety of video-language downstream tasks, such
as video-text retrieval, video question answering and video captioning [62,
64, 5, 60, 70, 33, 42, 56].

Inspired by the success of image-language pre-training paradigm, various
methods [32, 28, 11, 12, 31] have been proposed to adapt it to video-language
pre-training. ClipBERT [26] and Singularity [25] directly build on
representations from image encoders and aggregate them via score aggregation
function and temporal encoder. Furthermore, MIL-NCE [44] and Frozen [2] switch
image encoder to video encoder for spatio-temporal video representation
learning and align the video with corresponding text. In addition, some
advanced pre-training tasks are designed through modeling entity [28],
reconstructing masked patches [11] and predicting frame order [31, 72].
Despite their promising performance on downstream tasks, they treat video
within global perspective illustrated in Figure 1(a), thus failing to consider
fine-grained temporal information and relations which are essential to video-
language pre-training.

Figure 1: Comparison between existing paradigms and ours for video-language
pre-training. (a) Previous methods align video and text within global
perspective as the pretext. (b) We introduce HiTeA by varying video in
different temporal views and modeling cross-modal alignment between moments
and texts, as well as the temporal relations between multi-modal pairs.

Since untrimmed video contains various temporal details, directly treating the
video globally has two main limitations: (1) Less effective in modeling the
fine-grained moment information including atomic actions and moments. As
illustrated in Figure 1(b), we vary time resolutions and generate two views
(long & short) for the input video. As a result, the shot-view video clip
tends to represent the moment information and the long-view video may express
more event-level information. For example, the short-view video clip in Figure
1(b) only describes the moment of "lick fingers" rather than "eating ice
cream". Such fine-grained moment information is hard to be captured by the
long-view video under global event perspective; (2) Ignoring the temporal
relations implicitly existed in the video. Knowing the event expressed by the
text, the moment "eating ice cream" can be inferred from the moment "lick
fingers" shown by short-view video. However, the implicit temporal relations
between the moments are rarely explored in previous works.

To address these problems, we propose a Hierarchical Temporal-Aware video-
language pre-training framework, HiTeA, for both multi-modal understanding and
generation. Except for the standard pre-training tasks, HiTeA introduces two
novel temporal-aware video-language pre-training tasks, named cross-modal
moment exploration (CME) and multi-modal temporal relation exploration (MTRE),
which not only model the fine-grained moments with partial cross-modal
alignment but also capture temporal relations between multi-modal pairs
hierarchically. Specifically, we first generate the long-view and short-view
videos with different time resolutions to build hierarchy of the input video.
Then, based on the similarities of words and short-view video, we select the
most relevant words as positive and leave the rest of the words as hard
negatives. The CME pre-training task is applied to align the positive words
and shot-view video representations in the same embedding space. Moreover, to
capture association between moments and the event, we match different views
for the same video. However, directly matching two views visually would be
noisy due to the background similarity [50]. To this end, we perform multi-
modal alignment between video-text pairs via the MTRE pre-training task. More
specifically, the shot-view video guided by most relevant words and the long-
view video guided by text will be aligned. Empowered by above two novel
temporal-aware video-language pre-training tasks, HiTeA captures both fine-
grained moment information and temporal relations between different views of
video.

In spite of a good performance, recent studies [25, 4] reveal most video-
language downstream datasets are biased towards still objects, scenes, _etc_.,
while the temporal dynamics are negligible. To evaluate the temporal
performance of the video-language pre-training model and temporal reliance of
downstream datasets, we introduce temporal shuffling test for these datasets.
This enables a more comprehensive evaluation of temporal modeling capability
in the video-language pre-training field. Besides, our method achieves
significant improvement on the datasets with heavy temporal reliance.

In summary, our key contributions are the followings:

  * •

We propose a novel hierarchical temporal-aware video-language pre-training
framework with both video-language understanding and generation capabilities.

  * •

We introduce additional temporal-aware pre-training tasks by performing cross-
modal and multi-modal alignment hierarchically, which not only model moment
information with fine-grained semantics but also capture temporal relations
between moments and event.

  * •

Extensive experiments demonstrate the effectiveness of HiTeA, and it achieves
state-of-the-art performance on 15 video-language downstream datasets
including video-text retrieval, video question answering, and video
captioning, especially on temporal-oriented datasets (_e.g_., SSv2-Template
and SSv2-Label) with 8.6% and 11.1% improvement respectively.

Figure 2: Illustration of the proposed HiTeA. We first generate two different
temporal views for the input video, where the long-view is the video itself
and the short-view is randomly truncated from the input video. To explore the
moment revealed in the short-view, cross-modal moment exploration (CME)
selects the candidate words from the input text with
$\mathcal{L}_{\text{CME}}$. Then, we perform multi-modal temporal relation
exploration (MTRE) for modeling the temporal relations between two video-text
pairs with different views by $\mathcal{L}_{\text{MTRE}}$. Note that the
multi-modal encoders and the text features are shared.

##  2 Related Work

##### Video-Language Pre-training

Benefiting from a large number of image/video-text pairs, video-language pre-
training (VLP) exhibits superior capabilities on various video-text
benchmarks. The method of VLP is constantly evolving. Traditional approaches
[53, 73, 39, 31] leverage offline-extracted dense video features for pre-
training to circumvent the expensive computation overhead. In contrast,
ClipBERT [26] suggests that sparse sampling can enable affordable end-to-end
learning and improve performance simultaneously. Recent emerging approaches
[2, 28, 12, 26, 15, 32] adopt this strategy and propose new model
architectures and pre-training tasks. Frozen [2] trains jointly on image and
video datasets via video-text contrastive learning (VTC). ALPRO [28] proposes
a new visually-grounded pre-training task combined with VTC, video-text
matching (VTM) and masked language modeling (MLM) [10] to learn fine-grained
region-entity alignment. LAVENDER [32] formulates all pre-training and
downstream tasks as MLM so that a unified architecture can used for all video-
text tasks. Apart from above representative works, frame order modeling (FOM)
[31, 72] and masked video modeling (MVM) [11] are designed for VLP. However,
the temporal characteristic of video still remains largely unexplored. To this
end, we introduce a novel hierarchical temporal-aware VLP framework which not
only models the fine-grained moment information but also captures their
correlations with different temporal granularities.

##### Temporal Modeling

The temporal characteristic acts as a vital role in VLP since it provides the
model with the capabilities of reasoning and understanding causality. Previous
efforts in this field can be roughly divided into three categories. First,
several methods directly transfer image-text models to video-text tasks by
simply concatenating video frame [29, 27] or building a additional temporal
encoder [40, 41]. Second, some works [2, 28, 11, 32] switch the image encoder
to video encoder for learning spatio-temporal contexts within videos. Third,
HERO [31] and MERLOT [72] design FOM task to explicitly recover the correct
temporal order of shuffled frames. Nonetheless, ATP [4] and Singularity [25]
reveal the existence of a static appearance bias in popular video-language
datasets, and they develop single-frame models to achieve surprisingly strong
performance, comparable or even better than above methods with explicit
temporal modeling. Therefore, they recommend SSv2 [25] and NExT-QA [60]
datasets to test the temporal ability of VLP models. Different from previous
approaches, we vary the temporal resolutions and generate two views of video
so as to construct the temporal hierarchy, which equips the model with the
ability to learn both fine-grained moment information and temporal relations
at the same time.

##  3 Method

###  3.1 Overview

Figure 2 sketches the overview of the HiTeA. In concrete, our model consists
of two unimodal encoders for encoding video and text separately, a multi-modal
encoder for video and text interaction, and a text decoder for generation
which is omitted here for simplicity and detailed in Appendix.

For video representation, previous methods [26, 28, 32] encode the whole input
video as a single-view feature, ignoring the rich temporal details contained
in the video. Thus, we first treat the video into two views with different
time resolutions to build hierarchy of the input video. Specially, the
untrimmed video is regarded as a long-view video $V^{L}$ for capturing event
information, and a video segment is randomly truncated from the input video as
the short-view for capturing moment information denoted as $V^{S}$. Then, we
use the video encoder to encode an arbitrary view of video
$V\in\mathbb{R}^{T\times H\times W}$ into a sequence of embeddings:
$\mathcal{V}=\\{v_{\text{cls}},v_{1},\cdots,v_{M}\\}\in\mathbb{R}^{M\times
D}$, where $M$ is the number of flattened patches, $D$ is hidden size, and
$v_{\text{cls}}$ is the embedding of the visual [CLS] token which provides
global representation of the video. For text representation, we use the text
encoder to transform the text T into a sequence of embeddings:
$\mathcal{T}=\\{w_{\text{cls}},w_{1},\cdots,w_{N}\\}\in\mathbb{R}^{N\times
D}$, where $N$ is length of the text. After that, the multi-modal encoder
takes video features $\mathcal{V}$ and text features $\mathcal{T}$ as inputs
and yields the multi-modal representation $\bm{v}_{\text{cls}}$ for the video.

In order to take full advantage of the different view of the video, we
introduce cross-modal moment exploration (CEM) to explore the proper words or
phrases from input text to align the short-view video with
$\mathcal{L}_{\text{CME}}$ for capturing the moment information in Section
3.2. Furthermore, to model the relations between the short-view video
containing moment information and the long-view video with event information,
we propose multi-modal temporal relation exploration (MTRE) to align the
multi-modal representation of short-view and long-view videos by
$\mathcal{L}_{\text{MTRE}}$ in Section 3.3. Lastly, we introduce the overall
pre-training objective for training the model in Section 3.4.

###  3.2 Cross-Modal Moment Exploration

To understand the fine-grained moment information, the video with short
temporal range (_i.e_. short-view of video) should be aligned with the
corresponding text. However, since the video is partially aligned with the
paired text which describes the whole video, directly aligning short-view of
video with paired text would bring noise to model learning and degrade the
performance. Therefore, we propose a novel pre-training task named cross-modal
moment exploration (CME), which enables the model to understand fine-grained
moment information.

Formally, we first discover the possible positive words for the video in
short-view by computing the cosine similarity of the word embedding sequence
$\\{w_{1},\cdots,w_{N}\\}$ from text encoder and the short-view video
representation $v_{\text{cls}}^{S}$ from video encoder as:

$\displaystyle\mathcal{K}=\\{\pi(1),\cdots,\pi(K)\\},$ (1)

where $\pi:\\{1,\cdots,N\\}\rightarrow\\{1,\cdots,N\\}$ is a permutation
function for ranking such that $s(w_{\pi(1)},v_{\text{cls}}^{S})\geq\cdots\geq
s(w_{\pi(N)},v_{\text{cls}}^{S})$, and $\mathcal{K}$ is the set of selected
word indices, $K$ is the number of possible selected words, and
$s(x,y)=x^{T}y/\|x\|_{2}\|y\|_{2}$ represents the cosine similarity between
$x$ and $y$. After obtaining the words for the video in short-view as the
positive pair, the cross-modal moment exploration loss
$\mathcal{L}_{\text{CME}}$ is computed with negative pairs from other words in
the input text, which is defined as:

$\displaystyle\mathcal{L}_{\text{CME}}=-\frac{1}{B}\sum_{i=1}^{B}\left(\frac{1}{|\mathcal{K}|}\sum_{k\in\mathcal{K}}\log\frac{\exp((v_{\text{cls}}^{S})^{\top}_{i}w_{i,k}/\tau)}{\sum_{n=1}^{N}\exp((v_{\text{cls}}^{S})^{\top}_{i}w_{i,n}/\tau)}\right),$
(2)

where $\tau$ is the learnable temperature hyper-parameter that controls the
sharpness of the output distribution, and it is initialized as 0.07. As a
consequence, the model is able to understand moment information via the
proposed cross-modal exploration scheme.

|  | MSRVTT | DiDeMo | LSMDC | ActivityNet Caption
---|---|---|---|---|---
Method | # PT Data | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
ClipBERT [26] | 0.2M | 22.0 | 46.8 | 59.9 | 20.4 | 48.0 | 60.8 | - | - | - | 21.3 | 49.0 | 63.5
Frozen [2] | 5M | 31.0 | 59.5 | 70.5 | 31.0 | 59.8 | 72.4 | 15.0 | 30.8 | 39.8 | - | - | -
ALPRO [28] | 5M | 33.9 | 60.7 | 73.2 | 35.9 | 67.5 | 78.8 | - | - | - | - | - | -
BridgeFormer [12] | 5M | 37.6 | 64.8 | 75.1 | 37.0 | 62.2 | 73.9 | 17.9 | 35.4 | 44.5 | - | - | -
Singularity [25] | 5M | 36.8 | 65.9 | 75.5 | 47.4 | 75.2 | 84.0 | - | - | - | 43.0 | 70.6 | 81.3
LAVENDER [32] | 5M | 37.8 | 63.8 | 75.0 | 47.4 | 74.7 | 82.4 | 22.2 | 43.8 | 53.5 | - | - | -
Models pre-trained on more data |  |  |  |  |  |  |  |  |
VIOLET [11] | 183M | 34.5 | 63.0 | 73.4 | 32.6 | 62.8 | 74.7 | 16.1 | 36.6 | 41.2 | - | - | -
All-in-one [58] | 138M | 37.9 | 68.1 | 77.1 | 32.7 | 61.4 | 73.5 | - | - | - | 22.4 | 53.7 | 67.7
Clip4Clip [40] | 400M | 42.1 | 71.9 | 81.4 | 43.4 | 70.2 | 80.6 | 21.6 | 41.8 | 49.8 | 40.5 | 72.4 | -
X-CLIP [41] | 400M | 46.1 | 73.0 | 83.1 | 45.2 | 74.0 | - | 23.3 | 43.0 | - | 44.3 | 74.1 | -
HiTeA | 5M | 44.4 | 69.3 | 78.9 | 51.8 | 79.1 | 85.3 | 27.1 | 46.2 | 54.5 | 45.1 | 73.5 | 84.2
HiTeA | 17M | 46.8 | 71.2 | 81.9 | 56.5 | 81.7 | 89.7 | 28.7 | 50.3 | 59.0 | 49.7 | 77.1 | 86.7

Table 1: Performance comparison on text-to-video retrieval. All results are
reported on R@1/R@5/R@10. We gray out methods that use significantly more pre-
training data for fair comparison. # PT Data is the number of video-text pairs
for pre-training.

###  3.3 Multi-Modal Temporal Relation Exploration

While the video encoder has demonstrated its effectiveness in learning
temporal representation implicitly [11, 28, 68], it remains a challenge to
discover the inherent temporal relations. As a result, the limited
capabilities in temporal modeling deteriorate the downstream task in temporal
reasoning. This is in particular a missing point for the existing video-
language pre-training paradigm [26, 28, 12, 32], which usually focuses on
bridging video and text neglecting the function of text for guiding the video
context representation learning thus losing the temporal cues.

To this end, we introduce multi-modal temporal relation exploration (MTRE), a
novel temporal-aware pre-training task that improves models’ capacities in
capturing temporal correlation of moments in video with fine-grained text
guidance by aligning multi-modal pairs. Specially, the short-view video
$V^{S}$ would represent moment information with respect to the whole video. On
the contrary, the long-view video $V^{L}$ expresses the event and topical
information. To obtain the text-guided video features, we feed videos in
different temporal views into the video encoder individually. Then, the text
features are extracted and interact with the video features by the multi-modal
encoder and yield text-guided video representations
$\bm{v}_{\text{cls}}^{L}\in\mathbb{R}^{D}$ and
$\bm{v}_{\text{cls}}^{S}\in\mathbb{R}^{D}$ as follows:

$\displaystyle\bm{v}_{\text{cls}}^{L}$
$\displaystyle=f(\\{v_{\text{cls}}^{L},v_{1}^{L},\cdots,v_{M}^{L}\\},\\{w_{\text{cls}},w_{1},\cdots,w_{N}\\}),$
(3) $\displaystyle\bm{v}_{\text{cls}}^{S}$
$\displaystyle=f(\\{v_{\text{cls}}^{S},v_{1}^{S},\cdots,v_{M}^{S}\\},\\{w_{\text{cls}},w_{1},\cdots,w_{N}\\}),$
(4)

where $f(\mathcal{V},\mathcal{T})$ represents the multi-modal encoder with
video features $\mathcal{V}$ and text features $\mathcal{T}$. However, since
the short-view of the video is partially aligned with the text, using the
whole text is not reasonable for generating accurate text-guided video feature
for short-view. Meanwhile, improper video-text pairs would yield noisy multi-
modal representation thus degrading the performance of the model. Therefore,
thanks to the positive words mined by cross-modal moment exploration, we can
calibrate representation for short-view video by:

$\displaystyle\bm{v}_{\text{cls}}^{S}$
$\displaystyle=f(\\{v_{\text{cls}}^{S},v_{1}^{S},\cdots,v_{M}^{S}\\},\\{w_{\mathcal{K}_{1}},\cdots,w_{\mathcal{K}_{K}}\\}),$
(5)

where $\mathcal{K}_{i}\in\mathcal{K}$ is the index of the set for selected
positive words. Then, we aim to match the representation of produced text-
guided video features in different granularities in order to enable the model
to predict the past and the future from the short-view of video, which
benefits for capturing the general structure of the video. Specifically, we
adopt the SimSiam framework [6] for minimizing their negative cosine
similarity:

$\displaystyle\mathcal{D}(p^{S},z^{L})=-\frac{p^{S}}{\|p^{S}\|_{2}}\cdot\frac{z^{L}}{\|z^{L}\|_{2}},$
(6)

where $p^{S}=h(g(\bm{v}_{\text{cls}}^{S}))$ and
$z^{L}=g(\bm{v}_{\text{cls}}^{L})$. The $g$ and $h$ are projection MLP head
and prediction MLP head [14, 7]. Minimizing $\mathcal{D}(p^{S},z^{L})$ is
equivalent for minimizing the mean square error between $p^{S}$ and $z^{L}$,
which encourages the videos in different temporal magnitudes to be similar.
Following [6, 14], we defined a symmetrized loss as:

$\displaystyle\mathcal{L}_{\text{MTRE}}=\frac{1}{2}\left[\mathcal{D}(p^{L},\texttt{sg}(z^{S}))+\mathcal{D}(p^{S},\texttt{sg}(z^{L}))\right],$
(7)

where $\texttt{sg}(\cdot)$ is the stop-gradient operation that prevents the
model from collapse during training [6].

###  3.4 Pre-training Objectives

Apart from the two proposed temporal-aware pre-training tasks, we follow
proven video-text pre-training approaches [28, 2, 32] to adopt the standard
pre-training tasks including video-text contrastive (VTC), video-text matching
(VTM), masked language modeling (MLM), and prefix language modeling (PrefixLM)
described in the related work. Precisely, VTC and VTM align the video and text
from the global perspective, while MLM and PrefixLM contribute to multi-modal
understanding and generation capabilities of the model. Details of these
objectives are described in the Appendix. We simply combine these as the base
training objective $\mathcal{L}_{base}$ for our model. Therefore, the full
pre-training objective is computed as:

$\mathcal{L}=\mathcal{L}_{\text{base}}+\mathcal{L}_{\text{CME}}+\mathcal{L}_{\text{MTRE}}.$
(8)

##  4 Experiments

###  4.1 Experiment Setup

##### Pre-training Datasets

Following the recent work [2, 28, 25, 12, 32], we pre-train our model on a
webly-sourced video dataset WebVid-2M [2] with 2.5M video-text pairs and a
image-text dataset Google Conceptual Captions (CC3M) [52] with 3M image-text
pairs. Unlike previous methods, we do not pre-train our model on the large-
scale video-text datasets like HowTo100M [44] with 136M video-text pairs and
YT-Temporal-180M [72] due to the heavy computation. For scaling up, we also
trained our model on the widely used image-text pre-training datasets
including MS COCO [38], Visual Genome [22], SBU Captions [45] and Conceptual
12M [52], we refer this setting as 17M corpus.

##### Downstream Datasets

We evaluate our pre-trained model on 18 video-language benchmarks including
video-text retrieval, video question answering, and video captioning tasks.
Specifically, video question answering (VideoQA) can be categorized as
Multiple-Choice (MC) and Open-Ended (OE) settings. The evaluation datasets are
briefly summarized in below. The details can be found in the Appendix.

  * •

Video-Text Retrieval: MSRVTT [64], DiDeMo [1], LSMDC [49], ActivityNet Caption
[21], SSv2-Label [25], and SSv2-Template [25];

  * •

VideoQA (MC): TGIF-Action, TGIF-Transition [16], MSRVTT-MC [69], LSMDC-MC
[56], and NExT-QA [60];

  * •

VideoQA (OE): TGIF-Frame [16], MSRVTT-QA, MSVD-QA [62], LSMDC-FiB [42] and
ActivityNet-QA [70].

  * •

Video Captioning: MSRVTT [64] and MSVD [5].

##### Implementation Details

Our implementation of HiTeA is based on PyTorch [46]. In detail, we
instantiate the video encoder with MViT-Base model [36] pretrained on
ImageNet-21K [48]. The text encoder is initialized from first six layers of
pre-trained BERT-Base [10], and the multi-modal encoder is initialized with
last six layers of pre-trained BERT-Base. We pre-train HiTeA for 10 epochs,
using a batch size of 16 on 8 NVIDIA A100 GPUs. We use AdamW [20] optimizer
with a weight decay of 0.02 and betas (0.9, 0.98). The learning rate is first
warmed up to 5e-5 in the first 1000 iterations, and decays following a cosine
schedule. During pre-training, we sparsely sample 8 frames for short and long
view while preserving their order in-between and resize them to 224 $\times$
224\. The duration of short view is restricted as the 1/8 of the whole video
duration. $K$ is empirically set to 5. The MLM mask ratio is set to $15\%$.
Details of fine-tuning stage are described in Appendix.

|  | TGIF | MSRVTT | LSMDC | MSVD | ActivityNet
---|---|---|---|---|---|---
Method | #PT Data | Action | Transition | Frame | MC | QA | MC | FiB | QA | QA
ClipBERT [26] | 0.2M | 82.8 | 87.8 | 60.3 | 88.2 | 37.4 | - | - | - | -
ALPRO [28] | 5M | - | - | - | - | 42.1 | - | - | 46.3 | -
Singularity [25] | 5M | - | - | - | 92.0 | 42.7 | - | - | - | 41.8
LAVENDER [32] | 5M | 96.6 | 99.1 | 72.2 | 96.6 | 44.2 | 86.0 | 56.9 | 55.4 | -
Clover [15] | 5M | 94.9 | 98.0 | 71.4 | 95.0 | 43.9 | 83.2 | 54.1 | 51.9 | -
Models pre-trained on more data |  |  |  |  |  |
VIOLET [11] | 183M | 92.5 | 95.7 | 68.9 | 91.9 | 43.9 | 82.8 | 53.7 | 47.9 | 38.9
JustAsk [65] | 69M | - | - | - | - | 41.5 | - | - | 46.3 | 38.9
MERLOT [72] | 180M | 94.0 | 96.2 | 69.5 | 90.9 | 43.1 | 81.7 | 52.9 | - | 41.4
All-in-one [58] | 283M | 95.5 | 94.7 | 66.3 | 92.3 | 46.8 | 84.4 | - | 48.3 | -
HiTeA | 5M | 96.8 | 98.8 | 72.5 | 97.2 | 45.4 | 85.8 | 54.6 | 55.6 | 45.1
HiTeA | 17M | 97.2 | 98.8 | 73.2 | 97.4 | 45.9 | 85.3 | 54.5 | 55.3 | 46.4

Table 2: Performance comparison on video question answering. Accuracy is
reported for evaluation. We gray out methods that use significantly more pre-
training data for fair comparison.

Method | # PT Data | MSRVTT | MSVD
---|---|---|---
UniVL [39] | 180M | 49.9 | -
SwinBERT [37] | - | 53.8 | 120.6
MV-GPT [51] | 53M | 60.0 | -
CLIP4Caption [55] | 400M | 57.7 | -
LAVENDER [32] | 5M | 58.0 | 142.9
HiTeA | 5M | 62.5 | 145.1
HiTeA | 17M | 65.1 | 146.9

Table 3: Performance comparison on video captioning. CIDEr [57] is reported
for evaluation.

Method | MSRVTT-Retrieval | SSv2 Template-Retrieval | NExT-QA (Hard) | MSVD-QA
---|---|---|---|---
R@1 | R@5 | R@10 | AveR | R@1 | R@5 | R@10 | AveR | Acc@C | Acc@T | Acc.
$\mathcal{L}_{\text{base}}$ | 40.0 | 68.0 | 77.1 | 61.7 | 80.5 | 100.0 | 100.0 | 93.5 | 44.0 | 46.4 | 52.7
$\mathcal{L}_{\text{base}}$ \+ $\mathcal{L}_{\text{MTRE}}$ | 41.6 | 69.1 | 78.2 | 63.0 | 83.3 | 98.9 | 100.0 | 94.1 | 46.3 | 46.4 | 54.8
$\mathcal{L}_{\text{base}}$ \+ $\mathcal{L}_{\text{CME}}$ | 42.0 | 69.3 | 79.7 | 63.7 | 83.9 | 99.4 | 100.0 | 94.4 | 46.3 | 48.3 | 54.3
$\mathcal{L}_{\text{base}}$ \+ $\mathcal{L}_{\text{CME}}$ \+ $\mathcal{L}_{\text{MTRE}}$ | 44.4 | 69.3 | 78.9 | 64.2 | 85.6 | 100.0 | 100.0 | 95.2 | 47.8 | 48.6 | 55.6

Table 4: Evaluation of the proposed methods on four downstream video-language
tasks. For text-to-video retrieval, R@1, R@5, R@10, and the average are
reported. For video question answering, we report the accuracy.

|  | SSv2-Label | SSv2-Template
---|---|---|---
Method | # PT Data | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Frozen [2] | 5M | - | - | - | 52.9 | 94.8 | 99.4
Clip4Clip [40] | 400M | 43.1 | 71.4 | 80.7 | 77.0 | 96.6 | 98.3
Singularity [25] | 5M | 44.1 | 73.5 | 82.2 | 77.0 | 98.9 | 99.4
HiTeA | 5M | 55.2 | 81.4 | 89.1 | 85.6 | 100.0 | 100.0

Table 5: Comparison of existing methods on Something-to-Something (SSv2) text-
to-video retrieval.

Method | # PT Data | Acc@C | Acc@T | Acc@D | Acc.
---|---|---|---|---|---
Full Set |  |  |  |  |
Human | - | 87.6 | 88.6 | 90.4 | 88.4
HCRN [23] | - | 45.9 | 49.3 | 53.7 | 48.2
HGA [18] | - | 46.3 | 50.7 | 59.3 | 49.7
VGT [61] | 0.18M | 53.4 | 56.4 | 69.5 | 56.9
HGA* [18] | 400M | 46.8 | 52.1 | 59.3 | 50.4
ATP [4] | 400M | 51.3 | 50.2 | 66.8 | 54.3
HiTeA | 5M | 62.4 | 58.3 | 75.6 | 63.1
Hard Split |  |  |  |  |
ATP [4] | 400M | 38.4 | 36.5 | / | /
HGA [18] | - | 43.3 | 45.3 | / | /
HiTeA | 5M | 47.8 | 48.6 | / | /

Table 6: Comparison of existing methods on NExT-QA [60]. We report accuracy on
the Causal (C), Temporal (T), Descriptive (D) splits and overall accuracy on
validation set. * stands for using CLIP as the initialization of visual
encoder.

###  4.2 Comparison to Prior Arts

In this section, we compare HiTeA with numerous state-of-the-art video-
language pre-training methods on several downstream datasets under fine-tuning
setting.

####  4.2.1 Text-to-Video Retrieval

Table 1 summarizes the results on MSRVTT [64], DiDeMo [1], LSMDC [49], and
ActivityNet Caption [21] under fine-tuning settings. Our method outperforms
all of the existing video-language pre-training model by a large margin under
the same data scale. In particular, our method yields 6.6% lift in terms of
R@1 on MSRVTT dataset while only exploiting 5M video-text pairs. Note that we
also include the comparison with the recent works that utilize the powerful
encoder from CLIP [47], our method still can be comparable with them even
surpass them, which shows the validness of the proposed method. Besides, we
can notice that our method achieves the best result among all of listed
methods on LSMDC dataset, which proves that our model can leverage the various
moments presented in fruitful movie clips with cross-modal moment exploration.

####  4.2.2 Video Question Answering

Table 2 lists the results of HiTeA and current state-of-the-art approaches on
nine VideoQA datasets. It can be noticed that our method achieves the best
performance in most of VideoQA datasets even with less pre-training data.
Specifically, it achieves absolute improvement 1.1% on TGIF-FrameQA, 2.2% on
MSRVTT-MC, 1.5% on MSRVTT-QA, 0.2% on MSVD-QA, and 3.3% on ActivityNet-QA. We
believe the moments learned by the cross-modal exploration are useful for
finding the clue of answers in VideoQA.

####  4.2.3 Video Captioning

Table 3 compares HiTeA with existing mthods on video captioning datasets
MSRVTT and MSVD. As shown in the table, although we use less pre-training data
than compared approaches, HiTeA still obtains significant improvement compared
to those large-scale pre-trained models. On MSRVTT Caption, our method
surpasses SoTA method MV-GPT [51] by 2.5% CIDEr. Note that MV-GPT is pre-
trained for multi-modal video captioning and it leverages the ASR transcripts
from audio as the additional input. By contrast, our method only utilizes
video as the input during generation.

Dataset |  Original $\uparrow$ |  Shuffled $\downarrow$ |  Gap $\uparrow$
---|---|---|---
MSRVTT [64] | 64.2 | 63.3 | 0.9
DiDeMo [1] | 72.1 | 70.2 | 1.9
LSMDC [49] | 42.6 | 41.7 | 0.9
ActivityNet Caption [21] | 67.6 | 66.8 | 0.8
SSv2 Template [25] | 95.2 | 72.4 | 22.8
SSv2 Label [25] | 76.7 | 73.5 | 3.2

Table 7: Dependency on temporal information for text-to-video retrieval
datasets with temporal shuffling test. The average recall of Recall@1,
Recall@5, and Recall@10 are reported. We evaluate the performance drop when
shuffling the input during inference. “Original” and “Shuffled” denote the
original and shuffled input videos, respectively, and “Gap” is the difference
between the Original and Shuffled metric. The larger "Gap" indicates the
dataset relies on temporal information, and the model utilizes more temporal
information to solve the task.

Dataset |  Original $\uparrow$ |  Shuffled $\downarrow$ |  Gap $\uparrow$
---|---|---|---
MSRVTT-QA [62] | 45.4 | 45.2 | 0.2
MSVD-QA [62] | 55.6 | 55.5 | 0.1
TGIF-FrameQA [16] | 72.5 | 72.1 | 0.4
ActivityNet-QA [70] | 45.1 | 45.0 | 0.1
NExT-QA (Hard) [60] | 47.1 | 45.6 | 0.5

Table 8: Dependency on temporal information for video question answering
datasets by temporal shuffling test. We report the accuarcy for each dataset.
For NExT-QA dataset, we evaluate with the hard split of the validation set
[4].

###  4.3 Discussion

In this section, we discuss the temporal characteristics of our model and the
datasets.

##### Impact of Loss Terms.

We investigate the contribution of individual loss terms and the results are
shown in Table 4. It can be observed that the combining both
$\mathcal{L}_{\text{CME}}$ and $\mathcal{L}_{\text{MTRE}}$ improves the
performance of text-to-video retrieval and video question answering by at
least 1.7% and 2.9% in Average Recall and Average accuracy respectively. In
addition, we also find that the performance of $\mathcal{L}_{\text{CME}}$
surpasses that of $\mathcal{L}_{\text{MTRE}}$ on MSRVTT retrieval dataset that
largely dominated by the appearance information. This can be explained that
the cross-modal moment exploration loss not only select the positive verbs for
the video from the text but also choose the acting object for alignment, which
can boost the retrieval performance.

##### Evaluation on Temporal-aware Tasks.

Lei _et al_. [25] reveal that the previous four retrieval datasets are prone
to being biased for appearance while rarely relying on temporal information,
thus introducing Something-to-Something v2 (SSv2) Template and SSv2 Label
retrieval datasets to test models’ true temporal modeling capability. In
particular, SSv2 Template retrieval task requires a deeper understanding of
the moment and temporal relation since no objects information are presented.
The performance on these datasets are summarized in Table 5. It can be
observed that HiTeA achieves significant improvement with $+8.5\%$ gains in
terms of R@1 on these two temporal-oriented text-to-video retrieval datasets,
which demonstrates the effectiveness of our proposed method through exploring
fine-grained moment information and modeling temporal relation. In addition,
we evaluate our model on NExT-QA [60] dataset that explicitly designed for
temporal and causal understanding. As presented in Table 6, our method
significantly surpasses its competitive counterparts, even those methods
equipped with powerful image-text pre-trained encoders. Quantitatively, HiTeA
obtains an absolute improvement $+9\%$ on the causality split with the help of
intrinsic temporal relation. Recently, Buch _et al_. [4] filter out the
trivial question for the dataset, and build the hard split for causality and
temporal related questions for evaluate the causality and temporal of the
model. As we can see in the table, even for the questions that heavily rely on
causality, our model can still achieves a relative gain of 4.1% on the model
with specific design for VideoQA, which indicates that our model do not solely
depend on static appearance.

Figure 3: Visualizations of learned cross-attention maps from multi-modal
encoder. We present samples from MSRVTT [64] and SSv2 Template [25] retrieval
dataset. HiTeA attends to the patches related to objects motion by tracking
trajectory. Best viewed in color.

|  | MSRVTT | DiDeMo | LSMDC
---|---|---|---|---
Method | # PT Data | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Frozen [2] | 5M | 18.7 | 39.5 | 51.6 | 21.1 | 46.0 | 56.2 | 9.3 | 22.0 | 30.1
ALPRO [28] | 5M | 24.1 | 44.7 | 55.4 | 23.8 | 47.3 | 57.9 | - | - | -
BridgeFormer [12] | 5M | 26.0 | 46.4 | 56.4 | 25.6 | 50.6 | 61.1 | 12.2 | 25.9 | 32.2
Singularity [25] | 5M | 28.4 | 50.2 | 59.5 | 36.9 | 61.6 | 69.3 | - | - | -
Models pre-trained on more data |  |  |  |  |  |
VideoCLIP [63] | 138M | 10.4 | 22.2 | 30.0 | 16.6 | 46.9 | - | - | - | -
VIOLET [11] | 183M | 25.9 | 49.5 | 59.7 | 23.5 | 49.8 | 59.8 | - | - | -
Clip4Clip [40] | 400M | 31.2 | 53.7 | 64.2 | - | - | - | 11.3 | 22.7 | 29.2
HiTeA | 5M | 29.9 | 54.2 | 62.9 | 36.1 | 60.1 | 70.3 | 15.5 | 31.1 | 39.8
HiTeA | 17M | 34.4 | 60.0 | 69.9 | 43.2 | 69.3 | 79.0 | 18.3 | 36.7 | 44.2

Table 9: Zero-shot evaluation on text-to-video retrieval. All results are
reported on R@1/R@5/R@10. We gray out methods that use significantly more pre-
training data for fair comparison.

Method | # PT Data | MSRVTT-QA | MSVD-QA
---|---|---|---
Just Ask [65] | 69M | 2.9 | 7.5
LAVENDER [32] | 5M | 4.5 | 11.6
MERLOT Reserve [71] | 1B | 5.8 | -
FrozenBiLM [66] | 10M | 6.4 | 11.7
HiTeA | 5M | 8.6 | 18.2
BLIP [29] | 129M | 19.2 | 35.2
mPLUG [27] | 400M | 21.1 | 37.2
HiTeA | 5M | 21.7 | 37.4

Table 10: Zero-shot evaluation on video question answering. Accuracy is
reported. We gray out those methods additionally supervised pre-training on
VQA v2 [13] dataset.

##### Temporal Reliance of Datasets.

Previous methods [28, 26, 12] only evaluate the performance of models on the
existing datasets to demonstrate the superiority of the methods. However, Buch
_et al_. [4] and Lei _et al_. [25] reveal that the most of the evaluation are
biased towards the static concepts. Here, we investigate the temporal reliance
for the evaluated datasets by introducing the temporal shuffling test.
Specifically, we compute the performance changes between running inference on
the ordered video versus its shuffled version. The large performance drop
indicates the dataset has less spatial bias and needs for temporal
information. Table 7 and Table 8 conclude the performance gap between ordered
and shuffled input video for text-to-video retrieval and VideoQA datasets. For
text-to-video retrieval task, SSv2 Template shows the large performance drop
after shuffling the input video, which demonstrates that it depends most on
the dynamic information thus verifying our assumption. On the contrary, the
performance on ActivityNet Caption dataset is barely affected (-0.8 on Mean
Recall) since the text almost describes the static objects without relying on
temporal information. For video question answering dataset, we observe that
the MSVD-QA and ActivityNet-QA are less sensitive to the order of video
frames. This is because these two datasets contain more questions requiring
frame-region information, such as object categories, scenes, and species. We
believe this can be used to evaluate the temporal reliance of the datasets as
well as the utilization for temporal cue by models in the future work.

##### Qualitative Analysis.

To verify that our model can capture the motion information with respected to
the given text rather than inferring from the static signal, we present the
query text in SSv2 Template dataset which has masked all of the object
information, and also visualize the query in MSRVTT dataset. As we can see in
the Figure 3, the attention map of atomic action "talking" mainly focuses on
the mouse of the cartoon characters while the baseline largely focusing on the
characters, which indicates that our method can understand the moment better
when adopting the temporal-aware pre-training tasks. In another example, the
word "spins" can reveal the trajectory of the object showing that our method
is able to capture the temporal motion presented in the video.

###  4.4 Zero-shot Generalizability

To demonstrate the generalizability of proposed video-text pre-trained model,
we perform zero-shot evaluation on video-language downstream tasks. Table 9
summarizes the performance of our model and compared approaches on text-to-
video retrieval. We can observe that our model yields more than 3.4% lift in
R@1 on MSRVTT dataset [64] while exploiting less video-text pairs. Besides,
our method surpasses all of the compared models in terms of LSMDC dataset
showing the superiority of our model’s generalizability. We also evaluate the
zero-shot performance on VideoQA task in Table 10. Our method attains
competitive zero-shot performance on MSRVTT-QA and MSVD-QA datasets even
without help of audio signal supervision [71] or additional generated video
question pairs [65]. In particular, less pre-training data (_i.e_. 5M $<$ 69M)
are used while our method can still outperform other SoTA approaches. We also
evaluate the zero-shot performance of models supervised on VQA v2 [13]. We can
find that our method surpasses the powerful multi-modal SoTA methods (_e.g_.
mPLUG [27]) with only 5M pre-training data showing the better generalization
ability of HiTeA.

##  5 Conclusion

In this work, we introduce HiTeA, a novel hierarchical temporal-aware video-
language pre-training framework with both understanding and generation
capabilities. We vary the video with different views and model cross-modal
alignment between moments and texts as well as their temporal relations in a
hierarchical way. Specifically, a cross-modal moment exploration pre-training
task is proposed to explore the alignment between the text and video moment,
which helps to overcome the partially semantic alignment between video and
text. Moreover, multi-modal pairs are constructed to learn temporal relations
between moments and the event presented by the video with multi-modal temporal
relation exploration pre-training task. Even pre-trained on less data, HiTeA
still achieves state-of-the-art performance on a wide range of video-language
downstream datasets, which clearly shows the superiority of our method.

## References

  * [1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In Proceedings of the IEEE international conference on computer vision, pages 5803–5812, 2017.
  * [2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  * [3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, volume 2, page 4, 2021.
  * [4] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles.  Revisiting the" video" in video-language understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917–2927, 2022.
  * [5] David Chen and William B Dolan.  Collecting highly parallel data for paraphrase evaluation.  In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190–200, 2011\.
  * [6] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [7] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021.
  * [8] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In European conference on computer vision, pages 104–120. Springer, 2020.
  * [9] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.  Unifying vision-and-language tasks via text generation.  In International Conference on Machine Learning, pages 1931–1942. PMLR, 2021.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [11] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu.  Violet: End-to-end video-language transformers with masked visual-token modeling.  arXiv preprint arXiv:2111.12681, 2021.
  * [12] Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo.  Bridging video-text retrieval with multiple choice questions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16167–16176, 2022.
  * [13] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * [14] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [15] Jingjia Huang, Yinan Li, Jiashi Feng, Xiaoshuai Sun, and Rongrong Ji.  Clover: Towards a unified video-language alignment and fusion model.  arXiv preprint arXiv:2207.07885, 2022.
  * [16] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766, 2017.
  * [17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [18] Pin Jiang and Yahong Han.  Reasoning with heterogeneous graph alignment for video question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11109–11116, 2020.
  * [19] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In International Conference on Machine Learning, pages 5583–5594. PMLR, 2021.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In Proceedings of the IEEE international conference on computer vision, pages 706–715, 2017.
  * [22] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [23] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran.  Hierarchical conditional relation networks for multimodal video question answering.  International Journal of Computer Vision, 129(11):3027–3050, 2021\.
  * [24] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He.  Stacked cross attention for image-text matching.  In Proceedings of the European conference on computer vision (ECCV), pages 201–216, 2018.
  * [25] Jie Lei, Tamara L Berg, and Mohit Bansal.  Revealing single frame bias for video-and-language learning.  arXiv preprint arXiv:2206.03428, 2022.
  * [26] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331–7341, 2021.
  * [27] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al.  mplug: Effective and efficient vision-language learning by cross-modal skip-connections.  arXiv preprint arXiv:2205.12005, 2022.
  * [28] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi.  Align and prompt: Video-and-language pre-training with entity prompts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4953–4963, 2022.
  * [29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [30] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in Neural Information Processing Systems, 34:9694–9705, 2021.
  * [31] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  arXiv preprint arXiv:2005.00200, 2020.
  * [32] Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang.  Lavender: Unifying video-language understanding as masked language modeling.  arXiv preprint arXiv:2206.07160, 2022.
  * [33] Ping Li, Qinghao Ye, Luming Zhang, Li Yuan, Xianghua Xu, and Ling Shao.  Exploring global diverse attention via pairwise temporal relation for video summarization.  Pattern Recognition, 111:107677, 2021.
  * [34] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang.  Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning.  arXiv preprint arXiv:2012.15409, 2020.
  * [35] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [36] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804–4814, 2022.
  * [37] Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Swinbert: End-to-end transformers with sparse attention for video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17949–17958, 2022.
  * [38] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [39] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  arXiv preprint arXiv:2002.06353, 2020.
  * [40] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.  Neurocomputing, 508:293–304, 2022.
  * [41] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji.  X-clip: End-to-end multi-grained contrastive learning for video-text retrieval.  arXiv preprint arXiv:2207.07285, 2022.
  * [42] Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal.  A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6884–6893, 2017.
  * [43] Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and Stéphane Marchand-Maillet.  Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17(4):1–23, 2021.
  * [44] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879–9889, 2020.
  * [45] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  Advances in neural information processing systems, 24, 2011.
  * [46] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [48] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  arXiv preprint arXiv:2104.10972, 2021.
  * [49] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele.  A dataset for movie description.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3202–3212, 2015.
  * [50] Karsten Roth, Oriol Vinyals, and Zeynep Akata.  Integrating language guidance into vision-based deep metric learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16177–16189, 2022.
  * [51] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid.  End-to-end generative pretraining for multimodal video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17959–17968, 2022.
  * [52] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of ACL, 2018.
  * [53] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464–7473, 2019.
  * [54] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [55] Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li.  Clip4caption: Clip for video caption.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 4858–4862, 2021.
  * [56] Atousa Torabi, Niket Tandon, and Leonid Sigal.  Learning language-visual embedding for movie understanding with natural-language.  arXiv preprint arXiv:1609.08124, 2016.
  * [57] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [58] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  All in one: Exploring unified video-language pre-training.  arXiv preprint arXiv:2203.07303, 2022.
  * [59] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  arXiv preprint arXiv:2108.10904, 2021.
  * [60] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9777–9786, 2021.
  * [61] Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan.  Video graph transformer for video question answering.  arXiv preprint arXiv:2207.05342, 2022.
  * [62] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017.
  * [63] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  arXiv preprint arXiv:2109.14084, 2021.
  * [64] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016.
  * [65] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686–1697, 2021.
  * [66] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Zero-shot video question answering via frozen bidirectional language models.  arXiv preprint arXiv:2206.08155, 2022.
  * [67] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [68] Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang.  Temporal cue guided video highlight detection with low-rank audio-visual fusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7950–7959, 2021.
  * [69] Youngjae Yu, Jongseok Kim, and Gunhee Kim.  A joint sequence fusion model for video question answering and retrieval.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 471–487, 2018.
  * [70] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019.
  * [71] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi.  Merlot reserve: Neural script knowledge through vision and language and sound.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375–16387, 2022.
  * [72] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  Advances in Neural Information Processing Systems, 34:23634–23651, 2021.
  * [73] Linchao Zhu and Yi Yang.  Actbert: Learning global-local video-text representations.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8746–8755, 2020.

##  Appendix A Additional Experimental Results

In this section, we provide more experimental results for completeness of our
proposed method.

###  A.1 Transfer to Image-Text Downstream Tasks

Since images can be viewed as the single-frame videos, we evaluate the
proposed method on image-text tasks including image-text retrieval and visual
question answering.

##### Image-Text Retrieval

We perform the Image-to-Text and Text-to-Image retrieval on COCO datasets, and
the results are summarized in Table 11. We can observe that our method
surpasses Singularity [25] with same amount of pre-train data, especially 1%
improvement on Recall@1 for Text-to-Image retrieval task. Moreover, although
some methods [30, 8] leverage 4M dataset which contains the COCO dataset as a
part of the pre-training dataset, HiTeA can still attain comparable results
showing the good generalization ability.

##### Visual Question Answering

We also evaluate our method on visual question answering task. Table 12
concludes the image question answering results on VQAv2 [13] datasets. We
observe that HiTeA demonstrates competitive performance on the VQA tasks. It
is worthwhile noting that our method achieves the better performance compared
to Singularity [25] same pre-training datasets, which indicates the video-text
pre-training would boost the performance of image-text downstream tasks.
However, we still see a gap with state-of-the-art image-text pre-trained
models since our method do not use the in-domain data (_e.g_. COCO) during
pre-training, thus leading to the gap with SoTA performance. One future
direction is to use more image-text data during video-text pre-training for
better generalization.

###  A.2 Additional Ablation Studies

##### Impact of positive candidate words size $K$.

We investigate the effect of choosing different positive words size $K$ during
cross-modal moment exploration. As depicted in Figure 4, it can be observed
that with the increment of $K$, the performance on each dataset is increasing
then start to decrease. In addition, there is a trade-off between the choice
of $K$ and performance with respected to different datasets, and $K=5$ gives
relative good results among these datasets. It also suggests that the small
$K$ would give more deterministic results since the model would only select
the word with the largest similarity, thus more focusing on the single action
or object. Then, as number of positive words increased, more accurate words
are selected to align with the short-view of video. However, the model no
longer benefits from cross-modal moment exploration when $K$ is large enough
(_i.e_., $K=11$ or $K=13$) due to the increased noise in the selected
candidate words.

Method | #PT Data | COCO (5K test)
---|---|---
TR | IR
R1 | R5 | R10 | R1 | R5 | R10
ViLT [19] | 4M | 61.5 | 86.3 | 92.7 | 42.7 | 72.9 | 83.1
UNITER [8] | 4M | 65.7 | 88.6 | 93.8 | 52.9 | 79.9 | 88.0
OSCAR [35] | 4M | 70.0 | 91.1 | 95.5 | 54.0 | 80.8 | 88.5
ALBEF [30] | 4M | 73.1 | 91.4 | 96.0 | 56.8 | 81.5 | 89.2
BLIP [29] | 14M | 80.6 | 95.2 | 97.6 | 63.1 | 85.3 | 91.1
ALIGN [17] | 1.2B | 77.0 | 93.5 | 96.9 | 59.9 | 83.3 | 89.8
Singularity [25] | 5M | 71.9 | 90.8 | 95.4 | 54.6 | 80.0 | 87.8
HiTeA | 5M | 72.4 | 90.9 | 95.4 | 55.6 | 80.6 | 87.8
Table 11: Comparison to existing methods on image-text retrieval on COCO dataset. We show results for both text retrieval (image-to-text retrieval, TR) and image retrieval (IR). Method | #PT Data | test-dev | test-std
---|---|---|---
ClipBERT [26] | 0.2M | 69.08 | 69.43
ViLT [19] | 4M | 70.94 | -
VL-BART [9] | 0.2M | - | 71.30
LXMERT [54] | 4M | 72.42 | 72.54
UNITER [8] | 4M | 72.70 | 72.91
UNIMO [34] | 4M | 73.79 | 74.02
OSCAR [35] | 4M | 73.16 | 73.44
ALBEF [30] | 4M | 74.54 | 74.70
BLIP [29] | 14M | 77.54 | 77.62
Singularity [25] | 5M | 70.30 | 70.53
HiTeA | 5M | 74.06 | 74.28
Table 12: Comparison to existing methods on VQA. Figure 4: Variations in
performance by changing the number of selected positive words $K$.

|  MSRVTT [64] |  SSv2-Template [25] |  SSv2-Label [25]
---|---|---|---
Method |  Original $\uparrow$ |  Shuffled $\downarrow$ |  Gap $\uparrow$ |  Original $\uparrow$ |  Shuffled $\downarrow$ |  Gap $\uparrow$ |  Original $\uparrow$ |  Shuffled $\downarrow$ |  Gap $\uparrow$
$\mathcal{L}_{\text{base}}$ | 61.7 | 60.8 | 0.9 | 93.5 | 75.1 | 18.4 | 74.6 | 71.9 | 2.7
$\mathcal{L}_{\text{base}}$ \+ $\mathcal{L}_{\text{CME}}$ | 63.7 | 62.9 | 0.8 | 94.4 | 72.6 | 21.8 | 74.8 | 71.8 | 3.0
$\mathcal{L}_{\text{base}}$ \+ $\mathcal{L}_{\text{MTRE}}$ | 63.0 | 62.6 | 0.4 | 94.1 | 73.0 | 21.1 | 75.8 | 72.2 | 3.6
$\mathcal{L}_{\text{base}}$ \+ $\mathcal{L}_{\text{CME}}$ \+ $\mathcal{L}_{\text{MTRE}}$ | 64.2 | 63.3 | 0.9 | 95.2 | 72.4 | 22.8 | 76.7 | 73.5 | 3.2

Table 13: Evaluation of proposed methods for temporal dependency with temporal
shuffling test. We evaluate the performance drop when shuffling the input
during inference. “Original” and “Shuffled” denote the original and shuffled
input videos, respectively, and “Gap” is the difference between the Original
and Shuffled metric. The larger "Gap" indicates the dataset relies on temporal
information, and the model utilizes more temporal information to solve the
task.

##### Temporal evaluation of loss terms.

To further validate the temporal dependency for the proposed method, we adopt
the shuffling test for models with different loss terms, as shown in Table 13.
Table 13 shows that our loss terms contribute more significantly when the
dataset requires more temporal understanding. In concrete,
$\mathcal{L}_{\text{CME}}$ and $\mathcal{L}_{\text{MTRE}}$ consistently
improve the performances of Original and Gap on more temporal relied datasets
(_i.e_. SSv2-Template and SSv2-Label). For example, model with two loss terms
largely surpasses the baseline model in the metric of Gap by achieving 4.4 and
0.5 improvement on SSv2-Template and SSv2-Label, respectively.

Method | MSRVTT | DiDeMo | SSv2-Template
---|---|---|---
TimeSformer ($\mathcal{L}_{\text{base}}$) | 57.30 | 62.38 | 92.91
\+ $\mathcal{L}_{\text{MTRE}}$ | 59.23 | 63.18 | 93.68
\+ $\mathcal{L}_{\text{CME}}$ | 59.03 | 63.78 | 93.30
\+ $\mathcal{L}_{\text{CME}}$ \+ $\mathcal{L}_{\text{MTRE}}$ | 59.93 | 65.34 | 94.25

Table 14: Effectiveness of the proposed methods on different video backbone.
We use TimeSformer [3] pre-trained on ImageNet-21K [48] to verify the
generalization ability of our proposed method. For text-to-video retrieval,
the Mean Recall of Recall@1, Recall@5, and Recall@10 is reported. For video
question answering task, we report the Top-1 accuracy.

##### Generalization to other vision backbone.

Table 14 shows that our proposed method is generalizable to different vision
backbones. In details, we instantiate the video encoder with TimeSformer [3]
pretrained on ImageNet-21K [48]. It can be observed both CME and MTRE
consistently improve the model performance across the video backbones
considered showing the generalization of proposed hierarchical temporal-aware
pre-training framework. It is worth noting that, TimeSformer generates long
video tokens compared to that of Multi-scale ViT [36], which brings extra
memory cost for the multi-modal encoder and decoder since the computation of
self-attention is quadratic. This makes TimeSformer expensive to scale to more
input frames with longer sequences.

Figure 5: Variations in performance by adopting language during Multi-modal
Temporal Relation Exploration (MTRE). We report the Mean Recall of Recall@1,
Recall@5, and Recall@10.

##### Influence of language for MTRE.

We investigate the influence of language for multi-modal temporal relation
exploration. Instead of utilizing the language signals, we directly adopt the
video representation $v_{\text{cls}}$ from the video encoder during the
learning. The results are sketched in Figure 5. It can be observed that the
model trained with multi-modal pairs attains better performance than the model
without text. In concrete, it achieves 2.1% gains on SSv2-Template which
mainly depends on the understanding of actions, which indicates that our
method can better understanding the actions via multi-modal temporal relation
exploration. Besides, we also notice that performance of the model trained
with correct multi-modal pairs surpasses that of model trained by multi-modal
pairs with same text, which indicates that improper video-text pair yields
noisy multi-modal representation thus degrading the performance of the model.

Figure 6: Examples of similarities between words and videos generated by our
method. Our method captures the atomic actions in the videos as well as the
object information with the help of cross-modal moment exploration.

##  Appendix B Discussion

###  B.1 Qualitative Analysis

We sample some videos and corresponding texts and compute similarities between
words and videos in Figure 6. As we can see in the figure, our model can
effectively capture the moments such as "spreading", "moving", and "preparing"
etc. in the video, which is essential for understanding videos. Besides, we
can notice that the video would also attend the object appeared in the video,
showing the capability for modeling fine-grained moment information.

###  B.2 Connection to Other Fine-Grained Methods

Some efforts [24, 67, 43] have been made to learn the fine-grained correlation
and alignment between two modalities by leveraging the token-wise similarities
in vision-language pre-training. FILIP [67] and TERAN [43] aggregate the
maximum token similarity scores and assign the optimal patch-word transport
matrix. SCAN [24] utilizes the similarity scores to attend each tokens for
soft fine-grained alignment. These approaches are originally tailored for
image-text pre-training, which aims to locate the fine-grained static object.
However, different from image-text pre-training, video-text pre-training needs
to understand the correlation between words and moments, which not only
contains static objects but also consists of atomic actions. Our proposed
cross-modal moment exploration leverages the short-view of video to reflect
the moment information and discover the relationship between short-view videos
and words, which results in fine-grained moment representations for video-
language pre-training.

###  B.3 Limitations and Boarder Impact

Despite the effectiveness of the proposed method on various downstream tasks,
our method still has some limitations that would make for promising directions
for future work. (1) Currently, we only pre-train our model on 5M data with
the base-size encoders, and the scalability of the model is not explored which
deserves more in-depth investigation in the future. (2) Our method shares
similar risks like other pre-training methods that the pre-training data might
consist bias and unsafe content which requires further analysis before the
deployment.

##  Appendix C Implementation Details

###  C.1 Number of Parameters

Method | # of Parameter
---|---
ClipBERT [26] | 137M
Frozen [2] | 232M
BridgeFormer [12] | 152M
All-in-one [58] | 110M
VIOLET [11] | 198M
ALPRO [28] | 231M
Singularity [25] | 209M
LAVENDER [11] | 198M
HiTeA | 297M
Table 15: Comparison to other models in the number of parameters.

We include some of previous models with their parameter counts (which were
reported in the original paper or calculated by follow-up work), and we
compare them with HiTeA in Table 15. Compared with other models, our model is
of comparable model size and requires less pair of video-text pre-training
data to achieve better performance in terms of both video-language
understanding and generation.

Figure 7: Architecture of the proposed HiTeA and other pre-training
objectives.

###  C.2 Model Architecture

As sketched in Figure 7, our model consists two unimodal encoders for text and
video respectively, a multi-modal encoder for video-text interaction, and a
text decoder for generation. In concrete, given an arbitrary view of video
$V\in\mathbb{R}^{T\times H\times W}$ is encoder into a sequence of embeddings:
$\\{v_{\text{cls}},v_{1},\cdots,v_{M}\\}\in\mathbb{R}^{(M+1)\times D}$, where
$M$ is the number of flattened patches for video $V$, and $v_{\text{cls}}$ is
the embedding of the visual [CLS] token and used to provide global
representation of the video. The text encoder transforms the text into a
sequence of embeddings:
$\\{w_{\text{cls}},w_{1},\cdots,w_{N}\\}\in\mathbb{R}^{(N+1)\times D}$, where
$N$ is the number of words in the text. To efficiently encode multi-modal
information while preserving unimodal information, we fuse the video and text
features from uni-modal encoders following [27]. The output of the multi-modal
encoder
$\\{\bm{v}_{\text{cls}},\bm{v}_{1},\cdots,\bm{v}_{M},\bm{w}_{\text{cls}},\bm{w}_{1},\cdots,\bm{w}_{N}\\}\in\mathbb{R}^{(M+N+2)\times
D}$ is fed into a transformer decoder for sequence to sequence generation,
which equips HiTeA with the capabilities of both multi-modal understanding and
generation.

###  C.3 Pre-training Objectives

During pre-training, we also perform four pre-training tasks including Video-
Text Contrastive Learning ($\mathcal{L}_{\text{VTC}}$), Video-Text Matching
($\mathcal{L}_{\text{VTM}}$), Masked Language Modeling
($\mathcal{L}_{\text{MLM}}$), and Prefix Language Modeling
($\mathcal{L}_{\text{PrefixLM}}$). The VTC task first is applied to align the
unimodal representation of video and text. And the multi-modal representation
can be learned by VTM and MLM tasks. Upon on the video-language
representations obtained from multi-modal encoder, the decoder is trained by
PrefixLM loss with text completion task.

##### Video-Text Contrast (VTC)

Following [28, 58], we align the unimodal encoders via this task. Specially,
the softmax-normalized video-to-text and text-to-video similarities are
computed, and we employ memory queues in MoCo [7] to increase the number of
negative samples during learning. Formally, the video-text contrastive loss is
calculated as:

$\displaystyle\mathcal{L}_{v2t}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s(\mathcal{V}_{i},\mathcal{T}_{i}))}{\sum_{j=1}^{B}\exp(s(\mathcal{V}_{i},\mathcal{T}_{j}))},$
(9) $\displaystyle\mathcal{L}_{t2v}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s(\mathcal{V}_{i},\mathcal{T}_{i}))}{\sum_{j=1}^{B}\exp(s(\mathcal{V}_{j},\mathcal{T}_{i}))},$
$\displaystyle\mathcal{L}_{\text{VTC}}$
$\displaystyle=\frac{1}{2}(\mathcal{L}_{v2t}+\mathcal{L}_{t2v}),$

where $\mathcal{V}_{i}$ and $\mathcal{T}_{j}$ are the projected
representations of $v_{\text{cls}}$ and $w_{\text{cls}}$ for $i$-th video-text
pair in the batch.

##### Video-Text Matching (VTM)

This task aims to predict whether a video and a text is paired or not based on
the multi-modal representation. As suggested in [30, 28], hard negative video-
text pairs are selected based on the similarity of video and text during
contrastive learning. Formally, the video-text matching loss is calculated as:

$\mathcal{L}_{\text{VTM}}=-\mathbb{E}_{(\mathcal{W},\mathcal{V})}\log
p(y|\mathcal{W},\mathcal{V}),$ (10)

where $\mathcal{W}$ denotes the word tokens, and $\mathcal{V}$ denotes the
video features of long-view video.

##### Masked Language Modeling (MLM)

The setup of this pre-training task is same as that used in BERT [10], where
15% of tokens in the text are randomly masked, and the model needs to predict
the masked tokens based on the multi-modal representation. Formally, the
masked language modeling loss is calculated as:

$\mathcal{L}_{\text{MLM}}=-\mathbb{E}_{(\mathcal{W},\mathcal{V})}\log
p(w_{i}|\mathcal{W}_{\backslash i},\mathcal{V}),$ (11)

where $w_{i}$ denotes the masked word token.

##### Prefix Language Modeling (PrefixLM)

This pretext task requires model to complete the truncated texts based on
given videos and prefix sequence of truncated texts [27, 29]. The model can be
trained by maximizing the likelihood of the truncated text in an auto-
regressive manner. Formally, the prefix language modeling loss is calculated
as:

$\mathcal{L}_{\text{PrefixLM}}=-\mathbb{E}_{(\mathcal{W},\mathcal{V})}\left[\sum_{l=L_{p}}^{L}\log
p(w_{l}|\mathcal{W}_{[L_{p},l)},\mathcal{W}_{<L_{p}},\mathcal{V})\right],$
(12)

where $L$ denotes the total number of words in the text, and $L_{p}$ is the
length of a prefix sequence of tokens which is randomly selected.

Dataset | Optimizer | Learning Rate | Weight Decay | LR Schedule | Batch Size $\times$ # GPUs | Epochs
---|---|---|---|---|---|---
MSRVTT-Ret [64] | AdamW | 2e-5 | 0.02 | Cosine Decay | $24\times 8$ | 10
DiDeMo [1] | AdamW | 1e-5 | 0.02 | Cosine Decay | $24\times 8$ | 20
LSMDC [49] | AdamW | 2e-5 | 0.02 | Cosine Decay | $24\times 8$ | 10
Activity Caption [21] | AdamW | 2e-5 | 0.02 | Cosine Decay | $24\times 8$ | 20
SSv2-Template [25] | AdamW | 5e-5 | 0.02 | Cosine Decay | $24\times 8$ | 20
SSv2-Label [25] | AdamW | 2e-5 | 0.02 | Cosine Decay | $24\times 8$ | 20
MSRVTT-QA [62] | AdamW | 2e-5 | 0.02 | Cosine Decay | $16\times 8$ | 8
MSVD-QA [62] | AdamW | 2e-5 | 0.02 | Cosine Decay | $16\times 8$ | 8
TGIF-FrameQA [16] | AdamW | 2e-5 | 0.02 | Cosine Decay | $16\times 8$ | 8
LSMDC-FIB [42] | AdamW | 2e-5 | 0.02 | Cosine Decay | $16\times 8$ | 8
ActivityNet-QA [70] | AdamW | 2e-5 | 0.02 | Cosine Decay | $16\times 8$ | 8
TGIF-Action [16] | AdamW | 3e-5 | 0.02 | Cosine Decay | $16\times 8$ | 56
TGIF-Transition [16] | AdamW | 3e-5 | 0.02 | Cosine Decay | $16\times 8$ | 30
LSMDC-MC [56] | AdamW | 2e-5 | 0.02 | Cosine Decay | $16\times 8$ | 10
NExT-QA [60] | AdamW | 2e-5 | 0.02 | Cosine Decay | $16\times 8$ | 10
MSRVTT-Caption [64] | AdamW | 2e-5 | 0.02 | Cosine Decay | $24\times 8$ | 10
MSVD-Caption [5] | AdamW | 2e-5 | 0.02 | Cosine Decay | $24\times 8$ | 10
Table 16: End-to-end fine-tuning configurations for video-language downstream
tasks.

###  C.4 Downstream Task Implementation Details

We evaluate HiTeA on various downstream video-language tasks, including Text-
to-Video Retrieval, Open-ended VideoQA, Multiple Choice VideoQA, and Video
Captioning. The fine-tuning procedures are described as follows:

  * •

For retrieval tasks, we jointly optimize the VTC loss and VTM loss for video-
text alignment during fine-tuning. During inference, we first select top-k
candidates by computing the dot-product similarity between the video and text
features, and then reranking the selected candidates based on their VTM
scores. $k$ is set to 128 by default.

  * •

For open-ended VideoQA, we first generate video features and text features
with two unimodal encoders, and then fuse them with multi-modal encoder. The
output of multi-modal features are fed to text decoder for answer generation.
We use the language modeling loss to optimize the model. During inference, the
answer would be generated by the text decoder.

  * •

For multiple choice VideoQA, we treat the problem as the text-to-video
retrieval task where the correct answer should have the highest matching
probability. During training, we compute the VTM scores for each candidate
answer and video, then optimize the model with cross entropy loss. During the
inference, the answer with highest VTM score is the prediction answer.

  * •

For Video Captioning, we use the video features from video encoder and
directly feed it into text decoder for caption generation. The language
modeling loss is utilized for model optimization.

For all above video-language downstream tasks, we resize video frames to
$224\times 224$. During fine-tuning, following [25, 28], we randomly sample 12
frames for text-to-video retrieval, 16 frames for video question answering and
video captions. We perform uniform sampling during inference. We use
RandomCrop with minimum ratio 0.5 and HorizontalFlip with 0.5 probability for
data augmentation. The hyperparameters that we used for fine-tuning on
downstream tasks are summarized in Table 16. For the video caption task, we
use a prefix prompt “A video of” to improve the quality of generated captions.

###  C.5 Datasets Description

In this section, we describe all of the downstream video-language datasets
used during evaluation. The details of the datasets are represented below:

##### Text-to-Video Retrieval.

We evaluate HiTeA on 6 popular text-to-video retrieval datasets including
MSRVTT [64], DiDeMo [1], LSMDC [49], ActivityNet Caption [21], SSv2 Template
[25], and SSv2 Label [25]. Details of these datasets: MSRVTT [64] contains 10K
YouTube sourced videos with 200K text descriptions. Following [32, 40, 15], we
train the video on 9K videos and evaluate on the rest 1K video. DiDeMo [1]
contains of 10K videos from Flickr and 4 descriptions for each video.
Following [28, 41, 32], we concatenate all of the given descriptions from the
same video as a paragraph, and evaluate the paragraph-to-video retrieval
performance. The number of video in training set is 8K, leaving 1K for
validation set and 1K for test set. LSMDC [49] consists of 118K video clips
from 202 movies, and each clip is accompanied with a caption from video
scripts. It has 101K video clips for training and 1K clips for testing. We use
the standard splits from [49]. ActivityNet Caption [21] is built on 20K
YouTube videos with 100K captions. We use the train split with 10K videos for
training, and report the performance on the val1 split with 4.9K videos.
SSv2-Template and SSv2-Label [25] contain 169K videos for training and 2K
videos for testing. The text queries in SSv2-Template are templates without
object information (_e.g_. "Throwing [something] in the air and catching it").
By contrast, SSv2-Label contains annotated text queries with specific object
information (_e.g_. "Throwing keys in the air and catching it"). Therefore,
SSv2-Template mainly focuses on temporal understanding of actions, while
SSv2-Label needs a more comprehensive understanding of both appearance and
temporal dynamic.

##### Multiple-choice Video QA.

Five datasets are evaluated for multiple-choice video question answering
tasks. TGIF-Action and TGIF-Transition [16] are adopted to evaluate model’s
capability to recognize the repeated actions and state transitions in short
GIFs. Each video and question is equipped with 5 candidate answers. We
concatenate the question and answer as the text and use the highest similarity
among the video and candidate texts. TGIF-Action contains 18K GIFs for
training and 2K for testing. TGIF-Transitions has 47K GIF-question pairs for
training and 6K for testing. MSRVTT-MC [69] and LSMDC-MC [56] are originally
retrieval task, but reformulated as the multiple choice video QA task. It
requires the model to find the optimal caption that describes the video out of
5 candidate texts. NExT-QA [60] is explicitly designed for temporal and causal
understanding. Questions in the dataset are categorized into three types:
Descriptive, Temporal, and Causal. Each question in the dataset are paired
with 5 candidate answers. Therefore, this dataset is able to evaluate model’s
ability in video question answering in different aspects.

##### Open-ended Video QA.

For open-ended video QA, we evaluate the model on five datasets. MSRVTT-QA is
composed of 243K open-ended questions over 10K videos, while MSVD-QA [62]
consists 2K videos with 47K questions. TGIF-Frames [16] collects the
answerable with just a single frame in the video, and is divided into training
set with 35K questions and test set with 14K questions.. For LSMDC-FiB [42],
the model needs to predict a correct word for the blank with a given video and
a sentence with blank. It contains 297K sentences for training and 30K
sentences for testing. ActivityNet-QA [70] .

##### Video Captioning.

We use MSRVTT [64] and MSVD [5] for video captioning evaluation. As described
before, MSRVTT is composed of 10K videos with 20 captions per video, and MSVD
contains 2K videos with around 40 captions per video. We follow the standard
splits from [37, 32]. During inference, we generate the caption with beam
search until the model outputs a [SEP] that indicates the end of sentence or
when it reaches the maximum generation step 40.