# VIOLET : End-to-End Video-Language Transformers with
Masked Visual-token Modeling

Tsu-Jui Fu†, Linjie Li‡, Zhe Gan‡, Kevin Lin‡, William Yang Wang†, Lijuan
Wang‡, Zicheng Liu‡
†UC Santa Barbara ‡Microsoft
{tsu-juifu<EMAIL_ADDRESS>
{lindsey.li, zhe.gan, keli, lijuanw<EMAIL_ADDRESS>
###### Abstract

A great challenge in video-language (VidL) modeling lies in the disconnection
between fixed video representations extracted from image/video understanding
models and downstream VidL data. Recent studies try to mitigate this
disconnection via end-to-end training. To make it computationally feasible,
prior works tend to “imagify” video inputs, i.e., a handful of sparsely
sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or
concatenation to obtain the overall video representations. Although achieving
promising results, such simple approaches may lose temporal information that
is essential for performing downstream VidL tasks. In this work, we present
VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video
transformer to explicitly model the temporal dynamics of video inputs.
Further, unlike previous studies that found pre-training tasks on video inputs
(e.g., masked frame modeling) not very effective, we design a new pre-training
task, Masked Visual-token Modeling (MVM), for better video modeling.
Specifically, the original video frame patches are “tokenized” into discrete
visual tokens, and the goal is to recover the original visual tokens based on
the masked patches. Comprehensive analysis demonstrates the effectiveness of
both explicit temporal modeling via video transformer and MVM. As a result,
VIOLET achieves new state-of-the-art performance on 5 video question answering
tasks and 4 text-to-video retrieval tasks.111Code is available at
https://github.com/tsujuifu/pytorch_violet

##  1 Introduction

Humans are born to perceive this world from multiple modalities such as
vision, sound, and touch. Video, containing multiple modalities in nature, has
been used as an epitome to test how AI systems perceive. Video-language (VidL)
research aims at extending this ability to convey perception via language.
Popular VidL tasks were introduced, such as text-to-video retrieval [1, 2, 3,
4, 5], video question answering [6, 7, 8, 9], text-based video moment
retrieval [10, 11, 2, 4], and video captioning [12, 13, 1, 3].

Figure 1: End-to-end VIdeO-LanguagE Transformer (VIOLET). VIOLET performs
large-scale visual-text pre-training and can be applied to various video
question answering and text-to-video retrieval tasks.

Previous works [6, 14, 15, 16, 5, 17] attempt cross-modal fusion over dense
video features and text features to tackle VidL tasks, but suffer from domain
disconnection due to offline feature extraction [18, 16]. To address this
issue, ClipBERT [18] proposes to “imagify” the dense video frame inputs.
First, it adopts a sparse sampling strategy to employ only a handful of frames
from the entire video for efficient end-to-end training. Second, the overall
video representations are obtained through mean-pooling a sequence of frame
features, individually computed by a 2D Convolutional Network. Although
obtaining promising results, the brash mean pooling over individual frame
features forfeits the crucial temporal information in video. To improve
temporal modeling, recent works [19, 20] concatenate all sparse-sampled frame
features in chronological order, and directly enforce VidL learning along with
the text inputs. However, these methods still treat video frames as static
images, and rely heavily on cross-modal fusion module to capture both temporal
dynamics in videos and the alignment between visual and textual elements
simultaneously.

We propose fully end-to-end VIdeO-LanguagE Transformer (VIOLET) to enhance
video modeling for better VidL modeling from two perspectives: ($i$) model
architecture, and ($ii$) pre-training task design. In terms of _model
architecture_ , instead of naive mean pooling or concatenation over a sequence
of individual frame features, VIOLET contains Video Swin Transformer that
models video temporal explicitly for VidL learning [21, 22]. Since the self-
attention over spatial-temporal locality allows modeling variable sequence
lengths, our video transformer support flexible VidL learning from both videos
and static images.

In terms of _pre-training tasks_ , though the direct adoption of Masked
Language Modeling [23] has proven effective in pre-training vision-language
models, the attempt on similar masked modeling on vision inputs is not as
successful. For example, Masked Region Modeling (MRM) [24] or Masked Frame
Modeling (MFM) [5] aim to recover masked image regions or video frames.
Despite of the different variants of MRM/MFM that model object category or
distilled region/frame features, it suffers from imperfect patch labels,
excessive feature dimensions, rendering unsatisfactory performance [24, 5].
Recent VidL works [18, 19, 25] even completely discard such pre-training tasks
due to limited performance improvements.

To promote better video representations for VidL learning, we present a new
pre-training task: Masked Visual-token Modeling (MVM), as shown in the left of
Fig. 1. By using the pre-trained discrete VAE [26] from DALL-E [27], we
“tokenize” the video frames into discrete visual tokens, which can be used to
reconstruct the original video frames. During pre-training, we mask out some
proportions of the video input along both spatial and temporal dimensions, and
the model learns to recover the discrete visual tokens of these masked
patches. MVM improves over previous MRM/MFM in two ways: ($i$) MVM predicts
over a discrete space, which avoids falling ill with the similar training
issues of excessive feature dimensions as in [24, 5]; ($ii$) MVM is based on
latent visual tokens obtained from a self-reconstruction training procedure,
instead of distilling from a well-supervised visual backbone. Our
comprehensive comparison shows that MVM enhances the model’s capability to
better understand video scenes and in turn benefits downstream VidL tasks.

In summary, our contributions are four-fold. ($i$) We present VIOLET, a fully
end-to-end transformer to model the spatial-temporal dynamic in videos for
VidL learning. ($ii$) We propose a new pre-training task, Masked Visual-token
Modeling, which recovers the masked video frame patches into a discrete visual
token space. ($iii$) VIOLET achieves state-of-the-art results on 4 text-to-
video retrieval and 5 out of 8 video question answering tasks. ($iv$)
Comprehensive ablation studies demonstrate the necessity of temporal video
modeling and the effectiveness of MVM across different VidL pre-training
settings.

Figure 2: Overview of the proposed end-to-end VIdeO-LanguagE Transformer
(VIOLET), with Video Swin Transformer, Language Embedder, and Cross-modal
Transformer. VIOLET adopts Discrete VAE to extract discrete visual tokens to
perform Masked Visual-token Modeling along with Visual-Text Matching and
Masked Language Modeling during large-scale visual-text pre-training.

##  2 Related Work

Video-Language Understanding. Joint video-language (VidL) understanding [28,
29, 30, 14, 31, 32] aims at interpreting the physical world via both vision
and text perception. Researchers have explored such capability on VidL tasks
including text-based video retrieval [1, 2, 3, 4, 5], video question answering
[6, 7, 8, 9], moment retrieval [10, 11, 2, 4], and video captioning [12, 13,
1, 3]. Prior arts before the large-scale pre-training era [33, 34, 35, 36, 14,
9] leverage offline extracted video features [37, 38, 39, 40, 41, 42, 43, 44,
45]. Later on, VidL pre-trained models [46, 15, 5, 16] built on the above pre-
extracted features have shown promising results. To enhance the performance,
there have been parallel interests in bringing in more modalities from raw
video inputs [31, 47, 48] and end-to-end training till the raw pixels [49, 18,
19, 25], both aiming to elevate video representations for VidL modeling.

Our work further explores the second direction for general VidL understanding,
in contrast to [25] focusing on text-to-video retrieval only. Instead of
encoding each video frame individually as static image and applying simple
mean-pooling or concatenation along the temporal dimension [18, 19], we
demonstrate the necessity of temporal modeling by our video transformer over
the input video frames, even when they are sparsely sampled [18].

Masked Visual Modeling. Aligned with the success of transformer-based [50]
language pre-training models [51, 52, 53, 54, 55], image-text pre-training
[56, 57, 58, 59, 60, 61, 62, 63, 64, 65] and video-text pre-training [66, 67,
68] have shown promising results on diverse vision-language (VL) tasks.
Popular VL pre-training tasks include Visual-Text Matching and Masked Language
Modeling, which are directly adapted from language pre-training [23]. Similar
masked modeling on visual inputs [24, 5] has also been introduced to VL pre-
training, but are not as useful. We propose Masked Visual-token Modeling
(MVM), adopting the latent codes of discrete VAE [27, 69, 26] as the
reconstruction target for masked patches, which eases the auto-encoding
prediction and can lead to a more significant improvement.

Among the literature, BEiT [70] and VIMPAC [71] are two relevant studies of
masked visual modeling for image classification [42] and action recognition
[37]. Specifically, BEiT [70] proposes a BERT-like pre-training strategy to
recover the original visual tokens from some masked image patches. Our MVM
takes inspiration from BEiT, but extends to more complex video inputs with an
additional temporal dimension for VidL modeling. To prevent the model from
taking shortcuts in recovering visual tokens from its spatial or temporal
neighbors, we further introduce a combination of blockwise masking and
attended masking. VIMPAC [71] takes a step further to completely remove the
raw pixel inputs from the training procedure. It employs visual tokens as the
discrete representation of video inputs and applies a mask-then-predict pre-
training task. The removal of raw pixel inputs renders a weaker baseline on
popular action recognition tasks. In our work, we leverage visual tokens as
prediction targets for MVM, instead of replacing the raw video frame patches.

##  3 VIOLET

###  3.1 Model Architecture

Fig. 2 illustrates the overall architecture of our end-to-end video-language
transformer (VIOLET). VIOLET contains 3 components: Video Swin Transformer
(VT), Language Embedder (LE), and Cross-modal Transformer (CT). VIOLET takes
video $\mathcal{V}$ and sentence $\mathcal{X}$ as inputs. Sparse-sampled
frames $\\{f_{1},f_{2},...\\}$ from $\mathcal{V}$ are first processed by VT to
compute video features $v=\\{v_{1},v_{2},...\\}$. LE extracts the word
embeddings $w=\\{w_{1},w_{2},...\\}$ for each word token
$\\{x_{1},x_{2},...\\}$ in $\mathcal{X}$. Then CT performs cross-modal fusion
on top of $v$ and $w$ to produce joint video-language (VidL) representations
$h$ for pre-training and downstream finetuning. We explain each component in
detail below.

Video Swin Transformer (VT). Instead of mean pooling or concatenating
individual frame features, we adopt Video Swin Transformer [22] (VT) to model
$T$ sparse-sampled frames $\\{f_{t}\\}_{t=1}^{T}$ along both spatial and
temporal dimensions as video features $\\{v_{t}\\}_{t=1}^{T}$. VT first splits
each frame as non-overlapping $H\times W$ patches [72] and adopts a linear
projection layer to obtain the preliminary video patch embeddings
$u\in\mathbb{R}^{T\times H\times W\times d}$:

$u_{t}=\text{LinearProj}(f_{t}).$ (1)

The multi-layer 3D-shifted window [22] then considers different levels of
spatial-temporal attention over these video patch embeddings. We add learnable
positional embedding $p^{\text{v}}$ to $u$, including spatial
$p^{\text{s}}\in\mathbb{R}^{H\times W\times d}$ and temporal ordering
$p^{\text{t}}\in\mathbb{R}^{T\times d}$, and extracts the video features $v$:

$\begin{split}p^{\text{v}}_{t}&=p^{\text{s}}+p^{\text{t}}_{t},\\\
v&=\text{VT}(\\{u_{t}+p_{t}^{\text{v}}\\}_{t=1}^{T}).\end{split}$ (2)

All patches from the $t^{\text{th}}$ frame shares the same $p^{\text{t}}_{t}$
and all patches with the same spatial position are given the same
$p^{\text{s}}$. In particular, each 3D window is in the size of
$T^{\prime}\times M\times M$ and considers video temporal across $T^{\prime}$
consecutive frames. By adopting 3D windows upon blocks of video patches, VT
can model image spatial and video temporal simultaneously through the self-
attended computation procedure. Note that we make a slight modification to
remove the temporal down-sampling from the original Video Swin Transformer and
ensure the same temporal dimension as the input video for Masked Visual-token
Modeling during pre-training (Sec. 3.2).

VT enforces spatial-temporal modeling via 3D-shifted window to compute the
initial video representations for VidL modeling. We demonstrate the advantages
of VT over simple mean-pooling or concatenation of “imagified” frame
representations under different VidL pre-training settings in Sec. 4.3. In
addition, as VT encodes video frame patches through a fully self-attended
computation, it can support a variable length of visual inputs. This video
encoding enables VIOLET to carry out static images (i.e., $T$ = 1). We discuss
the flexibility of pre-training VIOLET on both large-scale image-text data and
video-text data in Sec. 5.

Language Embedder (LE). For a language input $\mathcal{X}$, we follow
WordPiece [73] and tokenize it into word tokens $\\{x_{i}\\}_{i=1}^{L}$, where
$L$ is the number of tokens in $\mathcal{X}$. LE embeds the discrete word
token $x_{i}$ into high-dimensional word representation
$w_{i}\in\mathbb{R}^{d}$ :

$\\{w_{i}\\}_{i=1}^{L}=\text{LE}(\\{x_{i}\\}_{i=1}^{L}).$ (3)

Cross-modal Transformer (CT). Given video features $v$ and word features $w$,
CT performs cross-modal fusion over all $\\{v_{i}\\}_{i=1}^{T}$ and
$\\{w_{i}\\}_{i=1}^{L}$ for joint VidL learning. We add different positional
embeddings $p^{\text{v}}$ or $p^{\text{x}}$ to video features $v$ or word
features $w$, to incorporate sequence ordering and distinguish between the two
modalities. In particular, we reuse $p^{\text{v}}$ from VT, containing both
spatial position and temporal ordering information. We concatenate the video
and text representations after position embedding as the input sequence to CT.
In addition, a special [CLS] token is added to compute the global VidL
representation, used in pre-training and downstream finetuning. The joint VidL
features $h\in\mathbb{R}^{(T+1+L)\times d}$ are computed as:

$\begin{split}h&=\text{CT}([v+p^{\text{v}},\texttt{[CLS]},w+p^{\text{x}}]),\\\
[h^{\text{v}},&~{}h^{\text{c}},h^{\text{x}}]=h,\end{split}$ (4)

###  3.2 Pre-training Tasks.

To benefit from large-scale data [19, 25, 74], we incorporate three pre-
training tasks, including our proposed Masked Visual-token Modeling. Masked
Language Modeling [23, 24, 5] predicts the masked word tokens to improve
language reasoning with the aid of visual perception. Masked Visual-token
Modeling recovers the masked video patches to enhance the video scene
understanding. Visual-Text Matching [24, 18, 25] learns the alignments between
video and text modality, improving the cross-modal fusion.

Masked Language Modeling (MLM). In MLM, we randomly mask out some word tokens
with a probability of 15%.222Following BERT [23], We replace 80% of masked
word tokens as the [MASK] token, 10% as a random token, and 10% as its
original token. The goal is to recover these masked tokens $x$ from the joint
VidL features $h$ modeled by Cross-modal Transformer (CT). Specifically, the
corresponding $h^{\text{x}}$ for these masked tokens are fed in a fully-
connected (FC) layer ($\text{FC}^{\text{MLM}}$) and projected to the discrete
word token space for classification:

$\begin{split}x^{\prime}_{i}&=\text{FC}_{\text{MLM}}(h^{\text{x}}_{i}),\\\
\mathcal{L}_{\text{MLM}}&=-\mathbb{E}~{}[\frac{1}{|\mathcal{M}^{\text{MLM}}|}\sum\nolimits_{i\in\mathcal{M}^{\text{MLM}}}\log
P(x_{i}~{}|~{}x^{\prime}_{i})],\end{split}$ (5)

where $\mathcal{M}^{\text{MLM}}$ denotes the index set of masked word tokens.

Visual-Text Matching (VTM). VTM enhances the cross-modal fusion via modeling
the alignments between visual and textual inputs. At each training step, we
randomly replace the corresponding text $\mathcal{X}_{\text{pos}}$ for a given
video $\mathcal{V}$ with the text description $\mathcal{X}_{\text{neg}}$ from
a different video in the same batch. Both the positive pair
$(\mathcal{V},\mathcal{X}_{\text{pos}})$ and negative pair
$(\mathcal{V},\mathcal{X}_{\text{neg}})$ are modeled by CT, and VTM is to tell
them apart from the global VidL representation $h^{\text{c}}$ of the [CLS]
token. In particular, $h^{\text{c}}$ will be processed by a FC layer
($\text{FC}^{\text{VTM}}$) to perform binary classification:

$\begin{split}b_{\text{pos}}&=\text{FC}^{\text{VTM}}(h^{\text{c}}_{\text{pos}}),b_{\text{neg}}=\text{FC}^{\text{VTM}}(h^{\text{c}}_{\text{neg}}),\\\
\mathcal{L}_{\text{VTM}}&=-\mathbb{E}[\log(b_{\text{pos}})+\log(1-b_{\text{neg}})],\end{split}$
(6)

where $h^{\text{c}}_{\text{pos}}$ or $h^{\text{c}}_{\text{neg}}$ is
$h^{\text{c}}$ of positive or negative pairs.

Masked Visual-token Modeling (MVM). Previous Masked Region Modeling (MRM) [24]
and Masked Frame Modeling (MFM) [5] extends MLM to visual inputs but sometimes
leads to unsatisfactory performance. Different from MRM and MFM, which rely on
distilled visual categories or features from a well-supervised visual backbone
[45, 41], we present Masked Visual-token Modeling (MVM) to perform masked
visual modeling in a self-reconstruction scenario. We consider the discrete
variational autoencoder (dVAE) [26, 27] to quantize video inputs into masked
prediction targets. dVAE is learned to tokenize images into discrete visual
tokens $q$ from a finite vocabulary and then reconstruct the original visual
scene based on $q$, where $q$ should have a one-to-one correspondence with the
input image patches spatially. We first adopt dVAE to tokenize the
$t^{\text{th}}$ video frame $f_{t}$ into $q_{t}$:

$q_{t}=\text{dVAE}(f_{t}).$ (7)

Similar to MLM, we mask out some video patches by replacing the pixel values
with all zeros. MVM aims at recovering the visual tokens $q$ of those masked
video patches $v$ from the corresponding joint VidL features $h^{\text{v}}$.
$h^{\text{v}}$ is fed into a FC layer ($\text{FC}^{\text{MVM}}$) and projected
to the discrete visual token space for classification:

$\displaystyle q^{\prime}_{t,i}$
$\displaystyle=\text{FC}^{\text{MVM}}(h^{\text{v}}_{t,i}),$ (8)
$\displaystyle\mathcal{L}_{\text{MVM}}$
$\displaystyle=-\mathbb{E}~{}[\sum_{t=1}^{T}\frac{1}{|\mathcal{M}^{\text{MVM}}_{t}|}\sum\nolimits_{i\in\mathcal{M}^{\text{MVM}}_{t}}\log
P(q_{t,i}~{}|~{}q^{\prime}_{t,i})],$

where $\mathcal{M}^{\text{MVM}}_{t}$ is the index set of masked video patches
for the $t^{\text{th}}$ frame. Using discrete visual tokens as masked
prediction targets has two main advantages: ($i$) The finite vocabulary size
of these discrete visual tokens eases the learning of MVM, avoid the previous
difficulty in model training with MRM/MFM from imperfect patch categories or
excessive feature dimensions; ($ii$) MVM does not require a well-supervised
visual backbone to distill the masking labels. The latent visual tokens can be
learned in a self-supervised way without human annotations.

###  3.3 Masking Strategy of MLM and MVM

We introduce a combination of Blockwise Masking and Attended Masking to
amplify the effectiveness of MLM and MVM, as shown in Fig. 3.

Blockwise Masking (BM). Video usually presents analogous visual patterns in
spatial-temporal neighbors (i.e., nearby patches within current frame or
neighboring frames). While these neighbors make the masked video patches easy
to recover, they may lead to spurious success in MVM evaluation. To make MVM
more challenging, we adopt Blockwise Masking [71, 70] that masks blocks of
video patches along spatial-temporal dimension rather than independently
masking randomly sampled patches for each frame. Specifically, we randomly
sample an $(H^{\prime},W^{\prime},T^{\prime})$ as a masking block, where all
$H^{\prime}\times W^{\prime}$ visual patches in the following $T^{\prime}$
consecutive frames will be masked; We repeat this process until $>$15% of
video patches are masked to perform MVM pre-training. The model cannot merely
rely on similar neighboring visual cues but requires actual visual reasoning
to recover a group of missing patterns.

Figure 3: Masking Strategy of MLM and MVM, including Blockwise Masking (BM)
and Attended Masking (AM).

Attended Masking (AM). The conventional practice is to sample masked visual
patches or textual tokens with the same probability over all visual and
textual inputs. However, the important elements (e.g., visual patches
containing the main object or content words) receive the same weight as the
less relevant elements (e.g., scene background or stop words) in masked
modeling. Attended Masking tries to put more weights on the more important
elements based on the attention weights computed by Cross-modal Transformer
(CT). A similar idea has been explored in [19] for MLM. In this paper, we
extend AM to both visual and textual modalities. We first keep the video-text
inputs intact, feed them into CT to compute the attention weights, to decide
which portions in video and text are more important. We then select the top
15% of most-attended tokens to be masked in both video and text inputs to
perform MVM and MLM.

| Text-to-Video Retrieval
---|---
Method | MSRVTT | DiDeMo | YouCook2 | LSMDC
Models using Pre-extracted Features |
HT100M [16] | 14.9 / 40.2 / 52.8 | - |  08.2 / 24.5 / 35.3 |  07.1 / 19.6 / 27.9
MMT [31] | 26.6 / 57.1 / 67.1 | - | - | 12.9 / 29.9 / 40.1
HERO [5] | 16.8 / 43.4 / 57.7 | - | - | -
AVLnet [47] | 27.1 / 55.6 / 66.6 | - | 33.2 / 61.0 / 71.5 | 17.0 / 38.0 / 48.6
Support-Set [32] | 30.1 / 58.3 / 69.3 | - | - | -
TACo [75] | 28.4 / 57.8 / 71.2 | - | 29.6 / 59.7 / 72.7 | -
VideoCLIP [17] | 30.9 / 55.4 / 66.8 | - | 32.2 / 62.6 / 75.0 | -
Models with End-to-end Training |
ClipBERT [18] | 22.0 / 46.8 / 59.9 | 20.4 / 48.0 / 60.8 | - | -
Frozen [25] | 32.5 / 61.5 / 71.2 | 31.0 / 59.8 / 72.4 | - | 15.0 / 30.8 / 39.8
Clip4Clip [76] | 42.1 / 71.9 / 81.4 | 43.4 / 70.2 / 80.6 | - | 21.6 / 41.8 / 49.8
VIOLET |  34.5 / 63.0 / 73.4 |  32.6 / 62.8 / 74.7 |  35.7 / 66.7 / 78.2 |  16.1 / 36.6 / 41.2
(a)

| Zero-Shot Retrieval
---|---
Method | MSRVTT | DiDeMo
Models using Pre-extracted Features
HT100M [16] |  07.5 / 21.2 / 29.6 | -
MMT [31] | 0-0 / 06.9 / 0-0 | -
AVLnet [47] | 19.6 / 40.8 / 50.7 | -
Support-Set [32] | 12.7 / 27.5 / 36.2 | -
TACo [75] |  09.8 / 25.0 / 33.4 | -
VideoCLIP [17] | 10.4 / 22.2 / 30.0 |  16.6 / 46.9 / 0-0
Models with End-to-end Training
MIL-NCE [49] |  09.9 / 24.0 / 32.4 | -
VATT [77] | 0-0 / 00-0 / 29.7 | -
Frozen [25] | 24.7 / 46.2 / 57.2 | 21.1 / 46.0 / 56.2
CLIP [78, 76] | 31.2 / 53.7 / 64.2 | -
VIOLET |  25.9 / 49.5 / 59.7 |  23.5 / 49.8 / 59.8
(b)

Table 1: Comparison with SOTA on text-to-video-retrieval tasks under different
settings: (a) pre-train then finetune and (b) pre-train then zero-shot
evaluation. All resutls are reported on R@1 / R@5 / R@10. All models perform
visual-text pre-training. Rows highlighted in blue use additional modalities
such as sound and speech besides video frames.

##  4 Experiments

###  4.1 Experimental Setup

Pre-training Datasets. As mentioned in Sec. 3.1, VIOLET is flexible in taking
both video and image as inputs. Hence, We follow [25] to jointly pre-train our
model on image-text and video-text data, which we briefly describe below.
($i$) YT-Temporal-180M (YT-Temporal) [19] contains 6M YouTube videos with
subtitle texts from Automatic Speech Recognition (ASR). Following [19], we
divide a long video into several video segments, with an average length of
9.29 seconds. We treat every 4 consecutive segments with their ASR as a video
clip, leading to 180M video-subtitle pairs. ($ii$) WebVid-2.5M (WebVid) [25]
scrapes 2.5M video-text pairs from the web. Different from YT-Temporal, text
data in WebVid describes the global video semantic. ($iii$)
ConceptualCaptions-3M (CC) [74] consists of 3.3M image-text pairs harvested
from the web. We compare the effects of different pre-training data on
downstream tasks in Sec. 5.

Downstream Tasks. We evaluate VIOLET on both text-to-video retrieval and video
question answering, across 12 downstream benchmarks. For text-to-video
retrieval, we report performance of Recall at K (R@K) on MSRVTT [1], DiDeMo
[79], YouCook2 [13] and LSMDC [3]. For video question answering, we consider 8
datasets in multiple-choice and open-ended settings: TGIF-Action, TGIF-
Transition and TGIF-Frame [6], MSRVTT-MC [80], MSRVTT-QA, MSVD-QA [7], LSMDC-
MC and LSMDC-FiB [81]. Accuracy is used as evaluation metric. More details are
provided in Appendix A.

Implementation Details. We initialize our Video Swin Transformer with
VideoSwin-Base [22], pre-trained on Kinetics-400 [37]. Language Embedder and
Cross-modal Transformer are initialized from pre-trained BERT-Base [59]. We
train VIOLET in a end-to-end manner for both pre-training and downstream
finetuning.

During pre-training, we sparsely sample $T$ = 4 video frames and resize them
into 224x224 to split into patches with $H$ = $W$ = 32. We use pre-trained
DALL-E [27] as our dVAE to generate discrete visual tokens for MVM. For WebVid
[25] and CC [74], we perform VTM+MLM+MVM to pre-train on videos or images with
the globally-aligned alt-text descriptions. We follow [19] to concatenate all
ASR descriptions for each middle frame as text input for YT-Temporal. VTM is
performed for each pair of middle frame and its ASR text to learn the temporal
reasoning over YT-Temporal video clips. Our implementation of VIOLET is based
on PyTorch [82]. We adopt AdamW [83] as the optimizer with an initial learning
rate of 2e-5, betas of (0.9, 0.98), and weight decay of 1e-3 for all pre-
training experiments. VIOLET follows a simple curriculum learning strategy,
where we first pre-train on YT-Temporal with noisy ASR text for 5 epochs and
then on WebVid+CC with alt-text descriptions for another 5 epochs.

For all downstream tasks, we adopt the same video frame size (224x224) and
patch size (32x32) but 5 sparse-sampled frames. Due to various data scales and
domains, we use task-specific learning rates and training epochs based on the
performance of the validation set for each downstream task.

###  4.2 Comparison to Prior Arts

Text-to-Video Retrieval. Table LABEL:table:retrieval summarizes results on
text-to-video retrieval. VIOLET achieves significant gain over existing VidL
pre-trained models across all text-to-video retrieval datasets considered.
Specifically, VIOLET surpasses most previous methods focus on modeling multi-
modal fusion with pre-extracted video features. Notably, VIOLET is still
competitive even when compared with MMT [31], HERO [5], and AVLnet [47] that
use additional modalities, such as sound and speech besides video frames.

For comparisons to end-to-end pre-trained models, VIOLET outperforms ClipBERT
[18] by $+10\%$ on R@1 on both MSRVTT and DiDeMo, even though VIOLET uses even
less frames (Ours: 5 frames vs. ClipBERT: 16 frames). These results highlight
the deficiency of ‘imagifying‘ video representations. When compared with
Frozen [25], designed specifically for text-to-video retrieval tasks, VIOLET
can achieve notable performance improvements with $+2.0\%$, $+1.6\%$ and
$+1.1\%$ on R@1 for MSRVTT, DiDeMo and LSMDC, respectively. We also include
results from Clip4Clip [76] that leverages pre-trained CLIP [78] on over 400M
image-text data, which is a few magnitude larger than our pre-training data.
VIOLET closes the gap between previous end-to-end pre-trained models and
Clip4Clip, and we believe pre-training VIOLET with larger-scale data can
further reduce the gap.

Zero-shot text-to-video retrieval. We further conduct generalizability
evaluation under the zero-shot setting on MSRVTT and DiDeMo in Table
LABEL:table:retrieval-zs. Similarly, VIOLET achieves remarkable performance
improvements over the existing methods by large margins. Specifically, we
observe $+6\%$ gain on R@1 over previous models using pre-extracted video
features and $+1.2-2\%$ on R@1 over end-to-end pre-trained models, excluding
CLIP [78, 76].

| TGIF | MSRVTT | LSMDC | MSVD
---|---|---|---|---
Method | Action | Transition | Frame | MC | QA | MC | FiB | QA
ClipBERT [18] | 82.8 | 87.8 | 60.3 | 88.2 | 37.4 | - | - | -
JustAsk [68] | - | - | - | - | 41.5 | - | - | 46.3
MERLOT [19] | 94.0 | 96.2 | 69.5 | 90.9 | 43.1 | 81.7 | 52.9 | -
VIOLET | 92.5 | 95.7 | 68.9 | 91.9 | 43.9 | 82.8 | 53.7 | 47.9

Table 2: Comparison with SOTA methods on video question answering. We gray out
MERLOT due to its excessive computational cost (e.g., 30K TPU hours vs. 2K GPU
hours (ours) for pre-training and frame resolution 704 vs. 224 for downstream
tasks).

Video Question Answering. We compare with prior arts on video question
answering (QA) tasks in Table 2. VIOLET surpasses ClipBERT [18] with
significant performance gain of $+9.7\%$ on TGIF-Action, $+7.9\%$ on TGIF-
Transition, $+8.6\%$ on TGIF-Frame, $+3.7\%$ on MSRVTT-MC and $+6.5\%$ on
MSRVTT-QA. These results suggest the explicit temporal modeling introduced by
our video transformer is essential for video QA tasks, and pre-training with
image-text data alone may not be sufficient for VidL modeling. We provide more
detailed discussions in Sec. 4.3.

Note that both JustAsk [68] and MERLOT [19] specifically focus on video QA.
JustAsk automatically generates 69M video-question-answer triplets from
narrated videos for training, which is hardly extendable to text-to-video
retrieval tasks. MERLOT is pre-trained for 40 epochs and with extensive
hyperparameter tuning on the frame resolution from 384x704 to 704x704 for
downstream tasks. The overall pre-training of MERLOT takes 30,720 TPU hours on
TPU v3. In contrast, we pre-train VIOLET for 5 epochs, which results in 2,240
GPU hours on V100 GPUs. We also adopt a much lower frame resolution of
224x224. With a much lower computational cost, VIOLET achieves around $+1.0\%$
performance gain over MERLOT on 4 video QA tasks for MSRVTT and LSMDC videos,
while remains competitive on TGIF. We believe VIOLET can further improve with
larger frame resolution and longer pre-training epoch if computational
resources permit.

Video | TGIF- | TGIF- | MSRVTT- | DiDeMo-
---|---|---|---|---
Encoding | Action | Transition | Retrieval | Retrieval
Random initialized visual encoder
Mean | 72.1 | 83.5 |  08.4 / 22.7 / 35.3 |  09.1 / 24.9 / 36.7
Concat | 72.9 | 83.7 |  09.0 / 23.5 / 35.5 |  09.4 / 25.8 / 38.1
VT | 73.6 | 84.6 |  09.2 / 24.0 / 35.8 |  10.3 / 30.1 / 40.5
ImageNet pre-trained visual encoder
Mean | 77.5 | 86.5 |  09.6 / 26.7 / 39.5 |  09.5 / 27.5 / 40.9
Concat | 78.0 | 87.0 | 10.4 / 30.5 / 42.0 | 10.6 / 30.8 / 42.9
VT | 79.6 | 87.8 |  11.8 / 32.3 / 44.6 |  12.0 / 32.4 / 43.5
\+ Video-text pre-training on WebVid
Mean | 80.3 | 88.7 | 20.8 / 44.9 / 58.1 | 17.9 / 43.5 / 51.3
Concat | 82.5 | 91.2 | 23.5 / 51.9 / 63.0 | 22.2 / 50.5 / 62.6
VT | 85.8 | 92.1 |  27.0 / 56.5 / 68.8 |  26.1 / 56.9 / 68.9

Table 3: Impact of different temporal modeling methods over video inputs under
different settings: ($i$) random initialized visual encoder; ($ii$) ImageNet
[84] pre-trained visual encoder and ($iii$) Adding video-text pre-training on
WebVid [25].

###  4.3 Analysis of VIOLET

We conduct ablation experiments on two video question answering datasets
(TGIF-Action and TGIF-Transition) and two text-to-video retrieval datasets
(MSRVTT and DiDeMo) to study the factors leading to VIOLET’s success.

Impact of Temporal Video Modeling. To demonstrate the necessity of temporal
modeling even under sparse sampling, we compare three variants for temporal
modeling in Table 3. ($i$) Mean: mean-pooling over independently computed
frame features via ResNet-50 [43] as in [18]; ($ii$) Concat: concatenation of
the aforementioned frame features along the temporal dimension as in [19];
($iii$) VT: enforcing spatial-temporal modeling altogether on input video
frame sequences via Video Swin Transformer in VIOLET. The final video
representations are then concatenated with corresponding text embeddings and
fed in Cross-modal Transformer for downstream VidL modeling. We show results
under different settings: random-initialized visual encoder, ImageNet-
pretrained visual encoder, and with additional VidL pre-training on WebVid
[25].

VT consistently outperforms Mean and Concat over the 4 datasets across all
settings. The loss of temporal information in naive mean pooling (Mean) result
in worst performance among the three. Although Concat can preserve the
temporal order of input video frames, it solely relies on Cross-modal
Transformer to model both the temporal dynamics in video and the
correspondence between visual and textual elements, brings unsatisfactory
performance.

When taking a closer look into different pre-training settings, multimodal
pre-training on WebVid significantly boosts the model performance, compared to
unimodal pre-training of visual encoder on ImageNet. In addition, VT benefits
more from VidL pre-training, leading to a bigger performance gap when compared
to Mean or Concat. As the exposure to video data during pre-training utmostly
enhances the learning of temporal dynamics.

Pre-training | TGIF- | TGIF- | MSRVTT- | DiDeMo-
---|---|---|---|---
Task | Action | Transition | Retrieval | Retrieval
None | 81.9 | 88.5 | 13.0 / 36.5 / 49.6 | 18.3 / 46.4 / 56.5
VTM+MLM | 85.4 | 91.6 | 24.4 / 54.4 / 68.1 | 25.8 / 54.2 / 67.0
\+ MCM | 85.0 | 91.6 | 26.0 / 56.0 / 68.4 |  25.8 / 55.9 / 68.1
\+ MFM | 85.5 | 92.0 | 26.2 / 55.5 / 68.4 | 25.4 / 55.5 / 67.8
\+ MPM | 85.0 | 91.8 |  26.6 / 56.2 / 68.4 |  26.0 / 56.5 / 68.0
\+ MVM | 85.8 | 92.1 |  27.0 / 56.5 / 68.8 |  26.1 / 56.9 / 68.9

Table 4: Impact of self-supervised pre-training on video inputs. All pre-
training are conducted on WebVid [25]. Figure 4: MVM accuracy vs. downstream
performance. We adopt only MVM during pre-training (using 0% (w/o MVM), 10%
(8.9% MVM accuracy), 20% (13.4%), 50% (19.2%), and 100% (21.6%) of YT-Temporal
videos).

Effectiveness of MVM. To demonstrate the effectiveness of MVM, we compare
different variants of masked visual modeling when pre-trained on WebVid [25]
in Table 4. First, we establish two baselines: without pre-training (None) and
pre-training with only Visual-Text Matching and Masked Language Modeling
(VTM+MLM) following [18, 19, 25]. Then we augment VTM+MLM with different
variants of masked visual modeling tasks. Masked Classification Modeling (MCM)
mimics MRC in [24] to predict the ImageNet [84] category of the masked image
patch from a pre-trained ResNet-50 [43]; Masked Feature Modeling (MFM) [5]
distills the fixed frame features extracted from a pre-trained visual encoder.
We use the output feature of the masked patches from the last CNN layer of a
ImageNet pre-trained ResNet-50 and adopt linear regression as the training
objective; Masked Patch Modeling (MPM) [5] distinguishes the correct masked
visual patch from negative patches in the same batch with Noise Contrastive
Estimation loss [85], similar to MFM-NCE in [5].

Our results suggest that not all masked visual modeling methods bring
consistent improvement. MCM and MPM give worse results on TGIF-Action over
VTM+MLM; similar trends have been observed in [24, 5]. MFM seems to favor QA
tasks, and MPM benefits more on retrieval tasks. In contrast, MVM leads to the
best performance on all tasks, as it recovers masked patches into a finite
discrete set, making the learning of masked visual modeling easier.

We further investigate the relationship between MVM performance and downstream
performance. We pre-train VIOLET with MVM-only on 10%, 20%, 50%, and 100% of
video scenes from YT-Temporal [19], discarding the corresponding text. As
illustrated in Fig. 4, such MVM pre-training on video inputs only can greatly
lift the performance on all 4 datasets, even without text information.
Moreover, better MVM performance also leads to better downstream performance.
For example, with a 21.6% MVM accuracy on 100% YT-Temporal data, our model
achieves +2.3% improvement on TGIF-Action and +2.5% R@5 increase on MSRVTT. In
summary, results in Table 4 and Fig. 4 suggest that MVM is vital in the
success of VIOLET, as it learns a better video representation to benefit
downstream VidL tasks.

Method | TGIF- | TGIF- | MSRVTT- | DiDeMo-
---|---|---|---|---
Action | Transition | Retrieval | Retrieval
Without Pre-training |
VIOLET | 81.9 | 88.5 | 13.0 / 36.5 / 49.6 | 18.3 / 46.4 / 56.5
Pre-training on COCO+VG |
ClipBERT [18] | 82.8 | 87.8 | 22.0 / 46.8 / 59.9 | 20.4 / 48.0 / 60.8
VIOLET | 84.8 | 90.2 |  23.5 / 50.5 / 63.9 |  22.8 / 51.2 / 62.0
Pre-training on WebVid+CC |
Frozen [25] | - | - | 31.0 / 59.5 / 70.5 | 31.0 / 59.8 / 72.4
VIOLET | 87.1 | 93.6 |  34.2 / 63.5 / 73.6 |  32.9 / 63.0 / 74.5
Pre-training on YTTemporal |
MERLOT [19] | 94.0 | 96.2 | - | -
VIOLET | 91.0 | 94.7 | 25.4 / 54.3 / 64.6 | 26.7 / 56.4 / 64.6

Table 5: Impact of using different pre-training data. We gray out MERLOT due
to its excessive computational cost (e.g., 30K TPU hours vs. 2K GPU hours
(ours) for pre-training and frame resolution 384x704 vs. 224x224 (ours) for
downstream tasks).

Impact of different pre-training data. Table 5 establishes a fair comparison
to the recent SOTA methods, ClipBERT [18], Frozen [25] and MERLOT [19], with
the same pre-training data, respectively. We also compare the effects of
different pre-training data on downstream tasks.

Under fair comparison, our model consistently outperforms ClipBERT and Frozen
by large margins. When both pre-trained on COCO+VG [86, 44], VIOLET surpasses
ClipBERT by $>$+2.0% on Video QA tasks, and $>$+1.5% on R@1 for retrieval
tasks. Frozen adopts a two-stream architecture specifically designed for text-
to-video retrieval applications. VIOLET not only is applicable to video QA
tasks but also achieves a gain of $>$+1.9% on R@1 for retrieval tasks over
Frozen, when both pre-trained on WebVid+CC [25, 74]. On YT-Temporal [19],
VIOLET achieves competitive results with MERLOT on TGIF-Action and TGIF-
Transition with a much lower training cost, as discussed in Sec. 4.2. We
further examine the effect of different pre-training data on downstream tasks
with VIOLET. YT-Temporal is designed to promote video temporal reasoning and
not surprisingly leads to the best QA result. However, the noisy ASR
descriptions lead to smaller gains in retrieval tasks, with a similar
performance to COCO+VG, but much worse than WebVid+CC with a smaller data size
(5.5M vs. 180M). Therefore, we take advantage of both YT-Temporal and
WebVid+CC as our final pre-training corpus, which leads to strong performance
on both video QA and retrieval tasks as presented in Sec. 4.2.

Figure 5: Qualitative examples of self-reconstruction (highlighted with orange
bounding boxes) from predicted visual tokens during our Masked Visual-token
Modeling (MVM).

Qualitative Examples. Fig. 5 illustrates the qualitative examples of self-
reconstruction from predicted visual tokens during MVM, under both Blockwise
Masking (BM) and Attended Masking (AM). As shown, BM masks blocks of video
patches along with consecutive frames and AM masks the most-attended video
patches based on text input (e.g., drawing with “hand” and “cartoon image” in
the $2^{\text{nd}}$ row or “chicken” and “ground” in the $4^{\text{th}}$ row).
VIOLET improves visual reasoning through this video reconstruction during MVM,
and the better video scene understanding further benefits downstream VidL
tasks.

##  5 Conclusion

We present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which
contains Video Swin Transformer to explicitly model the vital video temporal
for video-language learning. We further enhance VIOLET with a new pre-training
task, Masked Visual-token Modeling (MVM), that learns video scene
understanding through a mask-the-predict procedure with self-reconstructable
visual tokens. Experiments on various text-to-video retrieval and video
question answering tasks show that VIOLET achieves SOTA (or competitive)
performance. Comprehensive ablation studies demonstrate the necessity of
temporal video modeling and the effectiveness of MVM over previous MRM/MFM for
video-language reasoning under different pre-training settings.

##  Appendix A Experimental Setup of Downstream Tasks

We evaluate our pre-trained VIOLET on text-to-video retrieval and video
question answering tasks across 12 downstream datasets. For text-to-video
retrieval, we report model performance on MSRVTT [1], DiDeMo [79], YouCook2
[13], and LSMDC [3] and use Recall at K (R@K) as the evaluation metric. For
video question answering, we consider datasets in both multiple-choice and
open-ended settings, including TGIF-Action, TGIF-Transition, TGIF-Frame [6],
MSRVTT-MC, MSRVTT-QA, MSVD-QA [7], LSMDC-MC and LSMDC-FiB [81]. We evaluate
our models using accuracy.

We follow the standard training/validation/testing splits of the original
datasets. If not otherwise stated, we sparsely sample $T$ = 5 video frames and
adopt video frame size 224 with patch size 32. We use AdamW [83] to fine-tune
VIOLET for each downstream task with an initial learning rate of 1.2e-5, betas
of (0.9, 0.98), and weight decay of 1e-3. All finetuning experiments are
conducted on Microsoft Azure [87] with 8 Nvidia V100 GPUs (32GB VRAM).

###  A.1 Text-To-Video Retrieval

For text-to-video retrieval, similar to visual-text matching (VTM) during pre-
training, we treat corresponding video-text pairs as positives and all other
pairwise combinations as negatives. We adopt a fully-connected (FC) layer
(FC${}^{\text{T2V}}$) over the global VidL representation $h^{\text{c}}$ of
the [CLS] token to perform binary classification:

$\begin{split}b_{\text{pos}}&=\text{FC}^{\text{T2V}}(h^{\text{c}}_{\text{pos}}),b_{\text{neg}}=\text{FC}^{\text{T2V}}(h^{\text{c}}_{\text{neg}}),\\\
\mathcal{L}_{\text{T2V}}&=-\mathbb{E}[\log(b_{\text{pos}})+\log(1-b_{\text{neg}})],\end{split}$
(9)

where $h^{\text{c}}_{\text{pos}}$ or $h^{\text{c}}_{\text{neg}}$ is
$h^{\text{c}}$ of positive or negative pairs. In particular, we use pre-
trained FC${}^{\text{VTM}}$ for zero-shot text-to-video retrieval and to
initialize FC${}^{\text{T2V}}$ for further fine-tuning on each downstream
text-to-video retrieval task.

MSRVTT [1] contains 10K YouTube videos with 200K human annotations. For fair
comparison [25, 18], we train on 9K training+validation splits and evaluate on
the 1K-A testing split. We adopt batch size 56 and train for 20 epochs.

DiDeMo [79] consists of 10K videos annotated with 40K sentences from Flickr.
Following [25, 18], we concatenate all sentences from the same video into a
paragraph and perform paragraph-to-video retrieval for DiDeMo. We adopt batch
size 48 and train for 20 epochs.

YouCook2 [13] contains 14K video clips from 2K cooking videos and 89 recipes.
Each clip is annotated with one sentence. We follow [16, 17] to report
retrieval performance on the entire validation clips. We adopt batch size 56
and train for 40 epochs.

LSMDC [3] is built upon 118K video clips from 202 movies. Each clip has a
caption from movie scripts or descriptive video services. Following [25, 16],
we evaluate on 1K testing clips that disjoint from the training+validation
splits. We adopt batch size 56 and train for 40 epochs.

VideoQA | Task | #Option
---|---|---
Multiple- Choice |  TGIF-Action [6] | 5
TGIF-Transition [6] | 5
MSRVTT-MC [7] | 5
LSMDC-MC [3] | 5
Open- Ended |  TGIF-Frame [6] | 1,540
MSRVTT-QA [7] | 1,500
MSVD-QA [88] | 1,000
LSMDC-FiB [81] | 908
Table 6: Summary of video question answering tasks.

###  A.2 Video Question Answering

We test our model on video question answering (QA) tasks in both multiple-
choice and open-ended settings, as summarized in Table 6. For multiple-choice
QA tasks, we concatenate question with each answer option and add a separating
blank token to form the input text (Q+[SEP]+A). We adopt a FC layer upon
$h^{\text{c}}$ to predict the model confidence on each answer option. Cross-
entropy loss is used to train a classifier over all answer options for each
video-question pair. For open-ended QA tasks, we follow the common practice to
convert it to a classification task with a finite set of answer classes. We
build a specific answer vocabulary that can cover most common answers in the
training split of each dataset. Similarly, our model predicts the answer to a
given question over all answer vocabulary through a FC layer upon
$h^{\text{c}}$.

TGIF-Action, TGIF-Transition, and TGIF-Frame [6] require spatial-temporal
reasoning to answer questions regarding GIF videos in TGIF-QA [6]
Specifically, we aim to test our model along three dimensions: ($i$) Action:
to recognize the repeated action; ($ii$) Transition: to identify the
transition between the before and after states; ($iii$) Frame: to answer
questions about a specific frame from the GIF video. Among them, TGIF-Action
and TGIF-Transition are collected under multiple-choice setting, and TGIF-
Frame is an open-ended video QA task with free-form answers. In our
implementation, we select 1,540 most common answers as answer candidates for
TGIF-frame. We adopt batch size 48 and train for 20 epochs.

MSRVTT-MC and MSRVTT-QA [7] are created based on videos and captions in MSRVTT
[1]. MSRVTT-MC is a multiple-choice task with videos as questions, and
captions as answers. Each video contains 5 captions, with only one positive
match. MSRVTT-QA contains 243K open-ended questions over 10K videos. We select
1,500 most common answers as the answer candidates. We adopt batch size 48 and
training epochs 20 for both datasets.

MSVD-QA [7] consists of 47K open-ended questions over 2K videos, based on
video-caption pairs from MSVD [88]. We use 1,000 most common answers as the
answer vocabulary. We adopt batch size 80 and train for 40 epochs.

LSMDC-MC and LSMDC-FiB [81] are built from LSMDC dataset [3]. Similar to
MSRVTT-MC, LSMDC-MC requires the model to select the only positive caption
that describes the video from 5 caption candidates and formulates it as a
multiple-choice QA task. LSMDC-FiB replaces a word in the question sentence
with the [BLANK] token, and the model is to recover the missing word. We
regard LSMDC-FiB as an open-ended Video QA task. In particular, we use a FC
layer over the joint VidL representation $h$ of the [BLANK] token to predict
from 908 answer candidates. We adopt batch size 80 and train for 40 epochs.

Masking | TGIF- | TGIF- | MSRVTT- | DiDeMo-
---|---|---|---|---
Strategy | Action | Transition | Retrieval | Retrieval
Without pre-training
None | 81.9 | 88.5 | 13.0 / 36.5 / 49.6 | 18.3 / 46.4 / 56.5
Pre-train on WebVid [25] with VTM+MLM+MVM
Random | 83.7 | 90.8 | 24.3 / 54.8 / 66.7 | 24.2 / 53.5 / 67.6
BM | 85.4 | 91.8 |  27.0 / 56.2 / 68.6 |  25.8 / 56.8 / 68.8
AM | 85.5 | 91.6 |  26.8 / 56.5 / 68.7 |  26.0 / 56.8 / 68.6
BM+AM | 85.8 | 92.1 |  27.0 / 56.5 / 68.8 |  26.1 / 56.9 / 68.9

Table 7: Impact of masking strategy in MVM and MLM.

##  Appendix B Impact of Masking Strategy

To amplify the effectiveness of our MLM and MVM, we introduce Blockwise
Masking (BM) and Attended Masking (AM) in Sec. 3.3. Table 7 compares different
masking strategies when pre-trained on WebVid [25]. Specifically, we compare 4
masking strategies, random masking, BM only, AM only and BM+AM. Although
improving from the non-pretrained baseline, random masking results in the
least performance improvement on both video QA and retrieval tasks. In
contrast, BM or AM alone brings more significant performance improvements on
both tasks, while seems to benefit different tasks (e.g., 91.8% on TGIF-
Transition and 27.0% R@1 on MSRVTT-Retrieval for BM, and 85.5% on TGIF-Action
and 26.0% R@1 on DideMo-Retrieval for AM). Finally, by leveraging both BM and
AM (BM+AM), we lead to the best performance among the four.

Unlike random masking, BM cuts down the spurious success in MVM evaluation
through neighboring patches that are visually similar to the masked patches,
and AM puts more masking weights on more important video-text elements based
on the attention pattern from our Cross-modal Transformer. These results
demonstrate that both BM and AM contribute to the success of VIOLET.

##  Appendix C Extending VIOLETto Image Question Answering Task

In this section, we show that VIOLET is also extendable to image question
answering task by evaluating it on VCR [89], which requires commonsense
reasoning about the image content. We follow MERLOT [19] to draw colored
highlights around the referenced entity (e.g., [PERSON-1] and [CHAIR-2]) in
the given image and report performance on the multiple-choice Q→A subtask. To
finetune our model, we concatenate the question and each answer choice from
the 4 possible answer candidates. Similarly, a FC layer upon the global cross-
modal representation $h^{\text{c}}$ of the [CLS] token is adopted to predict
the answer and cross-entropy loss is used to supervise the model training. We
adopt batch size 48 and train for 20 epochs.

Method | Frame Size | VCR
---|---|---
MERLOT [19] | 384x704 | 75.1
VIOLET | 224x224 | 74.9
VIOLET | 384x384 | 76.3
Table 8: Comparison with MERLOT [19] under the same pre-training epoch on VCR
[89]. The pre-training are conducted on YT-Temporal [19] for 5 epochs.

The results are shown in Table 8. For a fair comparison, both VIOLET and
MERLOT are pre-trained on YT-Temporal [19] for 5 epochs. Note that MERLOT
adopts a input image resolution of 384x704. With input image size of 224x224,
our VIOLET achieves comparable performance as MERLOT (74.9% vs. 75.1%). When
increasing the input image resolution to 384x384, though still smaller than
the input image size in MERLOT, VIOLET can achieve a superior performance with
an absolute gain of +1.2% over MERLOT.

As mentioned in Sec. 4.2, the full MERLOT pre-training requires excessive
computation power (30K TPU hours), while VIOLET, only pre-trained for 5 epochs
(2K GPU hours), can achieve competitive performance on video-language
downstream tasks with lower input resolution (ours: 224x224, MERLOT:384x704).
When comparing the results from VIOLET with different input resolutions in
Table 8, we observe that higher input resolution results in better downstream
performance. It is also worth noting that longer pre-training leads to
monotonic performance improvement, as shown in [19]. Hence we believe VIOLET
can further improve with higher frame resolution and more pre-training epochs
if computational resources permit.

##  Appendix D Qualitative Examples of Zero-shot Text-to-Video Retrieval

We visualize some qualitative examples of zero-shot text-to-video retrieval in
Fig. 6-9 on MSRVTT [1], DiDeMo [79], LSMDC [3], and YouCook2 [13],
respectively. These examples show that pre-training on large-scale visual-text
data (YT-Temporal [19], WebVid [25], and CC [74]) enables VIOLET to learn
cross-modal alignment to perform text-to-video retrieval in a zero-shot
scenario. For MSRVTT (Fig. 6), “grand theft auto 5” (a video game) is not a
commonly seen phrase, but we can still retrieve relevant video clips depicting
the video game. For paragraph-to-video retrieval in DiDeMo (Fig. 7), the
textual query is a concatenation of multiple sentences, much longer than the
input text during pre-training. Surprisingly, VIOLET can still retrieve videos
that contain relevant semantics mentioned in the textual query. For instance,
the top-2/3/5 of the retrieved videos on the upper left of Fig. 7 correspond
to the textual cues, such as “traveling away, comes back, red signs flaps”,
Moreover, visualizations of zero-shot text-to-video retrieval on LSMDC (Fig.
8) and YouCook2 (Fig. 9) show that VIOLET is generalizable to more specific
video domains, such as movie or cooking videos.

##  Appendix E Limitation and Broader Impact

The broader impact of this paper falls in applications of video-language
(VidL) reasoning, including video question answering and text-to-video
retrieval. Our end-to-end VIdeO-LanguagE Transformer (VIOLET) has the
potential to be applied to various VidL tasks, such as video captioning and
video grounded dialogue, which is worth exploration in future study. In
addition, the newly introduced Masked Visual-token Modeling can further
improve the performance when scaling up the pre-training to even larger-scale
visual-text data. There are also several potential limitations of VIOLET that
would make for promising avenues of future work, including: 1) extending
VIOLET to model the full-length videos with densely sampled frames for
downstream VidL tasks like TGIF-Count [6]; and 2) exploring extra input
signals from videos, such as audio, into the VIOLET framework for better
performance.

We do not anticipate major ethical issues in our work. As a data-driven
system, the self-supervised method is sensitive to the distribution of the
pre-training data. Therefore, we consider diverse types of video, including
VLOGs, instructional videos, short-duration GIFs, and even static images
across YT-Temporal [19], WebVid [25], and CC [74]. The accompanying textual
inputs used to train our model are from various sources, such as alt-text
descriptions, human annotations, and ASR outputs, at different levels,
including temporally-specified and globally-aligned descriptions. We conduct a
comprehensive downstream evaluation over 12 VidL tasks, trying to mitigate the
bias of our learned cross-modal representation for better VidL reasoning.

Figure 6: Qualitative examples of zero-shot text-to-video retrieval on MSRVTT
[1].

Figure 7: Qualitative examples of zero-shot text-to-video retrieval on DiDeMo
[79].

Figure 8: Qualitative examples of zero-shot text-to-video retrieval on LSMDC
[3].

Figure 9: Qualitative examples of zero-shot text-to-video retrieval on
YouCook2 [13].

## References

  * [1] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  MSR-VTT: A Large Video Description Dataset for Bridging Video and Language.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [2] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-Captioning Events in Videos.  In International Conference on Computer Vision (ICCV), 2017.
  * [3] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele.  A Dataset for Movie Description.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [4] Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal.  TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval.  In European Conference on Computer Vision (ECCV), 2020.
  * [5] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
  * [6] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [7] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video Question Answering via Gradually Refined Attention over Appearance and Motion.  In ACM Multimedia (ACMMM), 2017.
  * [8] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg.  TVQA: Localized, Compositional Video Question Answering.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
  * [9] Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal.  TVQA+: Spatio-Temporal Grounding for Video Question Answering.  In Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
  * [10] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing Moments in Video with Natural Language.  In International Conference on Computer Vision (ICCV), 2017.
  * [11] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  TALL: Temporal Activity Localization via Language Query.  In International Conference on Computer Vision (ICCV), 2017.
  * [12] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research.  In International Conference on Computer Vision (ICCV), 2019.
  * [13] Luowei Zhou, Chenliang Xu, and Jason J. Corso.  Towards Automatic Learning of Procedures from Web Instructional Videos.  In AAAI Conference on Artificial Intelligence (AAAI), 2018.
  * [14] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran.  Hierarchical Conditional Relation Networks for Video Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [15] Linchao Zhu and Yi Yang.  ActBERT: Learning Global-Local Video-Text Representations.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [16] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  In International Conference on Computer Vision (ICCV), 2019.
  * [17] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  * [18] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu.  Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [19] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  MERLOT: Multimodal Neural Script Knowledge Models.  In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [20] Peng Wu, Xiangteng He, Mingqian Tang, Yiliang Lv, and Jing Liu.  HANet: Hierarchical Alignment Networks for Video-Text Retrieval.  In ACM Multimedia (ACMMM), 2021.
  * [21] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is Space-Time Attention All You Need for Video Understanding?  In International Conference on Machine Learning (ICML), 2021.
  * [22] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video Swin Transformer.  In arXiv:2106.13230, 2021.
  * [23] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
  * [24] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: UNiversal Image-TExt Representation Learning.  In European Conference on Computer Vision (ECCV), 2020.
  * [25] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval.  In International Conference on Computer Vision (ICCV), 2021.
  * [26] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural Discrete Representation Learning.  In Conference on Neural Information Processing Systems (NeurIPS), 2017.
  * [27] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-Shot Text-to-Image Generation.  In arXiv:2102.12092, 2021.
  * [28] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara Lee Berg, Mohit Bansal, Jingjing Liu, Lijuan Wang, and Zicheng Liu.  VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation.  In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [29] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman.  Use What You Have: Video Retrieval Using Representations From Collaborative Experts.  In British Machine Vision Conference (BMVC), 2020.
  * [30] Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao.  Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering.  In AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * [31] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal Transformer for Video Retrieval.  In European Conference on Computer Vision (ECCV), 2020.
  * [32] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao Henriques, and Andrea Vedaldi.  Support-set bottlenecks for video-text representation learning.  In International Conference for Learning Representations (ICLR), 2021.
  * [33] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia.  Motion-Appearance Co-Memory Networks for Video Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [34] Bowen Zhang, Hexiang Hu, and Fei Sha.  Cross-Modal and Hierarchical Modeling of Video and Text.  In European Conference on Computer Vision (ECCV), 2018.
  * [35] Jie Lei, Tamara L Berg, and Mohit Bansal.  QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries.  In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [36] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang.  Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [37] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The Kinetics Human Action Video Dataset.  In arXiv:1705.06950, 2017.
  * [38] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition.  In European Conference on Computer Vision (ECCV), 2016.
  * [39] Joao Carreira and Andrew Zisserman.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [40] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification.  In European Conference on Computer Vision (ECCV), 2018.
  * [41] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  SlowFast Networks for Video Recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: a Large-Scale Hierarchical Image Database.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
  * [43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [44] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li.  Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations.  In International Journal of Computer Vision (IJCV), 2017.
  * [45] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [46] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  VideoBERT: A Joint Model for Video and Language Representation Learning.  In International Conference on Computer Vision (ICCV), 2019.
  * [47] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, and James Glass.  AVLnet: Learning Audio-Visual Language Representations from Instructional Videos.  In INTERSPEECH, 2021.
  * [48] Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, and Zhongyuan Wang.  HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval.  In arXiv:2103.15049, 2021.
  * [49] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-End Learning of Visual Representations from Uncurated Instructional Videos.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need.  In Conference on Neural Information Processing Systems (NeurIPS), 2017.
  * [51] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  In arXiv:1907.11692, 2019.
  * [52] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In Conference on Neural Information Processing Systems (NeurIPS), 2019.
  * [53] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  In arXiv:1910.10683, 2020.
  * [54] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  In International Conference for Learning Representations (ICLR), 2020.
  * [55] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.  In International Conference for Learning Representations (ICLR), 2020.
  * [56] Wonjae Kim, Bokyung Son, and Ildoo Kim.  ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision.  In International Conference on Machine Learning (ICML), 2021.
  * [57] Karan Desai and Justin Johnson.  VirTex: Learning Visual Representations from Textual Annotations.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [58] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-Task Vision and Language Representation Learning.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [59] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  In International Conference for Learning Representations (ICLR), 2020.
  * [60] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao.  Unified Vision-Language Pre-Training for Image Captioning and VQA.  In AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * [61] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou.  Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training.  In AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * [62] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  VisualBERT: A Simple and Performant Baseline for Vision and Language.  In arXiv:1908.03557, 2019.
  * [63] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-Scale Adversarial Training for Vision-and-Language Representation Learning.  In Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [64] Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Jingjing Liu.  LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval.  In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
  * [65] Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu.  UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [66] Seonhoon Kim, Seohyeong Jeong, Eunbyul Kim, Inho Kang, and Nojun Kwak.  Self-supervised Pre-training and Contrastive Representation Learning for Multiple-choice Video QA.  In AAAI Conference on Artificial Intelligence (AAAI), 2021.
  * [67] Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura.  BERT Representations for Video Question Answering.  In Winter Conference on Applications of Computer Vision (WACV), 2020\.
  * [68] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just Ask: Learning to Answer Questions from Millions of Narrated Videos.  In International Conference on Computer Vision (ICCV), 2021.
  * [69] Jason Tyler Rolfe.  Discrete Variational Autoencoders.  In International Conference for Learning Representations (ICLR), 2017.
  * [70] Hangbo Bao, Li Dong, and Furu Wei.  BEiT: BERT Pre-Training of Image Transformers.  In arXiv:2106.08254, 2021.
  * [71] Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal.  VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning.  In arXiv:2106.11250, 2021.
  * [72] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In International Conference for Learning Representations (ICLR), 2021.
  * [73] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.  In arXiv:1609.08144, 2016.
  * [74] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.  In Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
  * [75] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao.  TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentl.  In International Conference on Computer Vision (ICCV), 2021.
  * [76] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.  In arXiv:2104.08860, 2021.
  * [77] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text.  In arXiv:2104.11178, 2021.
  * [78] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In arXiv:2103.00020, 2021.
  * [79] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing Moments in Video with Natural Language.  In International Conference on Computer Vision (ICCV), 2017.
  * [80] Youngjae Yu, Jongseok Kim, and Gunhee Kim.  A Joint Sequence Fusion Model for Video Question Answering and Retrieval.  In European Conference on Computer Vision (ECCV), 2018.
  * [81] Atousa Torabi, Niket Tandon, and Leonid Sigal.  Learning Language-Visual Embedding for Movie Understanding with Natural-Language.  In arXiv:1609.08124, 2016.
  * [82] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In Conference on Neural Information Processing Systems (NeurIPS), 2019.
  * [83] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In International Conference for Learning Representations (ICLR), 2019.
  * [84] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  In Conference on Neural Information Processing Systems (NeurIPS), 2012.
  * [85] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu.  Exploring the Limits of Language Modeling.  In arXiv:1602.02410, 2016.
  * [86] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick.  Microsoft COCO Captions: Data Collection and Evaluation Server.  In arXiv:1504.00325, 2015.
  * [87] Microsoft Azure.  https://azure.microsoft.com/.
  * [88] David L. Chen and William B. Dolan.  Collecting Highly Parallel Data for Paraphrase Evaluation.  In Annual Meetings of the Association for Computational Linguistics (ACL), 2011.
  * [89] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From Recognition to Cognition: Visual Commonsense Reasoning.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.