#  Lavender: Unifying Video-Language Understanding as Masked Language Modeling

Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, Lijuan
Wang
Microsoft
<EMAIL_ADDRESS>
###### Abstract

Unified vision-language frameworks have greatly advanced in recent years, most
of which adopt an encoder-decoder architecture to unify image-text tasks as
sequence-to-sequence generation. However, existing video-language (VidL)
models still require task-specific designs in model architecture and training
objectives for each task. In this work, we explore a unified VidL framework
Lavender, where Masked Language Modeling [14] (MLM) is used as the common
interface for all pre-training and downstream tasks. Such unification leads to
a simplified model architecture, where only a lightweight MLM head, instead of
a decoder with much more parameters, is needed on top of the multimodal
encoder. Surprisingly, experimental results show that this unified framework
achieves competitive performance on 14 VidL benchmarks, covering video
question answering, text-to-video retrieval and video captioning. Extensive
analyses further demonstrate the advantage of Lavender over existing VidL
methods in: ($i$) supporting all downstream tasks with just a single set of
parameter values when multi-task finetuned; ($ii$) few-shot generalization on
various downstream tasks; and ($iii$) enabling zero-shot evaluation on video
question answering tasks. Code is available at
https://github.com/microsoft/LAVENDER.

Figure 1: Overview of Lavender (LAnguage-VidEo uNDERstanding) model. Lavender
unifies both pre-training and downstream finetuning as Masked Language
Modeling.

##  1 Introduction

Large-scale transformer-based pre-training is now the _de facto_ practice for
NLP and vision-language research [14, 36, 51, 50, 24]. Together with the great
success of image-text pre-training [62, 40, 11, 34], video-language (VidL)
pre-training [61, 87, 32, 28, 84] has also received an increasing amount of
attention. By pre-training an end-to-end multimodal transformer on a large
number of video-text pairs, state-of-the-art performance has been achieved
across a wide range of VidL tasks, including video question answering (QA)
[23, 73], text-to-video retrieval [21, 54], and video captioning [76, 69].
These advances are encouraging; however, on the other hand, all existing VidL
works require designing task-specific heads on top of the transformer encoder
for each pre-training or downstream task. For example, during pre-training,
separate Masked Language Modeling [14] (MLM) and Video Text Matching (VTM)
heads are used, while a new, separately parameterized head needs to be added
for each downstream adaptation. Furthermore, due to the particular nature of
different tasks, they are typically modeled using different training
objectives. For example, multiple-choice video QA is formulated as a
classification problem, while video captioning is inherently a generation
task. A natural but challenging question arises: _can we have a unified
architecture that supports all the popular VidL tasks simultaneously without
introducing task-specific heads?_

To answer this, we present Lavender, a unified VidL framework where all pre-
training and downstream tasks are formulated as simple MLM. As shown in Figure
1, we use two pre-training tasks: MLM and VTM. However, for VTM, instead of
adding a head on top of the output of the commonly used [CLS] token, as used
in all existing works, we propose to append the same [MASK] token that is used
for MLM at the end of the video-text input, and use the same MLM head to
predict whether the input video-text pair matches or not. Note that VTM is
typically formulated as a binary classification problem; here, we simply treat
the output of true or false from VTM as natural language tokens directly
predicted from the whole vocabulary, so that the same set of parameters can be
used for both MLM and VTM.

During downstream adaptation, instead of discarding the MLM head used in pre-
training and adding new heads, which is the standard practice for all previous
VidL works, we use the same MLM head used in pre-training for all downstream
tasks. Specifically,

  * •

For text-to-video retrieval, we train the model in the same way as in the VTM
pre-training task. During inference, for each text query, we concatenate it
with each candidate video, and calculate the corresponding probability of the
[MASK] token being predicted as true, and then rank all candidate videos based
on that score.

  * •

For multiple-choice video QA, we concatenate the question and each answer
candidate sequentially, and add a [MASK] token at the end of the sequence, and
use the same MLM head to predict the answer as “n” (assuming the ground-truth
choice is the n-th answer).

  * •

For open-ended video QA, since most of the ground-truth answers in our tested
datasets only contain one word, we simply append a [MASK] token to the end of
the video-question input, and let the model predict the answer from the whole
vocabulary.

  * •

For video captioning, during training, we mask a certain percentage of the
tokens, and then predict the masked tokens using a seq2seq attention mask [34,
85]. During inference, the full caption is auto-regressively predicted, by
inserting [MASK] tokens one at a time.

Table 1: New state-of-the-art performance with Lavender across 12 video-
language understanding tasks. Accuracy, average(R1, R5, R10) and CIDEr scores
are reported for video QA, retrieval and captioning.

| TGIF | MSRVTT | LSMDC | MSVD | Captioning | Retrieval
---|---|---|---|---|---|---
| Action | Transition | Frame | MC | QA | MC | FiB | QA | MSRVTT | MSVD | DiDeMo | LSMDC
Published SOTA | 94.0 | 96.2 | 69.5 | 90.9 | 43.1 | 81.7 | 52.9 | 46.3 | 60.0 | 120.6 | 65.1 | 41.9
[84] | [84] | [84] | [84] | [84] | [84] | [84] | [78] | [57] | [35] | [6] | [12]
Lavender | 96.6 | 99.1 | 73.5 | 97.4 | 45.0 | 87.0 | 57.1 | 56.6 | 60.1 | 150.7 | 72.4 | 43.3
$\Delta$ |  2.6$\uparrow$ |  2.9$\uparrow$ |  4.0$\uparrow$ |  6.5$\uparrow$ |  1.9$\uparrow$ |  5.3$\uparrow$ |  4.2$\uparrow$ |  10.3$\uparrow$ |  0.1$\uparrow$ |  30.1$\uparrow$ |  7.3$\uparrow$ |  1.4$\uparrow$

Lavender is inspired by VL-T5 [13], UNICORN [80] and OFA [68] that aim to
provide a unified pre-training framework for image-text tasks. However, ours
is very different from theirs, as we use an encoder-only model and an
additional lightweight MLM head on top of it, while a heavy transformer
decoder is needed in [13, 80, 68]. By unifying all the VidL tasks as MLM,
Lavender can seamlessly adapt to different VidL tasks, and enables new
capabilities over existing task-specific methods, such as ($i$) supporting
different VidL tasks with a single set of parameter values when multi-task
finetuned; ($ii$) better generalizability to test data under few-shot
finetuning; and ($iii$) zero-shot inference on video question answering.
Surprisingly, by using this simple generative approach, we outperform
previously published state-of-the-arts on 12 out of 14 downstream tasks (Table
1), even when pre-trained with much fewer data (see Section 4.5 for detailed
comparisons).

##  2 Related Work

Video-Language Pre-training. Branching out from large-scale image-text pre-
training [11, 62], researchers have been leveraging large-scale multimodal
data [10, 27, 58, 46, 84, 3, 83] to build pre-trained video-language (VidL)
models [42, 49, 2, 79, 75, 55, 74] for a wide range of generative [86, 76, 30]
and discriminative [23, 44, 29] tasks. Prominent examples include VideoBERT
[61], HERO [32], ActBERT [87], ClipBERT [28] and MERLOT [84]. Popular pre-
training tasks include Masked Language Modeling (MLM) [60], Video Text
Matching (VTM) [28], frame order modeling [32, 84] and masked visual modeling
[32, 16]. For simplicity, we only use MLM and VTM losses for pre-training, as
in our experiments, we observe that the other pre-training tasks are not that
essential to improving the final performance on downstream tasks (results in
Appendix A).

Although achieving strong performance, existing methods all require task-
specific architectures or objectives for different downstream tasks. For
example, text-to-video retrieval [21, 54] is modeled as binary classification
[28] or via contrastive learning [45, 17]; video question answering [23, 73]
is often formulated as multi-class classification with a set of pre-defined
answer candidates [84, 81]; and video captioning can be tackled via MLM with a
multi-layer perceptron [35] or prefix language modeling with a text decoder
[57].

Unified Frameworks for Multimodal Understanding. There have been attempts in
building an omnipotent model that can simultaneously handle different tasks
with a unified architecture, which can be broadly categorized into two
directions. The first is to insert expert-designed task-specific heads for
each downstream task [20, 59, 22, 33]. These task-specific output layers not
only require expert knowledge, but are also unlikely to generalize to new
tasks. For example, when a new question answering task comes in, a new fully-
connected layer with output dimension of answer vocabulary size is required.
The second direction is to unify the input-output format of different
downstream tasks [70, 13, 80, 68]. With a unified vocabulary, different
downstream tasks (e.g., image question answering [19], image captioning [10]
and visual grounding [26]) can be formulated as the sequence-to-sequence
generation with the shared encoder-decoder architecture [80, 68].

Our work aims to provide a unified framework for VidL understanding, in
contrast to task-specific architectures and objectives used in existing VidL
models (Figure 2(c) vs. Figure 2(a)). Lavender differs from the previous
unified image-text models in that all pre-training and downstream tasks are
unified as MLM, and a simple encoder-only architecture with a lightweight MLM
head is used, instead of sequence-to-sequence modeling as in [13, 70, 68, 80]
that also requires a heavy transformer decoder (Figure 2(c) vs. Figure 2(b)).

(a) Existing VidL Methods

(b) Existing Unified Image-text Models

(c) Lavender

Figure 2: Illustration of the differences between Lavender and existing
methods with image/video question answering (QA) as an example. Unlike task-
specific designs in existing VidL methods, Lavender unifies all tasks as MLM
(Figure 1). We adopt an encoder-only architecture, with a lightweight MLM
head, instead of the heavy decoder in unified image-text models (_e.g._ ,
VL-T5 [13] with task-specific prefix in text input).

##  3 Lavender

###  3.1 Model Architecture

Given a pair of text sentence $\\{{\bm{w}}_{n}\\}|_{n=1}^{N}$ and a video
$\\{{\bm{v}}_{t}\\}|_{t=1}^{T}$, we first encode them separately via unimodal
encoders (i.e., vision encoder and text encoder) to generate unimodal
features. Here, $N$ is the number of tokens in a sentence and $T$ is the
number of frames sampled from the input video. We follow previous works [28,
84] to only sparsely sample a few frames to ease the computational burden. A
multimodal fusion encoder (dubbed as fusion encoder) projects textual features
and visual features into a shared embedding space to learn cross-modal
representations. As Lavender unifies both pre-training and downstream tasks as
Masked Language Modeling (MLM), the same MLM head is used to generate the
final outputs from the cross-modal representations, across different tasks.
Next, we explain each component in detail.

Vision Encoder. Inspired by the success of vision transformers in modeling
spatial details in images [15, 37], different transformer architectures [5,
38, 77] have been proposed to model the long-range temporal modeling in
videos, achieving promising results on action recognition [25]. Recent video-
language works [16, 35] have started to embrace the success of video
transformers, demonstrating stronger performance than encoding each video
frame independently [28]. In our work, we adopt Video Swin Transformer [38]
(VidSwin) as the vision encoder to encode the raw video frame inputs as a
sequence of visual features. Given $T$ input video frames
$\\{{\bm{v}}_{t}\\}|_{t=1}^{T}$ of size $H\times W\times 3$, we first split
each frame into non-overlapping patches of size $h\times w$. VidSwin
additionally enforces temporal downsampling of size 2 as a preprocessing step.
To allow Lavender to have the flexibility of utilizing both video-text and
image-text data for pre-training, we remove this temporal downsampling. As a
result, we can extract a sequence of visual features of size
$T\times\frac{H}{h}\times\frac{W}{w}$ from the last encoder block of VidSwin.
Each feature is of size $8C$ ($C$ is the channel dimension), which is
projected to the same dimensional space as text features via a fully-connected
layer. We follow [16] to add learnable positional embedding layers along both
spatial and temporal dimensions. The resulting visual features are used as
input to the fusion encoder to learn cross-modal representations.

Text Encoder. The input text sentence is first tokenized into the sequence of
word tokens $\\{{\bm{w}}_{n}\\}|_{n=1}^{N}$, following [72]. Two special
tokens [CLS] and [SEP] are inserted at the beginning and the end of the token
sequence. We follow previous works [28, 84, 16] to adopt a lightweight word
embedding layer [14] as the text encoder. The high-dimensional text embeddings
are concatenated with visual features and then fed into the fusion encoder.

Multimodal Fusion Encoder. The fusion encoder is a 12-layer, 768-dimensional
Transformer [65], mirroring the BERT-base architecture [14]. To compute the
cross-modal representations, the unimodal features from vision and text
encoders are fused together via self-attention operations.

###  3.2 Our Unified Framework

Now, we introduce how Lavender can be trained in a unified way.

Video-language Pre-training. We adopt two objectives to pre-train Lavender.
The first is Masked Language Modeling (MLM), which is directly adopted from
language model pre-training [14, 36]. In MLM, we randomly replace 15% of word
tokens with a [MASK] token, a random word, or the same word. The goal is to
reconstruct the correct tokens based on the corresponding hidden
representations from the output of the fusion encoder at the masked position.
A multi-layer perceptron (MLP) with output dimension as vocab_size projects
these hidden representations into the discrete word token space. Cross-entropy
loss is used to supervise the model training. The second is Video Text
Matching, but reformatted as MLM (VTM as MLM). Specifically, we append a
[MASK] token to the textual sentence to mimic the masked textual inputs in
MLM. At each training step, we randomly replace the corresponding text for a
given video with a text description from a different video in the same batch.
At the masked position, Lavender reuses the exact same MLP used in MLM to make
a prediction. Although the ground-truth label is restricted to two tokens
(i.e., true (false) for a positive (negative) video-text pair), but the model
predictions are made across all vocabularies.

Downstream Adaptation. As shown in Figure 1, we can readily apply the pre-
trained Lavender to 4 types of downstream tasks, including text-to-video
retrieval, multiple-choice video question answering, open-ended video question
answering and video captioning. For each task, we transform the text input by
inserting or replacing existing tokens with [MASK] tokens, so that all tasks
can be supervised with cross-entropy loss, and the final predictions are made
based on the word token predicted at the masked position. Here, we explain in
detail how to construct the masked textual inputs and generate model
predictions for each downstream task.

For text-to-video retrieval, similar to VTM during pre-training, we insert a
[MASK] token at the end of the text input. During training, we treat
corresponding video-text pairs as positives (with ground-truth label true) and
all other pairwise combinations constructed by replacing the ground-truth text
with a randomly sampled one as negatives (with ground-truth label false).
During inference, given a textual query, we rank the videos according to the
model confidence of predicting true at the masked position. For multiple-
choice video QA, we concatenate each answer choice (A${}_{\text{n}}$)
sequentially to the question (Q) with a [SEP] token in between. A [MASK] token
is then added at the end, to allow the model to make a prediction of the
correct index for the ground-truth answer choice. For example, for a question
and 5 answer choices, we take Q+[SEP]+A0+[SEP]+...+A4+[MASK] as the text
input. If A${}_{\text{n}}$ is the correct answer, the ground-truth label for
the masked token is n. Through the MLM head, the model makes a prediction at
the masked position over the whole vocabulary. During inference, to ensure a
valid answer, we take the most probable predictions over all answer indices
(e.g., {0,1,2,3,4}). For open-ended video QA, we similarly inject [MASK]
tokens after the question. For simplicity, we only add one [MASK]
token.111Note that we can optionally insert multiple [MASK] tokens to allow
answer predictions with variant lengths. However, as we will show in Appendix
E, over 95% of the questions in the evaluation datasets considered can be
answered with a single word. We then tokenize the ground-truth answers as the
ground-truth label for masked prediction. If the tokenized answer is longer
than 1 word, we simply ignore it during training, and regard it as a wrong
prediction during inference. For video captioning, we use a causal self-
attention mask where the caption token can only attend to the existing output
tokens, which simulates a uni-directional seq2seq generation process,
following [35]. During training, we randomly “mask” some words with [MASK]
token and apply the MLM objective. During inference, the caption is generated
in an auto-regressive manner. At each generation step, the model sees the
entire video input and previously generated tokens, plus a [MASK] token, at
which the model makes a prediction for the current token.

##  4 Experiments

In this section, we first describe our experimental settings (4.1), and show
the superiority of Lavender over comparable task-specific baselines under both
single-task (4.2) and multi-task (4.3) finetuning settings. We then show that
our model can better generalize to testing data under few-shot finetuning and
has strong zero-shot capability on video question answering benchmarks (4.4).
Lastly, we compare Lavender with prior arts and show we outperform published
methods on 12 out of 14 benchmarks, even when pre-trained with much fewer data
(4.5).

###  4.1 Experimental Settings

Pre-training Data. In our default setting, we follow [3] to aggregate video-
text pairs in WebVid2.5M [3] and image-text pairs in CC3M [58] to pre-train
Lavender. As a scale-up recipe, we additionally crawl 11.9M video-text pairs
from the web, following the same procedure in [3]. We similarly scale up
image-text pairs by assembling COCO [10], Visual Genome [27], SBU Captions
[47], CC12M [8] and CC3M. Combining these video-text and image-text datasets
together results in 14M videos + 16M images. Unless otherwise specified, all
results reported in this section as Lavender are pre-trained under the default
setting with 2.5M videos + 3M images. In Section 4.5, we show that scaling up
our pre-training data further improves model performance.

Downstream Tasks. We evaluate Lavender on 14 video-language benchmarks over
popular VidL tasks, including text-to-video retrieval, video question
answering (QA) in both multiple-choice (MC) and open-ended (OE) settings and
video captioning. We briefly list the evaluation datasets for each task type
below; detailed descriptions are included in Appendix E.

  * •

Text-to-video Retrieval: MSRVTT [76], DiDeMo [21], MSVD [9] and LSMDC [54];

  * •

MC Video QA: TGIF-Action, TGIF-Transition [23], MSRVTT-MC [82] and LSMDC-MC
[64];

  * •

OE Video QA: TGIF-Frame [23], MSRVTT-QA, MSVD-QA [73] and LSMDC-FiB [44];

  * •

Video Captioning: MSRVTT [76] and MSVD [9].

Implementation Details. We initialize our Vision Encoder with VideoSwin-Base
[38], pre-trained on Kinetics-600 [25]. Text Encoder and Multimodal Fusion
Encoder are initialized from pre-trained BERT-Base [14]. We train Lavender in
an end-to-end manner for both pre-training and downstream finetuning. Our
implementation of Lavender is based on PyTorch [48]. We adopt AdamW [39] as
the optimizer with an initial learning rate of 2e-5, betas of (0.9, 0.98), and
weight decay of 1e-3 for all pre-training and finetuning experiments. During
pre-training, we sparsely sample $T$=4 video frames and resize them into
$H$=$W$=224 to split into patches with size $h$=$w$=32. For default setting
with 2.5M videos + 3M images, we pre-train Lavender for 10 epochs on 32 NVIDIA
V100 GPUs with a batch size of 28 per GPU, which takes about 2 days. The
scale-up pre-training takes about 10 days on 64 NVIDIA V100 GPUs. For all
downstream tasks, we adopt the same video frame size and patch size, but $T$=5
video frames. More details can be found in Appendix C.

Table 2: Comparison to task-specific baseline under single-task (ST) and multi-task (MT) finetuning (FT), with or without video-langauge pre-training (PT). We report accuracy for TGIF-Action and MSVD-QA, average (R1, R5, R10) for DiDeMo Retrieval (Ret.) and CIDEr score for MSRVTT Captioning (Cap.). Meta-Ave. is the average score across all evaluation datasets. P and H denote the total parameter count in the backbone of Lavender (vision encoder + text encoder + fusion encoder) and top output layer. Note that the baseline with task-specific heads (L3-6) is Lavender-ts, introduced in Section 4.2.  | Task-specific |  |  |  | Meta | TGIF | MSVD | DiDeMo | MSRVTT
---|---|---|---|---|---|---|---|---|---
PT | designs | FT | #Params |  # | Ave. | Action | QA | Ret. | Cap.
- | - | ST | 4(P+H) |  1 | 45.5 | 93.5 | 40.8 | 0.0 | 47.7
MT | P+H |  2 | 58.5 | 95.9 | 47.4 | 41.2 | 50.0
Head | ST | 4(P+H) |  3 | 40.1 | 31.9 | 44.2 | 36.7 | 47.4
MT | P+4H |  4 | 55.6 | 94.1 | 44.6 | 35.4 | 48.3
VTM+MLM | Head | ST | 4(P+H) |  5 | 64.0 | 94.5 | 46.7 | 59.0 | 55.7
MT | P+4H |  6 | 62.4 | 95.5 | 47.7 | 53.0 | 53.3
VTM (as MLM) +MLM | - | ST | 4(P+H) |  7 | 68.9 | 95.8 | 54.4 | 68.2 | 57.3
- | MT | P+H |  8 | 68.3 | 96.5 | 53.5 | 65.8 | 57.4
Task Prompt |  9 | 67.9 | 96.2 | 53.4 | 65.6 | 56.4
Task Token |  10 | 67.9 | 96.5 | 53.6 | 64.9 | 56.7

###  4.2 Comparison to Task-specific Baseline

To make a fair comparison to task-specific methods, we train a task-specific
version of Lavender (denoted as Lavender-ts). We replace the shared Masked
Language Modeling (MLM) head in Lavender with task-specific heads and adopt
task-specific objectives. For text-to-video retrieval (and similarly for video
text matching during pre-training), a multi-layer perceptron (MLP) with output
dimension 1 is applied over the global video-language representation of the
[CLS] token and binary cross-entropy loss is adopted to supervise the model
training. For multiple-choice video question answering (QA), we concatenate
questions with all answer candidates to form the input text, similar to what
was described in Section 3.2, but without the added [MASK] token. A task-
specific MLP with output dimension as the number of answer choices is applied
over the representation of [CLS] token and cross-entropy loss is used to train
a classifier over all answer indices (e.g., $\\{0,1,2,3,4\\}$ with 5 answer
choices). For open-ended video QA, we follow the common practice [84, 16] to
build a finite set of answer vocabularies covering the most common answers in
the training split of each dataset. Similarly, a MLP with output dimension as
the number of answers is added and cross-entropy loss is used to train a
classifier over all answers. For video captioning, we simply adopt the same
training strategy with MLM head as in our unified model.

Table 2 compares Lavender to the task-specific baseline Lavender-ts under
different settings, on four representative benchmarks. For easier comparisons,
we use Meta-Ave, the average across scores over all evaluation tasks, to
measure the average model performance. Here, we focus our discussion on
results under single-task finetuning, and delay the analysis on multi-task
finetuning to Section 4.3. Without video-language (VidL) pre-training, task-
specific baseline with different task heads (L3) outperforms our unified model
(L1) on MSVD-QA and DiDeMo Retrieval. Captioning performance are similar as we
adopt the same MLM head and finetuning strategy for both models. The only
outlier is TGIF-Action, on which we empirically find the training of Lavender-
ts struggles to converge, leading to a low Meta-Ave score.

We also compare the two models under VidL pre-training. We follow [28] to pre-
train task-specific Lavender-ts with MLM and the standard Video Text Matching
(VTM) task, which is modeled as binary classification with an additional MLP
layer. In comparison, the unified model Lavender is pre-trained with MLM and
VTM as MLM, with the shared MLM head (Section 3.2). The unified VidL pre-
training (L7) significantly enhances the performance of Lavender, with a gain
of +23.4 on Meta-Ave over without pre-training (L1). Comparing both models
under VidL pre-training, we also observe that Lavender (L7) outperforms
Lavender-ts (L5) by a notable margin across all 4 tasks (+4.9 on Meta-Ave).

###  4.3 Multi-task Finetuning

In this section, we aim to answer the important question raised in Section 1:
can we have a unified architecture that supports all downstream tasks
simultaneously without introducing task-specific heads? We first compare
Lavender with several multi-task learning baselines with different task-
specific designs in Table 2 and report results under the extreme multi-task
setting - a single model that can support all 14 datasets (all-in-one) in
Table 3.

Comparison to task-specific baseline. We begin our comparison with the most
common multi-task baseline in literature [33, 41] \- adopting a task-specific
head and objective for each task, while sharing the backbone. This is
equivalent to finetuning the task-specific model Lavender-ts under multi-task
setting. We compare the two models in Table 2 and summarize our findings
below:

  * •

Single-task vs. Multi-task: Without video-language (VidL) pre-training (L1-4),
we find that multi-task training greatly improves model performance in both
cases, as it can take advantages of additional and diverse supervision.
Combining multi-task finetuning with VidL pre-training renders slight
performance drop (L5-8).

  * •

Single shared head vs. Task-specific heads: Results show that Lavender (single
shared head) outperforms Lavender-ts (task-specific heads), with (L8 vs. L6,
+5.9) or without (L2 vs. L4, +2.9) VidL pre-training. Lavender also saves more
parameters (approximately 3H) from the additional 3 task-specific heads in
Lavender-ts under multi-task finetuning. In addition, the performance drop of
multi-task finetuning from single-task finetuning is less severe with the
shared MLM head (-0.6 for Lavender vs. -1.6 for Lavender-ts) under VidL pre-
training.

Multi-task Variants with Lavender. We also explore different multi-task
variants with our unified framework Lavender in Table 2: ($i$) L8: the vanilla
version without any task-specific design; ($ii$) L9: with human-readable task-
specific prompts (_e.g._ , “is the video-text paired, true or false?" for
video text matching), which has shown promising results for language
understanding [56]; and ($iii$) L10: with learnable task-specific tokens
(_e.g._ , a special token [VTM] for video text matching), which is in analogy
to the task prefixes in [13]. Different from observations in [56, 13], both
task-specific prompts and tokens do not show a clear advantage over the
vanilla version. We conjecture the differences may be due to the weaker text
encoder and the less diverse text prompts, which we leave as interesting
directions for future study. Based on the above analyses, we simply extend the
vanilla multi-task finetuning method from 4 to all 14 VidL benchmarks
considered.

All-in-One. In Table 3, we finally attempt to answer the question with one
model that can conquer all 14 downstream tasks simultaneously. We first
establish the baseline performance by training single-task (ST) models with
Lavender. We then report multi-task results with ($i$) a single set of
parameters for all tasks (MT (all-in-one)); ($ii$) the best-performing
checkpoint for each task (MT (best)); and ($iii$) with multi-task finetuning
as 2nd stage pre-training and then finetune on each single task
(MT$\rightarrow$ST), from the learned weights with MT (all-in-one). As the
results show, MT$\rightarrow$ST achieves slightly better Meta-Ave across all
settings. Surprisingly, the all-in-one model is very competitive, with only
-0.5 performance drop on Meta-Ave, when compared with ST models.

Table 3: Multi-task Finetuning. Accuracy, average (R1, R5, R10) and CIDEr
score are used as evaluation metrics for video QA, retrieval and captioning
tasks. Meta-Ave. is the average score across all evaluation datasets. P
denotes the total parameter count in Lavender. All results are reported under
VidL pre-training.

Finetune |  | Meta | TGIF | MSRVTT | LSMDC | MSVD | DiDeMo
---|---|---|---|---|---|---|---
Method | # Params | Ave. | Act. | Trans. | Frame | MC | QA | Ret | Cap | MC | FiB | Ret | QA | Ret | Cap | Ret
ST | 14P | 73.9 | 95.8 | 99.1 | 72.2 | 96.6 | 44.2 | 58.9 | 57.3 | 84.5 | 56.9 | 39.8 | 54.4 | 67.6 | 139.4 | 68.2
MT (all-in-one) | P | 73.4 | 95.8 | 98.0 | 70.7 | 93.9 | 44.1 | 56.3 | 57.1 | 85.3 | 56.5 | 39.4 | 53.4 | 69.2 | 141.1 | 66.1
MT (best) | 14P | 73.8 | 95.8 | 98.3 | 71.6 | 94.3 | 44.2 | 56.4 | 57.2 | 86.0 | 56.7 | 39.4 | 55.4 | 69.3 | 141.6 | 66.5
MT $\rightarrow$ ST | 14P | 74.2 | 96.6 | 98.5 | 71.2 | 96.0 | 44.1 | 58.8 | 58.0 | 85.3 | 56.9 | 39.8 | 53.5 | 69.7 | 142.9 | 67.7

(a) TGIF-Action

(b) MSVD-QA

(c) DiDeMo Retrieval

(d) MSRVTT Captioning

Figure 3: Few-shot Evaluation under VidL Pre-training. Each experiment are repeated 5 times with different random seeds. The shaded areas highlight the standard error. Percentage of training data needed to achieve 90% of the full model performance: (a) 40%, (b) 10%, (c) 10%, (d) 6% for Lavender (unified as MLM, red) and (a) 60%, (b) 60%, (c) 25%, (d) 10% for task-specific baseline Lavender-ts (blue). Table 4: Zero-shot Evaluation on Video QA. Top-1 accuracy is reported. We gray out BLIP [31], which is additionally supervised with VQA v2 [19], and Merlot Reserve [83], pre-trained with additional audio modality and uses GPT-3 [7] to reword questions into masked statements. ⋆: unpublished preprints. | # pre-train | TGIF | MSRVTT | LSMDC | MSVD
---|---|---|---|---|---
Method | video/images | Act. | Trans. | Frame | MC | QA | MC | FiB | QA
JustAsk [78] | 69M / - | - | - | - | - | 2.9 | - | - | 7.5
Merlot Reserve [83] |  1B/ - | - | - | - | - | 5.8 | - | 31.0 | -
BLIP⋆ [31] | \- / 129M | - | - | - | - | 19.2 | - | - | 35.2
All-in-one⋆ [67] | 283M / - | - | - | - | 80.3 | - | 56.3 | - | -
Lavender-ts | 2.5M / 3M | 48.5 | 47.9 | 0.0 | 84.6 | 0.0 | 66.9 | 0.0 | 0.0
Lavender | 2.5M / 3M | 52.6 | 54.1 | 16.7 | 86.7 | 4.5 | 73.8 | 34.2 | 11.6
14M / 16M | 55.1 | 53.8 | 19.6 | 87.2 | 2.7 | 73.9 | 36.7 | 9.2

###  4.4 Few-shot and Zero-shot Evaluation

Next, we showcase two new capabilities enabled by Lavender over task-specific
baseline.

Few-Shot Generalizability. We first study how Lavender can generalize to
testing data with limited training examples. Figure 3 presents the results of
the proposed Lavender (red line), which is unified as Masked Language Modeling
(MLM) for all downstream tasks, in comparison to the task-specific baseline
Lavender-ts with different heads and finetuning objectives (blue line) on 4
representative benchmarks. For easier comparison, we further plot two dotted
lines, which denotes 90% of model performance when trained with all the
training data for each model. Lavender has shown a clear advantage, as it can
easily achieve 90% of the full model performance, with much less training
data. Specifically, approximately 40%, 10%, 10%, 6% of training data is needed
for Lavender on TGIF-Action, MSVD-QA, DiDeMo-Retrieval and MSRVTT-Captioning,
while Lavender-ts requires 60%, 60%, 25% and 10%, respectively.

Zero-Shot Evaluation on Video QA. Even without a heavy text decoder as in [13,
80, 68], a pre-trained Lavender can be directly evaluated on video question
answering (QA) tasks without finetuning. Table 4 compares zero-shot (ZS)
performance of Lavender with task-specific baseline Lavender-ts on 8 video QA
benchmarks.

Since the model has neither learned to perform the multiple-choice QA task nor
seen similar data during pre-training, we transform multiple-choice QA as
Video Text Matching (VTM) for better ZS performance. Specifically, we let
Lavender to predict true or false via MLM head, given a video-question-answer
input, and we rank the probability of model prediction as true across all
answer choices. Similarly, for Lavender-ts with binary classification head for
VTM, we simply rank the probability of model prediction as “matched”. With the
same pre-training data, Lavender evidently outperforms Lavender-ts on all
multiple-choice QA benchmarks. On open-ended QA tasks, Lavender can be applied
seamlessly, thanks to the shared MLM head. However, for Lavender-ts, the
randomly initialized task-specific heads give meaningless ZS predictions.

We also compare Lavender against previous methods. Without the help of
additional audio modality in [83] or supervision signals in [31], Lavender
achieves competitive ZS performance, even when pre-trained with much less data
(5.5M vs. >69M). When scaling up the pre-training data by roughly 5 times, we
observe notable performance improvements on most QA benchmarks. The
performance drop on a few datasets may be due to the inclusion of more noisy
data when scaling up.

###  4.5 Comparison to Prior Arts

In this section, we compare Lavender with prior arts, which are mostly
designed to tackle a single type of video-language (VidL) task.

Table 5 summarizes results of Lavender on video question answering (QA) and
video captioning. For video QA, Lavender achieves significant gains over
existing VidL pre-trained models on 7 out 8 video QA benchmarks considered. On
MSRVTT-QA, Lavender is only 1.8 points behind All-in-one [67] pre-trained with
283M videos, which is 9 times more than ours (30M). It is worth mentioning
that VIOLET [16] adopts the same model architecture and finetuning objectives
as our task-specific baseline Lavender-ts. Even when scaling up the VidL pre-
training to 186M videos+images, the task-specific model VIOLET still
underperforms Lavender, which further demonstrates the advantages of our
unified framework. For video captioning, Lavender achieves the new state-of-
the-arts on both datasets. Note that MV-GPT [57] is pre-trained for multi-
modal video captioning, where the auto-transcribed text from audio is used as
additional input. With video-only inputs, Lavender is able to achieve
comparable performance.

Table 6 presents the comparison on text-to-video retrieval. The most
competitive methods [12, 6, 18] on text-to-video retrieval are based on CLIP
[50] pre-trained on 400M images. However, with much fewer pre-training data,
Lavender can still perform competitively on all 4 benchmarks, especially when
compared to non-CLIP pre-trained methods [67, 16, 3]. Notably, on DiDeMo and
LSMDC, Lavender surpasses all baseline methods in Table 6. We hypothesize that
the fusion encoder in Lavender is more effective in modeling interactions
between video and long paragraph query (_i.e._ , DiDeMo) or the contextualized
queries collected from movie scripts (_i.e._ , LSMDC), than the late dot-
product fusion in [12, 6].

Table 5: Comparison with SOTA on video question answering and video captioning. Accuracy and CIDEr scores are reported for QA and captioning evaluation. ⋆: unpublished preprints.  | # Pretrain | TGIF | MSRVTT | LSMDC | MSVD | Captioning
---|---|---|---|---|---|---
Method | videos/images | Act. | Trans. | Frame | MC | QA | MC | FiB | QA | MSRVTT | MSVD
ClipBERT [28] | \- / 200K | 82.8 | 87.8 | 60.3 | 88.2 | 37.4 | - | - | - | - | -
JustAsk [78] | 69M / - | - | - | - | - | 41.5 | - | - | 46.3 | - | -
MERLOT [84] | 180M / - | 94.0 | 96.2 | 69.5 | 90.9 | 43.1 | 81.7 | 52.9 | - | - | -
VIOLET⋆ [16] | 183M / 3M | 92.5 | 95.7 | 68.9 | 91.9 | 43.9 | 82.8 | 53.7 | 47.9 | - | -
All-in-one⋆ [67] | 283M / - | 95.5 | 94.7 | 66.3 | 92.3 | 46.8 | 84.4 | - | 48.3 | - | -
SwinBERT [35] | \- / - | - | - | - | - | - | - | - | - | 53.8 | 120.6
MV-GPT [57] | 53M / - | - | - | - | - | 41.7 | - | - | - | 60.0 | -
Lavender | 2.5M / 3M | 96.6 | 99.1 | 72.2 | 96.6 | 44.2 | 86.0 | 56.9 | 55.4 | 58.0 | 142.9
14M / 16M | 96.3 | 98.7 | 73.5 | 97.4 | 45.0 | 87.0 | 57.1 | 56.6 | 60.1 | 150.7
Table 6: Comparison with SOTA on text-to-video-retrieval tasks. All results
are reported on R1/5/10. The row highlighted in blue assumes the model can see
all textual queries during testing. ⋆: unpublished preprints.

| # Pretrain | Text-to-Video Retrieval
---|---|---
Method | videos/images | MSRVTT | DiDeMo | MSVD | LSMDC
ClipBERT [28] | \- / 200K | 22.0 / 46.8 / 59.9 | 20.4 / 48.0 / 60.8 | - | -
Frozen [3] | 2.5M / 3.2M | 32.5 / 61.5 / 71.2 | 31.0 / 59.8 / 72.4 | 45.6 / 79.8 / 88.2 | 15.0 / 30.8 / 39.8
VIOLET⋆ [16] | 183M / 3M | 34.5 / 63.0 / 73.4 | 32.6 / 62.8 / 74.7 | - | 16.1 / 36.6 / 41.2
All-in-one⋆ [67] | 103M / - | 37.9 / 68.1 / 77.1 | 32.7 / 61.4 / 73.5 | - | -
BridgeFormer [18] | \- / 400M | 44.9 / 71.9 / 80.3 | - |  54.4 / 82.8 / 89.4 | 21.8 / 41.1 / 50.6
QB-Norm [6] | \- / 400M | 47.2 / 73.0 / 83.0 | 43.3 / 71.4 / 80.8 | 47.6 / 77.6 / 86.1 | 22.4 / 40.1 / 49.5
CAMoE [12] | \- / 400M | 47.3 / 74.2 / 84.5 | 43.8 / 71.4 / 79.9 | 49.8 / 79.2 / 87.0 | 25.9 / 46.1 / 53.7
Lavender | 2.5M / 3M | 37.8 / 63.8 / 75.0 | 47.4 / 74.7 / 82.4 | 46.3 / 76.9 / 86.0 | 22.2 / 43.8 / 53.5
14M / 16M | 40.7 / 66.9 / 77.6 |  53.4 / 78.6 / 85.3 | 50.1 / 79.6 / 87.2 |  26.1 / 46.4 / 57.3

##  5 Conclusion and Discussion of Broader Impact

We introduce Lavender, the first unified video-language (VidL) framework, that
can tackle various VidL tasks with a unified Masked Language Modeling
objective. Without any task-specific architectures, Lavender outperforms the
prior state-of-the-art on 12 out of 14 benchmarks considered. Experiments show
that Lavender is better suited for multi-task learning, few-shot
generalization and zero-shot evaluation on video question answering tasks.
There are several potential limitations of Lavender that would make for
promising directions for future work, including: ($i$) extension to fine-
grained VidL tasks (e.g., video corpus moment retrieval [30]); and ($ii$) more
effective in-context few-shot learning or prompt tuning. Like other data-
driven systems, Lavender shares similar risks that may have negative societal
impact, such as biases in training data and energy consumption with large-
scale training. However, we believe that our unified framework combined with
multi-task learning can most likely reduce both memory and energy costs, and
potentially lead to more economical deployment in real-world applications.

## References

  * [1] Microsoft Azure.  https://azure.microsoft.com/.
  * [2] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.  VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text.  In NeurIPS, 2021.
  * [3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval.  In ICCV, 2021.
  * [4] Hangbo Bao, Li Dong, and Furu Wei.  BEiT: BERT Pre-Training of Image Transformers.  In ICLR, 2022.
  * [5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, 2021.
  * [6] Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie.  Cross modal retrieval with querybank normalisation.  arXiv preprint arXiv:2112.12777, 2021.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [8] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts.  In CVPR, 2021.
  * [9] David L. Chen and William B. Dolan.  Collecting Highly Parallel Data for Paraphrase Evaluation.  In ACL, 2011.
  * [10] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick.  Microsoft COCO Captions: Data Collection and Evaluation Server.  In arXiv preprint arXiv:1504.00325, 2015.
  * [11] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  UNITER: UNiversal Image-TExt Representation Learning.  In ECCV, 2020.
  * [12] Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen.  Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss.  In AAAI, 2021.
  * [13] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.  Unifying vision-and-language tasks via text generation.  In ICML, 2021.
  * [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In NAACL, 2019.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In ICLR, 2021.
  * [16] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu.  Violet: End-to-end video-language transformers with masked visual-token modeling.  arXiv preprint arXiv:2111.12681, 2021.
  * [17] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal Transformer for Video Retrieval.  In ECCV, 2020.
  * [18] Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo.  Bridgeformer: Bridging video-text retrieval with multiple choice questions.  In CVPR, 2022.
  * [19] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In CVPR, 2017.
  * [20] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem.  Towards general purpose vision systems.  In CVPR, 2022.
  * [21] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing Moments in Video with Natural Language.  In ICCV, 2017.
  * [22] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In ICCV, 2021.
  * [23] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering.  In CVPR, 2017.
  * [24] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, and Tom Duerig Zhen Li.  Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision.  In ICML, 2021.
  * [25] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The Kinetics Human Action Video Dataset.  In arXiv preprint arXiv:1705.06950, 2017.
  * [26] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg.  Referit game: Referring to objects in photographs of natural scenes.  In EMNLP, 2014.
  * [27] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 2017.
  * [28] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu.  Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling.  In CVPR, 2021.
  * [29] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg.  TVQA: Localized, Compositional Video Question Answering.  In EMNLP, 2018.
  * [30] Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal.  TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval.  In ECCV, 2020.
  * [31] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  arXiv preprint arXiv:2201.12086, 2022.
  * [32] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training.  In EMNLP, 2020.
  * [33] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara Lee Berg, Mohit Bansal, Jingjing Liu, Lijuan Wang, and Zicheng Liu.  VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation.  In NeurIPS, 2021.
  * [34] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks.  In ECCV, 2020.
  * [35] Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Swinbert: End-to-end transformers with sparse attention for video captioning.  In CVPR, 2022.
  * [36] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  In arXiv preprint arXiv:1907.11692, 2019.
  * [37] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  In ICCV, 2021.
  * [38] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video Swin Transformer.  In CVPR, 2022.
  * [39] Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In ICLR, 2019.
  * [40] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.  In NeurIPS, 2019.
  * [41] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-Task Vision and Language Representation Learning.  In CVPR, 2020.
  * [42] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  arXiv preprint arXiv:2002.06353, 2020.
  * [43] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.  In arXiv preprint arXiv:2104.08860, 2021.
  * [44] Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal.  A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering.  In CVPR, 2017.
  * [45] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-End Learning of Visual Representations from Uncurated Instructional Videos.  In CVPR, 2020.
  * [46] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  In ICCV, 2019.
  * [47] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  In NeurIPS, 2011.
  * [48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In NeurIPS, 2019.
  * [49] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao Henriques, and Andrea Vedaldi.  Support-set bottlenecks for video-text representation learning.  In ICLR, 2021.
  * [50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In ICML, 2021.
  * [51] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  In arXiv preprint arXiv:1910.10683, 2020.
  * [52] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-Shot Text-to-Image Generation.  In ICML, 2021.
  * [53] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In KDD, 2020.
  * [54] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele.  A Dataset for Movie Description.  In CVPR, 2015.
  * [55] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, and James Glass.  AVLnet: Learning Audio-Visual Language Representations from Instructional Videos.  In INTERSPEECH, 2021.
  * [56] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al.  Multitask prompted training enables zero-shot task generalization.  arXiv preprint arXiv:2110.08207, 2021.
  * [57] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid.  End-to-end generative pretraining for multimodal video captioning.  In CVPR, 2022.
  * [58] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.  In ACL, 2018.
  * [59] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In CVPR, 2022.
  * [60] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  In ICLR, 2020.
  * [61] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  VideoBERT: A Joint Model for Video and Language Representation Learning.  In ICCV, 2019.
  * [62] Hao Tan and Mohit Bansal.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers.  In EMNLP, 2019.
  * [63] Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal.  VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning.  arXiv:2106.11250, 2021.
  * [64] Atousa Torabi, Niket Tandon, and Leonid Sigal.  Learning Language-Visual Embedding for Movie Understanding with Natural-Language.  In arXiv preprint arXiv:1609.08124, 2016.
  * [65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need.  In NeurIPS, 2017.
  * [66] Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko.  Translating videos to natural language using deep recurrent neural networks.  arXiv preprint arXiv:1412.4729, 2014.
  * [67] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  All in one: Exploring unified video-language pre-training.  arXiv preprint arXiv:2203.07303, 2022.
  * [68] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  arXiv preprint arXiv:2202.03052, 2022.
  * [69] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research.  In ICCV, 2019.
  * [70] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  In ICLR, 2022.
  * [71] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  arXiv preprint arXiv:2112.09133, 2021.
  * [72] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.  In arXiv preprint arXiv:1609.08144, 2016.
  * [73] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video Question Answering via Gradually Refined Attention over Appearance and Motion.  In ACMMM, 2017.
  * [74] Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer.  Vlm: Task-agnostic video-language model pre-training for video understanding.  In Findings of ACL-IJCNLP, 2021.
  * [75] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding.  In EMNLP, 2021.
  * [76] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  MSR-VTT: A Large Video Description Dataset for Bridging Video and Language.  In CVPR, 2016.
  * [77] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid.  Multiview transformers for video recognition.  In CVPR, 2022.
  * [78] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just Ask: Learning to Answer Questions from Millions of Narrated Videos.  In ICCV, 2021.
  * [79] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao.  TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentl.  In ICCV, 2021.
  * [80] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang.  Crossing the format boundary of text and boxes: Towards unified vision-language modeling.  arXiv preprint arXiv:2111.12085, 2021.
  * [81] Weijiang Yu, Haoteng Zheng, Mengfei Li, Lei Ji, Lijun Wu, Nong Xiao, and Nan Duan.  Learning from inside: Self-driven siamese sampling and reasoning for video question answering.  In NeurIPS, 2021.
  * [82] Youngjae Yu, Jongseok Kim, and Gunhee Kim.  A joint sequence fusion model for video question answering and retrieval.  In ECCV, 2018.
  * [83] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi.  Merlot reserve: Neural script knowledge through vision and language and sound.  In CVPR, 2022.
  * [84] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  MERLOT: Multimodal Neural Script Knowledge Models.  In NeurIPS, 2021.
  * [85] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In CVPR, 2021.
  * [86] Luowei Zhou, Chenliang Xu, and Jason J. Corso.  Towards Automatic Learning of Procedures from Web Instructional Videos.  In AAAI, 2018.
  * [87] Linchao Zhu and Yi Yang.  ActBERT: Learning Global-Local Video-Text Representations.  In CVPR, 2020.

##  Appendix A Additional Results

Multi-task Finetuning. For completeness, we include additional results under
multi-task finetuning. Table 7 presents the results of single-task finetuning
and multi-task variants from the scale-up pre-training on 14M videos + 16M
images. For easier reference in future work, we report detailed retrieval
results on R1/5/10 in Table 8.

Table 7: Multi-task Finetuning of Lavender with scale-up VidL Pre-training on
14M videos + 16M images. Accuracy, average(R1, R5, R10) and CIDEr score are
used as evaluation metrics for video QA, retrieval and captioning tasks. Meta-
Ave. is the average score across all evaluation datasets. P denotes the total
parameter count in Lavender (backbone + MLM head).

Finetune |  | Meta | TGIF | MSRVTT | LSMDC | MSVD | DiDeMo
---|---|---|---|---|---|---|---
Method | # Params | Ave. | Act. | Trans. | Frame | MC | QA | Ret | Cap | MC | FiB | Ret | QA | Ret | Cap | Ret
ST | 14P | 76.0 | 94.8 | 98.7 | 73.5 | 97.2 | 45.0 | 61.4 | 59.4 | 85.9 | 57.1 | 41.9 | 55.6 | 72.3 | 150.3 | 72.4
MT (all-in-one) | P | 75.2 | 95.4 | 98.4 | 71.0 | 94.5 | 44.7 | 58.9 | 57.9 | 87.0 | 56.6 | 41.7 | 54.0 | 71.7 | 150.0 | 71.1
MT (best) | 14P | 75.4 | 95.6 | 98.6 | 72.5 | 94.5 | 44.8 | 59.1 | 58.9 | 87.0 | 56.9 | 41.9 | 56.6 | 71.9 | 150.1 | 71.3
MT $\rightarrow$ ST | 14P | 76.2 | 96.3 | 98.6 | 71.8 | 97.4 | 45.0 | 61.7 | 60.1 | 87.0 | 57.1 | 43.3 | 54.3 | 71.8 | 150.7 | 71.5

Table 8:  Detailed results of Lavender on text-to-video retrieval tasks under single-task (ST) finetuning, and different multi-task (MT) finetuning settings. All results are reported on R1/5/10.  # Pretrain | Finetune | Text-to-Video Retrieval
---|---|---
videos/images | Method | MSRVTT | DiDeMo | MSVD | LSMDC
2.5M / 3M | ST |  37.8 / 63.8 / 75.0 |  47.4 / 74.7 / 82.4 | 45.8 / 75.7 / 85.0 |  22.2 / 43.8 / 53.5
MT (all-in-one) | 33.7 / 61.3 / 73.9 | 44.1 / 72.4 / 81.8 | 45.4 / 76.5 / 85.8 | 22.4 / 42.8 / 53.1
MT (best) | 34.9 / 61.1 / 73.2 | 46.3 / 72.2 / 81.1 | 46.2 / 76.2 / 85.5 | 22.4 / 42.8 / 53.1
MT $\rightarrow$ ST |  36.8 / 63.4 / 75.2 |  45.9 / 73.0 / 84.1 |  46.3 / 76.9 / 86.0 |  22.8 / 43.5 / 53.2
14M / 16M | ST |  39.7 / 66.7 / 77.8 |  53.4 / 78.6 / 85.3 |  50.1 / 79.6 / 87.2 | 25.1 / 44.6 / 56.0
|  MT (all-in-one) | 37.6 / 63.4 / 75.6 | 50.3 / 77.8 / 85.2 | 49.4 / 78.7 / 86.8 | 23.4 / 77.8 / 85.2
|  MT (best) | 37.5 / 64.2 / 75.7 | 51.0 / 78.1 / 84.7 | 49.9 / 79.0 / 86.8 | 24.1 / 46.3 / 55.4
|  MT $\rightarrow$ ST |  40.7 / 66.9 / 77.6 | 51.6 / 77.7 / 85.1 | 49.8 / 78.8 / 86.8 |  26.1 / 46.4 / 57.3

Investigation on Other Pre-training Tasks. As mentioned in the main text, we
only adopt Masked Language Modeling (MLM) and Video Text Matching (VTM) as
pre-training tasks for both the proposed Lavender and the task-specific
baseline Lavender-ts. Here we briefly discuss other popular pre-training
objectives with Lavender-ts. The first is Frame Order Modeling [32, 84], where
the input video frames are randomly shuffled and the goal is to revert back
its original order. Different from the video-ASR pairs utilized in these
works, the paired text in our pre-training data is not temporally grounded. In
most cases, the shuffled frame sequence will probably still be globally
aligned with the textual description. Hence, such fine-grained temporal
reasoning objective is not applicable in our case. The second is Masked Visual
Modeling (MVM), where the model learns to reconstruct high-level semantics or
low-level details for a certain percentage of “masked” visual inputs (i.e.,
features or patches). Different variants have been proposed and shown little-
to-none effect in vision-language pre-training, such as predicting the object
category of masked image regions [11] and distilling region/frame features
from well-supervised vision encoders [11, 32]. More recently, by taking
advantage of pre-trained DALL-E [52], researchers [4, 16, 63] have shown
potentials in masked visual token modeling, which asks the model to recover
the discrete latent codes of the masked image patches. [71] explores image
feature descriptors such as Histograms of Oriented Gradients (HOG) as the
prediction target for self-supervised visual pre-training. In Table 9, we
investigate three different MVM objectives on top of VTM + MLM pre-training
for Lavender-ts: ($i$) VQ Token: to recover the discrete codes extracted from
pre-trained DALL-E following [16]; ($ii$) Pixel: to regress the RGB colors as
in [71]; and ($iii$) HOG: to regress the HOG values, following [71]. Results
show that only MVM with HOG achieves a marginal performance improvement of
+0.3 on average. Therefore, we adopt a simple recipe for all other pre-
training experiments in the paper, that is with only MLM and VTM.

Table 9: We investigate different Masked Visual Modeling tasks for pre-training Lavender-ts. Accuracy, average (R1, R5, R10) and CIDEr score are used as evaluation metrics for video QA, retrieval and captioning tasks. Meta-Ave. is the average score across all evaluation datasets. VidL pre-training is conducted on WebVid2.5M [3].  Pre-raining | Meta | TGIF | MSRVTT | LSMDC | MSVD | DiDeMo
---|---|---|---|---|---|---
Tasks | Ave. | Act. | Trans. | Frame | QA | Ret | Cap | FiB | QA | Ret
VTM+MLM | 60.9 | 91.5 | 98.6 | 64.6 | 40.7 | 50.6 | 53.0 | 51.9 | 45.3 | 52.2
\+ VQ Token [16] | 60.8 | 92.4 | 98.6 | 63.9 | 40.3 | 52.1 | 52.5 | 51.1 | 44.5 | 51.6
\+ Pixel [71] | 60.3 | 91.0 | 98.4 | 63.4 | 40.5 | 52.3 | 50.7 | 51.6 | 42.5 | 52.2
\+ HOG [71] | 61.2 | 91.7 | 98.6 | 64.4 | 40.4 | 51.8 | 53.4 | 50.5 | 45.8 | 53.9

(a)

(b)

(c)

Figure 4: Visualization of model predictions from Lavender (ours) and
Lavender-ts (task-specific) on video question answering. Green (Red)
highlights the correct (wrong) predictions.

Qualitative Comparisons to Task-specific Baseline. Figure 4 provides
qualitative comparisons between Lavender and task-specific baseline Lavender-
ts on video question answering (QA). The model predictions are sampled from
MSVD-QA.

In Figure 4(a), the ground-truth answer “fold” is not in the top-$k$ ($k=1000$
for MSVD-QA, following [16]) most common answers in training split, hence
excluded from the pre-defined answer vocabulary for training Lavender-ts. In
Figure 4(b), the ground-truth answer “bowl” appears roughly 9 times more than
“bag" in the training split. These visualization results on video QA suggest
that ($i$) our Lavender can better fit the open-ended setting for QA tasks, as
it does not restrict the predictions to be from a pre-defined answer
vocabulary as in Lavender-ts (Figure 4(a)); and ($ii$) the task-specific
baseline is easier to fail on questions with out-of-distribution answers than
Lavender (Figure 4(b)). Additionally, we show in Figure 4(c) when both models
can provide reasonable answers to the question, which do not exactly match the
ground-truth answer. This result reveals potential problems with the current
evaluation metrics or existing datasets on video QA. Future work may consider
collecting additional annotations to enrich the dataset and improve the
evaluation metric to handle multiple ground-truth answers (e.g., similar to
VQA scores [19]).

##  Appendix B Additional Comparison with Existing Work

(a) Existing VidL Methods

(b) Existing Unified Image-text Models

(c) Lavender

Figure 5: Illustration of the differences between Lavender and existing
methods during pre-training. Lavender unifies both masked language modeling
(MLM) and video text matching (VTM) as MLM, without task-specific heads in
existing video-language (VidL) models. Take VL-T5 [13] as an example; most
unified image-text models are pre-trained with a combination of complex pre-
training tasks (e.g., visual question answering (VQA), visual grounding (VG),
grounded captioning (GC)).

Figure 2 in the main text shows the detailed comparison between Lavender and
existing methods with the image/video question answering task as an example.
In Figure 5, we illustrate the differences among these methods in pre-
training. Unlike existing video-language models, which design task-specific
heads and objectives for different pre-training tasks. Lavender unifies masked
language modeling (MLM) and video text matching (VTM) as MLM. Compared with
unified image-text models (e.g., VL-T5 [13]), which are typically pre-trained
with a combination of complex pre-training tasks, such as visual question
answering and grounded captioning. Although these pre-training tasks may
enable the model with new abilities (e.g., generating region proposals as in
[80, 68]), the supervision often comes from human-annotated data. It remains
unclear how to design and effectively pre-train a unified model with such
capability but without dependency on human-labeling.

##  Appendix C Implementation Details

Task-specific Prompts and Tokens. As mentioned in Section 4.3 of the main
text, we explore the vanilla multi-task finetuning without any task-specific
designs and two additional variants with task-specific prompts and tokens for
Lavender. Here, we describe the prompts and tokens used in these baselines.

For task-specific prompts, we insert a human-readable text prompt at the
beginning of the text input.

  * •

For text-to-video retrieval and video-text matching during pre-training, the
text prompt is “is the video-text paired, true or false”;

  * •

For multiple-choice video question answering (QA), the text prompt is “which
answer choice is correct, choose from 0, 1, 2, 3, 4.”;

  * •

For open-ended video QA, the text prompt is “answer the question about the
video.”;

  * •

For video captioning, the text prompt is “write a description about the
video.”.

As discussed in the Experiments section of the main text, we only briefly
investigate prompt tuning with Lavender. How to design more diverse prompts
for more effective prompt tuning is an interesting direction for future work.

For task-specific tokens, we add several new tokens to the whole vocabulary,
and learn these token embeddings from scratch during multi-task finetuning.
For both training and inference, the task-specific token is inserted right
after [CLS] token in the text input. Specifically, the new task-specific
tokens are [VTM] for text-to-video retrieval and video-text matching, [MC] for
multi-choice video QA, [OE] for open-ended video QA, [CAP] for video
captioning.

Additional Training Details. We summarize the training configurations for
downstream finetuning in Table 10. Due to various data scales and domains, we
use task-specific batch size and training epochs based on the performance of
the validation set for each downstream task (Table LABEL:subtab:task-
specific). All other settings are shared across all datasets (Table
LABEL:subtab:common). All experiments are conducted on Microsoft Azure [1],
adopting mixed-precision training with DeepSpeed [53]. All video data are pre-
processed by evenly extracting 32 frames to avoid expensive decoding on-the-
fly. During training, we randomly sample $T$ frames from 32 frames, resize the
shorter side of all frames to 224 and random crop (224x224) at the same
location for all the frames in a given video. During inference, we evenly
sample $T$ frames from 32 frames and center crop (224x224) for all the frames.

For multi-task finetuning, since the same set of videos are shared among
several downstream tasks, there might be overlaps between one’s training split
and others’ validation or testing split (e.g., some video-text pairs in
MSRVTT-Retrieval 9K-train is in the testing split of MSRVTT-Captioning). To
avoid data contamination, we filter out validation and testing videos in all
downstream datasets from the training splits, and use this cleaned version for
multi-task finetuning. At each training step, we randomly sample one dataset
from all 14 of them, and construct a batch of examples from that dataset. The
training is conducted on 16$\times$80GB A100 for 20 epochs, and we adopt the
same batch size for retrieval tasks as shown in Table LABEL:subtab:task-
specific, and batch size 60 for all other tasks.

Table 10: Training Configurations For Downstream Finetuning.

Learning Rate | 2e-5
---|---
Weight Decay | 1e-3
Optimizer |  AdamW [39]
$\beta$s | (0.9, 0.98)
Warmup Ratio | 10%
# Frames ($T$) | 5
Frame Size ($H$, $W$) | (224, 224)
Patch Size ($h$, $w$) | (32, 32)
(d)

Dataset | # GPUs | Batch Size / GPU | # Epochs
---|---|---|---
Video Question Answering |
TGIF-Action | 8$\times$32GB V100 | 24 | 56
TGIF-Transition | 15
TGIF-Frame | 10
MSRVTT-QA | 8
LSMDC-MC | 10
LSMDC-FiB | 5
MSVD-QA | 8
Text-to-Video Retrieval |
MSRVTT |  16$\times$80GB A100 | 20 | 10
LSMDC | 8$\times$80GB A100 | 5
MSVD | 5
DiDeMo | 16 | 10
Video Captioning |
MSRVTT | 8$\times$32GB V100 | 24 | 20
MSVD
(e)

##  Appendix D Pre-training Data

Public Datasets We use the following publically available datasets to pretrain
Lavender:

  * •

WebVid2.5M [3] scrapes 2.5M video-text pairs from the web. The texts in this
data are alt-text descriptions, which generally describe the global video
semantics.

  * •

Conceptual Captions 3M (CC3M) [58] consists of 3.3M image-text pairs, which
are also harvested from the web. CC12M [8] further enlarges CC3M by 4 times.
Both have been used to pre-train large-scale image-text models.

  * •

SBU-Captions [47] is another widely used dataset for image-text pre-training,
web-crawled from Flickr. It contains 1M image-text pairs.

  * •

COCO [10] and Visual Genome (VG) are two human-annotated image-text datasets.
COCO contains 5 captions per image over 120K images. Unlike COCO captions that
can describe the whole scene, VG collects 5M regional descriptions over more
than 100K images.

Video-Text Data Collection For the scale-up pre-training, we additionally
crawl 11.9M video-text pairs from the web, following the same procedure in
[3]. Here, we briefly describe how we collected the data.

WebVid2.5M has led to promising results in text-to-video retrieval tasks as
shown in [3]. This motivates us to further crawl more video-text pairs from
the same source. We first use a search engine to identify the potential data
sources based on sampled textual descriptions in WebVid2.5M, and then we
scrape the video-text pairs from these data sources. Similarly, we follow [3,
58] to filter out offensive content and hide person and location names. In
total, we have collected 11.9M videos, each accompanied with an alt-text
description. The collected dataset shares similar characteristics as
WebVid2.5M, with the average video duration as $\sim$ 20 seconds, and the
average number of words in the textual description as $\sim$20\. Note that at
the time when we started this project, WebVid10M in [3] has not been released
yet. We later found our 11.9M data largely overlaps with WebVid10M. Hence, we
refer future work to WebVid10M for scale-up pre-training.

##  Appendix E Downstream Datasets

In this section, we introduce all downstream datasets used for evaluating
Lavender and discuss some dataset-specific training details below. Table 11
summarizes the number of examples in training/validation/testing split for
each dataset.

Table 11: Data Distribution of Downstream Datasets.

| TGIF | MSRVTT | LSMDC | MSVD
---|---|---|---|---
| Action | Transition | Frame | QA | MC | FiB | MC | QA
Training | 18K / 18K | 26K / 47K | 30K / 35K | 6.5K / 149K | \- / - | 95K / 297K | 101K / 101K | 1.2K / 30K
Validation | 2K / 2K | 5K / 5K | 4K/ 4K | 0.5K / 123K | \- / - | 7K / 22K | 7K / 7K | 0.2K / 6K
Testing | 2K / 2K | 3K / 6K | 7K/ 14K | 3K / 73K | 3K / 3K | 9.5K / 30K | 10K/ 10K | 0.5K / 13K
# answer choices | 5 | 5 | - | - | 5 | - | 5 | -
(MC-QA)

(f)

| MSRVTT | MSVD | LSMDC | DiDeMo
---|---|---|---|---
| Ret. | Cap. | Ret. | Cap. | Ret | Ret
Training | 9K / 180K | 6.5K / 130K | 1.2K / 49K | 1.2K / 49K | 101K / 101K | 8K / 8K
Validation |  1K / 1K† | 0.5K / 10K | 0.1K / 4K | 0.1K / 4K | 7K / 7K | 1K / 1K
Testing | 1K / 1K | 3K / 60K | 0.7K / 28K | 0.7K / 28K | 1K / 1K | 1K / 1K
(g)

Text-to-video Retrieval We evaluate Lavender on 4 popular text-to-video
retrieval datasets, namely MSRVTT [76], DiDeMo [21], MSVD [9] and LSMDC [54].
MSRVTT contains 10K YouTube videos with 200K descriptions. We follow [3] to
train on 9K videos and evaluate on 1K-A testing split. DiDeMo consists of 10K
Flickr videos, each annotated with 4 sentences. We concatenate all sentences
from the same video into a paragraph and perform paragraph-to-video retrieval,
following [28, 3]. Although this dataset comes with localisation annotations
(ground-truth temporal proposals) for each sentence, we perform all
experiments without leveraging this fine-grained information for both training
and evaluation. Instead, we use the same procedure as described in Appendix C
to sample frames from videos. MSVD is based on 2K YouTube videos and
crowdsourced 40 textual descriptions per video. LSMDC is built upon 118K video
clips from 202 movies. Each clip has a caption from movie scripts or
descriptive video services. We use the standard splits for DiDeMo, MSVD and
LSMDC, following [43]. For paragraph-to-video retrieval on DiDeMo, we adopt
the text augmentation technique proposed in Frozen [3], which is to randomly
sample and concatenate a variable number of sentences as paragraph for each
video.

Multiple-choice Video QA We evaluate Lavender on four multiple-choice QA
datasets: TGIF-Action, TGIF-Transition [23], MSRVTT-MC [73] and LSMDC-MC [44].
Among them, TGIF-Action and TGIF-Transition aim to test the model’s ability to
recognize repeating actions and state transitions in short GIFs. Each video-
question pair is accompanied with 5 answer choices. We concatenate the 5
answer choices sequentially with the question, and the model is asked to
predict the ground-truth answer index. MSRVTT-MC and LSMDC-MC are based on
retrieval tasks, but reformulated as multiple-choice QA. A model needs to find
the caption that describes the video out of 5 candidate captions. Due to its
similarity to video-to-text retrieval, we formulate it as video-text matching,
which is the same as zero-shot evaluation described in the Experiments section
of the main text. Specifically, we let Lavender to predict true or false via
MLM head, given a video-question-answer input, and we rank the probability of
model prediction as true across all answer choices. As there is no training
and validation data constructed in the same way for MSRVTT-MC, we follow [28]
to evaluate the retrieval model trained on MSRVTT to rank the 5 candidate
answers.

Open-ended Video QA Four datasets are considered for open-ended video QA:
TGIF-Frame [23], MSRVTT-QA, MSVD-QA [73] and LSMDC-FiB [44]. Among them, the
question-answer pairs in all but TGIF-Frame are based on the linguistic
transformation of captions for each video. Questions in TGIF-Frame is
collected via crowd-sourcing, which are answerable with just a single frame in
the video. MSRVTT-QA contains 243K open-ended questions over 10K videos and
MSVD-QA consists of 47K questions over 2K videos. The Fil-in-the-blank (FiB)
task of LSMDC-FiB is, given a video and a sentence with a blank in it, to
predict a correct word for the blank. We replace the blank with a [MASK]
token, and naturally it becomes a Masked Language Modeling (MLM) task.

As mentioned in the main text, Lavender answers the open-ended questions in
these datasets with only one word, as there is only one [MASK] token appended
to the text input. Table 12 summarizes the max answer length and the
percentage of examples with answers longer than one word in all four datasets.
As the statistics show, more than 92% of examples in these datasets are
answerable with a single word.

Table 12: Answer Length Distribution for Open-ended Video Question Answering. In summary, there are < 8% of examples across training, validation and testing split of each dataset, with answer length > 1\.  | TGIF-Frame | MSRVTT-QA | LSMDC-FiB | MSVD-QA
---|---|---|---|---
| Train | Val | Test | Train | Val | Test | Train | Val | Test | Train | Val | Test
Max answer_len | 4 | 5 | 5 | 4 | 7 | 6 | 3 | 3 | 3 | 4 | 4 | 4
% of data w/ answer_len > 1 | 2.8 | 2.8 | 2.5 | 2.9 | 4.9 | 4.6 | 2.4 | 2.7 | 2.6 | 6.6 | 7.9 | 7.0

Video Captioning MSRVTT [76] and MSVD [9] are used for captioning evaluation.
As introduced before, MSRVTT consists of 10K videos with 20 captions per
video, and MSVD contains 2K videos, with 40 captions per video. We follow the
standard captioning splits in [76] and [66] for MSRVTT and MSVD, respectively.
During training, we set the probability of random masking caption tokens to be
0.15, the same as what is used in MLM during pre-training. During inference,
we perform generation until the model outputs a [SEP], which is defined as the
sentence ending token or when it reaches the maximum generation step 50.

##  Appendix F Data Usage

We strictly comply with the data usage agreement for each public dataset used
in this paper, which is to only use for non-commercial research purposes. To
the best of our knowledge, there is no personal identification information or
offensive content in any of the public datasets.


# New static spheroidal solution in Jordan-Brands-Dicke theory.

S.M.KOZYREV
Scientific center gravity wave studies ”Dulkyn”,
PB 595, Kazan, 420111, Russia ,Kazan, Russian Federation
Email address<EMAIL_ADDRESS>
###### Abstract

The static spheroidal solutions of Jordan-Brands-Dicke theory (JBD) are
studied. We consider the effect of the anisotropic stresses of scalar field on
the shape of JBD self-graviting objects. It is shown that scalar fields can
have significant effect on the structure and properties of self-graviting
objects. In contrast with general relativity in JBD theory there are nonflat
static spheroidal solutions.

PACS: 04.20.Jb, 04.40.Dg, 95.35.+d

##  1 Introduction

A wide spread assumption in the study of stellar structure is that the shape
of star can be modeled as a spherical symmetry object. This approach has been
used extensively in the study of star, star system and galaxies [1]. However,
in many systems, deviation from spherical symmetry may play an important role
in determining of them properties. Physical situation where unspherical shape
may be relevant are very diverse. Scalar self-graviting objects resulting from
the non minimal coupling scalar fields to gravity are a system where
anisotropic pressure occurs naturally [2]. A model for the Universe where the
dark matter and energy are the scalar nature can be realistic and could
explain most of the observed structures [3]. The self-interaction of the
scalar field could explain the behavior of galaxy rotation curves all along
the background.

Anisotropy appears as an extra assumption on the behavior of scalar fields and
on the shape of equilibrium configuration. Since we still do not have a
formulation of the possible anisotropic stresses is emerging in these or other
contexts, we take the approach of finding several exact solutions representing
physical situations, modelled by ellipsoid of revolution. Our goals hear is to
find exact spheroidal solution, offering an analysis of the change in the
physical properties of the stellar and galaxy models due to presence of non
minimally coupled scalar fields. In this context, particularly interesting is
the case of JBD theory [4], [5] where pressure anisotropies come in action.

##  2 Static spheroidal vacuum solutions of Jordan-Brans-Dicke theory.

JBD theory can be thought of as a minimal extension of general relativity
designed to properly accommodate both Mach’s principle and Dirac’s large
number hypothesis. The progress in the understanding of scalar-tensor theories
of gravity is closely connected with finding and investigation of exact
solutions. Shortly after JBD theory was proposed, Heckmann obtained parametric
form of the exact static vacuum solution to the JBD equations [7]. Later Brans
[5] find the static, spherically symmetric, vacuum solution of the JBD
equations in isotropic coordinates. In the Jordan conformal frame, the JBD
action takes the form [4] (we use geometrized units such that G = c = 1 and we
follow the signature +,-,-,-).

$\displaystyle S$ $\displaystyle=$ $\displaystyle\int dx\sqrt{-g}(\phi
R-\omega g^{\mu\nu}\nabla_{\mu}\phi\nabla_{\nu}\phi)+S_{m}.$ (1)

Here, $\omega$ coupling constant, R is the Ricci scalar curvature with respect
to the space-time metric gμν and Sm denotes action of matter fields. (We use
units in which gravitational constant G=1 and speed of light c=1.)

Variation of (1) with respect to gμν and $\phi$ gives, respectively, the field
equations:

$R_{\mu\nu}-\frac{1}{2}Rg_{\mu\nu}=\frac{1}{\phi\
}T_{\mu\nu}^{M}+T_{\mu\nu}^{JBD},$ (2)

where

$\displaystyle T_{\mu\nu}^{JBD}$ $\displaystyle=$
$\displaystyle[\frac{\omega}{\phi^{2}}\left(\nabla_{\mu}\phi\nabla_{\nu}\phi-\frac{1}{2}g_{\mu\nu}\nabla_{\alpha}\phi\nabla^{\alpha}\phi\right)+$
(3) $\displaystyle\ \ +\frac{1}{\phi}\left(\nabla_{\mu}\nabla_{\nu}\phi-
g_{\mu\nu}\nabla_{\alpha}\nabla^{\alpha}\phi\right)],$

and

$\nabla_{\alpha}\nabla^{\alpha}\phi=\frac{T_{\lambda}^{M\
\lambda}}{3+2\omega},$ (4)

and $T_{\lambda}^{M\ \lambda}$ is the energy momentum tensor of ordinary
matter which obeys the conservation equation $T_{\mu\nu;\lambda}^{M\ }$ gνλ=
0. One can derive the energy density and pressure of JBD field from (3)
$\rho^{JBD}=T_{00}$, $P^{JBD}=T_{ii}$ $(i=1,2,3)$.

As we have already mentioned we consider standard static spheroidal space-
time. Solutions to the equation in spheroidal coordinates have application to
a wide range of problems in physics [6]. The two-dimensional elliptic
coordinate system is defined from the set of all ellipses and all hyperbolas
with a common set of two focal points. Oblate spheroidal coordinates are
derived from elliptic coordinates by rotating the elliptical coordinate system
about the perpendicular bisector of the focal points. Similarly, one can
obtain the prolate spheroidal coordinates by rotating it about the parallel
bisector. We adopt coordinates that allow us to write spheroidal geometry in
prolate form

$\displaystyle
ds^{2}=-B\left(\xi\right)dt^{2}+A\left(\xi\right)\left(c\sqrt{\frac{\xi^{2}-\eta^{2}}{\xi^{2}-1}}d\xi^{2}+c\sqrt{\frac{\xi^{2}-\eta^{2}}{1-\eta^{2}}}d\eta^{2}+c\sqrt{(\xi^{2}-1)(1-\eta^{2})}d\varphi^{2}\right),$
(5)

and in oblate form

$\displaystyle
ds^{2}=-B\left(\xi\right)dt^{2}+A\left(\xi\right)\left(c\sqrt{\frac{\xi^{2}-\eta^{2}}{\xi^{2}-1}}d\xi^{2}+c\sqrt{\frac{\xi^{2}-\eta^{2}}{1-\eta^{2}}}d\eta^{2}+c\xi\eta
d\varphi^{2}\right),$ (6)

where A, B are function of $\xi$ and $\xi\geq 1$, $-1\leq\eta\leq 1$,
$0\leq\varphi\leq 2\pi$. We denote the separation of the two focal points by
$c$.

Then the solutions of the gravitational field equations in the vacuum
$T_{\mu\nu}^{M\ }$= 0 take in oblate case the form

$A=a_{0}\left(\frac{1+\sqrt{1-\xi^{2}}}{-1+\sqrt{1-\xi^{2}}}\right)^{\beta\left(1+\frac{1}{\sqrt{-3-2\omega},}\right)},$
$B=b_{0}\left(\frac{1+\sqrt{1-\xi^{2}}}{-1+\sqrt{1-\xi^{2}}}\right)^{\beta\left(-1+\frac{1}{\sqrt{-3-2\omega},}\right)},\\\
$ (7)
$\phi=\phi_{0}\left(\frac{1+\sqrt{1-\xi^{2}}}{-1+\sqrt{1-\xi^{2}}}\right)^{-\frac{\beta}{\sqrt{-3-2\omega},}},$

where $\beta,a_{0},b_{0},\phi_{0}$ arbitrary constants. For the prolate case

$A=a_{0}\left(\frac{1+\xi}{-1+\xi}\right)^{\beta\left(\frac{1-\sqrt{-3-2\omega}}{1+\sqrt{-3-2\omega}}\right)},$
$B=b_{0}\left(\frac{1+\xi}{-1+\xi}\right)^{\beta},\\\ $ (8)
$\phi=\phi_{0}\left(\frac{1+\xi}{-1+\xi}\right)^{\left(\frac{-\beta}{1+\sqrt{-3-2\omega}}\right)}.$

These solutions can take some possible forms, depending on the values of
arbitrary constants appearing in the solution. Now choose the value
$\omega=-2$ and redefine the arbitrary constant. Then the metric and scalar
function become

$A=1,$
$B=b_{0}\left(\frac{1+\sqrt{1-\xi^{2}}}{-1+\sqrt{1-\xi^{2}}}\right)^{\beta},\\\
$ (9)
$\phi=\phi_{0}\left(\frac{1+\sqrt{1-\xi^{2}}}{-1+\sqrt{1-\xi^{2}}}\right)^{-\frac{\beta}{2,}},$

and

$A=1,$ $B=b_{0}\left(\frac{1+\xi}{-1+\xi}\right)^{\beta},\\\ $ (10)
$\phi=\phi_{0}\left(\frac{1+\xi}{-1+\xi}\right)^{-\frac{\beta}{2}}.$

To see that all these metrics is asymptotically flat it is enough to show that
the metric components behave in an appropriate way at large $\xi$-coordinate
values, e.g., $g_{\mu\nu}=\eta_{\mu\nu}+O(1/\xi)$ as $\xi\rightarrow\infty$.
By inspection of the coefficients, we verify that this is so.

It is certainly true that any vacuum solution of Einstein’s equations is also
a solution of JBD equations (2) with $\phi$ strictly constant, and that
$\phi=const$ is the solution of the equation(4) for $\omega\rightarrow\infty$.
However, this by no means implies that all JBD solutions satisfy Einstein’s
equations in the limit $\omega\rightarrow\infty$ or in the limit
$\phi\rightarrow const$. In fact, it is easy to show that Einstein’s field
equations yield only the flat space $A=1$, $B=1$.

We now deduce the results obtained in the oblate and prolate spheroidal
coordinates. Firstly, in both cases, the JBD scalar field appears to play the
role of dark matter component. Secondly, the directional components of
equation of state of JBD field are anisotropic and in both cases indicate that
the JBD scalar field appears to be an ”exotic” type of dark matter.

##  3 Discussion

In this article we delineated the qualitative features one would expect from
spheroidal scalar field object. It is demonstrated that our model can
successfully predict the spheroidal configuration in terms of a self-
gravitating spacetime solution to the JBD field equations and reproduce the
not spherically-symmetric shape in terms of the non-trivial energy density and
anisotropic pressure of the JBD scalar field which was absent in the context
of general relativity. We believe that following this hypotheses the shape of
galaxy and rotation curve may be explained by action of scalar fields. The
solution presented here could be a first approximation at the galactic space-
time provided the presence of scalar field. Therefore, it is necessary to
study how these results modify the standard method of interpretation rotation
data. Further investigation into the nature solutions with view to separating
the real rotational effects from the scalar fields anisotropy might be
rewarding.

## References

  * [1] M.R.Finch, J.E.F. Skea, A review of Relativistic Static Fluid Sphere, http://www.dft.if.uerj.br/users/JimSkea/papers/pfrev.ps
  * [2] S. Lee, Anisotropy due to Brans-Dicke Theory, arXiv:0811.1643 (2008).
  * [3] Clifford M.Will,Theory and Experiment in Gravitational Physics, Cambridge University Press, Cambridge, England, 1993
  * [4] P. Jordan, Schwerkraft und Weltall, Vieweg (Braunschweig) 1955.
  * [5] C. Brans and R. H. Dicke, Mach’s Principle and a Relativistic Theory of Gravitation, Phys. Rev. 124, 925-935, (1961).
  * [6] C. Flammer. Spheroidal Wave Functions. Stanford University Press, 1957.
  * [7] O. Heckmann, P. Jordan, R.Fricke, Astroph., Zur erweiterten Gravitationstheorie, Z. 28, 113-149, (1951).