#  : Recipes for Training and Evaluating Large
Video Description Models

Jiawei Wang Liping Yuan∗ Yuchen Zhang∗

ByteDance Research
<EMAIL_ADDRESS>ByteDance
Research. All authors contributed equally and are sorted in alphabetical
order.

###### Abstract

Generating fine-grained video descriptions is a fundamental challenge in video
understanding. In this work, we introduce , a family of large-scale video-
language models designed to generate high-quality video descriptions. employs
CLIP-ViT to encode frames separately and then uses an LLM to model temporal
relationships. Despite its simple architecture, we demonstrate that with a
meticulously designed two-stage training procedure, the models exhibit
substantially stronger video description capabilities than any existing open-
source model, showing a $+51.4\%$ advantage in human side-by-side evaluation
over the strongest model. Additionally, they are comparable to state-of-the-
art proprietary models, with a $+12.3\%$ advantage against GPT-4V and a
$-6.7\%$ disadvantage against Gemini 1.5 Pro. Besides video description,
proves to be a versatile generalist model, achieving new state-of-the-art
results across nine public benchmarks, including multi-choice VQA, open-ended
VQA, and zero-shot video captioning. Our second contribution is the
introduction of a new benchmark for evaluating video description models,
consisting of a new challenging dataset featuring videos from diverse sources
and varying complexity, along with an automatic method specifically designed
to assess the quality of fine-grained video descriptions. We make our models
and evaluation benchmark publicly available at
https://github.com/bytedance/tarsier.

##  1 Introduction

Large-scale Video-Language Models (LVLMs) have made significant progress
recently. These models combine visual encoders with large language models
(LLMs) to achieve zero-shot video understanding capabilities. Proprietary
models, such as GPT-4V [51] and Gemini [63, 57], are currently considered
state-of-the-art. Open-source models [35, 4, 32, 26, 81] have achieved high
scores on public video understanding benchmarks, especially in video question
answering [77, 85, 76, 32, 45]. However, they have not yet demonstrated
performance comparable to proprietary models in challenging open-ended
generative tasks.

In this paper, we focus on _fine-grained video description_ , a task that
encapsulates some core challenges in video understanding. In this task, the
model generates a long answer in free-form text to give a detailed coverage of
the video content. The model must be both comprehensive and faithful, meaning
that it must cover all notable events in the video without introducing
hallucinations. The challenge increases when the video includes subtle
movements, such as small but meaningful body actions, or quick motions. The
challenge also increases as the video grows in complexity, capturing multiple
events, subjects, and shots. Additionally, since the task does not have a
unique answer, how to evaluate the model is a non-trivial question.

Empirically, we found that existing open source models lag far behind
proprietary model in fine-grained video description. Our main contribution is
the introduction of models, a family of LVLMs with simple architectures that
generate high-quality video descriptions (see Figure 1 for some examples). We
trained two model sizes: 7B and 34B. Each model consists of a CLIP-ViT encoder
[55], a projection layer and a LLM. We train the model in two stages. The
first stage is multi-task video-to-text pre-training, which enables the model
to understand videos from different perspectives. The second stage is
instruction tuning focused on generating multi-grained video descriptions.
Both stages utilize high-dynamic videos with accurately matched text. Despite
the simple architecture, we demonstrate that superior video description
capability can be achieved by increasing the scale, diversity, and quality of
the data.

Figure 1: Example dialogue between a user and , demonstrating the model’s
ability to generate video descriptions with varying levels of detail. The
model is also able to answer video-related questions in a zero-shot fashion.
(Source movie: _Turning Red, 2022_)

Our second contribution is the introduction of a new benchmark for automatic
evaluation of video description models. Existing video captioning datasets [9,
80, 71] typically pair each video with a single-sentence caption. These
captions are too short to stress-test the full capacity of models. We
introduce a new video description dataset called DREAM-1K, which features
videos of diverse sources and complexity. DREAM-1K contains 1,000 video clips
selected from five categories: live-action movies, animated movies, stock
videos, YouTube videos, and TikTok-style short videos. Typical DREAM-1K videos
contain multiple events, possibly performed by multiple subjects in multiple
shots, all carefully described in detail by human annotators. Additionally, we
propose AutoDQ, an automatic evaluation method specifically designed to assess
the quality of fine-grained video descriptions. AutoDQ breaks down the
evaluation process into two steps: event extraction and entailment. This
approach allows us to compute and compare the precision and recall of
different models in an interpretable way.

Based on DREAM-1K, we demonstrate that has video description capability
comparable to state-of-the-art proprietary models and significantly
outperforms any existing open-source model. Specifically, the -34B model
outperforms all open-source models as well as GPT-4V and Gemini 1.5 Pro in
automatic evaluations. In human side-by-side evaluations, -34B significantly
outperforms the strongest open-source model with a $+52\%$ advantage, is
favored over GPT-4V with a $+12.3\%$ advantage, and slightly falls behind
Gemini 1.5 Pro with a $-6.7\%$ disadvantage. Moreover, proves to be a
versatile generalist model, achieving new state-of-the-art results on public
benchmarks in multi-choice video Q&A [32, 76, 45], open-ended video Q&A [77,
24, 85], and zero-shot video captioning [9, 80, 71]. Finally, we present
extensive ablation studies to gain insights into what makes a good LVLM. Our
findings indicate that several factors contribute to the model’s strong
performance: conducting large-scale multi-task pre-training, scaling up the
LLM, tuning all LLM parameters, and fine-tuning the model with carefully
annotated multi-grained video descriptions.

##  2 Related Work

##### Video-Language Modeling

Video-language models typically consist of a visual encoder and a text
decoder. On the encoder side, existing work either encodes video frames
separately using a ViT [68, 12, 13], or encodes them jointly with
architectures that can model spatial-temporal relationships [11, 78, 22, 53,
72]. For example, VALOR [11] employs a Video Swin Transformer [42], and
Mirasol3B [72] uses TubeViT [52] to model the spatial-temporal relationship,
while InternVideo2 [72] concatenates all the visual tokens and feeds them to a
standard ViT. On the decoder side, standard Transformer [65] is usually used
for autoregressive generation. Typical methods for pre-training the video-
language model include video-text contrastive learning [79], masked video
modeling [64], generative next-token prediction [2, 61, 68], or unifying
multiple training objectives [69, 73, 72, 89]. takes the simpler approach: it
uses a frozen CLIP-ViT [55] to encode individual frames, then rely solely on
the LLM for temporal modeling. The model is trained with the next-token
prediction loss only.

##### Video-LLMs

Recently, there has been a surge of research on how to connect a pre-trained
video encoder to a pre-trained LLM to achieve zero-shot video understanding
capabilities [31, 32, 44, 43, 87, 35, 4, 67, 81]. The two components are
typically connected by either an adapter, such as Q-Former in BLIP2 [30], or a
simple projection layer, such as a linear layer or MLP in LLaVA [38]. adopts
the later architecture. The combined model is usually fine-tuned on a
relatively small instruction tuning set. For example, Video-LLaVA [35] is
fine-tuned on 100K video-text pairs, while VideoChat2 [32] and PLLaVA [81] are
fine-tuned on 788K video-text pairs, including 7K detailed video descriptions.
Another line of research explores training-free approaches that utilize
existing image understanding models or short clip captioners to achieve video
understanding [86, 26, 70, 25]. In contrast, our models are pre-trained with a
larger and more diverse dataset, and then fine-tuned with high-quality, fine-
grained description data.

##### Video Description

Earlier research [68, 78, 22, 11, 13] typically pre-trains a video-language
model and then fine-tunes it on specific video caption datasets, with
performance measured by BLEU, ROUGE [36], METEOR [7], or CIDEr [66]. Although
these fine-tuned models can achieve high scores on specific datasets, they
often do not generalize well to open-world videos. Some recent models report
zero-shot results [2, 83, 87, 89, 72]. However, since the standard video
captioning benchmarks [9, 80, 71] are relatively simple, they do not stress-
test the models’ full capacity. As video complexity grows and descriptions
contain more details, traditional metrics cannot accurately reflect the
discrepancy between the generated descriptions and the ground truth. Maaz et
al.[44] propose using ChatGPT[1] for automatic evaluation, which has been
widely used to evaluate open-ended question answering [35, 26, 67, 81]. We
propose DREAM-1K and AutoDQ as a more challenging video description benchmark
and a more interpretable method for automatic evaluation.

##  3 The model

###  3.1 Model Architecture

Figure 2: model architecture. We use a frozen CLIP-ViT to encode frames, then
project them to the LLM token space through an MLP. The LLM is then trained
with the next-token prediction loss. (Source movie: _Titanic, 1997_)

employs a simple model architecture. As depicted in Figure 2, it encodes each
video frame separately. Initially, each frame is encoded using CLIP-ViT and
then mapped to the downstream LLM’s token embedding space via a multi-layer
perceptron (MLP). The resulting visual tokens are concatenated, separated by a
special [end-of-image] token, allowing to uniformly encode both videos and
images. Then, all tokens are fed into the LLM for auto-regressive generation.
This architecture captures the inter-frame relationship solely through the
LLM, leveraging on its ability to do complex reasoning, which we believe is
essential for understanding the connection between frames. During training,
the CLIP-ViT component and the MLP are frozen, and the LLM is trained.

###  3.2 Training Recipe

We initialize with the weights from the LLaVA-NeXT image models [39] and
employ a two-stage training strategy. Initially, we conduct multi-task pre-
training using large-scale, high-quality data. Subsequently, we perform
instruction tuning on moderate-scale, human annotated data. We describe the
training recipe as follows; for a detailed list of all the training
hyperparameters, please see Appendix A.

####  3.2.1 Multi-task pre-training

Dataset | Size | Task | Dataset | Size | Task
---|---|---|---|---|---
WebVid-10M [6] | 2.8M | Video Captioning |  LSMDC [58] | 108K | Video Captioning
TGIF [33] | 104K | Video Captioning |  ActivityNet [27] | 37K | Video Captioning
Charades [60] | 16K | Video Captioning |  Charades-Ego [59] | 6K | Video Captioning
YouCook2 [90] | 9K | Video Captioning |  TACoS [56] | 18K | Video Captioning
Ego4D [20] | 1.08M | Video Captioning |  Spoken Moments [49] | 492K | Video Captioning
TREC-VTT [5] | 64K | Video Captioning |  |  |
SSV2 [19] | 168K | Action Recognition |  HMDB [28] | 6K | Action Recognition
Kinetics-700 [8] | 536K | Action Recognition |  COIN [62] | 10K | Action Recognition
20BN-jester [46] | 46K | Action Recognition |  Multi-Moments [50] | 1M | Action Recognition
FineAction [41] | 82K | Action Recognition |  RareAct [47] | 2K | Action Recognition
DiDeMo [3] | 41K | Action Localization |  AVA [21] | 27K | Action Localization
CLEVRER [84] | 83K | Video QA |  TGIF-QA [24] | 72K | Video QA
EgoQA [18] | 5K | Video QA |  |  |
Oops! [17] | 14K | Intent Recognition |  VideoInstruct [44] | 100K | Video Instructions
VIST [23] | 38K | Visual Storytelling |  |  |
ShareGPT4V [10] | 94K | Image Captioning |  MS COCO [37] | 566K | Image Captioning
Flicker [54] | 145K | Image Captioning |  LLaVA-1.5 [38] | 665K | Image Instructions
OpenOrca [34] | 1M | Text |  ShareGPT [16] | 88K | Text
Table 1: Public datasets used in pre-training.

In the pre-training stage, we trained our model across diverse tasks such as
video captioning, video question answering, action recognition, multi-image
understanding, and text generation, utilizing a broad array of public datasets
as detailed in Table 1. To unify the training process, all tasks were
converted into text generation tasks. For each dataset, we crafted specific
prompts to define the output format. For instance, video captioning tasks may
use prompts like _Describe the video in one sentence_ or _Describe the video
in detail_ , tailored to the length of the annotated description; Video QA
tasks may use prompts like _Give a short answer to the following question_.
Once prompts were assigned to each dataset, we leveraged ChatGPT [1] to
enhance prompt diversity. The model was subsequently trained to produce the
annotated response based on the video/image input and the associated prompt.
For datasets that have train/val/test splits, we use the training splits only.

Note that larger-scale video-text datasets such as HowTo100M [48], HD-VILA
[82], and Panda-70M [14] exist. We opted not to use the video-text pairs in
these datasets due to issues such as narrow domains or low-quality texts. We
found that human-annotated descriptions of high-dynamic videos are the most
helpful for training our models. Notably, the WebVid-10M dataset [6] includes
many instances where the video content can be described by just looking at a
single frame, which are less valuable for video understanding. To address
this, we used instance re-sampling to prioritize videos involving human
actions and animal movements, as these are more likely to feature dynamic
events. This resulted in 2.8M video-text pairs from WebVid-10M being used in
pre-training.

In addition to public datasets, we also added 3.5M in-house data to pre-
training, including 2.4M high-quality video caption data similar to WebVid and
1.1M videos with object-tracking annotations. The later annotations are
automatically generated for a subset of videos in WebVid-10M [6] and HD-VILA
[82]. We use DEVA [15] to generate object-tracking bounding boxes for these
videos at high frame rates, then train the model to predict the bounding box
coordinates of every object in every frame.

Overall, our pre-training dataset comprises 13.6M video-text pairs. We trained
the model on 48 A100 GPUs for 100K steps using a batch size of 288. Throughout
the training process, we randomly sampled between 8 to 32 frames per video. To
sample $n$ frames, we first partitioned the video evenly into $n$ segments,
then randomly sampled one frame from each segment. Each sampled frame was
resized to 336x336 before being sent to the CLIP-ViT encoder.

####  3.2.2 Instruction Tuning

In the instruction tuning stage, we use 500K of in-house instruction tuning
data. The dataset contains a wide spectrum of video complexities and diverse
prompts. In addition to the tasks from pre-training, we added the following
new tasks:

  * •

Describing Complex Videos at Different Levels of Granularity: We collected a
set of 100K movie clips featuring multiple shots, subjects, or events, and had
annotators provide descriptions varying in length and detail, from brief
motion summaries to comprehensive narratives of visual details. During
training, we used different prompts to precisely specify the requirements for
each level of granularity.

  * •

Describing Camera Motions: We found that precisely describing camera motion is
a weak point of existing models. To address this, we created a dataset rich in
camera motions, including zooming, translating, panning, and rotating. Then we
ask annotators to accurately describe these motions.

  * •

Creative Writing: The model was also trained to generate creative outputs
based on videos, such as imagining characters’ dialogues, drafting
motivational speeches, and writing poems. We used self-instruct [74] to create
diverse prompts and relied on GPT-4V [51] to generate responses based on
human-annotated video descriptions.

By incorporating these tasks, we further enhanced the model’s capabilities in
multi-grained video description. We fine-tuned the model for 10K steps on the
instruction tuning set using 16 A100 GPUs and a batch size of 96. During
training, we sampled 16 frames from each of the 100K complex video clips
mentioned above, while sampling 8 frames from other videos. This
differentiation is necessary because movies often contain rapid actions and
movements that can only be fully captured with a higher frame rate. The first
and last frames of the video were always selected, with the intermediary
frames evenly sampled throughout the video.

##  4 Evaluation of Video Description Models

In this section, we present DREAM-1K, a new benchmark and AutoDQ, an automatic
evaluation method to comprehensive assess the quality of video description
models.

###  4.1 DREAM-1K: A challenging video description benchmark

Generating high-quality video descriptions requires an understanding of
multiple frames together. This understanding demands two primary capabilities.
Firstly, the model should be capable of comparing frames at the pixel level to
detect subtle motions, such as a person speaking or the camera panning.
Secondly, it must link frames on a semantic level to reason about higher-level
events, like the characters’ intentions and relationships. Current video
captioning benchmarks, such as MSR-VTT [80] and VATEX [71], are relatively
simplistic. Typically, each video in these benchmarks involves a single major
action performed by one subject in a single shot. Additionally, the _single
frame bias_ [29] indicates that many actions in these datasets can be
identified from still images, without the need for considering temporal
relations.

| Live-action | Animation | YouTube | Shorts | Stock | Total
---|---|---|---|---|---|---
Number of videos | 200 | 200 | 200 | 200 | 200 | 1,000
Avg. video length (second) | 7.3 | 6.1 | 7.8 | 9.5 | 13.7 | 8.9
Avg. text length (word) | 47.5 | 56.7 | 81.0 | 67.2 | 43.9 | 59.3
Avg. number of events | 5.5 | 6.5 | 6.6 | 7.5 | 5.4 | 6.3
Avg. number of subjects | 2.4 | 2.6 | 2.2 | 1.9 | 1.8 | 2.2
Avg. number of shots | 2.4 | 2.1 | 2.2 | 1.5 | 1.0 | 1.9
Table 2: DREAM-1K data statistics. Figure 3: This video features six actions,
each highlighted in a unique color. The descriptions capture more actions and
contains fewer hallucinations (indicated by underlining and red color)
compared to existing models.

To quantitatively assess the true capabilities of models, we introduce
DREAM-1K (Description with Rich Events, Actions, and Motions), a collection of
1,000 annotated video clips with diverse complexities. We sourced these videos
from five different origins: live-action movies, animated movies, stock
videos, long YouTube videos, and TikTok-style short videos, with each category
contributing 200 clips. We manually pick all videos from Internet to ensure
that the entire dataset is disjoint from public video-language training data.
Each clip in DREAM-1K includes at least one dynamic event that cannot be
accurately identified from a single frame alone. Table 2 shows the statistics
for the dataset. On average, each video lasts 8.9 seconds, includes 6.3
events, 2.2 subjects (people, animals, etc.) and 1.9 shots. We provide a fine-
grained manual annotation for each video, covering all events, actions, and
motions, with an average length of 59.3 words.

Figure 4 illustrates an example video from DREAM-1K. The video is 14-second
long and features six actions. From this example, a noticeable performance gap
emerges between models. The descriptions generated by existing models are less
comprehensive, often missing key events, or less accurate, introducing
hallucinations. In contrast, the descriptions generated by our model capture
more events and exhibit far fewer hallucinations.

###  4.2 AutoDQ: Automatic evaluation of description quality

Figure 4: The AutoDQ workflow. AutoDQ uses an extraction model to extract
events from two video descriptions, then uses an entailment model to examine
how many events extracted from one description are entailed by the other
description. We use ChatGPT to implement both models. (Source movie:
_Fireheart, 2022_)

Traditional $n$-gram based metrics, such as CIDEr [66], fall short when
evaluating long video descriptions. This is primarily because there are
numerous ways to produce a high-quality description, many of which have
minimal $n$-gram overlap with the reference description. Additionally,
manually assigning quality scores to descriptions is both costly and
subjective. To address this, Maaz et al. [44] propose using ChatGPT to rate
descriptions on a scale from 1 to 5. However, the meaning of each rating is
ambiguous, and the ratings themselves are not calibrated.

We propose AutoDQ (Automatic Description Quality) as a more interpretable
approach to automatic evaluation. Given a reference description $D_{\rm ref}$
and a model-generated description $D_{\rm model}$, our method compares them
through a two-step procedure. In the first step, we use an _event extraction
model_ to extract a sequence of events from both descriptions. In the second
step, we employ a _natural language inference model_ to compute two ratios:
the ratio of events in $D_{\rm ref}$ that are entailed by $D_{\rm model}$, and
the ratio of events in $D_{\rm model}$ that are entailed by $D_{\rm ref}$. The
first ratio is defined as the recall of $D_{\rm model}$, while the second
ratio is defined as its precision. See Figure 4 for a concrete example. Since
both event extraction and natural language inference are well-studied NLP
tasks, we can rely on ChatGPT to perform them effectively. For the specific
prompts used in these tasks, please refer to Appendix B. Finally, by
aggregating these results over all examples, we obtain the precision and
recall of the model across the entire dataset, from which we can compute its
F1 score.

##  5 Experiments

In this section, we compare the models with state-of-the-art open-source and
proprietary models on a variety of video understanding tasks.

###  5.1 Results on DREAM-1K

In this test, we generate detailed descriptions for each video in DREAM-1K. By
default, for Tarsier and GPT-4V, we uniformly sample 8 frames per video. For
Gemini 1.5 Pro, it dynamically samples frames at 1 FPS. For other open-source
models, we sample frames following their official settings: 8 frames for
Video-LLaVA; 90 frames for MiniGPT-4V; 32 frames for LLaVA-NeXT-Video; 16
frames for VideoChat2 and PLLaVA.

The default prompt is “Describe the video in detail.” For GPT-4V and Gemini,
we found that more specific instructions yield better descriptions, leading us
to use a different prompt for these models. For other models, we use the
default prompt unless the official implementation recommends a specific one
[81, 88]. See Appendix C for the specific prompts that we have used.

####  5.1.1 Automatic evaluation

Model | Live-action | Animation | YouTube | Shorts | Stock | Overall
---|---|---|---|---|---|---
Video-LLaVA [35] | 19.4/24.3/16.2 | 15.3/21.2/11.9 | 21.2/31.9/15.8 | 18.5/29.4/13.5 | 27.0/33.5/22.7 | 20.4/28.1/16.0
MiniGPT-4V† [4] | 23.6/23.8/23.4 | 16.4/19.0/14.5 | 24.8/28.4/22.1 | 23.7/29.1/20.0 | 30.7/30.3/31.1 | 24.0/26.1/22.2
LLaVA-NeXT-Video⋆ [88] | 25.8/33.4/21.0 | 22.0/32.0/16.7 | 26.5/36.2/20.9 | 24.2/39.8/17.4 | 31.1/37.3/26.7 | 26.1/35.7/20.5
VideoChat2‡ [32] | 26.0/28.1/24.3 | 18.8/23.9/15.5 | 27.3/32.7/23.4 | 26.5/36.8/20.7 | 33.2/33.6/32.7 | 26.6/31.0/23.3
PLLaVA-34B [81] | 29.3/34.9/25.2 | 20.9/32.0/15.6 | 28.9/40.8/22.3 | 25.6/41.9/18.4 | 35.1/42.5/29.9 | 28.2/38.4/22.3
GPT-4V [51] | 34.8/39.2/31.3 | 27.4/31.9/24.0 | 33.8/40.1/29.2 | 34.8/46.1/28.0 | 40.7/46.7/36.1 | 34.4/40.8/29.7
Gemini 1.5 Pro [57] | 36.4/36.4/36.4 | 30.7/31.8/29.7 | 34.0/36.7/31.6 |  37.0/42.4/32.7 |  42.2/40.7/43.8 | 36.2/37.6/34.8
-7B | 36.6/38.5/34.8 | 29.3/34.6/25.5 | 33.0/39.2/28.4 | 33.6/44.6/26.9 | 39.6/44.7/35.5 | 34.6/40.3/30.2
-34B |  38.5/39.6/37.5 |  32.2/35.8/29.2 |  34.5/41.1/29.7 | 34.0/44.1/27.7 | 41.7/46.4/37.8 |  36.3/41.4/32.4

Table 3: DREAM-1K evaluation by AutoDQ. We report F1/Precision/Recall scores
for each category and for the overall dataset. †: MiniGPT-4V-Mistral. ‡:
VideoChat2-Mistral. $\star$: LLaVA-NeXT-Video-34B-DPO. The models achieve
significantly higher scores than all open source models in all categories.

Figure 5: Description quality (F1 score) as a function of increasing video
complexity.

First, we report the automatic evaluation results computed by the method in
Section 4.2. The ChatGPT model used to implement AutoDQ is gpt-3.5-turbo-0125.
As shown in Table 3, the models outperform all open-source models in terms of
precision, recall, and F1 scores across all categories. Among the existing
open-source models, the PLLaVA-34B model exhibits the best overall description
quality, yet there is a large gap between its scores and those of models. Both
the -7B and -34B models achieve higher F1 scores than GPT-4V. The 34B model
also outperforms Gemini 1.5 Pro in the overall score, though the 7B model is
slightly weaker than Gemini. This is a promising result, especially
considering that our training procedure utilizes significantly less compute
resources than these proprietary models.

Next, we partitioned DREAM-1K according to the number of events, subjects, and
shots to test the models against increasing video complexities. As shown in
Figure 5, while F1 scores generally decline for all models as video complexity
increases, Tarsier-34B consistently outperforms other open-source models and
GPT4-V across all levels of complexity, while achieving comparable scores
consistently with Gemini 1.5 Pro.

####  5.1.2 Human evaluation

Comparison |  wins | Same | Baseline wins | Advantage
---|---|---|---|---
-34B vs PLLaVA-34B | 71.7% | 8.0% | 20.3% | $+51.4\%$
-34B vs GPT-4V | 50.0% | 12.3% | 37.7% | $+12.3\%$
-34B vs Gemini 1.5 Pro | 28.0% | 37.3% | 34.7% | $-6.7\%$
Table 4: Side-by-side human evaluation results on DREAM-1K. The advantage rate
is defined as the win rate subtracted by the loss rate.

Human side-by-side comparison is a gold standard for evaluating video
descriptions. In this test, we randomly sampled 300 videos from DREAM-1K, then
asked experienced annotators to compare the descriptions generated by two
different models, collecting their preferences. Every pair of descriptions was
randomly shuffled to ensure that the annotators were blind to the
corresponding models. We compared the -34B model against PLLaVA-34B, GPT4-V,
and Gemini 1.5 Pro. The results, as shown in Table 4, align with our automatic
evaluation findings. Our model significantly outperforms PLLaVA, demonstrating
strong capabilities in describing dynamic events with less hallucination. It
is also preferred over GPT-4V with a notable margin. When compared to Gemini
1.5 Pro, has a slightly negative advantage rate, but in a significant
percentage of cases (37.3%), the two models are labeled as “Same.” This
indicates that their performance is comparable in many instances. See Appendix
D for more information about the human evaluation process.

###  5.2 Results on other public benchmarks

Model | MVBench | Next-QA | EgoSchema
---|---|---|---
ST-LLM [40] | 54.9 | - | -
PLLaVA [81] | 58.1 | - | -
VideoChat2 [32] | 60.4 | - | 63.6/54.4
InternVideo2 [72] | 60.9 | - | -
IG-VLM [26] | - | 70.9 | -
LLoVi [86] | - | 67.7 | 57.6/50.3
VideoAgent [70] | - | 71.3 | 60.2/54.1
LangRepo [25] | - | 60.9 | 66.2/41.2
VideoTree [75] | - | 73.5 | 66.2/61.1
-7B | 62.6 | 71.6 | 56.0/49.9
-34B | 67.6 | 79.2 |  68.6/61.7
(a) Accuracy of zero-shot multi-choice video question answering. On EgoSchema,
we report the results on subset/fullset respectively.

Model | MSVD | MSR-VTT | VATEX
---|---|---|---
BLIP2 [30] | 27.4 | 19.0 | -
Panda [14] | 49.2 | 31.5 | -
VideoCoca [83] | - | 27.1 | 22.8
Video-LLaMA [87] | 47.0 | 29.1 | -
Flamingo [2] | - | - | 39.5
VideoPrism [89] | - | 38.5 | 31.7
InternVideo2 [72] | 93.1 | 43.5 | 49.2
-7B | 98.0 | 24.1 | 51.1
-34B | 125.9 | 44.7 | 57.1
(b) The CIDEr scores ($\uparrow$) of zero-shot video captioning.

Model | MSVD-QA | MSR-VTT-QA | ActivityNet-QA | TGIF-QA
---|---|---|---|---
Video-LLaVA [35] | 70.7/3.9 | 59.2/3.5 | 45.3/3.3 | 70.0/4.0
MiniGPT-4V [4] | 73.9/4.1 | 58.3/3.5 | 44.3/3.4 | 72.2/4.1
VideoChat2 [32] | 70.0/3.9 | 54.1/3.3 | 49.1/3.3 | -
ST-LLM [40] | 74.6/3.9 | 63.2/3.4 | 50.9/3.3 | -
IG-VLM LLaVA-34B [26] | 79.6/4.1 | 62.4/3.5 | 58.4/3.5 | 79.1/4.2
IG-VLM GPT-4V [26] | 76.3/4.0 | 63.8/3.5 | 57.0/3.5 | 65.3/3.7
Elysium [67] | 75.8/3.7 | 67.5/3.2 | 43.4/2.9 | 66.6/3.6
PLLaVA-7B [81] | 76.6/4.1 | 62.0/3.5 | 56.3/3.5 | 77.5/4.1
PLLaVA-34B [81] | 79.9/4.2 | 68.7/3.8 | 60.9/3.7 | 80.6/4.3
-7B | 77.0/4.1 | 62.0/3.5 | 59.5/3.6 | 79.2/4.2
-34B | 80.3/4.2 | 66.4/3.7 | 61.6/3.7 | 82.5/4.4
(c) Accuracy and quality score of zero-shot open-ended video question
answering. We employ ChatGPT to evaluate the scores.

Table 5: Zero-shot results on public benchmarks. When citing a baseline model
that has multiple variants (e.g., different sizes), we always report the
results of the best-performing variant.

We evaluated models on popular video understanding benchmarks. As is a
generalist model, we report its zero-shot performance, meaning that the
training sets of these datasets (if they exist) were not used in any stage of
training. The results for existing models were sourced directly from their
respective papers. models sampled 8 frames per video, except for the long-form
video understanding benchmark, EgoSchema, where we sampled 16 frames per
video.

####  5.2.1 Multi-choice video question answering

In multi-choice video question answering, the model is given a video, a
question, and several answer choices, and is asked to select the correct
answer. We compare models on MVBench [32], Next-QA [76], and EgoSchema [45].
MVBench is a comprehensive video understanding benchmark covering 20 tasks;
Next-QA is a benchmark for causal and temporal action reasoning; EgoSchema is
a benchmark for evaluating long video understanding capabilities. To prevent
data leakage, we have removed all EgoSchema-related videos from Ego4D [20]
during training.

Results presented in Table 5(a) demonstrate that the models have established
new state-of-the-art performance across three benchmarks. We observe a
consistent increase in accuracy as the size of the LLM increases, which we
attribute to the enhanced reasoning capabilities of larger LLMs.

####  5.2.2 Open-ended video question answering

In open-choice video question answering, the model is provided with a video
and a question and is asked to generate a free-form answer. We report the
results on four standard benchmarks: MSVD-QA [77], MSR-VTT-QA [77],
ActivityNet-QA [85], and TGIF-QA [24]. Following the approach of Video-ChatGPT
[44], we utilize ChatGPT to assess performance, using the same model version
(gpt-3.5-turbo-0125) and the same evaluation prompt as previous work to ensure
comparability of results.

The results presented in Table 5(c) show that -34B outperforms all existing
models on MSVD-QA, ActivityNET-QA, and TGIF-QA, establishing new state-of-the-
art results. Additionally, -7B outperforms all existing models on these
datasets, with the exception of PLLaVA-34B. Similar to multi-choice VQA, we
observe a consistent increase in accuracy and quality as the size of the LLM
increases.

####  5.2.3 Zero-shot video captioning

In this experiment, we evaluated our model on MSVD [9], MSR-VTT [80], and
VATEX [71] video captioning benchmarks. Since our model was not exposed to the
training sets of these benchmarks, we compared our results with previous zero-
shot video captioning models. Table 5(b) demonstrates that our models
established new state-of-the-art for zero-shot video captioning. It is
important to note that CIDEr scores for zero-shot evaluation are typically
much lower than those for models specifically fine-tuned on these datasets.
However, fine-tuned models are optimized to match specific benchmarks in terms
of caption length, word usage, and language style, which can boost the CIDEr
scores but does not necessarily reflect the true description quality.

###  5.3 Ablation studies

Model | Original setting | Ablating setting | DREAM-1K | MVBench
---|---|---|---|---
-7B | - | - | 34.6 | 62.6
w/o pre-training | stage1+stage2 | stage2 only | 30.6 | 46.6
w/o instruction tuning | stage1+stage2 | stage1 only | 25.8 | 60.2
w/o multi-task pre-train | stage1: multi-task data | stage1: WebVid-10M | 31.4 | 48.4
w/o multi-grained | stage2: multi-grained |  stage2: VideoChat2 [32] | 22.4 | 61.0
description data | description data | training data |  |
w/o full LLM tuning | train all LLM layers | train top-8 LLM layers | 29.9 | 49.3
Table 6: Ablation study results. We report the F1 scores on DREAM-1K and the
accuracy scores on MVBench.

In this section, we conduct ablation studies to assess the effectiveness of
our training methodology. We examine the impact of five different ablation
settings: (1) omitting pre-training, (2) omitting instruction tuning, (3)
substituting our 13.6M multi-task pre-training data with the WebVid-10M [6]
video captioning data, (4) substituting our 500K instruction tuning data with
the VideoChat2 [32] training data, which contains 788K video instruction
tuning pairs with only 7K detailed video descriptions, and (5) rather than
tuning all parameters of the LLM, we train only its top-8 layers to reduce
computational costs. Due to the high computational costs involved, our
experiments are conducted on the 7B model.

Table 6 presents the results of our ablation studies, revealing that each
modification leads to a significant decline in performance across both
DREAM-1K and MVBench. Specifically, removing stage 1 pre-training or replacing
the pre-training dataset with WebVid-10M results in a substantial drop in
MVBench accuracy, with decreases of $-16.0$ and $-14.2$ absolute points
respectively. The table also shows a notable drop in the DREAM-1K scores.
These results underscore the importance of multi-task pre-training in
developing a robust generalist model. On the other hand, omitting instruction
tuning or changing the instruction tuning data significantly diminishes the
DREAM-1K score by $-8.8$ and $-12.2$ absolute points, highlighting the
critical role of our human-annotated, multi-grained video description data in
enhancing the model’s video description capabilities. Furthermore, training
only the top-8 layers of the LLM leads to inferior outcomes on both datasets,
emphasizing the importance of full model tuning.

##  6 Conclusion and Future Work

In this paper, we release the family of models, which outperforms existing
open-source video description models in both automatic and human evaluations.
Our ablation studies reveal that the strong performance of can be attributed
to its extensive multi-task pre-training, as well as its fine-tuning on human-
annotated, multi-grained video description data. Looking ahead, several
directions for further exploration appear promising. First, there is potential
for scaling up the pre-training data to the next level, which would require
innovative methods to acquire video-text pairs from the internet. Second,
scaling up both the visual encoder and the LLM simultaneously could yield
improvements. Third, refining the model’s ability to follow complex
instructions could further enhance its utility.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in neural information processing systems, 35:23716–23736, 2022.
  * [3] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In Proceedings of the IEEE international conference on computer vision, pages 5803–5812, 2017.
  * [4] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny.  Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.  arXiv preprint arXiv:2404.03413, 2024.
  * [5] George Awad, Keith Curtis, Asad Butt, Jonathan Fiscus, Afzal Godil, Yooyoung Lee, Andrew Delgado, Eliot Godard, Lukas Diduch, Jeffrey Liu, et al.  An overview on the evaluated video retrieval tasks at trecvid 2022.  arXiv preprint arXiv:2306.13118, 2023.
  * [6] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  * [7] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  * [8] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [9] David Chen and William B Dolan.  Collecting highly parallel data for paraphrase evaluation.  In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190–200, 2011.
  * [10] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  arXiv preprint arXiv:2311.12793, 2023.
  * [11] Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu.  Valor: Vision-audio-language omni-perception pretraining model and dataset.  arXiv preprint arXiv:2304.08345, 2023.
  * [12] Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, and Jing Liu.  Cosa: Concatenated sample pretrained vision-language foundation model.  arXiv preprint arXiv:2306.09085, 2023.
  * [13] Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu.  Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.  Advances in Neural Information Processing Systems, 36, 2024.
  * [14] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al.  Panda-70m: Captioning 70m videos with multiple cross-modality teachers.  arXiv preprint arXiv:2402.19479, 2024.
  * [15] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee.  Tracking anything with decoupled video segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1316–1326, 2023.
  * [16] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [17] Dave Epstein, Boyuan Chen, and Carl Vondrick.  Oops! predicting unintentional action in video.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 919–929, 2020.
  * [18] Chenyou Fan.  Egovqa-an egocentric video question answering benchmark dataset.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [19] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The” something something” video database for learning and evaluating visual common sense.  In Proceedings of the IEEE international conference on computer vision, pages 5842–5850, 2017.
  * [20] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
  * [21] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al.  Ava: A video dataset of spatio-temporally localized atomic visual actions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6047–6056, 2018.
  * [22] Xingjian He, Sihan Chen, Fan Ma, Zhicheng Huang, Xiaojie Jin, Zikang Liu, Dongmei Fu, Yi Yang, Jing Liu, and Jiashi Feng.  Vlab: Enhancing video language pre-training by feature adapting and blending.  arXiv preprint arXiv:2305.13167, 2023.
  * [23] Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al.  Visual storytelling.  In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1233–1239, 2016.
  * [24] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766, 2017.
  * [25] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo.  Language repository for long video understanding.  arXiv preprint arXiv:2403.14622, 2024.
  * [26] Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee.  An image grid can be worth a video: Zero-shot video question answering using a vlm.  arXiv preprint arXiv:2403.18406, 2024.
  * [27] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In International Conference on Computer Vision (ICCV), 2017.
  * [28] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  Hmdb: a large video database for human motion recognition.  In 2011 International conference on computer vision, pages 2556–2563. IEEE, 2011.
  * [29] Jie Lei, Tamara L Berg, and Mohit Bansal.  Revealing single frame bias for video-and-language learning.  arXiv preprint arXiv:2206.03428, 2022.
  * [30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [31] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  arXiv preprint arXiv:2305.06355, 2023.
  * [32] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al.  Mvbench: A comprehensive multi-modal video understanding benchmark.  arXiv preprint arXiv:2311.17005, 2023.
  * [33] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo.  Tgif: A new dataset and benchmark on animated gif description.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2016.
  * [34] W Lian, B Goodson, E Pentland, et al.  Openorca: An open dataset of gpt augmented flan reasoning traces, 2023.
  * [35] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.  Video-llava: Learning united visual representation by alignment before projection.  arXiv preprint arXiv:2311.10122, 2023.
  * [36] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In Text summarization branches out, pages 74–81, 2004.
  * [37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [38] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv preprint arXiv:2310.03744, 2023.
  * [39] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
  * [40] Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li.  St-llm: Large language models are effective temporal learners.  arXiv preprint arXiv:2404.00308, 2024.
  * [41] Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao.  Fineaction: A fine-grained video dataset for temporal action localization.  IEEE transactions on image processing, 31:6937–6950, 2022.
  * [42] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022.
  * [43] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei.  Valley: Video assistant with large language model enhanced ability.  arXiv preprint arXiv:2306.07207, 2023.
  * [44] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  arXiv preprint arXiv:2306.05424, 2023.
  * [45] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik.  Egoschema: A diagnostic benchmark for very long-form video language understanding.  Advances in Neural Information Processing Systems, 36, 2024.
  * [46] Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic.  The jester dataset: A large-scale video dataset of human gestures.  In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
  * [47] Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  Rareact: A video dataset of unusual interactions.  arXiv preprint arXiv:2008.01018, 2020.
  * [48] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630–2640, 2019.
  * [49] Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva.  Spoken moments: Learning joint audio-visual representations from video descriptions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14871–14881, 2021.
  * [50] Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry A McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogério Schmidt Feris, and Aude Oliva.  Multi-moments in time: Learning and interpreting models for multi-action video understanding.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9434–9445, 2021.
  * [51] OpenAI.  Gpt-4v system card.  https://openai.com/index/gpt-4v-system-card/, 2023.
  * [52] AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova.  Rethinking video vits: Sparse video tubes for joint image and video learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2214–2224, 2023.
  * [53] AJ Piergiovanni, Isaac Nobel, Dahun Kim, Michael S Ryoo, Victor Gomes, and Anelia Angelova.  Mirasol3b: A multimodal autoregressive model for time-aligned and contextual modalities.  arXiv preprint arXiv:2311.05698, 2023.
  * [54] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [56] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  Transactions of the Association for Computational Linguistics, 1:25–36, 2013.
  * [57] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  arXiv preprint arXiv:2403.05530, 2024.
  * [58] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele.  Movie description.  International Journal of Computer Vision, 123:94–120, 2017.
  * [59] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari.  Charades-ego: A large-scale dataset of paired third and first person videos.  arXiv preprint arXiv:1804.09626, 2018.
  * [60] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 510–526. Springer, 2016.
  * [61] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Generative pretraining in multimodality.  arXiv preprint arXiv:2307.05222, 2023.
  * [62] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou.  Coin: A large-scale dataset for comprehensive instructional video analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1207–1216, 2019.
  * [63] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [64] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  Advances in neural information processing systems, 35:10078–10093, 2022.
  * [65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [66] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [67] Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, and Can Huang.  Elysium: Exploring object-level perception in videos via mllm.  arXiv preprint arXiv:2403.16558, 2024.
  * [68] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100, 2022.
  * [69] Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, et al.  All in one: Exploring unified video-language pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6598–6608, 2023.
  * [70] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy.  Videoagent: Long-form video understanding with large language model as agent.  arXiv preprint arXiv:2403.10517, 2024.
  * [71] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581–4591, 2019.
  * [72] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al.  Internvideo2: Scaling video foundation models for multimodal video understanding.  arXiv preprint arXiv:2403.15377, 2024.
  * [73] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al.  Internvideo: General video foundation models via generative and discriminative learning.  arXiv preprint arXiv:2212.03191, 2022.
  * [74] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  arXiv preprint arXiv:2212.10560, 2022.
  * [75] Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal.  Videotree: Adaptive tree-based video representation for llm reasoning on long videos.  arXiv preprint arXiv:2405.19209, 2024.
  * [76] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021.
  * [77] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017.
  * [78] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al.  mplug-2: A modularized multi-modal foundation model across text, image and video.  In International Conference on Machine Learning, pages 38728–38748. PMLR, 2023.
  * [79] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  arXiv preprint arXiv:2109.14084, 2021.
  * [80] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016.
  * [81] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng.  Pllava: Parameter-free llava extension from images to videos for video dense captioning.  arXiv preprint arXiv:2404.16994, 2024.
  * [82] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo.  Advancing high-resolution video-language representation with large-scale video transcriptions.  In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [83] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu.  Videococa: Video-text modeling with zero-shot transfer from contrastive captioners.  arXiv preprint arXiv:2212.04979, 2022.
  * [84] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum.  Clevrer: Collision events for video representation and reasoning.  arXiv preprint arXiv:1910.01442, 2019.
  * [85] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019.
  * [86] Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius.  A simple llm framework for long-range video question-answering.  arXiv preprint arXiv:2312.17235, 2023.
  * [87] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  arXiv preprint arXiv:2306.02858, 2023.
  * [88] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li.  Llava-next: A strong zero-shot video understanding model, April 2024.
  * [89] Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al.  Videoprism: A foundational visual encoder for video understanding.  arXiv preprint arXiv:2402.13217, 2024.
  * [90] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg.  Visual to sound: Generating natural sound for videos in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3550–3558, 2018.

##  Appendix A Training hyperparameters

The following table shows the training hyperparameters in Stage 1 and Stage 2.

Configuration | Stage 1 (Pre-training) | Stage 2 (Instruction Tuning)
---|---|---
ViT init. | clip-vit-large-patch14-336
LLM init. | LLaVA-NeXT
VL Adapter init. | LLaVA-NeXT
Optimizer name | AdamW | AdamW
Optimizer $\beta_{1}$ | $0.9$ | $0.9$
Optimizer $\beta_{2}$ | $0.999$ | $0.999$
Optimizer eps | $1e^{-6}$ | $1e^{-6}$
Learning rate | $2e^{-5}$ | $2e^{-5}$
Learning rate schedule | cosine | cosine
Training steps | 100,000 | 10,000
Warm-up steps | 1,000 | 1,500
Weight decay | 0.01 | 0.01
Gradient clip | 1.0 | 1.0
Dropout rate | 0 | 0
Global batch size | 288 | 96
Frame resolution | 336x336 | 336x336
Tokens per frame | 576 | 576
Frames per video | [8,32] | {8,16}
Numerical precision | bfloat16 | bfloat16
Table 7: Training hyperparameters of

##  Appendix B AutoDQ prompts

This appendix presents the ChatGPT prompts used by AutoDQ to execute automatic
evaluation. Below is the prompt for event extraction. There is a placeholder
to put in the video description text from which events will be extracted:

Below is a description of a video clip: [The video description] Extract at
most 10 key events from the above video description paragraph. Requirements:
\- An event must include an action, motion or movement (NOT STATIC
INFOMATION). DON’T repeat same events. \- Every event is represented by a
brief sentence with in 10 words, with a subject, a predicate and optionally an
object, avoid unnecessary appearance descriptions. \- Every event must be
atomic, meaning that it cannot be further split into multiple events. \- Scene
cuts and camera motions are NOT events. \- Substitute pronouns by the nouns
they refer to. Please generate the response in the form of a Python dictionary
string with keys "events". The value of "events" is a List(str), of which each
item is an event. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only
provide the Python dictionary string. For example, your response should look
like this:{"events": [event1, event2, ...]}

Below is the ChatGPT prompt for predicting entailment. There are placeholders
to put in the video description text, as well as the events extracted by the
previous prompt:

Given a video description and a list of events. For each event, classify the
relationship between the video description and the event into three classes:
entailment, neutral, contradiction. \- "entailment" means that the video
description entails the event. \- "contradiction" means that some detail in
the video description contradicts with the event. \- "neutral" means that the
relationship is neither "entailment" or "contradiction". Output a list in Json
format: [ {"event": "copy an event here", "relationship": "put class name
here", "reason": "give a reason"}, ... ] Video description: [The video
description] Events: [The events in Json list-of-string format] DO NOT PROVIDE
ANY OTHER OUTPUT TEXT OR EXPLANATION. Only output the JSON. Output:

##  Appendix C Video description prompts

This appendix presents the prompts used for the DREAM-1K experiments in
Section 5.1. The default prompt is:
Describe the video in detail.

The prompt for GPT-4V:
Given 8 frames uniformly sampled from a video clip, describe the video (not
the individual images!) in detail, focusing on the main subjects, their
actions and the background scene. DON’T describe feelings or atmosphere.

The prompt for Gemini 1.5 Pro:
Describe the video in one paragraph, mainly focusing on the dynamic events in
the video. Don’t describe feelings or atmosphere.

The prompt for PLLaVA-34B, as recommended by [81]:
You are to assist me in accomplishing a task about the input video. Reply to
me with a precise yet detailed response. For how you would succeed in the
recaptioning task, read the following Instructions section and Then, make your
response with a elaborate paragraph. # Instructions 1\. Avoid providing over
detailed information such as color, counts of any objects as you are terrible
regarding observing these details 2\. Instead, you should carefully go over
the provided video and reason about key information about the overall video
3\. If you are not sure about something, do not include it in you response. #
Task Describe the background, characters and the actions in the provided
video.

The prompt for LLaVA-NeXT-Video, as recommended by [88]:
Please provide a detailed description of the video, focusing on the main
subjects, their actions, and the background scenes.

##  Appendix D Human evaluation details

In the human evaluation, we randomly sampled a subset of DREAM-1K, consisting
of 300 videos. Three experienced annotators were involved in the evaluation.
For each model pair, each annotator was presented with 100 video description
pairs. Each pair contained two descriptions generated by the two models, but
their order was randomly swapped to ensure that the annotator could not
identify the model that produced each description. The annotators were asked
to assign one of three labels to each pair: “Description A is better,”
“Description B is better,” or “Same quality.” Before the annotation process
began, we provided them with the following annotation guide:

> Imagine you are reading video descriptions to a blind person. Which version
> of the description better helps the blind person understand the content of
> the video? You should choose the more helpful version. This judgment may be
> subjective, but in principle, you can consider both accuracy and
> comprehensiveness:
>
>   * •
>
> When equally accurate, the more comprehensive description is more helpful.
>
>   * •
>
> When equally comprehensive, the more accurate description is more helpful.
>
>   * •
>
> If both descriptions have flaws in accuracy and comprehensiveness, you need
> to judge which flaw has a lesser impact on understanding the main content of
> the video. In this case, you can use your own subjective feelings and common
> sense to make a judgment, without relying on explicit rules.
>
>   * •
>
> We value the description of dynamic events. Therefore, your evaluation
> should focus on dynamic events (actions, behaviors, changes, etc.). Detailed
> descriptions of static aspects (appearance, color, texture, static
> background) do not earn extra points, unless they help understand key
> events. The model does not accept audio input, so you can completely ignore
> the sounds in the video.
>
>   * •
>
> Please make an effort to discern the better description, and try to use the
> ”Same quality” option as little as possible.
>
>