#  VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video
Foundation Model

Xinhao Li1,2, Zhenpeng Huang1, Jing Wang1, Kunchang Li2, Limin Wang1,2
1State Key Laboratory for Novel Software Technology, Nanjing University
2Shanghai AI Laboratory
https://github.com/MCG-NJU/VideoEval Corresponding author (lmwang@nju.edu.cn).

###### Abstract

With the growth of high-quality data and advancement in visual pre-training
paradigms, Video Foundation Models (VFMs) have made significant progress
recently, demonstrating their remarkable performance on traditional video
understanding benchmarks. However, the existing benchmarks (e.g. Kinetics) and
their evaluation protocols are often limited by relatively poor diversity,
high evaluation costs, and saturated performance metrics. In this paper, we
build a comprehensive benchmark suite to address these issues, namely
VideoEval. Specifically, we establish the Video Task Adaption Benchmark
(VidTAB) and the Video Embedding Benchmark (VidEB) from two perspectives:
evaluating the task adaptability of VFMs under few-shot conditions and
assessing their representation power by directly applying to downstream tasks.
With VideoEval, we conduct a large-scale study on 20 popular open-source
vision foundation models. Our study reveals some insightful findings on VFMs:
1) overall, current VFMs exhibit weak generalization across diverse tasks, 2)
increasing video data, whether labeled or weakly-labeled video-text pairs,
does not necessarily improve task performance, 3) the effectiveness of some
pre-training paradigms may not be fully validated in previous benchmarks, and
4) combining different pre-training paradigms can help improve the
generalization capabilities. We believe this study serves as an important
complement to the current evaluation for VFMs and offers valuable insights for
the future research.

##  1 Introduction

Figure 1:  Overview of VideoEval. We propose a novel, vision-centric
evaluation method for video foundation models that is comprehensive,
challenging, indicative, and low-cost.

The field of deep learning is experiencing a paradigm shift of pre-training
and transfer learning due to the emergence of foundation models (FMs). These
models, exemplified by BERT [1], GPT [2, 3, 4], CLIP [5] and Stable Diffusion
[6], are pre-trained on massive and diverse data at scale and demonstrate
remarkable generalization power to a broad spectrum of downstream tasks.

In the realm of video understanding, early researchers train backbone networks
[7, 8, 9, 10, 11, 12] using visual classification tasks on large-scale labeled
datasets like ImageNet [13] and Kinetics [14]. However, the high cost
associated with labeled data promotes the development of self-supervised
learning methods that capitalize on unlabeled data for visual pre-training
[15, 16, 17, 18, 19]. Furthermore, researchers delve into multimodal pre-
training utilizing large-scale weakly-supervised visual-text pairs [20, 21,
22, 23, 24], thereby enhancing their models’ capabilities and demonstrating
impressive zero-shot performance. Overall, fueled by the accumulation of high-
quality image and video data and advancements in visual pre-training
paradigms, Video Foundation Models (VFMs) witness remarkable progress in
recent years. A new generation of VFMs [17, 18, 25, 26, 27, 28, 29] emerges,
demonstrating outstanding performance on conventional video understanding
benchmarks.

The rapid development of VFMs raises a new problem: How to evaluate a video
foundation model? In image realm, previous works assess the generalization
capability of Image Foundation Models (IFMs) by evaluating their performance
on numerous downstream visual tasks, encompassing diverse scenarios and
evaluation protocols [30, 31, 32, 33, 34, 35, 36]. In contrast, previous works
primarily evaluates VFMs through benchmarks simply focusing on action
recognition tasks [18, 26, 37]. Some studies [38, 29, 27] have also considered
combining VFMs with Large Language Models (LLMs) to evaluate their performance
on multimodal video understanding tasks. There are several issues with current
evaluation benchmarks: (1) Limited domain and task: benchmarks like Kinetics
[14], Something-Something [39] and AVA [40], which focus on action
recognition, overlook other video understanding scenarios (e.g., video quality
assessment), limiting their applicability in evaluating the generalization
capabilities of visual foundation backbones across diverse video understanding
applications. (2) Saturated performance: the performance of VFMs on the
standard benchmarks [41] has reached a saturation point ( around 90% Top-1
accuracy), making it challenging to differentiate between the true
capabilities of different VFMs. (3) High evaluation cost: the conventional
validation protocols often necessitate end-to-end training on the entire
dataset, resulting in a significant challenge for large VFMs. (4) Less
indicative: Incorporating LLM into VFMs for evaluation on multi-modal video
tasks may introduce bias and is less indicative for vision representation
power, as performance differences might stem from the language model rather
than the VFMs itself.

To tackle these problems, we build a comprehensive vision-centric benchmark
suite for evaluating VFMs, namely VideoEval. As shown in Figure 1, our method
has the following key features: Comprehensive: First, we create the Video Task
Adaptation Benchmark (VidTAB) to evaluate the adaptability of VFMs to unseen
tasks with limited training samples. We collect public datasets from various
video task domains, ranging from action recognition in special scenarios, AI
for science, video content moderation, video quality/aesthetic assess, to
emotion analysis. From these domains, we construct eight adaptation tasks and
propose evaluation protocols and adaptation methods suitable for current VFMs.
Additionally, to assess the capability of VFMs as a direct feature embedding
extractor for downstream applications, we create the Video Embedding Benchmark
(VidEB), which includes four tasks to evaluate their embedding power at
different granularities. Challenging & Indicative: Due to the diversity of
test data and the more challenging evaluation protocols, our VideoEval can
effectively distinguish between various VFMs that perform similarly on
traditional benchmarks, providing deeper insights and more accurate evaluation
on their true capabilities. Low-cost: Thanks to our training-light few-shot
evaluation and training-free feature embedding evaluation protocols, VideoEval
requires significantly fewer evaluation cost compared to previous benchmarks,
while still maintaining a comparable number of testing samples to ensure
accurate and stable evaluations. Vision-centric: Our evaluation focuses solely
on the capabilities of video foundation models themselves, avoiding the bias
due to the introduction of LLMs.

Based on VideoEval, we evaluate 20 open-source vision foundation models,
including Video Foundation Models (VFMs), Image Foundation Models (IFMs), and
IFMs with image-to-video adaptors. Our main findings as following: First,
current VFMs still struggle to adapt to unseen video tasks with limited
training samples. Second, while more data and larger models generally improve
performance, augmenting video training data can sometimes negatively affect
certain tasks. Third, the effectiveness of certain pre-training paradigms,
such as VideoMAEv2 [25], may not have been adequately validated in previous
benchmarks. Finally, combining multiple pre-training paradigms can lead to
models with better generalization capabilities, such as performing multimodal
contrastive learning after unimodal visual self-supervised pre-training [24,
29].

Table 1: Comparison of VFMs Benchmark. "Num. training" denotes number of
training samples, "Num. test" denotes number of test samples, and "Beyond
Action" denotes the tasks in this benchmark extend beyond action
understanding.

Benchmark | Num. training | Num. test | Beyond Action | Task Diversity | Domain Diversity | VFMs-specific protocol
---|---|---|---|---|---|---
Single-dataset Benchmarks
Kinetics-400 [41] | 240,436 | 19,165 | ✗ | ✗ | ✗ | ✗
Sth-Sth V2 [42] | 168,913 | 24,777 | ✗ | ✗ | ✗ | ✗
Moment-in-Time [43] | 791,246 | 33,898 | ✗ | ✗ | ✗ | ✗
UCF101 [44] | 9,537 | 3,783 | ✗ | ✗ | ✗ | ✗
Multi-dataset Benchmarks
SEVERE [45] | 868,446 | 144,830 | ✗ | ✓ | ✓ | ✗
BEAR [46] | 240,236 | 140,436 | ✗ | ✓ | ✓ | ✗
VideoGLUE [37] | 1,896,621 | 239,011 | ✗ | ✓ | ✓ | ✓
VideoEval | 5,704 | 20,497 | ✓ | ✓ | ✓ | ✓

##  2 Related Work

#### Video foundation models (VFMs).

With the continuous growth of image [47, 48, 49] and video data [50, 23, 51,
52, 53] and advancements in pre-training paradigms, research on Video
Foundation Models (VFMs) has progressed rapidly. Current VFMs are primarily
built with different pre-training paradigms: contrastive or predictive self-
supervised learning methods [54, 15, 55, 56, 26] on unimodal video data,
masked video modeling based on unimodal video data [17, 18, 25, 19, 57, 58,
59] and video-text contrastive learning based on multimodal visual-text pairs
[20, 21, 60, 22, 61, 23]. Some works [28, 24, 29, 27] combine these paradigms,
enabling VFMs to extend further into multimodal understanding. Additionally,
some studies introduce modalities like audio and speech on top of video and
text [51, 52, 29], further expanding the capabilities of VFMs. Recently,
InternVideo2 [29] combines pre-training paradigms and large-scale high-quality
data to scale VFMs to 6 billion parameters, achieving remarkable performance
improvements.

#### Evaluation of VFMs.

Previous works primarily utilize action recognition benchmarks [14, 42, 40] to
evaluate VFMs with a focus on understanding the appearnace and motion from
videos. To enhance evaluation diversity, some studies explore richer domains
and tasks [62, 46, 63], but they still remain limited to the action
recognition tasks. The InternVideo series [28, 29] and VideoGLUE [37] attempt
to provide a more comprehensive evaluation on VFMs by reporting performance on
a number of benchmarks and evaluation protocols. However, these efforts are
still based on existing benchmarks and incurred high validation costs. In
contrast, our VideoEval considers the characteristics and application
scenarios of VFMs, offering a comprehensive and more challenging evaluation
solution from the perspectives of task definition and evaluation protocols.

##  3 Building VideoEval

Figure 2:  Illustration of building VideoEval. We build VideoEval through the
following steps: (1) conducting task selection by considering our expected
capabilities for VFMs. (2) performing data filtration from the perspectives of
quality, difficulty, and diversity, (3) standardizing the task format through
task construction, and (4) defining evaluation methods and targets.

We argue that a powerful video foundation model should possess two key
capabilities: (1) strong task adaptation ability, i.e., the ability to adapt
to diverse and new tasks with limited training samples, and (2) the capacity
of general video encoder to extract feature embedding that retain and distill
key information from videos, directly supporting various downstream tasks.
From these two perspectives, we construct VideoEval, which includes the Video
Task Adaptation Benchmark (VidTAB) and the Video Embedding Benchmark (VidEB).
By collecting diverse tasks from different scenarios and employing low-cost
evaluation scheme, our VideoEval can efficiently and comprehensively assess
the generalization ability of VFMs in video understanding. In this section, we
present our VideoEval in detail. The construction pipeline for VideoEval is
illustrated in Figure 2, and the evaluation tasks we ultimately constructed
are presented in Table 2.

###  3.1 Video Task Adaption Benchmark

#### Collecting diverse datasets from public source.

Previous benchmarks primarily focus on evaluating video models for action
recognition, overlooking many other video understanding tasks. Therefore, we
consider five different application scenarios: Action Recognition in Special
Scenarios (Action), AI for Science (Science), Video Content Moderation
(Safety), Video Quality Assessment (Quality), and Emotion Analysis (Emotion).
We collect publicly annotated video datasets from the Internet. Subsequently,
we manually filter out data with low video or annotation quality. We also
consider the difficulty level of tasks to ensure that our benchmark can
distinguish between models. We discard tasks that are very easy or can even be
accomplished without video understanding, as well as tasks that are extremely
difficult even for humans. It does not imply that we have low expectations for
foundation model capabilities relative to humans; Rather, we aim to ensure
meaningful and effective evaluation in the context of task adaptation, because
overly challenging tasks would not provide clear distinctions between models.

Table 2:  Task details of VideoEval. All videos are collected from the public
datasets for building tasks of VidTAB and VidEB.

Domain | Task | Source | Task Description
---|---|---|---
Video Task Adaptation Benchmark (VidTAB)
Action Recognition in Special Scenarios | Action Recognition | ARID [64] | Recognizing 11 distinct human actions in dark scenarios.
in Dark Scene | e.g. Run / Walk / Drink
Action Recognition | BreakFast [65] | Classifying 10 types of long-duration cooking videos.
in Long Video | e.g. Milk / Tea / Sandwich
AI for Science | Medical Surgery | SurgicalActions160 [66] |  Classifying 16 surgical actions in gynecologic laparoscopy. e.g. Knotting / Suction / Injection
Animal Behavior |  Animal Kingdom [67] | Classifying 12 behaviors of wild animals from diverse environmental footage.
e.g. Flying / Chirping / Preening
Video Content Moderation |  Fake Face |  FaceForensics++ [68] | Determine whether the faces in the video have been tampered with by AI technology (such as DeepFake).
e.g. Origin video / Video with fake face
Harmful Content | mob [69] | Detecting 3 degrees of malicious content within videos.
e.g. Obscene / Indecent activity / Violent activity
Video Quality Assessment |  Quality Assess | DOVER [70] | Evaluating videos from an aesthetic and technical perspective and categorizing them into low and high quality.
e.g. Low quality / High quality
Emotion Analysis |  Emotion Analysis | CAER [71] | Classifying 7 different human emotions in video.
e.g. Happy / Fear / Anger
Video Embedding Benchmark (VidEB)
Scene Understanding in Temporal Contexts |  Duplicate Scene Retrieval | FIVR5K [72] | Retrieve Duplicate Scene Videos (DSV):
Videos captured by the same camera and sharing at least one scene (without
considering any application transformations).
Complementary Scene Retrieval | FIVR5K [72] | Retrieve Complementary Scene Videos (CSV):
Retrieve a portion of the same spatiotemporal segment captured from different
perspectives.
Incident Scene Retrieval | FIVR5K [72] | Retrieving Incident Scene Videos (ISV):
The same event is close in both space and time, but there are no overlapping
videos.
Copy Detection | DVSC23 [73] | Detecting edited versions of the same source video.
Given a query inserted with one or more copied segments, detect the source
video from the database.
|  |  |

#### Constructing the adaptation tasks based on the existing annotations.

Classification tasks are straightforward and well-defined, with high
classification performance often indicating effective feature learning.
Therefore, they are suitable for evaluating video foundation models. We
construct adaptation classification tasks based on the collected data and
annotations. For datasets that originally include category labels, such as
ARID [64] and Animal Kingdom [67], we select categories with sufficient
samples to ensure evaluation accuracy and stability. We also control the final
number of categories to avoid making the adaptation task overly difficult. For
datasets that are not originally classification tasks, we transform the tasks
into classification tasks. For example, for DOVER [70], which is used for
video aesthetics and technical quality assessment (a regression task), we
assume that videos with quality scores in the top 40% are "high-quality
videos" and those with scores in the bottom 40% are "low-quality videos", thus
converting the original task into a binary classification task. In total, we
construct eight classification tasks to evaluate the adaptation capabilities
of video foundation models. Specific construction details of each task are
provided in the Appendix.

#### Determining the evaluation protocol and metric.

Previous studies [28, 29, 37] typically fine-tune video foundation models
using the entire samples of training set, and most popular benchmarks have
large training sample sizes. We argue that this evaluation method overlooks
the examination of the adaptation capability of VFMs under few-shot setting.
As illustrated in Figure 3, under the scenario of using full training samples,
the differences between VFMs are difficult to discern. However, under a low-
sample protocol, different foundation models exhibit varying degrees of task
adaptation capabilities. We observe that for the task of Action Recognition in
Dark Scenes, which VFMs usually excel at, there are significant differences in
adaptation capabilities among different models when training samples are
extremely limited (4 shot and 16 shot). As the number of samples gradually
increases to 100 shot, these differences diminish. Conversely, for more
challenging tasks like Emotion Analysis, the performances of different models
are uniformly weak when training samples are extremely limited, showing no
discernible differences until a certain number of training samples (100 shot)
are reached, at which point different models begin to demonstrate distinct
adaptation capabilities. Therefore, to account for the adaptation capabilities
of models with different numbers of training samples, we define a new task
adaptation capability evaluation score (TA-score):

$\text{TA-score}=\frac{{Acc}^{4s}+Acc^{16s}+Acc^{100s}}{3},$ (1)

where $Acc^{4s}$, $Acc^{16s}$, $Acc^{100s}$ represent the model’s top-1
accuracy for 4-shot, 16-shot, and 100-shot classifications, respectively.
Unless otherwise specified, we will use the TA-score as the metric to evaluate
the performance of various tasks in VidTAB.

Figure 3: Performance comparison on different training data scales. We
evaluate the performance variation of multiple video foundation models across
tasks from two different domains as the scale of the training data changed.
’FT’ and ’AP’ denote full finetuning and attentive probe, respectively. Figure
4:  Illustration of different adaptation method: (a) Full Finetuning, (b)
Adapter, (c) Attentive Probe, and (d) Linear Probe. Table 3:  Comparison of
adaptation method on V-JEPA-H [26] All results are obtained using A100-80G
with PyTorch-builtin mixed precision, using a batch size of 4 to measure CUDA
memory and training time. "Dark" and "Emotion" denote the tasks of Action
Recognition in Dark Scenes and Emotion Analysis, respectively.

| Adaptation
---
method
| Tunable
---
Params (M)
| CUDA
---
Memory (G)
| Training
---
Time (h)
| Dark
---
TA-score
| Emotion
---
TA-score
full finetuning | 663.7 | 52.1 | 1.0 | 68.8 | 25.3
adapter | 52.6 | 45.0 | 1.0 | 62.4 | 24.7
attentive probe | 19.7 | 6.4 | 0.4 | 54.7 | 23.8
linear probe | 0.0 | 6.0 | 0.3 | 12.9 | 16.2
|  |  |  |  |

#### Identifying efficient adaptation method for evaluation.

We also need to investigate how to adapt the video foundation models to the
downstream tasks. Previous work [74, 75, 76, 77, 78, 79, 80] has explored
various strategies for efficient adapting the foundation models. Here, we
consider several popular methods, as illustrated in Figure 4: Full Finetuning:
Fine-tuning all the parameters of the pre-trained model. Adapter: Freezing the
pre-trained model and inserting learnable low-rank adapter [81] modules into
each block of the pre-trained model for adaptation. Attentive Probe: Freezing
the pre-trained model and adding an additional learnable cross-attention block
at the end of the model to achieve attentive pooling, followed by a linear
projection for classification. Linear Probe: Directly using the features from
the pre-trained model, performing mean pooling, and then using a linear
projection for classification. We evaluate the performance of these adaptation
methods based on the V-JEPA-H model, as shown in Table 3. Full finetuning and
adapter achieve the best adaptation performance, but require high training
costs. Linear probe is highly efficient but showed weak adaptation
performance. Attentive probe offers a good trade-off between efficiency and
adaptation performance. Therefore, in subsequent evaluation experiments, we
employ attentive probe to adapt various video foundation models.

###  3.2 Video Embedding Benchmark

Apart from the traditional classification tasks, the more direct evaluation of
representations typically involves the retrieval tasks such as video action
retrieval [82, 83, 84], which primarily rely on text information. However,
these benchmarks often overlook the overall scene context and exhibit an
overlap with recognition tasks. In contrast, inspired by previous works [85,
73, 86, 87, 88], we focus on vision-centric embedding evaluation in Table 2.
Specifically, we require models to rank and recall instance-level samples
based on similarity, rather than solely on action categories. This evaluation
protocol focuses on the model’s capability of capturing the subtle visual
information.

Evaluation protocol. To facilitate fine-grained embedding evaluation, we
include two tasks for assessment: (1) Hierarchical Video Retrieval aims to
retrieve videos from a database that closely matches the query video in terms
of scene, viewpoint, and temporal context. According to previous work [72],
videos related to the query are categorized into three levels based on their
similarity to the query: Duplicate Scene Videos (DSVs), Complementary Scene
Videos (CSVs), and Incident Scene Videos (ISVs), as shown in Table 2.
Consequently, the retrieval tasks are structured into three hierarchical
levels:

  * •

Duplicate Scene Video Retrieval: only DSVs are positive instances.

  * •

Complementary Scene Video Retrieval: both DSVs and CSVs are positive
instances.

  * •

Incident Scene Video Retrieval: DSVs, CSVs, and ISVs are all positive
instances.

For the evaluation metric, we follow [72] to utilize the mean Average
Precision (mAP) to assess the quality of video ranking. (2) Video Copy
Detection aims to detect edited copies of the query video. In contrast to the
ranking/retrieval task where all video pairs need to be sorted according to
video embedding similarity, it is required to identify a set of video pairs
that contain edited versions of the given query. Following [73], we consider
the micro-AP ($\mu$AP) as our evaluation metric that operates on all queries
jointly and takes the confidence scores into account.

##  4 Benchmarking Video Foundation Models

###  4.1 Video foundation models and evaluation details

#### Evaluation targets.

We evaluate twenty open-source vision foundation models. Including: (1) twelve
video foundation models, covering different pre-training paradigms, model
scales, and training data scales, to analyze the impact of these factors on
the generalization capability of foundation models. (2) five image foundation
models to observe how much generalization capability trained on image data can
exhibit in video understanding. (3) three image-to-video methods based on
image foundation models to assess the effectiveness of current efficient
transfer methods.

#### Implementation details.

All models take 8 frames (16 frames if the model has temporal downsampling),
with each frame being 224x224 in size as input. For VidTAB, to ensure fair
comparison and efficient assessment, we train all models for the same number
of epochs and make minor adjustments to the hyperparameters to ensure
convergence. For VidEB, all models take 16 frames, with each frame of size
224x224. In hierarchical video retrieval, the similarity of video-level
embedding determines the ranking of retrieval results. In video copy
detection, each sample is segmented into 5 clips. The detection confidence
score for the entire video is derived from the maximum frame-wise similarity
computed for each query-reference pair. See the appendix for more details.

###  4.2 Results on VidTAB

Table 4: Evaluating state-of-the-art FMs on the VidTAB. The best and second-
best results of foundation models are noted by blue and underline,
respectively. ’I’, ’V’, and ’IV’ denote image data, video data, and mixed
image-video data, respectively. Data marked in gray indicates that the model
uses a model trained on that data as initialization. ’K710ft’ indicates that
the model was fine-tuned with supervision using the labeled action recognition
dataset Kinetics-710 (0.66M).

|  |  |  | Action | Science | Safety | Quality | Emotion
---|---|---|---|---|---|---|---|---
# Params (M) |  # Pretrain Data |  Average |  Dark Scene |  Long Video |  Medical Surgery |  Animal Behavior |  Harmful Content |  Fake Face |  Quality Assess |  Emotion Analysis
Random | - | - | 22.7 | 9.1 | 10.0 | 6.3 | 8.3 | 33.3 | 50.0 | 50.0 | 14.3
Zero-shot performance of visual language models
CLIP-L [5] | 428 |  I-400M | 35.7 | 29.2 | 34.6 | 12.5 | 32.9 | 42.1 | 56.3 | 65.5 | 12.9
EVA-CLIP-g [89] | 1365 |  I-2B | 36.0 | 32.8 | 37.2 | 9.4 | 28.5 | 39.6 | 52.8 | 69.5 | 17.9
ViCLIP-L [23] | 428 |  I-400M+V-200M | 33.6 | 26.2 | 37.5 | 8.3 | 29.3 | 32.1 | 52.2 | 53.9 | 29.0
InternVideo2stage2 [29] | 1350 |  IV-1.1M+IV-25.5M | 40.6 | 37.1 | 40.2 | 11.5 | 45.2 | 59.1 | 51.3 | 56.1 | 24.3
Image Foundation Model
CLIP-L [5] | 316 |  I-400M | 42.8 | 31.2 | 38.0 | 32.3 | 36.3 | 50.3 | 58.5 | 67.7 | 28.1
SigLiP-SO [90] | 444 |  I-4.11B | 42.6 | 26.9 | 37.6 | 36.5 | 35.7 | 49.4 | 58.2 | 68.1 | 28.2
EVA-g [91] | 1035 |  I-2B | 45.4 | 39.7 | 45.9 | 34.4 | 40.0 | 50.8 | 55.2 | 68.5 | 28.6
DINOv2-L [56] | 317 |  I-142M | 40.7 | 37.9 | 41.0 | 39.6 | 35.7 | 34.3 | 52.0 | 62.2 | 23.2
DINOv2-g [56] | 1165 |  I-142M | 43.0 | 35.8 | 43.5 | 42.7 | 35.6 | 44.0 | 53.2 | 63.7 | 25.3
Image Foundation Model with image-to-video adaptation method
ST-Adapter-CLIP-L [78] | 328 |  I-400M | 46.9 | 43.0 | 45.0 | 31.2 | 39.4 | 49.4 | 64.9 | 72.3 | 29.9
AIM-CLIP-L [79] | 328 |  I-400M | 48.7 | 42.3 | 49.9 | 38.5 | 38.9 | 47.0 | 68.3 | 74.6 | 30.1
ZeroI2V-CLIP-L [80] | 303 |  I-400M | 46.5 | 41.3 | 46.8 | 31.2 | 39.3 | 47.2 | 64.6 | 70.6 | 30.6
Video Foundation Model
ViCLIP-L-10M [23] | 316 |  I-400M+V-10M | 41.4 | 31.3 | 41.2 | 30.2 | 34.2 | 45.3 | 53.9 | 68.3 | 26.8
ViCLIP-L-200M [23] | 316 |  I-400M+V-200M | 42.7 | 36.7 | 43.9 | 30.2 | 36.8 | 46.9 | 54.8 | 65.4 | 27.2
VideoMAEv1-H [18] | 651 |  V-0.24M | 44.4 | 45.1 | 30.3 | 35.4 | 37.9 | 53.9 | 52.0 | 71.3 | 28.9
VideoMAEv2-g [25] | 1037 |  V-1.35M | 37.7 | 34.9 | 18.5 | 18.8 | 33.3 | 57.6 | 50.7 | 67.6 | 20.4
VideoMAEv2-g [25] | 1037 |  V-1.35M+K710ft | 53.5 | 76.1 | 72.4 | 50.0 | 42.6 | 43.3 | 56.6 | 61.5 | 25.1
UMT-Lstage1 [24] | 316 |  V-0.66M | 40.0 | 33.9 | 34.4 | 30.0 | 33.5 | 44.1 | 53.4 | 64.9 | 26.1
UMT-Lstage2 [24] | 316 |  V-0.66M+IV-25M | 43.6 | 32.6 | 44.0 | 22.9 | 39.3 | 62.3 | 52.8 | 67.7 | 27.2
V-JEPA-L [26] | 318 |  V-2M | 43.0 | 50.2 | 34.1 | 39.6 | 38.5 | 41.9 | 51.5 | 66.9 | 21.0
V-JEPA-H [26] | 653 |  V-2M | 44.5 | 54.7 | 36.0 | 35.4 | 40.5 | 45.1 | 52.9 | 67.7 | 23.8
InternVideo2-1Bstage1 [29] | 1037 |  IV-1.1M | 45.1 | 45.4 | 47.7 | 33.3 | 37.7 | 48.9 | 53.7 | 66.0 | 27.8
InternVideo2-1Bstage1 [29] | 1037 |  IV-1.1M+K710ft | 56.4 | 75.9 | 77.9 | 53.1 | 45.2 | 47.7 | 55.6 | 64.7 | 30.9
InternVideo2-1Bstage2 [29] | 1037 |  IV-1.1M+IV-25.5M | 53.1 | 67.6 | 69.5 | 38.5 | 48.6 | 51.2 | 55.2 | 64.5 | 29.4
|  |  |  |  |  |  |  |  |  |  |

#### Zero-shot evaluation.

To preliminarily test the difficulty of the dataset, we first evaluate the
zero-shot performance of the eight tasks using two image language models and
two video language models. As shown in the top section of Table 3, both image
and video models demonstrate relatively high performance for action-related
tasks, and video models achieve better. For tasks involving low-level
information understanding, such as Quality Assessment task, image models
perform significantly better. In contrast, for other tasks involving scenarios
typically unseen in training data, such as medical surgery videos or Safety
Review tasks requiring complex semantic reasoning, all models perform similar
to random guess.

#### Main results of adapting foundation models.

Table 4 presents the evaluation results on VidTAB. We summarize our findings
as follows.

In general, (1) Despite exhibiting a degree of generalization capability,
current vision FMs still struggle to adapt to unseen video tasks with limited
training samples. VFMs outperform IFMs, particularly in tasks related to
action and behavior understanding. However, IFMs achieve superior performance
on more low-level tasks, particularly in the domains of safety and quality,
especially when combined with image-to-video adaptation techniques.

(2) The adaptation performance of models generally increases with the growth
of data and model size, as observed by the improvements observed from V-JEPA-L
to V-JEPA-H (+1.5) and ViCLIP-L-10M to ViCLIP-L-200M (+1.3).

For the training data, (3) While augmenting video training data is generally
beneficial, it can negatively impact the performance on some tasks. For both
VideoMAEv2-g and InternVideo2-1Bstage1, fine-tuning on Kinetics-710 data
significantly enhances Action-related tasks, but consistently degrades the
performance on the Safety and Quality tasks. Similar findings are observed
with ViCLIP-L, where post-pretraining on a large-scale video dataset improves
Action-related tasks but diminishes performance in other domains (Science,
Safety, Quality, Emotion). It could be attributed to the limited diversity of
the current video training data.

(4) For models trained on single-modal visual data, incorporating additional
weak-supervised post-pretraining with visual-text data leads to significant
improvements in adaptation capabilities. This is evident in the performance
gains observed from UMT-Lstage1 to UMT-Lstage2 (+3.6) and from
InternVideo2-1Bstage1 to InternVideo2-1Bstage2 (+8.0). Interestingly, this
finding contradicts previous conclusions drawn from commonly used action
recognition benchmarks, suggesting that these benchmarks may introduce bias.

For the pre-training paradigms of model, (5) The effectiveness of pre-training
paradigms in scaling model size might not be adequately validated on popular
action recognition benchmarks [41, 42]. While VideoMAEv2 [25] successfully
scaled a model to 1B parameters using the dual masking strategy, its
adaptation performance (37.7 vs 44.4) significantly declined compared to
VideoMAEv1-H. Interestingly, VideoMAEv2-g demonstrated remarkable performance
after fine-tuning on Kinetics-710 (0.66M), suggesting that the abundant
labeled data may have compensated for the shortcomings of its pre-training
performance.

(6) Single-modal self-supervised pre-training paradigms exhibit superior data
efficiency compared to multimodal weakly-supervised pre-training paradigms.
Notably, V-JEPA and VideoMAEv1, trained solely on relatively small-scale
unlabeled video data via self-supervised pre-training, demonstrate comparable
or even superior performance to ViCLIP, which is trained on a massive dataset
of video-text pairs.

Finally, (7) Effective adaptation method for image-to-video FMs is crucial.
Three image-to-video methods based on CLIP-L achieved significant performance
improvements compared to using an attentive probe directly. We believe this
represents a promising avenue for future research.

Table 5: Evaluation of State-of-the-Art Foundation Models on the VidEB
Dataset. "K400pt" and "K400ft" denote that the model is pre-trained and fine-
tuned, respectively, using the labeled action recognition dataset Kinetics-400
(0.31M). MCL: Multi-modal Contrastive Learning, SCL: Self-supervised
Contrastive Learning, MVM: Masked Video Modeling, SFT: Supervised Fine-tuning.
Other notations are consistent with those in Table 4.

|  |  |  | Scene
---|---|---|---|---
Pretrain Tasks | # Pretrain Data | Average | Duplicate | Complementary | Incident | Copyright
Image Foundation Model |
CLIP-L [5] | MCL |  I-400M | 43.0 | 41.1 | 46.4 | 52.0 | 32.3
EVA-g [91] | MCL |  I-2B | 37.1 | 41.4 | 46.1 | 51.7 | 9.3
SigLiP-SO [90] | MCL |  I-4.11B | 38.6 | 40.6 | 45.5 | 51.5 | 16.9
DINOv2-L [56] | SCL |  I-142M | 45.6 | 49.0 | 53.5 | 54.3 | 25.6
DINOv2-g [56] | SCL |  I-142M | 48.6 | 50.5 | 55.1 | 56.0 | 32.8
Video Foundation Model |
ViCLIP-L-10M [23] | MVM+MCL |  I-400M+V-10M | 43.7 | 40.0 | 44.4 | 46.9 | 43.4
ViCLIP-L-200M [23] | MVM+MCL |  I-400M+V-200M | 39.9 | 33.1 | 37.2 | 40.3 | 49.2
VideoMAEv1-L [18] | MVM | K400pt | 12.9 | 14.5 | 15.1 | 13.2 | 8.8
VideoMAEv1-L-K400ft [18] | MVM+SFT | K400pt+ft | 27.4 | 27.6 | 30.2 | 30.3 | 21.6
VideoMAEv2-g [25] | MVM |  V-1.35M | 11.6 | 14.8 | 15.4 | 13.4 | 2.8
VideoMAEv2-g-K710ft [25] | MVM+SFT |  V-1.35M+K710ft | 37.4 | 33.8 | 37.1 | 37.1 | 41.7
UMT-Lstage1 [24] | MVM |  V-0.66M | 41.1 | 42.2 | 46.6 | 49.6 | 25.7
UMT-Lstage1-K710ft [24] | MVM+SFT |  V-0.66M+K710ft | 29.0 | 26.4 | 29.4 | 30.3 | 30.0
UMT-Lstage2 [24] | MVM+MCL |  V-0.66M+IV-25M | 34.2 | 33.4 | 37.3 | 40.6 | 25.4
V-JEPA-L [26] | MVM |  V-2M | 19.7 | 21.3 | 23.9 | 21.7 | 12.0
V-JEPA-H [26] | MVM |  V-2M | 20.2 | 21.5 | 23.7 | 21.2 | 14.3
InternVideo2-1Bstage1 [29] | MVM |  IV-1.1M | 50.4 | 47.3 | 52.1 | 54.9 | 47.3
InternVideo2-1Bstage1-K710ft [29] | MVM+SFT |  IV-1.1M+K710ft | 33.9 | 30.5 | 34.2 | 34.1 | 36.9
InternVideo2-1Bstage2 [29] | MVM+MCL |  IV-1.1M+IV-25.5M | 34.6 | 32.4 | 36.8 | 39.9 | 29.3
|  |  |  |  |  |  |

###  4.3 Results on VidEB

The main results of VidEB are presented in Table 5. We evaluate the embedding
performance using different pre-training paradigms for IFMs and VFMs as frozen
feature extractors. Surprisingly, IFMs performs better than most VFMs, likely
due to the existing gap in spatial modeling capabilities between VFMs and
IFMs.

For the pre-training paradigms of the model, (1) The contrastive learning (CL)
based approach consistently excels in embedding evaluation. Due to CL’s
emphasis on the relationships between samples during training, DINOv2, which
focuses solely on vision, outperforms vision-language contrastive methods like
CLIP across multiple tasks.

(2) The effectiveness of masked video modeling is closely tied to the targets
it reconstructs or aligns with.  With higher semantic richness, it shows
progressive improvements in embedding quality for VideoMAE-L, V-JEPA-L, and
UMT-Lstage1.

(3) Vision-centric pretraining outperforms multi-modal pretraining in vision-
centric scenarios. Comparing UMT-Lstage1 and InternVideo2-1Bstage1 with their
multi-modal counterparts UMT-Lstage2 and InternVideo2-1Bstage2, the
introduction of visual-text pair data in multi-stage training does not enhance
performance in vision-centric scenarios. This is also consistent with the
performance differences observed between DINO and CLIP-style pre-training
methods.

Additionally, we assess the impact of fine-tuning on the embedding evaluation
of these pre-trained models. (4) Labels bring new semantic information or
disrupt existing finer-grained semantic information. The performance
variations after fine-tuning differ based on the pre-training strategy. For
UMT-Lstage1 and InternVideo2-1Bstage1, fine-tuning leads to a significant drop
in performance (-12.1 for UMT and -16.5 for InternVideo) due to the
introduction of more singular label information, which causes catastrophic
forgetting. In contrast, VideoMAE and VideoMAEv2 show substantial performance
gains (+14.5 and +25.8, respectively) because the low-level semantics learned
during pre-training are less abstract and benefit more from the addition of
high-level label information.

##  5 Conclusions

We have presented the VideoEval, a comprehensive benchmark suite for
efficiently evaluating the VFMs. To this end, we first establish the VidTAB
benchmark, which explores suitable evaluation tasks and protocols for VFMs
from the perspective of assessing their adaptability to unknown tasks with
limited samples. Additionally, we create the VidEB benchmark to evaluate the
capability of VFMs as direct feature embedding extractors for the downstream
tasks. Based on the VideoEval, we conduct a large-scale study involving 20
popular open-source vision foundation models, providing valuable insights for
future research directions.

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  ArXiv, abs/1810.04805, 2018.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [3] OpenAI.  Gpt-4 technical report.  ArXiv, abs/2303.08774, 2023.
  * [4] OpenAI.  Gpt-4v(ision) system card.  https://api.semanticscholar.org/CorpusID:263218031, 2023.
  * [5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [7] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In ECCV, 2016.
  * [8] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, 2019.
  * [9] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu.  TDN: Temporal difference networks for efficient action recognition.  In CVPR, 2021.
  * [10] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, 2021.
  * [11] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In CVPR, 2022.
  * [12] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In ICCV, 2021.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [14] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  ArXiv, abs/1705.06950, 2017.
  * [15] Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu.  Videomoco: Contrastive video representation learning with temporally adversarial examples.  In CVPR, 2021.
  * [16] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  In CVPR, 2022.
  * [17] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He.  Masked autoencoders as spatiotemporal learners.  NeurIPS, 2022.
  * [18] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  In NeurIPS, 2022.
  * [19] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan.  Bevt: Bert pretraining of video transformers.  CVPR, 2022.
  * [20] Tianhao Li and Limin Wang.  Learning spatiotemporal features via video and text pair discrimination.  CoRR, abs/2001.05691, 2020.
  * [21] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  In EMNLP, 2021.
  * [22] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu.  Video-text modeling with zero-shot transfer from contrastive captioners.  ArXiv, abs/2212.04979, 2022.
  * [23] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Jian Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Y. Qiao.  Internvid: A large-scale video-text dataset for multimodal understanding and generation.  ICLR, 2024.
  * [24] Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao.  Unmasked teacher: Towards training-efficient video foundation models.  In ICCV, 2023.
  * [25] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao.  Videomae v2: Scaling video masked autoencoders with dual masking.  In CVPR, 2023.
  * [26] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas.  V-jepa: Latent video prediction for visual representation learning.  ArXiv, abs/2404.08471, 2023.
  * [27] Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al.  Videoprism: A foundational visual encoder for video understanding.  ArXiv, abs/2402.13217, 2024.
  * [28] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao.  Internvideo: General video foundation models via generative and discriminative learning.  ArXiv, abs/2212.03191, 2022.
  * [29] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al.  Internvideo2: Scaling video foundation models for multimodal video understanding.  ArXiv, abs/2403.15377, 2024.
  * [30] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  The visual task adaptation benchmark.  Arxiv, abs/1910.04867, 2019.
  * [31] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In CVPR, 2021.
  * [32] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In ICML, 2019.
  * [33] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In ICCV, 2021.
  * [34] Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric P. Xing.  Learning robust global representations by penalizing local predictive power.  In NeurIPS, 2019.
  * [35] Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim.  Imagenet-x: Understanding model mistakes with factor of variation annotations.  In ICLR, 2023.
  * [36] Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, Rama Chellappa, Andrew Gordon Wilson, and Tom Goldstein.  Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks.  In NeurIPS, 2023.
  * [37] Liangzhe Yuan, Nitesh Bharadwaj Gundavarapu, Long Zhao, Hao Zhou, Yin Cui, Lu Jiang, Xuan Yang, Menglin Jia, Tobias Weyand, Luke Friedman, et al.  Videoglue: Video general understanding evaluation of foundation models.  ArXiv, abs/2307.03166, 2023.
  * [38] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao.  Mvbench: A comprehensive multi-modal video understanding benchmark.  CoRR, abs/2311.17005, 2023.
  * [39] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic.  The "something something" video database for learning and evaluating visual common sense.  In ICCV, 2017.
  * [40] Chunhui Gu, Chen Sun, Sudheendra Vijayanarasimhan, Caroline Pantofaru, David A. Ross, George Toderici, Yeqing Li, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik.  Ava: A video dataset of spatio-temporally localized atomic visual actions.  CVPR, 2017.
  * [41] Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev, Mustafa Suleyman, and Andrew Zisserman.  The kinetics human action video dataset.  ArXiv, abs/1705.06950, 2017.
  * [42] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The" something something" video database for learning and evaluating visual common sense.  In ICCV, 2017.
  * [43] Mathew Monfort, Bolei Zhou, Sarah Adel Bargal, Alex Andonian, Tom Yan, Kandan Ramakrishnan, Lisa M. Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, and Aude Oliva.  Moments in time dataset: One million videos for event understanding.  TPAMI, 2020.
  * [44] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A dataset of 101 human actions classes from videos in the wild.  Arxiv, abs/1212.0402, 2012.
  * [45] Fida Mohammad Thoker, Hazel Doughty, Piyush Bagad, and Cees GM Snoek.  How severe is benchmark-sensitivity in video self-supervised learning?  In ECCV, 2022.
  * [46] Andong Deng, Taojiannan Yang, and Chen Chen.  A large-scale study of spatiotemporal representation learning with a new benchmark on action recognition.  In ICCV, 2023.
  * [47] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [48] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In CVPR, 2021.
  * [49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: an open large-scale dataset for training next generation image-text models.  In NeurIPS, 2022.
  * [50] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In ICCV, 2021.
  * [51] Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu.  VALOR: vision-audio-language omni-perception pretraining model and dataset.  Arxiv, abs/2304.08345, 2023.
  * [52] Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu.  Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.  NeurIPS, 2024.
  * [53] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov.  Panda-70m: Captioning 70m videos with multiple cross-modality teachers.  Arxiv, abs/2402.19479, 2024.
  * [54] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [55] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv’e J’egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [56] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  ArXiv, abs/2304.07193, 2023.
  * [57] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang.  Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning.  In CVPR, 2023.
  * [58] Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Omnimae: Single model masked pretraining on images and videos.  In CVPR, 2023.
  * [59] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer.  Hiera: A hierarchical vision transformer without the bells-and-whistles.  In ICML, 2023.
  * [60] Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  All in one: Exploring unified video-language pre-training.  In CVPR, 2023.
  * [61] Feng Cheng, Xizi Wang, Jie Lei, David J. Crandall, Mohit Bansal, and Gedas Bertasius.  Vindlu: A recipe for effective video-and-language pretraining.  ArXiv, abs/2212.05051, 2022.
  * [62] Fida Mohammad Thoker, Hazel Doughty, Piyush Bagad, and Cees G. M. Snoek.  How severe is benchmark-sensitivity in video self-supervised learning?  In ECCV, 2022.
  * [63] Madeline Chantry Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh S Rawat.  A large-scale robustness analysis of video action recognition models.  In CVPR, 2023.
  * [64] Yuecong Xu, Jianfei Yang, Haozhi Cao, Kezhi Mao, Jianxiong Yin, and Simon See.  Arid: A new dataset for recognizing action in the dark.  In IJCAI, 2021.
  * [65] Hilde Kuehne, Ali Arslan, and Thomas Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  In CVPR, 2014.
  * [66] Klaus Schoeffmann, Heinrich Husslein, Sabrina Kletz, Stefan Petscharnig, Bernd Muenzer, and Christian Beecks.  Video retrieval in laparoscopic video recordings with dynamic content descriptors.  Multimedia Tools and Applications, 77:16813–16832, 2018.
  * [67] Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu.  Animal kingdom: A large and diverse dataset for animal behavior understanding.  In CVPR, 2022.
  * [68] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  Faceforensics++: Learning to detect manipulated facial images.  In ICCV, 2019.
  * [69] Syed Hammad Ahmed, Muhammad Junaid Khan, HM Qaisar, and Gita Sukthankar.  Malicious or benign? towards effective content moderation for children’s videos.  ArXiv, abs/2305.15551, 2023.
  * [70] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin.  Exploring video quality assessment on user generated contents from aesthetic and technical perspectives.  In ICCV, 2023.
  * [71] Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn.  Context-aware emotion recognition networks.  In ICCV, 2019.
  * [72] Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, and Ioannis Kompatsiaris.  Fivr: Fine-grained incident video retrieval.  IEEE Transactions on Multimedia, 21, 2019.
  * [73] Ed Pizzi, Giorgos Kordopatis-Zilos, Hiral Patel, Gheorghe Postelnicu, Sugosh Nagavara Ravindra, Akshay Gupta, Symeon Papadopoulos, Giorgos Tolias, and Matthijs Douze.  The 2023 video similarity dataset and challenge.  Computer Vision and Image Understanding, 2024.
  * [74] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In ICML, 2019.
  * [75] Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al.  Visual tuning.  ACM Computing Surveys, 2023.
  * [76] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In ECCV, 2022.
  * [77] Chen Xu, Haocheng Shen, Fengyuan Shi, Boheng Chen, Yixuan Liao, Xiaoxin Chen, and Limin Wang.  Progressive visual prompt learning with contrastive feature re-formation.  CoRR, abs/2304.08386, 2023.
  * [78] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li.  Parameter-efficient image-to-video transfer learning.  arXiv, abs/2206.13559, 2022.
  * [79] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li.  Aim: Adapting image models for efficient video action recognition.  In ICLR, 2023.
  * [80] Xinhao Li and Limin Wang.  Zeroi2v: Zero-cost adaptation of pre-trained transformers from image to video.  ArXiv, abs/2310.01324, 2023.
  * [81] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych.  Adapterhub: A framework for adapting transformers.  In EMNLP, 2020.
  * [82] Tengda Han, Weidi Xie, and Andrew Zisserman.  Self-supervised co-training for video representation learning.  In NeurIPS, 2020.
  * [83] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang.  Self-supervised spatiotemporal learning via video clip order prediction.  In CVPR, 2019.
  * [84] Tengda Han, Weidi Xie, and Andrew Zisserman.  Memory-augmented dense predictive coding for video representation learning.  In ECCV, 2020.
  * [85] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In ICCV, 2015.
  * [86] Xiao Wu, Alexander G Hauptmann, and Chong-Wah Ngo.  Practical elimination of near-duplicates from web video search.  In Proceedings of the 15th ACM international conference on Multimedia, pages 218–227, 2007.
  * [87] Yu-Gang Jiang, Yudong Jiang, and Jiajun Wang.  Vcdb: a large-scale database for partial copy detection in videos.  In ECCV, 2014.
  * [88] Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, et al.  The 2021 image similarity dataset and challenge.  arXiv preprint arXiv:2106.09672, 2021.
  * [89] Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  ArXiv, abs/2303.15389, 2023.
  * [90] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  In ICCV, 2023.
  * [91] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  In CVPR, 2023.

##  Appendix A Appendix

In this appendix, we provide more details of VideoEval from the following
aspects:

  * •

Details of our benchmark are in § A.1.

  * •

Details of training and evaluation, can be found in § A.2.

  * •

Ethics etatement of the datasets are in § A.3

  * •

Limitations and potential negative societal impacts are in § A.4

###  A.1 Details of Benchmark

Figure 5:  Examples of VidTAB. We present video examples for each task in
VidTAB, demonstrating that successfully completing these tasks requires VFMs
to possess strong generalization capabilities.

#### Examples of VidTAB.

As shown in Figure 5, we present some examples of tasks in VidTAB.

Table 6:  Task construction details of VidTAB. All videos are collected from
the public datasets for building tasks of VidTAB.

Task | Source | Num. test sample | Details of Task Construction
---|---|---|---
Action Recognition | ARID [64] | 2011 | We directly employ the original classification task definition. Specifically, 11 categories.
Action Recognition | BreakFast [65] | 822 | We directly employ the original classification task definition. Specifically, 12 categories.
Medical Surgery | SurgicalActions160 [66] | 96 | We directly employ the original classification task definition. Specifically, 16 categories.
Animal Behavior |  Animal Kingdom [67] |  2268 | Since the annotations in this dataset included multiple labels, we filtered out all categories with only single labels and then
selected categories with more than 150 samples. This resulted in a final set
of 12 categories.
Fake Face |  FaceForensics++ [68] |  1800 | We used the original 1000 videos as positive samples. Then, we divided the original videos into five parts and used the Deepfakes,
Face2Face, FaceShifter, FaceSwap, and NeuralTextures methods to generate 1000
negative samples by face-swapping. We then
selected 1800 of these samples as the test set and the remaining as the
training set.
Harmful Content |  mob [69] |  1661 | We categorized videos into three classes based on their content: those containing fast repetitive movements and violence activities,
those containing unpleasant appearances and obscene scenes, and those
containing no malicious information at all. This resulted in a
three-class classification task.
Quality Assess |  DOVER [70] |  724 | To convert the task into a classification problem, we sorted the "overall score" label and divided the videos into positive and negative
samples, with the top and bottom 40% constituting the respective categories.
Emotion Analysis | CAER [71] | 3953 | We directly employ the original classification task definition. Specifically, 7 categories.
|  |  |

#### Details of VidTAB.

The detals of task construction are presented in Table 6. For each category in
one task, we sample 4, 16, and 100 samples, respectively. Given the limited
volume of medical surgery data, we only sample 4 samples from each category
for few-shot evaluation. To mitigate the impact of randomness, we sampled two
sets of data for four tasks and obtained the benchmark results. We found that
the randomness of sampling had negligible effects on the final rankings of
VFMs in the benchmark.

###  A.2 Details of Training and Evaluation

#### Code and annotations

We release the evaluation code and the modified annotation file on
https://github.com/leexinhao/VideoEval, see README.md for details.

#### Checkpoints of Evaluation Models

We provide checkpoints of the models we evaluate for reproducibility of our
results.

  * •

CLIP [5]: https://huggingface.co/openai/clip-vit-large-patch14

  * •

EVA-CLIP [5]: https://huggingface.co/QuanSun/EVA-CLIP

  * •

ViCLIP [23]: https://github.com/OpenGVLab/InternVideo/tree/main/Data/InternVid

  * •

InternVideo2 [29]:
https://huggingface.co/collections/OpenGVLab/internvideo2-6618ccb574bd2f91410df5cd

  * •

SigLiP [90]: https://huggingface.co/google/siglip-so400m-patch14-384

  * •

DINOv2 [56]: https://huggingface.co/facebook/dinov2-giant

  * •

VideoMAE [18]: https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md

  * •

VideoMAEv2 [25]:
https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md

  * •

UMT [24]: https://github.com/OpenGVLab/unmasked_teacher

  * •

V-JEPA [26]: https://github.com/facebookresearch/jepa

#### Trainging strategies.

Specific hyperparameter configurations are available in the configs provided
in our code repository. In essence, we train all models for 25 epochs using a
similar training strategy, employing the Adam optimizer, a learning rate of
5e-5, and only utilizing RandomResizedCrop for data augmentation. And we use a
single clip to obtain the final evaluation performance.

#### Total amount of compute and the type of resources used.

Leveraging the low cost of our evaluation protocol, we conducted each
experiment involving a single VFM and a single task on one A100-80G GPU. We
performed approximately 300 such experiments, each taking around 1-2 hours,
resulting in a total of around 400 GPU hours.

###  A.3 Ethics Statement

#### License of the datasets

The dataset we are using is collected from publicly accessible sources, all
licensed under Creative Commons (CC-BY) or other open-source licenses. We have
diligently followed all legal requirements to integrate this data into our
research, emphasizing the importance of transparency in data licensing for
proper attribution and appropriate use. Although we have taken steps to ensure
the inclusion of suitable content, we recognize that some problematic content
may still exist. If you encounter any such content, please notify us
immediately so we can take corrective action to maintain a dataset free from
inappropriate material. We are dedicated to maintaining a high-quality,
ethically responsible dataset and pledge to uphold principles of privacy and
transparency in all our work.

#### Privacy or safety concerns in video

For personally identifiable information or offensive content in video, our
data collection sources have been carefully considered, and we believe these
issues are not present. However, if you discover any oversights, please do not
hesitate to contact us promptly.

###  A.4 Limtiations and Societal Impacts

#### Limitations.

First, due to the limitations of diversity and accuracy in our video sources
and annotations, which were gathered from public resources, we plan to further
enrich the task in the future by incorporating manual annotations and self-
collected data. Secondly, considering the evaluation cost and simplicity, we
currently only evaluate tasks like classification and retrieval, which
primarily rely on VFMs’ global information extraction capabilities. We have
not yet considered tasks like spatio-temporal action detection and temporal
grounding, which assess other aspects of VFMs’ capabilities. We will expand
the scope of evaluation in the future.

#### Potential negative societal impacts.

While our evaluation includes tasks like synthetic video recognition and
harmful information recognition, these serve only as indicators of the model’s
overall performance in this area and cannot be used to accurately evaluate the
actual performance of a specific task. If researchers or engineers in society
attempt to use VFMs to perform these specific tasks, our benchmark can serve
as a reference for their choice of VFMs, but it cannot be used as the final
standard for evaluating that task. Otherwise, it may have negative impacts on
the corresponding real-world applications.