in tackling Inference-based questions that require a deeper understanding of
the video content, including both dense spatio-temporal relationships (how
objects and actions unfold over time) and causal relationships (understanding
cause-and-effect).

Overall, Multi-modal video tasks like retrieval and VQA remain far from
achieving their full potential. A deeper analysis reveals a more critical
issue: the limited scale of video-text datasets compared to image-text
datasets. This data scarcity hinders effective learning for video foundation
models. Further compounding the problem is the data quality of many video-text
datasets, often generated by scraping online videos and text, which introduces
significant noise. The presence of noisy data further complicates the already
challenging task of multi-modal learning for video models.

####  6.2.2. Video Captioning.

Table 8 (right column) compares video captioning performance on various
datasets (MSRVTT (Xu et al., 2016), MSVD (Chen and Dolan, 2011), and YouCook2
(Zhou et al., 2018) datasets). The results show specialization: mPLUG-2 (Xu et
al., 2023b) (image-video-text) performs well on MSRVTT (Xu et al., 2016),
MaMMUT (Kuo et al., 2023) (image-video-text) excels on MSVD (Chen and Dolan,
2011), and VAST (Chen et al., 2023d) (video-audio-text) dominates YouCook2
(Zhou et al., 2018) (cooking videos). This suggests the importance of
incorporating relevant modalities for specific video content. Notably, VAST’s
strong performance on YouCook2 highlights the informative nature of audio in
video captioning. Overall, the superiority of unified models across datasets
underscores the potential of exploring additional modalities for captioning
tasks.

Discussion. While Table 8 focuses on video captioning (Vinyals et al., 2016;
Gu et al., 2017; Xu et al., 2016), dense video captioning (Chang et al., 2022;
Deng et al., 2021), which generates multiple captions describing events
throughout a video, remains an unsolved challenge. This task demands a deeper
understanding of spatio-temporal relationships, an area where ViFMs are still
under development.

###  6.3. Video Content Generation and Manipulation

We discuss Text-To-Video(T2V) generation tasks, generates video given its
textual description. For T2V generation GANs, auto-regressive, and diffusion
models are the dominant choice.

Table 9. Comparing the fine-tuned performance of state-of-the-art (SOTA) generalist and specialist models for zero-shot Text-To-Video Generation on MSR-VTT (Xu et al., 2016) and UCF-101 (Soomro et al., 2012) dataset. The highlighted entries indicate the best performing methods. | Method | MSR-VTT | UCF-101
---|---|---|---
|  ClipSIM ($\uparrow$) |  FVD ($\downarrow$) |  FVD ($\downarrow$) |  IS ($\uparrow$)
Gen |  InternVid (Wang et al., 2023f) | 0.2951 | - | 616.51 | 21.04
Make-A-Video (Singer et al., 2022) | 0.3049 | - | 367.23 | 33.00
|  PYoCo (Ge et al., 2023) | - | - | 355.19 | 47.76
|  SVD (Blattmann et al., 2023a) | - | - | 242.02 | -
|  VideoPoet (Kondratyuk et al., 2023) | 0.3049 | 213.00 | 355.00 | 38.44
|  VideoLaVIT (Jin et al., 2024) | 0.3010 | 169.51 | 274.96 | 37.96
|  VideoLDM (Blattmann et al., 2023c) | 0.2929 | - | 550.61 | 33.45
|  VideoComposer (Wang et al., 2024c) | 0.2932 | 580.00 | - | -
Special |  CogVideo (Hong et al., 2022) | 0.2631 | 1294.00 | 702 | 25.27
MagicVideo (Zhou et al., 2022c) | - | 998.00 | 655 | -
VideoFactory (Wang et al., 2023j) | 0.3005 | - | 410 | -

####  6.3.1. Text-To-Video Generation

Table 9 showcases the performance of different models on the Text-to-Video
generation task, tested on MSR-VTT (Xu et al., 2016) and UCF101 (Soomro et
al., 2012) datasets.

Looking at the MSR-VTT (Xu et al., 2016) results, VideoPoet (Kondratyuk et
al., 2023), a model utilizing a unified image-video framework, stands out by
outperforming all others across both CLIPSim (Wu et al., 2021a) and FVD
(Unterthiner et al., 2018) metrics. However, on UCF101, different strengths
emerge. InternVid, another ViFM, takes the lead when using the IS (Saito et
al., 2020) metric. Interestingly, for this dataset, SVD (Blattmann et al.,
2023a), a diffusion-based foundation model, surpasses its competitors in terms
of FVD (Unterthiner et al., 2018). This indicates SVD’s (Blattmann et al.,
2023a) ability to generate videos with exceptionally high visual fidelity
within the context of UCF101 (Soomro et al., 2012).

Discussion. Despite significant progress in video generation over the past few
years, real-world integration remains challenging. High computational
complexity and time-consuming processes are major hurdles. Generating a single
minute of video can take hours. However, generative models are continuously
improving their ability to maintain temporal consistency. This paves the way
for integration with existing ViFMs to enhance the representation space.

(a) Fine-tuning results of Image-based, Video-based, and Universal
foundational models against Specialist models for video action recognition
task on k400 (Kay et al., 2017) (left), JHMDB (Kuehne et al., 2011) (center)
and SSv2 (Goyal et al., 2017) (right) datasets. (b) Fine-tuning results of
Image-based, Video-based, and Universal foundational models against Specialist
models for video retrieval task on MSRVTT (Xu et al., 2016) (left), DIDEMO
(Anne Hendricks et al., 2017) (center) and LSMDC (Rohrbach et al., 2015)
(right) datasets.

(c) Fine-tuning results of Image-based, Video-based, and Universal
foundational models against Specialist models for video captioning task on
MSRVTT (Xu et al., 2016) (left), MSVD (Chen and Dolan, 2011) (center) and
YouCook2 (Zhou et al., 2018) (right) datasets.

###  6.4. Analysis of Results

Figure 8(a), 8(b), and 8(c) presents a general comparison of image-based
foundational models, video-based models, unified-models and task-specific
specialist models. Note that task-specific models are finetuned for a
particular task being analyzed and have been shown by a separate color (red).
Across all the tasks a general trend is being observed: i.e. Image based
foundational models perform better than video based models. We attribute this
to the fact that there is a large amount of pretraining data available for
image-based models whereas video-based models significantly lag behind this
aspect. A notable exception is video models performing better on SSV2 in
Action Recognition: this might be because SSV2 contains challenging action
classes whose definition changes with the order of frames and requires
understanding the arrow of time. Video Modelling helps establish much needed
temporal correspondance in this case.

Across all the tasks, unified models perform much better than both image/video
based models: this shows that joint representational learning of image/video
modalities helps achieve significant performance gains. Finally, we note that
all unified models outperform task-specific models. This highlights a
desirable generalization behaviour: we are able to train foundational models
on massive datasets, outperform task-specific specialists, as well as adapt a
given model for several downstream tasks with minimal compute.

##  7\. Challenges and Future Directions

Multi-modal video foundation models encounter challenges in accessing
extensive training data, limiting their development compared to image-text
models. Existing video-text datasets are often small and noisy, hindering
robust representation learning. Future research must prioritize creating high-
quality, large-scale video-text datasets and exploring data cleaning,
augmentation, and alternative pretraining methods. Additionally, existing
video foundation models struggle with temporal consistency, object-centric
tasks, and adaptability to diverse contexts. However, recent advancements,
such as the generative auto-regressive model SORA (Brooks et al., 2024), offer
insights into addressing temporal consistency issues. ViFMs can draw
inspiration from such models to improve their performance. Integrating diverse
self-supervised learning techniques (Madan et al., 2024) and adapter modules
provides avenues for domain adaptation and various video understanding tasks.
The subsequent subsections present possible future research directions in the
domain of ViFMs.

###  7.1. Addressing Ethical Considerations.

As Video Foundation Models (ViFMs) find increasing application in real-world
scenarios, akin to ChatGPT (OpenAI, 2023a) and Amazon SageMaker (Perrone et
al., 2021), addressing ethical concerns becomes crucial. Future research
should focus on mitigating biases through debiasing datasets and fairness
metrics, promoting transparency and explainability to build trust, and
establishing responsible use guidelines throughout the ViFM development
lifecycle. By actively addressing these ethical considerations, we can ensure
ViFMs are deployed responsibly, maximizing their positive impact on real-world
applications.

###  7.2. Long-Form Video Understanding.

Achieving long-form video understanding (Mangalam et al., 2023) with ViFMs
presents a significant challenge due to the extensive memory requirements for
processing extended sequences. Recent efforts have recognized this hurdle and
begun to explore solutions, such as memory consolidation mechanisms (Song et
al., 2023; Balažević et al., 2024) and memory-efficient attention (Liu et al.,
2024). However, to truly unlock the potential of ViFMs in this domain,
integrating causal reasoning could be pivotal. By incorporating causal
reasoning (Yi et al., 2019) into ViFMs, we can enhance their ability to
comprehend extended video content by enabling them to answer fundamental
questions like ”why,” ”what next,” and ”what if.” This deeper understanding
facilitated by causal reasoning could revolutionize long-form video
understanding, allowing ViFMs to recognize event order, direction of
causality, and detailed relationships between actors, actions, and objects.
Moreover, integrating causal reasoning may enhance the robustness (Schiappa et
al., 2023a) of ViFMs and improve their ability to handle occlusions (Modi et
al., 2024) and other challenges commonly encountered in real-world video data.
Therefore, while addressing the memory constraints is crucial, integrating
causal reasoning into ViFMs offers a promising avenue for achieving
comprehensive long-form video understanding.

###  7.3. Viewpoint Invariance.

ViFMs excel in traditional video settings (i.e., third-person viewpoint), but
limitations arise in understanding different viewpoints like egocentric
(Grauman et al., 2022) or birds-eye view (Peng et al., 2023a). Future research
can delve into viewpoint-invariant representations for dynamic scenes.
Inspired by the human ability to mentally rotate objects, neural
representations like NeRF (Mildenhall et al., 2020) can be explored to encode
a continuous 3D representation within ViFMs. Additionally, methods that
project an agent’s limited view to a common reference frame (Philion and
Fidler, 2020) and establish correspondences across views hold promise for
learning robust representations despite dynamic exploration. By pursuing these
directions, ViFMs can be equipped to handle different viewpoints and varying
camera paths, ultimately leading to significant advancements in action
localization and a deeper understanding of dynamic video content.

###  7.4. Domain Adaptation

Domain adaptation refers to the ability of models to perform well in new
environments (lighting, locations, etc.) or domains that differ from those
they were trained on. Some studies (Schiappa et al., 2023b; Kumar et al.,
2023) provide directions to make traditional video models robust against such
changes by suggesting augmentations (Zara et al., 2024) and specific tuning
(Kareer et al., 2024). However, research in domain adaptation for foundation
models is still lacking. To seamlessly integrate these powerful models into
real-world applications, future work should explore methods to make ViFMs
robust against domain shifts.

###  7.5. Improving Efficiency

Despite promising results, ViFM’s resource demands pose a challenge for edge
deployment. These models often have hundreds of millions to billions of
parameters, leading to longer training and inference times. This consequently
limits their deployment on edge devices for real-time inference. To address
this challenge, a key future direction in the integration of video foundation
models for edge devices involves developing efficient deployment strategies to
overcome resource constraints and enable seamless inferencing at the edge,
citing (Lin et al., 2023e) as an example. This entails exploring novel
approaches to optimize model architecture and reduce computational overhead,
as well as investigating innovative techniques for model compression and
quantization to facilitate deployment on resource-constrained edge devices
without compromising performance. By addressing these challenges, researchers
can pave the way for widespread adoption of video foundation models in edge
computing environments, unlocking their potential to power a diverse range of
high-impact applications across industries.

##  8\. Conclusion

This survey offers a comprehensive and, to the best of our knowledge, the
first-of-its-kind in-depth exploration of Video Foundation Models (ViFMs). We
commenced by establishing a foundation with discussions on video understanding
tasks, relevant architectures, pretraining datasets, and approaches for ViFM
pretraining. We categorize core methodologies for ViFM creation into three
primary techniques: Inflating Image Models, Video-based models (focusing on
video or video-text pretraining), and Unified Image-Video ViFMs (applicable to
both image and video tasks). By comparing the performance of various ViFMs on
video tasks and offering insights based on methodologies and results, we aim
to equip the research community with a comprehensive overview of existing
ViFMs, while also highlighting critical areas for future exploration. This, we
believe, will foster further advancements in video modeling and unlock the
full potential of ViFMs.

##  9\. Acknowledgements

This project is funded by Innovation Funds, Denmark.

## References

  * (1)
  * Ahmad et al. (2023) Shahzad Ahmad, Sukalpa Chanda, and Yogesh S Rawat. 2023.  EZ-CLIP: Efficient Zeroshot Video Action Recognition.  _arXiv:2312.08010_ (2023).
  * Akbari et al. (2024) Hassan Akbari, Dan Kondratyuk, Yin Cui, Rachel Hornung, Huisheng Wang, and Hartwig Adam. 2024.  Alternating gradient descent and mixture-of-experts for integrated multimodal perception. In _Proceedings of NeurIPS_.
  * Akbari et al. (2021) Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In _Proceedings of NeurIPS_. 24206–24221.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022\.  Flamingo: a visual language model for few-shot learning. In _Proceedings of NeurIPS_. 23716–23736.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023\.  Palm 2 technical report.  _arXiv:2305.10403_ (2023).
  * Anne Hendricks et al. (2017) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017.  Localizing moments in video with natural language. In _Proceedings of ICCV_. 5803–5812.
  * Arnab et al. (2021) Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021.  Vivit: A video vision transformer. In _Proceedings of ICCV_. 6836–6846.
  * Ashutosh et al. (2023) Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. 2023.  HierVL: Learning Hierarchical Video-Language Embeddings. In _Proceedings of CVPR_. 23066–23078.
  * Athar et al. (2023) Ali Athar, Jonathon Luiten, Paul Voigtlaender, Tarasha Khurana, Achal Dave, Bastian Leibe, and Deva Ramanan. 2023.  Burst: A benchmark for unifying object recognition, segmentation and tracking in video. In _Proceedings of WACV_. 1674–1683.
  * Awais et al. (2023) Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2023.  Foundational models defining a new era in vision: A survey and outlook.  _arXiv:2307.13721_ (2023).
  * Bagad et al. (2023) Piyush Nitin Bagad, Makarand Tapaswi, and Cees G. M. Snoek. 2023.  Do Video-Language Foundation Models have a Sense of Time?. In _Proceedings of ICLR-Workshop on Mathematical and Empirical Understanding of Foundation Models_.
  * Bain et al. (2020) Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. 2020.  Condensed movies: Story based retrieval with contextual embeddings. In _Proceedings of ACCV_.
  * Bain et al. (2021) Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021.  Frozen in time: A joint video and image encoder for end-to-end retrieval. In _Proceedings of CVPR_. 1728–1738.
  * Balažević et al. (2024) Ivana Balažević, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. 2024.  Memory Consolidation Enables Long-Context Video Understanding.  _arXiv:2402.05861_ (2024).
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the ACL-W_. 65–72.
  * Bar-Tal et al. (2024) Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. 2024\.  Lumiere: A space-time diffusion model for video generation.  _arXiv:2401.12945_ (2024).
  * Ben-Shabat et al. (2021) Yizhak Ben-Shabat, Xin Yu, Fatemeh Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hongdong Li, and Stephen Gould. 2021.  The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. In _Proceedings of WACV_. 847–859.
  * Bernardin and Stiefelhagen (2008) Keni Bernardin and Rainer Stiefelhagen. 2008.  Evaluating multiple object tracking performance: the clear mot metrics.  _EURASIP Journal on Image and Video Processing_ 2008 (2008), 1–10.
  * Bertasius et al. (2021) Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021.  Is space-time attention all you need for video understanding?. In _Proceedings of ICML_.
  * Blattmann et al. (2023a) Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. 2023a.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _arXiv:2311.15127_ (2023).
  * Blattmann et al. (2023b) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023b.  Align your latents: High-resolution video synthesis with latent diffusion models. In _Proceedings of CVPR_. 22563–22575.
  * Blattmann et al. (2023c) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023c.  Align your latents: High-resolution video synthesis with latent diffusion models. In _Proceedings of CVPR_. 22563–22575.
  * Bogolin et al. (2022) Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. 2022.  Cross modal retrieval with querybank normalisation. In _Proceedings of CVPR_. 5194–5205.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv_ (2021).
  * Brooks et al. (2024) Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024.  Video generation models as world simulators.  https://openai.com/sora.
  * Buch et al. (2022) Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022.  Revisiting the” video” in video-language understanding. In _Proceedings of CVPR_. 2917–2927.
  * Caba Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  Activitynet: A large-scale video benchmark for human activity understanding. In _Proceedings of CVPR_. 961–970.
  * Carreira et al. (2018) Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. 2018.  A short note about kinetics-600.  _arXiv preprint arXiv:1808.01340_ (2018).
  * Carreira et al. (2019) Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. 2019.  A short note on the kinetics-700 human action dataset.  _arXiv preprint arXiv:1907.06987_ (2019).
  * Carreira and Zisserman (2017a) Joao Carreira and Andrew Zisserman. 2017a.  Quo vadis, action recognition? a new model and the kinetics dataset. In _proceedings of CVPR_. 6299–6308.
  * Carreira and Zisserman (2017b) Joao Carreira and Andrew Zisserman. 2017b.  Quo vadis, action recognition? a new model and the kinetics dataset. In _Proceedings of CVPR_. 6299–6308.
  * Castro et al. (2022a) Santiago Castro, Naihao Deng, Pingxuan Huang, Mihai Burzo, and Rada Mihalcea. 2022a.  In-the-Wild Video Question Answering. In _Proceedings of International Conference on Computational Linguistics (ICCL)_. 5613–5635.
  * Castro and Heilbron (2022) Santiago Castro and Fabian Caba Heilbron. 2022.  FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks.  _arXiv_ (2022).
  * Castro et al. (2022b) Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan Stroud, and Rada Mihalcea. 2022b.  FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework. In _Proceedings of ACL_. 2925–2940.
  * Chai et al. (2023) Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. 2023.  Stablevideo: Text-driven consistency-aware diffusion video editing. In _Proceedings of ICCV_. 23040–23050.
  * Chang et al. (2022) Zhi Chang, Dexin Zhao, Huilin Chen, Jingdan Li, and Pengfei Liu. 2022.  Event-centric multi-modal fusion method for dense video captioning.  _Neural Networks_ 146 (2022), 120–129.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In _Proceedings of CVPR_. 3558–3568.
  * Chatterjee et al. (2024) Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. 2024.  Opening the vocabulary of egocentric actions. In _Proceedings of NeurIPS_.
  * Chen and Dolan (2011) David Chen and William B Dolan. 2011.  Collecting highly parallel data for paraphrase evaluation. In _Proceedings of ACL:HLT_. 190–200.
  * Chen et al. (2023a) Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. 2023a.  X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages.  _arXiv:2305.04160_ (2023).
  * Chen et al. (2023f) Guangyi Chen, Xiao Liu, Guangrun Wang, Kun Zhang, Philip HS Torr, Xiao-Ping Zhang, and Yansong Tang. 2023f.  Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer. In _Proceedings of ICCV_. 13945–13955.
  * Chen et al. (2020b) Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020b.  Vggsound: A large-scale audio-visual dataset. In _Proceedings of ICASSP_. IEEE, 721–725.
  * Chen et al. (2023b) Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. 2023b.  VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset.  _arXiv_ (2023).
  * Chen et al. (2023c) Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, and Jing Liu. 2023c.  COSA: Concatenated Sample Pretrained Vision-Language Foundation Model.  _arXiv_ (2023).
  * Chen et al. (2023d) Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. 2023d.  VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset.  _arXiv_ (2023).
  * Chen et al. (2023h) Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. 2023h.  BEATs: Audio Pre-Training with Acoustic Tokenizers. In _Proceedings of ICML_. 5178–5193.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a.  Simclr: A simple framework for contrastive learning of visual representations. In _Proceedings of ICLR_.
  * Chen et al. (2023e) Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. 2023e.  Motion-conditioned diffusion model for controllable video synthesis.  _arXiv:2304.14404_ (2023).
  * Chen et al. (2023i) Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. 2023i.  Control-a-video: Controllable text-to-video generation with diffusion models.  _arXiv:2305.13840_ (2023).
  * Chen et al. (2023g) Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. 2023g.  Pali-3 vision language models: Smaller, faster, stronger.  _arXiv:2310.09199_ (2023).
  * Chen et al. (2022) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022\.  Pali: A jointly-scaled multilingual language-image model.  _arXiv:2209.06794_ (2022).
  * Cheng et al. (2023c) Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. 2023c.  Vindlu: A recipe for effective video-and-language pretraining. In _Proceedings of CVPR_. 10739–10750.
  * Cheng et al. (2023b) Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. 2023b.  Tracking anything with decoupled video segmentation. In _Proceedings of ICCV_. 1316–1326.
  * Cheng et al. (2021) Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. 2021.  Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss.  _arXiv:2109.04290_ (2021).
  * Cheng et al. (2023a) Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. 2023a.  Segment and Track Anything.  _arXiv_ (2023).
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023\.  Palm: Scaling language modeling with pathways.  _JMLR_ 24, 240 (2023), 1–113.
  * Computer (2023) Together Computer. 2023.  RedPajama: an Open Dataset for Training Large Language Models.  https://github.com/togethercomputer/RedPajama-Data
  * Dai et al. (2023a) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023a.  InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning.  _arXiv preprint arXiv:2305.06500_ (2023).
  * Dai et al. (2023b) Zuozhuo Dai, Fangtao Shao, Qingkun Su, Zilong Dong, and Siyu Zhu. 2023b.  Fine-grained Text-Video Retrieval with Frozen Image Encoders.  _arXiv preprint arXiv:2307.09972_ (2023).
  * Damen et al. (2018) Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2018\.  Scaling egocentric vision: The epic-kitchens dataset. In _Proceedings of ECCV_. 720–736.
  * Dave et al. (2020) Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. 2020.  Tao: A large-scale benchmark for tracking any object. In _Proceedings of ECCV_. 436–454.
  * Demir et al. (2021) Ugur Demir, Yogesh S Rawat, and Mubarak Shah. 2021.  Tinyvirat: Low-resolution video action recognition. In _Proceedings of ICPR_. 7387–7394.
  * Dendorfer et al. (2020) Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Qinfeng Shi, Daniel Cremers, Ian D. Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taix’e. 2020.  MOT20: A benchmark for multi object tracking in crowded scenes.  _ArXiv_ (2020).
  * Deng et al. (2023) Chaorui Deng, Qi Chen, Pengda Qin, Da Chen, and Qi Wu. 2023.  Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval. In _Proceedings of ICCV_. 15648–15658.
  * Deng et al. (2021) Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu. 2021.  Sketch, ground, and refine: Top-down dense video captioning. In _Proceedings of CVPR_. 234–243.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _Proceedings of CVPR_. 248–255.
  * Doersch et al. (2022) Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. 2022.  Tap-vid: A benchmark for tracking any point in a video. In _Proceedings of NeurIPS_. 13610–13626.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Duarte et al. (2018) Kevin Duarte, Yogesh Rawat, and Mubarak Shah. 2018.  Videocapsulenet: A simplified network for action detection. In _Proceedings of NeurIPS_ , Vol. 31.
  * Duarte et al. (2019) Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. 2019.  Capsulevos: Semi-supervised video object segmentation using capsule routing. In _Proceedings of the IEEE/CVF international conference on computer vision_. 8480–8489.
  * Engin and Avrithis (2023) Deniz Engin and Yannis Avrithis. 2023.  Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts. In _Proceedings of ICCV_. 2804–2810.
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021.  Taming transformers for high-resolution image synthesis. In _Proceedings of CVPR_. 12873–12883.
  * Fabbri et al. (2021) Matteo Fabbri, Guillem Brasó, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljoša Ošep, Simone Calderara, Laura Leal-Taixé, and Rita Cucchiara. 2021.  MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?. In _Proceedings of ICCV_. 10849–10859.
  * Fan et al. (2019) Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. 2019.  Lasot: A high-quality benchmark for large-scale single object tracking. In _Proceedings of CVPR_. 5374–5383.
  * Fang et al. (2021) Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. 2021.  Clip2video: Mastering video-text retrieval via image clip.  _arXiv:2106.11097_ (2021).
  * Fang et al. (2022) Han Fang, Pengfei Xiong, Luhui Xu, and Wenhan Luo. 2022.  Transferring image-clip to video-text retrieval via temporal relations.  _IEEE Transactions on Multimedia_ (2022).
  * Fang et al. (2023b) Han Fang, Zhifei Yang, Yuhan Wei, Xianghao Zang, Chao Ban, Zerun Feng, Zhongjiang He, Yongxiang Li, and Hao Sun. 2023b.  Alignment and Generation Adapter for Efficient Video-Text Understanding. In _Proceedings of ICCV_. 2791–2797.
  * Fang et al. (2023a) Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023a.  Eva-02: A visual representation for neon genesis.  _arXiv:2303.11331_ (2023).
  * Feichtenhofer et al. (2022) Christoph Feichtenhofer, Yanghao Li, and Kaiming He. 2022.  Masked autoencoders as spatiotemporal learners. In _Proceedings of NeurIPS_ , Vol. 35. 35946–35958.
  * Feichtenhofer et al. (2016) Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. 2016.  Convolutional two-stream network fusion for video action recognition. In _Proceedings of CVPR_. 1933–1941.
  * Fu et al. (2022) Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. 2022.  VIOLET: End-to-End Video-Language Transformers with Masked Visual-token Modeling.  _arXiv_ (2022).
  * Fu et al. (2023a) Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. 2023a.  An empirical study of end-to-end video-language transformers with masked visual modeling. In _Proceedings of CVPR_. 22898–22909.
  * Fu et al. (2023b) Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng-Yang Fu, Jong-Chyi Su, William Yang Wang, and Sean Bell. 2023b.  Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. In _Proceedings of CVPR_. 10681–10692.
  * Gavrilyuk et al. (2018) Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. 2018.  Actor and action video segmentation from a sentence. In _Proceedings of CVPR_. 5958–5966.
  * Ge et al. (2023) Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. 2023.  Preserve your own correlation: A noise prior for video diffusion models. In _Proceedings of ICCV_. 22930–22941.
  * Ge et al. (2022a) Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. 2022a.  Bridging video-text retrieval with multiple choice questions. In _Proceedings of CVPR_. 16167–16176.
  * Ge et al. (2022b) Yuying Ge, Yixiao Ge, Xihui Liu, Jinpeng Wang, Jianping Wu, Ying Shan, Xiaohu Qie, and Ping Luo. 2022b.  Miles: Visual bert pre-training with injected language semantics for video-text retrieval. In _Proceedings of ECCV_. 691–708.
  * Geiger et al. (2013) Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013.  Vision meets robotics: The KITTI dataset.  _International Journal of Robotics Research (IJRR)_ 32 (2013), 1231 – 1237.
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  Audio set: An ontology and human-labeled dataset for audio events. In _Proceedings of ICASSP_. 776–780.
  * Georgescu et al. (2023) Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, and Anurag Arnab. 2023.  Audiovisual masked autoencoders. In _Proceedings of ICCV_. 16144–16154.
  * Girdhar et al. (2023) Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023.  Omnimae: Single model masked pretraining on images and videos. In _Proceedings of CVPR_. 10406–10417.
  * Girdhar et al. (2022) Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. 2022.  Omnivore: A single model for many visual modalities. In _Proceedings of CVPR_. 16102–16112.
  * Gong et al. (2021) Yuan Gong, Yu-An Chung, and James Glass. 2021.  Ast: Audio spectrogram transformer.  _arXiv preprint arXiv:2104.01778_ (2021).
  * Gong et al. (2023) Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. 2023.  Contrastive Audio-Visual Masked Autoencoder.  _arXiv_ (2023).
  * Gorti et al. (2022) Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. 2022.  X-pool: Cross-modal language-video attention for text-video retrieval. In _Proceedings of CVPR_. 5006–5015.
  * Goyal et al. (2017) Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. 2017\.  The” something something” video database for learning and evaluating visual common sense. In _Proceedings of the ICCV_. 5842–5850.
  * Grauman et al. (2022) Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. 2022\.  Ego4d: Around the world in 3,000 hours of egocentric video. In _Proceedings of CVPR_. 18995–19012.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020\.  Bootstrap your own latent – a new approach to self-supervised learning. In _Proceedings of NeurIPS_. 21271–21284.
  * Gu et al. (2018) Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. 2018\.  Ava: A video dataset of spatio-temporally localized atomic visual actions. In _Proceedings of CVPR_. 6047–6056.
  * Gu et al. (2017) Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2017.  Non-autoregressive neural machine translation.  _arXiv:1711.02281_ (2017).
  * Gu et al. (2023) Xin Gu, Guang Chen, Yufei Wang, Libo Zhang, Tiejian Luo, and Longyin Wen. 2023.  Text with knowledge graph augmented transformer for video captioning. In _Proceedings of CVPR_. 18941–18951.
  * Gupta et al. (2019) Agrim Gupta, Piotr Dollar, and Ross Girshick. 2019.  Lvis: A dataset for large vocabulary instance segmentation. In _Proceedings of CVPR_. 5356–5364.
  * Gupta et al. (2018) Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. 2018.  Imagine this! scripts to compositions to videos. In _Proceedings of ECCV_. 598–613.
  * Hayes et al. (2022) Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, and Devi Parikh. 2022.  Mugen: A playground for video-audio-text multimodal understanding and generation. In _Proceedings of ECCV_. 431–449.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022.  Masked autoencoders are scalable vision learners. In _Proceedings of CVPR_. 16000–16009.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  _IEEE TPAMI_ 37, 3 (2016), 630–643.
  * Hernandez et al. (2023) Jefferson Hernandez, Ruben Villegas, and Vicente Ordonez. 2023.  Visual Representation Learning from Unlabeled Video using Contrastive Masked Autoencoders.  _arXiv_ (2023).
  * Ho et al. (2022) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. 2022\.  Imagen video: High definition video generation with diffusion models.  _arXiv:2210.02303_ (2022).
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models. In _Proceedings of NeurIPS_. 6840–6851.
  * Hong et al. (2022) Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2022.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  _arXiv:2205.15868_ (2022).
  * Hori et al. (2017) Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. 2017.  Attention-based multimodal fusion for video description. In _Proceedings of ICCV_. 4193–4202.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP. In _Proceedings of ICML_. 2790–2799.
  * Hu et al. (2023) Zhiming Hu, Angela Ning Ye, Salar Hosseini Khorasgani, and Iqbal Mohomed. 2023.  AdaCLIP: Towards Pragmatic Multimodal Video Retrieval. In _Proceedings of ACM-MM_. 5623–5633.
  * Huang et al. (2023c) Bingkun Huang, Zhiyu Zhao, Guozhen Zhang, Yu Qiao, and Limin Wang. 2023c.  MGMAE: Motion Guided Masking for Video Masked Autoencoding. In _Proceedings of ICCV_. 13493–13504.
  * Huang et al. (2023b) Jingjia Huang, Yinan Li, Jiashi Feng, Xinglong Wu, Xiaoshuai Sun, and Rongrong Ji. 2023b.  Clover: Towards a unified video-language alignment and fusion model. In _Proceedings of CVPR_. 14856–14866.
  * Huang et al. (2023a) Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. 2023a.  VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval. In _Proceedings of CVPR_. 6565–6574.
  * Huang et al. (2023d) Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. 2023d.  FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition. In _Proceedings of ICLR_.
  * Hwang et al. (2023) Sunil Hwang, Jaehong Yoon, Youngwan Lee, and Sung Ju Hwang. 2023.  Efficient Video Representation Learning via Motion-Aware Token Selection.  _arXiv_ (2023).
  * Idrees et al. (2017) Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. 2017.  The thumos challenge on action recognition for videos “in the wild”.  _CVIU_ 155 (2017), 1–23.
  * Ilharco et al. (2021) Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021.  OpenCLIP.
  * Jang et al. (2017) Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In _Proceedings of CVPR_. 2758–2766.
  * Jhuang et al. (2013) Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black. 2013.  Towards understanding action recognition. In _Proceedings of ICCV_. 3192–3199.
  * Jia et al. (2021a) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021a.  Scaling up visual and vision-language representation learning with noisy text supervision. In _Proceedings of ICML_. 4904–4916.
  * Jia et al. (2021b) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021b.  Scaling up visual and vision-language representation learning with noisy text supervision. In _Proceeedings of ICML_. 4904–4916.
  * Jian et al. (2023a) Yiren Jian, Chongyang Gao, and Soroush Vosoughi. 2023a.  Bootstrapping Vision-Language Learning with Decoupled Language Pre-training.  _arXiv:2307.07063_ (2023).
  * Jian et al. (2023b) Yiren Jian, Chongyang Gao, and Soroush Vosoughi. 2023b.  Bootstrapping Vision-Language Learning with Decoupled Language Pre-training.  _arXiv_ (2023).
  * Jiang et al. (2022) Haojun Jiang, Jianke Zhang, Rui Huang, Chunjiang Ge, Zanlin Ni, Jiwen Lu, Jie Zhou, Shiji Song, and Gao Huang. 2022.  Cross-Modal Adapter for Text-Video Retrieval.  _arXiv_ (2022).
  * Jin et al. (2023) Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. 2023.  Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding.  _arXiv_ (2023).
  * Jin et al. (2024) Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. 2024\.  Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization.  _arXiv:2402.03161_ (2024).
  * Ju et al. (2022a) Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. 2022a.  Prompting visual-language models for efficient video understanding. In _Proceedings of ECCV_. 105–124.
  * Ju et al. (2022b) Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. 2022b.  Prompting Visual-Language Models for Efficient Video Understanding. In _Proceedings of ECCV_. 105–124.
  * Kang and Cho (2023) Seong-Min Kang and Yoon-Sik Cho. 2023.  MEME: Multi-Encoder Multi-Expert Framework with Data Augmentation for Video Retrieval. In _Proceedings of SIGIR_. 475–484.
  * Kareer et al. (2024) Simar Kareer, Vivek Vijaykumar, Harsh Maheshwari, Prithvijit Chattopadhyay, Judy Hoffman, and Viraj Prabhu. 2024.  We’re Not Using Videos Effectively: An Updated Domain Adaptive Video Segmentation Baseline.  _arXiv:2402.00868_ (2024).
  * Karpathy et al. (2014) Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014.  Large-scale video classification with convolutional neural networks. In _Proceedings of CVPR_. 1725–1732.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020.  Transformers are rnns: Fast autoregressive transformers with linear attention. In _Proceedings of ICML_. 5156–5165.
  * Kay et al. (2017) Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017\.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ (2017).
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL-HLT_. 4171–4186.
  * Khoreva et al. (2019) Anna Khoreva, Anna Rohrbach, and Bernt Schiele. 2019.  Video object segmentation with language referring expressions. In _Proceedings of ACCV_. 123–141.
  * Kirillov et al. (2023a) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023a.  Segment Anything. In _Proceedings of ICCV_. 4015–4026.
  * Kirillov et al. (2023b) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023b.  Segment Anything. In _Proceedings of ICCV_. 4015–4026.
  * Ko et al. (2023) Dohwan Ko, Joonmyung Choi, Hyeong Kyu Choi, Kyoung-Woon On, Byungseok Roh, and Hyunwoo J. Kim. 2023.  MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models. In _Proceedings of CVPR_. 20105–20115.
  * Kondratyuk et al. (2023) Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. 2023\.  Videopoet: A large language model for zero-shot video generation.  _arXiv:2312.14125_ (2023).
  * Krishna et al. (2017a) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017a.  Dense-captioning events in videos. In _Proceedings of ICCV_. 706–715.
  * Krishna et al. (2017b) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017b.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _IJCV_ 123 (2017), 32–73.
  * Kuehne et al. (2014) Hilde Kuehne, Ali Arslan, and Thomas Serre. 2014.  The language of actions: Recovering the syntax and semantics of goal-directed human activities. In _Proceedings of CVPR_. 780–787.
  * Kuehne et al. (2011) Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. 2011.  HMDB: a large video database for human motion recognition. In _Proceedings of ICCV_. 2556–2563.
  * Kumar et al. (2023) Akash Kumar, Ashlesha Kumar, Vibhav Vineet, and Yogesh Singh Rawat. 2023.  Benchmarking self-supervised video representation learning.  _arXiv:2306.06010_ (2023).
  * Kuo et al. (2023) Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Benjamin Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew M Dai, Zhifeng Chen, Claire Cui, et al. 2023\.  MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks.  _Transactions on Machine Learning Research_ (2023).
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020\.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _IJCV_ 128, 7 (2020), 1956–1981.
  * Lei et al. (2022) Jie Lei, Tamara L. Berg, and Mohit Bansal. 2022.  Revealing Single Frame Bias for Video-and-Language Learning.  _arXiv_ (2022).
  * Lei et al. (2021) Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021.  Less is more: Clipbert for video-and-language learning via sparse sampling. In _Proceedings of CVPR_. 7331–7341.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The Power of Scale for Parameter-Efficient Prompt Tuning. In _Proceedings of EMNLP_. 3045–3059.
  * Li et al. (2005) A Li, M Thotakuri, DA Ross, J Carreira, A Vostrikov, and A Zisserman. 2005.  The AVA-kinetics localized human actions video dataset (2020).  _arXiv:2005.00214_ (2005).
  * Li et al. (2022a) Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. 2022a.  Align and prompt: Video-and-language pre-training with entity prompts. In _Proceedings of CVPR_. 4953–4963.
  * Li et al. (2023d) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023d.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv:2301.12597_ (2023).
  * Li et al. (2022b) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _Proceedings of ICML_. 12888–12900.
  * Li et al. (2022c) Jiangtong Li, Li Niu, and Liqing Zhang. 2022c.  From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In _Proceedings of CVPR_. 21273–21282.
  * Li et al. (2023b) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023b.  VideoChat: Chat-Centric Video Understanding.  _arXiv_ (2023).
  * Li et al. (2022d) Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. 2022d.  Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer.  _arXiv preprint arXiv:2211.09552_ (2022).
  * Li et al. (2023e) Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. 2023e.  Unmasked Teacher: Towards Training-Efficient Video Foundation Models. In _Proceedings of ICCV_.
  * Li et al. (2020) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  _arXiv:2005.00200_ (2020).
  * Li et al. (2023a) Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. 2023a.  Lavender: Unifying video-language understanding as masked language modeling. In _Proceedings of CVPR_. 23119–23129.
  * Li et al. (2023f) Pandeng Li, Chen-Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge, Yun Zheng, Deli Zhao, and Yongdong Zhang. 2023f.  Progressive spatio-temporal prototype matching for text-video retrieval. In _Proceedings of ICCV_. 4100–4110.
  * Li and Wang (2023) Xinhao Li and Limin Wang. 2023.  ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video.  _arXiv_ (2023).
  * Li et al. (2023c) Yizhuo Li, Kunchang Li, Yinan He, Yi Wang, Yali Wang, Limin Wang, Yu Qiao, and Ping Luo. 2023c.  Harvest Video Foundation Models via Efficient Post-Pretraining.  _arXiv_ (2023).
  * Li et al. (2018) Yingwei Li, Yi Li, and Nuno Vasconcelos. 2018.  Resound: Towards action recognition without representation bias. In _Proceedings of ECCV_. 513–528.
  * Li et al. (2022e) Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. 2022e.  Mvitv2: Improved multiscale vision transformers for classification and detection. In _Proceedings of CVPR_. 4804–4814.
  * Li et al. (2023g) Yonglin Li, Jing Zhang, Xiao Teng, and Long Lan. 2023g.  RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation.  _arXiv_ (2023).
  * Lin et al. (2023d) Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023d.  Video-LLaVA: Learning United Visual Representation by Alignment Before Projection.  _arXiv_ (2023).
  * Lin et al. (2022c) Chengzhi Lin, Ancong Wu, Junwei Liang, Jun Zhang, Wenhang Ge, Wei-Shi Zheng, and Chunhua Shen. 2022c.  Text-adaptive multiple visual prototype matching for video-text retrieval. In _Proceedings of NeurIPS_. 38655–38666.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries. In _Proceedings of ACL-W_. 74–81.
  * Lin et al. (2023f) Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. 2023f.  VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning.  _arXiv_ (2023).
  * Lin et al. (2023e) Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. 2023e.  VILA: On Pre-training for Visual Language Models.  _arXiv:2312.07533_ (2023).
  * Lin et al. (2023a) Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, Ce Liu, and Lijuan Wang. 2023a.  MM-VID: Advancing Video Understanding with GPT-4V(ision).  _arXiv_ (2023).
  * Lin et al. (2022b) Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022b.  Swinbert: End-to-end transformers with sparse attention for video captioning. In _Proceedings of CVPR_. 17949–17958.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _Proceedings of ECCV_. 740–755.
  * Lin et al. (2023b) Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger, Mateusz Kozinski, Rameswar Panda, Rogerio Feris, Hilde Kuehne, and Horst Bischof. 2023b.  Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge. In _Proceedings of ICCV_. 2851–2862.
  * Lin et al. (2023c) Yuanze Lin, Chen Wei, Huiyu Wang, Alan Yuille, and Cihang Xie. 2023c.  Smaug: Sparse masked autoencoder for efficient video-language pre-training. In _Proceedings of CVPR_. 2459–2469.
  * Lin et al. (2022a) Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022a.  Frozen CLIP Models are Efficient Video Learners. In _Proceedings of ECCV_. 388–404.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a.  Improved baselines with visual instruction tuning.  _arXiv:2310.03744_ (2023).
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b.  Visual instruction tuning. In _Proceedings of NeurIPS_.
  * Liu et al. (2024) Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024.  World Model on Million-Length Video And Language With RingAttention.  _arXiv:2402.08268_ (2024).
  * Liu et al. (2020) Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. 2020.  VIOLIN: A large-scale dataset for video-and-language inference. In 2020 IEEE. In _Proceedings of CVPR_. 13–19.
  * Liu et al. (2023c) Xin Liu, Fatemeh Karimi Nejadasl, Jan C van Gemert, Olaf Booij, and Silvia L Pintea. 2023c.  Objects do not disappear: Video object detection by single-frame object location anticipation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 6950–6961.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv:1907.11692_ (2019).
  * Liu et al. (2022b) Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. 2022b.  Fineaction: A fine-grained video dataset for temporal action localization.  _IEEE TIP_ 31 (2022), 6937–6950.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of ICCV_. 10012–10022.
  * Liu et al. (2022a) Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2022a.  Video swin transformer. In _Proceedings of CVPR_. 3202–3211.
  * Luiten et al. (2021) Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. 2021.  Hota: A higher order metric for evaluating multi-object tracking.  _IJCV_ 129 (2021), 548–578.
  * Luo et al. (2020) Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020.  UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation.  _arXiv_ (2020).
  * Luo et al. (2022) Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022.  Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.  _Neurocomputing_ 508 (2022), 293–304.
  * Luo et al. (2023) Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. 2023.  Valley: Video Assistant with Large Language model Enhanced abilitY.  _arXiv_ (2023).
  * Lyu et al. (2023a) Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023a.  Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration.  _arXiv_ (2023).
  * Lyu et al. (2023b) Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023b.  Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration.  _arXiv_ (2023).
  * Ma et al. (2022) Yue Ma, Tianyu Yang, Yin Shan, and Xiu Li. 2022.  SimVTP: Simple Video Text Pre-training with Masked Autoencoders.  _arXiv_ (2022).
  * Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023.  Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models.  _arXiv_ (2023).
  * Madan et al. (2024) Neelu Madan, Nicolae-Cătălin Ristea, Kamal Nasrollahi, Thomas B Moeslund, and Radu Tudor Ionescu. 2024.  CL-MAE: Curriculum-Learned Masked Autoencoders. In _Proceedings of WACV_. 2492–2502.
  * Mangalam et al. (2023) Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023.  Egoschema: A diagnostic benchmark for very long-form video language understanding. In _Proceedings of NeurIPS_.
  * Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021.  Docvqa: A dataset for vqa on document images. In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_. 2200–2209.
  * Miech et al. (2020) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020.  End-to-end learning of visual representations from uncurated instructional videos. In _Proceedings of CVPR_. 9879–9889.
  * Miech et al. (2019) Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In _Proceedings of ICCV_. 2630–2640.
  * Milan et al. (2016) Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. 2016.  MOT16: A benchmark for multi-object tracking.  _arXiv_ (2016).
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _Proceedings of ECCV_. 405–421.
  * Modi et al. (2022) Rajat Modi, Aayush Jung Rana, Akash Kumar, Praveen Tirupattur, Shruti Vyas, Yogesh Singh Rawat, and Mubarak Shah. 2022.  Video action detection: Analysing limitations and challenges.  _arXiv:2204.07892_ (2022).
  * Modi et al. (2024) Rajat Modi, Vibhav Vineet, and Yogesh Rawat. 2024.  On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes.  _Proceedings of NeurIPS_ (2024).
  * Mokady et al. (2021) Ron Mokady, Amir Hertz, and Amit H Bermano. 2021.  Clipcap: Clip prefix for image captioning.  _arXiv:2111.09734_ (2021).
  * Molad et al. (2023) Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. 2023.  Dreamix: Video diffusion models are general video editors.  _arXiv preprint arXiv:2302.01329_ (2023).
  * Momeni et al. (2023) Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. 2023.  Verbs in action: Improving verb understanding in video-language models. In _Proceedings of ICCV_. 15579–15591.
  * Monfort et al. (2019) Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. 2019\.  Moments in time dataset: one million videos for event understanding.  _IEEE TPAMI_ 42, 2 (2019), 502–508.
  * Monfort et al. (2021a) Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. 2021a.  Spoken moments: Learning joint audio-visual representations from video descriptions. In _Proceedings of CVPR_. 14871–14881.
  * Monfort et al. (2021b) Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. 2021b.  Spoken moments: Learning joint audio-visual representations from video descriptions. In _Proceedings of CVPR_. 14871–14881.
  * Munasinghe et al. (2023) Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. 2023.  PG-Video-LLaVA: Pixel Grounding Large Video-Language Models.  _arXiv preprint arXiv:2311.13435_ (2023).
  * Nagaraja et al. (2016) Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. 2016.  Modeling context between objects for referring expression understanding. In _Proceedings of ECCV_. 792–807.
  * Nagrani et al. (2022) Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. 2022.  Learning audio-video modalities from image captions. In _Proceedings of ECCV_. 407–426.
  * Ni et al. (2022) Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. 2022.  Expanding Language-Image Pretrained Models for General Video Recognition. In _Proceedings of ECCV_. 1–18.
  * Ning et al. (2023) Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. 2023.  Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models.  _arXiv_ (2023).
  * Ning et al. (2021) Xu Ning, Yang Linjie, Yang Jianchao, Yue Dingcheng, Fan Yuchen, Liang Yuchen, and S. Huang. Thomas. 2021.  Youtubevis dataset 2021 version.  https://youtube-vos.org/dataset/vis/.
  * Oncescu et al. (2021) Andreea-Maria Oncescu, Joao F Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. 2021.  Queryd: A video dataset with high-quality text and audio narrations. In _Proceedings of ICASSP_. 2265–2269.
  * OpenAI (2023a) OpenAI. 2023a.  ChatGPT [Large language model].  https://chat.openai.com/chat Accessed: 2024-03-28.
  * OpenAI (2023b) OpenAI. 2023b.  GPT-4 Technical Report.  _arXiv:2303.08774_ (2023).
  * Ordonez et al. (2011a) Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011a.  Im2Text: Describing Images Using 1 Million Captioned Photographs. In _Proceedings of NeurIPS_ , J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.).
  * Ordonez et al. (2011b) Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011b.  Im2text: Describing images using 1 million captioned photographs. In _Proceedings of NeurIPS_.
  * Pan et al. (2022) Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. 2022.  St-adapter: Parameter-efficient image-to-video transfer learning.  _Proceedings of NeurIPS_ (2022), 26462–26477.
  * Pan et al. (2021) Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. 2021.  Videomoco: Contrastive video representation learning with temporally adversarial examples. In _Proceedings of CVPR_. 11205–11214.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation. In _Proceedings of ACL_. 311–318.
  * Park et al. (2023) Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. 2023.  Dual-path Adaptation from Image to Video Transformers. In _Proceedings of CVPR_. 2203–2213.
  * Parthasarathy et al. (2023) Nikhil Parthasarathy, SM Ali Eslami, Joao Carreira, and Olivier J. Henaff. 2023.  Self-supervised video pretraining yields robust and more human-aligned visual representations. In _Proceedings of NeurIPS_.
  * Peng et al. (2023a) Lang Peng, Zhirong Chen, Zhangjie Fu, Pengpeng Liang, and Erkang Cheng. 2023a.  Bevsegformer: Bird’s eye view semantic segmentation from arbitrary camera rigs. In _Proceedings of WACV_. 5935–5943.
  * Peng et al. (2023b) Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023b.  Kosmos-2: Grounding Multimodal Large Language Models to the World.  _arXiv:2306.14824_ (2023).
  * Perazzi et al. (2016) Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. 2016.  A benchmark dataset and evaluation methodology for video object segmentation. In _Proceedings of CVPR_. 724–732.
  * Perrone et al. (2021) Valerio Perrone, Huibin Shen, Aida Zolic, Iaroslav Shcherbatyi, Amr Ahmed, Tanya Bansal, Michele Donini, Fela Winkelmolen, Rodolphe Jenatton, Jean Baptiste Faddoul, et al. 2021\.  Amazon sagemaker automatic model tuning: Scalable gradient-free optimization. In _Proceedings of SIGKDD_. 3463–3471.
  * Phan et al. (2023) Thinh Phan, Khoa Vo, Duy Le, Gianfranco Doretto, Donald Adjeroh, and Ngan Le. 2023.  ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection.  _arXiv_ (2023).
  * Philion and Fidler (2020) Jonah Philion and Sanja Fidler. 2020.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In _Proceedings of ECCV_. 194–210.
  * Qi et al. (2023) Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. 2023.  Fatezero: Fusing attentions for zero-shot text-based video editing. In _Proceedings of ICCV_. 15932–15942.
  * Qi et al. (2022) Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. 2022.  Occluded video instance segmentation: A benchmark.  _IJCV_ 130, 8 (2022), 2022–2039.
  * Qian et al. (2022) Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui. 2022.  Multimodal open-vocabulary video classification via pre-trained vision and language models.  _arXiv:2207.07646_ (2022).
  * Qing et al. (2023) Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Yingya Zhang, Changxin Gao, Deli Zhao, and Nong Sang. 2023.  Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning. In _Proceedings of ICCV_. 13934–13944.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _Proceedings of ICML_. 8748–8763.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.  _OpenAI blog_ (2019).
  * Rajič et al. (2023) Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. 2023.  Segment Anything Meets Point Tracking.  _arXiv_ (2023).
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation. In _Proceedings of ICML_. 8821–8831.
  * Ranasinghe and Ryoo (2023) Kanchana Ranasinghe and Michael S Ryoo. 2023.  Language-based action concept spaces improve video self-supervised learning. In _Proceedings of NeurIPS_.
  * Ranftl et al. (2021) René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021.  Vision transformers for dense prediction. In _Proceedings of ICCV_. 12179–12188.
  * Rasheed et al. (2023) Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023.  Fine-tuned CLIP Models are Efficient Video Learners. In _Proceedings of CVPR_. 6545–6554.
  * Rohrbach et al. (2015) Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015.  A dataset for movie description. In _Proceedings of CVPR_. 3202–3212.
  * Romero and Solorio (2024) David Romero and Thamar Solorio. 2024.  Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering.  _arXiv:2402.10698_ (2024).
  * Rujikietgumjorn and Watcharapinchai (2017) Sitapa Rujikietgumjorn and Nattachai Watcharapinchai. 2017.  Vehicle detection with sub-class training using R-CNN for the UA-DETRAC benchmark.  _Proceedings of AVSS_ (2017), 1–5.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115 (2015), 211–252.
  * Saito et al. (2020) Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. 2020.  Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan.  _IJCV_ 128, 10-11 (2020), 2586–2606.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Schiappa et al. (2023a) Madeline Chantry Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh S Rawat. 2023a.  A large-scale robustness analysis of video action recognition models. In _Proceedings of CVPR_. 14698–14708.
  * Schiappa et al. (2023b) Madeline C Schiappa, Yogesh S Rawat, and Mubarak Shah. 2023b.  Self-supervised learning for videos: A survey.  _Comput. Surveys_ 55, 13s (2023), 1–37.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022\.  Laion-5b: An open large-scale dataset for training next generation image-text models. In _Proceedings of NeurIPS_ , Vol. 35. 25278–25294.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv:2111.02114_ (2021).
  * Seo et al. (2022) Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. 2022.  End-to-end generative pretraining for multimodal video captioning. In _Proceedings of CVPR_. 17959–17968.
  * Seo et al. (2020) Seonguk Seo, Joon-Young Lee, and Bohyung Han. 2020.  URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark. In _Proceeding of the ECCV_. 208–223.
  * Shao et al. (2019) Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019.  Objects365: A large-scale, high-quality dataset for object detection. In _Proceedings of ICCV_. 8430–8439.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In _Proceedings of ACL_. 2556–2565.
  * Sigurdsson et al. (2016) Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016.  Hollywood in homes: Crowdsourcing data collection for activity understanding. In _Proceedings of ECCV_. 510–526.
  * Singer et al. (2022) Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022\.  Make-a-video: Text-to-video generation without text-video data.  _arXiv:2209.14792_ (2022).
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics. In _Proceedings of ICML_. 2256–2265.
  * Song et al. (2023) Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. 2023.  MovieChat: From Dense Token to Sparse Memory for Long Video Understanding.  _arXiv_ (2023).
  * Song et al. (2022) Yuxin Song, Min Yang, Wenhao Wu, Dongliang He, Fu Li, and Jingdong Wang. 2022.  It Takes Two: Masked Appearance-Motion Modeling for Self-supervised Video Transformer Pre-training.  _arXiv_ (2022).
  * Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012.  UCF101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ (2012).
  * Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019.  Videobert: A joint model for video and language representation learning. In _Proceedings of ICCV_. 7464–7473.
  * Sun et al. (2023b) Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023b.  Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models.  _arXiv:2310.05863_ (2023).
  * Sun et al. (2023a) Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023a.  Eva-clip: Improved training techniques for clip at scale.  _arXiv preprint arXiv:2303.15389_ (2023).
  * Sun et al. (2022) Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. 2022.  Long-form video-language pre-training with multimodal temporal contrastive learning. In _Proceedings of NeurIPS_. 38032–38045.
  * Sundararaman et al. (2021) Ramana Sundararaman, Cedric De Almeida Braga, Eric Marchand, and Julien Pettre. 2021.  Tracking pedestrian heads in dense crowd. In _Proceedings of CVPR_. 3865–3875.
  * Tan et al. (2021) Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. 2021.  Vimpac: Video pre-training via masked token prediction and contrastive learning.  _arXiv preprint arXiv:2106.11250_ (2021).
  * Tang et al. (2021) Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. 2021.  Clip4caption: Clip for video caption. In _Proceedings of ACM-MM_. 4858–4862.
  * Tang et al. (2019) Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019.  Coin: A large-scale dataset for comprehensive instructional video analysis. In _Proceedings of CVPR_. 1207–1216.
  * Tang et al. (2023) Zineng Tang, Jaemin Cho, Jie Lei, and Mohit Bansal. 2023.  Perceiver-vl: Efficient vision-and-language modeling with iterative latent attention. In _Proceedings of WACV_. 4410–4420.
  * Tang et al. (2022) Zineng Tang, Jaemin Cho, Yixin Nie, and Mohit Bansal. 2022.  TVLT: Textless vision-language transformer.  _Proceedings of NeurIPS_ (2022), 9617–9632.
  * Tay et al. (2022a) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022a.  Ul2: Unifying language learning paradigms. In _Proceedings of ICLR_.
  * Tay et al. (2022b) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022b.  Ul2: Unifying language learning paradigms. In _Proceedings of ICLR_.
  * Teed and Deng (2020) Zachary Teed and Jia Deng. 2020.  Raft: Recurrent all-pairs field transforms for optical flow. In _Proceedings of ECCV_. 402–419.
  * Tong et al. (2022) Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In _Proceedings of NeurIPS_. 10078–10093.
  * Torabi et al. (2016) Atousa Torabi, Niket Tandon, and Leonid Sigal. 2016.  Learning language-visual embedding for movie understanding with natural-language.  _arXiv:1609.08124_ (2016).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv:2302.13971_ (2023).
  * Tran et al. (2015) Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015.  Learning spatiotemporal features with 3d convolutional networks. In _Proceedings of ICCV_. 4489–4497.
  * Unterthiner et al. (2018) Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. 2018.  Towards accurate generative models of video: A new metric & challenges.  _arXiv:1812.01717_ (2018).
  * Van Den Oord et al. (2017) Aaron Van Den Oord, Oriol Vinyals, et al. 2017\.  Neural discrete representation learning. In _Proceedings of NeurIPS_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Proceedings of NeurIPS_.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation. In _Proceedings of CVPR_. 4566–4575.
  * Vinyals et al. (2016) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016.  Show and tell: Lessons learned from the 2015 mscoco image captioning challenge.  _IEEE PAMI_ 39, 4 (2016), 652–663.
  * Wang et al. (2023i) Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. 2023i.  Towards open-vocabulary video instance segmentation. In _Proceedings of ICCV_. 4057–4066.
  * Wang et al. (2023b) Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. 2023b.  Chatvideo: A tracklet-centric multimodal and versatile video understanding system.  _arXiv:2304.14407_ (2023).
  * Wang et al. (2022b) Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. 2022b.  Omnivl: One foundation model for image-language and video-language tasks. In _Proceedings of NeurIPS_ , Vol. 35. 5696–5710.
  * Wang et al. (2022d) Jinpeng Wang, Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022d.  Object-aware video-language pre-training for retrieval. In _Proceedings of CVPR_. 3313–3322.
  * Wang et al. (2023e) Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, and Ying Shan. 2023e.  All in one: Exploring unified video-language pre-training. In _Proceedings of CVPR_. 6598–6608.
  * Wang et al. (2023g) Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. 2023g.  Videomae v2: Scaling video masked autoencoders with dual masking. In _Proceedings of CVPR_. 14549–14560.
  * Wang et al. (2024b) Mengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen, Jianbiao Mei, Xingxing Zuo, Guang Dai, Jingdong Wang, and Yong Liu. 2024b.  M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition.  _arXiv:2401.11649_ (2024).
  * Wang et al. (2021) Mengmeng Wang, Jiazheng Xing, and Yong Liu. 2021.  Actionclip: A new paradigm for video action recognition.  _arXiv:2109.08472_ (2021).
  * Wang et al. (2023d) Qiang Wang, Junlong Du, Ke Yan, and Shouhong Ding. 2023d.  Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learning. In _Proceedings of ACM Multimedia_. 5339–5347.
  * Wang et al. (2022a) Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. 2022a.  Bevt: Bert pretraining of video transformers. In _Proceedings of CVPR_. 14733–14743.
  * Wang et al. (2023c) Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. 2023c.  Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In _Proceedings of CVPR_. 6312–6322.
  * Wang et al. (2023j) Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. 2023j.  VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation.  _arXiv:2305.10874_ (2023).
  * Wang et al. (2023h) Xiao Wang, Yaoyu Li, Tian Gan, Zheng Zhang, Jingjing Lv, and Liqiang Nie. 2023h.  RTQ: Rethinking Video-language Understanding Based on Image-text Model. In _Proceedings of ACM-MM_. 557–566.
  * Wang et al. (2024c) Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. 2024c.  Videocomposer: Compositional video synthesis with motion controllability. In _Proceedings of NeurIPS_.
  * Wang et al. (2022c) Yuxuan Wang, Difei Gao, Licheng Yu, Weixian Lei, Matt Feiszli, and Mike Zheng Shou. 2022c.  Geb+: A benchmark for generic event boundary captioning, grounding and retrieval. In _Proceedings of ECCV_. 709–725.
  * Wang et al. (2023f) Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. 2023f.  InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation.  _arXiv_ (2023).
  * Wang et al. (2024a) Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. 2024a.  InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding.  _arXiv:2403.15377_ (2024).
  * Wang et al. (2022e) Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. 2022e.  InternVideo: General Video Foundation Models via Generative and Discriminative Learning.  _arXiv_ (2022).
  * Wang et al. (2022f) Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. 2022f.  InternVideo: General Video Foundation Models via Generative and Discriminative Learning.  _arXiv_ (2022).
  * Wang et al. (2023a) Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. 2023a.  Paxion: Patching Action Knowledge in Video-Language Foundation Models. In _Proceedings of NeurIPS_.
  * Wasim et al. (2023a) Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. 2023a.  Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting. In _Proceedings of CVPR_. 23034–23044.
  * Wasim et al. (2023b) Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. 2023b.  Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting. In _Proceedings of CVPR_. 23034–23044.
  * Wei et al. (2022) Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. 2022.  Masked feature prediction for self-supervised visual pre-training. In _Proceedings of CVPR_. 14668–14678.
  * Wu et al. (2021b) Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 2021b.  Star: A benchmark for situated reasoning in real-world videos. In _Proceedings of NeurIPS_.
  * Wu et al. (2021a) Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. 2021a.  Godiva: Generating open-domain videos from natural descriptions.  _arXiv:2104.14806_ (2021).
  * Wu et al. (2022b) Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. 2022b.  Nüwa: Visual synthesis pre-training for neural visual world creation. In _Proceedings of ECCV_. 720–736.
  * Wu and Krahenbuhl (2021) Chao-Yuan Wu and Philipp Krahenbuhl. 2021.  Towards long-form video understanding. In _Proceedings of CVPR_. 1884–1894.
  * Wu et al. (2023a) Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, and Radu Soricut. 2023a.  Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts.  _arXiv preprint arXiv:2312.00968_ (2023).
  * Wu et al. (2023b) Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. 2023b.  General Object Foundation Model for Images and Videos at Scale.  _arXiv:2312.09158_ (2023).
  * Wu et al. (2022a) Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. 2022a.  Language as queries for referring video object segmentation. In _Proceedings of the CVPR_. 4974–4984.
  * Wu et al. (2023c) Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. 2023c.  Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models. In _Proceedings of CVPR_. 6620–6630.
  * Xiao et al. (2021) Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021.  Next-qa: Next phase of question-answering to explaining temporal actions. In _Proceedings of CVPR_. 9777–9786.
  * Xiao et al. (2024) Junfei Xiao, Zheng Xu, Alan Yuille, Shen Yan, and Boyu Wang. 2024.  PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter.  _arXiv:2402.10896_ (2024).
  * Xu et al. (2017) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017.  Video question answering via gradually refined attention over appearance and motion. In _Proceedings of ACM-MM_. 1645–1653.
  * Xu et al. (2021a) Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer. 2021a.  VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding. In _Proceedings of ACL-IJCNLP_. 4227–4239.
  * Xu et al. (2021b) Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021b.  VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding. In _Proceedings of EMNLP_. 6787–6800.
  * Xu et al. (2023b) Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jingren Zhou. 2023b.  mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video.  _arXiv_ (2023).
  * Xu et al. (2023a) Jiaqi Xu, Bo Liu, Yunkuo Chen, Mengli Cheng, and Xing Shi. 2023a.  MuLTI: Efficient Video-and-Language Understanding with MultiWay-Sampler and Multiple Choice Modeling.  _arXiv_ (2023).
  * Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016.  Msr-vtt: A large video description dataset for bridging video and language. In _Proceedings of CVPR_. 5288–5296.
  * Xue et al. (2022) Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. 2022.  Advancing high-resolution video-language representation with large-scale video transcriptions. In _Proceedings of CVPR_. 5036–5045.
  * Xue et al. (2023b) Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. 2023b.  CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment.  _arXiv_ (2023).
  * Xue et al. (2023a) Zihui Xue, Kumar Ashutosh, and Kristen Grauman. 2023a.  Learning Object State Changes in Videos: An Open-World Perspective.  _arXiv:2312.11782_ (2023).
  * Yan et al. (2023) Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. 2023.  VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners.  _arXiv_ (2023).
  * Yang et al. (2021) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021.  Just ask: Learning to answer questions from millions of narrated videos. In _Proceedings of ICCV_. 1686–1697.
  * Yang et al. (2023) Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. 2023.  Track Anything: Segment Anything Meets Videos.  _arXiv_ (2023).
  * Yang et al. (2019) Linjie Yang, Yuchen Fan, and Ning Xu. 2019.  Video instance segmentation. In _Proceedings of ICCV_. 5188–5197.
  * Yang et al. (2022) Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. 2022.  AIM: Adapting Image Models for Efficient Video Action Recognition. In _Proceedings of ICLR_.
  * Ye et al. (2023) Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. 2023.  Hitea: Hierarchical temporal-aware video-language pre-training. In _Proceedings of CVPR_. 15405–15416.
  * Yi et al. (2019) Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. 2019.  CLEVRER: Collision Events for Video Representation and Reasoning. In _Proceedings of ICLR_.
  * Yousaf et al. (2023) Adeel Yousaf, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. 2023.  Videoprompter: an ensemble of foundational models for zero-shot video understanding.  _arXiv preprint arXiv:2310.15324_ (2023).
  * Yu et al. (2020) Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In _Proceedings of CVPR_. 2636–2645.
  * Yu et al. (2016b) Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. 2016b.  Video paragraph captioning using hierarchical recurrent neural networks. In _Proceedings of CVPR_. 4584–4593.
  * Yu et al. (2022) Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022.  Coca: Contrastive captioners are image-text foundation models.  _arXiv:2205.01917_ (2022).
  * Yu et al. (2016a) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016a.  Modeling context in referring expressions. In _Proceedings of ECCV_. 69–85.
  * Yu et al. (2018) Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018.  A joint sequence fusion model for video question answering and retrieval. In _Proceedings of ECCV_. 471–487.
  * Yu et al. (2019) Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019.  Activitynet-qa: A dataset for understanding complex web videos via question answering. In _Proceedings of AAAI_. 9127–9134.
  * Yuan et al. (2023) Liangzhe Yuan, Nitesh Bharadwaj Gundavarapu, Long Zhao, Hao Zhou, Yin Cui, Lu Jiang, Xuan Yang, Menglin Jia, Tobias Weyand, Luke Friedman, et al. 2023\.  VideoGLUE: Video General Understanding Evaluation of Foundation Models.  _arXiv:2307.03166_ (2023).
  * Yuan et al. (2024) Zhengqing Yuan, Ruoxi Chen, Zhaoxu Li, Haolong Jia, Lifang He, Chi Wang, and Lichao Sun. 2024.  Mora: Enabling Generalist Video Generation via A Multi-Agent Framework.  _arXiv:2403.13248_ (2024).
  * Zara et al. (2024) Giacomo Zara, Victor Guilherme Turrisi da Costa, Subhankar Roy, Paolo Rota, and Elisa Ricci. 2024.  Simplifying open-set video domain adaptation with contrastive learning.  _CVIU_ (2024), 103953.
  * Zellers et al. (2021a) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021a.  Merlot: Multimodal neural script knowledge models. In _Proceedings of NeurIPS_. 23634–23651.
  * Zellers et al. (2021b) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021b.  Merlot: Multimodal neural script knowledge models. In _Poceedings of NeurIPS_. 23634–23651.
  * Zeng et al. (2023) Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou. 2023.  X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks.  _arXiv_ (2023).
  * Zhai et al. (2022) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022.  Scaling vision transformers. In _Proceedings of CVPR_. 12104–12113.
  * Zhang (2016) Bill Zhang. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ (2016).
  * Zhang et al. (2023a) Binjie Zhang, Yixiao Ge, Xuyuan Xu, Ying Shan, and Mike Zheng Shou. 2023a.  TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter.  _arXiv_ (2023).
  * Zhang et al. (2023b) Hang Zhang, Xin Li, and Lidong Bing. 2023b.  Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding.  _arXiv_ (2023).
  * Zhang et al. (2023c) Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. 2023c.  Temporal sentence grounding in videos: A survey and future directions.  _IEEE TPAMI_ (2023).
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022\.  Opt: Open pre-trained transformer language models.  _arXiv:2205.01068_ (2022).
  * Zhang et al. (2023d) Zhenghao Zhang, Zhichao Wei, Shengfan Zhang, Zuozhuo Dai, and Siyu Zhu. 2023d.  UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model.  _arXiv_ (2023).
  * Zhao et al. (2019) Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. 2019.  Hacs: Human action clips and segments dataset for recognition and temporal localization. In _Proceedings of ICCV_. 8668–8678.
  * Zhao et al. (2024a) Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. 2024a.  VideoPrism: A Foundational Visual Encoder for Video Understanding.  _arXiv:2402.13217_ (2024).
  * Zhao et al. (2022) Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. 2022.  CenterCLIP: Token Clustering for Efficient Text-Video Retrieval. In _Proceedings of SIGIR_. 970–981.
  * Zhao and Krähenbühl (2023) Yue Zhao and Philipp Krähenbühl. 2023.  Training a large video model on a single machine in a day.  _arXiv:2309.16669_ (2023).
  * Zhao et al. (2023) Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. 2023.  Learning Video Representations from Large Language Models. In _Proceedings of CVPR_. 6586–6597.
  * Zhao et al. (2024b) Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, et al. 2024b.  Distilling vision-language models on millions of videos.  _arXiv preprint arXiv:2401.06129_ (2024).
  * Zheng et al. (2023) Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. 2023.  Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In _Proceedings of ICCV_. 19855–19865.
  * Zhou et al. (2022c) Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. 2022c.  Magicvideo: Efficient video generation with latent diffusion models.  _arXiv:2211.11018_ (2022).
  * Zhou et al. (2018) Luowei Zhou, Chenliang Xu, and Jason Corso. 2018.  Towards automatic learning of procedures from web instructional videos. In _Proceedings of AAAI_.
  * Zhou et al. (2022a) Qianyu Zhou, Xiangtai Li, Lu He, Yibo Yang, Guangliang Cheng, Yunhai Tong, Lizhuang Ma, and Dacheng Tao. 2022a.  TransVOD: end-to-end video object detection with spatial-temporal transformers.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022).
  * Zhou et al. (2022b) Tianfei Zhou, Fatih Porikli, David J Crandall, Luc Van Gool, and Wenguan Wang. 2022b.  A survey on deep learning technique for video segmentation.  _IEEE TPAMI_ 45, 6 (2022), 7099–7122.
  * Zhu et al. (2023b) Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2023b.  LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment.  _arXiv_ (2023).
  * Zhu et al. (2023a) Jiawen Zhu, Zhenyu Chen, Zeqi Hao, Shijie Chang, Lu Zhang, Dong Wang, Huchuan Lu, Bin Luo, Jun-Yan He, Jin-Peng Lan, Hanyuan Chen, and Chenyang Li. 2023a.  Tracking Anything in High Quality.  _arXiv_ (2023).