#  : Self-Training Enables Video Instruction Tuning with Any Supervision

Orr Zohar1†, Xiaohan Wang1, Yonatan Bitton2, Idan Szpektor2 & Serena Yeung-
Levy1†
1Stanford University, 2Google Research
<EMAIL_ADDRESS>
Project page: https://orrzohar.github.io/projects/video-star/

###### Abstract

The performance of Large Vision Language Models (LVLMs) is dependent on the
size and quality of their training datasets. Existing video instruction tuning
datasets lack diversity as they are derived by prompting large language models
with video captions to generate question-answer pairs, and are therefore
mostly descriptive. Meanwhile, many labeled video datasets with diverse labels
and supervision exist - however, we find that their integration into LVLMs is
non-trivial. Herein, we present Video Self-Training with augmented Reasoning
(), the first video self-training approach. allows the utilization of any
labeled video dataset for video instruction tuning. In , an LVLM cycles
between instruction generation and finetuning, which we show (I) improves
general video understanding and (II) adapts LVLMs to novel downstream tasks
with existing supervision. During generation, an LVLM is prompted to propose
an answer. The answers are then filtered only to those that contain the
original video labels, and the LVLM is then re-trained on the generated
dataset. By only training on generated answers that contain the correct video
labels, utilizes these existing video labels as weak supervision for video
instruction tuning. Our results demonstrate that -enhanced LVLMs exhibit
improved performance in (I) general video QA, where TempCompass performance
improved by $10\%$, and (II) on downstream tasks, where improved
Kinetics700-QA accuracy by $20\%$ and action quality assessment on FineDiving
by $15\%$.

##  1 Introduction

The advent of large Vision-Language Models (LVLMs) marked a significant
milestone in artificial intelligence. These models aim to create versatile
systems capable of understanding and executing vision-and-language tasks
aligned with human intentions. Early advancements in LVLMs, as exemplified by
works such as BLIP (Li et al., 2022; 2023a) and LLaVA (Liu et al., 2023b; a),
have been driven by the dissemination of pre-trained large language models
(LLMs) (e.g., LLaMA (Touvron et al., 2023a; b)) and pre-trained vision/vision-
language models (e.g., CLIP (Radford et al., 2021)). LVLMs connect the two
model types via visual-language alignment and instruction tuning.

Figure 1: Overview. can utilize any labeled video dataset, including AR
(Action Recognition), AQA (Action Quality Assessment), and TAL (Temporal
Action Localization) – from which it generates video instruction tuning data
(video, question, answer triplets). Internally, cycles between: (I) Answer
Generation, where an LVLM is prompted to generate candidate answers for the
questions. (II) Label Verification where generated answers are filtered to
only those that contain the video labels. And (III) Instruction Tuning, where
a model is retrained on answers that pass verification. These cycles continue
until performance plateaus.

Wei et al. (2022); Liu et al. (2023a); Karamcheti et al. (2024) demonstrated
the importance of visual instruction tuning on the resulting LVLM’s
performance. However, while much progress has been made in image-LVLMs, video-
LVLMs still face challenges due to the relative difficulty in generating
quality video instruction tuning datasets. Even though videos contain more
complex scene dynamics and temporal information, indicating a need for larger
and more diverse training datasets compared to images, the largest video
instruction dataset, VideoInstruct-$100K$ (VI-$100K$) (Maaz et al., 2023),
comprises $100K$ video-text pairs but only $13K$ unique videos. This is small
compared to image instruction datasets like LLaVA-$1.5$-Instruct (Liu et al.,
2023a), which has $\sim 665K$ image question pairs and $\sim 350K$ unique
images. Such limitation in video instruction tuning leads to performance
saturation (Li et al., 2023b; Maaz et al., 2023; Jin et al., 2023; Wang et
al., 2023; Lin et al., 2023).

Furthermore, due to video instruction tuning dataset construction - mainly
prompting large language models to produce question-answer pairs - these video
datasets often degrade to simplistic questions, prompting for video captions —
$75$% of VI-$100K$’s questions are of this type (see App. Fig. 10). Combining
different sources of supervision has the potential to generate more diverse
video instruction tuning datasets, enhancing video understanding. Such
supervision exists, as the broader computer vision community has developed an
extensive collection of video benchmarks tailored for diverse tasks such as
action recognition (Smaira et al., 2020; Soomro et al., 2012), action quality
assessment (Xu et al., 2022b; Zhang et al., 2023c), among others (Heilbron et
al., 2015; Wu et al., 2021; Grunde-McLaughlin et al., 2021; Grauman et al.,
2023; Luo et al., 2022).

Beyond improving overall LVLM performance, adapting LVLMs to novel or out-of-
domain tasks is also crucial. While LVLMs have many novel and impactful
applications, many remain out of reach — such as analyzing radiology images
(Senkaiahliyan et al., 2023), meteorological data (Lawson et al., 2024),
traffic analysis (Zhou & Knoll, 2024), judging sporting events (e.g.,
gymnastics, Olympic diving), and assisting in surgical procedures, among
others (Zheng et al., 2024; Jiang et al., 2024; Deng et al., 2024). These
tasks require expert, in-domain knowledge that LVLMs lack, necessitating
adaptation through instruction tuning. However, collecting video instruction
tuning datasets is complex and requires extensive manual effort. For instance,
training an ‘AI judge’ to judge Olympic diving would traditionally involve
collecting detailed expert critiques of each dive. On the other hand, these
tasks often include auxiliary annotations that could be leveraged, such as
surgical outcomes in medical procedures or judging scores in Olympic events.

To address these challenges, we take inspiration from LLM’s capability for
self-improvement (a.k.a self-training) (Huang et al., 2022; Zelikman et al.,
2022), which involves training a model on its generated data and filtering to
exclude low-quality outputs. A model’s performance is improved by cycling
between generation, filtering, and training. For example, Zelikman et al.
(2022) introduced Self-Taught Reasoners which generate chain-of-thought by
prompting an LLM to generate answers and rationalize responses, retaining only
correctly answered questions for further training. Herein, we explore self-
training in LVLMs and introduce Video Self-Training with augmented Reasoning
(, see Fig. 1). enables the incorporation of _any_ labeled video dataset of
any format or task by prompting an LVLM with the video and a question to
generate answers (Fig. 2, 2.1) containing the video content’s label. If the
model cannot correctly answer the question, we provide the video label and ask
it to rationalize it (Fig. 2, 2.2). We then utilize the labels as weak
supervision, rejecting answers that do not contain the correct label (Fig. 2,
2.3). By facilitating the use of any supervision for video instruction tuning,
enables the creation of diverse datasets.

Our experimental setup initializes with Video-LLaVA (Lin et al., 2023),
focusing on assessing its impact on video question-answering (VQA)
performance. After a few training cycles, we compare the performance of to
other state-of-the-art LVLMs and strong baselines to gauge the effectiveness
of the framework. Our findings demonstrate notable enhancements in accuracy
and reasoning capabilities, highlighting ’s role in overcoming the constraints
posed by conventional video instruction tuning datasets. We show that the
integration of not only boosts Video-LLaVA’s performance on standard zero-shot
VQA benchmarks but also significantly improves its adaptability to various
downstream video understanding tasks. This underscores ’s capacity to advance
LVLM training while improving overall performance and versatility.

Our contributions can be summarized as follows:

  1. 1.

We introduce Video Self-Training with augmented Reasoning (), the first video
self-training Large Video-Language Model method. Using self-training, enables
the use of any labeled video dataset for video instruction tuning.

  2. 2.

improves zero-shot video question answering performance on various benchmark
datasets, as evidenced by increased accuracy on TempCompass (Liu et al., 2024)
from $45.7$ to $50.3$ ($+10\%$) and on MSVD-QA (Maaz et al., 2023) from $69.7$
to $71.3$ ($+2.3\%$).

  3. 3.

We demonstrate that can adapt LVLMs to diverse video tasks, notably enhancing
action quality assessment on FineDiving (Xu et al., 2022b), where accuracy
rose from $17.6$ to $20.2$ ($+15\%$) and in action recognition on Kinetics700
(Smaira et al., 2020), where accuracy increased from $50.0$ to $59.9$
($+20\%$).

  4. 4.

Utilizing , we create a large, $1M$ video instruction tuning dataset -
VSTaR-1M, sourced from diverse datasets and tasks, and show that it benefits
LVLM performance.

##  2 Video Self-Training with augmented Reasoning ()

Given a dataset of videos $v$ and their corresponding labels
$l:\mathcal{D}=\\{(v_{i},l_{i})\\}_{i=1}^{d}$, ’s objective is to create
question $q$ answer $a$ pairs to instruction-tune the pre-trained model $M$ on
the dataset $\hat{\mathcal{D}}=\\{(v_{i},q_{i},a_{i})\\}_{i=1}^{d_{f}}$,
producing the instruction-tuned model $\hat{M}$. Note that videos need not be
from the same task, and may contain multiple labels. We start by prompting a
large language model with a task description $T$ and video labels $L$ to
generate candidate questions $q$:

$\displaystyle Y_{T,L}$ $\displaystyle=\parbox[t]{348.26173pt}{ A video is
labeled \\{L\\} for the task of \\{T\\}. What questions could you ask someone
about the video that should contain the video labels in the response? }$

performs generation-training cycles, where in cycle $i$ the instruction-tuned
model $\hat{M}^{i\star}$ is produced, while the instruction-tuned model from
the previous cycle $\hat{M}^{(i-1)\star}$ is utilized for training data
generation. We initialize the process with $\hat{M}^{0\star}$, an existing
instruction-tuned model.

To prepare the training data in cycle $i$, answers are generated either
directly via Answer Generation or through backward rationalization via Label
Rationalization. In Answer Generation, $\hat{M}^{(i-1)\star}$ is prompted with
questions (Sec. 2.1). Candidate answers are then filtered using the original
video labels (Sec. 2.3). Videos rejected during direct Answer Generation are
rationalized, where $\hat{M}^{(i-1)\star}$ is provided both a video $v_{i}$
and labels $l_{i}$, and then prompted with the question again (Sec. 2.2).
Candidate answers are filtered again, creating the instruction tuning dataset
in cycle $i$, $\hat{\mathcal{D}_{i}}$ of size $d_{i}$. A pre-trained model $M$
is then finetuned on $\mathcal{D}_{i}$, producing $\hat{M}^{i\star}$. The next
cycle generates data using $\hat{M}^{i\star}$, until the performance plateaus
(see Fig. 2).

Figure 2:  Video Self-Training with augmented Reasoning. (2.1) We initialize
by prompting an LVLM to generate an answer for a particular video. (2.3) We
then filter the generated answers to those only containing the original video
labels. (2.2) The videos whose generated answer did not contain the ground-
truth labels are then sent to label rationalization, where given the video,
question, and label - the model is expected to rationalize the label. (2.3)
The generated answers are filtered again to those only containing the ground-
truth labels, and (2) the LVLM is instruction-tuned from the pre-trained
checkpoint on the resulting dataset. The cycle is then repeated.

###  2.1 Answer Generation

Each cycle begins in direct Answer Generation. In this phase,
$\hat{M}^{(i-1)\star}$ is prompted with the video-question pair to provide an
answer along with a detailed rationale:

$\displaystyle Y_{Q}$ $\displaystyle=\parbox[t]{348.26173pt}{ Question:
\\{Q\\}. Rationalize your answer step-by-step; how can one arrive at this
conclusion? }$

When prompted with the question $q_{i}$ on a particular video,
$\hat{M}^{(i-1)\star}$ is expected to generate an answer $a_{i}$ that contains
the label $\hat{l}_{i}$ and the rationale $r_{i}$
($a_{i}=r_{i}\cup\hat{l}_{i}$, see Fig. 2). We observe that answers containing
the correct labels are of higher quality and suffer less from hallucination.
Therefore, we filter the generated answers to include only those that contain
the correct label $(\hat{l}_{i}=l_{i})$ utilizing a verifier (Sec. 2.3). For
an example of Answer Generation, see Fig. 3.

###  2.2 Label Rationalization

Answer generation has two main drawbacks: (i) in some applications, especially
on challenging/out-of-domain tasks, initial answer generation yield is low,
resulting in almost no training samples after filtering (e.g., FineDiving, see
Fig. 3); (ii) improvement plateaus as the model fails to solve new problems in
the training set, and it is only trained on examples it answers correctly.

Inspired by Zelikman et al. (2022), for videos whose $\hat{M}^{(i-1)\star}$
generated answers did not contain the ground-truth labels, we introduce label
rationalization as part of . Concretely, we provide $\hat{M}^{(i-1)\star}$ the
video, question, and video label and instruct the model to rationalize the
label:

$\displaystyle Y_{Q,L}$ $\displaystyle=\parbox[t]{348.26173pt}{ Question:
\\{Q\\}. Answer: \\{L\\}.\\\ Can you rationalize the answer step-by-step? How
can one arrive at this conclusion? }$

Given the correct label, the model can reason backward and more easily
generate a rationale leading to the correct answer. However, label
rationalization is more prone to hallucinations, so we prefer direct answer
generation and use rationalizations only if answer generation fails. For an
example of Label Rationalization, see Fig. 3, top. The generated answers are
then filtered again, keeping only those that contain the correct label
$(\hat{l}_{i}=l_{i})$ utilizing a verifier (Sec. 2.3). Label Rationalization
is only utilized in training cycles; only direct answers are used to produce
the final model $\hat{M}^{\star}$.

###  2.3 Label Verification

aims to utilize the labels as weak supervision in instruction tuning data
generation. Gold labels are a grounding aspect of our datasets and represent
some ground-truth knowledge. Therefore, we assume that answers that contain
the ground-truth labels in their responses are higher quality than those that
don’t. While we would like to validate the existence of the different labels
in the generated text, this can be non-trivial.

To this end, we introduce the Parser-Verifier. The Parser, $P$ extracts the
predicted labels from the generated text ($\hat{l_{i}}=P(a_{i})$), using a
mixture of named entity recognition and Regex. Regex is used to identify
easily identifiable string patterns, such as bounding boxes and time ranges,
while named entity recognition is used for more nuanced entities, such as
timestamps. The Verfier, $V$ compares the extracted labels with the gold ones
using the appropriate metrics ($V(l_{i},\hat{l_{i}})\rightarrow\mathbb{R}$).
For example, IoU for bounding boxes/temporal action localization, and BERT
(Devlin et al., 2018) embedding similarity for sentence ordering. Each video
has between 1-3 associated labels. To be classified as correct, the predicted
labels must be within a 5% margin of error from the gold.

Source Videos Labels Avg. Dur. Source Task Kinetics700 (2020) 650K 700 5.1
Action Recognition STAR benchmark (2021) 22K 207 28.6 Video Reasoning,
Temporal Action Localization, Bounding Box FineDiving (2022b) 3K 1065 3.2
Action Quality Assessment, Action Sequence

Table 1: Source Dataset Summary. Video datasets used as source datasets for
instruction tuning data generation, their source tasks, and the number of
unique labels.

##  3 Generated Dataset - VSTaR-1M

In this section, we detail the different source datasets utilized in our study
(Sec. 3.1) and analyze the generated Dataset, VSTaR-1M (Sec. 3.2).

Figure 3:  Qualitative Improvement of Data Generation over Cycles on
FineDiving. We initialize the model with Video-LLaVA (Cycle 0), where the
model cannot generate an answer ($\rightarrow|\times$) or rationalize the
label correctly ($|\rightarrow\times$). In the second cycle (Cycle 1), the
model still cannot generate an answer ($\rightarrow|\times$) but can
rationalize the video label ($\checkmark|\rightarrow$), which is selected for
instruction tuning. Finally, in the third cycle (Cycle 2), the model directly
generates a correct answer ($\checkmark|\rightarrow$), which is selected for
visual instruction tuning. We highlight in green correct answers, in red wrong
answers, and in yellow - hallucinations.

###  3.1 Source Datasets

In selecting source datasets, we selected datasets that contain diverse video
content and label types, please see Tab. 1. These include Kinetics700 (Smaira
et al., 2020), which has action recognition annotations and is particularly
large and diverse. FineDiving (Xu et al., 2022b) is an action quality
assessment dataset of Olympic diving events and has both an overall score and
action sequence annotations. Finally, STAR-benchmark (Wu et al., 2021), a
video reasoning dataset, also contains bounding box and temporal action
localization annotations. Tab. 1 contains the relevant dataset statistics,
e.g., the number of videos and labels per dataset.

###  3.2 Generated Dataset Analysis

#### Quantitative Analysis.

Through the application of , significant dataset augmentation was achieved
over two cycles of, illustrated in Fig. 4. This figure displays the Answer
Generation and Label Rationalization yield across the datasets source.
Notably, the initial application of Video-LLaVA on datasets like Kinetics700
and STAR-Benchmark showed significant Answer Generation success rates.
However, the FineDiving dataset presented a notable challenge, with Answer
Generation having no answers generated directly, underscoring the complexity
of the dataset and the critical role of Label Rationalization. By the end of
the second cycle, a substantial number of high-quality instances had been
produced, showcasing both the effectiveness of in converting labeled video
datasets into video instruction tuning dataset, as evidenced in Fig. 4.

#### Qualitative Analysis.

See Tab. 2 for examples of generated question-answer pairs. From Kinetics700,
we extracted an instance showcasing a video labeled ‘smashing’. Video-STaR
correctly identified a more fine-grained label, ‘chopping wood’. In the
FineDiving dataset, a clip depicting a complex dive was accompanied by the
question ‘On a scale from 1-100$\cdots$’ The model’s output text provided a
breakdown of the dive’s components, leading to a score (label), as would be
desired from an LVLM visual assistant. Finally, in the STAR benchmark,
questions are already provided; therefore, we utilized them directly.

| Kinetics700 | STAR-Benchmark | FineDiving
---|---|---|---
Video |  |  |
Question |  What type of activity is being performed in this video? |  Which object was put down by the person? |  On a scale from 1-100, how would you evaluate the execution of the actions in the video?
Generated Answer |  In the video, a man is seen holding a large axe and swinging it around while a crowd watches. He then proceeds to hit a block of wood with the axe. Based on these observations, it can be concluded that the activity is chopping wood. |  In the video, the person is seen picking up a bag and putting it down on the floor. After that, he picks up a book and puts it down on the floor. Based on this information, we can conclude that the person put down the bag and picked up the book. |  First, we identify the diver’s action sequence: inward somersault $\cdots$ entry into the water. Evaluating each movement’s difficulty, we assign $\cdots$ inward somersault: 2.5, $\cdots$. By averaging these scores $\cdots$ the overall score is 63.0 out of 100.
Label | Smashing | The bag. | Overall Score 63.0

Table 2: Examples of Generated Data. Examples of the video, question,
generated answer, and ground-truth label from each source dataset.

Figure 4:  Dataset Yield vs. Cycles. Percentage of the videos converted to
instruction tuning by the Answer Generation and Label Rationalization per
dataset. As can be seen, on difficult datasets, such as FineDiving, no videos
are converted by Answer Generation in the first cycle. By utilizing Label
Rationalization, the model is able to improve to eventually generate answers
correctly.

In 3, we show the qualitative improvement of the generated data over cycles.
In the first cycle (Cycle 0), Video-LLaVA failed at Answer Generation and
Label Rationalization. After one cycle (Cycle 1), still failed at Answer
Generation but succeeded in Label Rationalization. After the final cycle
(Cycle 2), managed to generate the answer without requiring the label via
Answer Generation.

##  4 Experiments

We experimented with and evaluated its enhanced video understanding
capabilities. In Sec. 4.3, we evaluate how adapts Large Vision-Language Models
(LVLMs) to the source datasets and how these capabilities are transferred
zero-shot to similar benchmarks. In Sec. 4.2, we evaluate the video question-
answering capabilities on video benchmark datasets.

###  4.1 Experimental Setting

#### Implementation Details.

We initialize from the Video-LLaVA (Lin et al., 2023) model, which utilizes
the Vicuna-7B v1.5 (Chiang et al., 2023). We ran three cycles, and each cycle
was initialized with the pre-trained Video-LLaVA weights. We train for one
epoch using a $128$ batch size, AdamW optimizer, and a cosine learning rate
schedule. The learning rate is $2e-5$ with a $0.03$ warmup ratio. In
combination with the generated instruction tuning dataset, we additionally
utilized the VideoInstruct-$100K$ (Maaz et al., 2023) and the LLaVA v$1.5$
instruction tuning datasets (Liu et al., 2023a). Additional details are
available in the appendix.

#### Baselines.

Besides comparing to Video-LLaVA, we also wanted to evaluate the effect of
utilizing additional data and naively adapting the source datasets. Therefore,
we utilized simple templates to generate question-answer pairs from the video
labels and trained Video-LLaVA on the resulting dataset. We will reference
this baseline as Video-LLaVA+. Another baseline for adapting Large Vision-
Language Models to novel tasks is model distillation, where a stronger video
model - in this work, Gemini $1.5$ pro-vision - is utilized to label/annotate
a small set of videos ($500$ from each dataset) and used to finetune the
models. We will reference this baseline as Vid-LLaVAGemini.

#### Evaluation Details.

We evaluate on the following benchmarks; the Zero-shot question-answer (QA)
benchmarks: MSVD-QA, MSRVTT-QA, TGIF-QA, and ActivityNet-QA (Xu et al., 2017;
2016; Jang et al., 2017; Heilbron et al., 2015). TempCompass (Liu et al.,
2024), a multiple-choice fine-grained QA benchmark. Adapted task performance
is evaluated by converting source datasets using simple templates and applying
the same evaluation protocol as Maaz et al. (Maaz et al., 2023), producing
Kinetics700-QA, STAR-benchmark-QA, and FineDiving-QA. This protocol reports
two metrics: accuracy (the percentage of correctly answered questions) and the
average score (where ChatGPT rates each response on a scale of 1-5 and
calculates the mean of these scores). All evaluations utilize the same GPT
model (Wu, 2024) (“gpt-3.5-turbo”) to ensure consistent comparisons. Due to
cost considerations, 1000 videos were randomly selected from each dataset for
Gemini evaluation. The reported values are used on ActivitlyNet-QA.

| Action | Direction | Speed | Event | Attribute Change | Avg.
---|---|---|---|---|---|---
| Fine | Coarse | Obj. | Cam. | Abs. | Rel. | Order | Color | Size | Both | Other |
Random | 39.7 | 40.1 | 39.8 | 39.0 | 40.8 | 42.0 | 41.5 | 40.4 | 39.9 | 38.9 | 39.4 | 40.5
mPLUG-Owl (2023) | 48.8 | 66.1 | 38.7 | 36.8 | 42.2 | 38.4 | 42.0 | 41.7 | 44.7 | 41.9 | 39.9 | 44.4
Video-LLaVA (2023) | 63.4 | 93.5 | 36.1 | 34.8 | 42.7 | 26.5 | 39.1 | 52.6 | 37.1 | 43.3 | 33.3 | 45.7
Video-LLaVA+ | 62.1 | 93.0 | 35.0 | 32.6 | 41.1 | 38.7 | 36.4 | 59.0 | 40.2 | 36.7 | 44.4 | 47.2
Vid-LLaVAGemini | 30.7 | 30.1 | 37.8 | 40.0 | 41.8 | 42.4 | 21.5 | 50.4 | 49.9 | 38.0 | 37.4 | 38.2
| 68.6 | 94.1 | 35.8 | 38.0 | 38.7 | 37.6 | 37.1 | 53.8 | 48.5 | 45.0 | 55.6 | 50.3$(+10\%)$
Gemini-1.5 (2024) | 94.8 | 98.4 | 43.6 | 42.4 | 65.3 | 48.7 | 55.6 | 79.5 | 59.8 | 70.0 | 66.7 | 66.0

Table 3: Comparison with state-of-the-art methods on TempCompass. TempCompass
(Liu et al., 2024) assesses the temporal understanding capabilities of video
language models across five dimensions improves Video-LLaVA performance on
TempCompass by $10\%$.

Methods Dataset MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA size Accuracy Score
Accuracy Score Accuracy Score Accuracy Score VideoChat (2023b) 4K 56.3 2.8
45.0 2.5 34.4 2.3 - 2.2 Video-LLaMA (2023a) 4K 51.6 2.5 29.6 1.8 - - 12.4 1.1
Video-ChatGPT (2023) 100K 64.9 3.3 49.3 2.8 51.4 3.0 35.2 2.7 Video-LLaVA∗
(2023) 100K 69.7 3.9 57.4 3.5 46.5 3.3 43.2 3.4 Video-LLaVA+ 650K 67.8 3.8
56.0 3.4 46.5 3.3 42.2 3.3 Vid-LLaVAGemini 2k 67.2 3.9 55.9 3.5 44.5 3.2 39.6
3,3 550K 71.3 4.0 58.2 3.5 47.3 3.3 43.2 3.3 Gemini-1.5-pro (2024) - 71.6 3.9
52.6 3.2 45.0 3.1 56.7 -

Table 4: Zero-shot Video QA benchmarks. As can be seen, many models are
approaching Gemini performance - indicating that LVLMs may be operating near
the noise level on these benchmarks.

###  4.2 Quantitative Evaluation on Zero-Shot Benchmarks

To evaluate ’s effect on general video question answering, we evaluated its
effect on Video-LLaVA’s performance on TempCompass, see Tab. 3. On
TempCompass, outperformed Video-LLaVA across the board– by $\sim 10\%$. To see
if this performance boost is simply a factor of training on a larger dataset,
we also evaluated Video-LLaVA+. Video-LLaVA+was trained on even a larger video
dataset by naively utilizing video labels, and yields a more modest
improvement of $3\%$, showing the utility of . TempCompass is also a fine-
grained dataset that would be sensitive to hallucinations, indicating that is
not more prone to hallucinations compared to existing methods. Gemini 1.5 pro
scored an impressive $66.0$ on TempCompass, showing there is still much room
for improvement on this benchmark.

We then continued and evaluated ’s effect on zero-shot video QA performance on
the MSVD-QA, MSRVTT-QA, TGIF-QA and ActivityNet-QA benchmarks. As can be seen
in Tab. 4, achieves performance improvements where, for instance, on the MSVD-
QA dataset, attains the highest accuracy of $71.3$% vs Video-LLaVA’s 69.7. On
MSRVTT-QA, leads with an accuracy of $58.2$% and maintains a competitive edge
in other datasets like TGIF-QA and ActivityNet-QA. Seeing the relatively small
performance gains compared to TempCompass, we additionally evaluated Gemini
$1.5$ pro-vision on $1000$ video subsets of each dataset and found that its
performance is on par with existing open-source models. We believe this shows
that we are near the ‘noise’ limit of these benchmarks. Our qualitative
analysis indicated that many of the questions selected as ‘wrong’ are actually
due to the benchmark design—overly general questions with multiple correct
answers. Concurrent work (Wu, 2024) has similarly concluded that the
ChatGPT-$3.5$ version utilized in evaluation can lead to variations of $\pm
10$ in accuracy.

Methods Kinetics700-QA STAR-bench-QA FineDiving-QA Accuracy Score Accuracy
Score Accuracy Score Video-LLaVA 50.0 3.2 24.9 2.6 17.6 2.2 Video-LLaVA+ 49.5
3.2 28.8 2.8 19.1 2.2 Vid-LLaVAGemini 41.9 2.9 22.3 2.6 16.3 2.1 59.9 (+20%)
3.5 (+10%) 33.0 (+33%) 2.9 (+12%) 20.2 (+15%) 2.3 (+5%)

Table 5: Adapted Dataset Performance. Performance metrics on test sets of
Kinetics700, FineDiving, and STAR-benchmark datasets via converting them to QA
following Maaz et al. (2023). shows significant improvement over Video-LLaVA
and Video-LLaVA+, showing the potential of for LVLM adaptation to new tasks.

###  4.3 Quantitative Evaluation on Adapted Datasets

Besides improving general visual question-answering performance, can also
adapt Large Vision-Language models to novel takes. To demonstrate this, we
converted the test sets (not included in training) of the source datasets –
Kinetics700, STAR-benchmark, and FineDiving. The results of these evaluations
are reported in Tab. 5. Adapting LVLMs with easier-to-collect labels can be
helpful in various applications, leading to a more versatile, multi-domain
capable assistant. When evaluating ’s impact on LVLM performance on the
diverse source datasets, we found that it significantly improves model
performance, particularly on complex tasks. For instance, on Kinetics700,
known for its extensive action categories, enhanced Video-LLaVA’s performance
accuracy by an average of 20% (as can be seen in Tab. 5), showcasing its
ability to develop generalized models adept across multiple domains.
Interestingly, Video-LLaVA+’s performance did not improve compared to Video-
LLaVA, and in some cases, even worsened, showing that one cannot directly
utilize labeled datasets for LVLM adaptation.

Action Quality Assessment (AQA) is a complex video task requiring detailed
action understanding, where significantly enhanced LVLM performance on the
FineDiving dataset. Our results show a notable improvement from $17.6$ to
$20.2$ in score prediction accuracy, highlighting ’s effectiveness in refining
LVLM’s temporal reasoning. However, allows LVLMs to not only rate a particular
dive but also explain the rationale behind each assessment. This rationale is
invaluable for many applications, effectively providing potential user
feedback for improvement. This advancement enables novel applications, from
sports coaching to automated feedback systems, by offering evaluations and
constructive feedback. The ability to interpret and improve action quality
underscores the potential of , underscoring the potential of utilizing LVLMs
as intelligent and informative visual assistants. For more, please see App.
Sec. A.2.

Ablations Kinetics700-QA STAR-bench-QA FineDiving-QA Accuracy Score Accuracy
Score Accuracy Score 59.9 3.5 33.0 2.9 20.2 2.3 \- Rationalization 59.8 3.5
26.6 2.7 12.8 2.0 \- Generation 50.0 3.2 24.9 2.6 17.6 2.2

Table 6: Ablations on Adapted Datasets. Performance metrics on test sets of
Kinetics700, STAR-benchmark, and FineDiving datasets. Label Rationalization
impacts mostly the difficult datasets, such as FineDiving, whose initial
Answer Generation yields are low.

Ablations MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA Accuracy Score Accuracy
Score Accuracy Score Accuracy Score 71.3 4.0 58.2 3.5 46.8 3.3 42.2 3.3 \-
Rationalization 70.6 3.9 57.5 3.5 47.7 3.4 42.2 3.3 \- Generation 69.7 3.9
57.4 3.5 46.5 3.3 43.2 3.4

Table 7: Ablations on Zero-Shot Benchmarks. In simpler benchmarks, Answer
Generation proved more critical for zero-shot generalization than Label
Rationalization.

###  4.4 Ablations

In our ablation studies, we evaluated the impact of removing Label
Rationalization and Answer Generation from , focusing on adapted datasets
(Kinetics700, FineDiving, STAR-benchmark) and zero-shot benchmarks (MSVD-QA,
MSRVTT-QA, TGIF-QA, ActivityNet-QA).

#### Adapted Datasets

For adapted datasets (Tab. 6), excluding Label Rationalization led to a
significant performance drop in FineDiving, from $20.2$ to $12.8$ in accuracy,
highlighting its critical role in complex reasoning tasks. This is likely due
to the lack of conversion of any examples from the data. However, the removal
of Answer Generation resulted in a more pronounced and uniform decline across
all datasets. For example, Kinetics700’s accuracy was reduced from $59.9$ to
$50.0$, underscoring its foundational role in generating context-relevant
responses.

#### Zero-shot benchmarks

In zero-shot benchmarks (Tab. 7), the removal of Label Rationalization had a
mixed impact, slightly affecting MSVD-QA where accuracy decreased from $71.3$
to $70.6$. The elimination of Answer Generation consistently lowered
performance, such as a decrease in MSRVTT-QA accuracy from $58.2$ to $57.4$.
ActivityNet-QA performance improved, probably because $100K$-Instruct utilizes
ActivityNet for instruction tuning. Therefore, the introduction of additional
videos decreases performance.

##  5 Related Works

###  5.1 Large Vision-Language Models

Initial LVLMs, such as LLaVA (Liu et al., 2023b; a) and BLIP-2 (Li et al.,
2023a), demonstrated the potential of merging image inputs with large language
models. Methods like mPLUG-Owl (Ye et al., 2023) and Flamingo (Alayrac et al.,
2022) further allowed for multiple image inputs without architectural changes.
Li et al. (2023b) and Zhang et al. (2024) led the transition to video
understanding, integrating video/image encoders and LLMs while training on
small video instruction tuning datasets. Jin et al. (2023) introduced Chat-
UniVi, a unified model employing dynamic visual tokens for both images and
videos, optimizing visual token usage and higher frame count sampling. LLaMA-
VID (Li et al., 2023c) showed that the token count can be further reduced by
pooling the tokens selectively via the text prompt using Q-Former. Recently,
Video-LLaVA (Lin et al., 2023) used modality-specific encoders for video and
image inputs to leverage LanguageBind encoders as they are constructively
aligned during pretraining and utilized a shared projection.

Maaz et al. (2023) expanded the field with the first large video instruction
tuning dataset, VideoInstruct-$100K$. VideoInstruct-$100K$ was generated from
ActivityNet (Heilbron et al., 2015) by prompting chatGPT with the video
captions, generating question-answer pairs. While driving much of the
performance improvement in the field (Jin et al., 2023; Wang et al., 2023; Lin
et al., 2023; Li et al., 2023c), upon examination of VideoInstruct-$100K$, it
is evident that it suffers from quality issues. The questions often degrade
into de facto prompts for a video caption (see Fig. 10) and rarely require
many spatiotemporal capabilities, which may limit LVLM performance.

###  5.2 Large Language Models and Self-Training

The advent of GPT (Radford et al., 2018; Brown et al., 2020) marked
significant milestones in natural language processing, showcasing LLMs’ power
in understanding and generating human-like text. Open-source LLMs like LLaMA
(Touvron et al., 2023a; b) and their instruction-tuned variants like Alpaca
and Vicuna (Taori et al., 2023; Chiang et al., 2023) further tailored these
models for nuanced human-AI interactions. However, even LLMs have found it
challenging to scale annotated datasets for training, prompting work on self-
training and self-improvement (Singh et al., 2023; Huang et al., 2022; Ho et
al., 2023; Marasović et al., 2022; Hosseini et al., 2024). In this line of
work, LLMs cycle between instruction-tuning data generation and instruction
tuning, iteratively improving LLM performance over cycles. For instance,
Zelikman et al. (2022) introduced the the Self-Taught Reasoners method, used
rationalization to generate chain-of-thought (CoT) reasoning, filtering poor
rationalizations to retain correctly answered questions. Other self-training
approaches include expectation-maximization-based approaches (Singh et al.,
2023), which alternate between data generation and improvement between
training cycles. Alternatively, majority voting has also been utilized to
generate answers and rationale for unlabeled questions (Huang et al., 2022).
These methods show the effectiveness of iterative self-training. In our work,
we aim to introduce a weakly supervised self-training approach for video
instruction tuning, leveraging video supervision that is often easier to
collect and exists in many large and diverse datasets.

##  6 Conclusions

In conclusion, Video Self-Taught Reasoners () presents a novel approach to
enhance Large Vision-Language Models (LVLMs) by enabling the use of diverse
labeled video datasets for visual instruction tuning. This method addresses
critical data diversity and quality challenges, leading to significant
performance improvements across various video understanding tasks. Our
experiments demonstrate ’s effectiveness in both source dataset adaptation and
zero-shot generalization, showcasing its potential in advancing LVLM
capabilities for complex video content.

The promising results of open new research avenues, particularly in expanding
LVLM knowledge bases using readily available image and video datasets. Future
work could explore advanced self-training techniques and integration with
emerging LVLM architectures, focusing on long-form video understanding to
further boost LVLM understanding. Additional work is also needed to reduce
hallucinations, perhaps by using grounded VLMs as auxiliary input.

Acknowledgements. We thank Google Research for providing financial support
through a Stanford HAI–Google collaboration. This work was partially supported
by the National Science Foundation under Grant No. 2026498. We thank the
Knight-Hennessy Scholars Foundation for funding OZ.

##  7 Limitations

While introduces a novel approach to visual instruction tuning, it is not
without its limitations. Firstly, the methodology can be computationally
intensive due to the cycling of both generating and rationalizing question-
answer and instruction tuning. Secondly, the assumption that all labels
necessitate a rationale may not always hold true. Certain labels might be
straightforward enough not to require elaborate rationalization, potentially
leading to unnecessary computational overhead. Lastly, hallucinations,
especially in label rationalization can be further reduced by perhaps
implementing additional verifiers.

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning, 2022.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Cohen (2020) Adam Cohen.  seatgeek/fuzzywuzzy : Release 0.18.0, February 2020.  URL https://pypi.org/project/fuzzywuzzy/.
  * Deng et al. (2024) Jiawen Deng, Kiyan Heybati, and Matthew Shammas-Toma.  When vision meets reality: Exploring the clinical applicability of gpt-4 with vision, 2024.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Grauman et al. (2023) Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, David Crandall, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray.  Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives.  _arXiv preprint arXiv:2311.18259_ , 2023.
  * Grunde-McLaughlin et al. (2021) Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala.  Agqa: A benchmark for compositional spatio-temporal reasoning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 961–970, 2015.  doi: 10.1109/CVPR.2015.7298698.
  * Ho et al. (2023) Namgyu Ho, Laura Schmid, and Se-Young Yun.  Large language models are reasoning teachers.  _arXiv preprint arXiv:2212.10071_ , 2023.
  * Hosseini et al. (2024) Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal.  V-star: Training verifiers for self-taught reasoners, 2024.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han.  Large language models can self-improve.  _ArXiv_ , abs/2210.11610, 2022.  URL https://api.semanticscholar.org/CorpusID:253080328.
  * Jang et al. (2017) Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2758–2766, 2017.
  * Jiang et al. (2024) Yixing Jiang, Jesutofunmi A Omiye, Cyril Zakka, Michael Moor, Haiwen Gui, Shayan Alipour, Seyed Shahabeddin Mousavi, Jonathan H Chen, Pranav Rajpurkar, and Roxana Daneshjou.  Evaluating general vision-language models for clinical medicine.  _medRxiv_ , pp. 2024–04, 2024.
  * Jin et al. (2023) Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan.  Chat-univi: Unified visual representation empowers large language models with image and video understanding.  _arXiv preprint arXiv:2311.08046_ , 2023.
  * Karamcheti et al. (2024) Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh.  Prismatic vlms: Investigating the design space of visually-conditioned language models.  _arXiv preprint arXiv:2402.07865_ , 2024.
  * Lawson et al. (2024) John R Lawson, Montgomery L Flora, Kevin H Goebbert, Seth N Lyman, Corey K Potvin, David M Schultz, Adam J Stepanek, and Joseph E Trujillo-Falcón.  Pixels and predictions: Potential of gpt-4v in meteorological imagery analysis and forecast communication.  _arXiv preprint arXiv:2404.15166_ , 2024.
  * Levenshtein (1965) Vladimir I. Levenshtein.  Binary codes capable of correcting deletions, insertions, and reversals.  _Soviet physics. Doklady_ , 10:707–710, 1965.  URL https://api.semanticscholar.org/CorpusID:60827152.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pp. 12888–12900. PMLR, 2022.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 19730–19742. PMLR, 23–29 Jul 2023a.  URL https://proceedings.mlr.press/v202/li23q.html.
  * Li et al. (2023b) Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_ , 2023b.
  * Li et al. (2023c) Yanwei Li, Chengyao Wang, and Jiaya Jia.  Llama-vid: An image is worth 2 tokens in large language models, 2023c.
  * Lin et al. (2023) Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.  Video-llava: Learning united visual representation by alignment before projection.  _arXiv preprint arXiv:2311.10122_ , 2023.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023a.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023b.
  * Liu et al. (2024) Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou.  Tempcompass: Do video llms really understand videos?  _arXiv preprint arXiv: 2403.00476_ , 2024.
  * Luo et al. (2022) Zelun Luo, Zane Durante, Linden Li, Wanze Xie, Ruochen Liu, Emily Jin, Zhuoyi Huang, Lun Yu Li, Jiajun Wu, Juan Carlos Niebles, et al.  Moma-lrg: language-refined graphs for multi-object multi-actor activity parsing.  _Advances in Neural Information Processing Systems_ , 35:5282–5298, 2022.
  * Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  _arXiv preprint arXiv:2306.05424_ , 2023.
  * Marasović et al. (2022) Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E. Peters.  Few-shot self-rationalization with natural language prompts.  _arXiv preprint arXiv:2111.08284_ , 2022.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning (ICML)_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 8748–8763. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/radford21a.html.
  * Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_ , 2024.
  * Senkaiahliyan et al. (2023) Senthujan Senkaiahliyan, Augustin Toma, Jun Ma, An-Wen Chan, Andrew Ha, Kevin R An, Hrishikesh Suresh, Barry Rubin, and Bo Wang.  Gpt-4v (ision) unsuitable for clinical care and education: a clinician-evaluated assessment. medrxiv.  _Preprint posted online on November_ , 16, 2023.
  * Singh et al. (2023) Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel.  Beyond human data: Scaling self-training for problem-solving with language models, 2023.
  * Smaira et al. (2020) Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman.  A short note on the kinetics-700-2020 human action dataset, 2020.  URL https://deepmind.com/research/open-source/kinetics.
  * Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wang et al. (2023) Yizhou Wang, Ruiyi Zhang, Haoliang Wang, Uttaran Bhattacharya, Yun Fu, and Gang Wu.  Vaquita: Enhancing alignment in llm-assisted video understanding.  _arXiv preprint arXiv:2312.02310_ , 2023.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=gEZrGCozdqR.
  * Wu et al. (2021) Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan.  STAR: A benchmark for situated reasoning in real-world videos.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.  URL https://openreview.net/forum?id=EfgNF5-ZAjM.
  * Wu (2024) Wenhao Wu.  Freeva: Offline mllm as training-free video assistant.  _arXiv preprint arXiv:2405.07798_ , 2024.
  * Xu et al. (2022a) Angchi Xu, Ling-An Zeng, and Wei-Shi Zheng.  Likert scoring with grade decoupling for long-term action assessment.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3222–3231, 2022a.  doi: 10.1109/CVPR52688.2022.00323.
  * Xu et al. (2017) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In _ACM Multimedia_ , 2017.
  * Xu et al. (2022b) Jinglin Xu, Yongming Rao, Xumin Yu, Guangyi Chen, Jie Zhou, and Jiwen Lu.  Finediving: A fine-grained dataset for procedure-aware action quality assessment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2949–2958, June 2022b.
  * Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5288–5296, 2016.
  * Ye et al. (2023) Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou.  mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023.
  * Yu et al. (2021) Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, and Jie Zhou.  Group-aware contrastive regression for action quality assessment.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 7919–7928, October 2021.
  * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.  STar: Bootstrapping reasoning with reasoning.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=_3ELRdg2sgI.
  * Zhang et al. (2023a) Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023a.
  * Zhang et al. (2024) Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=d4UiXAHN2W.
  * Zhang et al. (2023b) Shiyi Zhang, Wenxun Dai, Sujia Wang, Xiangwei Shen, Jiwen Lu, Jie Zhou, and Yansong Tang.  Logo: A long-form video dataset for group action quality assessment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2405–2414, June 2023b.
  * Zhang et al. (2023c) Shiyi Zhang, Wenxun Dai, Sujia Wang, Xiangwei Shen, Jiwen Lu, Jie Zhou, and Yansong Tang.  Logo: A long-form video dataset for group action quality assessment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2405–2414, June 2023c.
  * Zheng et al. (2024) Ziqiang Zheng, Yiwei Chen, Jipeng Zhang, Tuan-Anh Vu, Huimin Zeng, Yue Him Wong Tim, and Sai-Kit Yeung.  Exploring boundary of gpt-4v on marine analysis: A preliminary case study.  _arXiv preprint arXiv:2401.02147_ , 2024.
  * Zhou & Knoll (2024) Xingcheng Zhou and Alois C Knoll.  Gpt-4v as traffic assistant: An in-depth look at vision language model on complex traffic events.  _arXiv preprint arXiv:2402.02205_ , 2024.

##  Appendix A Appendix

In Sec. A.1, we provide additional implementation details and compute used in
developing . In Sec. A.2, we introduce explainable action quality assessment
and provide good and bad examples of on the FineDiving test dataset. Finally,
we provide additional qualitative Answer Generation and Label Rationalization
examples in Sec. A.3 and A.4.

###  A.1 Implementation Details

utilized the Video-LLaVA model, which integrated the Vicuna-7B v$1.5$ for
language processing and ViT-L/14 video and image encoders from LanguageBind
for visual encoding. The system’s tokenizer, adapted from LLaMA, has a
vocabulary size of around $32,000$ classes and a dimensionality of $4096$. Two
cycles of were executed, each initialed with the pre-trained Video-LLaVA model
(before instruction tuning). The training data was augmented by incorporating
VideoInstruct-$100$K and LLaVA v$1.5$’s visual instruction datasets.

Four clusters of $10$ NVIDIA Titan RTX GPUs were employed for $64$ hours. The
structured prompts for these tasks were as follows:

  * •

Answer Generation

> Question: {Q}.
> Can you explain step-by-step how one can arrive at this
> conclusion?

  * •

Label Rationalization

> Question: {Q}
> Answer: {L}.
> Can you explain step-by-step how one can arrive at this
> conclusion?

Figure 5:  Action Quality Assessments by on the FineDiving Test Set. Different
diving sequences with corresponding evaluations, from a high score of $85.78$
for complex sequences (top) to $74.8$ for intermediate (middle), and a lower
score of $54.6$ for basic sequences (bottom), showcasing ’s proficiency in
scoring dives with varying degrees of difficulty and execution quality.

These prompts guided the model in producing detailed answers and
rationalizations, enhancing the depth and utility of the generated
instruction-tuning dataset. Answer correctness was evaluated using template
matching with Levenshtein Distance-based Levenshtein (1965) fuzzy logic Cohen
(2020), considering an answer correct if all keywords from the label were
present in the generated response with a minimum similarity score of $80$%.
For example, if in Kinetics the action label is ‘eating apple pie’, we would
only consider a generated answer correct of ‘eating’, ‘apple’, ‘pie’ all
appeared with a similarity score of $80$.

###  A.2 Explainable Action Quality Assessment

Explainable Action Quality Assessment (AQA) is critical for detailed analysis
of performances in precision sports, such as competitive diving, where
execution and complexity significantly impact scores. Unlike previous AQA
works, which only provide a score Xu et al. (2022b; a); Yu et al. (2021);
Zhang et al. (2023b) not only generates scores but also offers detailed
justifications akin to expert analysis.

Fig. 5 illustrates instances where ’s predictions align closely with
established scoring criteria in diving. For example, the model breaks down a
dive sequence, Reverse$\rightarrow$$3.5$ Soms.Tuck$\rightarrow$Entry, with an
assigned difficulty of $3.5$, into its components. It then logically assigns
scores to each element, such as $1$ for more straightforward maneuvers and $4$
for more complex twists, culminating in an overall difficulty score of $3.3$,
close to the GT $3.5$. The final predicted score was $79.2$, close to the GT
score of $85.78$.

Figure 6:  Challenges in Action Quality Assessment by on the FineDiving Test
Set. Illustrating scenarios where diverges from optimal scoring: a complete
absence of prediction (top), an overestimation of a dive’s complexity and
execution (middle), and underestimation (bottom).

Figure 7:  Answer Generation Across Datasets by . Interpreting actions on
Kinetics700 (top), detailing action sequences in STAR-benchmark (middle), and
evaluating diving action quality in FineDiving (bottom).

’s proficiency extends to dives with varying levels of performance. It can
discern relatively complex dives with ground-truth scores of $74.8$ and
$54.6$, which scored as $76.5$ and $47.0$, respectively. In both cases, the
model breaks down the actions into sub-actions and rates them in terms of
difficulty and execution. While it manages to rate the dives themselves well,
in one instance, the model erroneously calculated the average for
$2.5,3.4,3.4,3.4$ as $3.4$.

Challenges in maintaining consistent AQA accuracy are depicted in Fig. 6,
showcasing instances of either the model not following instructions or
estimating the score incorrectly. In Fig. 6, top, the model did not produce a
score and resorted to explaining how a score might be derived. The model
occasionally produces an inaccurate score, and either grossly over (Fig. 6,
middle) or under (Fig. 6, bottom) estimates the score. For example, it
attributed a score of $72.0$ to a sequence deemed to have a lower difficulty
level of $35.0$. These discrepancies underscore the necessity for ongoing
improvements in the model’s grasp of nuanced scoring criteria to enhance
reliability in AQA predictions.

In summary, enhances Action Quality Assessment (AQA) by supplementing scores
with detailed rationales, an advancement over traditional AQA approaches that
only provide numerical scores. Although it effectively dissects the components
of diving performances, indicating both complexity and execution as seen in
Fig. 5, it faces challenges in maintaining consistent accuracy, particularly
in aligning scores with established benchmarks, as evidenced in Fig. 6. These
instances highlight the need for a deeper understanding of each movement’s
difficulty and execution quality to ensure the model’s scoring aligns with
professional judging standards. Additionally, they emphasize the ongoing
requirement to improve the model’s accuracy and ground its rationalizations in
the verifiable aspects of the performances it evaluates.

###  A.3 Qualitative Analysis of Answer Generation

Fig. 7 presents our model’s capabilities in generating answers and
rationalizing actions across varied video contexts, demonstrating its
adeptness in interpreting complex scenes.

#### Kinetics700.

In the first example from Kinetics700 (Fig. 7, Kinetics700, top), the model
effectively identifies a man’s act of pretending to be a statue and further
discerns the performance’s subtle aspects, such as the engagement level and
the humor conveyed through a sign. In another Kinetics700 example (Fig. 7,
Kinetics700, bottom), the model processes a scene with multiple concurrent
actions. first correctly identifies the man in the red shirt talking towards
the camera while holding a phone and texting. It correctly identifies the next
scene, where another man is texting while driving. This precision in
temporally locating different actions in the videos is invaluable for visual
instruction tuning and could potentially enhance models’ The capability to
analyze scenes with multiple focal points is an essential feature for
comprehensive video understanding.

Figure 8:  Successful Label Rationalizations by . The model accurately infers
a preparatory behavior from a person’s interaction with household items (top),
correctly interprets box-handling actions (middle), and provides a nuanced
breakdown of a complex diving sequence, assigning an overall score of $71.4$
(bottom), exemplifying precise action understanding on FineDiving and STAR-
benchmark.

Figure 9:  Challenges in Label Rationalization by . Instances of
rationalization errors are shown: an incorrect inference about dishwashing
(top) and fabricated details about closet interactions (middle). A
rationalization for a diving sequence (bottom) is accurate but demonstrates
the model’s vulnerability to hallucinations in complex action sequences within
FineDiving and STAR-benchmark datasets.

#### STAR-benchmark.

In the first STAR-benchmark example (Fig. 7, STAR- benchmark, top), the model
uses inferential reasoning to deduce the purpose behind a person consuming
medicine despite the absence of explicit details about the medicine. This
instance showcases the model’s ability to apply logical assumptions to fill in
informational gaps, a valuable trait for interpreting actions without fully
detailed context. In the next example (Fig. 7, STAR-benchmark, bottom),
identified additional details, such as the person sitting at the table eating
cake.

#### FineDiving.

in Fig. 7, FineDiving, ’s approach to evaluating a diving sequence in
FineDiving illustrates its proficiency in detailed performance analysis. By
deconstructing the dive into individual elements and assessing each for
difficulty, the model mirrors the evaluative processes of human judges,
providing a comprehensive performance score. This depth of analysis, which
includes a critique of the dive’s complexity and execution, underscores the
model’s utility in contexts requiring nuanced assessment, such as athletic
performance evaluation.

These examples from Fig. 7 show how the proposed Answer Generation is capable
of creating valuable and informative video question-answer pairs that can be
utilized in instruction tuning, highlighting its potential applicability in
various domains that demand a deep understanding of video scenes.

Figure 10:  VideoInstruct-$100K$ Qualitative Evaluation. Evaluation of
VideoInstruct-$100K$’s question distribution. A wide gap can be seen between
‘grounding’ questions, which contains ‘where/when’ (bottom) and the top-50
most common questions (top). when analyzing the top-50 most common questions
(top), it is clear they are all prompting for video captions.

###  A.4 Qualitative Examples of Label Rationalizations

Label rationalization in is initiated when direct answer generation by the
Large Multimodal Model (LMM) fails. Although this method proves beneficial in
certain situations, it occasionally leads to the generation of hallucinated
content—incorrect details or inferences not supported by the video.

#### STAR-benchmark.

Illustrated in Fig. 8, effective label rationalizations provide added context,
enriching the model’s responses. A notable example from the STAR benchmark
demonstrates the model’s capacity to build upon a basic action, like lying on
a couch, by inferring additional activities, such as donning a shirt and
tidying up, hinting at the individual’s subsequent actions. This example
illustrates ’s ability to navigate ambiguous labels and furnish more nuanced,
informative responses, crucial for Visual Instruction Tuning.

However, Label Rationalization is also more prone to hallucinations. Fig. 9
exposes the model’s tendency for hallucinations, mainly where it introduces
actions and details not evidenced in the video. In Fig. 9, STAR-benchmark,
top; the LMM hallucinated that after taking the dish, the man washed it and
placed it on a dry rack, which did not occur in the video. In Fig. 9, STAR-
benchmark, bottom; the model hallucinated that the camera panned to the right,
that it could see the reflection of the man in the mirror, and that he took
out shoes from the closet — none of which occur in the video.

#### FineDiving.

Label rationalization proved especially useful in challenging, domain-expert
datasets such as the Olympic diving scoring dataset - FineDiving. Answer
Generation initially had zero yield, and Label Rationalization allowed the
model to start learning about this challenging task. For example, in Olympic
diving events, to get an overall score for the dive, one removes the top and
bottom $2$ scores (out of a total of $7$), then multiplies this score with the
dive difficulty. In Fig. 8, FineDiving, the LMM correctly deduced that the
execution score is multiplied by the difficulty. With no supervision, the
model correctly deduced the rules, deducing that the final execution score is
obtained by multiplying the execution score and the action difficulty.

Fig. 9, FineDiving, Label Rationalization failed to generate an answer with
sufficient depth. Rather than providing additional insights, the model
resorted to re-iterating the labels.

In summary, Label Rationalization produces shorter, less informative answers
than Answer Generation and is more prone to hallucinations. It is primarily
utilized so complex datasets, such as FineDiving, can be learned, especially
in cases with initial zero yield.

###  A.5 Evaluation of Video Instruction Tuning Datasets

We performed a qualitative evaluation of VideoInstruct-$100K$ and found that
the broad majority of the questions essentially prompt the Large Vision-
Language model for video captions - see Fig. 10. As can be seen, $\sim 75\%$
of VideoInstruct-$100K$’s questions are of this type.