#  i-SRT: Aligning Large Multimodal Models for Videos
by Iterative Self-Retrospective Judgment

Daechul Ahn1,3,∗ Yura Choi1,3,∗ San Kim1,3
Youngjae Yu1 Dongyeop Kang2 Jonghyun Choi3,†
1Yonsei University 2University of Minnesota 3Seoul National University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Aligning Video Large Multimodal Models (VLMMs) face challenges such as
modality misalignment and verbose responses. Although iterative approaches
such as self-rewarding or iterative direct preference optimization (DPO)
recently showed a significant improvement in language model alignment,
particularly on reasoning tasks, self-aligned models applied to large video-
language models often result in lengthy and irrelevant responses. To address
these challenges, we propose a novel method that employs self-retrospection to
enhance both response generation and preference modeling, and call iterative
self-retrospective judgment (i-SRT). By revisiting and evaluating already
generated content and preference in loop, i-SRT improves the alignment between
textual and visual modalities, reduce verbosity, and enhances content
relevance. Our empirical evaluations across diverse video question answering
benchmarks demonstrate that i-SRT significantly outperforms prior arts. We are
committed to open-sourcing our code, models, and datasets to encourage further
investigation.††∗: equal contribution. †: corresponding author. Code:
https://github.com/snumprlab/SRT

Figure 1: Illustration of the proposed i-SRT. During iterative direct
preference optimization (DPO) in VLMM, we select preferences from responses
based on not only video content but also visual context $c_{t}$, _i.e_.,
detailed video description, to ensure preferences are grounded in video
information. Specifically, we enhance the context on the self-retrospective
manner by leveraging previously generated context $c_{t-1}$, a process we call
_self-retrospective_ preference modeling.

##  1 Introduction

Direct Preference Optimization (DPO) Rafailov et al. (2023), an alternative to
reinforcement learning for preference optimization Ouyang et al. (2022), has
become integral in training advanced Large Language Models (LLMs) Rafailov et
al. (2023) and Video Large Multimodal Models (VLMMs) Zhang et al. (2024a). On
top of DPO, recent work Yuan et al. (2024) has proposed a self-rewarding
approach that _iteratively_ utilizes a single LLM for both response generation
and preference generation (_i.e_., LLM-as-a-judge) and optimization using DPO.
This approach has shown promising results in aligning LLMs by iteratively
constructing more informative preference datasets.

However, the adaptation of self-rewarding strategies to VLMMs presents
different challenges than those in LLMs. VLMMs often face modality
misalignment during both response generation and preference modeling. This
misalignment leads to self-generated preferences that are linguistically
plausible but not grounded in the visual content of the associated video Ahn
et al. (2024). Moreover, it is well known that preference fine-tuning methods
such as RLHF, RLAIF, and DPO tend to produce responses that are longer than
those generated prior to preference optimization Prasann Singhal and Durrett
(2023). Although somewhat longer responses might enhance the quality of the
predicted response, excessively long responses can introduce irrelevant
content to the actual video or question, _i.e_., _hallucination_ , without
necessarily enhancing quality. The iterative nature of self-rewarding could
further exacerbate this verbosity, increasing the hallucination, which we term
_length bias_ within _self-rewarding cycle_.

Figure 2: Example of naively applying self-rewarding to VLMMs. (a) The average
length of generated responses increases steadily as the number of DPO
iterations rise in the video question answering benchmark Xu et al. (2017).
(b) Example of hallucination in longer responses. At the 1st iteration, the
response is concise and directly addresses the question. By the 5th iteration,
the response becomes more detailed, referencing explicit text overlays in the
video. By the 10th iteration, however, the response starts to include
excessive details and assumptions, leading to irrelevant content. This figure
highlights how self-rewarding for VLMM can initially enhance detail but
eventually cause overly verbose responses and hallucinations.  Figure 3:
Overview of iterative self-retrospective judgment. Our i-SRT consists of three
steps (Sec. 3.1): 1) At time $t$, the current VLMM ($\pi_{\theta^{t}}$)
generates responses $y_{1}$ and $y_{2}$ for the given video $V$ and question
$x$. In addition, a visual description, _i.e_., visual context, is generated
by a self-retrospective manner (Sec. 3.2), providing the necessary input for
the next stage as indicated in black dotted line, 2) Using the input from the
previous stage, the VLMM ($\pi_{\theta^{t}}$) classifies the preferred
response $y_{w}$ and rejected response $y_{l}$ from generated responses. This
preference data $D_{t}^{\text{pref}}$ is recorded for use in the subsequent
stage, 3) Then, the VLMM at time $t$ is optimized using DPO to update the
parameters to $\pi_{\theta^{t+1}}$.

We empirically examine the effect of applying self-rewarding in VLMMs on
response length, as illustrated in Fig. 2. As the iterative training
progresses, we observe that the response length increases, resulting in
detailed but increasingly redundant explanations, as shown in Fig. 1-(b). This
verbosity phenomenon is due to a bias in the preference data favoring longer
responses Prasann Singhal and Durrett (2023); Park et al. (2024). Despite
minimal differences in length between preferred and rejected responses, the
increase in verbosity is statistically significant Park et al. (2024).

To address these challenges of multimodal alignment in VLMMs using self-
rewarding, we introduce i-SRT: an iterative Self-Retrospective judgmenT
approach that generates responses and performs preference modeling (_i.e_.,
VLMM-as-a-judge) based on self-retrospection, as illustrated in Fig. 1. In
contrast to the naive application of self-rewarding in VLMM, we enhance our
model’s ability to select preferences by conditioning not only on the video
content – the primary visual and audio elements of the video – but also on the
visual context. This context includes detailed video descriptions generated by
the VLMM itself and serves a critical role similar to attention or focusing
within diverse content. During iterative preference modeling, this dual
conditioning on both video content and visual context improves the model’s
understanding and interpretation of video.

Moreover, we employ an iterative self-retrospective method to refine visual
context. By reviewing previously generated video description, _i.e_., visual
context, we generate more accurate and fine-grained descriptions in the
current iteration, thus deepening the understanding of visual content during
preference selection process, a process we refer to _self-retrospective_
preference modeling.

Our empirical studies demonstrate that our i-SRT exhibits superior performance
compared to state-of-the-art VLMMs on various video question answering
benchmarks, as discussed in Sec. 4.2.

We summarize our contributions as follows:

  * •

We propose a novel modality alignment method for video large multimodal models
(VLMMs), utilizing iterative direct preference optimization (DPO) to align
video-text modalities effectively.

  * •

We enhance AI’s feedback by proposing self-retrospective preference modeling,
which improves clarity and comprehension in video through the use of
iteratively refined visual context for preference selection.

  * •

We demonstrate the effectiveness of our proposed i-SRT on various video
question answering benchmarks by a noticeable margin.

##  2 Related Work

##### Aligning large multimodal models for videos.

Video Large Multimodal Models (VLMMs) Maaz et al. (2024); Liu et al. (2023a);
Lin et al. (2023) have achieved notable success in capabilities, such as video
temporal understanding Liu et al. (2023c), question answering Lin et al.
(2023), and instruction-following Maaz et al. (2024). These models integrate
publicly available Large Language Models (LLMs) Touvron et al. (2023a, b) with
visual encoders Radford et al. (2021) and additional learnable parameters Hu
et al. (2022), undergoing Supervised Fine-Tuning (SFT) and, more recently,
preference optimization Rafailov et al. (2023); Zhang et al. (2024a); Ahn et
al. (2024).

##### Self-rewarding for alignment.

Recent efforts to align LLMs with human intentions adopt self rewarding Yuan
et al. (2024); Lee et al. (2024); Pang et al. (2024); Chen et al. (2024),
where a model judges its own outputs to build feedback data and then trains on
this data using the DPO pipeline, known as iterative DPO or self-rewarding.
This iterative process of training and generating feedback with a single model
enhances the model’s ability to produce better responses, leading to
increasingly informative preference datasets and showing promising results in
aligning LLMs.

##  3 Iterative Self-Retrospective Judgment

To effectively align the multimodalities between video and text, we propose to
use an iterative approach that combines direct preference optimization (DPO)
with self-retrospective preference modeling. Figure 3 illustrates the overall
training pipeline of our proposed i-SRT, which iteratively executes three
stages: 1) generating context and responses, 2) selecting preferences, and 3)
optimization (Sec. 3.1).

Throughout this iterative process, we enhance our model’s ability to select
preferences by conditioning not only the video content but also on the visual
context. This visual context, consisting of detailed video descriptions
generated by the VLMM, functions like an attention mechanism, indicating which
parts of diverse video content to focus on. This integration enhances the
focus on relevant visual details, thereby improving the relevance of responses
by grounding them in the video content. It also mitigates the bias toward
lengthy responses, producing shorter, concise, yet accurate answers (see Sec.
4.3.1). Additionally, our approach utilize previously generated visual context
to construct a more comprehensive visual context at time $t$, a process we
call _self-retrospective_ judgment (Sec.3.2).

###  3.1 Iterative Direct Preference Optimization in VLMM

For notation, we denote the current VLMM at the $t$-th iteration as
$\pi_{\theta^{t}}$. This model generates responses and selects preferences
from those responses, thereby generating preference data denoted as
$D_{t}^{\text{pref}}$. Using generated preference data $D_{t}^{\text{pref}}$,
we train the subsequent iterative VLMM, denoted as $\pi_{\theta^{t+1}}$, at
the $t+1$-th iteration.

##### Initial model.

Given a seed of preference data annotated in Zhang et al. (2024a), we conduct
preference fine-tuning using DPO, starting from the SFT model provided from
previous work Zhang et al. (2024a). This preference fine-tuned model is
referred to as the initial model $\pi_{\theta^{1}}$.

##### Preference modeling.

Given the current VLMM $\pi_{\theta^{t}}$, we generate two different responses
for the input video $V$ and question $x$ using a high temperature hyper-
parameter value (_e.g_., 0.7). This high temperature flattens the token
sampling probability distribution, producing varied responses from the same
input in the current VLMM $\pi_{\theta^{t}}$:

$y_{1}\sim\pi_{\theta^{t}}(V,x),~{}y_{2}\sim\pi_{\theta^{t}}(V,x).$

We then select the better response between two responses by leveraging the
current VLMM to evaluate its own responses, _i.e_., VLMM-as-a-judge. In
particular, we provide the VLMM with the visual context $c_{t}$ for enhanced
visual clarity (more detailed in Sec. 3.2). We can present this preference
selection procedure as follows:

$(y_{w},y_{l})\sim\pi_{\theta^{t}}(V,x,c_{t},y_{1},y_{2}),$

where $y_{1}$ and $y_{2}$ are two sampled responses, $y_{w}$ is the chosen
response, and $y_{l}$ is the rejected response.

After constructing the preference data at $t$-th iteration as
$D_{t}^{\text{pref}}=\\{V,x,y_{w},y_{l}\\}$, we use this dataset to perform
preference optimization on the current VLMM $\pi_{\theta^{t}}$ using DPO. The
DPO objective for the current VLMM $\pi_{\theta^{t}}$ is represented as
follows:

$\begin{aligned} &\hskip
28.45274pt\mathcal{L}_{\text{DPO}}(\pi_{\theta^{t}};\pi_{\text{ref},t})=\\\
&-\mathbb{E}_{(V,x,y_{w},y_{l})\sim\mathcal{D}_{t-1}^{\text{pref}}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta^{t}}(y_{w}\mid
V,x)}{\pi_{\text{ref},t}(y_{w}\mid V,x)}\right.\right.\\\ &\hskip
105.2751pt\left.\left.-\beta\log\frac{\pi_{\theta^{t}}(y_{l}\mid
V,x)}{\pi_{\text{ref},t}(y_{l}\mid V,x)}\right)\right],\end{aligned}$

where $\pi_{ref,{t}}$ is the current base reference model, $\beta$ is a hyper-
parameter controlling the deviation from the current base reference model and
$\sigma$ is the sigmoid function.

##### Iterative training.

Our overall iterative training procedure follows previous work Yuan et al.
(2024), where a series of models $\pi_{\theta^{1}},\ldots,\pi_{\theta^{T}}$ is
trained sequentially. Each successive model at iteration of $t+1$ uses
preference data $D_{t}^{\text{pref}}$ generated by the VLMM at iteration $t$,
defined as follows:

$\pi_{\theta^{t+1}}:\text{Training with }D_{t}^{\text{pref}}\text{ initialized
from }\pi_{\theta^{t}},$

where the $t$-th model $\pi_{\theta^{t}}$ generates preference data
$D_{t}^{\text{pref}}$ through self-judgment.

Methods | ActivityNet-QA | VIDAL-QA | WebVid-QA
---|---|---|---
Acc. | Score | Acc. | Score | Acc. | Score
Video-ChatGPT Maaz et al. (2024) | 34.17 | 2.19 | 29.35 | 2.10 | 38.88 | 2.27
LLaMA-VID Li et al. (2023b) | 36.54 | 2.27 | 30.58 | 2.15 | 36.99 | 2.24
Chat-UniVi Jin et al. (2023) | 39.35 | 2.32 | 31.40 | 2.16 | 40.05 | 2.31
Video-LLaVA Lin et al. (2023) | 41.35 | 2.38 | 34.30 | 2.24 | 42.47 | 2.39
VLM-RLAIF Ahn et al. (2024) | 53.27 | 2.56 | 44.82 | 2.40 | 53.69 | 2.62
PLLaVA† Xu et al. (2024) | 48.44 | 2.50 | 42.45 | 2.39 | 53.55 | 2.59
LLaVA-NeXT-DPO† Zhang et al. (2024b) | 68.05 | 2.88 | 61.52 | 2.72 | 73.35 | 3.00
LLaVA-Hound-DPO† Zhang et al. (2024a) | 76.62 | 3.18 | 70.06 | 3.04 | 79.82 | 3.29
i-SRT | 82.99 | 3.26 | 79.00 | 3.13 | 88.11 | 3.40

Table 1: Quantitative comparison between different VLMMs on _in-domain_ video
question answering with detailed captions as supporting evidence proposed in
Zhang et al. (2024a). i-SRT consistently outperforms all other models in both
accuracy and score across these benchmarks, demonstrating its superior
performance in in-domain video question answering task. The best results are
bold and the second-best results are underlined. ${\dagger}$: reproduced by
the authors’ implementation. All results except ${\dagger}$ are directly
sourced from Zhang et al. (2024a).

Methods | MSVD-QA | MSRVTT-QA | TGIF-QA | SSV2-QA
---|---|---|---|---
Acc. | Score | Acc. | Score | Acc. | Score | Acc. | Score
Video-ChatGPT Maaz et al. (2024) | 34.06 | 2.20 | 25.65 | 1.98 | 31.35 | 2.09 | 19.36 | 1.75
LLaMA-VID Li et al. (2023b) | 34.14 | 2.21 | 25.02 | 1.99 | 27.18 | 2.00 | 22.16 | 1.84
Chat-UniVi Jin et al. (2023) | 35.61 | 2.23 | 25.89 | 2.01 | 33.23 | 2.13 | 20.59 | 1.79
Video-LLaVA Lin et al. (2023) | 39.46 | 2.37 | 30.78 | 2.15 | 32.95 | 2.18 | 24.31 | 1.90
VLM-RLAIF Ahn et al. (2024) | 51.16 | 2.55 | 41.44 | 2.30 | 46.52 | 2.41 | 29.78 | 1.94
PLLaVA† Xu et al. (2024) | 48.92 | 2.53 | 38.26 | 2.28 | 43.83 | 2.40 | 30.92 | 2.07
LLaVA-NeXT-DPO† Zhang et al. (2024b) | 65.08 | 2.82 | 59.12 | 2.65 | 60.80 | 2.70 | 40.14 | 2.24
LLaVA-Hound-DPO† Zhang et al. (2024a) | 73.64 | 3.12 | 68.29 | 2.98 | 74.00 | 3.12 | 48.89 | 2.53
i-SRT | 80.36 | 3.20 | 75.42 | 3.05 | 78.58 | 3.12 | 54.66 | 2.59

Table 2: Quantitative comparison between different VLMMs on _out-domain_ video
question answering with detailed captions as supporting evidence proposed in
Zhang et al. (2024a). i-SRT consistently outperforms all other models in both
accuracy and score across these benchmarks, demonstrating its superior
performance in out-domain video question answering task. The best results are
bold and the second-best results are underlined. ${\dagger}$: reproduced by
the authors’ implementation. All results except ${\dagger}$ are directly
sourced from Zhang et al. (2024a).

###  3.2 Self-Retrospective Judgment

A key aspect of iterative direct preference optimization in VLMMs involves
leveraging a pre-trained VLMM to iteratively select preferences, _i.e_., VLMM-
as-a-judge. These preferences need to be visually grounded in the input video
and accurately answer the posed questions Ahn et al. (2024). To ensure this,
we provide the VLMM with detailed visual descriptions, _i.e_., visual context,
generated by the VLMM itself at each iteration, in addition to the video
content. This dual conditioning enhances the understanding and interpretation
of the video during preference selection.

Furthermore, inspired by how humans refine their written text, we introduce
self-retrospective judgment to refine the quality of visual context during the
iterative process. Retrospection allows humans to make better decisions by
reflecting on past experiences Simon (1962); Madaan et al. (2023). Similarly,
we leverage previously generated visual context to enhance the current visual
context, improving the accuracy and relevance of the preference selection
process, defined as follows:

$c_{t}\sim\pi_{\theta^{t}}(V,c_{t-1}),$

where $c_{t-1}$ is the previously generated visual context at time $t-1$.

Using the generated context $c_{t}$, question $x$, video $V$, and responses
$\\{y_{1},y_{2}\\}$, we classify the chosen $y_{w}$ and rejected data $y_{l}$
from responses using the current aligned VLMM $\pi_{\theta^{t}}$, a process we
call _self-retrospective judgment_ , thereby constructing preference data
$D_{t}^{\text{pref}}=\\{V,x,y_{w},y_{l}\\}$ at time $t$.

Methods | MSVD-QA | MSRVTT-QA | TGIF-QA
---|---|---|---
Acc. | Score | Acc. | Score | Acc. | Score
VideoLLaMA Zhang et al. (2023a) | 51.6 | 2.5 | 29.6 | 1.8 | - | -
LLaMA-Adapter Zhang et al. (2023b) | 54.9 | 3.1 | 43.8 | 2.7 | - | -
VideoChat Li et al. (2023a) | 56.3 | 2.8 | 45.0 | 2.5 | 34.4 | 2.3
Valley Luo et al. (2023) | 60.5 | 3.3 | 51.1 | 2.9 | - | -
BT-Adapter Liu et al. (2023b) | 67.5 | 3.7 | 57.0 | 3.2 | - | -
Video-ChatGPT Maaz et al. (2024) | 68.6 | 3.8 | 58.9 | 3.4 | 47.8 | 3.2
Chat-UniVi Jin et al. (2023) | 70.0 | 3.9 | 53.1 | 3.1 | 46.1 | 3.1
VideoChat2 Li et al. (2024) | 70.0 | 3.9 | 54.1 | 3.3 | - | -
Video-LLaVA Lin et al. (2023) | 71.8 | 3.9 | 59.0 | 3.4 | 48.4 | 3.2
LLaMA-VID Li et al. (2023b) | 72.6 | 3.9 | 58.7 | 3.4 | 49.2 | 3.3
PLLaVA† Xu et al. (2024) | 78.8 | 4.0 | 65.6 | 3.4 | 57.9 | 3.5
LLaVA-NeXT-DPO† Zhang et al. (2024b) | 78.6 | 4.0 | 63.4 | 3.1 | 58.2 | 3.4
VLM-RLAIF† Ahn et al. (2024) | 81.0 | 4.2 | 69.2 | 3.7 | 62.3 | 3.5
LLaVA-Hound-DPO Zhang et al. (2024a) | 80.7 | 4.1 | 70.2 | 3.7 | 61.4 | 3.5
i-SRT | 85.8 | 4.3 | 78.7 | 3.9 | 67.8 | 3.5

Table 3: Quantitative comparison between different VLMMs on _out-domain_ video
question answering benchmark proposed in Maaz et al. (2024). i-SRT outperforms
previous work across three video question answering datasets. The best results
are bold and the second-best results are underlined. ${\dagger}$: reproduced
by the authors’ implementation. All results except ${\dagger}$ are directly
sourced from Zhang et al. (2024a).

##  4 Experiments

###  4.1 Experimental Setup

##### Dataset details.

For iterative direct preference optimization, we utilize only the open-source
video-text preference dataset (16k) proposed in Zhang et al. (2024a), in
contrast to previous work Yuan et al. (2024); Chen et al. (2024) that
incorporated additional data at each iteration. Except for the initial VLMM
$\pi_{\theta^{1}}$, we generate preference datasets $D_{t}^{\text{pref}}$ at
each iteration using questions $x$ and videos $V$ from the provided preference
dataset (16k) Zhang et al. (2024a). Generated preference dataset is then used
to train the subsequent VLMM $\pi_{\theta^{t+1}}$ using DPO training pipeline.

For evaluation, we employ two types of video question answering datasets
across seven video datasets Maaz et al. (2024); Zhang et al. (2024a). The
first type requires brief sentences or keywords, providing concise responses,
while the second type demands detailed sentences, necessitating comprehensive
and elaborate answers, as specified in Maaz et al. (2024) and Zhang et al.
(2024a), respectively.

##### Training details.

During training, we conduct full-parameter fine-tuning, using DPO. We set the
number of total iteration $N$ to 9, three times compared to previous self-
rewarding approach in LLMs alignment Yuan et al. (2024). All generative
processes, _i.e_., context, response, and preference as illustrated in Fig. 3,
entail their own prompts, as further detailed in Appendix Sec. A. All models
are trained using 8$\times$NVIDIA A100 GPUs (80G). We used a 7B-sized model
and compared results with other 7B models for fair comparison.

###  4.2 Quantitative Analysis

For quantitative evaluation, we use various video question answering
benchmarks Xu et al. (2017); Zhang et al. (2024a) that require both concise
and detailed responses, providing a comprehensive assessment of the model’s
capabilities in handling multimodalities of video and text. We use the 9-th
iteration of the VLMM (_i.e_., $N=9$) for our evaluation unless otherwise
specified.

##### In-domain video question answering.

We evaluate our model using three in-domain video question answering
benchmarks Zhang et al. (2024a). As shown in Tab. 1, our proposed method,
i-SRT, outperforms all existing models across these benchmarks in both
accuracy and score by a noticeable margin. We attribute this performance
improvement to the better modality alignment provided by the proposed
components.

Figure 4: Length analysis of preference dataset during iterative DPO. (a)
Average (Avg.) length of chosen response $|y_{w}|$ in preference dataset
$D_{t}^{\text{pref}}$ across DPO iterations. The naive application of self-
rewarding results in longer responses compared to the i-SRT. (b) Ratio of the
lengths of chosen responses ($|y_{w}|$) to rejected responses ($|y_{l}|$).
i-SRT consistently maintains a lowered ratio compared to the self-rewarding,
indicating reduced response length after optimized. ‘# DPO iteration’ means
the number of DPO iterations.  Figure 5: Comparison of average (Avg.) response
length between self-rewarding Yuan et al. (2024) and i-SRT on various video
question answering benchmarks. i-SRT yields more compact and concise responses
at the same iteration compared to self-rewarding.

##### Out-domain video question answering.

For evaluating out-domain video question answering, we use two types of
datasets. The first requires long, complex answers to questions, while the
second requires concise answers in the form of keywords or short phrases.
Tables 2 and 3 show the comparative results for datasets requiring both
complex answers and concise keyword answers, respectively. i-SRT outperforms
previous work by a large margin in both cases, demonstrating its effectiveness
in generating detailed and precise responses.

Moreover, the performance improvements observed during the iteration DPO
training are available in Appendix Sec. 10, exhibiting consistent enhancement
in performance.

###  4.3 Detailed Analysis

To evaluate the effectiveness of i-SRT, we address the following research
questions, specifically exploring the effect and design of visual context:

  * •

RQ1: What are the effects and benefits of visual context during iterative DPO?
(Sec. 4.3.1)

  * •

RQ2: How should the visual context be designed? (Sec. 4.3.2)

Context Design | Out-of-domain Video QA Benchmark | In-domain Video QA Benchmark
---|---|---
MSVD | MSRVTT | TGIF | SSV2 | ActivityNet | VIDAL | WebVid
Acc. | Score | Acc. | Score | Acc. | Score | Acc. | Score | Acc. | Score | Acc. | Score | Acc. | Score
| N/A
---
78.73 | 3.14 | 73.42 | 3.00 | 77.10 | 3.09 | 54.34 | 2.56 | 81.96 | 3.23 | 76.71 | 3.09 | 87.24 | 3.39
Fixed | 79.17 | 3.15 | 74.35 | 3.02 | 77.88 | 3.09 | 54.29 | 2.57 | 82.25 | 3.24 | 77.90 | 3.12 | 87.49 | 3.39
Renew | 79.49 | 3.19 | 74.04 | 3.04 | 77.63 | 3.12 | 53.03 | 2.56 | 82.03 | 3.26 | 77.73 | 3.12 | 86.68 | 3.38
| Self-retro.
---
80.36 | 3.20 | 75.42 | 3.05 | 78.58 | 3.12 | 54.66 | 2.59 | 82.99 | 3.26 | 79.00 | 3.13 | 88.11 | 3.40

Table 4: Quantitative comparison of various designs for generating visual
context. ‘N/A’ indicates no context, ‘Fixed’ uses context generated in the
first iteration for all subsequent iterations, ‘Renew’ generates new context
each iteration, and ‘Self-retro.’ employs self-retrospective context. Figure
6: Head-to-head performance comparison with and without visual context for
iterative DPO. We use the same 9-th iteration’s model on both i-SRT and self-
rewarding, and i-SRT overall outperforms the naive application of self-
rewarding.  Figure 7: Visualization of predicted context over iteration. We
visualize generated context of our i-SRT over DPO iteration. Red indicates
irrelevant responses, while blue indicates well-grounded responses.

####  4.3.1 Effect of visual context during iterative direct preference
optimization

##### Length analysis.

Figure 4 compares the average length of chosen responses and the ratio of
chosen responses to rejected responses. As shown in Fig.4-(a), i-SRT generates
shorter chosen responses compared to the naive application of self-rewarding
as training iterations progress. Similarly, Fig. 4-(b) shows a lower ratio of
the lengths of chosen responses to rejected responses in i-SRT. We believe
that dual conditioning on video content and context during preference
selection enables the VLMM to choose preferences based on video information
rather than length bias, resulting in relatively shorter chosen responses than
with the naive application of self-rewarding. Consequently, as shown in Fig.
5, i-SRT generates more concise responses than naive self-rewarding.

Figure 8: Qualitative comparison of self-rewarding _vs_. i-SRT at multiple
iterations. The figure contrasts descriptions generated without visual
context, i.e., naive self-rewarding (left), against those with visual context,
i.e., i-SRT (right), at the 5th and 9th iterations. Visual context results in
more accurate, concise, and relevant descriptions. Red indicates irrelevant
responses, while blue indicates well-grounded responses.

##### Performance analysis.

Following Yuan et al. (2024), we compare the 9-th iteration models of naive
self-rewarding and our i-SRT for both, using GPT-4 as the evaluator on video
question answering benchmarks. GPT-4 assesses performance by selecting the
response closer to the ground truth, _i.e_., evaluating win-rate with ground-
truth reference (details in Appendix Sec. C). Figure 6 shows the percentage
gap between self-rewarding and i-SRT across all benchmarks, demonstrating the
effectiveness of i-SRT.

##### Quality comparison of preference dataset.

Following Lee et al. (2023), We compare human labeler alignment accuracy for
preference dataset quality between self-rewarding and i-SRT as shown in Tab.
5. i-SRT shows higher alignment accuracy (75.0 %) compared to self-rewarding
(59.0 %), indicating a stronger correlation with human preferences. We believe
that by considering visual context in addition to video content, i-SRT
achieves a deeper understanding of the video, enabling the model to select
visually grounded preferences, similar to human judgment. More details are in
Appendix Sec. B.

Task | Human Alignment Accuracy (%)
---|---
Self-rewarding | i-SRT
Preference selection | 59.0 | 75.0

Table 5: Human labeler alignment accuracy for preference selection. We measure
human alignment accuracy to evaluate the amount of correlation between human
and aligned models, _i.e_., self-rewarding _vs_. i-SRT.

####  4.3.2 Various design choices for visual context

We compare different designs for context generation, as shown in Tab. 4: 1)
without context (‘N/A’), 2) using fixed context generated in the first
iteration for all subsequent iterations (‘Fixed’), 3) generating new context
at each iteration without any condition (‘Renew’) and 4) self-retrospective
context (‘Self-retro.’). ‘Self-retro.’ performs best, likely because it
leverages previously generated context to refine the current one, leading to a
better understanding of the video and demonstrating its efficacy.

###  4.4 Qualitative Analysis

##### Enhanced visual context over iteration.

Figure 7 shows visual contexts generated during the iterative DPO training. As
training iterations progress, the context includes more detailed visual
information about the video, such as specific species of goldfish. This
improved context enhances the overall understanding of the video’s content and
aids preference selection process.

##### Comparison of self-rewarding _vs_. i-SRT.

Figure 8 compares the performance of the naive application of self-rewarding
and i-SRT, highlighting their multimodal understanding capabilities over
multiple iterations during iterative DPO. This comparison demonstrates i-SRT’s
advantage in generating better-grounded responses to visual input, whereas
naive self-rewarding falls short as training iterations progress. More
examples are in the Appendix Fig. 11 for the space sake.

##  5 Conclusion

We propose a novel alignment strategy for VLMMs that addresses modality
misalignment and verbosity issues through an iterative self-retrospective
judgment (i-SRT) method. Our empirical validations demonstrate that i-SRT
significantly outperforms previous models across various video question
answering benchmarks. This indicates a robust improvement in the instruction-
following ability of multimodal video-text models.

## Limitations

Given that our approach, iterative self-retrospective judgement (i-SRT),
utilizes feedback generated by the AI model, the effectiveness of i-SRT
largely depends on the quality of AI responses. Recent studies Koo et al.
(2023); Das et al. (2024) have highlighted the challenges in enhancing the
quality of synthetically generated data, which is critical for reliable RL
framework. Additionally, our method’s performance can also be limited by the
computational resources required for processing and analyzing large-scale
video data. The iterative nature of i-SRT, while beneficial for improving
model alignment, also increases the computational overhead, potentially
limiting its scalability and practicality for real-time applications.
Moreover, the current implementation of i-SRT may not adequately address the
nuances of different languages and cultural contexts, which can affect the
model’s performance in global applications. The predominantly English-centric
data used in training may not generalize well to other languages or dialects,
potentially limiting the effectiveness of the model in non-English speaking
regions. Addressing these cultural and linguistic variations is another
critical area for future development to ensure the global applicability of our
approach.

## Ethical Considerations

##### Bias and fairness.

The iterative self-retrospective judgment (i-SRT) used in Video Large
Multimodal Models (VLMMs) risks inheriting and amplifying biases present in
their training datasets. Although the iterative nature of i-SRT may make it
less susceptible to these biases, they can still skew model performance and
decision-making, potentially leading to unfair outcomes for specific user
groups or scenarios. To mitigate this risk, it is crucial to regularly conduct
bias audits and employ fairness algorithms that adjust the model’s outputs.
Continuous monitoring for emergent biases as the model interacts with real-
world data is also essential.

##### Explainability.

The inherent complexity of multimodal models such as the iterative self-
retrospective judgment (i-SRT) often categorizes them as "black box" systems,
characterized by non-transparent decision-making processes. This lack of
transparency can render the models challenging for users to comprehend and
trust, which might impede their effective management and broader acceptance,
particularly in applications where a clear understanding of AI decisions is
critical. Nonetheless, i-SRT mitigates some of these transparency issues by
generating intermediate results during each iteration, which provide
incremental insights into its decision-making process. This feature of i-SRT
contributes to a reduced “black box” perception by offering users greater
visibility into how conclusions are incrementally built, thereby enhancing
both the trustworthiness and manageability of the system in its practical
applications.

## References

  * Ahn et al. (2024) Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. 2024.  Tuning large multimodal models for videos using reinforcement learning from ai feedback.  In _ACL_.
  * Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024.  Self-play fine-tuning converts weak language models to strong language models.
  * Das et al. (2024) Debarati Das, Karin De Langis, Anna Martin-Boyle, Jaehyung Kim, Minhwa Lee, Zae Myung Kim, Shirley Anugrah Hayati, Risako Owan, Bin Hu, Ritik Parkar, Ryan Koo, Jonginn Park, Aahan Tyagi, Libby Ferland, Sanjali Roy, Vincent Liu, and Dongyeop Kang. 2024.  Under the surface: Tracking the artifactuality of llm-generated data.
  * Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jin et al. (2023) Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. 2023.  Chat-univi: Unified visual representation empowers large language models with image and video understanding.  _arXiv preprint arXiv:2311.08046_.
  * Koo et al. (2023) Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023.  Benchmarking cognitive biases in large language models as evaluators.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2023.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_.
  * Lee et al. (2024) Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu. 2024.  Aligning large language models by on-policy self-judgment.  In _ACL_.
  * Li et al. (2023a) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023a.  Videochat: Chat-centric video understanding.  _arXiv preprint arXiv:2305.06355_.
  * Li et al. (2024) Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2024.  Mvbench: A comprehensive multi-modal video understanding benchmark.  _arXiv_.
  * Li et al. (2023b) Yanwei Li, Chengyao Wang, and Jiaya Jia. 2023b.  Llama-vid: An image is worth 2 tokens in large language models.  _arXiv preprint arXiv:2311.17043_.
  * Lin et al. (2023) Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023.  Video-llava: Learning united visual representation by alignment before projection.  _arXiv preprint arXiv:2311.10122_.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a.  Visual instruction tuning.  In _NeurIPS_.
  * Liu et al. (2023b) Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H. Li, and Ge Li. 2023b.  One for all: Video conversation is feasible without video instruction tuning.  _arXiv preprint arXiv:2309.15785_.
  * Liu et al. (2023c) Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. 2023c.  St-llm: Large language models are effective temporal learners.  _https://arxiv.org/abs/2404.00308_.
  * Luo et al. (2023) Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. 2023.  Valley: Video assistant with large language model enhanced ability.
  * Maaz et al. (2024) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)_.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Pang et al. (2024) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024.  Iterative reasoning preference optimization.
  * Park et al. (2024) Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024.  Disentangling length from quality in direct preference optimization.
  * Prasann Singhal and Durrett (2023) Jiacheng Xu Prasann Singhal, Tanya Goyal and Greg Durrett. 2023.  A long way to go: Investigating length correlations in rlhf.  _arXiv_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Simon (1962) Herbert A. Simon. 1962.  The architecture of complexity.  _Proceedings of the American Philosophical Society_ , 106(6):467–482.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Xu et al. (2017) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017.  Video question answering via gradually refined attention over appearance and motion.  In _ACM Multimedia_.
  * Xu et al. (2024) Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024.  Pllava : Parameter-free llava extension from images to videos for video dense captioning.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024.  Self-rewarding language models.
  * Zhang et al. (2023a) Hang Zhang, Xin Li, and Lidong Bing. 2023a.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_.
  * Zhang et al. (2023b) Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023b.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_.
  * Zhang et al. (2024a) Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. 2024a.  Direct preference optimization of video large multimodal models from language model reward.
  * Zhang et al. (2024b) Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024b.  Llava-next: A strong zero-shot video understanding model.

##  Appendix A Input Prompts for Generating Preference Dataset

We leverage three types of input prompts to generate preference dataset,
_i.e_., 1) response generation, 2) self-retrospective context generation and
3) preference judgment, as discussed in Sec. 3.1 and Sec. 4.1. The upper part
of Figure 9 presents a designed input prompt given to our method, resulting in
two responses. Also, the center part of Figure 9 illustrates the prompt to
generate a context based on the previous context. Lastly, the bottom part of
Figure 9 demonstrates the prompt for preference judgment using the current
VLMM.

Figure 9: Various input prompts for constructing preference dataset. This
shows various input prompts: the upper part for generating two responses, the
center part for context generation based on previous context, and the bottom
part for preference judgment using the current VLMM..  Figure 10: Accuracy of
i-SRT over iteration on video question aswering benchmarks. Our i-SRT
consistently improves its performance over DPO iteration.  Figure 11: More
qualitative example of prediction from self-rewarding _vs_. i-SRT. We compare
responses generated at the 9-th iteration for both models. Integrating visual
context leads to more accurate, concise, and relevant descriptions that align
more closely with the ground-truth answer. Red indicates irrelevant or wrong
responses, while blue indicates well-grounded responses.

##  Appendix B Details on Human Evaluation for Human Preference Alignment

Following Lee et al. (2023), we conduct a human evaluation to measure how well
the AI-generated preferences align with human preferences, similar to the
method used in the previous study. We recruit 15 annotators for each of the
100 questions using the Amazon Mechanical Turk platform. Specifically, we
randomly sample 100 questions from the dataset used in the previous study Xu
et al. (2017). Specific instructions and examples for the questions can be
found in Fig. 12.

Figure 12: Evaluation criteria provided to Amazon MTechanical Turk annotators.
We carefully instructed the annotators to penalize the outputs that includes
unaligned contents with the provided video, or the answer that contains overly
verbose sentences thar deviates from the question’s purposes.

##  Appendix C Details on Measuring Win Ratio with GPT-4

To assess the head-to-head comparison of generated responses between self-
rewarding Yuan et al. (2024) and i-SRT, we prompted GPT-4 to decide which of
the two responses is better, across in-domain and out-domain video question
answering benchmarks, as shown in Fig. 13. Here, we also introduced ‘Tie’
where it is hard to determine the preference between two responses.

Figure 13: Evaluation criteria provided to GPT-4. To compare the generated
responses of method without context and with context in Fig. 6, we prompted
GPT-4 to choose better response regrading three criteria: Relevance, Accuracy,
and Conciseness.


Evaluating and Crafting Datasets Effective for Deep Learning With Data Maps

Bishnu, Jay Gondoputro, Andrew

###### Abstract

Rapid development in deep learning model construction has prompted an
increased need for appropriate training data. The popularity of large datasets
- sometimes known as “big data” - has diverted attention from assessing their
quality. Training on large datasets often requires excessive system resources
and an infeasible amount of time. Furthermore, the supervised machine learning
process has yet to be fully automated: for supervised learning, large datasets
require more time for manually labeling samples. We propose a method of
curating smaller datasets with comparable out-of-distribution model accuracy
after an initial training session using an appropriate distribution of samples
classified by how difficult it is for a model to learn from them.

##  1 Introduction

Though larger datasets are often preferred for deep learning to maximize out-
of-distribution accuracy, their benefits begin to diminish at a particular
size ([1]). A dataset’s efficiency can be defined qualitatively as the
accuracy of a model trained on it relative to the amount of time and resources
needed to produce the model. Our work utilizes the PyTorch Dataset Cartography
project ([2]) to evaluate metrics of individual samples (dubbed “training
dynamics”) in a dataset after training. By using training dynamics to
represent the correctness, confidence, and variability in a model’s prediction
for a sample’s label, we can categorize samples by their difficulty for the
model to learn. With groups of these samples, we can craft a new data subset.
Ideally, a data subset constructed from a certain proportion of hard-to-learn,
easy-to-learn, and ambiguous samples could achieve comparable real-world
performance while requiring a fraction of the memory and training time.

##  2 Background

###  2.1 Datasets

The Dataset Cartography project is designed for use with several GLUE-style
datasets. For this project, we used the Stanford Natural Language Inference
(SNLI) Corpus ([3]). All GLUE-style datasets are stored in the TSV (tab-
separated value) format.

The SNLI dataset contains two statements per sample. During model production,
each pair is assigned a gold label111A label declared accurate, usually by a
human. \- entailment, contradiction, or neutral - based on how the second
statement relates to the first. The model is trained to predict the label of a
new sentence pair using the train split222A portion of the entire dataset. of
the data. The model’s out-of-distribution accuracy is measured by predicting
labels for samples in the out-of-distribution dev split before returning to
training. Once training has concluded, the model’s performance is evaluated on
the out-of-distribution test split.

Sentence 1 |  Sentence 2 |  Gold Label
---|---|---
Three black dogs on grass. |  Three dogs on grass. |  Entailment
Three men are sitting on chairs. |  The men are dancing together on the dance floor. |  Contradiction
Short dark hair woman with a black phone is on the phone on the sidewalk. |  A woman is on the phone with her husband. |  Neutral
Table 1: Samples from the train split of the SNLI dataset.

###  2.2 Data Maps

Manually estimating the effectiveness of each sample in a dataset for training
can be costly and time-consuming. The Dataset Cartography project was first
proposed as a way to characterize samples in a dataset through a scatter plot.
Following model evaluation, the samples in the model’s training sequence are
plotted according to their training dynamics. The y-axis represents a model’s
confidence in its prediction for the sample, and the x-axis represents the
variability in its confidence. These metrics can allow humans and computers to
identify from which samples the model learns easily and from those it
struggles with.

The Dataset Cartography project records new metrics after every epoch of
training. These metrics are used to label every example in a training split
with its correctness, confidence, and variability. Correctness represents the
fraction of epochs in which the model correctly predicted the label for a
given sample. Confidence is a numeric metric between 0 and 1 that measures the
mean of the model’s probabilities of a sample being labeled correctly across
epochs. Variability is the standard deviation of these probabilities.

Based on the confidence and variability of a data point, the project
classifies it into one of three categories: easy-to-learn, hard-to-learn, or
ambiguous. Easy-to-learn samples generally have low variability and high
confidence, hard-to-learn samples have low variability and low confidence, and
ambiguous samples have high variability.

Labeling data points in this way allows for data subsets to be filtered from
the original dataset based on specific criteria; for example, the samples the
model found easiest to learn from can be described as those with the highest
confidence. The project allows for a selection of samples - i.e., the 33% of
samples with the highest confidence - to be filtered into a new data subset.
Our method uses this feature to create new data subsets.

The original paper introducing the Dataset Cartography project describes
several out-of-dataset accuracy tests. The authors evaluated the accuracy of
models trained on the top 33% easiest-to-learn, hardest-to-learn, and most
ambiguous samples. The model trained on the ambiguous samples predicted the
correct label for both in-distribution and out-of-distribution data most
consistently.

However, further evaluations revealed that small ambiguous datasets performed
worse. This phenomenon could indicate that ambiguous samples alone were
insufficient for learning and that there may be a combination of sample types
with superior performance.

Figure 1: A data map for a DistilRoBERTa-base model trained on the SNLI
dataset without shuffling.

##  3 Methods

Our goal was to expand upon the training analysis from the original Dataset
Cartography project paper. We used the DistilRoBERTa-base transformers model
and the SNLI corpus for training and evaluation.

We trained a preliminary model on the entire dataset for six epochs to
classify samples as easy-to-learn, hard-to-learn, or ambiguous. Using the
filtering function, we sorted by the confidence metric to filter by the
hardest-to-learn and easiest-to-learn samples, and by the variability metric
to filter by the most ambiguous samples. We then created nine new subsets
after filtering the labeled data: a copy of the original dataset, but with
shuffled samples; a randomly selected 33% sample of the original dataset;
three 33% datasets, or one for the top 33% of each of the categories; three
16.67% + 16.67% datasets, with the three possible combinations of the
categories (i.e., 16.67% easy-to-learn and 16.67% hard-to-learn); and a
dataset containing the top 11% of samples from all three categories.

##  4 Findings

###  4.1 Results

After training all nine models on the new datasets, we recorded their accuracy
in labeling the in-distribution train set and the out-of-distribution dev and
test sets.

The model trained on the entire SNLI dataset had the highest out-of-
distribution accuracies, while the model trained on easiest-to-learn samples
had the highest in-distribution accuracy. The model trained on only the
hardest-to-learn and most ambiguous samples had the lowest out-of-distribution
accuracy, and the model trained on the hardest-to-learn samples had the lowest
in-distribution accuracy.

We found that the model trained on only the most ambiguous samples had
comparable out-of-distribution accuracies to the entire dataset’s, but poor
in-distribution accuracy. Excluding the model trained on the entire dataset,
the model with the highest overall accuracies was the one trained on the
easiest-to-learn and most ambiguous samples.

|  Final Training Accuracy (ID) |  Dev
Accuracy (OOD) |  Test
Accuracy (OOD)
---|---|---|---
100.00% |  0.8936 |  0.8997 |  0.8976
33.33% random |  0.8782 |  0.8776 |  0.8785
33.33% easy-to-learn |  0.9996 |  0.8293 |  0.8286
33.33% hard-to-learn |  0.5680 |  0.5966 |  0.5856
33.33% ambiguous |  0.7684 |  0.8878 |  0.8894
16.67% easy-to-learn
16.67% hard-to-learn |  0.7581 |  0.5938 |  0.5999
16.67% easy-to-learn
16.67% ambiguous |  0.8835 |  0.8802 |  0.8807
16.67% hard-to-learn
16.67% ambiguous |  0.5900 |  0.4076 |  0.4023
11.11% easy-to-learn
11.11% hard-to-learn
11.11% ambiguous |  0.7401 |  0.5249 |  0.5145
Table 2: Results from training and testing models based on the data subsets.

###  4.2 Mislabeled Data

The original Dataset Cartography paper discussed the idea of noisy data
(mislabeled samples), which could be more easily identified by filtering hard-
to-learn samples. The SNLI Dataset classifies pairs of sentences as either
entailment, neutral, or contradiction, but some samples with inaccurate gold
labels could interfere with training and evaluation. While exploring some of
the samples categorized as hard-to-learn, we found several mislabeled data
points. We strongly suspect these were partially responsible for the poor
performance of the models trained on subsets with the hardest-to-learn
samples.

Sentence 1 |  Sentence 2 |  Gold Label |  Our Label
---|---|---|---
A hiker travels along a rocky path bordered by greenery. |  The hiker is carrying a big backpack on his back. |  Entailment |  Neutral
Two beige dogs are playing in the snow. |  Two dogs are outside. |  Contradiction |  Entailment
A man with a gray beard is standing next to a woman in a brown jacket. |  Nobody is standing. |  Neutral |  Contradiction
Table 3: Mislabeled samples in the SNLI train split.

##  5 Conclusion

Our research demonstrated that a model trained on only specific samples in a
dataset could achieve accuracy comparable to one trained on the entire
dataset. In particular, models trained on subsets with a strong focus on
ambiguous samples and little-to-no focus on hard-to-learn samples generally
had excellent out-of-distribution performance, while those trained on easy-to-
learn samples had near-perfect in-distribution performance. We believe that
evaluating and refining a dataset based on the training dynamics of its
samples can allow for smaller, more efficient models with equivalent - or
potentially even superior - performance.

##  6 Further Research

The Dataset Cartography project has other applications in dataset and model
construction. Not only can it be used to identify mislabeled or low-quality
samples, but the filtering process could also be automated further to find an
ideal ratio of easy-to-learn, hard-to-learn, and ambiguous samples for any
particular dataset. Though this proof-of-concept project is currently
constrained to a select few datasets, the method can be generalized to fields
beyond natural language processing and categorization projects. A similar
process could allow model engineers to evaluate the quality of data used to
train generative adversarial networks (GANs) by analyzing the quality of
generated samples used by the GAN’s corresponding discriminator ([4]).

##  7 Acknowledgments

We are grateful to have performed our research under the guidance of Mark
Galassi, Rhonda Crespo, Maria de Hoyos, and the rest of the Institute for
Computing in Research team. Furthermore, the Software Freedom Conservancy
generously funded all of our research. We would also like to thank Dr. Ameeta
Agrawal and her Ph.D. student Yufei Tao at Portland State University for
spearheading our project. Finally, we thank the Allen Institute for Artificial
Intelligence for the original Dataset Cartography project and accompanying
research.

## References

  * [1] Michele Banko and Eric Brill  “Mitigating the Paucity-of-Data Problem: Exploring the Effect of Training Corpus Size on Classifier Performance for Natural Language Processing”  In _Proceedings of the First International Conference on Human Language Technology Research_ , 2001  URL: https://aclanthology.org/H01-1052
  * [2] Swabha Swayamdipta et al.  “Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics”  In _Proceedings of EMNLP_ , 2020  URL: https://arxiv.org/abs/2009.10795
  * [3] Samuel R. Bowman, Gabor Angeli, Christopher Potts and Christopher D. Manning  “A large annotated corpus for learning natural language inference”  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ Association for Computational Linguistics, 2015
  * [4] Ian J. Goodfellow et al.  “Generative Adversarial Networks”  In _Advances in Neural Information Processing Systems 27 (NIPS 2014)_ , 2014  URL: https://arxiv.org/pdf/1406.2661.pdf