(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: National University of Singapore (NUS), Singapore
22institutetext: Nanyang Technological University (NTU), Singapore
33institutetext: Carnegie Mellon University (CMU), USA 44institutetext: VinAI
Research, Vietnam
44email<EMAIL_ADDRESS>
# Meta-optimized Angular Margin Contrastive Framework for Video-Language
Representation Learning

Thong Nguyen First author11 Yi Bin Corresponding author11 Xiaobao Wu 22
Xinshuai Dong 33 Zhiyuan Hu 11
Khoi Le 44 Cong-Duy Nguyen 22 See-Kiong Ng 11 Luu Anh Tuan 22

###### Abstract

Data quality stands at the forefront of deciding the effectiveness of video-
language representation learning. However, video-text pairs in previous data
typically do not align perfectly with each other, which might lead to video-
language representations that do not accurately reflect cross-modal semantics.
Moreover, previous data also possess an uneven distribution of concepts,
thereby hampering the downstream performance across unpopular subjects. To
address these problems, we propose a contrastive objective with a subtractive
angular margin to regularize cross-modal representations in their effort to
reach perfect similarity. Furthermore, to adapt to the non-uniform concept
distribution, we propose a multi-layer perceptron (MLP)-parameterized
weighting function that maps loss values to sample weights which enable
dynamic adjustment of the model’s focus throughout the training. With the
training guided by a small amount of unbiased meta-data and augmented by
video-text data generated by large vision-language model, we improve video-
language representations and achieve superior performances on commonly used
video question answering and text-video retrieval datasets.

###### Keywords:

Video-language representation learning Large vision-language models
Contrastive learning Meta learning

##  1 Introduction

Figure 1: Examples of video inputs and their textual descriptions.

Figure 2: Topic distribution of the MSRVTT dataset. We use Latent Dirichlet
Allocation (LDA) to extract topics from manually annotated descriptions of
videos.

Learning vision and language representations has advanced state-of-the-art
across multiple cross-modal tasks, such as video question answering (VideoQA)
[27, 32, 16, 17, 8, 57, 66] and text-video retrieval [89, 64, 80, 26, 33, 67,
6]. The success of vision-language representation learning mainly results from
the effectiveness of contrastive learning, which projects video and text
inputs into a common latent space based on their semantic similarities.

Data quality stands at the forefront of influencing the efficacy of video-
language representation learning, particularly through its cleanliness and
diversity, which are pivotal in optimizing model performance and bolstering
generalization capabilities [88]. Within this context, we pinpoint two
interrelated issues that significantly impact the cleanliness and diversity of
video-text data: (1) the imperfection in alignment, characterized by a
scarcity of fine-grained details, and (2) the imbalance among the concepts of
data samples. First, a video and its textual description commonly does not
perfectly align with each other, e.g. the description may omit certain details
in the video, such as the green grass on a soccer field or the pink flowers at
the background in two videos of Fig. 1. As such, aggressively minimizing a
contrastive loss to pull video and language representations together might
result in distorted video-language representations that do not closely capture
the semantic similarity of video-text pairs, thus compromising the
interpretive video-language understanding. Second, as an example, Fig. 2 shows
a majority of video-text pairs in the well-known MSRVTT dataset denote the
video game or singing topic (the topic bars), while a minority of them
describe the fashion topic (the bottom bar). This might skew the model’s
exposure towards certain topics at the expense of others, undermining the
model’s ability to perform uniformly across a broad spectrum of subjects [28].
Addressing these intertwined challenges is crucial for advancing video-
language representation learning and facilitating more effective video-
language understanding.

In this paper, we propose a Subtractive Angular Margin Contrastive Loss, in
which we subtract a margin between a positive video and text sample in the
angular space. Our mathematical derivation shows that the subtracted margin
can decay the gradient norm, thus providing a regularization effect to
constrain positive but imperfectly aligned samples from reaching perfect
similarity. For the imbalance issue, to enable the network to dynamically
adjust its focus during training, we propose a sample reweighting strategy
that maps loss values to sample weights. A natural idea is to assign higher
weights to larger losses in order to emphasize minority classes for which the
network is likely to make mistakes [15, 63, 42]. Nevertheless, with respect to
the imperfect alignment issue, it is advisable to assign higher weights to
smaller losses, as there is a greater chance that these samples better align
with each other, hence forming cleaner samples which the network should focus
on [22, 87, 70]. To avoid the exhaustive effort in manually specifying a
weighting function and enhance the function generalization, we parameterize
our weighting function as a multi-layer perceptron (MLP) as theoretically a
universal approximator for any continuous function [10], and use a small
unbiased validation set (meta-data) to train the MLP.

Moreover, to further enhance the diversity in video-language representation
learning, we devise a strategy to utilize off-the-shelf large vision-language
model (LVLM) to augment downstream video-text data. In particular, given an
additional video input, we adopt the density peak-based clustering approach to
extract its key frames, then concatenate the extracted frames into one grid
image, and forward the image with a relevant input prompt to obtain the text
pairing. To sum up, our contributions are three-fold:

  * •

We propose a subtractive angular margin contrastive training objective to
regularize the training of positive video-text pairs to resolve the imperfect
alignment issue.

  * •

We propose an MLP-parameterized weighting function that reweighs the loss
values of training samples to control the imbalance of concepts described by
video-text data.

  * •

Combined with a strategy that utilizes an off-the-shelf LVLM to augment video-
text data, our framework outperforms previous state-of-the-art video-language
representation learning methods on standard MSRVTT, DiDeMo, ActivityNet, TGIF-
QA-R, NExT-QA, and Causal-VidQA datasets.

##  2 Related Work

Video-language representation learning. Recent approaches utilizing
contrastive learning have demonstrated impressive performance in video-
language understanding tasks including videoQA [65, 8, 86, 26, 67, 6] and
text$\leftrightarrow$video retrieval [1, 40, 25, 61, 39, 88, 81, 3, 36, 18, 4,
83, 49, 47]. VideoQA requires the model to infer an answer based on the fusion
of video and question representations, while text$\leftrightarrow$video
retrieval requires a model to project text query and video to a common latent
space, where their similarity is directly calculated. Collecting video-text
data is expensive and time-consuming. Therefore, previous methods take
advantage of audio transcripts [84, 85, 39, 71, 72] or adapt image-text models
to videos [26, 6, 27, 73]. Unfortunately, audio transcripts often do not
temporally align with the video content [19] and image-text models are
ineffective in temporal reasoning [26]. Instead, Zhao et al. [88] and Yang et
al. [84] fine-tune an LLM to be conditioned on visual frames to augment
textual descriptions for videos. However, due to the huge size of LLMs, their
approaches incur huge computational and storage cost.

Deep metric learning (DML). Contrastive learning is an implementation of DML.
Its target is to learn a function that maps data samples into a latent space,
where semantically similar objects stay close while dissimilar ones are far
away [60, 41, 50, 48, 52, 45, 46, 51, 76, 77, 74, 75]. Numerous efforts
introduce angular margin between two input samples to enhance discriminative
power of input representations [12, 9]. Nevertheless, little attention is paid
to the imperfect alignment between two positive samples. The situation is more
challenging for video and language, since these modalities exhibit a more
distinct semantic gap than any cross-modal input pairs.

Sample Reweighting. To manage the impact of each sample upon the training
process, sample reweighting pre-evaluates samples by mapping training loss to
sample weight, and dynamically modulating the weight throughout training.
There are two approaches to design the mapping function. The first approach
monotonically increases the weight when the loss increases, which adapts to
the class imbalance case as minor classes tend to engender higher loss values.
Methods that belong to this first group include focal loss [5, 35], hard
example mining [29, 42], and boosting algorithm [15]. The second approach
monotonically decreases the weight for larger training loss values [22, 14,
87]. In particular, self-paced learning (SPL) [22] and iterative reweighting
[14, 87] put more emphasis upon cleaner samples with smaller losses. Both
approaches need to pre-specify the scheme of their weighting function. This
could restrain their flexibility to adapt to diverse scenarios in practice,
e.g. training data might possess both label noise and and class imbalance
issue.

Figure 3: Our meta-optimized angular margin contrastive framework, which also
uses an off-the-shelf large vision-language model to augment textual
descriptions.

##  3 Methodology

In this section, we explain our meta-optimized angular margin contrastive
framework with our augmentation strategy for video-language representation
learning. We provide an overall illustration of our framework in Figure 3.

###  3.1 Video-Language Representation Learning

We are given a corpus of video data, in which each video $\mathbf{v}_{i}$ is
attached with a textual description $\mathbf{t}_{i}$ and possibly a label
$y_{i}$. In the beginning, we embed the video input $\mathbf{v}_{i}$ into a
sequence of visual representations
$\mathbf{V}_{i}^{f}=\\{\mathbf{h}_{i,j}^{f}\\}_{j=1}^{N_{\mathbf{v}_{i}}}$,
where $N_{\mathbf{v}_{i}}$ is the number of randomly sampled video frames in
$\mathbf{v}_{i}$. We also embed the textual description into a sequence of
representations
$\mathbf{T}^{w}_{i}=\\{\mathbf{h}_{i,j}^{w}\\}_{j=1}^{N_{\mathbf{t}_{i}}}$,
where $N_{\mathbf{t}_{i}}$ is the number of words in $\mathbf{t}_{i}$.

In this work, we conduct video-language representation learning for two types
of video-language model, i.e. dual and bidirectional model. For the dual type,
we will pool the visual representations $\mathbf{V}^{f}_{i}$ and textual
representations $\mathbf{T}^{w}_{i}$ to obtain global representations
$\mathbf{h}^{V}_{i}$ and $\mathbf{h}^{T}_{i}$ for the video and text input,
respectively. We use cosine similarity to calculate the cross-modal similarity
$S_{\mathbf{v}_{i},\mathbf{t}_{i}}$. Then, we aim to maximize the similarities
$S_{\mathbf{v}_{i},\mathbf{t}_{i}}$ of video-text pairs within the data,
relative to those of the unaligned video-text pairs.

The bidirectional model will concatenate the visual and textual
representations into a sequence, then feed the sequence to Transformer
attention layers to capture the video-text relations. Subsequently, we will
forward the [CLS] token which represents the video-text sequence to a fully-
connected (FC) layer. The FC layer will calculate the similarity
$S_{\mathbf{v}_{i},\mathbf{t}_{i}}$ if the video input $\mathbf{v}_{i}$ has a
text pairing $\mathbf{t}_{i}$, or calculate the log-likelihood $\log
q(y_{i}|\mathbf{v}_{i},\mathbf{t}_{i})$ of the answer if the video input
$\mathbf{v}_{i}$ has a label $y_{i}$. In the former case, similar to the dual
type we also maximize the similarity of in-distribution video-text pairs,
whereas for the latter case we maximize the log-likelihood of the answer label
$y_{i}$.

In general, for all video inputs, we maximize the similarity
$S_{\mathbf{v},\mathbf{t}}$ through minimizing the cross-modal contrastive
loss as:

$\small\mathcal{L}^{v,t}_{\text{contrastive},i}=-\log\frac{e^{S_{\textbf{v}_{i},\textbf{t}_{i}}/\tau}}{\sum\limits_{j=1}^{B}e^{S_{\textbf{v}_{i},\textbf{t}_{j}}/\tau}},\;\;\;\;\mathcal{L}^{t,v}_{\text{contrastive},i}=-\log\frac{e^{S_{\textbf{t}_{i},\textbf{v}_{i}}/\tau}}{\sum\limits_{j=1}^{B}e^{S_{\textbf{t}_{i},\textbf{v}_{j}}}/\tau},$
(1)

where $B$ is the batch size and $\tau$ is the temperature hyperparameter. If a
video $\mathbf{v}_{i}$ additionally exhibits an answer label $y_{i}$, we
jointly combine the cross-entropy loss to maximize the log-likelihood of the
label with the above contrastive loss:

$\displaystyle\small\mathcal{L}_{\text{CE},i}=-\sum\limits_{y_{i}}p(y_{i}|\mathbf{v}_{i},\mathbf{t}_{i})\log
q(y_{i}|\mathbf{v}_{i},\mathbf{t}_{i}),$ (2)
$\displaystyle\mathcal{L}_{\text{train},i}=\mathcal{L}^{v,t}_{\text{contrastive},i}+\mathcal{L}^{t,v}_{\text{contrastive},i}+\eta\cdot\mathcal{L}_{\text{CE},i},$
(3)

where $\eta$ is a hyperparameter to balance both losses. It is worth noting
that $\eta=0$ if a video input $\mathbf{v}_{i}$ is not attached with $y_{i}$.

###  3.2 Meta-optimized Angular Margin Contrastive Framework

Algorithm 1 Our meta-optimized learning framework

0: Training data $\mathcal{D}$, meta-data $\widehat{\mathcal{D}}$, training
batch size $B$, meta-data batch size $M$, initialized video-language model
parameter $\Theta^{(0)}$ and MLP network parameter $\theta^{(0)}$

0: Video-language model parameter $\Theta^{(K)}$

1: for $k=0$ to $K-1$ do

2: Sample a training minibatch $\\{\textbf{v},\textbf{t},y\\}$.

3: Sample a meta-data minibatch
$\\{\textbf{v}^{\text{meta}},\textbf{t}^{\text{meta}},y^{\text{meta}}\\}$.

4: Estimate $\hat{\Theta}^{(t)}$ by Eq. (9).

5: Update $\theta^{(k+1)}$ by Eq. (10).

6: Update $\Theta^{(k+1)}$ by Eq. (11).

7: end for

Subtractive angular margin contrastive learning. As cosine similarity is used
to calculate $S_{\mathbf{v}_{i},\mathbf{t}_{j}}\in[-1,1]$, we denote the angle
between the representation of video $i$ and text $j$ as:

$\small\lambda_{i,j}=\arccos\left(S_{\mathbf{v}_{i},\mathbf{t}_{j}}\right).$
(4)

The original video-language representation learning minimizes $\lambda_{i,i}$
to approach a 0-degree angle. However, since the video and its textual
description commonly does not perfectly align with each other, we want the
gradient to be regularized when the similarity of positive pair
$\textbf{v}_{i}$ and $\textbf{t}_{i}$ becomes small. Particularly, we replace
$\mathcal{L}_{\text{contrastive},i}^{v,t}$ by a new training objective:

$\mathcal{L}_{\text{angular},i}^{v,t}=\begin{cases}-\log\frac{e^{\cos\left([\lambda_{i,i}-\mu]_{+}\right)/\tau}}{e^{\cos\left([\lambda_{i,i}-\mu]_{+}\right)/\tau}+\sum\limits_{j\neq
i}e^{\cos(\lambda_{i,j})/\tau}},&\text{if }\lambda_{i,i}\leq\frac{\pi}{2}\\\
-\log\frac{e^{\cos(\lambda_{i,i})/\tau}}{e^{\cos(\lambda_{i,i})/\tau}+\sum\limits_{j\neq
i}e^{\cos(\lambda_{i,j})/\tau}},&\text{otherwise},\end{cases}$

A similar objective $\mathcal{L}_{\text{angular},i}^{t,v}$ is formulated
between $\mathbf{t}_{i}$ and $\mathbf{v}_{i}$. As long as the angular
difference $\lambda_{i,i}$ is smaller than $\mu$, the similarity score will
become 1. On the other hand, if $\lambda_{i,i}$ is larger but starts to turn
small, i.e. $\mu\leq\lambda_{i,i}\leq\frac{\pi}{2}$, the subtractive margin
$\mu$ will temporarily pull the positive video and text closer to regularize
the gradient of the positive samples, thus restraining them from reaching
perfect similarity. This intuition is formalized by the following theorem:

###### Theorem 3.1

Let $\lambda_{i,j}$ denote the angle between the representation of two samples
$i,j$, $\mathcal{L}^{v,t}_{\textup{angular},i}$ and
$\mathcal{L}^{v,t}_{\textup{contrastive},i}$ denote the training objectives
with and without the angular margin, respectively. Then, if
$\lambda_{i,i}\leq\frac{\pi}{2}$, the following inequality holds:

$\small\left|\frac{\partial\mathcal{L}_{\textup{angular},i}^{v,t}}{\partial\lambda_{i,i}}\right|\leq\left|\frac{\partial\mathcal{L}_{\textup{contrastive},i}^{v,t}}{\partial\lambda_{i,i}}\right|$
(5)

We provide the proof in Appendix 0.A. In order to speed up the training in the
beginning and constrain the update of positive pairs in the latter stage, we
adopt an adaptive strategy that gradually increases the margin $\mu$ towards a
limit:

$\small\mu^{(k)}=\frac{a_{0}}{a_{1}+e^{-a_{2}\cdot k}},$ (6)

where $k$ denotes the training step, and $a_{0}$, $a_{1}$, and $a_{2}$ denote
hyperparameters.

MLP-parameterized weighting function. To control the effect of the data
imbalance issue, we construct a weighting function as a $\theta$-parameterized
MLP to map each training loss to a sample weight:

$\small
w(\theta,\mathcal{L}_{i}^{\text{train}}(\Theta))=\text{MLP}_{\theta}(\mathcal{L}_{i}^{\text{train}}(\Theta)),\;\;i\in\\{1,2,…,B\\},$
(7)

where $\mathcal{L}_{i}^{\text{train}}(\Theta)$ denotes the training loss of
sample $i$ in a batch of size $B$, calculated using the $\Theta$-parameterized
video-language model.

Then, the final training objective is the sum of the training losses of data
samples weighted by $w$:

$\small\mathcal{L}(\Theta,\theta)=\frac{1}{B}\sum\limits_{i=1}^{B}w(\theta,\mathcal{L}_{i}^{\text{train}}(\Theta))\cdot\mathcal{L}_{i}^{\text{train}}(\Theta).\vspace{-5pt}$
(8)

Meta-optimized learning framework. At present, we have a
$\theta$-parameterized MLP network and a $\Theta$-parameterized video-language
model to train. Since jointly training these two models might be unstable
[59], we instead develop a meta-learning approach. In our procedure, we
manually extract a small amount of unbiased meta-data set
$\\{\textbf{v}_{i}^{\text{meta}},\textbf{t}_{i}^{\text{meta}},y_{i}^{\text{meta}}\\}_{i=1}^{M}$,
i.e. with semantically aligned textual description and balanced label
distribution, representing the meta-knowledge of the groundtruth sample-label
distribution, where $M$ is the number of the meta-samples and $M\ll N$.

We first estimate the update of the video-language model parameters $\Theta$:

$\small\hat{\Theta}^{(k)}=\Theta^{(k)}-\frac{\alpha}{B}\sum\limits_{j=1}^{B}w_{i}\left(\theta^{(k)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)\cdot\nabla_{\Theta}\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right),\vspace{-5pt}$
(9)

where $\alpha$ is the learning rate for the video-language model.
Subsequently, we update the MLP network using the estimated $\hat{\Theta}$:

$\small\theta^{(k+1)}=\theta^{(k)}-\frac{\eta}{M}\sum\limits_{i=1}^{M}\nabla_{\theta}\mathcal{L}^{\text{meta}}_{i}\left(\hat{\Theta}^{(k)}\right),\vspace{-5pt}$
(10)

where $\beta$ is the learning rate for the MLP network. Lastly, we obtain the
new video-language model’s parameters as:

$\vspace{-5pt}\begin{split}\Theta^{(k+1)}=\Theta^{(k)}-\frac{\alpha}{B}\sum\limits_{j=1}^{B}w\left(\theta^{(k+1)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)\cdot\nabla_{\Theta}\mathcal{L}^{\text{train}}_{j}\left(\Theta\right)\Big{|}_{\Theta^{(k)}}.\end{split}\vspace{-5pt}$
(11)

We summarize our algorithm in Algorithm 1. From an empirical perspective, we
compare the effectiveness of joint learning and our meta-optimized framework
in Section 4.3. From a theoretical perspective, we conduct further derivation
for Eq. (10), resulting in the following formulation (more details in Appendix
0.B):

$\displaystyle\theta^{(k+1)}=\theta^{(k)}+\frac{\alpha\beta}{BM}\sum_{i=1}^{M}\sum_{j=1}^{B}G_{ij}\frac{\partial
w\left(\theta^{(k)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)}{\partial\theta}\Big{|}_{\theta^{(k)}},$
(12) $\displaystyle
G_{ij}=\nabla_{\hat{\Theta}}\mathcal{L}_{i}^{\text{meta}}\left(\hat{\Theta}\right)\Big{|}_{\hat{\Theta}^{(k)}}\cdot\nabla_{\Theta}\mathcal{L}_{j}^{\text{train}}\left(\Theta\right)\Big{|}_{\Theta^{(k)}},$
(13)

where $G_{ij}$ denotes the coefficient of the gradient. The coefficient will
increase for training samples whose gradient is in the same direction with the
gradient of the meta-data (meta gradient), as this is more likely to be a
clean learning direction. In contrast, the coefficient will decrease for those
whose gradient is opposite from the meta gradient, since the learning
direction is likely to be noisy.

###  3.3 Large Vision-Language Model for Augmentation

To further enhance our video-language representation learning, we devise a
strategy to utilize large vision-language model to augment additional video-
text data. Our strategy consists of two stages, i.e. extracting key frames and
generating textual descriptions, which we delineate as follows:

Key frame extraction. Inspired by [23], we adopt a density peak-based
clustering approach to extract key video frames. Starting with a sequence of
visual frame features
$\textbf{V}_{i}^{f}=\\{\mathbf{h}^{f}_{i,j}\\}_{j=1}^{N_{\textbf{v}_{i}}}$, we
calculate the locality density $d_{i,j}$ of each feature
$\mathbf{h}^{f}_{i,j}$ based on its $K$-nearest neighbors:

$\vspace{-10pt}\small
d_{i,j}=\exp\left(-\frac{1}{K}\sum\limits_{\textbf{h}^{f}_{i,l}\in\text{KNN}(\mathbf{h}^{f}_{i,j})}\left|\left|\textbf{h}^{f}_{i,l}-\textbf{h}^{f}_{i,j}\right|\right|^{2}\right),$
(14)

where $\text{KNN}(\mathbf{h}^{f}_{i,j})$ denotes the $K$-nearest neighbors of
$\mathbf{h}^{f}_{i,j}$.

Subsequently, we estimate the distance index $\gamma_{i,j}$ of each frame
$\mathbf{h}_{i,j}^{f}$:

$\displaystyle\small\gamma_{i,j}=\begin{cases}\min\limits_{l:d_{i,j}>d_{i,l}}||\mathbf{h}^{f}_{i,l}-\mathbf{h}^{f}_{i,j}||^{2},&\text{if
}\exists l\;\text{s.t.}\;d_{i,l}>d_{i,j}\\\
\max\limits_{l}||\mathbf{h}^{f}_{i,l}-\mathbf{h}^{f}_{i,j}||,&\text{otherwise}\end{cases}.$
(15)

Our intuition is that $d_{i,j}$ denotes the local density of a video frame,
and $\gamma_{i,j}$ denotes the distance from other frames of high density. We
proceed to extract the top $Q$ video frames of the highest
$d_{i,j}\times\gamma_{i,j}$ values as key video frames.

Textual description generation. Since large vision-language models have
established impressive results on natural instruction tuning and visual
reasoning capabilities, we leverage a LVLM to augment textual descriptions for
video data [11, 31, 37]. Particularly, given the top $Q$ video frames, we
concatenate them into a single image $I$ of the $W\times H$ grid. We then
forward the image $I$ along with the textual prompt “Write a short caption
sentence for the video in order from left to right, top to bottom” to a large
vision-language model to generate the sentence-level textual description t,
which we combine with the video v to form a video-text pair
$(\textbf{v},\textbf{t})$. We illustrate the generated textual descriptions of
our strategy in Appendix 0.F.

##  4 Experiments

###  4.1 Experimental Settings

Downstream tasks. Following previous works [23, 62, 88, 43], we evaluate our
framework on two popular video question answering and
text$\leftrightarrow$video retrieval tasks. We provide the details of the
datasets in Appendix 0.C:

  * •

Video question answering (VideoQA): We experiment with two videoQA settings,
i.e. open-ended and multi-choice videoQA. Open-ended videoQA classifies a pair
of video and question into one of the pre-defined set of answer labels. Multi-
choice videoQA chooses the correct answer from five choices given the video
and the language question. We assess our video-language representations by the
VideoQA task using the following five datasets, MSRVTT [82], MSVD [7], TGIF-
QA-R [53], NExT-QA [78], and Causal-VidQA [30].

  * •

Text$\leftrightarrow$video retrieval: The retrieval task is to extract the
corresponding video given the textual query, or extract the textual
description given the video. We evaluate the retrieval ability of our video-
language representations using three datasets: MSRVTT [82], DiDeMo [2],
ActivityNet [24].

Evaluation metrics. For the videoQA task, we use the answer accuracy as the
evaluation metric. For the retrieval task, we employ Recall at rank $K$
($R@K$), with $K\in\\{1,5,10\\}$ to evaluate the performance.

Video-language backbones. To extensively validate the effectiveness of our
framework, we conduct experiments on various models of bidirectional and dual
architectures. Particularly, for the dual architecture, we experiment with
CLIP-ViP [83] and video graph transformer (VGT) [79], while for the
bidirectional architecture, we experiment with the well-known VIOLET
architecture [17].

Baseline methods. We compare our method with a comprehensive list of video-
language understanding models, along with the approaches that use LVLM to
augment video-text data: (i) CLIP4Clip [40], a model that transfers image-text
CLIP model [54] to text$\leftrightarrow$video retrieval tasks; (ii) MERLOT
[86], a method that trains on both spatial and temporal objectives to learn
video-language representations; (iii) LAVENDER [33], a model that learns
video-language representations using a unified framework of masked language
modeling; (iv) Singularity [26], curates a pre-training dataset and uses an
early fusion strategy to improve single-frame video-language representation
learning; (v) OmniVL [67]: a model that is trained on both image-language and
video-language data to enhance video-language representation learning; (vi)
VindLU [8], follows a recipe to select pre-training objectives, pre-training
data, and model scale for effective video-language representation learning;
(vii) All-in-one [66], a model that embeds raw video and textual signals into
hidden representations without using pre-trained unimodal encoders; (viii) (
DRL [68], a disentangled representation method that decouples sequential from
hierarchical representations to specifically improve both of them for cross-
modal retrieval; (ix) CLIP2Video [69], consists of two normalization methods
to improve the effectiveness and robustness of cross-modal retrieval models;
(x) LaViLa [88], an approach that fine-tunes LLM to be conditioned on visual
frames to generate additional textual descriptions, which are used to improve
video-language representation learning; (xi) Vid2Seq [84], a pre-trained dense
video captioning model which is used to generate additional textual
descriptions for video data, which are employed to enhance video-language
representation learning.

Implementation details. We utilize LLaVA model as the LVLM [37] to augment
video inputs of the HowTo100M dataset [44], which is a large-scale dataset for
video-language representation learning but has been known for its weakly
aligned video-text pairings [19]. To construct the input for the LVLM, we use
$K=6$ and $Q=12$, then concatenate key frames into an image as a grid of
$3\times 4$ frames. Based upon validation, we adopt $\tau=1$,
$a_{0}=0.2,a_{1}=10$, and $a_{2}=-0.1$ for angular margin contrastive
learning, and $\eta=1$ for our optimization objective. For videoQA, to fairly
compare with previous methods, we fine-tune our VIOLET-based model using AdamW
with an initial learning rate of $\alpha=\beta=2e-5$, betas of (0.9, 0.98),
weight decay of 1e-3, and batch size $B$ of 4 for 10 epochs. Similarly, for
the VGT-based model, we use Adam optimizer with an initial learning rate of
1e-5 of a cosine annealing schedule, and batch size $B$ of 4 for 30 epochs.
For the text$\leftrightarrow$video retrieval task, we fine-tune the model with
an initial learning rate of 5e-6 and a fixed weight decay of 5e-2, using a
batch size $B$ of 4 for 5, 20, and 20 epochs on MSRVTT, DiDeMo, and
ActivityNet datasets, respectively. More details can be found in Appendix 0.D.

###  4.2 Main Results

Table 1: VideoQA results on MSRVTT [82], MSVD [7], and TGIF-QA-R [53]
datasets. Open-ended videoQA is evaluated on MSRVTT, MSVD, and TGIF-Frame
datasets. Multi-choice videoQA is evaluated upon TGIF-Action and TGIF-
Transition.

Method | MSRVTT-QA | MSVD-QA | TGIF-QA-R
---|---|---|---
Action | Frame | Transition
MERLOT | 43.1 | 51.9 | 61.4 | 69.3 | 84.0
LAVENDER | 44.2 | 52.4 | 57.1 | 66.9 | 84.0
Singularity | 43.5 | 49.6 | 53.1 | 65.1 | 81.5
OmniVL | 44.1 | 51.0 | 62.0 | 69.5 | 85.5
VindLU | 44.6 | 51.0 | 59.5 | 65.8 | 87.2
VGT | 40.0 | 36.4 | 61.0 | 61.7 | 73.2
VIOLET | 44.5 | 52.5 | 62.6 | 70.0 | 86.3
All-in-one | 44.3 | 47.9 | 34.9 | 65.0 | 53.8
LaViLa (VIOLET) | 44.9 | 53.7 | 64.3 | 70.7 | 87.0
Vid2Seq (VIOLET) | 44.8 | 53.1 | 64.2 | 70.1 | 86.6
Our method (VGT) | 41.6 | 37.1 | 61.7 | 62.7 | 74.0
Our method (VIOLET) | 46.4 | 55.8 | 66.5 | 71.7 | 89.5

Table 2: VideoQA results on NExT-QA [78] and Causal-VidQA [30] datasets. All
of the datasets target multi-choice videoQA.

Method | NExT-QA | Causal-VidQA
---|---|---
Descriptive | Temporal | Causal | All | Descriptive | Explanatory | Predictive | Counterfactual
MERLOT | 66.6 | 62.5 | 58.2 | 59.4 | 67.2 | 65.7 | 57.2 | 57.0
LAVENDER | 63.4 | 56.5 | 54.6 | 56.7 | 62.0 | 61.6 | 46.3 | 50.4
Singularity | 60.0 | 61.1 | 50.9 | 54.8 | 54.3 | 49.6 | 41.3 | 46.9
OmniVL | 67.1 | 63.8 | 55.9 | 59.9 | 67.0 | 66.0 | 56.5 | 57.1
VindLU | 68.4 | 59.7 | 55.7 | 59.8 | 63.6 | 54.8 | 57.7 | 54.4
VGT | 69.6 | 65.4 | 56.2 | 61.8 | 74.4 | 75.6 | 60.7 | 65.6
VIOLET | 67.7 | 58.0 | 50.7 | 58.5 | 67.6 | 66.6 | 57.1 | 57.6
All-in-one | 64.8 | 63.9 | 50.9 | 57.9 | 60.4 | 51.3 | 51.2 | 50.7
LaViLa (VGT) | 70.9 | 69.2 | 59.3 | 64.8 | 74.7 | 76.1 | 61.6 | 65.7
Vid2Seq (VGT) | 70.3 | 67.0 | 58.5 | 64.2 | 73.7 | 76.0 | 61.3 | 65.6
Our method (VGT) | 72.7 | 70.6 | 62.2 | 66.3 | 75.3 | 77.1 | 62.2 | 68.2
Our method (VIOLET) | 71.2 | 68.6 | 61.1 | 65.8 | 72.7 | 68.0 | 60.5 | 59.5

Table 3: Text$\rightarrow$video retrieval results on MSRVTT [82], DiDeMo [2],
and ActivityNet [24] datasets.

Method | MSRVTT | DiDeMo | ActivityNet
---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
VIOLET | 37.2 | 64.8 | 75.8 | 47.9 | 76.5 | 84.1 | 18.1 | 43.1 | 56.7
VindLU | 48.8 | 72.4 | 82.2 | 59.8 | 86.6 | 91.5 | 55.9 | 82.3 | 90.9
CLIP4Clip | 44.5 | 71.4 | 81.6 | 42.8 | 68.5 | 79.2 | 40.5 | 72.4 | 83.4
CLIP2Video | 47.2 | 73.0 | 83.0 | - | - | - | - | - | -
DRL | 47.4 | 74.6 | 83.8 | 47.9 | 73.8 | 82.7 | 44.2 | 74.5 | 86.1
CLIP-ViP | 55.7 | 77.7 | 86.6 | 55.7 | 78.1 | 86.1 | 59.1 | 83.9 | 91.3
LaViLa (CLIP-ViP) | 56.0 | 79.8 | 87.2 | 56.6 | 79.8 | 87.1 | 58.7 | 82.8 | 90.5
Vid2Seq (CLIP-ViP) | 55.3 | 77.9 | 86.1 | 57.6 | 79.9 | 88.4 | 55.1 | 79.0 | 87.4
Our method (CLIP-ViP) | 60.0 | 82.2 | 89.2 | 62.7 | 89.9 | 96.0 | 60.0 | 85.0 | 91.7

Table 4: Video$\rightarrow$text retrieval results on MSRVTT [82], DiDeMo [2],
and ActivityNet [24] datasets.

Method | MSRVTT | DiDeMo | ActivityNet
---|---|---|---
R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
VIOLET | 36.6 | 64.1 | 75.1 | 44.8 | 72.9 | 82.4 | 15.8 | 39.8 | 54.8
VindLU | 46.0 | 71.8 | 80.2 | 57.2 | 83.1 | 88.9 | 47.1 | 77.5 | 86.6
CLIP4Clip | 43.1 | 70.5 | 81.2 | 17.5 | 37.5 | 49.4 | 42.6 | 73.4 | 85.6
CLIP2Video | 43.5 | 72.3 | 82.1 | - | - | - | - | - | -
DRL | 45.1 | 72.9 | 83.5 | 45.4 | 72.6 | 82.1 | 42.2 | 74.0 | 86.2
CLIP-ViP | 48.0 | 72.4 | 82.9 | 46.3 | 73.2 | 81.9 | 50.2 | 78.3 | 87.5
LaViLa (CLIP-ViP) | 49.0 | 74.5 | 83.4 | 47.1 | 74.2 | 83.0 | 50.3 | 78.7 | 88.4
Vid2Seq (CLIP-ViP) | 49.1 | 74.1 | 82.5 | 47.2 | 74.4 | 83.1 | 50.9 | 79.5 | 89.6
Our method (CLIP-ViP) | 50.1 | 76.8 | 84.8 | 48.1 | 78.1 | 85.5 | 52.3 | 80.8 | 90.1

We denote the results for videoQA in Table 1 and 2, for text$\rightarrow$video
retrieval in Table 3, and for video$\rightarrow$text retrieval in Table 4.

VideoQA. In terms of open-ended videoQA, we substantially outperform the LVLM
approaches, e.g. improve upon LaViLa with improvements of 1.5% on MSRVTT, 2.1%
on MSVD, and 3.7% on TGIF-Frame. We also surpass previous video-language
understanding models, e.g. VindLU by 1.8% on MSRVTT, VIOLET by 3.3% on MSVD,
and OmniVL by 2.2% on TGIF-Frame. Moreover, for multi-choice videoQA, we
consistently surpass both LVLM and video-language understanding models, e.g.
achieving an overall gain of 1.5% on NExT-QA and 1.2% on Causal-VidQA over
LaViLa, while the gains over VGT are 4.5% and 1.6%, respectively on these two
datasets.

Text$\leftrightarrow$video retrieval. Our observation in videoQA applies for
the text$\leftrightarrow$video retrieval task. In the text$\rightarrow$video
direction, we enhance upon CLIP-ViP by 2.6 R@10 points on MSRVTT, 5.5 R@1
points on DiDeMo, and 1.1 R@5 points on ActivityNet. We also significantly
outperform the LVLM approaches, e.g. by 4.0 R@1 points and by 2.4 R@5 points
on MSRVTT over LaViLa. Our superiority on video$\rightarrow$text retrieval is
analogous to the text$\rightarrow$video case.

These results substantiate that our framework is applicable to various video-
language understanding tasks and model architectures. We hypothesize that we
can better control the alignment between the positive video and language, and
their effect upon the model training, thus leading to more reasonable video-
language representations.

###  4.3 Ablation Study

We ablate our meta-optimized framework to investigate which factor results in
the demonstrated effectiveness and explore intriguing properties. We conduct
all experiments of our ablation study on the MSRVTT and MSVD for videoQA, and
MSRVTT and DiDeMo for the text$\leftrightarrow$video retrieval task.

Varying the angular margin. The angular margin, represented as $\mu$, is
critical to control the relationship between semantically close video and
language inputs. To better understand $\mu$, we experiment with manually
varying the margin $\mu$ from 0.1 to 0.4 in Figure 4 (left). We observe that
the optimal performance is achieved when $\mu\sim 0.2$. Based upon this
observation, we adopt $a_{0}=2$ and $a_{1}=10$. Subsequently, we evaluate the
impact of $a_{2}\in\\{0.01,0.05,0.1,0.15,0.2\\}$ in Figure 4 (right). We
discover that $a_{2}=0.1$ achieves the best performance and we adopt it for
all experiments.

Reweighting strategies. In addition to MLP, there exist various approaches to
weigh training losses to control noisy pairings and data imbalance issues
within the training data, e.g. focal loss [34], self-paced learning (SPL)
[22], and L2RW [55]. We replace our MLP by these approaches and demonstrate
the results in Table 6. We find that our MLP-parameterized function
significantly improves upon previous weighting functions that use manually
designed formulation, i.e. focal loss and SPL, and also the meta-learning
approach without using MLP, i.e. L2RW. The possible reason could be that MLP
is a universal approximator, so that it can adaptively learn a reasonable
weighting function based on data, thus outperforming methods that do not
employ MLP.

Table 5: Experiments on the LVLM choice for our prompting strategy.

LVLM choice | VideoQA | Text$\rightarrow$video retrieval
---|---|---
MSRVTT | MSVD | MSRVTT | DiDeMo
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
No augmentation | 45.6 | 54.7 | 59.0 | 80.3 | 87.6 | 58.0 | 82.9 | 89.6
BLIP-2 | 45.7 | 54.9 | 59.1 | 80.4 | 87.7 | 58.1 | 82.9 | 89.7
InstructBLIP | 46.1 | 55.2 | 59.2 | 80.9 | 88.1 | 58.5 | 83.5 | 89.8
LLaVA-ChatGPT | 46.1 | 55.0 | 59.0 | 80.3 | 87.7 | 58.7 | 83.9 | 90.0
LLaVA | 46.4 | 55.8 | 60.0 | 82.2 | 89.2 | 62.7 | 89.9 | 96.0

Table 6: Experiments on weighting functions.

Weighting function | VideoQA | Text$\rightarrow$video retrieval
---|---|---
MSRVTT | MSVD | MSRVTT | DiDeMo
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Focal loss | 45.0 | 54.1 | 58.3 | 80.0 | 86.9 | 59.7 | 85.6 | 91.8
SPL | 45.3 | 54.3 | 58.7 | 80.8 | 88.1 | 61.4 | 87.5 | 93.8
L2RW | 45.9 | 55.2 | 59.5 | 81.8 | 89.6 | 62.0 | 88.8 | 94.6
MLP | 46.4 | 55.8 | 60.0 | 82.2 | 89.2 | 62.7 | 89.9 | 96.0

Optimization strategies. We explore different strategies to optimize the MLP
and video-language model. Particularly, we experiment with the joint learning
strategy, which simultaneously updates the parameters of the MLP and the
video-language model to minimize the objective in Eq. 3. As shown in Table 8,
our meta-learning approach outperforms the joint learning one and achieves the
best performance. This empirically validates the effectiveness of our meta-
learning strategy which enables the training to follow gradient of the meta-
data.

Table 7: Experiments on the number of key frames $Q$ and the concatenated
grid.

$Q$ | Grid | VideoQA | Text$\rightarrow$video retrieval
---|---|---|---
MSRVTT | MSVD | MSRVTT | DiDeMo
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
1 | $1\times 1$ | 45.0 | 55.1 | 58.2 | 79.3 | 86.2 | 60.2 | 85.8 | 93.2
2 | $1\times 2$ | 45.7 | 55.3 | 58.8 | 80.0 | 87.0 | 60.5 | 85.8 | 93.3
$2\times 1$ | 45.9 | 55.4 | 58.4 | 79.2 | 87.0 | 60.9 | 86.1 | 94.2
4 | $1\times 4$ | 45.9 | 55.2 | 59.1 | 80.3 | 88.4 | 61.3 | 86.3 | 94.6
$2\times 2$ | 45.6 | 55.2 | 59.2 | 80.4 | 88.4 | 61.3 | 86.7 | 94.6
8 | $2\times 4$ | 45.9 | 55.6 | 59.8 | 81.3 | 88.7 | 61.8 | 89.0 | 95.3
$4\times 2$ | 45.8 | 55.5 | 59.3 | 80.7 | 88.5 | 61.8 | 88.9 | 95.3
12 | $3\times 4$ | 46.4 | 55.8 | 60.0 | 82.2 | 89.2 | 62.7 | 89.9 | 96.0
$2\times 6$ | 45.8 | 55.6 | 59.3 | 81.7 | 89.1 | 62.7 | 89.2 | 95.6
16 | $2\times 8$ | 45.8 | 55.1 | 58.3 | 79.7 | 87.6 | 61.9 | 88.7 | 94.8
$4\times 4$ | 45.7 | 55.1 | 58.9 | 80.8 | 88.7 | 62.6 | 89.1 | 95.9

Figure 4: (Left) Validation videoQA accuracy on MSVD with respect to $\mu$;
(Middle) Validation videoQA accuracy on MSVD with respect to $a_{0}$; (Right)
Relationship between loss values and weight values generated by our MLP-
parameterized weighting function.

Video-text data augmentation. We assess the influence of different video-text
data augmentation strategies. In particular, we ablate the LLaVA-generated
samples and also explore different LVLM choices, i.e. BLIP-2 [31] and
InstructBLIP [11]. In addition, we also use LLaVA model to generate image
caption for each video frame, and ask ChatGPT to write a summary for the
concatenated captions of all frames as the textual description. Table 5 shows
that neglecting LVLM-generated data results in slight performance degradation,
which can be resolved by using other LVLMs, but still lack behind LLaVA model.
We conjecture that LLaVA can better follow the language instruction to produce
more accurate textual descriptions for videos [37]. Moreover, LLaVA-ChatGPT
approach also leads to performance degradation, possibly because it is still
challenging for ChatGPT to infer temporal relations among captions of separate
video frames.

Table 8: Experiments on the learning strategy.

Training strategy | VideoQA | Text$\rightarrow$video retrieval
---|---|---
MSRVTT | MSVD | MSRVTT | DiDeMo
R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Joint learning | 45.5 | 54.9 | 59.3 | 81.1 | 88.4 | 59.2 | 84.9 | 90.2
Meta-learning | 46.4 | 55.8 | 60.0 | 82.2 | 89.2 | 62.7 | 89.9 | 96.0

Figure 5: a) Relative R@1 improvement in the text-video retrieval task on
MSRVTT for each topic; b) Relative accuracy improvement in the videoQA task on
MSRVTT with respect to the proportion of questions for which each label is the
answer.

Table 9: Case study of similarity scores for video-text pairs.

Video | Caption | CLIP-ViP score | Our score
---|---|---|---
|  a group of people dancing on a stage in front of a crowd. | 0.1727 | 0.1698
the man in black reaches his dance partner. man walks to the woman in red the camera zooms out on the dancers. we see a dancer in the back left spinning | 0.2175 | 0.3342
|  a boy in a red shirt and a blue hat is playing. | 0.2594 | 0.2441
a girl throwing a rock. girl in the red shirt looks back over his shoulder the boy in a blue hat is seen for the first time. the boy in the baseball cap first looks at the camera boy wearing a blue ball cap | 0.2869 | 0.3164

Number of extracted key frames. We explore the impact of varying number of
extracted key frames and varying the approach to concatenate in Table 7. While
the performance is quite invariant to the concatenation strategy, we observe
an increasing trend of the performance when the number of the extracted frames
$Q$ increases, since there exists more information for LVLM to put into the
descriptions. However, after surpassing $Q=12$ frames, the performance
dramatically decreases. The reason could be that when $Q$ is excessively
large, visual frames become excessively small for LVLM to precisely detect
details. Furthermore, balanced grids, i.e. $3\times 4$ and $4\times 4$, tend
to outperform skewed grids, i.e. $2\times 6$ and $2\times 8$, respectively.
The cause might be that video details are more difficult for an LVLM to
capture when the video frames are presented on a long rectangle than when they
are presented on a balanced one.

###  4.4 Analysis

Effect of angular margin contrastive learning. To better understand the effect
of our angular margin contrastive method, we show in Table 9 examples of
videos and language captions, along with the similarity scores which are
generated by the baseline CLIP-ViP model and the model trained with our
proposed subtractive contrastive learning strategy. We observe that when the
caption consists of less details, i.e. it aligns more weakly with the video
content, our model assigns a lower similarity score compared to CLIP-ViP. In
contrast, when the language caption becomes more detailed, we assign a higher
score while CLIP-ViP only slightly elevates the score. These examples
demonstrate that our angular margin contrastive strategy can control the
similarity level for weakly aligned video-text pairs while being able to adapt
to cases of strongly aligned pairs. We provide more examples in Appendix 0.E.

Effect of MLP-parameterized weighting function. To closely study the impact of
our MLP-based weighting function, we show the relative R@1 and the relative
accuracy improvement of the text-video retrieval tasks in Figure 5a and 5b,
respectively. Both figures demonstrate that we accomplish higher level of
performance improvement on samples that belong to minority groups, i.e.
unpopular topic groups and answer labels. Such results intuitively
substantiate the productivity of our MLP-based weighting function and showcase
its capacity to learn the effective strategy to control the training effect of
the training samples.

##  5 Conclusion

In this paper, we propose a meta-optimized contrastive framework to enhance
video-language representation learning. In particular, we propose a
contrastive learning framework with a subtractive margin between positive
video and language to regularize their representations from reaching perfect
similarity. Our framework also utilizes an MLP network that maps training
losses to sample weights to enable the model to dynamically adjust the focus
on data samples during training. Combined with a strategy to utilize large-
vision language model to augment video-text data, our framework achieves
superior results on commonly used video question answering and text-video
retrieval tasks. Our framework is also applicable to a wide array of model
architectures, which can promote its implementation in practical applications.

Acknowledgement. This research/project is supported by the National Research
Foundation, Singapore under its AI Singapore Programme (AISG Award No:
AISG3-PhD-2023-08-051T). Thong Nguyen is supported by a Google Ph.D.
Fellowship in Natural Language Processing.

## References

  * [1] Akbari, H., Yuan, L., Qian, R., Chuang, W.H., Chang, S.F., Cui, Y., Gong, B.: Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems 34, 24206–24221 (2021)
  * [2] Anne Hendricks, L., Wang, O., Shechtman, E., Sivic, J., Darrell, T., Russell, B.: Localizing moments in video with natural language. In: Proceedings of the IEEE international conference on computer vision. pp. 5803–5812 (2017)
  * [3] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1728–1738 (2021)
  * [4] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: A clip-hitchhiker’s guide to long video retrieval. arXiv preprint arXiv:2205.08508 (2022)
  * [5] Bin, Y., Han, M., Shi, W., Wang, L., Yang, Y., Ng, S.K., Shen, H.: Non-autoregressive math word problem solver with unified tree structure. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 3290–3301 (2023)
  * [6] Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., Niebles, J.C.: Revisiting the" video" in video-language understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2917–2927 (2022)
  * [7] Chen, D., Dolan, W.B.: Collecting highly parallel data for paraphrase evaluation. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. pp. 190–200 (2011)
  * [8] Cheng, F., Wang, X., Lei, J., Crandall, D., Bansal, M., Bertasius, G.: Vindlu: A recipe for effective video-and-language pretraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10739–10750 (2023)
  * [9] Coria, J.M., Ghannay, S., Rosset, S., Bredin, H.: A metric learning approach to misogyny categorization. In: Proceedings of the 5th Workshop on Representation Learning for NLP. pp. 89–94 (2020)
  * [10] Csáji, B.C., et al.: Approximation with artificial neural networks. Faculty of Sciences, Etvs Lornd University, Hungary 24(48), 7 (2001)
  * [11] Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.06500
  * [12] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690–4699 (2019)
  * [13] Fabian Caba Heilbron, Victor Escorcia, B.G., Niebles, J.C.: ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
  * [14] Fernando, D.l.T., Mkchael, J.B.: A framework for robust subspace learning. International Journal of Computer Vision 54(1), 117–142 (2003)
  * [15] Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences 55(1), 119–139 (1997)
  * [16] Fu, T.J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681 (2021)
  * [17] Fu, T.J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: An empirical study of end-to-end video-language transformers with masked visual modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22898–22909 (2023)
  * [18] Gao, Z., Liu, J., Sun, W., Chen, S., Chang, D., Zhao, L.: Clip2tv: Align, match and distill for video-text retrieval. arXiv preprint arXiv:2111.05610 (2021)
  * [19] Han, T., Xie, W., Zisserman, A.: Temporal alignment networks for long-term video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2906–2916 (2022)
  * [20] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [21] Jang, Y., Song, Y., Yu, Y., Kim, Y., Kim, G.: Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2758–2766 (2017)
  * [22] Jiang, L., Meng, D., Yu, S.I., Lan, Z., Shan, S., Hauptmann, A.: Self-paced learning with diversity. Advances in neural information processing systems 27 (2014)
  * [23] Jin, P., Huang, J., Xiong, P., Tian, S., Liu, C., Ji, X., Yuan, L., Chen, J.: Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2472–2482 (2023)
  * [24] Krishna, R., Hata, K., Ren, F., Fei-Fei, L., Carlos Niebles, J.: Dense-captioning events in videos. In: Proceedings of the IEEE international conference on computer vision. pp. 706–715 (2017)
  * [25] Lei, C., Luo, S., Liu, Y., He, W., Wang, J., Wang, G., Tang, H., Miao, C., Li, H.: Understanding chinese video and language via contrastive multimodal pre-training. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 2567–2576 (2021)
  * [26] Lei, J., Berg, T.L., Bansal, M.: Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428 (2022)
  * [27] Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., Liu, J.: Less is more: Clipbert for video-and-language learning via sparse sampling. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7331–7341 (2021)
  * [28] Li, B., Han, Z., Li, H., Fu, H., Zhang, C.: Trustworthy long-tailed classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6970–6979 (2022)
  * [29] Li, H., Bin, Y., Liao, J., Yang, Y., Shen, H.T.: Your negative may not be true negative: Boosting image-text matching with false negative elimination. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 924–934 (2023)
  * [30] Li, J., Niu, L., Zhang, L.: From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21273–21282 (2022)
  * [31] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [32] Li, L., Chen, Y.C., Cheng, Y., Gan, Z., Yu, L., Liu, J.: Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200 (2020)
  * [33] Li, L., Gan, Z., Lin, K., Lin, C.C., Liu, Z., Liu, C., Wang, L.: Lavender: Unifying video-language understanding as masked language modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23119–23129 (2023)
  * [34] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
  * [35] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. IEEE transactions on pattern analysis and machine intelligence (2018)
  * [36] Lin, Y.B., Lei, J., Bansal, M., Bertasius, G.: Eclipse: Efficient long-range video retrieval using sight and sound. In: European Conference on Computer Vision. pp. 413–430. Springer (2022)
  * [37] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023)
  * [38] Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. arXiv preprint arXiv:2106.13230 (2021)
  * [39] Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, J., Bharti, T., Zhou, M.: Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353 (2020)
  * [40] Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., Li, T.: Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing 508, 293–304 (2022)
  * [41] Ma, X., Santos, C.N.d., Arnold, A.O.: Contrastive fine-tuning improves robustness for neural rankers. arXiv preprint arXiv:2105.12932 (2021)
  * [42] Malisiewicz, T., Gupta, A., Efros, A.A.: Ensemble of exemplar-svms for object detection and beyond. In: ICCV (2011)
  * [43] Man, X., Shao, J., Chen, F., Zhang, M., Shen, H.T.: Tevl: Trilinear encoder for video-language representation learning. ACM Transactions on Multimedia Computing, Communications and Applications 19(5s), 1–20 (2023)
  * [44] Miech, A., Zhukov, D., Alayrac, J.B., Tapaswi, M., Laptev, I., Sivic, J.: Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2630–2640 (2019)
  * [45] Nguyen, C.D., Nguyen, T., Vu, D., Luu, A.: Improving multimodal sentiment analysis: Supervised angular margin-based contrastive learning for enhanced fusion representation. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 14714–14724 (2023)
  * [46] Nguyen, C.D., Nguyen, T., Wu, X., Luu, A.T.: Kdmcse: Knowledge distillation multimodal sentence embeddings with adaptive angular margin contrastive learning. arXiv preprint arXiv:2403.17486 (2024)
  * [47] Nguyen, T., Bin, Y., Xiao, J., Qu, L., Li, Y., Wu, J.Z., Nguyen, C.D., Ng, S.K., Tuan, L.A.: Video-language understanding: A survey from model architecture, model training, and data perspectives. arXiv preprint arXiv:2406.05615 (2024)
  * [48] Nguyen, T., Luu, A.T.: Contrastive learning for neural topic model. Advances in neural information processing systems 34, 11974–11986 (2021)
  * [49] Nguyen, T., Wu, X., Dong, X., Le, K.M., Hu, Z., Nguyen, C.D., Ng, S.K., Luu, A.T.: Read-pvla: Recurrent adapter with partial video-language alignment for parameter-efficient transfer learning in low-resource video-language modeling. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 18824–18832 (2024)
  * [50] Nguyen, T., Wu, X., Dong, X., Nguyen, C.D., Ng, S.K., Luu, A.: Demaformer: Damped exponential moving average transformer with energy-based modeling for temporal language grounding. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 3635–3649 (2023)
  * [51] Nguyen, T., Wu, X., Dong, X., Nguyen, C.D.T., Ng, S.K., Luu, A.T.: Topic modeling as multi-objective contrastive optimization. arXiv preprint arXiv:2402.07577 (2024)
  * [52] Nguyen, T., Wu, X., Luu, A.T., Nguyen, C.D., Hai, Z., Bing, L.: Adaptive contrastive learning on multimodal transformer for review helpfulness predictions. arXiv preprint arXiv:2211.03524 (2022)
  * [53] Peng, L., Yang, S., Bin, Y., Wang, G.: Progressive graph attention network for video question answering. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 2871–2879 (2021)
  * [54] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763 (2021)
  * [55] Ren, M., Zeng, W., Yang, B., Urtasun, R.: Learning to reweight examples for robust deep learning. In: International conference on machine learning. pp. 4334–4343. PMLR (2018)
  * [56] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015)
  * [57] Seo, P.H., Nagrani, A., Arnab, A., Schmid, C.: End-to-end generative pretraining for multimodal video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17959–17968 (2022)
  * [58] Shang, X., Di, D., Xiao, J., Cao, Y., Yang, X., Chua, T.S.: Annotating objects and relations in user-generated videos. In: Proceedings of the 2019 on International Conference on Multimedia Retrieval. pp. 279–287 (2019)
  * [59] Shu, J., Xie, Q., Yi, L., Zhao, Q., Zhou, S., Xu, Z., Meng, D.: Meta-weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems 32 (2019)
  * [60] Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems 29 (2016)
  * [61] Sun, C., Baradel, F., Murphy, K., Schmid, C.: Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.05743 (2019)
  * [62] Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model for video and language representation learning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 7464–7473 (2019)
  * [63] Sun, Y., Kamel, M.S., Wong, A.K., Wang, Y.: Cost-sensitive boosting for classification of imbalanced data. Pattern recognition 40(12), 3358–3378 (2007)
  * [64] Tang, Z., Lei, J., Bansal, M.: Decembert: Learning from noisy instructional videos via dense captions and entropy minimization. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2415–2426 (2021)
  * [65] Wang, A., Luu, A.T., Foo, C.S., Zhu, H., Tay, Y., Chandrasekhar, V.: Holistic multi-modal memory network for movie question answering. IEEE Transactions on Image Processing 29, 489–499 (2019)
  * [66] Wang, J., Ge, Y., Yan, R., Ge, Y., Lin, K.Q., Tsutsui, S., Lin, X., Cai, G., Wu, J., Shan, Y., et al.: All in one: Exploring unified video-language pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6598–6608 (2023)
  * [67] Wang, J., Chen, D., Wu, Z., Luo, C., Zhou, L., Zhao, Y., Xie, Y., Liu, C., Jiang, Y.G., Yuan, L.: Omnivl: One foundation model for image-language and video-language tasks. Advances in neural information processing systems 35, 5696–5710 (2022)
  * [68] Wang, Q., Zhang, Y., Zheng, Y., Pan, P., Hua, X.S.: Disentangled representation learning for text-video retrieval. arXiv preprint arXiv:2203.07111 (2022)
  * [69] Wang, Q., Zhang, Y., Zheng, Y., Pan, P., Hua, X.S.: Disentangled representation learning for text-video retrieval. arXiv preprint arXiv:2203.07111 (2022)
  * [70] Wang, Y., Kucukelbir, A., Blei, D.M.: Robust probabilistic modeling with bayesian data reweighting. In: International Conference on Machine Learning. pp. 3646–3655. PMLR (2017)
  * [71] Wei, J., Hu, G., Tuan, L.A., Yang, X., Zhu, W.: Multi-scale receptive field graph model for emotion recognition in conversations. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2023)
  * [72] Wei, J., Hu, G., Yang, X., Luu, A.T., Dong, Y.: Audio-visual domain adaptation feature fusion for speech emotion recognition. In: INTERSPEECH. pp. 1988–1992 (2022)
  * [73] Wei, J., Hu, G., Yang, X., Luu, A.T., Dong, Y.: Learning facial expression and body gesture visual information for video emotion recognition. Expert Systems with Applications 237, 121419 (2024)
  * [74] Wu, X., Dong, X., Nguyen, T., Liu, C., Pan, L.M., Luu, A.T.: Infoctm: A mutual information maximization perspective of cross-lingual topic modeling. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 13763–13771 (2023)
  * [75] Wu, X., Dong, X., Pan, L., Nguyen, T., Luu, A.T.: Modeling dynamic topics in chain-free fashion by evolution-tracking contrastive learning and unassociated word exclusion. In: Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics (2024)
  * [76] Wu, X., Li, C., Zhu, Y., Miao, Y.: Short text topic modeling with topic distribution quantization and negative sampling decoder. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1772–1782. Online (Nov 2020), https://aclanthology.org/2020.emnlp-main.138.pdf
  * [77] Wu, X., Luu, A.T., Dong, X.: Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 2748–2760. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Dec 2022), https://aclanthology.org/2022.emnlp-main.176
  * [78] Xiao, J., Shang, X., Yao, A., Chua, T.S.: Next-qa: Next phase of question-answering to explaining temporal actions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9777–9786 (2021)
  * [79] Xiao, J., Zhou, P., Yao, A., Li, Y., Hong, R., Yan, S., Chua, T.S.: Contrastive video question answering via video graph transformer. arXiv preprint arXiv:2302.13668 (2023)
  * [80] Xu, H., Ghosh, G., Huang, P.Y., Arora, P., Aminzadeh, M., Feichtenhofer, C., Metze, F., Zettlemoyer, L.: Vlm: Task-agnostic video-language model pre-training for video understanding. arXiv preprint arXiv:2105.09996 (2021)
  * [81] Xu, H., Ghosh, G., Huang, P.Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., Feichtenhofer, C.: Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084 (2021)
  * [82] Xu, J., Mei, T., Yao, T., Rui, Y.: Msr-vtt: A large video description dataset for bridging video and language. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5288–5296 (2016)
  * [83] Xue, H., Sun, Y., Liu, B., Fu, J., Song, R., Li, H., Luo, J.: Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint arXiv:2209.06430 (2022)
  * [84] Yang, A., Nagrani, A., Seo, P.H., Miech, A., Pont-Tuset, J., Laptev, I., Sivic, J., Schmid, C.: Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10714–10726 (2023)
  * [85] Yu, T., Dai, W., Liu, Z., Fung, P.: Vision guided generative pre-trained language models for multimodal abstractive summarization. arXiv preprint arXiv:2109.02401 (2021)
  * [86] Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J.S., Cao, J., Farhadi, A., Choi, Y.: Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems 34, 23634–23651 (2021)
  * [87] Zhang, Z., Sabuncu, M.R.: Generalized cross entropy loss for training deep neural networks with noisy labels. In: NeurIPS (2018)
  * [88] Zhao, Y., Misra, I., Krähenbühl, P., Girdhar, R.: Learning video representations from large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6586–6597 (2023)
  * [89] Zhu, L., Yang, Y.: Actbert: Learning global-local video-text representations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8746–8755 (2020)

## Appendix

##  Appendix 0.A Proof of Theorem 1

###### Theorem 1.

Let $\lambda_{i,j}$ denote the angle between the representation of two samples
$i,j$, $\mathcal{L}^{v,t}_{\textup{angular},i}$ and
$\mathcal{L}^{v,t}_{\textup{contrastive},i}$ denote the training objectives
with and without the angular margin, respectively. Then, if
$\lambda_{i,i}\leq\frac{\pi}{2}$, the following inequality holds:

$\left|\frac{\partial\mathcal{L}_{\textup{angular},i}^{v,t}}{\partial\lambda_{i,i}}\right|\leq\left|\frac{\partial\mathcal{L}_{\textup{contrastive},i}^{v,t}}{\partial\lambda_{i,i}}\right|$
(16)

###### Proof

From the formulation of $\mathcal{L}_{\textup{angular},i}^{v,t}$, we have the
following equation:

$\mathcal{L}_{\textup{angular},i}^{v,t}=-\frac{\cos\left([\lambda_{i,i}-\mu]_{+}\right)}{\tau}+\log\left(e^{\frac{\cos\left([\lambda_{i,i}-\mu)]_{+}\right)}{\tau}}+\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}\right).$ (17)

Differentiating $\mathcal{L}_{\textup{angular},i}^{v,t}$ with respect to
$\lambda_{i,i}$, we obtain:

$\displaystyle\frac{\partial\mathcal{L}_{\textup{angular},i}^{v,t}}{\partial\lambda_{i,i}}$
$\displaystyle=\frac{\sin\left([\lambda_{i,i}-\mu]_{+}\right)}{\tau}-\frac{\sin\left([\lambda_{i,i}-\mu]_{+}\right)\cdot
e^{\frac{\cos(\lambda_{i,j})}{\tau}}}{e^{\frac{\cos\left([\lambda_{i,i}-\mu)]_{+}\right)}{\tau}}+\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}}$ (18)
$\displaystyle=\frac{\frac{\sin\left([\lambda_{i,i}-\mu]_{+}\right)}{\tau}\cdot\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}}{e^{\frac{\cos\left([\lambda_{i,i}-\mu)]_{+}\right)}{\tau}}+\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}}$ (19)

Analogously, we achieve the derivative of
$\mathcal{L}_{\textup{contrastive},i}^{v,t}$ with respect to $\lambda_{i,i}$:

$\frac{\partial\mathcal{L}_{\textup{contrastive},i}^{v,t}}{\partial\lambda_{i,i}}=\frac{\frac{\sin\left([\lambda_{i,i}]_{+}\right)}{\tau}\cdot\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}}{e^{\frac{\cos\left([\lambda_{i,i})]_{+}\right)}{\tau}}+\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}}$ (20)

Dividing the norm of
$\frac{\partial\mathcal{L}_{\textup{angular},i}^{v,t}}{\partial\lambda_{i,i}}$
by the norm of
$\frac{\partial\mathcal{L}_{\textup{contrastive},i}^{v,t}}{\lambda_{i,i}}$
yields that:

$\begin{split}\left|\frac{\partial\mathcal{L}_{\textup{angular},i}^{v,t}}{\partial\lambda_{i,i}}\right|/\left|\frac{\partial\mathcal{L}_{\textup{contrastive},i}^{v,t}}{\partial\lambda_{i,i}}\right|=\frac{\sin\left([\lambda_{i,i}-\mu]_{+}\right)}{\sin\left(\lambda_{i,i}\right)}\cdot\frac{e^{\frac{\cos\left([\lambda_{i,i})]_{+}\right)}{\tau}}+\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}}{e^{\frac{\cos\left([\lambda_{i,i}-\mu)]_{+}\right)}{\tau}}+\sum\limits_{j\neq
i}e^{\frac{\cos(\lambda_{i,j})}{\tau}}}.\end{split}$ (21)

Since, $\lambda_{i,i}\leq\frac{\pi}{2}$, we have
$\sin\left([\lambda_{i,i}-\mu]_{+}\right)\leq\sin(\lambda_{i,i})$ and
$\cos(\lambda_{i,i})\leq\cos\left([\lambda_{i,i}-\mu]_{+}\right)$. Therefore,

$\left|\frac{\partial\mathcal{L}_{\textup{angular},i}^{v,t}}{\partial\lambda_{i,i}}\right|/\left|\frac{\partial\mathcal{L}_{\textup{contrastive},i}^{v,t}}{\partial\lambda_{i,i}}\right|\leq
1.$ (22)

This concludes our proof. ∎

##  Appendix 0.B Derivation of the Update Rule in Theoretical Analysis

###### Proof

Recalling Eq. (10), we have the update rule as follows:

$\theta^{(k+1)}=\theta^{(k)}-\frac{\beta}{M}\sum\limits_{i=1}^{M}\nabla_{\theta}\mathcal{L}_{i}^{\text{meta}}\left(\hat{\Theta}^{(k)}\right).$
(23)

We continue our derivation using chain rule:

$\theta^{(k+1)}=\theta^{(k)}-\frac{\beta}{M}\sum\limits_{i=1}^{M}\frac{\partial\mathcal{L}_{i}^{\text{meta}}\left(\hat{\Theta}^{(k)}\right)}{\partial\hat{\Theta}^{(k)}}\frac{\partial\hat{\Theta}^{(k)}}{\partial\theta}.$
(24)

Taking into account the update rule for the parameters of our video-language
model in Eq. (11), we have:

$\begin{split}&\theta^{(k+1)}=\theta^{(k)}-\frac{\beta}{M}\sum\limits_{i=1}^{M}\frac{\partial\mathcal{L}_{i}^{\text{meta}}\left(\hat{\Theta}^{(k)}\right)}{\partial\hat{\Theta}^{(k)}}\cdot\left(-\frac{\partial\frac{\alpha}{B}\sum\limits_{j=1}^{B}w_{i}\left(\theta^{(k)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)\nabla_{\Theta}\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)}{\partial\theta}\right)\\\
&=\theta^{(k)}+\frac{\alpha\beta}{BM}\sum\limits_{i=1}^{M}\sum\limits_{j=1}^{B}\nabla_{\hat{\Theta}^{(k)}}\mathcal{L}_{i}^{\text{meta}}\left(\hat{\Theta}\right)\Big{|}_{\hat{\Theta}^{(k)}}\nabla_{\Theta}\mathcal{L}_{j}^{\text{train}}\left(\Theta\right)\Big{|}_{\Theta^{(k)}}\nabla_{\theta}w\left(\theta^{(k)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)\Big{|}_{\theta^{(k)}}.\end{split}$
(25)

If we assign:

$G_{ij}=\nabla_{\hat{\Theta}^{(k)}}\mathcal{L}_{i}^{\text{meta}}\left(\hat{\Theta}\right)\Big{|}_{\hat{\Theta}^{(k)}}\nabla_{\Theta}\mathcal{L}_{j}^{\text{train}}\left(\Theta\right)\Big{|}_{\Theta^{(k)}},$
(26)

then we obtain the target result:

$\theta^{(k+1)}=\theta^{(k)}+\frac{\alpha\beta}{BM}\sum_{i=1}^{B}\sum_{j=1}^{M}G_{ij}\frac{\partial
w\left(\theta^{(k)},\mathcal{L}_{j}^{\text{train}}\left(\Theta^{(k)}\right)\right)}{\partial\theta}\Big{|}_{\theta^{(k)}}$
(27)

##  Appendix 0.C Datasets

MSRVTT [82] is composed of 10K YouTube videos, each of which possesses 20
sentence-level descriptions. For text-video retrieval, we follow the standard
protocol to train on 9K videos and evaluate on the 1K-A testing split. For
videoQA, MSRVTT is formatted into 243K open-ended questions. We adopt a split
of 149K/12K/73K train/val/test split to evaluate our framework.

MSVD [7] comprises 47K open-ended questions over 2K videos. We employ a ratio
of 30K/6K/13K to split the questions into training, validation, and testing
sets, respectively.

TGIF-QA-R [21, 53] assesses the spatial-temporal reasoning ability to answer
questions about GIF videos. There are three subsets in the dataset: (i)
Action: to identify the repeated action; (ii) Transition: to recognize the
action before or after some event; (iii) Frames: to answer questions regarding
a particular frame in the video. TGIF-Action and TGIF-Transition support
multi-choice videoQA, and TGIF-Frame supports open-ended videoQA. Peng et al.
(2021) [53] discover that TGIF-QA exhibits an answer bias that enables a model
to determine the answer without relating to the video and question. They
construct a debiased version of TGIF-QA, i.e. TGIF-QA-R, in which they gather
answer choices and randomly re-draw the distractor options for each question.
To prevent the bias from impact the evaluation of our learned video-language
representations, we employ the TGIF-QA-R variant in our experiments.

NExT-QA [78] specifically judges the causal and temporal reasoning. Its videos
are inherited from the VidOR dataset [58], and used to construct three
question categories, i.e. Causal, Descriptive, and Temporal. NExT-QA supports
the multi-choice videoQA setting, where each question consists of five options
with one correct answer and four distractor answers.

Causal-VidQA [30] focuses upon visual commonsense reasoning and designs four
types of questions, i.e. Counterfactual, Descriptive, Explanatory, and
Predictive. Analogous to NExT-QA, it mainly supports multi-choice videoQA
which tasks a model to select the correct answer among five available options.

DiDeMo [2] is composed of 10K Flickr videos. Theses videos are manually
annotated to result in 40K sentences. We follow common practice [26, 27, 33]
to concatenate all annotated sentences of a video and evaluate paragraph-to-
video and video-to-paragraph retrieval tasks.

ActivityNet [13] consists of 20K YouTube videos annotated with 100K captions.
Similar to DiDeMo, we also concatenate captions of a video to conduct
paragraph-to-video and video-to-paragraph retrieval experiments. We train our
model and prior baselines on 10K videos, validate on 5K, and test on the
remaining 5K videos.

##  Appendix 0.D Implementation Details

We test our learned video-language representations on video question answering
(videoQA) and text$\leftrightarrow$video retrieval tasks.

Video question answering (VideoQA). For videoQA, with respect to the VIOLET-
based model, to fairly compare with previous works [16, 17], we sparsely
sample $N_{V}=5$ video frames and utilize frame size of $224\times 224$. We
also adopt a max length of 30 words for every language question. We then use
Video Swin Transformer [38] with VideoSwin-Base pretrained on Kinetics-600 to
encode the frame sequence, and pretrained BERT-Base embeddings to encode the
language question and the answer. For the VGT-based model [79], we randomly
sample $N_{V}=32$ frames given a video input. We utilize Faster-RCNN [56] with
the ResNet-101 backbone [20] pretrained on the Visual Genome dataset to
extract visual frame features. For textual features, we employ the pretrained
BERT-Base embedding layer.

Text$\leftrightarrow$video retrieval. For text$\leftrightarrow$video
retrieval, given a video, we randomly sample $N_{V}=12$ frames and resize them
to $224\times 224$ images. We split a language query into word tokens with a
maximum length of 70. We use CLIP visual encoder and CLIP textual encoder to
encode video and language query, respectively.

##  Appendix 0.E More examples of similarity scores for video-text pairs

Table 10: Case study of similarity scores for video-text pairs.

Video | Caption | CLIP-ViP score | Our score
---|---|---|---
|  a baby playing with a keyboard. | 0.3297 | 0.3277
the cameraman’s hand moves into the screen to play keys and grab the babies hand. camera pans down to baby the cameraman reaches down and plays some keys on the piano. | 0.3580 | 0.3843
|  a person in a striped shirt is holding a spoon. | 0.2123 | 0.1949
a man takes a bite of food. Man eats thing on chopstick. Man pulls item off finger and eats it. | 0.3385 | 0.3504
|  a bird is sitting on the ground in the middle of a row of doors. | 0.2470 | 0.2175
bird picks up a bag bird folds paper over. the bird lifts the napkin with its head. the bird begins to eat the bread | 0.3070 | 0.3242
|  a bird is flying above the water. | 0.1991 | 0.1216
bird makes a few splashes in the water before flying off we see a bird skim the water ’ fly away two birds fly away | 0.2771 | 0.3013
|  a plane flying through the sky. | 0.2331 | 0.2188
jet approaches the crowd then takes off again the plane is flying over the crowd. no crowd in these frames | 0.3503 | 0.3723
|  a man is standing in front of a building with a sign on it. | 0.1849 | 0.1713
guy enters into view camera pans left to show man with a map a person is standing at the gates of a location. a man holding up a pamphlet in front of a building. the person can be seen holding something | 0.3082 | 0.3153

In this appendix, we provide more examples of video-text pairs and their
corresponding similarity scores computed by our model and the CLIP-ViP
baseline in Table 10.

##  Appendix 0.F Additional Qualitative Results

In this appendix, we provide additional qualitative results of our proposed
strategy which utilizes LVLM to augment video-text data.

Figure 6: Examples of video input and our textual description.