# Lightweight Boosting Models for User Response Prediction Using Adversarial
Validation

Hyeonwoo Kim<EMAIL_ADDRESS>Upstage AI ResearchRepublic of Korea and
Wonsung Lee<EMAIL_ADDRESS>Upstage AI ResearchRepublic of Korea

(2018)

###### Abstract.

The ACM RecSys Challenge 2023, organized by ShareChat, aims to predict the
probability of the app being installed. This paper describes the lightweight
solution to this challenge. We formulate the task as a user response
prediction task. For rapid prototyping for the task, we propose a lightweight
solution including the following steps: 1) using adversarial validation, we
effectively eliminate uninformative features from a dataset; 2) to address
noisy continuous features and categorical features with a large number of
unique values, we employ feature engineering techniques.; 3) we leverage
Gradient Boosted Decision Trees (GBDT) for their exceptional performance and
scalability. The experiments show that a single LightGBM model, without
additional ensembling, performs quite well. Our team achieved ninth place in
the challenge with the final leaderboard score of 6.059065. Code for our
approach can be found here: https://github.com/choco9966/recsys-
challenge-2023.

ACM RecSys Challenge 2023, User Response Prediction, Adversarial Validation,
Gradient Boosting Decision Trees, CTR Prediction

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation email; June 03–05, 2018; Woodstock,
NY††booktitle: RecSys ’23: ACM Symposium on Neural Gaze Detection, June 03–05,
2018, Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Information systems Recommender systems

##  1\. Introduction

User response prediction is challenging due to the inherent nature of the
task, such as data sparsity, noisy and incomplete data, concept drift, and
temporal dynamics. Researchers have explored diverse approaches to address
this issue and improve prediction accuracy, including logistic regression (LR)
(McMahan et al., 2013), decision trees (He et al., 2014), factorization
machines (FM) (Juan et al., 2016), and deep learning-based methods (Wang et
al., 2017; Zhou et al., 2018; Gu et al., 2020). Deep learning models for user
response prediction have recently gained popularity due to their ability to
provide an inductive bias suitable for the input data type and capture
intricate non-linear relationships in complex high-dimensional data spaces.
However, despite the success of deep learning in predicting user responses, it
is also widely known that tree-based methods often outperform neural networks
in tabular data prediction tasks, especially in competitions.

The goal of ACM RecSys Challenge 2023 is to estimate the probability of the
app being installed. This task can be formulated naturally as a click-through
rate (CTR) or conversion rate (CVR) prediction problem, which is a type of
user response prediction. For rapid prototyping for the task, we propose a
lightweight solution including the following steps: 1) given a dataset
consisting of user and ad features, we employ adversarial validation to
exclude non-informative features effectively; 2) we perform feature
engineering to deal with noisy continuous features and categorical features
with high cardinality; 3) we utilize GBDT due to their superior performance
and scalability. Our contributions are as follows. First, we provide a
lightweight solution that combines adversarial validation and a set of feature
engineering techniques. Second, we show that even a single LightGBM model
without an additional ensembling performs quite well in user response
prediction, ranking ninth in the leaderboard.

##  2\. Related Work

###  2.1. User Response Prediction

Early studies that tackled user response prediction tasks include the
following significant studies (mainly CTR prediction): (McMahan et al., 2013;
He et al., 2014; Juan et al., 2016). Google (McMahan et al., 2013) presented a
regularized LR model based on an FTRL-Proximal algorithm for massive-scale
sponsored search advertising. They explored several memory-saving guidances
essential for building an industrial-scale CTR prediction system, such as
bloom filter, fewer bit encoding, and train data sub-sampling. Another work
(He et al., 2014) published by Facebook employed a combination of decision
trees and LR. The boosted decision tree performs non-linear supervised feature
encoding and feeds the encoded features as an input to LR. They also stressed
the importance of train data sub-sampling, negative down-sampling, and model
calibration to handle large-scale real-world data. Field-aware FM (Juan et
al., 2016), which is a variant of FM (Rendle, 2010), also showed an impressive
performance, showing the critical importance of capturing predictive feature
interactions.

Recent years have seen a growth of interest in the adoption of deep learning-
based models for CTR prediction. Due to the superior ability of representation
learning with multiple levels of abstraction, the use of deep neural networks
(DNN) in industrial-scale CTR prediction is gradually becoming the industry
standard (Wang et al., 2017; Zhou et al., 2018; Gu et al., 2020). The Deep &
Cross Network (DCN) (Wang et al., 2017) adopts a memory-efficient cross
network that explicitly learns predictive cross features without manual
feature engineering. This simple yet effective architecture with significantly
fewer parameters outperforms other baselines, including DNN, FM, and LR.
Alibaba proposed a Deep Interest Network (DIN) (Zhou et al., 2018), which
utilizes a local activation unit, similar to an attention mechanism, to
dynamically learn the representation of user interests based on their past
behaviors related to a specific advertisement. Deep Multifaceted Transformers
(DMT) (Gu et al., 2020), which models multiple types of behaviors (e.g.,
click, cart, and order) with multiple Transformers (Vaswani et al., 2017), has
been successfully deployed to JD.com, one of the largest E-commerce sites in
the world. DMT also shows that properly adopting multi-task learning that
recognizes the user’s different objectives can significantly improve real-
world online performance metrics such as CTR, Conversion Rate (CVR), and Gross
Merchandise Volume (GMV).

###  2.2. Competitive Data Science

Data science competitions have evolved from a niche community of passionate
participants to a widely popular platform attracting millions of data
scientists worldwide (Banachewicz et al., 2022). In 1997, the KDD Cup, one of
the earliest data science competitions, was held. In 2006, the Netflix Prize
(Bennett et al., 2007) was held, significantly impacting the development of
personalized recommender systems and collaborative filtering. Since then,
various data science competitions focusing on personalized recommendations and
user response prediction have been held at academic conferences and
competitive data science platforms.

We observed a wide variety of methodologies being used for personalized
predictions in competitions, from TF-IDF to advanced deep learning
architectures. Surprisingly, several top-ranking solutions utilized tree-based
models. For example, in 2017, 2018, 2019, 2020, and 2022 ACM RecSys
challenges111https://recsys.acm.org/challenges, the winning solutions utilized
GBDT such as LightGBM (Ke et al., 2017), CatBoost (Dorogush et al., 2018), and
XGBoost (Chen and Guestrin, 2016) with substantial feature engineering. Also,
the winning solution222https://www.kaggle.com/competitions/h-and-m-
personalized-fashion-recommendations/discussion/324070 of the H&M Personalized
Fashion Recommendations competition, recently held at Kaggle, took advantage
of different kinds of decision tree models (LightGBM and CatBoost) as a ranker
module. Although deep learning has achieved tremendous success in the computer
vision and natural language processing domains, it is known that tree-based
methods often outperform neural networks in tabular data prediction tasks,
especially when dealing with skewed distributions, heavy-tailed feature
distributions, and dataset irregularities (Grinsztajn et al., 2022a; McElfresh
et al., 2023; Grinsztajn et al., 2022b).

##  3\. Problem Formulation

###  3.1. Dataset Description

The data available for the challenge was provided by
ShareChat333https://sharechat.com/recsys2023, India’s largest homegrown social
media company with over 400M MAUs across all its platforms. The dataset
consists of 10M users who visited the ShareChat and Moj apps over three
months. The organizers preprocessed the dataset to have ten impressions for
each user. The objective of the challenge is to predict the probability of the
app being installed. Each row of the dataset consists of user and ad features.
The train data consists of subsampled history from the past $22$ days, and the
target variable is the probability of the app being installed on the $23$rd
day.

###  3.2. Evaluation

The objective of the challenge is to predict whether the user will install or
not for a given ad impression. The more accurately the probability is
estimated, the higher the expected revenue of the platform. The metric used in
the challenge was Normalized Cross Entropy (NCE):

(1) $\displaystyle
NCE=\frac{-\frac{1}{N}\sum_{i=1}^{N}(\frac{1+y_{i}}{2}\log{p_{i}}+\frac{1-y_{i}}{2}\log{(1-p_{i})})}{-(p\log{p}+(1-p)\log{(1-p)})},$

where $N$, $p_{i}$, $p$, and $y_{i}\in\\{-1,1\\}$ are the number of the
dataset, the estimated probability of the app being installed, the average
empirical probability of installation, and the label, respectively.

NCE is calculated by dividing the average log loss per impression by the
average log loss per impression that would occur if the model predicted the
background CTR (in our case, CVR) for every impression. In other words, NCE is
the predictive log loss normalized by the entropy of the empirical probability
of installation. The lower the NCE value, the better the model performance. In
our experiments, we use log loss as a proxy evaluation metric for local
validation because we can not access the ground-truth empirical CVR, $p$.

##  4\. Data Preparation & Preprocessing

Figure 1. Flow chart of our lightweight pipeline

###  4.1. Local Validation Strategy

The dataset includes temporal information as the $f_{1}$ variable, which
indicates the date for each row. We aim to predict $p_{i}$ as accurately as
possible for test data with $f_{1}=67$. Therefore, to reflect recent temporal
trends and to make the validation set’s distribution mimic the test set’s
distribution, we select the data with $f_{1}=66$ as a local validation set.
The entire procedure of the pipeline is shown in Fig. 1.

###  4.2. Adversarial Validation

Figure 2. AUC metrics of each feature in adversarial validation for detecting
a covariate shift.

In our preliminary experiments, we observed a severe discrepancy between the
local validation score and the leaderboard score. In particular, this
phenomenon was amplified when certain variables were added as features. This
overfitting issue seems to be due to a covariate shift, the discrepancy
between the distribution of train data and test data. To identify and address
the problem, we employ an adversarial validation approach.

The adversarial validation approach can be used to detect and address the
covariate shift. This approach involves training a binary classifier to
determine whether a sample belongs to the training or test set. When the
classifier’s performance is close to random guessing ($\mathtt{AUC}=0.5$), it
suggests that distinguishing between the training and test sets is
challenging, indicating that the distribution of both is relatively
consistent. On the other hand, if the classifier performs significantly better
than random guessing, it suggests the discrepancy between the distribution of
the training and test sets. We train an adversarial classifier for each
feature to detect potential features that exhibit the covariate shift between
train and test data. As shown in Fig. 2, some variables have very high
$\mathtt{AUC}$ scores, and these variables seem to show the covariate shift
that may cause the gap between validation performance and test performance.
Therefore, we can exclude these features from model training to reduce the
inconsistency between the local validation and the leaderboard.

##  5\. Feature Engineering

Based on the analysis in 4.2, we filter out the variables with
$\mathtt{AUC}>=0.75$ (indicating a potential covariate shift). Next, we encode
categorical features to informative continuous values to deal with high
cardinality and facilitate effective tree splits for GBDT models. In the
following paragraphs, we introduce how to deal with noisy continuous features
and how to encode categorical features. Please refer to our source
code444https://github.com/choco9966/recsys-challenge-2023 for other details of
feature engineering.

###  5.1. Dealing with Noisy Continuous Features

Figure 3. Pairwise correlation plot for continuous features: $f_{42}\sim
f_{79}$.

Fig. shows that some continuous variables exhibit a block structure indicating
a strong correlation with each other. Based on this observation, we found a
pattern presumed to be artificial noise injection. Specifically, some
variables represent an arithmetic sequence of unique values. More concretely,
given the ordered unique values $\\{v_{1},v_{2},...v_{M}\\}$ for a particular
feature, where $v_{n-1}<v_{n}$ for $n=2,...,M$, the general formula for the
$n$-th term is as follows: $v_{n}=v_{1}+(n-1)\Delta$. For example, variable
groups ($f_{42}$, $f_{52}\sim f_{57}$, $f_{74}\sim f_{76}$) and ($f_{44}\sim
f_{50}$ and $f_{71}\sim f_{73}$) have $\Delta\simeq 0.0385$ and $\Delta\simeq
0.5711$, respectively. We denoise these feature values by dividing the
corresponding $\Delta$, converting a data type from float to integer. The
transformed integer features are expected to be more descriptive and
informative when crossing features.

###  5.2. Encoding Categorical Features

The dataset includes many categorical features with high cardinality. We use
frequency encoding and CatBoost encoding to avoid the curse of dimensionality.
Frequency encoding, a.k.a. count encoding, replaces a category value with its
number of occurrences for a given categorical feature. This simple strategy
can be effective when the frequency is somewhat related to the target
variable. We compared the following three alternatives for frequency encoding:
(1) the number of occurrences during the previous day, (2) the number of
occurrences during the previous week, and (3) the number of occurrences up to
one day ago. We selected the second one due to its highest performance. Next,
we conduct CatBoost encoding, a target encoding variation that supports time-
aware encoding and regularization to avoid over-fitting. Similar to the
sliding window, the category of the target day was transformed after fitting
the encoder based on the data up to the previous day. The operations were
performed for the two target variables, click and install, respectively.

##  6\. Models

We employ GBDT as our prediction model. Specifically, we use LightGBM, which
has been proven to show state-of-the-art performance in many previous
competitions (Jankiewicz et al., 2019; Zzh et al., 2022). We also tested other
baseline models known to perform well in predicting tabular data, such as
TabNet (Arik and Pfister, 2021) and CatBoost. However, LightGBM was more
robust and performed the best in our experiments. The tuned hyperparameters of
LightGBM are as follows: number of leaves = $491$, max depth = $-1$, boosting
type = gbdt, and number of iterations = $10,000$ with early stopping rounds =
$100$. Note that we did not apply an ensemble technique that blends the
prediction results of several different models because we focus on a
lightweight solution for rapid prototyping.

##  7\. Experiments

Table 1. Performance of the prediction models. The best-performance model is denoted in bold. Model | Local Validation Score | Leaderboard Score
---|---|---
(1) Vanilla LightGBM | 0.375506 | 6.466970
(2) = (1) + Frequency Encoding | 0.368485 | 6.264767
(3) = (2) + Denoisng Features | 0.361400 | 6.218796
(4) = (3) + CatBoost Encoding + Additional Feature Engineering | 0.361309 | 6.071975
(5) Final LightGBM = (4) + Hyperparameter Tuning | 0.360991 | 6.059065
(6) CatBoost | 0.359850 | 6.106765
(7) TabNet | 0.369832 | 6.221323

To evaluate the effectiveness of our method, we conduct detailed experiments
and illustrate the results in Table 1. By incorporating validated features, we
can observe an improvement in the performance of our model. It is worth noting
that CatBoost reported a good performance on local validation but poor
performance on the leaderboard. Although both LightGBM and CatBoost belong to
GBDT, in our study, LightGBM showed a much more robust performance than
CatBoost. Fig. 4 shows the top 20 important features according to the LightGBM
feature importance measure based on the number of times the feature is used in
tree splitting. All experiments were conducted on virtual machines with 64
vCPUs, 120GB RAM, and NVIDIA RTX 3090 GPU.

Figure 4. LightGBM feature importance plot based on the number of tree splits.

##  8\. Conclusion

This paper presents our approach to the ACM RecSys Challenge 2023. Our
solution comprises adversarial validation, feature engineering, and a
prediction model. In the adversarial validation phase, we detect and exclude
potential features exhibiting the covariate shift. Next, in the feature
engineering phase, we focused on denoising noisy continuous features and
transforming categorical features into informative continuous values by
performing frequency or CatBoost encoding. We employ LightGBM as our primary
prediction model with tuned hyperparameters, and the final model showed
excellent performance on the leaderboard even without applying an additional
ensemble method. The proposed lightweight solution is simple yet effective for
the following reasons: 1) adversarial validation can filter out non-
informative features in the early stage, and 2) it does not apply the ensemble
method that often requires much effort to find the optimal configuration. We
believe our solution can be used for rapid prototyping for CTR/CVR prediction
tasks.

## References

  * (1)
  * Arik and Pfister (2021) Sercan Ö Arik and Tomas Pfister. 2021.  Tabnet: Attentive interpretable tabular learning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 6679–6687.
  * Banachewicz et al. (2022) Konrad Banachewicz, Luca Massaron, and Anthony Goldbloom. 2022\.  _The Kaggle Book: Data analysis and machine learning for competitive data science_.  Packt Publishing Ltd.
  * Bennett et al. (2007) James Bennett, Stan Lanning, et al. 2007\.  The netflix prize. In _Proceedings of KDD cup and workshop_ , Vol. 2007. New York, 35.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_. 785–794.
  * Dorogush et al. (2018) Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018\.  CatBoost: gradient boosting with categorical features support.  _arXiv preprint arXiv:1810.11363_ (2018).
  * Grinsztajn et al. (2022a) Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. 2022a.  Why do tree-based models still outperform deep learning on typical tabular data?  _Advances in Neural Information Processing Systems_ 35 (2022), 507–520.
  * Grinsztajn et al. (2022b) Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. 2022b.  Why do tree-based models still outperform deep learning on typical tabular data?  _Advances in Neural Information Processing Systems_ 35 (2022), 507–520.
  * Gu et al. (2020) Yulong Gu, Zhuoye Ding, Shuaiqiang Wang, Lixin Zou, Yiding Liu, and Dawei Yin. 2020\.  Deep multifaceted transformers for multi-objective ranking in large-scale e-commerce recommender systems. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2493–2500.
  * He et al. (2014) Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. 2014\.  Practical lessons from predicting clicks on ads at facebook. In _Proceedings of the eighth international workshop on data mining for online advertising_. 1–9.
  * Jankiewicz et al. (2019) Paweł Jankiewicz, Liudmyla Kyrashchuk, Paweł Sienkowski, and Magdalena Wójcik. 2019\.  Boosting algorithms for a session-based, context-aware recommender system in an online travel domain. In _Proceedings of the Workshop on ACM Recommender Systems Challenge_. 1–5.
  * Juan et al. (2016) Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. 2016\.  Field-aware factorization machines for CTR prediction. In _Proceedings of the 10th ACM conference on recommender systems_. 43–50.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017\.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ 30 (2017).
  * McElfresh et al. (2023) Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Micah Goldblum, Colin White, et al. 2023\.  When Do Neural Nets Outperform Boosted Trees on Tabular Data?  _arXiv preprint arXiv:2305.02997_ (2023).
  * McMahan et al. (2013) H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. 2013\.  Ad click prediction: a view from the trenches. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1222–1230.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization machines. In _2010 IEEE International conference on data mining_. IEEE, 995–1000.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2017) Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017\.  Deep & cross network for ad click predictions.  In _Proceedings of the ADKDD’17_. 1–7.
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018\.  Deep interest network for click-through rate prediction. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 1059–1068.
  * Zzh et al. (2022) Zzh, Wei Zhang, and Wentao. 2022.  Industrial Solution in Fashion-domain Recommendation by an Efficient Pipeline using GNN and Lightgbm.  In _Proceedings of the Recommender Systems Challenge 2022_. 45–49.


# Contrastive Video-Language Learning with Fine-grained Frame Sampling

Zixu Wang1, Yujie Zhong2, Yishu Miao3, Lin Ma2, Lucia Specia1
1Language and Multimodal AI Lab (LAMA), Imperial College London
2Meituan Inc., 3Haiper.ai
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Despite recent progress in video and language representation learning, the
weak or sparse correspondence between the two modalities remains a bottleneck
in the area. Most video-language models are trained via pair-level loss to
predict whether a pair of video and text is aligned. However, even in paired
video-text segments, only a subset of the frames are semantically relevant to
the corresponding text, with the remainder representing noise; where the ratio
of noisy frames is higher for longer videos. We propose FineCo (Fine-grained
Contrastive Loss for Frame Sampling), an approach to better learn video and
language representations with a fine-grained contrastive objective operating
on video frames. It helps distil a video by selecting the frames that are
semantically equivalent to the text, improving cross-modal correspondence.
Building on the well established VideoCLIP model as a starting point, FineCo
achieves state-of-the-art performance on YouCookII, a text-video retrieval
benchmark with long videos. FineCo also achieves competitive results on text-
video retrieval (MSR-VTT), and video question answering datasets (MSR-VTT QA
and MSR-VTT MC) with shorter videos.

Contrastive Video-Language Learning with Fine-grained Frame Sampling

Zixu Wang1, Yujie Zhong2, Yishu Miao3, Lin Ma2, Lucia Specia1 1Language and
Multimodal AI Lab (LAMA), Imperial College London 2Meituan Inc., 3Haiper.ai
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Human perception is multimodal, including visual, textual, and audial
information. To achieve human-level perceptional ability, intelligent systems
need to understand and interpret these multimodal signals and summarise the
relevant information in them. Learning from video and language data has
received significant attention in recent multimodal machine learning work for
downstream tasks that require joint understanding of video and textual
information, including text-video retrieval (Lin et al., 2014; Liu et al.,
2019; Miech et al., 2018; Wang et al., 2016; Bain et al., 2021), video
question answering (Fan et al., 2019; Yang et al., 2021; Huang et al., 2020;
Jiang et al., 2020; Le et al., 2020; Lei et al., 2021), and video captioning
(Ging et al., 2020; Luo et al., 2020; Zhang et al., 2020b). In most of this
work, contrastive learning (Gutmann and Hyvärinen, 2010) is used as training
objective.

The aim of a cross-modal contrastive loss is to maximise the similarity
between an aligned video-text pair while minimising the similarity for all
other pairs. One issue with standard cross-modal contrastive loss is that it
focuses on pair-level alignment but ignores the negative effects of irrelevant
frames that are present in a single video clip, even in a pair of aligned
video and text. We define irrelevant frames as those with no or little shared
semantics with the text. These irrelevant frames may negatively affect the
contribution of frames that are semantically similar to the text, which
further results in less informative video representation. Therefore, we posit
that frame-level learning is a better strategy for video-language tasks.

Figure 1: Illustration of the weak correspondence problem in video-language
learning. Given a pair of video and its text (e.g. caption, instruction, or
transcription), only a subset of the frames (here indicated by coloured
bounding boxes) is semantically aligned to the textual content. The remaining
frames represent irrelevant visual information and will not contribute to
language grounding on videos.

In this paper, we propose FineCo, an approach that has a frame selector to
sample relevant frames in a video and is trained with a fine-grained
contrastive loss on frame-text pairs, in order to mitigate the problem of weak
correspondence in video-language representation learning. Existing video-
language learning approaches (Miech et al., 2020; Xu et al., 2021) only
optimise pair-level alignment but do not explicitly learn which part of a
video contributes to its alignment with the text. FineCo focuses on aligning
relevant frames with the text. It is inspired by the text-based temporal
localisation task (Zhang et al., 2020a), however, the motivation of FineCo is
different: to learn better video-level representation by adding a frame-level
contrastive learning signal to the pair-level objective, with no need for
temporal annotation within a video-text pair.

We hypothesise that FineCo is particularly beneficial for long videos, where
each video provides more information and only a small proportion of frames
will be relevant to its text counterpart, as shown in Figure 1. FineCo is able
to model frame-text similarity through fine-grained contrastive learning,
where the most informative frames are paired with the text as positive pairs
and the remaining frames, as negatives. It then explicitly contrasts the
selected informative frames against the noisy frames, without the need for
frame-text annotations. This frame-level distillation provides a strong
learning signal, which encourages the alignment of semantically equivalent
video-text pairs. The fine-grained contrastive loss abstracts the learning
signal from pair-level annotations and is trained in an end-to-end manner.
This combination of pair-level learning signal and frame-level contrastive
loss is novel and effective, and boosts the performance on two important
video-language benchmark tasks, especially in text-video retrieval with longer
videos. We devised FineCo by building on the recently proposed and well
performing VideoCLIP (Xu et al., 2021), in which a video clip is represented
as sequence of frame features.

Our contributions are summarised as follows: (1) We propose FineCo, an
approach trained with fine-grained contrastive loss to mitigate the weak
correspondence problem in video-text pairs; (2) We use FineCo to distil a
video clip by sampling frames that are relevant to its text counterpart
according to frame-text similarities; (3) On text-video retrieval and video
question answering benchmarks, we show that FineCo achieves state-of-the-art
performance on YouCookII and MSR-VTT MC (multiple choice).

##  2 Related Work

##### Contrastive Learning

The use of contrastive loss (Gutmann and Hyvärinen, 2010) has become the
dominant paradigm for learning video-language representations. The aim is to
maximise the similarity of video-text pairs that are aligned to each other
(positive pairs) while pushing away irrelevant (negative) pairs. However, the
semantic alignment between most video-text pairs is weak, which makes it
difficult to ground textual information on the videos. In order to mitigate
the pair-level weak alignment issue, MIL-NCE (Miech et al., 2020) leverages
multiple surrounding captions as the positive pairs and makes use of multiple
instance learning (MIL) (Dietterich et al., 1997) with contrastive loss to
mitigate noise in cross-modal correspondences. The main idea is to consider
multiple contextual sentences for matching a video, instead of only comparing
a video against a single sentence. To alleviate the issue that semantically
equivalent videos and texts from different pairs may be taken as dissimilar in
contrastive learning, support-set (Patrick et al., 2021) introduces a
generative approach for captioning over a set of visual candidates that
ensures that video-language representation does not over specialise to
individual samples. MIL-NCE and support-set focus on pair-level contrastive
signals to align relevant video-text pairs. However, even within a positive
video-text pair, the video is likely to contain many irrelevant frames.
Therefore, it can be beneficial to distil the video such that only the
relevant frames, i.e. those which have similar content to the text, are
selected for cross-modal learning.

##### Video-language Learning

(Sun et al., 2019; Zhu and Yang, 2020; Gabeur et al., 2020; Li et al., 2020a;
Miech et al., 2020; Ging et al., 2020; Luo et al., 2020) have shown promising
results for video-language learning with pre-training followed by fine-tuning.
This strategy has become very prominent since the release of BERT (Devlin et
al., 2019) and many image-text pre-training frameworks (Tan and Bansal, 2019;
Li et al., 2019, 2020b; Zhang et al., 2021; Chen et al., 2020; Zhang et al.,
2019; Kim et al., 2021; Li et al., 2021, 2022). The release of datasets such
as HowTo100M (Miech et al., 2019) and WebVid-2M (Bain et al., 2021) has
enabled large-scale pre-training on unlabelled video-text pairs to improve
representation learning of video and language. Many approaches (Miech et al.,
2020; Zhu and Yang, 2020; Patrick et al., 2021) use HowTo100M as their pre-
training dataset. FiT (Bain et al., 2021) uses WebVid-2M and Google Conceptual
Captions (CC3M) to take advantage of the large collection of video-text and
image-text pairs for pre-training. However, large pre-training datasets rely
on loosely aligned video-text pairs, without any fine-grained supervision on
alignment. This makes it difficult to learn cross-modal cues present in the
given video-text pairs. It is also computationally expensive to improve video-
language representation learning, given that videos can contain a large number
of frames, especially longer videos. ClipBERT (Lei et al., 2021) randomly
samples a few frames from a video for video-language representation learning.
Their motivation is to minimise memory and computation costs from processing
the full sequence of frames. This sampling strategy is over simplistic and can
thus be improved by better approaches to select frames based on their
relevance to the paired text.

##  3 FineCo

###  3.1 Preliminaries

The most widely used objective function for video-language learning is
contrastive loss, specifically the softmax version of noise-contrastive
estimation (NCE) (Gutmann and Hyvärinen, 2010). It is formulated as

$\sum^{n}_{i=1}\log\left(\frac{e^{f(x_{i})^{T}g(y_{i})}}{e^{f(x_{i})^{T}g(y_{i})}+\sum\limits_{(x^{\prime},y^{\prime})\in\mathcal{N}_{i}}e^{f(x^{\prime}_{i})^{T}g(y^{\prime}_{i})}}\right)$
(1)

where $x_{i}$ denotes a video clip and $y_{i}$ represents the corresponding
text (e.g. a caption, an instruction, or transcription); $f$ and $g$ are video
encoder and text encoder respectively; $e^{f(x_{i})^{T}g(y_{i})}$ denotes the
similarity of a positive video-text pair, calculated as the exponentiated dot
product of the video representation $f(x_{i})$ and text representation
$g(y_{i})$; $\mathcal{N}_{i}$ is a set of negative video-text pairs
$x^{\prime}_{i}$ and $y^{\prime}_{i}$ that are not aligned.

This contrastive loss leverages pair-level similarity of video and text, but
ignores the fact that weak video-language correspondence does not stem only
from entirely negative pairs of video and text, but also from frame-level
noise, which happens even when a video-text pair is aligned as a whole.
Standard contrastive loss does not explicitly model frame-text relevance, i.e.
it does not differentiate between frames that are semantically equivalent to
the corresponding text and frames that are not. It can thus suffer by learning
from noisy signals, particularly in long videos with various scenes.

Figure 2: FineCo architecture. Given a sequence of frames in a video clip $x$,
the video encoder $f$ transforms them into a sequence of video features. The
corresponding sentence $y$ is fed into the text encoder $g$ to get the text
representation. The frame selector $FS$ takes the text representation and the
sequence of video features as inputs and outputs the similarities
(probabilities of each frame being relevant). The top $k$ frames are then used
as the positive candidates and the remaining ones as negative, both of which
are combined with the text representation to compute the fine-grained
contrastive loss.

###  3.2 Fine-grained Contrastive Learning

A video consists of a sequence of frames. For video-language learning, the
video is paired with a text which describes/refers to some of the content of
the video. For most tasks, only some of the visual information has an
equivalent textual signal, e.g. a video description is only a summary of the
visual information. To sample and optimise for the relevant visual information
from a video, we propose a fine-grained contrastive loss to distil each video-
text pair.

Formally, a video-text pair is denoted as ($x$, $y$), where $x$ is a video
clip consisting of a sequence of $N$ video frames
$\\{x_{1},x_{2},\dots,x_{K}\\}$ where $K$ is the number of frames in the video
clip, and $y$ is the paired text. We assume that a video $x$ contains a set of
$C$ positive frames $\mathcal{P}(x)$ and a set of $(K-C)$ negative frames
$\mathcal{N}(x)$, where positive frames contains relevant information to the
text while negative frames are noisy/irrelevant ones. The aim is to maximise
the joint probability of relevant frame-text pairs $(x_{k},y)$ by
exponentiating the similarity of the two representations:

$p(x_{k},y)=h(f(x_{k}),g(y))\propto e^{\text{sim}(f(x_{k}),g(y))}$ (2)

####  3.2.1 Objective Function

Given $n$ pairs of video representation $f(x)$ and text representation $g(y)$,
the $i$th pair is denoted as
$f(x_{i})=\\{f(x_{i_{1}}),f(x_{i_{2}}),\dots,f(x_{i_{K}})\\}$ and $g(y_{i})$,
our fine-grained contrastive loss $\mathcal{L}$ is defined as:

$\displaystyle\mathcal{A}_{i}$
$\displaystyle=\sum\limits_{x_{i_{k}}\in\mathcal{P}(x_{i})}e^{\text{sim}(f(x_{i_{k}}),g(y_{i}))}$
(3) $\displaystyle\mathcal{B}_{i}$
$\displaystyle=\sum\limits_{x^{\prime}_{i_{k}}\in\mathcal{N}(x_{i})}e^{\text{sim}(f(x^{\prime}_{i_{k}}),g(y_{i}))}$
$\displaystyle\mathcal{L}$
$\displaystyle=\sum^{n}_{i=1}\log\left(\frac{\mathcal{A}_{i}}{\mathcal{A}_{i}+\mathcal{B}_{i}}\right)$

where $\mathcal{P}(x_{i})$ contains the positive frames in a video that have
higher similarities to the text representation $g(y_{i})$, and
$\mathcal{N}(x_{i})$ is the set of remaining frames in the same video, which
refers to the negative frames. The similarity is calculated by our frame
selector ($\mathcal{FS}$) (Section 3.2.2) with the frame $x_{i_{k}}$ and text
representations $y_{i}$ as inputs. $\mathcal{A}_{i}$ and $\mathcal{B}_{i}$
represent the sum of similarity scores for positive and negative frames,
respectively. This objective function aims to maximise the similarity between
the positive frames and the text, while increasing the dissimilarity between
the negative frames and the text. Therefore, the sampled relevant frames can
directly contribute to the cross-modal learning of video-text alignments.

####  3.2.2 Assignment of Positives and Negatives

Inspired by MIL-NCE (Miech et al., 2020), which makes use of multiple
sentences for matching a video and its corresponding text, we extract multiple
positive frames from the complete set according to the similarity score
between each frame and the text. Consider an example $(x,y)$ with $K$ frames
$\\{x_{1},x_{2},\dots,x_{K}\\}$, we introduce a frame selector $\mathcal{FS}$,
a cross-modal module which takes video and text representation as the input
and outputs the similarity scores between each frame and the text, denoted as:

$\text{sim}_{k}=\mathcal{FS}(f(x_{k}),g(y));x_{k}\in\\{x_{1},x_{2},\dots,x_{K}\\}$
(4)

where $f(x_{k})$ is the representation of the $k$th frame; $g(y)$ is the
representation that encodes the meaning of the complete text sequence, which
is used to find semantically similar frames in the corresponding video $x$;
$\text{sim}_{k}$ is the similarity score between the $k$th frame and the text
$y$.

By ranking the similarity scores of $K$ frames, we choose top $C$ frames to
form the positive set and the remaining $(K-C)$ as the negative set. This is
an explicit sampling strategy which extracts the relevant frames in a video.
There is no constraint on the architecture of $\mathcal{FS}$. In this work, we
use a multi-layer perceptron (MLP) with a softmax layer to compute the
similarity scores.

###  3.3 Model Architecture

As our methodology focuses on fine-grained contrastive learning signal for a
single pair of video and its text, it makes no assumptions on the encoder
architectures and can work with pre-training frameworks with different video
and text backbones. In our experiments, we use Transformer (Vaswani et al.,
2017) as both the video encoder and the text encoder, as we detail below.

####  3.3.1 Text Encoder

We use BERT (Devlin et al., 2019) as the text encoder $g$ to get text
representation $g(y)$. The text encoder is trained together with the video
encoder to learn better text representations. Following VideoCLIP (Xu et al.,
2021), we use average pooling (instead of using the [CLS] token) as the final
text encoding. The text representation is used as the guiding element and
anchor to calculate the frame-text similarity scores and to sample the most
semantically similar frames in a video clip.

####  3.3.2 Video Encoder

Our video encoder $f$ is composed of an S3D (Xie et al., 2018; Miech et al.,
2020) and a Transformer (Vaswani et al., 2017), following VideoCLIP (Xu et
al., 2021). To speed up training, we use a S3D pre-trained on HowTo100M (Miech
et al., 2019) to extract pre-trained video features, where the video feature
of a video clip is represented by a sequence of video frames. The output from
the S3D is formulated as $x=[x_{1},x_{2},\dots,x_{K}]$, where $x$ is the
representation of a sequence of video frames. We extract the frames at a rate
of one frame per second, so the number of video frames equals the number of
seconds. $x$ is concatenated with learnable tokens [CLS] and [SEP] at the
beginning and the end of the sequence, respectively. We then train the
Transformer using the pre-extracted video representation as the input, to
obtain the last hidden states as the representation of the sequence of video
frames.

###  3.4 Training

Training with the pair-level contrastive loss is challenging due to the
intractability of computing the normalisation constant over all possible pairs
of videos and texts. It is however more feasible in our fine-grained
contrastive loss as the number of possible frames in a single video clip is
limited. The normalising constant is computationally tractable and can be
directly computed by summing over exponentiated similarity scores across all
the frame-text pairs. The overall training objective ($\mathcal{L}$) is
defined by combining our fine-grained contrastive loss ($\mathcal{L}_{1}$) and
task-specific losses ($\mathcal{L}_{2}$), denoted by
$\mathcal{L}=\mathcal{L}_{1}+\mathcal{L}_{2}$; where in text-video retrieval,
the task loss $\mathcal{L}_{2}$ is pair-level contrastive loss and in video
question answering, it is cross-entropy.

###  3.5 Inference

For text-video retrieval, there is no cross-modal fusion module at inference
time. It requires only video and text representations which are first
projected to a common dimension via linear layers. The similarity between a
video-text pair is calculated by performing the exponentiated dot product
between the two projected embeddings. This ensures retrieval inference is of
trivial cost, since it is indexable and scalable to large-scale retrieval at
inference time. For video question answering, we follow the pipeline in Figure
2, where we concatenate the video and text representations, and feed it into
an MLP module to obtain the final representation for answer prediction.

##  4 Experiments

In this section, we describe the tasks and datasets used in our experiments
with FineCo.

###  4.1 Datasets and Metrics

FineCo is mainly beneficial for long videos, therefore we focus our evaluation
on YouCookII (Zhou et al., 2018) \- a text-video retrieval dataset with long
videos. YouCookII consists of 2K cooking videos with 14K video clips. The
videos are of a total duration of 176 hours with average 5.26 minutes per
video. Each video clip is annotated with one sentence on a cooking
instruction. It is collected from YouTube and contains 89 types of recipes. We
split the dataset according to Miech et al. (2020) where 9.6k video-text pairs
are used for training and 3.3k pairs for validation.

We further evaluate FineCo on other benchmark datasets for text-video
retrieval and video question answering with shorter videos. MSR-VTT (Xu et
al., 2016) is another popular benchmark dataset for text-video retrieval. It
contains 10K YouTube videos (an average 20 seconds per video) with 200K
captions. We report the results on the 1k test split and use the remaining 9k
videos for training. MSVD (Chen and Dolan, 2011) consists of 80K captions for
1,970 videos from YouTube, with each video containing 40 sentences. We use the
standard split of 1200, 100, and 670 videos for training, validation, and
testing as in (Liu et al., 2019; Patrick et al., 2021). DiDeMo (Hendricks et
al., 2018) contains 10K Flickr videos with 40K sentences. Following (Liu et
al., 2019; Lei et al., 2021), we evaluate paragraph-to-video retrieval, where
all sentence descriptions from a video are concatenated into a single query.
MSR-VTT QA contains 10K videos and 243K open-ended questions, which is created
using the videos and captions from original MSR-VTT. We use 1500 most frequent
answers as the answer vocabulary, which covers over 93% samples. MSR-VTT MC
(multiple choice) is also created from original MSR-VTT. Multiple choice QA is
formulated as a video-text retrieval task where the videos are the questions
and captions are the answers.

##### Evaluation Metrics

Following the standard evaluation protocols as described in most video-
language work (Miech et al., 2019; Zhang et al., 2018; Mithun et al., 2018;
Miech et al., 2018, 2020), we report the text-video retrieval performance
using recall-based metrics: Recall at rank K (R@K) which measures the rate at
which the correct video is retrieved amongst the top ranked results, and
Median Rank (MdR) which calculates the median of a list of indices
representing the rank of the ground truth video; where the higher R@K and
lower median rank indicate better performance. For MSR-VTT QA and MSR-VTT MC,
accuracy is reported, as in Xu et al. (2021).

###  4.2 Training Details

To minimise computation costs, we use S3D (Xie et al., 2018) for video feature
extraction, which is pre-trained on HowTo100M (Miech et al., 2019) following
MIL-NCE (Miech et al., 2020). The feature dimensionality is 512 (e.g. given a
10-second video, the shape of the video feature extracted is $[10,512]$). We
apply video feature pre-extraction to all the downstream datasets in our
experiments. We follow the pre-training steps as in VideoCLIP (Xu et al.,
2021) where pre-training is done using HowTo100M, which contains uncurated
instructional videos. A total of 1.1M videos are used for pre-training after
cleaning and filtering.

For the video Transformer encoder, we use 6 attention blocks, while for the
text Transformer encoder, we use 12 blocks. The weights for both encoders are
initialised with bert-base-uncased. The maximum length of a video is 32; for
text inputs it is 64. Before feeding video and text inputs into their
respective encoders, [CLS] and [SEP] tokens are concatenated to the beginning
and end of each modality. All the models are trained on one NVIDIA Tesla V100
GPU with 32 GB of RAM memory for 15 epochs, with fp16 precision for 2-3 hours.
We select the final checkpoint according to the loss on the validation set.
Optimisation is performed using Adam (Kingma and Ba, 2015) with a learning
rate of 5e-5. The model takes 1000 steps for warm-up, and we use a learning
rate schedule with polynomial decay.

##  5 Results

In this section, we describe the experimental results and compare FineCo with
state-of-the-art approaches (Section 5.1). We further explore different
sampling strategies to select positive frames (Section 5.2), and fine-grained
word sampling (Section 5.3). We also provide examples of the frames selected
by FineCo (Section 5.4).

###  5.1 Comparison to State-of-the-art

Overall, as we detail below, FineCo outperforms its base model VideoCLIP
across all benchmark datasets. Additionally, it achieves state-of-the-art
performance on YouCookII and MSR-VTT MC.

####  5.1.1 Text-video Retrieval

We start by evaluating on YoucookII, which contains longer videos than other
text-video benchmarks, and is therefore more challenging for video-language
representation learning. As shown in Table 1, FineCo outperforms all previous
approaches by a large margin. We report results w/ and w/o Dual Softmax (DS)
following Cheng et al. (2021) and Gao et al. (2021). In Dual Softmax, given a
similarity matrix in text-video retrieval, a prior probability is calculated
in the cross direction, which is then multiplied with the original similarity
matrix as an efficient regulariser. FineCo surpasses previous state-of-the-art
with fine-grained contrastive loss (3.5% gains for R@1). Dual Softmax further
improves the results (1.6% for R@1) and achieves an even higher state-of-the-
art (37.3% R@1).

YouCookII | R@1 | R@5 | R@10 | MedR
---|---|---|---|---
HowTo100M (Miech et al., 2019) | 8.2 | 24.5 | 35.3 | 24.0
MIL-NCE (Miech et al., 2020) | 15.1 | 38.0 | 51.2 | 10.0
COOT (Ging et al., 2020) | 16.7 | 40.2 | 52.3 | 9.0
UniVL (Luo et al., 2020) | 28.9 | 57.6 | 70.0 | 4.0
VideoCLIP (Xu et al., 2021) | 32.2 | 62.6 | 75.0 | 3.0
Ours w/o DS | 35.7 | 65.9 | 77.5 | 3.0
Ours w DS | 37.6 | 66.6 | 78.2 | 3.0

Table 1: YouCookII Retrieval Results. DS denotes Dual Softmax.

We provide additional results on text-video retrieval across MSR-VTT 111We
omit the results of text-video retrieval on MSR-VTT from CLIP (Radford et al.,
2021) models (Cheng et al., 2021; Luo et al., 2021; Fang et al., 2021; Gao et
al., 2021) as it would not be a fair comparison since CLIP-based models
benefit mainly from large-scale image-text pre-training, which we do not use.
(Table 2), MSVD (Table 3), and DiDeMo (Table 4). Our reported scores of
VideoCLIP on MSVD and DiDeMo are from our implementation as their paper does
not test on the datasets. As FineCo builds on VideoCLIP (Xu et al., 2021), our
results are directly comparable with the scores reported in VideoCLIP. 222 We
also implemented FineCo in FiT (Bain et al., 2021), however the improvements
are not obvious as in VideoCLIP. The reason might be the difference of video
encoding in VideoCLIP and FiT. FineCo contributes more to complete frame
features where a video is encoded into a long sequence of video features with
more temporally contextual information, rather than only a few visual frames
in ViT (Dosovitskiy et al., 2021) and Timesformer (Bertasius et al., 2021).
From the additional results, it can be seen that FineCo outperforms VideoCLIP
on all text-video retrieval datasets by a large margin. This shows that FineCo
is generalisable to various types of text-video retrieval data. The smaller
improvements (e.g., 30.9% $\rightarrow$ 32.6% R@1 on MSR-VTT 1k in Table 2)
compared to those on YouCookII (32.2% $\rightarrow$ 37.6% R@1) might be due to
the less varied scenes in shorter videos of MSR-VTT, which makes it
challenging to distinguish among intra-video frames in a short video.

MSR-VTT 1k | R@1 | R@5 | R@10 | MedR
---|---|---|---|---
JSFusion (Yu et al., 2018) | 10.2 | 31.2 | 43.2 | 13.0
HowTo100M (Miech et al., 2019) | 14.9 | 40.2 | 52.8 | 9.0
ClipBERT (Lei et al., 2021) | 22.0 | 46.8 | 59.9 | 6.0
Support-set (Patrick et al., 2021) | 30.1 | 58.5 | 69.3 | 3.0
FiT (Bain et al., 2021) | 32.5 | 61.5 | 71.2 | 3.0
VideoCLIP (Xu et al., 2021) | 30.9 | 55.4 | 66.8 | 4.0
Ours | 32.6 | 62.1 | 71.4 | 3.0

Table 2: MSR-VTT Results - 1k

MSVD | R@1 | R@5 | R@10 | MedR
---|---|---|---|---
VSE (Kiros et al., 2014) | 12.3 | 30.1 | 42.3 | 14.0
VSE ++ (Faghri et al., 2018) | 15.4 | 39.6 | 53.0 | 9.0
CE (Liu et al., 2019) | 19.8 | 49.0 | 63.8 | 6.0
Support-set (Patrick et al., 2021) | 28.4 | 60.0 | 72.9 | 4.0
FiT (Bain et al., 2021) | 33.7 | 64.7 | 76.3 | 3.0
VideoCLIP (Xu et al., 2021) | 26.4 | 52.2 | 63.3 | 5.0
Ours | 27.2 | 54.0 | 64.0 | 5.0

Table 3: MSVD Results

DiDeMo | R@1 | R@5 | R@10 | MedR
---|---|---|---|---
S2VT (Venugopalan et al., 2015) | 11.9 | 33.6 | - | 13.0
FSE (Zhang et al., 2018) | 13.9 | 36.0 | - | 11.0
CE (Liu et al., 2019) | 16.1 | 41.1 | - | 8.3
ClipBERT (Lei et al., 2021) | 20.4 | 44.5 | 56.7 | 7.0
FiT (Bain et al., 2021) | 31.0 | 59.8 | 72.4 | 3.0
VideoCLIP (Xu et al., 2021) | 16.6 | 46.9 | - | -
Ours | 19.5 | 48.8 | 55.9 | 7.0

Table 4: DiDeMo Results

Note that video-text pairs in these downstream datasets are constructed to be
aligned in order to provide strong supervision learning signals to video-
language representation learning. FineCo distils aligned video-text pairs and
achieves noticeable improvements over approaches without any frame sampling,
which corroborates our hypothesis that there are irrelevant or less useful
frames in a video even if it is annotated as aligned to its text counterpart.

MSR-VTT QA | Accuracy
---|---
AMU (Xu et al., 2017) | 32.5
HME (Fan et al., 2019) | 33.0
HCRN (Le et al., 2020) | 35.6
ClipBERT (Lei et al., 2021) | 37.4
VideoCLIP (Xu et al., 2021) | 35.9
Ours | 37.4

Table 5: MSR-VTT QA Results Table 6: Comparison of different sampling
strategies for positive frames.

Strategy | fixed-k ($k=1,10,30,50,100,256$) | median | ratio ($30\%,50\%,80\%$) | random
---|---|---|---|---
R@1 | 26.90 | 30.44 | 37.17 | 37.32 | 37.04 | 34.80 | 37.62 | 37.29 | 36.99 | 36.85 | 30.08

fixed-k | 1 | 5 | 10 | 15 | 20 | 25 | 32
---|---|---|---|---|---|---|---
MSR-VTT QA | 35.5 | 36.3 | 36.2 | 36.8 | 37.4 | 37.2 | 35.9
MSR-VTT MC | 90.3 | 92.3 | 92.6 | 92.4 | 92.6 | 92.7 | 92.1

Table 7: Effect of different number of positive frames on MSR-VTT QA and MSR-
VTT MC. When $k=32$, FineCo equals VideoCLIP.

MSR-VTT MC | Accuracy
---|---
MLB (Kim et al., 2016) | 76.1
JSFusion (Yu et al., 2018) | 83.4
ActBERT (Zhu and Yang, 2020) | 85.7
ClipBERT (Lei et al., 2021) | 88.2
VideoCLIP (Xu et al., 2021) | 92.1
Ours | 92.7

Table 8: MSR-VTT MC Results

####  5.1.2 Video Question Answering

Tables 5 and 8 show the results on video question answering (VideoQA) for MSR-
VTT QA and MSR-VTT MC, respectively. For both datasets, FineCo improves over
VideoCLIP. For MSR-VTT MC, it achieves a new state-of-the-art (92.7%
accuracy). This further shows the generalisation ability of FineCo across
different video-language tasks and datasets.

For MST-VTT QA, the score reported for VideoCLIP is from our implementation as
their paper does not test on this dataset. For MSR-VTT MC, the score reported
is from the original paper. For VideoQA, we note that ClipBERT also achieves
good results, which might be because it employs a multimodal Transformer
encoder after two separate encoders for the video and the question to learn
better cross-modal relationships. The improvement is particularly noticeable
on MSR-VTT MC, which quantitatively suggests that FineCo can distil question-
relevant frames to improve answer accuracy. We speculate that this is because
a question only needs partial information in some frames of a video clip to be
answered, which is addressed by FineCo.

###  5.2 Decision on Number of Frames

Given a pair of video clip and text, we choose the positive frames according
to the similarities between each frame and the text. The number of positive
frames $k$ is the key factor, deciding the set of frames to be treated as
positive, and hence the extent of the contribution of the fine-grained
contrastive learning signal. We propose four strategies to choose positive
frames in a video clip.

Fixed-k: We select a fixed number of positive frames which have the highest
similarities to the text. We experiment with $k=[1,10,30,50,100,256]$ as the
number of positive frames, with 256 as the maximum number of frames (one frame
per second).333We set the maximum length of a video sequence to 256 frames for
YouCookII, but 32 frames for other datasets with much shorter videos. Median:
We use the averaged similarity medians in a mini-batch as the thresholds for
each video: in a sequence of video frames, the ones with higher similarities
than the median are used as the positive frames. The number of positive frames
will vary across different mini-batches, depending on the distribution of
similarities. Ratio: We apply 30%, 50%, and 80% of the original video length
(without padding or trimming) as the positive frames. Note that different
video clips have different lengths, so the number of sampled frames will
differ from video to video. Random: We randomly sample $k=50$ frames in a
video clip as the positives.

We show the performance of the four strategies on YouCookII in Table 6. Median
has the best performance ($37.62$), which is followed by fixed-k with $k=50$
($\approx 20\%$ of the data) ($37.32$), and similarly to ratio with $30\%$
($37.29$). This indicates that on average only $\approx 20\%-30\%$ frames in
the long videos from YouCookII are informative for the retrieval task. Fixed-k
with $k=1$ has the lowest score, which makes sense given that the entire
videos are summarised by the one most similar frame to be used as the positive
candidate. This mistakenly treats many other possibly relevant frames as
negative frames, hence degrading the performance significantly. The best
number $50$ indicates that for most video-text pairs in YouCookII, 50 frames
($=$50 seconds as we extract video features at a rate of one feature per
second, so the length of the extracted video features is the same as the
number of seconds) ($\approx 20\%$) are the most relevant and sufficient. For
random, we choose $k=50$ as this was the best number according to the fixed-k
analysis. The comparison between random and fixed-k clearly shows that
sampling positive pairs based on their similarity to the text is an effective
strategy to improve performance on the downstream task: on the same number of
positive frames, fixed-k improves over random by 7.24%.

We also compare the performance of fixed-k on MSR-VTT QA and MSR-VTT MC. In
Table 7, we show that FineCo has the best performance on MSR-VTT QA with
$k=20$ and on MSR-VTT MC with $k=25$, where both have a sequence with maximum
number of $32$ frames. The ratio of positive frames ($\approx 70\%-80\%$) is
higher than in YouCookII. This corroborates our hypothesis that fine-grained
sampling is more applicable to longer videos, which tend to contain more
varied scenes and where there is more scope to filter out noisy or irrelevant
frames. Therefore, in video-language datasets with shorter videos, a higher
proportion of frames is needed as positive frames for effective contrastive
learning. As the number of informative frames $k$ in a video clip varies
across different types of videos, we recommend that this is treated as
hyperparameter that is tuned for each new dataset, following our fixed-k
strategy to select the number $k$ on a development set.

###  5.3 Fine-grained Word Sampling

Given the improvements of FineCo with fine-grained frame sampling, we were
curious about potential improvements if applying the same strategy to the text
instead of the video, i.e. sampling most relevant words. Therefore, we
experiment with this idea over a sequence of words to sample the most
informative words as those with the highest similarity to the entire video
clip in YouCookII. The text-video retrieval results in this setup are
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>These figures are similar to those obtained
by VideoCLIP<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>but substantially lower than our
results from FineCo in Table 1. The reason is intuitive: by removing certain
words, the meaning of the sentence or paragraph can be substantially
compromised, and having an understanding of the meaning of the complete text
is important for video-language tasks. Video frames, on the other hand, can be
more redundant or contribute less to the complete video understanding, and
therefore fine-grained sampling from frames proves more effective.

###  5.4 Qualitative Examples

To further elaborate the contribution of FineCo and understand the effect of
fine-grained contrastive loss, we show two examples where FineCo improves over
VideoCLIP in Figure 3.444We only show a subset of informative and irrelevant
frames for each example due to space limitations. As we can observe from the
examples, some of the information in each video clip can be considered
irrelevant, given the meaning of the text. For example, in the first case, the
long video (82 seconds) describes the cooking instruction “brush the circles
with egg washa and sprinkle with sesame seeds” but there are only two frames
delivering this meaning. This is a common feature in the YouCookII dataset,
hence the positive results from sampling subsets of frames. In the third
example we show a failure case where FineCo does not distinguish between
similar videos hence a similar but incorrect video retrieved. We also observed
failure cases where the video is either relatively short or less dynamic.
FineCo might not effectively distil these types of videos to find the most
informative frames. The issues could be potentially mitigated by incorporating
FineCo into large-scale video-language pre-training to learn from more dynamic
videos of various lengths.

Figure 3: Qualitative examples. FineCo makes correct retrieval predictions on
the frist two examples from YouCookII dataset. We calculate the frame-text
similarities and highlight the frames with the highest scores.

##  6 Conclusions

We propose FineCo, an approach with a fine-grained contrastive loss to
mitigate the weak correspondence problem in video-language representation
learning. Experiments conducted on text-video retrieval and video question
answering datasets suggest that FineCo can distil video frames that are
relevant to its corresponding text and contribute to significant gains in
performance, especially on the text-video retrieval dataset YouCookII with
long videos. FineCo achieves state-of-the-art on YouCookII and MSR-VTT MC, and
for text-video retrieval datasets with shorter videos, it substantially
improves over the base model. Ablation studies analyse the key factors in
FineCo including number of positive frames and word sampling. Our strategy for
frame selection is simple and can generalise to different video-language
frameworks, as long as they are based on contrastive learning, which is
standard in this area. In addition, we posit that FineCo can be useful for
video-language pre-training on large loosely or misaligned video-text
datasets. We hope that our work will draw attention to the need for frame-
level alignment to improve video-language representation learning.

## References

  * Bain et al. (2021) Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _IEEE International Conference on Computer Vision_.
  * Bertasius et al. (2021) Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021.  Is space-time attention all you need for video understanding?  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Chen and Dolan (2011) David Chen and William Dolan. 2011.  Collecting highly parallel data for paraphrase evaluation.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 190–200, Portland, Oregon, USA. Association for Computational Linguistics.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _ECCV_.
  * Cheng et al. (2021) Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. 2021.  Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss.  _arXiv preprint arXiv:2109.04290_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dietterich et al. (1997) Thomas G. Dietterich, Richard H. Lathrop, and TomÃ¡s Lozano-PÃ©rez. 1997.  Solving the multiple instance problem with axis-parallel rectangles.  _Artificial Intelligence_ , 89(1):31–71.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_.
  * Faghri et al. (2018) Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018.  Vse++: Improving visual-semantic embeddings with hard negatives.
  * Fan et al. (2019) Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. 2019.  Heterogeneous memory enhanced multimodal attention model for video question answering.  In _CVPR_.
  * Fang et al. (2021) Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. 2021.  Clip2video: Mastering video-text retrieval via image clip.  _arXiv preprint arXiv:2106.11097_.
  * Gabeur et al. (2020) Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020.  Multi-modal Transformer for Video Retrieval.  In _European Conference on Computer Vision (ECCV)_.
  * Gao et al. (2021) Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. 2021\.  Clip2tv: An empirical study on transformer-based methods for video-text retrieval.  _arXiv preprint arXiv:2111.05610_.
  * Ging et al. (2020) Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox. 2020.  Coot: Cooperative hierarchical transformer for video-text representation learning.  In _Advances on Neural Information Processing Systems (NeurIPS)_.
  * Gutmann and Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. 2010.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , volume 9 of _Proceedings of Machine Learning Research_ , pages 297–304, Chia Laguna Resort, Sardinia, Italy. PMLR.
  * Hendricks et al. (2018) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2018.  Localizing moments in video with temporal language.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Huang et al. (2020) Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. 2020\.  Location-aware graph convolutional networks for video question answering.  In _AAAI_.
  * Jiang et al. (2020) Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. 2020.  Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(07):11101–11108.
  * Kim et al. (2016) Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. 2016.  Hadamard product for low-rank bilinear pooling.  _arXiv preprint arXiv:1610.04325_.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5583–5594. PMLR.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kiros et al. (2014) Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014.  Unifying visual-semantic embeddings with multimodal neural language models.  _ArXiv_ , abs/1411.2539.
  * Le et al. (2020) Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2020.  Hierarchical conditional relation networks for video question answering.  _arXiv preprint arXiv:2002.10698_.
  * Lei et al. (2021) Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. 2021.  Less is more: Clipbert for video-and-language learningvia sparse sampling.  In _CVPR_.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
  * Li et al. (2021) Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  In _NeurIPS_.
  * Li et al. (2020a) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020a.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  In _EMNLP_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  In _Arxiv_.
  * Li et al. (2020b) Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020b.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  _ECCV 2020_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham. Springer International Publishing.
  * Liu et al. (2019) Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. 2019.  Use what you have: Video retrieval using representations from collaborative experts.  In _arXiv preprint arxiv:1907.13487_.
  * Luo et al. (2020) Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  _arXiv preprint arXiv:2002.06353_.
  * Luo et al. (2021) Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2021\.  CLIP4Clip: An empirical study of clip for end to end video clip retrieval.  _arXiv preprint arXiv:2104.08860_.
  * Miech et al. (2020) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020.  End-to-End Learning of Visual Representations from Uncurated Instructional Videos.  In _CVPR_.
  * Miech et al. (2018) Antoine Miech, Ivan Laptev, and Josef Sivic. 2018.  Learning a text-video embedding from incomplete and heterogeneous data.  _arXiv_ , abs/1804.02516.
  * Miech et al. (2019) Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  In _ICCV_.
  * Mithun et al. (2018) Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K. Roy-Chowdhury. 2018.  Learning joint embedding with multimodal cues for cross-modal video-text retrieval.  In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_ , ICMR ’18, page 19–27, New York, NY, USA. Association for Computing Machinery.
  * Patrick et al. (2021) Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F. Henriques, and Andrea Vedaldi. 2021.  Support-set bottlenecks for video-text representation learning.  In _International Conference on Learning Representations_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019.  Videobert: A joint model for video and language representation learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111, Hong Kong, China. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Venugopalan et al. (2015) Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. 2015.  Translating videos to natural language using deep recurrent neural networks.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1494–1504, Denver, Colorado. Association for Computational Linguistics.
  * Wang et al. (2016) Liwei Wang, Yin Li, and Svetlana Lazebnik. 2016.  Learning deep structure-preserving image-text embeddings.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Xu et al. (2017) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017.  Video question answering via gradually refined attention over appearance and motion.  In _ACM Multimedia_.
  * Xu et al. (2021) Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021.  VideoCLIP: Contrastive pre-training for zero-shot video-text understanding.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6787–6800, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).
  * Yang et al. (2021) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021\.  Just ask: Learning to answer questions from millions of narrated videos.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1686–1697.
  * Yu et al. (2018) Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018.  A joint sequence fusion model for video question answering and retrieval.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Zhang et al. (2018) Bowen Zhang, Hexiang Hu, and Fei Sha. 2018.  Cross-modal and hierarchical modeling of video and text.  In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII_ , pages 385–401.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  Vinvl: Making visual representations matter in vision-language models.  _CVPR 2021_.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced language representation with informative entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2020a) Zhu Zhang, Zhou Zhao, Zhijie Lin, jieming zhu, and Xiuqiang He. 2020a.  Counterfactual contrastive learning for weakly-supervised vision-language grounding.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18123–18134. Curran Associates, Inc.
  * Zhang et al. (2020b) Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. 2020b.  Object relational graph with teacher-recommended learning for video captioning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhou et al. (2018) Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018.  Towards automatic learning of procedures from web instructional videos.  In _AAAI Conference on Artificial Intelligence_.
  * Zhu and Yang (2020) Linchao Zhu and Yi Yang. 2020.  Actbert: Learning global-local video-text representations.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.