# Learning to Trust Your Feelings:
Leveraging Self-awareness in LLMs for Hallucination Mitigation

Yuxin Liang1 Work done in IDEA  Zhuoyang Song2 Hao Wang1 Jiaxing Zhang2
1X2Robot
2International Digital Economy Academy
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{songzhuoyang<EMAIL_ADDRESS>

###### Abstract

We evaluate the ability of Large Language Models (LLMs) to discern and express
their internal knowledge state, a key factor in countering factual
hallucination and ensuring reliable application of LLMs. We observe a robust
self-awareness of internal knowledge state in LLMs, evidenced by over 85%
accuracy in knowledge probing. However, LLMs often fail to express their
internal knowledge during generation, leading to factual hallucinations. We
develop an automated hallucination annotation tool, DreamCatcher, which merges
knowledge probing and consistency checking methods to rank factual preference
data. Using knowledge preference as reward, We propose a Reinforcement
Learning from Knowledge Feedback (RLKF) training framework, leveraging
reinforcement learning to enhance the factuality and honesty of LLMs. Our
experiments across multiple models show that RLKF training effectively
enhances the ability of models to utilize their internal knowledge state,
boosting performance in a variety of knowledge-based and honesty-related
tasks.

Learning to Trust Your Feelings:
Leveraging Self-awareness in LLMs for Hallucination Mitigation

##  1 Introduction

Figure 1: Internal knowledge state categorization of LLM, based on the
possession of corresponding internal knowledge and the capacity to express it
honestly.

Large Language Models (LLMs), including notable examples such as GPT-3 Brown
et al. (2020), LLaMA (Touvron et al. (2023a), Touvron et al. (2023b)), and
PaLM Chowdhery et al. (2023), have emerged as a transformative tool in diverse
fields due to their robust capabilities in various tasks. However, despite
this significant progress and success, an inherent challenge continues to
persist: their tendency to "hallucinate", i.e., generate content misaligned
with actual facts. This issue is particularly problematic in critical
applications, such as clinical or legal scenarios, where the generation of
reliable and accurate text is vital. Therefore, mitigating hallucinations in
LLMs is a crucial step toward enhancing their practical application scope and
improving the overall trust in these emerging technologies.

Hallucinations of LLMs can be categorized into three types Zhang et al.
(2023b): input conflict, context conflict, and factual conflict. This paper
focus on the issue of fact-conflicting hallucination, where LLM produces
fluent and seemingly plausible content, but conflicts with real-world facts,
pose risks of misleading users and compromise the models’ fact-based
reasoning.

Commonly used hallucination mitigation methods, such as retrieval augmentation
generation (RAG), address fact-conflict hallucination of LLM by bringing in
external knowledge, but at the cost of introducing a costly and complex
retrieval system. In this paper, we propose to mitigate the factual
hallucination problem from the perspective of enhancing the model’s
utilization of internal knowledge.

Previous works (Kadavath et al. (2022), Azaria and Mitchell (2023), Agrawal et
al. (2023)) have shown that LLMs have the capability to discern the validity
of factual statements, supported further by Kadavath et al. (2022) suggesting
these models’ capacity to assess their ability in responding to specific
questions. Nevertheless, the universality and extent of models’ self-awareness
of their knowledge remains an open question. In light of this, we conducted
exploratory experiments to probe the knowledge state of various models across
different scales, employing linear probes to ascertain the accuracy of models’
judgments regarding their internal knowledge states. The results revealed that
all models under analysis demonstrated proficient accuracy in recognizing
whether they have the internal knowledge about certain facts.

However, during generation, such accurate judgments do not translate into
honest output; instead, in the absence of specific internal knowledge, models
often manifest a tendency towards hallucinations. Therefore, to mitigate
factual hallucinations, it is crucial that models leverage their self-assessed
judgments concerning their knowledge status.

We propose a training framework named reinforcement learning from knowledge
feedback (RLKF) to improve the factuality and honesty of LLM with
reinforcement learning using factual preferences as reward. Through the
hallucination annotation method DreamCatcher – a blend of knowledge probing
and consistency-based judgments – we rank the knowledge-based Question-
Answering (QA) data adhering to a preference hierarchy delineated as:
$factuality>uncertainty>hallucination$. This factual preference data is then
utilised to train the reward model which is deployed to optimize the Large
Language Model via Proximal Policy Optimisation (PPO) algorithm.

The primary contributions of this paper are articulated as follows:

  1. 1.

We carried out extensive experiments on different models’ capacity to discern
their own internal knowledge. The results indicate that LLMs are highly adept
at discerning their internal knowledge, with an impressive accuracy over 85%
in most cases with a limited amount of data.

  2. 2.

We develop DreamCatcher111https://github.com/liangyuxin42/dreamcatcher, an
automatic hallucination detection tool for scoring the degree of hallucination
in LLM generations. DreamCatcher integrates knowledge probing methods and
consistency judgments, achieving 81% agreement with human annotator.

  3. 3.

We introduce the Reinforcement Learning from Knowledge Feedback (RLKF)
training framework to optimize LLM against the factual preference. The
experiment results on multiple knowledge and reasoning tasks indicate that
RLKF not only enhances the honesty and factuality of LLMs but also improves
their general capabilities.

##  2 Problem Setup

(a) Llama2-chat-13B

(b) Qwen-chat-14B

(c) Ziya-reader-13B

Figure 2: Accuracy of knowledge state probing across different models with
different internal representations. The light-colored area in the figure shows
the range of accuracy for ten repetitions of the experiment, and the solid
line shows the mean accuracy. More results shown in A.2

Hallucination, in the context of Large Language Models, refers to a set of
inconsistencies during model generation. The central focus of this paper is
exploring the fact-conflict hallucination which is defined as the
inconsistency between the generated content of the model and the established
facts. Although the definition provides a description of the generation
results, the causes underlying this phenomenon are multifaceted.

In general, LLMs encode factual knowledge into parameters during training and
utilize this internal knowledge for generation during inference. However, LLMs
do not always honestly express the knowledge in its parameters, which is one
of the major causes of fact-conflict hallucination.

For a given question that requires factual knowledge, the model output can be
classified into one of four states, depending on the model’s internal
knowledge and its honesty. These states are illustrated in Figure 1:

State 1: The model has relevant internal knowledge and expresses it
faithfully.

State 2: Despite having the relevant internal knowledge, the model fails to
express it honestly. This discrepancy could be due to various factors such as
the decoding strategy (Lee et al., 2022; Chuang et al., 2023), hallucination
snowballing (Zhang et al., 2023a), or misalignment issues (Schulman, 2023).

State 3: The model lacks the necessary internal knowledge but honestly
indicates unawareness.

State 4: The model lacks the necessary internal knowledge and instead produces
a hallucinated response.

Outputs in State 2 and State 4 are both considered forms of hallucination,
despite the differing conditions of internal knowledge.

In the upper section of Figure 1, the model’s outputs are devoid of
hallucinations, honestly mirroring its internal knowledge reservoir. Here,
State 1 stands out as the most desirable state, where the model both possesses
and faithfully produces the relevant knowledge.

Many efforts have been deployed to transition model toward state 1.

Retrieval-augmented generation (RAG) attempts to bypass the lack of internal
knowledge by providing knowledge via context, thereby enabling the model to
transition from State 3/4 to State 1. On another front, certain strategies,
like those of Li et al. (2023b) and Chuang et al. (2023), seek to move the
model from State 2 to State 1 by intervening the model’s internal
representation or the decoding process during inference. While these methods
improve the model’s capacity to express existing internal knowledge, they
disregard scenarios where the model lacks relevant internal knowledge. Also,
interference at inference time can potentially lead to unpredictable effects
on other types of tasks.

Without the introduction of external knowledge, the mitigation of the model’s
fact-conflict hallucination correspond to an upward movement of the state in
Figure 1. In essence, this symbolizes the enhancement of the model’s capacity
to accurately represent its internal knowledge state. A critical question,
then, is how to discern the internal knowledge state of a model?

##  3 Knowledge State Probing

This section delves into the complexities of discerning a model’s internal
knowledge state. It comprises two perspectives. The first, an external
perspective, discuss how to determine if a model possesses specific knowledge
based on the model generations; The second perspective, an internal view,
questions if it is possible to determine whether a model possesses specific
knowledge by its internal activation.

For the following pilot experiments, we have selected three families of models
with different sizes: Llama2-chat(Touvron et al., 2023b) (13B and 7B); Qwen-
chat(Bai et al., 2023) (14B and 7B); Ziya-reader(Junqing et al., 2023) (13B).

As for data, We randomly select passages from Chinese and English Wikipedia
and instruct GPT3.5 to generate a knowledge-related question-answer pair. The
answer generated by GPT3.5 based on the original Wikipedia is considered as
the correct answer. We refer to the QA pairs obtained by this method as wiki-
QA in this paper. Examples of instructions and corresponding output are shown
in Appendix A.1.

###  3.1 External perspective

Determining whether a model has specific knowledge through its generation is a
straightforward way. But it is challenging to accurately assess the model’s
knowledge state through a singular generation result, due to the uncertainty
of generation caused by sampling Lee et al. (2022) and different generation
tendencies Chuang et al. (2023). Multiple generation results can more
faithfully reflect the knowledge state of the model.

In the presence of a correct answer, the consistency of the model’s multiple
generation with the correct answer is a reliable method for assessing
knowledge state. The consistency of model generation with the correct answer
can be computed using methods such as unigram overlap and cosine similarity of
text representation.

However, the correct answer is hard to obtain in many scenarios, in which case
self-consistency becomes a critical tool for assessing the validity of the
generation. As evidenced by multiple research (Manakul et al. (2023), Agrawal
et al. (2023), Hase et al. (2023), Elaraby et al. (2023)), there is a general
conclusion that higher consistency across multiple generations is often
indicative of validity of the generation. Intuitively, if the model has the
corresponding knowledge, multiple generation are likely to contain consistent
facts, resulting in higher consistency. Whereas, the contents of the
hallucinations often varies, leading to lower self-consistency. We evaluate
the self-consistency of a certain generation by the average of the cosine
similarity representations among other generated answers.

###  3.2 Internal perspective

Previous work (Azaria and Mitchell (2023), Kadavath et al. (2022), Li et al.
(2023b)) prove that LLMs can discern the factual accuracy of certain
statements, even when the false statements are self-generated. This supports
the existence of state 2 in Figure 1 where the model has the corresponding
knowledge but generates incorrect outputs. But are LLMs capable of discerning
its own state of knowledge? The question can be rephrased as follows: for a
given knowledge-related question, can a model discern its capability to output
the correct answer before the actual generation of an answer? The following
linear probing experiments on multiple models implies that the answer is yes.

We sample questions from the wiki-QA data, and LLM generates $k=5$ answers for
each question separately. We use the consistency method described earlier to
pre-label the questions. The sum of these normalized consistency scores
computed to derive the final score.

To categorize the questions, straightforward thresholds are utilized. The
upper threshold is set at the 65th percentile score, and the lower at the 35th
percentile score. Under this setup, responses with scores exceeding the upper
threshold are labeled as correct, while those falling below the lower
threshold are labeled as incorrect. If all of the k generated responses
related to a specific question are deemed correct, the model is presumed to
possess the relevant internal knowledge, and thus the question is labeled as
’Known’. Conversely, if all k responses are incorrect, the model is considered
to lack the necessary internal knowledge, and hence the question is labeled as
’Unknown’.

A single linear layer classifier (probe) is trained on the internal
representation corresponding to the last token of each question. Its task is
to predict the corresponding Known/Unknown label.

For our experiments, we select three types of internal representations:

The attention output, which refers to the output of the dot product attention
and before the attention linear layer in the decoder layer. This setup aligns
with the probe’s positioning within Li et al. (2023b); The MLP output, i.e.,
the feed-forward layer’s output within the decoder layer, occurring prior to
residual linkage; The hidden states, defined as each decoder layer’s output.

The results of the internal knowledge probe experiment are shown in Figure 2,
which presents the accuracy of the trained probes across different models with
different internal representation and at different layers.

Comparative analysis of the experimental results across models of varying
sizes yields consistent observations:

1\. The linear probes of the internal state accurately predict the knowledge
representation of the model. The probes’ maximum accuracy surpasses 0.85 in
most setups. This suggests that information about whether the model has the
corresponding knowledge is linearly encoded in the internal representation of
the model with high accuracy.

2\. The accuracy of the probes increases rapidly in the early to middle layer,
indicating that the model needs some layers of computation before it can
determine its own knowledge states.

3\. Hidden state probes exhibit the highest accuracy in discerning the
knowledge state of the model, sustaining high accuracy from the middle layer
to the output layer, which opens up the possibility of utilizing internal
knowledge state when generating responses.

###  3.3 DreamCatcher

We integrated the above methods of knowledge state probing and consistency
judgments to develop an automatic hallucination labeling tool, DreamCatcher.

We start by collect the LLMś generation for each question in the question set,
in our case, the wiki-QA dataset. This process features two modes: normal
generation and uncertainty generation. Normal generation is when the prompt
contains only the question and model generates k responses, while uncertainty
generation refers to where the prompt contains a request for the model to
output answers that show uncertainty or lack of knowledge.

Subsequently, we assess the degree of hallucination of the generated responses
using multiple scorers using the methods described above. Concretely, we
compute the following scores:

$\begin{array}[]{ll}s_{s2g}&=\text{avg}_{ij}(\text{cos}(\textbf{r}_{G_{i}},\textbf{r}_{G_{j}}))\\\
s_{p}&=\text{probe}(\textbf{r}_{Q})\\\
s_{o2a}&=\text{count}(token_{overlap})/\text{count}(token_{A})\\\
s_{s2a}&=\text{cos}(\textbf{r}_{G},\textbf{r}_{A})\\\ \end{array}$

where $Q$ denotes the question, $A$ the correct answer, $G$ the generation and
r the embedding representation of text.

$s_{p}$ (Probe Score): rates the questions by utilizing the probes trained in
Section 3.2, which are intended to discern the model’s knowledge state for the
corresponding questions.

$s_{o2a}$ (Overlap with Answer Score): calculates the ratio of token overlap
between the generated output and the correct answer ($A$).

$s_{s2a}$ (Similarity to Answer Score): computes the cosine similarity between
the embedding of the generated response ($G$) and the correct answer ($A$),
using the bge-large model for text embedding.

The scores are normalized and summed to provide an overall factuality score
for each generation. The generations are then classified as "correct" or
"incorrect" based on whether their total score is above or below the median
score, respectively. Questions are categorized as "Known", "Unknown", or
"Mixed" based on whether the responses are consistently correct, incorrect, or
a combination of correct and incorrect across multiple generations, with
"Mixed" being a less frequent occurrence.

The categories correspond to three ranking hierarchies as shown in Figure 3:
Known (corresponding to state 1 in Fig. 1): factual > uncertainty; Mixed
(state 2): factual > uncertainty > hallucination; Unknown (state 4):
uncertainty > hallucination. Here, "factual" refers to the generation with the
highest factuality score, while "hallucination" denotes the generation with
the lowest score.

We randomly sampled 200 entries, half Chinese and half English, from the
DreamCatcher labeled data. Then the human annotator annotate the same data,
without access to the labels of DreamCatcher. The consistency between
DreamCatcher and human annotator is shown in Table 1, with an overall accuracy
of 81%.

Language | Accuracy | Precision | Recall
---|---|---|---
All | 81% | 77% | 86%
Chinese | 77% | 79% | 76%
English | 86% | 76% | 98%

Table 1: The consistency between DreamCatcher and human annotator. For
precision and recall, we treat "correct" as a positive label and "incorrect"
as negative.

##  4 Method

Figure 3: RLKF training

From the above knowledge-probing experiments, we discover that LLMs are
capable of evaluating their own knowledge states in response to specific
knowledge-based questions. This implies that LLMs demonstrate a self-awareness
of their knowledge state, which does not consistently translate into their
generational output.

Frequently, when faced with questions outside of internal knowledge, LLMs
tends to generate hallucinations. Additionally, even with questions within
internal knowledge, LLMs may potentially generate incorrect responses due to
other influences. One possible explanation could be that LLMs did not learn to
generate with respect to the internal knowledge state during model training.
Instead, the fine-tuning process often requires the model to generate
seemingly reasonable answers to all factual questions.

We therefore emphasize on enhancing the model’s utilization of internal
knowledge state so that the model can choose to rely on internal knowledge to
answer or honestly express its lack of relevant knowledge.222This intuition
could also be used for efficient RAG, enabling direct responses when the LLM
possesses relevant internal knowledge, while relying on the retrieval tool in
case of a knowledge gap.

Consequently, we propose the RLKF (Reinforce Learning from Knowledge Feedback)
training framework. This introduces model knowledge state assessments into the
reinforcement learning feedback mechanism, enhancing model honesty and
factuality. The RLKF training process shares similarities with the standard
RLHF (Reinforce Learning from Human Feedback), and can integrate smoothly with
the existing RLHF framework, but reduces data collection costs by substituting
human labeling with automatic knowledge labeling.

The RLKF training framework consists of the following components, as shown in
Figure 3.

Knowledge state annotation: We annotate factual preference data using the
Dreamcatcher tool.

Knowledge Feedback Modeling: Having obtained the factual preference data, we
train the reward model following Ouyang et al., 2022. The language modelling
head in reward model is replaced with a linear layer to produce a scalar
output, corresponding to the reward of the generated response. In line with
Köpf et al. (2023), an additional regularization parameter is introduced to
prevents the predicted values from diverging too much.

By initiating the PPO Policy training and the reward model training from the
same model, we can ensure that the Reward model can leverage the same internal
knowledge.

PPO Optimizing: Based on our factual reward model, we optimize the policy,
i.e., the initial generative model, using the PPO algorithm once again
following Ouyang et al., 2022. To improve the efficiency of model exploration
towards honesty, we use guidance technique in reinforcement learning.
Concretely, we concatenate the first few tokens of the preferred responses to
the input prompts in a portion of the training data. The added tokens do not
participate in the loss calculation, but can guide the model to generate
desired responses, thus improving learning efficiency.

The core of the training framework is to establish the factual preference
reward mechanism. The reinforcement learning algorithms in the RLKF framework
can also be replaced by other optimization algorithms such as DPO Rafailov et
al. (2023), reject sampling, etc. We choose PPO to be consistent with the
common practice in RLHF training.

##  5 Experiments

In the following experiments, We chose three different models of varying
sizes: llama2-chat (13B and 7B); Qwen-chat (14B and 7B); and Ziya-reader
(13B), which is consistent with the choice of models for the knowledge-probing
experiments detailed in Section 3.

Model | Known | Unknown | Mixed
---|---|---|---
Qwen-chat-14B | 82.7% | 87.1% | 77.8%
Qwen-chat-7B | 65.7% | 81.6% | 61.1%
Llama2-chat-13B | 85.4% | 85.4% | 60.0%
Llama2-chat-7B | 78.9% | 89.2% | 57.6%
Ziya-reader-13B | 93.5% | 82.4% | 64.5%

Table 2: Accuracy of trained reward model for each knowledge state category.

Models | MMLU | WinoGrande | ARC | BBH | GSM8K | MATH | C-Eval | CMMLU | Avg
---|---|---|---|---|---|---|---|---|---
Qwen-chat-14B | before | 64.2% | 53.8% | 76.5% | 34.5% | 47.3% | 18.9% | 65.0% | 64.1% | 53.0%
after | 64.5% | 59.1% | 87.2% | 37.3% | 49.9% | 20.3% | 64.6% | 66.4% | 56.2%
Qwen-chat-7B | before | 54.2% | 49.6% | 63.1% | 28.8% | 50.0% | 12.6% | 57.8% | 58.1% | 46.8%
after | 55.3% | 52.2% | 75.4% | 28.1% | 50.9% | 12.5% | 57.5% | 56.0% | 48.5%
Llama2-chat-13B | before | 52.3% | 51.9% | 72.4% | 21.7% | 35.2% | 3.2% | 34.6% | 34.5% | 38.2%
after | 52.8% | 54.3% | 72.1% | 23.4% | 35.6% | 3.1% | 34.3% | 34.6% | 38.8%
Llama2-chat-7B | before | 45.9% | 51.5% | 59.2% | 23.3% | 25.9% | 1.6% | 32.1% | 31.6% | 33.9%
after | 46.2% | 52.4% | 61.1% | 24.4% | 23.7% | 2.0% | 34.0% | 32.1% | 34.5%
Ziya-reader-13B | before | 49.5% | 50.8% | 64.7% | 44.7% | 29.3% | 4.3% | 44.7% | 46.1% | 41.7%
after | 50.3% | 51.9% | 67.9% | 42.6% | 33.2% | 3.8% | 42.6% | 45.1% | 42.2%

Table 3: Evaluation of RLKF-trained models on various knowledge and reasoning
related tasks: MMLU Hendrycks et al. (2020), WinoGrande Sakaguchi et al.
(2021), ARC Chollet (2019), BBH Suzgun et al. (2022), GSM8K Cobbe et al.
(2021), MATH Hendrycks et al. (2021), C-Eval Huang et al. (2023), CMMLULi et
al. (2023a). Tasks are evaluated by the open-source evaluation tool TLEM
SUSTech (2023), employing a 0-shot setting with greedy generation.

###  5.1 Data collection

We used the wiki-QA data collection method same as in Section 3, obtaining
about 7,000 QA pairs each for Chinese and English. To add variety to the
questions, we have also modified the prompt to include multiple choice
question types. Since our approach relies on the internal knowledge of the
models and the boundaries of the internal knowledge are different for each
model, we need to perform automatic annotation for each model individually.
The generated responses are labeled using Dreamcatcher to obtain factual
preference data. The statistics of the factual preference data are shown in
Table 7.

###  5.2 RLKF Training

We train the reward model using the factual preference data in Table 7. To
maintain the generalization of the RM, we include same amount of general
purpose data as the wiki-QA data in the training. Accuracy of the trained RM
on factual preference data testset are shown in Table 2. Interestingly, the
reward model is able to quickly achieve high accuracy for both known/unknown
categories during training, suggesting that reward model may utilize the
internal knowledge state of the initial model to determine whether the
uncertainty response should be preferred.

Using the trained reward model, the RL process optimizes policy model using
the PPO algorithm, where policy model is initialized from the same base model
as reward model. The detailed training settings and hyper-parameters are
described in A.4.

We conduct an evaluation of the trained model, focusing on its factuality and
truthfulness. A comparative analysis of the models is performed between pre-
and post- RLKF training on various tasks related to knowledge and reasoning as
shown in Table 3. The RLKF-trained models demonstrate improvements on the
majority of the benchmarks. While RLHF typically results in a reduction of
benchmark performance, termed as ’alignment tax’ (Askell et al., 2021), RLKF
avoids this decline specifically on knowledge-related tasks, and even lead to
improvements. Note that our training methodology does not employ any benchmark
data, and the overall volume of training data utilized is small.

Regarding the truthfulness of trained models, we evaluated their performance
using the widely recognized TruthfulQA task. Notably, all models, with the
exception of llama2-chat-13B, show increase in honesty, as shown in Table 4.

Models | TruthfulQA
---|---
Qwen-chat-14B | before | 43.7%
after | 49.1%
Qwen-chat-7B | before | 49.1%
after | 50.3%
Llama2-chat-13B | before | 21.5%
after | 20.9%
Llama2-chat-7B | before | 27.5%
after | 28.3%
Ziya-reader-13B | before | 34.8%
after | 37.9%
Table 4: Evaluation of RLKF-trained models on TruthfulQA, again using TLEM
SUSTech (2023), employing a 0-shot setting with greedy generation.

##  6 Related Work

Hallucination in large language models (LLMs) has been the focal point of
research, spanning its causes, detection, and mitigation. Our work relates to
all three aspects.

Causes of hallucination: Studies have linked LLM hallucination to various
causes. McKenna et al. (2023) ascribes it to memorization of training data,
indicating a direct correlation between the training data and the resultant
hallucination. Other works such as Schulman (2023) pinpoint improper model
fine-tuning as contributive, and Perez et al. (2022) argues that RLHF induce
model "sycophancy" which in turn degrades honesty.

Other studies link hallucinations to the generation process. For example, Lee
et al. (2022) suggests that sampling-induced randomness could be responsible.
One perspective provided by Chuang et al. (2023) proposes that "lower-level"
prior layer information might overshadow factual information from subsequent
layers. Furthermore, some works relate hallucinations to the overconfidence of
LLMs Ren et al. (2023).

Hallucination detecting: In terms of detecting hallucination, the consistency
of multiple generations has been recognized as an effective indicator.
SelfCheckGPT Manakul et al. (2023) capitalizes on the consistent nature of
internal knowledge-based generations compared to the variable nature of
hallucination, leading to the proposal of several consistency checks to
identify hallucinations. The idea is echoed by Agrawal et al. (2023), who
suggest evaluating the generation consistency of generated references to spot
hallucination. Similarly, Elaraby et al. (2023) proposes a metric involving
the calculation of sentence-level entailment between response pairs as a
measure of hallucination.

Employing large language models (LLMs) to recognize their own hallucinations
has been suggested in Saunders et al. (2022), suggesting that discrimination
is more accurate than generation for LLMs (G-D gap). This notion is furthered
by Kadavath et al. (2022) and Agrawal et al. (2023) by directly prompting LLMs
to assess the validity of their own output.

Another approach examines the factualness of statements by analyzing the
model’s internal representation. Studies Li et al. (2023b) and Burns et al.
(2022) identify a "factualness" direction in the model’s internal
representation, with Li et al. (2023b) showcasing a high accuracy attention
head through linear probing, and Burns et al. (2022) locating factualness
direction through consistency of facts. Additionally, Kadavath et al. (2022)
trains the model to predict the probability that it knows. Base on these
works, we shifts focus onto the model’s self-evaluation of knowledge state.

Hallucination mitigation: The common approach of hallucination mitigation
involves enhancing the model with additional information. Elaraby et al.
(2023) propose the use of larger models to provide additional information when
hallucinations is detected.

Some research efforts focus on the optimization of decoding strategies to
address hallucinations. Chuang et al. (2023) suggests that contrastive
decoding can augment the factualness of model generation. Others leverage the
direction of factualness in the model representation space; Li et al. (2023b)
enhances factualness by adjusting the output of attention heads along the
direction of factualness during inference. Our work seeks to optimizes the
utilization of the model’s internal knowledge state, in line with the
direction proposed by Schulman (2023) leveraging reinforcement learning to
tackle hallucinations.

##  7 Conclusion

In our research, we have thoroughly explored the capability of large language
models (LLMs) to discern and express their internal knowledge, a key factor in
mitigating factual hallucinations and ensuring reliable applications. Our
research, manifested through a series of knowledge probing experiments,
identifies the model’s self-awareness of its knowledge state. We released the
open-source tool DreamCatcher which scores and labels the degree of
hallucination in the LLM’s response to knowledge-oriented question and rank
responses based on their factuality.

We further validated our findings through the introduction of a training
framework: Reinforcement Learning from Knowledge Feedback (RLKF). Utilizing
DreamCatcher to annotate factual preference data, we trained a reward model
and leveraging reinforcement learning to enhances LLM’s factuality and
truthfulness. Our results indicate RLKF’s effectiveness in improving the
model’s utilization of its internal knowledge state, enhancing its performance
in various knowledge and honesty related tasks. We posit that RLKF is a
promising solution to address LLM’s hallucination issues and, combined with
RLHF, offers significant potential for enhancing the model’s overall
capabilities.

## References

  * Agrawal et al. (2023) Ayush Agrawal, Lester Mackey, and Adam Tauman Kalai. 2023.  Do language models know when they’re hallucinating references?  _arXiv preprint arXiv:2305.18248_.
  * Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021.  A general language assistant as a laboratory for alignment.  _arXiv preprint arXiv:2112.00861_.
  * Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell. 2023.  The internal state of an llm knows when its lying.  _arXiv preprint arXiv:2304.13734_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Burns et al. (2022) Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022.  Discovering latent knowledge in language models without supervision.  _arXiv preprint arXiv:2212.03827_.
  * Chollet (2019) François Chollet. 2019.  On the measure of intelligence.  _arXiv preprint arXiv:1911.01547_.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113.
  * Chuang et al. (2023) Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023.  Dola: Decoding by contrasting layers improves factuality in large language models.  _arXiv preprint arXiv:2309.03883_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Elaraby et al. (2023) Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu. 2023.  Halo: Estimation and reduction of hallucinations in open-source weak large language models.  _arXiv preprint arXiv:2308.11764_.
  * Hase et al. (2023) Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2023.  Methods for measuring, updating, and visualizing factual beliefs in language models.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2706–2723.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_.
  * Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _arXiv preprint arXiv:2305.08322_.
  * Junqing et al. (2023) He Junqing, Pan Kunhao, Dong Xiaoqun, Song Zhuoyang, Liu Yibo, Liang Yuxin, Wang Hao, Sun Qianguo, Zhang Songxin, Xie Zejian, et al. 2023.  Never lost in the middle: Improving large language models via attention strengthening question answering.  _arXiv preprint arXiv:2311.09198_.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023.  Openassistant conversations–democratizing large language model alignment.  _arXiv preprint arXiv:2304.07327_.
  * Lee et al. (2022) Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022.  Factuality enhanced language models for open-ended text generation.  _Advances in Neural Information Processing Systems_ , 35:34586–34599.
  * Li et al. (2023a) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023a.  Cmmlu: Measuring massive multitask language understanding in chinese.  _arXiv preprint arXiv:2306.09212_.
  * Li et al. (2023b) Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023b.  Inference-time intervention: Eliciting truthful answers from a language model.  _arXiv preprint arXiv:2306.03341_.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * McKenna et al. (2023) Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. 2023.  Sources of hallucination by large language models on inference tasks.  _arXiv preprint arXiv:2305.14552_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Perez et al. (2022) Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2022.  Discovering language model behaviors with model-written evaluations.  _arXiv preprint arXiv:2212.09251_.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_.
  * Ren et al. (2023) Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023.  Investigating the factual knowledge boundary of large language models with retrieval augmentation.  _arXiv preprint arXiv:2307.11019_.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022.  Self-critiquing models for assisting human evaluators.  _arXiv preprint arXiv:2206.05802_.
  * Schulman (2023) John Schulman. 2023.  Reinforcement learning from human feedback: Progress and challenges, 2023.
  * SUSTech (2023) SUSTech. 2023.  Introducing tlem: The future of language model evaluation.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _arXiv preprint arXiv:2210.09261_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Zhang et al. (2023a) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023a.  How language model hallucinations can snowball.  _arXiv preprint arXiv:2305.13534_.
  * Zhang et al. (2023b) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023b.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_.

##  Appendix A Appendix

###  A.1 Example of wiki-QA Instruction

Instruction template: Based on the following Wikipedia article snippet, ask a
knowledge-based question and provide a corresponding answer. Article snippet:
{Wikipedia passage} Requirements: 1\. there is a unique correct answer to the
question, and the answer can be found in the given article fragment. 2\. the
question can be answered independently of the article fragment, i.e. the
answer to the question cannot depend on contextual information, e.g. a
question about a character in a literature needs to specify the work to which
the character belongs, and a question such as "What is the article about?"
cannot be asked. 3\. Provide the question, answer, and category (e.g.,
literature, physics, etc.) at the same time, and reply in the following
format: {"question":question,"answer":answer,"type":category}. If you are
unable to ask a question that meets the above requirements, you can simply
reply "Unable to ask". Reply: Wikipedia passage: House Arrest (1996 film)
House Arrest is a 1996 American comedy film directed by Harry Winer, written
by Michael Hitchcock, and starring Jamie Lee Curtis, Kevin Pollak, Jennifer
Tilly, Christopher McDonald, Wallace Shawn, and Ray Walston with supporting
roles done by Kyle Howard, Amy Sakasitz, Mooky Arizona, Russel Harper, and an
up-and-coming Jennifer Love Hewitt. It tells the story of two children who
trap their parents in their basement upon their plans for a separation as the
other children they know get involved by trapping their respective problem
parents as well. The film was released on August 14, 1996 and went on to gross
just over $7 million at the box office. The film was panned by critics. The
film was shot at various locations in the U.S. states of California and Ohio.
Monrovia, California was the location for several exterior house scenes while
most interior shots were done at the CBS/Radford lot in Studio City,
California. The story was set in Defiance, Ohio, although another town,
Chagrin Falls, Ohio, actually doubled for it. GPT3.5 response:
{"question":"Who directed the film House Arrest?","answer":"Harry
Winer","type":"film"}
---

Table 5: Example of instruction and corresponding GPT3.5 output of English
wiki-QA.

Instruction template:
根据下面的维基百科文章片段，提出一个简短的知识型问题并给出对应回答，要求这个问题存在唯一正确答案，并且答案可以在给出的文章片段中找到。 文章片段：
{Wikipedia passage}
问题需要在脱离文章片段的情况下仍能够被回答，例如针对文学作品中人物提问需要指明所属的作品，以免引起歧义。问题的回答不能依赖于上下文的信息，不能提出类似“这篇文章的内容是什么？”的问题。
同时给出问题，回答和问题分类（比如文学类或物理类等），按如下格式回复：{"question":问题,"answer":回答,"type":分类}。如果无法提出满足上述要求的问题，可以直接回复“无法提问”。
回复： Wikipedia passage: M25 M25，也称为IC 4725，是一个由恒星组成，在南天人马座的疏散星团。Philippe Loys
de
Chéseaux在1745年对这个星团进行了第一次有记录的观测，查尔斯·梅西耶1764年将它收录进他的星云天体清单[6]。这个星团位于模糊的特征附近，因此有一条暗带通过中心附近[3]。
M25距离地球大约2,000光年，年龄约为6,760万岁[2]。这个星团在空间的维度大约是13光年，估计质量是1,937
M☉，其中大约24%是星际物质[4]。星团成员中的人马座U是一颗分类为造父变星的变星[7]，还有两颗红巨星，且其中一颗是联星系统[8]。 GPT3.5
response: {"question":"M25是位于哪个星座的疏散星团？","answer":"南天人马座","type":"天文学"}
---

Table 6: Example of instruction and corresponding GPT3.5 output of Chinese
wiki-QA.

###  A.2 More probing results

(a) Llama2-chat-7B

(b) Qwen-chat-7B

Figure 4: Accuracy of knowledge state probing in 7B models. The light-colored
area in the figure shows the range of accuracy for ten repetitions of the
experiment, and the solid line shows the mean accuracy.

###  A.3 Statistics of factual preference data

Model | Total | Known | Unknown | Mixed
---|---|---|---|---
Qwen-chat-14B | 12799 | 49% | 43% | 8%
Qwen-chat-7B | 7201 | 52% | 40% | 8%
Llama2-chat-13B | 6600 | 48% | 44% | 8%
Llama2-chat-7B | 6680 | 45% | 45% | 10%
Ziya-reader-13B | 12558 | 49% | 41% | 10%

Table 7: Statistics of factual preference data and percentage of each
knowledge state category used for reward modeling. The Llama2 models use
English-only wiki-QA data, Qwen-chat-7B uses Chinese-only data, and Qwen-
chat-14B and Ziya-reader-13B use a mixture of English and Chinese data.

###  A.4 RLKF Training details

We use the AdamW optimizer, with $\beta_{1}=0.9$, $\beta_{2}=0.99$, $eps$ =
$1e-5$ for all models. The learning rate for reward model training is $5e-6$
with 1% warmup and linear decay scheduler. The batch size is 16 for 13/14B
models and 64 for 7B models. We train the reward model for 1 epoch. For PPO
training, we use learning rate of $1e-6$ with cosine scheduler. The batch size
is 32 for 13/14B models and 64 for 7B models. We set the KL penalty to 0 for
all models.

###  A.5 More Observation

We observe that, some of the responses to the unknown questions are indicating
uncertainty in RLHF-trained models, but there is also a significant percentage
of responses that are hallucinations. This indicates an increase in model
honesty achieved through RLHF, but there is still room for improvement.


# An Empirical Study of Frame Selection for Text-to-Video Retrieval

Mengxia Wu1, Min Cao1, Yang Bai1, Ziyin Zeng1, Chen Chen2,
Liqiang Nie3, Min Zhang1
1 Soochow University, 2 Institute of Automation, Chinese Academy of Sciences,
3 Harbin Institute of Technology, Shenzhen
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author

###### Abstract

Text-to-video retrieval (TVR) aims to find the most relevant video in a large
video gallery given a query text. The intricate and abundant context of the
video challenges the performance and efficiency of TVR. To handle the
serialized video contexts, existing methods typically select a subset of
frames within a video to represent the video content for TVR. How to select
the most representative frames is a crucial issue, whereby the selected frames
are required to not only retain the semantic information of the video but also
promote retrieval efficiency by excluding temporally redundant frames. In this
paper, we make the first empirical study of frame selection for TVR. We
systemically classify existing frame selection methods into text-free and
text-guided ones, under which we detailedly analyze six different frame
selections in terms of effectiveness and efficiency. Among them, two frame
selections are first developed in this paper. According to the comprehensive
analysis on multiple TVR benchmarks, we empirically conclude that the TVR with
proper frame selections can significantly improve the retrieval efficiency
without sacrificing the retrieval performance.

##  1 Introduction

Text-to-video retrieval (TVR) aims to search for the most relevant video given
a query text, involving intra-modal understanding and inter-modal alignment.
Videos are composed of a sequence of frames, each of which can be considered
as the primary processing unit for video understanding and alignment with the
text. However, due to the constraints of computing resources, it is
impractical to process all frames of one video for TVR. Therefore, existing
methods Liu et al. (2019); Gabeur et al. (2020); Lei et al. (2021); Luo et al.
(2022) select a subset of frames from the original videos to perform TVR.
Generally speaking, selecting fewer frames can benefit higher computational
efficiency, while leading to the loss of visual details and thus impairing
retrieval performance. Considering both effectiveness and efficiency, it is
imperative to adopt an optimal frame selection method for TVR.

Current frame selection methods can be divided into text-free frame selection
and text-guided frame selection. The text-free frame selection only explores
the video contents themselves while ignoring their relevancy to the texts in
selecting frames. Specifically, the conventional techniques Chang et al.
(1999); Divakaran et al. (2002); Chang (2003) for frame selection primarily
rely on the color features of frames. However, these approaches fall short of
comprehending the semantic content of videos and are not readily applicable to
end-to-end training in various video-related tasks. For simplicity, existing
works Lei et al. (2021); Luo et al. (2022); Li et al. (2022a) typically employ
uniform frame selection (Uni) or random frame selection (Rand) to eliminate
the temporally redundant frames that contribute little to the retrieval. In
contrast, the text-guided frame selection is aimed at utilizing textual
information to filter the inessential frames. We partition the text-guided
frame selection into two categories: non-interactive frame selection (N-InT)
Gorti et al. (2022); Han et al. (2022) and interactive frame selection (InT)
Buch et al. (2022); Yang et al. (2023); Wang et al. (2022); Lin et al. (2022).
The former directly calculates the similarities between the representations of
frames and the text from two unimodal encoders without interaction, and then
selects the frames with high similarities as the representatives of the video.
The latter constructs a cross-modal interactive module with extra parameters
to assess the relevancy between frames and the text.

Although persistent efforts have been made in frame selection, there still
exist three main problems. 1) The critical clues to the retrieval in a video
typically exhibit a non-uniform distribution. The text-free frame selection
with the Uni/Rand does not follow this distribution, which may filter out the
key frames that carry the critical clues and eventually hamper the performance
of TVR. And the text-guided frame selection circumvents this problem with the
aid of text information, but at the expense of simpleness. Nevertheless, both
of them have not considered the distribution of the video content itself. 2)
The videos usually contain low-quality frames that are out-of-interest and
less meaningful to TVR, leading to increased noise and decreased efficiency.
These frames could caused by transitions, out-of-focus shots, inappropriate
angles, etc. Yet, such frames are overlooked by the current text-free frame
selection methods. 3) Currently, there is a noticeable absence of a
comprehensive comparison and analysis of frame selection methods that take
into account the aforementioned two factors. As a result, the optimal frame
selection method for TVR still remains elusive, with no definitive answer yet.

To this end, we extend the current frame selection methods and conduct a
thorough empirical study of the frame selection methods to explore the optimal
one in this paper. Specifically, beyond the existing frame selections, we
first develop a redundancy-aware frame selection (Redun-A) method for the
first problem and a low-quality-aware frame selection method (LQ-A) for the
second problem. Redun-A selects the most representative and diverse frames by
clustering the video frames, and LQ-A chooses the frames of high quality by
designing a scorer network and retaining frames with high scores, both of
which belong to the text-free frame selection. Then, based on the existing
frame selections and the proposed ones, we investigate the effect of each
frame selection method on TVR, as well as the potential combinations of them.
Through extensive study, we conclude that: (i) giving priority to the text-
free frame selection, combining the proposed Redun-A and LQ-A is an optimal
policy; (ii) by further considering the text-guided frame selection, the
better trade-off between effectiveness and efficiency is achieved by the
combination of Redun-A and N-InT.

Our main contributions can be summarized as follows:

  * •

We present an empirical study of frame selection for TVR, encompassing four
text-free frame selection methods and two text-based frame selection methods.
Through a comprehensive analysis, we aim to explore the optimal frame
selection policy that balances effectiveness and efficiency.

  * •

We introduce two text-free frame selection methods, i.e., the redundancy-aware
and low-quality-aware frame selection, thus providing broader views for the
frame selection.

  * •

We conduct a comprehensive evaluation of the frame selection methods on the
three common benchmarks, i.e., MSR-VTT Xu et al. (2016), DiDemo Anne Hendricks
et al. (2017) and ActivityNet Captions Caba Heilbron et al. (2015). A
significant trade-off between performance and efficiency on the benchmarks can
be achieved by the explored optimal frame selection policy.

##  2 Related Work

###  2.1 Text-to-Video Retrieval

The early TVR methods Yu et al. (2018); Liu et al. (2019); Gabeur et al.
(2020) focused on designing a fusion module for cross-modal learning from the
offline extracted video and text features. In recent years, visual-language
pre-training technologies have begun to push the development of TVR
dominantly. CLIP4Clip Luo et al. (2022) conducted an empirical study on video
aggregation schemes for transferring the CLIP Radford et al. (2021), which has
been pre-trained on large-scale text-image pairs, to TVR. This study
demonstrated impressive performance in TVR with the aid of CLIP. As a result,
a series of works have focused on excavating the power of CLIP by designing
various alignment strategies Ma et al. (2022); Fang et al. (2023); Gorti et
al. (2022); Chen et al. (2023) for performance improvement or plugging token
selection strategies Zhao et al. (2022); Liu et al. (2022) for efficiency
improvement. CenterCLIP Zhao et al. (2022) performed token selection by
clustering numerous video patch tokens within each short video segment.
However, this method limited the selection to the local content of the video
and resulted in inefficient information selection through token clustering.
Beyond adopting CLIP, a two-stream network without the fusion module, as TVR
backbone, LiteVL Chen et al. (2022) constructed a BLIP-based Li et al. (2022b)
model, which incorporates a pre-trained deep fusion module into TVR, and
exhibits better performance but lower efficiency than the CLIP-based works.
Differently, this paper makes an empirical study of frame selection and
explores an optimal frame selection policy for TVR, aiming at achieving the
trade-off between retrieval performance and efficiency.

###  2.2 Frame Selection for TVR

Frame selection is an important process for TVR. It can filter out the
inessential frames in the video for a high-effectiveness and high-performance
TVR. Uniform and random frame selections are two common frame selection
methods, by which some works Luo et al. (2022); Lei et al. (2021) empirically
explored the proper frame rate for TVR to balance efficiency and
effectiveness. However, due to no reasonable guidance for frame selection,
both methods are prone to lose some key frames beneficial to TVR, resulting in
a limited trade-off between efficiency and effectiveness. Actually, texts are
the natural guidance for frame selection in TVR. Directly calculating the
similarities between representations of the frame and text is one of the ways
Gorti et al. (2022); Han et al. (2022). Beyond that, TW-BERT and FinCo Yang et
al. (2023); Wang et al. (2022) proposed to simply concatenate the
representations of frames with the text, and the concatenated representations
were then input to a scorer network, which comprised a fully connected layer
followed by a softmax layer, to assign a score to each frame. The resulting
scores are adopted to guide the frame selection. To better fuse the inter-
modal information during frame selection, some works Buch et al. (2022); Lin
et al. (2022) constructed a self-attention-based encoder to fuse the frames
and text representations. Different from the above text-guided frame selection
methods, we employ a cross-attention-based encoder to fuse the frames and text
representations, and then select the textual-relevant frames based on the
attention weights in InT frame selection. Moreover, this paper provides a
comprehensive analysis of the frame selections to figure out the optimal one
for TVR.

Figure 1: An illustration of the model architecture and frame selection
methods. (a) The model architecture consists of three encoders, i.e., vision,
text, and multimodal encoders. The uniform and random frame selections work in
image frame space, while the other four in feature space. (b) We group all the
frame selection methods into two types, i.e., text-free frame selection (I
~IV) and text-guided frame selection (V ~VI). $\\{s_{1},...,s_{K}\\}$ denotes
the sequence of key frames. The low-quality-aware and interactive frame
selection methods bring in extra parameters, while the others are parameter-
free.

##  3 Methods

In this section, we first introduce the model architecture used for the
empirical study of frame selection. Then, we elaborate on the core content of
our empirical study, including four text-free frame selection methods (i.e.,
uniform, random, redundancy-aware, and low-quality-aware frame selections) and
two text-guided frame selection methods (i.e., non-interactive and interactive
frame selections). Finally, we present the training and inference processes.

###  3.1 Model Architecture

Following BLIP Li et al. (2022b), the model architecture comprises three main
components: a vision encoder, a text encoder, and a multimodal encoder, as
shown in Figure 1 (a). Details for each component are as follows.

Vision Encoder. We employ the ViT-B/16 Dosovitskiy et al. (2020) as our vision
encoder. Given a video $V$, following the general practice in TVR methods Lei
et al. (2021); Luo et al. (2022), we first uniformly pre-sample $N$ frames
$\mathcal{V}=\\{F_{1},F_{2},\cdots,F_{N}\\}$ from the video to represent video
contents. Then we conduct frame selection based on $\mathcal{V}$ to improve
the retrieval efficiency. For the uniform and random frame selections, we
directly operate on $\mathcal{V}$ and obtain $K$ $(K<N)$ frames
$\tilde{\mathcal{V}}=\\{F_{s_{1}},F_{s_{2}},\cdots,F_{s_{K}}\\}$, where
$s_{k}\in\\{1,2,\dots,N\\}$ denotes the index of the selected frame. Then,
each frame $F_{s_{k}}$ is fed into ViT-B/16 and encoded as a sequence of
representations
$f_{s_{k}}=\\{f_{s_{k}}^{cls},f_{s_{k}}^{1},\cdots,f_{s_{k}}^{P}\\}$, where
$P$ is the number of the frame patches and $f_{s_{k}}^{cls}$ denotes the
representation of [CLS] token and can be viewed as the global representation
of $F_{s_{k}}$. For the other four frame selections (i.e., redundancy-aware,
low-quality-aware, non-interactive, and interactive frame selections), we
first encode the frames in $\mathcal{V}$ using ViT-B/16, based on which we
select $K$ key frames. To sum up, the uniform and random frame selections are
performed in image frame space, while the redundancy-aware, low-quality-aware,
non-interactive, and interactive frame selections are carried out in feature
representation space. These frame selection methods will be detailed in
Section 3.2.

To this end, we can obtain the representations of the selected $K$ frames.
Based on this, we compute the global representation of the video $f_{V}$. We
first map the [CLS] representations $f^{cls}_{s_{k}}$ of all $K$ frames to a
lower dimension (256-d) by a linear transformation, denoted as
$f^{cls\prime}_{s_{k}}$. For the uniform, random and redundancy-aware frame
selections, we conduct mean-pooling on the $f^{cls\prime}_{s_{k}}$ for
generating the global representation of the video:

$f_{V}=\frac{1}{K}\sum_{k=1}^{K}f_{s_{k}}^{cls\prime}.\vspace{-3pt}$ (1)

While for the other four frame selection methods, we obtain $f_{V}$ by a
weighted average of $f^{cls\prime}_{s_{k}}$:

$f_{V}=\sum_{k=1}^{K}softmax(\alpha_{s_{k}})f_{s_{k}}^{cls\prime},\vspace{-3pt}$
(2)

where $\alpha_{s_{k}}$ is the score of the frame $F_{s_{k}}$ computed by the
corresponding frame selection method.

Text Encoder. We apply BERT-base Devlin et al. (2018) as the text encoder that
transforms an input text $T$ with $M$ tokens into a sequence of
representations $\\{t^{cls},t^{1},\cdots t^{M}\\}$, where the $t^{cls}$ is the
representation of the [CLS] token. We also map $t^{cls}$ to the lower
dimension (256-d) as the global representation of the text, denoted as
$f_{T}$.

Multimodal Encoder. We adopt the BERT-base together with a cross-attention
layer inserted in each transformer block as the multimodal encoder. The BERT-
base shares the parameters with the text encoder. The representations of the
selected K frames from the vision encoder act as the visual inputs of the
multimodal encoder. For the textual inputs, following BLIP Li et al. (2022b),
we append an [Encode] token to $T$, and its encoded representation $t^{enc}$
is denoted as the multimodal representation of the video-text pair. In the
cross-attention layer, we take textual inputs as the query and visual inputs
as the key and value.

###  3.2 Frame Selection

According to the participation of text, we categorize the frame selection
methods into two groups: the text-free and text-guided frame selections, as
shown in Figure 1 (b). In the following, we detail each frame selection
method.

####  3.2.1 Text-Free Frame Selection

The text-free frame selection chooses frames only from the perspective of
video contents themselves, including the uniform, random, redundancy-aware,
and low-quality-aware frame selection methods. The former two frame selection
methods are performed on the image frame level, while the latter two are on
the feature representation level.

Uniform/Random Frame Selection. The uniform and random frame selections are
the most straightforward frame selection methods. Given the pre-sampled frames
$\mathcal{V}$, the uniform frame selection evenly selects $K$ frames with the
same intervals from $\mathcal{V}$, while the random sampling selects with
random intervals. The selected frames are then sent to the vision encoder for
frame representation.

Redundancy-aware Frame Selection. The uniform or random frame selection
neglects the irregular distribution of crucial information within video
contents, leading to the tendency to lose the crucial frames. For this, we
develop a redundancy-aware frame selection, which is dedicated to reserving
the most representative and diverse frames from each video. Specifically,
given a sequence of lower-dimensional [CLS] representations of each frame in
$\mathcal{V}$, denoted as
$f^{cls^{\prime}}=\\{f_{1}^{cls^{\prime}},f_{2}^{cls^{\prime}},\cdots,f_{N}^{cls\prime}\\}$,
we utilize k-medoids++ clustering algorithm Zhao et al. (2022) to partition
them into $K$ groups. The frames in the same group typically exhibit similar
video contents, indicating redundancy within the video. The center frame of
each group is considered the most representative one. Naturally, we select the
frame whose [CLS] representation corresponds to the center of each group.

Low-quality-aware Frame Selection. The presence of low-quality frames in
videos, which are generally out of interest, has a negative impact on both
retrieval performance and efficiency. Nonetheless, the low-quality frames are
ignored by the existing text-free frame selection methods. In this situation,
we propose a low-quality-aware frame selection to filter them. In detail, we
append a scorer network after the vision encoder to assess the quality score
$\alpha_{i}$ of each frame:

$\alpha_{i}=softmax(FC(f_{i}^{cls\prime})),\vspace{-3pt}$ (3)

where $FC(\cdot)$ denotes the fully connected layer that map the
$f_{i}^{cls\prime}$ $(i=1,2,\cdots,N)$ to one dimension. Given the absence of
explicit annotations indicating frame quality, we instead leverage the paired
text as weak supervision to optimize the scorer network. This strategy is
based on the assumption that frames highly relevant to the text are less
likely to be of low quality. Specifically, during the training process, we
employ the scores of selected frames as weights to aggregate the frames
features as Eq. 2, and then utilize the VTC loss (detailed in section 3.3.1)
to optimize the scorer network. Finally, we retain the frames with the top $K$
scores. During inference, we score each video frame by the trained scorer
network to filter out the low-quality frames.

####  3.2.2 Text-guided Frame Selection

In contrast to the text-free frame selection, the text-guided frame selection
considers the query text information to help filter the inessential frames,
which contain the non-interactive and interactive frame selections.

Non-interactive Frame Selection. It is efficient to directly calculate the
similarities between the representations of frames and the given text for the
frame selection. Specifically, we compute the cosine distances, followed by a
softmax layer, as the relevancy score of frame $F_{i}$ and the given text $T$,
denoted as $\alpha_{i}$:

$\alpha_{i}=softmax(s(f_{i}^{cls\prime},f_{T})),\vspace{-3pt}$ (4)

where $s(\cdot,\cdot)$ denotes the cosine similarity. The frames with the
highest $K$ similarities to the given text are selected.

Interactive Frame Selection. The interactive frame selection constructs a
specific multimodal-based selection module to model the relevancy between
frames and the given text. In our implementation, we design a cross-attention-
based encoder for frame selection, which consists of a single transformer
layer. Specifically, in the cross-attention module, the text global
representation $f_{T}$ is projected into a query $Q_{t}$, and the frames’
[CLS] representations $f^{cls\prime}_{i}$ ($i=1,\cdots,N$) are projected into
key $K_{f}$ and value $V_{f}$. We then adopt the scaled dot product attention
to aggregate these representations, denoted as $f_{T\prime}$:

$\displaystyle f_{T\prime}=Attn(Q_{t},K_{f})V_{f},$ (5) $\displaystyle
Attn(Q_{t},K_{f})=softmax(\frac{Q_{t}K_{f}^{\top}}{\sqrt{d}}),\vspace{-0.5cm}$
(6)

where $d$ is the dimension of $K_{f}$. The dot product attention
$Attn(Q_{t},K_{f})\in\mathcal{R}^{1\times N}$ provides relevancy weights from
a text to each frame. Through the attention mechanism, the fused video feature
$f_{T\prime}$ is encouraged to incorporate the text semantics. To optimize the
selection module, we employ the Contrastive Loss (CL) to pull closer the
$f_{T\prime}$ and the paired $f_{T}$ in each batch $B$, denoted as
$\mathcal{L}_{cl}$:

$\mathcal{L}_{cl}=-\frac{1}{B}\sum_{i}^{B}\log\frac{e^{s(f^{i}_{T},f^{i}_{T\prime})/\lambda}}{\sum\nolimits_{j=1}^{B}e^{s(f^{j}_{T},f^{i}_{T\prime})/\lambda}},\vspace{-3pt}$
(7)

where $\lambda$ is a learnable temperature parameter. Finally, we select the
frames with the top $K$ attention scores to the text.

Table 1: Performance comparison of different frame selection methods on MSR-VTT and DiDeMo. We gray out the best performance in each row. The ‘base’ in the Frame column represents the video processing only with the pre-sampling and without further using the frame selection.  Dataset | Frame | Text-Free | Text-Guided
---|---|---|---
Uni | Rand | Redun-A | LQ-A | N-InT | InT
R@1$\uparrow$ | R@sum$\uparrow$ | R@1$\uparrow$ | R@sum$\uparrow$ | R@1$\uparrow$ | R@sum$\uparrow$ | R@1$\uparrow$ | R@sum$\uparrow$ | R@1$\uparrow$ | R@sum$\uparrow$ | R@1$\uparrow$ | R@sum$\uparrow$
MSR-VTT | 16(base) | 52.9 | 212.9 | 52.9 | 212.9 | 52.9 | 212.9 | 52.9 | 212.9 | 52.9 | 212.9 | 52.9 | 212.9
16$\Rightarrow$12 | 51.7 | 212.6 | 51.8 | 212.8 | 52.3 | 213.0 | 51.1 | 209.6 | 50.4 | 210.6 | 50.8 | 207.7
16$\Rightarrow$8 | 50.9 | 209.9 | 52.4 | 211.1 | 50.8 | 210.5 | 49.6 | 206.5 | 47.3 | 202.1 | 48.9 | 204.3
16$\Rightarrow$6 | 51.4 | 209 | 49.7 | 206.9 | 52.2 | 211.4 | 47.8 | 202.5 | 46.3 | 200.3 | 48.5 | 204.8
16$\Rightarrow$4 | 47.9 | 202.0 | 47.8 | 201.9 | 49.0 | 205.3 | 45.2 | 196.6 | 44 | 193.8 | 46.0 | 197.4
16$\Rightarrow$1 | 24.3 | 125.6 | 26.5 | 136.3 | 39.4 | 176.1 | 33.8 | 158 | 41.7 | 185.4 | 32.6 | 156.3
DiDeMo | 32(base) | 58.4 | 229.2 | 58.4 | 229.2 | 58.4 | 229.2 | 58.4 | 229.2 | 58.4 | 229.2 | 58.4 | 229.2
32$\Rightarrow$24 | 57.8 | 228.1 | 58.7 | 228.6 | 58.3 | 229.7 | 59.2 | 230.7 | 58.6 | 228.6 | 58.6 | 226.4
32$\Rightarrow$16 | 58.1 | 227.4 | 58.0 | 227.6 | 59.3 | 231.1 | 57.7 | 228.7 | 56.3 | 223.9 | 57.6 | 225.6
32$\Rightarrow$12 | 56.9 | 225.7 | 57.6 | 226.5 | 58.3 | 231.3 | 56.6 | 226.7 | 55.2 | 224.2 | 55.7 | 222.8
32$\Rightarrow$8 | 54.6 | 223.4 | 53.8 | 222.9 | 57.7 | 230.7 | 55.9 | 224.5 | 54.0 | 221.2 | 55.2 | 220.9
32$\Rightarrow$4 | 52 | 215 | 52.5 | 215.4 | 57.0 | 225.8 | 51.2 | 213.5 | 51.6 | 217.2 | 49.8 | 215.5
32$\Rightarrow$1 | 33.2 | 166.3 | 34.2 | 164.5 | 38.5 | 180.6 | 36.8 | 173.5 | 47.2 | 203.6 | 41.7 | 191.8

###  3.3 Training and Inference

####  3.3.1 Training

During training, we optimize the model mainly with two losses, i.e., video-
text contrastive loss (VTC) and video-text matching loss (VTM). In particular,
we also append the Contrastive Loss (CL) to optimize the selection encoder of
the interactive frame selection.

Video-Text Contrastive (VTC) pulls the representations of paired video and
text close and pushes away the unpaired ones. Specially, given the [CLS]
representation of video $f_{V}$ and text $f_{T}$, we calculate in-batch
symmetric VTC loss as follows:

$\mathcal{L}_{vtc}=\frac{1}{2}(\mathcal{L}_{t2v}+\mathcal{L}_{v2t}),\vspace{-3pt}$
(8)

where $\mathcal{L}_{t2v}$ and $\mathcal{L}_{v2t}$ denotes text-to-video and
video-to-text loss, respectively, formulated as:

$\displaystyle\mathcal{L}_{t2v}=-\frac{1}{B}\sum_{i}^{B}\log\frac{e^{s(f^{i}_{V},f^{i}_{T})/\tau}}{\sum\nolimits_{j=1}^{B}e^{s(f^{j}_{V},f^{i}_{T})/\tau}},$
(9)
$\displaystyle\mathcal{L}_{v2t}=-\frac{1}{B}\sum_{i}^{B}\log\frac{e^{s(f^{i}_{V},f^{i}_{T})/\tau}}{\sum\nolimits_{j=1}^{B}e^{s(f^{i}_{V},f^{j}_{T})/\tau}},$
(10)

where $\tau$ is a learnable temperature parameter and $B$ is the batch size.

Video-Text Matching (VTM) determines whether a video-text pair is matched or
not. We feed the multimodal encoder’s output $t^{enc}$ into a binary
classifier to predict a two-class probability $p_{vtm}$. The VTM loss is
formulated as:

$\mathcal{L}_{vtm}=\mathbb{E}_{(V,T)\sim D}H(y_{vtm},p_{vtm}),\vspace{-3pt}$
(11)

where $y_{vtm}$ is a 2-dimensional one-hot vector representing the ground-
truth label.

In conclusion, the full training objective is:

$\mathcal{L}=\mathcal{L}_{vtm}+\mathcal{L}_{vtc}+\beta\mathcal{L}_{cl},\vspace{-3pt}$
(12)

where $\beta=1$ when we use the interactive frame selection, otherwise
$\beta=0$.

####  3.3.2 Inference

To speed up the inference, as BLIP Li et al. (2022b), we first compute the
feature similarities $S_{t2v}$ for all text-video pairs by the vision and text
encoders, and then choose the first top-128 video candidates according to
$S_{t2v}$ for further ranking by the multimodal encoder. In addition, with our
frame selections, the inference is more efficient.

##  4 Experiment

###  4.1 Experimental Settings

##### Datasets and Implementation.

We conduct experiments on three common TVR benchmarks, including MSR-VTT Xu et
al. (2016), DiDeMo Anne Hendricks et al. (2017) and ActivityNet Captions Caba
Heilbron et al. (2015). The details of each benchmark and implementation are
introduced in Appendix A and B, respectively.

##### Evaluation Metric.

To evaluate the retrieval performance, we employ three retrieval metrics:
recall at rank M (R@M, M=1, 5, 10), median rank (MdR) and the sum of the
recall (R@sum). R@M calculates the percentage of the successful retrieval in
which the paired sample to the query is found in the top-M retrieved results.
MdR measures the median rank of correct items in the retrieved ranking list.
R@sum is calculated by summing R@1, R@5, and R@10, reflecting the overall
retrieval performance.

Table 2: Efficiency comparison of frame selection methods under the best performance. MeM is the max GPU memory cost during training. Time is the interaction duration per query with all videos during inference.  Method | MSRVTT | DiDeMo
---|---|---
Frame | R@1 | MeM(GB) | GLOPs | Time(ms) | Frame | R@1 | MeM(GB) | GLOPs | Time(ms)
Base | 16 | 52.9 | 19.2 | 1107.6 | 245.8 | 32 | 58.4 | 33.4 | 1997.1 | 462.9
Uni | 16$\Rightarrow$12 | 51.7 | 15.8 | 836.7 | 191.9 | 32$\Rightarrow$24 | 57.8 | 26.4 | 1345.1 | 359.4
Rand | 16$\Rightarrow$12 | 51.8 | 15.8 | 836.7 | 191.9 | 32$\Rightarrow$24 | 58.7 | 26.4 | 1345.1 | 359.4
Redun-A | 16$\Rightarrow$6 | 52.2 | 14.9 | 861.7 | 112.1 | 32$\Rightarrow$16 | 59.3 | 26.4 | 1564.1 | 241.9
LQ-A | 16$\Rightarrow$12 | 51.1 | 17.6 | 1042.2 | 189.8 | 32$\Rightarrow$24 | 59.2 | 30.0 | 1842.8 | 359.9
N-InT | 16$\Rightarrow$12 | 50.4 | 17.6 | 1070.1 | 197.2 | 32$\Rightarrow$24 | 58.6 | 30.0 | 1854.6 | 365.7
InT | 16$\Rightarrow$12 | 50.8 | 17.6 | 1098.6 | 210.6 | 32$\Rightarrow$24 | 58.6 | 30.0 | 1895.7 | 410.1
Table 3: Performance of combinations of frame selection methods. The best
results are bold. The second best results are underline.

Redun-A | LQ-A | N-InT | MSR-VTT | DiDeMo
---|---|---|---|---
Frame | R@1$\uparrow$ | R@sum$\uparrow$ | Frame | R@1$\uparrow$ | R@sum$\uparrow$
|  |  | 16 | 52.9 | 212.9 | 32 | 58.4 | 229.2
✓ |  |  | 16$\Rightarrow$12 | 52.3 | 213.0 | 32$\Rightarrow$16 | 59.3 | 231.1
| ✓ |  | 16$\Rightarrow$12 | 51.1 | 209.6 | 32$\Rightarrow$24 | 59.2 | 230.7
|  | ✓ | 16$\Rightarrow$12 | 50.4 | 210.6 | 32$\Rightarrow$24 | 58.6 | 228.6
✓ | ✓ |  | 16$\Rightarrow$12 | 52.7 | 213.2 | 32$\Rightarrow$16 | 59.5 | 232.1
✓ |  | ✓ | 16$\Rightarrow$12 | 52.6 | 213.9 | 32$\Rightarrow$16 | 59.7 | 232.2
✓ | ✓ | ✓ | 16$\Rightarrow$12 | 52.6 | 213.1 | 32$\Rightarrow$16 | 59.2 | 232.3

###  4.2 Analysis of Frame Selection

In this section, we first compare the frame selection methods on retrieval
performance. Subsequently, we investigate the impact of each frame selection
method on retrieval efficiency. At last, based on the performance and
efficiency analysis, we explore combining multiple frame selection methods to
determine the optimal method. All the analyses are based on the experimental
results on MSR-VTT and DiDeMo, representing two canonical TVR types, i.e.,
standard sentence-to-video retrieval and paragraph-to-video retrieval,
respectively.

####  4.2.1 Effectiveness of Each Frame Selection

Frame selection aims to select fewer $K$ frames to represent a video for
retrieval. Table 1 shows the retrieval performance under different frame
selection methods with the setting of $K\in\\{12,8,6,4,1\\}$ for the MSR-VTT
and $K\in\\{24,16,12,8,4,1\\}$ for the DiDeMo. We can see that: (1) Redun-A
exhibits an overall superiority compared to the existing methods (i.e., Uni
and Rand) in reducing temporal redundancy in videos. Compared to the widely-
used Uni and Rand which have not considered the distribution of crucial
information in the video contents, Redun-A clusters the frames with similar
contents to select the most representative ones and discard other redundant
frames. As a result, Redun-A can effectively mitigate the performance decrease
caused by the reduction of frame number. (2) When only selecting a single
frame (i.e., $K=1$) for TVR to maximize the retrieval efficiency, the text-
guided frame selections, especially the N-InT, present a significant advantage
compared to the text-free frame selections. The text-guided frame selection
can retain the most query-relevant frame in a video and facilitate the
retrieval. (3) When considering more frames (i.e., $K>1$) for higher
performance, the text-free frame selection performs better than the text-
guided frame selection. We conjecture that the text-guided frame selection
introduces the textual bias for TVR. This means that by emphasizing the text-
related frames, the similarities of videos partially related to the given text
may be improved, potentially resulting in extra disturbance for TVR.

Table 4: Results of text-video retrieval on MSR-VTT and DiDeMo datasets. The numbers on the left and right of $\Rightarrow$ respectively denote the number of frames sampled from raw videos and frames used for inter-modal interaction after frame selection. $\dagger$ denotes our implementation of baselines. Method | MSR-VTT | DiDeMo
---|---|---
Frames | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | R@sum$\uparrow$ | MdR$\downarrow$ | Frames | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | R@sum$\uparrow$ | MdR$\downarrow$
CE Liu et al. (2019) | - | 20.9 | 48.8 | 62.4 | 132.1 | 6 | - | 16.1 | 41.1 | 82.7 | 139.9 | 8.3
ClipBERT Lei et al. (2021) | 16 | 22.0 | 46.8 | 59.9 | 128.7 | 6 | 16 | 20.4 | 48.0 | 60.8 | 129.2 | 6
Frozen Bain et al. (2021) | 4 | 32.5 | 61.5 | 71.2 | 165.2 | 3 | 4 | 34.6 | 65.0 | 74.7 | 174.3 | 3
TW-BERT Yang et al. (2023) | 20$\Rightarrow$8 | 38.4 | 65.1 | 76.6 | 180.1 | 3 | 20$\Rightarrow$8 | 41.8 | 71.1 | 81.2 | 194.1 | 4
CLIP4Clip Luo et al. (2022) | 12 | 44.5 | 71.4 | 81.6 | 197.5 | 2 | 64 | 43.4 | 69.9 | 80.2 | 193.5 | 2
MOF Han et al. (2022) | 12$\Rightarrow$4 | 40.5 | 68.2 | 79.5 | 188.2 | 2 | 12$\Rightarrow$4 | 41.3 | 68.5 | 79.4 | 189.2 | 2
CAMoE Cheng et al. (2021) | 16 | 47.3 | 74.2 | 84.5 | 206 | 2 | - | - | - | - | - | -
CenterCLIP Zhao et al. (2022) | 12 | 48.4 | 73.8 | 82.0 | 204.2 | 2 | - | - | - | - | - | -
X-CLIP Ma et al. (2022) | 12 | 49.3 | 75.8 | 84.8 | 209.9 | 2 | 64 | 47.8 | 79.3 | - | - | -
Ts2net Liu et al. (2022) | 12 | 49.4 | 75.6 | 85.3 | 210.3 | 2 | 64 | 41.8 | 71.6 | 82.0 | 195.4 | 2
TABLE Chen et al. (2023) | 12 | 47.1 | 74.3 | 82.9 | 204.3 | 2 | 32 | 47.9 | 74.0 | 82.1 | 204 | 2
HBI Jin et al. (2023) | 12 | 48.6 | 74.6 | 83.4 | 206.6 | 2 | 64 | 46.9 | 74.9 | 82.7 | 204.5 | 2
Cap4Video Wu et al. (2022) | 12 | 51.4 | 75.7 | 83.9 | 211 | 2 | 64 | 52.0 | 79.4 | 87.5 | 218.9 | 1
X-pool$\dagger$ Gorti et al. (2022) | 12 | 46.6 | 73 | 82.9 | 202.5 | 2 | 32 | 44.7 | 72.4 | 80.5 | 197.6 | 2
X-pool(Redun-A+LQ-A) | 12$\Rightarrow$8 | 46.2 | 72.7 | 82.5 | 201.4 | 2 | 32$\Rightarrow$16 | 44.9 | 73.1 | 82.0 | 200.0 | 2
X-pool(Redun-A+N-InT) | 12$\Rightarrow$8 | 46.7 | 72.6 | 83.2 | 202.5 | 2 | 32$\Rightarrow$16 | 45.2 | 73.1 | 82.5 | 200.8 | 2
BLIP$\dagger$ Li et al. (2022b) | 16 | 52.9 | 75.9 | 84.1 | 212.9 | 1 | 32 | 58.4 | 82.8 | 88.0 | 229.2 | 1
BLIP(Redun-A+LQ-A) | 16$\Rightarrow$12 | 52.7 | 75.9 | 84.6 | 213.2 | 1 | 32$\Rightarrow$16 | 59.5 | 83.5 | 89.1 | 232.1 | 1
BLIP(Redun-A+N-InT) | 16$\Rightarrow$12 | 52.6 | 76.5 | 84.8 | 213.9 | 1 | 32$\Rightarrow$16 | 59.7 | 83.7 | 88.8 | 232.2 | 1
Table 5: Results of text-to-video retrieval on ActivityNet Captions dataset.  Method | Frames | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | R@sum$\uparrow$ | MdR$\downarrow$
---|---|---|---|---|---|---
CE Liu et al. (2019) | - | 18.2 | 47.7 | 91.4 | 157.3 | 6
ClipBERT Lei et al. (2021) | 20 | 21.3 | 49.0 | 63.5 | 133.8 | 6
Frozen Bain et al. (2021) | 4 | 28.8 | 60.9 | - | - | 3
CLIP4Clip Luo et al. (2022) | 64 | 40.5 | 72.4 | - | - | 2
Ts2net Liu et al. (2022) | 64 | 41.0 | 73.6 | 84.5 | 199.1 | 2
X-CLIP Ma et al. (2022) | - | 46.2 | 75.5 | - | - | 6.8
HBI Jin et al. (2023) | 64 | 42.2 | 73.0 | 84.6 | 199.8 | 2
BLIP$\dagger$ Li et al. (2022b) | 32 | 54.3 | 80.6 | 88.8 | 223.7 | 1
BLIP(Redun-A+LQ-A) | 32$\Rightarrow$24 | 54.6 | 80.0 | 88.8 | 223.4 | 1
BLIP(Redun-A+N-InT) | 32$\Rightarrow$24 | 54.8 | 80.4 | 89.0 | 224.2 | 1

####  4.2.2 Efficiency of Each Frame Selection

In this subsection, we explore the retrieval efficiency of different frame
selection methods under the best performance in Table 2. Uni and Rand
determine the $K$ key frames prior to the vision encoder, which means only
fewer $K$ frames need to be encoded, resulting in the lowest memory cost and
GLOPs. The Redun-A filters out most of the redundant frames through clustering
and only retains 16 most representative frames, which gives rise to the
fastest inference speed, simultaneously accompanied with a remarkable
performance of $59.3\%$ at R@1. Although the LQ-A achieves competitive
performance compared to Redun-A, it comes at the cost of increased frame
count, resulting in lower efficiency. The N-InT simply uses a dot product
between the whole frames and the given query to select the key frames,
bringing a marginal additional time consumption and comparable efficiency.
Since the InT requires an additional fusion for the selection, it eventually
exhibits an inferiority of retrieval efficiency. In conclusion, the Redun-A
and N-InT are two reasonable individual frame selection methods for text-free
and text-guided frame selection, respectively.

####  4.2.3 Combinations of Frame Selections

We combine various frame selections to explore a better trade-off between
efficiency and performance. According to the aforementioned analysis of each
individual frame selection method, we choose Redun-A for the text-free frame
selection and N-InT for the text-guided frame selection. In addition,
considering the competitive performance of the LQ-A, we also combine it with
the Redun-A to further towards better performance of text-free frame
selection. In the combinations, we calculate the cumulative scores by summing
the frame scores obtained from each single frame selection method and select
the top-K frames with the highest scores. In particular, when combining the
Redun-A, we fix the clustering number (denoted as $Z$) referred to the
retrieval performance of the single Redun-A method, that is $Z=6$ for MSR-VTT
and 16 for DiDeMo. We assign a score of $1/Z$ to each selected frame, while
for the others, it is set to $0$. Table 3 presents the results of the various
combinations. By combining multiple frame selection methods, we can get better
performance than a single frame selection.

For example, compared with the individual Redun-A, the combination of Redun-A
and LQ-A can achieve an R@sum improvement of $0.9\%$ on DiDeMo due to the
elimination of the low-quality frames. When taking the text information into
consideration for frame selection, the combination of Redun-A and N-InT gets
the best performance at R@sum, consistently surpassing all of the single frame
selection methods. We also observe that combining all of the three frame
selections could not bring additional gains compared with the combinations of
the two methods. We hypothesize that the N-InT could cover the ability to
filter the low-quality frames to some extent. In addition, we provide the
performance comparison between the combination of text-free frame
selection(i.e., the combination of Redun-A and LQ-A) and the traditional key
frame selection technology in the Appendix C.

###  4.3 Comparison to State-of-the-Art

To further demonstrate the effectiveness of the frame selection in TVR, we
compare our method with the state-of-the-art methods in Table 4 and Table 5.
It is worth noting that the pure BLIP has outperformed the current SOTA
methods, which indicates that a pretrained model with a deeper fusion module
benefits the TVR. However, the accompanying drawback of the deeper fusion lies
in the low efficiency. Through our frame selection, we get a better trade-off
between performance and efficiency. Specifically, with proper frame selections
(e.g., Redun-A + N-InT), the performance of BLIP could have a 1.0%, 3.0%, and
0.5% improvement at R@sum, while the frame number used for deeper fusion has a
considerable decrease to 75%, 50% and 75% on MSR-VTT, DiDeMo and ActivityNet
Captions, respectively. To verify the generalization ability of our method, we
also conduct the frame selection methods on other baseline X-pool Gorti et al.
(2022) on MSR-VTT and DiDeMo datasets, as shown in Table 4. The table also
clearly indicates that frame selection (e.g., Redun-A + N-InT) can effectively
mitigate performance degradation despite using fewer frames, which typically
results in a higher retrieval efficiency.

##  5 Conclusion

In this paper, we make an empirical study to explore the optimal frame
selection methods for balancing the retrieval performance and efficiency in
TVR. Beyond the existing four frame selection methods, we additionally develop
two text-free paradigms, i.e., redundancy-aware and low-quality-aware frame
selection. We conduct abundant experiments to analyze the impact on TVR of the
above six frame selections under the condition of various frame number, and
make a comprehensive comparison from the perspective of retrieval efficiency
and resource consumption. Extended experimental results on multiple TVR
benchmarks effectively demonstrate that with proper frame selection, we can
not only get a competitive performance but also realize a higher retrieval
efficiency under lower resource consumption. Hopefully, this study could spur
further study on deeper interaction between text and video in TVR.

## Limitations

This empirical study provides insight into the optimal frame selections for
TVR. The major limitation may be manually searching for a shared number of key
frames in a dataset. However, a video typically contains a various number of
key frames. Therefore, a more desirable frame selection method would be able
to automatically determine an appropriate number of key frames based on the
content of each video individually. We leave the related investigations to
future work.

## References

  * Anne Hendricks et al. (2017) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017.  Localizing moments in video with natural language.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5803–5812.
  * Bain et al. (2021) Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1728–1738.
  * Buch et al. (2022) Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022.  Revisiting the" video" in video-language understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2917–2927.
  * Caba Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  Activitynet: A large-scale video benchmark for human activity understanding.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ , pages 961–970.
  * Chang et al. (1999) Hyun Sung Chang, Sanghoon Sull, and Sang Uk Lee. 1999.  Efficient video indexing scheme for content-based retrieval.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 9(8):1269–1279.
  * Chang (2003) Shih-Fu Chang. 2003.  Content-based video summarization and adaptation for ubiquitous media access.  In _12th International Conference on Image Analysis and Processing, 2003. Proceedings._ , pages 494–496. IEEE.
  * Chen et al. (2022) Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. 2022.  Litevl: Efficient video-language learning with enhanced spatial-temporal modeling.  _arXiv preprint arXiv:2210.11929_.
  * Chen et al. (2023) Yizhen Chen, Jie Wang, Lijian Lin, Zhongang Qi, Jin Ma, and Ying Shan. 2023.  Tagging before alignment: Integrating multi-modal tags for video-text retrieval.  _arXiv preprint arXiv:2301.12644_.
  * Cheng et al. (2021) Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. 2021.  Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss.  _arXiv preprint arXiv:2109.04290_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Divakaran et al. (2002) Ajay Divakaran, Regunathan Radhakrishnan, and Kadir A Peker. 2002.  Motion activity-based extraction of key-frames from video shots.  In _Proceedings. International Conference on Image Processing_ , volume 1, pages I–I. IEEE.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Fang et al. (2023) Bo Fang, Chang Liu, Yu Zhou, Min Yang, Yuxin Song, Fu Li, Weiping Wang, Xiangyang Ji, Wanli Ouyang, et al. 2023.  Uatvr: Uncertainty-adaptive text-video retrieval.  _arXiv preprint arXiv:2301.06309_.
  * Gabeur et al. (2020) Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020.  Multi-modal transformer for video retrieval.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 214–229. Springer.
  * Gorti et al. (2022) Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. 2022.  X-pool: Cross-modal language-video attention for text-video retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5006–5015.
  * Han et al. (2022) Ning Han, Xun Yang, Ee-Peng Lim, Hao Chen, and Qianru Sun. 2022.  Efficient cross-modal video retrieval with meta-optimized frames.  _arXiv preprint arXiv:2210.08452_.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1026–1034.
  * Jin et al. (2023) Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. 2023.  Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning.  _arXiv preprint arXiv:2303.14369_.
  * KeplerLab (2019) KeplerLab. 2019.  Katna: Tool for automating video keyframe extraction, video compression, image autocrop and smart image resize tasks.
  * Lei et al. (2021) Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7331–7341.
  * Li et al. (2022a) Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. 2022a.  Align and prompt: Video-and-language pre-training with entity prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4953–4963.
  * Li et al. (2022b) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer.
  * Lin et al. (2022) Yuanze Lin, Chen Wei, Huiyu Wang, Alan Yuille, and Cihang Xie. 2022.  Smaug: Sparse masked autoencoder for efficient video-language pre-training.  _arXiv preprint arXiv:2211.11446_.
  * Liu et al. (2019) Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. 2019.  Use what you have: Video retrieval using representations from collaborative experts.  _arXiv preprint arXiv:1907.13487_.
  * Liu et al. (2022) Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. 2022.  Ts2-net: Token shift and selection transformer for text-video retrieval.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIV_ , pages 319–335. Springer.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Luo et al. (2022) Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022.  Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.  _Neurocomputing_ , 508:293–304.
  * Ma et al. (2022) Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022.  X-clip: End-to-end multi-grained contrastive learning for video-text retrieval.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 638–647.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Wang et al. (2022) Zixu Wang, Yujie Zhong, Yishu Miao, Lin Ma, and Lucia Specia. 2022.  Contrastive video-language learning with fine-grained frame sampling.  _arXiv preprint arXiv:2210.05039_.
  * Wu et al. (2022) Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. 2022.  Cap4video: What can auxiliary captions do for text-video retrieval?  _arXiv preprint arXiv:2301.00184_.
  * Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5288–5296.
  * Yang et al. (2023) Xu Yang, Zhangzikang Li, Haiyang Xu, Hanwang Zhang, Qinghao Ye, Chenliang Li, Ming Yan, Yu Zhang, Fei Huang, and Songfang Huang. 2023.  Learning trajectory-word alignments for video-language tasks.  _arXiv preprint arXiv:2301.01953_.
  * Yu et al. (2018) Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018.  A joint sequence fusion model for video question answering and retrieval.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 471–487.
  * Zhao et al. (2022) Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. 2022.  Centerclip: Token clustering for efficient text-video retrieval.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 970–981.

Figure 2: Visualization of the selected frames under different frame selection
methods. We select 6 key frames with the highest scores in the raw pre-sampled
frames. Bar plots show the corresponding scores of the displayed frames.

##  Appendix A Datasets

We introduce the three involving benchmarks for TVR as follows:

  * •

MSR-VTT contains 10K videos, each of which ranges from 10 to 32 seconds and is
captioned with 20 descriptions. Each description provides a brief overview of
the corresponding video for standard sentence-to-video retrieval. We follow
the widely-used data split in previous methods Gabeur et al. (2020); Luo et
al. (2022) to train the model on the training-9K set and test on the test 1k-A
set.

  * •

DiDeMo includes 10K unedited videos paired with 40K descriptions. Each video
is divided into multiple segments, and each segment is accompanied by a
corresponding description. Following previous works Lei et al. (2021); Luo et
al. (2022), we conduct paragraph-to-video retrieval by concatenating all
descriptions of each video into a single query.

  * •

ActivityNet Captions contains 20K videos annotated with 100K descriptions. The
training set contains 10K videos, and we use the val1 set with 4.9K videos to
report results. Following Gabeur et al. (2020); Luo et al. (2022), all
descriptions of a video are also concatenated together for paragraph-to-video
retrieval.

##  Appendix B Implementation Details.

All experiments are conducted on 4 NVIDIA A40 GPUs. We initialize the model
parameters used for empirical study from the BLIP Li et al. (2022b) that has
been fine-tuned on the COCO Lin et al. (2014). As for the extra parameters of
the frame selection module of the LQ-A and InT, we use Kaiming Normal He et
al. (2015) initialization. To convert a video into a frame sequence, we
uniformly pre-sample 16, 32, and 32 frames from each video of MSR-VTT, DiDeMo,
and ActivityNet Captions, respectively, and then resize each frame to
224$\times$224\. The maximum textual length is set as 35 for MSR-VTT and 64
for the other two datasets. We set the batch size to 4 and set the learning
rate for BLIP-initialized weights to 5e-6 and for other parameters to 5e-4. We
optimize our model for 5 epochs using the AdamW optimizer Loshchilov and
Hutter (2017) with a weight decay of 0.05 and decay the learning rate using a
cosine schedule.

Except for the BLIP, we also employ another baseline X-pool Gorti et al.
(2022) to verify the effectiveness of the optimal frame selection methods in
Section 4.3. We follow the default settings of the X-pool for MSR-VTT. For the
DiDeMO, we set the batch size to 16 and the pre-sampled frames to 32. For the
scorer network of LQ-A, we set its learning rate to 5e-4.

##  Appendix C Comparison to Traditional Key Frame Selection Method

Table 6: Performance comparison between ours Redun-A+LQ-A and Katna on DiDeMo. Both methods are designed to eliminate redundant and low-quality frames.  Method | Frames | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | R@sum$\uparrow$ | MdR$\downarrow$
---|---|---|---|---|---|---
Katna KeplerLab (2019) | 8 | 52.3 | 77.8 | 85.7 | 215.8 | 1
16 | 56.0 | 81.0 | 86.6 | 223.6 | 1
Redun-A+LQ-A | 8 | 57.9 | 82.3 | 88.3 | 228.5 | 1
16 | 59.5 | 83.5 | 89.1 | 232.1 | 1

Key frame extraction techniques Chang et al. (1999); Divakaran et al. (2002);
Chang (2003) have been developed in the past decades. However, these
techniques are generally based on the low-level pixel-level understandings of
videos (e.g. frames’ color feature) for the key frame selection. For instance,
Katna KeplerLab (2019) identifies redundant and blurry frames by initially
applying K-means Clustering to frames using image histograms. It subsequently
selects the best frames from clusters based on the variance of the Laplacian
and the overall sharpness of the frames. In Table 6, we compare the retrieval
performance achieved by applying Redun-A+LQ-A and Katna for frame selection,
respectively. Both methods are designed to eliminate redundant and low-quality
frames. Our Redun-A+LQ-A exhibits a significant advantage in TVR compared to
Katna, with performance improvements of 3.5% and 5.6% at R@1 when the frame
number is set to 16 and 8, respectively. Our Redun-A+LQ-A select frames using
frame features computed by the video encoder, which encapsulate the
understanding of video contents. In comparison, the Katna pre-processes videos
based on color features. On one hand, Katna fails to prioritize the primary
area of interest in frames during the selection process. On the other hand,
the combination of the Katna and deep retrieval model could only be done by
video pre-processing. Therefore, our frame selection methods are more friendly
to the deep retrieval model and thus get better performance.

##  Appendix D Visualization of Frame Selection

We display visualization results under various frame selection methods in
Figure 2. We can see that the selected frames could effectively reflect the
purpose of each selection method. Specifically, the redundancy-aware frame
selection retains the most diverse information of the pre-sampled frames,
while the low-quality-aware frame selection effectively recognizes the blurry
frames (i.e., the $2^{nd}$ ~$4^{th}$ frames). In contrast, the text-guided
frame selection, including the non-interactive and the interactive frame
selection, can retain almost all the frames that are related to the query
text. The visualization illustrates that the involved frame selection methods
could effectively filter the inessential frames based on the respective
criteria.