#  Correlation-guided Query-Dependency Calibration in
Video Representation Learning for Temporal Grounding

WonJun Moon, Sangeek Hyun, SuBeen Lee, Jae-Pil Heo⋆
Sungkyunkwan University
{wjun0830, hsi1032, leesb7426<EMAIL_ADDRESS>
###### Abstract

Recent endeavors in video temporal grounding enforce strong cross-modal
interactions through attention mechanisms to overcome the modality gap between
video and text query. However, previous works treat all video clips equally
regardless of their semantic relevance with the text query in attention
modules. In this paper, our goal is to provide clues for query-associated
video clips within the crossmodal encoding process. With our Correlation-
Guided DEtection TRansformer (CG-DETR), we explore the appropriate clip-wise
degree of cross-modal interactions and how to exploit such degrees for
prediction. First, we design an adaptive cross-attention layer with dummy
tokens. Dummy tokens conditioned by text query take a portion of the attention
weights, preventing irrelevant video clips from being represented by the text
query. Yet, not all word tokens equally inherit the text query’s correlation
to video clips. Thus, we further guide the cross-attention map by inferring
the fine-grained correlation between video clips and words. We enable this by
learning a joint embedding space for high-level concepts, i.e., moment and
sentence level, and inferring the clip-word correlation. Lastly, we use a
moment-adaptive saliency detector to exploit each video clip’s degrees of text
engagement. We validate the superiority of CG-DETR with the state-of-the-art
results on various benchmarks for both moment retrieval and highlight
detection. Codes are available at github.com/wjun0830/CGDETR.

†† ⋆ Corresponding author  Figure 1:  Comparison of degrees of text-to-video
correlation in attention layers. In the middle row, we compare the clip-wise
correspondence score to the text query (sum of attention weights over all
words) with its corresponding GT (saliency scores). While the use of (b) self-
attention or (c) cross-attention fails to distinguish target clips based on
the degree of cross-modal attention, our approach (a) exhibits a high
activation level for the text query to attend only the query-relevant clips
since the dummies occupy a portion of the attention degree on irrelevant
clips. We also investigate the fine-grained correlation between clips and
words. Despite the absence of supervision, our method learns to attend more to
salient words.

##  1 Introduction

Video temporal grounding is to detect temporal moments that align with user-
specified language requests. Aside from the moment localization, recent works
also explore how well each video clip corresponds with the text query. To
address these tasks, it is crucial to align the representation space across
modalities. In this regard, the use of transformers [64] has become a common
approach that facilitates easy integration of multimodal representations.
While some studies [32, 43] use transformer encoders to establish a shared
embedding space, others also leverage detection transformers for making
predictions [27, 71]. Recently, QD-DETR [46] highlighted previous works’
insufficient text reflection in predictions and suggested explicitly forcing
text engagement with cross-attention. Yet, in moment retrieval scenarios where
users seek specific video segments, we argue that uncalibrated text
involvement in each video clip also has a large stake in insufficient text
engagement. We suggest that the adaptive textual conditioning on each clip
enables the model to attend to desired moments based on the degree of text
condition, thereby maintaining the high text involvement across all video
clips may not necessarily be the optimal solution. In Fig. 1, we display the
clip-wise degree of cross-modal attention. Although only a few clips are
relevant to the given query, previous works using (b) self-attention and (c)
cross-attention exhibit similar degrees of cross-modal interaction on every
clip both in terms of the entire query and each word. For the importance of
the adaptive degree in attention, we examine the positive relationship between
the alignment of clip-wise query attention degrees with the corresponding
ground truth (GT) saliency scores and their impact on performance in Appendix
A.1.

Building upon the observation that the exploration for optimal cross-modal
interaction among visual and texture representations remains a yet-to-be-
explored challenge in temporal grounding, our Correlation-Guided DEtection
TRansformer (CG-DETR) focuses on discovering the appropriate degree of
interaction and utilizing such degrees in predictions. To this end, we first
propose an Adaptive Cross-Attention layer (ACA) that modifies the operation of
the cross-attention layer; by adding dummy tokens to the key in the multi-head
attention, we adjust the intensity of text query’s clip-wise engagement based
on the relevance between video clips and a text query. Subsequently, we delve
into the relationship between clips and each word in a text query. It is
evident that not all words bear close relevance to a video clip, even if they
are part of a highly relevant text query. Still, studying a fine-grained
correlation is demanding since word-level supervision requires huge resources.
Thus, we approximate the clip-word relevance by calculating similarity in a
modality-aligned space at a more coarse video and sentence level. The
similarity map is then, distilled to adjust the magnitude of the attention map
in the cross-attention layer. Lastly, whereas previous saliency detectors rely
on learnable parameters, we propose a moment-adaptive saliency detector that
leverages the magnitude of the cross-modal interaction. A saliency token
designed with two principles, i.e., preserving original context and
representing diverse characteristics of desired moments, is aggregated with
textual information. This allows us to incorporate scaled interaction degrees
into prediction.

To sum up, our contributions are (i) We propose adaptive cross-attention with
dummy tokens to enable the manipulation in the degree of video-text
interaction with respect to their correlation, (ii) To further calibrate the
interaction degree, we discover and distill the clip-word correlation from the
aligned space at video-sentence level, (iii) We introduce a moment-adaptive
saliency detector to exploit the calibrated degree of cross-modal interaction,
and (iv) Extensive experiments demonstrate the effectiveness of CG-DETR.

##  2 Related Work

###  2.1 Video Temporal Grounding

Video temporal grounding, a task to link the text query to corresponding video
segments, can be further divided into moment retrieval and highlight detection
which localizes the desired moments and scores the clip-wise correspondence to
the query. Moment retrieval has been introduced with the goal of retrieving
user-desired moments [1, 15, 59, 34, 33, 10, 78]. Conventional approaches fall
into either proposal-based or proposal-free methods. Proposal-based methods
employ predefined proposals, e.g., sliding windows [1, 38, 82, 16, 77] and
temporal anchors [8, 77, 85, 79, 35], or learn to generate proposals [84, 37,
67, 56, 69] to prepare moment candidates. With the candidates, they treat the
task as the matching problem between candidates and the text query. On the
other hand, proposal-free methods learn to encode multimodal knowledge and
predict the temporal spans with the regression head. Highlight detection aims
to score the importance of every clip with respect to either visual or textual
information. It has been studied extensively with varying sources, e.g.,
visual only [70, 4, 54, 68] and visual-audio [19, 21, 68, 3], and different
granularity levels on the given labels; there exist supervised [18, 70, 63],
weakly supervised [6, 49, 68], and unsupervised methods [4, 45, 25, 53] for
highlight detection.

Recently, since the advent of QVHighlights [27], these problems are now being
considered together. Since then, newly proposed approaches can be divided into
DETR- or regression-based. Employing either streams, diverse kinds of
practices were conducted; UMT [43] exploited additional audio modality, QD-
DETR [46] and EaTR [23] developed the DETR architecture, and [32, 72] remarked
the importance of pretraining. Also, the study for the query dependency is
being spotlighted [86, 46, 5, 20, 39]. Our motivation resembles these works,
especially QD-DETR [46]. However, whereas they only focus on enforcing text
engagement in every video clip, we aim to discover an appropriate degree and
calibrate the query dependency in cross-modal interaction.

Figure 2: An overview of CG-DETR. From left to right, the model consists of
three phases: (i) feature extraction, (ii) correlation-guided feature
interaction, and (iii) predictions for grounding tasks. (i) Along with video
and text feature extraction, dummy tokens are conditioned by the query to
represent the query-excluding meaning. (ii) Correlation-guided feature
interaction is performed with adaptive cross-attention. In addition to
calibration of text query engagement as a whole, we also guide the word-wise
engagement with the clip-word correlation learner. At the bottom, we utilize
the video context and a calibrated attention map to generate a saliency token.
A saliency token is processed through the same cross-attention layer to be
similarly encoded with video tokens to prevent a modality gap. Details for
correlation learner and saliency token are in Fig. 4, 5. (iii) Finally, tokens
are processed through the encoder and decoder to make predictions.

###  2.2 Vision-text Alignment

Driven by the recognition that learning the joint embedding spaces for visual
and textual modalities yields effective representations, there has been a
surge of interest in vision-text alignment [29, 30, 75, 76, 50, 28, 24, 31].
Subsequently, outcomes of the studies have been introduced in multimodal
downstream tasks, e.g., CLIP has gained prominence for video grounding.
However, even large pretrained models do not invariably possess a perfect
shared embedding space. Hence, efforts have been directed toward refining
cross-modal interactions in downstream tasks to address such limitations. In
the context of video grounding, dual contrastive learning [48] and multimodal
reasoning [88] have been proposed to improve text-video alignment. Similarly,
QD-DETR [46] employed negative-pair learning to learn general relationships
and fostered active cross-modal interaction. Our work strives for a more
precise understanding of relations. Subsequent to forming an aligned space
between the video and sentence, we infer fine-grained clip-word similarity
within the space, enabling the manipulation of the behavior of cross-modal
interaction for enhanced video temporal grounding.

##  3 Method

###  3.1 Overview

Given a video $V=[v_{1},...,v_{L_{v}}]$ of $L_{v}$ clips and a text query
$Q=[q_{1},...,q_{L_{q}}]$ of $L_{q}$ tokens, our objective is to predict clip-
wise saliency scores $\\{s_{1},s_{2},...,s_{L_{v}}\\}$ and to localize the
target moments in the form of $(m_{c},m_{\sigma})$ where $m_{c}$ and
$m_{\sigma}$ denote the center temporal coordinate and span of the moment. The
overview of our model is illustrated in Fig. 2.

The aim of CG-DETR is to leverage the interrelation between modalities to
improve the feature interaction operation. To achieve that, in Sec. 3.2, we
propose to employ the dummy tokens $D$; these serve to distinguish the degree
of cross-modal attention for each video clip and are conditioned by query
tokens to encapsulate representations that exclude query contexts. In Sec.
3.3, we provide fine-grained correlation guidance to the adaptive cross-
attention layers by aligning the modalities at the video-sentence level where
supervision is available and inferring at a more granular clip-word level.
Lastly, in Sec. 3.4, we introduce a saliency token that incorporates both
video context and moment-adaptive knowledge derived from clip-wise query
correspondence in the learned attention map of cross-attention layers. Then, a
saliency token processed via cross-attention exclusively with text tokens is
concatenated with clip-wise tokens to be forwarded for the prediction.

Figure 3: Illustration of deriving clip-wise query correspondence $\bar{a}$.

###  3.2 Adaptive Cross-Attention:
Reflecting the Video-Text Correlation

Adaptive cross-attention stems from the insight that not all contents in a
video always correspond to the semantics of a text query. Meanwhile, cross-
attention is one of the most popular methods to incorporate the features of
different modalities, especially for retrieval tasks [60, 46]. Yet, as the
softmax applied in the cross-attention forces the text query to be equally
aggregated across all video clips, the cross-attention is inadequate to learn
the degree of video-text correspondence. A naïve solution is to change the
softmax activation to sigmoid. However, we point out the vulnerability of
sigmoid in ranking the text-relevance scores of clips since it eliminates the
dependency among other clips.

Instead, we introduce dummy tokens that are concatenated to the text tokens
and possess a portion of attention weights in cross-attention layers. Assuming
that we have $L_{d}$ learnable dummy tokens $D=[D_{1},...,D_{L_{d}}]$, these
dummy tokens are conditioned by the text query $Q$ through attention layers to
contain the query-excluding context. We denote the encoded dummy tokens as
$\tilde{D}$. Then, in the cross-attention layer where query
$\textit{Q}=[p_{\textit{Q}}(v_{1}),...,p_{\textit{Q}}(v_{L_{v}})]$ and value
$\textit{V}=[p_{\textit{V}}(t_{1}),...,p_{\textit{V}}(t_{L_{q}})]$ are
prepared by projecting the video clips and text queries, dummy tokens take
part in key
$\textit{K}=[p_{\textit{K}}(t_{1}),...,p_{\textit{K}}(t_{L_{q}}),p_{\textit{K}}(\tilde{D}_{1}),...,p_{\textit{K}}(\tilde{D}_{L_{d}})]$,
concatenated with text tokens before key projection. $p_{\textit{Q}}(\cdot)$,
$p_{\textit{K}}(\cdot)$, and $p_{\textit{V}}(\cdot)$ are projection layers for
query, key, and value. Formally, the operation of adaptive cross-attention
with dummy tokens for $i$-th video clip, $\text{ACA}(v_{i})$, is expressed as:

$\displaystyle\text{ACA}(v_{i})\\!=\\!\sum_{j=1}^{L_{q}}W_{i,j}\\!\odot\\!\textit{V}_{j};\;W_{i,j}\\!=\frac{{\text{exp}\left(\frac{\textit{Q}_{i}\odot\textit{K}_{j}}{\sqrt{h}}\right)}}{\sum_{k=1}^{L_{q}+L_{d}}\\!\text{exp}\\!\left(\frac{\textit{Q}_{i}\odot\textit{K}_{k}}{\sqrt{h}}\right)},\\!\\!\\!$
(1)

where $h$ denotes the projected hidden dimension and $\odot$ stands for the
dot product. To guide the dummy tokens to take desired portions of attention
weights that are inversely proportional to the saliency scores, we define a
query correspondence score for $i$-th clip,
$\bar{a_{i}}=\sum_{j=1}^{L_{q}}W_{i,j}$, as shown in Fig. 3 and train
$\bar{a_{i}}$ by applying the objectives for highlight detection. Details for
the objective are in Sec. 3.5 and Appendix A.4. In addition, we employ binary
cross-entropy to discretize between moment and non-moments within every
instance and enforce orthogonality among encoded dummy tokens to prevent them
from playing the same role:

$\mathcal{L}_{\text{bce}}=\\!\frac{1}{L_{v}}\\!\sum^{L_{v}}_{i=1}\\!\left(a_{i}\\!\odot\\!\text{log}\;\bar{a}_{i}+\left(1-a_{i}\right)\\!\odot\\!\text{log}\left(1-\bar{a}_{i}\right)\right),$
(2)
$\mathcal{L}_{\text{ortho}}=\frac{1}{L_{d}\left(L_{d}-1\right)}\sum^{L_{d}}_{m=1}{\sum^{L_{d}}_{n=1}{\mathbbm{1}_{m\neq
n}|\tilde{D}_{m}\odot\tilde{D}_{n}|}},$ (3)

where $a_{i}$ is 1 if the saliency GT is non-zero, otherwise set to 0. By
allowing such flexible cross-attention, the model learns better separation of
relevant and irrelevant video segments.

Figure 4: Clip-word correlation learner. We discover and reflect the relevance
between clips and text words into cross-attention. (a) We establish visual
moment, non-moment, query, and dummy representative tokens
($\hat{M}^{b+},\hat{M}^{b-},\hat{S}^{b+},\text{and }\hat{S}^{b-}$) using
learnable moment and sentence tokens. (b) To learn the aligned space, we
implement contrastive learning. Whereas the text query prototype $\hat{S}^{+}$
learns to be aligned with the paired visual moment token $\hat{M}^{+}$, dummy
prototype $\hat{S}^{-}$ learn to exclude the moment-specific knowledge. (c)
Within the aligned space, the correlation in clip-word level is employed as
the guidance for the attention map in the cross-attention.

###  3.3 Clip-Word Correlation Learner: Align, Discover, and Distill the Fine-
grained Correlation

By employing ACA, the model considers the clip-wise relevance with text query.
Delving into the clip-word relation, it is evident that not all words are
equally associated with visual clues. With this motivation, we further aim to
discover the proper attention weights for cross-attention that represent the
correlation between video clips and semantic words.

However, determining the proper clip-word correlation is challenging in the
absence of supervision. Hence, we take a two-step approach; 1) learning the
vision-text-aligned embedding space at the video-query level where supervision
is given, 2) inferring the vision-text correlation at a more fine-grained
level, i.e., clip-word level, and distill the correlation to the attention map
in the cross-attention layers.

To align the embedding space of both modalities at a video-sentence level, we
generate moment and non-moment prototypes for both modality domains with
domain-specific prototype tokens, i.e., moment and sentence tokens, and apply
contrastive learning between them [11, 50, 41, 31]. Let $V^{b+}$ $=$
$\left[v^{b}_{i}|i\in\\{1,2,...,L_{v}\\},a_{i}=1\right]$ and $V^{b-}$ $=$
$\left[v^{b}_{i}|i\in\\{1,2,...,L_{v}\\},a_{i}\neq 1\right]$ be the video
clips in the specified target moment (query-relevant) and otherwise (query-
irrelevant) in $b$-th video instance of batch size $B$, respectively. As
mentioned in Sec. 3.2, $a$ indicates the GT for query correspondence score
$\bar{a}$. In Fig. 4 (a), we illustrate prototype generation. By processing
learnable moment token $M$ with each of these visual inputs through the self-
attention block (SA), we derive moment prototype $\hat{M}^{b+}$ and non-moment
prototypes $\hat{M}^{b-}$ for $b$-th video instance in the visual domain each
with the projected video representations $\hat{V}^{b+}$ and $\hat{V}^{b-}$:

$\displaystyle\hat{M}^{b+},\hat{V}^{b+}=\text{SA}([M;V^{b+}],[M;V^{b+}],[M;V^{b+}]),$
(4)
$\displaystyle\hat{M}^{b-},\hat{V}^{b-}=\text{SA}([M;V^{b-}],[M;V^{b-}],[M;V^{b-}]),$
(5)

where $\left[~{};\right]$ and $\;\hat{\cdot}\;$ denote concatenation and
projected output via SA, respectively. Symmetrical to processing the moment
token $M$ in the video domain, we leverage the learnable sentence token $S$
with each of the $b$-th text query $Q^{b}$ and dummy tokens $D^{b}$ to yield
moment and non-moment prototypes in the semantic domain:

$\displaystyle\hat{S}^{b+},\hat{Q}^{b}=\text{SA}([S;Q^{b}],[S;Q^{b}],[S;Q^{b}]),$
(6)
$\displaystyle\hat{S}^{b-},\hat{D}^{b}=\text{SA}([S;\tilde{D}^{b}],[S;\tilde{D}^{b}],[S;\tilde{D}^{b}]).$
(7)

Consequently, with $\mathcal{B}$ as the index set in the batch, we formulate
the contrastive learning objective for $b$-th instance using the batch-wise
labels in Fig. 4 (b):

$\displaystyle\mathcal{L}^{+}_{\text{align}}\\!=\\!-\text{log}\frac{\text{exp}(\hat{S}^{b+}\odot\hat{M}^{b+}/\tau)}{\sum\limits_{a\in\mathcal{B}}\sum\limits_{*\in\left\\{+,-\right\\}}\text{exp}(\hat{S}^{b+}\odot\hat{M}^{a*}/\tau)},$
(8)
$\displaystyle\mathcal{L}^{-}_{\text{align}}=-\text{log}\bigg{(}1-\frac{\text{exp}(\hat{S}^{b-}\odot\hat{M}^{b-}/\tau)}{\sum\limits_{a\in\mathcal{B}}\sum\limits_{*\in\left\\{+,-\right\\}}\\!\\!\\!\\!\\!\text{exp}(\hat{S}^{b-}\\!\odot\\!\hat{M}^{a*}/\tau)}\bigg{)},$
(9)
$\displaystyle\mathcal{L}_{\text{align}}=\mathcal{L}^{+}_{\text{align}}+\mathcal{L}^{-}_{\text{align}}.$
(10)

To illustrate, $\mathcal{L}^{+}_{\text{align}}$ is to enforce the proximity
between the visual moment prototypes $\hat{M}^{b+}$ and semantic moment
prototypes $\hat{S}^{b+}$ whereas $\mathcal{L}^{-}_{\text{align}}$ is to
assign the moment-excluding visual representation in the dummy tokens. The
design choice for assigning such a large semantic coverage in the dummy tokens
is due to the obscureness of the opposite meaning in terms of high-dimensional
moment and sentence space. For instance, the sentence consists of words and
there exist opposite terms for every word which makes it hard to define the
opposite definition of the high-dimensional sentence. Hence, we instead learn
the dummy tokens to cover large semantics that excludes only the particular
semantics.

Then, we infer the word-clip correlation from the aligned space to derive the
guidance $G$ (Fig. 4 (c)) for the attention map in the adaptive cross-
attention layer:

$\displaystyle
G_{i,j}=\frac{\text{exp}\left(\hat{v}^{+}_{i}\odot{[\hat{Q};\hat{D}]}_{j}\right)}{\sum_{k=1}^{L_{q}+L_{d}}{\text{exp}\left(\hat{v}^{+}_{i}\odot{[\hat{Q};\hat{D}]}_{k}\right)}},$
(11)

where $\hat{v}^{+}_{i}$ is $i$-th video clip of $\hat{V}^{+}$. Note that we
only derive the guidance for positive clips, i.e., clips belonging to the GT
moments, since the model has not learned the positive textual relationships
for non-moment clips which can make it susceptible to inaccuracies. Finally,
we provide clip-word level guidance to the attention map in the cross-
attention layer. Given the weights in the attention map $W$ defined in Eq. 1
and the guidance $G$ defined in Eq. 11, distillation loss
$\mathcal{L}_{\text{distill}}$ is expressed as:

$\displaystyle\mathcal{L}_{\text{distill}}=\frac{1}{L_{v}}\sum_{i=1}^{L_{v}}\sum^{L_{q}+L_{d}}_{j=1}\mathbbm{1}_{{v_{i}}\in{V^{+}}}W_{i,j}~{}\text{log}\frac{W_{i,j}}{G_{i,j}}.$
(12)

Via establishing the shared semantic space at the video-sentence level and
distilling the inferred clip-word correlation, we expect interpretable
attention layers that also take the fine-grained semantics into account.

###  3.4 Moment-adaptive Saliency Detector

We have discussed ways to calibrate the magnitude of cross-modal interaction
concerning the correlation between video and text modalities. Here, we
introduce the novel design of the saliency detector that exploits the
calibrated degree of cross-modal interaction in each clip representation.

Figure 5:  Saliency token generation. The saliency token is obtained by
combining video-averaged context with a moment-descriptive token which is
calculated by aggregating top-K moment-descriptive candidates. Specifically,
we yield a moment-descriptive token by subtracting a context token from clip
tokens and then, use their correlation to saliency candidates as the moment-
descriptiveness scores after scaling with clip-wise query correspondence
$\bar{a}$.

Following [46], we employ a saliency token to estimate the saliency scores by
computing the similarity. This implies that there should be no contextual
discrepancy between a saliency token and video representations while a
saliency token also should encapsulate the dynamic characteristics of moments,
e.g., actions, within each video for discrimination.

Accordingly, we combine a video context token and a moment-descriptive token
to generate the saliency token $T$ as described in Fig. 5. We maintain the
video context by utilizing the averaged video token so-called a context token
$V_{\text{ctx}}$. Conversely, we identify a moment-descriptive token from
learnable candidate pools $P$. To effectively incorporate distinct
characteristics of varying moments using a set number of candidate tokens, we
initiate by eliminating context from the video clips. Subsequently, weighted
similarity matching is applied between $L_{v}$ clip tokens and $L_{p}$
candidates in the pool. The top-K candidates are then aggregated to construct
a moment-descriptive token based on their similarity values. Formally, the
process is expressed as:

$C_{j}=\sum^{L_{v}}_{i=1}\bar{a}_{i}\cdot\frac{\text{exp}\left((v_{i}-V_{\text{ctx}})\odot
P_{j}\right)}{\sum_{k=1}^{L_{p}}\text{exp}\left((v_{i}-V_{\text{ctx}})\odot
P_{k}\right)},$ (13)
$T=V_{\text{ctx}}+\sum_{j=1}^{L_{p}}\mathbbm{1}_{C_{j}\in\\{C_{\left(1\right)},C_{\left(2\right)},...,C_{\left(K\right)}\\}}P_{j}\odot
C_{j},$ (14)

where $C_{\left(k\right)}$ denotes the $k$-th biggest value in the candidate
weight vector $C$ and $\mathbbm{1}$ is an indicator. Note that we use clip-
wise query correspondence $\bar{a}$ from the cross-attention map as a weight
to assign greater significance to clips pertinent to the desired moment during
weighted similarity matching.

2.5 Split | test | val
---|---|---
Method | MR | HD | MR | HD
R1 | mAP | >= Very Good | R1 | mAP | >= Very Good
@0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1 | @0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1
2.5 BeautyThumb [62] | - | - | - | - | - | 14.36 | 20.88 | - | - | - | - | - | - | -
DVSE [42] | - | - | - | - | - | 18.75 | 21.79 | - | - | - | - | - | - | -
MCN [1] | 11.41 | 2.72 | 24.94 | 8.22 | 10.67 | - | - | - | - | - | - | - | - | -
CAL [12] | 25.49 | 11.54 | 23.40 | 7.65 | 9.89 | - | - | - | - | - | - | - | - | -
XML [26] | 41.83 | 30.35 | 44.63 | 31.73 | 32.14 | 34.49 | 55.25 | - | - | - | - | - | - | -
XML+[26] | 46.69 | 33.46 | 47.89 | 34.67 | 34.90 | 35.38 | 55.06 | - | - | - | - | - | - | -
Moment-DETR [27] | 52.89 | 33.02 | 54.82 | 29.40 | 30.73 | 35.69 | 55.60 | 53.94 | 34.84 | - | - | 32.20 | 35.65 | 55.55
UMT [43] | 56.23 | 41.18 | 53.38 | 37.01 | 36.12 | 38.18 | 59.99 | 60.26 | 44.26 | - | - | 38.59 | 39.85 | 64.19
QD-DETR [46] | 62.40 | 44.98 | 62.52 | 39.88 | 39.86 | 38.94 | 62.40 | 62.68 | 46.66 | 62.23 | 41.82 | 41.22 | 39.13 | 63.03
UniVGT [32] | 58.86 | 40.86 | 57.6 | 35.59 | 35.47 | 38.20 | 60.96 | 59.74 | - | - | - | 36.13 | 38.83 | 61.81
EaTR [23] | - | - | - | - | - | - | - | 61.36 | 45.79 | 61.86 | 41.91 | 41.74 | 37.15 | 58.65
CG-DETR (Ours) | 65.43 | 48.38 | 64.51 | 42.77 | 42.86 | 40.33 | 66.21 | 67.35 | 52.06 | 65.57 | 45.73 | 44.93 | 40.79 | 66.71
2.5 |  |  |  |  |  |  |  |  |  |  |  |  |  |
Table 1:  Performance comparison on QVHighlights test and val splits with the
features from Slowfast and CLIP. We calculate the average mAP score with IoU
thresholds ranging from 0.5 to 0.95 in 0.05 intervals.

2.5 Method | TACoS | Charades-STA
---|---|---
|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| mIoU |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| mIoU
2D-TAN [83] | 40.01 | 27.99 | 12.92 | 27.22 | 58.76 | 46.02 | 27.50 | 41.25
VSLNet [80] | 35.54 | 23.54 | 13.15 | 24.99 | 60.30 | 42.69 | 24.14 | 41.58
Moment-DETR [27] | 37.97 | 24.67 | 11.97 | 25.49 | 65.83 | 52.07 | 30.59 | 45.54
QD-DETR [46] | - | - | - | - | - | 57.31 | 32.55 | -
LLaViLo [44] | - | - | - | - | - | 55.72 | 33.43 | -
UniVTG [32] | 51.44 | 34.97 | 17.35 | 33.60 | 70.81 | 58.01 | 35.65 | 50.10
CG-DETR (Ours) | 52.23 | 39.61 | 22.23 | 36.48 | 70.43 | 58.44 | 36.34 | 50.13
2.5 |  |  |  |  |  |  |  |
Table 2: Results on moment retrieval datasets, i.e., TACoS and Charades-STA. Video features are extracted using Slowfast and CLIP. 2.5 Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---
2.5 SAP [9] | 27.42 | 13.36
SM-RL [66] | 24.36 | 11.17
2D-TAN [83] | 40.94 | 22.85
FVMR [14] | 42.36 | 24.14
APGN [36] | 44.23 | 25.64
SSRN [87] | 46.72 | 27.98
UMT$\dagger$ [43] | 48.31 | 29.25
QD-DETR [46] | 52.77 | 31.13
CG-DETR (Ours) | 55.22 | 34.19
2.5 |  |
Table 3: Performances on Charades-STA with VGG backbone. $\dagger$ indicates
the usage of additional audio modality.

Utilizing the generated saliency token, we carry out query aggregation via
cross-attention that shares the parameter with ACA. However, unlike the
process with video tokens, we omit dummy tokens from key and engage in cross-
attention exclusively with pure text query tokens. In this fashion, we expect
the saliency detector to be equipped with query awareness and leverage a
finely calibrated degree of query involvement in each clip token for saliency
score prediction.

###  3.5 Training Objectives

Our proposed CG-DETR operates with losses for moment retrieval and highlight
detection. Briefly, we employ L1, gIoU [52], and cross-entropy objectives for
moment retrieval and use margin ranking loss, rank contrastive loss, and
entropy loss for highlight detection. Also, as discussed in Sec. 3.2, we use
the same highlight detection objectives on $\bar{a}$, the clip-wise query
correspondence which is derived by adding the attention weights of cross-
attention layers. To abbreviate the moment retrieval, highlight detection, and
attention weights $\bar{a}$ learning objectives as $\mathcal{L}_{\text{mr}}$
and $\mathcal{L}_{\text{hl}}$, and $\mathcal{L}_{\text{attn}}$, our overall
objective can be formulated as:

$\begin{split}\mathcal{L}_{\text{obj}}=&\mathcal{L}_{\text{mr}}+\lambda_{\text{hl}}\left(\mathcal{L}_{\text{hl}}+\mathcal{L}_{\text{attn}}+\mathcal{L}_{\text{bce}}\right)+\\\
&\lambda_{\text{ortho}}\mathcal{L}_{\text{ortho}}+\lambda_{\text{align}}\mathcal{L}_{\text{align}}+\lambda_{\text{distill}}\mathcal{L}_{\text{distill}}.\end{split}$
(15)

For further details, we refer to the Appendix A.4.

##  4 Evaluation

To validate the generality of our proposed method, we conduct extensive
experiments on video grounding datasets. We use three datasets for moment
retrieval: QVHighlights [27], Charades-STA [15], and TACoS [51], and three for
highlight detection: QVHighlights, TVSum [61], and Youtube-hl [63]. For the
type of feature extractor, we follow previous works [27, 46, 23, 43] for a
fair comparison; we utilize Slowfast+CLIP [13, 50], I3D [7], and VGG [57]
networks. We refer to Appendix A.3 for dataset details and evaluation metrics.

2.5 Method | VT | VU | GA | MS | PK | PR | FM | BK | BT | DS | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
2.5 sLSTM [81] | 41.1 | 46.2 | 46.3 | 47.7 | 44.8 | 46.1 | 45.2 | 40.6 | 47.1 | 45.5 | 45.1
SG [45] | 42.3 | 47.2 | 47.5 | 48.9 | 45.6 | 47.3 | 46.4 | 41.7 | 48.3 | 46.6 | 46.2
LIM-S [68] | 55.9 | 42.9 | 61.2 | 54.0 | 60.3 | 47.5 | 43.2 | 66.3 | 69.1 | 62.6 | 56.3
Trailer [65] | 61.3 | 54.6 | 65.7 | 60.8 | 59.1 | 70.1 | 58.2 | 64.7 | 65.6 | 68.1 | 62.8
SL-Module [70] | 86.5 | 68.7 | 74.9 | 86.2 | 79.0 | 63.2 | 58.9 | 72.6 | 78.9 | 64.0 | 73.3
QD-DETR [46] | 88.2 | 87.4 | 85.6 | 85.0 | 85.8 | 86.9 | 76.4 | 91.3 | 89.2 | 73.7 | 85.0
UniVTG [32] | 83.9 | 85.1 | 89.0 | 80.1 | 84.6 | 81.4 | 70.9 | 91.7 | 73.5 | 69.3 | 81.0
MINI-Net [22]$\dagger$ | 80.6 | 68.3 | 78.2 | 81.8 | 78.1 | 65.8 | 57.8 | 75.0 | 80.2 | 65.5 | 73.2
TCG [74]$\dagger$ | 85.0 | 71.4 | 81.9 | 78.6 | 80.2 | 75.5 | 71.6 | 77.3 | 78.6 | 68.1 | 76.8
Joint-VA [2]$\dagger$ | 83.7 | 57.3 | 78.5 | 86.1 | 80.1 | 69.2 | 70.0 | 73.0 | 97.4 | 67.5 | 76.3
UMT [43]$\dagger$ | 87.5 | 81.5 | 88.2 | 78.8 | 81.4 | 87.0 | 76.0 | 86.9 | 84.4 | 79.6 | 83.1
CG-DETR (Ours) | 86.9 | 88.8 | 94.8 | 87.7 | 86.7 | 89.6 | 74.8 | 93.3 | 89.2 | 75.9 | 86.8
2.5 |  |  |  |  |  |  |  |  |  |  |

Table 4: Highlight detection results on TVsum. $\dagger$ denotes the methods
that utilize the audio modality.

###  4.1 Comparison with the State-of-the-arts

For joint moment retrieval and highlight detection tasks on QVHighlights, we
present comparisons to the state-of-the-art methods in Tab. 3.4. For a fair
comparison between methods, numbers are reported with the test and validation
splits, respectively (We leave cells blank if the results are not available).
As observed, our proposed method outperforms previous state-of-the-art methods
with notable margins, e.g., 8% boosts in mAP for MR in both splits and 8% and
12% increase in<EMAIL_ADDRESS>Our superior performance over baseline methods in both
tasks substantiates the importance of calibrating the degree of cross-modal
interactions. Specifically, we believe that the roles of the transformer
encoder and decoder become considerably streamlined as they process
discretized features based on their relevance to the given query during the
cross-modal interaction phase.

Tab. 3.4 and Tab. 3.4 present comparisons on moment retrieval datasets: TACoS
and Charades-STA. Interestingly, we observe variations in the performance gap
across different datasets compared to previous state-of-the-art methods. For
instance, while our results are notably superior on QVHighlights, the margin
is slightly reduced on TACoS and relatively small on Charades-STA. We
attribute this phenomenon to different dataset characteristics. To illustrate,
QVHighlights offers long context-rich queries, e.g., A woman in a blue coat
sits and waves to the camera as a man in a lab coat takes a sample. In
contrast, other datasets feature relatively short and simple queries, e.g., a
person sits in a chair. In short, we observe that the performance improvements
in our proposed method are more pronounced when diverse textual semantics are
provided, i.e., when more unique words with longer queries are given. For more
detailed statistics and discussions, we refer to Appendix A.2.

Results for highlight detection benchmarks are reported in Tab. 4 and Tab.
4.1. Unlike the moment predictions made with decoder outputs, saliency scores
for highlight detection are calculated with the encoder outputs. Not only does
our work achieve state-of-the-art results compared to previous methods that
utilize only the video modality but also exceeds methods that use additional
audio modality.

2.5 Method | Dog | Gym. | Par. | Ska. | Ski. | Sur. | Avg.
---|---|---|---|---|---|---|---
2.5 RRAE [73] | 49.0 | 35.0 | 50.0 | 25.0 | 22.0 | 49.0 | 38.3
GIFs [18] | 30.8 | 33.5 | 54.0 | 55.4 | 32.8 | 54.1 | 46.4
LSVM [63] | 60.0 | 41.0 | 61.0 | 62.0 | 36.0 | 61.0 | 53.6
LIM-S [68] | 57.9 | 41.7 | 67.0 | 57.8 | 48.6 | 65.1 | 56.4
SL-Module [70] | 70.8 | 53.2 | 77.2 | 72.5 | 66.1 | 76.2 | 69.3
QD-DETR [46] | 72.2 | 77.4 | 71.0 | 72.7 | 72.8 | 80.6 | 74.4
UniVTG [32] | 71.8 | 76.5 | 73.9 | 73.3 | 73.2 | 82.2 | 75.2
MINI-Net [22]$\dagger$ | 58.2 | 61.7 | 70.2 | 72.2 | 58.7 | 65.1 | 64.4
TCG [74]$\dagger$ | 55.4 | 62.7 | 70.9 | 69.1 | 60.1 | 59.8 | 63.0
Joint-VA [2]$\dagger$ | 64.5 | 71.9 | 80.8 | 62.0 | 73.2 | 78.3 | 71.8
UMT [43]$\dagger$ | 65.9 | 75.2 | 81.6 | 71.8 | 72.3 | 82.7 | 74.9
CG-DETR (Ours) | 76.3 | 76.1 | 70.0 | 76.0 | 75.1 | 81.9 | 75.9
2.5 |  |  |  |  |  |  |
Table 5: Highlight detection performances on Youtube-hl. $\dagger$ indicates
the usage of audio modality.

2.5  | ACA | CCL | MSD | MR | HD
---|---|---|---|---|---
|  |  |  | R1 | mAP | >= Very Good
|  |  |  | @0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1
(a) |  |  |  | 58.9 | 39.0 | 58.4 | 34.4 | 35.4 | 39.0 | 62.4
(b) | ✓ |  |  | 62.9 | 46.0 | 62.46 | 41.3 | 41.1 | 39.6 | 62.5
(c) | ✓ | ✓ |  | 64.2 | 47.4 | 63.7 | 43.3 | 42.0 | 40.2 | 64.8
(d) | ✓ | ✓ | ✓ | 65.3 | 50.3 | 64.2 | 43.5 | 42.8 | 40.3 | 65.9
(e) | ✓ | ✓ | ✓ | 67.4 | 52.1 | 65.6 | 45.7 | 44.9 | 40.8 | 66.7
2.5 |  |  |  |  |  |  |  |  |  |

Table 6: Component analysis on QVHighlights val split. ACA, CCL, and MSD
denote adaptive cross-attention, clip-word correlation learner, and moment-
adaptive saliency detector introduced in Sec. 3.2, 3.3, and 3.4. With all
components combined in (d), we search the hyperparameters, i.e., number of
dummy tokens and number of layers, to train our final model in (e).

2.5  | MR | HD
---|---|---
| R1 | mAP | >= Very Good
| @0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1
Adaptiveness in cross-attention layer
(I) | 59.9 | 41.9 | 58.7 | 35.9 | 36.0 | 39.1 | 63.2
(II) | 62.9 | 46.0 | 62.5 | 41.3 | 41.1 | 39.6 | 62.5
Saliency detector
(III) | 65.5 | 50.3 | 64.7 | 44.8 | 43.8 | 39.8 | 64.8
(IV) | 66.0 | 49.8 | 65.1 | 44.3 | 43.7 | 39.9 | 64.9
(V) | 67.4 | 52.1 | 65.6 | 45.7 | 44.9 | 40.8 | 66.7
2.5 |  |  |  |  |  |  |
Table 7: Ablation studies on adaptive cross attention and saliency detector.
Studies are conducted on QVHighlights val split. Figure 6: Visualization of
prediction comparisons on QVHighlights. Figure 7:  Examination of learned
correlation between video clips and text query in CG-DETR. Each row comes from
the same example. (Left) Comparison between the degrees of multimodal
interaction and GT saliency scores. (Right) Learned correspondence between
each word and video clips.

###  4.2 Ablation Studies

In Tab. 4.1, we investigate the effectiveness of each component with a lighter
version of CG-DETR. For baseline (a), we employ the DETR-based architecture
[27, 40] with a negative pair learning strategy proposed by QD-DETR [46]. Rows
(b), (c), and (d) clearly validate the benefits of each component; (b)
discloses the importance of adjusting the degree of cross-modal interaction
according to the relevance between video clip and text query, (c) further
shows the fine-grained relevance consideration beyond sentence level brings
additional benefits, and finally with the representations built on the
discriminative degree of cross-modal interaction, we find that moment-adaptive
saliency detector easily captures the highlight clips in (d). Also, we briefly
give a short review of the impact of the increased capacity of the model in
(e) following [32, 23]. For detailed examinations of model capacity w.r.t. the
model size, number of dummy tokens, and number of prompt tokens, we refer to
extensive ablation studies in Appendix A.5.

Further ablation is conducted in Tab. 4.1. To be specific, we test the
effectiveness of the adaptiveness in the cross-attention layer and validate
the design choice for the saliency detector. Comparing (I) and (II), we
observe that granting the adaptivity to manipulate the degree of cross-modal
interaction in the cross-attention layer leads to notable differences in
numbers. For the saliency detector, we compare our design choice (V) with
naïvely employing a learnable saliency token (III) [46] and using video-
averaged context as the saliency detector. The dominance of (V) over (III) and
(IV) clearly demonstrates the importance of focusing on both the context gap
and the diverse nature of video moments.

###  4.3 Analysis

To examine the cross-modal interaction, we scrutinize how each clip
corresponds to the sentence as a whole and to each word. On the left side in
Fig. 7, we find that the degree of interactions is well-calibrated so that it
resembles the saliency distribution. Subsequently, we illustrate the fine-
grained interaction degree on the right. Although the activations may not be
intuitively ranked because of unavailable supervision, we observe that the
core words such as ”food, Two” are highly activated. Also, as the end token
$[\text{ED}]$ is regarded as the prototype for the text query in CLIP training
[50], we find that $[\text{ED}]$ is usually highly activated on moment clips.

###  4.4 Qualitative Results

In Fig. 6, we plot examples of qualitative results. To compare against
previous state-of-the-art methods, we illustrate the plots with [27, 23, 32,
46]. As more accurate moment predictions against other methods are observed in
the bar plot, we believe that the plot shows the significance of scaling the
degree of clip-wise cross-modal interaction. Furthermore, the line plot
showing highlight detection predictions also verifies the effectiveness of
exploiting the scaled degree. For additional results, we refer to Appendix
A.8.

##  5 Conclusion

In this paper, we introduced CG-DETR, calibrating the query-dependency of
video representation with respect to the correlation between video clips and
text query. To enable the calibration of the degree of video-query
interaction, we first proposed an adaptive cross-attention equipped with dummy
tokens. By granting dummy tokens a role to take the portion of attention
between irrelevant video clips and the text query, we modeled the correlation-
guided feature interaction. Then, we devised a clip-word correlation learner
to discover fine-grained correlation and further calibrate the interaction at
the clip-word level. Consequently, given the video clip tokens equipped with
calibrated query-dependency, we leverage the moment adaptive saliency token to
exploit the discrepancy in the degree of interactions. With extensive
experiments and studies, CG-DETR is verified as a strong tool to associate the
desired moments with given language descriptions. We hope CG-DETR can provide
new insights in terms of modality interaction for the video grounding
community.

##  A Appendix

Figure A1: Performance and data count variation across bins of similarity
between clip-wise query correspondence (degree of cross-modal interaction) and
GT saliency scores. To calculate the similarity between clip-wise-query
correspondence and GT saliency scores, we use the cosine similarity. Whereas
the histogram displays the number of data points in each interval, the points
in the line graph indicate the average score of averaged mAP for moment
retrieval in each bin.

###  A.1 Importance of Considering the Degree of Cross-Modal Interaction

In this section, we illustrate the importance of the degree of cross-modal
interaction to support our motivation. Fig. A1 visualizes the positive
correlation between how well clip-wise degrees of cross-modal interaction
align with the saliency scores and the performance. Along the X-axis, we
indicate the range of cosine similarity that is calculated between the ground-
truth saliency scores and the degree of cross-modal interaction for each
video-text paired data. For each bin in the plot, we calculate the average mAP
score for moment retrieval and plot the number of instances that belong to
each bin. As shown with the recent work, EaTR, we observe the tendency of
performance increase in bins as the degrees of cross-modal interaction
resemble the saliency scores which indicates the video-text relevance. By
modeling the appropriate correlation during the interaction phase, it is
verified with the number of instances in each bin that our model clearly
implements the correlation-reflected interaction. Furthermore, as the
performance drastically increases along with the quality of injected
correlation, we believe that our CG-DETR is a more explainable grounding model
compared to previous frameworks.

Figure A2: Examined text query complexity of each dataset. X- and Y-axis denote the number of unique words and the average length of text queries in each dataset split, respectively. We state that the query complexity is higher when the number of unique words in each set is bigger and the average query length is longer. Dataset | Feat | Hyperparameters | Layer | Loss
---|---|---|---|---
bs | epoch | lr | lr drop | $L_{d}$ | $L_{p}$ | K | Enc. | Dec. | ACA | D.Enc. | Mom. | Sent. | $\lambda_{\text{hl}}$ | $\lambda_{\text{L1}}$ | $\lambda_{\text{gIoU}}$ | $\lambda_{\text{CE}}$ | $\lambda_{\text{ortho}}$ | $\lambda_{\text{align}}$ | $\lambda_{\text{distill}}$
QVHighlights | SF+C | 32 | 200 | 1e-4 | x | 45 | 10 | 1 | 3 | 3 | 2 | 2 | 1 | 1 | 1 | 10 | 1 | 4 | 1 | 1 | 1
Charades-STA | SF+C | 32 | 200 | 2e-4 | x | 45 | 10 | 2 | 3 | 3 | 2 | 2 | 1 | 1 | 4 | 10 | 1 | 4 | 1 | 1 | 1
Charades-STA | VGG | 16 | 200 | 2e-4 | x | 45 | 10 | 2 | 3 | 3 | 2 | 2 | 1 | 1 | 4 | 10 | 1 | 4 | 1 | 1 | 1
TACoS | SF+C | 32 | 200 | 2e-4 | x | 50 | 10 | 2 | 3 | 3 | 2 | 2 | 1 | 1 | 4 | 10 | 1 | 4 | 1 | 1 | 1
TVSum | I3D | 4 | 1000 | 1e-3 | x | 3 | 10 |  Tab. A3 | 3 | 3 | 2 | 2 | 1 | 1 | 4 | 10 | 1 | 4 | 1 | 1 | 1
Youtube-HL | SF+C | 4 | 1000 | 2e-4 | x | 1 | 10 |  Tab. A3 | 3 | 3 | 2 | 2 | 1 | 1 | 4 | 10 | 1 | 4 | 1 | 1 | 1
Table A1: Implementation details. From the left to the right, ’bs’ denotes the
batch size; ’lr’ denotes the learning rate; ’lr drop’ denotes whether we drop
the learning rate or not; $L_{d}$ denotes the number of dummy tokens; $L_{p}$
denotes the number of candidates in the moment-representative candidate pool;
K denotes the number of selected candidates to form moment-representative
saliency token; Enc. denotes the number of transformer encoder layers; Dec.
denotes the number of transformer decoder layers; ACA denotes the number of
adaptive cross attention layers; D.Enc. denotes the number of dummy encoder
layers; Mom. denotes the number of moment encoder layers; Sent. denotes the
number of sentence encoder layers.

Domains | VT | VU | GA | MS | PK | PR | FM | BK | BT | DS
---|---|---|---|---|---|---|---|---|---|---
K | 2 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 1 | 2
Table A2: Number of selected candidates to form moment-representative saliency
token for TVSum.

Domains | Dog | Gym. | Park. | Ska. | Ski. | Surf.
---|---|---|---|---|---|---
K | 1 | 1 | 2 | 2 | 1 | 2
Table A3: Number of selected candidates to form moment-representative saliency
token for Youtube-hl.

###  A.2 When does CG-DETR Benefit the Most: Query Complexity of Datasets

The query complexity of datasets can be defined by the number of semantic
words used; we intuitively believe that query complexity is high if diverse
semantic words are used and the average query length is longer. In Fig. A2, we
have plotted the text query complexity of each moment retrieval dataset.
Comparing the query complexity with the moment retrieval performances (Tab. 1,
Tab. 2, and Tab. 3 in the manuscript), we find a strong correlation between
the two as indicated in Sec. 4.1 in the manuscript; our moment retrieval
results are notable on QVHighlights (7.5%p increase in<EMAIL_ADDRESS>compared to
[32]) which possess the highest query complexity and relatively less effective
on Charades-STA (0.7%p increase in<EMAIL_ADDRESS>compared to [32]). This is because
our aim is to discover and apply coarse-to-fine correlations between text and
video modalities; when dealing with overly simplistic or excessively short
text queries containing fewer semantic components, we anticipate a decrease in
benefits. Consequently, we claim that CG-DETR benefits the most in cases where
language descriptions are semantically rich and the model can infer and
exploit diverse correlations between queries and videos.

###  A.3 Experimental Settings

####  A.3.1 Datasets

##### Moment Retrieval.

QVHighlights is relatively recently publicized dataset by [27]. Consisting of
varying lengths of moments and diverse text queries, it is a challenging and
only dataset for joint moment retrieval and highlight detection tasks.
Providing 10,310 video-text pairs, it provides a test server on Codalab to
ensure fair comparisons. Charades-STA and TACoS are datasets for moment
retrieval. Charades-STA [15] consists of 9,848 videos regarding indoor
activities with an average duration of 30 seconds. There exist a few
benchmarks depending on the type of feature extractors. Among them, we follow
[46, 43] to test with two popular backbones, i.e., Slowfast+Clip and VGG.
TACoS [51] includes videos mostly about cooking as it is derived from the MPII
Cooking Composite Activities dataset [55]. With 127 cooking videos, it has
18,818 moment-query pairs with an average duration of 287 seconds. Among them,
we use 9790 pairs for training and 4436 pairs for testing.

##### Highlight Detection.

TVSum and Youtube-HL are datasets for highlight detection. TVSUM [61], Title-
based Video Summarization dataset, composes 50 videos of various genres, e.g.,
news, documentary, and vlog. Obtained via crowdsourcing, it has 20 saliency
score annotations per video. We follow the settings in [43, 46]. YouTube
Highlights [63] is composed of 433 videos from 6 domains: dog, gymnastics,
parkour, skating, skiing, and surfing. We follow [32] for the settings, as
well as the usage of the domain name as the text query.

####  A.3.2 Evaluation Metrics

To evaluate moment retrieval, we mainly use Recall@1 (R@1) and Mean Average
Precision (mAP) with different IoU thresholds. We also report mean
Intersection over Union (mIoU) for Charades-STA, TACoS, and NLQ datasets. For
highlight detection, we use mAP for QVHighlights, TVSum, and Youtube-hl
following [43, 27, 46, 32]. HIT@1, a metric to compute the hit ratio for the
most highlighted clip, is additionally used for QVHighlights.

Enc. | Dec. | ACA | D.Enc. | Mom | Sent. | MR | HD
---|---|---|---|---|---|---|---
R1 | mAP | >= Very Good
@0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1
2 | 2 | 2 | 2 | 1 | 1 | 65.29 | 50.32 | 64.21 | 43.47 | 42.79 | 40.23 | 65.87
2 | 2 | 2 | 2 | 2 | 2 | 64.13 | 48.06 | 64.07 | 42.81 | 42.33 | 39.91 | 64.84
3 | 3 | 2 | 2 | 1 | 1 | 67.35 | 52.06 | 65.57 | 45.73 | 44.93 | 40.79 | 66.71
3 | 3 | 2 | 2 | 2 | 2 | 66.39 | 50.90 | 65.40 | 46.17 | 44.92 | 40.31 | 66.71
3 | 3 | 3 | 2 | 2 | 2 | 65.87 | 50.45 | 65.00 | 44.61 | 44.14 | 40.66 | 66.13
3 | 3 | 3 | 3 | 3 | 3 | 65.29 | 50.90 | 64.85 | 44.27 | 43.69 | 40.62 | 65.87
Table A4: Layer ablation. In inference time, only the layers for encoder,
decoder, adaptive cross-attention, and dummy encoder are used. Figure A3:
Ablation studies on hyperparameters with averaged mAP metric for moment
retrieval task.

###  A.4 Training Details

In this section, we describe the training objectives, hyperparameters, and
implementation details.

##### Training Objectives.

As elaborated in the manuscript, CG-DETR consists of two sets of objectives:
moment retrieval and highlight detection. To predict the timestamp of the
target moments, we utilize L1 and generalized IoU losses [52] with cross-
entropy (CE) loss to classify the moment queries between foreground and
background. Let us denote the ground-truth moment and binary classification
label as $m=\left(m_{c},m_{\sigma}\right)$ and $y$, and corresponding
predictions as $\bar{m}=\left(\bar{m}_{c},\bar{m}_{\sigma}\right)$ and
$\bar{y}$, respectively, then the objective is formulated as:

$\displaystyle\mathcal{L}_{\text{mr}}\\!=\\!\lambda_{\text{L1}}\mathcal{L}_{\text{L1}}(m,\\!\bar{m})\\!+\\!\lambda_{\text{gIoU}}\mathcal{L}_{\text{gIoU}}(m,\\!\bar{m})\\!+\\!\lambda_{\text{CE}}L_{\text{CE}}(y,\\!\bar{y}),$
(16)

where $\lambda_{*}$ stands for the coefficients for corresponding losses.

Margin ranking loss, rank contrastive loss, and entropy loss for negative
pairs are used for highlight detection. Among them, margin ranking loss and
rank contrastive loss share the objective that ensures the ranking of ground-
truth saliency scores is preserved in the predicted scores. Entropy loss for
negative pairs is to suppress the saliency scores of unmatched pairs. These
losses can be formulated as:

$\displaystyle\mathcal{L}_{\text{marg}}=\text{m}$
$\displaystyle\text{ax}(0,\Delta+S(v^{\text{low}})-S(v^{\text{high}}));$ (17)
$\displaystyle\mathcal{L}_{\text{rctl}}=-\sum_{r=1}^{R}$
$\displaystyle\text{log}\frac{\sum_{v\in
V_{r}^{\text{pos}}}\text{exp}(S(v)/\tau)}{\sum_{v\in(V_{r}^{\text{pos}}\cup
V_{r}^{\text{neg}})}\text{exp}(S(v)/\tau)};$ (18)
$\displaystyle\mathcal{L}_{\text{neg}}$
$\displaystyle=-\log(1-S(v^{\text{neg}})),$ (19)
$\displaystyle\mathcal{L}_{\text{hl}}$
$\displaystyle=\mathcal{L}_{\text{marg}}+\mathcal{L}_{\text{rctl}}+\mathcal{L}_{\text{neg}},$
(20)

where $\Delta$, $S(\odot)$, $V^{\text{high}}$, $v^{\text{high}}$ and
$v^{\text{low}}$ denote a margin, saliency estimation process, and video
tokens from pairs of high and low-rank clips, respectively. $\tau$ is a
temperature scaling parameter and $v^{\text{neg}}$ is video token aggregated
with unmatched text query.

Consequently, with all the above objectives and the highlight detection losses
with attention weights $\mathcal{L}_{\text{attn}}$ combined, our final
objective can be formulated as indicated in the manuscript:

$\begin{split}\mathcal{L}_{obj}=&\mathcal{L}_{\text{mr}}+\lambda_{\text{hl}}\left(\mathcal{L}_{\text{hl}}+\mathcal{L}_{\text{attn}}+\mathcal{L}_{\text{bce}}\right)+\\\
&\lambda_{\text{ortho}}\mathcal{L}_{\text{ortho}}+\lambda_{\text{align}}\mathcal{L}_{\text{align}}+\lambda_{\text{distill}}\mathcal{L}_{\text{distill}}.\end{split}$
(21)

##### Hyperparamters.

Parameters for each benchmark are enumerated in Tab. A1. Following [27, 43,
46] on QVHighlights, Charades-STA, and TVSum, we utilize slowfast [13], CLIP
[50], VGG [58], and I3D [7] backbone features. For TACoS and Youtube-HL
datasets, we follow [32] to use slowfast and CLIP. Note that we differ the
hyperparameter $K$ per domain for highlight detection datasets as different
domains are treated as different splits [32]. Details for these datasets are
enumerated in Tab. A3 and Tab. A3. For other details, we use the Adam
optimizer with a weight decay of 1e-4 for all experiments. The hidden
dimension of the transformer architecture is set to 256, and $\tau$ for rank
contrastive loss is set to 0.5 for all experiments.

##### Additional implementation details.

Following [46, 32], we also employ a negative-pair learning strategy. However,
to alleviate the risk of forming false negative pairs during training, we do
not use the video clips from the same video as a negative pair. Moreover, as
we regard negative pairs to contain any query-corresponding moments, we do not
calculate $\mathcal{L}_{\text{distill}}$ with the negative pairs. Lastly, we
also adopt modality embedding following [32].

2.5 Split | test | val
---|---|---
Method | PT | MR | HD | MR | HD
R1 | mAP | >= Very Good | R1 | mAP | >= Very Good
@0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1 | @0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1
2.5 UniVGT [32] | ✗ | 58.86 | 40.86 | 57.6 | 35.59 | 35.47 | 38.20 | 60.96 | 59.74 | - | - | - | 36.13 | 38.83 | 61.81
CG-DETR (Ours) | ✗ | 65.43 | 48.38 | 64.51 | 42.77 | 42.86 | 40.33 | 66.21 | 67.35 | 52.06 | 65.57 | 45.73 | 44.93 | 40.79 | 66.71
UniVTG [32] | ✓ | 65.43 | 50.06 | 64.06 | 45.02 | 43.63 | 40.54 | 66.28 | 68.39 | - | - | - | 45.99 | 41.25 | 67.42
CG-DETR (Ours) | ✓ | 68.48 | 53.11 | 69.40 | 49.12 | 47.97 | 40.71 | 66.60 | 68.65 | 54.39 | 69.47 | 51.06 | 49.29 | 40.55 | 66.65
2.5 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table A5: Results with large-scale pretraining on QVHighlights test and val
splits. Pretraining was conducted on Ego4D and VideoCC datasets.

2.5 Split | test | val
---|---|---
Method | MR | HD | MR | HD
R1 | mAP | >= Very Good | R1 | mAP | >= Very Good
@0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1 | @0.5 | @0.7 | @0.5 | @0.75 | Avg. | mAP | HIT@1
2.5 Convolution (Ours) | 64.33 | 46.82 | 55.55 | 38.78 | 37.25 | 40.92 | 66.15 | 66.13 | 49.1 | 55.57 | 39.61 | 38.01 | 41.23 | 66.90
DETR (Ours) | 65.43 | 48.38 | 64.51 | 42.77 | 42.86 | 40.33 | 66.21 | 67.35 | 52.06 | 65.57 | 45.73 | 44.93 | 40.79 | 66.71
2.5 |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table A6: Performance comparison between ours with convolutional decoder and
DETR on QVHighlights test and val splits.

###  A.5 Ablation Study

##### Layer.

Previous works have demonstrated the effectiveness of the model capacity for
temporal grounding [23, 32]. Following them, we also provide the performance
variation according to the model size in Tab. A4. As others also observed, we
find that the increased capacity in the mainstream, i.e., encoder and decoder,
yields higher performances. Yet, we find that the capacity increase in the
moment and sentence encoder may result in decreased performance as these
encoders learning the alignment between clearly segmented moments and text
queries is a much easier task compared to the grounding task that requires
clip-level predictions.

##### Dummy tokens.

An ablation study for the number of dummy tokens can be found in Fig. A3 (a).
Simply put, as the increased number of dummy tokens can express more
complicated patterns, we find that there is a tendency for a performance boost
with the increased number of dummies. Yet, we also find that the number of
dummies should be determined according to the complexity of patterns in each
dataset as too large a number of dummies may disturb the training. For
example, in the case of youtube-hl, we empirically find that setting $L_{d}$
to 1 performs the best since not only examples in each domain are domain-
specific but also do not require fine-grained discrimination with complex
patterns.

##### Candidate pool for moment-adaptive saliency token.

For the candidate pool, we have two parameters; $L_{p}$ is the number of token
candidates in the pool and $K$ is the number of selected candidates. We
conduct an ablation study for these parameters in Fig. A3 (b) and (c).
Generally, our model is not very vulnerable to the size of the candidate pool
and the number of selected candidates. Furthermore, we observe that using the
maximum combinations with two tokens out of ten candidates can decently
express the diverse characteristics of moment clips in the $L_{p}$ and $K$
columns in Tab. A1. This validates the idea that matching clips without
context helps to efficiently cover varying properties of moment clips with the
limited size of the candidate pool.

###  A.6 Large-scale Pretraining

Recently, as many works have observed the effectiveness of pretraining
strategies, a lot of attention is being paid to pretraining strategies. In
this subsection, we scrutinize whether our CG-DETR benefits from the
pretraining. Particularly, we adopt the recent pretraining technique for
temporal grounding [32]. We use Ego4D [17], VideoCC [47] datasets with unified
annotations provided by UniVTG 111https://github.com/showlab/UniVTG.
Pretraining was conducted on 8 NVIDIA Tesla A100 GPUs.

Results are reported in Tab. A.4. As can be seen, our proposed method gains
superior performances after pretraining. Compared to the very recent baseline,
CG-DETR achieves 10% and 7.2% performance gains with the mAP metric for moment
retrieval. Furthermore, ours also shows relatively fewer symptoms of
overfitting whereas the performances of the baseline differ in test and val
splits.

###  A.7 Decoder for Moment Retrieval: Convolution v.s. DETR.

2.5 Method |  NLQ
---|---
|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| mIoU
Conv-based (Ours) | 7.33 | 4.16 | 2.01 | 5.21
DETR-based (Ours) | 6.07 | 3.10 | 1.19 | 4.71
2.5 |  |  |  |

Table A7: Performances of ours with convolutional decoder and DETR on NLQ.
Video features are extracted using Slowfast and CLIP.

Recent temporal grounding techniques can be categorized into convolution-based
and DETR-based. In this subsection, we briefly study the benefits of two types
of architecture. In short, each type has its strengths in different
circumstances. As can be expected intuitively, the convolution-based model has
its advantage when processing the datasets with the property of locality
whereas the DETR-based is superior in making predictions for more complex
datasets. To verify the claim, we implement and compare both versions on
QVHighlights and NLQ datasets where their variations in moment lengths vary;
QVHighlights is challenging due to the large variation in the length of
moments varies from pair to pair and moment length in NLQ does not vary much.

Results are in Tab. A.4 and Tab. A.7. While the results with DETR
architectures are more powerful in more challenging QVHighlights dataset, the
convolution decoder shows its strength when locality exists in moment length.
To provide detailed statistics for the locality, the standard deviation of
moment length is 0.237 for QVHighlights whereas it is 0.046 for NLQ (We
normalized every video length to 1 and calculated the standard deviation of
portions of the moment). Results for highlight detection with QVHighlights are
comparable because they share the encoder architecture where predictions for
highlight detection are yielded.

###  A.8 Visualizations

In Fig. A4, we show additional paired plots of the learned correlation between
video clips and text queries at sentence and word levels. Fig. A5, Fig. A6,
Fig. A7 visualize additional qualitative comparisons between methods on the
QVHighlights dataset.

Figure A4: Visualizations of learned correlation between paired video clips
and text queries. Figure A5: Qualitative results. Figure A6: Qualitative
results. Figure A7: Qualitative results.

## References

  * Anne Hendricks et al. [2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5803–5812, 2017.
  * Badamdorj et al. [2021a] Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng.  Joint visual and audio learning for video highlight detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8127–8137, 2021a.
  * Badamdorj et al. [2021b] Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng.  Joint visual and audio learning for video highlight detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8127–8137, 2021b.
  * Badamdorj et al. [2022] Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng.  Contrastive learning for unsupervised video highlight detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14042–14052, 2022.
  * Barrios et al. [2023] Wayner Barrios, Mattia Soldan, Fabian Caba Heilbron, Alberto Mario Ceballos-Arroyo, and Bernard Ghanem.  Localizing moments in long video via multimodal guidance.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Cai et al. [2018] Sijia Cai, Wangmeng Zuo, Larry S Davis, and Lei Zhang.  Weakly-supervised video summarization using variational encoder-decoder and web prior.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 184–200, 2018.
  * Carreira and Zisserman [2017] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6299–6308, 2017.
  * Chen et al. [2018] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua.  Temporally grounding natural sentence in video.  In _Proceedings of the 2018 conference on empirical methods in natural language processing_ , pages 162–171, 2018.
  * Chen and Jiang [2019] Shaoxiang Chen and Yu-Gang Jiang.  Semantic proposal for activity localization in videos via sentence query.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019.
  * Chen et al. [2020a] Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang.  Learning modality interaction for temporal sentence localization and event captioning in videos.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 333–351. Springer, 2020a.
  * Chen et al. [2020b] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020b.
  * Escorcia et al. [2019] Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell.  Temporal localization of moments in video collections with natural language.  _arXiv preprint arXiv:1907.12763_ , 2019.
  * Feichtenhofer et al. [2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6202–6211, 2019.
  * Gao and Xu [2021] Junyu Gao and Changsheng Xu.  Fast video moment retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1523–1532, 2021.
  * Gao et al. [2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  Tall: Temporal activity localization via language query.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5267–5275, 2017.
  * Ge et al. [2019] Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia.  Mac: Mining activity concepts for language-based temporal localization.  In _2019 IEEE winter conference on applications of computer vision (WACV)_ , pages 245–253. IEEE, 2019.
  * Grauman et al. [2022] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18995–19012, 2022.
  * Gygli et al. [2016a] Michael Gygli, Yale Song, and Liangliang Cao.  Video2gif: Automatic generation of animated gifs from video.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1001–1009, 2016a.
  * Gygli et al. [2016b] Michael Gygli, Yale Song, and Liangliang Cao.  Video2gif: Automatic generation of animated gifs from video.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1001–1009, 2016b.
  * Hendricks et al. [2018] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with temporal language.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1380–1390. Association for Computational Linguistics, 2018.
  * Hong et al. [2020a] Fa-Ting Hong, Xuanteng Huang, Wei-Hong Li, and Wei-Shi Zheng.  Mini-net: Multiple instance ranking network for video highlight detection.  In _European Conference on Computer Vision_ , pages 345–360. Springer, 2020a.
  * Hong et al. [2020b] Fa-Ting Hong, Xuanteng Huang, Wei-Hong Li, and Wei-Shi Zheng.  Mini-net: Multiple instance ranking network for video highlight detection.  In _European Conference on Computer Vision_ , pages 345–360. Springer, 2020b.
  * Jang et al. [2023] Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn.  Knowing where to focus: Event-aware transformer for video grounding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Khosla et al. [2013] Aditya Khosla, Raffay Hamid, Chih-Jen Lin, and Neel Sundaresan.  Large-scale video summarization using web-image priors.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2698–2705, 2013.
  * Lei et al. [2020] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal.  Tvr: A large-scale dataset for video-subtitle moment retrieval.  In _European Conference on Computer Vision_ , pages 447–463. Springer, 2020.
  * Lei et al. [2021] Jie Lei, Tamara L Berg, and Mohit Bansal.  Detecting moments and highlights in videos via natural language queries.  _Advances in Neural Information Processing Systems_ , 34:11846–11858, 2021.
  * Li et al. [2017] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten.  Learning visual n-grams from web data.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4183–4192, 2017.
  * Li et al. [2022a] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022a.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Li et al. [2022b] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10965–10975, 2022b.
  * Lin et al. [2023] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou.  Univtg: Towards unified video-language temporal grounding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Lin et al. [2020] Zhijie Lin, Zhou Zhao, Zhu Zhang, Zijian Zhang, and Deng Cai.  Moment retrieval via cross-modal interaction networks with query reconstruction.  _IEEE Transactions on Image Processing_ , 29:3750–3762, 2020.
  * Liu et al. [2018a] Bingbin Liu, Serena Yeung, Edward Chou, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles.  Temporal modular networks for retrieving complex compositional activities in videos.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 552–568, 2018a.
  * Liu et al. [2020] Daizong Liu, Xiaoye Qu, Xiao-Yang Liu, Jianfeng Dong, Pan Zhou, and Zichuan Xu.  Jointly cross-and self-modal graph attention network for query-based moment localization.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 4070–4078, 2020.
  * Liu et al. [2021a] Daizong Liu, Xiaoye Qu, Jianfeng Dong, and Pan Zhou.  Adaptive proposal generation network for temporal sentence localization in videos.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9292–9301, 2021a.
  * Liu et al. [2021b] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie.  Context-aware biaffine localizing network for temporal sentence grounding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11235–11244, 2021b.
  * Liu et al. [2018b] Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua.  Attentive moment retrieval in videos.  In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pages 15–24, 2018b.
  * Liu et al. [2023a] Meng Liu, Liqiang Nie, Yunxiao Wang, Meng Wang, and Yong Rui.  A survey on video moment localization.  _ACM Computing Surveys_ , pages 1–37, 2023a.
  * Liu et al. [2022a] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  DAB-DETR: Dynamic anchor boxes are better queries for DETR.  In _International Conference on Learning Representations_ , 2022a.
  * Liu et al. [2023b] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023b.
  * Liu et al. [2015] Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo.  Multi-task deep visual-semantic embedding for video thumbnail selection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3707–3715, 2015.
  * Liu et al. [2022b] Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie.  Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3042–3051, 2022b.
  * Ma et al. [2023] Kaijing Ma, Xianghao Zang, Zerun Feng, Han Fang, Chao Ban, Yuhan Wei, Zhongjiang He, Yongxiang Li, and Hao Sun.  Llavilo: Boosting video moment retrieval via adapter-based multimodal modeling.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2798–2803, 2023.
  * Mahasseni et al. [2017] Behrooz Mahasseni, Michael Lam, and Sinisa Todorovic.  Unsupervised video summarization with adversarial lstm networks.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 202–211, 2017.
  * Moon et al. [2023] WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo.  Query-dependent video representation for moment retrieval and highlight detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23023–23033, 2023.
  * Nagrani et al. [2022] Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid.  Learning audio-video modalities from image captions.  In _European Conference on Computer Vision_ , pages 407–426. Springer, 2022.
  * Nan et al. [2021] Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu.  Interventional video grounding with dual contrastive learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2765–2775, 2021.
  * Panda et al. [2017] Rameswar Panda, Abir Das, Ziyan Wu, Jan Ernst, and Amit K Roy-Chowdhury.  Weakly supervised summarization of web videos.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 3657–3666, 2017.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Regneri et al. [2013] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  _Transactions of the Association for Computational Linguistics_ , 1:25–36, 2013.
  * Rezatofighi et al. [2019] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 658–666, 2019.
  * Rochan et al. [2018] Mrigank Rochan, Linwei Ye, and Yang Wang.  Video summarization using fully convolutional sequence networks.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 347–363, 2018.
  * Rochan et al. [2020] Mrigank Rochan, Mahesh Kumar Krishna Reddy, Linwei Ye, and Yang Wang.  Adaptive video highlight detection by learning from user history.  In _European conference on computer vision_ , pages 261–278. Springer, 2020.
  * Rohrbach et al. [2012] Marcus Rohrbach, Michaela Regneri, Mykhaylo Andriluka, Sikandar Amin, Manfred Pinkal, and Bernt Schiele.  Script data for attribute-based recognition of composite activities.  In _Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part I 12_ , pages 144–157. Springer, 2012.
  * Shao et al. [2018] Dian Shao, Yu Xiong, Yue Zhao, Qingqiu Huang, Yu Qiao, and Dahua Lin.  Find and focus: Retrieve and localize video events with natural language queries.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 200–216, 2018.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _3rd International Conference on Learning Representations (ICLR 2015)_ , 2015.
  * Soldan et al. [2021] Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem.  Vlg-net: Video-language graph matching network for video grounding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3224–3234, 2021.
  * Song and Choi [2021] Juyong Song and Sunghyun Choi.  Image-text alignment using adaptive cross-attention with transformer encoder for scene graphs.  2021\.
  * Song et al. [2015] Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes.  Tvsum: Summarizing web videos using titles.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5179–5187, 2015.
  * Song et al. [2016] Yale Song, Miriam Redi, Jordi Vallmitjana, and Alejandro Jaimes.  To click or not to click: Automatic selection of beautiful thumbnails from videos.  In _Proceedings of the 25th ACM international on conference on information and knowledge management_ , pages 659–668, 2016.
  * Sun et al. [2014] Min Sun, Ali Farhadi, and Steve Seitz.  Ranking domain-specific highlights by analyzing edited videos.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13_ , pages 787–802. Springer, 2014.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2020] Lezi Wang, Dong Liu, Rohit Puri, and Dimitris N Metaxas.  Learning trailer moments in full-length movies with co-contrastive attention.  In _European Conference on Computer Vision_ , pages 300–316. Springer, 2020.
  * Wang et al. [2019] Weining Wang, Yan Huang, and Liang Wang.  Language-driven temporal activity localization: A semantic matching reinforcement learning model.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 334–343, 2019.
  * Xiao et al. [2021] Shaoning Xiao, Long Chen, Songyang Zhang, Wei Ji, Jian Shao, Lu Ye, and Jun Xiao.  Boundary proposal network for two-stage natural language video localization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2986–2994, 2021.
  * Xiong et al. [2019] Bo Xiong, Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman.  Less is more: Learning highlight detection from video duration.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1258–1267, 2019.
  * Xu et al. [2019] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko.  Multilevel language and vision integration for text-to-clip retrieval.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 9062–9069, 2019.
  * Xu et al. [2021] Minghao Xu, Hang Wang, Bingbing Ni, Riheng Zhu, Zhenbang Sun, and Changhu Wang.  Cross-category video highlight detection via set-based learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7970–7979, 2021.
  * Xu et al. [2023] Yifang Xu, Yunzhuo Sun, Yang Li, Yilei Shi, Xiaoxiang Zhu, and Sidan Du.  Mh-detr: Video moment and highlight detection with cross-modal transformer.  _arXiv preprint arXiv:2305.00355_ , 2023.
  * Yan et al. [2023] Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid.  Unloc: A unified framework for video localization tasks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Yang et al. [2015] Huan Yang, Baoyuan Wang, Stephen Lin, David Wipf, Minyi Guo, and Baining Guo.  Unsupervised extraction of video highlights via robust recurrent auto-encoders.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4633–4641, 2015.
  * Ye et al. [2021] Qinghao Ye, Xiyue Shen, Yuan Gao, Zirui Wang, Qi Bi, Ping Li, and Guang Yang.  Temporal cue guided video highlight detection with low-rank audio-visual fusion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7950–7959, 2021.
  * Yu et al. [2022] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  _Transactions on Machine Learning Research_ , 2022.
  * Yu et al. [2023] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal.  Self-chained image-language model for video localization and question answering.  _Advances in Neural Information Processing Systems_ , 2023.
  * Yuan et al. [2019] Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu.  Semantic conditioned dynamic modulation for temporal sentence grounding in videos.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Zeng et al. [2020] Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan.  Dense regression network for video grounding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10287–10296, 2020.
  * Zhang et al. [2019a] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis.  Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1247–1257, 2019a.
  * Zhang et al. [2020a] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou.  Span-based localizing network for natural language video localization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6543–6554, Online, 2020a. Association for Computational Linguistics.
  * Zhang et al. [2016] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Video summarization with long short-term memory.  In _European conference on computer vision_ , pages 766–782. Springer, 2016.
  * Zhang et al. [2019b] Songyang Zhang, Jinsong Su, and Jiebo Luo.  Exploiting temporal relationships in video moment localization with natural language.  In _Proceedings of the 27th ACM International Conference on Multimedia_ , pages 1230–1238, 2019b.
  * Zhang et al. [2020b] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.  Learning 2d temporal adjacent networks for moment localization with natural language.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 12870–12877, 2020b.
  * Zhang et al. [2020c] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.  Learning 2d temporal adjacent networks for moment localization with natural language.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 12870–12877, 2020c.
  * Zhang et al. [2019c] Zhu Zhang, Zhijie Lin, Zhou Zhao, and Zhenxin Xiao.  Cross-modal interaction networks for query-based moment retrieval in videos.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 655–664, 2019c.
  * Zhijian et al. [2021] Hou Zhijian, Ngo Chong-Wah, and Wing-Kwong Chan.  Conquer: Contextual query-aware ranking for video corpus moment retrieval.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021.
  * Zhu et al. [2022a] Jiahao Zhu, Daizong Liu, Pan Zhou, Xing Di, Yu Cheng, Song Yang, Wenzheng Xu, Zichuan Xu, Yao Wan, Lichao Sun, et al.  Rethinking the video sampling and reasoning strategies for temporal sentence grounding.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 590–600, 2022a.
  * Zhu et al. [2022b] Jiahao Zhu, Daizong Liu, Pan Zhou, Xing Di, Yu Cheng, Song Yang, Wenzheng Xu, Zichuan Xu, Yao Wan, Lichao Sun, et al.  Rethinking the video sampling and reasoning strategies for temporal sentence grounding.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , 2022b.