# RGNet: A Unified Retrieval and Grounding Network for Long Videos

Tanveer Hannan1,2 Md Mohaiminul Islam3 Thomas Seidl1,2 Gedas Bertasius3
1 LMU Munich 2 MCML 3 UNC Chapel Hill
Corresponding author<EMAIL_ADDRESS>
###### Abstract

We present a novel end-to-end method for long-form video temporal grounding to
locate specific moments described by natural language queries. Prior long-
video methods for this task typically contain two stages: proposal selection
and grounding regression. However, the proposal selection of these methods is
disjoint from the grounding network and is not trained end-to-end, which
limits the effectiveness of these methods. Moreover, these methods operate
uniformly over the entire temporal window, which is suboptimal given redundant
and irrelevant features in long videos. In contrast to these prior approaches,
we introduce RGNet, a unified network designed for jointly selecting proposals
from hour-long videos and locating moments specified by natural language
queries within them. To achieve this, we redefine proposal selection as a
video-text retrieval task, i.e., retrieving the correct candidate videos given
a text query. The core component of RGNet is a unified cross-modal RG-Encoder
that bridges the two stages with shared features and mutual optimization. The
encoder strategically focuses on relevant time frames using a sparse sampling
technique. RGNet outperforms previous methods, demonstrating state-of-the-art
performance on long video temporal grounding datasets MAD and Ego4D. The code
is released at https://github.com/Tanveer81/RGNet.

##  1 Introduction

The exponential rise in online videos has created the demand for efficient
search engines. The search results of user queries, particularly in long
videos, often require finding specific moments. Most existing solutions,
tailored for short videos (5-30 seconds in length) [58], struggle when applied
to hour-long videos, slowing down searches and impacting the quality of the
search results [44]. This challenge emerges because it is very difficult to
locate short moments in a long video based on text queries, a task known as
Long Video Temporal Grounding (LVTG) [14, 44].

Figure 1: The Long Video Temporal Grounding (LVTG) task requires detecting the
precise moment boundary based on a natural language query within a long video
(several hours or longer). Traditional methods involve a separate proposal
retriever and grounding network, which are usually trained disjointly, thus
limiting the effectiveness of these approaches. In contrast, we propose RGNet,
an end-to-end trainable architecture that unifies the two stages with shared
features and mutual optimization.

A straightforward solution is to divide the video into shorter clips and
predict moments individually from each clip [59, 56, 43]. However, handling
all the clips is computationally intensive, resulting in numerous proposed
moments and making the task even more challenging. In contrast, several
proposal-based methods [16, 38] apply their grounding model only to a selected
set of proposal clips, offering a more efficient solution while demonstrating
better performance. However, these models have a separate module for selecting
the proposals, which is disjoint from the grounding network. The separate
optimization of these two modules leads to suboptimal results and complicates
the integration of the two training stages. Additionally, it results in
redundant computations, as both stages necessitate video-text modeling. Since
both stages require a cross-modal similarity function between the text query
and the proposal clips, we believe that unifying the cross-modal modeling of
both stages can facilitate effective end-to-end optimization and enhance
moment detection with shared features.

Recently, there has been a growing interest in detection transformers [3, 28]
for grounding tasks [23]. Hence, we could unify the stages with vanilla
transformers on the combined clip and text features [23]. However, such a
framework encounters challenges in handling the extensive number of frames and
textual content using self-attention mechanisms. Alternatively, more effective
techniques involve processing the two modalities into distinct phases [29,
36]. Nevertheless, these techniques are limited to moment-detection in short
videos. Moreover, most of these transformer-based approaches have been
predominantly crafted for uni-modal tasks, such as object detection in images.
Adapting these models for multi-modal tasks like moment detection in videos
necessitates further exploration. These limitations underscore the need to
develop an effective multi-modal detection transformer tailored explicitly for
long video temporal grounding.

Additionally, prior works on video-text retrieval [13, 24, 30] demonstrate the
significance of sampling video frames highly correlated with the text.
However, video retrieval tasks generally lack supervision for explicitly
learning such a sampler. In contrast, grounding datasets offer temporal
boundaries for the moments, which could enable the optimization of the
sampling process for the proposal retriever. Based on these observations, we
postulate that shared features and mutual optimization could enhance both the
proposal generation and the grounding phases.

Leveraging the synergy between video-text retrieval and grounding techniques,
we propose RGNet, A Unified Retrieval and Grounding Network for Long Videos.
We reformulate proposal generation as a video-text retrieval task and design a
Unified Retrieval and Grounding Encoder (RG-Encoder) that simultaneously
retrieves proposals and provides cross-modal features for grounding.
Consequently, the encoder unifies the cross-modal modeling for both stages, as
illustrated in Fig. 1. Additionally, based on moment supervision, the encoder
is trained end-to-end with a grounding decoder. We apply intra-proposal
sampling loss, which explicitly trains a sparse sampling module to focus on
relevant video frames for the retrieval phase. An inter-proposal contrastive
loss further aligns the textual and visual information. The proposed encoder,
along with the losses, facilitates shared features and the mutual optimization
of both stages. Together, these components of RGNet bridge the gap between the
two stages of Long Video Temporal Grounding (LVTG), leading to improved
performance.

Specifically, our proposed RGNet achieves state-of-the-art performance across
both LVTG datasets, Ego4D [14] and MAD [44]. Compared to the previous best
Ego4D baseline[42], RGNet improves the primary evaluation metric, the<EMAIL_ADDRESS>and<EMAIL_ADDRESS>score average by 9.6%. In the case of the MAD dataset, RGNet
outperforms the<EMAIL_ADDRESS>and<EMAIL_ADDRESS>scores of the recently proposed SooNet [38]
by 1.2% and 1.9%.

##  2 Related Literature

#### Long Video Temporal Grounding.

Recently, video temporal grounding has been adapted for long videos with MAD
[44] and Ego4D [14] datasets. Several large-scale models [40, 8, 41, 25, 60]
had been trained to extract better features from these long videos. Typically,
the methods designed for long videos involve two stages. Proposal-free methods
[59, 56, 43, 2, 27] segment lengthy videos into smaller parts to predict
candidate moments and then rank them to obtain the final predictions.
Proposal-based methods [38, 16] generate proposal clips or anchors and
subsequently apply their grounding model to the retrieved proposals. These
proposal-based models are more efficient and perform better than their
counterparts. However, the proposal retrieval phase of these methods is
typically disjoint from their grounding network and trained independently. In
contrast, our approach unifies the proposal generation and grounding phases
within a single network, training them end-to-end.

#### Transformers for Grounding.

Transformers [3, 61, 35, 28] have demonstrated impressive performance in
unimodal tasks such as object detection [57] and tracking [22]. This success
has led to their adoption to the video temporal grounding tasks. Moment-DETR
[23] pioneered this approach by applying transformers to combined video and
text features. Subsequent works, such as UMT [29] and QD-DETR [36], split the
cross-modal and unimodal modeling of video and text features for enhanced
performance. EATR [18] improves its decoder with video-specific moment
queries. Some methods, like CONE [16] and M-Guidance [2], leverage the
standard detection transformer for long videos. However, these transformers
were originally designed for object detection, and subsequent research focused
mostly on the decoder query design. In contrast, we redesigned its encoder to
fuse the video-text features effectively to facilitate moment detection.

#### Video-text Retrieval.

The early retrieval methods [4, 6, 21, 53, 52] designed multimodal fusion
techniques to align pre-trained video and text models. To bridge the gap
between pre-training and downstream tasks, large-scale pre-trained video-
language representations [1, 9, 11, 5, 32, 45, 48, 49, 50] have been
introduced in various works. Several studies [7, 10, 19, 20, 33, 46, 51, 54,
47] have transferred the knowledge from CLIP [41] to text-video retrieval
tasks. In recent studies, various sampling strategies [24, 13, 30] have been
explored, enabling the model to selectively concentrate on pertinent video
frames based on provided text inputs. For instance, Clip-BERT [24] introduces
a sparse sampling strategy, X-Pool [13] utilizes a cross-modal attention
model, and TS2-Net [30] adapts CLIP to capture temporal information and
eliminate unimportant tokens. These methods implicitly incorporate the
sampling method due to the absence of moment supervision. In contrast, our
approach trains a sampling mechanism using the precise moment boundaries
present in LVTG datasets.

##  3 RGNet

This section presents a detailed description of our proposed unified LVTG
method. As illustrated in Fig. 2, RGNet takes a long video and a text query as
input and predicts the precise moment semantically correlated with the query
in an end-to-end manner. The whole network is divided into the proposed RG-
Encoder (Fig. 3) and a grounding decoder (Sec. 3.4). We uniformly divide the
long video into shorter fixed-size proposal candidates from which the encoder
retrieves top-k proposals. The decoder then processes the selected proposals
to predict the precise moment boundary. Additionally, an Intra-Proposal
Sampling Loss and an Inter-Proposal Contrastive Loss are incorporated to
facilitate the end-to-end training of both stages.

###  3.1 Feature Extraction

Let us assume we have an untrimmed long video $V$ consisting of $T$ frames and
a query sentence $S$ with $N$ words corresponding to a target moment’s center
$\tau_{c}$ and width $\tau_{w}$. Following previous methods [16, 38], we use
pre-trained frozen models to extract visual features
${V}=\\{{f}^{1},{f}^{2},...,{f}^{T}\\}\in{R}^{T\times D_{f}}$ as well as
textual features ${S}=\\{{w}^{1},{w}^{2},...,{w}^{N}\\}\in{R}^{N\times
D_{w}}$, where $D_{f}$ and $D_{w}$ represent the feature dimensions of video
frames and query words. We slice the long video $V$ into proposal candidates
of length $L_{c}$ and feed them into our model. We employ a sliding window of
length $L_{c}$ and stride $L_{c}/2$ to obtain a collection of $T_{c}$
candidates ${C}=\\{{C}^{1},{C}^{2},...,{C}^{T_{c}}\\}\in{R}^{T_{c}\times
L_{c}\times D_{f}}$. Here, the $i^{th}$ candidate,
${C}^{i}=\\{{f}^{s_{i}+1},{f}^{s_{i}+2},...,{f}^{s_{i}+L_{c}}\\}\in{R}^{L_{c}\times
D_{f}}$, where $s_{i}$ is the start index of the proposal candidate.

Figure 2: Overview of RGNet. To begin, we uniformly split the video into
fixed-size proposal candidates, serving as the input for RG-Encoder. This
encoder extracts proposal features and determines their relevancy score. The
top-k proposals are then retrieved based on these scores. For brevity, we
visualize only the top-$1$ retrieved proposal. Subsequently, the grounding
decoder processes the selected proposal features to predict the precise moment
boundary.

###  3.2 RG-Encoder

All the proposal candidates ${C}$ from a single video comprise instances in a
single batch. The RG-Encoder (Fig. 3) processes them with the query ${S}$ to
produce the encoded proposals $P$. First, the cross-attention generates query-
conditioned frame features ${F}$. Then, based on the frame-query correlation
calculated from $F$, the sampler determines the relevant frames inside the
proposals to produce a mask ${M}$ for subsequent retrieval attention. We
incorporate a learnable token ${R}$ to capture the context of the proposals.
Additionally, the frame features ${F}$ undergo restricted attention based on
the predicted mask $M$ to generate the content feature $F_{c}$. The
combination of context $R$ and content $F_{c}$ forms the encoded proposal $P$.

#### Cross Attention:

To evaluate the relevance of each frame within the proposal $C^{i}$, we
utilize cross-attention, where frame features act as queries and text features
serve as keys and values. Cross-attention represents the frame features as a
weighted average of the textual features, scaled by their mutual correlation.
This process results in text-conditioned frame features $F^{i}$, establishing
a fine-grained correspondence between the modalities. The query, key and
values are calculated as ${Q}^{i}=f_{Q}({C}^{i})\in{R}^{L_{c}\times D_{f}}$,
${K}=f_{K}({S})\in{R}^{N\times D_{q}}$ and ${V}=f_{V}({S})\in{R}^{N\times
D_{q}}$ respectively. Here, $f_{Q}(\cdot)$, $f_{K}(\cdot)$, and $f_{V}(\cdot)$
denote the projection layers for the query, key, and value. Then, we apply the
cross-attention layer as follows:

$\begin{gathered}F^{i}=\operatorname{softmax}({{Q}}^{i}{K}^{T}){V}+{Q}^{i}\\\
\end{gathered}$ (1)

#### Sampler:

To retrieve proposals, we need to assess the relevance of each proposal based
on the input query. Proposal relevance is determined by aggregating the
relevancy of its frames. However, given the small moment duration, most frames
are unrelated to the query. Hence, we classify frames into relevant and non-
relevant categories using the sampler to assist our encoder in focusing on
these pertinent regions. We calculate the relevancy ${G}^{j}\in[0,1]$ of the
${j}^{th}$ frame with Eq. 2. We use a differentiable softmax function [15, 17]
for enabling end-to-end training.

${G}^{j}=\operatorname{softmax}(\text{linear}({f}^{j}))$ (2)

The sampling loss optimizes this classification using the ground truth frame
relevancy information described in Sec. 3.5. To distinguish irrelevant frames
from their counterparts, we constrain their updates in the retrieval
attention. This restriction is implemented with an attention mask computed
from the learned frame relevancy ${G}^{j}$. The attention mask ${M}$ between
$j^{{th}}$ and $k^{{th}}$ frame is calculated with Eq. 3. Here, the $j^{th}$
and $k^{th}$ frames are the attention query and key, respectively.

${M}(j,k)=\left\\{\begin{array}[]{ll}0&\text{ if }{G}^{j}>0.5\hskip
5.0pt\text{or}\hskip 5.0ptj=k\\\ -\infty&\text{ otherwise }\end{array}\right.$
(3)

#### Retrieval Attention

We aim for a learnable cross-modal retrieval function to train end-to-end with
the moment supervision. Thus, we devise the retrieval attention for
aggregating the text-conditioned frame features based on their relevancy. To
condense the $i^{th}$ proposal into a contextual feature embedding, we
introduce a randomly initialized learnable vector ${R}^{i}$. We call this
retrieval token, which summarizes the entire proposal and is trained with an
inter-proposal contrastive loss. First, we concatenate the retrieval token
with the text-conditioned frame features to create the query
${Q}^{i}=[{R}^{i},{F}^{i}]$ for retrieval attention. We also generate an
attention mask containing all zeros for the retrieval token and concatenate it
with ${M}$. We set the key ${K}={Q}^{i}$, and the value ${V}={Q}^{i}$ to
calculate the retrieval attention according to Eq. 4.

$\begin{gathered}{Q}^{i}=\operatorname{softmax}({Q}^{i}{K}^{T}+{M}){V}+{Q}^{i}\\\
\end{gathered}$ (4)

We get the proposal context feature, ${R}^{i}={F}^{i,1}$ and frame-level
content features, ${F_{c}}_{i,j}=[{F}^{i,2},{F}^{i,3},...,{F}^{i,L_{c}+1}]$.
Here, ${F_{c}}^{i,j}$ is the j$th$ frame feature of the i$th$ proposal. The
combined context and content features create the encoded proposals as follows:

${P}^{i,j}={F_{c}}^{i,j}+{R}^{i}\times{G}^{j},\hskip 1.0pti\in P,\forall j\\\
$ (5)

Figure 3: Overview of the RG-Encoder It initiates with a cross-attention
mechanism that computes the frame-query correlation. The red frames are highly
correlated. Utilizing this correlation, the sampler generates a mask for the
subsequent retrieval attention, assigning a value of 1 to the masked frames.
In this attention, we employ a learnable token to distill the proposal into
its context feature while simultaneously generating frame-level content
features. The fusion of context with content forms the encoded proposals.

###  3.3 Proposal Retrieval

The context features ${R}$, trained with the Inter-Proposal Contrastive Loss,
are instrumental in ranking and retrieving the proposals. It undergoes a
projection linear layer, $f_{s}(\cdot)$, to obtain context scores
${S_{r}}=\text{sigmoid}(f_{s}({R}))$ for each query-proposal pair. Based on
the context score ${S_{r}}$, we select the top-$k$ proposal set to be
processed in the decoder. Therefore, we limit the number of candidate
proposals from $T_{c}$ to a constant $k$ to ensure an efficient and
predictable computation cost.

###  3.4 Grounding Decoder

We process the retrieved proposals with a grounding decoder to predict precise
moment boundaries. Previous methods [23, 16] feed both modalities into the
decoder, introducing multiple positional information (e.g., time in the video
and word order in the text) that complicates the detection task. In contrast,
our encoder injects text information directly into the proposal, eliminating
the requirement to feed both modalities into the decoder. We incorporate
learnable anchor queries [28] to represent the moment’s center and width as
$(\tau_{c},\tau_{w})$.

###  3.5 Loss Functions

#### Intra-Proposal Sampling Loss

The sampling loss guides our sampler to distinguish frames within and outside
ground truth moments. Our approach computes the sampling loss using a hinge
loss between two in-moment and out-of-moment frames. Specifically, the in-
moment frames are required to maintain higher sampling scores than those
outside the ground truth moment. In short, $L_{\text{samp}}$ is defined as:

$\displaystyle\begin{split}\mathcal{L}_{\text{samp}}=&\operatorname{max}(0,\Delta+S_{c}(i,j_{\text{out}})-S_{c}(i,j_{\text{in}}))\end{split}$
(6)

In the equation, $S_{c}$ is the sampling score, $\Delta$ represents the
margin, and $j_{\text{in}}$ and $j_{\text{out}}$ are randomly chosen indexes
for in-moment and out-of-moment frames, respectively. To calculate the
sampling score, the context features ${R_{i}}$, and encoded proposal $P^{i,j}$
undergoes linear layers , $f_{r}(\cdot)$ and $f_{c}(\cdot)$, to obtain
projected context ${R}^{i}_{{proj}}=f_{r}({R^{i}})\in{R}^{D_{f}}$ and content
features, $P^{i,j}_{proj}=f_{c}(P^{i,j})\in{R}^{{L_{c}}\times{D_{f}}}$. Then,
we calculate the frame-level sampling score as,

$S_{c}(i,j)={R}^{i}_{{proj}}\cdot P^{i,j}_{{proj}}$ (7)

#### Inter-Proposal Contrastive loss

Retrieving relevant proposals from a huge candidate set requires aligning the
query-proposal pairs effectively. We learn these relationships by contrasting
the positive pairs with many negatives. In Fig. 4, the proposal from a
positive pair contains the moment defined by the text, while our negative
samples consist of other pairs from the same batch. For this section, we
denote the context feature, $R^{i,j}$ for $i^{th}$ proposal and $j^{th}$ text
query. We train our model contrastively with positive query-proposal pairs
$[R^{i,j}]_{i=j}$ and negative pairs $[R^{i,j}]_{i\neq j}$. We use InfoNCE
loss [37] to identify the positive query-proposal pair amongst a set of
unrelated negative samples:

$\mathcal{L}_{\text{cont}}=-\sum_{i}log\frac{\exp(f_{\text{cont}}({R^{i,i}}))}{\sum_{j}\exp(f_{\text{cont}}({R^{i,j}}))}$
(8)

Here, $f_{cont}(\cdot)$ is a projection layer that transforms the $D_{f}$
dimensional context feature into a one-dimensional logit.

#### Grounding loss

The objective functions for grounding, which aim to locate desired moments,
are adopted from the baseline approach [23]. The grounding loss
$\mathcal{L}_{\text{g}}$ measures the discrepancy between the ground truth
(GT) and predicted moments. A one-to-one correspondence between GT and
predicted moments is established using the Hungarian algorithm. This loss
includes both an $\mathcal{L}1$ loss and a generalized IoU loss
($\mathcal{L}{\text{gIoU}}$). Additionally, a cross-entropy loss
$\mathcal{L}_{\text{CE}}$ is employed to classify predicted moments as either
foreground or background. Thus, $\mathcal{L}_{\text{g}}$ is defined as
follows:

$\displaystyle\mathcal{L}_{\text{g}}=\lambda_{L1}||\tau-\hat{\tau}||+\lambda_{\text{gIoU}}\mathcal{L}_{\text{gIoU}}(\tau,\hat{\tau})+\lambda_{\text{CE}}\mathcal{L}_{\text{CE}},$
(9)

where $\tau$ and $\hat{\tau}$ represent the ground-truth moment and its
corresponding prediction, containing center coordinates $\tau_{c}$ and width
$\tau_{w}$. The hyper-parameters $\lambda_{*}$ are used for balancing the
losses. Finally, with the sampling and contrastive loss, the total loss
$\mathcal{L}_{\text{total}}$ is defined as follows:

$\displaystyle\mathcal{L}_{\text{total}}=\lambda_{\text{samp}}\mathcal{L}_{\text{samp}}+\lambda_{\text{cont}}\mathcal{L}_{\text{cont}}+\mathcal{L}_{\text{g}}$
(10)

Figure 4: Contrastive Loss. We train our retrieval encoder using a contrastive
loss on query-proposal pairs. In contrast to previous methods [16] that use a
single negative example to learn the distinction between positive and negative
pairs, we sample negative pairs from all the videos in a batch. Sampling
multiple negative examples closely resembles the inference process, as long
videos contain numerous negative query-proposal pairs.

##  4 Experimental Setup

Model | Venue |  Ego4D-NLQ [14] |  |  MAD [44] | Avg
---|---|---|---|---|---
R1.3 | R5.3 | R1.5 | R5.5 | Avg |  | R1.1 | R5.1 | R1.3 | R5.3 | R1.5 | R5.5 | Avg
2D-TAN[59] | AAAI’20 | 5.04 | 12.89 | 2.02 | 5.88 | 6.46 |  | 3.22 | 11.90 | 2.52 | 9.25 | 1.58 | 5.69 | 5.69 | 6.07
UniVTG[26] | ICCV’23 | 11.74 | 7.54 | 3.25 | 7.88 | 7.60 |  | - | - | - | - | - | - | - | -
ReLER[27] | CVPR’22 | 10.79 | 13.19 | 6.74 | 8.85 | 9.89 |  | - | - | - | - | - | - | - | -
VLG-Net [43] | ICCV’21 | - | - | - | - | - |  | 3.64 | 11.66 | 2.76 | 9.31 | 1.65 | 5.99 | 5.84 | -
M-DETR[23] | NIPS’21 | 8.23 | 23.23 | 5.01 | 13.37 | 12.46 |  | 3.60 | 12.98 | 2.81 | 9.86 | 1.67 | 5.58 | 6.08 | 9.27
CLIP [41] | ICML’21 | - | - | - | - | - |  | 6.57 | 15.05 | 3.13 | 9.85 | 1.39 | 5.44 | 6.90 | -
M-Guide [2] | ICCV’23 | - | - | - | - | - |  | 9.30 | 18.96 | 4.65 | 13.06 | 2.16 | 7.40 | 9.26 | -
SOONet[38] | ICCV’23 | 8.00 | 22.40 | 3.76 | 11.09 | 11.31 |  | 11.26 | 23.21 | 9.00 | 19.64 | 5.32 | 13.14 | 13.59 | 12.45
H-Hands[55] | ICCV’23 | 13.20 | 23.30 | 7.90 | 15.60 | 15.00 |  | - | - | - | - | - | - | - | -
NaQ∗ [42] | CVPR’23 | 19.31 | 23.62 | 11.59 | 15.75 | 17.57 |  | - | - | - | - | - | - | - | -
CONE [16] | ACL’23 | 14.15 | 30.33 | 8.18 | 18.02 | 17.67 |  | 8.90 | 20.51 | 6.87 | 16.11 | 4.10 | 9.59 | 11.01 | 14.34
RGNet (Ours) | Review | 20.63 | 41.67 | 12.47 | 25.08 | 24.96 |  | 12.43 | 25.12 | 9.48 | 18.72 | 5.61 | 10.86 | 13.70 | 19.33
Table 1: Performance on the Ego4D and MAD dataset combined. RGNet achieves
state-of-the-art performance on both datasets. The Best score is reported in
bold text, while the Second Best is underlined. NaQ∗ is trained with the model
from ReLER.

###  4.1 Datasets

We evaluate RGNet on the recently proposed LVTG datasets, Ego4D-NLQ [14], and
MAD [44].

MAD [44] is an extensive dataset comprising 1.2K hours of full-length movies
and 384K natural language queries, each associated with specific moments in
the videos. The videos are orders of magnitude longer than previous datasets,
with an average duration of 110 minutes, while the specified text moments are,
on average, 4.1 seconds. This small moment-to-video ratio poses a significant
challenge for the grounding task.

Ego4D [14] is a large-scale egocentric video data set with multiple
challenges. Specifically, we use the episodic memory benchmark Ego4D-NLQ,
which requires localizing where the answer to a natural language query can be
seen. It contains around $13$ template questions, amounting to around 74K
queries. The train, val, and test sets contain 11.3K, 3.9K, and 4.0K queries.
The video length ranges from 8 to 20 minutes, with an average of 8.25 minutes,
and the average moment duration is 8.3 seconds. This means the moments
constitute only 2% of the input video on average.

###  4.2 Evaluation Metric

Grounding Metric: For grounding, we follow previous methods [44, 16], and
adopt the standard metric Recall@$k$ at IoU=$\theta$ (R$k_{\theta}$). This
metric represents the percentage of testing samples with at least one
grounding prediction whose intersection over union (IoU) with the ground truth
(GT) is larger than $\theta$ among the top-$k$ predictions.

Retrieval Metric: To assess our proposal retrieval stage independently of the
grounding, we utilize the standard retrieval metric [13], Recall at Rank $k$
(R@$k$). This metric calculates the percentage of GT moments present in the
top-$k$ retrieved proposals.

###  4.3 Implementation Details:

We use pre-trained CLIP [41] and EgoVLP [25] models to extract video frames
from MAD and Ego4D. Text features for both datasets are extracted using CLIP.
The feature extractors are frozen, and their pre-trained weights remain
unchanged during training. For Ego4D, we pre-train our model with NaQ
augmentation from previous work [42]. The proposal length, $W_{c}$ is set to
180s for MAD and 48s for Ego4D, with only the top-30 and top-5 proposals
retrieved, respectively. Training on Ego4D and MAD is conducted on four
Nvidia-RTX-A6000 GPUs while fine-tuning on Ego4D is performed on a single GPU.
Considering the longer proposals in MAD, we utilize a batch size of 32 for
Ego4D and 8 for MAD. The model is developed using pytorch [39], and
distributed training is implemented. The number of moment queries is set to 5.
For loss computation, we set hyperparameters as $\lambda_{L1}=10$,
$\lambda_{\text{gIoU}}=1$, $\lambda_{\text{CE}}=4$, $\lambda_{\text{samp}}=1$,
$\lambda_{\text{cont}}=10$, and $\Delta=0.2$. We use Xavier initialization
[12] and employ AdamW [31] with an initial learning rate of $1\times 10^{-4}$.
RGNet is trained for 35 and 200 epochs on MAD and Ego4D datasets. The initial
learning rate is reduced by one order of magnitude at epochs 25 for MAD and
120 for Ego4D.

##  5 Results and Analysis

Data | Model |  | Proposal Retrieval |  | Oracle Grounding
---|---|---|---|---|---
| R@1 | R@5 | R@10 | R@30 |  | R1.3 | R5.3 | R1.5 | R5.5
Ego4d | Disjoint |  | 31.71 | 64.63 | 83.31 | 98.58 |  | 29.84 | 54.01 | 17.90 | 33.42
Unified (ours) |  | 42.08 | 76.28 | 90.03 | 99.13 |  | $\mathbf{36.53}$ | $\mathbf{63.64}$ | $\mathbf{21.39}$ | $\mathbf{37.97}$
MAD | Disjoint |  | 12.41 | 24.50 | 32.90 | 50.42 |  | 29.49 | 53.02 | 17.19 | 32.58
Unified (ours) |  | 25.01 | 50.02 | 64.98 | 88.56 |  | $\mathbf{33.42}$ | $\mathbf{63.43}$ | $\mathbf{18.75}$ | $\mathbf{38.03}$
Table 2: Our unified network improves both proposal retrieval and oracle
grounding compared to the disjoint baseline. Oracle Grounding runs on the
window where the ground truth moment is present. This means that the
performance of the grounding is judged separately from the performance of the
retrieval.

We first compare our performance with previous state-of-the-art models. Then,
we report detailed ablation studies of our proposed method and visualize the
qualitative results compared to the disjoint baseline.

#### Result on the MAD Dataset

We compare our model’s performance on the MAD dataset in Tab. 1. RGNet
achieves state-of-the-art performance on both the R$1_{.1}$ and R$5_{.1}$
scores by outperforming the previous best SooNet [38] by 1.2% and 1.9%.
Together with its competitive performance on other metrics, RGNet demonstrates
the importance of end-to-end modeling of long videos. Specifically, our
performance of R$1$ is superior at all IoU thresholds, which testifies to the
effectiveness of the proposed retrieval method. Notably, prior models rely on
heuristic dot product similarity measures applied to CLIP [41] features for
the retrieval phase. In contrast, our shared retrieval feature and its unified
optimization with the grounding objective drastically enhance the first stage
selection, as seen in Tab. 2. It further helps the network capture more
discriminative features crucial for detecting extremely tiny moments in hour-
long videos.

#### Result on the Ego4D Dataset

RGNet demonstrates state-of-the-art performance on all metrics for the
Ego4D-NLQ benchmark. As shown in Table 1, RGNet improves the primary
evaluation metric, the R$1_{.3}$ and R$5_{.3}$ score average by 9.6%.
Separately, we outperform the previous best methods on R$1_{.3}$ and R$5_{.3}$
by 1.32% and 11.3%, respectively. This improvement is consistent across other
metrics as well. The unified proposal and frame-level modeling in RGNet
parallelly capture long- and short-range temporal dependencies, contributing
to its superior performance. By modeling the entire video as a whole, RGNet
learns better cross-modal alignment by contrasting visual scenarios from
various events within the same video. Notably, compared to SooNet [38] and
CONE [16], two prominent proposal-based methods, RGNet achieves an average
score improvement of $13.7\%$ and $7.3\%$. This substantial improvement is
attributed to our unified modeling of proposal retrieval, whereas they
disjointly employ a dot product similarity measure.

###  5.1 Ablation Studies

This section reports detailed ablation studies on the proposed modules and
loss functions and unveils the impact of the unified architecture
independently for proposal retrieval and grounding. We also ablate with
various proposal lengths and proposal numbers. Unless stated, the experiments
in this section are done w/o NaQ augmentation [42] for the Ego4D dataset with
CONE [16] as the baseline.
Modules: In our initial experiments, we validate the effectiveness of each
proposed module with a cumulative ablation in Tab. 3. The unified network
demonstrates superior performance across all evaluation metrics, notably
achieving an R$1_{.3}$ score of $18.28\%$. Note that the disjoint baseline
performs $4.2\%$ worse than our unified network. This performance difference
can be attributed to our encoder’s parallel proposal and frame-level temporal
modeling, which help generate better features for detecting small moments.
With the removal of the sampler, there is a decline in R$1_{.3}$ performance
of approximately $0.5\%$. This verifies the impact of the sampler and its
optimization using moment supervision. Subsequently, eliminating the context
feature of the encoder for proposal retrieval considerably drops the score by
about $1.7\%$. This proves the efficacy of our unified training and end-to-end
proposal selection. Finally, removing the RG-Encoder yields a notable
degradation of approximately $2.0\%$ in the R$1_{.3}$ score, which
substantiates our improved cross-modal encoding method.

Model | R1.3 | R5.3 | R1.5 | R5.5
---|---|---|---|---
Unified (ours) | 18.28 | 34.02 | 12.04 | 22.89
w/o Sampler | 17.80 | 33.99 | 11.25 | 22.32
w/o Proposal Ret. | 16.12 | 31.57 | 9.91 | 20.47
w/o RG-Encoder | 14.15 | 30.33 | 8.18 | 18.02
Table 3:  Cumulative ablation study for various proposed modules of our
method. The last row is the Disjoint baseline.

Proposal Retrieval: Retrieving proposals where the ground truth moment exists
is inherently crucial, independent of the subsequent moment localization
accuracy. Regardless of the effectiveness of moment boundary detection, if the
proposal containing the moment boundary cannot be accurately retrieved, the
overall performance is likely to suffer significantly. As stated in Tab. 2,
our unified method consistently improves all retrieval metrics compared to the
disjoint baseline. Notably, we achieve a staggering $10.4\%$ and $12.6\%$
improved R@1 score for Ego4D and MAD.
Oracle Grounding: Since the ultimate Long Video Temporal Grounding (LVTG)
performance relies on the first stage of proposal retrieval, it may not
perfectly reflect the actual moment localization capability. To specifically
assess the grounding performance, we designed an oracle experiment operating
exclusively on the proposal where the ground truth moment is present. Similar
to the enhanced retrieval, we observed a significant improvement compared to
the disjoint baseline. Specifically, in comparison to the disjoint baseline,
the R$1_{.3}$ scores improved by about $6.7\%$ and $4.0\%$ for Ego4D and MAD,
respectively. Hence, the unified architecture proves more effective at
capturing discriminative frame-level features necessary for precisely
detecting the moment boundary.

|
---|---
Figure 5: Reducing the number of proposals speeds up the network execution
time. While the baseline model experiences a significant drop in performance
with this reduction, RGNet shows a noticeably smaller decline in performance
under the same conditions.  Figure 6:  Top-1 predictions on Ego4D-NLQ. We show
the GT moments with a red bar, while the green and yellow bars represent the
RGNet and baseline predictions, respectively. The disjoint baseline fails to
retrieve the correct proposal in the first query and struggles to pinpoint the
precise moment in the second one. RGNet succeeds in all cases, demonstrating
the effectiveness of our unified approach.

Number of Proposals: For the proposal-based methods, the number of retrieved
proposals significantly influences the Long Video Temporal Grounding (LVTG)
performance. Our model consistently outperforms the baseline across various
choices of top-k retrieved proposals, as shown in Fig. 5. Even with only a
single retrieved proposal, our performance surpasses the baseline’s optimal
result for both datasets. Moreover, our model exhibits a less steep
performance decline when fewer proposals are chosen. Importantly, with fewer
proposals, the grounding network runs fewer times, leading to an overall
speedup. Therefore, our model strikes a favorable balance between efficiency
and performance, showcasing superior performance compared to the baseline.
Proposal Length: Varying proposal length impacts both proposal retrieval and
grounding performance. The number of proposals decreases with longer
proposals, making the retrieval task easier. Fig. 7 reports a monotonically
improving retrieval with longer proposals. However, after a certain length,
moment localization quality starts to deteriorate. With longer proposals, the
task becomes extremely difficult. We see the decline after the 180s and 48s
for the MAD and Ego4D datasets. Hence, we set the default proposal length of
our method to these optimal lengths. Importantly, the unified network helps
outscore the disjoint baseline on all tested proposal lengths, inversely
proportional to the proposal numbers in Fig. 5.

|
---|---
Figure 7: Longer proposals result in improved retrieval due to fewer
candidates. However, grounding becomes exceedingly difficult with longer
clips. For example, the grounding performance drops after 180 seconds and 48
seconds for the MAD and Ego4D datasets. Hence, we use this as the optimal
proposal length.

Loss Functions: We provide further evidence of the effectiveness of our
proposed loss functions through a cumulative ablation study, as shown in Table
4. Training RGNet without the Inter-Proposal Contrastive loss results in an
almost $0.9\%$ decline in the R$1_{.3}$ score. By eliminating the sampling
loss, we also assess the impact of unified supervision and observe a reduction
of $1.2\%$ in grounding performance. This experiment validates the importance
of the proposed proposal-text mining technique for contrastive loss and
moment-guided sampling loss for introducing sparsity in the retrieval module.

Loss Functions | R1.3 | R5.3 | R1.5 | R5.5
---|---|---|---|---
Combined Loss | 18.28 | 34.23 | 12.04 | 22.89
w/o Contrastive Loss | 17.41 | 34.02 | 10.79 | 22.80
w/o Sampling Loss | 16.21 | 31.59 | 9.91 | 20.53
Table 4:  Cumulative ablation study on the validation set of Ego4D for various
Loss functions.

###  5.2 Qualitative Analysis

We visualize the qualitative predictions of RGNet compared to the disjoint
baseline in Fig. 6. For the first query, the baseline struggles to retrieve
the correct proposal where the scissor appears. RGNet accurately retrieves the
proposal with the searched object and precisely localizes the moment it
appeared in the video. Disjoint retrieval often leads to incorrect proposals,
complicating the final grounding task. For the second query, the baseline
selects the correct proposal but fails to localize the precise moment the man
closes the car door, indicating a gap between the two stages. RGNet mitigates
this gap by leveraging features from the retrieval phase to guide moment
localization. These results attest to the successful unification of the two
stages. More visual outputs are in the supplementary.

##  6 Conclusion

We introduce an end-to-end model for long video temporal grounding, unifying
the two stages of prevailing methods with shared features and mutual
optimization. This parallel training of the two stages of long video modeling
optimally fuses the two effortlessly. Our approach demonstrates state-of-the-
art performance on challenging long video grounding datasets, validating its
effectiveness. We conduct independent analyses of the two stages, shaping our
solution based on their distinct impacts. We hope to inspire heuristic-free
modeling approaches for long videos.

## References

  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _ICCV_ , pages 1728–1738, 2021.

  * Barrios et al. [2023] Wayner Barrios, Mattia Soldan, Alberto Mario Ceballos-Arroyo, Fabian Caba Heilbron, and Bernard Ghanem.  Localizing moments in long video via multimodal guidance.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13667–13678, 2023.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Chen et al. [2020] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu.  Fine-grained video-text retrieval with hierarchical graph reasoning.  In _CVPR_ , pages 10638–10647, 2020.
  * Cheng et al. [2023] Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius.  Vindlu: A recipe for effective video-and-language pretraining.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Dong et al. [2021] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang.  Dual encoding for video retrieval by text.  _IEEE TPAMI_ , pages 4065–4080, 2021.
  * Fang et al. [2023] Bo Fang, Chang Liu, Yu Zhou, Min Yang, Yuxin Song, Fu Li, Weiping Wang, Xiangyang Ji, Wanli Ouyang, et al.  Uatvr: Uncertainty-adaptive text-video retrieval.  _arXiv preprint arXiv:2301.06309_ , 2023.
  * Feichtenhofer et al. [2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In _ICCV_ , pages 6202–6211, 2019.
  * Gabeur et al. [2020] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal transformer for video retrieval.  In _ECCV_ , pages 214–229, 2020.
  * Gao et al. [2021] Zijian Gao, Jingyu Liu, Weiqi Sun, Sheng Chen, Dedan Chang, and Lili Zhao.  Clip2tv: Align, match and distill for video-text retrieval.  _arXiv preprint arXiv:2111.05610_ , 2021.
  * Ge et al. [2022] Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo.  Bridging video-text retrieval with multiple choice questions.  In _CVPR_ , pages 16167–16176, 2022.
  * Glorot and Bengio [2010] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * Gorti et al. [2022] Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu.  X-pool: Cross-modal language-video attention for text-video retrieval.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5006–5015, 2022.
  * Grauman et al. [2022] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _CVPR_ , pages 18995–19012, 2022.
  * Hannan et al. [2023] Tanveer Hannan, Rajat Koner, Maximilian Bernhard, Suprosanna Shit, Bjoern Menze, Volker Tresp, Matthias Schubert, and Thomas Seidl.  Gratt-vis: Gated residual attention for auto rectifying video instance segmentation.  _arXiv preprint arXiv:2305.17096_ , 2023.
  * Hou et al. [2022] Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan.  Cone: An efficient coarse-to-fine alignment framework for long video temporal grounding.  _arXiv preprint arXiv:2209.10918_ , 2022.
  * Jang et al. [2016] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Jang et al. [2023] Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn.  Knowing where to focus: Event-aware transformer for video grounding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13846–13856, 2023.
  * Jiang et al. [2022] Haojun Jiang, Jianke Zhang, Rui Huang, Chunjiang Ge, Zanlin Ni, Jiwen Lu, Jie Zhou, Shiji Song, and Gao Huang.  Cross-modal adapter for text-video retrieval.  _arXiv preprint arXiv:2211.09623_ , 2022.
  * Jin et al. [2022] Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David A Clifton, and Jie Chen.  Expectation-maximization contrastive learning for compact video-and-language representations.  _NeurIPS_ , 2022.
  * Kiros et al. [2014] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  _arXiv preprint arXiv:1411.2539_ , 2014.
  * Koner et al. [2023] Rajat Koner, Tanveer Hannan, Suprosanna Shit, Sahand Sharifzadeh, Matthias Schubert, Thomas Seidl, and Volker Tresp.  Instanceformer: An online video instance segmentation framework.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1188–1195, 2023.
  * Lei et al. [2021a] Jie Lei, Tamara L Berg, and Mohit Bansal.  Detecting moments and highlights in videos via natural language queries.  _Advances in Neural Information Processing Systems_ , 34:11846–11858, 2021a.
  * Lei et al. [2021b] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7331–7341, 2021b.
  * Lin et al. [2022] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al.  Egocentric video-language pretraining.  _arXiv preprint arXiv:2206.01670_ , 2022.
  * Lin et al. [2023] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou.  Univtg: Towards unified video-language temporal grounding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2794–2804, 2023.
  * Liu et al. [2022a] Naiyuan Liu, Xiaohan Wang, Xiaobo Li, Yi Yang, and Yueting Zhuang.  Reler@ zju-alibaba submission to the ego4d natural language queries challenge 2022.  _arXiv preprint arXiv:2207.00383_ , 2022a.
  * Liu et al. [2022b] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  Dab-detr: Dynamic anchor boxes are better queries for detr.  _arXiv preprint arXiv:2201.12329_ , 2022b.
  * Liu et al. [2022c] Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie.  Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3042–3051, 2022c.
  * Liu et al. [2022d] Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin.  Ts2-net: Token shift and selection transformer for text-video retrieval.  In _ECCV_ , pages 319–335, 2022d.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Luo et al. [2020] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou.  Univl: A unified video and language pre-training model for multimodal understanding and generation.  _arXiv preprint arXiv:2002.06353_ , 2020.
  * Ma et al. [2022] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji.  X-clip: End-to-end multi-grained contrastive learning for video-text retrieval.  In _ACM MM_ , pages 638–647, 2022.
  * Maddison et al. [2016] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  _CoRR_ , abs/1611.00712, 2016.
  * Meng et al. [2021] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang.  Conditional detr for fast training convergence.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3651–3660, 2021.
  * Moon et al. [2023] WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo.  Query-dependent video representation for moment retrieval and highlight detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23023–23033, 2023.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Pan et al. [2023] Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng, and Deli Zhao.  Scanning only once: An end-to-end framework for fast temporal grounding in long videos.  _arXiv preprint arXiv:2303.08345_ , 2023.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Pramanick et al. [2023] Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang.  Egovlpv2: Egocentric video-language pre-training with fusion in the backbone.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5285–5297, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  pages 8748–8763, 2021.
  * Ramakrishnan et al. [2023] Santhosh Kumar Ramakrishnan, Ziad Al-Halah, and Kristen Grauman.  Naq: Leveraging narrations as queries to supervise episodic memory.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6694–6703, 2023.
  * Soldan et al. [2021] Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem.  Vlg-net: Video-language graph matching network for video grounding.  In _ICCV_ , pages 3224–3234, 2021.
  * Soldan et al. [2022] Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem.  Mad: A scalable dataset for language grounding in videos from movie audio descriptions.  In _CVPR_ , pages 5026–5035, 2022.
  * Wang et al. [2022a] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  All in one: Exploring unified video-language pre-training.  _arXiv preprint arXiv:2203.07303_ , 2022a.
  * Wang et al. [2022b] Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and Xian-Sheng Hua.  Disentangled representation learning for text-video retrieval.  _arXiv preprint arXiv:2203.07111_ , 2022b.
  * Wang et al. [2023] Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal.  Unified coarse-to-fine alignment for video-text retrieval.  In _The IEEE International Conference on Computer Vision (ICCV)_ , 2023.
  * Xu et al. [2021a] Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer.  Vlm: Task-agnostic video-language model pre-training for video understanding.  _ACLliu2021hit_ , 2021a.
  * Xu et al. [2021b] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  In _EMNLP_ , pages 6787–6800, 2021b.
  * Xue et al. [2022] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo.  Advancing high-resolution video-language representation with large-scale video transcriptions.  In _CVPR_ , pages 5036–5045, 2022.
  * Xue et al. [2023] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo.  Clip-vip: Adapting pre-trained image-text model to video-language representation alignment.  _ICLR_ , 2023.
  * Yu et al. [2017] Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim.  End-to-end concept word detection for video captioning, retrieval, and question answering.  In _CVPR_ , pages 3165–3173, 2017.
  * Yu et al. [2018] Youngjae Yu, Jongseok Kim, and Gunhee Kim.  A joint sequence fusion model for video question answering and retrieval.  In _ECCV_ , pages 471–487, 2018.
  * Zhang et al. [2023a] Bowen Zhang, Xiaojie Jin, Weibo Gong, Kai Xu, Zhao Zhang, Peng Wang, Xiaohui Shen, and Jiashi Feng.  Multimodal video adapter for parameter efficient video text retrieval.  _arXiv preprint arXiv:2301.07868_ , 2023a.
  * Zhang et al. [2023b] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman.  Helping hands: An object-aware ego-centric video recognition model.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13901–13912, 2023b.
  * Zhang et al. [2020a] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou.  Span-based localizing network for natural language video localization.  _arXiv preprint arXiv:2004.13931_ , 2020a.
  * Zhang et al. [2022a] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  _arXiv preprint arXiv:2203.03605_ , 2022a.
  * Zhang et al. [2022b] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou.  The elements of temporal sentence grounding in videos: A survey and future directions.  _arXiv preprint arXiv:2201.08071_ , 1(2), 2022b.
  * Zhang et al. [2020b] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.  Learning 2d temporal adjacent networks for moment localization with natural language.  In _AAAI_ , pages 12870–12877, 2020b.
  * Zhao et al. [2023] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar.  Learning video representations from large language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6586–6597, 2023.
  * Zhu et al. [2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  _arXiv preprint arXiv:2010.04159_ , 2020.

##  Appendix A Supplementary

###  A.1 Further Ablation Studies

This section provides an in-depth analysis of various components of RGNet,
such as the sampler and transformer decoder. Furthermore, we conduct ablation
studies on pre-training data and frame rate. All experiments are carried out
on the Ego4D-NLQ dataset.

#### Sampler

We use a differentiable softmax function to enable the end-to-end training of
the sampler. Let’s denote $p^{j}$ as the linear projection of $f^{j}$ from Eq.
2.

$p^{j}=\text{linear}({f}^{j})$ (11)

To derive a categorical variable $G^{j}$ characterized by probabilities
${\pi_{1}}^{j}=\sigma(p^{j})$ and ${\pi_{0}}^{j}=1-\sigma(p^{j})$, where
$\sigma$ is the sigmoid operation, we can reframe the sampling procedure for
$G^{j}$ through the utilization of the Gumbel-Max trick, outlined as follows:

$G^{j}={\arg\max}_{k}\left\\{\log\left({\pi}_{k}^{j}\right)+g_{k}:k=0,1\right\\}$
(12)

Here, the set $\\{g_{k}\\}_{k={0,1}}$ consists of independently and
identically distributed (i.i.d.) random variables sampled from the
$Gumbel(0,1)$ distribution. Considering the non-differentiable characteristic
of the argmax operation, we employ an approximation for $G^{j}$ using a
differentiable, soft version $\hat{G}^{j}$, derived from the Gumbel-Softmax
relaxation [17, 34].

$\hat{G}^{j}=\frac{\exp\left(\left(\log\left({\pi}_{1}^{j}\right)+g_{1}\right)/{\tau}\right)}{\Sigma_{k\in{0,1}}\exp\left(\left(\log\left({\pi}_{k}^{j}\right)+g_{k}\right)/\tau\right)}$
(13)

To ensure differentiability with respect to the discrete samples $G^{j}$, we
employ the straight-through trick [17] and utilize the gradients of
$\hat{G}^{j}$ as an approximation for the gradients of $G^{j}$ in the backward
pass.

The Gumbel-Softmax distribution serves as an interpolation between discrete
one-hot-encoded categorical distributions and continuous categorical
densities. For low temperatures ($\tau$ = 0.3), the expected value of a
Gumbel-Softmax random variable approaches the expected value of a categorical
random variable with the same logits. As the temperature increases ($\tau$ =
0.9), the expected value converges to a uniform distribution over the
categories. Fig. 8(a) shows that we achieve the best performance with $\tau$
to be $0.3$, which means the differentiation between the relevant and
irrelevant frames is beneficial during the retrieval.

#### Transformer Decoder

RGNet utilizes learnable decoder queries [28] to localize the moments. The
number of queries equates to the number of predicted moments from each
retrieved proposal. With more queries, the decoder can detect moments in
different temporal locations of various widths. However, for long videos, the
decoder runs on the top-$k$ retrieved proposals, and increasing the queries
beyond $5$ leads to decreased retrieval performance overall (ref. Fig. 8(b)).
Consequently, we set the number of queries in our decoder to $5$.

#### Frame Rate

Manual reduction of frame rate (FPS) yields enhanced retrieval accuracy but
significantly degrades grounding performance, as shown in Fig. 8(c). Lowering
the FPS serves as a heuristic sampling strategy that aids retrieval but
results in temporal information loss, leading to diminished overall
performance. In contrast, our learned sampler enhances retrieval accuracy
without compromising grounding performance, presenting a superior alternative
to manual sampling.

#### NaQ Pretraining

Similar to the prior state-of-the-art model [42], we employ NaQ augmentation
to pre-train RGNet on the Ego4D-NaQ dataset. With and without this
pretraining, the previously leading ReLER [27] achieves a R$1_{.3}$ score of
$19.31$ and $14.66$, whereas our model demonstrates an impressive $20.63$ and
$18.28$ (refer to Table 5). It is important to note that the evaluation of the
Ego4D test set is exclusively available on their official server, which is
presently closed. Therefore, in alignment with recent publications [55, 38] in
ICCV’23, we present our performance of Ego4D-NLQ on the validation set.

Method | NaQ | R1.3 | R5.3 | R1.5 | R5.5
---|---|---|---|---|---
VSLNet | ✗ | 5.45 | 10.74 | 3.12 | 6.63
EgoVLP | ✗ | 10.84 | 18.84 | 6.81 | 13.45
ReLER | ✗ | 14.66 | 17.84 | 8.67 | 11.54
RGNet (Ours) | ✗ | 18.28 | 34.02 | 12.04 | 22.89
VSLNet | ✓ | 10.26 | 19.01 | 5.81 | 12.67
EgoVLP | ✓ | 15.90 | 26.38 | 9.46 | 17.80
ReLER | ✓ | 19.31 | 23.62 | 11.59 | 15.75
RGNet (Ours) | ✓ | 20.63 | 41.67 | 12.47 | 25.08

Table 5: Results on Ego4D-NLQ dataset with and w/o NaQ augmentation. RGNet
achieves the best performance in both cases.

(a) Low temperature enables our model to approximate a categorical
distribution over the relevancy of frames and helps achieve better
performance.

(b) Increasing the number of decoder queries up to $5$ enhances performance,
while further increments start to deteriorate the results.

(c) While decreasing the FPS improves retrieval accuracy, it deteriorates the
grounding performance due to the loss of temporal information.

Figure 8: Ablation studies on the sampler, transformer decoder, and input FPS
with Ego4d-NLQ Dataset.

###  A.2 Qualitative Results

We visualize the relevance score of all the candidate proposals and proposal
frames for a single video in Fig. 10. For both the proposal and frame levels,
our model scores higher on the ground truth regions than the baseline. Then,
we visualize some successful moment localization on both MAD and Ego4D
datasets in Tab. 9 and 11.

Figure 9: Qualitative results on MAD. RGNet successfully localizes moments
defined by various length queries from hour-long movies.  Figure 10: We
illustrated the relevancy score of the proposal candidates (left) and proposal
frames (right) for the query “Did I leave the car door open?” from Ego4D-NLQ.
We present the score for both RGNet and the disjoint baseline model in green
and yellow, respectively. RGNet approximates the ground truth (GT) proposal
and moment better than the baseline for both stages.  Figure 11: Qualitative
results of Ego4D. RGNet captures fine-grained details, enabling it to detect
activities across various circumstances.