capbtabboxtable[][] [table]footnoterule=normal,capposition=bottom

# BAM-DETR: Boundary-Aligned Moment Detection Transformer
for Temporal Sentence Grounding in Videos

Pilhyeon Lee Hyeran Byun
Department of Computer Science, Yonsei University Correspondence to: Pilhyeon
Lee<EMAIL_ADDRESS>
###### Abstract

Temporal sentence grounding aims to localize moments relevant to a language
description. Recently, DETR-like approaches have shown notable progress by
decoding the center and length of a target moment from learnable queries.
However, they suffer from the issue of center misalignment raised by the
inherent ambiguity of moment centers, leading to inaccurate predictions. To
remedy this problem, we introduce a novel boundary-oriented moment
formulation. In our paradigm, the model no longer needs to find the precise
center but instead suffices to predict any anchor point within the interval,
from which the onset and offset are directly estimated. Based on this idea, we
design a Boundary-Aligned Moment Detection Transformer (BAM-DETR), equipped
with a dual-pathway decoding process. Specifically, it refines the anchor and
boundaries within parallel pathways using global and boundary-focused
attention, respectively. This separate design allows the model to focus on
desirable regions, enabling precise refinement of moment predictions. Further,
we propose a quality-based ranking method, ensuring that proposals with high
localization qualities are prioritized over incomplete ones. Extensive
experiments verify the advantages of our methods, where our model records new
state-of-the-art results on three benchmarks. Code is at
https://github.com/Pilhyeon/BAM-DETR.

##  1 Introduction

Recent years have witnessed a notable surge in the popularity of short-form
video content on social media platforms like YouTube Shorts, Instagram Reels,
and TikTok. In the same vein, users prefer to selectively engage with short
moments of interest rather than passively watch an entire long video. This
trend highlights the importance of localizing desired moments. As a result,
moment localization tasks have emerged as pivotal research topics in the field
of video understanding, encompassing temporal action detection [75, 29, 20],
highlight detection [48, 58, 22], and video summarization [71, 50]. Within
this context, we particularly tackle temporal sentence grounding [1], aiming
to retrieve moments corresponding to free-form language descriptions.

To address temporal sentence grounding, numerous efforts have been undertaken
in the last decade [11, 49, 63, 9, 73, 32]. Especially, taking inspiration
from DETR [3], query-based approaches have become a promising research
direction owing to the architectural simplicity [22, 37, 41, 17, 26]. By
decoding temporal spans from a small set of learnable queries, they achieve
promising grounding performance while maintaining high inference speed.

Figure 1: Comparison of moment modeling approaches under the scenario of an
ambiguous center from QVHighlights [22]. (a) The conventional method
formulates the moment with a tuple of ($c$, $l$). (b) In contrast, we propose
to model it with a triplet of ($p$, $d_{s}$, $d_{e}$).

Existing query-based models typically predict a moment using its center and
width, _i.e_., ($c$, $l$), under the assumption that the boundaries are
equidistant from the center. However, such a formulation can be problematic as
the center of a moment might be ambiguous. An illustrative example is
presented in Fig. 1 (top), where the frame at the center of the ground-truth
moment primarily shows the audience, which is less relevant to the given
sentence. This ambiguity can challenge the model’s ability to precisely locate
the centers, resulting in low-quality predictions (Fig. 1a). To probe the
impact of center misalignment on performance, we conducted a diagnostic
experiment in Footnote 1. The results reveal that existing methods struggle
with detecting accurate center points and suffer from significant performance
drops when the predicted centers deviate from the ground truths.

To address the challenge, we present a novel boundary-oriented formulation for
moments, as illustrated in Fig. 1b, where each moment is represented by a
triplet consisting of an anchor point and its distances to the boundaries,
_i.e_., $(p,d_{s},d_{e})$. This formulation liberates the model from the
stringent requirement of predicting the center. Instead, it is sufficient for
the model to predict any salient anchor within the target moment, and the
distances from the anchor to the onset and offset are predicted subsequently.
By directly locating the boundaries based on the anchor point, the model can
achieve improved boundary alignment, even when the anchor point does not
coincide with the actual center.

Building upon the proposed moment modeling, we introduce a new framework
equipped with a dedicated decoder, dubbed Boundary-Aligned Moment Detection
Transformer (BAM-DETR). The design of its decoding layers originates from the
intuition that the refining processes for an anchor and boundaries should be
distinct from each other. That is to say, a model needs to scan over the whole
video to find a potential anchor point that enables estimating the rough
location of the target moment. On the other hand, it is required to focus on
fine-grained details in the vicinity to refine the boundaries to be aligned
with those of the target moments. From this motivation, in contrast to
existing methods, our model adopts a dual-pathway decoding pipeline to predict
an anchor point and boundaries in a parallel way. To be specific, we leverage
two different types of queries respectively for anchor and boundary
refinement. The former aggregates global information with standard attention,
while the latter concentrates on the sparse local neighborhood of the
boundaries using the proposed boundary-focused attention. These distinct
designs of two pathways allow for effective moment localization with minimal
computational overhead increase.

Method |  [0, 0.1) |  [0.1, 0.2) |  [0.2, 0.3) |  [0.3, 0.4) |  [0.4, 0.5) |  All
---|---|---|---|---|---|---
M-DETR [22] |  83.20 |  65.00 |  54.82 |  43.77 |  34.98 |  67.84
(46 %) |  (24 %) |  (14 %) |  (9 %) |  (7 %) |  (100 %)
QD-DETR [41] |  87.79 |  67.95 |  55.40 |  44.93 |  36.45 |  74.09
(56 %) |  (18 %) |  (11 %) |  (9 %) |  (5 %) |  (100 %)
BAM-DETR† |  77.62 |  78.20 |  77.52 |  78.96 |  71.08 |  77.21
(Ours) |  (24 %) |  (21 %) |  (21 %) |  (22 %) |  (12 %) |  (100 %)

†anchor points are utilized for grouping.

Table 1: Impact of misaligned centers on QVHighlights [22]. The top-1
predictions are grouped based on their center errors normalized by ground-
truth lengths. For each group, we present the mean IoU (%) and the proportion
(in parentheses). Only the predictions whose centers fall within the ground
truths are considered here111In case of interest, the mean IoUs of the
predictions with their centers falling outside the ground truths are 11.76,
12.46, and 22.51, respectively..

In addition, we identify the problem of the conventional scoring method, where
binary classification (or matching) scores are used for proposal ranking. This
leads to suboptimal results for the grounding task since a fractional moment
may have high matching scores with the given sentence. To handle this issue,
we propose to rank proposals based on their localization qualities.
Accordingly, we modify the typical matching function and training objectives
to be localization-oriented by discarding the role of classification scores.
In this way, our model can prioritize the moment proposals exhibiting high
overlap with ground truths at inference, leading to improved grounding
performance.

The advantage of our BAM-DETR over existing methods is shown in Footnote 1. It
is observed that the anchor points predicted by our model are evenly
distributed within the ground-truth intervals. Importantly, our model
consistently produces precise moments with high IoUs ($>0.7$) across different
groups, indicating that it does not depend on center prediction. On average,
our model shows superior grounding performances with the help of our moment
formulation and quality-based scoring. In a later section, we validate the
effectiveness of the proposed methods through extensive experiments. Notably,
our model outperforms previous state-of-the-art methods by large margins on
three public benchmarks, especially under strict evaluation metrics.

##  2 Related Works

###  2.1 Temporal Sentence Grounding in Videos

Temporal sentence grounding requires seeking temporal spans semantically
relevant to the given sentence in a video. Proposal-based approaches adopt the
two-stage pipeline, _i.e_., proposal generation and ranking. They generate
moment proposals by relying on sliding windows [11, 34, 72, 13] or utilizing
pre-defined anchors [67, 59, 63, 4, 74]. Several works process all possible
candidates at once with 2D maps [73, 56, 31, 57, 25]. On the other hand,
proposal-free methods are developed for efficient grounding by directly
predicting the moments [64, 42] or estimating the probabilities of each frame
being starting and ending positions [15, 69]. Several approaches perform dense
regression by predicting the boundaries from individual frames [66, 5, 39].
Recently, query-based models streamline the complicated sentence grounding
pipeline by removing handcrafted techniques [2, 37, 60, 17, 26]. There are
also attempts to unify temporal sentence grounding with other video
understanding tasks into a single framework [28, 61].

Figure 2: (a) Overview of the proposed BAM-DETR. (b) Details of the proposed
dual-pathway decoding layer. It consists of two parallel pathways respectively
for anchor and boundary updates, which refine the previous moment predictions
in a sequential manner.

Our method belongs to the query-based group [22, 41], inheriting the benefit
of architectural simplicity. In contrast to others, we employ a new boundary-
oriented formulation of moments so as to relieve the heavy reliance on center
predictions, leading to better boundary alignment. Meanwhile, our method also
relates to the dense regression methods [66, 28, 39, 5] that predict distances
to the boundaries considering each frame an anchor. Compared to them, our
model leverages dynamic anchor points that are gradually adjusted for better
prediction through decoding, enabling precise localization with a small number
of predictions.

###  2.2 Detection Transformers

Query-based temporal sentence grounding models by design are closely related
to the family of detection transformer (DETR) [3]. Since the advent of DETR, a
number of variants have been introduced to improve it from various
perspectives [33, 53, 30, 36]. Some works focus on reducing the excessive
computational costs of vanilla transformers to leverage multi-scale features
[7, 78, 47, 24, 76]. Other methods attempt to speed up the model convergence
by manipulating the attention operations [12, 35, 68, 62] or incorporating the
denoising process during the training [23, 70].

The most relevant ones to ours are those which propose explicit anchor
modeling for object queries with points [40, 55] or boxes [78, 35]. In
comparison to an object, a moment in temporal sentence grounding has its own
challenges such as center ambiguity and indistinct boundaries. To accommodate
the discrepancies, we propose a novel boundary-oriented modeling of moments to
replace the conventional 1D box-based modeling. The advantages of our approach
are clearly verified in the experiments.

##  3 Method

Given an untrimmed video and a sentence, the goal of temporal sentence
grounding is to localize relevant moments
$\\{\varphi_{n}=(t_{s_{n}},t_{e_{n}})\\}_{n=1}^{N}$, where $N$ denotes the
number of ground truths in the video and $\varphi_{n}$ indicates the temporal
interval of the $n$-th moment. Note that a video may have multiple moments
that match the sentence, _i.e_., $N\geq 1$. During the test time, the model is
expected to produce a total of $M$ predictions,
$\\{(\hat{\varphi}_{m},q_{m})\\}_{m=1}^{M}$, where
$\hat{\varphi}_{m}=(\hat{t}_{s_{m}},\hat{t}_{e_{m}})$ is the $m$-th
prediction, while $q_{m}$ is its score for ranking.

#### Motivation.

We tackle two main problems in the current prediction process. On one hand,
existing query-based approaches adopt the symmetric design of $(c,l)$ to
predict a moment, _i.e_., $\hat{\varphi}=(c-0.5l,c+0.5l)$. As we discuss in
Sec. 1, this strategy suffers from the issue of over-reliance on the center
prediction, leading to unstable performances. To address this, we propose a
boundary-oriented modeling with a triplet of ($p,d_{s},d_{e}$), where a moment
is represented by $\hat{\varphi}=(p-d_{s},p+d_{e})$. This asymmetric design
enables direct alignment of boundaries without relying on precise prediction
of the center. On the other hand, previous models utilize the classification
score as $q_{m}$. This scoring is prone to sub-optimal solutions since only a
fraction of the moment may well match the sentence, leading incomplete
predictions to be highly ranked. To handle this, we propose quality-based
scoring to sort the proposals based on their localization qualities rather
than the degree of matching.

###  3.1 Overview

As shown in Fig. 2a, our BAM-DETR follows the encoder-decoder pipeline.
Briefly, it first extracts multimodal video features using the encoders.
Taking them as memory, the boundary-aligning decoder predicts a set of
temporal spans and their quality scores by progressively refining learnable
initial spans using anchor and boundary queries.

###  3.2 Feature Extraction

Following the convention [73, 42, 69], we employ pre-trained encoders for
token-level unimodal feature extraction. It is worth noting that all unimodal
encoders are kept frozen during training to avoid memory exploding. In this
stage, we obtain $D_{v}$-dim video features $E_{v}\in\mathbb{R}^{N_{v}\times
D_{v}}$ and $D_{t}$-dim text features $E_{t}\in\mathbb{R}^{N_{t}\times
D_{t}}$, where $N_{v}$ and $N_{t}$ are the numbers of clips and words,
respectively.

###  3.3 Multimodal Encoder

We feed the unimodal features of the video and text to the multimodal encoder,
so as to fuse them into text-aware video representations for temporal sentence
grounding. While various multimodal encoders are explored [22, 37, 28], we
adopt the text-to-video encoder design [41] consisting of cross- and self-
attention blocks [54]. Before getting into the encoder, unimodal features are
projected into a shared space to facilitate cross-modal interaction, _i.e_.,
$\mathcal{V}=f_{v}(E_{v})\in\mathbb{R}^{N_{v}\times D}$,
$\mathcal{T}=f_{t}(E_{t})\in\mathbb{R}^{N_{t}\times D}$, where $D$ is the
embedding dimension. Afterward, multi-head cross-attention blocks are used to
inject textual information into the clip-level video representations. In
specific, we project $\mathcal{V}$ to $\mathbf{Q}_{\mathcal{V}}$ (query) while
$\mathcal{T}$ to $\mathbf{K}_{\mathcal{T}}$ (key) and
$\mathbf{V}_{\mathcal{T}}$ (value). Then a single-head attention block can be
formulated as:

$\displaystyle\mathcal{V}^{\prime}$
$\displaystyle=\text{softmax}\Big{(}\frac{\mathbf{Q}_{\mathcal{V}}\mathbf{K}_{\mathcal{T}}^{\top}}{\sqrt{D}}\Big{)}\mathbf{V}_{\mathcal{T}}+\mathcal{V},$
(1) $\displaystyle\mathcal{V}^{\prime\prime}$
$\displaystyle=\text{FFN}(\mathcal{V^{\prime}})+\mathcal{V}^{\prime},$

where $\text{FFN}(\cdot)$ is a feed-forward network. We denote the resulting
multimodal representations obtained after $L_{E}$ multi-head cross-attention
blocks by $\tilde{\mathcal{V}}\in\mathbb{R}^{N_{v}\times D}$. Although we here
present the single-head attention block, it can readily generalize to a multi-
headed version [54].

Subsequently, self-attention blocks are leveraged to enhance the
representations by allowing the inter-clip interaction. Here we project
$\tilde{\mathcal{V}}$ to $\mathbf{Q}_{\tilde{\mathcal{V}}}$,
$\mathbf{K}_{\tilde{\mathcal{V}}}$, and $\mathbf{V}_{\tilde{\mathcal{V}}}$. We
note that the query and the key are supplemented with fixed sinusoidal
positional encoding [54, 3] for temporal awareness. Then the self-attention
block is defined in a similar way to Eq. 1 but with different inputs. The
enhanced clip-level representations after $L_{E}$ self-attention blocks are
denoted by $\hat{\mathcal{V}}\in\mathbb{R}^{N_{v}\times D}$, which serves as
memory for the decoder.

It is widely known that providing the multimodal features with saliency
guidance helps the model to better understand the semantic relationship
between the video and text [22, 28]. As in the previous works [17, 26], we
impose saliency score constraints on the memory features. Specifically, we
leverage a saliency predictor $S(\cdot)$ and train the model with the
following margin-based training objective.

$\mathcal{L}_{\text{margin}}=\max(0,\alpha+S(\hat{v}^{\text{low}})-S(\hat{v}^{\text{high}})),$
(2)

where $\alpha$ is a margin and $(\hat{v}^{\text{low}},\hat{v}^{\text{high}})$
is the sampled feature pair satisfying that the saliency label of
$\hat{v}^{\text{low}}$ is lower than that of $\hat{v}^{\text{high}}$. In case
of the absence of saliency labels, we collect clips within and outside the
ground-truth moment intervals. In addition, we employ the rank-aware
contrastive loss $\mathcal{L}_{\text{cont}}$ and the negative relation loss
$\mathcal{L}_{\text{neg}}$, following Moon  _et al_. [41]. Due to space
limits, we refer the readers to Appendix for the loss formulations. In
summary, the overall saliency loss is defined as
$\mathcal{L}_{\text{sal}}=\mathcal{L}_{\text{margin}}+\mathcal{L}_{\text{cont}}+\mathcal{L}_{\text{neg}}$.

###  3.4 Dual-pathway Decoder

With the multimodal representations $\hat{\mathcal{V}}$ as memory features, we
aim to localize temporal spans corresponding to the sentence. We adopt the new
boundary-oriented formulation for moment prediction, where each prediction is
represented by a triplet of $(p,d_{s},d_{e})$, where $p$ is the anchor point,
while $d_{s}$ and $d_{e}$ are the distances from the anchor to the starting
and ending points, respectively. To make full use of the modified formulation,
we design a dual-pathway decoding layer with two parallel pathways as
described in Fig. 2b. The inputs of the $l$-th layer are anchor queries
$\mathbf{C}_{p}^{l}\in\mathbb{R}^{M\times D}$, boundary queries
$\mathbf{C}_{s}^{l},\mathbf{C}_{e}^{l}\in\mathbb{R}^{M\times D}$, and previous
moment predictions
$\mathbf{A}^{l}=\begin{bmatrix}\mathbf{p}^{l};\mathbf{d}_{s}^{l};\mathbf{d}_{e}^{l}\end{bmatrix}\in\mathbb{R}^{M\times
3}$, where $M$ is the number of queries (predictions). Note that the initial
queries and spans, _i.e_., $\\{\mathbf{C}_{p}^{0}$, $\mathbf{C}_{s}^{0}$,
$\mathbf{C}_{e}^{0}$, $\mathbf{A}^{0}\\}$, are learnable parameters. We
elaborate on the two pathways in the following.

#### Anchor updating pathway.

Given the predictions and the anchor queries from the preceding layer, the
goal of this pathway is to adjust the anchor position so that the boundaries
can be accurately predicted based on it. Intuitively, in order to obtain a
valuable anchor point without redundancy, an anchor query should communicate
with other queries as well as the memory features. To this goal, the anchor
updating pathway consists of a self-attention layer, a cross-attention layer,
and a feed-forward network. In the self-attention layer, the anchor queries
$\mathbf{C}_{p}^{l}$ are first projected into
$\mathbf{Q}_{\mathbf{C}_{p}^{l}}$, $\mathbf{K}_{\mathbf{C}_{p}^{l}}$, and
$\mathbf{V}_{\mathbf{C}_{p}^{l}}$. Since an anchor query itself lacks
positional information, we build positional encoding. Following the previous
works [41, 17], we extend the current spans $\mathbf{A}^{l}$ to the positional
information of the queries, _i.e_.,
$\mathbf{P}_{\mathbf{A}^{l}}=\text{MLP}(\text{PE}(\mathbf{A}^{l}))\in\mathbb{R}^{M\times
D}$, where $\text{PE}(\cdot)$ denotes the point-wise mapping from a position
to the corresponding sinusoidal encoding while $\text{MLP}(\cdot)$ is multi-
layer perceptron. Then, the self-attention for the anchor queries is defined
as:

$\begin{aligned}
\tilde{\mathbf{C}}_{p}^{l}=\text{softmax}\Big{(}\frac{\big{(}\mathbf{Q}_{\mathbf{C}_{p}^{l}}+\mathbf{P}_{\mathbf{A}^{l}}\big{)}\big{(}\mathbf{K}_{\mathbf{C}_{p}^{l}}+\mathbf{P}_{\mathbf{A}^{l}}\big{)}^{\top}}{\sqrt{D}}\Big{)}\mathbf{V}_{\mathbf{C}_{p}^{l}}+\mathbf{C}_{p}^{l}.\end{aligned}$

(3)

After inter-query interaction, we employ a global cross-attention layer to
aggregate multi-modal features from the memory. The anchor query
$\tilde{\mathbf{C}}_{p}^{l}$ is projected into
$\mathbf{Q}_{\tilde{\mathbf{C}}_{p}^{l}}$ while the memory $\hat{\mathcal{V}}$
is projected into $\mathbf{K}_{\hat{\mathcal{V}}}$ and
$\mathbf{V}_{\hat{\mathcal{V}}}$. To make the query location-aware, we
leverage the sinusoidal encoding of current anchor positions, _i.e_.,
$\mathbf{P}_{\mathbf{p}^{l}}=\text{PE}(\mathbf{p}^{l})\in\mathbb{R}^{M\times
D}$. Similarly, the memory leverages the positional encoding, _i.e_.,
$\mathbf{P}_{\hat{\mathcal{V}}}=\text{PE}(\hat{\mathcal{V}})\in\mathbb{R}^{N_{v}\times
D}$. We use concatenation instead of summation to separate the roles of
features and positional encoding [40, 35]. Then the global cross-attention
between anchor queries and the memory can be expressed as follows.

$\hat{\mathbf{C}}_{p}^{l}=\text{softmax}\Big{(}\frac{\big{(}\mathbf{Q}_{\tilde{\mathbf{C}}_{p}^{l}}\mathbin{\|}\mathbf{P}_{\mathbf{p}^{l}}\big{)}\big{(}\mathbf{K}_{\hat{\mathcal{V}}}\mathbin{\|}\mathbf{P}_{\hat{\mathcal{V}}})\big{)}^{\top}}{\sqrt{2D}}\Big{)}\mathbf{V}_{\hat{\mathcal{V}}}+\tilde{\mathbf{C}}_{p}^{l}.$
(4)

Then the anchor queries are updated with a feed-forward network, _i.e_.,
$\mathbf{C}_{p}^{(l+1)}=\text{FFN}(\hat{\mathbf{C}}_{p}^{l})+\hat{\mathbf{C}}_{p}^{l}$.
Lastly, we adjust the anchor positions using sigmoid-based refinement [78]:
$\hat{\mathbf{A}}^{l}=\begin{bmatrix}\mathbf{p}^{(l+1)};\mathbf{d}_{s}^{l};\mathbf{d}_{e}^{l}\end{bmatrix}$
where
$\mathbf{p}^{(l+1)}=\sigma(\sigma^{-1}(\mathbf{p}^{l})+\Delta\mathbf{p}^{l})$
with
$\Delta\mathbf{p}^{l}=\text{MLP}(\mathbf{C}_{p}^{(l+1)})\in\mathbb{R}^{M}$ and
sigmoid function $\sigma$.

#### Boundary updating pathway.

After the anchor update, we need to refine the boundaries based on the anchor.
It is widely perceived that a model needs to focus on fine-grained features in
the neighborhood rather than far ones to adjust temporal boundaries [29, 27,
51]. Inspired by this, we devise a boundary-focused attention layer (Fig. 3).
For brevity, we explain the process of starting point update.

The plain memory $\hat{\mathcal{V}}$ lacks the inductive bias due to the
property of attentional layers [8]. Thus we first build locality-enhanced
memory features for effective boundary refinement. To this goal, we obtain
boundary-sensitive features with several 1D convolutional layers, _i.e_.,
$\hat{\mathcal{V}}_{\text{s}}=f_{\text{s}}(\hat{\mathcal{V}})$. Then we
encourage them to highly activate around the starting position of target
moments. In detail, on the clip-wise activation scores obtained by channel-
mean, _i.e_.,
$\hat{g}^{s}=\text{mean}(\sigma(\hat{\mathcal{V}_{s}}))\in\mathbb{R}^{N_{v}}$,
we impose regularization as:

$\mathcal{L}_{\text{regul}}^{s}=-\frac{1}{N_{v}}\sum_{i=1}^{N_{v}}\big{(}g_{i}^{s}\text{log}(\hat{g}_{i}^{s})+(1-g_{i}^{s})\text{log}(1-\hat{g}_{i}^{s})\big{)},$
(5)

where $g_{i}^{s}$ is the binary label obtained as
$g_{i}^{s}=\mathbbm{1}[i\in\mathcal{B}^{s}]$, where $\mathcal{B}^{s}$ is the
neighbor clip set around starting points with a radius of $r^{s}$. We merge
boundary-sensitive features with the plain ones to form locality-enhanced
memory, _i.e_.,
$\hat{\mathcal{V}}^{\prime}_{s}=[\hat{\mathcal{V}}\mathbin{\|}\hat{\mathcal{V}}_{s}]\in\mathbb{R}^{N_{v}\times
2D}$. We can obtain $\mathcal{L}_{\text{regul}}^{e}$ and
$\hat{\mathcal{V}}^{\prime}_{e}$ in the same way. The total regularization
term is
$\mathcal{L}_{\text{regul}}=\mathcal{L}_{\text{regul}}^{s}+\mathcal{L}_{\text{regul}}^{e}$.

Given the locality-enhanced memory, the queries need to capture fine-grained
details around the boundaries for refinement. To aggregate useful features
near the boundaries, we employ deformable attention [78, 77]. Regarding the
starting boundary ($\mathbf{p}^{(l+1)}-\mathbf{d}_{s}^{l}$) as the origin, we
predict offsets and weights to select $K$ neighbors, _i.e_.,
$\mathbf{o}^{s}=\phi_{o}(\mathbf{C}_{s}^{l})\in\mathbb{R}^{M\times K}$,
$\mathbf{w}^{s}=\text{softmax}(\phi_{w}(\mathbf{C}_{s}^{l}))\in\mathbb{R}^{M\times
K}$, where $\phi_{*}$ are fully-connected layers. Features from the sampled
neighbors are then aggregated into the starting queries as:

$\hat{\mathbf{C}}_{s}^{l}=\sum_{k=1}^{K}\big{[}\mathbf{w}_{k}^{s}\cdot\hat{\mathcal{V}}^{\prime}_{s}[\mathbf{p}^{(l+1)}-\mathbf{d}_{s}^{l}+\mathbf{o}_{k}^{s}]\big{]}+\mathbf{C}_{s}^{l},$
(6)

where we denote the sampling process from memory by
$\hat{\mathcal{V}}^{\prime}_{s}[\cdot]$. Lastly, we adopt a feed-forward
network to obtain the updated starting queries, _i.e_.,
$\mathbf{C}_{s}^{(l+1)}=\text{FFN}(\hat{\mathbf{C}}_{s}^{l})$. We also obtain
$\mathbf{C}_{e}^{(l+1)}$ in the same way. Using boundary queries, we refine
the boundaries to be better aligned with those of ground truths, _i.e_.,
$\mathbf{A}^{(l+1)}=[\mathbf{p}^{(l+1)};\mathbf{d}_{s}^{(l+1)};\mathbf{d}_{e}^{(l+1)}]$
with sigmoid-based refinement as similar in the anchor update. Note that we
utilize deformable attention for the purpose of local feature aggregation,
which differs from the work [78] that uses it for efficient multi-scale global
operation. We provide comparison experiments of their roles in Appendix.

Figure 3: Boundary-focused attention layer for starting queries.

Moment prediction.  We repeatedly update the predictions through a total of
$L_{D}$ dual-pathway decoding layers. We denote the final predictions by
$\mathbf{A}=\begin{bmatrix}\mathbf{p};\mathbf{d}_{s};\mathbf{d}_{e}\end{bmatrix}$,
the anchor queries by $\mathbf{C}_{p}$, and the boundary queries by
$\mathbf{C}_{s}$ and $\mathbf{C}_{e}$. Then we cast the final predictions in
the form of starting and ending timestamps, _i.e_.,
$\hat{\varphi}=\begin{bmatrix}\mathbf{p}-\mathbf{d}_{s};\mathbf{p}+\mathbf{d}_{e}\end{bmatrix}\in\mathbb{R}^{M\times
2}$, which will serve as the final moment prediction results.

###  3.5 Quality-based Scoring

After producing the moment predictions, we opt to rank them for evaluation. In
the convention, query-based models utilize classification scores as the
measure, which exhibits how well the proposals semantically match the
sentence. However, it does not necessarily represent the localization
qualities of proposals. Hence we propose to estimate the localization quality
of each moment prediction. Owing to the complementarity of anchor and boundary
queries, we leverage all of them to predict the quality scores of the current
predictions. Formally, the quality score can be derived as
$\mathbf{q}=\sigma(\text{MLP}([\mathbf{C}_{p}\mathbin{\|}\mathbf{C}_{s}\mathbin{\|}\mathbf{C}_{e}]))\in\mathbb{R}^{M}$,
where $\sigma$ is sigmoid activation. The quality loss is defined as follows.

$\mathcal{L}_{\text{qual}}=\sum_{m=1}^{M}\Big{|}q_{m}-\max_{\forall
n}\big{(}\frac{|\hat{\varphi}_{m}\cap\varphi_{n}|}{|\hat{\varphi}_{m}\cup\varphi_{n}|}\big{)}\Big{|},$
(7)

where the objective of the quality head is to predict the maximum IoUs of the
proposals with ground-truth moments.

###  3.6 Matching

Following the standard [3, 22], we perform Hungarian matching [19] between
predictions and ground truths. The optimal matching results $\psi^{*}$ can be
derived as follows.

$\begin{aligned}
&\psi^{*}={\arg\min}_{\psi\in\mathfrak{G}_{N}}\sum_{n=1}^{N}\mathcal{C}(\varphi_{n},\hat{\varphi}_{\psi(n)}),\\\
\mathcal{C}(\varphi_{n},\hat{\varphi}_{\psi(n)})&=\lambda_{l_{1}}\mathcal{L}_{l_{1}}(\varphi_{n},\hat{\varphi}_{\psi(n)})+\lambda_{\text{iou}}\mathcal{L}_{\text{iou}}(\varphi_{n},\hat{\varphi}_{\psi(n)}),\end{aligned}$

where $\mathfrak{G}_{N}$ denotes the combination pool and $\psi(n)$ is the
index of the prediction matched by the $n$-th ground truth.
$\mathcal{L}_{l_{1}}$ and $\mathcal{L}_{\text{iou}}$ respectively represent
the $L_{1}$ distance and the generalized IoU [46] between the moments, while
$\lambda_{l_{1}}$ and $\lambda_{\text{iou}}$ are their weights. Note that in
contrast to other works, the classification term is not involved in the
matching process.

Once the matching is completed, we minimize the matching cost between each
pair of the matching results $\psi^{*}$. The moment localization loss can be
formulated as follows.

$\mathcal{L}_{\text{loc}}=\sum_{n=1}^{N}\mathcal{C}(\varphi_{n},\hat{\varphi}_{\psi^{*}(n)}).$
(8)

#### Overall training objectives.

Our model is trained in an end-to-end fashion and the overall objective is
defined as:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{loc}}+\lambda_{\text{qual}}\mathcal{L}_{\text{qual}}+\lambda_{\text{sal}}\mathcal{L}_{\text{sal}}+\lambda_{\text{regul}}\mathcal{L}_{\text{regul}},$
(9)

where $\lambda_{*}$ are the balancing parameters.

##  4 Experiments

###  4.1 Experimental Settings

Datasets. QVHighlights [22] is a recently built dataset, containing a total of
10,148 videos and 10,310 sentences from vlog and news domains. In addition to
moments, it provides segment-level saliency score annotations within each
moment. Importantly, it allows a single sentence corresponding to multiple
disjoint moments (1.8 on average). Due to this practical setup, we utilize
QVHighlights as the main benchmark for experiments. Charades-STA [11] includes
16,128 sentence-moment pairs with 9,848 indoor videos. The average duration of
videos and moments is 30.6 and 8.1 seconds, respectively. TACoS [45] contains
127 cooking videos encompassing a total of 18,818 sentence-moment pairs. This
dataset is known to be challenging since the moments occupy only a small
portion (6.1 sec on average) within considerably long videos (4.8 min on
average).

Evaluation metrics. Following the standard protocol, we measure the Recall@1
(R1) under the IoU thresholds of 0.3, 0.5, and 0.7 by default. Since
QVHighlights contains multiple ground-truth moments per sentence, we report
the mean average precision (mAP) with IoU thresholds of 0.5 and 0.75 as well
as the average mAP over a set of IoU thresholds [0.5:0.05:0.95]. Meanwhile, we
compute the mean IoU of top-1 predictions on Charades-STA and TACoS. Note that
the performances at high IoU thresholds (_e.g_., 0.7) exhibit how well the
predictions align with the ground truths.

| R1 |  | mAP
---|---|---|---
Method |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
w/o pre-training
MCN [1] |  11.41 |  2.72 |  |  24.94 |  8.22 |  10.67
CAL [9] |  25.49 |  11.54 |  |  23.40 |  7.65 |  9.89
CLIP [44] |  16.88 |  5.19 |  |  18.11 |  7.00 |  7.67
XML [21] |  46.69 |  33.46 |  |  47.89 |  34.67 |  34.90
Moment-DETR [22] |  52.89 |  33.02 |  |  54.82 |  29.40 |  30.73
UMT† [37] |  56.23 |  41.18 |  |  53.83 |  37.01 |  36.12
MH-DETR [60] |  60.05 |  42.48 |  |  60.75 |  38.13 |  38.38
QD-DETR [41] |  62.40 |  44.98 |  |  62.52 |  39.88 |  39.86
QD-DETR† [41] |  63.06 |  45.10 |  |  63.04 |  40.10 |  40.19
UniVTG [28] |  58.86 |  40.86 |  |  57.60 |  35.59 |  35.47
EaTR‡ [17] |  57.98 |  42.41 |  |  59.95 |  39.29 |  39.00
MomentDiff [26] |  57.42 |  39.66 |  |  54.02 |  35.73 |  35.95
BAM-DETR |  62.71 |  48.64 |  |  64.57 |  46.33 |  45.36
BAM-DETR† |  64.07 |  48.12 |  |  65.61 |  47.51 |  46.91
w/ pre-training on 4.2M data labeled by CLIP
UniVTG [28] |  65.43 |  50.06 |  |  64.06 |  45.02 |  43.63
w/ pre-training on 236K ASR captions
Moment-DETR [22] |  59.78 |  40.33 |  |  60.51 |  35.36 |  36.14
UMT† [37] |  60.83 |  43.26 |  |  57.33 |  39.12 |  38.08
QD-DETR [41] |  63.18 |  45.19 |  |  63.37 |  40.35 |  39.96
BAM-DETR |  63.88 |  47.92 |  |  66.33 |  48.22 |  46.67

†additional use of audio modality ‡reproduced by official checkpoint

Table 2: Results on the QVHighlights test split.

###  4.2 Implementation Details

For a fair comparison, we adopt the same feature extraction strategy with the
competitors [22, 41, 26]. Specifically, we adopt the CLIP [44] text features
for text and the concatenation of Slowfast [10] (ResNet-50) and CLIP [44]
(ViT-B/32) features for videos unless otherwise specified. The video features
are extracted every 1 second for Charades-STA and 2 seconds for QVHighlihgts
and TACoS. To compare with audio-augmented models [37], we optionally employ
audio features extracted by PANNs [18] pre-trained on AudioSet [14]. Due to
the long video duration, we uniformly sample 200 feature vectors from each
video for TACoS.

We set the embedding dimension $D$ to 256, the number of attention heads to 8,
the number of queries $M$ to 10, the number of boundary points $K$ to 3, and
the margin $\alpha$ to 0.2. We determine the numbers of encoding and decoding
layers as same with the prior work [41], _i.e_., $L_{E}=L_{D}=2$. The
balancing parameters are set as: $\lambda_{l1}=10$,
$\lambda_{\text{iou}}=\lambda_{\text{sal}}=\lambda_{\text{regul}}=1$,
$\lambda_{\text{qual}}=2$. As in previous works [26, 41], we increase
$\lambda_{\text{sal}}$ to 4 when saliency labels are unavailable, _i.e_., for
Charades-STA and TACoS. Our model is trained from scratch for 200 epochs on
QVHighlights and 100 epochs on the other datasets using the AdamW optimizer
[38] with a learning rate of 1e-4 and a batch size of 32.

###  4.3 Comparison with State-of-the-arts

Results on QVHighlights. We compare our model with existing state-of-the-arts
including recent query-based approaches [22, 37, 60, 41, 17, 26] on the test
split222Codalab QVHighlights Evaluation Server. As shown in Table 2, our BAM-
DETR consistently outperforms the comparative models under various settings.
In detail, without pre-training, our model surpasses the previous state-of-
the-art model [41] by large margins, _e.g_., 3.66% in<EMAIL_ADDRESS>and 6.45% in
<EMAIL_ADDRESS>These improvements under strict IoU thresholds verify the superior
localization ability of our method. When leveraging the audio features, the
performance further boosts especially in<EMAIL_ADDRESS>and mAPs, enlarging the gap
between the competitors including those with audio features [37, 41]. To
compare with the methods with pretraining, we pretrain our model on middle-
scale ASR caption data [22]. Again, our BAM-DETR achieves state-of-the-art
results, while showing the least gap with the method [28] that leverages
large-scale data for pre-training. Notably, our model even outperforms it in
terms of mAPs with much fewer (about 18$\times$) pre-training data,
manifesting the effectiveness of the proposed methods.

Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  mIoU
---|---|---|---|---
2D-TAN [73] |  58.76 |  46.02 |  27.40 |  41.25
VSLNet [69] |  60.30 |  42.69 |  24.14 |  41.58
Moment-DETR [22] |  65.83 |  52.07 |  30.59 |  45.54
QD-DETR [41] |  - |  57.31 |  32.55 |  -
UniVTG [28] |  70.81 |  58.01 |  35.65 |  50.10
MomentDiff [26] |  - |  55.57 |  32.42 |  -
BAM-DETR |  72.93 |  59.95 |  39.38 |  52.33
Table 3: Results on the Charades-STA test split.

Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  mIoU
---|---|---|---|---
2D-TAN [73] |  40.01 |  27.99 |  12.92 |  27.22
VSLNet [69] |  35.54 |  23.54 |  13.15 |  24.99
Moment-DETR [22] |  37.97 |  24.67 |  11.97 |  25.49
QD-DETR‡ [41] |  52.39 |  36.77 |  21.07 |  35.76
UniVTG [28] |  51.44 |  34.97 |  17.35 |  33.60
MomentDiff‡ [26] |  46.64 |  28.92 |  12.37 |  30.36
BAM-DETR |  56.69 |  41.54 |  26.77 |  39.31

‡reproduced by official codebase

Table 4: Results on the TACoS test split.

Results on Charades-STA. Experimental results on the test split are shown in
Table 3. Not only does our BAM-DETR outperform the query-based competitors
[22, 41, 26], but it achieves a new state-of-the-art by surpassing the best
performing anchor-free model [28] for all metrics. Notably, a large
performance gap of 3.73% is observed in<EMAIL_ADDRESS>which confirms the strong
localization ability of our model.

Results on TACoS. We present the comparison results on the test set in Table
4. It can be observed that our BAM-DETR achieves a new state-of-the-art with
pronounced performances under the strict IoU thresholds, which is consistent
with the above results on other datasets. On this challenging benchmark, our
model surpasses the previous best model [41] by 5.7% (relatively 27%) in
<EMAIL_ADDRESS>These results clearly exhibit the superiority of the proposed model.

###  4.4 Robustness Evaluation

Query-based models potentially have a temporal bias [65, 16] against the
locations and lengths of moments. To measure robustness, we evaluate our model
on the anti-biased Charades-STA [26] with distribution shifts of the moment
location and length between training and test sets. Table 5 summarizes the
results, where our model outperforms all competitors under both anti-biased
settings. Especially, it shows significant performance gaps under the moment
length bias. This can be expected since our model directly localizes
boundaries instead of predicting lengths, which lessens the effect of bias.
This robustness test corroborates the advantage of our boundary-oriented
moment modeling.

Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  mAPavg
---|---|---|---|---
w.r.t. moment location
2D-TAN [73] |  27.81 |  20.44 |  10.84 |  17.23
MMN [56] |  33.58 |  27.20 |  14.12 |  19.18
Moment-DETR [22] |  29.94 |  22.16 |  11.56 |  18.66
QD-DETR‡ [41] |  56.17 |  46.82 |  28.13 |  30.70
MomentDiff [26] |  48.39 |  33.59 |  15.71 |  21.37
BAM-DETR |  59.83 |  50.00 |  32.08 |  31.68
w.r.t. moment length
2D-TAN [73] |  39.68 |  28.68 |  17.72 |  22.79
MMN [56] |  43.58 |  34.31 |  19.94 |  26.85
Moment-DETR[22] |  42.73 |  34.39 |  16.12 |  24.02
QD-DETR‡ [41] |  67.39 |  54.44 |  32.87 |  36.99
MomentDiff [26] |  51.25 |  38.32 |  23.38 |  28.19
BAM-DETR |  68.40 |  55.46 |  40.74 |  43.21

‡reproduced by official codebase

Table 5: Results on the anti-biased Charades-STA test split against the moment length and location. To enable a fair comparison, the VGG [52] and Glove [43] features are employed for all models. | R1 |  | mAP
---|---|---|---
Method |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
Baseline |  62.39 |  47.87 |  |  62.64 |  41.54 |  41.75
$+$ boundary-oriented modeling |  63.42 |  49.23 |  |  62.86 |  43.24 |  42.42
$+$ dual-pathway decoder |  63.61 |  50.26 |  |  63.01 |  44.98 |  44.16
$+$ quality-based scoring |  65.10 |  51.61 |  |  65.41 |  48.56 |  47.61
Table 6: Ablation study of components on QVHighlights. Table 7: Ablative
experiments on QVHighlights. Default settings are marked gray.

| R1 |  | mAP
---|---|---|---
$M$ |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
5 |  62.19 |  49.03 |  |  62.26 |  46.50 |  45.18
10 |  65.10 |  51.61 |  |  65.41 |  48.56 |  47.61
15 |  65.48 |  50.32 |  |  65.34 |  49.77 |  48.23
20 |  65.23 |  51.61 |  |  64.56 |  49.40 |  48.01

(a)

| R1 |  | mAP
---|---|---|---
Memory |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
plain |  63.74 |  49.68 |  |  64.10 |  47.74 |  46.21
sensitive |  63.81 |  49.87 |  |  65.08 |  46.62 |  46.34
merged |  65.10 |  51.61 |  |  65.41 |  48.56 |  47.61
(b)

| R1 |  | mAP
---|---|---|---
$K$ |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
1 (fixed) |  63.23 |  48.32 |  |  63.53 |  46.42 |  45.17
1 |  64.32 |  49.87 |  |  65.01 |  48.72 |  47.14
3 |  65.10 |  51.61 |  |  65.41 |  48.56 |  47.61
5 |  64.90 |  50.45 |  |  65.38 |  47.58 |  46.70
(c)

|  |  | R1 |  |  | mAP
---|---|---|---|---|---|---
$\mathbf{C}_{p}$ |  $\mathbf{C}_{s}$ |  $\mathbf{C}_{e}$ |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
✓ |  |  |  62.97 |  49.03 |  |  64.44 |  47.50 |  46.26
|  ✓ |  |  64.39 |  50.71 |  |  64.60 |  47.69 |  46.66
|  |  ✓ |  64.58 |  49.23 |  |  64.38 |  47.85 |  46.71
✓ |  ✓ |  ✓ |  65.10 |  51.61 |  |  65.41 |  48.56 |  47.61
(d)

Anchor |  Boundary | R1 |  | mAP |  |  |  |
---|---|---|---|---|---|---|---|---
query |  query |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg. |  |  FLOPs |  |  Params
Global (shared) |  62.26 |  49.03 |  |  62.63 |  45.90 |  44.18 |  |  0.71G |  |  8.2M
Global |  Global |  63.61 |  49.81 |  |  64.11 |  45.38 |  44.54 |  |  0.72G |  |  11.5M
Global |  Focused |  63.74 |  49.68 |  |  64.10 |  47.74 |  46.21 |  |  0.65G |  |  9.5M
Focused |  Focused |  62.48 |  49.16 |  |  63.90 |  47.31 |  45.87 |  |  0.62G |  |  7.6M

(e)

(f) Boundary hit rate with varying band widths.

(g) Visualization of the ground-truth moment and predictions by competitors.

Figure 4: Analytical experiments on QVHighlights.

###  4.5 Analysis

We conduct analytic experiments on the QVHighlights validation split. More
experiments are included in Appendix.

Effect of each component. We analyze the effect of each component in Table 6.
The direct adoption of our moment modeling solely improves the performance,
particularly at strict thresholds, showcasing its advantage in boundary
alignment. Employing the dual-pathway decoder leads to further gains, which
suggests the essential role of separate pathways. Lastly, the quality-based
scoring considerably elevates the scores, especially in terms of mAPs.

Number of predictions. We experiment with varying numbers of predictions in
LABEL:tab:num_queries, where the model achieves robust results when $M$ is
sufficiently large. By default, we set $M$ to 10 as same with the previous
works [22, 41].

Locality-enhanced features. We analyze the effect of the choice of input
features for boundary-focused attention in LABEL:tab:boundary_features. The
results indicate that boundary-sensitive features are more helpful than plain
ones in precise localization. In addition, owing to their complementarity, the
merged features enable achieving the best performance.

Number of sampled boundary points. We analyze the effect of the number of
sampled boundary points $K$ in LABEL:tab:num_boundary_points. For comparisons,
we report the case where the fixed boundary features are sampled without using
offsets (1${}^{\text{st}}$ row). As shown in the table, the dynamic selection
of neighborhoods rather than fixed boundaries is important for accurate moment
localization. The performance improves when sampling multiple points, while it
saturates at $K=3$.

Queries for quality prediction. We leverage both anchor and boundary queries
for quality prediction. We investigate the effect of query choices in
LABEL:tab:quality_input. As a result, utilizing all the queries shows better
localization performance than using one of them. This might be trivial as
different queries contain complementary information for quality prediction.

Attention layers. LABEL:tab:attn_combination compares different combinations
of attention layers in terms of performances and costs. For an apple-to-apple
comparison, we leverage the plain memory features for both attention layers.
First of all, employing separate global cross-attention for different queries
leads to a huge parameter increase yet limited score gains. Replacing the
global one for boundary queries with our focused attention layer substantially
reduces the cost, while achieving the best performance. Meanwhile, the use of
focused attention for both queries leads to inferior results.

Boundary alignment. Standard IoU-based metrics are indirect measures for
boundary alignment. To precisely diagnose the ability, inspired by Deeplab
[6], we compute the boundary hit rate of predictions with varying band widths.
We expand ground-truth boundary points with a band width to form
starting/ending zones and regard a prediction as correct if both of its
boundaries fall within the corresponding zones. More details are in Appendix.
To disentangle the effect of ranking, we mark a video as correct if at least
one prediction is correct and measure the video-level hit rate. As shown in
Fig. 4, our model greatly outperforms the recent competitors. The sharp
increase in low band widths validates the superiority of our model in boundary
alignment.

Qualitative results. As shown in Fig. 4, previous models fail to localize
accurate moments with misleading center predictions. In contrast, our model
directly predicts well-aligned boundaries. More visualizations are in
Appendix.

##  5 Conclusion

In this paper, we identified the center misalignment issue of existing query-
based models for sentence grounding. To address it, we presented boundary-
oriented moment modeling where boundaries are directly predicted without
relying on centers. Based on the modeling, we designed the boundary-aligned
moment detection transformer characterized by dual-pathway decoding. Further,
we proposed localization quality-based scoring of predictions. The efficacy of
the proposed methods is validated by thorough examinations. We hope this work
sheds light on the issue of current center-based moment modeling in detection
transformers.

Acknowledgements. This project was supported by the National Research
Foundation of Korea grant funded by the Korea government (MSIT) (No.
2022R1A2B5B02001467).

## References

  * Anne Hendricks et al. [2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In _ICCV_ , pages 5803–5812, 2017.
  * Cao et al. [2021] Meng Cao, Long Chen, Mike Zheng Shou, Can Zhang, and Yuexian Zou.  On pursuit of designing multi-modal transformer for video grounding.  In _EMNLP_ , pages 9810–9823, 2021.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _ECCV_ , pages 213–229. Springer, 2020.
  * Chen et al. [2018] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua.  Temporally grounding natural sentence in video.  In _EMNLP_ , pages 162–171, 2018.
  * Chen et al. [2020] Long Chen, Chujie Lu, Siliang Tang, Jun Xiao, Dong Zhang, Chilie Tan, and Xiaolin Li.  Rethinking the bottom-up framework for query-based video localization.  In _AAAI_ , pages 10551–10558, 2020.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _PAMI_ , 40(4):834–848, 2017.
  * Dai et al. [2021] Xiyang Dai, Yinpeng Chen, Jianwei Yang, Pengchuan Zhang, Lu Yuan, and Lei Zhang.  Dynamic detr: End-to-end object detection with dynamic attention.  In _ICCV_ , pages 2988–2997, 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Escorcia et al. [2019] Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell.  Temporal localization of moments in video collections with natural language.  _arXiv preprint arXiv:1907.12763_ , 2019.
  * Feichtenhofer et al. [2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In _ICCV_ , pages 6202–6211, 2019.
  * Gao et al. [2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  Tall: Temporal activity localization via language query.  In _CVPR_ , pages 5267–5275, 2017.
  * Gao et al. [2021] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fast convergence of detr with spatially modulated co-attention.  In _ICCV_ , pages 3621–3630, 2021.
  * Ge et al. [2019] Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia.  Mac: Mining activity concepts for language-based temporal localization.  In _WACV_ , pages 245–253. IEEE, 2019.
  * Gemmeke et al. [2017] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In _ICASSP_ , pages 776–780. IEEE, 2017.
  * Ghosh et al. [2019] Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander G Hauptmann.  Excl: Extractive clip localization using natural language descriptions.  In _NAACL_ , pages 1984–1990, 2019.
  * Hao et al. [2022] Jiachang Hao, Haifeng Sun, Pengfei Ren, Jingyu Wang, Qi Qi, and Jianxin Liao.  Can shuffling video benefit temporal bias problem: A novel training framework for temporal grounding.  In _ECCV_ , pages 130–147. Springer, 2022.
  * Jang et al. [2023] Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn.  Knowing where to focus: Event-aware transformer for video grounding.  In _ICCV_ , pages 13846–13856, 2023.
  * Kong et al. [2020] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  _IEEE/ACM Trans. on Audio, Speech, and Language Processing_ , 28:2880–2894, 2020.
  * Kuhn [1955] Harold W Kuhn.  The hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 2(1-2):83–97, 1955.
  * Lee et al. [2023] Pilhyeon Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, and Hyeran Byun.  Decomposed cross-modal distillation for rgb-based temporal action detection.  In _CVPR_ , pages 2373–2383, 2023.
  * Lei et al. [2020] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal.  Tvr: A large-scale dataset for video-subtitle moment retrieval.  In _ECCV_ , pages 447–463. Springer, 2020.
  * Lei et al. [2021] Jie Lei, Tamara L Berg, and Mohit Bansal.  Detecting moments and highlights in videos via natural language queries.  In _Neurips_ , pages 11846–11858, 2021.
  * Li et al. [2022] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang.  Dn-detr: Accelerate detr training by introducing query denoising.  In _CVPR_ , pages 13619–13627, 2022.
  * Li et al. [2023a] Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, and Lionel M Ni.  Lite detr: An interleaved multi-scale encoder for efficient detr.  In _CVPR_ , pages 18558–18567, 2023a.
  * Li et al. [2023b] Hongxiang Li, Meng Cao, Xuxin Cheng, Yaowei Li, Zhihong Zhu, and Yuexian Zou.  G2l: Semantically aligned and uniform video grounding via geodesic and game theory.  In _ICCV_ , pages 12032–12042, 2023b.
  * Li et al. [2023c] Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang.  Momentdiff: Generative video moment retrieval from random to real.  In _Neurips_ , 2023c.
  * Lin et al. [2021] Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu.  Learning salient boundary feature for anchor-free temporal action localization.  In _CVPR_ , pages 3320–3329, 2021.
  * Lin et al. [2023a] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou.  Univtg: Towards unified video-language temporal grounding.  In _ICCV_ , pages 2794–2804, 2023a.
  * Lin et al. [2018] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In _ECCV_ , pages 3–19, 2018.
  * Lin et al. [2023b] Yutong Lin, Yuhui Yuan, Zheng Zhang, Chen Li, Nanning Zheng, and Han Hu.  Detr does not need multi-scale or locality design.  In _ICCV_ , pages 6545–6554, 2023b.
  * Liu et al. [2021a] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie.  Context-aware biaffine localizing network for temporal sentence grounding.  In _CVPR_ , pages 11235–11244, 2021a.
  * Liu et al. [2021b] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie.  Context-aware biaffine localizing network for temporal sentence grounding.  In _CVPR_ , pages 11235–11244, 2021b.
  * Liu et al. [2021c] Fanfan Liu, Haoran Wei, Wenzhe Zhao, Guozhen Li, Jingquan Peng, and Zihao Li.  Wb-detr: transformer-based detector without backbone.  In _ICCV_ , pages 2979–2987, 2021c.
  * Liu et al. [2018] Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua.  Cross-modal moment localization in videos.  In _ACM MM_ , pages 843–851, 2018.
  * Liu et al. [2022a] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  Dab-detr: Dynamic anchor boxes are better queries for detr.  In _ICLR_ , 2022a.
  * Liu et al. [2023] Shilong Liu, Tianhe Ren, Jiayu Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, and Lei Zhang.  Detection transformer with stable matching.  In _ICCV_ , pages 6491–6500, 2023.
  * Liu et al. [2022b] Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie.  Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection.  In _CVPR_ , pages 3042–3051, 2022b.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _ICLR_ , 2019.
  * Lu et al. [2019] Chujie Lu, Long Chen, Chilie Tan, Xiaolin Li, and Jun Xiao.  Debug: A dense bottom-up grounding approach for natural language video localization.  In _EMNLP-IJCNLP_ , pages 5144–5153, 2019.
  * Meng et al. [2021] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang.  Conditional detr for fast training convergence.  In _ICCV_ , pages 3651–3660, 2021.
  * Moon et al. [2023] WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo.  Query-dependent video representation for moment retrieval and highlight detection.  In _CVPR_ , pages 23023–23033, 2023.
  * Mun et al. [2020] Jonghwan Mun, Minsu Cho, and Bohyung Han.  Local-global video-text interactions for temporal grounding.  In _CVPR_ , pages 10810–10819, 2020.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _EMNLP_ , pages 1532–1543, 2014.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763. PMLR, 2021.
  * Regneri et al. [2013] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  _Trans. Assoc. Comput. Linguistics_ , 1:25–36, 2013.
  * Rezatofighi et al. [2019] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In _CVPR_ , pages 658–666, 2019.
  * Roh et al. [2022] Byungseok Roh, JaeWoong Shin, Wuhyun Shin, and Saehoon Kim.  Sparse detr: Efficient end-to-end object detection with learnable sparsity.  In _ICLR_ , 2022.
  * Rui et al. [2000] Yong Rui, Anoop Gupta, and Alex Acero.  Automatically extracting highlights for tv baseball programs.  In _ACM MM_ , pages 105–115, 2000.
  * Shao et al. [2018] Dian Shao, Yu Xiong, Yue Zhao, Qingqiu Huang, Yu Qiao, and Dahua Lin.  Find and focus: Retrieve and localize video events with natural language queries.  In _ECCV_ , pages 200–216, 2018.
  * Sharghi et al. [2017] Aidean Sharghi, Jacob S Laurel, and Boqing Gong.  Query-focused video summarization: Dataset, evaluation, and a memory network based approach.  In _CVPR_ , pages 4788–4797, 2017.
  * Shi et al. [2023] Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao.  Tridet: Temporal action detection with relative boundary modeling.  In _CVPR_ , pages 18857–18866, 2023.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_ , 2015.
  * Sun et al. [2021] Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris M Kitani.  Rethinking transformer-based set prediction for object detection.  In _ICCV_ , pages 3611–3620, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Neurips_ , 2017.
  * Wang et al. [2022a] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun.  Anchor detr: Query design for transformer-based detector.  In _AAAI_ , pages 2567–2575, 2022a.
  * Wang et al. [2022b] Zhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li, and Gangshan Wu.  Negative sample matters: A renaissance of metric learning for temporal grounding.  In _AAAI_ , pages 2613–2623, 2022b.
  * Xiao et al. [2021] Shaoning Xiao, Long Chen, Songyang Zhang, Wei Ji, Jian Shao, Lu Ye, and Jun Xiao.  Boundary proposal network for two-stage natural language video localization.  In _AAAI_ , pages 2986–2994, 2021.
  * Xiong et al. [2019] Bo Xiong, Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman.  Less is more: Learning highlight detection from video duration.  In _CVPR_ , pages 1258–1267, 2019.
  * Xu et al. [2019] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko.  Multilevel language and vision integration for text-to-clip retrieval.  In _AAAI_ , pages 9062–9069, 2019.
  * Xu et al. [2023] Yifang Xu, Yunzhuo Sun, Yang Li, Yilei Shi, Xiaoxiang Zhu, and Sidan Du.  Mh-detr: Video moment and highlight detection with cross-modal transformer.  In _ACM MM_ , 2023.
  * Yan et al. [2023] Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid.  Unloc: A unified framework for video localization tasks.  In _ICCV_ , pages 13623–13633, 2023.
  * Ye et al. [2023] Mingqiao Ye, Lei Ke, Siyuan Li, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu.  Cascade-detr: Delving into high-quality universal object detection.  In _ICCV_ , pages 6704–6714, 2023.
  * Yuan et al. [2019a] Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu.  Semantic conditioned dynamic modulation for temporal sentence grounding in videos.  In _Neurips_ , 2019a.
  * Yuan et al. [2019b] Yitian Yuan, Tao Mei, and Wenwu Zhu.  To find where you talk: Temporal sentence localization in video with attention based location regression.  In _AAAI_ , pages 9159–9166, 2019b.
  * Yuan et al. [2021] Yitian Yuan, Xiaohan Lan, Xin Wang, Long Chen, Zhi Wang, and Wenwu Zhu.  A closer look at temporal sentence grounding in videos: Dataset and metric.  In _Proc. 2nd Int. Workshop on Human-Centric Multimedia Analysis_ , pages 13–21, 2021.
  * Zeng et al. [2020] Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan.  Dense regression network for video grounding.  In _CVPR_ , pages 10287–10296, 2020.
  * Zhang et al. [2019a] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis.  Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment.  In _CVPR_ , pages 1247–1257, 2019a.
  * Zhang et al. [2022] Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu.  Accelerating detr convergence via semantic-aligned matching.  In _CVPR_ , pages 949–958, 2022.
  * Zhang et al. [2020a] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou.  Span-based localizing network for natural language video localization.  In _ACL_ , 2020a.
  * Zhang et al. [2023] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  In _ICLR_ , 2023.
  * Zhang et al. [2016] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Video summarization with long short-term memory.  In _ECCV_ , pages 766–782. Springer, 2016.
  * Zhang et al. [2019b] Songyang Zhang, Jinsong Su, and Jiebo Luo.  Exploiting temporal relationships in video moment localization with natural language.  In _ACM MM_ , pages 1230–1238, 2019b.
  * Zhang et al. [2020b] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.  Learning 2d temporal adjacent networks for moment localization with natural language.  In _AAAI_ , pages 12870–12877, 2020b.
  * Zhang et al. [2019c] Zhu Zhang, Zhijie Lin, Zhou Zhao, and Zhenxin Xiao.  Cross-modal interaction networks for query-based moment retrieval in videos.  In _ACM SIGIR_ , pages 655–664, 2019c.
  * Zhao et al. [2017] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  In _ICCV_ , pages 2914–2923, 2017.
  * Zheng et al. [2023] Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang.  Less is more: Focus attention for efficient detr.  In _ICCV_ , pages 6674–6683, 2023.
  * Zhu et al. [2019] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In _CVPR_ , pages 9308–9316, 2019.
  * Zhu et al. [2021] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In _ICLR_ , 2021.

This appendix provides the following content.

  * •

Formulation of saliency losses (Sec. A)

  * •

Comparison with Deformable DETR (Sec. B)

  * •

Details of boundary alignment evaluation (Sec. C)

  * •

More analyses of the proposed methods (Sec. D)

  * •

Further qualitative results (Sec. E)

##  A Formulation of Saliency Losses

As mentioned in the main paper, we adopt saliency losses for effective
multimodal alignment in the encoder, following the convention [22, 41]. In
specific, our total saliency-based loss is composed of three losses, _i.e_.,
$\mathcal{L}_{\text{sal}}=\mathcal{L}_{\text{margin}}+\mathcal{L}_{\text{cont}}+\mathcal{L}_{\text{neg}}$.
The margin-based loss $\mathcal{L}_{\text{margin}}$, as defined in Eq. 2 of
the main paper, aims to encourage the model to produce higher saliency scores
for the clips relevant to the given sentence compared to less related clips.
Meanwhile, the rank-aware contrastive loss $\mathcal{L}_{\text{cont}}$ is
utilized to preserve the ground-truth clip ranking in prediction scores. To be
concrete, we define the positive and negative sets based on a reference score
$r$: clips with saliency score labels higher than $r$ form the positive set
$\mathcal{B}_{r}^{+}$, and the remaining clips constitute the negative set
$\mathcal{B}_{r}^{-}$. The rank-aware contrastive loss is then formulated with
a set of reference scores $\mathcal{R}$ as:

$\mathcal{L}_{\text{cont}}=-\sum_{\forall
r\in\mathcal{R}}\log\frac{\sum_{\forall\hat{v}\in\mathcal{B}_{r}^{+}}\text{exp}(S(\hat{v})/\tau)}{\sum_{\forall\hat{v}\in(\mathcal{B}_{r}^{+}\cup\mathcal{B}_{r}^{-})}\text{exp}(S(\hat{v})/\tau)},$
(10)

where $S(\cdot)$ is a saliency score predictor and $\tau$ is a temperature
(set to 0.5). We define $\mathcal{R}$ to be the set of saliency score labels
of positive clips within ground-truth moments.

The negative relation loss is based on the assumption that all video clips
should exhibit low saliency scores when paired with unmatched (negative)
sentences. Formally, the loss is defined as follows.

$\mathcal{L}_{\text{neg}}=-\sum_{\forall\hat{v}^{\text{neg}}\in\hat{\mathcal{V}}^{\text{neg}}}\log(1-S(\hat{v}^{\text{neg}})),$
(11)

where $\hat{\mathcal{V}}^{\text{neg}}$ denotes the memory features obtained by
processing the video with a negative sentence through the encoder. In our
implementation, a negative sentence is sampled from a different video-sentence
pair in the mini-batch.

##  B Comparison with Deformable DETR

Our boundary-focused attention layer incorporates deformable attention, which
is first proposed in Deformable DETR [78]. It was originally designed for
computationally efficient global attention with multi-scale features in object
detection. In contrast, we employ deformable attention for local aggregation
of neighbor features, aiding precise boundary prediction in temporal sentence
grounding. To elucidate the discrepancy in their roles, we conduct a
comparative experiment. For this study, we implement a 1D variant of single-
scale Deformable DETR, tailored for temporal sentence grounding. We apply the
proposed quality-based scoring to this model for a fair comparison. Its key
differences with our BAM-DETR lie in the moment formulation (center-based vs.
boundary-oriented) and the design of decoding layers (single-pathway vs. dual-
pathway).

Figure A1: Offset histogram on the QVHighlights validation split. | R1 |  | mAP
---|---|---|---
Method |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
Deformable DETR |  60.52 |  49.35 |  |  62.53 |  46.41 |  44.73
BAM-DETR (Ours) |  65.10 |  51.61 |  |  65.41 |  48.56 |  47.61
Table A1: Results on the QVHighlights validation split.

To analyze the behavior of deformable attention, we look into the absolute
values of predicted offsets, _i.e_., how distant features are referenced
during the attention process. These offsets can indicate whether the attention
is responsible for global or local interaction. Fig. A1 shows a comparison
between the normalized histograms of predicted offsets from two comparative
methods. In our BAM-DETR, the deformable attention primarily concentrates on
the neighbor features near boundaries, _e.g_., over 80% of offsets are shorter
than 5 seconds. Conversely, in the case of Deformable DETR, the deformable
attention strives to aggregate global information, _e.g_., about 45% of
offsets are longer than 10 seconds. These results clearly confirm the
different roles of deformable attention in the two models. In addition, we
compare the grounding performance in Table A1, where our BAM-DETR
substantially outperforms Deformable-DETR. This underscores the importance of
our boundary-oriented moment modeling as well as the design of dual-pathway
decoding layers.

##  C Details of Boundary Alignment Evaluation

We provide more details regarding the experimental setup of boundary alignment
evaluation performed in Fig. 4 of the main manuscript. Inspired by the trimap
evaluation of DeepLab [6], we propose a novel metric of boundary hit rate
under varying band widths to evaluate the degree of boundary alignment. In
detail, we expand boundary points of the $n$-th ground truth
$\\{t_{s_{n}},t_{e_{n}}\\}$ with a given band width of $l_{w}$ to form
boundary zones. We can denote the starting and ending zones by
$Z_{s_{n}}=[t_{s_{n}}-0.5l_{w},t_{s_{n}}+0.5l_{w}]$ and
$Z_{e_{n}}=[t_{e_{n}}-0.5l_{w},t_{e_{n}}+0.5l_{w}]$, respectively. Then, for
the $m$-th proposal $\\{\hat{t}_{s_{m}},\hat{t}_{e_{m}}\\}$, we check whether
both of its boundaries fall in the corresponding zones. We iterate this
process for all combinations of ground truths and predictions, and mark a
video as correct if any pair is positive. Formally, the binary variable of $h$
of a video is defined as:

$\displaystyle h=$ $\displaystyle\max_{\forall
n,m}\Big{[}\text{Hit}^{s}(n,m)\cdot\text{Hit}^{e}(n,m)\Big{]},$
$\displaystyle\text{where}~{}~{}\text{Hit}^{z}(n,m)$
$\displaystyle=\mathbbm{1}\big{[}|\hat{t}_{z_{m}}-t_{z_{n}}|\leq
0.5l_{w}\big{]},~{}~{}z\in\\{s,e\\}.$

We measure the hit rate over the whole validation set.

##  D More Analyses

Ablation on loss functions. Our model employs several loss functions for
training. We conduct an ablative experiment to diagnose their effects. Table
A2 summarizes the results, where the upper part adopts the typical
classification-based scoring whereas the lower one leverages our proposed
quality-based scoring. We first examine the benefit of saliency losses.
Consistent with the recent findings [22], we observe notable performance gains
brought by the better cross-modal alignment in the encoder. Then we
investigate the importance of our regularization loss (_cf_., Eq. 5 of the
main paper) designed for boundary-sensitive feature construction. It can be
observed that regardless of the choice of scoring methods, the boundary
regularization leads to significant performance boosts. Putting together the
results in LABEL:tab:boundary_features of the main paper, it becomes clear
that boundary-sensitive features are essential for precise boundary updating.
Lastly, the comparison between the two separate parts validates the efficacy
of our quality-based scoring.

Comparison between scoring methods. We present the quality-based scoring
method to replace the conventional classification-based one. To compare two
scoring methods, we draw scatter plots of scores vs. IoUs with ground truths
using all predictions on the QVHighlights validation set. Fig. A2 indicates
that classification scores correlate with IoUs to an extent. Meanwhile, we
observe in Fig. A2 that our quality-based scoring shows a much stronger
correlation, leading to better proposal ranking. These results validate its
efficacy in estimating the localization qualities of proposals, rather than
the matching degree.

Generalizablity of the quality-based scoring. By design, our quality-based
scoring is generalizable to any query-based approach. To investigate this
property, we conduct experiments with three representative models: Moment-DETR
[22], QD-DETR [41], and EaTR [17]. The results are shown in Table A3, where
the proposed scoring method brings consistent improvements over different
baselines. Noticeably, we observe the pronounced gains at high IoU thresholds,
indicating better fitting of proposals with the ground truths. This
corroborates our claim that moment proposals ought to be ranked based on their
localization qualities rather than the degree of matching.

|  |  |  |  | R1 |  |  | mAP
---|---|---|---|---|---|---|---|---
$\mathcal{L}_{\text{loc}}$ |  $\mathcal{L}_{\text{cls}}$ |  $\mathcal{L}_{\text{qual}}$ |  $\mathcal{L}_{\text{sal}}$ |  $\mathcal{L}_{\text{regul}}$ |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
✓ |  ✓ |  |  |  |  56.77 |  41.03 |  |  58.63 |  39.25 |  39.12
✓ |  ✓ |  |  ✓ |  |  60.58 |  46.65 |  |  62.09 |  43.83 |  42.94
✓ |  ✓ |  |  ✓ |  ✓ |  63.61 |  50.26 |  |  63.01 |  44.98 |  44.16
✓ |  |  ✓ |  |  |  59.23 |  46.13 |  |  60.52 |  44.80 |  43.48
✓ |  |  ✓ |  ✓ |  |  63.23 |  50.00 |  |  64.03 |  47.42 |  46.64
✓ |  |  ✓ |  ✓ |  ✓ |  65.10 |  51.61 |  |  65.41 |  48.56 |  47.61
Table A2: Ablation study on the loss functions on QVHighlights.

(a) Classification-based scoring

(b) Quality-based scoring

Figure A2: Correlation between scores and IoUs with ground truths: (a) the
classification scores show a moderate correlation (Pearson’s r of 0.44); (b)
the quality scores exhibit a stronger correlation (Pearson’s r of 0.67).

| R1 |  | mAP
---|---|---|---
Method |  @0.5 |  @0.7 |  |  @0.5 |  @0.75 |  Avg.
Moment-DETR‡ [22] |  53.23 |  34.00 |  |  54.80 |  29.02 |  30.58
$+$ quality-based scoring |  56.77 |  38.65 |  |  55.09 |  35.30 |  34.98
QD-DETR‡ [41] |  62.90 |  46.77 |  |  62.66 |  41.51 |  41.24
$+$ quality-based scoring |  64.26 |  50.32 |  |  63.79 |  46.03 |  44.50
EaTR‡ [17] |  57.74 |  42.71 |  |  59.40 |  39.34 |  39.06
$+$ quality-based scoring |  59.42 |  45.61 |  |  60.24 |  42.29 |  41.61

‡All models are reproduced by official codebase

Table A3: Generalizability evaluation of quality-based scoring on the
QVHighlights validation split.

##  E Further Qualitative Results

We perform further qualitative comparisons with previous query-based methods
in Fig. A3 and Fig. A4. The comparison results across various scenarios
demonstrate the superiority of our BAM-DETR over strong competitors.

Figure A3: Qualitative comparison on the QVHighlights validation split. Figure
A4: Qualitative comparison on the QVHighlights validation split.