# MRTNet: Multi-Resolution Temporal Network for Video Sentence Grounding

Wei Ji  National University of SingaporeSingapore<EMAIL_ADDRESS>,  Long
Chen  Columbia UniversityUSA<EMAIL_ADDRESS>,  Yinwei Wei  National
University of SingaporeSingapore<EMAIL_ADDRESS>,  Yiming Wu  The
University of SydneyAustralia<EMAIL_ADDRESS>and  Tat-Seng Chua  National
University of SingaporeSingapore<EMAIL_ADDRESS>
###### Abstract.

Given an untrimmed video and natural language query, video sentence grounding
aims to localize the target temporal moment in the video. Existing methods
mainly tackle this task by matching and aligning semantics of the descriptive
sentence and video segments on a single temporal resolution, while neglecting
the temporal consistency of video content in different resolutions. In this
work, we propose a novel multi-resolution temporal video sentence grounding
network: MRTNet, which consists of a multi-modal feature encoder, a Multi-
Resolution Temporal (MRT) module, and a predictor module. MRT module is an
encoder-decoder network, and output features in the decoder part are in
conjunction with Transformers to predict the final start and end timestamps.
Particularly, our MRT module is hot-pluggable, which means it can be
seamlessly incorporated into any anchor-free models. Besides, we utilize a
loss function to supervise cross-modal features in MRT module for more
accurate grounding in three scales: frame-level, clip-level and sequence-
level. Extensive experiments on three prevalent datasets have shown the
effectiveness of MRTNet.

video sentence grounding, temporal resolution, cross-modal

††copyright: none††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze
Detection, June 03–05, 2018, Woodstock, NY††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††submissionid: 2627††ccs: Computing methodologies
Visual content-based indexing and retrieval

##  1\. Introduction

Video sentence grounding aims to localize the temporal moments from the
untrimmed video corresponding to the given descriptive language query. As an
important visual-language task, video sentence grounding involves both
computer vision (CV) and natural language processing (NLP) techniques, and is
beneficial to numerous down-stream video understanding tasks, such as video
dialog (Chu et al., 2021), video relationship detection (Shang et al., 2017,
2021; Ji et al., 2021), and video question answering (Zhong et al., 2022; Xiao
et al., 2022; Li et al., 2022), _etc_. Clearly, the cross-modal semantic
consistency is essential for video sentence grounding.

Figure 1. An illustration of localizing a temporal moment in an untrimmed
video by a language query. The 1D temporal map shows the response area in the
whole video sequence, according to the language query. As shown above, if
video is compressed with different sampling rates, these 1D temporal maps will
display in high, middle, and low resolutions, while the relative locations of
target regions stay unchanged.

Prior works primarily treat video sentence grounding as a ranking problem,
_i.e._ , applying a multi-modal matching architecture to find the best
matching moment for the given language query, which can be treated as
proposal-based methods. Recently, some works explore the cross-modal matching
between video and query, and then directly regress the temporal locations of
target moment, which is classified as proposal-free methods. Commonly, the
structure of current proposal-free video language grounding methods mainly
includes these modules: visual and language encoders module, multi-modal
attention module, and predictor module. The results calculated by the
predictor module heavily rely on the multi-modal features with attention.
However, these methods neglect the temporal consistency of video data in
multi-resolutions, which is beneficial for better video feature
representation.

In this paper, we try to deal with the video sentence grounding task from
another perspective: In the 1D temporal dimension, the target range in video
can be treated as the foreground region, and the rest of the video is viewed
as the background. The essence of video sentence grounding is to localize the
maximum response area in 1D temporal range according to the descriptive query.
As we known, attention map can also be treated as the maximum response area in
the temporal dimension. Hence, context information in the whole video is
crucial for locating the accurate boundaries of temporal segments.

On the basis of the previous network design, we propose a plug-and-play module
to constrains the temporal consistency of video in different resolutions. In
the module design, we aim to quickly localize the relative location of the
target range in low resolution of the video by downsampling, and then refine
the accurate boundaries in the decoder part when upsampling to the original
resolution. Inspired by the observations above, we propose a Multi-Resolution
Temporal Network (MRTNet) for the video sentence grounding task to
simultaneously address all the above-mentioned limitations. It mainly consists
of a multi-model feature encoder, a multi-resolution temporal (MRT) module,
and a predictor module. Our MRT model is hot-pluggable, and can be seamlessly
incorporated into any stronger proposal-free or proposal-based video sentence
grounding network. Besides, we utilize a vision-and-linguistic Transformer to
encode the multi-modal features with attention. Moreover, we propose a novel
loss function that fuses Cross Entropy (CE) loss, Structural Similarity Index
Measure (SSIM) loss, and Intersection over Union (IoU) loss to supervise the
training process of accurate video sentence grounding in three levels: frame-
level, clip-level, and sequence-level. We demonstrate the effectiveness of the
proposed MRTNet on multiple benchmarks. From the experimental results and
ablation study, we observe consistent improvements across extensive ablations
and settings.

In summary, our proposed MRTNet has the following advantages:

  * •

We propose a novel multi-resolution temporal video sentence grounding network:
MRTNet, which mainly consists of a multi-model feature encoder, a multi-
resolution temporal (MRT) module, and a predictor module. We use a multi-modal
transformer to encode the visual and linguistic features. Our MRT model is
hot-pluggable, and can be seamlessly incorporated into any stronger proposal-
free or proposal-based video sentence grounding network.

  * •

We propose a loss function that fuses CE loss, SSIM loss, and IoU loss to
supervise the training process of accurate video sentence grounding in three
levels: frame-level, clip-level, and sequence-level.

  * •

Experimental results show that our MRT module can effectively improve the
performance of different baseline methods in terms of popular evaluation
metrics.

Figure 2. The overview architecture of the MRTNet model, which consists of
three parts, including Multi-modal Feature Encoder Module, Multi-resolution
Temporal Module, and Predictor Module. Taking the untrimmed video and language
query as input, the Feature Encoder Module first extracts the video and query
feature by Video Encoder and Query Encoder separately. Then cross-modal
features are fused by video-query attention mechanism. Taking the cross-modal
features as input, multi-resolution temporal module use an encoder-decoder
architecture with downsampling and upsampling, and regularizes the context
temporal information from different resolutions. Finally, the temporal
boundary results (start and end timestamps) are predicted according to the
temporal context-aware features.

##  2\. Related Work

###  2.1. Video Sentence Grounding

Video sentence grounding is defined as retrieving video segments corresponding
to language queries (Anne Hendricks et al., 2017; Xiao et al., 2021; Liu et
al., 2020, 2021). Most proposed methods (Hendricks et al., 2018; Liu et al.,
2018a) treat this task as a ranking problem. They first generate a certain
amount of proposals of video segments, and then match the visual content of
proposals and linguistic information of language queries. After ranking
segment proposals according to the multi-modal matching, the best matching
moment for the query can be selected. Similar to object detection, which also
generates proposals first and then classifies the content in the bounding box,
mining appropriate numbers of positive and negative samples are important for
model performance. Thus, many methods always densely sample video moment
proposals to achieve good performance, which leads to large computation costs.

To overcome the above-mentioned drawbacks, (Yuan et al., 2019b) propose a
proposal-free method, which utilizes a Bi-LSTM to encode visual and sentence
features, and a co-attention interaction module to fuse multi-modal features.
The model treats the video as a whole and directly predicts the temporal
coordinates according to the sentence queries. (Lu et al., 2019) propose a
dense bottom-up framework, which treats all frames corresponding to the
language query as foreground, and then regresses the unique distances of each
frame in the foreground to bi-directional ground-truth boundaries, finally
fuses appropriate temporal candidates as final result. To mine the
relationship of sentence semantics with diverse video contents, Yuan et al.
(Yuan et al., 2019a) also propose a semantic conditioned dynamic modulation,
which leverages sentence semantic information to modulate the temporal
convolution processes in a hierarchical temporal convolutional network, and
establishes a precise matching relationship between sentence and video.
Recently, (Zhang et al., 2020a) propose a 2D temporal map to model the
temporal relations of different moments with variant length, in which the two
dimensions indicate the start and end timestamps, respectively. Apart from
these top-down and bottom-up methods, there are also works (Wang et al., 2019)
(He et al., 2019) which adopt reinforcement learning to make decisions on the
action space of candidate segments, such as the start or end boundaries move
left or right, corresponding to the language query matching result.

Among previous works, most similar to ours is DRN (Zeng et al., 2020). They
propose a dense regression network with three different heads to generate
accurate boxes. However, directly training three head branches is difficult
and unstable, the authors propose a three-step strategy to train the proposed
DRN (Zeng et al., 2020). Different from generating candidate boxes in
different scales, we propose a multi-resolution temporal network to further
leverage the benefit of 1D temporal attention to better model the context
information in different resolutions, which improves a lot for the accurate
temporal region in videos. Besides, to obtain accurate attention information,
we utilize a loss function to regularize the attention map from three
different levels respectively. Furthermore, our proposed module can be plugged
into other anchor-free baselines and can be easily trained in an end-to-end
manner.

###  2.2. Multi-modal Transformer

Transformer (Vaswani et al., 2017) is first proposed in natural language
processing community, which consists of the encoder and decoder with self-
attention to deal with the neural machine translation problem. Then,
transformer-based networks are proposed to deal with various NLP tasks, such
as language modeling (Dai et al., 2018), word sense disambiguation (Tang et
al., 2018), and learning sentence representations (Devlin et al., 2018). Due
to the great success of Transformer in NLP tasks, its applications to vision
tasks have received increasing attention (Dosovitskiy et al., 2020). By
splitting the image into fixed-size patches, an image can be treated as a
sequence of vectors, which is the same as a sentence. Visual Transformer can
be applied in a serious of CV tasks, such as image classification (Wang et
al., 2021b), semantic segmentation (Wang et al., 2021a), _etc_.

As for the multimedia area, several works have tried to utilize Transformer
architecture to deal with the multi-modal feature interaction. For example,
(Gabeur et al., 2020) propose a multi-modal transformer to jointly encode the
different modalities in video, so as to retrieve the corresponding video
according to language and audio. (Yu et al., 2019) propose a Multi-modal
Transformer model for image captioning, which simultaneously captures intra-
modal and inter-modal interactions in a unified attention block. In this
paper, we propose to use Transformers to encode the multi-modal features and
predict the start and end timestamps.

##  3\. Approach

In this paper, we propose an MRTNet model to deal with the video sentence
grounding task. To sum up, MRTNet mainly consists of three parts: multi-model
feature encoder module, multi-resolution temporal module, and a predictor
module. The whole architecture of MRTNet is illustrated in Figure 2. In this
section, we first define the problem formulation and describe the multi-model
feature encoder module. Then, we present the details of the Multi-Resolution
Temporal (MRT) module. Finally, we calculate the start and end timestamps in
the predictor module and show the training and inference processes of MRTNet
in detail.

###  3.1. Problem Formulation

Given an untrimmed video $V={f_{t}}_{t=1}^{T}$ and the language query
$Q={q_{j}}_{j=1}^{M}$, where $T$ and $M$ are the numbers of frames and words,
respectively, our goal is to predict the start and end timestamp ($\tau^{s}$,
$\tau^{e}$) in the video corresponding to query $Q$. For each video $V$, we
extract its visual features $\bm{V}={\bm{v}_{i}}_{i=1}^{n}$ with a pretrained
3D ConvNet, where $n$ is the length of extracted features, each feature
$\bm{v}_{i}$ here is a video feature vector. For each query $Q$, we initialize
the word features ${\bm{Q}=\bm{q}_{j}}_{j=1}^{m}$ by using GloVe embeddings.

###  3.2. Multi-modal Feature Encoder Module

For the visual feature $\bm{V}\in\mathbb{R}^{n\times d_{v}}$ and linguistic
feature $\bm{Q}\in\mathbb{R}^{m\times d_{q}}$, We first project them into the
same dimension $d$ by two linear layers. Then we adopt the same embedding
encoder layer as QANet (Yu et al., 2018a) to build the visual and linguistic
feature encoder, which consists of four convolution layers, a multi-head
attention layer (Vaswani et al., 2017), a layer normalization layer, and a
feed-forward layer with the residual connection. The encoded visual features
and query features are as follows:

(1) $\displaystyle\bm{\widetilde{V}}$
$\displaystyle=\mathtt{VisualEncoder}(\bm{VW^{v}}),$
$\displaystyle\bm{\widetilde{Q}}$
$\displaystyle=\mathtt{QueryEncoder}(\bm{QW^{q}})$

where $\bm{W^{v}}$ and $\bm{W^{q}}$ are projection matrices to keep the
dimension consistency between two modalities. The parameters of feature
encoder are shared by two modal features.

Multi-modal Attention. With the encoded visual and query features, we
calculate the similarity of two modal features and fuse the multi-modal
features by a multi-modal attention mechanism. Similar to (Lu et al., 2019),
we first calculate the similarity scores, $\mathcal{S}\in\mathbb{R}^{n\times
m}$, between each visual feature and query feature. Then the attention weights
of visual-to-query ($\mathcal{A}$) and query-to-visual ($\mathcal{B}$) are
computed as:

(2)
$\displaystyle\mathcal{A}=\mathcal{S}_{r}\cdot\bm{\widetilde{Q}}\in\mathbb{R}^{n\times
d},$
$\displaystyle\mathcal{B}=\mathcal{S}_{r}\cdot\mathcal{S}_{c}^{T}\cdot\bm{\widetilde{V}}\in\mathbb{R}^{n\times
d},$

where $\mathcal{S}_{r}$ and $\mathcal{S}_{c}$ are the row-wise and column-wise
normalization of $\mathcal{S}$ by softmax operation, respectively. Finally,
the output of visual-query attention is written as:

(3)
$\bm{V}^{q}=\mathtt{FFN}\big{(}[\bm{\widetilde{V}};\mathcal{A};\bm{\widetilde{V}}\odot\mathcal{A};\bm{\widetilde{V}}\odot\mathcal{B}]\big{)},$

where $\bm{V}^{q}\in\mathbb{R}^{n\times d}$; $\mathtt{FFN}$ is a single feed-
forward layer; $\odot$ denotes element-wise multiplication. $\bm{V}^{q}$ is
the fused multi-modal semantic features with visual and query attention.

###  3.3. Multi-Resolution Temporal Module

The multi-modal features $\bm{V}^{q}$ we obtain above are in the same length
as the video feature $\bm{V}$. However, the temporal consistency of video
sequence in different resolutions lacks of exploration. Inspired by the U-Net
(Ronneberger et al., 2015), we design a multi-resolution temporal module in an
encoder-decoder fashion. This encoder-decoder design is able to capture
contexts globally in low resolution and refine accurate temporal boundary
information in high resolution at the same time. Both encoder and decoder have
three different scales of temporal features.

Specifically, the encoder has two stages with the output from high resolution
to low resolution, each stage is comprised of two convolution layers and a
non-overlapping max-pooling layer.

To further capture the global context in the lowest resolution, we propose to
utilize a convolutional layer to bridge the feature from the encoder and
decoder parts. For each convolutional layers mentioned above, we add a batch
normalization (Ioffe and Szegedy, 2015) and a ReLU activation function, _i.e._
,

(4)
$\displaystyle\bm{M}^{e}_{2}=\mathtt{Pooling}(\mathtt{Conv}(\bm{M}^{e}_{1})),$
$\displaystyle\bm{M}_{b}=\mathtt{Pooling}(\mathtt{Conv}(\bm{M}^{e}_{2})).$

where $\bm{M}^{e}_{1}$, $\bm{M}^{e}_{2}$ represents the multi-modal features
in encoder part, here $\bm{M}^{e}_{1}$ is the same value as $\bm{V}^{q}$. Here
we use the depth-wise convolution for efficient network structure. More choice
about the encoder layer is discussed in Section 7.

Then, the concentrated video feature in the lowest resolution (_i.e._ ,
$\bm{M}_{b}$) are concatenated with language feature $\bm{\widetilde{Q}}$
again. To make the length of language and visual feature consistent, we use an
weighted pooling on the language feature $\bm{\widetilde{Q}}$, _i.e._ ,

(5)
$\begin{split}\bm{M}_{b}&=\mathtt{FFN}([\mathtt{Pooling}(\bm{W}_{m}\bm{M}_{b}),\mathtt{Pooling}(\bm{W}_{q}\bm{\bm{\widetilde{Q}}})]),\end{split}$

For the decoder, we try to refine the accurate boundaries of the responding
region while recovering to high resolution. To be specific, the structure of
the decoder part is symmetrical to the encoder part according to layer design.
There are also two corresponding stages with the output from low resolution to
high resolution in the decoder design. Each stage consists of two convolution
layers, and is followed by a batch norm layer and a ReLU function, _i.e._ ,

(6)
$\displaystyle\bm{M}^{d}_{2}=\mathtt{Conv}([\mathtt{UpPooling}(\bm{M}_{b}),\bm{M}^{e}_{2}]),$
$\displaystyle\bm{M}^{d}_{1}=\mathtt{Conv}([\mathtt{UpPooling}(\bm{M}^{d}_{2}),\bm{M}^{e}_{1}]).$

As shown in Figure 2, the input of each stage in the decoder part is the
combination of output feature maps from the previous stage in the decoder part
and those of the corresponding stage in the encoder part. In total, there are
features in three different resolutions among the multi-resolution temporal
modules. To fully supervise the generated temporal features in different
resolutions, the output of each decoder block is supervised by the ground-
truth temporal labels in their corresponding resolutions. To be specific, the
ground-truth temporal labels can be transferred as 1D temporal maps $G$: The
length of the whole video can be divided as $n$ clips (_e.g._ , 128 clips),
and all clips lie in the corresponding region of the query are represented as
1, the other clips are 0.

Based on the structure mentioned above, we add a $3\times 3$ convolution layer
for each output feature map in the bridge stage and each decoder stage,
followed by the upsampling and sigmoid functions. Therefore, for input
temporal features, our model produces three temporal attention maps of
different resolutions in the training process. To obtain the 1D temporal maps
of result, we use an 1D convolution layer with kernel of $3\times 3$ to output
the features with one channel. And we take $\bm{M}^{d}_{2}$ as the final input
fed into the prediction module.

###  3.4. Predictor Module

We follow (Zhang et al., 2020b) and construct a predictor by using two
unidirectional LSTMs to predict the start and end logits according to feature
$\bm{M}^{d}_{1}$. Meanwhile, two LSTMs are stacked so that prediction of end
boundaries can be conditioned on those of start boundaries:

(7) $\displaystyle\mathbf{S}_{t}^{s}$
$\displaystyle=\mathbf{W}_{s}\times([\mathtt{UniLSTM}_{\textrm{start}}(\bm{M}_{1}^{d},\bm{h}_{t-1}^{s});\bm{M}_{1}^{d}])+\mathbf{b}_{s},$
$\displaystyle\mathbf{S}_{t}^{e}$
$\displaystyle=\mathbf{W}_{e}\times([\mathtt{UniLSTM}_{\textrm{end}}(\bm{S}_{t}^{s},\bm{h}_{t-1}^{e});\bm{M}_{1}^{d}])+\mathbf{b}_{e}.$

Here, $\bm{S}_{t}^{s}$ and $\bm{S}_{t}^{e}$ denote the scores of start and end
boundaries at position $t$; $\bm{v}_{t}^{q}$ represents the $t$-th feature in
$\bm{V}^{q}$.

Table 1. Statistics of Charades-STA, ActivityNet Caption, and TACoS datasets. Dataset | Charades-STA | ActivityNet Captions | TACoS
---|---|---|---
Source | Homes | YouTube | Lab Kitchen
Domain | Indoor Activity | Open | Kitchen Cooking
#Video | 6,672 | 14,926 | 1,27
#Annotation | 16,128 | 71,957 | 18,818
#Moments | 11,767 | 71,957 | 3,290
Vocabulary Size | 1,303 | 15,505 | 1,344
Average Video Length (seconds) | 30.60 | 117.60 | 286.59
Average Moment Length (seconds) | 8.09 | 37.14 | 6.10
Average Query Length (words) | 7.22 | 14.41 | 10.05
Annotation Split (Training/ Validation/ Testing) | 12,408 / - / 3,720 | 37,421/ 17,505 / 17,031 | 10,146 / 4,589 / 4,083

Then, the probability distributions of start and end boundaries
($\bm{P}_{s}\in\mathbb{R}^{n}$, $\bm{P}_{e}\in\mathbb{R}^{n}$) are calculated
by:

(8) $\displaystyle\bm{P}_{s}$ $\displaystyle=\mathtt{Softmax}(\bm{S}^{s}),$
$\displaystyle\bm{P}_{e}$ $\displaystyle=\mathtt{Softmax}(\bm{S}^{e}),$

Inference. Suppose the video duration is $\mathcal{T}$, the start (end) index
is calculated by $a^{s(e)}=\langle\tau^{s(e)}/\mathcal{T}\times n\rangle$,
where $\langle\cdot\rangle$ denotes the rounding operator. During the
inference, the predicted boundary can be easily converted to the corresponding
time via $\hat{\tau}^{s(e)}=\hat{a}^{s(e)}/n\times\mathcal{T}$, and the
predicted moment locations $(\hat{a}^{s},\hat{a}^{e})$ of a query is generated
by maximizing the joint probability of start and end boundaries by:

(9) $\displaystyle\mathtt{(}\hat{a}^{s},\hat{a}^{e})$
$\displaystyle=\arg\max_{\hat{a}^{s},\hat{a}^{e}}P_{s}(\hat{a}^{s})P_{e}(\hat{a}^{e})$
$\displaystyle\textrm{s.t. }0\leq\hat{a}^{s}\leq\hat{a}^{e}\leq n$

###  3.5. Loss function

To fully supervise the generated temporal features $M^{d}_{2}$, $M^{d}_{1}$,
and $M_{b}$ in different resolutions, our training loss of multi-resolution
temporal module is calculated by considering all outputs, _i.e._ ,

(10) $L={\sum}_{k=1}^{K}{\alpha_{k}l^{(k)}},$

where $l^{(k)}$ represents the loss of the $k$-th 1D temporal features, and
$\alpha_{k}$ is the weight parameters. In our experiments, MRTNet is deeply
supervised with outputs in three resolutions, _i.e._ , $K=3$.

In each $l^{(k)}$, we define $l^{(k)}$ as a loss function to obtain high
quality temporal location and clear boundaries:

(11) $\centering
L^{(k)}=L_{ce}^{(k)}+L_{ssim}^{(k)}+L_{iou}^{(k)},\@add@centering$

where $l_{ce}^{(k)}$, $l_{ssim}^{(k)}$, and $l_{iou}^{(k)}$ denote CE loss,
SSIM loss, and IoU loss, respectively.

CE loss is widely used in previous video sentence grounding models:

(12)
${L_{ce}=\frac{1}{2}\big{[}L_{\textrm{ce}}(P_{s},Y_{s})+L_{\textrm{ce}}(P_{e},Y_{e})\big{]}},$

where $L_{\textrm{CE}}$ represents cross-entropy loss function; $Y_{s}$ and
$Y_{e}$ are the labels for the start ($a^{s}$) and end ($a^{e}$) boundaries,
respectively. $P_{s}$ and $P_{e}$ are generated according to Eq. 8.

SSIM (Structural Similarity Index Measure) loss (Wang et al., 2003), is first
proposed in image quality assessment task. Since there exists structural
information in an image, SSIM can effectively measure the similarity of
images, which has been proven in relevant tasks (Qin et al., 2019). In the
video moment grounding task, 1D temporal structural information is also
important, which can be utilized as a training loss compared with ground
truth.

For two clips from prediction $S$ and the binary 1D temporal groundtruth $G$,
we calculate the SSIM loss by elements in each clip. We denote
$x=\\{x_{j}|j=1,...,N\\}$ and $y=\\{y_{j}|j=1,...,N\\}$, $x$ is cropped from
$S$ and $y$ is cropped from $G$. respectively. Then, the SSIM loss of $x$ and
$y$ can be calculated as introduced below:

(13)
$\centering{L_{ssim}=1-\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}}\@add@centering$

Here $\mu_{x}$, $\mu_{y}$ represents the mean deviations of $x$ and $y$,
$\sigma_{x}$, $\sigma_{y}$ represents the standard deviations of $x$ and $y$.
$\sigma_{xy}$ represents covariance. $C_{1}$ and $C_{2}$ are parameters that
are introduced in implementation details.

IoU loss is widely used in segmentation tasks, and can also be borrowed into
the video sentence grounding task. The calculation is listed below:

(14) $\centering
L_{iou}=1-\tfrac{\sum\limits_{c=1}\limits^{W}S(c)G(c)}{\sum\limits_{c=1}\limits^{W}[S(c)+G(c)-S(c)G(c)]},\@add@centering$

Here $G(c)\in\\{0,1\\}$ denotes the groundtruth label of the element $c$ and
$S(c)$ is the prediction of 1D temporal map.

##  4\. Experiments

###  4.1. Datasets

To evaluate the performance of our proposed MRTNet, we conduct experiments on
three challenging video sentence grounding datasets:

  * •

Charades-STA (Gao et al., 2017) is composed of daily indoor activities videos,
which is based on Charades dataset (Sigurdsson et al., 2016). This dataset
contains 6672 videos, 16,128 annotations, and 11,767 moments. The average
length of each video is 30 seconds. $12,408$ and $3,720$ moment annotations
are labeled for training and testing, respectively;

  * •

ActivityNet Caption (Caba Heilbron et al., 2015) is originally constructed for
dense video captioning, which contains about $20$k YouTube videos with an
average length of 120 seconds. As a dual task of dense video captioning, video
sentence grounding utilize the the sentence description as query and output
the temporal boundary of each sentence description.

  * •

TACoS (Regneri et al., 2013) is collected from MPII Cooking dataset (Regneri
et al., 2013), which has 127 videos with an average length of $286.59$
seconds. TACoS has 18,818 query-moment pairs, which are all about cooking
scenes. We follow the same splits in (Gao et al., 2017), where $10,146$,
$4,589$ and $4,083$ annotations are used for training, validation and test,
respectively.

The details of each dataset are summarized in Table 1. It it worth noting that
some methods make minor changes to the dataset when evaluating the
experimental performance. For example, CMIN (Zhang et al., 2019b) uses val_1
as validation set and val_2 as testing set in the ActivityNet Captions
dataset, while other methods (Zhang et al., 2020a) combine the val_1 and val_2
together as the testing set. And for TACoS dataset, 2D-TAN (Zhang et al.,
2020a) utilize a modified TACoS dataset for evaluation. To make a fair
comparison, we follow the setting of dataset splitting report in their
original papers when evaluating the performance of our method.

###  4.2. Evaluation Metrics

Following existing video grounding works, we evaluate the performance on two
main metrics:

mIoU: “mIoU” is the average predicted Intersection over Union over all testing
samples. The mIoU metric is particularly challenging for short video moments;

Recall: We adopt “$\textrm{R@}n,\textrm{IoU}=\mu$” as the evaluation metrics,
following (Gao et al., 2017). The “$\textrm{R@}n,\textrm{IoU}=\mu$” represents
the percentage of language queries having at least one result whose IoU
between top-$n$ predictions with ground-truth is larger than $\mu$. In our
experiments, we reported the results of $n=1$ and $\mu\in\\{0.3,0.5,0.7\\}$.

###  4.3. Implementation Details

For language query $Q$, we use the $300$-D GloVe (Pennington et al., 2014)
vectors to initialize each lowercase word, and these word embeddings are fixed
during training. For video $V$, we downsample frames and extracted RGB visual
features using the 3D ConvNet which was pre-trained on the Kinetics dataset.
We set the dimension of all the hidden layers in the model as $128$, the
kernel size of the convolutional layer as $7$, and the head size of multi-head
attention as $8$. For all datasets, models were trained for $100$ epochs The
batch size was set to $16$. Dropout and an early stopping strategies were
adopted to prevent overfitting. The whole framework was trained by Adam
optimizer with an initial learning rate 0.0001. $C_{1}=0.01^{2}$ and
$C_{2}=0.03^{2}$.

Table 2. Performance comparison (%) with the state-of-the-art models on TACoS dataset. Noted that the improvement achieved by MRTNet is significant. Methods | R@1,IoU=0.3 | R@1,IoU=0.5 | R@1,IoU=0.7 | mIoU
---|---|---|---|---
CTRL ICCV2017 | 18.32 | 13.30 | – | –
ACRN SIGIR 2018 | 19.52 | 14.62 | – | –
MCF IJCAI2018 | 18.64 | – | – | –
SM-RL CVPR2019 | 20.25 | 15.95 | – | –
ACL WACV2019 | 22.07 | 17.78 | – | –
SAP AAAI2019 | – | 18.24 | – | –
L-NET AAAI2019 | – | – | – | 13.41
TGN EMNLP2018 | 21.77 | 18.90 | – | –
ABLR-aw AAAI2019 | 18.90 | 9.30 | – | 12.50
ABLR-af AAAI2019 | 19.60 | – | – | 13.40
DEBUG EMNLP2019 | 23.45 | 11.72 | – | 16.03
CMIN SIGIR2019 | 24.64 | 18.05 | – | –
GDP AAAI2020 | 24.14 | – | – | 16.18
DRN CVPR2020 | – | 23.17 | – | –
SeqPAN ACL2021 | 31.72 | 27.19 | 21.65 | 25.86
VSLNet ACL2020 | 29.61 | 24.27 | 20.03 | 24.11
MRTNet (VSLNet) | 32.35 | 25.84 | 21.31 | 26.14
2D-TAN AAAI2020 | 37.29 | 25.32 | 13.32 | 25.19
MRTNet (2D-TAN) | 37.81 | 26.01 | 14.95 | 26.29
Table 3. Performance comparison (%) with the state-of-the-art models on ActivityNet Captions dataset. Noted that the improvement achieved by MRTNet is significant. Methods | R@1,IoU=0.3 | R@1,IoU=0.5 | R@1,IoU=0.7 | mIoU
---|---|---|---|---
CMIN SIGIR2019 | 63.61 | 43.40 | 23.88 | –
TGN EMNLP2018 | 43.81 | 27.93 | – | –
QSPN AAAI2019 | 45.30 | 27.70 | 13.60 | –
RWM AAAI2019 | – | 36.90 | – | –
ABLR-af AAAI2019 | 53.65 | 34.91 | – | 35.72
ABLR-aw AAAI2019 | 55.67 | 36.79 | – | 36.99
DEBUG EMNLP2019 | 55.91 | 39.72 | – | 39.51
GDP AAAI2020 | 56.17 | 39.27 | – | 39.80
DRN CVPR2020 | 58.52 | 41.51 | 23.07 | 41.13
CI-MHA SIGIR2021 | 61.49 | 43.97 | 25.13 | –
SeqPAN ACL2021 | 61.65 | 45.50 | 29.37 | 45.11
VSLNet ACL2020 | 63.16 | 43.22 | 26.16 | 43.19
MRTNet (VSLNet) | 64.17 | 44.09 | 27.43 | 44.82
2D-TAN AAAI2020 | 59.45 | 44.51 | 27.38 | 43.29
MRTNet (2D-TAN) | 60.71 | 45.59 | 28.07 | 44.54
Table 4. Performance (%) comparison with the state-of-the-art models on Charades-STA dataset. Noted that the improvement achieved by MRTNet is significant. Methods | R@1,IoU=0.3 | R@1,IoU=0.5 | R@1,IoU=0.7 | mIoU
---|---|---|---|---
CTRL ICCV2017 | – | 23.63 | 8.89 | –
ROLE MM2018 | 25.26 | 12.12 | – | –
ACL WACV2019 | – | 30.48 | 12.20 | –
SAP AAAI2019 | – | 27.42 | 13.36 | –
RWM AAAI2019 | – | 36.70 | – | –
SM-RL CVPR2019 | – | 24.36 | 11.17 | –
QSPN AAAI2019 | 54.70 | 35.60 | 15.80 | –
DEBUG EMNLP2019 | 54.95 | 37.39 | 17.92 | 36.34
GDP AAAI2020 | 54.54 | 39.47 | 18.49 | –
DRN CVPR2020 | - | 42.90 | 23.68 | 41.28
ExCL ACL2019 | – | 44.10 | 22.40 | –
MAN CVPR2019 | – | 46.53 | 22.72 | –
CI-MHA SIGIR2021 | 69.87 | 54.68 | 35.27 | –
VSLNet ACL2020 | 70.46 | 54.19 | 35.22 | 50.02
MRTNet (VSLNet) | 70.88 | 56.19 | 36.37 | 50.74
2D-TAN AAAI2020 | 57.31 | 42.80 | 23.25 | 39.23
MRTNet (2D-TAN) | 59.23 | 44.27 | 25.88 | 40.59

###  4.4. Comparison with State-of-the-Arts

####  4.4.1. Experimental Settings.

We compare our proposed MRTNet with state-of-the-art video sentence grounding
methods on three public datasets. These methods can be grouped into three
categories according to the viewpoints of proposal-based, proposal-free, and
other approaches:

1) Proposal-based models: CTRL (Gao et al., 2017), ACRN (Liu et al., 2018a),
ROLE (Liu et al., 2018b), MCF (Wu and Han, 2018), ACL (Ge et al., 2019), SAP
(Chen and Jiang, 2019), QSPN (Xu et al., 2019), CMIN (Zhang et al., 2019b) TGN
(Chen et al., 2018), 2D-TAN (Zhang et al., 2020a) MAN (Zhang et al., 2019a);
2) Proposal-free models: L-Net (Chen et al., 2019), ABLR-aw (Yuan et al.,
2019b), ABLR-af (Yuan et al., 2019b), DEBUG (Lu et al., 2019), CI-MHA (Yu et
al., 2021) ExCL (Ghosh et al., 2019), GDP (Chen et al., 2020), VSLNet (Zhang
et al., 2020b), LGI (Mun et al., 2020), DRN (Zeng et al., 2020) SeqPAN (Zhang
et al., 2021); 3) Others: RWM (He et al., 2019), SM-RL (Wang et al., 2019).

Figure 3. Qualitative results of MRTNet (VSLNet) on Charades-STA dataset.

####  4.4.2. Quantitative Results.

The results on three benchmarks are reported from Table 2 to Table 4,
respectively. According to the experimental results, we can observe that our
MRTNet can effectively improve the performance of baseline networks over all
metrics and benchmarks. We choose VSLNet (Zhang et al., 2020b) and 2D-TAN
(Zhang et al., 2020a) as baseline networks, which are known as typical
proposal-free and proposal-based models with published source codes.

  * •

For the implementation based on each baseline method, our MRTNet (VSLNet)
shares the same architecture as VSLNet in Feature Encoder Module.
Specifically, we implement the VSLNet with C3D feature followed the settings
they reported in original paper. VSLNet extracts the frame-level feature using
the backbone of QANet (Yu et al., 2018b) and utilizes two LSTMs to classify
the start and end boundary. For the MRTNet (2D-TAN), we add the MRT Module
before restructuring the 2D temporal feature map. For other modules in each
baseline network, we follow the same settings according to original papers.

  * •

Table 4 summarizes the experimental results on Charades-STA dataset. We can
observe that MRTNet (VSLNet) works well in even stricter metrics, _e.g._ ,
MRTNet (VSLNet) achieves a significant 1.28% absolute improvement in<EMAIL_ADDRESS>and MRTNet (2D-TAN) achieves 1.63% absolute improvement in<EMAIL_ADDRESS>which
demonstrates the effectiveness of proposed model. It is mainly because that
MRTNet better utilizes the temporal consistency of video sequences in
different resolutions and utilizes a loss function to better regularize the
multi-modal features. The results on TACoS and ActivityNet Captions dataset
are summarized in Table 2 and Table 3, respectively. Note that videos in TACoS
have a longer average duration and the ground-truth video segments in
ActivityNet Captions have a longer average length. Although there exists bias
in the data distribution of datasets (Yang et al., 2021), MRTNet significantly
outperforms the other methods on both benchmarks with the C3D feature, which
demonstrates that MRTNet is highly adaptive to videos and segments with
diverse lengths, and can bring in consistent improvement based on different
baseline methods.

  * •

Our proposed MRT module also has good generalization ability, since the input
multi-modal features of the MRT module keep in the same dimension as output
features, then this module can be easily plugged into any other proposal-based
or proposal-free video grounding network, and the whole network with proposed
loss function can be trained in an end-to-end manner.

Figure 4. Performance comparison (%) of MRTNet (VSLNet) in different layer
numbers on TACoS dataset.

####  4.4.3. Qualitative Results.

As illustrated in Figure 3, the qualitative results of MRTNet (VSLNet) on
Charades-STA dataset are reported. According to the three examples, the
localized moments generated by MRTNet are very close to ground-truth. If we
train the model without SSIM loss, the generated temporal region will be more
expansive compared with the ground-truth region, which is shown as a longer
temporal interval with high confidence. Hence, the model trained with SSIM
loss can better capture the accurate boundaries of query-corresponding
temporal regions.

###  4.5. Ablative Studies

We conduct ablative experiments to analyze the effectiveness of loss function
in our approach. Also, we analyze the effectiveness of the MRT module in
MRTNet. For simplification, the ablation experiments introduced below are
conducted on the TACoS dataset with VSLNet as the baseline.

####  4.5.1. Multi-resolution Temporal Module.

From the results in Figure 4 below, we can observe that three layers in the
encoder-decoder structure is the most appropriate setting when dealing with
videos of different lengths, which can achieve a trade-off between speed and
accuracy.

Table 5. Performance comparison (%) of MRTNet (VSLNet) with different encoding layers on TACoS dataset. Encoding Layer | R@1, IoU=0.5 | R@1, IoU=0.7 | mIoU
---|---|---|---
Convolution | 25.84 | 21.31 | 26.14
Transformer | 25.91 | 21.14 | 26.21

Then, we consider replacing the convolution layer with vanilla Transformer in
the encoder part of multi-resolution module, as shown in Eq. (4). According to
Table 5, we can observe that the performance of using Transformer layer in the
encoder part is comparative with using convolution, but causes larger
computation cost. To make a trade-off between speed and accuracy, our MRTNet
use depth-wise convolution in the encoder layer.

Table 6. Performance comparison (%) of MRTNet (VSLNet) with different losses on TACoS dataset.  Loss | $\ell_{ce}$ | $\ell_{ssim}$ | $\ell_{iou}$ | R@1, IoU=0.7 | mIoU
---|---|---|---|---|---
MRT Module | ✓ |  |  | 20.01 | 24.94
✓ | ✓ |  | 20.30 | 25.26
✓ |  | ✓ | 20.24 | 25.29
✓ | ✓ | ✓ | 21.31 | 26.14

####  4.5.2. Loss function.

To evaluate the effectiveness of each component in our proposed loss function,
we conduct a series of experiments with different losses:

  * •

The CE loss is computed in the frame-level. It mainly evaluates the
possibility of two peaks in time dimension: start and end timestamps. As shown
in Table 6, training the MRTNet (VSLNet) model with merely CE loss achieves
24.94 in mIoU.

  * •

The SSIM loss is a clip-level measure. The results in Table 6 indicate that
the proposed SSIM loss greatly improves the performance, which brings about
0.7% performance gain on average in the metric of mIoU. The SSIM loss can
regularize the features at clip-level and refine the temporal boundaries.

  * •

The IoU loss is a sequence-level measure. As shown in Table 6, utilizing SSIM
loss can bring about 0.3% performance gain in the metric of mIoU.

According to the detailed analysis of each loss, it is clear that our loss
function is beneficial to better feature learning and can achieve superior
qualitative results.

####  4.5.3. Computation Complexity.

We evaluate the computation complexity of MRTNet compared with other SOTA
methods on TACoS dataset.

  * •

According to Table 2 and Table 7, MRTNet achieves better performance compared
with baseline methods in all experimental cases. Due to the utilization of
depth-wise convolution, the model size of whole network has only minor
increment (almost 6$\%$) compared with the baseline 2D-TAN method.

  * •

We also try to replace the convolution layer in encoder part of MRT Module
with vanilla Transformer layer, which can further improve the performance but
brings in more parameters and increase the running time.

  * •

The experimental results show that utilizing the characteristic of multi-
resolution in video can bring in steady improvement based on each baseline
network and still have potential for better performance. To be specific, we
can use light-weighted version convolution or Transformer layers to achieve
better performance while relieving the burden of computation cost. Also, there
also exists a series of variants designed for multi-scale feature interaction,
such as pyramid pooling, multi-scale attention, and so on.

Table 7. Computation Complexity Comparison with SOTA methods on TACoS dataset. Method | Running Time | Model Size | R@1, IoU=0.5
---|---|---|---
CTRL | 2.23s | 22M | 13.30
ACRN | 4.31s | 128M | 14.62
TGN | 0.92s | 166M | 18.90
DRN | 0.15s | 214M | 23.17
2D-TAN | 0.57s | 232M | 25.32
MRTNet (2D-TAN) | 0.68s | 249M | 26.01
+Transformer | 0.82s | 284M | 26.21

##  5\. Conclusion

In this work, we propose a novel multi-resolution temporal video sentence
grounding network: MRTNet, which mainly consists of a multi-model feature
encoder, a Multi-Resolution Temporal (MRT) module, and a predictor. Our MRT
module is plug-and-play, which means it can be seamlessly incorporated into
any video sentence grounding baseline network. Besides, we propose a loss
function to supervise the cross-modal features of accurate video sentence
grounding in three levels: frame-level, clip-level, and sequence-level.
Through extensive experiments on three benchmark datasets, we show that MRTNet
outperforms the baseline methods; and the proposed MRT module is easy to plug
into other video sentence grounding frameworks. In future work, we will
explore more efficient and light-weighted models to make quick video sentence
grounding inference.

## References

  * (1)
  * Anne Hendricks et al. (2017) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017.  Localizing moments in video with natural language. In _ICCV_. 5803–5812.
  * Caba Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  Activitynet: A large-scale video benchmark for human activity understanding. In _CVPR_. 961–970.
  * Chen et al. (2018) Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. 2018.  Temporally grounding natural sentence in video. In _EMNLP_. 162–171.
  * Chen et al. (2019) Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. 2019.  Localizing natural language in videos. In _AAAI_ , Vol. 33. 8175–8182.
  * Chen et al. (2020) Long Chen, Chujie Lu, Siliang Tang, Jun Xiao, Dong Zhang, Chilie Tan, and Xiaolin Li. 2020.  Rethinking the bottom-up framework for query-based video localization. In _AAAI_ , Vol. 34. 10551–10558.
  * Chen and Jiang (2019) Shaoxiang Chen and Yu-Gang Jiang. 2019.  Semantic proposal for activity localization in videos via sentence query. In _AAAI_ , Vol. 33. 8199–8206.
  * Chu et al. (2021) Yun-Wei Chu, Kuan-Yen Lin, Chao-Chun Hsu, and Lun-Wei Ku. 2021\.  End-to-end Recurrent Cross-Modality Attention for Video Dialogue.  _TASLP_ (2021).
  * Dai et al. (2018) Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2018.  Transformer-xl: Language modeling with longer-term dependency.  (2018).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Gabeur et al. (2020) Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020.  Multi-modal transformer for video retrieval. In _ECCV_ , Vol. 5. Springer.
  * Gao et al. (2017) Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017\.  Tall: Temporal activity localization via language query. In _ICCV_. 5267–5275.
  * Ge et al. (2019) Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. 2019\.  Mac: Mining activity concepts for language-based temporal localization. In _WACV_. 245–253.
  * Ghosh et al. (2019) Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann. 2019.  Excl: Extractive clip localization using natural language descriptions.  _arXiv preprint arXiv:1904.02755_ (2019).
  * He et al. (2019) Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen. 2019\.  Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos. In _AAAI_ , Vol. 33. 8393–8400.
  * Hendricks et al. (2018) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2018.  Localizing moments in video with temporal language.  _arXiv preprint arXiv:1809.01337_ (2018).
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _ICML_. 448–456.
  * Ji et al. (2021) Wei Ji, Yicong Li, Meng Wei, Xindi Shang, Junbin Xiao, Tongwei Ren, and Tat-Seng Chua. 2021.  VidVRD 2021: The Third Grand Challenge on Video Relation Detection. In _Proceedings of the 29th ACM International Conference on Multimedia_. 4779–4783.
  * Li et al. (2022) Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. 2022.  Invariant grounding for video question answering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2928–2937.
  * Liu et al. (2021) Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie. 2021\.  Context-aware biaffine localizing network for temporal sentence grounding. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11235–11244.
  * Liu et al. (2020) Daizong Liu, Xiaoye Qu, Xiao-Yang Liu, Jianfeng Dong, Pan Zhou, and Zichuan Xu. 2020\.  Jointly cross-and self-modal graph attention network for query-based moment localization. In _Proceedings of the 28th ACM International Conference on Multimedia_. 4070–4078.
  * Liu et al. (2018a) Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua. 2018a.  Attentive moment retrieval in videos. In _SIGIR_. 15–24.
  * Liu et al. (2018b) Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua. 2018b.  Cross-modal moment localization in videos. In _ACM Multimedia_. 843–851.
  * Lu et al. (2019) Chujie Lu, Long Chen, Chilie Tan, Xiaolin Li, and Jun Xiao. 2019.  DEBUG: A dense bottom-up grounding approach for natural language video localization. In _EMNLP_. 5147–5156.
  * Mun et al. (2020) Jonghwan Mun, Minsu Cho, and Bohyung Han. 2020.  Local-Global Video-Text Interactions for Temporal Grounding. In _CVPR_. 10810–10819.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _EMNLP_. 1532–1543.
  * Qin et al. (2019) Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand. 2019\.  Basnet: Boundary-aware salient object detection. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 7479–7489.
  * Regneri et al. (2013) Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. 2013.  Grounding action descriptions in videos.  _ACL_ 1 (2013), 25–36.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-net: Convolutional networks for biomedical image segmentation. In _MICCAI_. 234–241.
  * Shang et al. (2021) Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. 2021.  Video visual relation detection via iterative inference. In _Proceedings of the 29th ACM International Conference on Multimedia_. 3654–3663.
  * Shang et al. (2017) Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. 2017.  Video visual relation detection. In _ACM Multimedia_. 1300–1308.
  * Sigurdsson et al. (2016) Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016.  Hollywood in homes: Crowdsourcing data collection for activity understanding. In _ECCV_. 510–526.
  * Tang et al. (2018) Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018.  Why self-attention? a targeted evaluation of neural machine translation architectures.  _arXiv preprint arXiv:1808.08946_ (2018).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ (2017).
  * Wang et al. (2019) Weining Wang, Yan Huang, and Liang Wang. 2019.  Language-driven temporal activity localization: A semantic matching reinforcement learning model. In _CVPR_. 334–343.
  * Wang et al. (2021a) Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021a.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  _arXiv preprint arXiv:2102.12122_ (2021).
  * Wang et al. (2021b) Wenxiao Wang, Lu Yao, Long Chen, Deng Cai, Xiaofei He, and Wei Liu. 2021b.  CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention.  _arXiv preprint arXiv:2108.00154_ (2021).
  * Wang et al. (2003) Zhou Wang, Eero P Simoncelli, and Alan C Bovik. 2003\.  Multiscale structural similarity for image quality assessment. In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, Vol. 2. Ieee, 1398–1402.
  * Wu and Han (2018) Aming Wu and Yahong Han. 2018.  Multi-modal Circulant Fusion for Video-to-Language and Backward.. In _IJCAI_ , Vol. 3. 8.
  * Xiao et al. (2022) Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. 2022\.  Video as Conditional Graph Hierarchy for Multi-Granular Question Answering. AAAI.
  * Xiao et al. (2021) Shaoning Xiao, Long Chen, Songyang Zhang, Wei Ji, Jian Shao, Lu Ye, and Jun Xiao. 2021.  Boundary Proposal Network for Two-Stage Natural Language Video Localization. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 2986–2994.
  * Xu et al. (2019) Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. 2019\.  Multilevel language and vision integration for text-to-clip retrieval. In _AAAI_ , Vol. 33. 9062–9069.
  * Yang et al. (2021) Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. 2021.  Deconfounded Video Moment Retrieval with Causal Intervention.  _arXiv preprint arXiv:2106.01534_ (2021).
  * Yu et al. (2018a) Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018a.  Fast and accurate reading comprehension by combining self-attention and convolution. In _ICLR_.
  * Yu et al. (2018b) Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018b.  Qanet: Combining local convolution with global self-attention for reading comprehension.  _arXiv preprint arXiv:1804.09541_ (2018).
  * Yu et al. (2019) Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. 2019\.  Multimodal transformer with multi-view visual representation for image captioning.  _TCSVT_ 30, 12 (2019), 4467–4480.
  * Yu et al. (2021) Xinli Yu, Mohsen Malmir, Xin He, Jiangning Chen, Tong Wang, Yue Wu, Yue Liu, and Yang Liu. 2021.  Cross Interaction Network for Natural Language Guided Video Moment Retrieval. In _SIGIR_.
  * Yuan et al. (2019a) Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu. 2019a.  Semantic conditioned dynamic modulation for temporal sentence grounding in videos.  _arXiv preprint arXiv:1910.14303_ (2019).
  * Yuan et al. (2019b) Yitian Yuan, Tao Mei, and Wenwu Zhu. 2019b.  To find where you talk: Temporal sentence localization in video with attention based location regression. In _AAAI_ , Vol. 33. 9159–9166.
  * Zeng et al. (2020) Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan. 2020\.  Dense regression network for video grounding. In _CVPR_. 10287–10296.
  * Zhang et al. (2019a) Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. 2019a.  Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In _CVPR_. 1247–1257.
  * Zhang et al. (2021) Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, and Rick Siow Mong Goh. 2021.  Parallel Attention Network with Sequence Matching for Video Grounding.  _arXiv preprint arXiv:2105.08481_ (2021).
  * Zhang et al. (2020b) Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. 2020b.  Span-based localizing network for natural language video localization.  _ACL_ (2020).
  * Zhang et al. (2020a) Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020a.  Learning 2d temporal adjacent networks for moment localization with natural language. In _AAAI_ , Vol. 34. 12870–12877.
  * Zhang et al. (2019b) Zhu Zhang, Zhijie Lin, Zhou Zhao, and Zhenxin Xiao. 2019b.  Cross-modal interaction networks for query-based moment retrieval in videos. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 655–664.
  * Zhong et al. (2022) Yaoyao Zhong, Wei Ji, Junbin Xiao, Yicong Li, Weihong Deng, and Tat-Seng Chua. 2022\.  Video Question Answering: Datasets, Algorithms and Challenges.  _EMNLP_ (2022).