# Redundancy-aware Transformer for Video Question Answering

Yicong Li1, Xun Yang2∗, An Zhang1, Chun Feng2, Xiang Wang2, Tat-Seng Chua1
1National University of Singapore, 2University of Science and Technology of
China liyicong<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2023)

###### Abstract.

This paper identifies two kinds of redundancy in the current VideoQA paradigm.
Specifically, the current video encoders tend to holistically embed all video
clues at different granularities in a hierarchical manner, which inevitably
introduces neighboring-frame redundancy that can overwhelm detailed visual
clues at the object level. Subsequently, prevailing vision-language fusion
designs introduce the cross-modal redundancy by exhaustively fusing all visual
elements with question tokens without explicitly differentiating their
pairwise vision-language interactions, thus making a pernicious impact on the
answering.

To this end, we propose a novel transformer-based architecture, that aims to
model VideoQA in a redundancy-aware manner. To address the neighboring-frame
redundancy, we introduce a video encoder structure that emphasizes the object-
level change in neighboring frames, while adopting an out-of-neighboring
message-passing scheme that imposes attention only on distant frames. As for
the cross-modal redundancy, we equip our fusion module with a novel adaptive
sampling, which explicitly differentiates the vision-language interactions by
identifying a small subset of visual elements that exclusively support the
answer. Upon these advancements, we find this Redundancy-aware transformer
(RaFormer) can achieve state-of-the-art results on multiple VideoQA
benchmarks.

Video Question Answering, Video-Language

$*$ Xun Yang is the corresponding author

Xiang Wang is also affiliated with Institute of Artificial Intelligence,
Institute of Dataspace, Hefei Comprehensive National Science Center.

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29-November 3, 2023;
Ottawa, ON, Canada††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON,
Canada††price: 15.00††doi: 10.1145/3581783.3612577††isbn:
979-8-4007-0108-5/23/10††ccs: Information systems Question answering††ccs:
Information systems Multimedia and multimodal retrieval

##  1\. Introduction

(a)

(b)

(c)

Figure 1. (a) Conventional multi-hierarchy video encoder. (b) Our design
philosophy for video encoder. (c) An illustration of how a small proportion of
interactions responds to answering, 5 critical interactions are highlighted.

Ever since the inception of intelligent systems, persistent interest has been
paid to revealing how machine reflects the physical world. In this regard,
Video Question Answering (VideoQA), a task that answers the natural language
questions in the context of videos, has gained steady progress over the last
few years. This advancement stems, in part from the recent surge of pre-
trained foundation models (Xiao et al., 2022b), and in part from the
enthusiasm for designing task-specific architectures (Park et al., 2021; Li et
al., 2022c; Xiao et al., 2022a).

Scrutinizing the recent progress (Jiang and Han, 2020; Le et al., 2020; Park
et al., 2021; Li et al., 2022c; Xiao et al., 2022a) in VideoQA, we systematize
a dominant paradigm as a composition of three modules: 1) a video encoder that
embeds the cross-frame dynamic on top of the frame representations, 2) a
question encoder that contextualizes the question semantic to acquire text
embeddings, and 3) a across-modal fuser that encapsulates the video and
question embedding to the multi-modal representations, and then makes the
prediction on top of that. Clearly, a correct prediction requires a careful
processing of visual clues and coordinating the interactions of vision-
language elements to reflect the answer reasoning as a mutual agreement
between video and question.

Although existing works have extensively adopted this paradigm, we argue that
the potential of current practices is seriously undermined due to their
inability in handling the substantial redundancy of VideoQA. Specifically, the
redundancy comes from two aspects:

  * •

The prevailing video encoder (Xiao et al., 2022b; Dang et al., 2021; Le et
al., 2020), typically models the input video as a multi-level hierarchy where
visual features at different granularities are merged together in a bottom-up
manner (as shown in fig. 1(a)). However, videos are temporally redundant, the
neighboring frames are similar in general because they share identical
environment or background, despite the minor differences in foreground
objects. Therefore, recklessly merging all adjacent frames together is neither
effective nor necessary, which introduces the neighboring-frame redundancy
that will overwhelms the detail movement of foreground objects in adjacent
frames. Thus, as shown in fig. 1(b), we argue that an effective video modeling
approach should focus on the object-level changes in adjacent frames rather
than merging all visual elements in each frame together. Moreover, frame-level
merging should only be imposed on distant frames, where the gradual changes of
objects have accumulated into a qualitative change at the frame level.

  * •

The existing cross-modal fuser (Jang et al., 2017; Le et al., 2020; Dang et
al., 2021; Peng et al., 2021) performs exhaustive feature fusing, that is,
merging all information from the video and language inputs, without
differentiating the pairwise interactions between them. However, various
interactions contribute differently, and only a small proportion of critical
interactions are responsive to answering the question, leaving the rest bulk
as cross-modal redundancy. This redundancy lead to a highly sparse target
signal in VideoQA, making the answer reasoning extremely challenging. Taking
fig. 1(c) as an example, the interactions between the query words ”girl,
apple, and dog” and the ”feeding” scene in the last two clips contain the
causal information that best supports the answer, while interactions involving
other clips contain only environmental information and should be ruled out to
avoid derailing the reasoning process (Ross et al., 2017; Chen et al., 2020;
Li et al., 2022c).

To resolve these limitations, we propose a novel VideoQA framework, named
Redundancy-aware Transformer (RaFormer) that introduces two fundamental
advancements: 1) a novel video encoder that emphasizes the object-level change
within a temporal neighborhood by modelling object movement in adjacent
frames, while avoid neighboring-frame redundancy by imposing attention only on
distant frames, and 2) an adaptive sampling module that enables the cross-
modal fuser to discover the answer critical frames in an interaction-aware
manner. Specifically, our video encoder aims to model the detailed change of
objects in adjacent frames, while imposing out-of-neighborhood message passing
at frame-level to avoid redundancy. To achieve this, we first introduce a
window cross-attention that enhances a single frame representation using
objects within its temporal window. Then, on top of all enhanced frame
representations, we adopt leap attention (Zhang et al., 2022) to impose
connection only between two temporal distant frames, which avoid the
neighboring-frame redundancy in a dilate manner. Notably, unlike conventional
hierarchy-based methods (Xiao et al., 2022b; Dang et al., 2021; Le et al.,
2020) that merge visual elements in different granularities into a single
packed representation (see fig. 1(a)), our video encoder uses a series of
object enhanced frames to represent a video, which coordinates our fuser
design. In cross-modal fusing, we first adopt a transformer-style encoder to
exert all pairwise interactions. Then, based on encoder’s cross-attention map,
we design an adaptive sampling module to pinpoint a small set of critical
interactions, which enables their corresponding frame being collected in an
adaptive manner. Upon these collected critical frames, we can naturally infer
the answer. Intuitively, such a frame down-sampling scheme acts as an
information bottleneck (Tishby and Zaslavsky, 2015) that condenses the
critical information for answering and makes it easier for a VideoQA model to
capture the learning pattern, thus leading to a significant improvement over
the original input.

Our contributions are summarized as follows:

  * •

We address the long-ignored redundancy issue in VideoQA, which is elicited by
the neighboring-frame redundancy in current practice of video encoder and the
cross-modal redundancy of fuser.

  * •

We propose RaFormer, a fully transformer-based VideoQA model that avoids
neighboring-frame redundancy by highlighting object-level change in adjacent
frames and the out-of-neighborhood message passing at frame-level. In
addition, it also handles the cross-modal redundancy via a novel adaptive
sampling module.

  * •

We perform extensive experiments on four benchmark datasets to demonstrate the
effectiveness of RaFormer. The results show significant improvements over
previous arts. (The absolute improvements w.r.t. accuracy are: NExT-QA (Xiao
et al., 2021) +3.5%, CausalVid-QA(Li et al., 2022a) +3.9%, MSVD-QA (Xu et al.,
2017) +2.3%, MSRVTT(Xu et al., 2017) +2.6%.)

##  2\. Related works

Video Question Answering (VideoQA). Video Question Answering (VideoQA) is a
fundamental extension of ImageQA that incorporates a temporal component.
Typically, previous efforts either leverage frame-level features as video
representation or incorporated region features as an additional input, which
brings extra benefit for detail-oriented question. Regardless of the input
representation, current designs tend to merge all visual elements together as
the video-level representation, using some inductive biases. (Jiang and Han,
2020) and (Park et al., 2021) pioneered graph-based structures for video
modeling based on the heterogeneity of input modality, while (Xiao et al.,
2022b) enabled relation reasoning via dynamic object graphs. In recent years,
a prevailing line of research tend to extract a multi-level hierarchy from the
video sequence. (Le et al., 2020) built a bottom-up pathway by assembling
information from clip-level to frame-level, while subsequent works such as
(Dang et al., 2021) and (Xiao et al., 2022a) extended their approaches with an
additional step in the object region. Despite these advances, existing methods
overlook the redundant nature of VideoQA tasks. In contrast, we developed
RaFormer to emphasize the two types of redundancy and mitigate their negative
impact on answer reasoning.

Redundancy in Video. Video related tasks (Dong et al., 2022; Yang et al.,
2022; Tan et al., 2021; Yang et al., 2021, 2020; Dong et al., 2021; Li et al.,
2021b; Ji et al., 2021, 2023b, 2023a; Zhong, 2022; Shang et al., 2021) have
seen significant advancements in recent years, primarily driven by the
adoption of 3D CNNs (Figurnov et al., 2017) or transformer architecture
(Vaswani et al., 2017; Li et al., 2023). While these models have shown
promising results on different video tasks (including video understanding
(Shang et al., 2019; Xiao et al., 2020; Li et al., 2021a) and video-language
tasks), there is a growing interest in developing more efficient techniques to
maintain or even improve the performance, due to the redundancy nature of
video. In video understanding domain, previous works have explored various
approaches (Pan et al., 2021; Wang et al., [n. d.]; Zhang et al., 2022) to
reduce the computational cost and redundancy, such as using hybrid 2D-3D
architectures, group convolution, or selecting salient clips. For the video-
language tasks, (Buch et al., 2022) is the first work that emphasizes the
“single-frame redundancy ” issue in video. However, its a analytical work that
aims to provide suggestion for a dataset construction view. To the best od our
knowledge, none of existing work focus on how ruling out redundancy in video
can bring extra benefit to the VideoQA. Intuitively, a concise representaion
with limited redundancy makes model easier to capture the causal pattern, thus
outperforms the model trained with original inputs (Tishby and Zaslavsky,
2015).

Rationale Discovery. The pursuit of explainability has led to recent
developments in deep neural networks (DNN) that aim to reveal the intuitive
evidence behind their predictions, known as rationales. Rationale discovery is
a widely adopted practice in the natural language processing (NLP) community
(Ribeiro et al., 2016), as well as in the graph (Wu et al., 2022) and vision
fields (Zhang et al., 2019). Recently, this trend has also extended to the
multi-modal community, with ImageQA-based tasks that request additional
textual evidence (Park et al., 2018; Dua et al., 2021) and causalvid that
applies the concept to videoQA. Despite the progress, existing solutions often
require a rationale finder with significant computation overhead (Li et al.,
2022c) or need to be trained in a contrastive manner (Li et al., 2022b).
RaFormer, however, can identify rationales (_i.e.,_ the critical frames)
without requiring additional training loss.

##  3\. Preliminaries

Taking a holistic view of existing methods, we summarize the paradigm of
VideoQA and provide a formal definition in this section. Throughout the paper,
we use upper-cased (_e.g.,_ $A$) and lower-cased (_e.g.,_ $a$) letters to
denote a variable and its deterministic value, respectively.

Modeling. Given the video $V$ and the question $Q$, the VideoQA model $G$ aims
to exploit information in both visual and textual streams to yield the
predictive answer $\hat{A}$. In leading VideoQA methods, $G$ is a combination
of a video encoder $G_{V}$, a question encoder $G_{Q}$ and a fuser $G_{F}$. It
aims to encapsulate the visual content and linguistic semantics into multi-
modal knowledge and then generate the predictive answer $\hat{A}$. Typically,
an entropy-based risk function is adopted to abridge the predictive answer
$\hat{A}$ and the ground-truth answer ${A}$:

(1) $\displaystyle\min\mathcal{L}((G_{V}(V),G_{Q}(Q))\circ G_{F},\,A).$

Data representation. We take a video of $T$ clips and select the frame in the
middle of each clip to serve as a representation of the entire video. Each of
these frames is presented by extracting a frame feature $\mathbf{f}_{t}$
through a pretrained image recognition backbone, as well as $S$ object
features $\mathbf{o}_{t,s}$ using a pretrained object detector, where $t$ and
$s$ are the indices for frame and object, respectively. To represent the text,
we use a pretrained language model as question encoder, which takes in the
question as a sequence of $L$ tokens, and produces a textual representation
$\mathbf{q}_{l}$ for each token. During training, the visual backbones are
fixed while the language backbone is fine-tuned end-to-end, following the
approach described in (Xiao et al., 2022b). To create a common $d$-dimensional
space for the representations, we apply three linear mappings to
$\mathbf{f}t$, $\mathbf{o}_{t,s}$, and $\mathbf{q}_{l}$, respectively,
resulting in
$\mathbf{F}\\!=\\!\left\\{\mathbf{f}_{t}\right\\}_{t=1}^{T}\\!\in\\!\mathbf{R}^{T\times
d}$,
$\mathbf{O}\\!=\\!\left\\{\mathbf{o}_{t,s}\right\\}_{t=1,s=1}^{T,S}\\!\in\\!\mathbf{R}^{T\times
S\times d}$, and
$\mathbf{Q}\\!=\\!\left\\{\mathbf{q}_{l}\right\\}_{l=1}^{L}\\!\in\\!\mathbf{R}^{L\times
d}$, which represent the features of the frame, object, and question,
respectively.

##  4\. Methodology

Figure 2. Overview of RaFormer. We made the advancement on two modules: the
video encoder and the cross-modal fuser. The video encoder incorporates a
window-based cross-attention that aggregates objects within a temporal window
to enhance the representation for each frame. On top of that, it also adopts a
leap self-attention mechanism that avoid neighboring-frame redundancy by
imposing attention only to distant frames. In the Cross-modal Fuser, a cross-
attention takes the video and question tokens as input, yielding multi-modal
knowledge and a cross-modal attention map. Based on this attention map, an
adaptive sampling module identifies critical frame representations, which are
then fed into a answer decoder for prediction.

In the existing literature, the temporal redundancy in video-language tasks
has been analyzed by (Buch et al., 2022) from the dataset construction angle.
However, a comprehensive solution for VideoQA has been missing until now. To
fill this gap, we propose a novel VideoQA framework named Redundancy-aware
Transformer (RaFormer). As depicted in fig. 2, RaFormer consists of a novel
video encoder that incorporates a window cross-attention and a leap self-
attention to mitigate the neighboring-frame redundancy. Additionally, it
tackles the cross-modal redundancy by introducing an adaptive sampling module
to cross-modal fuser, which discover the critical frames from a vision-
language interaction perspective.

###  4.1. Video Encoder

To encode a video instance, our video encoder aims to generate contextualized
frame representations that focus on the detailed changes of objects within
neighboring frames while avoiding neighboring-frame redundancy. Specifically,
unlike previous encoder designs that resort to multi-level hierarchy (Xiao et
al., 2022a; Dang et al., 2021; Xiao et al., 2022b), which introduces
neighboring-frame redundancy that can overwhelm the object details, RaFormer
focuses on the detail movement of objects within neighboring-frame by adopting
a window cross-attention. It enhances the representation of center frames by
aggregating the objects in its temporal neighborhood. On top that, we leverage
a leap attention (Zhang et al., 2022) to avoid imposing attention on
neighboring frames, resulting a concise representation for input video.

####  4.1.1. Window Cross-Attention

Window Cross-Attention (WCA) aims to model the detailed movement of objects
within a temporal window $W$, which helps to enhance a frame representation
$\mathbf{f}_{t}\in\mathbb{R}^{d}$ by aggregating all objects within its
temporal window. Concretely, we first add spatio-temporal position encoding to
each object feature following (Xiao et al., 2022b), and then apply an intra-
frame self-attention to contextualize all objects within a single frame
$\mathbf{o}_{t}\\!\in\\!\mathbb{R}^{S\times d}$. Based on the output tokens
$\mathbf{o^{\prime}}_{t}$, we flatten the objects within a temporal window $W$
(denote as
$\mathbf{o}_{t}^{W}\\!=\\!\left\\{\mathbf{o}_{t,s}\right\\}_{t-(W+1)/2,s=1}^{t+(W-1)/2,S}\\!\in\\!\mathbb{R}^{W\times
S\times d}$) to the shape of $(WS)\times d$ and feed it as query to window
cross-attention transformer decoder, along with the corresponding frame
embedding $\mathbf{f}_{t}$ as key and value:

(2) $\displaystyle\mathbf{o^{\prime}}_{t}=\text{Self-
Attention}(\mathbf{o}_{t})+\mathbf{o}_{t},$ (3)
$\displaystyle\mathbf{f^{\prime}}_{t}=\text{Cross-
Attention}_{W}(\mathbf{o}_{t}^{W})+\mathbf{o}_{t}^{W},\quad\text{s.t.}\,|W|=w$

where $\text{Cross-Attention}_{W}$ denotes window cross-attention with window
size $W$. In practice, we apply window cross-attention in multi-head manner,
where each head is set to different window size. We empirically show that such
multi-scale temporal reception fields are more flexible to cater different
video content, and thus bringing in extra benefits.

####  4.1.2. Leap Attention

To avoid neighbor-frame redundancy and preserve object details obtained in
window cross-attention, we resort to leap attention (Zhang et al., 2022) that
computes self-attention on discrete temporal frame pairs. With this dilated
strategy, we could build connection only between the distant frames, instead
of recklessly merging all neighboring frame together, thus maintaining the
detailed visual clue at object-level. Here, we use $E$ to denotes the temporal
skipped step size. For example in fig. 2, when $E$=2, we get attention
connection on following frame pairs for a sequence of four frames: (0,2),
(1,3). Formally, by applying leap attention on the enhanced frame
representations $\mathbf{f^{\prime}}$, we acquire the final output of video
encoder $\mathbf{f^{\prime\prime}}\in\mathbb{R}^{T\times d}$ as:

(4) $\displaystyle\mathbf{f^{\prime\prime}}=\text{Leap-
Attention}_{E}(\mathbf{f^{\prime}})+\mathbf{f^{\prime}},\quad\text{s.t.}\,|E|=e,$

where $\text{Leap-Attention}_{E}$ refers leap attention with step size of $E$.
For simplicity, we omit the superscript and use
$\mathbf{f}\in\mathbb{R}^{T\times d}$ to denote the output of video encoder.

###  4.2. Cross-modal Fuser with Adaptive Sampling

Although we have alleviated neighboring-frame redundancy with the video
encoder design, cross-modal redundancy can still overwhelm the critical
information when fusing frame and question tokens. To address that, we
introduce Adaptive Sampling (AS), a parameter-free module that adaptively
down-samples frame tokens according to their interaction activeness with
question embedding. As shown in fig. 3, we first use the attention map between
frame and question tokens to indicate their interaction activeness. Then, we
flatten and normalize all $T\times L$ interactions to get a probability
distribution $p$ over all interactions. Next, we calculate cumulative
distribution function (CDF) based on $p$, and then select a subset of
interactions using inverse transform sampling. Finally, we softly down-sample
the frames by selecting tokens that corresponds to the sampled interactions,
which adaptively remove redundant frames with the minimal computational
overhead.

####  4.2.1. Interaction Scoring

To gather the critical frames, a naive solution is to generate an importance
vector in size of $1\times T$ by prepending a
$\left\langle\text{CLS}\right\rangle$ token to the question, which denotes the
contribution of each of frame tokens, and then we can collect the critical
frames by selecting from this importance vector. However, the collection under
such a scheme suffers from _uni-modal bias_ (Cadène et al., 2019). As a
result, tokens with similar background can exert a high attention score, and
will be more likely to be selected, while ignoring the the question-critical
ones. Instead, we identifies the critical frames via the cross-modal
interactions – a mutual agreement of vision and language modalities. As shown
in the right part of fig. 2, first apply a cross-attention using encoded
frames $\mathbf{f}$ as query and and question embedding $\mathbf{q}$ as key
and value, which yields the frame token $\mathbf{\bar{f}}$ as well as the pre-
normalized cross-attention map $\mathbf{z}\in\mathbb{R}^{T\times L}$ as
output:

(5) $\displaystyle\mathbf{\bar{f}},\mathbf{z}=\text{Cross-
Attention}(\mathbf{f},\mathbf{q})+\mathbf{f}.$

Based on the cross-attention map $\mathbf{z}$, we apply our adaptive sampling.

Figure 3. Illustration of Adaptive Sampling process.

####  4.2.2. Adaptive Frame Sampling

Adaptive Sampling (AS) aims to select $N$ interactions from
$\mathbf{z}\\!\in\\!\mathbb{R}^{T\times L}$ ($N\ll T\times L$), then we can
collect the corresponding 1-D tokens from the 2-D interactions view. Notably,
by keeping only one frame when selected repetitively, we end up with an
adaptive frame collection where only a small proportion of $\bar{f}$ are
selected as critical frames $\mathbf{f}^{c}$ (_i.e.,_
$\left|\mathbf{f}^{c}\right|<=N$).

Specifically, to select N interactions from $\mathbf{z}$, a naive approach is
to select top-N with highest score. However, this deterministic approach does
not perform well, because it discards all interactions with lower scores. Some
of these interactions, however, can be useful particularly at the earlier
stages when the features are less discriminative. Therefore, we resort to
inverse transform sampling to sample interactions in an differentiable manner.
As shown in fig. 3, we first flatten $\mathbf{z}$ into the shape of
${1\times(TL)}$, and then apply a softmax over all interactions to obtain the
significance scores that can be interpreted as probabilities
$\mathbf{p}\in\mathbb{R}^{1\times(TL)}$:

(6) $\displaystyle\mathbf{p}=\text{Softmax}(\text{Flatten}(\mathbf{z})).$

Then we calculate the cumulative distribution function (CDF) of $\mathbf{p}$:

(7)
$\displaystyle\text{DCF}_{k}=\sum_{1}^{k}\mathbf{p}_{k},\quad\text{s.t.}\,k=(t-1)\times
L+l,$

where $k\in\left[1,2,...,TL\right]$ is the flatten index of the $\mathbf{p}$.

Next, we obtain the sampling function by taking the inverse of the CDF as:

(8) $\displaystyle k=\text{DCF}^{-1}(n),\quad n\in\left[0,1\right].$

In other words, the significance score $\mathbf{p}$ is used to calculate the
mapping function between the indices of the original interactions and the
sampled interactions. To obtain $N$ samples, we use a fixed sampling strategy
by choosing $n=\left\\{\frac{1}{2N},\frac{3}{2N},...\frac{2N-1}{2N}\right\\}$.
Since $\text{DCF}^{-1}\in\mathbb{R}$, we consider the indices of the tokens
with the nearest significant scores as the sampling indices.

After acquiring the $N$ sampled interactions, we collect their corresponding
frame tokens as critical frames $\mathbf{f}_{c}$. It is worth noticing that
gathering 1-D tokens from the 2-D interaction view can incur repetition. Thus,
we only keep one frame if it is selected multiple times, which enables an
adaptive collection of critical frames.

Figure 4. Illustration of our decoding process in Multi-choice QA (left) and
Open-end QA (right).

###  4.3. Answer Prediction

To generate the predicted answer, we use a transformer-based answer decoder
that receives an answer query to extract the predictive information from the
multi-modal features, and select the the predicted answer $\hat{A}$ from the
answer candidates. According to task setting, the answer candidates can be
provided as $\left|A_{mc}\right|$ sentences that are tailored for each
instance in the case of multi-choice QA, or as global answer categories of
size $\left|A_{oe}\right|$ that are shared among all instances (_a.k.a._ ,
Open-ended QA). This lead to the two formations of answer query (see in fig.
4).

####  4.3.1. Multi-Choice QA

In the Multi-Choice setting (see fig. 4 left), the answer candidates are given
as multiple sentences tailored for each instance. To generate a holistic
representation for each candidate answer, we prepend a
$\left\langle\text{CLS}\right\rangle$ token and use the same language model in
the question encoder to process the each answer candidate in parallel. Then we
gather the processed $\left\langle\text{CLS}\right\rangle$ tokens from
$\left|A_{mc}\right|$ answer candidates representations and form the answer
query $\mathcal{Q}_{mc}\in\mathbb{R}^{\left|A_{mc}\right|\times d}$. During
decoding, we feed the answer query $\mathcal{Q}_{mc}$ as the query and the
concatenation of critical frames and question (_i.e.,_
$\left[\mathbf{f}_{c};\mathbf{q}\right]$) as key and value to the transformer
decoder, and the decoder outputs the embedding
$\mathbf{h}_{mc}\in\mathbb{R}^{\left|A_{mc}\right|\times d}$ as:

(9) $\displaystyle\mathbf{h}_{mc}=\text{Transformer-
Decoder}(\mathcal{Q}_{mc},\left[\mathbf{f}_{c};\mathbf{q}\right])$

where $\left[;\right]$ refers to the concatenation operation. Notably, since
the correctness of a answer candidate is invariant to its position, answer
query is free of position encoding. Finally, we apply a linear projection on
$\mathbf{h}_{mc}$ to obtain the predictive answer
$\mathbf{\hat{a}_{mc}}\in\mathbb{R}^{\left|A_{mc}\right|}$.

(10) $\displaystyle\mathbf{\hat{a}}_{mc}=\text{MLP}(\mathbf{h}_{mc})$

####  4.3.2. Open-ended QA

In the open-ended setting, it is common practice (Xiao et al., 2022a; Li et
al., 2022c) to treat each candidate answer in the global answer set as a
category, which makes the size of the answer set too large to be processed as
multi-choice. Therefore, we initialize a single learnable embedding, denoted
as $\mathcal{Q}_{oe}\in\mathbb{R}^{d}$, as the answer query for this setting.

Similar to eq. 9, we use the transformer decoder to process the answer query
$\mathcal{Q}{oe}$ along with the concatenation of critical frames and question
($[\mathbf{f}c;\mathbf{q}]$) as the key and value. This results in a decoded
representation $h_{oe}\in\mathbb{R}^{d}$. Finally, we project $h_{oe}$ to the
answer space $\mathbb{R}^{\left|A_{oe}\right|}$ to obtain the prediction
$\mathbf{\hat{a}}_{oe}$.

(11) $\displaystyle\mathbf{\hat{a}}_{oe}=\text{MLP}(h_{oe}).$

To align the predictive answer with the gold answer, we establish our
objective on a cross-entropy loss.

##  5\. Experiments

Table 1. Dataset statistics. MC: Multi-Choice OE: Open-Ended

Dataset | Challenge | #QA pair | V-Len | Q-Len | QA
---|---|---|---|---|---
NExT-QA | Causal & Temporal | 48K | 44s | 11.6 | MC
Causal-VidQA | Evidence & Commonsense | 161K | 9s | 9.5 | MC
MSVD-QA | Description | 50K | 10s | 6.6 | OE
MSRVTT-QA | Description | 244K | 15s | 7.4 | OE

Table 2. Accuracy (%) comparison on Causal-VidQA. D: Description, E:
Explanation, P: Prediction, C: Counterfactual. *: Reproduced result using
official implementation.

Methods | Venue | Causal-VidQA
---|---|---
Acc@D | Acc@E | Acc@P | Acc@C | Acc@All
Q $\rightarrow$ A |  Q $\rightarrow$ R |  Q $\rightarrow$ AR |  Q $\rightarrow$ A |  Q $\rightarrow$ R |  Q $\rightarrow$ AR
HCRN (Le et al., 2020) | CVPR20 | 56.4 | 61.6 | 51.7 | 51.3 | 32.6 | 51.6 | 53.4 | 32.7 | 48.1
HGA (Jiang and Han, 2020) | AAAI20 | 65.7 | 63.5 | 49.4 | 50.6 | 32.2 | 52.4 | 55.9 | 34.3 | 48.9
B2A (Park et al., 2021) | CVPR21 | 66.2 | 62.9 | 49.0 | 50.2 | 31.2 | 53.3 | 56.3 | 35.2 | 49.1
VGT* (Xiao et al., 2020) | ECCV22 | 70.8 | 70.3 | 55.2 | 56.9 | 38.4 | 61.0 | 59.3 | 42.0 | 55.4
RaFormer (Ours) | - | 71.8 | 73.8 | 58.5 | 58.5 | 41.2 | 67.1 | 64.1 | 48.9 | 58.9
Abs. Improve | - | +1.0 | +3.5 | +2.7 | +1.6 | +2.8 | +6.1 | +4.8 | +6.9 | +3.5

Table 3. Accuracy (%) comparison on NExT-QA, MSVD-QA, and MSRVTT-QA. VIB:
Visual Inductive Bias, G,H denotes graph- and hierarchy- based method,
respectively. ROI: whether use additional region feature as additional input.
Acc@C, T, D, denote questions type of Causal, Temporal, and Descriptive in
NExT-QA, respectively. The best and 2nd best results are highlighted.

Methods | venue | NExT-QA | MSVD | MSRVTT
---|---|---|---|---
Acc@C | Acc@T | Acc@D | Acc@All
HCRN (Le et al., 2020) | CVPR20) | 47.1 | 49.3 | 54.0 | 48.9 | 36.1 | 35.6
HGA (Jiang and Han, 2020) | AAAI20 | 48.1 | 49.1 | 57.8 | 50.0 | 34.7 | 35.5
HOSTR (Dang et al., 2021) | IJCAI21 | - | - | - | - | 39.4 | 35.9
PGAT (Peng et al., 2021) | MM21 | - | - | - | - | 39.0 | 38.1
MHN (Peng et al., 2022) | IJCAI22 | - | - | - | - | 40.4 | 38.6
IGV (Li et al., 2022c) | CVPR22 | 48.6 | 51.7 | 59.6 | 51.3 | 40.8 | 38.3
HQGA (Xiao et al., 2022a) | AAAI22 | 49.0 | 52.3 | 59.4 | 51.8 | 41.2 | 38.6
EIGV (Li et al., 2022b) | MM22 | 51.2 | 51.5 | 61.0 | 52.9 | 43.7 | 39.3
VGT (Xiao et al., 2022b) | ECCV22 | 51.6 | 51.9 | 63.7 | 53.7 | - | 39.7
VGT-PT (Xiao et al., 2022b) | ECCV22 | 52.8 | 54.5 | 67.3 | 55.7 | - | -
RaFormer | - | 58.2 | 57.7 | 67.8 | 59.6 | 46.0 | 42.3
Abs.Improve | - | +5.4 | +3.2 | +0.5 | +3.9 | +2.3 | +2.6

In this section, we show the experimental results that answer the following
research questions:

  * •

RQ1: How effective is RaFormer compared with the State-of-the-Art (SoTA)
models?

  * •

RQ2: How do modules of Raformer and parameter setting affect the performance?

  * •

RQ3: What learning pattern does RaFormer capture?

Datasets: To evaluate the effectiveness of our proposed design, we conduct
experiments on four benchmark datasets that challenge RaFormer from different
aspects. Specifically, we use the prevalent Multi-Choice setting of NExT-QA
(Xiao et al., 2021) and Causal-VidQA (Li et al., 2022a) to test the model’s
temporal reasoning ability with complex causal and commonsense relations. As a
complement, we also use the Open-Ended setting of MSVD-QA (Xu et al., 2017)
and MSRVTT-QA (Xu et al., 2017), which mainly emphasize the description of
video objects, activities, and their attributes. For detailed statistics of
the datasets, please refer to Tab. 1.

Implementation Details: We followed the standard procedure established in
(Xiao et al., 2022b; Li et al., 2022c, b) for sampling videos, where each
video is represented as a sequence of $T=16$ frames, and each frame is encoded
using a ViT-L (Rao et al., 2021) model pre-trained on ImageNet-21k. To extract
object-level features, we used a Faster-RCNN (Ren et al., 2015) model pre-
trained on the Visual Genome, which detects $S=20$ objects on each frame. For
encoding the textual input (questions and answers), we used a pre-trained
Deberta-base model (He et al., 2021) as the question encoder. During training,
we optimized the model using an Adam optimizer with a learning rate of 1e-5,
and set the hidden dimension to 768. For for all datasets, we set skip step-
size $E$=4 in leap attention and sample $N=10$ interactions in adaptive
sampling.

###  5.1. Main Result (RQ1)

Tab. 2 and Tab. 3 show that RaFormer’s performance far exceeds existing SoTA
methods. Specifically, our observations are as follows:

  * •

QA settings. When comparing RaFormer’s performance across four datasets, we
observe that it provides a greater improvement to Multi-Choice QA than to
Open-Ended QA. Specifically, NExT-QA and Causal-VidQA saw an increase of 3.9%
and 3.5%, respectively, while MSVD-QA and MSRVTT only saw an increase of 2.6%
and 2.3%, respectively, compared to previously reported results. This
difference can be explained by two aspects: Firstly, Multi-Choice QA datasets
tend to have longer videos with more objects, which inevitably introduce more
neighboring-frame redundancy, compared to the Open-ended setting. Luckily, our
video encoder is better suited to handle neighboring-frame redundancy, thus
can bring more favorable improvement to Multi-Choice QA. Secondly, Multi-
Choice datasets involve complex reasoning scenarios with composite question
sentences and long videos, leading to more cross-modal redundancy. In
contrast, Open-Ended datasets typically feature simpler questions and shorter
videos. Therefore, our adaptive sampling strategy is particularly well-suited
for this scenario, which achieves a larger gain in Multi-Choice QA by wiping
out more environmental frames.

  * •

Question type. Narrowing down the analysis to Multi-Choice datasets (NExT-QA
and CausalVid-QA), it is apparent that the performance gain is primarily due
to improvement in the composite question type (including: Acc@C&Acc@T in NExT-
QA, Acc@E& Acc@P& Acc@C in Causal-VidQA), which involves deeper understanding
such as causal relation and counterfactual thinking. In contrast, the
improvement in the descriptive question type is much more moderate (Acc@D:
+0.5% on NExT-QA, and +1.0% on CausalVid). This highlights RaFormer’s
exceptional reasoning ability for complex questions. In particular, there is a
significant improvement in reason-based questions on Causal-VidQA (Q$\to$AR:
Acc@P +2.8%, Acc@C +6.9%), where the model has to justify its prediction by
selecting the correct evidence. This naturally aligns with the mechanism of
adaptive sampling, making RaFormer an ideal solution for this question type.

  * •

Benchmark size. Regarding the performance improvement on Open-Ended datasets,
it is interesting to note that the gain on MSRVTT-QA (+2.6%) is larger than
that on MSVD-QA (+2.3%), despite sharing the exactly same question type. We
suggest that the difference in dataset volume may have played a role in this
observation, since MSVD-QA is five times smaller than MSRVTT-QA (_cf._ Tab.
1). This could limit the reasoning ability of Raformer on MSVD-QA due to a
smaller amount of training data. This highlights the potential of Raformer
when trained on larger and more diverse datasets.

###  5.2. In-Depth Study (RQ2)

####  5.2.1. Ablative Results

We scrutinize some key designs in RaFormer by performing model ablation and
discussing other implementation alternatives. The observations of Tab. 4 are
as follows:

Table 4. Ablative study on NExT-QA and MSRVTT-QA.

Variants | NExT-QA | MSRVTT
---|---|---
Acc@C | Acc@T | Acc@D | Acc@All
w/o WCA | 57.7 | 57.3 | 67.9 | 59.2 | 41.1
w/o LA | 58.4 | 57.0 | 65.7 | 59.2 | 41.3
w/o WCA & LA | 57.1 | 56.3 | 65.6 | 58.2 | 40.8
w/o AS | 57.1 | 56.6 | 67.3 | 58.7 | 41.4
Hard TopN | 56.9 | 57.6 | 68.3 | 59.0 | 41.7
$\left\langle\text{CLS}\right\rangle$ Sampling | 57.0 | 56.2 | 65.0 | 58.1 | 41.3
w/o Enc & AS | 57.3 | 55.4 | 64.1 | 57.9 | 40.3
RaFormer | 58.2 | 57.7 | 67.8 | 59.6 | 42.3

  * •

In the top section, we study the design of our video encoder. In the 1st row,
we study the effectiveness of window cross-attention by replacing it with a
conventional object merging method (Xiao et al., 2022b, a) (_i.e.,_ apply mean
pooling to merge all objects in a single frame, then add the pooled
representation to the frame feature), where the performance drop of this
alternative (“w/o WCA”) demonstrates the necessity of capturing the object
transition details in adjacent frames. In 2nd row, we verify the efficacy of
Leap-Attention (LA) in reducing the neighboring-frame redundancy by replacing
it with a vanilla self-attention. As a result, we also witness a performance
drop on this variant (“w/o LA”), which demonstrates cutting off connections
between adjacent frames and restricts them to the distant frames can remove
the redundancy from temporal dimension. In the 3rd row, we ablate the whole
encoder design by removing both window attention and leap attention, the
denote this variant as “w/o WCA & LA”. We notice the performance of this
variant drops more significantly, which means the benefits of WCA and LA focus
on different aspect of video encoding, (WCA for modelling object detail in
adjacent frames, LA help to restrict neighboring-frame redundancy) and their
contribution can be added together.

  * •

In the bottom section, some mutants of the adaptive sampling are investigated.
First, we study the impact of our sampling strategy by removing the adaptive
sampling module (“w/o AS”), where all frames are used for answer prediction
without differentiating their interactions with question. As expected, the
accuracy drop demonstrates the negative impact of in redundant interactions in
answer reasoning as well as the necessity of critical frame down-sampling.
Next, we show that collecting critical interactions via hard ranking of
(_i.e.,_ Hard TopN) interaction scores results in sub-optimal performance. The
reason is that, in hard topN, interaction with a relatively smaller score will
not be explored, even if the scores itself are far from discriminative at the
early stage, thus leading to poor optimization. However, RaFormer circumvents
this issue by sampling from inversed cumulative distribution function, which
enable a better exploration of the interactions in a differentiable manner. As
discussed in Sec. 4.2.1, we also analyze the benefits of interaction-aware
sampling by comparing to an alternative that collects critical frames from a
probability vector generated $\left\langle\text{CLS}\right\rangle$ token. As
expected, this variant causes a sharp performance decline. We empirically find
that this is because of the _uni-modal bias_ issue mentioned in Sec. 4.2.1,
_i.e.,_ , the high correlated adjacent frames can induce a selection bias that
prompts visually similar tokens as critical frames, instead of the answer-
responsive ones.

  * •

Finally, we show the effectiveness of our overall design by ablating both
encoder (implemented as “w/o WCA & LA”) and adaptive sampling (implemented as
“w/o AS”). We show this variant, denoted as “w/o Enc & LA”, performs poorly
comparing to RaFormer, which testifies that our overall design is reasonable.

####  5.2.2. Study of Window Size

In order to investigate the effect of window cross-attention, the performance
of RaFormer is studied using different window sizes $W$. Specifically, the
default setting uses 4-head attention, where different heads can have the same
window size or varied ones. As shown in Tab. 5, when setting all heads’ window
size to 1, the model delivers the worst accuracy, which demonstrates the
necessity of modeling the objects in the neighboring frame. As the window size
increases, the model delivers better results on MSRVTT when the window size is
equal to 3, while NExT-QA reaches its peak at $W$=5. This discrepancy may be
caused by the complexity of the video source. Videos in NExT-QA feature
relational reasoning among multiple objects, where a larger window size can
incorporate more temporal related objects, thus benefiting the relational
reasoning. In comparison, MSRVTT typically questions the attributes of a
single object, making a smaller window size sufficient to capture the answer
information. Finally, the results in last row show that none of the single-
size window settings can beat the multi-scale encoding, where four attention
heads are equipped with different window sizes. This demonstrates that multi-
scale window settings are more flexible in catering to different video
content.

Figure 5. Case-Study on NExT-QA. Each video comes with two questions that
focus on different parts of the video. The blue and green windows indicate the
selected critical frames. In addition, we also mark the question tokens that
correspond to the sampled interactions.

####  5.2.3. Study of Sample Size N

To validate how the size of the sampled interaction $N$ affect RaFormer, we
conduct experiments with a variation of $N$ and show the results in fig. 7. In
both datasets, we observe constant performance gains as $N$ increases from 1,
and the peaks are reached at around 10. Then, as $N$ keeps growing, the
accuracies in both datasets drop since more redundancies are introduced. It is
worth noticing that, RaFormer can surpass all existing methods even with $N$
equal to 1 (_i.e.,_ only one frame is select and used to infer the answer),
which shed light on the redundancy nature of VideoQA task, and the deficiency
of currents methods as their ignorance of this inherent redundancy. Compare to
MSRVTT-QA, fluctuation on NExT-QA is more drastic when $N$ is small, where the
altering can cause a 5% difference in accuracy. This is because questions of
MSRVTT-QA generally focus on the description of a single entity, where the
crtical frames are more perceivable even with small $N$. In contrast, the
NExT-QA typically questions the temporal relation in a composite scenario,
where a small $N$ is unable to cover the golden frames that delivers the
answer information, thus leading to sub-optimal performance.

$W$ | NExT-QA | MSRVTT
---|---|---
1 | 58.7 | 41.7
3 | 59.2 | 42.0
5 | 59.4 | 41.8
7 | 58.9 | 41.9
$\left[1,3,5,7\right]$ | 59.6 | 42.3

Table 5. WCA’s window size W.

Figure 6. Hyper-parameter N in adaptive sampling.

Figure 7. The number of the critical frames and removed tokens after adaptive
sampling.

####  5.2.4. Study of Critical Frames

To grasp the learning insight of RaFormer, we analyzed the number of
identified critical frames in the original videos and present the statistics
in fig. 7. As expected, our adaptive sampling can identify a small group of
tokens as critical frames while leaving the rest as redundant. Specifically,
we observed that NExT-QA requires more frames to answer questions than MSRVTT-
QA. This is due to the relational reasoning nature of NExT-QA, which requires
clues from multiple entities, and are typically associated with more frames.
In contrast, MSRVTT-QA mainly focuses on the description of a single object,
which can be encoded with fewer frames. Interestingly, although RaFormer’s
peak performance on MSRVTT-QA and NExT-QA is achieved with almost the same
size of critical interaction group (both around $N$=10 _cf._ fig. 7), the
number of critical frames differs significantly between the two datasets. This
demonstrates that our frame selection strategy can adaptively cater to
different VideoQA instances.

###  5.3. Qualitative analysis (RQ3)

Intuitively, the design of adaptive sampling naturally empowers RaFormer with
visual explainability. Following this line, fig. 5 shows some predictive
results on NExT-QA along with the identified critical frames. To better
understand learning pattern of adaptive sampling, we also mark the question
tokens that corresponds to the sampled interactions, which can be interpreted
as ”critical words”. Here, each video instance comes with two questions that
inquire about visual clues at different frames. Generally, RaFormer can locate
very few indicative elements that involve the interaction of high activitness.
For the critical words, RaFormer tends to imply the entity in the video
(_e.g.,_ “man”, “lady”) as well as words that serve as a temporal indicator
(_e.g.,_ “after”, “end”). As for the identified critical frames, we notice
that our adaptive sampling tends to select frames that contains the key
entities in the question (_e.g.,_ “man”, “lady”). Similarly, it also select
frames that corresponds to the temporal indicator, which benefits critical
frames selection by narrowing down the target scope. This is attributed to our
cross-modal sampling strategy, where the answer is referred to as a mutual
agreement between critical frames and question words. Apart from that, we also
notice that, even for the same video, RaFormer can accredit different scenes
as the critical frames in response to the question clues. Such adaptiveness
demonstrates the design philosopy of our sampling module, where frames are
differentiate base on their interaction activeness with question, making
different question target at different scene.

##  6\. Conclusions

In this paper, we pinpoint the redundancy issue in current VideoQA paradigm.
Specifically, we design a novel video encoder to emphasize the modelling of
detailed object movement within the neighboring frames, while address the
neighboring-frame redundancy by imposing the leap attention that models the
frame-level representations in a dilate manner. To tackle the cross-model
redundancy in prevailing fuser design, we incorporate an adaptive sampling
strategy that select a small set of critical frames according to their
interactions with question tokens. Extensive experiments on four benchmark
datasets have demonstrated the superior of RaFormer. We hope this simple yet
effective design can spark more future efforts in handling VideoQA redundancy.

##  7\. Acknowledgments

This work was supported by NExT search center, the National Natural Science
Foundation of China (NSFC) under Grant 9227010114, Grant62272435, Grant
U22A2094, and the University Synergy Innovation Program of Anhui Province
(GXXT-2022-040).

## References

  * (1)
  * Buch et al. (2022) Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022\.  Revisiting the ”Video” in Video-Language Understanding. In _CVPR_. 2917–2927.
  * Cadène et al. (2019) Rémi Cadène, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2019\.  RUBi: Reducing Unimodal Biases for Visual Question Answering. In _NeurIPS_. 839–850.
  * Chen et al. (2020) Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. 2020\.  Counterfactual Samples Synthesizing for Robust Visual Question Answering. In _CVPR_. 10797–10806.
  * Dang et al. (2021) Long Hoang Dang, Thao Minh Le, Vuong Le, and Truyen Tran. 2021\.  Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering. In _IJCAI_. 636–642.
  * Dong et al. (2022) Jianfeng Dong, Xianke Chen, Minsong Zhang, Xun Yang, Shujie Chen, Xirong Li, and Xun Wang. 2022.  Partially Relevant Video Retrieval. In _Proceedings of the 30th ACM International Conference on Multimedia_. 246–257.
  * Dong et al. (2021) Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang. 2021.  Dual encoding for video retrieval by text.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44, 8 (2021), 4065–4080.
  * Dua et al. (2021) Radhika Dua, Sai Srinivas Kancheti, and Vineeth N. Balasubramanian. 2021\.  Beyond VQA: Generating Multi-Word Answers and Rationales to Visual Questions. In _CVPR_. 1623–1632.
  * Figurnov et al. (2017) Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry P. Vetrov, and Ruslan Salakhutdinov. 2017.  Spatially Adaptive Computation Time for Residual Networks. In _CVPR_. 1790–1799.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021\.  Deberta: decoding-Enhanced Bert with Disentangled Attention. In _ICLR_.
  * Jang et al. (2017) Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In _CVPR_. 2758–2766.
  * Ji et al. (2021) Wei Ji, Yicong Li, Meng Wei, Xindi Shang, Junbin Xiao, Tongwei Ren, and Tat-Seng Chua. 2021.  Vidvrd 2021: The third grand challenge on video relation detection. In _ACM MM workshop_. 4779–4783.
  * Ji et al. (2023a) Wei Ji, Renjie Liang, Lizi Liao, Hao Fei, and Fuli Feng. 2023a.  Partial Annotation-based Video Moment Retrieval via Iterative Learning. In _ACM MM_.
  * Ji et al. (2023b) Wei Ji, Renjie Liang, Zhedong Zheng, Wenqiao Zhang, Shengyu Zhang, Juncheng Li, Mengze Li, and Tat-seng Chua. 2023b.  Are binary annotations sufficient? video moment retrieval via hierarchical uncertainty-based active learning. In _CVPR_. 23013–23022.
  * Jiang and Han (2020) Pin Jiang and Yahong Han. 2020.  Reasoning with Heterogeneous Graph Alignment for Video Question Answering. In _AAAI_. 11109–11116.
  * Le et al. (2020) Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2020\.  Hierarchical Conditional Relation Networks for Video Question Answering. In _CVPR_. 9969–9978.
  * Li et al. (2022a) Jiangtong Li, Li Niu, and Liqing Zhang. 2022a.  From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering. In _CVPR_. 21241–21250.
  * Li et al. (2023) Kun Li, Jiaxiu Li, Dan Guo, Xun Yang, and Meng Wang. 2023.  Transformer-based Visual Grounding with Cross-modality Interaction.  _ACM Transactions on Multimedia Computing, Communications and Applications_ 19, 6 (2023), 1–19.
  * Li et al. (2022b) Yicong Li, Xiang Wang, Junbin Xiao, and Tat-Seng Chua. 2022b.  Equivariant and Invariant Grounding for Video Question Answering.  _CoRR_ abs/2207.12783 (2022).
  * Li et al. (2022c) Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. 2022c.  Invariant Grounding for Video Question Answering. In _CVPR_. 2928–2937.
  * Li et al. (2021a) Yicong Li, Xun Yang, Xindi Shang, and Tat-Seng Chua. 2021a.  Interventional Video Relation Detection. In _ACM MM_. 4091–4099.
  * Li et al. (2021b) Yicong Li, Xun Yang, Xindi Shang, and Tat-Seng Chua. 2021b.  Interventional video relation detection. In _ACM MM_. 4091–4099.
  * Pan et al. (2021) Bowen Pan, Rameswar Panda, Camilo Luciano Fosco, Chung-Ching Lin, Alex J. Andonian, Yue Meng, Kate Saenko, Aude Oliva, and Rogério Feris. 2021\.  VA-RED2: Video Adaptive Redundancy Reduction. In _ICLR_.
  * Park et al. (2018) Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018.  Multimodal Explanations: Justifying Decisions and Pointing to the Evidence. In _CVPR_. 8779–8788.
  * Park et al. (2021) Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. 2021.  Bridge To Answer: Structure-Aware Graph Interaction Network for Video Question Answering. In _CVPR_. 15526–15535.
  * Peng et al. (2021) Liang Peng, Shuangji Yang, Yi Bin, and Guoqing Wang. 2021\.  Progressive Graph Attention Network for Video Question Answering. In _ACM MM_.
  * Peng et al. (2022) Min Peng, Chongyang Wang, Yuan Gao, Yu Shi, and Xiang-Dong Zhou. 2022.  Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering. In _IJCAI_. 1276–1282.
  * Rao et al. (2021) Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021\.  DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. In _NeurIPS_ , Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 13937–13949.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In _NeurIPS_. 91–99.
  * Ribeiro et al. (2016) Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In _KDD_ , Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (Eds.). 1135–1144.
  * Ross et al. (2017) Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017.  Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations. In _IJCAI_. 2662–2670.
  * Shang et al. (2019) Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. 2019\.  Annotating objects and relations in user-generated videos. In _Proceedings of the 2019 on International Conference on Multimedia Retrieval_. 279–287.
  * Shang et al. (2021) Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. 2021.  Video Visual Relation Detection via Iterative Inference. In _ACM MM_. 3654–3663.
  * Tan et al. (2021) Yi Tan, Yanbin Hao, Xiangnan He, Yinwei Wei, and Xun Yang. 2021.  Selective dependency aggregation for action classification. In _Proceedings of the 29th ACM International Conference on Multimedia_. 592–601.
  * Tishby and Zaslavsky (2015) Naftali Tishby and Noga Zaslavsky. 2015.  Deep Learning and the Information Bottleneck Principle.  _CoRR_ abs/1503.02406 (2015).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NeurIPS_. 5998–6008.
  * Wang et al. ([n. d.]) Junke Wang, Xitong Yang, Hengduo Li, Li Liu, Zuxuan Wu, and Yu-Gang Jiang. [n. d.].  Efficient Video Transformers with Spatial-Temporal Token Selection. In _ECCV_.
  * Wu et al. (2022) Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. 2022.  Discovering Invariant Rationales for Graph Neural Networks. In _ICLR_.
  * Xiao et al. (2020) Junbin Xiao, Xindi Shang, Xun Yang, Sheng Tang, and Tat-Seng Chua. 2020.  Visual relation grounding in videos. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_. Springer, 447–464.
  * Xiao et al. (2021) Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021\.  NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions. In _CVPR_. 9777–9786.
  * Xiao et al. (2022a) Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. 2022a.  Video as Conditional Graph Hierarchy for Multi-Granular Question Answering. In _AAAI_. 2804–2812.
  * Xiao et al. (2022b) Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan. 2022b.  Video Graph Transformer for Video Question Answering. In _ECCV_. Springer, 39–58.
  * Xu et al. (2017) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017.  Video Question Answering via Gradually Refined Attention over Appearance and Motion. In _ACM MM_. 1645–1653.
  * Yang et al. (2021) Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. 2021.  Deconfounded video moment retrieval with causal intervention. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1–10.
  * Yang et al. (2020) Xun Yang, Xueliang Liu, Meng Jian, Xinjian Gao, and Meng Wang. 2020.  Weakly-supervised video object grounding by exploring spatio-temporal contexts. In _Proceedings of the 28th ACM international conference on multimedia_. 1939–1947.
  * Yang et al. (2022) Xun Yang, Shanshan Wang, Jian Dong, Jianfeng Dong, Meng Wang, and Tat-Seng Chua. 2022\.  Video moment retrieval with cross-modal neural architecture search.  _IEEE Transactions on Image Processing_ 31 (2022), 1204–1216.
  * Zhang et al. (2022) Hao Zhang, Lechao Cheng, Yanbin Hao, and Chong-wah Ngo. 2022\.  Long-term leap attention, short-term periodic shift for video classification. In _ACM MM_. 5773–5782.
  * Zhang et al. (2019) Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. 2019\.  Interpreting CNNs via Decision Trees. In _CVPR_. 6261–6270.
  * Zhong (2022) Yaoyao Zhong. 2022\.  Video Question Answering: Datasets, Algorithms and Challenges.  _EMNLP_ (2022).