# DSANet: Dynamic Segment Aggregation Network
for Video-Level Representation Learning

Wenhao Wu1∗†, Yuxiang Zhao1,2∗, Yanwu Xu3, Xiao Tan1, Dongliang He1,
Zhikang Zou1, Jin Ye1, Yingying Li1, Mingde Yao1, Zichao Dong1, Yifeng Shi1 1
Baidu Inc. 2 Shenzhen Institute of Advanced Technology, CAS 3 University of
Pittsburgh

(2021)

###### Abstract.

Long-range and short-range temporal modeling are two complementary and crucial
aspects of video recognition. Most of the state-of-the-arts focus on short-
range spatio-temporal modeling and then average multiple snippet-level
predictions to yield the final video-level prediction. Thus, their video-level
prediction does not consider spatio-temporal features of how video evolves
along the temporal dimension. In this paper, we introduce a novel _D ynamic
Segment Aggregation_ (DSA) module to capture relationship among snippets. To
be more specific, we attempt to generate a dynamic kernel for a convolutional
operation to aggregate long-range temporal information among adjacent snippets
adaptively. The DSA module is an efficient plug-and-play module and can be
combined with the off-the-shelf clip-based models (_i.e.,_ TSM, I3D) to
perform powerful long-range modeling with minimal overhead. The final video
architecture, coined as DSANet. We conduct extensive experiments on several
video recognition benchmarks (_i.e._ , Mini-Kinetics-200, Kinetics-400,
Something-Something V1 and ActivityNet) to show its superiority. Our proposed
DSA module is shown to benefit various video recognition models significantly.
For example, equipped with DSA modules, the top-1 accuracy of I3D ResNet-50 is
improved from 74.9% to 78.2% on Kinetics-400. Codes are available at
https://github.com/whwu95/DSANet.

neural networks, action recognition, video representation learning

$*$ Equal contribution. ${\dagger}$ Corresponding author
(wuwenhao17@mails.ucas.edu.cn).

††journalyear: 2021††copyright: acmlicensed††conference: Proceedings of the
29th ACM International Conference on Multimedia; October 20–24, 2021; Virtual
Event, China††booktitle: Proceedings of the 29th ACM International Conference
on Multimedia (MM ’21), October 20–24, 2021, Virtual Event, China††price:
15.00††doi: 10.1145/3474085.3475344††isbn: 978-1-4503-8651-7/21/10††ccs:
Computing methodologies Activity recognition and understanding

##  1\. Introduction

With the rapid development of the Internet and mobile devices, video data has
exploded over the past years. Video action recognition, as a fundamental
problem in video analytics, has become one of the most active research topics.

Figure 1.  The pipeline of our method for video recognition. Input video is
divided into several segments and a short snippet is randomly sampled from
each segment. Green trapezoids denote the snippet-level spatio-temporal
networks (_e.g._ , I3D, TSM) which share parameters on all snippets. DSA
module can be easily inserted into the networks to capture long-range temporal
structure.

Previous methods have obtained noticeable improvements for video recognition.
3D CNNs are intuitive spatio-temporal networks and natural extension from
their 2D counterparts, which directly tackle 3D volumetric video data (Tran et
al., 2015; Carreira and Zisserman, 2017). However, the heavy computational
cost of 3D CNNs limits its application. Recent studies have shown that
factorizing 3D convolutional kernel into spatial (_e.g._ ,
1$\times$3$\times$3) and temporal components (_e.g._ , 3$\times$1$\times$1) is
preferable to reduce complexity as well as boost accuracy, _e.g._ , P3D (Qiu
et al., 2017), R(2+1)D (Tran et al., 2018), S3D-G (Xie et al., 2018), _etc_.
In practice, however, compared with their 2D counterparts, the increased
computational cost cannot be ignored. Thus, 3D channel-wise convolution are
also applied to video recognition in CSN (Tran et al., 2019) and X3D
(Feichtenhofer, 2020). To capture temporal information with reasonable
training resources, the other alternative 2D CNN based architectures are
developed _i.e.,_ TSM (Lin et al., 2019), TEI (Liu et al., 2020a), TEA (Li et
al., 2020), MVFNet (Wu et al., 2021), _etc_. There methods aim to design
efficient temporal module based on existing 2D CNNs to perform efficient
temporal modeling. Temporal Shift Module (TSM) (Lin et al., 2019) is the
representative 2D-based model, which achieves a good trade-off between
performance and complexity meanwhile introduces zero FLOPs for the 2D CNN
backbone.

However, the above architectures, both based on 2D CNNs and 3D CNNs, are
mainly designed to learn short-range spatio-temporal representation (snippet-
level) instead of long-range representation (video-level). Specifically,
during training, these snippet-level methods sample sub-frames (4 frames, 8
frames or 16 frames) from a random snippet of the original video. While for
inference, they uniformly sample multiple short snippets from a video and
compute the classification scores for all snippets individually and the final
prediction is just the average voting from the snippet-level predictions.
Thus, during training, these clip-based models take no account of modeling
long-range temporal structure due to the limited observation of only a single
snippet. Also, simply averaging the prediction scores of all snippets could
yield a sub-optimal solution during inference.

To alleviate the problem mentioned above, Temporal Segment Network (Wang et
al., 2016) provides a simple framework to perform video-level learning, which
just simply fuses snippet-level predictions. Lately, V4D (Zhang et al., 2020)
presents an intuitive 4D convolutional operation to capture inter-clip
interaction, which is a natural extension of enhancing the representation of
the original clip-based 3D CNNs. However, TSN fails on capturing the finer
temporal structure, (_e.g._ , the temporal pattern of the ordering of
features), and V4D causes high computational cost compared with the TSN-
equipped counterpart and might lack the ability to describe the dynamics by
simply employing a fixed number of video invariant kernels. This inspires us
to focus on designing a temporal module, which keeps both advantages of high
efficiency and strong ability of capturing long-range temporal structure.

In this paper, we creatively introduce a module to adaptively aggregate the
features of multiple snippets, namely _D ynamic Segment Aggregation Module_
(DSA Module). Figure 1 illustrates the pipeline of our solution, DSA module is
a novel generic module that can be easily plugged into the existing clip-based
networks (_e.g.,_ TSM (Lin et al., 2019), I3D (Carreira and Zisserman, 2017))
to perform long-range modeling with the ignorable cost. Briefly, the DSA
module firstly gets global context information from features of all snippets,
then it utilizes the contextual view to learns the weight of aggregation in a
channel-wise convolution manner. Thus, DSA module can capture relation-ship
among snippets via dynamic fusion along snippet dimension. To make this module
more efficient, we separate the features across the channel dimension. One
keeps the original activation and the other performs dynamic segment
aggregation separately followed by a concatenation operation. In this way, DSA
only introduces a bit more FLOPs which is close to zero, while bringing
remarkable improvement. In practice, we integrate the DSA module into the
existing spatio-temporal block (_e.g._ , TSM block, I3D block) to form our
DSANet.

We identify the effectiveness of the proposed DSA module via comprehensive
ablations. Extensive experimental results on multiple well-known datasets,
including Mini-Kinetics-200 (Xie et al., 2018), Kinetics-400 (Kay et al.,
2017), Something-Something V1 (Goyal et al., 2017) and ActivityNet-v1.3 (Caba
Heilbron et al., 2015) show the superiority of our solution. All of these
prove DSA boosts the performance over its snippet-level counterpart. Overall,
our major contributions are summarized as follows:

  * •

Instead of snippet-level temporal modeling, we propose to exploit an effective
and efficient video-level framework for learning video representation. To
tackle this, the proposed DSA module provides a novel mechanism to adaptively
aggregate snippet-level features.

  * •

The DSA module works in a plug-and-play way and can be easily integrated into
existing snippet-level methods. Without any bells and whistles, the DSA module
brings consistent improvements when combined with both 2D CNN-based and 3D
CNN-based networks (_e.g._ , TSM, I3D, _etc_).

  * •

Extensive experiments on four public benchmark datasets demonstrate that the
proposed DSA obtain an evident improvement over previous long-range temporal
modeling methods with a minimal computational cost.

##  2\. Related Work

###  2.1. CNNs for Video Recognition.

Using cutting edge 2D CNN architecture is desirable to address the video
recognition task. Two-stream(Simonyan and Zisserman, 2014) proposes to jointly
learn spatial and temporal context through two parallel branch structure. 3D
CNN expands a new dimension of 2D convolution whereby intuitively apply to
assemble spatio-temporal feature. C3D(Tran et al., 2015) is the first one to
using VGG of 3D type in video recognition. I3D (Carreira and Zisserman, 2017)
is a milestone of 3D convolution which used pre-trained parameters of 2D CNNs
onto the 3D CNNs by inflating the parameter on a new axis, boost the
performance by a large margin. Along with significant study on separable
convolution, some methods (_e.g.,_ S3D-G (Xie et al., 2018), R(2+1)D (Tran et
al., 2018), P3D (Qiu et al., 2017)) decompose the full 3D convolution into a
2D convolution and a 1D temporal convolution to reduce the burdensome
parameters and accelerate the training.

To alleviate the high computational cost of video classification networks, the
other alternative 2D CNN based architectures (_e.g.,_ STM (Jiang et al.,
2019), TSM (Lin et al., 2019), ECO (Zolfaghari et al., 2018), TEI (Liu et al.,
2020a), TEA (Li et al., 2020), GSM (Sudhakaran et al., 2020), MVFNet (Wu et
al., 2021)) and channel-wise 3D CNN based architectures (_e.g.,_ CSN (Tran et
al., 2019), X3D (Feichtenhofer, 2020)) are developed. Also, there is active
research on dynamic inference (Wu et al., 2020; Wang et al., 2021). These
methods can get good computation/accuracy trade-offs.

Besides, recently we notice that the concurrent work to ours is TANet (Liu et
al., 2020b), which composed of two branches, the local branch aims to learn a
location sensitive importance map to enhance discriminative features and the
global branch produces the location invariant weights to aggregate temporal
information. Both the global branch of TANet and our DSANet are inspired by
the previous dynamic convolution mechanisms (Chen et al., 2020; Wang et al.,
2019). However, the TANet aims to capture _clip-level_ temporal information
and our DSANet targets on _video-level_ temporal information.

###  2.2. Long-Term Modeling for Video Recognition

The obvious difference between long-term modeling method and short-term
modeling method is whether the method can capture the relation of different
video segments for video-level representation learning. Various long-term
modeling frameworks have been developed for capturing more complex temporal
structure in video action recognition fields. In (Donahue et al., 2015), CNN
features of video frames are fed into the LSTM network for video recognition.
ShuttleNet (Shi et al., 2017) introduces biologically-inspired feedback
connections to model long-term dependencies of spatial CNN descriptors.
NetVlad (Arandjelovic et al., 2016), ActionVlad (Girdhar et al., 2017),
AttentionClusters (Long et al., 2018) are proposed for better local feature
integration instead of directly average pooling as used. MARL(Wu et al., 2019)
uses multiple agents as frame selectors instead of the general uniform sampler
from the entire video for better global temporal modeling. Each agent learns a
flexibly moving policy through the temporal axis to get a vital representation
frame and other agents’ behavior as well. (Korbar et al., 2019) proposes a new
solution as dynamic sampler. Ranking the high score clips generated by a small
classifier network, a top-k pooling method is then applied to feed the most
related parts into a large network.

Figure 2. Illustration of our framework for video action recognition. (a)
shows the TSM block equipped with a DSA module. (b) shows the I3D block
equipped with a DSA module. (c) depicts the architecture of the DSA module.
(d) shows our DSA block which integrates the DSA module into the ResNet-Style
block.

Closely related works are TSN (Wang et al., 2016), T-C3D(Liu et al., 2018),
StNet(He et al., 2019) and V4D (Zhang et al., 2020). (Wang et al., 2016)
operates on a sequence of short snippets sparsely sampled from the entire
video and the parameter is updated according to segmental consensus derived
from all snippet-level prediction. T-C3D replaces 2D backbones with C3D. So
these models only capture coarse co-occurrence rather than the finer temporal
structures. StNet applies 2D ConvBlocks on “superimage” for local spatial-
temporal modeling and 1D temporal convolutions for global temporal dependency.
V4D (Zhang et al., 2020) tends to focus on the intuitive 4D convolutional
operation to capture inter-clip interaction, which is a natural extension to
enhance the representation power of the original snippet-level 3D CNNs,
however, causes high computational cost. Therefore we present a flexible long-
range modeling module that shares the capacity of learning video-level
representations yet still keeps the efficiency of the snippet-level
counterpart.

##  3\. Dynamic Segment Aggregation

###  3.1. Preliminary: 4D Convolution

To aggregate the snippet-level features, V4D cast 3D convolution into 4D
convolution. Formally, following the notations in V4D (Zhang et al., 2020),
the whole video formed by a series of short snippets of
$A_{i}\in\mathbb{R}^{C\times T\times H\times W}$, where $C$ is the number of
the input channel, $C^{\prime}$ denotes the number of output channel, $T$ is
the dimension of sampled frames from each snippet, and $H$, $W$ represent
height and width of the frames respectively. Further, video can then be
represented as a 5-D tensor $V:=\left\\{A_{1},A_{2},\ldots,A_{U}\right\\}$,
where $U$ is the number of snippets as a new dimension. Then, the 4D
convolutional operation can be formulated as:

(1) $O_{c^{\prime},u,t,h,w}=\sum_{c,i,j,k,l}K_{c^{\prime},c,l,k,i,j}\
V_{c,u+\hat{l},t+\hat{k},h+\hat{i},w+\hat{j}},$

where $O$ is the output tensor, $K$ is the 4D convolution kernel, $i,\ j,\ k,\
l$ index along the $H$, $W$, $T$, $U$ dimensions of kernel and $c^{\prime}$
represents the number of kernel.

###  3.2. Dynamic Kernel Generation

Due to the flowing dynamics of videos, we learn an adaptive, context-aware
kernel to handle video variations along the temporal dimension. To be
simplified, we denote the input to the DSA module as a tensor
$V\in\mathbb{R}^{C\times U\times T\times H\times W}$. The overall DSA module
is illustrated in Figure 2(c). We can formalize this dynamic kernel generation
as follows:

(2) $K=Softmax(\mathcal{F}(\theta,g(V))),$

where $g$ aggregates video feature maps $V$ for global view by using global
average pooling, which produces ${\hat{V}}\in\mathbb{R}^{C\times U}$.
$\mathcal{F}$ is a mapping function that could generate kernel $K$ with
learnable parameters of $\theta$. And $K\in\mathbb{R}^{C\times L}$ is
generated dynamic convolutional kernel for each channel, where $L$ is the
kernel size. In this paper, we set $L$ to 3. To capture context information
effectively, we use multi-layer perceptron (MLP) network with batch
normalization and activation function (_e.g._ , ReLU) to instantiate
$\mathcal{F}$ in this paper. $\alpha$ is a super-parameters of fully connected
layer in MLP to control the size of feature. To conduct better capability, we
constraint $K$ with a _Softmax_ function to generate normalized kernel
weights.

Table 1. Two backbone instantiations of the I3D network. The dimensions of
kernels are denoted by $\\{$$T$$\times$$S^{2}$, $C$$\\}$ for temporal,
spatial, and channel sizes. Strides are denoted as $\\{$temporal stride,
spatial stride2$\\}$.

stage | I3D ResNet18 | I3D ResNet50 | output sizes
---|---|---|---
raw clip | - | - | 32$\times$224${}^{\text{2}}$
data layer | stride 8, 1${}^{\text{2}}$ | stride 8, 1${}^{\text{2}}$ | 4$\times$$224^{2}$
conv1 | 1$\times$7${}^{\text{2}}$, 64 | 1$\times$7${}^{\text{2}}$, 64 | 4$\times$$112^{2}$
stride 1, 2${}^{\text{2}}$ | stride 1, 2${}^{\text{2}}$
pool1 | 1$\times$3${}^{\text{2}}$ max | 1$\times$3${}^{\text{2}}$ max | 4$\times$$56^{2}$
stride 1, 2${}^{\text{2}}$ | stride 1, 2${}^{\text{2}}$
res2 | $\left[\begin{array}[]{c}\text{1$\times$3${}^{\text{2}}$, {64}}\\\\[-1.00006pt] \text{1$\times$3${}^{\text{2}}$, {64}}\end{array}\right]$$\times$2 | $\left[\begin{array}[]{c}\text{1$\times$1${}^{\text{2}}$, {64}}\\\\[-1.00006pt] \text{1$\times$3${}^{\text{2}}$, {64}}\\\\[-1.00006pt] \text{1$\times$1${}^{\text{2}}$, {256}}\end{array}\right]$$\times$3 | 4$\times$$56^{2}$
res3 | $\left[\begin{array}[]{c}\text{1$\times$3${}^{\text{2}}$, {128}}\\\\[-1.00006pt] \text{1$\times$3${}^{\text{2}}$, {128}}\end{array}\right]$$\times$2 | $\left[\begin{array}[]{c}\text{1$\times$1${}^{\text{2}}$, {128}}\\\\[-1.00006pt] \text{1$\times$3${}^{\text{2}}$, {128}}\\\\[-1.00006pt] \text{1$\times$1${}^{\text{2}}$, {512}}\end{array}\right]$$\times$4 | 4$\times$$28^{2}$
res4 | $\left[\begin{array}[]{c}\text{3$\times$3${}^{\text{2}}$, {256}}\\\\[-1.00006pt] \text{3$\times$3${}^{\text{2}}$, {256}}\end{array}\right]$$\times$2 | $\left[\begin{array}[]{c}\text{\text@underline{3$\times$1${}^{\text{2}}$}, {256}}\\\\[-1.00006pt] \text{1$\times$3${}^{\text{2}}$, {256}}\\\\[-1.00006pt] \text{1$\times$1${}^{\text{2}}$, {1024}}\end{array}\right]$$\times$6 | 4$\times$$14^{2}$
res5 | $\left[\begin{array}[]{c}\text{3$\times$3${}^{\text{2}}$, {512}}\\\\[-1.00006pt] \text{3$\times$3${}^{\text{2}}$, {512}}\end{array}\right]$$\times$2 | $\left[\begin{array}[]{c}\text{\text@underline{3$\times$1${}^{\text{2}}$}, {512}}\\\\[-1.00006pt] \text{1$\times$3${}^{\text{2}}$, {512}}\\\\[-1.00006pt] \text{1$\times$1${}^{\text{2}}$, {2048}}\end{array}\right]$$\times$3 | 4$\times$$7^{2}$
global average pool, fc | # classes

###  3.3. DSA as a Specific Case of 4D Convolution

In this section, we formulate our proposed DSA as a specific case of 4D
convolution. While capturing the long-term perception structure, the learned
kernel $K$ above can be viewed as aggregation weights to perform dynamic
weighted fusion along $U$ dimensions via channel-wise convolutional operation

(3) $O_{c,u,t,h,w}=\sum_{l}K_{c,l}V_{c,u+l,t,h,w}.$

At this point, we can see that DSA has a reduced kernel size along $T,H,W$
dimensions, which is also a 4D convolutional kernel.

To make this module more efficient, we decompose the feature maps into two
parts as shown in Figure 2(c). Let $V_{1}\in\mathbb{R}^{\beta C\times U\times
T\times H\times W}$ and $V_{2}\in\mathbb{R}^{(1-\beta)C\times U\times T\times
H\times W}$ be the two splits of $V$ along the channel dimension, where
$\beta\in[0,1]$ denoted the proportion of input channels for dynamic
aggregation. Finally, we get the DSA module output $Y\in\mathbb{R}^{C\times
U\times T\times H\times W}$ as:

(4) $Y_{c,u,t,h,w}=\mathit{Concat}(\sum_{l}K_{c,l}V_{1_{c,u+l,t,h,w}},V_{2}),$

where _Concat_ represents concatenation operation along dimension of channel.

###  3.4. DSA Network Architecture

The proposed DSA module is a flexible module that can be easily plugged into
the existing clip-based networks to capture long-range temporal structure.
Here we illustrate how to adapt it into a residual style block, and then build
up the _Dynamic Segment Aggregation Network_ (DSANet). To evaluate the
generality of the DSA module, we build up the DSANet on both 2D and 3D
backbone networks (_i.e._ , TSM, I3D). TSM (Lin et al., 2019) and I3D
(Feichtenhofer et al., 2019), the two widely used networks, are respectively
the representatives of the 2D CNN-based model and 3D CNN-based model. To avoid
confusion, as shown in Table 1, I3D refer to the slow-only branch of SlowFast
(Feichtenhofer et al., 2019) which has no temporal downsampling operations.

For a fair comparison with many recent state-of-the-art video recognition
methods, we follow the same practices and instantiate the DSANet using ResNet-
style backbone in this instantiation. Specifically, we integrate the proposed
module into the I3D ResNet block or TSM ResNet block to construct our DSA
block. The overall design of the DSA block is shown in Figure 2(a) and Figure
2(b). And there are four optional positions (denoted as I, II, III, IV) to
insert DSA module and we will perform ablation studies in Sec. 4.3. Then
following Figure 2(d), we replace some residual blocks with DSA blocks to form
our DSANet.

###  3.5. Discussion

We have noticed that our DSA module gets global context information from
features of all snippets by _global average pooling_ , which is same with the
_squeeze operation_ in SENet (Hu et al., 2018). However, there are essential
differences between SENet and DSA as follows:

1) We propose to aggregate the snippet-level features by utilizing an
efficient convolutional operation.

2) In corresponding to the dynamics of video data, we further build a method
to learn an adaptive convolutional kernel with ignorable computation cost for
the convolutional operation. Essentially, our method is different from SENet,
which applies a self-attention mechanism to generate a channel-wise attention.

##  4\. Experiments

###  4.1. Datasets and Evaluation Metrics

We evaluate our method on several standard video recognition benchmarks,
including Mini-Kinetics-200 (Xie et al., 2018), Kinetics-400 (Kay et al.,
2017), Something-Something V1 (Goyal et al., 2017) and ActivityNet-v1.3 (Caba
Heilbron et al., 2015). Kinetics-400 contains 400 human action categories and
provides around 240k training video clips and 20k validation video clips. Each
example is temporally trimmed to be around 10 seconds. Mini-Kinetics-200
consists of the 200 categories with the most training examples and is a subset
of the Kinetics-400 dataset. Since the full Kinetics-400 dataset is quite
large, we use Mini-Kinetics-200 for our ablation experiments. For Something-
Something V1 datasets, the actions therein mainly include object-object and
human-object interactions, which require strong temporal relation to well
categorizing them. Something-Something V1 includes about 110k videos for 174
fine-grained classes. To elaborately study the effectiveness and
generalization ability of our solution, we also conduct experiments for
untrimmed video recognition. We choose ActivityNet-v1.3 which is a large-scale
untrimmed video benchmark, contains 19,994 untrimmed videos of 5 to 10 minutes
from 200 activity categories.

We report top-1 and top-5 accuracy (%) for Mini-Kinetics-200, Kinetics-400 and
Something-Something V1. For ActivityNet-v1.3, we follow the original official
evaluation scheme using mean average precision (mAP). Also, we report the
computational cost (in FLOPs) as well as the number of model parameters to
depict model complexity. In this paper, we only use the RGB frames of these
datasets and don’t use extra modalities (_e.g.,_ optical flow, audio).

###  4.2. Implementation Details

Here we report the implementation details.

Table 2. Ablation studies on Mini-Kinetics-200. We show top-1 and top-5
classification accuracy (%), as well as computational complexity measured in
FLOPs (floating-point operations).

(a) Study on the effectiveness of DSA module. $T$ denotes the number of frames sampled from each video snippet, $U$ denotes the number of snippets. Backbone: I3D R18. Model |  $T_{train}$$\times$$U_{train}$ |  $T_{infer}$$\times$$U_{infer}$$\times$#crop | Top-1 | Top-5 | Params
---|---|---|---|---|---
I3D R18 | $4\times 1$ | $4\times 10\times 3$ | 72.2 | 91.2 | 32.3M
I3D R18 | $16\times 1$ | $16\times 10\times 3$ | 73.4 | 91.1 | 32.3M
TSN+I3D R18 | $4\times 4$ | $4\times 10\times 3$ | 73.0 | 91.3 | 32.3M
V4D+I3D R18 | $4\times 4$ | $4\times 10\times 3$ | 75.6 | 92.7 | 33.1M
DSA+I3D R18 | $4\times 4$ | $4\times 8\times 3$ | 77.3 | 93.9 | 32.3M

(b) Study on different position to insert DSA module. Setting: I3D R50, $\alpha$=2, $\beta$=1, stage: res5.  Position | Top-1 | Top-5
---|---|---
I | 81.4 | 95.4
II | 81.5 | 95.2
III | 80.8 | 95.2
IV | 81.4 | 95.1

(c) Parameter choices of $\bm{\alpha}$. Setting: I3D R50, Position II, $\beta$=1, inserted stage: res5. Setting | Top-1 | Top-5
---|---|---
$\alpha$=1 | 81.0 | 95.1
$\alpha$=2 | 81.5 | 95.2
$\alpha$=4 | 81.2 | 95.0
$\alpha$=8 | 81.3 | 95.0

(d) The DSA blocks in different stage of I3D R50. Setting: Position II, $\alpha$=2, $\beta$=1. Stage | Top-1 | Top-5
---|---|---
res{2} | 81.4 | 94.7
res{3} | 81.3 | 95.1
res{4} | 81.3 | 95.3
res{5} | 81.5 | 95.2

(e) Parameter choices of $\bm{\beta}$. Setting: I3D R50, Position II, $\alpha$=2, inserted stage: res5. Setting | Top-1 | Top-5
---|---|---
$\beta$=1 | 81.5 | 95.2
$\beta$=1/2 | 81.7 | 95.4
$\beta$=1/4 | 81.6 | 95.0
$\beta$=1/8 | 81.5 | 95.0

(f) The number of DSA block inserted into I3D R50. Setting: Position II, $\alpha$=2, $\beta$=1/8. Stages | Blocks | Top-1 | Top-5
---|---|---|---
res{5} | 1 | 81.5 | 95.0
res{4,5} | 4 | 81.5 | 95.3
res{3,4} | 5 | 81.8 | 95.4
res{2,3} | 3 | 81.4 | 95.1

(g) Different short-term temporal structure for DSA module. Model | Top-1 | Top-5
---|---|---
TSM R50 | 77.4 | 93.4
DSA+TSM R50 | 80.4 | 95.0
I3D R50 | 78.0 | 93.9
DSA+I3D R50 | 81.8 | 95.4

(h) Study on the effectiveness of DSA module with different backbones (I3D R18, I3D R50). SENet+I3D uses SE module to replace the DSA module in DSANet. Arch. | I3D | SENet+I3D | DSA+I3D
---|---|---|---
ResNet18 | 72.2 | 73.8 | 77.3
ResNet50 | 78.0 | 78.5 | 81.8

(i) Training FLOPs. Comparison with V4D, the extra computation cost brought by the DSA module is close to zero. Model | Input size | FLOPs
---|---|---
TSN+I3D R50 | $4\times 4\times 224^{2}\times 3$ | 83.8G
V4D+I3D R50 | $4\times 4\times 224^{2}\times 3$ | 143.0G
DSA+I3D R50 | $4\times 4\times 224^{2}\times 3$ | 83.8G

Training. We evaluate the DSA module on both 2D and 3D backbone networks.
Specifically, the slow-only branch of SlowFast (Feichtenhofer et al., 2019) is
applied as our 3D backbone baseline (denoted as I3D) and TSM (Lin et al.,
2019) is applied as our 2D backbone baseline. We use random initialization for
our DSA module and train the model in an end-to-end manner. Following the
similar practice in V4D (Zhang et al., 2020) and TSN (Wang et al., 2016), we
uniformly divide the whole video into $U$ segments and we set $U=4$ in this
paper, then randomly select a snippet of consecutive 32 or 64 frames from each
segment as temporal augmentation. For each snippet, we uniformly sample 4 or 8
frames with a fixed stride of 8. The size of the short side of these frames is
fixed to 256 and then random scaling is utilized for data augmentation.
Finally, we resize the cropped regions to 224$\times$224 for network training.
On the Mini-Kinetics-200 dataset, the learning rate is 0.01 and will be
reduced by a factor of 10 at 40 and 60 epochs (70 epochs in total)
respectively. On the Kinetics-400 dataset, our model is trained for 100 epochs
starting with a learning rate of 0.01 and reducing it by a factor of 10 at 50,
75, and 90 epochs. For Something-Something V1 dataset, our model is trained
for 50 epochs starting with a learning rate 0.01 and reducing it by a factor
of 10 at 30, 40 and 45 epochs. For these three datasets, our models are
initialized by pre-trained models on ImageNet (Deng et al., 2009). As for
ActivityNet-v1.3, we followed the common practice to fine-tune from
Kinetics-400 pre-trained weights and start training with a learning rate of
0.01 for 200 epochs. The learning rate is decayed by a factor of 10 at 80 and
160 epochs.

Inference. We follow the common setting in (Feichtenhofer et al., 2019) to
just uniformly sample multiple clips from a video along its temporal axis. We
sample 8 clips unless otherwise specified. For each clip, we resize the short
side to 256 pixels and take 3 spatial crops in each frame.

###  4.3. Ablation Studies

This section provides ablation studies on Mini-Kinetics-200 to investigate
different aspects of the DSA module as shown in Table 2(i). Accordingly,
concrete analysis is as follows.

Effectiveness of DSA. We first investigate the performance of our DSA module
with a toy backbone I3D ResNet-18 (R18). Specifically, we follow the Slow path
in SlowFast (Feichtenhofer et al., 2019) to build the I3D R18 backbone which
is the same as the structure in V4D (Zhang et al., 2020). As it can be seen in
Table 2(a), long-term modeling is complementary to clip-based method I3D R18.
Specifically, when equipped with TSN, V4D, and our DSA, I3D R18 obtains 0.8%,
3.4%, and 5.1% gain respectively. In comparison to these video-level methods,
DSA significantly outperforms TSN and V4D by 4.3% and 1.7% accuracy with the
same training protocols and even using less number of snippets for inference.
In the following, unless stated otherwise, we apply I3D ResNet-50 (R50) as the
backbone.

Inserted Position. As in Figure 2(b), there are four optional positions to
integrate our DSA module into the standard I3D ResNet block. Table 2(b) shows
the results of DSA module in different position. We find that position II gets
slightly higher than position I and IV. Hence, the DSA module is inserted
before the second convolution of the I3D ResNet Block by default in the
following ablations.

Table 3. Comparison with the state-of-the-art models on Mini-Kinetics-200 dataset. $T$ denoted the temporal length of video snippet, $U$ denoted the number of video snippets. Method | Backbone | $T_{train}\times U_{train}$ |  $T_{infer}\times U_{infer}\times$ #crop | Top-1 | Top-5
---|---|---|---|---|---
S3D (Xie et al., 2018) | S3D Inception | $64\times 1$ | N/A | 78.9% | -
I3D (Yue et al., 2018) | 3D ResNet50 | $32\times 1$ | $32\times 10\times 3$ | 75.5% | 92.2%
I3D (Yue et al., 2018) | 3D ResNet101 | $32\times 1$ | $32\times 10\times 3$ | 77.4% | 93.2%
I3D+NL (Yue et al., 2018) | 3D ResNet50 | $32\times 1$ | $32\times 10\times 3$ | 77.5% | 94.0%
I3D+CGNL (Yue et al., 2018) | 3D ResNet50 | $32\times 1$ | $32\times 10\times 3$ | 78.8% | 94.4%
I3D+NL (Yue et al., 2018) | 3D ResNet101 | $32\times 1$ | $32\times 10\times 3$ | 79.2% | 93.2%
I3D+CGNL (Yue et al., 2018) | 3D ResNet101 | $32\times 1$ | $32\times 10\times 3$ | 79.9% | 93.4%
V4D+I3D (Zhang et al., 2020) | 3D ResNet18 | $4\times 4$ | $4\times 10\times 3$ | 75.6% | 92.7%
V4D+I3D (Zhang et al., 2020) | 3D ResNet50 | $4\times 4$ | $4\times 10\times 3$ | 80.7% | 95.3%
DSA+I3D (Ours) | 3D ResNet18 | $4\times 4$ | $4\times 8\times 3$ | 77.3% | 93.9%
DSA+I3D (Ours) | 3D ResNet50 | $4\times 4$ | $4\times 8\times 3$ | 81.8% | 95.4%

Parameter choices of $\bm{\alpha}$. We experiment with different $\alpha$ to
figure out the optimal super-parameters of fully connected layers in the DSA
module as shown in Figure 2(c). And in Table 2(c), our module with $\alpha=2$
get better performance thus we choose it in our experiments.

Table 4. Comparison with the state-of-the-art models on Kinetics-400 dataset.
We report the inference cost by computing the GFLOPs. ∗ indicates the result
of our calculation using the official model.

Method | Backbone |  $T_{infer}\times U_{infer}\times$ #crop | GFLOPs | Top-1 | Top-5
---|---|---|---|---|---
TSM (Lin et al., 2019) | ResNet-50 | 8$\times$10$\times$3 | 33$\times$30$=$990 | 74.1% | 91.2%
TEINet (Liu et al., 2020a) | ResNet-50 | 8$\times$10$\times$3 | 33$\times$30$=$990 | 74.9% | 91.8%
TEA (Li et al., 2020) | ResNet-50 | 8$\times$10$\times$3 | 35$\times$30$=$1050 | 75.0% | 91.8%
TANet (Liu et al., 2020b) | ResNet-50 | 8$\times$10$\times$3 | 43$\times$30$=$1290 | 76.1% | 92.3%
MVFNet (Wu et al., 2021) | ResNet-50 | 8$\times$10$\times$3 | 33$\times$30$=$990 | 76.0% | 92.4%
NL+I3D (Wang et al., 2018) | 3D ResNet-50 | 32$\times$10$\times$3 | 70.5$\times$30$=$2115 | 74.9% | 91.6%
NL+I3D (Wang et al., 2018) | 3D ResNet-50 | 128$\times$10$\times$3 | 282$\times$30$=$8460 | 76.5% | 92.6%
X3D-L (Feichtenhofer, 2020) | - | 16$\times$10$\times$3 | 24.8$\times$30$=$744 | 77.5% | 92.9%
Slowfast (Feichtenhofer et al., 2019) | 3D R50+3D R50 | (4+32)$\times$10$\times$3 | 36.1$\times$30$=$1083 | 75.6% | 92.1%
Slowfast (Feichtenhofer et al., 2019) | 3D R50+3D R50 | (8+32)$\times$3$\times$10 | 65.7$\times$30$=$1971 | 77.0% | 92.6%
Slowfast (Feichtenhofer et al., 2019) | 3D R101+3D R101 | (8+32)$\times$3$\times$10 | 106$\times$30$=$3180 | 77.9% | 93.2%
Slowonly (Feichtenhofer et al., 2019) | 3D ResNet-50 | 8$\times$10$\times$3 | 41.9$\times$30$=$1257 | 74.9% | 91.5%
V4D+I3D (Zhang et al., 2020) | 3D ResNet-50 | 8$\times$10$\times$3 | 286.1$\times$2.5$\times$3$=$2146∗ | 77.4% | 93.1%
DSA+I3D (Ours) | 3D ResNet-50 | 4$\times$8$\times$3 | 83.8$\times$2$\times$3$=$503 | 77.7% | 93.1%
DSA+I3D (Ours) | 3D ResNet-50 | 8$\times$8$\times$3 | 167.7$\times$2$\times$3$=$1006 | 78.2% | 93.2%
DSA+I3D (Ours) | 3D ResNet-50 | (4+8)$\times$8$\times$3 | 251.5$\times$2$\times$3$=$1509 | 79.0% | 93.7%

Which stage to insert DSA blocks. We denote the conv2_x to conv5_x of ResNet
architecture as res{2} to res{5}. To evaluate which stage may be important for
DSA design, we insert DSA blocks into a different stage of I3D R50 and only
replace one with DSA block for every two I3D ResNet blocks. Table 2(d) shows
that the performance of DSA blocks on res{2}, res{3}, res{4} or res{5} is
similar, which suggests that DSA can perform effective adaptive fusion on both
low-level features and high-level features of snippets.

Parameter choices of $\bm{\beta}$. As shown in Table 2(e), we compare networks
with different proportion of input channels ($\beta=1/8,1/4,1/2,1$) for long-
term temporal modeling. Here we add DSA blocks into res{5} and we observe that
the change in $\beta$ appeared to have little impact on performance thus we
adopt $\beta=1/8$ for efficiency in the following.

The number of DSA Blocks. We here expect to figure out if more DSA blocks can
further improve performance. Table 2(f) compares DSA blocks added to multiple
stages of I3D ResNet. The res{3,4} achieves the highest performance and will
be used in the following experiments.

Generality of DSA. Being a general module, DSA could be combined with both 2D
and 3D clip-based networks. To show this, we add DSA to TSM (DSA+TSM R50)
which is the recent state-of-the-art model based on 2D CNN and train such a
combination in an end-to-end manner. For the 3D backbone, the slow-only branch
of SlowFast is applied as our backbone network (denoted as I3D R50) due to its
promising performance on various datasets. As shown in Table 2(g), adding DSA
to TSM R50 could boost the top-1 accuracy by 3.0%. A consistent improvement is
observed as in I3D R50, which has an increase of 3.8%.

Different Backbone. In Table 2(h) we compare various instantiations of DSA
networks on Mini-Kinetics-200. As expected, using deeper backbones is
complementary to our method. Comparing with the I3D R18 counterparts, our DSA
gets additional improvement on I3D R50.

Comparison with SENet. To avoid misunderstanding, here we clarify the
essential difference between SENet and DSANet is that SENet is a kind of self-
attention mechanism by using its global feature to calibrate the features in
channel-level, while our DSA generates dynamic convolution kernel to
adaptively aggregate the snippet-level features with a global view. To prove
the effectiveness of DSA, we conduct the comparative experiments in Table
2(h). Under the same setting, our DSA outperforms the SE module with a clear
margin (77.3% _vs._ 73.8%, 81.8% _vs._ 78.5%) on I3D R18 and I3D R50.

Training Cost. Here we make a comparison with the state-of-the-art video-level
method V4D+I3D under the same setting of backbone and inputs. We list the
number of FLOPs for all models in Table 2(i), our DSA+I3D is more lightweight
(83.8G _vs._ 143.0G) than V4D+I3D (Zhang et al., 2020) and achieve a similar
cost with the baseline TSN+I3D.

###  4.4. Comparison with State-of-the-arts

In this section, we compare our approach with the state-of-the-art methods on
trimmed video datasets (_i.e.,_ Mini-Kinetics-200 (Xie et al., 2018),
Kinetics-400 (Kay et al., 2017) and Something-Something V1 (Goyal et al.,
2017)) and untrimmed video dataset (_i.e.,_ ActivityNet (Caba Heilbron et al.,
2015)).

Results on Mini-Kinetics-200. We evaluate the DSA-equipped networks on Mini-
Kinetics-200. As shown in Table 3, DSA with less input already outperforms the
previous methods. Notably, DSA-R18 (77.3%) has similar accuracy with I3D R101
(77.4%) and I3D R50-NL (77.5%). Compared with V4D, DSA achieves better
accuracy on R18 (77.3% _vs._ 75.6%) and R50 (81.8% _vs._ 80.7%) with fewer
clips for prediction (8 _vs._ 10) which shows the superiority of our method.

Results on Kinetics-400. We make a thorough comparison in Table 4, where our
DSA outperforms the recent SOTA approaches on Kinetics-400. Here we only list
the models using RGB as inputs for fair comparisons. We provide results of our
DSA network trained with two different inputs (_i.e._ , 4 frames ($T=4$) or 8
frames ($T=8$) sampled from each snippet). The upper part of the Table 4 shows
the results of clip-based models. Compared with the 2D CNN-based model, during
inference, when utilizing less frames as input (4$\times$8 _vs._ 8$\times$10),
our DSA still outperforms these current state-of-the-art efficient methods
(_e.g._ , TSM, TEINet, TEA and MVFNet) with a clear margin (77.7% _vs._
74.1%/74.9%/75.0%/76.0%) meanwhile with (2$\times$) less calculation cost.
Compared with computationally expensive models of 3D architecture, DSA with
4$\times$8 frames input outperforms NL I3D-R50 with 32$\times$10 frames (77.7%
_vs._ 74.9%) but only uses 4.2$\times$ less GFLOPs, and uses 16.8$\times$ less
GFLOPs than NL I3D-R50 with 128$\times$10 frames meanwhile achieves a better
accuracy (77.7% _vs._ 76.5%). Moreover, DSA with 8$\times$8 frames input
achieves a better accuracy than Slowfast-R50 and Slowfast-R101 (78.2% _vs._
77.9%/77.0%) when using 3.2$\times$, 2$\times$ less GFLOPs respectively.
Compared with X3D (Feichtenhofer, 2020), our DSA achieves better performance
(77.7% _vs._ 77.5%) while using less inference cost (503G _vs._ 744G).
Remarkably, equipped with the DSA module, the top-1 accuracy of Slowonly
(namely I3D R50 above) improves from 74.9% to 78.2% on Kinetics-400. Then
compared with video-level method, our DSA with 4$\times$8 frames as input
outperform V4D with 8$\times$10 frames (78.2% _vs._ 77.4%) when using much
less GFLOPs (4.3$\times$). Furthermore, our DSA has better results with longer
inputs, and exceed V4D by 0.8%. For readers’ reference, here we also report
the results of ensemble the two models. As shown in Figure 3, comparison with
prior works (SlowFast (Feichtenhofer et al., 2019), X3D (Feichtenhofer, 2020),
TSM (Lin et al., 2019), TEA (Li et al., 2020), MVFNet (Wu et al., 2021), V4D
(Zhang et al., 2020)), our DSANet achieves state-of-the-art performance on
Kinetics-400 and get better accuracy-computation trade-off than other methods.

Figure 3. Accuracy-computation trade-off on Kinetics-400 for different methods in the inference phase. The horizontal axis shows the inference cost per video. Table 5. Comparison with state-of-the-art on ActivityNet v1.3. The results using Multi-scale Temporal Window Integration (Wang et al., 2016) (abbreviated as M-TWI) with only RGB frames. Model | Backbone | mAP
---|---|---
TSN (Wang et al., 2016) | BN-Inception | 79.7%
TSN (Wang et al., 2016) | Inception V3 | 83.3%
TSN-Top3 (Wang et al., 2016) | Inception V3 | 84.5%
V4D+I3D (Zhang et al., 2020) | 3D ResNet50 | 88.9%
DSA+I3D (Ours) | 3D ResNet50 | 90.5%

Results on ActivityNet-v1.3. To verify the generalization ability of our
DSANet, we further evaluate the performance of our method on a large-scale
untrimmed video recognition benchmark, ActivityNet (Caba Heilbron et al.,
2015). We finetuned the model pre-trained on Kinetics-400 on the
Activitynet-v1.3 dataset and report the mean average precision (mAP) follow
the official evaluation metrics. As shown in Table 5, our method outperforms
the previous methods (_i.e.,_ V4D, TSN-Top3, TSN) with a clear margin (90.5%
vs. 88.9%, 84.5%, 83.3%).

Table 6. Comparison with the state-of-the-art models on Something-Something V1 dataset. Method | Backbone | Top-1
---|---|---
MultiScale TRN (Zhou et al., 2018) | BN-Inception | 34.4%
ECO (Zolfaghari et al., 2018) | BN-Inception+3D ResNet 18 | 46.4%
S3D-G (Xie et al., 2018) | S3D Inception | 45.8%
Nonlocal+GCN (Wang and Gupta, 2018) | 3D ResNet50 | 46.1%
TSM (Lin et al., 2019) | ResNet50 | 47.2%
I3D (our impl.) | 3D ResNet50 | 48.7%
V4D+I3D (Zhang et al., 2020) | 3D ResNet50 | 50.4%
DSA+I3D (Ours) | 3D ResNet50 | 51.8%
Figure 4. Visualization of predictions with DSANet and I3D. (Green: correct
predictions, Red: wrong predictions.)

Results on Something-Something V1. We also evaluate our method on Something-
Something V1 dataset which focuses on modeling temporal relation and is more
complicated than Kinetics dataset. As shown in Table 6, comparing with prior
works, our DSA obtains comparable or better performance. To verify the
effectiveness of DSA, we report the baseline result of I3D ResNet-50 in our
implementation for fair comparison. Remarkably, equipped with the DSA module,
the top-1 accuracy of I3D ResNet-50 improves from 48.7% to 51.8% on Something-
Something V1. In this way, we believe our DSA can achieve higher accuracy when
using better snippet-level counterpart.

##  5\. Visualization

We also visualize some examples of predictions generated by DSANet and I3D
model on the validation data of Kinetics-400 in Figure 4. Compared with I3D
which simply averaging the prediction scores of all snippets to yield a video-
level prediction, we see that our DSANet can learn global context information
and adaptively capture relation-ship among snippets via dynamic aggregation,
then provide more accurate results with contextual perspective.

##  6\. Conclusion

In this paper, we presented the Dynamic Segment Aggregation (DSA) module to
better exploit long-range spatiotemporal representation in videos. The DSA
module can be easily applied to the existing snippet-level methods in a plug-
and-play manner with a negligible computational cost, bringing consistent
improvements on both 2D/3D CNN backbone. A series of empirical studies verify
that our DSA network is an accurate and efficient model for large scale video
classification. The experimental results show that our method has achieved
state-of-the-art performance on four public benchmarks with a high inference
efficiency.

## References

  * (1)
  * Arandjelovic et al. (2016) Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. 2016.  NetVLAD: CNN architecture for weakly supervised place recognition. In _Proc. CVPR_.
  * Caba Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  Activitynet: A large-scale video benchmark for human activity understanding. In _Proc. CVPR_.
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo vadis, action recognition? a new model and the kinetics dataset. In _Proc. CVPR_.
  * Chen et al. (2020) Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. 2020\.  Dynamic convolution: Attention over convolution kernels. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11030–11039.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _Proc. CVPR_.
  * Donahue et al. (2015) Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015\.  Long-term recurrent convolutional networks for visual recognition and description. In _Proc. CVPR_.
  * Feichtenhofer (2020) Christoph Feichtenhofer. 2020\.  X3D: Expanding Architectures for Efficient Video Recognition. In _Proc. CVPR_. 203–213.
  * Feichtenhofer et al. (2019) Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019.  Slowfast networks for video recognition.  _Proc. ICCV_ (2019).
  * Girdhar et al. (2017) Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell. 2017.  ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification. In _Proc. CVPR_.
  * Goyal et al. (2017) Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. 2017\.  The” Something Something” Video Database for Learning and Evaluating Visual Common Sense.. In _Proc. ICCV_.
  * He et al. (2019) Dongliang He, Zhichao Zhou, Chuang Gan, Fu Li, Xiao Liu, Yandong Li, Limin Wang, and Shilei Wen. 2019\.  Stnet: Local and global spatial-temporal modeling for action recognition. In _AAAI_.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7132–7141.
  * Jiang et al. (2019) Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. 2019.  Stm: Spatiotemporal and motion encoding for action recognition. In _Proc. ICCV_. 2000–2009.
  * Kay et al. (2017) Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017\.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ (2017).
  * Korbar et al. (2019) Bruno Korbar, Du Tran, and Lorenzo Torresani. 2019.  Scsampler: Sampling salient clips from video for efficient action recognition. In _Proc. ICCV_. 6232–6242.
  * Li et al. (2020) Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. 2020\.  TEA: Temporal Excitation and Aggregation for Action Recognition. In _Proc. CVPR_. 909–918.
  * Lin et al. (2019) Ji Lin, Chuang Gan, and Song Han. 2019.  TSM: Temporal Shift Module for Efficient Video Understanding. In _Proc. ICCV_.
  * Liu et al. (2018) Kun Liu, Wu Liu, Chuang Gan, Mingkui Tan, and Huadong Ma. 2018.  T-C3D: Temporal convolutional 3D network for real-time action recognition. In _AAAI_ , Vol. 32.
  * Liu et al. (2020a) Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu. 2020a.  TEINet: Towards an Efficient Architecture for Video Recognition.. In _Proc. AAAI_. 11669–11676.
  * Liu et al. (2020b) Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. 2020b.  TAM: Temporal Adaptive Module for Video Recognition.  _arXiv preprint arXiv:2005.06803_ (2020).
  * Long et al. (2018) Xiang Long, Chuang Gan, Gerard de Melo, Jiajun Wu, Xiao Liu, and Shilei Wen. 2018\.  Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification. In _Proc. CVPR_.
  * Qiu et al. (2017) Zhaofan Qiu, Ting Yao, and Tao Mei. 2017.  Learning spatio-temporal representation with pseudo-3d residual networks. In _Proc. ICCV_.
  * Shi et al. (2017) Yemin Shi, Yonghong Tian, Yaowei Wang, Wei Zeng, and Tiejun Huang. 2017.  Learning long-term dependencies for action recognition with a biologically-inspired deep network. In _Proc. ICCV_.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Two-stream convolutional networks for action recognition in videos. In _Neurips_.
  * Sudhakaran et al. (2020) Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. 2020\.  Gate-Shift Networks for Video Action Recognition. In _Proc. CVPR_. 1102–1111.
  * Tran et al. (2015) Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015.  Learning spatiotemporal features with 3d convolutional networks. In _Proc. ICCV_.
  * Tran et al. (2019) Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. 2019\.  Video classification with channel-separated convolutional networks. In _Proc. ICCV_. 5552–5561.
  * Tran et al. (2018) Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. 2018\.  A Closer Look at Spatiotemporal Convolutions for Action Recognition. In _Proc. CVPR_.
  * Wang et al. (2019) Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, and Dahua Lin. 2019\.  Carafe: Content-aware reassembly of features. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3007–3016.
  * Wang et al. (2016) Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016.  Temporal segment networks: Towards good practices for deep action recognition. In _Proc. ECCV_.
  * Wang et al. (2018) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018.  Non-Local Neural Networks. In _Proc. CVPR_.
  * Wang and Gupta (2018) Xiaolong Wang and Abhinav Gupta. 2018.  Videos as space-time region graphs. In _Proc. ECCV_.
  * Wang et al. (2021) Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. 2021\.  Adaptive Focus for Efficient Video Recognition.  _arXiv preprint arXiv:2105.03245_ (2021).
  * Wu et al. (2021) Wenhao Wu, Dongliang He, Tianwei Lin, Fu Li, Chuang Gan, and Errui Ding. 2021\.  MVFNet: Multi-View Fusion Network for Efficient Video Recognition. In _Proc. AAAI_.
  * Wu et al. (2019) Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen. 2019.  Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition. In _Proc. ICCV_.
  * Wu et al. (2020) Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, Yi Yang, and Shilei Wen. 2020\.  Dynamic Inference: A New Approach Toward Efficient Video Action Recognition. In _Proceedings of CVPR Workshops_. 676–677.
  * Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In _Proc. ECCV_.
  * Yue et al. (2018) Kaiyu Yue, Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding, and Fuxin Xu. 2018\.  Compact Generalized Non-local Network.  _Neurips_ (2018).
  * Zhang et al. (2020) Shiwen Zhang, Sheng Guo, Weilin Huang, Matthew R Scott, and Limin Wang. 2020.  V4D: 4D Convolutional Neural Networks for Video-level Representation Learning. In _ICLR_.
  * Zhou et al. (2018) Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. 2018.  Temporal relational reasoning in videos. In _Proc. ECCV_.
  * Zolfaghari et al. (2018) Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. 2018\.  ECO: Efficient Convolutional Network for Online Video Understanding. In _Proc. ECCV_.