# Gate-Shift-Fuse for Video Action Recognition

Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz  S. Sudhakaran was
with FBK, Trento, Italy with this work and is now with Samsung AI Center,
Cambridge, UK. E-mail<EMAIL_ADDRESS>S. Escalera is with Universitat
de Barcelona and Computer Vision Center, Spain. E-mail<EMAIL_ADDRESS>O.
Lanz was with FBK, Trento, Italy with this work and is now with Free
University of Bozen-Bolzano, Italy. E-mail<EMAIL_ADDRESS>submitted October 28, 2021; revised August 9, 2022, revised December 20, 2022;
revised April 7, 2023

###### Abstract

Convolutional Neural Networks are the de facto models for image recognition.
However 3D CNNs, the straight forward extension of 2D CNNs for video
recognition, have not achieved the same success on standard action recognition
benchmarks. One of the main reasons for this reduced performance of 3D CNNs is
the increased computational complexity requiring large scale annotated
datasets to train them in scale. 3D kernel factorization approaches have been
proposed to reduce the complexity of 3D CNNs. Existing kernel factorization
approaches follow hand-designed and hard-wired techniques. In this paper we
propose Gate-Shift-Fuse (GSF), a novel spatio-temporal feature extraction
module which controls interactions in spatio-temporal decomposition and learns
to adaptively route features through time and combine them in a data dependent
manner. GSF leverages grouped spatial gating to decompose input tensor and
channel weighting to fuse the decomposed tensors. GSF can be inserted into
existing 2D CNNs to convert them into an efficient and high performing spatio-
temporal feature extractor, with negligible parameter and compute overhead. We
perform an extensive analysis of GSF using two popular 2D CNN families and
achieve state-of-the-art or competitive performance on five standard action
recognition benchmarks.

###### Index Terms:

Action Recognition, Video Classification, Spatial Gating, Channel Fusion

##  1 Introduction

Understanding human actions in videos is one of the challenging and
fundamental problems in computer vision with a wide range of applications from
video surveillance to robotics and human-computer interaction. Recently, video
action recognition research has witnessed great progress owing to the adoption
of Deep Neural Networks. However, such Deep Neural Models are yet to achieve
the success of their image recognition counterparts. A key challenge lies in
addressing the question of how to better encode the space-time features.

Inspired from the improved performance of 2D Convolutional Neural Networks on
image recognition tasks, [25] explores several approaches to extend 2D CNNs
for action recognition by considering videos as a set of frames. This includes
using a single frame from the video to stacking multiple frames and applying
it to a 2D CNN. However it is found that both of these configurations perform
similarly, indicating that the learned video representation is not capable of
capturing motion information in the video. Two-Stream networks [51], [16]
address this problem by stacking optical flow images to capture short-term
motion features and incorporate them with the appearance features obtained
from a single frame. Such approaches require costly optical flow computations
and ignore valuable appearance features from the video. TSN [62] and
ActionVLAD [18] propose to extract features from multiple frames instead,
followed by a late temporal aggregation via mean pooling for improved
appearance feature extraction. Such approaches still relied on optical flow
images for capturing motion information. The works of [7], [36], [54]
formulate late temporal aggregation as a sequence learning problem by encoding
the frame-level features using variants of Recurrent Neural Networks. All
these approaches perform temporal modelling in a shallow manner.

Figure 1: 3D kernel factorization for spatio-temporal learning in video.
Majority of previous approaches decompose into channel-wise (CSN), spatial
followed by temporal (S3D, TSM), or grouped spatial and spatio-temporal (GST).
In these, spatial, temporal, and channel-wise interaction is hard-wired. Our
Gate-Shift-Fuse (GSF) leverages group spatial gating and fusion (blocks in
green) to control interactions in spatial-temporal decomposition. GSF is
lightweight and a building block of high performing video feature extractors.

Fine-grained recognition can benefit from deeper temporal modeling. Full-3D
CNNs (C3D [58, 21]) process the video in space-time by expanding the kernels
of a 2D ConvNet along the temporal dimension. Deep C3Ds are designed to learn
powerful representations in the joint spatio-temporal feature space with more
parameters (3D kernels) and computations (kernels slide over 3 densely sampled
dimensions). In practice however, they may under-perform due to the lack of
sufficiently large datasets for training them at scale. To cope with these
issues arising from the curse of dimension one can narrow down network
capacity by design. Fig. 1 shows several C3D kernel decomposition approaches
proposed for spatio-temporal feature learning in video. A most intuitive
approach is to factorize 3D spatio-temporal kernels into 2D spatial plus 1D
temporal, resulting in a structural decomposition that disentangles spatial
from temporal interactions (P3D [46], R(2+1)D [60], S3D [72]). An alternative
design is separating channel interactions and spatio-temporal interactions via
group convolution (CSN [59]), or modeling both spatial and spatio-temporal
interactions in parallel with 2D and 3D convolution on separated channel
groups (GST [38]). Temporal convolution can be constrained to hard-coded time-
shifts that move some of the channels forward in time or backward (TSM [37]).
These approaches learn structured kernels with a hard-wired connectivity and
propagation pattern across the network. There is no data dependent decision
taken at any point in the network to route features selectively through
different branches, for example, group-and-shuffle patterns are fixed by
design and learning how to shuffle is combinatorial complexity. Recently, [9]
introduced RubiksNet, a special case of TSM, that replaces spatial convolution
and temporal shifting with learnable shifting operations to achieve a flexible
spatio-temporal decomposition.

In this paper we introduce spatial gating and channel fusion in spatial-
temporal decomposition of 3D kernels. We implement this concept with Gate-
Shift-Fuse (GSF) as shown in Fig. 1. GSF is lightweight and turns a 2D-CNN
into a highly efficient spatio-temporal feature extractor. GSF first applies
2D convolution, then decomposes the output tensor using a learnable spatial
gating into two tensors: a gated version of it, and its residual. The gated
tensor goes through a 1D temporal convolution while its residual is skip-
connected to its output. We implement spatial gating as group spatio-temporal
convolution with single output plane per group. We use hard-coded time-shift
of channel groups instead of learnable temporal convolution. Channel fusion is
implemented by generating channel weights via a light weight 2D convolution.
The two feature groups, gate-shifted and residual, are then fused by weighted
averaging across the channels. With GSF plugged in, a 2D-CNN learns to
adaptively route features through time and combine them, at almost no
additional parameters and computational overhead. For example, when GSF is
plugged into TSN [62], an absolute gain of $+32$ percentage points in accuracy
is obtained on Something Something-V1 dataset with just $0.48\%$ additional
parameters and $0.58\%$ additional Floating Point Operations.

The contributions of this paper can be summarized as follows:

  * •

We propose a novel spatio-temporal feature extraction module that can be
plugged into existing 2D CNN architectures with negligible overhead in terms
of computations and memory;

  * •

We perform an extensive ablation analysis of the proposed module to study its
effectiveness in video action recognition;

  * •

We perform an extensive analysis to study how to effectively integrate the
proposed module with two popular 2D CNN architecture families and achieve
state-of-the-art or competitive results on five public benchmarks with less
parameters and FLOPs compared to existing approaches.

This paper extends our earlier work [55] published in CVPR 2020 in many
aspects. In particular, we introduce a weighted fusion of the two feature
groups, gate-shifted and its residual, instead of the naive average fusion
presented in [55]. Secondly, we present an extensive analysis to identify the
best design choice for applying our light weight module to ResNet based
backbone architectures. Thirdly, we evaluate the performance of our module on
two additional datasets: Kinetics 400 and Something Something-v2 and replace
two datasets (EPIC-Kitchens-55 and Diving48) used in [55] with their updated
versions. We also extend the related works section by reviewing more recent
works on video action recognition and the experimental section by performing
an extensive comparison of the performance of our work with more recently
developed approaches.

The rest of the paper is organized as follows. Sec. 2 reviews recent works on
action recognition. Sec. 3 presents the technical details of Gate-Shift-Fuse
(GSF). Experimental results are reported in Sec. 4 and Sec. 5 concludes the
paper.

##  2 Related Work

The development of large-scale video datasets such as Kinetics [26] [52],
Moments in Time [43], Something Something [19], EPIC-Kitchens-100 [5], _etc_.,
to mention a few, inspired video action recognition research significantly.
Such datasets with varying properties, such as requirement of spatial
reasoning [26], temporal reasoning [19], spatio-temporal reasoning [5],
resulted in the development of a plethora of action recognition approaches.
The development of recent large-scale datasets such as HowTo100M [42], HVU
[6], Ego4D [20], _etc_., are expected to further video understanding research
beyond simple tasks such as action recognition.

###  2.1 Action Recognition

Fusing appearance and flow. A popular extension of 2D CNNs to handle video is
the Two-Stream architecture by Simonyan and Zisserman [51]. Their method
consists of two separated CNNs (streams) that are trained to extract features
from a sampled RGB video frame paired with the surrounding stack of optical
flow images, followed by a late fusion of the prediction scores of both
streams. The image stream encodes the appearance information while the optical
flow stream encodes the motion information, that are often found to complement
each other for action recognition. Several works followed this approach to
find a suitable fusion of the streams at various depths [16] and to explore
the use of residual connections [14] or multiplicative gating functions [15]
between them. These approaches rely on optical flow images for motion
information, and a single RGB frame for appearance information, which is
limiting when reasoning about the temporal context is required for video
understanding.

Video as a set or sequence of frames. Later, other approaches were developed
using multiple RGB frames for video classification. These approaches sparsely
sample multiple frames from the video, which are applied to a 2D CNN followed
by a late integration of frame-level features using average pooling [62],
multilayer perceptrons [79], recurrent aggregation [7, 36], or attention [17,
54]. To boost performance, most of these approaches also combine video frame
sequence with externally computed optical flow. This shows to be helpful, but
computationally intensive.

Modeling short-term temporal dependencies. Other research has investigated the
middle ground between late aggregation (of frame features) and early temporal
processing (to get optical flow), by modeling short-term dependencies. This
includes differencing of intermediate features [44] and combining Sobel
filtering with feature differencing [57]. Other works [10, 45] develop a
differentiable network that performs TV-L1 [74], a popular optical flow
extraction technique. The work of [30] instead uses a set of fixed filters for
extracting motion features, thereby greatly reducing the number of parameters.
DMC-Nets [50] leverage motion vectors in the compressed video to synthesize
discriminative motion cues for two-stream action recognition at low
computational cost compared to raw flow extraction.

Video as a space-time volume. Unconstrained modeling and learning of action
features is possible when considering video in space-time. Since video can be
seen as a temporally dense sampled sequence of images, expanding 2D
convolution operation in 2D-CNNs to 3D convolution is a most intuitive
approach to spatio-temporal feature learning [58, 21, 3]. The major drawback
of 3D CNNs is the huge number of parameters involved. This results in
increased computations and the requirement of large scale datasets for pre-
training. Carreira and Zisserman [3] addressed this limitation by inflating
video 3D kernels with the 2D weights of a CNN trained for image recognition.
Several other approaches focused on reducing the number of parameters by
disentangling the spatial and temporal feature extraction operations. P3D [46]
proposes three different choices for separating the spatial and temporal
convolutions and develops a 3D-ResNet architecture whose residual units are a
sequence of such three modules. R(2+1)D [60] and S3D-G [72] also show that a
2D convolution followed by 1D convolution is enough to learn discriminative
features for action recognition. CoST [31] performs 2D convolutions, with
shared parameters, along the three orthogonal dimensions of a video sequence.
CT-Net [32] proposes to tensorize the channel dimension of features as a
multiplication of K sub-dimensions, followed by 3D separable convolutions
along each of the K channel sub-dimensions. This allows the model to gradually
increase the spatio-temporal receptive field with reduced computational
complexity while maintaining interaction of features across both spatio-
temporal and channel dimensions. MultiFiber [4] uses multiple lightweight
networks, the fibers, and multiplexer modules that facilitate information flow
using point-wise convolutions across the fibers. CIDC [33] performs channel
independent directional convolution by applying 2D convolution independently
on individual feature channels while considering the temporal dimension as
channels. Other works involve Neural Architecture Search (NAS) for developing
an optimum architecture [49], [48], [65]. X3D [12] develops a progressive
algorithm for expanding the spatio-temporal resolution of inputs, network
depth, number of channels, _etc_., of 3D CNN architectures. The final
architecture developed with the search algorithm results in improved
performance with less computational complexity compared to hand-designed 3D
CNNs.

Modeling long-term dependencies. Majority of existing approaches rely on the
network depth to combine the information present in frames that are further
apart in time. [67] develops a feature bank that stores features from the
entire video which can then be integrated as contextual information to a video
recognition model for understanding the present. Non-local [63] is a plug and
play module that computes the features at a specific position in space-time as
the weighted sum of features at all other positions, thereby capturing long-
range dependencies. Timeception [22] introduces multi-scale separable temporal
convolutions to reason about videos of longer duration. Eidectic 3D LSTM [66]
integrates 3D convolution into RNNs and introduces a gated controlled self-
attention module to enable the memory to interact with its past values. The
former enables short-term temporal modeling while the latter allows the model
to encode long-term temporal information. VideoGraph [23] constructs a
unidirectional graph whose nodes represent the latent action concepts. A novel
graph embedding layer is then developed to learn the relationship between
these concepts. Zhang  _et al_. [76] introduces 4D convolution to perform
long-term modeling hierarchically. Inspired by the performance improvement
obtained by Transformers in Natural Language Processing, Vision Transformer
(ViT) [8] is developed for image recognition. TimeSformer [1] extends ViT by
developing different variants of self-attention schemes for spatio-temporal
feature learning. However, even the efficient variant of TimeSformer suffers
from increased computational complexity and larger parameter count.

Figure 2: C3D [58] decomposition approaches in comparison to GSF schematics.
S3D [72] decomposes C3D into spatial and temporal convolutions while TSM [37]
replaces the 1D temporal convolution with fixed channel-wise shifting. GST
[38] decomposes C3D into grouped spatial and spatio-temporal convolutions by
splitting across the channel dimension. RubiksNet [9] implements learnable
spatio-temporal shifting to approximate expensive 3D convolution operation.
GSF is inspired by GST and TSM but replaces the hard-wired channel split and
concat aggregation with learnable spatial gating and fusion blocks. Compared
to RubiksNet that needs to be trained from scratch, GSF can be plugged into
any existing 2D CNNs thereby leveraging ImageNet pre-trained weights.

Multi-scale architectures. Inspired by the effectiveness of Two-stream
architectures using RGB and optical flow images, [25] [13], [11] developed
multi-stream architectures operating on two streams of RGB images. A multi-
resolution CNN architecture is developed in [25] where low resolution frames
are processed by one stream and high resolution center crops are processed by
the second stream. [11] proposes to apply alternate frames of high resolution
through a compact network in addition to a second stream processing low
resolution images. This allows the network to encode features at multiple
spatial scales and increase the number of processed frames while maintaining
reasonable computational complexity. SlowFast [13] encodes frames using a dual
pathway network. One pathway encodes frames sampled at a slower frame rate
while the second encodes frames sampled at a faster frame rate, enabling the
network to capture information at multiple temporal scales. These approaches
rely on multiple networks to encode information at different scales. To
address this issue, TPN [73] introduces a feature level pyramid network to
encode features at multiple hierarchies thereby allowing the network to
extract features at different temporal scales. Since the pyramid structure is
constructed at feature level, TPN do not require multiple architectures,
thereby reducing the number of parameters and computational complexity. MSTI-
Net [68] proposes to split the feature tensors across the channel dimension
into four groups and apply convolution operation with different kernel size
and aggregate them hierarchically. Varying the kernel size of the convolution
operation enables the network to encode spatio-temporal features at multiple
scales.

###  2.2 Efficient Video Understanding

Recently, the focus of research is moving to the development of efficient
(from a computational point of view) and effective (from a performance point
of view) approaches for video understanding. Such approaches perform efficient
modeling by (1) reducing redundant computations or (2) developing novel
efficient architectures.

Reducing computations by selective processing. [47] and [70] develop a
conditional computing approach that decides whether to compute fine resolution
features in a data dependent manner, to avoid additional compute overhead.
AdaFrame [71] trains a policy gradient network to determine which frame to
process next, thereby skipping processing of non-informative frames in a
video. SCSampler [28] develops a light-weight model for identifying the most
salient clip for recognizing the action present in a video. AdaFuse [41]
learns a decision policy to reduce temporal redundancy by removing feature
channels that are uninformative and reuse the feature channels from past
frames for efficient inference. A reinforcement policy based learning approach
for selecting the optimum input frame resolution is developed in [40]. [2]
follows the approach of knowledge distillation to train a student network that
processes a smaller number of frames using a teacher network trained with a
larger number of frames. All these approaches are based on existing spatio-
temporal feature extractors or architectures that require large memory
requirements due to the model size. Even though these approaches reduce the
computations in inference time, majority of them suffer from heavy
computations during the training stage.

Efficient architectures. CNNs provide different levels of feature abstractions
at different layers of the hierarchy. It has been found that the bottom layer
features are less useful for extracting discriminative motion cues [56, 82,
72]. In [56] it is proposed to apply 1D convolution layers on top of a 2D CNN
for video action recognition. The works of [82] and [72] show that it is more
effective to apply full 3D and separable 3D convolutions at the top layers of
a 2D CNN for extracting spatio-temporal features. These approaches resulted in
performance improvement over full 3D architectures with less parameters and
computations. Static features from individual frames represent scenes and
objects and can also provide important cues in identifying the action. This is
validated by the improved performance obtained with two-path structures that
apply a parallel 2D convolution in addition to the 3D convolution [80, 38].
MiCT [80] is designed by adding 3D convolution branches in parallel to the 2D
convolution branches of a BN-Inception-like CNN. GST [38] makes use of the
idea of grouped convolutions for developing an efficient architecture for
action recognition. They separate the features at a hierarchy across the
channel dimension and separately perform 2D and 3D convolutions followed by a
concatenation operation. In this way, the performance is increased while
reducing the number of parameters. MVFNet [69] extends the idea of GST by
replacing the 3D convolution with three 1D convolutions applied to each of the
two spatial dimensions and temporal dimension. STM [24] proposes two parallel
blocks for extracting motion features and spatio-temporal features. Their
network rely only on 2D and 1D convolutions and feature differencing for
encoding motion and spatio-temporal features. TSM [37] proposes to shift the
features across the channel dimension as a way to perform temporal interaction
between the features from adjacent frames of a video. This parameter-less
approach has resulted in similar performance to 3D CNNs. RubiksNet [9]
replaces the convolution operation in 3D CNNs with learnable spatio-temporal
shifting. TEA [34] develops a novel ResNet based architecture that weights
channels with motion information while suppressing those without relevant
information and that can encode long range temporal information in a
hierarchical aggregation fashion. MSNet [29] develops a plug in module that
can estimate the correspondences between adjacent frame level features and
convert these into effective motion features that can be injected into the
backbone CNN for efficient spatio-temporal modeling.

In all previous approaches, spatial, temporal, and channel-wise interaction is
hard-wired. Here, we propose the Gate-Shift-Fuse (GSF), which control
interactions in spatial-temporal decomposition and learns to adaptively route
features through time and combine them in a data dependent manner, at almost
no additional parameters and computational overhead.

##  3 Gate-Shift-Fuse

We present Gate-Shift-Fuse (GSF), a module capable of converting a 2D CNN into
a high performing spatio-temporal feature extractor with minimal overhead.

###  3.1 Gate-Shift-Fuse Module

shift_fwshift_bw$3\times 3$$sigmoid$poolconcat$tanh$$3\times 3\times
3$C2D$X_{1}$$R_{1}$$Y_{1}$$R_{1}$$Z_{1}$$X$$Y_{1}$$Y_{2}$$Y_{1_{shift}}$$F_{1}$$1$-$F_{1}$[$R_{1}$;
$R_{2}$][$Y_{1_{shift}}$; $Y_{2_{shift}}$][$Z_{1}$; $Z_{2}$]

Figure 3: GSF implementation with group gating, forward-backward temporal
shift, and fusion. A gate is a single 3D convolution kernel with tanh
calibration while fusion consists of a single 2D convolution kernel with
sigmoid calibration. Thus very few parameters are added when GSF is used to
turn a C2D base model into a spatio-temporal feature extractor.

Several 3D kernel factorization approaches have been developed for efficient
video recognition. Fig. 2 illustrates the network schematics of some of these
approaches. S3D, or R(2+1)D, P3D, decompose 3D convolutions into 2D spatial
plus 1D temporal convolutions. TSM replaces 1D temporal convolution with
parameter-free and zero FLOP channel-wise temporal shift operations. GST uses
group convolution where one group applies 2D spatial and the other 3D spatio-
temporal convolution. GST furthermore applies point-wise convolution before
and after the block to allow for interactions between spatial and spatio-
temporal groups, and for channel reduction and up-sampling. In these modules,
the feature flow is hard-wired by design, meaning that features are forwarded
from one block to the next without data-dependent pooling, gating or routing
decision.

GSF design, in Fig. 2, is inspired by GST and TSM but replaces the hard-wired
channel split and concat aggregation with a learnable spatial gating block
followed by a learnable fusion block to fuse the two splits of input features.
The function of gate block is to selectively route the features through time-
shifts to encode temporal information. The fuse block then merge the shifted
(temporal) features with the residual (spatial) features. The gate and fuse
blocks are learnable and perform gating and fusion operations in a data
dependent manner.

Based on the conceptual design in Fig. 2, we instantiate GSF as in Fig. 3. We
first describe the equations governing the implementation of GSF. Let $X$ be
the $C\times T\times W\times H$ shaped input tensor to GSF, where $C$ is the
number of channels and $WH$, $T$ are the spatial and temporal dimensions,
respectively. Let $X=[X_{1},X_{2}]$ be the group=2 split of $X$ along the
channel dimension, $W_{g}=[W_{g_{1}},W_{g_{2}}]$ be two
$1\times\nicefrac{{C}}{{2}}\times 3\times 3\times 3$ shaped gating kernels and
$W_{f}=[W_{f_{1}}$, $W_{f_{2}}]$ be two fusion kernels of shape $1\times
2\times 3\times 3$. Then, the GSF output $Z=[Z_{1},Z_{2}]$ is computed as

$\displaystyle Y_{1}$ $\displaystyle=$ $\displaystyle
tanh(W_{g_{1}}*X_{1})\odot X_{1}$ (1) $\displaystyle Y_{2}$ $\displaystyle=$
$\displaystyle tanh(W_{g_{2}}*X_{2})\odot X_{2}$ (2) $\displaystyle R_{1}$
$\displaystyle=$ $\displaystyle X_{1}-Y_{1}$ (3) $\displaystyle R_{2}$
$\displaystyle=$ $\displaystyle X_{2}-Y_{2}$ (4) $\displaystyle Y_{1_{shift}}$
$\displaystyle=$ $\displaystyle\verb+shift_fw+(Y_{1})$ (5) $\displaystyle
Y_{2_{shift}}$ $\displaystyle=$ $\displaystyle\verb+shift_bw+(Y_{2})$ (6)
$\displaystyle F_{1}$ $\displaystyle=$
$\displaystyle\sigma(W_{f_{1}}*[\phi(Y_{1_{shift}};R_{1})])$ (7)
$\displaystyle F_{2}$ $\displaystyle=$
$\displaystyle\sigma(W_{f_{2}}*[\phi(Y_{2_{shift}};R_{2})])$ (8)
$\displaystyle Z_{1}$ $\displaystyle=$ $\displaystyle F_{1}\odot
Y_{1_{shift}}+(1-F_{1})\odot R_{1}$ (9) $\displaystyle Z_{2}$ $\displaystyle=$
$\displaystyle F_{2}\odot Y_{2_{shift}}+(1-F_{2})\odot R_{2}$ (10)

where ‘$*$’ represents convolution, ‘$\odot$’ is Hadamard product, shift_fw,
shift_bw is forward, backward temporal shift, $\sigma$ is sigmoid function, [.
; .] is concatenation operation and $\phi$ is spatial average pooling.

The output features $X$ obtained from the spatial convolution, inherited from
the 2D CNN base model, is applied to the gating module. The gating module
splits the features into two groups across the channel dimension and generates
two gating planes of shape $1\times T\times W\times H$, each for the two
groups. The gating planes are then applied as spatial weight maps to each of
the two feature groups to generate a pair of gated features and their
residuals. The gated features are then group-shifted forward and backward in
time to inject temporal information. The group-shifted features and their
corresponding residuals are then applied to the fusion module. The fusion
module generates a channel weight map of shape $C\times T\times 1\times 1$ for
each of the two groups. The shifted features and their residuals are then
fused via weighted averaging using the channel weight maps. This way, GSF
selectively mixes spatial and temporal information through learnable spatial
gating and fusion mechanisms.

Gating. Gating is implemented with a single spatio-temporal 3D kernel and tanh
activation. With a 3D kernel we utilize short-range spatio-temporal
information in the gating. $tanh$ provides spatial gating planes with values
in the range $(-1,+1)$ and is motivated as follows. When the gating value at a
feature location is 0 and that of the time-shifted feature was +1, then a
weighted temporal feature averaging is performed at that location. If the
gating value of the time-shifted feature was -1 instead, then a weighted
temporal feature differencing is performed. Using $tanh$, the gating can thus
learn to apply either of the two modes, location-wise.

Fusion. The simplest choice for fusing the time-shifted temporal features with
the residual is summation, as done in our preliminary work Gate-Shift Module
(GSM) [55]. However, since different channels of a feature tensor encode
different level of information, fusion by summation may be sub-optimal.
Instead GSF performs weighted averaging on the two feature groups. For this,
the group-shifted features and their corresponding residuals are concatenated
followed by spatial average pooling operation, resulting in a tensor of shape
$2\times C\times T$. Then a 2D convolution is applied on this tensor followed
by $sigmoid$ non-linearity to obtain a weight map with shape $1\times C\times
T$. This weight map is then reshaped to $C\times T\times 1\times 1$ and is
used to fuse the time-shifted features and their residuals via weighted
averaging. When the fusion weight at a feature channel is 0, only the residual
features with the spatial information is retained for further processing. On
the other hand, if the channel weight is 1, the network will discard the
residual spatial features and propagate the time-shifted temporal features.
With other weight values, the model mixes the two feature groups selectively
by mixing the channels.

Parameter count, complexity. The parameter count in the gating is
$2\times(27\cdot\nicefrac{{C}}{{2}})=27\cdot C$ and that of the fusion module
is $2\times(2\cdot 3\cdot 3)=36$. The total parameter overhead introduced by
GSF is $27\cdot C+36\approx 27\cdot C$ since $C\gg 36$ in practice. On the
other hand, a typical C3D block such as the $3\times 1\times 1$ used in S3D
[72] (Fig. 2) has a parameter count of $C_{out}\cdot C\cdot 3$ where
$C_{out}\geq C$. Thus the parameter overhead introduced by GSF is far less
than that of a C3D block since $27\ll 3\cdot C_{out}$.

The complexity of gating module is $H\cdot W\cdot T\cdot 27\cdot C$ while that
of the fusion module is $C\cdot T\cdot 36$. The total complexity of GSF is
$H\cdot W\cdot T\cdot 27\cdot C+C\cdot T\cdot 18\approx H\cdot W\cdot T\cdot
27\cdot C$. For a C3D block performing 1D temporal convolution, the complexity
is $H\cdot W\cdot T\cdot 3\cdot C\cdot C_{out}$. Since $27\ll 3\cdot C_{out}$,
one can see that the complexity introduced by GSF is far less than that of a
1D temporal convolution layer. Thus GSF is far more efficient in terms of
parameters and computations compared to the temporal convolution used in
efficient implementations such as S3D or R(2+1)D.

###  3.2 Gate-Shift-Fuse Networks

GSF can be plugged into any 2D CNN architecture to inject temporal information
into the features extracted by the backbone CNN. In this work, we choose two
widely adopted 2D CNN families, Inception and ResNet, as the backbone CNNs. As
mentioned in Sec. 3.1, GSF can be inserted into any layer in the backbone CNN
after a 2D convolution. For the Inception family, GSF is applied to the least
parametrized branch, _i.e_., the branch with the least number of convolutions
(cf. Fig.4). For ResNet based CNN, GSF is applied after the second convolution
layer in the bottle neck layer of the backbone (cf. Fig. 5). Sec. 4.3 presents
the experiments conducted to determine the optimum configuration for applying
GSF to the two CNN families considered in this work.

We train the backbone CNN with GSF using the TSN framework. A fixed number of
frames are uniformly sampled from the video and applied to the network. The
network outputs a class prediction for each of the frames. The frame level
predictions are then averaged to obtain the video level prediction. Training
is done using cross-entropy loss computed from the video level prediction and
the ground truth video label.

##  4 Experiments and Results

This section presents an extensive set of experiments to evaluate GSF.

Figure 4: BN-Inception blocks with GSF. Kernel size and stride of conv and
pool layers are annotated inside each block.

###  4.1 Datasets

We evaluate Gate-Shift-Fuse (GSF) on five standard action recognition
benchmarks, Something Something [19] (SS-V1 and SS-V2), Kinetics 400 [26],
Diving48 [35] and EPIC-Kitchens-100 [5]. Something-V1 and Something-V2
consists of fine-grained object manipulation actions comprising of 174
categories. Something-V1 consists of 100K videos while Something-V2 is an
updated version of Something-V1 with 220K videos. Kinetics 400 is a large
scale dataset consisting of 400 action categories with at least 400 clips per
action class. Diving48 dataset contains around 18K videos with 48 fine-grained
diving categories. EPIC-Kitchens-100 is the largest egocentric action
recognition dataset with 90K video clips. Each clip in the dataset is labelled
with a verb and noun class from a possible set of 97 verb classes and 300 noun
classes. The verb and noun category pairs are combined to form an action
class.

All the datasets considered for evaluating GSF differ considerably in the type
of actions present in the videos. Something Something datasets consist of
actions that are object agnostic. Thus understanding the actions carried out
in the videos require strong temporal reasoning. On the other hand, the videos
in EPIC-Kitchens-100 consists of actions that are object specific and requires
encoding of the spatio-temporal patterns in the videos for recognizing an
action. Both Kinetics 400 and Diving48 datasets contain human actions captured
from a third person perspective and require understanding of the temporal
dynamics of the human body in the video. While the background of the videos in
Kinetics 400 can reveal useful information regarding the action category, the
videos in Diving48 are captured with a uniform background thereby requiring
encoding of the temporal progression of the actions.

###  4.2 Implementation Details

Training. We train the models end-to-end using Stochastic Gradient Descent
(SGD) with momentum 0.9, weight decay $10^{-4}$ and an initial learning rate
of 0.01, for 60 epochs. We use a cosine learning rate schedule with warmup (10
epochs) for adjusting the learning rate. The batch size used is 32 and a
dropout of 0.5 is applied to avoid overfitting. 8 or 16 frames sampled from
each video is used for predicting the action present in the video. Random
scaling, cropping and flipping along with temporal jittering as presented in
[62] are used as data augmentations during training. The backbone CNNs are
pretrained on ImageNet. Unless otherwise specified, we use the same setting
for training GSFs on all the datasets used in this work.

Testing. Similar to contemporary works, we follow two approaches for
evaluation: efficiency protocol and accuracy protocol. In the efficiency
protocol, a single clip consisting of 8 or 16 frames are uniformly sampled
from the video with each frame obtained by cropping the center portion of the
video frame. In the accuracy protocol, we sample two clips for all the dataset
except for Kinetics 400 from each video and extract three different crops from
each frame. Thus a total of 6 clips are sampled from the input video. For
Kinetics 400, we sample 10 clips and 3 crops from each video. The predictions
of the individual clips are averaged to obtain the final prediction for the
video.

Source code of our implementation can be found online at
https://github.com/swathikirans/GSF.

###  4.3 Model Development

The model development is performed using SS-V1 dataset. We use 8 frames
sampled from the video as input to the models and use the efficiency protocol
for evaluating the testing performance.

Branch | Accuracy (%) | Params. | FLOPs
---|---|---|---
Branch 1 | 47.92 | 10.48M | 16.51G
Branch 2 | 47.97 | 10.49M | 16.52G
Branch 3 | 48.05 | 10.5M | 16.5G
Branch 4 | 48.44 | 10.5M | 16.53G
All branches | 46.55 | 10.62M | 16.72G
TABLE I: Model design study done to determine the Inception branch that is most suitable for plugging in GSF. % of channels shifted | Accuracy (%) | Params. | FLOPs
---|---|---|---
12.5 | 45.62 | 10.46M | 16.45G
25 | 47.09 | 10.46M | 16.46G
50 | 47.65 | 10.47M | 16.48G
75 | 48.14 | 10.49M | 16.51G
100 | 48.44 | 10.5M | 16.53G
TABLE II: Model design study on GSF-BNInception to analyze recognition
accuracy vs number of channels shifted

Inception backbones. We use BNInception and InceptionV3 as the two CNN
backbones from the Inception family of models. Since the architectural designs
are similar for the two models, we choose BNInception for identifying the
optimum model configuration.

Inception models have a multi-branch architecture as shown in Fig. 4. We first
perform analysis to identify which branch is more suitable for GSF to be
plugged into. The results of this analysis are provided in Tab. I. From the
table, one can see that GSF performs the best when GSF is plugged into the
branch with the least number of convolution layers.

As mentioned in Sec. 3.1, temporal information is injected into the features
by channel shifting. One can control the amount of temporal information by
choosing the number of channels shifted. Shifting all the channels will allow
the model to encode more temporal information. However, this is achieved at
the expense of spatial information present in the features. There should be an
optimum balance between the two. In Tab. II we report the performance obtained
by varying the number of channels shifted in each layers of the BNInception
backbone. It can be seen from the table that shifting all the channels results
in the best performance.

We use the configuration found from these two experiments for constructing
GSFs with BNInception and InceptionV3 backbones.

Figure 5: ResNet bottleneck layer with GSF. The kernel size and stride of conv layers are annotated inside each block. Branch | Accuracy (%) | Params. | FLOPs
---|---|---|---
After conv1 | 46.95 | 23.97M | 33.53G
After conv2 | 47.15 | 23.97M | 33.4G
After conv3 | 47.63 | 24.3M | 34.43G
After all conv | 46.26 | 24.52M | 35.25G
TABLE III: Model design analysis done to determine where to plug in GSF on ResNet bottleneck layers. % of channels shifted | Accuracy (%) | Params. | FLOPs
---|---|---|---
12.5 | 49.17 | 23.92M | 33.23G
25 | 50.17 | 23.97M | 33.4G
50 | 50.08 | 24.08M | 33.75G
75 | 49.63 | 24.19M | 34.09G
100 | 47.63 | 24.3M | 34.43G
TABLE IV: Model design study on GSF-ResNet-50 to analyze recognition accuracy
vs number of channels shifted

ResNet backbones. The block diagram of the bottleneck layer of ResNet based
CNNs are shown in Fig. 5. Tab. III reports the performance of different
configurations obtained by applying GSF after different convolution layers
present in the block of ResNet50. Following the results from Tab. III we apply
GSF to after the second convolution of the ResNet block as shown in Fig. 5.

Similar to Inception architectures, we also conduct analysis on ResNet by
shifting different number of channels. The results are reported in Tab. IV.
Unlike with the case of Inception models, the best performance is obtained
when only 25% of the channels are shifted. This can be attributed to the fact
that shifting all the channels will affect the spatial modelling capability of
the network. In the case of Inception based models, the branches without GSF
will encode the spatial information. Hence, GSF can be applied to all the
channels in the corresponding branch to inject temporal information to the
features.

###  4.4 Ablation Analysis

In this section, we report the ablation analysis performed on the validation
set of Something-V1 dataset. In all the experiments, we apply 8 frames as
input to the network and report accuracy using the efficiency protocol.

Gating | Fusion | Model | Accuracy (%)
---|---|---|---
Learnable | Learnable | GSF | 48.44
Learnbale | Sum | GSM | 47.24
1 | Sum | TSM | 45.57
-1 | Sum | TDN | 45.89
0 | Sum | TSN | 17.25
TABLE V: Ablation study on GSF

We first ablate the various components of GSF and validate the design choices.
We choose BNInception as the backbone model due to its reduced complexity. The
results are reported in Tab. V. By removing the channel fusion, GSF converges
to GSM [55], our prior work on top of which GSF is built upon, and
underperforms by $1.2\%$. This validates the importance of the learned channel
fusion. We then fix the spatial gating to a fixed value of 1 which results in
an accuracy of $45.57\%$. This is similar to Temporal Shift Module (TSM) [37]
where temporal shifting is applied to all the channels. With a fixed gating of
-1, the networks performs differencing operation between the adjacent frames
and results in $45.89\%$. This is a special case of Temporal Difference
Network (TDN) [44] without the image subnetwork. Finally, we fix the gating to
0 which converges the model to the baseline, TSN, without any temporal
interaction between the frame level features, and drops the accuracy by
$31\%$.

Model | Accuracy (%) | Params. | FLOPs
---|---|---|---
BNInception (baseline) | 17.25 | 10.45M | 16.43G
BNInception + 1 GSF | 31.9 | 10.46M | 16.44G
BNInception + 5 GSF | 45.24 | 10.48M | 16.46G
BNInception + 10 GSF | 48.44 | 10.5M | 16.53G
TABLE VI: Recognition Accuracy by varying the number of GSF added to BNInception backbone. Model | Accuracy (%) | Params. | FLOPs
---|---|---|---
ResNet-50 (baseline) | 17.22 | 23.86M | 33.06G
ResNet-50 + 4 GSF | 45.97 | 23.92 | 33.09G
ResNet-50 + 8 GSF | 48.54 | 23.95M | 33.14G
ResNet-50 + 16 GSF | 50.17 | 23.97M | 33.4G
TABLE VII: Recognition Accuracy by varying the number of GSF added to ResNet50
backbone.

Next we compare the performance improvement obtained by adding different
number of GSF to the backbone CNN. Tab. VI shows the results of this
experiment conducted on BNInception backbone. Baseline is the standard
Temporal Segment Network (TSN) architecture, with an accuracy of $17.25\%$. We
then applied GSF at the last Inception block of the CNN. This improved the
recognition performance by $14.65\%$. By adding GSF to the last 5 layers of
the backbone, the performance improves by $27.99\%$. Thus, a consistent
improvement in performance can be observed by plugging in GSF to the backbone
layers of the CNN. With GSF added to all the Inception blocks in the CNN, an
absolute improvement of $31\%$ is obtained, over the baseline, to reach an
accuracy of $48.44\%$. This huge improvement is obtained only with an overhead
of $0.48\%$ and $0.55\%$ in terms of parameters and complexity, respectively.

We also conducted the above experiment on ResNet-50 backbone. The GSF follows
the setting mentioned in Sec. 4.3 and the results are reported in Tab. VII. By
plugging in GSF to the last four bottleneck layers of the backbone, a
significant improvement of $28\%$ in recognition accuracy is obtained. This is
achieved only with a very small overhead in number of parameters and
computations. As observed in the case of BNInception backbone, the recognition
performance increased consistently by plugging in GSF to all the bottleneck
layers of the backbone CNN.

###  4.5 State-of-the-Art Comparison

Method | Backbone | #Frames | Params (M) | GFLOPs | Accuracy (%)
---|---|---|---|---|---
SS-V1 | SS-V2
TSN [62] | BN-Inception | $16\times 1\times 1$ | 10.45 | $32.87\times 1\times 1$ | 17.44 | 32.71
MultiScale TRN [79] | BN-Inception |  $8\times$NA | NA | NA | 34.44 | 48.8
MFNet [30] | ResNet-101 | $10\times 1\times 1$ | NA | NA | 43.92 | -
S3D-G [72] | InceptionV1 | $64\times 1\times 1$ | 11.6 | $71.38\times 1\times 1$ | 48.2 | -
TrajectoryNet [78] | ResNet-18 | 16$\times$NA | NA | NA | 47.8 | -
ABM [81] | ResNet-50 | $16\times 3\times 1$ | NA | NA | 46.81 | 61.25
TSM [37] | ResNet-50 | $16\times 1\times 1$ | 24.3 | $65\times 1\times 1$ | 47.3 | 61.2
RubiksNet [9] | - | $8\times 1\times 1$ | 8.5 | $15.8\times 1\times 1$ | 46.4 | 59.0
STM [24] | ResNet-50 | $16\times 3\times 10$ | 24 | $66.5\times 3\times 10$ | 50.7 | 64.2
bLVNet-TAM [11] | bLResNet-50 | $16\times 1\times 2$ | 25 | $47\times 1\times 2$ | 48.4 | 61.7
bLVNet-TAM [11] | bLResNet-101 | $32\times 1\times 2$ | 40.2 | $128.6\times 1\times 2$ | 53.1 | 65.2
CorrNet [61] | ResNet-101 | 32$\times 1\times 10$ | NA | $224\times 1\times 10$ | 51.1 | -
TPN [73] | ResNet-50 | $8\times 10\times 1$ | NA | NA | 49.0 | 62
TEA [34] | ResNet-50 | $8\times 3\times 10$ | NA | $35\times 3\times 10$ | 51.7 | NA
TEA [34] | ResNet-50 | 16$\times$3$\times$10 | NA | 70$\times$3$\times$10 | 52.3 | -
MSNet [29] | ResNet-50 | $8\times 1\times 1$ | 24.6 | $34\times 1\times 1$ | 50.9 | 63
MSNet [29] | ResNet-50 | $16\times 1\times 1$ | 24.6 | $67\times 1\times 1$ | 52.1 | 64.7
MVFNet [69] | ResNet-50 | $16\times 3\times 2$ | - | $66\times 3\times 2$ | 52.6 | 65.2
CT-Net [32] | ResNet-50 | $16\times 3\times 2$ | - | $75\times 3\times 2$ | 53.4 | 65.9
I3D [3] from [64] | ResNet-50 | $32\times 1\times 2$ | 28 | $153\times 1\times 2$ | 41.6 | -
Non-local from [64] | ResNet-50 | $32\times 1\times 2$ | 35.3 | $168\times 1\times 2$ | 44.4 | -
GCN+Non-local [64] | ResNet-50 | $32\times 1\times 2$ | 62.2 | $303\times 1\times 2$ | 46.1 | -
GST [38] | ResNet-50 | $16\times 1\times 1$ | 21 | $59\times 1\times 1$ | 48.6 | 62.6
Martinez _et al_. [39] | ResNet-50 | NA | NA | NA | 50.1 | -
Martinez _et al_. [39] | ResNet-152 | NA | NA | NA | 53.4 | -
MoViNet-A3 [27] | - | $50\times 1\times 1$ | 5.3 | $23.7\times 1\times 1$ | - | 64.1
GSF | BNInception | $8\times 1\times 1$ | 10.5 | $16.53\times 1\times 1$ | 48.44 | 61.29
ResNet-50 | 23.97 | $33.4\times 1\times 1$ | 50.17 | 62.19
InceptionV3 | 22.22 | $26.96\times 1\times 1$ | 51.52 | 62.83
BNInception | $8\times 3\times 2$ | 10.5 | $16.53\times 3\times 2$ | 50.17 | 62.76
ResNet-50 | 23.97 | $33.4\times 3\times 2$ | 51.53 | 64.01
InceptionV3 | 22.22 | $26.85\times 3\times 2$ | 52.53 | 64.59
BNInception | $16\times 1\times 1$ | 10.5 | $33.06\times 1\times 1$ | 50.63 | 63.4
ResNet-50 | 23.97 | $66.8\times 1\times 1$ | 51.54 | 63.97
InceptionV3 | 22.22 | $53.93\times 1\times 1$ | 53.13 | 64.48
ResNet-50+MS | 24.05 | $68.05\times 1\times 1$ | 52.7 | 64.54
BNInception | $16\times 3\times 2$ | 10.5 | $33.06\times 3\times 2$ | 51.51 | 64.57
ResNet50 | 23.97 | $66.8\times 3\times 2$ | 52.33 | 65.23
InceptionV3 | 22.22 | $53.93\times 3\times 2$ | 54.09 | 65.60
ResNet-50+MS | 24.05 | $68.05\times 3\times 2$ | 54.04 | 65.73
TABLE VIII: Comparison to state-of-the-art on Something-V1 and Something-V2
datasets.

Something-Something. Tab. VIII compares the recognition performance of GSF
with state-of-the-art approaches on SS-V1 and SS-V2 datasets. For fair
comparison, we compare only with those methods that use RGB frames for feature
extraction. We also report the number of frames used by each approach along
with the number of parameters and the computational complexity in terms of
FLOPs in the table. Since different approaches use different backbone CNNs, we
also report the backbones used for fair comparison. The table is split into
two, the first part lists approaches that use 2D CNN or efficient
implementations of 3D CNN as backbone. The second part shows the methods that
rely on heavier 3D CNNs for spatio-temporal feature extraction. Our baseline
TSN [62] obtains 17.52% and 32.71% recognition accuracy on SS-V1 and SS-V2
datasets, respectively. Applying GSF to the backbone CNN, an absolute gain of
$+33.19\%$ and $+30.69\%$ is obtained on the two datasets. This huge
improvement is obtained with an increase of only +0.48% and +0.55% on the
number of parameters and FLOPs. Compared to the top performing approaches that
use 2D CNN, such as TEA [34] and MSNet [29], GSF performs competitively. On
the other hand, approaches that uses 3D CNN such as I3D [3] and Non-local
networks [63], considerably falls behind GSF. The approach from [39] uses a
deeper 3D CNN (I3D ResNet-152). When compared to the performance obtained with
a backbone CNN of equivalent depth (ResNet-50), GSF outperforms the approach
from [39] by $+2.2\%$. By plugging in the MotionSqueeze module [29] on the GSF
version of ResNet-50, our model surpasses the recently developed state-of-the-
art approaches MVF-Net [69] and CT-Net [32], with same backbone, achieving an
accuracy of $54.04\%$ on SS-V1.

Kinetics400. We compare the performance of GSF with state-of-the-art methods
on Tab.IX. All two GSF models are trained with 16 frames and the accuracy
protocol with 10 clip sampling during inference. As done previously, we split
the table into two sections: top section comparing methods that use 2D CNN
backbones and bottom section lists the approaches that use 3D CNN backbone. As
mentioned in Sec. 4.1, Kinetics400 consists of videos of short duration ( 10s)
that do not require strong temporal reasoning. This is reflected in Tab. IX,
where the approaches that use 3D CNN for feature extraction performs
significantly better than those that use 2D CNNs. Compared to approaches that
use 2D CNN, the two GSF models perform competitively with the InceptionV3
based model outperforming its ResNet-50 counterpart by $0.7\%$.

Method | Backbone | Accuracy (%)
---|---|---
bLVNet-TAM [11] | bLResNet-50 | 73.5
STM [24] | ResNet-50 | 73.7
TSM [37] | ResNet-50 | 74.1
Martnez _et al_. [39] | NA | 74.3
TEA [34] | ResNet-50 | 76.1
MFNet [30] | ResNet-50 | 76.4
MVFNet [69] | ResNet-50 | 77.0
CT-Net [32] | ResNet-50 | 77.3
I3D [30] | ResNet-101 + NL | 77.7
Martinez _et al_. [39] | NA | 78.8
ip-CSN [30] | ResNet-152 | 79.2
TPN [30] | ResNet-101 | 78.9
SlowFast [30] | ResNet-101 + NL | 79.8
X3D-XXL [30] | - | 80.4
MoViNet-A6 [27] | - | 81.5
GSF | ResNet-50 | 74.81
InceptionV3 | 75.57
TABLE IX: Comparison to state-of-the-art efficient architectures on
Kinetics400.

EPIC-Kitchens-100. Comparison of GSF with state-of-the-art approaches on the
validation set of EPIC-Kitchens-100 is performed on Tab. X. For this dataset,
we apply autoaugment [77] in addition to random cropping and random flipping
as augmentations. All models are trained with 16 frames as a multitask problem
for predicting verb, noun and action classes. Accuracy protocol is used during
inference. InceptionV3 and ResNet-50 models are initialized with Kinetics400
pretrained weights while ResNet-101 backbone is initialized with ImageNet
pretrained weights. From the table one can see that GSF models outperform the
other approaches significantly, except for MoViNet [27] which is a NAS based
approach using 3D convolutions. One interesting finding is that in contrast to
what was observed for Something and Kinetics datasets, the ResNet based
backbones result in better performance than InceptionV3 based model.

Method | Backbone | Accuracy (%)
---|---|---
Verb | Noun | Action
TSN [62]∗ | ResNet-50 | 60.18 | 46.03 | 33.19
TRN [79]∗ | ResNet-50 | 65.88 | 45.43 | 35.34
TSM [37]∗ | ResNet-50 | 67.86 | 49.01 | 38.27
SlowFast [13]∗ | ResNet-50 | 65.56 | 50.02 | 38.54
MoViNet-A6 [27] | - | 72.2 | 57.3 | 47.7
GSF | InceptionV3 | 68.35 | 52.71 | 43.42
ResNet-50 | 68.76 | 52.74 | 44.04
ResNet-101 | 69.97 | 54.01 | 44.78
TABLE X: Comparison to state-of-the-art on the validation set of EPIC-Kitchens-100. ∗: Results reported from [5] Method | Accuracy (%)
---|---
I3D [3] from [75] | 48.3
S3D [72] from [75] | 50.6
SlowFast [13] from [1] | 77.6
GST [38] from [75] | 78.9
TimeSformer-L [1] | 81
TQN [75] | 81.8
GSF | 90.51
TABLE XI: Comparison to state-of-the-art on Diving48.

Diving48. We compare the performance of Diving48 dataset on Tab. XI. We use 16
frames to train the GSF model with InceptionV3 backbone and inference is
performed following the accuracy protocol. All training settings are the same
as mentioned in Sec. 4.2 except for the batch size. For Diving48, we used a
smaller batch size of 8 during training. From the table, one can see that GSF
model outperforms approaches that uses 3D CNN backbones such as I3D [3],
SlowFast [13] and GST [38] by a significant margin.

(a)

(b)

Figure 6: t-SNE projection of output features obtained from the layer
preceeding the final linear layer for (a) TSN baseline and (b) TSN baseline
with GSF. Samples from the 10 action groups defined in [19] are visualized.
Figure 7: Action classes with the highest improvement over TSN baseline.
X-axis shows the the number of corrected samples for each class. Y-axis labels
are in the format true label (GSF)/predicted label (TSN).

###  4.6 Discussion

We perform further analysis of GSF in this section. The t-SNE plot of features
obtained from the output of the layer preceding the final Fully Connected (FC)
layer of BNInception backbone is plotted in Fig. 6(a). We visualize the 10
action groups defined in [19] in the plot. From the figure, one can see better
separation of the features when GSF is plugged into the backbone CNN. This
allows the model to correctly recognize the action present in the video and
results in improved recognition performance compared to the baseline TSN.

We further analyse the gating weights generated by GSF on BNInception backbone
CNN across its various layers. We plot a histogram of the weights averaged
across all the videos on Fig. 8. From the figure, one can see that the gating
weights are negative for the bottom layers while it becomes positive at the
top layers. The negative weights result in feature differencing while the
positive weights result in feature averaging, as explained in Sec. 3.1 of the
manuscript. Image difference captures the salient motion occurring between the
image pair. Since image features obtained from a CNN are invariant to
translation and small changes to appearance, image feature difference allows
in encoding of strong temporal information. Once the temporal information is
encoded via feature differencing in the bottom layers, the top layers perform
feature aggregation via feature averaging. Instead of simply shifting the
features as done in TSM, the gating module allows this flexibility of
performing feature differencing and averaging to the model, resulting in an
improved recognition performance

(a) Layer 1

(b) Layer 2

(c) Layer 3

(d) Layer 4

(e) Layer 5

(f) Layer 6

(g) Layer 7

(h) Layer 8

(i) Layer 9

(j) Layer 10

Figure 8: Histogram of the gating weights generated by the gating module on
videos from SS-v1 dataset.

The action categories that benefited the most by plugging in GSF to the
backbone CNN are reported in Fig. 7. The Y-axis labels show the true labels
(predicted by GSF) and the labels predicted by TSN, separated by “/”. From the
plot, it can be seen that the baseline TSN confuses actions that are of
similar nature but following a different (reversed) order, such as unfolding
something and folding something, moving something and something closer to each
other and moving something and something away from each other, _etc_.With GSF
modules in the backbone CNN, the model is capable of encoding the temporal
ordering of frames to correctly recognize the action. This can further be seen
from the visualization of the saliency tubes [53] for the samples from the
most improved classes as shown in Fig. 9. From the figure, one can see that
GSF predicts the action category by focusing on the relevant frames where the
corresponding action is taking place as opposed to the baseline TSN. In Figs.
9(c) and 9(d), even though the baseline model is focusing on the relevant
spatio-temporal features, the lack of temporal reasoning renders the model to
make wrong predictions.

folding something

(a)

unfolding something

(b)

plugging something into something but pulling it right out as you remove your
hand

(c)

plugging something into something

(d)

Figure 9: Saliency tubes generated by TSN (left) and GSF (right) on sample
videos taken from the validation set of Something Something-V1 dataset. Action
labels are shown as text on columns. Figure 10: Accuracy (%) vs complexity
(GFLOPs) of state-of-the-art approaches on Something-V1 dataset.

We analyse the computational complexity and memory requirement for GSFs and
state-of-the-art models in Fig. 10. X-axis shows the computational complexity
in GFLOPs and Y-axis shows the recognition accuracy. The size of the marker
indicate the number of parameters present in each model. It can be seen from
the plot that GSF outperforms existing approaches in terms of recognition
accuracy while maintaining a reasonable compute and memory requirements.Tab.
XII compares the inference latency of GSF against TSM and GST. We report the
time taken for a forward pass with a single input video consisting of 8
frames. We measure the latency on a single NVIDIA Titan V GPU. Compared to the
other two architectures, GSF is slightly slow with a latency of 12.4ms
compared to 7.59ms of TSM , 8.28ms of GST and 10.46ms of RubiksNet. However,
this latency of 12.4ms is equivalent to $\approx$80 videos per second, which
is a reasonable speed for real time applications. On the other hand, the
improvement in accuracy obtained by GSF on SS-V1 is large compared to the
other three architectures (+4.57% over TSM , +3.17% over GST and +3.77% over
RubiksNet). Note that all models except RubiksNet are run with half precision.
We report additional runtime analyses in the supplementary document.

Model | Latency (ms) | Accuracy on SS-V1 (%)
---|---|---
TSM | 7.59 | 45.6
GST | 8.28 | 47.0
RubiksNet∗ | 10.46 | 46.4
GSF | 12.40 | 50.17
TABLE XII: Comparison of latency incurred by GSF with respect to other
efficient approaches. We report the time taken for one forward pass of a video
clip consisting of 8 frames. ∗: All models except RubiksNet are run with half
precision.

##  5 Conclusion

We presented Gate-Shift-Fuse (GSF), a novel spatio-temporal feature extraction
module capable of converting a 2D CNN into a high performing spatio-temporal
feature extractor with negligible computational overhead. GSF relies on
spatial gating and channel fusion to exchange information between neighbouring
frames in a data dependent manner. We performed an extensive analysis to study
the effectiveness of GSF using two CNN model families on five standard action
recognition benchmarks. Compared to other approaches, GSF achieves state-of-
the-art or competitive performance on these datasets with far less model
complexity, indicating its effectiveness as an efficient spatio-temporal
feature extractor.

## Acknowledgements

This work has been partially supported by the Spanish projects
PID2019-105093GB-I00, TED2021-131317B-I00 and PDC2022-133305-I00. We
gratefully acknowledge the support from Amazon AWS Machine Learning Research
Awards (MLRA), ICREA under the ICREA Academia programme, start-up grant IN2814
of Free University of Bozen-Bolzano and NVIDIA AI Technology Centre (NVAITC),
EMEA. We acknowledge the CINECA award under the ISCRA initiative, for the
availability of high performance computing resources and support.

## References

  * [1] G. Bertasius, H. Wang, and L. Torresani.  Is Space-Time Attention All You Need for Video Understanding?  In Proc. Int. Conf. Mach. Learn., pages 813–824, July 2021.
  * [2] S. Bhardwaj, M. Srinivasan, and M. M. Khapra.  Efficient video classification using fewer frames.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 354–363, 2019.
  * [3] J. Carreira and A. Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 6299–6308, 2017.
  * [4] Y. Chen, Y. Kalantidis, J. Li, S. Yan, and J. Feng.  Multi-fiber networks for video recognition.  In Proc. Eur. Conf. Comput. Vis., pages 352–367, 2018.
  * [5] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al.  Rescaling egocentric vision.  arXiv preprint arXiv:2006.13256, 2020.
  * [6] A. Diba, M. Fayyaz, V. Sharma, M. Paluri, J. Gall, R. Stiefelhagen, and L. Van Gool.  Large Scale Holistic Video Understanding.  In Proc. Eur. Conf. Comput. Vis., pages 593–610. Springer, 2020\.
  * [7] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell.  Long-term recurrent convolutional networks for visual recognition and description.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 2625–2634, 2015.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In Proc. Int. Conf. Learn. Represent., 2021.
  * [9] L. Fan, S. Buch, G. Wang, R. Cao, Y. Zhu, J. C. Niebles, and L. Fei-Fei.  RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition.  In Proc. Eur. Conf. Comput. Vis., pages 505–521, 2020.
  * [10] L. Fan, W. Huang, C. Gan, S. Ermon, B. Gong, and J. Huang.  End-to-end learning of motion representation for video understanding.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 6016–6025, 2018.
  * [11] Q. Fan, C.-F. R. Chen, H. Kuehne, M. Pistoia, and D. Cox.  More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation.  In Proc. Adv. Neural Inform. Process. Syst., pages 2264–2273, 2019\.
  * [12] C. Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 203–213, 2020.
  * [13] C. Feichtenhofer, H. Fan, J. Malik, and K. He.  SlowFast Networks for Video Recognition.  In Proc. Int. Conf. Comput. Vis., pages 6201–6210, 2019.
  * [14] C. Feichtenhofer, A. Pinz, and R. Wildes.  Spatiotemporal residual networks for video action recognition.  In Proc. Adv. Neural Inform. Process. Syst., pages 3468–3476, 2016\.
  * [15] C. Feichtenhofer, A. Pinz, and R. P. Wildes.  Spatiotemporal multiplier networks for video action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4768–4777, 2017.
  * [16] C. Feichtenhofer, A. Pinz, and A. Zisserman.  Convolutional two-stream network fusion for video action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1933–1941, 2016.
  * [17] R. Girdhar and D. Ramanan.  Attentional pooling for action recognition.  In Proc. Adv. Neural Inform. Process. Syst., volume 30, pages 34–45, 2017.
  * [18] R. Girdhar, D. Ramanan, A. Gupta, J. Sivic, and B. Russell.  ActionVLAD: Learning spatio-temporal aggregation for action classification.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 971–980, 2017.
  * [19] R. Goyal, S. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al.  The” Something Something” Video Database for Learning and Evaluating Visual Common Sense.  In Proc. Int. Conf. Comput. Vis., pages 5842–5850, 2017.
  * [20] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al.  Ego4D: Around the World in 3,000 Hours of Egocentric Video.  arXiv preprint arXiv:2110.07058, 2021.
  * [21] K. Hara, H. Kataoka, and Y. Satoh.  Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 6546–6555, 2018.
  * [22] N. Hussein, E. Gavves, and A. W. Smeulders.  Timeception for complex action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 254–263, 2019.
  * [23] N. Hussein, E. Gavves, and A. W. Smeulders.  Videograph: Recognizing minutes-long human activities in videos.  In Proc. Int. Conf. Comput. Vis. Worksh., 2019.
  * [24] B. Jiang, M. Wang, W. Gan, W. Wu, and J. Yan.  STM: SpatioTemporal and Motion Encoding for Action Recognition.  Proc. Int. Conf. Comput. Vis., pages 2000–2009, 2019.
  * [25] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei.  Large-scale video classification with convolutional neural networks.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1725–1732, 2014.
  * [26] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al.  The Kinetics Human Action Video Dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [27] D. Kondratyuk, L. Yuan, Y. Li, L. Zhang, M. Tan, M. Brown, and B. Gong.  MoViNets: Mobile Video Networks for Efficient Video Recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 16020–16030, 2021.
  * [28] B. Korbar, D. Tran, and L. Torresani.  SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition.  In Proc. Int. Conf. Comput. Vis., pages 6232–6242, 2019.
  * [29] H. Kwon, M. Kim, S. Kwak, and M. Cho.  MotionSqueeze: Neural Motion Feature Learning for Video Understanding.  In Proc. Eur. Conf. Comput. Vis., pages 345–362, 2020.
  * [30] M. Lee, S. Lee, S. Son, G. Park, and N. Kwak.  Motion feature network: Fixed motion filter for action recognition.  In Proc. Eur. Conf. Comput. Vis., pages 387–403, 2018.
  * [31] C. Li, Q. Zhong, D. Xie, and S. Pu.  Collaborative Spatiotemporal Feature Learning for Video Action Recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 7872–7881, 2019.
  * [32] K. Li, X. Li, Y. Wang, J. Wang, and Y. Qiao.  Ct-net: Channel tensorization network for video classification.  In Proc. Int. Conf. Learn. Represent., 2021.
  * [33] X. Li, B. Shuai, and J. Tighe.  Directional Temporal Modeling for Action Recognition.  In Proc. Eur. Conf. Comput. Vis., pages 275–291, 2020.
  * [34] Y. Li, B. Ji, X. Shi, J. Zhang, B. Kang, and L. Wang.  TEA: Temporal Excitation and Aggregation for Action Recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 909–918, 2020.
  * [35] Y. Li, Y. Li, and N. Vasconcelos.  RESOUND: Towards action recognition without representation bias.  In Proc. Eur. Conf. Comput. Vis., pages 520–535, 2018.
  * [36] Z. Li, K. Gavrilyuk, E. Gavves, M. Jain, and C. G. Snoek.  VideoLSTM convolves, attends and flows for action recognition.  Comp. Vis. Image Und., 166:41–50, 2018.
  * [37] J. Lin, C. Gan, and S. Han.  Temporal Shift Module for Efficient Video Understanding.  In Proc. Int. Conf. Comput. Vis., pages 7082–7092, 2019.
  * [38] C. Luo and A. Yuille.  Grouped Spatial-Temporal Aggregation for Efficient Action Recognition.  In Proc. Int. Conf. Comput. Vis., pages 5512–5521, 2019.
  * [39] B. Martinez, D. Modolo, Y. Xiong, and J. Tighe.  Action recognition with spatial-temporal discriminative filter banks.  In Proc. Int. Conf. Comput. Vis., pages 5481–5490, 2019.
  * [40] Y. Meng, C.-C. Lin, R. Panda, P. Sattigeri, L. Karlinsky, A. Oliva, K. Saenko, and R. Feris.  Ar-net: Adaptive frame resolution for efficient action recognition.  In Proc. Eur. Conf. Comput. Vis., pages 86–104. Springer, 2020\.
  * [41] Y. Meng, R. Panda, C.-C. Lin, P. Sattigeri, L. Karlinsky, K. Saenko, A. Oliva, and R. Feris.  AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition.  In Proc. Int. Conf. Learn. Represent., 2021.
  * [42] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  In Proc. Int. Conf. Comput. Vis., pages 2630–2640, 2019.
  * [43] M. Monfort, A. Andonian, B. Zhou, K. Ramakrishnan, S. A. Bargal, T. Yan, L. Brown, Q. Fan, D. Gutfruend, C. Vondrick, et al.  Moments in Time Dataset: one million videos for event understanding.  IEEE Trans. Pattern Anal. Mach. Intell., pages 1–8, 2019.
  * [44] J. Ng and L. Davis.  Temporal difference networks for video action recognition.  In Proc. IEEE Wint. Conf. Appl. Comput. Vis., pages 1587–1596, 2018.
  * [45] A. J. Piergiovanni and M. S. Ryoo.  Representation flow for action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 9945–9953, 2019.
  * [46] Z. Qiu, T. Yao, and T. Mei.  Learning spatio-temporal representation with pseudo-3d residual networks.  In Proc. Int. Conf. Comput. Vis., pages 5533–5541, 2017.
  * [47] N. Quader, J. Lu, P. Dai, and W. Li.  Towards Efficient Coarse-to-Fine Networks for Action and Gesture Recognition.  In Proc. Eur. Conf. Comput. Vis., pages 35–51. Springer, 2020.
  * [48] M. S. Ryoo, A. Piergiovanni, J. Kangaspunta, and A. Angelova.  AssembleNet++: Assembling Modality Representations via Attention Connections.  In Proc. Eur. Conf. Comput. Vis., pages 654–671. Springer, 2020\.
  * [49] M. S. Ryoo, A. Piergiovanni, M. Tan, and A. Angelova.  AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures.  In Proc. Int. Conf. Learn. Represent., 2020.
  * [50] Z. Shou, X. Lin, Y. Kalantidis, L. Sevilla-Lara, M. Rohrbach, S. Chang, and Z. Yan.  DMC-Net: Generating discriminative motion cues for fast compressed video action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1268–1277, 2019.
  * [51] K. Simonyan and A. Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Proc. Adv. Neural Inform. Process. Syst., pages 568–576, 2014\.
  * [52] L. Smaira, J. Carreira, E. Noland, E. Clancy, A. Wu, and A. Zisserman.  A short note on the kinetics-700-2020 human action dataset.  arXiv preprint arXiv:2010.10864, 2020.
  * [53] A. Stergiou, G. Kapidis, G. Kalliatakis, C. Chrysoulas, R. Veltkamp, and R. Poppe.  Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions.  In Proc. IEEE Int. Conf. Image Process., pages 1830–1834. IEEE, 2019.
  * [54] S. Sudhakaran, S. Escalera, and O. Lanz.  LSTA: Long Short-Term Attention for Egocentric Action Recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 9954–9963, 2019.
  * [55] S. Sudhakaran, S. Escalera, and O. Lanz.  Gate-Shift Networks for Video Action Recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1102–1111, 2020.
  * [56] L. Sun, K. Jia, D. Yeung, and B. E. Shi.  Human action recognition using factorized spatio-temporal convolutional networks.  In Proc. Int. Conf. Comput. Vis., pages 4597–4605, 2015.
  * [57] S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang.  Optical flow guided feature: a fast and robust motion representation for video action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1390–1399, 2018.
  * [58] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In Proc. Int. Conf. Comput. Vis., pages 4489–4497, 2015.
  * [59] D. Tran, H. Wang, L. Torresani, and M. Feiszli.  Video Classification With Channel-Separated Convolutional Networks.  In Proc. Int. Conf. Comput. Vis., pages 5552–5561, 2019.
  * [60] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 6450–6459, 2018.
  * [61] H. Wang, D. Tran, L. Torresani, and M. Feiszli.  Video Modeling With Correlation Networks.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 349–358, 2020.
  * [62] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In Proc. Eur. Conf. Comput. Vis., pages 20–36, 2016.
  * [63] X. Wang, R. Girshick, A. Gupta, and K. He.  Non-local neural networks.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 7794–7803, 2018.
  * [64] X. Wang and A. Gupta.  Videos as space-time region graphs.  In Proc. Eur. Conf. Comput. Vis., pages 399–417, 2018.
  * [65] X. Wang, X. Xiong, M. Neumann, A. Piergiovanni, M. S. Ryoo, A. Angelova, K. M. Kitani, and W. Hua.  AttentionNas: Spatiotemporal Attention Cell Search for Video Classification.  In Proc. Eur. Conf. Comput. Vis., pages 449–465. Springer, 2020\.
  * [66] Y. Wang, L. Jiang, M.-H. Yang, L.-J. Li, M. Long, and L. Fei-Fei.  Eidetic 3D LSTM: A Model for Video Prediction and Beyond.  In Proc. Int. Conf. Learn. Represent., 2019.
  * [67] C. Wu, C. Feichtenhofer, H. Fan, K. He, P. Krahenbuhl, and R. Girshick.  Long-Term Feature Banks for Detailed Video Understanding.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 284–293, 2019.
  * [68] H. Wu, J. Liu, X. Zhu, M. Wang, and Z.-J. Zha.  Multi-Scale Spatial-Temporal Integration Convolutional Tube for Human Action Recognition.  In Proc. Int. Joint Conf. Artif. Intell., pages 753–759, 2020.
  * [69] W. Wu, D. He, T. Lin, F. Li, C. Gan, and E. Ding.  MVFNet: Multi-View Fusion Network for Efficient Video Recognition.  In Proc. AAAI, pages 2943–2951, 2021.
  * [70] Z. Wu, H. Li, Y. Zheng, C. Xiong, Y.-G. Jiang, and L. S. Davis.  A coarse-to-fine framework for resource efficient video recognition.  Int. J. Comput. Vis., pages 1–13, 2021.
  * [71] Z. Wu, C. Xiong, C.-Y. Ma, R. Socher, and L. S. Davis.  AdaFrame: Adaptive Frame Selection for Fast Video Recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 1278–1287, 2019.
  * [72] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In Proc. Eur. Conf. Comput. Vis., pages 305–321, 2018.
  * [73] C. Yang, Y. Xu, J. Shi, B. Dai, and B. Zhou.  Temporal Pyramid Network for Action Recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 591–600, 2020.
  * [74] C. Zach, T. Pock, and H. Bischof.  A duality based approach for realtime TV-L 1 optical flow.  In Joint pattern recognition symposium, pages 214–223, 2007.
  * [75] C. Zhang, A. Gupta, and A. Zisserman.  Temporal Query Networks for Fine-grained Video Understanding.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4486–4496, 2021.
  * [76] S. Zhang, S. Guo, W. Huang, M. R. Scott, and L. Wang.  V4D: 4D Convolutional Neural Networks for Video-level Representation Learning.  In Proc. Int. Conf. Learn. Represent., 2020.
  * [77] L. Zhao, T. Liu, X. Peng, and D. Metaxas.  Maximum-entropy adversarial data augmentation for improved generalization and robustness.  arXiv preprint arXiv:2010.08001, 2020.
  * [78] Y. Zhao, Y. Xiong, and D. Lin.  Trajectory Convolution for Action Recognition.  In Proc. Adv. Neural Inform. Process. Syst., pages 2208–2219, 2018\.
  * [79] B. Zhou, A. Andonian, A. Oliva, and A. Torralba.  Temporal relational reasoning in videos.  In Proc. Eur. Conf. Comput. Vis., pages 803–818, 2018.
  * [80] Y. Zhou, X. Sun, Z. Zha, and W. Zeng.  MiCT: Mixed 3d/2d convolutional tube for human action recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 449–458, 2018.
  * [81] X. Zhu, C. Xu, L. Hui, C. Lu, and D. Tao.  Approximated Bilinear Modules for Temporal Modeling.  In Proc. Int. Conf. Comput. Vis., pages 3493–3502, 2019.
  * [82] M. Zolfaghari, K. Singh, and T. Brox.  ECO: Efficient Convolutional Network for Online Video Understanding.  In Proc. Eur. Conf. Comput. Vis., pages 695–712, 2018.

  *[CNNs]: Convolutional Neural Network
  *[CNN]: Convolutional Neural Network
  *[GSF]: Gate-Shift-Fuse
  *[FLOPs]: Floating Point Operation
  *[RNNs]: Recurrent Neural Network
  *[TSM]: Temporal Shift Module
  *[GSM]: Gate-Shift Module
  *[SGD]: Stochastic Gradient Descent
  *[GSFs]: Gate-Shift-Fuse
  *[TDN]: Temporal Difference Network
  *[TSN]: Temporal Segment Network
  *[TRN]: Temporal Relation Network
  *[GST]: Grouped Spatial-temporal Aggregation
  *[FC]: Fully Connected