#  Spatial-Temporal Alignment Network for Action Recognition and Detection

Junwei Liang1 Liangliang Cao2 Xuehan Xiong2 Ting Yu2 Alexander Hauptmann1
1Carnegie Mellon University 2Google Cloud AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work partially done
during a research internship at Google.

###### Abstract

This paper studies how to introduce viewpoint-invariant feature
representations that can help action recognition and detection. Although we
have witnessed great progress of action recognition in the past decade, it
remains challenging yet interesting how to efficiently model the geometric
variations in large scale datasets. This paper proposes a novel Spatial-
Temporal Alignment Network (STAN) that aims to learn geometric invariant
representations for action recognition and action detection. The STAN model is
very light-weighted and generic, which could be plugged into existing action
recognition models like ResNet3D and the SlowFast with a very low extra
computational cost. We test our STAN model extensively on AVA, Kinetics-400,
AVA-Kinetics, Charades, and Charades-Ego datasets. The experimental results
show that the STAN model can consistently improve the state of the arts in
both action detection and action recognition tasks. We will release our data,
models and code.

##  1 Introduction

Human vision can recognize video actions efficiently despite the variations of
viewpoints. Convolutional neural networks (CNNs) [41, 43, 3, 35, 8] fully
utilize the power of GPUs/TPUs and employ spatial-temporal filters to
recognize actions, which outperforms traditional models including oriented
filtering in space time (HOG3D) [21], spatial-temporal interest points [24],
motion history images [42], and trajectories [47]. However, due to the high
variations in space-time, the state of the art of action recognition is still
far from satisfactory, compared with the success of 2D CNNs in image
recognition [13].

A key challenge of action recognition is to capture the variations across
space and time. Since CNN assumes the filters share weights at different
locations, it can not explicitly model the viewpoint changes and other
variations. To solve such limitations, a practical way is to expand feature
representations to accommodate a higher degree of freedom. For example, the
two-stream network [38] proposes to integrate optical flow with RGB features.
More recently, SlowFast [7] combines both slow and fast pathways to learn
different temporal information, and obtain good performance. However, such
feature expanding approaches quickly lead to cumbersome, high-dimensional
feature maps, which not only make the computation more expensive but also miss
the geometric interpretation of the subjects.

Figure 1: An illustrative example of how STAN can address the bounding box
misalignment issue in spatial temporal localization. In prior work [10, 7],
the detected person box (solid line) is expanded along the temporal axis
(dotted line) to form a 3D cube for the subsequent temporal ROI-align (TROI-
align). If the actor’s movement is large across frames as shown in the top row
(or there are substantial camera movements), the TROI-aligned features will be
noisy with the majority being background pixels. STAN learns a spatial
temporal transformation that puts feature maps in the same coordinate system
and in the bottom row example it is the central frame.

This paper proposes a different approach to capture the variation in action
recognition. Instead of relying on stacking deeper CNN layers, this paper aims
to explicitly learn geometric transformations and viewpoint invariant
features. Our idea is motivated by [22], which believes that human vision
relies on coordinate frames. However, the stacked capsule autoencoder in [22]
is designed for 2D images, and too expensive for large scale visual
recognition.

Fig. 1 illustrates the importance of alignment in the problem of spatial
temporal localization. Following the existing action detection pipeline [10,
7], the action representation is obtained by first cropping a 3D cube within
the spatial-temporal feature maps centered at the detected actor’s bounding
box, followed by a Max Pooling operation. Without alignment, the
representation is contaminated by background pixels as shown in the top row in
Fig. 1. With alignment the representation as shown in the bottom row is much
more focused on the actor.

In this paper, we propose a Spatial Temporal Alignment Network (STAN) that
aims to learn viewpoint invariant representations for action recognition and
action detection. The STAN block is very light-weighted and generic, which
could be plugged into existing action recognition models like ResNet3D, Non-
Local Network [48] and the SlowFast network [7]. As discussed in Section 3.6,
we insert a STAN block between $res_{2}$ and $res_{3}$ in the ResNet3D
architecture, and add it at the same location on the Fast pathway in the
SlowFast model. For the SlowFast + STAN model, the FLOPS increase is only
relatively 2.1% (134.5 GFLOPS vs. 131.7 GFLOPS) for action recognition on
Charades, but we achieve 5% relative improvement on mean average precision.

The contribution of this paper is three-fold: (1) To the best of our
knowledge, this is the first work to explore explicit spatial-temporal
alignment in 3D CNNs for action detection. (2) Our STAN requires very low
extra FLOPS in addition to the backbone network. (3) Extensive experiments on
different datasets suggest the model using STAN can outperform the state of
the art.

##  2 Related Work

The research of action recognition has advanced with both new datasets and new
models. As one of the earliest action recognition benchmarks, KTH[34] collects
videos of individual actors repetitively performing six types of human actions
(walking, jogging, running, boxing, hand waving and clapping) with a clean
background. Because these videos are very simple, KTH dataset turns out to be
a very easy benchmark since studies quickly obtained near-perfect accuracy on
it [1, 46].

To overcome the limitation of KTH, the HMDB dataset [23] was proposed in 2011
with 51 actions in 7000 video clips, while UCF101 [39] extended this effort by
collecting 101 action classes in 13000 clips. Both benchmarks are captured
with more diversified backgrounds. In the past decade, we have witnessed a
steady improvement of accuracy on these two datasets by different methods
including features fusion [32], two-stream network [38], C3D [43], I3D [3],
graph-based approaches [49, 5, 33, 54] and others [51, 4, 14]. However, some
clips in the UCF101 test set are taken from the same YouTube video as the
training set [20], which makes it relatively easy to obtain good accuracy on
UCF101. As a result, the SOTA on UCF101 dataset is more than 98%[20].

The modern benchmarks for action recognition and detection is the Kinetics
dataset [20] and the AVA dataset [25], respectively. The Kinetics dataset
proposes a bigger benchmark with more categories and more videos (e.g., 400
categories 160,000 clips in [20]) as a harder benchmark. The action labels in
AVA [25] are annotated with spatial temporal locations, which is more
challenging than the setting of one label per clip. Many new approaches [44,
55, 29, 7, 53] have been carried on these two datasets, of which the SlowFast
network [7] obtains the state of the art performance. Note that the SlowFast
contains more parameters than C3D or I3D networks, by integrating features at
both high and low frame rates. We can see the trend of action recognition in
the last two decades is to collect bigger datasets (e.g. Kinetics) as well as
build bigger models (e.g., I3D and SlowFast).

This paper considers action recognition using the spatial-temporal alignment
to overcome the viewpoint variations in videos. In recent years, there has
been a consistent effort to use alignment for image recognition [15, 50, 18,
28, 22]. Some data augmentation methods [27, 26] using 3D simulation have been
proposed to tackle the viewpoint changes. However, many previous studies show
that alignment models are not as competitive as data-driven approaches like
data augmentation or spatial pooling for image recognition. Some recent works
have to rely on very expensive models such as recurrent networks [28] or
stacked capsules [22]. As a result, a lot of alignment-based recognition
methods are limited to MNIST [22] and face recognition [50]. Some follow-up
works on capsule network [6] and 2D alignment network [16] have been proposed
but they are limited to action recognition on small datasets like UCF101 [39]
and JHMDB [19]. This paper shows that it is possible to build an efficient
spatial-temporal alignment for both action recognition and detection, and
improve the state of the arts with very few extra parameters.

Figure 2: Spatial Temporal Alignment Network (STAN) design. In our design,
different temporal slices of the feature map can undergo different
transformations to account for intra-clip camera motion and actor movement.
The same applies to subgroups of the channel, an idea similar to multihead
attention [45]. We show four alignment examples with a reference line where
the actor is aligned on for demonstration. After the features are transformed
and processed through the backbone network, they are passed to a temporal ROI-
Align layer (TROI-Align) if the task is action detection, and then finally to
the classification layer.

##  3 The STAN Model

In this section, we describe our spatial-temporal alignment network for action
recognition and detection, which we call STAN. Motivated by the previous works
in image understanding [15, 18, 28], our work considers a generalized model in
video domain, so that it can handle dynamic viewpoint changes in action
recognition and detection. The key idea of STAN is to utilize spatial-temporal
alignment for feature maps to account for viewpoint changes and actor
movements in the videos. In general, given an input spatial-temporal feature
map $\mathcal{I}_{in}\in\mathbf{R}^{C\times T\times H\times W}$ where $H$
stands for height, $W$ for width, $T$ for time and $C$ for channels, the
alignment function is defined as

$\displaystyle\text{STAN}(\mathcal{I}_{in})=\mathcal{T}(\theta,\mathcal{I}_{in})+\mathcal{I}_{in},\text{where}\;\theta=\mathcal{D}(\mathcal{I}_{in})$
(1)

In this paper, the output of STAN function is of the same dimension as
$\mathcal{I}_{in}$. Function $\mathcal{D}$ represents the deformation network,
where the feature map alignment parameter $\theta\in\mathbf{R}^{4\times 4}$ is
computed based on the input feature map. Function $\mathcal{D}$ can be in the
form of a simple feed-forward network using spatial-temporal features [44].
Function $\mathcal{T}$ is defined as the warping function, where input feature
maps are warped based on the alignment parameter. In this paper, we add a
residual connection between the input feature maps and output feature maps for
faster training and avoiding boundary effect described in [28]. The STAN layer
can be added to different locations of the backbone to account for the
alignment needs for different level of feature maps.

###  3.1 Network Architecture

The overall STAN architecture is shown in Fig. 2. Our model uses a 3D CNN as
the backbone network to extract spatial-temporal feature maps from video
frames. In addition, STAN has the following key components:

Deformation Network produces the alignment / deformation parameter $\theta$ in
Eqn. 1.

Warping Module samples from the input feature maps based on $\theta$ and
outputs the final transformed feature maps.

Channel Grouping allows different alignment for different subsets of the
features similar to multihead attention [45].

Temporal Grouping learns a separate alignment for different temporal segments
within the video clip.

In the following, we will introduce the above modules in details.

###  3.2 Deformation Network

The deformation network $\mathcal{D}$ produces a transformation parameter,
$\theta\in\mathbf{R}^{4\times 4}$. Our network is based on R(2+1)D [44]
although other options are possible, such as a simple feed-forward network, or
a recurrent network and compositional function as proposed in [28]. Suppose
the number of input channels is $C$, the details of our network architecture
is presented in Table 1.

Layer | Filter size | Output channels
---|---|---
conv | $1\times 7^{2}$ | $C\sslash 4$
conv | $7\times 1^{2}$ | $C\sslash 2$
maxpool | $4\times 7^{2}$ | $C\sslash 2$
stride $4\times 7^{2}$
conv | $7\times 7^{2}$ | $C$
globalpool |  | $C$
fc |  | variable
Table 1: Deformation network architecture used in our experiments. The filter
sizes are $T\times S^{2}$ where $T$ is the temporal kernel size and $S$ is the
spatial size. The global pool layer averages features across all spatial-
temporal locations. “fc” is the final fully-connected layer.

All convolution layers are followed by batch normalization [17] and ReLU [11,
9]. The dimension of the final FC layer depends on the type of
parameterization we choose for the spatial-temporal alignment. Taking affine
transformation for example, the dimension of the network output
($\mathbf{p}_{\text{affine}}=[p_{1}\;...\;p_{12}]^{T}$) is of size 12 and
$\theta$ is constructed as

$\displaystyle\theta(\mathbf{p}_{\text{affine}})=\begin{pmatrix}1+p_{1}&p_{2}&p_{3}&p_{4}\\\
p_{5}&1+p_{6}&p_{7}&p_{8}\\\ p_{9}&p_{10}&1+p_{11}&p_{12}\\\ 0&0&0&1\\\
\end{pmatrix}$ (2)

$\theta$ can be more restrictive as in the case of attention [52], where
cropping, translation and scaling are allowed for transformation, the
dimension of the network output $\mathbf{p}_{\text{att}}=[p_{1},...p_{6}]^{T}$
is a vector of size 6 and $\theta$ is constructed as

$\displaystyle\theta(\mathbf{p}_{\text{att}})=\begin{pmatrix}1+p_{1}&0&0&p_{4}\\\
0&1+p_{2}&0&p_{5}\\\ 0&0&1+p_{3}&p_{6}\\\ 0&0&0&1\\\ \end{pmatrix}$ (3)

The design of STAN is flexible and can be any type of transformation. The key
intuition of the deformation on CNN feature maps is to compensate for the fact
that CNNs are not rotation, scale, and shear transformation equivariant [22].

###  3.3 Warping Module

After computing the transformation matrix $\theta$, we utilize a
differentiable warping function $\mathcal{T}$ to transform the feature maps
with better alignment of the content. See Fig. 3. The warping function is
essentially a resampling of features from the input feature maps to the output
at each corresponding pixel location. Note that the feature maps could also be
images. Extending from the notation of 2D alignment [18] , we define the
output feature maps $\mathcal{I}\in\mathbf{R}^{C\times T\times H\times W}$ to
lie on a spatial-temporal regular grid $G=\\{G_{i}\\}$, where each element of
the grid $G_{i}=(t^{o}_{i},x^{o}_{i},y^{o}_{i})$ corresponds to a vector of
output features of size $\mathbf{R}^{C}$. Hence, the pointwise sampling
between the input and output feature maps is written as

$\displaystyle\begin{bmatrix}t^{s}_{i}\\\ x^{s}_{i}\\\ y^{s}_{i}\\\ 1\\\
\end{bmatrix}=\mathcal{T}_{g}(\theta,G_{i})=\begin{pmatrix}1\\!+\\!p_{1}&0&0&p_{4}\\\
0&1\\!+\\!p_{2}&0&p_{5}\\\ 0&0&1\\!+\\!p_{3}&p_{6}\\\ 0&0&0&1\\\
\end{pmatrix}\begin{bmatrix}t^{o}_{i}\\\ x^{o}_{i}\\\ y^{o}_{i}\\\ 1\\\
\end{bmatrix}$ (4)

where ($t^{o}_{i},x^{o}_{i},y^{o}_{i}$) are the output feature map coordinates
in the regular grid and ($t^{s}_{i},x^{s}_{i},y^{s}_{i}$) are the
corresponding input feature map coordinates for feature sampling. Here we use
attention transformation as an example, where the deformation matrix is
$\theta$ parameterized by $\mathbf{p}_{\text{att}}$ (Eqn. 3). Given the
coordinates mapping, as the computed corresponding coordinates in the input
feature maps might not be integers, we utilize the differentiable trilinear
interpolation to sample input features from the eight closest points based on
their distance to the computed point ($t^{s}_{i},x^{s}_{i},y^{s}_{i}$). In
this way, we iterate through every point in the regular grid,
($t^{o}_{i}\in[1,...,T],x^{o}_{i}\in[1,...,W],y^{o}_{i}\in[1,...,H]$) and
compute the output feature maps identically for each channel.

Figure 3: Two examples of applying spatial-temporal warping/transform to an
input sequence. The top row shows an identity transformation while the bottom
row shows an example 3D warping where the actor is rotated and zoomed in.

###  3.4 Channel Grouping

To increase the flexibility of spatial-temporal alignment, we introduce
channel grouping to allow multiple alignments and deformations of feature maps
at the same time. This is inspired by the fact that there could be multiple
attention regions in the video that are relevant to recognizing the actions.
Specifically, we define STAN with channel grouping as follows:

$\displaystyle\text{STAN}_{cg}(\mathcal{I}_{in})=\text{concat}\left(\text{STAN}(\mathcal{I}^{1}_{in}),...,\text{STAN}(\mathcal{I}^{C_{g}}_{in})\right)$
(5)

where ${C_{g}}$ is the number of channel groups. Essentially, we apply
transformation on each channel group by
$\text{STAN}(\mathcal{I}^{C_{g}}_{in})\in\mathbf{R}^{C\sslash{C_{g}}\times
T\times H\times W}$ so that multiple alignments can be utilized.

###  3.5 Temporal Grouping

Many actions are composed of a sequence of sub-actions that require different
alignments. For example, action “High Jump” may consist of “stand”, “run” and
“jump” (see Fig. 2), and the actor may have moved around between video frames.
Based on this observation, we propose temporal grouping to allow the model to
learn different alignments at different time periods. Similar to channel
grouping, it is written as

$\displaystyle\text{STAN}_{tg}(\mathcal{I}_{in})=\text{concat}\left(\text{STAN}(\mathcal{I}^{1}_{in}),...,\text{STAN}(\mathcal{I}^{T_{g}}_{in})\right)$
(6)

where ${T_{g}}$ is the number of temporal groups and concatenation is operated
on the temporal axis. Each group alignment
$\text{STAN}(\mathcal{I}^{T_{g}}_{in})$ is of size $\mathbf{R}^{C\times
T\sslash{T_{g}}\times H\times W}$.

###  3.6 Integration with Backbone CNNs

Now that we have defined STAN function we now discuss effective ways to add it
into existing backbone CNN networks. In this paper, we focus on designing STAN
for the RGB stream and the extension to optical flow will be left as one of
our future work. Recent works [48, 7] and their variants [31] achieve single-
stream state-of-the-art performance on action recognition and detection with
3D CNNs. We therefore explore adding STAN layers to the ResNet3D model [48, 7]
and the SlowFast [7] network. Intuitively, placing STAN layer at shallower
layers allows earlier feature alignments that could potentially lead to
cleaner representation for better action recognition. However, shallow layers
may not have enough abstraction in the feature maps for STAN to learn the
right alignments. Based on this trade-off, we experiment with adding one STAN
layer after $res_{2}$ block. For SlowFast, we only add STAN layers on the Fast
pathway since it contains more temporal information compared to the Slow
pathway. Other insertion locations are explored in the ablation experiments
(Section 4.1.2).

###  3.7 Detection Architecture

For action recognition, the final CNN outputs are passed through global
averaging (and a concatenation for SlowFast) and a fully-connected layer to
get the action class probabilities. For action detection, following previous
works [10, 40, 7], we use pre-computed actor proposals. The bounding boxes are
used to extract region-of-interest (RoI) features using RoI-Align [12] at the
last feature map of “$res_{5}$” after temporal global average pooling. Our
STAN can fix the actor misalignment problem by aligning the 2D RoIs along the
temporal axis. Finally, the RoI features are then max-pooled and fed to a
fully-connected layer.

##  4 Experiments

To demonstrate the efficacy of our models, specifically, the viewpoint-
invariant design that helps action models to generalize, we experiment on two
recent action detection datasets, including AVA [10] and AVA-Kinetics [25],
and several major action recognition datasets, Kinetics-400 [20], Charades
[37], Charades-Ego [36].

###  4.1 Action Detection

This section evaluates our STAN model for the task of spatial temporal
localization for actions using AVA and AVA-Kinetics. We show significant
improvement over baselines while only adding a fraction of computation.

Datasets. The AVA dataset [10] is an action detection dataset where models are
required to output action classification results with bounding boxes. Spatial-
temporal labels are provided for one frame per second, with every person
annotated with a bounding box and (possibly multiple) actions. There are 211k
training and 57k validation video clips. We use AVA v2.1 and follow the
standard protocol [10, 7] of evaluating on 60 classes. The performance metric
is mean Average Precision (mAP) over 60 classes, using an IoU threshold of
0.5. We report mAP on the official validation set.

The AVA-Kinetics dataset [25] is a recent action detection dataset which
follows AVA annotation protocol to annotate relevant videos on the
Kinetics-700 [2] dataset. We utilize the part where all videos are from
Kinetics for training and testing. There are about 141k training videos and
32k validation videos. The AVA-Kinetics also contains the same 60 classes for
evaluation and we utilize the mAP metric. Following [25], we evaluate action
detection model performance with both ground truth person boxes and pre-
computed person boxes. We report mAP on the official validation set as well.

Action Proposals. The action detection models have to output action/person
bounding boxes, and only predicted boxes with IoU (intersection over union)
area w.r.t the ground truth boxes above a threshold of 0.5 are considered true
positives. As mentioned in Section 3.7, we follow previous works [10, 40, 7]
and use pre-computed person boxes as action proposals. For AVA, we utilize the
same person proposals from [7]. For AVA-Kinetics, we finetune a Mask-RCNN [12]
with ResNet-101 backbone trained on COCO [30] on the person boxes in the
training set and extract boxes as described in Section 3.7. The region
proposals for action detection are detected person boxes with a confidence
score of larger than 0.8, which has a recall of 83.1% and a precision of 62.1%
for the person class, given IoU threshold of 0.5. The average precision of the
person class is 0.732 on the validation set.

Training. We initialize the network weights from the Kinetics-400
classification models, following previous works [7]. We use a learning rate of
0.1 and cosine learning rate decay. We use synchronized SGD to train for 10
epochs with a batch size of 16 on a 4-GPU machine, with a linear warm-up from
0.000125 for the first 2 epochs. We use a weight decay of $10^{-7}$. Ground-
truth boxes and video clips centered at the annotated key frames are sampled
for training. The video is first resized to 256x320, and then we use random
224×224 crops and horizontal flipping following [7].

Inference. Since the annotations on AVA and AVA-Kinetics are one (key) frame
per second, we sample a single video clip temporally centered around the key
frame for evaluation. Following [7], we resize the spatial dimension such that
its shorter side is 256 pixels. Ground truth boxes or pre-computed boxes are
used as inputs. We report the inference time computational cost (in FLOPs) of
a single 256x320 clip using Tensorflow’s Profiler.

Implementation Details. We add one STAN layer to the backbone CNN network as
described in Section 3.6. We use affine transformation (Eqn. 2) and the
deformation network is defined in Table 1. We use a temporal group of 2 and
the number of base convolution filters in the deformation network is capped at
8 for ResNet3D to keep the FLOPs low. For SlowFast, we set this number as
described in Table 1.

Baselines. To demonstrate the effectiveness of our proposed model, we
experiment with recent 3D-CNN based models for action recognition and
detection. ResNet3D is a model based on ResNet-50 [13] with additional 3D
convolutional filters. The number of input frames is 8 and we sample 1 frame
every 8 frames (i.e., 8x8 frames). ResNet3D + STAN is our proposed model added
to ResNet3D as described in Section 3.6 with the same inputs. SlowFast is a
recent efficient model [7] with a Slow pathway and a Fast pathway, which takes
8x8 and 32x2 frames as inputs respectively. We use ResNet-50 backbone for
SlowFast as well. If we use only the Slow pathway, the model will become the
same as ResNet3D. SlowFast + STAN is our proposed model added to SlowFast as
described in Section 3.6.

Models | mAP | GFLOPs | MParams
---|---|---|---
ResNet3D (8x8) | 0.234 | 208.0 | 31.75
ResNet3D + STAN | 0.241 | 216.6 | 32.02
SlowFast [7] (32x2) | 0.252 | 242.6 | 33.77
SlowFast + STAN | 0.263 | 247.4 | 33.96
Table 2: Experiment results on AVA dataset. We show the model performance on mAP, computation cost (in billions) and number of parameters (in millions). The computational cost is of a single 256x320 video clip of length 8x8 or 32x2 (number of frames x sampling rate) frames.  Models | mAP | GFLOPs
---|---|---
Action Transformer [25] | 0.337 / 0.191 | -
ResNet3D (8x8) | 0.315 / 0.224 | 208.0
ResNet3D + STAN |  0.336 / 0.238 | 216.6
SlowFast [7] (32x2) | 0.341 / 0.242 | 242.6
SlowFast + STAN |  0.358 / 0.253 | 247.4
Table 3: Experiment results on AVA-Kinetics dataset. We show both mAP with
ground truth boxes and detected boxes. We are not able to get Action
Transformer’s FLOPs as the code is not public. Note that the pre-computed
boxes for Action Transformer is different from ours.

####  4.1.1 Main Results

AVA. Table 2 shows the results on AVA dataset [10]. We follow the SlowFast [7]
paper’s evaluation protocol and use the same predicted person bounding boxes
provided by the authors with ROIAlign to classify actions. For both ResNet3D
and SlowFast, we use ResNet-50 as their base architecture. Compared with
previous methods, our model is able to achieve 3% relative improvement on mAP
for ResNet3D, with only relatively 4% more computation. The parameter increase
is also minimal. On SlowFast, our method’s improvement is more efficient, with
4% relative improvement on mAP and only 2% more computation.

AVA-Kinetics is a recent action detection dataset with Internet videos from
the Kinetics-700 dataset [2]. Table 3 shows the results on AVA-Kinetics
dataset [25]. We follow the baseline [25] paper’s evaluation protocol and
experiment with both ground truth person boxes and detected person boxes from
the described finetuned Mask-RCNN model. Our model is able to achieve 7%
relative improvement on mAP for ResNet3D with only relatively 4% more
computation and 5% improvement with 2% more computation on SlowFast.

####  4.1.2 Ablation Experiments

| Diff | mAP | GFLOPs
---|---|---|---
SlowFast | - | 0.252 | 242.55
\+ STAN | +1.1% | 0.263 | 247.40
\+ STAN ($res_{3}$) | +0.7% | 0.259 | 247.35
\+ STAN ($res_{4}$) | +0.3% | 0.255 | 247.39
\+ STAN (Att, 6) | +0.7% | 0.259 | 247.40
\+ STAN (Sp, 6) | +0.6% | 0.258 | 247.40
\+ STAN (H, 15) | +0.3% | 0.255 | 247.40
\+ STAN (no tg) | +0.8% | 0.260 | 247.40
\+ STAN (cg=8) | +1.0% | 0.262 | 243.30
\+ STAN (fixed $W_{\theta}$) | +0.7% | 0.259 | 247.40
Table 4: Ablation experiment results (on AVA). The computational cost is of a
single 256x320 video clip of length 32x2 (number of frames x sampling rate)
frames. The “Diff” column shows absolute improvement of the variant models
compared to the baseline SlowFast model. “(Att, 6)” means attention
transformation with 6 degree-of-freedom (DoF), “(Sp, 6)” means spatial
transformation with 6 DoF (defined in Eqn. 7) and “(H, 15)” means homography
transformation with 15 DoF. “tg” and “cg” means temporal grouping and channel
grouping, respectively. See text for details.

In this section, we perform ablation studies on the AVA dataset with the
SlowFast model as the backbone network. To understand how action models can
benefit from STAN, we explore the following questions (results are shown in
Table 4):

Where to insert STAN layer? In CNN networks, shallower layers tend to encode
low-level visual features like edges and patterns while deeper layers may
contain more abstract information. Placing STAN layer at shallower layers
allows earlier feature alignments that could potentially lead to cleaner
representation. To verify this hypothesis, we experiment with adding one STAN
layer at deeper layers than $res_{2}$ block. We add STAN layer after $res_{3}$
and $res_{4}$. As we see, the model performance deteriorates significantly,
suggesting that STAN should be placed at earlier layers.

What is the best parameterization for the deformation network? In Section 3.2,
we have discussed two ways of parameterization for the deformation network,
affine transformation (Eqn. 2) and attention transformation (Eqn. 3). Each has
12 and 6 degree-of-freedom (DoF), respectively. We use affine transformation
in our main experiments and here we experiment with attention transformation,
spatial transformation and homography transformation. Given the deformation
network output of $\mathbf{p}_{\text{sp}}=[p_{1},...p_{6}]^{T}$, the spatial
transformation is defined as follows:

$\displaystyle\theta(\mathbf{p}_{\text{sp}})=\begin{pmatrix}1+p_{3}&0&0&p_{4}\\\
0&1+p_{1}&-p_{2}&p_{5}\\\ 0&p_{2}&1+p_{1}&p_{6}\\\ 0&0&0&1\\\ \end{pmatrix}$
(7)

where the second and third row is for the width and height dimension. For
homography transformation, the last element in the $4\times 4$ matrix is set
as 1 hence the DoF is 15, which is shown in Table 4 (“H, 15”). Results are
shown in Table 4. As we see, the model with the most free parameters performs
the worst, while attention and spatial transformation perform worse than
affine transformation.

Does channel/temporal grouping help? In this experiment, we validate the
efficacy of temporal grouping (Section 3.5) and channel grouping (Section
3.4). The main experiments are conducted with a temporal group of 2, and the
model performance drops by a small margin if temporal grouping is removed.
Interestingly, when we apply a channel grouping of 8 to the original model, we
can achieve similar performance improvement but with significantly less
computation (with channel grouping, we only need to add 0.3% relatively more
computation compared to 2% as before to achieve nearly the same performance.)
This result is also observed in the Kinetics-400 experiments (Table 5).

Does STAN transfer well? Finally, we conduct an experiment to see whether the
deformation network learned from a dataset can be generalized to another. We
train the original STAN model on Kinetics-400 dataset, and then only fine-tune
the layers after the STAN layer on AVA. As we see in Table 4 (“fixed
$W_{\theta}$”), STAN can still achieve reasonable improvement on AVA,
suggesting the deformation can be transferred from one dataset to another.

Models | top-1 | top-5 | GFLOPs
---|---|---|---
NonLocal R50 [48] | 0.749 | 0.916 | -
ResNet3D (8x8) | 0.735 | 0.908 | 109.2
ResNet3D + STAN | 0.751 | 0.916 | 113.2
SlowFast [7] (32x2) | 0.759 | 0.920 | 131.7
SlowFast + STAN | 0.769 | 0.928 | 134.5
SlowFast + STAN (cg=8) | 0.770 | 0.928 | 132.3
Table 5: Experiment results on Kinetics-400 dataset. The nonlocal baseline has
an input of 32 frames. “cg” means channel grouping. The computational cost is
of a single 256x256 video clip.

###  4.2 Action Recognition

The action recognition task is defined to be a classification task given a
trimmed video clip. To evaluate the generalization abilities of our proposed
model, we consider three major datasets, Kinetics-400 [20], Charades [37] and
Charades-Ego [36].

Datasets. Kinetics-400 [20] consists of about 240k training videos and 20k
validation videos in 400 human action classes. The videos are about 10 seconds
long. Following previous works [7, 48], we report top-1 and top-5
classification accuracy. Charades [37] is a dataset with longer ( about 30
seconds on average) videos of indoor activities. There are about 9.8k training
videos and 1.8k validation videos in 157 classes in a multi-label
classification setting. Charades-Ego [36] has the same 157 action labels but
consists of both third-person view and first-person view videos. Essentially,
this dataset shows different perspectives of the same actions, which makes it
ideal to test our proposed method. Performance is measured in mean Average
Precision (mAP).

Training. Our models on Kinetics are trained from scratch with random
initialization, without using any pre-training (same as in [7]). We use a
learning rate of 0.2 and cosine learning rate decay. We use synchronized SGD
to train for 100 epochs with a batch size of 16 on a 4-GPU machine, with a
linear warm-up from 0.01 for the first 20 epochs. We use a weight decay of
$10^{-7}$. On Charades and Charades-Ego, we initialize the models using models
trained on Kinetics-400. We use a learning rate of 0.02 and train for 50
epochs, where learning rate reduces to its 1/10 at epoch 40. We use a linear
warm-up from 0.000125 for the first 2 epochs. For all three datasets, we use
random $224\times 224$ crops and horizontal flipping from a video clip, which
is randomly sampled from the full-length video and resized to a shorter edge
side of randomly sampled in [256, 320] pixels.

Inference. Following previous works [7, 48], we sample $3\times 10$ clips for
each video during testing: we uniformly sample 10 clips for the temporal
domain and 3 spatial crops of size $256\times 256$ after the shorter edge size
are resized to 256 pixels. We average the softmax scores across all clips for
final prediction for Kinetics-400 and use the maximum of the softmax scores
for Charades and Charades-Ego. We report the computational cost of a single,
spatially center-cropped clip of size $256\times 256$.

####  4.2.1 Recognition Results

We compare the same baselines, as mentioned in the previous section. For
Kinetics-400, the input frames are the same as the previous section. For
Charades and Charades-Ego, we use $16\times 8$ input frames for ResNet3D and
$32\times 4$ input frames for SlowFast.

Kinetics-400. Table 5 shows the experiments on Kinetics-400. Our method can
improve top-1 accuracy by 1.6 and 1 point for ResNet3D and SlowFast,
respectively, at the cost of 3.6% and 2.1% relatively more computation. In
addition, we experiment with channel grouping with 8 groups (cg=8). The
improvement of accuracy is similar, but due to a smaller number of filters per
group, the computation cost is further reduced.

Models | mAP | GFLOPs | MParams
---|---|---|---
ResNet3D (16x8) | 0.354 | 218.4 | 32.40
ResNet3D + STAN | 0.377 | 226.4 | 32.47
SlowFast [7] (32x4) | 0.386 | 131.7 | 34.51
SlowFast + STAN | 0.406 | 134.5 | 34.53
Table 6: Experiment results on the Charades dataset. We show the model
performance on mAP, computation cost (in billions), and number of parameters
(in millions). The computational cost is of a single 256x256 video clip of
length 16x8 or 32x4 (number of frames x sampling rate) frames.

Charades. Table 6 shows the experiments on the Charades dataset. Our method is
able to improve mAP by 2.3 and 2 points for ResNet3D and SlowFast,
respectively, with only 3.6% and 2.1% relatively more computation. Our method
only contains very minimal parameters.

Models | 1st-person | 3rd-person
---|---|---
Baseline v1.0 [36] | 0.282 | 0.232
ResNet3D (16x8) | 0.298 | 0.361
ResNet3D + STAN | 0.318 | 0.366
SlowFast [7] (32x4) | 0.316 | 0.391
SlowFast + STAN | 0.326 | 0.396
Table 7: Experiment results on Charades-Ego dataset. Models are trained on
both 1st-person and 3rd-person view videos. We show the test set with 1st-
person and 3rd-person view videos separately. The computation cost and number
of parameters are the same as the Charades experiment.

Charades-Ego. Table 7 show the results on the Charades-Ego dataset. The test
set is divided into 1st-person videos and 3rd-person videos. Note that the
training set of Charades-Ego dataset includes mostly 3rd-person videos, and we
can observe from the results that our STAN model achieves more significant
improvement on the 1st-person test set, verifying the efficacy of our model’s
generalization ability. With SlowFast and STAN model, we are able to achieve
state-of-the-art performance on this dataset.

Figure 4: Qualitative analysis. Videos are from Charades. (a) and (c) are
original frames. (b) and (d) are transformed frames. The green bounding box
denotes a center reference. The sentences below the images are the ground
truth labels for the video clip. See main text for details of analysis.

####  4.2.2 Qualitative Analysis.

STAN can provide intuitive geometric interpretation of human actions. To
illustrate such effects, we visualize example transformations of our model on
Charades videos on two videos with correct classification labels in Fig. 4. We
use the output ($\theta$, Eqn. 2) from the STAN layer (deformation network) of
the first temporal group and visualize the spatial transformation using the
middle frame. Fig. 4 (a) and (c) are the original frames. Fig. 4 (b) and (d)
are the transformed frames using the transformation matrix predicted by STAN.
We use a center bounding box as a reference in the visualization. As we see,
for the first example, the person originally is located to the left of the
scene and STAN learns a transformation that centers the main actor. In the
second example, the person is already in the center and the transformation
does not alter much of the frames.

##  5 Conclusion

This paper has introduced a new spatial-temporal alignment network, STAN, for
action recognition and detection. Our study is the first to explore explicit
spatial-temporal alignment in 3D CNNs for action detection. Our model can be
conveniently inserted into existing networks and provides significant
improvement with a low extra computation cost. We have shown that our method
achieves state-of-the-art performance on multiple challenging action
recognition and detection benchmarks. We believe our models will facilitate
future research and applications on viewpoint-invariant feature representation
learning for actions. In the future, we would like to extend this work to the
optical flow stream and investigate how to learn the two-stream network more
accurately and more efficiently.

## References

  * [1] Liangliang Cao, Zicheng Liu, and Thomas S. Huang.  Cross-dataset action detection.  In CVPR, 2010.
  * [2] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman.  A short note on the kinetics-700 human action dataset.  arXiv preprint arXiv:1907.06987, 2019.
  * [3] João Carreira and Andrew Zisserman.  Quo vadis, action recognition? A new model and the kinetics dataset.  In CVPR, 2017.
  * [4] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the faster r-cnn architecture for temporal action localization.  In CVPR, 2018.
  * [5] Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis.  Graph-based global reasoning networks.  In CVPR, 2019.
  * [6] Kevin Duarte, Yogesh Rawat, and Mubarak Shah.  Videocapsulenet: A simplified network for action detection.  In NeurIPS, 2018.
  * [7] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, 2019.
  * [8] Christoph Feichtenhofer, Axel Pinz, and Richard P. Wildes.  Spatiotemporal residual networks for video action recognition.  In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, NeurIPS, 2016.
  * [9] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315–323, 2011.
  * [10] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al.  Ava: A video dataset of spatio-temporally localized atomic visual actions.  In CVPR, 2018.
  * [11] Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Mahowald, Rodney J Douglas, and H Sebastian Seung.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.  Nature, 405(6789):947–951, 2000.
  * [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [14] Rui Hou, Chen Chen, and Mubarak Shah.  Tube convolutional neural network (t-cnn) for action detection in videos.  In ICCV, 2017.
  * [15] Gary B. Huang, Marwan A. Mattar, Honglak Lee, and Erik G. Learned-Miller.  Learning to align from scratch.  In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, NeurIPS, 2012\.
  * [16] Linjiang Huang, Yan Huang, Wanli Ouyang, and Liang Wang.  Part-aligned pose-guided recurrent network for action recognition.  Pattern Recognition, 92:165–176, 2019.
  * [17] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [18] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In NeurIPS, 2015.
  * [19] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black.  Towards understanding action recognition.  In ICCV, 2013.
  * [20] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [21] Alexander Kläser, Marcin Marszalek, and Cordelia Schmid.  A spatio-temporal descriptor based on 3d-gradients.  In Mark Everingham, Chris J. Needham, and Roberto Fraile, editors, BMVC, 2008.
  * [22] Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton.  Stacked capsule autoencoders.  In NeurIPS, 2019.
  * [23] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre.  HMDB: a large video database for human motion recognition.  In ICCV, 2011.
  * [24] Ivan Laptev, Marcin Marszalek, Cordelia Schmid, and Benjamin Rozenfeld.  Learning realistic human actions from movies.  In CVPR, 2008.
  * [25] Ang Li, Meghana Thotakuri, David A Ross, João Carreira, Alexander Vostrikov, and Andrew Zisserman.  The ava-kinetics localized human actions video dataset.  arXiv preprint arXiv:2005.00214, 2020.
  * [26] Junwei Liang, Lu Jiang, and Alexander Hauptmann.  Simaug: Learning robust representations from simulation for trajectory prediction.  2020\.
  * [27] Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, and Alexander Hauptmann.  The garden of forking paths: Towards multi-future trajectory prediction.  In CVPR, 2020.
  * [28] Chen-Hsuan Lin and Simon Lucey.  Inverse compositional spatial transformer networks.  In CVPR, 2017.
  * [29] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In ICCV, 2019.
  * [30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [31] Junting Pan, Siyu Chen, Zheng Shou, Jing Shao, and Hongsheng Li.  Actor-context-actor relation network for spatio-temporal action localization.  arXiv preprint arXiv:2006.07976, 2020.
  * [32] Xiaojiang Peng, Limin Wang, Xingxing Wang, and Yu Qiao.  Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice.  Comput. Vis. Image Underst., 150:109–125, 2016.
  * [33] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu.  Learning human-object interactions by graph parsing neural networks.  In ECCV, 2018.
  * [34] Christian Schüldt, Ivan Laptev, and Barbara Caputo.  Recognizing human actions: A local SVM approach.  In ICPR, 2004.
  * [35] Gunnar A Sigurdsson, Santosh Divvala, Ali Farhadi, and Abhinav Gupta.  Asynchronous temporal fields for action recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 585–594, 2017.
  * [36] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari.  Charades-ego: A large-scale dataset of paired third and first person videos.  arXiv preprint arXiv:1804.09626, 2018.
  * [37] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In ECCV, 2016.
  * [38] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, NeurIPS, 2014.
  * [39] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [40] Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid.  Actor-centric relation network.  In ECCV, 2018.
  * [41] Graham W. Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler.  Convolutional learning of spatio-temporal features.  In Kostas Daniilidis, Petros Maragos, and Nikos Paragios, editors, ECCV, 2010.
  * [42] Yingli Tian, Liangliang Cao, Zicheng Liu, and Zhengyou Zhang.  Hierarchical filtered motion for action recognition in crowded videos.  IEEE Trans. Syst. Man Cybern. Part C, 42(3):313–323, 2012.
  * [43] Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, 2015.
  * [44] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In CVPR, 2018.
  * [45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [46] Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu.  Action recognition by dense trajectories.  In CVPR, 2011.
  * [47] Heng Wang and Cordelia Schmid.  Action recognition with improved trajectories.  In ICCV, 2013.
  * [48] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [49] Xiaolong Wang and Abhinav Gupta.  Videos as space-time region graphs.  In ECCV, 2018.
  * [50] Xuehan Xiong and Fernando De la Torre.  Supervised descent method and its applications to face alignment.  In CVPR, 2013.
  * [51] Huijuan Xu, Abir Das, and Kate Saenko.  R-c3d: Region convolutional 3d network for temporal activity detection.  In ICCV, 2017.
  * [52] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In ICML, 2015.
  * [53] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou.  Temporal pyramid network for action recognition.  In CVPR, 2020.
  * [54] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia Schmid.  A structured model for action detection.  In CVPR, 2019.
  * [55] Yue Zhao, Yuanjun Xiong, and Dahua Lin.  Trajectory convolution for action recognition.  In NeurIPS, 2018.