# Interpretable Deep Feature Propagation for Early Action Recognition

He Zhao, and Richard P. Wildes He Zhao is with the Department of Electrical
Engineering and Computer Science, York University, Toronto, ON, 30332.
E-mail<EMAIL_ADDRESS>Richard P. Wildes is with Department of Electrical
Engineering and Computer Science, York University, Toronto, ON.

###### Abstract

Early action recognition (action prediction) from limited preliminary
observations plays a critical role for streaming vision systems that demand
real-time inference, as video actions often possess elongated temporal spans
which cause undesired latency. In this study, we address action prediction by
investigating how action patterns evolve over time in a spatial feature space.
There are three key components to our system. First, we work with
intermediate-layer ConvNet features, which allow for abstraction from raw
data, while retaining spatial layout. Second, instead of propagating features
per se, we propagate their residuals across time, which allows for a compact
representation that reduces redundancy. Third, we employ a Kalman filter to
combat error build-up and unify across prediction start times. Extensive
experimental results on multiple benchmarks show that our approach leads to
competitive performance in action prediction. Notably, we investigate the
learned components of our system to shed light on their otherwise opaque
natures in two ways. First, we document that our learned feature propagation
module works as a spatial shifting mechanism under convolution to propagate
current observations into the future. Thus, it captures flow-based image
motion information. Second, the learned Kalman filter adaptively updates prior
estimation to aid the sequence learning process.

###### Index Terms:

Early action recognition, action prediction, motion analysis, Kalman
filtering, interpretability

##  1 Introduction

In many scenarios it is desirable to recognize actions that are being captured
in video as early as feasible, rather than await their completion. Autonomous
systems that need to interact with their environment in real-time can
especially benefit from such action prediction (_e.g_. autonomous vehicles and
interactive robots). Nevertheless, computational research in vision-based
early action recognition is limited in comparison to recognition based on
processing of entire action sequences. Action prediction shares many
challenges with action recognition based on full sequences, _e.g_. the need to
deal with viewpoint and performance variations as well as the fact that the
information about the actions per se often is mixed with distracting
information, _e.g_. clutter, camera motion, occlusion and motion blur.
Additional challenges present themselves for the case of prediction, _e.g_.
different action categories might share similar sub-components at different
stages (_e.g_. pushing and patting both start with stretching of arms), which
makes distinctions especially difficult when only partial information is
available. More generally, incomplete executions resulting from lack of
extended temporal context can lead to data that is not discriminative enough
for early classification.

Figure 1:  Overview of Proposed Feature Residual Propagation Approach to
Action Prediction. Intermediate layer ConvNet features are extracted from an
initial set of input frames; in the depicted example, these are given as
$[X_{t},X_{t+1},X_{t+2}]$; subsequent frames (_e.g_.
$[X_{t+3},\ldots,X_{t+7}]$ are not seen by the system during testing (although
they are during training) and are shown here merely for context. Inital
feature residuals, CNN Residuals, are extracted via pointwise differencing of
temporally adjacent feature maps. A generative model, Residual Generator
Network (RGN), then recursively estimates future residuals, P-Residuals.
Predicted features, P-Features, are recovered via addition of residuals to the
initial reference feature map. A Kalman Filter serves to minimize error
accumulation across time. The Kalman Filter operates across an entire video
sequence during training, but only across the initially observed partial
sequence during testing. Final action classification (not shown in figure) is
performed with reference to both the initially observed and predicted
features.

Action prediction often is formulated by transferring between full video
information and partial observations. Such approaches typically overlook the
rich motion patterns contained in videos, which has been demonstrated to play
a pivotal role in action recognition [1, 2, 3, 4]. With the recent success of
deep networks on action recognition (_e.g_. [5, 6, 7, 8]) deep representation
learning based approaches offer additional possibilities. For example, one can
design a temporally adaptive objective function that encourages the model to
produce the correct label as early as possible [9, 10]. Alternatively, one can
adopt a recurrent neural network to infer recursively the next features
conditioned on previous observations [11, 12]. However, the fact that such
approaches depend on the activation of fully-connected layers may compromise
performance, as the vectorized feature format collapses local information and
contains much more noise [13].

In response to the challenges noted above, we focus on exploring the subtle
changes in spatial features across time and propose a feature Residual
Generator Network (RGN) to propagate into the future. We choose intermediate
level activations of a pretrained deep network for propagation (_e.g_. final
ConvLayer output, _c.f_.,[14]), because features at such layers capture rich
spatial structures [15]. Rather than propagate the features per se, we
propagate feature residuals as they lead to a compact representation that
still captures essentials of how features change over time. To ameliorate
error accumulation over time, we incorporate a Kalman filter mechanism.
Empirical evaluation shows that our approach yields state-of-the-art
performance on three popular action prediction datasets. We also provide a
detailed analysis of the representations that have been learned by our system,
thereby providing interpretability. A preliminary description of this work has
appeared previously [16]. Most significantly, the current paper extends the
previous work by providing a detailed analysis of what has been learned by all
learning-based system components, something that was not considered in the
previous work. Also presented is a revised Kalman filter formulation that
better corrects predictions as well as new empirical results. Figure 1
provides a pictorial overview of the approach.

##  2 Related work

Action prediction. Early work on video-based action prediction concentrated on
use of handcrafted features. One such effort built temporally regularized
discriminative models for this purpose [17, 1]. Others proposed to solve a
posterior maximization on sparse feature encodings [2] or to enforce
consistency for varied temporal scales[4]. More recent work has focused on
deep learning. Some such work based prediction on action tubes over deep
detectors [18, 19]. In other work, a ConvNet with an LSTM was used to define a
temporally adaptive objective function to assign labels as early as possible
[9]. An alternative approach learned mappings between semantic features of
full and partial videos [20, 21], which was extended with an LSTM to handle
hard samples for improved performance [10]. By concentrating on relatively
high-level semantic features, these approaches tend to overlook more
temporally local information. To compensate for this potential shortcoming,
yet other work has generated sequential features based on current observations
[22, 11, 12]. Knowledge distillation is another approach that has shown
improvements on action prediction [23]. Moreover, some recent efforts focus on
discovering hard-to-discriminate details [24] or exploring the relationship of
multiple instances in action videos [25]. Finally, a lone effort has extended
single action prediction to group action prediction [26].

Dynamically adaptive filters. Convolution has became the standard operation
for a wide variety of vision tasks, from low level image processing (_e.g_.
denoising) to high level semantic problems (_e.g_. recognition). Standard
convolutional approaches learn a set of fixed filters from a training set and
apply them uniformly to all test data. Recent efforts have found success by
conditioning the convolutional filters on input test images [27, 28, 29, 30,
31] for a variety of vision problems. Our work follows a similar idea to
dynamically infer motion kernels from historical observations for temporal
feature propagation, as action videos often exhibit diverse (sometimes even
contrastive) movements that could be difficult for fixed filter banks.

Intermediate features. Recent work has shown solid benefits from explicitly
exploiting intermediate layer features in a multi-layer network. As examples:
Intermediate features have been used for local frame aggregation [32],
building compact feature correlations [33, 34], spatial warping for real-time
recognition [14], recovering images from various deep abstraction stages [35]
and modulating information of distinct network branches [36, 8]. The positive
results these approaches have yielded may be explained by the fact that in
comparison to fully-connected layers, intermediate layers preserve more
spatial structure and thereby support finer distinctions (_e.g_. in motion
layout) as well as have fewer parameters and thereby combat overfitting. For
these reasons, we build on intermediate layer features in our work on action
prediction.

Residual processing. Residual information can play an important role in
processing of redundant data even while capturing important subtle differences
in an efficient fashion. MPEG-4 compression is a well established outstanding
example of such processing [37], as is more general coarse-to-fine motion
estimation (_e.g_. [38]). Recent work that exploits residual processing has
considered optical-flow estimation [39], image denoising [40], video artifact
removal [41] and action recognition [42]. Our approach to action prediction
provides a novel use of residual processing.

Deep Kalman filtering. Temporal models are useful tools for video
understanding; the Kalman filter [43] is one such example, which is well known
for sequence modelling. It combines the idea of data assimilation with state-
space representations and recently has been adapted with deep networks for
video action detection [44], video prediction [45], tracking [46] and temporal
regularization [47]. Yet, existing work lacks sufficient understanding of why
deep Kalman structure helps. Our work also uses a learnable Kalman filtering
to assist the sequence training process and we provide detailed analysis on
the working mechanism of the deep Kalman filter. In particular, we demonstrate
that it operates as an adaptive version of the schedule sampling training
technique.

Interpretable deep learning. A variety of techniques have been developed for
understanding the operations and representations that are learned by deep
networks, which otherwise remain opaque. Early work resorted to visualizing
what has been learned to interpret deep networks, e.g., by showing the learned
convolutional kernels [15, 48]. A parallel line of research focuses on
visualizing the heat map relating the input image and output activations [49,
50]. Some others choose to fit traditional filters (_e.g_. Gabor filters [51])
to the deep framework and show that the training results in these traditional
filters being learned automatically [52, 53]. Other recent efforts emphasize
dissecting deep models by either identifying the functionality of each neuron
[54] or unrolling layers into consecutive single units [55]. Our work combines
visualization with analytic modelling. We visualize learned components,
provide analytic interpretations and further empirically evaluate these
interpretations to show the correlation between learned warping kernels and
image motion information. As noted above, we also provide an explanation of
our deep Kalman filter’s operation.

##  3 Technical approach

###  3.1 Overview

We seek to predict the correct action label, $y$, given the initial portion of
a partially observed video, $X_{1:k},$ where $k$ represents the $k$th frame of
a video that in total has $K$ frames. The key ingredient in support of our
goal is an effective approach for propagating the information contained in
initially observed consecutive frames $X_{1:k}$ to unobserved $X_{k+1:K}$. The
video action label, $y$, is then recovered via classification of the entire
concatenated sequence $X_{1:K}=Cat\\{X_{1:k},X_{k+1:K}\\}$. Follow existing
methods, we define the term observation ratio, $g$, as the fraction of the
observed frame set, $X_{1:k}$, to the full set, $X_{1:K}$. We present results
from experiments with $g\in[0.1,1.0]$.

Rather than predict future frames per se, we instead predict intermediate
layer features of a ConvNet trained for action recognition. We are
particularly interested in intermediate layer features, because features at
such layers enjoy a level of abstraction from the raw data that focuses on
action relevant components, even while preserving spatial layout to capture
relations between action components as well as scene context.

We decouple the prediction process into two steps: feature residual
propagation and feature reconstruction. As discussed in Section 2, feature
residual information previously has been used as a convenient proxy for full
data observations as it retains salient changes to objects and motions, even
while reducing redundancy entailed in explicit representation of non-changing
portions of observed data. Here, we advance the application of residual
extraction and processing in the domain of ConvNet features to yield a novel
framework for action prediction.

For illustrative purposes, we use the TSN architecture for initial feature
extraction and final classification, because of its generally strong
performance on action recognition [5]. While we use the TSN features and
classifier, our prediction does not rely on the specifics of that approach and
therefore should be more widely applicable to action prediction.

###  3.2 Feature residuals

Given a partially observed video with a set of frames $X_{1:k}$, let (mid-
level) features extracted at time $t$ be denoted as
$\mathbf{d}_{t}\in\Re^{C\times W\times H}$, with $C$ the number of feature
channels, $W$ the feature map width and $H$ the feature map height. Temporal
feature residuals at time $t$ are then calculated via pointwise differencing
along each channel

$\mathbf{r}_{t}\big{\rvert}_{c}=\mathbf{d}_{t}\big{\rvert}_{c}-\mathbf{d}_{t-1}\big{\rvert}_{c},\hskip
10.0pt2\leq t\leq k,1\leq c\leq C$ (1)

where $\big{\rvert}_{c}$ indicates application to channel $c$, _i.e_. the
value at spatial position $(w,h)$ in channel $c$ at time $t-1$ is subtracted
from the value at time $t$ and assigned to the residual,
$\mathbf{r}_{t}\in\Re^{C\times W\times H}$, at the same spatial position and
channel. Owing to the differencing operation, the cardinality of the set of
calculated residuals, $\\{\mathbf{r}_{2:k}\\}$, is one less than the set of
features, $\\{\mathbf{d}_{1:k}\\}$.

From the limited feature set $\\{\mathbf{d}_{1:k}\\}$ and their residuals set
$\\{\mathbf{r}_{2:k}\\}$, we seek to recover the feature representation of
$\\{\mathbf{d}_{k+1:K}\\}$. To achieve this result, we proceed in two steps.
First, we recursively generate feature residuals $\\{\mathbf{r}_{k+1:K}\\}$
via appeal to a feature Residual Generator Network (RGN). Second, we
sequentially add the residuals to the features that have been observed or
generated so far to reconstruct features into the future according to

$\mathbf{d}_{t+1}=\mathbf{d}_{t}+\mathbf{r}_{t+1},\hskip 10.0ptk\leq t\leq
K-1.$ (2)

In Figure 1, P-Residuals and P-Features are used to distinguish predicted
residuals and features, resp. In the next subsection, we define our feature
residual generator.

Figure 2:  Temporal Extrapolation Residual Generator Network (RGN) for
Predicting Next Time Step Residual. Our model recursively generates motion
kernels, $K_{n}$, using a ConvNet, $\mathcal{G}$, based on a short historical
temporal window, $m=3$, and performs transformations in a convolutional
fashion on the most recent residual. The newly generated residual joins the
ongoing prediction sequence until the end of the desired sequence. Subscripts
$n$ specify kernel size (_i.e_. $n\times n$). Convolutions on the residuals
are performed on a channel-by-channel basis; so, multiple kernels are depicted
for each $n$.

###  3.3 Residual Generator Network (RGN)

Our Residual Generator Network (RGN) is schematized in Figure 2. At its core
is a kernel motion transformation, $\mathcal{G}$. Given a set of stacked
temporal observations, $\mathcal{G}$ produces a set of kernels, $\\{K_{n}\\}$,
that can be convolved with the most recent residual input to predict the next
(unobserved) result. We choose the kernel motion transformation because it has
proven useful in synthesis of future intensity frames [56, 57], can be applied
with various kernel sizes, $n\times n$, to capture multiple motion scales and
has lower complexity than its deep regression counterpart [12].

We generate motion kernels for each channel, $c$, with multiple sizes,
$n\times n$, according to

$K_{n}=\mathcal{G}(\mathbf{r}_{t},\mathbf{r}_{t-1},\ldots,\mathbf{r}_{t-m}\,|\,\mathbf{r}_{t-m-1},\ldots,\mathbf{r}_{2};\theta_{f})\big{\rvert}_{c},$
(3)

where $\mathcal{G}$ is a ConvNet with learnable parameters, $\theta_{f}$, that
inputs residuals over its current observation window, $m$, but through its
recurrent application depends on the entire history of residuals and thereby
follows the Markov-Chain conditional distribution. Each of these kernels is
normalized via the $l_{2}$ norm to avoid abnormal temporal changing,
_c.f_.,[56]. The architecture of $\mathcal{G}$ is depicted in Figure 2, with
implementation details provided in Section 4.2. Subsequent to kernel
generation, for each channel, $c$, we apply the kernels to the current
residual $\mathbf{r}_{t}$ and average the results to predict the next time
step residual

$\mathbf{r}_{t+1}\big{\rvert}_{c}=\dfrac{1}{N}\sum_{n=1}^{N}K_{n}\circledast\mathbf{r}_{t}\big{\rvert}_{c},$
(4)

where $\circledast$ stands for convolution. Based on preliminary experiments
we use $N=3$, with $n\in\\{3,5,7\\}$.

Figure 3:  Depiction of Kalman update procedure. Prior estimation of feature
$\hat{\mathbf{d}}_{t}^{-}$ is updated with Kalman gain $\Gamma_{t}$. The
transition of $\Gamma_{t}$ is modeled by a ConvNet with LSTM
($\Gamma_{t}$-LSTM) across time. At each time step, $\Gamma_{t}$ corrects
$\hat{\mathbf{d}}_{t}^{-}$ with observed measurement $Z_{t}$ and produces
posterior $\hat{\mathbf{d}}_{t}^{+}$ for next time step inference.

###  3.4 Kalman filter correction

Recent approaches to sequential feature generation prefer decomposing multi-
step prediction into single-step prediction for training and apply the same
model recursively for testing. Owing to error accumulation, such approaches
often lead to quality degeneration as the sequence becomes longer. Current
time-series optimization methods (_e.g_. BackPropagation Through Time (BPTT))
lack the ability to inject mid-stage supervision during optimization; thus,
errors in initial stages negatively impact the following results. To avoid
such scenarios, we incorporate a Kalman filter [43] into our approach,
_c.f_.,[47, 41]; see Figure 3.

The Kalman filter recursively estimates an internal state from a time series
of measurements via alternating Predict and Update steps along the temporal
axis. In our case, the internal state corresponds to the features recovered
from the predicted residuals according to (2), while Predict is formulated as
the RGN defined in Section 3.3 and Update is formulated as

$\hat{\mathbf{d}}_{t}^{+}=\hat{\mathbf{d}}_{t}^{-}+\Gamma_{t}(Z_{t}-\hat{\mathbf{d}}_{t}^{-}),$
(5)

where $\hat{}$ distinguishes the estimated as opposed to groundtruth values,
$Z_{t}$ is the real measurement at time $t$ (groundtruth framewise feature),
$\mathbf{d}_{t}^{-}$ is the prior estimate, (2), and $\Gamma_{t}$ is the
Kalman Gain. The Update corrects the current prediction by balancing the
observed measurement, $Z_{t}$, and prior estimate, $\hat{\mathbf{d}_{t}^{-}}$,
and the gain, $\Gamma_{t}$, defined as

$\Gamma_{t}=\psi(Z_{t-1}-\hat{\mathbf{d}}_{t-1}^{+};\theta_{z}).$ (6)

In the classic definition, the Kalman gain is estimated from a ratio of the
process and measurement noise, both of which are pre-defined by prior
knowledge of physical aspects of the system being modelled (e.g.,
environmental factors, like air resistance). Such noise factors can be
considered as the source of prediction errors and are readily modelled under
Gaussian assumptions.

However, this design is not feasible in our work, as the size of the action
states is too large, _i.e_. every feature point is viewed as an individual
state variable and the employed features are large in size, $(28,28,192)$.
Further, it is difficult to obtain useful prior knowledge of errors when using
deep networks. Instead, we treat the Kalman gain as the output of a nonlinear
function, $\psi$, of the difference between the real and predicted features,
$Z_{(t-1)}-\hat{d}_{(t-1)}^{+}$, which naturally reflects the prediction
error. We realize $\psi$ as a ConvNet with an LSTM and learnable parameters,
$\theta_{z}$, cf. [47]. The architecture is depicted in Figure 3, with details
provided in Section 4.2. Note that this specification of the Kalman gain
differs from that in our earlier work [16], which more simply directly input
both $Z_{(t-1)}$ and $\hat{d}_{(t-1)}^{+}$ without differencing, to the
ConvNet and thereby more poorly captured the desire for the update to be
conditioned on the prediction error. This new Kalman gain is named KF-2 in the
following.

We explicitly incorporate the Kalman filter Update step into the training of
the RGN, where correction happens after the estimate of
$\hat{\mathbf{d}}_{t}^{-}$ is obtained, as depicted in Fig. 3. The corrected
feature $\hat{\mathbf{d}}_{t}^{+}$ is subsequently used for $t+1$ prediction
and loss computation thereafter. During training, the Kalman filter has access
to true observations, $Z_{t}$ throughout the video. In testing, however, the
Kalman filter only has access to true observations up through the final input
partial observation, $X_{k}$, and is only applied through that point, as
detailed in Section 3.6. We find that the instantaneous correction offered by
the Kalman filter helps stablize long-term inference, as documented in Sec.
4.4.

###  3.5 Learning scheme

In our approach, there are two sets of trainable parameters, $\theta_{f}$ and
$\theta_{z}$, that are associated with the kernel motion generator,
$\mathcal{G}$, of the residual generative network and the Kalman gain
transition, $\psi$, resp. Both sets of parameters are trained using
backpropagation to minimze loss objective functions. We adopt a two stage
training strategy that initially learns the $\theta_{f}$ values and
subsequently learns the $\theta_{z}$ values, while also refining the
$\theta_{f}$ values. We first train $\theta_{f}$ because it is more central to
our overall approach in performing the essential prediction, rather than the
correction. This design choice conforms to the standard Kalman filter paradigm
that presupposes a sane transition module and a corrective module built on
rational prior estimates [43]. Nevertheless, ultimately the prediction and
correction must work together; so, $\theta_{f}$ and $\theta_{z}$ are trained
jointly in our second stage.

The parameters $\theta_{f}$ are optimized with respect to four losses. The
first loss pertains to the residuals

$\mathcal{L}_{2}^{res}(\theta_{f})=||\mathbf{r}_{t}-RGN(\mathbf{r}_{t-1},\mathbf{r}_{t-2},...,\mathbf{r}_{t-m};\theta_{f})||^{2}_{2}$
(7)

where $m$ is the temporal window size. (In (7), note that $\mathcal{G}$ is
embedded in $RGN$, but here we suppress the recursive dependence on all
previous residuals beyond the current observation window that was given in (3)
for the sake of compactness of notation.) The second loss pertains to the
features

$\mathcal{L}_{2}^{feat}(\theta_{f})=||\mathbf{Z}_{t}-\hat{\mathbf{d}}_{t}^{-}||^{2}_{2}=||\mathbf{d}_{t}-(\hat{\mathbf{d}}_{t-1}+\hat{\mathbf{r}}_{t})||^{2}_{2}.$
(8)

As reported elsewhere [58, 59, 60], $\mathcal{L}_{2}$ works under the Gaussian
assumption that data is draw from a single parameterized Gaussian distribution
and thus produces blurry outcomes. To counter this shortcoming, we include an
additional two losses by applying the Gradient Difference Loss [58], which
emphasizes high frequency content, on both the features and residuals to yield

$\mathcal{L}_{gdl}^{res}(\theta_{f})=||\frac{\partial}{\partial
x}\left(\mathbf{r}_{t}-\hat{\mathbf{r}}_{t}\right)||^{2}_{2}+||\frac{\partial}{\partial
y}\left(\mathbf{r}_{t}-\hat{\mathbf{r}}_{t}\right)||^{2}_{2}$ (9)

and

$\mathcal{L}_{gdl}^{feat}(\theta_{f})=||\frac{\partial}{\partial
x}\left(\mathbf{Z}_{t}-\hat{\mathbf{d}}_{t}^{-}\right)||^{2}_{2}+||\frac{\partial}{\partial
y}\left(\mathbf{Z}_{t}-\hat{\mathbf{d}}_{t}^{-}\right)||^{2}_{2}.$ (10)

The overall objective function for $\mathcal{G}$ is defined as

$\mathcal{L}_{2}^{\mathcal{G}}(\theta_{f})=\lambda_{1}\mathcal{L}_{2}^{res}+\lambda_{2}\mathcal{L}_{2}^{feat}+\lambda_{3}\mathcal{L}_{gdl}^{res}+\lambda_{4}\mathcal{L}_{gdl}^{feat},$
(11)

with the $\lambda_{i}$ scalar weighting factors. Note that during the first
stage of training, the Kalman filter would not be operating, as it has yet to
be trained.

After training the RGN parameters, $\theta_{f}$, the Kalman gain parameters,
$\theta_{z}$, are trained, while the $\theta_{f}$ parameters values are
refined to yield a joint optimization. Now, there are only two losses, both
pertaining to the features, $\mathbf{d}$, because that is where the Kalman
filter operates. The losses are analagous to (8) and (10), except that they
are calculated on the updated posterior $\hat{\mathbf{d}_{t}^{+}}$ according
to

$\mathcal{L}_{2}(\theta_{f},\theta_{z})=\mathbf{\alpha}\mathcal{L}_{2}^{feat}(\hat{\mathbf{d}_{t}^{+}};\theta_{f},\theta_{z})+\mathbf{\beta}\mathcal{L}_{gdl}^{feat}(\hat{\mathbf{d}_{t}^{+}};\theta_{f},\theta_{z}),$
(12)

with $\alpha$ and $\beta$ scalar weighting factors.

###  3.6 Unified model for all observation ratios

Learning a separate model for each observation ratio is not applicable in the
real world. To overcome this difficulty, we design a unified training and
testing strategy, as follows.

Training. The RGN begins by inputting the very first batch of residuals
$[\mathbf{r}_{m},\mathbf{r}_{m-1},\ldots,\mathbf{r}_{2}]$ and recursively
produces all the rest. In other words, our model is trained for predicting the
whole sequence from the same starting point, thereby entirely ignoring
observation ratios.

Testing. Our testing also is invariant to observation ratio by switching modes
of the Kalman filter operation so that it only corrects the estimates while
data observations are available according to $g$. For example, when $g=0.6$,
the proposed approach still starts from the beginning observations and
propagates to the end, but in two modes: While the observation ratio is not
yet reached, _i.e_. $g\in[0.1,0.6]$, we update predictions via reference to
the observed true data by using the Kalman filter update step, (5). After
entering $g\in[0.7,1.0]$, only prediction is performed, (4).

This procedure resembles tracking objects under a Kalman filter: When objects
are observed, the system corrects its estimated coordinates based on true
observation measurements; however, while objects are occluded, the system
extrapolates possible locations based on “up-to-now” system parameter values,
_i.e_. only the prediction step is performed.

0.00.10.20.30.40.50.60.70.80.9Observation Ratio0102030405060708090Recognition
Accuracy %AAPnet [21]C3D [6]DeepSCN [20]C3D+Linear SVM [20]C3D+IKSVM
[20]C3D+Chi-Square SVM [20]Integral BOW [1]Dynamic BOW [1]MTSSVM [4]MSSC [2]AE
[61]RGN-KF [16]RGN-KF-2

(a) UCF-101 dataset

0.00.10.20.30.40.50.60.70.80.9Observation Ratio0102030405060708090Recognition
Accuracy %AAPnet [21]DeepSCN [20]C3D+Linear SVM [20]C3D+IKSVM [20]C3D+Chi-
Square SVM [20]Integral BOW [1]Dynamic BOW [1]MTSSVM [4]MSSC [2]AE [61]SC
[2]RGN-KF [16]RGN-KF-2Baseline

(b) BIT dataset

Figure 4:  Action Prediction Results on the UCF101 and BIT Datasets at all
Observation Ratios $g\in[0.1,1]$.

##  4 Empirical evaluation

###  4.1 Datasets and experiment protocol

To evaluate our approach, we choose three widely examined datasets, UCF101
[62], JHMDB21 [63] and BIT [64]. UCF101 consists of 13,320 videos of 101
action categories containing a wide range of activities (_e.g_. sports, music
and others). JHMDB21, a subset of HMDB [65], contains 928 videos of 21
realistic, nuanced human action categories (_e.g_. catching, throwing,
picking). We use the provided RGB images rather than body joints of JHMDB21.
BIT consists of 8 classes of human interactions, with 50 videos per class.
Different from the other datasets, BIT has similar behaviors of people in the
initial stage of different actions (_e.g_. they tend to be standing still)
[21], which leads to challenges from limited discriminatory information.

For all datasets, we use their standard train/test splits: UCF101 and JHMDB21
come with multiple train/test splits and we average over the results in our
reporting, unless otherwise noted; BIT has a single train/test split, with the
first 34 videos in each class for training and the rest for testing.

We present action classification accuracy as a function of observation ratio,
$g$, which is the ratio of observed to total frames in a video, as used
elsewhere [4]. Classification is always based on the concatenation of features
derived from the observed frames and those that are predicted. For mid-layer
features, which are the subject our propagation, we use the intermediate
output of two convolutional layers and two max-poolings $\in\Re^{28\times
28\times 192}$, unless otherwise noted. This layer is selected because
empirical comparison to others generally yielded superior performance; see
Section 4.5. Beyond the results presented in this section, additional detailed
results are provided in the supplement.

###  4.2 Implementation details

To examine the propagation module with minimal influence from other factors,
classifiers for chosen datasets are obtained beforehand. While a pretrained
TSN model is available for UCF101 [5], models for JHMDB21 and BIT are not
available. To adapt the TSN model to the JHMDB21 and BIT datasets, we append a
simple two layer MLP classifier consisting of two hidden layers to TSN
pretrained for HMDB-RGB and UCF101-Flow. For JHMDB21, the two hidden layers
have 32 and 21 activations. For BIT, the two hidden layers have 64 and 8
activations. Softmax is used for final probability generation in all cases.
During the training process all pretrained weights are frozen. For training of
weights added for adaptation to JHMDB21 and BIT, we randomly select 3 RGB
samples or 3 Optical Flow samples (each sample has 5 frames) from the videos
and get video labels by segment consensus. We employ a learning rate of
0.0001, batch size of 64, Stochastic Gradient Descent and the Adam optimizer.
Data augmentation is the same as for the original TSN [5].

Network configurations. For the kernel generator of the RGN, $\mathcal{G}$,
stacked residuals are first convolved with a $1\times 1$ kernel that reduces
the feature dimension. Then, two residual convolutional blocks [66] with
kernel size $3\times 3$, bottleneck dimension 48 and stride 2 are used to
capture temporal evolution. Subsequently, with batch and channel axis fixed,
flattened spatial features are individually processed with 3 FC layers to
produce $3\times 3$, $5\times 5$ and $7\times 7$ kernels. So the shape of
feature map is (28, 28, 192$\times m$)-(28, 28, 192)-(28, 28, 192)-(28, 28,
192)-(9, 192), (25, 192) and (49, 192), with $m=3$ the empirically selected
temporal window size. Convolution is performed on each channel.

For Kalman Gain, $\Gamma_{t}$, a set of convolutional layers with kernel size
3x3 and stride 2 are used to capture the covariance. Each layer is appended
with a ReLU layer [67]. The shape of feature map is (28, 28, 128)-(28, 28,
64)-(28, 28, 32)-(28, 28, 1). Subsequently, the flattened feature is taken as
input by $\Gamma_{t}$-LSTM to produce Kalman gain, $\Gamma_{t}\in\Re^{wh}$,
which then is reshaped to $\Gamma_{t}\in\Re^{w\times h}$, corresponding to
feature map spatial dimensions. The hidden state of the LSTM has the same size
as the input feature (784). The gain is then applied according to the update,
(5).

Training strategy. We train our model with 4 NIVDIA TITAN X GPUs, under
Pytorch [68]. Training of the spatiotemporal feature residual generative
network (RGN) employs the Adam optimizer and a learning rate 0.005 with
$\beta_{1}=0.9$ and $\beta_{2}=0.99$ to minimize the loss, (11). Empirically,
we set $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ with ratios of
1:1:5:5, which places more emphasis on the spatial gradient rather than raw
mean square values. The batch size is set to 56. Following initial training of
the RGN, we fine-tune it together with the Kalman gain transition ConvNet with
LSTM, $\psi$, to minimze the loss (12). Mini-batch-Stochastic Gradient Descent
is used with a learning rate of $2e^{-4}$ and exponential decay of $1e^{-5}$.
$\alpha$ and $\beta$ are set empirically with a ratio of 1:5.

For training on UCF101, we sample 30 frames from each video and use the first
3 to initialize our entire prediction system. For BIT and JHMDB21, we sample
25 frames from each video and use the first 3 to initialize our system. The
TSN architecture [5] serves to provide feature extraction and classification.
We apply our system to the RGB stream for JHMDB21, flow stream for BIT and
both streams for UCF101. We make these choices following typical state-of-the-
art practice on JHMDB21 (_e.g_. RGB features previously yielded top
performance [9, 11]), BIT (_e.g_. flow features greatly outperform spatial
features [64, 20]) and UCF101 (_e.g_. two-stream previously yielded top
performance [21]).

Once features are generated, no additional modifications to TSN are needed to
yield action labels. Generated features are inserted into the selected TSN
mid-layer and processed up through the network tower until the MLPs produce
probability scores. Video level labeling is gathered by averaging scores from
each frame.

###  4.3 Overall prediction results

UCF101. Figure 4 $(a)$ shows comparative results for our algorihm RGN-KF-2
_vs_. various alternatives on UCF101 as well as our earlier version, RGN-KF
[16]. It is seen that both our approaches, RGN-KF-2 and RGN-KF, outperform all
others at all observation ratios, improving accuracy by $\approx$ 3-4% on
average. The performance improvement is especially striking at lower
observation ratios, _e.g_. $g=0.1$, where we outperform the second best
(AAPnet) by 83.78% _vs_. 59.85%. When being compared with our earlier work,
RGN-KF, our new method uniformly surpasses it on low observation ratios,
_e.g_. $g\in(0.1,0.2,0.3)$, with the largest improvement at $g=0.2$ (86.22%
_vs_. 85.16%). Beyond that, RGN-KF-2 performs equally well. Since the task is
early action recognition, improvements at the lowest observation ratios are of
particular interest.

Notably, AAPnet also builds on TSN; however, it apparently does so less
effectively than our approach does. There are likely two reasons for this
state of affairs. First, AAPnet is not trained exclusively for inferring
action labels, but also for adversarial learning on feature transformation,
which might lessen its optimization for action prediction. Second, AAPnet more
radically modifies the TSN architecture in aggregating across all frames at a
given state of progress, which underlines the fact that our approach may be
more adaptable to various architectures as it has less impact on their native
operations.

BIT. Figure 4 $(b)$ shows comparative results for our algorithm _vs_. various
alternatives on BIT. It is seen that our results are equal to or better than
all others, except at the lowest observation ratio, $g=0.1$. For example,
compared with AAPnet, our approach achieves 69.12% accuracy at $g=0.3$, which
is 4% higher. Notably, our RGN-KF-2 obtains non-trivial improvements over RGN-
KF on multiple lower ratios, _e.g_. $g\in(0.1,0.2,0.3)$. Particularly at
$g=0.2$, our new approach boosts the accuracy from 46.09% to 49.96%, which
demonstrates the solid benefit of explicitly modelling the Kalman gain from
error signals, as given by, (6). Again, improvements at lowest observation
ratios are of particular interest for early action recognition.

In interpreting the results on BIT it is important to recall that the
beginning and ending portions of the videos tend be very similar in appearance
(_e.g_. two people standing facing one another), so that the most
discriminatory information largely is limited to the intermediate portions.
Correspondingly, there is a tendency for rapid performance rises after the
initial portion, which levels out in the final portion. In our case, a peak
performance of 92.28% at $g=0.7$ increases that at the previous ratio by 4%,
whereas AAPnet achieves no significant increase (0.78%) at the same stage.

Given that we train a modified TSN architecture in adapting TSN to BIT
(Section 4.2), we compare how well that modified architecture works when
forced to classify on just the initially provided frames without propagation
into the future. These results are shown as Baseline in Fig. 4 $b$. It is seen
that by propagating into the future our approach exceeds the baseline by large
margins when $g\in[0.1,0.4]$. For higher observation ratios, as the
discriminating parts of the input videos become available to the baseline (as
well as our full approach), performance becomes on par.

JHMDB21. The standard reporting protocol on JHMDB21 is to report recognition
accuracy only when the initial 20% of the videos are observed, _i.e_. $g=0.2$,
which we show in Table I. It is seen that our algorithm once again is the top
performer, _e.g_. RGN-KF-2 exceeding the RBF-RNN by 6% and RGN-KF by 1%. We
also provide a baseline comparison, where we compare to classification made
purely on the basis of adapting the TSN architecture to the JHMDB21 dataset,
analogous to the baseline comparison provided on BIT. Once again, it is seen
that our full propagation approach adds considerably to the performance of the
baseline alone.

Method | Accuracy (%)
---|---
ELSTM [9] | 55
Within-class Loss [69] | 33
DP-SVM [70] | 5
S-SVM [70] | 5
Where/What [71] | 10
Context-fusion [72] | 28
RBF-RNN [11] | 73
RGN-KF [16] | 78
RGN-KF-2 | 79
Baseline | 74
TABLE I: Action Prediction Results on JHMDB21. Following the standard protocol, accuracy results are shown only for the case where initial observations are limited to the first 20% of frames, _i.e_. $g=0.2$.  Temporal Propagation Approach | Accuracy (%)
---|---
ConvLSTM(3x3)-128-192-Org | 71.1
ConvLSTM(3x3)-128-192-Org-KF | 73.4
ConvLSTM(3x3)-128-192-Org-KF-2 | 74.3
ConvLSTM(3x3)-128-192-Res | 76.8
ConvLSTM(3x3)-128-192-Res-KF | 77.1
ConvLSTM(3x3)-128-192-Res-KF-2 | 77.3
RGN-Org | 70.9
RGN-Org-KF | 74.4
RGN-Org-KF-2 | 74.7
RGN-Res | 77.4
RGN-Res-KF | 78.3
RGN-Res-KF-2 | 79.0
TABLE II:  Accuracy results for different temporal propagation approaches on JHMDB21 split 1. Org denotes applying motion kernel transformation on original features, Res denotes residual propagation and KF* denotes inclusion of the Kalman filter. For ConvLSTM, (3x3), 128 & 192 represent kernel, hidden state & feature dimension, resp. | UCF-101 | BIT | JHMDB-21
---|---|---|---
observation ratio | 0.1 | 0.2 | 0.3 | 0.8 | 0.1 | 0.2 | 0.3 | 0.8 | 0.2
56x56x64 | 83.16 | 85.03 | 87.92 | 92.10 | 33.41 | 46.65 | 67.51 | 84.38 | 75.83
28x28x192 | 83.78 | 86.22 | 89.01 | 93.07 | 37.33 | 49.96 | 69.12 | 88.82 | 79.00
14x14x576 | 83.29 | 85.46 | 88.19 | 93.06 | 36.62 | 48.70 | 68.19 | 87.50 | 77.63
7x7x1024 | 80.51 | 84.94 | 88.17 | 92.77 | 36.12 | 49.22 | 67.61 | 87.50 | 77.42
TABLE III:  Prediction Accuracy (%) at Various Intermediate Feature Stages
with RGN-KF-2, Ordered by Decreasing Spatial Receptive Field Size. Observation
ratio $g\in\\{0.1,0.2,0.3,0.8\\}$ for UCF101 and BIT datasets. Set $g=0.2$ for
JHMDB21.

###  4.4 Influence of temporal model

In this section, we examine the influence of different temporal modeling
approaches to feature propagation using JHMDB21, with ConvLSTM as an extra
baseline, cf. [73, 60]; see Table II. For both scenarios, we find that
propagation on residuals is superior to propagation on raw features and the
Kalman filter provides further benefits. Performance of ConvLSTM is on par
with our RGN approach applied to the original features without the Kalman
filter; however, for all other configurations our RGN approach performs
better. Especially, our new version of the Kalman filter, KF-2, seems to
benefit all used temporal models. Overall, we find that our full approach to
temporal modeling (mid-layer convolutional feature residuals, RGN propagation
and Kalman filtering) yields best performance.

###  4.5 Influence of feature layers

We now examine the influence of different intermediate feature spaces on
prediction. We consider layers that yield feature maps of [56, 56, 64], [28,
28, 192], [14, 14, 512] and [7, 7, 1024], where [w, h, c] indicate the width,
height and number of channels, resp. Table III shows the results. For JHMDB21
and BIT, the [28, 28, 192] feature stage almost always achieves best results.
Moreover, deeper layers, [14, 14, 512] and [7, 7, 1024], are more useful than
the shallower layer [56, 56, 64]. This pattern of results may be explained by
the earliest layer not providing adequate abstraction from the raw input,
while the highest layers have lost too much distinguishing detail.
Interestingly, for UCF101 different feature stages have less impact on
accuracy. This may be due to the fact that UCF101 is generally less difficult
than the other datasets, as indicated by the fact that for any given
observation ratio, $g$, in Table III the results on UCF101 are always better
than for the others; correspondingly, the specifics of feature selection are
less important. More generally, however, the results of Table III support our
use of intermediate layer features, especially as the prediction task becomes
more difficult.

Figure 5:  Visualization of Features and Residuals. The example shows a
sequence of frames for the action baseball-swing along the left side. Feature
maps extracted at various layers are shown along the upper/bottom row, while
their residuals are shown along the middle row. Figure 6:  Sparsity Comparison
of Features and Residuals. The top row shows frames from a video of a kicking
action. The bottom row shows sparsity as the ratio of near-zero value points
(absolute value $<0.01$) to total points over time. On average, residual and
original feaure points are 65% and 14% sparse, resp.

###  4.6 Visualization of feature residuals

To understand further why intermediate layer features and their residuals are
especially useful for action prediction, we show comparative visualizations as
well as associated statistics. Figure 5 provides an example from the action
baseball-swing. It is seen that the earliest layer features concentrate on
low-level features (_e.g_. lines and edges) that may be too closely tied to a
specific example, rather than the action class. In contrast, the latest layer
features tends to lose too much distinguishing detail (_e.g_. merely a blob in
the vicinity of the actor at the top-layer). Comparatively, the mid-layer
features tend to concentrate on the actor, but also delineate details of the
actors parts. In comparing the raw features to their residuals, it is seen
that the residuals concentrate more on temporal changes, which are good for
propagating variations into the future without redundant information. Thus,
intermediate layer residuals appear to capture information that is especially
useful for action prediction.

The provided visualization, Figure 5, suggests that the residuals provide a
more sparse (and hence compact) representation compared to the features per
se. To quantify this observation, we define feature sparsity as the percentage
of near-zero points (absolute value $<0.01$) vs. total points. Figure 6 shows
comparative results for original features and their residuals. It is seen that
the residuals have approximately five times the sparsity of the originals,
which quantitatively confirms the relative sparsity of the residuals. Overall,
both the visualizations and the quantitative analysis confirm that mid-layer
feature residuals are especially information laden for action prediction.

##  5 What has been learned?

Figure 7:  Visualization of Learned Motion Kernels. Left-to-right: Input
frames and symbolized extracted features; learned kernels ($3\times 3$,
$5\times 5$, $7\times 7$); groundtruth prediction target frames that follow
the inputs, with an overlaid vector showing the dominant motion direction.
Top-to-bottom are boxing, catching, lunges and opening bottle actions. Plots
of the learned kernels resemble unit impulses that capture the depicted
motion.

In this section, we study the learned components of our approach in detail.
Specifically, there are two major learned components: 1) dynamically inferred
motion kernels, $K_{n}$, that propagate feature residuals into the future; 2)
the Kalman Gain, $\Gamma_{t}$, that updates sub-optimal prior estimates. In
the following, we provide detailed examination of each using the TSN deep
network as feature extractor, as it is used in our empirical evaluation of
Section 4. Analysis of the features learned by TSN is available elsewhere [5].

###  5.1 Learned motion kernels

####  5.1.1 Convolutional spatial shifting

To understand what has been learned by the motion kernels, we being by
visualizing them, as typically seen in deep network interpretation, cf. [15,
48, 50]; representative results are shown in Figure 7. In all the cases, it is
seen that the learned kernel has a high intensity value displaced from its
center in the direction and magnitude of the depicted motion. For example, in
the top row the motion is mainly from the actor stretching his arm towards the
left by approximately one pixel, with additional minor motion towards the top
(denoted as red arrow). Correspondingly, the motion kernels have peak value
toward the left at approximately one pixel and also some smaller values in the
top and top-left. The other rows also show similar patterns of a displaced
bright spot capturing the motion direction Moreover, not only is the motion
direction captured, but also its magnitude: For the largest motion (second
row) the displacement in the largest kernel is displaced by 3 pixels from the
center, while the smaller kernels displace to the extent they can; for the
intermediate magnitude motion the displacement never goes beyond 2 pixels; for
the smallest displacements (top and bottom rows) the displacement is one pixel
for all kernels. Interestingly, learned kernels across all actions in all
datasets tend to show similar patterns. Here, it is important to note that the
motion kernels are computed on a per feature channel basis and that different
feature channels capture different components of the images: Some channels are
better matched to foreground actors or even actor parts, while others are
better matched to backgrounds. These different channels may therefore yield
different motions and our per channel learning automatically follows these
various motions.

What is the significance of such learned kernels? An explanation can had by
recalling that signal shifting via convolution is captured by operating on the
signal with a unit impulse that is shifted by the desired amount, _i.e_.

$g(t)\circledast\delta({t-\Delta t})=\delta({t-\Delta t})\circledast
g(t)=g(t-\Delta t)$ (13)

where $\delta(\cdot)$ denotes the unit impulse and $g(\cdot)$ denotes an
arbitrary function [74]. In this light, the learned kernels can be interpreted
as (noisy) unit impulses that through convolution shift the feature maps into
the future according to the motion that has been observed in previous frames.

Figure 8:  Illustration of spatial shifting via convolution using 2D unit
impulse kernels. The positions of the impulse signal (yellow block) specifies
the movement directions (a) and the displacement magnitude of the motion
kernel specifies the movement stride (b). Optical flow kernels, $K^{OF}_{n}$,
can be constructed to propagate frames in a convolutional manner (Section
5.1.2) and thereby can be compared to our learned motion kernels to verify
that they are capturing observed motion through convolution (c).

To give more intuition, Figure 8 depicts the convolutional shifting operation
with a toy example: Moving a triangle within an image using convolution. In
the left subfigure, we demonstrate that there are 8 degrees of freedom in
terms of motion directions for a $3\times 3$ unit impulse kernels, _i.e_. the
8 outer entries. Once applying the convolution with any of them, the targeted
triangle would be shifted accordingly by one pixel distance. Arguably, the
limited shifting stride can be a shortcoming for temporal feature propagation,
as it enforces a small motion assumption. In the middle subfigure, however, it
is shown that the shifting stride can be extended by expanding the size of
motion kernels, _i.e_. moving by 2 or 3 pixel distance per operation with
$5\times 5$ and $7\times 7$ kernels, respectively. Thus, kernels with multiple
scales are more flexible to capture motions that cross greater distances and
also support fine-grained orientations, _e.g_. $5\times 5$ kernels can provide
16 more directions. Our approach has adopted this multi-scale scheme.

Figure 9:  Discretization of flow magnitude for conversion to motion kernel
format. Motion kernels are divided into 0, 1, 2 or 3 rings, for no motion,
$3\times 3$, $5\times 5$ and $7\times 7$ kernels, resp., with rings
corresponding to flow magnitude.

####  5.1.2 Motion kernels from optical flow

To further verify that the learned motion kernels are capturing the observed
motion, we compare them to optical flow estimated from the same input frames.
Apparently, no previous approach for comparing learned motion kernels with
optical flow has been proposed. In response, we proceed by building on the
classic Histogram of Oriented Optical Flow (HOOF) [75, 76], with details as
follows.

Let $\mathbf{v}(x,y)=(u(x,y),v(x,y))$ be optical flow, where we use a standard
estimator to recover the flow [77]; although, that particular choice is not
critical, _i.e_. any reliable optical flow estimator could be used. Note that
since we will be comparing to kernels that are learned on a feature channel
basis, the corresponding feature channels are used as input to the flow
estimator, rather than raw grey-level images. We define the flow magnitude
$\|\mathbf{v}(u,v)\|=\sqrt{u^{2}+v^{2}}$ and direction
$\theta=\text{tan}^{-1}(\frac{v}{u})$ in the usual way, where we suppress
dependence on $(x,y)$ for compactness of notation. For conversion to the
motion kernel format, we quantize the flow magnitudes and directions across
$(x,y)$ into histograms with bins that correspond to the discrete kernel
elements. The magnitude is quantized as shown in Figure 9. Note that different
rings are available for quantization depending on the kernel size: $3\times 3$
kernels only encompass the inner most ring; the $5\times 5$ kernal encompasses
both the inner most and middle rings; the $7\times 7$ kernel encompasses all
three rings. Flow direction is discretized according to an angular range for
bin $b$, out of a total of $B$ bins as
$-\frac{\pi}{2}+\pi\frac{b-1}{B}\leq\theta<-\frac{\pi}{2}+\pi\frac{b}{B},$
where $1\leq b\leq B$ and $B$ is the number of orientations that can be
captured by the outer ring of the motion kernel (_i.e_. 8, 16 and 24 for
motion magnitudes mapped to the inner, middle and outer rings, respectively).
Each bin of the histogram includes designated flow vectors weighted by their
magnitude, as in classic HOOF. Finally, the histograms are smoothed with a
Gaussian filter [76] and normalized to complete the conversion of optical flow
to motion kernel format, yielding $K^{OF}_{n}$.

####  5.1.3 Match results

Figure 10(a) illustrates our methodology for quantifying the comparison of
learned motion kernels, $K_{n}$, with groundtruth optical flow kernels,
$K^{OF}_{n}$, as defined in the previous section. As specified in the figure,
we take the inner product between the vectorized versions of $\tilde{K_{n}}$
and $\tilde{K}^{OF}_{n}$ to quantify their match, with $\tilde{}$ denoting
vectorization. Notice that since both $K_{n}$ and $K^{OF}_{n}$ are normalized
by construction (Sections 3.3 and 5.1.2, resp.), their inner product is equal
to their cosine similarity. In the following, we present results for all three
datasets, JHMDB21, BIT and UCF101 as well as overall discussion as a series of
questions and answers.

Figure 10:  Pipeline for computing the match between learned motion kernels,
$K_{n}$, and optical flow kernels, $K^{OF}_{n}$, for the entire prediction
horizons on JHMDB21, in the deep feature residual domain, (a). Box plots of
correlation results across 192 feature channels, (b). In each column, the
vertical extent of the box encompasses the interquartile range, the horizontal
line in the box indicates the median score and the whiskers extending above
and below the box indicate the $90^{th}$ and $10^{th}$ percentiles, resp.

JHMDB21. Results for JHMDB21 are shown in Figure 10(b) for 10 timesteps that
are sampled out of all prediction horizons (_i.e_. 22 timesteps). It is seen
that all three sets of learned kernels can demonstrate high matches for the
first 7 timesteps: $3\times 3$ kernels have median correlations of
$\approx$$0.8$ with OF-kernels; $5\times 5$ are at $\approx$$0.7$; $7\times 7$
are at $\approx$$0.6$. Thereafter, the matches decline (_e.g_. beyond the
$9^{th}$ timestep, $3\times 3$ kernels can only achieve 45%.

It also is notable that for the $1^{st}$-$7^{th}$ prediction timesteps, the
$3\times 3$ kernels achieve the highest matches and the $7\times 7$ kernels
the lowest. It also is interesting to observe that the $3\times 3$ kernels
degenerate more remarkably than the other two when above the $9^{th}$
prediction step. We return to both of these observations below.

Figure 11:  Box plots of correlation scores between learned kernels, $K_{n}$,
and optical flow kernels, $K^{OF}_{n}$, for the BIT (a) and the UCF101 (b)
datasets.

BIT and UCF101. Similar patterns of correlation results for BIT and UCF101 are
given in Figure 11. As the BIT dataset includes mostly small actions (eg high-
five, patting and handshaking) and actors execute their actions smoothly, the
match scores for the $3\times 3$ kernels are generally higher than the other
two kernel sizes compared to that seen with JHMDB21. However, due to BIT
depending on stacked optical flow input (Section 4), which naturally contains
noise, its best correlation value is still lower than JHMDB21. Notably, the
variability of its $7\times 7$ kernels are much larger than those for JHMDB21,
which likely is because the depicted slow actions do not yield large
displacements that would be relevant for the largest kernel. For the UCF101
dataset, the results are not as well correlated as the others, _i.e_. the
median match scores for all three sets of kernels are lower (_e.g_. 0.56 at
the 1st timestep). These results may arise because of the greater complexity
of the motions captured in UCF101, _e.g_. arising from real-world sports
video. This result also helps explain that even while our approach outperforms
alternatives on this dataset (Figure 4 (a)), its relative advantage compared
to the other datasets is less.

Figure 12:  Visualization of three distinctive feature channels (top). Plots
of the averaged matches of the learned 3x3 kernels for three selected channels
across the test set (bottom). The visual quality of feature channels reflects
the match scores.

####  5.1.4 Discussion

Are there performance differences by feature channel? Recall that the employed
deep feature residual input to the motion kernels is a multi-channel tensor
from the TSN middle stage, _i.e_. intermediate feature with size (28, 28,
192). Our approach propagates each channel with a distinct set of kernels.
Here, we visualize a few representative channels and examine the match
differences between them; see Figure 12. It is seen that certain feature
channels, _e.g_. $8$ and $70$, do well in capturing the actions (_i.e_. high
feature map responses delineating the actor), while other feature maps (_e.g_.
184), are mainly noise. Thus, their matches differ greatly: Clean channels
show reasonable match scores between the learned and groundtruth kernels,
whereas noisy channels show very low values throughout the predictions.

Figure 13:  Plots of kernels for action videos whose deep feature residuals
have flow magnitudes larger than 2 (top) and smaller than 2 (bottom).

Is it necessary to use larger kernel sizes? From the $7\times 7$ kernel plots
in Figure 10(b), it is seen that these kernels have relative low correlation
scores even at early timesteps, which raises doubt regarding their usefulness.
Those results aggregate across all frames and actions. Examination of
particular cases, however, shows the importance of larger kernels. Figure 13
provides an illustrative comparison. The top row depicts a kicking action
where the execution leads to a relatively large displacement (_i.e_. half of
its pixels exhibit flow magnitude greater than 2) and our learned $7\times 7$
kernels have high valued impulses at the 3 pixel displacement positions (the
outermost ring). In the bottom row, however, the standing action executes
slowly and the learned kernels reject having high values at the outermost
ring. By our counting, $21$% of deep feature residuals across the entire test
set encompass flow magnitude larger than 2 and these are captured best by the
$7\times 7$ kernels.

What are the failure modes of the learned motion kernels? Another observation
from Figure 10(b) is that the $3\times 3$ kernels tend to degenerate more
severely at longer prediction horizons than the other two, _e.g_. its
correlation goes down to $0.4$ when prediction horizon reaches the $9^{th}$
timestep and onward. To probe further, we show visualizations of a sequence of
learned and groundtruth $3\times 3$ and $5\times 5$ motion kernels; see Figure
14. It is seen that the learned $3\times 3$ kernels eventually degenerate to
indicating no-motion (_i.e_. highest response at the center position), whereas
the learned $5\times 5$ kernels continue capturing the motion and thereby
yield higher correlations. This pattern is likely due to the depicted motion
being relatively large and the smaller kernel has limited ability to encompass
the displacement even at the beginning; this imitation is exacerbated as the
shifts that it propagates lead to ever less accurate predictions at longer
horizons. Thus, we see that a failure mode of the approach is capture of
larger motions at longer horizons, especially as predicted by the small motion
kernels. This limitation is the major one that we have discovered.

Figure 14:  Visualization of failure modes of 3x3 and 5x5 kernels on a picking
test action video.

Is easy-to-propagate equal to easy-to-predict? Our approach is based on the
assumption that propagation into the future would bring benefits, _e.g_.
discovering the unobserved information. To examine this assumption, we
consider the relationship between accuracy improvement vs. mean feature
propagation error on every action category. Results are show in Figure 15. The
lower left subfigure shows that when being compared with the simple baseline
approach (_i.e_. using only the first 20% of the video to predict the entire
action without any further processing, as in Section 4), our approach, RGN,
sees the most performance improvement on the sit and picking actions, _i.e_.
sit is improved from $33.4$% to $58.3$% accuracy and picking is improved from
$66.7$% to $91.7$% accuracy. The lower right subfigure shows a plot of mean-
square-error between propagated and groundtruth deep features. It is seen that
the smallest error arises for sit and pick features, showing that these
actions are relatively easy to propagate for our approach. These observations
suggest that our success on action prediction is from the effectiveness of our
feature propagation.

Figure 15:  (Top) Visual examples for action sit and climb stairs, along with
their most confused categories in red boxes). (Bottom) Early action
recognition accuracy from our framework and baseline approach (a). Mean
feature propagation error per action category (b).

Finally, visualizations of these most improved examples and their most
frequently confused categories are shown in the upper portion of Figure 15. It
is seen that sit can be easily confused with standing when looking only at the
static image, as they both share the same physical environments (_e.g_.
chairs). Similarly, picking and climbing stairs share some common subtle
movements, _e.g_. lifting-up legs. Nevertheless, our system yields greatest
advantage on these examples, suggesting that its approach to predicting the
future yields gains in discriminative power.

###  5.2 Learned Kalman gain

We now examine the learned Kalman gain by plotting its values across
prediction horizons. To place our results in context, we begin with a brief
discussion of related work.

Injecting groundtruth mid-stage information into a recursive generation
process has been seen in various previous efforts to stabilize learning,
_e.g_. scheduled sampling [78] and teacher forcing [79]. Long-term sequences
often suffer from error accumulation due to the poor quality of intermediate
results; therefore, these approaches use real midway data at scheduled times
to replace the undesired results. The Kalman filter update step serves similar
purposes, but does not simply replace some predictions with real measurements:
Instead of completely discarding the predictions, it corrects them
proportionally to current estimates of prediction quality according to the
Kalman gain. Moreover, the update is continual, rather than simply at
predetermined schedule times.

In our approach, the Kalman gain is derived (_i.e_. learned) from the
difference between the prediction and measurement, (6), and maps to a value
range $(0,1)$, with larger values providing a larger update. By
implementation, the gain update is continual and the question becomes whether
it follows the prediction and measurement difference, as desired. An example
is shown in Figure 16 showing an example of baseball swing. When the
prediction reaches the $7^{th}$ timestep, the propagated feature begins to
degnerate (shown in red boxes) and meanwhile the Kalman gain increases,
indicating that the prior estimation is less reliable. After the update at the
$11^{th}$, timestep, the corrupt feature is corrected (shown in green box) and
accordingly the gain value returns to a lower vale, _e.g_. declining from
$0.83$ to $0.47$.

Figure 16:  Illustration of learned Kalman gain values.

We further observe that for smooth actions, _e.g_. Figure 16, the Kalman gain
starts with small values and then smoothly increases to larger values until a
major update is needed and applied, after which it decreases. This behaviour
follows the standard scheduled sampling strategy [78] where the probability of
using on-going predictions decays as timestep increases, but in our case the
update is continual and is driven directly by the current state of the
estimation, rather than a predefined schedule.

In contrast to the case of smooth motion, Figure 17 illustrates performance in
more complicated scenarios. Subfigure (a) depicts the situation where there is
a sudden change in the direction of motion at the very beginning of the
propagation and the gain immediately goes from high to low values. This
behaviour relates to recent work that found reverse scheduled sampling useful
for video frame prediction training [80], but again our approach is adaptive
to the input. Subfigure (b) provides an even more complicated example
(clapping) of multiple changes in motion directions in short time periods and
the behaviour of the Kalman gain, which is seen to provide larger corrections
when the direction of motion changes. These examples, illustrate that
scheduling strategy should be adaptive for action videos, as there exists
quite diverse motion patterns, and our Kalman filter design fullfills the need
in providing updates when the motion direction changes and the prediction
fails.

Figure 17:  Visualization of learned Kalman gain of RGN-KF-2 and RGN-KF on
examples from the JHMDB21 dataset.

In addition to the plots of our current Kalman gain shown in Figures 16 and 17
we also show the values provided by our previous approach to learning the gain
[16]. As noted in Section 3.4, the earlier approach was based more simply on
inputting both the prediction and observation to the learning procedure,
rather than its difference. It is seen that our current approach does much
better at providing corrections when most needed. In contrast, the previous
approach provides uniformly high gain values, which will be prone to induce
exposure bias when testing [81].

Finally, we consider why the UCF101 dataset enjoys the least accuracy
improvements using the new _vs_. previous Kalman gain formulation among the
three datasets considered. We illustrate with representative examples in
Figure 18. It seems that certain action videos, _e.g_. ShotPut, WallPushup and
JumpingJack, have solid benefits and their Kalman gain values show a similar
pattern as for JHMDB21 (Figures 16 and 17), _i.e_. the gain increases only at
particular timesteps to correct predictions and otherwise stays relatively
low. In contrast, for the HighJump class, the gain remains uniformly high,
likely due to the fact that the videos in the class show continual dramatic
changes across time, _e.g_. actors are missing at the beginning of videos and
scene backgrounds drift in a substantial way due to camera motion. These
changes make it necessary for RGN-KF-2 to update with high Kalman gain across
time, similar to what was seen for RGN-KF in Figures 16 and 17; therefore, the
two approaches yield similar accuracies. UCF101 tends to depict the most
continual temporal changes compared to the other datasets considered, which
leads the gain, and therefore final result, of the two approaches to be
especially similar on this dataset. Indeed, cases where the two Kalman
formulations across all datasets yield similar performance typically arise
when the change within the videos is large and ongoing. Still, the newer
approach is important for cases where updates are required less frequently
(_e.g_. in the top three rows of Figure 18 as well as Figures 16 and 17) and
can thereby help avoid exposure bias, as discussed above.

Figure 18:  Visualization of changed early action recognition accuracies from
RGN-KF-2 vs. RGN-KF (in parentheses) on UCF101 at observation ratio $g=0.2$
(left). Plots of RGN-KF-2 gain at observation ratio $g=0.2$ across time for
all four classes (right).

##  6 Conclusions

We have presented a novel spatiotemporal feature residual propagation approach
to early action recognition. Our approach learns to propagate framewise
residuals in feature space to complete partial observations. The approach
enjoys the advantages of the spatial structure preservation of mid-layer
ConvNet features, compact representation that captures essential information
via residual processing and long-term stability via instaneous Kalman filter
corrections. The approach has been evaluated on the UCF101, JHMDB21 and BIT-
Interaction datasets, where it achieves state-of-the-art performance in
comparison to a variety of alternative approaches. We also provide a detailed
investigation of what is captured by all learned components of our system to
yield an overall interpretable approach.

## References

  * [1] M. S. Ryoo, “Human activity prediction: Early recognition of ongoing activities from streaming videos,” in _ICCV_ , 2011.
  * [2] Y. Cao, D. Barrett, A. Barbu, S. Narayanaswamy, H. Yu, A. Michaux, Y. Lin, S. Dickinson, J. Mark Siskind, and S. Wang, “Recognize human activities from partially observed videos,” in _CVPR_ , 2013.
  * [3] T. Lan, T.-C. Chen, and S. Savarese, “A hierarchical representation for future action prediction,” in _ECCV_ , 2014.
  * [4] Y. Kong, D. Kit, and Y. Fu, “A discriminative model with multiple temporal scales for action prediction,” in _ECCV_ , 2014.
  * [5] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. V. Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in _ECCV_ , 2016.
  * [6] J. Carreira and A. Zisserman, “Quo Vadis, Action recognition? a new model and the kinetics dataset,” in _CVPR_ , 2017.
  * [7] G. Varol, I. Laptev, and C. Schmid, “Long-term temporal convolutions for action recognition,” _IEEE Trans. PAMI_ , vol. 40, no. 6, pp. 1510–1517, 2018.
  * [8] C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in _ICCV_ , 2019.
  * [9] M. Sadegh Aliakbarian, F. Sadat Saleh, M. Salzmann, B. Fernando, L. Petersson, and L. Andersson, “Encouraging LSTMs to anticipate actions very early,” in _ICCV_ , 2017.
  * [10] Y. Kong, S. Gao, B. Sun, and Y. Fu, “Action prediction from videos via memorizing hard-to-predict samples,” in _AAAI_ , 2018.
  * [11] Y. Shi, B. Fernando, and R. Hartley, “Action anticipation with RBF kernelized feature mapping RNN,” in _ECCV_ , 2018.
  * [12] C. Vondrick, H. Pirsiavash, and A. Torralba, “Anticipating visual representations from unlabeled video,” in _CVPR_ , 2016.
  * [13] F. Tung and G. Mori, “Deep neural network compression by in-parallel pruning-quantization,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 3, pp. 568–579, 2018.
  * [14] X. Zhu, Y. Xiong, J. Dai, L. Yuan, and Y. Wei, “Deep feature flow for video recognition,” in _CVPR_ , 2017.
  * [15] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in _ECCV_ , 2014.
  * [16] H. Zhao and R. P. Wildes, “Spatiotemporal feature residual propagation for action prediction,” in _ICCV_ , 2019.
  * [17] M. S. Ryoo and J. K. Aggarwal, “Spatio-temporal elationship match: Video structure comparison for recognition of complex human activities,” in _ICCV_ , 2009.
  * [18] G. Singh, S. Saha, M. Sapienza, P. H. Torr, and F. Cuzzolin, “Online real-time multiple spatiotemporal action localisation and prediction,” in _ICCV_ , 2017\.
  * [19] G. Singh, S. Saha, and F. Cuzzolin, “Predicting action tubes,” in _ECCVW_ , 2018.
  * [20] Y. Kong, Z. Tao, and Y. Fu, “Deep sequential context networks for action prediction,” in _CVPR_ , 2017.
  * [21] ——, “Adversarial action prediction networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 3, pp. 539–553, 2018\.
  * [22] H. Gammulle, S. Denman, S. Sridharan, and C. Fookes, “Predicting the future: A jointly learnt model for action anticipation,” in _ICCV_ , 2019.
  * [23] X. Wang, J.-F. Hu, J.-H. Lai, J. Zhang, and W.-S. Zheng, “Progressive teacher-student learning for early action prediction,” in _CVPR_ , 2019.
  * [24] T. Li, J. Liu, W. Zhang, and L. Duan, “Hard-net: Hardness-aware discrimination network for 3D early activity prediction,” in _ECCV_ , 2020\.
  * [25] X. Wu, R. Wang, J. Hou, H. Lin, and J. Luo, “Spatial–temporal relation reasoning for action prediction in videos,” _IJCV_ , pp. 1–22, 2021.
  * [26] J. Chen, W. Bao, and Y. Kong, “Group activity prediction with sequential relational anticipation model,” in _ECCV_ , 2020.
  * [27] T. Xue, J. Wu, K. L. Bouman, and W. T. Freeman, “Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks,” _arXiv preprint arXiv:1607.02586_ , 2016.
  * [28] X. Jia, B. De Brabandere, T. Tuytelaars, and L. Van Gool, “Dynamic filter networks,” in _NIPS_ , 2016.
  * [29] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _ICCV_ , 2017.
  * [30] S. Bako, T. Vogels, B. McWilliams, M. Meyer, J. Novák, A. Harvill, P. Sen, T. Derose, and F. Rousselle, “Kernel-predicting convolutional networks for denoising monte carlo renderings.” _ACM Trans. Graph._ , vol. 36, no. 4, pp. 97–1, 2017.
  * [31] S. Pouyanfar, Y. Tao, A. Mohan, H. Tian, A. S. Kaseb, K. Gauen, R. Dailey, S. Aghajanzadeh, Y.-H. Lu, S.-C. Chen _et al._ , “Dynamic sampling in convolutional neural networks for imbalanced data classification,” in _MIPR_ , 2018.
  * [32] Z. Lan, Y. Zhu, A. G. Hauptmann, and S. Newsam, “Deep local video feature for action recognition,” in _CVPRW_ , 2017.
  * [33] A. Diba, V. Sharma, and L. Van Gool, “Deep temporal linear encoding networks,” in _CVPR_ , 2017.
  * [34] H. Wang, D. Tran, L. Torresani, and M. Feiszli, “Video modeling with correlation networks,” in _CVPR_ , 2020.
  * [35] A. Dosovitskiy and T. Brox, “Inverting visual representations with convolutional networks,” in _CVPR_ , 2016.
  * [36] C. Feichtenhofer, A. Pinz, and R. P. Wildes, “Spatiotemporal multiplier networks for video action recognition,” in _CVPR_ , 2017.
  * [37] D. L. Gall, “MPEG: A video compression standard for multimedia applications,” _Commun. ACM_ , vol. 34, pp. 46–58, 1991.
  * [38] P. Anandan, “A computational framework and an algorithm for the measurement of visual motion,” _IJCV_ , vol. 2, no. 3, pp. 283–310, 1989.
  * [39] A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in _CVPR_ , 2017.
  * [40] J. Jiao, W.-C. Tu, S. He, and R. W. H. Lau, “FormResNet: Formatted residual learning for image restoration,” in _CVPRW_ , 2017.
  * [41] G. Lu, W. Ouyang, D. Xu, X. Zhang, Z. Gao, and M.-T. Sun, “Deep Kalman filtering network for video compression artifact reduction,” in _ECCV_ , 2018\.
  * [42] C.-Y. Wu, M. Zaheer, H. Hu, R. Manmatha, A. J. Smola, and P. Krähenbühl, “Compressed video action recognition,” in _CVPR_ , 2018.
  * [43] R. E. Kalman, “A new approach to linear filtering and prediction problems,” _Journal of Basic Engineering_ , vol. 82, no. 1, pp. 35–45, 1960.
  * [44] A. Dave, O. Russakovsky, and D. Ramanan, “Predictive-corrective networks for action detection,” in _CVPR_ , 2017.
  * [45] V. L. Guen and N. Thome, “Disentangling physical dynamics from unknown factors for unsupervised video prediction,” in _CVPR_ , 2020.
  * [46] T. Haarnoja, A. Ajay, S. Levine, and P. Abbeel, “Backprop KF: Learning discriminative deterministic state estimators,” _arXiv preprint arXiv:1605.07148_ , 2016.
  * [47] H. Coskun, F. Achilles, R. DiPietro, N. Navab, and F. Tombari, “Long short-term memory Kalman filters: Recurrent neural estimators for pose regularization,” in _ICCV_ , 2017.
  * [48] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in _ICCV_ , 2015\.
  * [49] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _CVPR_ , 2016.
  * [50] C. Feichtenhofer, A. Pinz, R. P. Wildes, and A. Zisserman, “Deep insights into convolutional networks for video recognition,” _IJCV_ , vol. 128, no. 2, pp. 420–437, 2020.
  * [51] D. Gabor, “Theory of communication,” _Journal of the Institute of Electrical Engineers_ , vol. 93, pp. 429–457, 1946.
  * [52] S. Luan, C. Chen, B. Zhang, J. Han, and J. Liu, “Gabor convolutional networks,” _IEEE TIP_ , vol. 27, no. 9, pp. 4357–4366, 2018.
  * [53] I. Hadji and R. P. Wildes, “Why convolutional networks learn oriented bandpass filters: Theory and empirical support,” _arXiv preprint arXiv:2011.14665_ , 2020.
  * [54] D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba, “Understanding the role of individual units in a deep neural network,” _Proceedings of the National Academy of Sciences_ , vol. 117, no. 48, pp. 30 071–30 078, 2020.
  * [55] V. Monga, Y. Li, and Y. C. Eldar, “Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing,” _IEEE Signal Processing Magazine_ , vol. 38, no. 2, pp. 18–44, 2021.
  * [56] C. Finn, I. Goodfellow, and S. Levine, “Unsupervised learning for physical interaction through video prediction,” in _NIPS_ , 2016.
  * [57] F. A. Reda, G. Liu, K. J. Shih, R. Kirby, J. Barker, D. Tarjan, A. Tao, and B. Catanzaro, “SDCNet: Video prediction using spatially-displaced convolution,” in _ECCV_ , 2018.
  * [58] M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” _arXiv preprint arXiv:1511.05440_ , 2015.
  * [59] R. Villegas, J. Yang, S. Hong, X. Lin, and H. Lee, “Decomposing motion and content for natural video sequence prediction,” in _ICLR_ , 2017.
  * [60] W. Byeon, Q. Wang, R. K. Srivastava, and P. Koumoutsakos, “Contextvp: Fully context-aware video prediction,” in _ECCV_ , 2018.
  * [61] M. Chen, Z. Xu, K. Weinberger, and F. Sha, “Marginalized denoising autoencoders for domain adaptation,” _arXiv preprint arXiv:1206.4683_ , 2012\.
  * [62] K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” _arXiv preprint arXiv:1212.0402_ , 2012\.
  * [63] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black, “Towards understanding action recognition,” in _ICCV_ , 2013.
  * [64] Y. Kong, Y. Jia, and Y. Fu, “Interactive phrases: Semantic descriptions for human interaction recognition,” _IEEE Trans. PAMI_ , vol. 36, no. 9, pp. 1775–1788, 2014.
  * [65] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “HMDB: A large video database for human motion recognition,” in _ICCV_ , 2011.
  * [66] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [67] X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” in _AISTATS_ , 2011.
  * [68] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” in _NIPS Autodiff Workshop_ , 2017.
  * [69] S. Ma, L. Sigal, and S. Sclaroff, “Learning activity progression in lstms for activity detection and early detection,” in _CVPR_ , 2016.
  * [70] K. Soomro, H. Idrees, and M. Shah, “Online localization and prediction of actions and interactions,” _IEEE Trans. PAMI_ , vol. 41, no. 2, pp. 459–472, 2018.
  * [71] ——, “Predicting the where and what of actors and actions through online action localization,” in _CVPR_ , 2016.
  * [72] A. Jain, A. Singh, H. S. Koppula, S. Soh, and A. Saxena, “Recurrent neural networks for driver activity anticipation via sensory-fusion architecture,” in _ICRA_ , 2016.
  * [73] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-C. Woo, “Convolutional LSTM network: A machine learning approach for precipitation nowcasting,” in _NIPS_ , 2015.
  * [74] R. N. Bracewell, _The Fourier Transform and its Applications_. McGraw Hill, 1986.
  * [75] R. Chaudhry, A. Ravichandran, G. Hager, and R. Vidal, “Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions,” in _CVPR_ , 2009.
  * [76] D. G. Lowe, “Object recognition from local scale-invariant features,” in _ICCV_ , 1999.
  * [77] G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” in _SCIA_ , 2003.
  * [78] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” in _NIPS_ , 2015.
  * [79] R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” _Neural computation_ , vol. 1, no. 2, pp. 270–280, 1989.
  * [80] H. Wu, J. Zhang, Z. Gao, J. Wang, P. S. Yu, M. Long _et al._ , “PredRNN: A recurrent neural network for spatiotemporal predictive learning,” _arXiv preprint arXiv:2103.09504_ , 2021.
  * [81] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” in _CVPR_ , 2017.

|  He Zhao received the B. Eng. degree in communication engineering from
ZhengZhou University, China, in 2014, and the MSc degree in computer science
from the University of Florida, Gainesville, Florida, USA, in 2015. Currently,
he is pursuing the doctoral degree in the Department of Electrical Engineering
and Computer Science at York University, Toronto, Canada. He is a recipient an
Ontario Trillium Foundation graduate scholarship. His major field of interest
is computer vision with an emphasis on video dynamics and human motion
understanding.
---|---
|  Richard P. Wildes (Member, IEEE) received the PhD degree from the
Massachusetts Institute of Technology in 1989. Subsequently, he joined Sarnoff
Corporation in Princeton, New Jersey, as a Member of the Technical Staff in
the Vision Technologies Lab. In 2001, he joined the Department of Electrical
Engineering and Computer Science at York University, Toronto, where he is a
Professor, a member of the Centre for Vision Research and a Tier I York
Research Chair. Honours include receiving a Sarnoff Corporation Technical
Achievement Award, the IEEE D.G. Fink Prize Paper Award and twice giving
invited presentations to the US National Academy of Sciences. His main areas
of research interest are computational vision, especially video understanding,
and artificial intelligence.
---|---