# Spatio-Temporal Video Autoencoder with
Differentiable Memory

Viorica Pătrăucean, Ankur Handa & Roberto Cipolla
Department of Engineering
University of Cambridge, UK
<EMAIL_ADDRESS>

###### Abstract

We describe a new spatio-temporal video autoencoder, based on a classic
spatial image autoencoder and a novel nested temporal autoencoder. The
temporal encoder is represented by a differentiable visual memory composed of
convolutional long short-term memory (LSTM) cells that integrate changes over
time. Here we target motion changes and use as temporal decoder a robust
optical flow prediction module together with an image sampler serving as
built-in feedback loop. The architecture is end-to-end differentiable. At each
time step, the system receives as input a video frame, predicts the optical
flow based on the current observation and the LSTM memory state as a dense
transformation map, and applies it to the current frame to generate the next
frame. By minimising the reconstruction error between the predicted next frame
and the corresponding ground truth next frame, we train the whole system to
extract features useful for motion estimation without any supervision effort.
We present one direct application of the proposed framework in weakly-
supervised semantic segmentation of videos through label propagation using
optical flow.

##  1 Introduction

High-level understanding of video sequences is crucial for any autonomous
intelligent agent, _e.g._ semantic segmentation is indispensable for
navigation, or object detection skills condition any form of interaction with
objects in a scene. The recent success of convolutional neural networks in
tackling these high-level tasks for static images opens up the path for
numerous applications. However, transferring the capabilities of these systems
to tasks involving video sequences is not trivial, on the one hand due to the
lack of video labelled data, and on the other hand due to convnets’ inability
of exploiting temporal redundancy present in videos. Motivated by these two
shortcomings, we focus on reducing the supervision effort required to train
recurrent neural networks, which are known for their ability to handle
sequential input data Williams & Zipser (1995); Hochreiter et al. (2000).

In particular, we describe a spatio-temporal video autoencoder integrating a
differentiable short-term memory module whose (unsupervised) training is
geared towards motion estimation and prediction Horn & Schunck (1994). This
choice has biological inspiration. The human brain has a complex system of
visual memory modules, including iconic memory, visual short-term memory
(VSTM), and long-term memory Hollingworth (2004). Among them, VSTM is
responsible mainly for understanding visual changes (movement, light changes)
in dynamic environments, by integrating visual stimuli over periods of time
Phillips (1974); Magnussen (2000). Moreover, the fact that infants are able to
handle occlusion, containment, and covering events by the age of 2.5 months
Baillargeon (2004) could suggest that the primary skills acquired by VSTM are
related to extracting features useful for motion understanding. These features
in turn could generate objectness awareness based on the simple principle that
points moving together belong to the same object. Understanding objectness is
crucial for high-level tasks such as semantic segmentation or action
recognition Alexe et al. (2012). In this spirit, our approach is similar to
the recent work of Agrawal et al. (2015), who show that the features learnt by
exploiting (freely-available) ego-motion information as supervision data are
as good as features extracted with human-labelled supervision data. We believe
that our work is complementary to their approach and integrating them could
lead to an artificial agent with enhanced vision capabilities.

Our implementation draws inspiration from standard video encoders and
compression schemes, and suggests that deep video autoencoders should differ
conceptually from spatial image autoencoders. A video autoencoder need not
reproduce _by heart_ an entire video sequence. Instead, it should be able to
encode the significant differences that would allow it to reconstruct a frame
given a previous frame. To this end, we use a classic convolutional image
encoder – decoder with a nested memory module composed of convolutional LSTM
cells, acting as temporal encoder. Since we focus on learning features for
motion prediction, we use as temporal decoder a robust optical flow prediction
module together with an image sampler, which provides immediate feedback on
the predicted flow map. At each time step, the system receives as input a
video frame, predicts the optical flow based on the current frame and the LSTM
content as a dense transformation map, and applies it to the current frame to
predict the next frame. By minimising the reconstruction error between the
predicted next frame and the ground truth next frame, we are able to train the
whole system for motion prediction without any supervision effort. Other
modules handling other types of variations like light changes could be added
in parallel, inspired by neuroscience findings which suggest that VSTM is
composed of a series of different modules specialised in handling the low-
level processes triggered by visual changes, all of them connected to a shared
memory module Magnussen (2000). Note that at the hardware level, this
variations-centred reasoning is similar to event-based cameras Boahen (2005),
which have started to make an impact in robotic applications Kim et al.
(2014).

Summary of contributions: We propose a spatio-temporal version of LSTM cells
to serve as a basic form of visual short-term memory, and make available an
end-to-end differentiable architecture with built-in feedback loop, that
allows effortless training and experimentation with the goal of understanding
the role of such an artificial visual short-term memory in low-level visual
processes and its implications in high-level visual tasks. We present one
direct application of the proposed framework in weakly-supervised semantic
segmentation of videos through label propagation using optical flow. All our
code (Torch implementation) is available online Con .

##  2 Related Work

Architectures based on LSTM cells Hochreiter & Schmidhuber (1997) have been
very successful in various tasks involving one-dimensional temporal sequences:
speech recognition Sak et al. (2014), machine translation Sutskever et al.
(2014), music composition Eck & Schmidhuber , due to their ability to preserve
information over long periods of time. Multi-dimensional LSTM networks have
been proposed to deal with (2D) images Graves et al. (2007) or (3D) volumetric
data Stollenga et al. (2015), treating the data as spatial sequences. Since in
our work we aim at building a visual short-term memory, customised LSTM cells
that deal with temporal sequences of spatial data represent a natural choice.

Recently, Srivastava et al. (2015) proposed an LSTM-based video autoencoder,
which aims at generating past and future frames in a sequence, in an
unsupervised manner. However, their LSTM implementation treats the input as
sequences of vectors by flattening the frames or by using 1D frame
representations produced after the last fully-connected layer of a
convolutional neural network. This results in a large number of parameters,
since the activations of the LSTM cells use fully-connected linear operations,
which are not necessarily useful for 2D images, since natural image statistics
indicate only local correlations. Another difference is in the architecture
setup. We are not interested in training a black-box to produce past and
future frames in the sequence. Instead we aim at a transparent setup, and
train a more generic memory module together with specialised modules able to
decode the memory content and on which appropriate constraints (sparsity,
smoothness) can be imposed.

Our approach is partially related to optical flow estimation works like
DeepFlow Weinzaepfel et al. (2013) and FlowNet Fischer et al. (2015). However,
these methods use purely supervised training to establish matches between
consecutive pairs of frames in a sequence, and then apply a variational
smoothing. Our architecture is end-to-end differentiable and integrates a
smoothness penalty to ensure that nearby pixels follow a locally smooth
motion, and requires no labelled training data.

As mentioned in the previous section, our work is similar in spirit to Agrawal
et al. (2015), by establishing a direct link between vision and motion, in an
attempt to reduce supervision effort for high-level scene understanding tasks.
However, instead of relying on data provided by an inertial measurement unit
from which we can estimate only a global transformation representing the ego-
motion, we predict a dense flow map by integrating visual stimuli over time
using a memory module, and then we use a built-in feedback loop to assess the
prediction. The dense flow map is useful to explain dynamic environments,
where different objects can follow different motion models, hence a global
transformation is not sufficient Menze & Geiger (2015).

##  3 Architecture

Figure 1: Spatio-temporal video autoencoder.

Our architecture consists of a temporal autoencoder nested into a spatial
autoencoder (see Figure 1). At each time step, the network takes as input a
video frame $Y_{t}$ of size $H\times W$, and generates an output of the same
size, representing the predicted next frame, $\tilde{Y}_{t+1}$. In the
following, we describe each of the modules in detail.

###  3.1 Spatial autoencoder $E$ and $D$

The spatial autoencoder is a classic convolutional encoder – decoder
architecture. The encoder $E$ contains one convolutional layer, followed by
$\tanh$ non-linearity and a spatial max-pooling with subsampling layer. The
decoder $D$ mirrors the encoder, except for the non-linearity layer, and uses
nearest-neighbour spatial upsampling to bring the output back to the size of
the original input. After the forward pass through the spatial encoder
$Y_{t}\xrightarrow{E}x_{t}$, the size of the feature maps $x_{t}$ is $d\times
h\times w$, $d$ being the number of features, and $h$ and $w$ the height and
width after downsampling, respectively.

###  3.2 Temporal autoencoder

The goal of the temporal autoencoder is to capture significant changes due to
motion (ego-motion or movement of the objects in the scene), that would allow
it to predict the visual future, knowing the past and the present. In a
classic spatial autoencoder Masci et al. (2011), the encoder and decoder learn
proprietary feature spaces that allow an optimal decomposition of the input
using some form of regularisation to prevent learning a trivial mapping. The
encoder decides freely upon a decomposition based on its current feature
space, and the decoder constrains the learning of its own feature space to
satisfy this decomposition and to reconstruct the input, using usually
operations very similar to the encoder, and having the same number of degrees
of freedom. Differently from this, the proposed temporal autoencoder has a
decoder with a small number of trainable parameters, whose role is mainly to
provide immediate feedback to the encoder, but without the capacity of
amending encoder’s mistakes like in the spatial case. In optimisation terms,
the error during learning is attributed mainly to the encoder, which is now
more constrained to produce sensible feature maps.

####  3.2.1 Memory module $\mathit{LSTM}$

The core of the proposed architecture is the memory module playing the role of
a temporal encoder. We aim at building a basic visual short-term memory, which
preserves locality and layout ensuring a fast access and bypassing more
complicated addressing mechanisms like those used by Neural Turing Machines
Graves et al. (2014). To this end, we use customised spatio-temporal LSTM
cells with the same layout as the input. At each time step $t$, the
$\mathit{LSTM}$ module receives as input a new video frame after projection in
the spatial feature space. This is used together with the memory content and
output of the previous step $t-1$ to compute the new memory activations.

Classic LSTM cells operate over sequences of (one-dimensional) vectors and
perform biased linear (fully-connected) transformations, followed by non-
linearities to compute gate and cell activations. In our case, to deal with
the spatial and local nature of the video frames, we replace the fully-
connected transformations with spatial local convolutions. Therefore, the
activations of a spatio-temporal convolutional LSTM cell at time $t$ are given
by:

$\left\\{\begin{array}[]{ccl}i_{t}&=&\sigma(x_{t}\ast w_{xi}+h_{t-1}\ast
w_{hi}+w_{i\text{bias}})\\\ f_{t}&=&\sigma(x_{t}\ast w_{xf}+h_{t-1}\ast
w_{hf}+w_{f\text{bias}})\\\ \tilde{c}_{t}&=&\tanh(x_{t}\ast
w_{x\tilde{c}}+h_{t-1}\ast w_{h\tilde{c}}+w_{\tilde{c}\text{bias}})\\\
c_{t}&=&\tilde{c}_{t}\odot i_{t}+c_{t-1}\odot f_{t}\\\
o_{t}&=&\sigma(x_{t}\ast w_{xo}+h_{t-1}\ast w_{ho}+w_{o\text{bias}})\\\
h_{t}&=&o_{t}\odot\tanh(c_{t})\\\ \end{array}\right.$ (1)

where $x_{t}$ represents the input at time $t$, _i.e._ the feature maps of the
frame $t$; $i_{t}$, $f_{t}$, $\tilde{c}_{t}$, and $o_{t}$ represent the input,
forget, cell, and output gates, respectively; $c_{t}$, $c_{t-1}$, $h_{t}$, and
$h_{t-1}$ are the memory and output activations at time $t$ and $t-1$,
respectively; $\sigma$ and $\tanh$ are the sigmoid and hyperbolic tangent non-
linearities; $\ast$ represents the convolution operation, and $\odot$ the
Hadamard product. For input feature maps of size $d\times h\times w$, the
$\mathit{LSTM}$ module outputs a memory map of size $d_{m}\times h\times w$,
where $d_{m}$ is the number of temporal features learnt by the memory. The
recurrent connections operate only over the temporal dimension, and use local
convolutions to capture spatial context, unlike multi-dimensional LSTM
versions that use spatial recurrent connections Graves et al. (2007);
Stollenga et al. (2015). Note that a similar convolutional LSTM implementation
was recently used in Shi et al. (2015) for precipitation nowcasting.

####  3.2.2 Optical flow prediction $\Theta$ with Huber penalty $H$

The optical flow prediction module generates a dense transformation map
$\mathcal{T}$, having the same height and width as the memory output, with one
2D flow vector per pixel, representing the displacement in $x$ and $y$
directions due to motion between consecutive frames. $\mathcal{T}$ allows
predicting the next frame by warping the current frame. We use two
convolutional layers with relatively large kernels ($15\times 15$) to regress
from the memory feature space to the space of flow vectors. Large kernels are
needed since the magnitude of the predicted optical flow is limited by the
size of the filters. To ensure local smoothness, we need to penalise the local
gradient of the flow map $\triangledown\mathcal{T}$. We add a penalty module
$H$ whose role is to forward its input unchanged during the forward pass, and
to inject non-smoothness error gradient during the backward pass, towards the
modules that precede it in the architecture and that might have contributed to
the error. We use Huber loss as penalty, with its corresponding derivative
(2), due to its edge-preserving capability Werlberger et al. (2009). The
gradient of the flow map is obtained by convolving the map with a fixed (non-
trainable) $2\times 3\times 3$ filter, corresponding to a 5-point stencil, and
$0$ bias.

$\mathcal{H}_{\delta}(a_{ij})=\left\\{\begin{array}[]{ll}\frac{1}{2}a_{ij}^{2},&\text{for
}|a_{ij}|\leq\delta\\\
\delta(|a_{ij}|-\frac{1}{2}\delta),&\text{otherwise}\end{array},\right.\nabla\mathcal{H}_{\delta}(a_{ij})=\left\\{\begin{array}[]{ll}a_{ij}&\text{for}|a_{ij}|\leq\delta,\\\
\delta\text{sign}(a_{ij}),&\text{otherwise}\end{array}\right.$ (2)

Here, $a_{ij}$ represent the elements of $\triangledown\mathcal{T}$. In our
experiments, we used $\delta=10^{-3}$.

####  3.2.3 Grid generator $\mathit{GG}$ and image sampler $S$

The grid generator $\mathit{GG}$ and the image sampler $S$ output the
predicted feature maps $\tilde{x}_{t+1}$ of the next frame after warping the
current feature maps $x_{t}$ with the flow map produced by the $\Theta$
module. We use similar differentiable grid generator and image sampler as
Spatial Transformer Network (STN) Jaderberg et al. (2015) (implementation
available online (Moodstocks, 2015)). The output of $S$, of size $d\times
h\times w$, is considered as a fixed $h\times w$ grid, holding at each
$(x_{o},y_{o})$ position a feature map entry of size $1\times 1\times d$. We
modified the grid generator to accept one transformation per pixel, instead of
a single transformation for the entire image as in STN. Given the flow map
$\mathcal{T}$, $\mathit{GG}$ computes for each element in the grid the source
position $(x_{s},y_{s})$ in the input feature map from where $S$ needs to
sample to fill in the position $(x_{o},y_{o})$:

$\left(\begin{array}[]{c}x_{s}\\\
y_{s}\end{array}\right)=\mathcal{T}(x_{o},y_{o})\left(\begin{array}[]{c}x_{o}\\\
y_{o}\\\
1\end{array}\right),\mathcal{T}(\cdot,\cdot)=\left(\begin{array}[]{ccc}1&0&t_{x}\\\
0&1&t_{y}\end{array}\right).$ (3)

The $\Theta$ module outputs two parameters $(t_{x},t_{y})$ for each pixel. The
forward pass of $\mathit{GG}$ is given by equation (3). In the backward pass,
$\mathit{GG}$ simply backpropagates the derivative of equation (3) w.r.t. the
input parameters $(t_{x},t_{y})$.

####  3.2.4 Loss function

Training the network comes down to minimising the reconstruction error between
the predicted next frame and the ground truth next frame, with Huber penalty
gradient injected during backpropagation on the optical flow map:

$\mathcal{L}_{t}=\|\tilde{Y}_{t+1}-Y_{t+1}\|_{2}^{2}+w_{H}\mathcal{H}(\triangledown\mathcal{T}),$
(4)

where $w_{H}$ is a hard-coded parameter used to weight the smoothness
constraints w.r.t. the data term; in our experiments $w_{H}=10^{-2}$, which is
a common value in optical flow works.

###  3.3 Network parameters

The proposed network has 1,035,067 trainable parameters. The spatial encoder
and decoder have 16 filters each, size $7\times 7$. The memory module
($\mathit{LSTM}$) has 64 filters, size $7\times 7$, and the optical flow
regressor $\Theta$ has 2 convolutional layers, each with 2 filters of size
$15\times 15$, and a $1\times 1$ convolutional layer. The other modules:
$\mathit{GG}$, $H$, and $S$ have no trainable parameters.

##  4 Training

The training was done using _rmsprop_ , with a learning rate starting at
$10^{-4}$ and decaying by $0.9$ after every 5 epochs. The initialisation of
the convolutional layers except those involved in the memory module was done
using _xavier_ method Glorot & Bengio (2010). The parameters of the memory
module, except the biases, were initialised from a uniform distribution
$\mathcal{U}(-0.08,0.08)$. The biases of the forget gates are initialised to
1; for the other gates we set the biases to 0. During training, before each
parameter update, we clip the gradients to lie in a predefined range, to avoid
numerical issues Graves (2013).

Our implementation is based on Torch library Collobert et al.  and extends the
_rnn_ package Léonard et al. (2015). All our code is available online Con .
The training was done on an NVIDIA K40 GPU (12G memory).

##  5 Experiments

As a sanity check to confirm the ability of the grid generator $\mathit{GG}$
and image sampler $S$ to generate the next frame given the current frame and
the correct optical flow map, we ran simple warping tests using Sintel dataset
Butler et al. (2012), by isolating the two modules from the rest of the
architecture. Figure 2 shows an example of warping. Note that since the flow
displacement is significant in this dataset, the sampling result can contain
artefacts, especially near boundaries, caused by occlusions. The average per-
pixel error induced by the sampler on this dataset is $0.004$ (for pixel
values in $[0,1]$). However, this test was done by sampling directly from the
input frames, no spatial encoder – decoder was used. In our architecture,
these artefacts will be washed out to a certain extent by the convolutional
decoder. The optical flow throughout this section is displayed using the
colour code from Baker et al. (2011), illustrated in Figure 2, top-right
corner.

###  5.1 Unsupervised experiments

To assess qualitatively and quantitatively the behaviour of the proposed
architecture and of its components, we ran unsupervised experiments on
synthetic and real datasets111Existing datasets in the optical flow community
do not represent suitable training and/or testing data because they contain
flow maps for only pairs of images Fischer et al. (2015), or for very few
short video sequences (6 sequences of 2 or 8 frames in Middlebury dataset
Baker et al. (2011)). Sintel dataset Butler et al. (2012) contains indeed
ground truth optical flow for longer video sequences. However, it exhibits a
complex combination of camera self-motion and moving objects, with a
relatively low frame rate, leading to very large (non-realistic) displacements
between consecutive frames. Note also that the self-motion makes the entire
scene to _move_ in terms of optical flow, making the prediction very hard and,
in the same time, not useful for the purposes of our work: if all the points
in the scene move, it is difficult to delineate objects..

Figure 2: Warping performed by the $\mathit{GG}$ and $S$ modules, given a
video frame and the ground truth optical flow map. First line: input frame and
ground truth optical flow map displayed using the colour code shown in the
top-right corner of the image; the colour encodes the direction of movement
and the colour intensity reflects the magnitude; darker shades correspond to
higher magnitudes Baker et al. (2011). Second line: ground truth next frame
and sampled next frame; note the artifacts close to the boundaries. Figure 3:
Samples of images from moving MNIST dataset.

##### Synthetic dataset.

Moving MNIST dataset Srivastava et al. (2015) consists of sequences of 20
frames each, obtained by moving (translating) MNIST digit images inside a
square of size $64\times 64$, using uniform random sampling to obtain
direction and velocity; the sequences can contain several overlapping digits
in one frame. We generated 10k sequences for training and 3k sequences for
validation (see Figure 3 for sample images).

Figure 4: Baseline (autoencoder) next frame predictors. C = convolutional, T = $\tanh$, P = pooling, U = unpooling, R = reshape from 2D feature maps to 1D (flattened) vectors and vice-versa. Architecture | Parameters | Test error
---|---|---
AE-Conv | 109,073 | 0.0948
AE-fcLSTM | 33,623,649 | 0.0650
AE-ConvLSTM | 1,256,305 | 0.0641
AE-ConvLSTM-flow | 1,035,067 | 0.0439
Table 1: Number of parameters for the different architectures and their per-
pixel average error on moving MNIST.

To highlight the advantages of the proposed convolutional LSTM units (denoted
as ConvLSTM) and of the overall architecture (denoted as AE-ConvLSTM-flow), we
ran quantitative experiments using different architectures, illustrated in
Figure 4. AE-Conv is a simple convolutional autoencoder, which takes as input
the frames of a video sequence considered as independent channels, and outputs
the predicted next frame. It contains a similar encoder and decoder as AE-
ConvLSTM-flow, but replaces the (recurrent) memory module and optical flow
module with simple convolution-tanh layers. AE-ConvLSTM is a similar setup,
but uses convolutional LSTM blocks as temporal encoder and decoder
respectively; AE-fcLSTM replaces the convolutional LSTM blocks with classic
fully-connected LSTM blocks, resembling the architecture of Srivastava et al.
(2015), but predicting only one frame in the future. Table 1 summarises the
number of parameters for each architecture. Note that, although the depth of
all the architectures could be considered similar, the differences in the
number of parameters are significant: all the architectures using LSTM blocks
have more parameters than the simple convolutional autoencoder due to the
gating layers; among these, the fully-connected model is considerably larger.
AE-ConvLSTM-flow has less parameters than AE-ConvLSTM due to the fact that the
LSTM temporal decoder in the latter is replaced by simple convolutional
regression layers in the former.

The loss function for this experiment was the binary cross-entropy Srivastava
et al. (2015), and the sequences are fed in as binary data. Table 1 presents
the test errors and Figure 5 shows qualitative results. All the architectures
using LSTM blocks report lower errors than AE-Conv, showing the importance of
exploiting the temporal dependency between frames. Although the results of AE-
Conv seem qualitatively better than AE-fcLSTM, they are very blurred, leading
to the increased error. The architectures using the proposed convolutional
LSTM units are more efficient than the classic fully-connected units, since
the former preserve the spatial information; this can be observed in both the
quantitative and the qualitative results. Importantly, AE-ConvLSTM-flow
produces boundaries with better continuity than AE-ConvLSTM, suggesting that
it acquires a basic form of objectness awareness, _i.e._ it learns to reason
about objects as a whole due to their motion. Equally, these results show that
the optical flow maps produced by our network correspond to a basic form of
image segmentation into objects identified by their movement. Hence, this
setup could potentially be more useful for supervised video segmentation than
a classic video autoencoder.

Figure 5: Qualitative results on moving MNIST test dataset. First four
columns: the last frames (out of 10 in total) fed into the network (from $t-3$
to $t$); 5th column: the ground truth next frame ($t+1$); 6th column: AE-Conv;
7th column: AE-fcLSTM; 8th column: AE-ConvLSTM; last two columns: AE-ConvLSTM-
flow predicted next frame and optical flow.

##### Real dataset.

We train our architecture on real videos extracted from HMDB-51 dataset Kuehne
et al. (2011), with about 107k frames in total (1.1 hours), and present
qualitative results of the predicted next frame and the optical flow maps on
test sequences extracted from PROST Santner et al. (2010) and ViSOR Vezzani &
Cucchiara (2010) datasets. Figure 6 illustrates results obtained on different
test sequences and videos showing the optical flow for entire test sequences
are included in the supplemental material. The results indicate that the
moving elements of the scene are identified by the network. However, the
accuracy of the estimated flow is not comparable to supervised setups for
optical flow estimation. But this is expected since in our case we predict the
flow map that links the already seen frame $t$ to the unseen frame $t+1$,
whereas classic optical flow methods estimate the optical flow between two
already seen frames. Intuitively, we expect our network to extract more
meaningful features than a classic supervised setup, since the problem is more
challenging, at least for articulated objects.

Figure 6: Qualitative results on real videos. The first three columns show the
last frames (out of 12 in total) fed into the network (from $t-2$ to $t$), the
$t+1$ column shows the ground truth next frame, and the last two columns show
the predicted next frame and the predicted optical flow.

###  5.2 Application to weakly-supervised video semantic segmentation

Almost no labelled dataset of real videos exists for semantic
segmentation222SceneNet Handa et al. (2015) is a recent synthetic dataset
allowing to obtain synthetic labelled videos by rendering labelled synthetic
3D scenes. However, the models lack texture. Sun3D Xiao et al. (2013) contains
video sequences with labels obtained by label propagation. However, the
quality of the labelling is quite poor., limiting the applicability of deep
architectures for the task. There exist, however, datasets that contain video
sequences with one labelled frame per sequence Brostow et al. (2008); Nathan
Silberman & Fergus (2012). We take advantage of our autoencoder to enable
weakly-supervised video segmentation. The proposed framework, denoted SegNet-
flow, is sketched in Figure 7. We use AE-ConvLSTM-flow together with a per-
frame semantic segmentation module, SegNet333Although not state-of-the-art for
image semantic segmentation, we chose to use SegNet in our experiment due to
its reduced memory requirements. This is the only setup allowing us to train
the entire network on one GPU with 12G memory. Badrinarayanan et al. (2015).
Using the flow maps produced by AE-ConvLSTM-flow, we warp (propagate) SegNet’s
predictions until reaching the time step for which ground truth labels are
available. The label propagation is performed by a customised recurrent
module, which integrates a merging block $M$ and a sampler $S$ similar to the
one used in AE-ConvLSTM-flow. $M$ is a recurrent convolutional module, which
receives as input SegNet’s basic predictions $L_{t}^{b}$ at time $t$, and the
warped predictions $L_{t-1}^{w}$ from time $t-1$ generated by $S$, and
produces the final label predictions $\tilde{L}_{t}$ for time $t$. For the
time steps where ground truth labels $L_{t}^{gt}$ are available, we minimise
the cross-entropy segmentation error between $\tilde{L}_{t}$ and $L_{t}^{gt}$.
This error term is added to the existing reconstruction error term. In this
way, the two tasks, flow prediction and semantic segmentation, are learnt
jointly with a minimum of supervision required.

Figure 7: SegNet-flow: weakly-supervised video segmentation through label
propagation using optical flow.

We trained our architecture on Camvid dataset Brostow et al. (2008), which
consists of road scene RGB videos, with about 10k training frames in total,
among which 468 are labelled, and about 7k testing frames, among which 233 are
labelled. However, due to memory limitations, we were not able to use all the
(unlabelled) available frames, but only sequences of 6 frames around the
labelled ones. Hence about 2.5k frames were used for training. Additionally,
we downsampled the videos to $240\times 180$ pixels per frame444Note that the
results in Table 2 for basic SegNet are different from those reported in
Badrinarayanan et al. (2015), due to this downsampling.. We use the 12 classes
labelling, similar to Badrinarayanan et al. (2015). Table 2 summarises the
results of basic SegNet on the test set compared to our setup, and Figure 8
shows qualitative results. It can be observed that the quality of the
segmentation increases considerably for the large classes, but diminishes for
the small thin structures. We believe the main reason for this is the
relatively small size of the training set. SegNet-flow has about one million
parameters more than the basic SegNet, hence the former model is less
constrained. Equally, the fact that the flow estimation is performed on a
downsampled version of the input (after the spatial encoder) could lead to a
poor flow estimation for small structures. Nevertheless, we were able to train
end-to-end an architecture performing joint flow estimation and semantic
segmentation from videos. Understanding better and improving this framework
constitute an exciting direction of future work.

Figure 8: Qualitative results on Camvid. 1st column: input frame, 2nd column: ground truth, 3rd column: SegNet result, 4th column: SegNet-flow result. The flow constraints present in SegNet-flow make a significant difference in smoothing the segmentation results produced by SegNet. We expect this to hold irrespective of the segmentation module used. Network |  Sky |  Building |  Pole |  Road |  Sidewalk |  Vegetation |  Traffic sign |  Fence |  Car |  Pedestrian |  Cyclist |  Class Avg. |  Global Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
SegNet | 89.2 | 65.4 | 22.6 | 92.0 | 70.7 | 44.8 | 14.9 | 5.38 | 60.5 | 25.6 | 2.73 | 44.9 | 75.3
SegNet-flow | 92.9 | 73.4 | 3.22 | 97.2 | 62.9 | 52.3 | 2.37 | 2.35 | 63.3 | 14.9 | 1.70 | 42.4 | 76.9
Table 2: Per-class and global average segmentation accuracy.

##  6 Conclusion and Future Work

We proposed an original spatio-termporal video autoencoder based on an end-to-
end differentiable architecture that allows unsupervised training for motion
prediction. The core of the architecture is a module implementing a
convolutional version of long short-term memory (LSTM) cells to be used as a
form of artificial visual short-term memory. We showed that the proposed
convolutional LSTM module performs better than existing classic autoencoders
and than fully-connected LSTM implementations, while having a reduced number
of parameters. The usefulness of the overall setup was illustrated in the task
of weakly-supervised video semantic segmentation.

We believe that our work can open up the path to a number of exciting
directions. Due to the built-in feedback loop, various experiments can be
carried out effortlessly, to develop further the basic memory module that we
proposed. In particular, investigating on the size of the memory and its
resolution/downsampling could shed some light into the causes of some
geometric optical illusions; _e.g._ when storing in memory an overly-
downsampled version of the visual input, this could affect the capacity of
correctly perceiving the visual stimuli, in accordance with Shannon-Nyquist
sampling theorem Shannon (1949). Equally, we hypothesise that our
convolutional version of LSTM cells can lead to ambiguous memory activations,
_i.e._ the same activation would be produced when presenting a temporally
moving boundary or a spatially repeated boundary. This could imitate the
illusory motion experienced by biological VSTMs, _i.e._ static repeated
patterns that produce a false perception of movement Conway et al. (2005).

A necessary future development is the integration of the memory module with an
attention mechanism and a form of long-term memory module, to enable a
complete memory system, indispensable for supervised tasks.

Last but not least, the proposed architecture composed of memory module and
built-in feedback loop could be applied for static images as a compression
mechanism, similar to the inspirational work on jigsaw video compression
Kannan et al. (2007). The memory addressing in that case could use a content-
based mechanism similar to NTM Graves et al. (2014).

#### Acknowledgments

We are greatly indebted to the _Torch_ community for their efforts to maintain
this great library, and especially to Nicholas Léonard for his helpful advice.
We are grateful to CSIC–University of Cambridge for funding this work (grant
number EP/L010917/1).

## References

  * (1) Convlstm.  https://github.com/viorik/ConvLSTM.
  * Agrawal et al. (2015) Pulkit Agrawal, João Carreira, and Jitendra Malik.  Learning to see by moving.  _CoRR_ , abs/1505.01596, 2015.  URL http://arxiv.org/abs/1505.01596.
  * Alexe et al. (2012) B. Alexe, T. Deselaers, and V. Ferrari.  Measuring the objectness of image windows.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 34(11), 2012\.  doi: 10.3758/BF03203943.
  * Badrinarayanan et al. (2015) Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling.  _CoRR_ , abs/1505.07293, 2015.
  * Baillargeon (2004) Reneé Baillargeon.  Infants’ physical world.  _American Psychological Society_ , 13(3), 2004.
  * Baker et al. (2011) Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski.  A database and evaluation methodology for optical flow.  _Int. J. Comput. Vision_ , 92(1):1–31, 2011.
  * Boahen (2005) K. Boahen.  Neuromorphic microchips.  _Scientific American_ , 292(5), 2005.
  * Brostow et al. (2008) Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla.  Segmentation and recognition using structure from motion point clouds.  In _ECCV_ , pp. 44–57, 2008.
  * Butler et al. (2012) D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black.  A naturalistic open source movie for optical flow evaluation.  In _European Conf. on Computer Vision (ECCV)_ , pp. 611–625, 2012\.
  * (10) Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet.  Torch7: A matlab-like environment for machine learning.
  * Conway et al. (2005) Bevil R. Conway, Akiyoshi Kitaoka, Arash Yazdanbakhsh, Christopher C. Pack, and Margaret S. Livingstone.  Neural basis for a powerful static motion illusion.  _The Journal of Neuroscience_ , 25(23):5651–5656, 2005.
  * (12) Douglas Eck and Jürgen Schmidhuber.  A first look at music composition using lstm recurrent neural networks.  Technical Report IDSIA-07-02, IDSIA / USI-SUPSI.
  * Fischer et al. (2015) Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  _CoRR_ , abs/1504.06852, 2015.  URL http://arxiv.org/abs/1504.06852.
  * Glorot & Bengio (2010) Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _AISTATS_ , 2010.
  * Graves (2013) Alex Graves.  Generating sequences with recurrent neural networks.  _CoRR_ , abs/1308.0850, 2013.
  * Graves et al. (2007) Alex Graves, Santiago Fernández, and Jürgen Schmidhuber.  Multi-dimensional recurrent neural networks.  In _Artificial Neural Networks_ , volume 4668, pp. 549–558. 2007\.
  * Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural turing machines.  _CoRR_ , abs/1410.5401, 2014.
  * Handa et al. (2015) A. Handa, V. Pătrăucean, V. Badrinarayanan, S. Stent, and R. Cipolla.  SceneNet: Understanding real world indoor scenes with synthetic data.  _arXiv preprint arXiv:1511.07041_ , 2015.
  * Hochreiter & Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural Comput._ , 9(8):1735–1780, November 1997\.  ISSN 0899-7667.  doi: 10.1162/neco.1997.9.8.1735.  URL http://dx.doi.org/10.1162/neco.1997.9.8.1735.
  * Hochreiter et al. (2000) Sepp Hochreiter, Fakultat F. Informatik, Yoshua Bengio, Paolo Frasconi, and Jurgen Schmidhuber.  Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies.  In _Field Guide to Dynamical Recurrent Networks_. 2000.
  * Hollingworth (2004) A. Hollingworth.  Constructing visual representations of natural scenes: the roles of short- and long-term visual memory.  _J Exp Psychol Hum Percept Perform._ , 30(3), 2004.
  * Horn & Schunck (1994) Berthold K. P. Horn and B. G. Schunck.  Artificial intelligence in perspective.  chapter Determining Optical Flow: A Retrospective, pp. 81–87. 1994\.
  * Jaderberg et al. (2015) Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  _CoRR_ , abs/1506.02025, 2015.  URL http://arxiv.org/abs/1506.02025.
  * Kannan et al. (2007) Anitha Kannan, John Winn, and Carsten Rother.  Clustering appearance and shape by learning jigsaws.  In _Advances in Neural Information Processing Systems_ , pp. 657–664. 2007.
  * Kim et al. (2014) Hanme Kim, Ankur Handa, Ryad Benosman, Sio-Hoï Ieng, and Andrew J. Davison.  Simultaneous mosaicing and tracking with an event camera.  In _British Machine Vision Conference (BMVC)_. BMVC 2014, 2014.
  * Kuehne et al. (2011) H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre.  HMDB: a large video database for human motion recognition.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2011.
  * Léonard et al. (2015) Nicholas Léonard, Sagar Waghmare, Yang Wang, and Jin-Hwa Kim.  rnn : Recurrent library for torch.  _CoRR_ , abs/1511.07889, 2015.
  * Magnussen (2000) Svein Magnussen.  Low-level memory processes in vision.  _Trends in Neurosciences_ , 23(6):247–251, 2000\.  ISSN 0166-2236.  doi: http://dx.doi.org/10.1016/S0166-2236(00)01569-1.  URL http://www.sciencedirect.com/science/article/pii/S0166223600015691.
  * Masci et al. (2011) Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber.  Stacked convolutional auto-encoders for hierarchical feature extraction.  In _Artificial Neural Networks and Machine Learning_ , volume 6791, pp. 52–59. 2011.
  * Menze & Geiger (2015) Moritz Menze and Andreas Geiger.  Object scene flow for autonomous vehicles.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3061–3070, 2015.
  * Moodstocks (2015) Moodstocks.  Open Source Implementation of Spatial Transformer Networks.  URL https://github.com/qassemoquab/stnbhwd, 2015.
  * Nathan Silberman & Fergus (2012) Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _ECCV_ , 2012.
  * Phillips (1974) W.A. Phillips.  On the distinction between sensory storage and short-term visual memory.  _Perception & Psychophysics_, 16(2):283–290, 1974.  ISSN 0031-5117.  doi: 10.3758/BF03203943.  URL http://dx.doi.org/10.3758/BF03203943.
  * Sak et al. (2014) Hasim Sak, Andrew W. Senior, and Françoise Beaufays.  Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition.  _CoRR_ , abs/1402.1128, 2014.  URL http://arxiv.org/abs/1402.1128.
  * Santner et al. (2010) Jakob Santner, Christian Leistner, Amir Saffari, Thomas Pock, and Horst Bischof.  PROST Parallel Robust Online Simple Tracking.  In _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , San Francisco, CA, USA, 2010.
  * Shannon (1949) C.E. Shannon.  Communication in the presence of noise.  _Proceedings of the IRE_ , 37(1):10–21, Jan 1949\.  ISSN 0096-8390.  doi: 10.1109/JRPROC.1949.232969.
  * Shi et al. (2015) Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional LSTM network: A machine learning approach for precipitation nowcasting.  _CoRR_ , abs/1506.04214, 2015.
  * Srivastava et al. (2015) Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.  Unsupervised learning of video representations using lstms.  _ICML_ , 2015.
  * Stollenga et al. (2015) Marijn Stollenga, Wonmin Byeon, Marcus Liwicki, and Jürgen Schmidhuber.  Parallel multi-dimensional lstm, with application to fast biomedical volumetric image segmentation.  _CoRR_ , abs/1506.07452, 2015.  URL http://arxiv.org/abs/1506.07452.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks.  _NIPS_ , 2014.
  * Vezzani & Cucchiara (2010) Roberto Vezzani and Rita Cucchiara.  Video surveillance online repository (visor): an integrated framework.  _Multimedia Tools and Applications_ , 50(2):359–380, 2010.
  * Weinzaepfel et al. (2013) P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid.  Deepflow: Large displacement optical flow with deep matching.  In _Computer Vision (ICCV), 2013 IEEE International Conference on_ , pp. 1385–1392, Dec 2013.  doi: 10.1109/ICCV.2013.175.
  * Werlberger et al. (2009) Manuel Werlberger, Werner Trobin, Thomas Pock, Andreas Wedel, Daniel Cremers, and Horst Bischof.  Anisotropic Huber-L1 optical flow.  In _Proceedings of the British Machine Vision Conference (BMVC)_ , London, UK, September 2009.
  * Williams & Zipser (1995) Ronald J. Williams and David Zipser.  Backpropagation.  chapter Gradient-based Learning Algorithms for Recurrent Networks and Their Computational Complexity, pp. 433–486. 1995.
  * Xiao et al. (2013) J. Xiao, A. Owens, and A. Torralba.  Sun3d: A database of big spaces reconstructed using sfm and object labels.  In _ICCV_ , pp. 1625–1632, 2013.