Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

Forum for Information Retrieval Evaluation, December 15-18, 2023, India

<EMAIL_ADDRESS>] [1]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]<EMAIL_ADDRESS>]

[1]Corresponding author.

# Overview of the HASOC Subtrack at FIRE 2023: Identification of Tokens
Contributing to Explicit Hate in English by Span Detection

Sarah Masud  Indraprastha Institute of Information Technology, Delhi  Mohammad
Aflah Khan  Md. Shad Akhtar  Tanmoy Chakraborty  Indian Institute of
Technology, Delhi

(2023)

###### Abstract

As hate speech continues to proliferate on the web, it is becoming
increasingly important to develop computational methods to mitigate it.
Reactively, using black-box models to identify hateful content can perplex
users as to why their posts were automatically flagged as hateful. On the
other hand, proactive mitigation can be achieved by suggesting rephrasing
before a post is made public. However, both mitigation techniques require
information about which part of a post contains the hateful aspect, i.e., what
spans within a text are responsible for conveying hate. Better detection of
such spans can significantly reduce explicitly hateful content on the web. To
further contribute to this research area, we organized HateNorm at HASOC-FIRE
2023, focusing on explicit span detection in English Tweets111Disclaimer: The
paper contains samples of hate speech, which are only included for contextual
understanding. We do not support these views.. A total of $12$ teams
participated in the competition, with the highest macro-F1 observed at $0.58$.

###### keywords:

Hate Span Explicit Hate English Tweet HASOC’23

##  1 Introduction

Hate speech is a challenging social issue given its subjective nature: what is
hateful changes with time, geography, and cultural context. United Nations
defines hate speech222https://www.un.org/en/hate-speech/understanding-hate-
speech/what-is-hate-speech as “any form of communication that uses pejorative
or discriminatory language with reference to a person or a group based on who
they are." Further, hate speech has real-world implications; not only do real-
world biases drive up online hate speech, but online hate speech can lead to
an increase in hate crimes in the offline world. To reduce the burden on the
volume and velocity of hateful content accessed by content moderators,
analyzing, mitigating, and countering hateful content via computational
methods is binding. While computer-aided can help perform the first level of
mitigation, human involvement in subjective matters like hate speech is
critical and compulsory for improving social systems in the real world. To aid
the systems in better detection of hateful content, one can look into
developing systems that can capture and attend to the hateful spans [1] within
a sentence. Span detection can help develop a sense of rationale, act as a
tool for post hoc analysis, and improve the retrival of critical facts in
claim verification [2].

Table 1: A few examples of hateful posts of varying degrees from the dataset
curated by Masud et al. [3] and their corresponding hateful spans marked in
red.

Sample # |  Text
---|---
1 |  Women … Can’t live with them … Can’t shoot them
2 |  kathy griffin is the ultimate liberal attack somebody and suffer consequences and 5 & she is the victim c**t needs to have some dick forced up her dy** ass barron trumps

Shared Task Objective. A hate span is a set of continuous tokens that, in
tandem, communicate the explicit hatefulness in a sentence. Table 1 provides
some examples of harmful social media posts marked for hateful fragments. For
instance, in the first sentence of Table 1, “Women … Can’t live with them …
{Can’t shoot them}", the portion highlighted in red will be considered as a
hateful span. Formally, given a hate sample, tokenized as $t=\langle
w_{1},w_{2},\dots,w_{n}\rangle$, the hate span identification task looks for a
sequence of hateful tokens, $\langle w_{i},..,w_{i+l}\rangle$ [3].

Problem Definition: Given a hateful text, identify those specific fragments
within the sentence that are hateful. This is a sequence tagging task where
the aim is to label each word as either belonging to the hate span or not.

Share Task Details. Through the HateNorm shared task part of HASOC-FIRE
2023333https://hasocfire.github.io/hasoc/2023/, we aimed at engaging the
broader research community in understanding span detection techniques and
contributing towards the extraction of spans inside a hateful text. In this
task, we repurpose a part of the publicly available Hate Normalization dataset
[3], with each data point containing at least one hate span. The competition
ran for a month, from July 13, 2023, to August 16, 2023, PST. Hosted on
Kaggle, the task received $72$ submissions from $12$ teams.

Observations As opposed to a single label per input sentence in a general NLP
classification setup, for HateNorm, we had a label per token of the sentence
[4]. Given the sequential nature of the output, we observed initial hesitation
among participants in working with the dataset. However, their engagement
improved once a starter kit/codebook was shared. We also observed that among
the submissions that submitted a demo paper, the base architecture was more
than just a large language model (LLM)-based classifier. There was a mixed
usage of both LLM and Bi-directional LSTMs. Further, we noticed that half of
the teams did not apply a CRF layer to capture the sequential encoding of the
target label but instead relied on LLM’s ability to capture context while
making predictions for individual tags. The winning team ‘FiRC-NLP‘ with $8$
submissions, obtained macro-F1 scores of $0.53$ and $0.58$ on the public and
private leaderboards, respectively. While this beats the start-kit scores of
$0.34$, it is comparable to the SpanBERT-BiLSTM-CRF model from Masud et al.
[3], which also reported a macro-F1 of $0.58$. More work is needed to bring
mainstream attention beyond a text classification of hatefulness to detecting
spans. Shared task venues like HASOC and SemEval are the steps in the right
direction.

##  2 Related Work

Owing to the relevance and need for computational methods to tackle hate
speech, we now have a plethora of datasets [5, 6, 7] and techniques [8, 9]
exploring the same. Regarding explicit hate speech, hate lexicons [10, 11]
have been explored. Auxiliary tasks such as hate normalization [3, 12, 13] and
rationale prediction [14] underpinned by the presence or absence of hateful
phrases in a sentence led to the foray of hate span detection. In English, the
task has been explored from the point of view of detecting toxic and offensive
spans [1, 14, 3, 12]. In low-resource settings, the span detection has been
explored for Vietnamese [15]. Via the MUDES model, Ranasinghe and Zampieri
[16] explored the cross-lingual applicability of hateful span detection when
trained on English span datasets. In the multimodal aspect, video frames that
conveyed hatefulness were employed as hate spans [17]. Another work detected
phrases and sentences in long articles that contribute to hate [18]. In other
areas of social computing, span detection has been explored under the English
[2] and multilingual [19] factual claim detection settings. Meanwhile,
detecting tokens a model pays attention to while labeling a sample as hateful
has been employed in posthoc explanations [20].

Table 2: Examples of hateful posts from Masud et al. [3] dataset and their
corresponding BIO tags depicting harmful spans.

Text |  Span
---|---
lol what a stupid k*k* |  {O, O, O, B, I}
@user text me fa**ot. |  {O, O, O, B}
sad to say but I do not trust shit I know how bi****s operate |  {O, O, O, O, O, B, I, I, I, O, O, O, B, O}

##  3 Dataset

This task employed the existing dataset from Masud et al. [3] curated initially for $3$ processes – hate intensity prediction, hate span prediction, and hate normalization generation. We employ only the subset of samples labeled for hate span prediction for hosting HateNorm. This led to a dataset with $3027$ explicitly hateful sentences marked with hate spans. As outlined in Table 2, the spans are tagged via the BIO notation, marking the beginning and inclusion of span tokens as othering, marking the exclusion. Note that a single token can be a span with a corresponding ‘B’ tag. Meanwhile, an ‘I‘ tag is always preceded by a ‘B‘ tag. The $2421$ train samples contained $4695$ unique spans with an average of $1.939$ spans per training instance. Figure 1 outlines the distribution of the number of spans of a given length, and the majority of spans are $\leq 5$ in length. In the train set, each row contained an ‘id | space-separated token | list of span indices | space-separated gold span label.’ Meanwhile, the $606$ test instances were divided into $182$ public leader board and $424$ privately held instances.

Figure 1: The frequency plot of length of spans in train set. Most samples
have only 1 word spans, and majority have spans of length $\leq 5$.

##  4 Task Details

Hosting. HateNorm was hosted as a Kaggle444https://www.kaggle.com/competitions/hatenorm23 Competition from 13th July 2023 to 16th August 2023 PST. It received participation from $12$ teams, leading to $72$ submissions (an average of $6$ submissions per team) throughout the competition. As a part of the Kaggle competition, participants were given a sample codebook and a sample ‘submission.csv,‘ as outlined in Table 3. We required the ‘id | space-separated predicted label’ for the submission file.

Id | Predicted Span List
---|---
100 | O O O B I
200 | O O O O
606 | O O O O O B I I I O O O B O
Table 3: Submission format of test.csv with predictions corresponding to
inputs enlisted in Table 2.

Evaluation Metric. Unlike the classification of a single instance that can be
adjudged via accuracy or macro-F1, span detection requires evaluating the
correct ordering of spans, ‘B’ following a ‘I’ and ‘O’ being the default. To
capture this sequential nature of label prediction for individual tokens, we
employ the seqeval macro-F1 metric [21]. We hosted the custom metric of
seqeval as a script and loaded that to set up the competition so that each
incoming submission, by default, gets evaluated via seqeval macro-F1. Further,
held 70% of the test samples were private, based on which the final rankings
were revealed after the contest. During the contest, the participants saw
their rank compared against the public leaderboard 30%. Note that a public
leader board does not mean test cases are public.

Baselines. The codebook provided to the participant’s finetuned a
DistillBERT+FNN setup which reported a extremely low macro-F1 of $0.36$.
Meanwhile, the baselines provided by Masud et al. [3] consisted of BiLSTM+CRF
with a macro-F1 of $0.44$, and the best method555Note: We excluded the Elmo
based system from baseline due to reproducibility issues with Elmo on both
Tensorflow and Pytorch. being a SpanBERT [22]+BiLSTM+CRF system that reported
a macro-F1 of $0.58$.

##  5 Submitted System

Table 4 enlists the top $6$ submissions. Among the participating teams that
shared the overview notes, we observed that ‘FiRC-NLP’ employed an ensemble of
SpanBERT + CRF with teacher enforcing. When run under lowercase preprocessing,
the setup led to the highest macro-F1 of $0.58$. The SpanBERT-based method,
‘FiRC-NLP,’ is also at par with the SpanBERT system of the baseline solution
[3]. Note that owing to a one-to-one mapping of tokens to span tags, we
discouraged the users from performing additional preprocessing. Meanwhile, the
second-best team ‘Mohammadmostafa78’ with a macro-F1 of $0.52$, overcame the
skewness in BIO notations by converting the label space to only BO and
employing an XLM-RoBERTa [23]+FNN setup. The third highest scoring teams,
‘CNLP-NITS-PP’ and ‘IRLab@IITBHU,’ have a macro-F1 of $51$, differing only
fourth decimal place. However, both employ distinct methods. While the former
employs a BERT+BiLSTM+FNN setup, the latter employs contextual embedding
(Glove) based BiLSTM+CRF setup akin to the existing baseline. Similar to the
observations in our baseline solutions, we observe that BiLSTM and contextual
embedding-based solutions perform considerably well. Overall, while
Transformer systems either in the form of BERT or SpanBERT help improve the
performance, a BiLSTM system trained via CRF is equally viable. We also
observe that the proposed systems submissions more or less follow the
performance trends of the existing baseline solutions, further corroborating
that combining transformer-based systems with CRF and BiLSTM attention
mechanisms is the optimal way to detect hateful spans.

Rank (Change in Rank) | Team Name | macro-F1 | # Submissions
---|---|---|---
1 (-) | FiRC-NLP | 0.57605 | 8
2 ($\uparrow$3) | Mohammadmostafa78 | 0.51382 | 2
3 ($\downarrow$1) | CNLP-NITS-PP | 0.50888 | 22
4 ($\downarrow$1) | IRLab@IITBHU | 0.50861 | 9
5 ($\downarrow$1) | Niranjan Rao | 0.49563 | 4
6 ($\downarrow$1) | TextShield | 0.45661 | 5
Table 4: Top-6 teams based on seqeval macro-F1 on the private leader baord.
The ranks are the final rank obtained on private leaderboard, and change in
rank is how many positions a team moved up or down in the table when the final
ranks were computed on the private board. We also enlist the number of
submissions made by the team during the competition.

##  6 Conclusion

Despite engaging with malicious content, some online users are adaptable and
can be persuaded to change their beliefs through empathy and corrective
conduct. Through this task, we aimed to help these users whose social
interactions can eventually be nudged to becoming non-hateful. We believe that
the proposed systems can be effectively utilized to assist the moderators.

## References

  * Pavlopoulos et al. [2021] J. Pavlopoulos, J. Sorensen, L. Laugier, I. Androutsopoulos,  SemEval-2021 task 5: Toxic spans detection,  in: Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Association for Computational Linguistics, Online, 2021, pp. 59–69. URL: https://aclanthology.org/2021.semeval-1.6. doi:10.18653/v1/2021.semeval-1.6.
  * Sundriyal et al. [2022] M. Sundriyal, A. Kulkarni, V. Pulastya, M. S. Akhtar, T. Chakraborty,  Empowering the fact-checkers! automatic identification of claim spans on Twitter,  in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022, pp. 7701–7715. URL: https://aclanthology.org/2022.emnlp-main.525. doi:10.18653/v1/2022.emnlp-main.525.
  * Masud et al. [2022] S. Masud, M. Bedi, M. A. Khan, M. S. Akhtar, T. Chakraborty,  Proactively reducing the hate intensity of online posts via hate speech normalization,  in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 3524–3534. URL: https://doi.org/10.1145/3534678.3539161. doi:10.1145/3534678.3539161.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin,  Attention is all you need,  in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Waseem and Hovy [2016] Z. Waseem, D. Hovy,  Hateful symbols or hateful people? predictive features for hate speech detection on Twitter,  in: Proceedings of the NAACL Student Research Workshop, Association for Computational Linguistics, San Diego, California, 2016, pp. 88–93. URL: https://aclanthology.org/N16-2013. doi:10.18653/v1/N16-2013.
  * Davidson et al. [2017] T. Davidson, D. Warmsley, M. Macy, I. Weber,  Automated hate speech detection and the problem of offensive language,  Proceedings of the International AAAI Conference on Web and Social Media 11 (2017) 512–515. URL: https://ojs.aaai.org/index.php/ICWSM/article/view/14955.
  * Kulkarni et al. [2023] A. Kulkarni, S. Masud, V. Goyal, T. Chakraborty,  Revisiting hate speech benchmarks: From data curation to system deployment,  in: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 4333–4345. URL: https://doi.org/10.1145/3580305.3599896. doi:10.1145/3580305.3599896.
  * Schmidt and Wiegand [2017] A. Schmidt, M. Wiegand,  A survey on hate speech detection using natural language processing,  in: L.-W. Ku, C.-T. Li (Eds.), Proc. of the Fifth International Workshop on Natural Language Processing for Social Media, ACL, Valencia, Spain, 2017, pp. 1–10. URL: https://aclanthology.org/W17-1101. doi:10.18653/v1/W17-1101.
  * Founta and Specia [2021] A. Founta, L. Specia,  A survey of online hate speech through the causal lens,  in: A. Feder, K. Keith, E. Manzoor, R. Pryzant, D. Sridhar, Z. Wood-Doughty, J. Eisenstein, J. Grimmer, R. Reichart, M. Roberts, U. Shalit, B. Stewart, V. Veitch, D. Yang (Eds.), Proceedings of the First Workshop on Causal Inference and NLP, Association for Computational Linguistics, Punta Cana, Dominican Republic, 2021, pp. 74–82. URL: https://aclanthology.org/2021.cinlp-1.6. doi:10.18653/v1/2021.cinlp-1.6.
  * Polignano et al. [2022] M. Polignano, G. Colavito, C. Musto, M. de Gemmis, G. Semeraro,  Lexicon enriched hybrid hate speech detection with human-centered explanations,  in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’22 Adjunct, Association for Computing Machinery, New York, NY, USA, 2022, p. 184–191. URL: https://doi.org/10.1145/3511047.3537688. doi:10.1145/3511047.3537688.
  * Stamou et al. [2022] V. Stamou, I. Alexiou, A. Klimi, E. Molou, A. Saivanidou, S. Markantonatou,  Cleansing & expanding the HURTLEX(el) with a multidimensional categorization of offensive words,  in: K. Narang, A. Mostafazadeh Davani, L. Mathias, B. Vidgen, Z. Talat (Eds.), Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), Association for Computational Linguistics, Seattle, Washington (Hybrid), 2022, pp. 102–108. URL: https://aclanthology.org/2022.woah-1.10. doi:10.18653/v1/2022.woah-1.10.
  * Pavlopoulos et al. [2022] J. Pavlopoulos, L. Laugier, A. Xenos, J. Sorensen, I. Androutsopoulos,  From the detection of toxic spans in online discussions to the analysis of toxic-to-civil transfer,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 3721–3734. URL: https://aclanthology.org/2022.acl-long.259. doi:10.18653/v1/2022.acl-long.259.
  * Agarwal et al. [2023] V. Agarwal, Y. Chen, N. Sastry, Haterephrase: Zero- and few-shot reduction of hate intensity in online posts using large language models, 2023. arXiv:2310.13985.
  * Mathew et al. [2021] B. Mathew, P. Saha, S. M. Yimam, C. Biemann, P. Goyal, A. Mukherjee,  Hatexplain: A benchmark dataset for explainable hate speech detection,  in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, pp. 14867–14875.
  * Hoang et al. [2023] P. G. Hoang, C. D. Luu, K. Q. Tran, K. V. Nguyen, N. L.-T. Nguyen,  ViHOS: Hate speech spans detection for Vietnamese,  in: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, Dubrovnik, Croatia, 2023, pp. 652–669. URL: https://aclanthology.org/2023.eacl-main.47. doi:10.18653/v1/2023.eacl-main.47.
  * Ranasinghe and Zampieri [2021] T. Ranasinghe, M. Zampieri,  MUDES: Multilingual detection of offensive spans,  in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, Association for Computational Linguistics, Online, 2021, pp. 144–152. URL: https://aclanthology.org/2021.naacl-demos.17. doi:10.18653/v1/2021.naacl-demos.17.
  * Das et al. [2023] M. Das, R. Raj, P. Saha, B. Mathew, M. Gupta, A. Mukherjee,  Hatemm: A multi-modal dataset for hate video classification,  Proceedings of the International AAAI Conference on Web and Social Media 17 (2023) 1014–1023. URL: https://ojs.aaai.org/index.php/ICWSM/article/view/22209. doi:10.1609/icwsm.v17i1.22209.
  * Zhou et al. [2023] L. Zhou, A. Caines, I. Pete, A. Hutchings,  Automated hate speech detection and span extraction in underground hacking and extremist forums,  Natural Language Engineering 29 (2023) 1247–1274. doi:10.1017/S1351324922000262.
  * Mittal et al. [2023] S. Mittal, M. Sundriyal, P. Nakov,  Lost in translation, found in spans: Identifying claims in multilingual social media,  arXiv:2310.18205 (2023).
  * Kennedy et al. [2020] B. Kennedy, X. Jin, A. Mostafazadeh Davani, M. Dehghani, X. Ren,  Contextualizing hate speech classifiers with post-hoc explanation,  in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 5435–5442. URL: https://aclanthology.org/2020.acl-main.483. doi:10.18653/v1/2020.acl-main.483.
  * Nakayama [2018] H. Nakayama, seqeval: A python framework for sequence labeling evaluation, 2018. URL: https://github.com/chakki-works/seqeval, software available from https://github.com/chakki-works/seqeval.
  * Joshi et al. [2020] M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, O. Levy,  SpanBERT: Improving pre-training by representing and predicting spans,  Transactions of the Association for Computational Linguistics 8 (2020) 64–77. URL: https://aclanthology.org/2020.tacl-1.5. doi:10.1162/tacl_a_00300.
  * Conneau et al. [2020] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov,  Unsupervised cross-lingual representation learning at scale,  in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 8440–8451. URL: https://aclanthology.org/2020.acl-main.747. doi:10.18653/v1/2020.acl-main.747.


# A Log-likelihood Regularized KL Divergence for Video Prediction With a 3D
Convolutional Variational Recurrent Network

Haziq Razali and Basura Fernando
A*STAR, Singapore
{Haziq_Razali<EMAIL_ADDRESS>
###### Abstract

The use of latent variable models has shown to be a powerful tool for modeling
probability distributions over sequences. In this paper, we introduce a new
variational model that extends the recurrent network in two ways for the task
of video frame prediction. First, we introduce 3D convolutions inside all
modules including the recurrent model for future frame prediction, inputting
and outputting a sequence of video frames at each timestep. This enables us to
better exploit spatiotemporal information inside the variational recurrent
model, allowing us to generate high-quality predictions. Second, we enhance
the latent loss of the variational model by introducing a maximum likelihood
estimate in addition to the KL divergence that is commonly used in variational
models. This simple extension acts as a stronger regularizer in the
variational autoencoder loss function and lets us obtain better results and
generalizability. Experiments show that our model outperforms existing video
prediction methods on several benchmarks while requiring fewer parameters.

##  1 Introduction

Generating the future frames given the past has been a long standing problem
in Computer Vision. Currently, recurrent neural networks, specifically a
variant with Long Short Term Memory (LSTM) cells [13], hold the state-of-the-
art results in a wide range of sequence based tasks including future frame
prediction [32, 17, 37, 35]. At a high level, they belong to the family of
autoregressive models where the predicted element is conditioned on the
history of inputs received thus far. From video analysis [5] to speech
recognition [10], text generation [30], machine translation [31] and image
captioning [34], the versatility of recurrent networks has proven to be an
indispensable tool for machine learning practitioners.

There is evidence that the introduction of uncertainty into the hidden states
of a recurrent network can significantly improve its performance when
modelling complex sequences such as speech and music [1, 4, 7, 9]. These
methods integrate the Variational Autoencoder (VAE) [16] to infer the latent
variables which is shown to capture some form of semantic abstraction such as
the thickness or orientation of an MNIST digit from the observed data by
better capturing the input distribution.

Despite so, the state-of-the-art in future frame prediction using recurrent
networks have come from purely deterministic auto-encoder type models [37,
35]. We believe this to be the consequence of 2 factors: (a) that the use of
latent random variables to model sequential data for generation often result
in blurry reconstructions as evidenced in the literature [40], perhaps due to
the lack of a proper temporal model and (b) that the regularizer employed is
not sufficient for capturing the properties of the encoded distribution. In
particular, a VAE is an autoencoder where the training process is regularized
to ensure that the latent space captures the input distribution accurately.
This regularization is usually employed by minimizing the Kullback-Liebler
(KL) divergence between the encoded posterior and prior distributions. Here,
the posterior is assumed to be a standard Gaussian distribution and the
encoder is then trained to return the mean and the covariance of the posterior
Gaussian distribution. to the future). In this paper, we address the above
mentioned problems and propose a new architecture for the prediction of future
frames. We extend the Variational Recurrent Neural Network (VRNN) [4] firstly
by replacing all fully-connected and 2D convolutions in the architecture with
3D convolutions to increase its capacity to model temporal information. We use
a truncated ResNet [12, 11] with 3D convolutions for both the image encoder
and decoder, a shallow 3D convolutional network to generate the prior and
posterior distributions and, a 3D Convolutional LSTM (ConvLSTM) [26] as the
recurrent model. Since the architecture is fully fitted with 3D convolutions
that share parameters across both space and time, it can now better exploit
spatiotemporal dependencies and more importantly, preserve temporal
information across each component and operation thereby removing the LSTM’s
complete dependency on the hidden states for motion information. Similarly,
using 3D convolutions in the image decoder allows it to generate high quality
predictions by considering the spatiotemporal correlations of the 4D feature
maps in contrast to 2D convolutions. Finally, we choose to truncate our
3D-ResNet in order to leverage its power as a spatiotemporal feature extractor
while minimizing the total number of parameters. We argue that features
extracted by the lower layers of a 3D-ResNet are already abstract enough for
the ConvLSTM to further learn on and would therefore rather reallocate the
freed space to the ConvLSTM. This is in contrast to [3] that uses the full
2D-ResNet for feature extraction. On the other hand, we want to avoid forgoing
the image encoder as in [37, 35] since it puts too much reliance on the
ConvLSTM to jointly learn short-term spatiotemporal features and long term
dynamics from a raw sequence of images.

Next, we further regularize the latent space of the variational recurrent
model using a novel latent loss that combines the KL divergence and the log-
likelihood criterion. Specifically, we further constraint the latent space by
maximizing the likelihood of the prior mean with respect to the posterior
distribution assuming that conditionally, the prior given the posterior also
follows a normal distribution. We will mathematically show in section 3.2 that
this additional constraint lets the prior variance be larger while reducing
the divergence between the prior and posterior mean distributions. Ultimately,
we will show in our experiments that our novel objective function combined
with our architectural design choice lets us outperform the state of the art
while requiring fewer parameters.

In summary, we make three contributions. First, we present a VRNN that uses 3D
convolutions across the entire architecture and show their effectiveness for
future frame prediction. Second, we extend the KL divergence by introducing a
novel log-likelihood criterion to the latent loss used in variational models.
This new loss further regularizes the latent space and allows us to obtain
better results. Finally, we show through experiments that each individual
contribution improves the model and that their combination allows the model to
outperform existing state-of-the-art video prediction methods.

##  2 Related Work

Recurrent Networks used to predict the future frames can be grouped into two
categories: (1) those that are entirely deterministic and (2) those that
propagate uncertainty through the recurrent network via latent random
variables.

Recurrent networks were first used for future frame prediction in [21] when
Ranzato et al. learnt a model to predict a quantized space of image patches.
Srivastava et al. [29] proposed a model to predict the future as well as the
input sequence in order to prevent the model from storing information only
about the last few frames. Shi et al. [26] proposed an extension of the LSTM
by replacing the fully connected structure with one that is fully
convolutional which saw popular use to date for learning sequential data with
spatial information. Finn et al. [6] used an LSTM framework to model motion
via transformations of groups of pixels. Patraucean et al. [19] and Villegas
et al. [33] explicitly injected short term motion information through the use
of optical flow. Xu et al. [38] proposed a two-stream recurrent network to
deal with the high and low frequency content often present in natural videos.
Kalchbbrenner et al. [14] introduced a model that learns the joint
distribution of the raw pixels to generate them one at a time. Wang et al.
[37] proposed to improve the stacked LSTM by having the memory and hidden
states flow in a zig-zagged manner from the highest unit of the current
timestep to the lowest unit of the subsequent timestep. This was further
improved in [35] by replacing the 2D convolutions with 3D and a memory
attention in the LSTM itself. We also use 3D convolutions throughout our
entire architecture but in contrast, our model is stochastic.

Stochastic recurrent networks vary in way they propagate uncertainty across
time as well as the way inference is computed. For instance, Bayer et al. [1]
and Goyal et al. [9] conditioned the generation only on the hidden states of
the recurrent network whilst Chung et al. [4] and Fraccaro et al. [7] have the
output be some function over both the hidden states and the latent vector.
Next, the LSTM state transitions in [1, 4, 9] are additionally conditioned on
the latent vectors whereas in [7] is not. The work of [4] was later extended
in [3] through a hierarchy of latent variables for future frame prediction. We
propagate stochastic information in the same way as [4] except that the latent
tensors themselves now contain richer spatial-temporal information since they
are the result of 3D convolutions. For inference, both [7] and [9] run a
deterministic recurrent network backwards through the sequence to form the
approximate posterior whereas the posterior in [1] and [4] is computed using
only information up till the present. Similarly, our method for inference
follows that of [4] but in contrast to [4] and in fact all existing methods,
we jointly optimize both the KL divergence and a novel log likelihood
criterion.

Future frame prediction is useful for various application such as early action
prediction[23, 27, 24] and action anticipation [8, 18, 20]. Some anticipation
methods generate future RGB images and then classify them into human actions
using convolution neural networks [39, 36]. However, generation of RGB images
for the future is a very challenging task specially for longer action
sequences. Similarly, some methods aim to generate future motion images and
then try to predict action for the future [22].

##  3 Our model

Given a sequence of frames $\textbf{x}_{1:C-1}$ as context, our goal is to
learn a model that can predict $T$ frames into the future i.e.
$\hat{\textbf{x}}_{C:C+T}$. This task is challenging due to the variability
present in video sequences and the fact that there can exist multiple
plausible futures for any given input. To overcome this, we propose to use a
VRNN but with 3D convolutions in order to better capture both short and long
term relations. Specifically, in contrast to existing VRNN models, our
encoder, decoder, prior and posterior networks, and LSTM are all built using
3D convolutions and up-convolutions. We also further regularize the latent
space of the VRNN with an additional log-likelihood term. We begin the next
section with a brief review of the VRNN before describing our novel latent
loss function and the architecture of our 3D VRNN.

###  3.1 Variational Recurrent Neural Network

Figure 1 provides a graphical illustration of the VRNN. The VRNN uses a latent
variable $\textbf{z}_{t}$ at each timestep of a recurrent network to capture
the variations in the observed data. It contains a VAE at every timestep whose
mean $\mu_{t}$ and variance $\sigma_{t}$ are conditioned on the hidden unit
$h_{t}$ of a recurrent network. These parameters are then used to sample the
latent variable $\textbf{z}_{t}$ at each timestep. Concisely, the forward pass
can be completely described by the following set of recurrence equations where
the subscripts p and q denote the prior and posterior distributions
respectively, and the components $\text{f}_{\text{p}}$, $\text{f}_{\text{q}}$,
$\text{f}_{\text{enc}}$, $\text{f}_{\text{dec}}$ are functions implemented
using neural networks.

---
Figure 1: Graphical illustration of the Variational Recurrent Network. The
dotted lines denote the posterior network $\text{f}_{\text{q}}$ that is only
used during training and is discarded at test time.

$\displaystyle\mu_{\text{p},t},\sigma_{\text{p},t}$
$\displaystyle=\text{f}_{\text{p}}(h_{t-1})$ (1)
$\displaystyle\mu_{\text{q},t},\sigma_{\text{q},t}$
$\displaystyle=\text{f}_{\text{q}}(h_{t-1},\text{f}_{\text{enc}}(x_{t}))$ (2)
$\displaystyle z_{\text{p},t}$ $\displaystyle\sim
N(\mu_{\text{p},t},\sigma_{\text{p},t})$ (3) $\displaystyle z_{\text{q},t}$
$\displaystyle\sim N(\mu_{\text{q},t},\sigma_{\text{q},t})$ (4)
$\displaystyle\hat{x}_{t}$
$\displaystyle=\text{f}_{\text{dec}}(z_{\text{p},t},h_{t-1})$ (5)
$\displaystyle h_{t}$
$\displaystyle=\text{LSTM}(\text{f}_{\text{enc}}(x_{t}),h_{t-1},z_{\text{p},t})$
(6)

Here $N(\mu,\sigma)$ is a multivariate Gaussian distribution with mean $\mu$
and co-variance diag$(\sigma^{2})$. Note that the posterior network
$\text{f}_{\text{q}}$ is used only during training and is discarded at test
time. The entire model is then trained end-to-end for future frame prediction
by minimizing a sum of the reconstruction loss ($L_{\text{rec}}$), and latent
loss ($L_{\text{latent}}$) expressed as:

$\displaystyle
L=\lambda_{\text{rec}}L_{\text{rec}}+\lambda_{\text{latent}}L_{\text{latent}}$
(7)

where $\lambda_{\text{rec}}$ and $\lambda_{\text{latent}}$ are the trade off
hyper-parameters and the latent loss is the timestep-wise KL divergence
($L_{\text{KL}}$) between the prior ($p$) and posterior ($q$) distributions
and is expressed as:

$\displaystyle L_{\text{KL}}=\sum_{t=1}^{T}\text{KL}(q(z_{t}|X_{\leq
t},Z_{<t})||p(z_{t}|X_{<t},Z_{<t})))=$ (8)
$\displaystyle\sum_{t=1}^{T}\log(\sigma_{\text{q,t}})-\log(\sigma_{\text{p,t}})+\frac{\sigma_{\text{p,t}}^{2}+(\mu_{\text{p,t}}-\mu_{\text{q,t}})^{2}}{2\sigma_{\text{q,t}}^{2}}-0.5$
(9)

---
Figure 2: Our proposed architecture for future frame prediction. The
architecture inputs and outputs at each timestep a sequence of M video frames
with the prediction made H timesteps into the future. The entire architecture
is fitted with 3D convolutions. The 3D-ENC and 3D-DEC are mirrored versions of
the 2-block 3D-ResNet18 as shown in Figure 3. We use 2 LSTM layers and a
shallow 3D conv network to generate $\mu$ and $\sigma$ that are then sampled
from to produce z. The block $[\circ,\circ]$ indicates a concatenation along
the 4th axis. The values above each component (3D-ENC, CONVLSTM, Z, 3D-DEC)
indicate the sizes of the output tensor.

###  3.2 New log-likelihood regularized KL divergence

Typically, the KL divergence-based latent loss is used to regularize the
latent space, enforcing it to be a Gaussian distribution with known
parameters. We further enhance this regularization by appending the negative
of the log-likelihood term to the latent loss. The objective here is to
maximize the likelihood of the prior mean distribution w.r.t. the posterior.
This is done by minimizing the negative likelihood as shown in Eq. 11 by
assuming that the prior, posterior and _the conditional prior mean_ given the
posterior all follow a Gaussian distribution.

$\displaystyle-L_{\text{LL}}$
$\displaystyle=-\log\prod_{t=1}^{T}p(\mu_{\text{p},t}|\mu_{\text{q},t},\sigma_{\text{q},t})$
(10)
$\displaystyle=\sum_{t=1}^{T}\log({\sigma_{\text{q},t}})+(\frac{\mu_{\text{p},t}-\mu_{\text{q},t}}{\sigma_{\text{q},t}})^{2}$
(11)

The proposed latent loss is thus expressed together as:

$\displaystyle L_{\text{KL}}-L_{\text{LL}}$
$\displaystyle=\log(\sigma_{\text{q}})-\log(\sigma_{\text{p}})+\frac{\sigma_{\text{p}}^{2}+(\mu_{\text{p}}-\mu_{\text{q}})^{2}}{2\sigma_{\text{q}}^{2}}-0.5$
$\displaystyle+\log({\sigma_{\text{q}}})+(\frac{\mu_{\text{p}}-\mu_{\text{q}}}{\sigma_{\text{q}}})^{2}$
(12)
$\displaystyle=2\log(\sigma_{\text{q}})-\log(\sigma_{\text{p}})+\frac{\sigma_{\text{p}}^{2}+3(\mu_{\text{p}}-\mu_{\text{q}})^{2}}{2\sigma_{\text{q}}^{2}}-0.5$
(13)

Interestingly, the above equation is similar to the KL divergence (eq 9) but
with some of its components weighted differently. In particular, the log
posterior variance, $\log(\sigma_{q})$, has been scaled by a factor of 2 and
the squared difference of mean, $(\mu_{p}-\mu_{q})^{2}$, by a factor of 3.
This modification has 2 effects. First, it puts more emphasis on sample
diversity since the log prior variance $\log(\sigma_{p})$ put out by the
network must now be higher in order to match the scaled log posterior variance
$2\log(\sigma_{q})$. Secondly, the scaled difference of mean
$3(\mu_{p}-\mu_{q})^{2}$ serves to balance out the additional weight assigned
to the variance term and thus encourages the model to continue generating
samples that are representative of the dataset. As such, the log-likelihood
regularized KL divergence should have no adverse effects on the model since it
is simply the KL divergence with a reweighting of its components and would
argue it to be more forceful if one needs to have a greater emphasis on sample
diversity. Interestingly, the weights for each component can also be
customized although their individual effects will not be investigated since it
is not the purpose of this paper. All-in-all, our new loss function for
training the VRNN is expressed together as:
$L=\lambda_{\text{rec}}L_{\text{rec}}+\lambda_{\text{latent}}(L_{\text{KL}}-L_{\text{LL}})$.

###  3.3 Our 3D Convolutional VRNN

The ConvLSTM was proposed in [26] to address the shortcomings of Fully-
Connected LSTM, namely that latter always ends up decimating any spatial
information contained in the input tensor. Intuitively, if the states are
viewed as the hidden representations of moving objects, then a ConvLSTM with a
larger kernel should be able to capture faster motions while one with a
smaller kernel can capture slower motions. However, if the input at each
timestep is a single image, then the hidden states are the only component that
carry motion information in both Fully-Connected and ConvLSTMs.

In our work, we counteract this limitation by replacing all 2D convolutions
(and de-convolutions) with 3D to enable every component to retain motion
information instead of only the hidden states. The benefits of this are two-
fold. First, the 3D ConvLSTM is no longer completely reliant on the hidden
states for motion information since there is an additional source coming from
the 3D image encoder. Specifically, the use of 3D convolutions on multiple
frames result in an input tensor that carries short-term spatiotemporal
information as opposed to a VRNN that run a 2D convolution on a single frame
at every timestep. Second, we can now vary the window size and output horizon
at each timestep without needing to redesign the architecture. For example,
let us define M to be the window size, or the number of input frames to our
model at each timestep and $\mathcal{H}$ the output horizon (output frames),
or how far into the future should the model predict. Then, 3D convolutions
(and de-convolutions) allow us to set a large M to efficiently capture large
motions when dealing with datasets where the motion between frames is
prevalently large and conversely, a large $\mathcal{H}$ to predict many frames
into the future at once with minimal reconstruction errors if said motion
between frames is small. All in all, this upgrade renders our 3D convolutional
VRNN more effective and general than its 2D counterpart.

Our proposed architecture is shown in Figure 2. The encoder (3D-ENC) takes at
each timestep a clip of M video frames of shape [M,H,W,C] to produce a tensor
of shape [M/2, H/4, W/4, C/4] where H,W,C denote the height, width and
channels respectively. This tensor is then concatenated with the latent tensor
Z (a zero tensor in the 1st timestep) along the 4’th channel indicated by
$[\circ,\circ]$ then passed to the 3D ConvLSTM with 2 hidden layers for motion
learning. The LSTM hidden states at the second level with a shape of [M/2,
H/4, W/4, 128] are then fed through a shallow 3D CNN with two heads to produce
the parameters of the prior distribution with shape [M/2, H/4, W/4, 16] that
are later sampled to produce the latent tensor Z. This latent tensor is then
concatenated with the hidden states and finally propagated through the decoder
(3D-DEC) to predict the frames H timesteps into the future. The model is
applied recursively by using the newly generated frame as input if the ground
truth is not available. Specifically, if the frames are observed up to time C,
then the model will use the ground truth as input up till time C-M then a
combination of the ground truth and predicted frames between time C-M to C,
and then finally, only the predicted frames as input from time C onwards.
During training, a separate 3D encoder (not shown in Figure 2) is used to
generate the posterior distribution to optimize the KL divergence.

---
(a) 2-block 3D ResNet-18 encoder
(b) 2-block 3D ResNet-18 decoder
Figure 3: Architecture of our encoder and decoder. The encoder outputs a 4D
tensor with a spatial resolution of (H/4,W/4). Exception the decoder, each
filter output is followed by a 3D batch-norm and ReLU. An downsampling
operation with stride 2 is indicated by ”//2” and an upsampling with stride 2
by ”x2”.

As shown in Figure 3, we use a 2-block 3D ResNet-18 for both the encoder and
decoder in contrast to [3] that use a full ResNet. We find this to be
sufficient especially since the 3D ConvLSTM itself serve as an extension of
the 3D CNN for learning complex spatiotemporal features given a window of M
frames. Furthermore, by truncating the number of blocks to 2, we reduce the
total number of parameters significantly which allow us to devote additional
resources to our 3D ConvLSTM with 128 hidden units that contains 7m parameters
per level. However, we also want to avoid the other end of not having a
feature extractor at all [37, 35] since they have been shown to extract useful
features that tend to be task specific at the higher blocks and more general
purpose at the lower blocks. In short, we propose to use a smaller feature
extractor CNN and a larger LSTM. We show in our experiments that modelling the
architecture in such a manner allows us to outperform the state-of-the-art
while requiring fewer parameters.

##  4 Experiments

Ground Truth |
---|---
E3D LSTM |
Ours |
Ground Truth |
E3D LSTM |
Ours |
Figure 4: Prediction on the KTH action dataset. Our method recovers the disappearing man. Input |  |
---|---|---
GroundTruth |  |
ConvLSTM |  |
VarConvLSTM |  |
E3DLSTM |  |
Ours |  |
Figure 5: Prediction on the moving MNIST dataset. We obtain visually pleasing
results even on complex example shown in right. Furthermore, our results
indicate less blur.

###  4.1 Comparison to state-of-the-art

We compare our approach to several state-of-the-art methods using publicly
available source code and model where available with default parameters and
using standard metrics such as frame-wise Mean Squared Error (MSE), Structural
Similarity Index (SSIM) and Peak Signal to Noise Ratio (PSNR). We sample 50
predictions from the stochastic models for each ground truth test sequence and
average the metrics across the test set. Note that sampling is done only for
the purpose of evaluation in order to get the average performance and is not
required for deployment.

Training Details: We initialize the weights of our truncated 3D ResNet-18
encoder and decoder with weights pre-trained on the Kinetics-400 dataset [15]
and all other components using PyTorch’s default initializer. We use the Adam
optimizer with default hyperparameters, a learning rate of $10^{-3}$ with no
weight decay, a batch size of 6 and the L1+L2 reconstruction loss that was
also used in [37, 35]. We train the model using beta warm-up [28] and have it
gradually predict into the future using its own predictions as input [2].

The Moving MNIST dataset [29] consists of two digits (0 to 9) of size 28 x 28
moving inside a 64 x 64 patch. The digits are chosen randomly from the MNIST
training set and placed at random locations inside the patch. Each digit is
assigned a velocity whose direction is chosen uniformly at random on a unit
circle and whose magnitude is also chosen uniformly at random over a fixed
range. The digits bounce off the edges of the 64 x 64 frame and overlap as
they move past each other. The training set contains 10,000 sequences while
the validation and test sets 1,000 sequences each. By default, the sequences
are all 20 frames long and the models are trained to predict the next 10
frames given the first 5 or 10 as input.

Table 1 shows the performance of the models when using 5 frames to predict 10
and 15 frames into the future and when using 10 frames to predict 10 and 20
frames into the future. Our method demonstrates its promise, outperforming
both the state-of-the-art deterministic (E3D-LSTM [35]) and stochastic (VRNN
[4]) models, with the latter by a large margin. We also improve over the
E3D-LSTM [35] despite having fewer parameters. We were only able to train the
smallest variant of the models presented in [3] which nevertheless contains 62
million parameters. Interestingly, we also outperform them. These results thus
indicate the impact of our new design and the novel latent loss.

Type | Model | $\text{x}_{1:5}\rightarrow\hat{\text{x}}_{6:15}$ | $\text{x}_{1:5}\rightarrow\hat{\text{x}}_{6:20}$ | $\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:20}$ | $\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:30}$ | # Params
---|---|---|---|---|---|---
SSIM | MSE | SSIM | MSE | SSIM | MSE | SSIM | MSE
Deterministic | 2D ConvLSTM [26] | 0.662 | 111.1 | 0.482 | 154.3 | 0.763 | 82.2 | 0.660 | 112.3 | 2.8M
PredRNN++ [37] | 0.793 | 66.2 | 0.769 | 79.2 | 0.870 | 47.9 | 0.821 | 57.7 | 15.4M
E3D-LSTM [35] | 0.853 | 53.4 | 0.801 | 64.1 | 0.910 | 41.3 | 0.872 | 47.6 | 38.7M
Stochastic | Variational 2D ConvLSTM [4] | 0.733 | 91.1 | 0.564 | 126.4 | 0.816 | 60.7 | 0.773 | 83.5 | 2.9M
Improved VRNN [3] | 0.772 | 123.1 | 0.728 | 162.2 | 0.776 | 129.2 | 0.699 | 194.3 | 62M
| Variational 3D ConvLSTM (Ours) | 0.864 | 51.4 | 0.805 | 63.2 | 0.896 | 39.4 | 0.874 | 47.54 | 12.9M

Table 1: Results on the Moving MNIST dataset when using 5 frames to predict 10 ($\text{x}_{1:5}\rightarrow\hat{\text{x}}_{6:15}$) and 15 ($\text{x}_{1:5}\rightarrow\hat{\text{x}}_{6:20}$) frames into the future, and when using 10 frames to predict 10 ($\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:20}$) and 20 ($\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:30}$) frames into the future. The metrics are computed frame-wise. Higher SSIM or lower MSE scores indicate better results. Finally, the rightmost column indicate the number of parameters for the various models. | Model | $\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:30}$ | $\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:50}$ | $\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:70}$
---|---|---|---
SSIM | PSNR | SSIM | PSNR | SSIM | PSNR
2D ConvLSTM [26] | 0.712 | 23.58 | 0.639 | 22.85 | 0.551 | 20.13
PredRNN++ [37] | 0.865 | 28.47 | 0.741 | 25.21 | 0.702 | 23.51
E3D-LSTM [35] | 0.879 | 29.31 | 0.810 | 27.24 | 0.798 | 26.82
Variational 2D ConvLSTM [4] | 0.787 | 25.76 | 0.733 | 24.83 | 0.672 | 23.13
Variational 3D ConvLSTM (Ours) | 0.866 | 28.31 | 0.852 | 27.89 | 0.846 | 27.66
Table 2: Results on the KTH action dataset when using 10 frames to predict 20
($\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:30}$), 40
($\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:50}$), and 60
($\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:70}$) time steps into the
future. The metrics are computed frame-wise. Higher SSIM and PSNR scores
indicate better results. The bar chart on the right highlights the difference
between our model and the E3D-LSTM. Our model performs much better for longer
predictions.

We present some visual results in Figure 5 where the first row illustrates the
input sequence $\textbf{x}_{1:10}$, the second the ground truth for the
predicted sequence $\textbf{x}_{11:20}$ and all subsequent rows the
predictions made by the various models $\hat{\textbf{x}}_{11:20}$. It can
firstly be seen that the injection of stochasticity causes the Variational
ConvLSTM to output predictions that are blurrier than its deterministic
counterpart. This could principally be due to the fact that information coming
from the latent nodes act as noise and thus interferes with reconstruction.
Unlike the ConvLSTM however, the digits generated by the Variational ConvLSTM
are closer to the ground truth resulting in a performance that is generally
superior as evidenced by the quantitative scores in Table 1. We can then
observe our model producing the best results. This signals that the blurry
reconstructions manifested by the Variational ConvLSTM are counteracted by
replacing all 2D convolutions with 3D. Intuitively, our novel loss also acts
as a stronger regularizer for the reconstructions.

The KTH dataset [25] consists of humans performing 6 types of actions: boxing,
clapping, waving, jogging, running, and walking under 4 scenarios: outdoors,
outdoors with scale variation, outdoor with different clothes, and indoors
with a homogeneous and static background. Each video is recorded at 25 fps and
lasts an average of 4 seconds. We follow the experimental setup in [33] using
persons 1-16 for training and 17-25 for testing and resize each frame to
128x128 pixels. The models are trained to predict the next 10 frames given the
first 10 as input. Table 2 presents the performance of the various models when
predicting the next 20, 40 and 60 frames. It can be observed that our model
lags slightly behind the E3D-LSTM when predicting short term but performs much
better when tasked to predict further into the future. This difference is
highlighted on the bar chart beside Table 2 that shows the performance of our
model degrading at a slower rate than the E3D-LSTM. The results can be
explained by Figure 4 where each row represents the output sequence
$\hat{\textbf{x}}_{11:50}$ spaced 2 frames apart. It can be observed from the
figures that the predictions coming from our model are blurrier than E3D,
resulting in metrics that are inferior although we make up for it by being
able to predict the individual re-entering the scene. The quantitative scores
also show that our variational method is much better than the Variational 2D
ConvLSTM [4]. These findings once again demonstrate the effectiveness of our
architecture for modelling spatiotemporal data.

Model | $\text{x}_{1:5}\rightarrow\hat{\text{x}}_{6:15}$ | $\text{x}_{1:5}\rightarrow\hat{\text{x}}_{6:20}$ | $\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:20}$ | $\text{x}_{1:10}\rightarrow\hat{\text{x}}_{11:30}$ | # Params
---|---|---|---|---|---
SSIM | MSE | SSIM | MSE | SSIM | MSE | SSIM | MSE
2D ConvLSTM [26] | 0.662 | 111.1 | 0.482 | 154.3 | 0.763 | 82.2 | 0.660 | 112.3 | 2.8M
Variational 2D ConvLSTM [4] | 0.733 | 91.1 | 0.564 | 126.4 | 0.816 | 60.7 | 0.773 | 83.5 | 2.9M
Variational 3D ConvLSTM | 0.857 | 52.1 | 0.797 | 63.8 | 0.887 | 41.8 | 0.868 | 49.7 | 12.9M
Variational 3D ConvLSTM + LL Criterion (ours) | 0.864 | 51.4 | 0.805 | 63.2 | 0.896 | 39.4 | 0.874 | 47.5 | 12.9M

Table 3: Ablation study on the Moving MNIST dataset. The metrics are computed frame-wise. Higher SSIM or lower MSE scores indicate better results. |
---|---
|
Figure 6: The effect of M window size and the output horizon $\mathcal{H}$ on
the performance. The first row shows the SSIM and MSE scores on the moving
MNIST action dataset, while the second row the SSIM and PSNR on the KTH action
dataset.

###  4.2 Ablation Study

Effectiveness of each component: We quantize the effect of each contribution
in Table 3 on the moving MNIST dataset. It can be seen that the introduction
of stochasticity into the recurrent network allows it to better tackle
uncertainties in the recurrent dynamics which results in better predictions,
significantly lowering the MSE from 82.2 to 60.7. Additionally, swapping out
the 2D convolutions in place for 3D brings about significant improvements to
the model, lowering the MSE by approximately another 20 points. This makes
sense since 3D convolutions operate on both the spatial and temporal axis,
letting the architecture capture relationships in said dimensions. Finally, it
is quite apparent that the introduction of the log-likelihood criterion has a
noticeable effect, further bringing down the MSE by approximately 2 points.
This can be interpreted in various ways: (1) that the resulting loss function
empirically helps the network traverse towards a better local minima and (2)
that the added regularizer helps the recurrent model strengthen its ability at
expressing complex distributions. Intuitively, appending the log-likelihood
criterion to the KL divergence has some conceptual similarity to the use of
the L1+L2 loss functions that has been empirically shown in [37, 35] to be
better than the individual counterparts. In conclusion, each component brings
a definitive upgrade to the model and together, lend it the advantage it needs
to outperform the deterministic and stochastic state-of-the-art models [4, 35,
37, 3].

Window size and output horizon: Recall from section 3.3 that the advantages 3D
convolutions have over its 2D counterpart when paired with an LSTM is that (1)
the LSTM state transitions are no longer completely reliant on its hidden
states for motion information and (2) that one could vary the window size (M)
and the output horizon ($\mathcal{H}$) at each timestep without having to
change the architecture. In this experiment, we conduct additional studies on
our model where we varied M and $\mathcal{H}$, the number of input frames and
the output horizon respectively at each timestep to study the effects
different design choices have on our model. As expected, the plots in Figure 6
show a degradation in the metrics over time regardless of the configuration in
use. The KTH plots exhibits an exponential decay whereas the moving MNIST is
more linear. The plots also show that a smaller window size is better for the
MNIST dataset but has no clear difference for the KTH action dataset.
Interestingly, there are cases that favour a large window size and output
horizon and some other cases, that do not. For longer predictions however, the
plots show that the various configurations are quite similar in terms of
performance.

##  5 Conclusion

We have presented a deep neural network for future frame prediction that
performs well at predicting long term. Our method uses 3D convolutions
throughout the entire architecture and is trained using a latent loss that
includes a _specific log-likelihood criterion_. Theoretically and
experimentally, we have shown the effects of these two contributions. First,
the use of latent random variables in a 3D recurrent model enables it to
persistently generate predictions well beyond the time steps it was trained
for and second, the log-likelihood criterion helps direct the model towards a
better solution without an increase in model complexity. This model
outperforms prior stochastic methods by a good margin while obtaining a
performance that is on-par with the state-of-the-art deterministic models for
short-term future frame prediction while being superior when generating even
further into the future.

We also proposed the benefits of using a smaller convolutional network for
encoding and decoding videos as it alleviates the burden on the ConvLSTM to
jointly learn short-term spatiotemporal features and long-term dynamics, while
continuing to keep the total number of parameters manageable. We use a
truncated 3D ResNet-18 (2 blocks) which reduces the total number of parameters
by 60 million as the low-level spatiotemporal features captured by the first
few blocks of the 3D ResNet-18 are sufficient for the ConvLSTM to further
learn. We believe these findings are useful for the development of more
efficient and effective spatiotemporal models with variational recurrent
architectures.

Acknowledgment This research/project is supported by the National Research
Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-
RP-2019-010).

## References

  * [1] Justin Bayer and Christian Osendorfer.  Learning stochastic recurrent networks.  NIPS, 2016.
  * [2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer.  Scheduled sampling for sequence prediction with recurrent neural networks.  NIPS, 2015.
  * [3] Lluis Castrejon, Nicolas Ballas, and Aaron Courville.  Improved conditional vrnns for video prediction.  ICCV, 2019.
  * [4] Junyoung Chung, Kyle Kastner, Laurent Dinh, Aaron Courville Kratarth Goel, and Yoshua Bengio.  A recurrent latent variable model for sequential data.  NIPS, 2015.
  * [5] Jeff Donahue, Lisa Anne Hendricks, Subhashini Venugopalan Marcus Rohrbach and, Sergio Guadarrama, Kate Saenko, and Trevor Darrell.  Long-term recurrent convolutional networks for visual recognition and description.  CVPR, 2015.
  * [6] Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  NIPS, 2016.
  * [7] Marco Fraccaro, Soren Kaae Sonderbyz, Ulrich Paquet, and Ole Winther.  Sequential neural models with stochastic layers.  NIPS, 2016.
  * [8] Antonino Furnari and Giovanni Maria Farinella.  What would you expect? anticipating egocentric actions with rolling-unrolling lstms and modality attention.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6252–6261, 2019.
  * [9] Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Cote, Nan Rosemary Ke, and Yoshua Bengio.  Z-forcing: Training stochastic recurrent networks.  NIPS, 2017.
  * [10] Alex Graves and Navdeep Jaitly.  Towards end-to-end speech recognition with recurrent neural networks.  Proceedings of Machine Learning Research, 2014.
  * [11] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Learning spatio-temporal features with 3d residual networks for action recognition.  ICCVW, 2016.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CVPR, 2016.
  * [13] Sepp Hochreiter and Jurgen Schmidhuber.  Long short-term memory.  Neural computation, 1997.
  * [14] Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu.  Video pixel networks.  ICML, 2017.
  * [15] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The kinetics human action video dataset.  arXiv, 2017.
  * [16] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  International Conference on Learning Representations, 2013.
  * [17] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine.  Stochastic adversarial video prediction.  arXiv preprint arXiv:1804.01523, 2018.
  * [18] Antoine Miech, Ivan Laptev, Josef Sivic, Heng Wang, Lorenzo Torresani, and Du Tran.  Leveraging the present to anticipate the future in videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [19] Viorica Patraucean, Ankur Handa, and Roberto Cipolla.  Spatio-temporal video autoencoder with differentiable memory.  ICLR Workshop, 2016.
  * [20] Siyuan Qi, Siyuan Huang, Ping Wei, and Song-Chun Zhu.  Predicting human activities using stochastic grammar.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1164–1172, 2017.
  * [21] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra.  Video (language) modeling: a baseline for generative models of natural videos.  ArXiv, 2014.
  * [22] Cristian Rodriguez, Basura Fernando, and Hongdong Li.  Action anticipation by predicting future dynamic images.  In ECCV, pages 0–0, 2018.
  * [23] Michael S Ryoo.  Human activity prediction: Early recognition of ongoing activities from streaming videos.  In ICCV, pages 1036–1043. IEEE, 2011.
  * [24] Mohammad Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Basura Fernando, Lars Petersson, and Lars Andersson.  Encouraging lstms to anticipate actions very early.  In Proceedings of the IEEE International Conference on Computer Vision, pages 280–289, 2017.
  * [25] Christian Schuldt, Ivan Laptev, and Barbara Caputo.  Recognizing human actions: A local svm approach.  ICPR, 2004.
  * [26] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai kin Wong, and Wang chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  Neural Information Processing Systems, 2015.
  * [27] Yuge Shi, Basura Fernando, and Richard Hartley.  Action anticipation with rbf kernelized feature mapping rnn.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 301–317, 2018.
  * [28] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther.  Ladder variational autoencoders.  Neural Information Processing Systems, 2016.
  * [29] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.  Unsupervised learning of video representations using lstms.  ICML, 2015.
  * [30] Ilya Sutskever, James Martens, and Geoffrey E Hinton.  Generating text with recurrent neural networks.  In ICML, 2011.
  * [31] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks.  Neural Information Processing Systems, 2014.
  * [32] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Le, and Honglak Lee.  High fidelity video prediction with large stochastic recurrent neural networks.  In Advances in Neural Information Processing Systems, pages 81–91, 2019.
  * [33] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee.  Decomposing motion and content for natural video sequence prediction.  ICLR, 2017.
  * [34] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  Computer Vision and Pattern Recognition, 2015.
  * [35] Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei.  Eidetic 3d lstm: A model for video prediction and beyond.  ICLR, 2019.
  * [36] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip.  Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms.  In NIPS, pages 879–888, 2017.
  * [37] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S. Yu.  Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms.  NIPS, 2017.
  * [38] Jingwei Xu, Bingbing Ni, Zefan Li, Shuo Cheng, and Xiaokang Yang.  Structure preserving video prediction.  CVPR, 2018.
  * [39] Kuo-Hao Zeng, William B Shen, De-An Huang, Min Sun, and Juan Carlos Niebles.  Visual forecasting by imitating dynamics in natural sequences.  In ICCV, pages 2999–3008, 2017.
  * [40] Shengjia Zhao, Jiaming Song, and Stefano Ermon.  Towards a deeper understanding of variational autoencoding models.  arXiv, 2017.