# Patch-based Object-centric Transformers for
Efficient Video Generation

Wilson Yan1 &Ryo Okumura2 &Stephen James1 &Pieter Abbeel1 1 UC Berkeley 2
Panasonic. Correspondance to<EMAIL_ADDRESS>
###### Abstract

In this work, we present Patch-based Object-centric Video Transformer (POVT),
a novel region-based video generation architecture that leverages object-
centric information to efficiently model temporal dynamics in videos. We build
upon prior work in video prediction via an autoregressive transformer over the
discrete latent space of compressed videos, with an added modification to
model object-centric information via bounding boxes. Due to better
compressibility of object-centric representations, we can improve training
efficiency by allowing the model to only access object information for longer
horizon temporal information. When evaluated on various difficult object-
centric datasets, our method achieves better or equal performance to other
video generation models, while remaining computationally more efficient and
scalable. In addition, we show that our method is able to perform object-
centric controllability through bounding box manipulation, which may aid
downstream tasks such as video editing, or visual planning. Samples are
available at https://sites.google.com/view/povt-public

Figure 1: Video prediction samples along with predicted bounding boxes

##  1 Introduction

In recent years, generative modeling has seen a large amount of success in
various domains, including images (Kingma & Welling, 2013; Goodfellow et al.,
2014; Kingma & Dhariwal, 2018; Dinh et al., 2016), audio (van den Oord et al.,
2016a; Prenger et al., 2019; Oord et al., 2017a), and language (Radford et
al., 2019; Brown et al., 2020). Rapid progress in these domains can be largely
attributed to deep learning methods through a combination of algorithmic
design and the wider availability of more compute resources.

Although strong progress has been made in high-fidelity image generation (van
den Oord et al., 2016b; Brock et al., 2018; Karras et al., 2019; Ho et al.,
2020), videos remain a difficult modality to train, primarily due to the
complexity of the data distribution as well as the high compute requirements
for modeling larger temporal or spatial resolution videos. As such, most works
in high fidelity video generation have been limited to short sequences of 1-2
seconds at a resolution of $64\times 64$ (Clark et al., 2019; Yan et al.,
2021), with a few exceptions that aim to scale to higher spatial, temporal
resolutions, or both (Anonymous, 2022; Skorokhodov et al., 2021; Tian et al.,
2021).

In this work, we focus on video generation with autoregressive likelihood-
based models, since they are well understood, easy to train, and do not suffer
from posterior collapse or mode collapse. However, autoregressive modeling in
pixel space is prohibitively expensive (Weissenborn et al., 2019). As such,
recent work seeks to address this issue by learning autoregressive priors on
compressed latents from a discrete autoencoder (Yan et al., 2021; Rakhimov et
al., 2020; Le Moing et al., 2021; Walker et al., 2021). Doing so brings about
orders of magnitudes of savings in computational cost compared to modeling
pixels directly.

Our method, Patch-based Object-centric Video Transformer (POVT), proposes to
use object-centric information (via bounding boxes) as a means to acquire more
compressible representations for video sequences, resulting in improved
computational efficiency. In addition, object-centric representations encode a
high-level compositional hierarchy that may benefit models for better
generalization and robustness to new tasks. However, in order to generalize to
complex video, we cannot solely rely on local object-centric representations,
but also must consider the wider global context; for example, generating
scenes with dynamic backgrounds, or moving non-object entities, such as
changing lighting, camera movement, or flowing water. As such, we seek to
design an architecture that predicts object dynamics at the top level of a
hierarchy, and then uses this information for more informed prediction on the
whole scene image.

We build upon prior work in video generation architectures using
autoregressive discrete latent variable models that combine VQ-VAE (Oord et
al., 2017b) and transformers (Vaswani et al., 2017; Brown et al., 2020) by
introducing a novel two-stream attention module that jointly models VQ-VAE
discrete image latents along with predicting object bounding boxes for each
frame. We show that using object-centric representations constructed from
bounding box information allows for more efficient and scalable video
prediction by extending temporal context only along the object-stream.

In summary, we present the following key contributions in this paper:

  * •

To the best of our knowledge, we propose the first object-centric video
prediction model able to work on more complex in-the-wild video data

  * •

We demonstrate that POVT achieves comparable or better predictive accuracy
compared to similar models with more than 2x savings in computational cost,
and even better scaling on longer horizon videos (due to quadratic complexity
of attention in sequence length)

  * •

We show that better computational efficiency allows us to scale to larger
models on more complex datasets to achieve higher fidelity video prediction
quality

  * •

Lastly, we show that POVT allows for controllability and manipulation of
objects in videos

##  2 Related Work

###  2.1 Video Synthesis and Prediction

Video synthesis and prediction are two closely related areas, where prediction
can be commonly viewed as conditional video synthesis. Prior work in video
generation (prediction or synthesis) can generally be divided into GAN-based
methods and likelihood-based methods.

GAN (Goodfellow et al., 2014) video generation models are the most commonly
proposed methods, primarily centered around innovation through architecture
design. Prior work such as VGAN (Vondrick et al., 2016), MoCoGAN (Tulyakov et
al., 2018), and TGAN (Saito et al., 2017) leverage the notion of separate
content and motion as an inductive bias for encouraging smoother video
generation. MoCoGAN-HD (Tian et al., 2021) extends to larger resolution video
by learning to traverse the latent space of pretrained image GANs such as
StyleGANv2 (Karras et al., 2019). TGANv2 (Saito & Saito, 2018) proposes more
efficient GAN training through a hierarchical generation mechanism that allows
for training on sparse subsets of video sequences. DVD-GAN (Clark et al.,
2019) and TrIVD-GAN (Luc et al., 2020) investigate more efficient
discriminator designs. More recently, methods such as StyleGAN-V (Skorokhodov
et al., 2021) and DI-GAN (Yu et al., 2022) adopt continuous interpretations of
video modeling and are able to generate long spatio-temporal resolution videos
through training on very sparse videos. However, GAN-based video models still
suffer from mode collapse and have trouble generating diverse outputs on
complex datasets.

Another broad class of video generation models are likelihood-based methods,
such as variational and autoregressive models. Villegas et al. (2019) and
Castrejon et al. (2019) extend to videos through optimizing an ELBO consisting
of a temporal decomposition of latents. Lee et al. (2018) adds an adversarial
loss to encourage more realistic generation. FitVid (Babaeizadeh et al., 2021)
introduces a more efficient, high capacity architecture. In contrast to
optimizing an ELBO, autoregressive methods model log-likelihood. Video Pixel
Networks (Kalchbrenner et al., 2017) learns a PixelCNN (van den Oord et al.,
2016c) decoder conditioned on frame history encoded using an LSTM (Hochreiter
& Schmidhuber, 1997). Subscale Video Transformer (Weissenborn et al., 2019)
defines a unique spatio-temporal subscale pixel ordering to model videos. Due
to these models typically having long sampling times and high compute
requirements, more recent works (Rakhimov et al., 2020; Yan et al., 2021;
Walker et al., 2021; Le Moing et al., 2021; Anonymous, 2022) have investigated
using autoregressive models on the latent space output of a discrete encoder,
such as VQ-VAE (Oord et al., 2017b) or VQ-GAN (Esser et al., 2021). This
allows for training higher capacity models on less compute, as well as faster
inference time; however, there is an inherent trade-off due to the lossy
compression of discrete autoencoders.

In this work, we show that object-centric representations, through the use of
bounding boxes, allows for efficient and scalable video generation. While
hierarchical representations, such as keypoints (Villegas et al., 2017;
Minderer et al., 2019; Kim et al., 2019) and segmentation maps (Luc et al.,
2017, 2018; Lee et al., 2021) have been shown to improve long-horizon video
generation, our work is the first to explore a bounding box representation for
video generation which extends to realistic video.

###  2.2 Object-centric Generative Models

There is a substantial amount of literature on object-centric representation
learning and generative modeling, in which objects can be discovered in an
unsupervised manner. Image-based models such as MONet (Burgess et al., 2019),
GENESIS (Engelcke et al., 2019), SPACE (Lin et al., 2020b), and IODINE (Greff
et al., 2019) bias their models to encourage object-centric decomposition.
Sequential object-centric models such as SQAIR (Kosiorek et al., 2018),
R-SQAIR (Stanić & Schmidhuber, 2019), SCALOR (Jiang et al., 2019), STOVE
(Kossen et al., 2019), and G-SWM (Lin et al., 2020a) leverage the use of
videos to track and identify objects in a scene. Slot Attention (Locatello et
al., 2020) uses iterative attention on several fixed or randomly generated
slots to identify individual objects. SAVi (Kipf et al., 2021) extends on Slot
Attention to videos by propagating and updating slots over time. AG2Vid (Bar
et al., 2020) synthesizes videos given action graphs. OCVT (Wu et al., 2021b)
uses a transformer over object-centric representations derive from a pre-
trained SPACE model to predict videos. However, unlike our method, it does not
account for the global context of the scene, and is therefore unable to
generalize to diverse and dynamic backgrounds. In addition, it is unable to
model stochastic dynamics, a crucial component for any video prediction model.

Figure 2: Object-centric attention block for POVT. A base latent stream and
object stream are inputted, with a combination of self-attention on the base
latent stream, object-centric attention on the object stream, and cross-
attention for inter-stream communication. Each object (max $K$) is represented
by an $L\times D$ representation, which is a concatenation of bounding box and
patch encoding information

##  3 Method

In this section, we describe POVT, an extension on existing autoregressive
discrete latent variable models to incorporate object-centric information for
efficient and scalable video generation.

We first learn a VQ-VAE (Oord et al., 2017b) encoder to map individual video
frames to a 2D sequence of discrete latents. In order to incorporate object-
centric information, we consider the bounding boxes of a given set of tracked
objects in the scene. Let the VQ-VAE discrete latents for a video sequence of
length $T$ be denoted as $z_{1},\dots,z_{T}$, and the bounding boxes for $K$
objects be
$\\{(b^{1}_{1},\dots,b^{1}_{T}),\dots,(b^{K}_{1},\dots,b^{K}_{T})\\}$, where
we consider object tracks as a set and thus permutation invariant. Each
bounding box $b^{i}_{t}$ is represented as a tuple
$(o_{pres},o_{where},o_{scale})$. $o_{pres}$ is a binary random variable
defining the presence of the object in the scene, generally changed through
occlusion or movement in or out of a scene. $o_{where}$ and $o_{scale}$ are
only set when $o_{pres}=1$, and consist of the location and scale of the
bounding box parameters of a given object. Bounding box parameters are
normalized to $[0,1]$ and quantized uniformly over $64$ bins (the resolution
we train on).

Following VQ-VAE training, we train a transformer to model the joint
distribution over discrete codes and bounding boxes
$p(b_{1},z_{1},\dots,b_{T},z_{T})$ autoregressively with the following
factorization:

$\prod_{t=1}^{T}\left(\prod_{k=1}^{K}p(b^{k}_{t}|b^{<k}_{t},b_{<t},z_{<t})\right)p(z_{t}|b_{\leq
t},z_{<t})$

Each $b^{k}$ is modeled autoregressively in the order
$o_{pres},o_{where},o_{scale}$, and similarly for each $z_{t}$ in 2D raster
scan order. During inference, the model alternates between modeling bounding
box dynamics, and generating the corresponding set of discrete latents
conditioned on the predicted bounding boxes for each object.

In order to allow for more computationally efficient training, we introduce an
efficient object-centric attention architecture (Section 3.1) and a training
regime in which detailed object-centric information, along with coarse global
information, is extended back in time (Section 3.2).

Figure 3: Diversity of video prediction results for each model. For each
model, we condition on 1 frame and visualize the final frame of 6 different
predicted futures. For MoCoGAN-HD, we generate multiple futures starting with
1 randomly sampled latent vector. MoCoGAN-HD and FitVid generally show low
sample diversity compared to LVT and POVT

###  3.1 Object-centric Transformer

We introduce a two-stream attention architecture for modeling discrete latents
and object dynamics separately, with carefully added inter-stream attention
operations.

Constructing object representations
We use a Spatial Transformer (Jaderberg et al., 2015) to extract region
patches from each frame for every bounding box, where each region patch is
resized to a fixed resolution. In the case where a bounding box is invalid
with $o_{pres}=0$, we set bounding box latents and patches to zero. Each patch
is then downsampled using a strided convolution to a fixed resolution, similar
to ViT (Dosovitskiy et al., 2020). When training, we shift and pad all patch
representations one timestep forward since bounding boxes at a timestep $t$
should not be able to see their corresponding patch representations, as they
can only be extracted once the full image has been generated. For $t=0$, we
pad with a learned patch token that is broadcasted over all objects at that
timestep. Patch representations are then concatenated with their corresponding
discretized bounding boxes.

Object-centric attention
Figure 2 shows the proposed architecture of one attention block, and pseudo
code is provided in Appendix C. Attention over objects is decomposed into: (1)
attention over different objects of the same timesteps, and (2) attention over
same objects of different timesteps. Intuitively, the first step models object
interactions at each timestep, and the second step summarizes the temporal
evolution of an object’s history.

Attention over the base latent stream is full self-attention over all discrete
latents. We incorporate object-centric information into the discrete latents
through full cross-attention only with objects at the current timesteps.
Specifically, $z_{t}$ only attends to $b_{t}$ and its corresponding patch
representations, and does not attend to the full object history. This
architecture is far more efficient than attending over the entire video
representation; increasing computational efficiency.

Our transformer architecture follows GPT (Radford et al., 2019) with each
attention block replaced with our object-centric attention block. We use
spatio-temporally broadcasted positional embeddings (Yan et al., 2021) for the
base latent stream. Similarly, object-temporally broadcasted positional
embeddings are applied to the encoded objected bounding boxes and object
region encodings.

###  3.2 Combining Global and Local Representations for Efficient Training

In-scene dynamics are most relevant to the object representations; therefore,
a significant reduction in computational costs can be achieved by only
extending object representations back in time, rather than the full set of
discrete latents. However, we still require a few frames of discrete latents
in order to model the global, non-object dynamics, such as camera jitter or
changing backgrounds. We can more efficiently model video sequences of length
$T$ by defining $T^{\prime}$ such that we model the distribution:

$p(b_{T-T^{\prime}:T},z_{T-T^{\prime}:T}|b_{T-T^{\prime}-\tau:T-T^{\prime}-1})$

where $\tau$ is uniformly sampled every iteration from $[0,T-T^{\prime}]$.
$\tau$ is sampled in order to train on the full range of unconditional and
conditional distributions needed during inference. This allows us to
incorporate longer horizon temporal information about the scene in a more
computationally efficient manner due to the higher compressibility of object
latents compared to the global discrete latents from the VQ-VAE. In
experiments, $T$ is set to the same horizons as baseline models, and
$T^{\prime}$ set depending on the dataset, where datasets with more non-object
dynamics require greater $T^{\prime}$ to include more full-frame temporal
information.

Table 1: Results for video prediction on CATER-CAM, CATER-MOV, and MOVi,
comparing our method against baseline methods FitVid and LVT. Our method is
competitive with a fraction of the training and computational cost. Best
performing models for each metric are in bold, with multiple highlighted if
within a 95% confidence interval

CATER-CAM | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Params | GFLOP | GPU Days
---|---|---|---|---|---|---|---
FitVid | 275 | 25.2 | 0.819 | 0.106 | 78M | 239 | 1.5
LVT | 219 | 26.4 | 0.864 | 0.0662 | 26M | 181 | 2
POVT (ours) | 223 | 25.6 | 0.857 | 0.0701 | 26M | 81 | 1

CATER-MOV | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Params | GFLOP | GPU Days
---|---|---|---|---|---|---|---
FitVid | 181 | 30.4 | 0.87 | 0.0971 | 78M | 239 | 1.5
LVT | 166 | 30.2 | 0.903 | 0.0605 | 26M | 181 | 2
POVT (ours) | 164 | 30.1 | 0.902 | 0.0584 | 26M | 81 | 1

MOVi | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Params | GFLOP | GPU Days
---|---|---|---|---|---|---|---
FitVid | 983 | 22.5 | 0.807 | 0.313 | 78M | 239 | 1.5
LVT | 222 | 22.6 | 0.809 | 0.163 | 26M | 181 | 2
POVT (ours) | 226 | 22.9 | 0.815 | 0.157 | 26M | 81 | 1

Table 2: Quantitative evaluation of single-frame video prediction on Something-Something V2. Our method scales better while retaining access to longer temporal context. All methods perform similarly in terms of PSNR, SSIM, and LPIPS with little difference compared to the baseline of repeating the initial ground truth frame. We hypothesize that this is due to the highly stochastic nature of SSv2. As such, we believe FVD is a more reliable metric to measure the realism of predicted dynamic motion. SSv2 | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Params | GFLOP | GPU Days |
---|---|---|---|---|---|---|---|---
Repeat Frame | 600 | 19.0 | 0.611 | 0.172 | - | - | - | -
FitVid | 1028 | 19.5 | 0.618 | 0.241 | 315M | 930 | 16h |
LVT-S (8 frames) | 558 | 19.1 | 0.615 | 0.180 | 105M | 551 | 16 |
LVT-L (4 frames) | 530 | 19.2 | 0.624 | 0.173 | 206M | 482 | 16 |
POVT (ours) | 497 | 19.1 | 0.622 | 0.170 | 206M | 602 | 16 |

##  4 Experiments

In the following section, we evaluate our method to answer the following
questions:

  * •

Can we learn more efficient video generation models with object-centric
information?

  * •

Can we learn object-centric video models that are able to model real video?

  * •

Can we scale to larger models for generating higher-fidelity video?

###  4.1 Datasets

In this section, we briefly introduce each dataset we evaluate our method on.
More details on dataset generation and processing can be found in Appendix B.

CATER: CATER (Girdhar & Ramanan, 2019) consists of a set of simple objects
(cubes, spheres, cones) that randomly move around in a scene. The original
dataset does not contain any segmentation or bounding box information for
objects. Therefore, we use a version of CATER with object segmentations
introduced in Kabra et al. (2021) (that we call CATER-CAM), which contains
56,464 videos of 33 frames with random camera movement, and up to 10 objects,
where a maximum of 2 objects move at a time. In addition, we generate another
more complex version of CATER (called CATER-MOV) with 10,000 videos of 100
frames, with a maximum of 10 objects, and at most 10 objects moving around.

MOVi: Multi-Object Video (MOVi) is a dataset introduced in Kubric (Greff et
al., 2021) which simulates 3D rigid body interactions in a Blender
environment. Scenes generally consist of a subset of randomly generated
objects that fall with randomly sampled force vectors. We simulate 10,000
scenes of 5-10 objects and render videos of 24 frames.

Something-Something V2 (SSv2): SSv2 (Goyal et al., 2017) is a real-world,
action recognition dataset with a complex suite of action classes that
generally includes object interaction. ‘Something Else’ (Materzynska et al.,
2020) provides bounding box annotations for a large subset of the videos,
computed using a pretrained Faster R-CNN object detector. Following
Materzynska et al. (2020), we use a simple online tracker to group bounding
boxes into separate tracks, and restrict training to a subset of 150,000
videos with a maximum of $4$ objects.

Table 3: Unconditional video synthesis on CATER-CAM | CATER-CAM | FVD$\downarrow$ | Param | GFLOP | GPU Days
---|---|---|---|---
MoCoGAN-HD | 294 | 72M | 242 | 2
POVT (ours) | 273 | 26M | 81 | 1

Table 4: Unconditional video synthesis on CATER-MOV | CATER-MOV | FVD$\downarrow$ | Param | GFLOP | GPU Days
---|---|---|---|---
MoCoGAN-HD | 236 | 72M | 242 | 2
POVT (ours) | 193 | 26M | 81 | 1

Table 5: Unconditional video synthesis on MOVi | MOVi | FVD$\downarrow$ | Param | GFLOP | GPU Days
---|---|---|---|---
MoCoGAN-HD | 282 | 72M | 242 | 2
POVT (ours) | 295 | 26M | 81 | 1

Table 6: Unconditional video synthesis on SSv2 | SSv2 | FVD$\downarrow$ | Param | GFLOP | GPU Days
---|---|---|---|---
MoCoGAN-HD | 420 | 179M | 542 | 16
POVT (ours) | 562 | 206M | 602 | 16

###  4.2 Baselines

We compare our method against a baseline from each common class of video
generation models. To the best of our knowledge, there do not exist any video
prediction methods that use bounding box information (ground-truth or from a
pretrained object detector), so we compare against unconditional baselines,
and show that our method not only allows for a more efficient models, but also
is able to extend to real video unlike existing object-centric video models.
We do not compare against OCVT (Wu et al., 2021b) since it would perform
poorly on our datasets as OCVT is unable to model stochastic dynamics and is
limited to deterministic settings.

  * •

FitVid (Babaeizadeh et al., 2021) is a variational video prediction method
that outperforms popular baselines such as SVG (Denton & Fergus, 2018), SAVP
(Lee et al., 2018), SV2P (Babaeizadeh et al., 2017), and GHVAE (Wu et al.,
2021a). We choose FitVid over object-centric variational approaches such as
G-SWM (Lin et al., 2020a) since they exhibited worse performance than FitVid,
and have difficulty generalizing to realistic video such as SSv2.

  * •

MoCoGAN-HD (Tian et al., 2021) learns a video generation model by traversing
the latent space of a pretrained image generator. It is able to produce high
fidelity video generation with minimal computational cost compared to models
such as DVD-GAN (Clark et al., 2019) or TrIVD-GAN (Luc et al., 2020).

  * •

Latent Video Transformer (LVT) (Rakhimov et al., 2020) is an AR model that
learns a transformer over discrete latent codes of videos encoded using a VQ-
VAE. Other baseline models primarily propose modifications to learn better
discrete codes, such as through flow-conditional reconstruction (Le Moing et
al., 2021), using a VQ-GAN (Anonymous, 2022), or using a 3D VQ-VAE (Yan et
al., 2021), which we believe are orthogonal to our approach which focuses on
the transformer architecture. Therefore, we opt for the simplest baseline to
adapt our model to. Both LVT and our method POVT use the same VQ-VAE on each
dataset.

###  4.3 Training Details

For all datasets, we train on videos of 8 frames (16 frames did not increase
performance) scaled to $[0,1]$, and evaluate on generated videos of 16 frames.
For CATER-CAM, CATER-MOV, and MOVi, we condition on 1 full frame, and 7
timesteps of object representations. Full frame conditioning is to account for
random camera and lighting changes per video. For SSv2, we condition on 3 full
frames and 7 timesteps of object representations to model local non-object
dynamics of the complex videos. All baseline methods are trained on the full
$8$ frames. More details on hyperparameters for POVT and baselines can be
found in Appendix A. All experiments are trained on a maximum of 8 RTX A5000
GPUs each with 24GB of memory.

Figure 4: One object (in red) is translated to a random pixel. Note that occlusion is maintained during translation in both cases where the object occludes (top) and is occluded by other objects (bottom) Table 7: Ablation on the number of full frames to condition on. All experiments of POVT condition on 7 timesteps of object history in addition to the specified number of full frames Method | # Cond Frames | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | GFLOP
---|---|---|---|---|---|---
LVT | 1 | 218 | 28.4 | 0.884 | 0.0838 | 38
2 | 207 | 29.0 | 0.890 | 0.0751 | 56
4 | 165 | 29.6 | 0.895 | 0.0646 | 77
7 | 166 | 30.2 | 0.903 | 0.0605 | 181
POVT (ours) | 1 | 164 | 30.1 | 0.902 | 0.0583 | 81
2 | 158 | 30.0 | 0.897 | 0.0609 | 100
4 | 160 | 30.1 | 0.898 | 0.0599 | 121

###  4.4 Evaluation

Metrics: For video prediction tasks, we use Fréchet Video Distance (FVD)
(Unterthiner et al., 2019), PSNR (Huynh-Thu & Ghanbari, 2008), SSIM (Wang et
al., 2004), and LPIPS (Zhang et al., 2018). In order to account for
stochasticity of video prediction, we follow prior work (Babaeizadeh et al.,
2017; Villegas et al., 2019; Babaeizadeh et al., 2021) and take the best
matching trajectory with ground truth out of 100 samples. For final
evaluation, we take the average over $128$ test examples. For video synthesis,
we evaluate only on FVD. We compute FVD over batches of $256$ and average over
16 runs. Qualitative samples can be found on our paper website at
https://sites.google.com/view/povt-public.

CATER-CAM / CATER-MOV / MOVi: Tables 1, 6, 6 and 6 show video prediction and
synthesis results on CATER-CAM, CATER-MOV, and MOVi datasets. For CATER-MOV
and MOVi, our method performs competitively, or better than baseline methods
at a fraction of the computation cost in terms of GLOPs and GPU-days. Our
method slightly underperforms LVT for CATER-CAM, which we hypothesize may be
due to constant shifts in scale of object representations due to zooming
camera movement. Sample-wise, each model shows similar behavior across all
object datasets (CATER-CAM, CATER-MOV, and MOVi). MoCoGAN-HD produces high
quality images, but has difficult retaining object consistency across time,
where objects would constantly disappear, appear, and morph to other objects.
FitVid learns better reconstruction than LVT and POVT, but samples often
result in blurry movement. In addition, FitVid and MoCoGAN-HD generally have
difficulty modeling diversity of future predictions and generate predictions
with very low diversity compared to our method, shown in Figure 3.

Something-Something V2: Tables 2 and 6 show quantitative results on SSv2. For
consistency, we report PSNR, SSIM, and LPIPS. However, metrics across all
models show little difference compared to a naive baseline of repeating the
first frame over the entire trajectory (Repeat Frame), most likely due to the
highly stochastic nature of SSv2 (random camera jittering, translation, etc).
As such, we believe that FVD is a more suitable metric in this scenario since
it better measures the realism of dynamic motion in predicted trajectories. We
show that our method is able to scale better, comparing POVT to two variants
of LVT: LVT-S and LVT-L. LVT-S attends over 8 frames, restricted to similar
GLOPs as POVT by limiting $8$ transformer layers. LVT-L attends over 4 frames
(same as POVT, excluding object history) with 16 transformer layers, the same
as POVT. Therefore, POVT is able to benefit from scaling to larger models
under the same compute requirements as LVT while providing similar temporal
history. In addition, our model is able to benefit more from bounding box
information in complex video when "objectness" is less clear, compared to
CATER and MOVi datasets where objects are easily discovered through shape and
shading. FitVid does not perform well on this dataset, and collapses into a
sequence of blurry frames. MoCoGAN-HD performs the best, able to generate the
best samples according FVD. However, it has been demonstrated that GANs tend
to perform better on classifier-based metrics such as FVD due to focus on high
frequency image statistics whereas VQ-VAE based models throw away through
lossy compression (Razavi et al., 2019). In addition, we believe that
improving other classes of generative models such as likelihood-based models
is important, due to ease of training and mode-covering behavior. This can be
clearly seen in Figure 3 where GANs collapse to limited futures when
conditioned on a single frame.

Controllability of Object-centric Models: Given that POVT adopts a
hierarchical framework using bounding boxes, this makes it amenable to certain
controllability aspects in generation. Figure 4 shows our method able to
cleanly translated select objects which account for occlusion.

Table 8: Ablation comparing alternative representations for temporal context |  | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---
No Patch Encoding | 267 | 28.5 | 0.890 | 0.0722
Fixed Grid | 207 | 28.7 | 0.888 | 0.0770
BBox (ours) | 168 | 30.0 | 0.900 | 0.0598

Table 9: Ablation removing each attention component | Method | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---
POVT | 164 | 30.1 | 0.902 | 0.0584
$-$ Base-Base | 260 | 28.1 | 0.881 | 0.0722
$-$ Base-Obj | 209 | 29.6 | 0.896 | 0.0697
$-$ Obj-Obj (over time) | 396 | 27.3 | 0.863 | 0.098
$-$ Obj-Obj (per time) | 174 | 29.7 | 0.889 | 0.0643

###  4.5 Ablations

Object-centric representations provide sufficient temporal history. For
environments where dynamics are primarily composed of objects, we show that
object-centric representations are sufficient to represent scene history.
Table 7 shows ablations comparing our method against LVT, with varying number
of conditioned frames. Since LVT operates only on full image latents, it
benefits greatly from extending the number of conditioned frames further back
in time. However, our method, when already conditioned on the history of
object-centric representations, shows little to no change in performance when
conditioned on more full frames.

Object-centric representations provide better temporal history of a scene than
standard image representations. We investigate the use of other
representations to extend temporal context. In the most general case, we can
imagine just encoding previous frames. Table 9 shows comparisons of our method
which uses bounding box information against generic image latents constructed
by encoding fixed patches (similar to ViT) to provide temporal information. At
high levels of compression, using bounding box representations is far more
beneficial than generic image encodings. In addition, we show that primary
benefit of the object representations comes from bounding boxes in combination
with region encodings, as opposed to just using bounding boxes.

Modeling temporal dependencies for objects is crucial for accurate video
prediction. Table 9 shows the effects of removing each aspect of our object-
centric attention mechanism. Removing the object-temporal attention in our
models results in the greatest degradation, showing that POVT greatly benefits
in modeling longer temporal dynamics through object information.

##  5 Discussion

A primary limitation of our method is that it requires bounding box
information, as the benefit the model receives from bounding boxes is
restricted by their quality. Bounding boxes also enforce additional
restrictions compared to prior methods which only need raw video frames to
learn. Nevertheless, we show that there are large gains in computational
efficiency when leveraging bounding box information, as even using noisy
bounding boxes from a pretrained object detector in Something-Something V2
allows us to scale to larger models, as well produce higher fidelity video
samples. Overall, we believe that POVT provides a step in the right direction
for a more computationally efficient video generation architecture, a
necessity when extending to longer horizon and more complex videos.

## Acknowledgments and Disclosure of Funding

The work was in part supported by Panasonic through BAIR Commands, Darpa
RACER, and the Hong Kong Centre for Logistics Robotics. In addition, we thank
Cirrascale Cloud Services (https://cirrascale.com/) for providing compute
resources.

## References

  * Anonymous (2022) Anonymous.  Autoregressive latent video prediction with high-fidelity image generator.  In _Submitted to The Tenth International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=K-hiHQXEQog.  under review.
  * Babaeizadeh et al. (2017) Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S.  Stochastic variational video prediction.  _arXiv preprint arXiv:1710.11252_ , 2017.
  * Babaeizadeh et al. (2021) Babaeizadeh, M., Saffar, M. T., Nair, S., Levine, S., Finn, C., and Erhan, D.  Fitvid: Overfitting in pixel-level video prediction.  _arXiv preprint arXiv:2106.13195_ , 2021.
  * Bar et al. (2020) Bar, A., Herzig, R., Wang, X., Rohrbach, A., Chechik, G., Darrell, T., and Globerson, A.  Compositional video synthesis with action graphs.  _arXiv preprint arXiv:2006.15327_ , 2020.
  * Bewley et al. (2016) Bewley, A., Ge, Z., Ott, L., Ramos, F., and Upcroft, B.  Simple online and realtime tracking.  In _2016 IEEE international conference on image processing (ICIP)_ , pp. 3464–3468. IEEE, 2016.
  * Brock et al. (2018) Brock, A., Donahue, J., and Simonyan, K.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Burgess et al. (2019) Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A.  Monet: Unsupervised scene decomposition and representation.  _arXiv preprint arXiv:1901.11390_ , 2019.
  * Castrejon et al. (2019) Castrejon, L., Ballas, N., and Courville, A.  Improved conditional vrnns for video prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 7608–7617, 2019.
  * Clark et al. (2019) Clark, A., Donahue, J., and Simonyan, K.  Adversarial video generation on complex datasets, 2019.
  * Denton & Fergus (2018) Denton, E. and Fergus, R.  Stochastic video generation with a learned prior.  _arXiv preprint arXiv:1802.07687_ , 2018.
  * Dinh et al. (2016) Dinh, L., Sohl-Dickstein, J., and Bengio, S.  Density estimation using Real NVP.  _arXiv preprint arXiv:1605.08803_ , 2016.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Engelcke et al. (2019) Engelcke, M., Kosiorek, A. R., Jones, O. P., and Posner, I.  Genesis: Generative scene inference and sampling with object-centric latent representations.  _arXiv preprint arXiv:1907.13052_ , 2019.
  * Esser et al. (2021) Esser, P., Rombach, R., and Ommer, B.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12873–12883, 2021.
  * Girdhar & Ramanan (2019) Girdhar, R. and Ramanan, D.  Cater: A diagnostic dataset for compositional actions and temporal reasoning.  _arXiv preprint arXiv:1910.04744_ , 2019.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pp. 2672–2680, 2014.
  * Goyal et al. (2017) Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al.  The" something something" video database for learning and evaluating visual common sense.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5842–5850, 2017.
  * Greff et al. (2019) Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A.  Multi-object representation learning with iterative variational inference.  In _International Conference on Machine Learning_ , pp. 2424–2433. PMLR, 2019.
  * Greff et al. (2021) Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D. J., Gnanapragasam, D., Golemo, F., Herrmann, C., Kipf, T., Kundu, A., Lagun, D., Laradji, I., Liu, H.-T. D., Meyer, H., Miao, Y., Nowrouzezahrai, D., Oztireli, C., Pot, E., Radwan, N., Rebain, D., Sabour, S., Sajjadi, M. S. M., Sela, M., Sitzmann, V., Stone, A., Sun, D., Vora, S., Wang, Z., Wu, T., Yi, K. M., Zhong, F., and Tagliasacchi, A.  Kubric: a scalable dataset generator.  2021\.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _arXiv preprint arXiv:2006.11239_ , 2020.
  * Hochreiter & Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Huynh-Thu & Ghanbari (2008) Huynh-Thu, Q. and Ghanbari, M.  Scope of validity of psnr in image/video quality assessment.  _Electronics letters_ , 44(13):800–801, 2008\.
  * Jaderberg et al. (2015) Jaderberg, M., Simonyan, K., Zisserman, A., et al.  Spatial transformer networks.  _Advances in neural information processing systems_ , 28:2017–2025, 2015.
  * Jiang et al. (2019) Jiang, J., Janghorbani, S., De Melo, G., and Ahn, S.  Scalor: Generative world models with scalable object representations.  _arXiv preprint arXiv:1910.02384_ , 2019.
  * Kabra et al. (2021) Kabra, R., Zoran, D., Erdogan, G., Matthey, L., Creswell, A., Botvinick, M., Lerchner, A., and Burgess, C. P.  Simone: View-invariant, temporally-abstracted object representations via unsupervised video decomposition.  _arXiv preprint arXiv:2106.03849_ , 2021.
  * Kalchbrenner et al. (2017) Kalchbrenner, N., Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K.  Video pixel networks.  In _International Conference on Machine Learning_ , pp. 1771–1779. PMLR, 2017.
  * Karras et al. (2019) Karras, T., Laine, S., and Aila, T.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4401–4410, 2019.
  * Kim et al. (2019) Kim, Y., Nam, S., Cho, I., and Kim, S. J.  Unsupervised keypoint learning for guiding class-conditional video prediction.  _arXiv preprint arXiv:1910.02027_ , 2019.
  * Kingma & Dhariwal (2018) Kingma, D. P. and Dhariwal, P.  Glow: Generative flow with invertible 1x1 convolutions.  _arXiv preprint arXiv:1807.03039_ , 2018.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational Bayes.  _Proceedings of the 2nd International Conference on Learning Representations_ , 2013.
  * Kipf et al. (2021) Kipf, T., Elsayed, G. F., Mahendran, A., Stone, A., Sabour, S., Heigold, G., Jonschkowski, R., Dosovitskiy, A., and Greff, K.  Conditional object-centric learning from video.  _arXiv preprint arXiv:2111.12594_ , 2021.
  * Kosiorek et al. (2018) Kosiorek, A. R., Kim, H., Posner, I., and Teh, Y. W.  Sequential attend, infer, repeat: Generative modelling of moving objects.  _arXiv preprint arXiv:1806.01794_ , 2018.
  * Kossen et al. (2019) Kossen, J., Stelzner, K., Hussing, M., Voelcker, C., and Kersting, K.  Structured object-aware physics prediction for video modeling and planning.  _arXiv preprint arXiv:1910.02425_ , 2019.
  * Le Moing et al. (2021) Le Moing, G., Ponce, J., and Schmid, C.  Ccvs: Context-aware controllable video synthesis.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Lee et al. (2018) Lee, A. X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., and Levine, S.  Stochastic adversarial video prediction.  _arXiv preprint arXiv:1804.01523_ , 2018.
  * Lee et al. (2021) Lee, W., Jung, W., Zhang, H., Chen, T., Koh, J. Y., Huang, T., Yoon, H., Lee, H., and Hong, S.  Revisiting hierarchical approach for persistent long-term video prediction.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=3RLN4EPMdYd.
  * Lin et al. (2020a) Lin, Z., Wu, Y.-F., Peri, S., Fu, B., Jiang, J., and Ahn, S.  Improving generative imagination in object-centric world models.  In _International Conference on Machine Learning_ , pp. 6140–6149. PMLR, 2020a.
  * Lin et al. (2020b) Lin, Z., Wu, Y.-F., Peri, S. V., Sun, W., Singh, G., Deng, F., Jiang, J., and Ahn, S.  Space: Unsupervised object-oriented scene representation via spatial attention and decomposition.  _arXiv preprint arXiv:2001.02407_ , 2020b.
  * Locatello et al. (2020) Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T.  Object-centric learning with slot attention.  _arXiv preprint arXiv:2006.15055_ , 2020.
  * Luc et al. (2017) Luc, P., Neverova, N., Couprie, C., Verbeek, J., and LeCun, Y.  Predicting deeper into the future of semantic segmentation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 648–657, 2017.
  * Luc et al. (2018) Luc, P., Couprie, C., Lecun, Y., and Verbeek, J.  Predicting future instance segmentation by forecasting convolutional features.  In _Proceedings of the european conference on computer vision (ECCV)_ , pp. 584–599, 2018.
  * Luc et al. (2020) Luc, P., Clark, A., Dieleman, S., Casas, D. d. L., Doron, Y., Cassirer, A., and Simonyan, K.  Transformation-based adversarial video prediction on large-scale data.  _arXiv preprint arXiv:2003.04035_ , 2020.
  * Materzynska et al. (2020) Materzynska, J., Xiao, T., Herzig, R., Xu, H., Wang, X., and Darrell, T.  Something-else: Compositional action recognition with spatial-temporal interaction networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1049–1059, 2020.
  * Minderer et al. (2019) Minderer, M., Sun, C., Villegas, R., Cole, F., Murphy, K., and Lee, H.  Unsupervised learning of object structure and dynamics from videos.  _arXiv preprint arXiv:1906.07889_ , 2019.
  * Oord et al. (2017a) Oord, A. v. d., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., Driessche, G. v. d., Lockhart, E., Cobo, L. C., Stimberg, F., et al.  Parallel wavenet: Fast high-fidelity speech synthesis.  _arXiv preprint arXiv:1711.10433_ , 2017a.
  * Oord et al. (2017b) Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K.  Neural discrete representation learning.  _arXiv preprint arXiv:1711.00937_ , 2017b.
  * Prenger et al. (2019) Prenger, R., Valle, R., and Catanzaro, B.  Waveglow: A flow-based generative network for speech synthesis.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 3617–3621. IEEE, 2019.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9, 2019.
  * Rakhimov et al. (2020) Rakhimov, R., Volkhonskiy, D., Artemov, A., Zorin, D., and Burnaev, E.  Latent video transformer.  _arXiv preprint arXiv:2006.10704_ , 2020.
  * Razavi et al. (2019) Razavi, A., van den Oord, A., and Vinyals, O.  Generating diverse high-fidelity images with vq-vae-2.  In _Advances in Neural Information Processing Systems_ , pp. 14866–14876, 2019.
  * Saito & Saito (2018) Saito, M. and Saito, S.  Tganv2: Efficient training of large models for video generation with multiple subsampling layers.  _arXiv preprint arXiv:1811.09245_ , 2018.
  * Saito et al. (2017) Saito, M., Matsumoto, E., and Saito, S.  Temporal generative adversarial nets with singular value clipping.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2830–2839, 2017.
  * Skorokhodov et al. (2021) Skorokhodov, I., Tulyakov, S., and Elhoseiny, M.  Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2.  _arXiv preprint arXiv:2112.14683_ , 2021.
  * Stanić & Schmidhuber (2019) Stanić, A. and Schmidhuber, J.  R-sqair: relational sequential attend, infer, repeat.  _arXiv preprint arXiv:1910.05231_ , 2019.
  * Tian et al. (2021) Tian, Y., Ren, J., Chai, M., Olszewski, K., Peng, X., Metaxas, D. N., and Tulyakov, S.  A good image generator is what you need for high-resolution video synthesis.  _arXiv preprint arXiv:2104.15069_ , 2021.
  * Tulyakov et al. (2018) Tulyakov, S., Liu, M.-Y., Yang, X., and Kautz, J.  Mocogan: Decomposing motion and content for video generation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1526–1535, 2018.
  * Unterthiner et al. (2019) Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S.  Fvd: A new metric for video generation.  2019\.
  * van den Oord et al. (2016a) van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016a.
  * van den Oord et al. (2016b) van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K.  Pixel recurrent neural networks.  _International Conference on Machine Learning (ICML)_ , 2016b.
  * van den Oord et al. (2016c) van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., and Kavukcuoglu, K.  Conditional image generation with pixelcnn decoders.  _arXiv preprint arXiv:1606.05328_ , 2016c.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ , 2017.
  * Villegas et al. (2017) Villegas, R., Yang, J., Zou, Y., Sohn, S., Lin, X., and Lee, H.  Learning to generate long-term future via hierarchical prediction.  In _international conference on machine learning_ , pp. 3560–3569. PMLR, 2017.
  * Villegas et al. (2019) Villegas, R., Pathak, A., Kannan, H., Erhan, D., Le, Q. V., and Lee, H.  High fidelity video prediction with large stochastic recurrent neural networks.  _Advances in Neural Information Processing Systems_ , 32:81–91, 2019.
  * Vondrick et al. (2016) Vondrick, C., Pirsiavash, H., and Torralba, A.  Generating videos with scene dynamics.  In _Advances in neural information processing systems_ , pp. 613–621, 2016.
  * Walker et al. (2021) Walker, J., Razavi, A., and Oord, A. v. d.  Predicting video with vqvae.  _arXiv preprint arXiv:2103.01950_ , 2021.
  * Wang et al. (2004) Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Weissenborn et al. (2019) Weissenborn, D., Täckström, O., and Uszkoreit, J.  Scaling autoregressive video models.  _arXiv preprint arXiv:1906.02634_ , 2019.
  * Wu et al. (2021a) Wu, B., Nair, S., Martin-Martin, R., Fei-Fei, L., and Finn, C.  Greedy hierarchical variational autoencoders for large-scale video prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2318–2328, 2021a.
  * Wu et al. (2021b) Wu, Y.-F., Yoon, J., and Ahn, S.  Generative video transformer: Can objects be the words?  In _International Conference on Machine Learning_ , pp. 11307–11318. PMLR, 2021b.
  * Yan et al. (2021) Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A.  Videogpt: Video generation using vq-vae and transformers.  _arXiv preprint arXiv:2104.10157_ , 2021.
  * Yu et al. (2022) Yu, S., Tack, J., Mo, S., Kim, H., Kim, J., Ha, J.-W., and Shin, J.  Generating videos with dynamics-aware implicit generative adversarial networks.  _arXiv preprint arXiv:2202.10571_ , 2022.
  * Zhang et al. (2018) Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 586–595, 2018.

##  Appendix A Architecture Hyperparameters

###  A.1 VQ-VAE

Table 10: VQ-VAE hyperparameters for each dataset | CATER-CAM / CATER-MOV | MOVi | Something-Something v2
---|---|---|---
Input size | $64\times 64$ | $64\times 64$ | $64\times 64$
Latent size | $16\times 16$ | $16\times 16$ | $16\times 16$
$\beta$ (commitment loss coefficient) | 0.25 | 0.25 | 0.25
Batch size | 128 | 128 | 128
Learning rate | $7\times 10^{-4}$ | $7\times 10^{-4}$ | $7\times 10^{-4}$
Hidden units | 256 | 256 | 256
Residual units | 128 | 128 | 128
Residual layers | 2 | 4 | 4
Codebook size | 1024 | 2048 | 4096
Codebook dimension | 128 | 128 | 256
Encoder filter size | 3 | 3 | 3
Upsampling conv filter size | 4 | 4 | 4
Training steps | 200k | 200k | 200k

###  A.2 Prior Network

Table 11: POVT transformer hyperparameters for each dataset | CATER-CAM / CATER-MOV | MOVi | Something-Something v2
---|---|---|---
Full timesteps | 1 | 1 | 4
Object timesteps | 8 | 8 | 8
Object representation size | $1\times 1$ | $1\times 1$ | $2\times 2$
Object embedding size | 128 | 128 | 128
Batch size | 32 | 32 | 32
Learning rate | $1\times 10^{-3}$ | $1\times 10^{-3}$ | $1\times 10^{-3}$
Attention heads | 4 | 4 | 8
Attention Layers | 8 | 8 | 16
Embedding size | 512 | 512 | 1024
Feedforward hidden size | 2048 | 2048 | 4096
Dropout | 0.2 | 0.2 | 0.2
Attention dropout | 0.3 | 0.3 | 0.3
Training steps | 200k | 200k | 200k

##  Appendix B Dataset Details

###  B.1 CATER

Since the original CATER dataset does not contain object information, we use a
variant of CATER (CATER-CAM) introduced in Kabra et al. (2021). CATER-CAM
contains around 40k videos with segmentations masks for object. Random camera
movement is applied each episode with a maximum of 2 objects moving at a time.
Possible object motions include sliding and pick-place actions. We use the
default given train-test data split.

In order to generate CATER-MOV, we modify the original CATER generation
codebase (https://github.com/rohitgirdhar/CATER) to render segmentation masks
for each object. We accomplish this by removing all lights and shadows, and
performing shadeless rendering for each object set to white and everything
else black. Segmentations are then computed through a simple threshold of
$0.8$. Rendered videos do not include camera movement, and a maximum of 6
objects move at a time. Possible object motions include sliding, pick-place,
and rotation. We generate 10k videos with sequence lengths of 100. We use 10%
of the data as the test set.

For both datasets, we compute bounding boxes for each object using
segmentation masks. If the bounding box area is below a threshold ($5$), we
ignore the bounding box. CATER follows the Apache 2.0 license.

###  B.2 MOVi

We generate MOVi using Kubric (https://github.com/google-research/kubric). We
generate 10k trajectories of length 24 with random camera configuration with
5-10 KubricBasic objects. To compute bounding boxes, we follow the same
process as CATER using segmentation masks. We use 10% of the data as the test
set. MOVi follows the Apache 2.0 license.

###  B.3 Something-Something V2

Something-Something V2 Goyal et al. (2017) is an action recognition dataset
licensed for research use that contains 220,847 videos. We use a subset of the
dataset that has bounding box annotations provided by Something Else
Materzynska et al. (2020). We follow the original paper and use a simple
Kalman Filter-based tracking algorithm Bewley et al. (2016) to find bounding
box correspondances across frames. In addition, we only train on a subset of
data with a maximum of 4 objects. We use the default given train-test split
after filtering.

##  Appendix C POVT Attention Block Pseudocode

⬇

def povt_attention_block(base, obj):

# base: B x T’ x H x W x D

# obj: B x T x K x L x D

# all attention operations in each block share the same learnable parameters

# Full self-attention over base latent stream

attn_bb = MaskedMultiheadAttention(base, base)

# Cross attention from object stream to base latent stream

attn_bo = MultiheadATtention(base, obj)

# Per-object attention over time

oo_1 = reshape(obj, (B, K, T * L, D))

attn_oo_1 = MaskedMultiheadAttention(oo_1, oo_1)

attn_oo_1 = reshape(attn_oo_1, (B, T, K, L, D))

# Per-timestep attention over objects

oo_2 = reshape(obj, (B, T, K * L, D))

attn_oo_2 = MaskedMultiheadAttention(oo_2, oo_2)

attn_oo_2 = reshape(attn_oo_2, (B, T, K, L, D))

# Combine and return

base = (attn_bb + attn_bo) / 2

obj = (attn_oo_1 + attn_oo_2) / 2

return base, obj


# Enhancing Microgrid Performance Prediction with Attention-based Deep
Learning Models

Vinod Kumar Maddineni1, Naga Babu Koganti2 and Praveen Damacharla${}^{3}*$
1Power Systems Engineer, Schneider Electric, CA, USA
2Power Electronics Engineer, KineticAI Inc., MI, USA
3Research Scientist, KineticAI Inc., The Woodlands, TX, USA
<EMAIL_ADDRESS>
###### Abstract

In this research, an effort is made to address microgrid systems’ operational
challenges, characterized by power oscillations that eventually contribute to
grid instability. An integrated strategy is proposed, leveraging the strengths
of convolutional and Gated Recurrent Unit (GRU) layers. This approach is aimed
at effectively extracting temporal data from energy datasets to improve the
precision of microgrid behavior forecasts. Additionally, an attention layer is
employed to underscore significant features within the time-series data,
optimizing the forecasting process. The framework is anchored by a Multi-Layer
Perceptron (MLP) model, which is tasked with comprehensive load forecasting
and the identification of abnormal grid behaviors. Our methodology underwent
rigorous evaluation using the Micro-grid Tariff Assessment Tool dataset, with
Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and the coefficient
of determination (r2-score) serving as the primary metrics. The approach
demonstrated exemplary performance, evidenced by a MAE of 0.39, RMSE of 0.28,
and an r2-score of 98.89% in load forecasting, along with near-perfect zero
state prediction accuracy (approximately 99.9%). Significantly outperforming
conventional machine learning models such as support vector regression and
random forest regression, our model’s streamlined architecture is particularly
suitable for real-time applications, thereby facilitating more effective and
reliable microgrid management.

###### Index Terms:

Load forecasting, Microgrid management, CNN, GRU, Attention.

##  I Introduction

In the microgrid ecosystem, load forecasting and the identification of
anomalous generation patterns are integral to fulfilling demand requirements.
The energy provision within such networks may derive from various sources,
including diesel and solar energy, showcasing the system’s versatility in
harnessing power. [1]. However, this form of distributed energy source is
susceptible to fluctuations thereby presenting challenges in maintaining
stable output power.Thus, conducting a thorough analysis of the patterns
exhibited by the generated electricity is of paramount importance. On that
note, the proliferation of diverse energy generation capabilities within
microgrid configurations has markedly piqued the interest of a broader
spectrum of stakeholders [2]. In this research, the study delves into
predictive analysis of volatile load demand and the detection of anomalies in
power generation within microgrid infrastructures, aiming to enhance the
operational efficiency and reliability of these systems. The proposed solution
focuses on striking a balance between the demand and energy produced by the
grid; this approach could further facilitate an effective management of grid
energy even during the elevated usage in a typical scenarios such as warm
seasons. Another benefit of the devised strategy is accurate load forecasting
within the microgrids; this plays a crucial role in preventing unexpected
outages, enhancing the reliability of power supply. Furthermore, the adoption
of renewable energy sources in electricity generation significantly
contributes to the reduction of CO2 emissions, promoting a more sustainable
and environmentally friendly energy landscape. A typical representation of the
micro grid is shown in Figure 1.

Figure 1: The microgrid components and its structure.

As illustrated in the Figure 1, the microgrid integrates both renewable and
non-renewable energy sources. Central to this system’s efficacy is a
sophisticated controller tasked with the precise apportionment of generated
power. This includes a strategic division between direct residential supply
and allocation to energy storage solutions, underpinning a nuanced approach to
energy distribution and conservation.

Artificial Intelligence (AI) has exhibited substantial potential in the
analysis of time series data. This research is aimed at leveraging the
structural advantages of well-established memory models, specifically the
Gated Recurrent Unit (GRU) and attention mechanisms, to refine microgrid
behavior predictions [3]. Additionally, the study has explored the
incorporation of convolution layers for extracting time-dependent features to
further enhance the precision of microgrid predictions[4]. The proposed
structure integrates a one-dimensional convolution function (Conv1d) and a
Gated Recurrent Unit (GRU) to capture temporal features from the input time
series. An attention layer further refines this process by highlighting the
significance of specific time-dependent information extracted from the series,
thereby enhancing the model’s focus on relevant data for improved predictive
accuracy. Finally, a Multilayer Perceptron (MLP) [5] model is used for end to
end regression and classification.To expedite the performance of the final
model, the minimum number of units has been selected for both the GRU and MLP
components.

The literature review in recent studies illustrates a broad exploration of AI
techniques, particularly in Deep Learning and Machine Learning, for diagnosing
anomalies and forecasting behaviors in microgrid systems. Various approaches,
ranging from Feed-Forward Neural Networks (FNN), Recurrent Neural Networks
(RNNs) including Long Short-Term Memory (LSTM) and Gated Recurrent Units
(GRU), to more complex architectures like sequence-to-sequence models and
Temporal Convolutional Networks (TCN), have been scrutinized [6, 7]. The study
is notable for its extensive comparison of these models applied to real-world
data, identifying specific models that excel in short-term and long-term load
forecasting. Similarly, the research demonstrates the efficacy of integrating
full wavelet packet transforms with Multi-Layer Perceptrons (MLP) for detailed
electricity consumption forecasting during varying conditions, significantly
outperforming standard neural network approaches[8].

Further, the application of machine learning and deep learning for electrical
load prediction is elaborated in studies, which reveal the superior accuracy
of LSTM-based models when optimized with relevant data, including weather
variables[9]. The innovative use of federated learning for energy forecasting
showcases the potential for privacy-preserving models in residential energy
consumption analysis[10]. Moreover, the advancement in fault detection methods
within microgrids, as discussed in recent studies, highlights the significant
role of various machine learning classifiers, like Decision Trees and
AdaBoost, in enhancing microgrid reliability. The collective findings from
these studies underscore the evolving landscape of AI applications in
microgrid monitoring and management, steering towards more efficient and
reliable energy systems. This backdrop forms the foundation for our research,
where we propose a new model combining convolution, GRU, and attention layers
aimed at improving load forecasting precision and speed, addressing the
challenges associated with training deep learning models for real-time
microgrid applications.

The main focus of the proposed method is to extract effective information from
the original time series using a shallow deep learning architecture. This
research has three main contributions:

  * •

Proposing a shallow network for load and abnormal behavior forecasting.

  * •

Comparing the performance of the proposed shallow deep learning model and
traditional Machine Learning (ML) models.

  * •

Improving the performance of the proposed deep learning method using the fine-
tuning of the learning rate and batch size.

This research paper continues as follows: Section II delves into the latest
literature on load forecasting and the utilization trend in DL models, while
section II offers a statistical review of the datasets used. Section III, on
the other hand, focuses on the evaluation of model parameters and algorithm
structures for load forecasting and abnormal behavior prediction. The
experimental results, along with the performance ratings on these models, are
presented in Section IV. Finally, Section V gives a recap of the study’s
findings, limitations, and potential future directions.

##  II Methodology

###  II-A Dataset

The Microgrid Tariff Assessment Tool offers information on electricity
consumption in rural sub-Saharan Africa, allowing users to examine the impact
of system reliability on sizing and costs. It includes data on the levelized
cost of electricity for diesel-only, solar-plus-storage, and solar-storage-
diesel systems. The tool enables scenario comparison for locations in East,
West, and Southern Africa, factoring in local geographical and economic
parameters [11]. the tool allows comparison scenarios for three different
locations in sub-Saharan Africa: East Africa (Lodwar, Kenya), West Africa
(Accra, Ghana), and Southern Africa (Lusaka, Zambia). Each location has
specific geographical and economic parameters tailored to the region, such as
solar resource availability, diesel fuel prices, and distribution system
costs. The statistical characteristics of the dataset are shown in Table I.

TABLE I: Statistical characteristics of the dataset. Data Feature | Average | Variance
---|---|---
PV (KW) | 70.84 | 8.45
Battery (KW) | 7.55 | 1.99
Battery (KWh) | 382.21 | 14.92
Generator (KW) | 18.55 | 5.46
PV capital cost | 237,795,130.20 | 15,034.17
PV O and M | 6,894,229.04 | 2,559.88
Battery capital cost | 284,208,250.30 | 12,202.27
Battery O and M | 17,504,910.43 | 3,115.69
Diesel capital cost | 6,325,958.59 | 3,040.68
Diesel O and M (KW) | 853,071.00 | 1,173.45
Diesel O and M (KWh) | 11,055,836.65 | 1,930.88
Fuel Cost | 1,038,589,968.04 | 35,854.60
REopt LLC | 442,926,489.30 | 74,736.26

In this research, two columns are considered as the main targets of the
prediction. The first considered target in this research is the generator-
produced power (kW) and abnormal behavior of the generator-produced power
(kW).

###  II-B Proposed Model

####  II-B1 Convolution Layer

This investigation utilizes a one-dimensional convolution layer to delineate
the temporal associations among elements within a time series dataset[12]. The
essence of this layer lies in its employment of mathematical convolutions,
which process information across designated windows. The formula for computing
the output of this convolution layer is shown in Equation 1.

$y_{k}^{l}\ =\ b_{k}^{l}\ +\ {\displaystyle\sum_{i\ =\ 1}^{\
N_{l}-1}Conv_{1d}\left(w_{ik}^{l-1}*x_{{}_{i}}^{l-1}\right)\ \ \ \ \ \ \ \ }$
(1)

where the ${}_{{}_{i}}^{l-1}$ is defined as the input of the layer l-1,
$b_{k}^{l}$ is defined as the bias of the $k_{th}$ neuron at layer l,
$w_{ik}^{l-1}$ is the kernel from the $i_{th}$ neuron at layer $l-1$ to the
$k_{th}$ neuron at layer l, $Conv_{1d}$ is the 1-dimensional convolution layer
with zero padding function, $y_{k}^{l}$ is $k_{th}$ output of layer $l$.
Utilizing a one-dimensional convolution layer presents a dual benefit of
enabling the extraction of temporal features while concurrently facilitating
the optimization of the end regression or classifier model[13].

####  II-B2 Gated Recurrent Unit (GRU)

The Gated Recurrent Unit (GRU) is an improved version of the LSTM network.
Different from LSTM, the GRU prioritizes passing only the most critical
information while omitting non-vital data. It accomplishes this by employing
fewer gates in its structure [14]. The architecture of the GRU layer is shown
in Figure 2.

Figure 2: Architecture of the GRU.

the equations for components of the GRU are shown as follows:

$r_{t}\ =\ Sigmoid(X_{t}*W_{r}+W_{r}*H_{t-1}+b_{r})$ (2) $z_{t}\ =\
Sigmoid(X_{t}*W_{z}+W_{z}*H_{t-1}+b_{z})$ (3) $Out_{t}\ =\ (1-z_{t})\
*h_{t-1}\ +z_{t}\ *(Tanh(X_{t}*W+W*r_{t}*h_{t-1}+b))$ (4)

The process of calculating the reset gate is illustrated in Equation 2, while
the GRU unit’s output computations are demonstrated in Equation 3. GRUs
address the vanishing gradient descent issue in simple RNNs, improving
training speed. They use update and reset gates to control information
forwarded to the output, similar to LSTM cells but with one less gate, making
them more efficient.

####  II-B3 Attention Layer

Central to each attention mechanism is the process to compute attention scores
for every segment of the input sequence. These scores dictate the level of
focus each part should receive during output generation. Typically, a softmax
function is applied to calculate these attention scores, ensuring their sum
equals one, thus facilitating a weighted importance across the sequence[15].
The architecture of the attention layer is shown in Figure 3.

Figure 3: Attention unit mechanism.

The attention weights are determined by assessing the likeness between the
query and every key, using a measure of similarity such as dot product or
cosine similarity. These weights are subsequently normalized using a softmax
function to achieve a probability distribution. The importance of each
occurrence of the query is then dictated by these weights [16]. The formula
for calculating the emphasizing factor $\alpha$ is mentioned in Equation 5.

$\alpha_{t,i}\ =\ \frac{e^{\ score(X*K_{w},X*Q_{w},)}}{{\displaystyle\sum_{i\
=\ 1}^{\ n}}e^{\ score(X*K_{w},X*Q_{w},)}}$ (5)

####  II-B4 Multilayer Perceptron (MLP)

This study uses a Multilayer Perceptron (MLP) [17], a commonly used
architecture in artificial neural networks, as a regression model to forecast
the final layer [18]. Techniques such as the dropout layer and batch
normalization are employed to prevent overfitting. The hidden layer within MLP
modules consists of a combination of dropout and dense layers. The output
layer’s units depend on the grid’s bus numbers [19].

###  II-C Proposed structure

In this paper, the study introduces a novel framework combining convolutional,
GRU, attention, and MLP layers for forecasting load or detecting anomalies
within datasets. Initially, convolution and GRU layers process the input
signal for time-sensitive patterns, while the attention layer identifies key
insights from the GRU output. The outputs from these layers are merged and
normalized, with the architecture being iteratively applied. The culmination
of this process is the application of a multi-layer perceptron for final value
prediction, with the architecture detailed in the accompanying figure. 4.

Figure 4: The proposed method architecture

###  II-D Baseline Models

Similar to the reviewed research [20], this study conducts a comparative
analysis of the proposed model against traditional ML models using an
identical dataset. The evaluation encompasses KNN, SVM, tree-based methods,
and Bayesian strategies to assess the proposed model’s performance. Each model
applies a unique analytical perspective to the classification and regression
tasks, with subsequent sections delving into the structural composition of
each model for a comprehensive understanding.

####  II-D1 KNN

The K-Nearest-Neighbours (KNN) is a classification technique that uses the
nearest neighbors of a data record to determine its category. Selecting the
right value for ’k’ in this method can be tricky [21]. The KNN Model, a more
efficient approach, automatically optimizes ’k’, improving classification
speed and accuracy while reducing data dependency, making it a viable
alternative to traditional KNN classification [22].

####  II-D2 SVM

Support vector machines (SVMs) are popular supervised learning techniques used
for data classification and information retrieval in large, multidimensional
tasks [23]. They are noted for their high accuracy in text categorization and
pattern identification, especially with small and unbalanced training sets.
While primarily used in binary classification, SVMs can also handle multi-
class problems. Their theoretical and empirical benefits make them suitable
for active learning

####  II-D3 Tree base algorithms

A decision tree is a hierarchical tool used for data classification, boasting
accurate, straightforward analysis [24]. XGBoost and AdaBoost are ensemble
learning algorithms that combine weak learners into one effective learner.
XGBoost is a quick, efficient, scalable method that reduces overfitting,
whilst AdaBoost iteratively builds a strong classifier, focusing on hard-to-
categorize cases. Both are widely used in machine learning applications due to
their improved predicted accuracy [25].

####  II-D4 Beysian Methods

The Naive Bayes algorithm, based on Bayes’ Theorem, is effective for
classifying data such as tumors using gene expression profiling. Despite
assuming each property is independent, it can predict classes based on
assigned posterior class probabilities [26]. Highly parallelizable and
flexible, it’s suitable for large data analytics and applicable to fields like
statistics and bioinformatics. A Bayesian probabilistic model must be built to
use this method.

##  III Experimental Results And Analysis

This study conducted experiments using 80% of the dataset for training and 20%
for testing, with training capped at 10,000 iterations and discontinued if no
improvement was seen after 300 epochs. The initial learning rate was set at
0.001 and was reduced threefold in the absence of performance gains.
Evaluations were performed using a V-100 GPU and an Intel Cori@7 processor,
employing MAE and RMSE as the key metrics.The formulae for calculating the MAE
and RMSE is mentioned as follows:

$\displaystyle Mean\ Absolut\ Error\ (MAE)\ \ =$ (6) $\displaystyle\
\frac{\sum_{i=1}^{N}|Predicted_{i}\ -\ Real_{i}\ |}{N}$ $\displaystyle Root\
Mean\ Squared\ Error\ (RMSE)\ \ \ =$
$\displaystyle\sqrt{\frac{\sum_{i=1}^{N}\left(Predicted_{i}\ -\ Real_{i}\
\right)^{2}}{N}}$ (7) $Accuracy=\frac{TP+TN}{TP+FP+TN+FN}$ (8)
$Precision=\frac{TP}{TP+FP}$ (9)

$TP$, $TN$, $FP$, and $FN$ are abbreviations for true positive, true negative,
false positive, and false negative predictions, respectively.

TABLE II: Comparison between various DL models for load forecasting. Model | MAE | RMSE | R2-score
---|---|---|---
CNN-GRU-Attention | 0.39 | 0.28 | 98.89
SVR | 1.18 | 3.81 | 80.54
KNN | 0.41 | 0.82 | 95.70
XGB | 0.59 | 1.03 | 94.74
Bayesian Ridge | 0.49 | 0.44 | 97.74
RF | 0.25 | 0.48 | 97.53
CNN-GRU-Attention (Kenya) | 0.38 | 0.27 | 98.88
SVR (Kenya) | 1.31 | 4.63 | 76.42
KNN (Kenya) | 0.61 | 0.39 | 96.84
XGB (Kenya) | 0.64 | 1.03 | 94.57
Bayesian Ridge (Kenya) | 0.45 | 0.51 | 97.66
RF (Kenya) | 0.45 | 0.51 | 97.66

The evaluated results for forecasting the load by the proposed DL model amd ML
models has shown in Table II. As shown in Table II, the performance of the
proposed method is superior to ML models. The proposed model performance vs
real value for the predicted instances is shown in Figure 5.

Figure 5: The performance of the model for predicted versus real value.

The analysis across various machine learning models reveals competent accuracy
in identifying instances with zero output. Nevertheless, the models’ efficacy
in predicting values generated by the generator falls short. Specifically, the
Support Vector Regression (SVR) model from the experiment results shows
suboptimal performance for predicting peak generation values around 12kW.
Conversely, the Random Forest (RF) model excels at predicting zero-output
instances but struggles with accurately forecasting generated energy levels of
8kW, highlighting a nuanced challenge in energy production prediction.
The proposed methodologies exhibit robust performance in forecasting non-
generative states and various generation levels. A distinctive strength of the
model lies in its precision for predicting instances with zero generated
power. The model’s versatility is further demonstrated by assessing its
performance under zero-generation conditions and other scenarios through a
modification in its final layer for classification purposes. The efficacy of
this model, particularly in predicting zero-output states, is illustrated in
the accompanying figure.

Figure 6: ROC Curve Showing Performance of Final CNN-GRU Model Figure 7: True
Table Showing Performance of Final CNN-GRU Model

Figures 6 and 7 illustrate that the model’s capability in classifying non-
generative states shows promising results, highlighting its potential in
analyzing fluctuations in power production. This suggests the model could be
valuable in predicting power outputs, even when faced with significant
oscillations.

##  IV Discussion

This study employs a synergistic approach utilizing CNN, GRU, and attention
mechanisms to forecast microgrid operations and identify anomalies, extracting
temporal patterns from various energy sources for predictive load management.
It employs an attention layer to highlight crucial features, validated against
the Micro-grid Tariff Assessment Tool dataset, achieving approximately 99.9%
accuracy in zero state predictions and load forecasting accuracy with metrics
of 0.39 MAE, 0.28 MSE, and an r2-score of 98.89%. The significance of features
on predictions is assessed using Shapley values. The results of shapely values
for each feature is shown in Figure 8.

Figure 8: The importance of each feature based on shapely value.

Shapley values are a concept from cooperative game theory and are used as a
method of attributing a fair reward to players based on their contribution to
the game. Within the context of predictive modeling, these values offer a
quantitative metric to assess the relative importance of each feature,
enabling a nuanced understanding of their impact on model outcomes. As shown
in Figure 8, ”Battery O&M” and ”Reopt LLC” are the two most important features
for forecasting the generated power. Also, the ”PV O&M” and ”Battery O&M” are
the least important features for abnormal behavior detection and generated
load forecasting.
The model exhibits near-optimal performance in precise load forecasting and
anomaly detection, attributed to the optimized number of neurons and units
within the MLP and GRU layers, respectively. This optimization renders the
method relatively shallow, ensuring rapid response times that rival those of
machine learning models and surpass the base algorithm in test phases.
Consequently, this approach is well-suited for real-time applications in load
forecasting and the detection of abnormal behaviors.
Future work will focus on enhancing the model’s accuracy in predicting
microgrid behaviors by exploring various optimization techniques to fine-tune
the hyperparameters,thereby striving to achieve superior performance of the
proposed approach.

##  V Conclusion

A microgrid system consists of many diverse renewable and non-renewable energy
sources that feed into either a standalone system or a utility grid. The
energy produced varies significantly due to the range of sources, like solar
and diesel, leading to fluctuations between no power and full power
generation. These fluctuations can result in abnormal behavior in the
microgrid. Incorporating a mix of renewable and non-renewable sources, this
study devises a predictive model for microgrid dynamics, utilizing convolution
and GRU layers for time-sensitive data extraction, emphasized by an attention
layer. An MLP model refines the prediction and abnormality detection.
Evaluated with the Micro-grid Tariff Assessment Tool dataset, this method
demonstrated accuracy with a 0.39 MAE, 0.28 RMSE, and 98.89% r2-score in load
forecasting, and nearly 99.9% accuracy in zero power state prediction,
showcasing its effectiveness in microgrid behavior forecasting.

## References

  * [1] Haytham Abdelgawad and Vijay K Sood  “A comprehensive review on microgrid architectures for distributed generation”  In _2019 IEEE electrical power and energy conference (EPEC)_ , 2019, pp. 1–8  IEEE
  * [2] Muhammad Qamar Raza and Abbas Khosravi  “A review on artificial intelligence based load demand forecasting techniques for smart grid and buildings”  In _Renewable and Sustainable Energy Reviews_ 50 Elsevier, 2015, pp. 1352–1372
  * [3] Roberto Cahuantzi, Xinye Chen and Stefan Güttel  “A comparison of LSTM and GRU networks for learning symbolic sequences”  In _Science and Information Conference_ , 2023, pp. 771–785  Springer
  * [4] Hongyi Pan, Xin Zhu, Salih Furkan Atici and Ahmet Cetin  “A hybrid quantum-classical approach based on the hadamard transform for the convolutional layer”  In _International Conference on Machine Learning_ , 2023, pp. 26891–26903  PMLR
  * [5] Luis B Almeida  “Multilayer perceptrons”  In _Handbook of Neural Computation_ CRC Press, 2020, pp. C1–2
  * [6] Praveen Damacharla, Hamid Rajabalipanah and Mohammad Hosein Fakheri  “LSTM-CNN Network for Audio Signature Analysis in Noisy Environments”  In _2023 International Conference on Computational Science and Computational Intelligence (CSCI)_ , Dec. 13-15, 2023, pp. 78–82  DOI: 10.1109/CSCI62032.2023.00019
  * [7] Bibi Ibrahim, Luis Rabelo, Edgar Gutierrez-Franco and Nicolas Clavijo-Buritica  “Machine learning for short-term load forecasting in smart grids”  In _Energies_ 15.21 MDPI, 2022, pp. 8079
  * [8] Mohamed El-Hendawi and Zhanle Wang  “An ensemble method of full wavelet packet transform and neural network for short term electrical load forecasting”  In _Electric Power Systems Research_ 182 Elsevier, 2020, pp. 106265
  * [9] Xinfang Chen et al.  “Short-term load forecasting and associated weather variables prediction using ResNet-LSTM based deep learning”  In _IEEE Access_ 11 IEEE, 2023, pp. 5393–5405
  * [10] Paniz Hosseini, Saman Taheri, Javid Akhavan and Ali Razban  “Privacy-preserving federated learning: Application to behind-the-meter solar photovoltaic generation forecasting”  In _Energy Conversion and Management_ 283 Elsevier, 2023, pp. 116900
  * [11] Nina Rangel, Hu Li and Petros Aristidou  “An optimisation tool for minimising fuel consumption, costs and emissions from Diesel-PV-Battery hybrid microgrids”  In _Applied Energy_ 335 Elsevier, 2023, pp. 120748
  * [12] Teja Kattenborn, Jens Leitloff, Felix Schiefer and Stefan Hinz  “Review on Convolutional Neural Networks (CNN) in vegetation remote sensing”  In _ISPRS journal of photogrammetry and remote sensing_ 173 Elsevier, 2021, pp. 24–49
  * [13] Serkan Kiranyaz et al.  “1D convolutional neural networks and applications: A survey”  In _Mechanical systems and signal processing_ 151 Elsevier, 2021, pp. 107398
  * [14] Gensheng Qian and Jingquan Liu  “Fault diagnosis based on gated recurrent unit network with attention mechanism and transfer learning under few samples in nuclear power plants”  In _Progress in Nuclear Energy_ 155 Elsevier, 2023, pp. 104502
  * [15] Xiang Li et al.  “Deep learning attention mechanism in medical image analysis: Basics and beyonds”  In _International Journal of Network Dynamics and Intelligence_ , 2023, pp. 93–116
  * [16] Meng-Hao Guo et al.  “Attention mechanisms in computer vision: A survey”  In _Computational visual media_ 8.3 Springer, 2022, pp. 331–368
  * [17] Chenggang Yan et al.  “STAT: Spatial-temporal attention mechanism for video captioning”  In _IEEE transactions on multimedia_ 22.1 IEEE, 2019, pp. 229–241
  * [18] Praveen Damacharla, Achuth Rao M.., Jordan Ringenberg and Ahmad Javaid  “TLU-Net: A Deep Learning Approach for Automatic Steel Surface Defect Detection”  In _2021 International Conference on Applied Artificial Intelligence (ICAPAI)_ , 2021, pp. 1–6  DOI: 10.1109/ICAPAI49758.2021.9462060
  * [19] Reza Sabzehgar, Diba Zia Amirhosseini and Mohammad Rasouli  “Solar power forecast for a residential smart microgrid based on numerical weather predictions using artificial intelligence methods”  In _Journal of Building Engineering_ 32 Elsevier, 2020, pp. 101629
  * [20] Salah Bouktif, Ali Fiaz, Ali Ouni and Mohamed Adel Serhani  “Optimal deep learning lstm model for electric load forecasting using feature selection and genetic algorithm: Comparison with machine learning approaches”  In _Energies_ MDPI, 2018, pp. 1636
  * [21] Padraig Cunningham and Sarah Jane Delany  “k-Nearest neighbour classifiers-A Tutorial”  In _ACM computing surveys (CSUR)_ 54.6 ACM New York, NY, USA, 2021, pp. 1–25
  * [22] Abhishek Sahu, Peter H. Aaen and Praveen Damacharla  “An Automated Machine Learning Approach to Inkjet Printed Component Analysis: A Step Toward Smart Additive Manufacturing”  In _2024 IEEE Texas Symposium on Wireless & Micrwowave Circuits and Systems_, April 3 - 4th, 2024, pp. 1–6
  * [23] Mariette Awad, Rahul Khanna, Mariette Awad and Rahul Khanna  “Support vector machines for classification”  In _Efficient learning machines: Theories, concepts, and applications for engineers and system designers_ Springer, 2015, pp. 39–66
  * [24] WP Zhao et al.  “XGB model: Research on evaporation duct height prediction based on XGBoost algorithm”  In _Radioengineering_ 29.1, 2020, pp. 81–93
  * [25] Arundhati Navada, Aamir Nizam Ansari, Siddharth Patil and Balwant A Sonkamble  “Overview of use of decision tree algorithms in machine learning”  In _2011 IEEE control and system graduate research colloquium_ , 2011, pp. 37–42  IEEE
  * [26] Fernando G Zampieri et al.  “Using Bayesian methods to augment the interpretation of critical care trials. An overview of theory and example reanalysis of the alveolar recruitment for acute respiratory distress syndrome trial”  In _American journal of respiratory and critical care medicine_ 203.5 American Thoracic Society, 2021, pp. 543–552