# Style-Guided Inference of Transformer for High-resolution Image Synthesis

Jonghwa Yim Minjae Kim
Vision AI Lab., AI Center, NCSOFT
South Korea
{jonghwayim<EMAIL_ADDRESS>
###### Abstract

Transformer is eminently suitable for auto-regressive image synthesis which
predicts discrete value from the past values recursively to make up full
image. Especially, combined with vector quantised latent representation, the
state-of-the-art auto-regressive transformer displays realistic high-
resolution images. However, sampling the latent code from discrete probability
distribution makes the output unpredictable. Therefore, it requires to
generate lots of diverse samples to acquire desired outputs. To alleviate the
process of generating lots of samples repetitively, in this article, we
propose to take a desired output, a style image, as an additional condition
without re-training the transformer. To this end, our method transfers the
style to a probability constraint to re-balance the prior, thereby specifying
the target distribution instead of the original prior. Thus, generated samples
from the re-balanced prior have similar styles to reference style. In
practice, we can choose either an image or a category of images as an
additional condition. In our qualitative assessment, we show that styles of
majority of outputs are similar to the input style.

##  1 Introduction

Image generation is a task that learns a mapping from a prior density to real
image distribution. Recent works of deep generative models developed for this
task derive from two major branches; one is Generative Adversarial Nets (GAN)
[11] which is a pioneering research of implicit density models; the other is
explicit density models which include Variational Auto-Encoder (VAE) [17] and
Autoregressive generative models. Until recently, studies derive from GAN and
VAE show distinguished achievement in image quality, however, they yet have a
couple of limitations; first, they have a difficulty achieving high resolution
and satisfactory output quality due to inherent limitations in either loss
design or structurally restricted receptive field; second, the training
process is relatively unstable.

To improve the problems of GAN and VAE based methods, another direction of
research heads towards more direct formulation. To formulate tractable
density, PixelRNN [45] proposes an auto-regressive architecture to compute the
global relationship explicitly and model the image components recursively.
Thanks to the advantage in viewing global context, this research stably
produces locally and globally coherent results. Following research,
Conditional PixelCNN [31], introduces a condition in representing the target
distribution and displays qualitatively coherent results to given labels.
Despite solving the two major problems in GAN and VAE, its computation
complexity is proportional to the square of the output size, hampering
extending the size of an architecture and output resolution.

Recently, a transformer architecture from previous research [46] provides clue
for the above scalability problem in a tractable density model. By replacing
recursive computation for long range relationships to transformer with masked
self-attention module, the study achieves both of full receptive field and
computational efficiency. Thus, following researches start adopting
transformer instead of the recursive architecture. The research [34]
introduces transformer in image generation task and shows comparable or even
better performance than recursive neural network (RNN) based architectures
with a better computational efficiency. However, the limitation of these
methods lies in insufficient quality compared to GAN-based methods since
modeling low-level vision, pixel values in most cases, is qualitatively
inaccurate compared to GAN-based methods.

Instead of predicting low-level vision, predicting quantised latent
representation becomes a substitute. The first of this appears in VAE
approaches. The previous researches [44], [36] introduce vector quantisation
to latent representation in VAE, called Vector Quantised-Variational
AutoEncoder (VQ-VAE), thereby the latent prior becomes discrete and
categorical. Inspired by this study, the recent study [10] adopts VQ-VAE to
represent latent representation while the transformer infers quantized code
instead of pixel value. Hence, the output of the transformer is a prior,
probability density over discrete codebook. After the transformer infers the
prior, sampling a code from the prior probability creates a sequence of codes
which is deterministically translated to an image. Because a quantised latent
representation relieves the transformer from modeling the complex low-level
vision, instead it can focus more on learning the global context. With the
advantage of efficient role allocation, this study displays high-resolution
and natural-looking outputs. However, what would be generated from prior
sampling is unknown until it generates and decodes the full sequence of
quantised codes, which is much more time consuming as the target resolution is
getting higher.

To resolve the unpredictable behavior of sampling from prior density, our
proposal confers controllability on the above research. We propose a novel
method to guide prior sampling by adopting a style image as a conditional
input. To this end, we reformulate conditional generative model to posterior
expression where conditional input is a given observation. Therefore, our
method guides the inference of pre-trained transformer using a conditional
input without re-training the model. Our method inherits the merits of the
previous transformer-based generative approach, creating high-resolution
photo-realistic output images. Besides, our method accepts conditional input
image and generates outputs of similar styles to the given condition.

Our method can work both on a non-conditional and conditional pre-trained
transformer, and its output is visually well persuasive, especially in
landscape scenes where we can feed a semantic label map as a condition. In the
evaluation section, we show experiments of a single image and multiple images
as input styles. In the case of multiple images, we perform sampling and
averaging the distribution of latent features over the entire images.

##  2 Related Works

GAN-based Image Synthesis. Over the last years, the field of image synthesis
has been massively researched. Since the debut of Generative Adversarial
Network [11], this field has progressed to higher resolution and better
quality than ever before. However, the original GAN holds a few drawbacks; the
training process is unstable, enlarging the resolution is limited by the
architectural design and machine to run the model, and the output quality is
not yet that of the real world. Recently, a few cornerstone researches break
the limitations and enable to synthesize the real-world scene. The pioneering
research, PG-GAN [15], suggests GAN generating higher resolution by
progressively stacking more layers and consequently enlarging the resolution.
Meanwhile, to alleviate the instability in training process, we can find
various former researches including Wasserstain GAN [2], PGGAN [15], SN-GAN
[25], techniques for training GANs [38], LSGAN [22], and BEGAN [3]. Improving
the architecture is another branch of research; DCGAN [35] embraces
convolutional architecture, SA-GAN [52] adds self-attention blocks to its
backbone, and BigGAN [4] composes many previously-existing modules such as
self-attention, spectral normalization, and a discriminator with projection
[26]. Lastly, StyleGAN [16] is a foundation in its way. It proposes
disentangled latent priors from mapping network instead of random noise for
attribute control, stochastic variation, and better output quality.

Conditional Generative Model. Conditional image generation refers to the task
that generates images conditioning on a particular input data. For example,
conditioning on a class label [24], [27], [4], [23], [26] allows the generated
samples to be in the class. Another case of conditional generative models
[37], [54], [13], [49] is based on text to synthesize images. Recently, many
researches [47], [20], [32], [9], [43], [55], [56], [57], [42], [41] use
semantic segmentation map to indicate what to draw. SPADE [32] suggests a
spaital-varying normalization layer and becomes a significant backbone for the
following researches. CC-FPSE [20] applies spatially-varying convolutional
kernels which are based on the semantic label map to fully utilize semantic
layout information to generate high-quality and semantically high-fidelity
images. SEAN [56] displays per-semantic styling by applying region-adaptive
normalization. CLADE [42] proposes class-adaptive normalization layer. INADE
[41] presents an instance-adaptive modulation sampling and improves diversity.
Our work also extends the series of conditional generative model, except that
it suggests guided inference to follow a given condition instead of
architecturally accepting conditional input with additional parameters and
requiring full training process.

Figure 1: System overview of image-generative process of baseline method [10]
(top) and our method (bottom). The major difference from the baseline work is
applying style likelihood $P(z_{u}|T)$ in the middle of the process to apply
an additional condition $u$. $P(z_{u})$ and $P_{D}(T)$ are probability
distributions of all indices in latent maps. Latent map for $P(z_{u})$ is from
conditional image $u$. Latent maps for $P_{D}(T)$ are from images in training
dataset $D$. Encoder $E$, decoder $G$, and transformer $T$ are pre-trained. In
case of landscape synthesis, baseline method accepts semantic segmentation map
as a conditional input, while in case of face synthesis, baseline is
unconditional.

Autoregressive Generative Model. Autoregressive model formulates a tractable
density function as a generative approach. Due to its simplicity and
directness, it is broadly used in image and audio generation. In image
generation, autoregressive deep generative model predicts the probability over
discrete pixel value regressively using the architectures called PixelRNN and
PixelCNN [45], [31], [6]. In audio generation, WaveNet [30] introduces
PixelCNN architecture to the 1D audio data by applying dilated causal
convolution layer, followed by many researches [48], [1], [39], [29] that
improve the quality of synthesis. Afterwards, a transformer architecture [46]
is newly introduced in autoregressive generative method [34] replacing the
former RNN based architecture. Transformer becomes prevalent and also appears
in GAN based approaches [52], [14]. Recently, the state-of-the-art study,
Taming Transformer [10], adopts vector quantisation introduced in VQ-VAE [44],
[36] and predicts probability density over the entries of a quantised latent
vector instead of direct prediction of a discrete pixel value. Thanks to
lessen the burden of transformer from predicting low-level complex vision
feature, this work generates photo-realistic and high-resolution scenes.

Arbitrary Photorealistic Style Transfer. Arbitrary photorealistic style
transfer is partially in line with our method in that it stylizes the output,
though it requires a real input image. Some recent studies [21], [18], [19],
[40], [51], [50], [33], [7] introduce various methods to transfer style from a
single image, but they demand a large dataset for training. Besides, they
often yield undesirable transfer of color and texture due to the nature of the
neural network.

##  3 Method

###  3.1 Baseline Work

The upper half of Figure 1 overviews the baseline method [10]. The baseline
model is possible to be both unconditional and conditional. Specifically, it
is conditional upon a semantic label map in the case of landscape synthesis.
The training process of the baseline is a two-stage. In the first stage,
encoder $E$ and decoder $G$ are trained. While in the second stage,
transformer $T$ is trained, and $E$ is omitted later. $Z$ is a discrete
codebook of size $|Z|$. Transformer model $T$ auto-regressively outputs a
probability distribution $p(s_{i})$ over indices $s$ in $Z$, where an index
$s_{i}$ is random-sampled. Latent code $z_{i}$ is a paired entry of $s_{i}$ in
$Z$. $z_{i}$ fills the $i$-th entry in the latent map $z_{q}$. As a result of
an auto-regressive process, $z_{q}$ is formed and fed into $G$ to synthesize
the output. In this process, random sampling from $p(s_{i})$ makes the output
unpredictable. Thus it is hard to find desired styles such as $u$ from random
samples.

###  3.2 Style-guided Inference of Transformer

Our method aims for high-resolution conditionally guided image generation
without re-training the transformer. On top of the latest progress of
transformer [10], which incorporates an auto-regressive prediction and a
quantized prior, our method will be a missing-piece to guide the inference of
transformer conditionally and to stylize the output. To this end, we present a
method to control the probability distribution over an auto-regressive latent
prediction.

The bottom half of Figure 1 indicates an overview of the proposed method.
$P(z_{u})$ is a probability distribution of indices in the latent map of style
image $u$, and $P_{D}(T)$ is the model’s probability distribution of all
indices in latent maps of images sampled from the training dataset $D$. A
style likelihood $P(z_{u}|T)$ is drawn from $P(z_{u})$ and $P_{D}(T)$. Then,
it is repetitively multiplied to $p(s_{i})$ to form a posterior $P(T|z_{u})$.
(Throughout the paper, we write $p(s_{i})$ as $P(T)$ to denote a model’s prior
probability distribution.) From $P(T|z_{u})$, a style-guided index $s_{i}$ is
sampled, and $z^{(i)}$ is the paired entry of $s_{i}$.

To be specific, if we condition $P(T)$, the inference of pre-trained model, on
a style input $u$, then conditional probability is written as $P(T|u)$.
Previously, to make $T$ represent this distribution, $T$ needed supervised
learning on a style-labeled dataset. Our method, however, proposes the
conditionally guided-inference of pre-trained transformer without learning
such condition. Without re-train, we represent the given problem to a style
likelihood and a model’s prior: (Note that $\odot$ and $\oslash$ we use in
this paper are element-wise multiplication and division of two vectors; $i$-th
entries of $A\odot B$ and $A\oslash B$ are $A_{i}\times B_{i}$ and
$A_{i}/B_{i}$ each)

$\displaystyle P(T|u)\propto P(u|T)P(T)\text{.}$ (1)

Instead of $u$, we replace it with the probability distribution of latent
codes, $P(z_{u})$, by ignoring the sequence of latent map $z_{u}$ and
expressing all indices in $z_{u}$ as a probability distribution. (Note that
the generator part is deterministic, and the specific sequence of $z_{u}$
restores the correct $u$.) However, ignoring the sequence makes $P(z_{u})$
ill-posed to rebuild $u$. Instead, we rely on the last term, $P(T)$, to build
the sequence as this term is recursive. In the same way we get $P(z_{u})$, we
also obtain the probability distribution of $T$ of the training dataset $D$,
$P_{D}(T)$, by counting indices in latent maps from images in $D$. Then, we
can view the likelihood term is proportional to the events of latent indices
of conditional input $u$ divided by model’s general event counts:
$P(z_{u})\oslash P_{D}(T)$. Accordingly, the problem is rewritten into

$\displaystyle P(z_{u}|T)\odot P(T)\propto\\{P(z_{u})\oslash P_{D}(T)\\}\odot
P(T)\text{,}$ (2)

where $P(\cdot)\in\mathcal{R}^{|Z|}$, probability over the codebook indices.

Practically, instead of calculating all latent maps from images in $D$ to get
$P_{D}(T)$, we perform Monte-Carlo sampling and take the average,
$1/K\sum_{k=1}^{K}P(z_{x_{k}})$, for all $K$ number of randomly sampled $x$
from the training dataset $D$. Hence, the Eq. (2) becomes

$\displaystyle P(z_{u}|T)P(T)\propto
P(z_{u})\oslash\\{\frac{1}{K}\sum_{k=1}^{K}P(z_{x_{k}})\\}\odot P(T)$ (3)

where $\sum$ is element-wise summation.

Consequently, as the auto-regressive generative process continues, $P(z)$, the
distribution of sampled indices from $P(T|u)$, is getting closer to
$P(z_{u})$, making the style of the generated sample similar to $u$.

In practice, to reinforce accuracy lost from dropping the spatial and
sequential information when representing $P(u)$ as $P(z_{u})$, we can use a
semantic label map or coordinates in latent map. For example, when we have a
semantic label map as a conditional input, we use it to count $P(z_{u}|T)$
partially by semantic regions: $P(z_{u,j})\oslash P_{D}(T_{j})$ for $j$
labeled semantic region. (So we have $J$ number of style likelihoods where $J$
is the number of semantic labels.) In this way, we can accurately transfer the
style of each semantic to a corresponding semantic. In the following chapter,
we show experiments on single style image and a categorized set of style
images. In case of a categorized set of style images, we can compute the
average style likelihood.

##  4 Experiments

In this section, firstly, we design experiments on a landscape images because
these have visually distinguishable styles like snowy or rocky mountain. To
prepare a landscape dataset for training, we collect 80k images from various
sources, including Flickr, Pixabay, and Google search. Then, we generate
pseudo-semantic labels using the well-known semantic segmentation method,
DeepLabv3 [5] with ResNeSt-101 [53] backbone. As a base architecture for image
generation, we configure an autoregressive transformer conditioning on a
semantic label map presented by the previous paper [10]. To get the model’s
probability $P_{D}(T)$ over the training dataset, we randomly select $K=700$
number of images from dataset. Since our method can accept one or more style
inputs, firstly, we show experiments on a single style image, then on a style-
categorized set of multiple images sequentially for diversity. Note that we
can also choose different source of style for each semantic.

Figure 2: Qualitative results of our proposed method. Images on the left are
style references of the sky, and on the top are styles of the mountain. Style
of one of the sky references is applied to where semantic label matches the
sky, while one of the mountain references is applied to where the semantic
label matches the mountain. Three different semantic types (a), (b), and (c)
are used. Thus, each cell is a mixture of semantically one of (a), (b), and
(c), sky from sky reference on the left, and mountain from mountain reference
on the top.

In the first experiment shown in Figure 2, we experiment with our method
conditioned on a single style image. For exact modeling of style likelihood
probability distribution from a style image, we divide regions by semantic
labels and extract the style likelihood regionally. Then, we apply each
distribution of semantic labels to the target semantic label, which we call
semantic-aware stylization. For this experiment, we choose 11 different styles
for the sky from the Flickr dataset and two different styles for the mountain
from randomly synthesized samples. Then, style from one of the sky images and
one of the mountain images are applied to the sky and mountain semantics each.
Thus, the results have stylized sky and mountain in each image. (It is also
possible to stylize more semantic labels with different style images.) As a
conditional input, we prepare three types of semantic label maps, (a), (b),
and (c). (It can be inferred from result images.) We repeat the guided-
synthesis four times and pick the best one for each result image.

Figure 3: Qualitative results for categorized styles. We create six categories
(a-f) and collect around 30 to 50 images each by keyword-search and manual
inspection. Those images are used as style images to get average probabilities
for categorized styles. Then we generate images of each category four times

In the following experiment shown in Figure 3, we perform experiment on
categorized set of style images. At first, we search images by keywords -
“mountain” + “rock”, “mountain” + “snow”, “night”, “sunset”, and “winter” - in
Pixabay website. Then we choose 30 to 50 images for each category.
Additionally, category “green” is collected by manually selecting images of
green fields. After collecting images by style category, we get the likelihood
probabilities for every semantic labels from semantic regions in all style
images. Therefore, results in Figure 3 well reflects the common styles of
collected images in the category. In this figure, we repeat the inference four
times for each category and contain all of them.

Figure 4: Qualitative comparison with arbitrary style transfer. Given a style
images, our semantic-aware conditional generation shows plausible outputs in
every semantic regions. “Segmap” is a semantic segmentation map to feed as a
conditional input to baseline and our method for semantic-aware stylization.
For comparison, we run semantic-aware style transfer, WCT2 [51], and texture
swapping, SwappingAE [33] (official “Flickr Mountains” pretrained model), on
randomly generated samples from the baseline method

To the best of our knowledge, this is the first conditional image generation
using pre-trained generative model without learning such condition. To achieve
a similar result using any existing method, we have a few workarounds.
Firstly, there are a few researches on few-shot learning of image generative
model, however, they requires hundreds of samples and hardly generate
plausible samples on complex scenes. Instead, we can stylize the randomly
generated samples using an arbitrary style transfer method. Since our method
can apply style conditions by semantic regions, we compare it with one of the
state-of-the-art arbitrary style transfer, WCT2 [51], especially which can
perform a semantic-aware style transfer. Also we prepare recently well-known
texture transferring method, SwappingAE [33]. Although the style transfer
method produces reasonable outputs in some cases, it does not consistently
generate pleasant outputs in most cases. Furthermore, style transfer approach
still yields unmatched or ruined color and texture, which is not desirable in
high-resolution realistic scene generation. SwappingAE is excellent at
transferring tones and texture details. However, it fails to maintain semantic
contents. Moreover, it tends to follow the spatial scene compositions of style
image. (For example, it alters mountain to sky in the first two images, river
to ground in the second last, and ground to river in the last image.) Detailed
results are shown in Figure 4.

Figure 5: FFHQ babies/sunglasses-guided samples using our method. (Top): Babies-guided random samples. (Bottom): Sunglasses-guided random samples. Methods | Babies | Sunglasses
---|---|---
Random sampling | 155.30 | 94.21
[28] | 74.39 | 42.13
Ours | 49.96 | 39.15
Table 1: Quantitative evaluation on FFHQ. FID scores on two categories of FFHQ
dataset are measured. Our work synthesizes images that are closer to the
category of given guidance than the state-of-the-art competitor [28]. “Random
sampling” means random sampling using unconditional baseline [10].

Our work uses pseudo-segmentation map which is already available for semantic
image synthesis of our baseline work [10]. Having a semantic segmentation map
improves extracting the exact per-semantic style and lets us partially stylize
the output. On the contrary, in cases of unconditional face synthesis, we can
use aligned dataset and collect the probability vectors by spatial locations.
To perform a quantitative assessment of an unconditional model, we measure
Frechet Inception Distance (FID) [12] score on Flickr-Faces-HQ (FFHQ) [16]
dataset and compare with the state-of-the-art few shot image generation [28].
To use as a guidance, we randomly select 10 images of baby and sunglasses
each. Then we calculate FID using 2492 and 2683 images of baby and sunglasses
each which are the same test sets made public by [28]. We use the same face
synthesis model that the baseline work [10] released. Quantitative results of
guided-inference of unconditional transformer are given in Table 1. One
limitation is, however, our proposed method is not able to stylize out of
learnt domain because our method guides the inference of transformer only, not
re-training the model. Thus, in this experiment, we exclude “Sketches”
category from the experiment shown in the competitor [28].

##  5 Limitations

Our method provides a style image to guide the inference of transformer
introduced in our baseline work [10]. To achieve similar effect, most of the
recent studies require a re-training process and thousands of style images as
well, and this makes recent studies harder to apply to the cases where only
one or dozens of style images are provided. Our paper is unique in that,
instead of requiring a re-training process, it only needs a single or a few
style images to get the output of the desired style, while it inherits
advantages of the pre-trained backbone. However, since it does not include a
re-training process, it cannot generate samples out of the distribution of the
training dataset. Therefore, as shown in Table 1, it is impossible to generate
“Sketches” style output using the model trained on real face images.

Styles |  | Random sampling
---
| Ours
---
Abstract Expressionism | 94.24 | 80.82
Art Nouveau Modern | 65.73 | 58.13
Baroque | 83.02 | 68.63
Cubism | 96.18 | 69.45
Naive Art Primitivism | 79.25 | 70.58
Northern Renaissance | 84.42 | 80.21
Rococo | 96.79 | 82.26
Table 2: Ablation study of unconditional image synthesis on WikiArt dataset
[8]. Unlike above experiments, in this experiment, our method is not provided
any spatial aid for style likelihood. FID scores on the styles of the art by
the history are measured. We only count styles that have more than 2000 images
and less than 4500 images to prevent biased random sampling. “Baseline” stands
for random sampling-based synthesis of baseline transformer, while “Ours” does
for our style-guided synthesis.

To model the ill-posed $P(u|T)$ with the known variables, we use probability
distributions of $P(z_{u})$ and $P_{D}(T)$. To improve accuracy in the
process, we use semantic label maps in case of landscape synthesis and
coordinates of a latent map in case of aligned face synthesis. However, if we
aggregate probability distribution over all entries of a latent map without
spatial separation, it generates a limited result. To show this limitation,
where we claim the minimal performance, we carry out an ablation study on
WikiArt dataset [8]. In this experiment of unconditional image synthesis, we
collect style vectors from the entire region of the image. As shown in Table
2, FID scores using our method are lower than random sampling-based synthesis.
However, it is not dramatic gab like proven in Table 1.

##  6 Conclusion and Future Work

We believe that this is the first study on the guided inference of transformer
to accept an additional arbitrary condition to stylize the output without
requiring any parameter nor training procedure. To this end, we interpret
probability distribution conditioned on style to a product of style
distribution and a prior from transformer. In the experiment section, we prove
our method on landscape dataset where styles are diverse and distinguishable.
Compared to random generation of the original architecture [10], our method
can generate scenes of an arbitrary style that are rare from the original
method such as night scene. As an additional arbitrary condition, our work can
use a single style image as well as a categorized set of style images. Our
work is also similar to style transfer in a way that it stylizes the output
image. However, our method uses the pretrained architecture and lets it
describe target condition without re-training nor requiring any additional
architecture, and thus our method does not need a condition-labeled dataset
either.

Since our approach is built upon autoregressive latent code prediction, our
work is currently applied on a limited previous work. In the future, we
therefore plan to build a general conditional autoregressive module to support
an arbitrary conditional input. In addition, our work relies on transformer to
build the proper sequence, not the relative layout or sequence of latent codes
of style image. Therefore, for more accurate spatial application of style, we
will work on space-aware style-guided inference of transformer in the near
future.

## References

  * [1] Sercan Ö Arık, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al.  Deep voice: Real-time neural text-to-speech.  In International Conference on Machine Learning, pages 195–204. PMLR, 2017.
  * [2] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In International Conference on Machine Learning, pages 214–223. PMLR, 2017.
  * [3] David Berthelot, Thomas Schumm, and Luke Metz.  Began: Boundary equilibrium generative adversarial networks.  arXiv preprint arXiv:1703.10717, 2017.
  * [4] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In International Conference on Learning Representations, 2018.
  * [5] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [6] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel.  Pixelsnail: An improved autoregressive generative model.  In International Conference on Machine Learning, pages 864–872. PMLR, 2018.
  * [7] Jiaxin Cheng, Ayush Jaiswal, Yue Wu, Pradeep Natarajan, and Prem Natarajan.  Style-aware normalized loss for improving arbitrary style transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 134–143, 2021.
  * [8] Cs-Chan.  Artgan/wikiart dataset.  https://github.com/cs-chan/ArtGAN/tree/master/WikiArt Dataset.
  * [9] Aysegul Dundar, Karan Sapra, Guilin Liu, Andrew Tao, and Bryan Catanzaro.  Panoptic-based image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8070–8079, 2020.
  * [10] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873–12883, 2021.
  * [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in Neural Information Processing Systems, 27, 2014.
  * [12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in Neural Information Processing Systems, 30, 2017.
  * [13] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee.  Inferring semantic layout for hierarchical text-to-image synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7986–7994, 2018.
  * [14] Drew A Hudson and C. Lawrence Zitnick.  Generative adversarial transformers.  In International Conference on Machine Learning, pages 4487–4499. PMLR, 2021.
  * [15] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In International Conference on Learning Representations, 2018.
  * [16] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [17] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  International Conference on Learning Representations, 2014.
  * [18] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.  Universal style transfer via feature transforms.  Advances in Neural Information Processing Systems, 30, 2017.
  * [19] Yijun Li, Ming-Yu Liu, Xueting Li, Ming-Hsuan Yang, and Jan Kautz.  A closed-form solution to photorealistic image stylization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 453–468, 2018.
  * [20] Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hongsheng Li.  Learning to predict layout-to-image conditional convolutions for semantic image synthesis.  In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 570–580, 2019.
  * [21] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala.  Deep photo style transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4990–4998, 2017.
  * [22] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  * [23] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  In International Conference on Machine Learning, pages 3481–3490. PMLR, 2018.
  * [24] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [25] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In International Conference on Learning Representations, 2018.
  * [26] Takeru Miyato and Masanori Koyama.  cgans with projection discriminator.  In International Conference on Learning Representations, 2018.
  * [27] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In International Conference on Machine Learning, pages 2642–2651. PMLR, 2017.
  * [28] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10743–10752, 2021.
  * [29] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al.  Parallel wavenet: Fast high-fidelity speech synthesis.  In International Conference on Machine Learning, pages 3918–3926. PMLR, 2018.
  * [30] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  In 9th ISCA Speech Synthesis Workshop, pages 125–125, 2016.
  * [31] Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu.  Conditional image generation with pixelcnn decoders.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 4797–4805, 2016.
  * [32] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.
  * [33] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang.  Swapping autoencoder for deep image manipulation.  Advances in Neural Information Processing Systems, 33:7198–7211, 2020.
  * [34] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In International Conference on Machine Learning, pages 4055–4064. PMLR, 2018.
  * [35] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In International Conference on Learning Representations, 2016.
  * [36] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In Advances in Neural Information Processing Systems, pages 14866–14876, 2019.
  * [37] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In International Conference on Machine Learning, pages 1060–1069. PMLR, 2016.
  * [38] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2234–2242, 2016.
  * [39] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779–4783. IEEE, 2018.
  * [40] Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang.  Avatar-net: Multi-scale zero-shot style transfer by feature decoration.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8242–8250, 2018.
  * [41] Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Bin Liu, Gang Hua, and Nenghai Yu.  Diverse semantic image synthesis via probability distribution modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7962–7971, 2021.
  * [42] Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Gang Hua, and Nenghai Yu.  Efficient semantic image synthesis via class-adaptive normalization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [43] Hao Tang, Song Bai, and Nicu Sebe.  Dual attention gans for semantic image synthesis.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1994–2002, 2020.
  * [44] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309–6318, 2017.
  * [45] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  In International Conference on Machine Learning, pages 1747–1756. PMLR, 2016.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, pages 5998–6008, 2017.
  * [47] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [48] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous.  Tacotron: Towards end-to-end speech synthesis.  In Interspeech 2017, pages 4006–4010, 2017.
  * [49] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [50] Jonghwa Yim, Jisung Yoo, Won-joon Do, Beomsu Kim, and Jihwan Choe.  Filter style transfer between photos.  In European Conference on Computer Vision, pages 103–119. Springer, 2020.
  * [51] Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha.  Photorealistic style transfer via wavelet transforms.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9036–9045, 2019.
  * [52] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In International Conference on Machine Learning, pages 7354–7363. PMLR, 2019.
  * [53] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al.  Resnest: Split-attention networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2736–2746, 2022.
  * [54] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017.
  * [55] Haitian Zheng, Haofu Liao, Lele Chen, Wei Xiong, Tianlang Chen, and Jiebo Luo.  Example-guided image synthesis using masked spatial-channel attention and self-supervision.  In European Conference on Computer Vision, pages 422–439. Springer, 2020.
  * [56] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka.  Sean: Image synthesis with semantic region-adaptive normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5104–5113, 2020.
  * [57] Zhen Zhu, Zhiliang Xu, Ansheng You, and Xiang Bai.  Semantically multi-modal image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5467–5476, 2020.


capbtabboxtable[][]

20cm(1cm,1cm) Accepted by 2023 IEEE ISGT North America. To appear in IEEE
Xplore.

# Short-term Load Forecasting with Distributed Long Short-Term Memory
††thanks: This work is supported by the UK EPSRC through End-to-End Conceptual
Guarding of Neural Architectures [EP/T026995/1]).

Yi Dong  Department of Computer Science
University of Liverpool
Liverpool, the UK
<EMAIL_ADDRESS>Yang Chen  Department of R & D
Suzhou SeeEx Technology Co., Ltd
Jiangsu, China
<EMAIL_ADDRESS>Xingyu Zhao, Xiaowei Huang  Department of Computer
Science
University of Liverpool
Liverpool, the UK
{xingyu.zhao<EMAIL_ADDRESS>
###### Abstract

With the employment of smart meters, massive data on consumer behaviour can be
collected by retailers. From the collected data, the retailers may obtain the
household profile information and implement demand response. While retailers
prefer to acquire a model as accurate as possible among different customers,
there are two major challenges. First, different retailers in the retail
market do not share their consumer’s electricity consumption data as these
data are regarded as their assets, which has led to the problem of data
island. Second, the electricity load data are highly heterogeneous since
different retailers may serve various consumers. To this end, a fully
distributed short-term load forecasting framework based on a consensus
algorithm and Long Short-Term Memory (LSTM) is proposed, which may protect the
customer’s privacy and satisfy the accurate load forecasting requirement.
Specifically, a fully distributed learning framework is exploited for
distributed training, and a consensus technique is applied to meet
confidential privacy. Case studies show that the proposed method has
comparable performance with centralised methods regarding the accuracy, but
the proposed method shows advantages in training speed and data privacy.

###### Index Terms:

short-term load forecasting, long short term memory, distributed learning,
consensus, multi-agent system

##  I Introduction

Electricity load forecasting is an essential basis for not only industrial
production but also social life. The analysis of these load data could help in
revealing household profile information and enabling other uses [1, 2, 3].
Specifically, retailers could obtain the electricity consumption behaviour of
consumers and provide social and behavioural incentive signals to optimise
customers’ electricity usage. However, customers’ data are not able to be
shared among different power supply companies. Therefore, an appropriate
algorithm that can generate accurate short-term load forecasting (STLF) while
protecting custom privacy is essential for power system operation.

Load forecasting has been an essential part of the field of power system
research since the 1970s, starting with some of the earliest methods, such as
linear regression (LR) [4], stochastic time series (STS) [5] and general
exponential smoothing (GES) [6]. The mean accuracy of these methods is about
90%$\sim$96%. With the introduction of machine learning and artificial
intelligence methods, scholars started applying artificial intelligence
algorithms to STLF and achieved great success, which increased the accuracy to
more than 97% [7, 2, 8].

Due to the vigorous development of hardware equipment and the superb
performance of deep learning in terms of the characteristics of nonlinear
functions, deep learning algorithms have become particularly popular in load
prediction research in recent years. [9] proposed a novel model multi-scale
convolutional neural network with time-cognition (TCMS-CNN), which combines
sufficient and discriminative features to extract potential law in the dataset
providing an excellent result. [10] proposed a deep learning framework based
on a combination of a convolutional neural network (CNN) and long short-term
memory (LSTM), which could provide a significant improvement in the accuracy
of individual household load forecasting. [11] improved deep belief networks
(DBN) with Gauss-Bernoulli restricted Boltzmann machine (GB-RBM) and gray
relational analysis (GRA). The developed method has a better performance than
DBN and other traditional methods. However, the dataset required by the
aforementioned methods includes all customers’ data together without
considering individual customers’ privacy.

With the rapid development of big data, customer data privacy has gradually
received attention. Over the last few years, several governments also have
committed to data privacy protection, e.g., the European Commission’s General
Data Protection Regulation [12] and the Consumer Privacy Bill of Rights in the
US [13]. As a result, decentralised learning methods, such as federal
learning, have been proposed to solve these problems. [14] proposed a
federated learning method for load forecasting with smart meter data that is
capable of training a machine learning model in a distributed manner without
requiring the participant to share their local data. [15] proposed a fully
distributed STLF method based on Distributed Deep Belief Networks (DDBN),
which can solve the STLF model by local computing agents (CA) and update the
model parameters by communicating with connected neighbours. However, these
distributed learning algorithms still require a data centre to communicate
with individual nodes to update global parameters. In the case of huge data
volumes, the central-based distributed method could cause traffic jams and
scalability issues.

To this end, a novel fully distributed LSTM-based STLF method is proposed in
this paper. It can be trained with a local dataset and converge to global
optima by communicating with its connected neighbours, which does not need to
transfer any customers’ data. Different from the federated learning framework,
the proposed consensus-based framework does not have a single computing centre
to deal with an integrated global model among all local agents, and therefore
the proposed algorithm will not meet computing burdens. Apart from data
privacy and communication congestion, the distributed training framework can
also avoid the local over-fitting problem since its parameters will be amended
based on its neighbouring information.

The major contributions of this paper include:

  1. 1.

A novel distributed LSTM algorithm is proposed for STLF, which can train the
model at local and only communicate with its connected neighbours, preserving
data privacy.

  2. 2.

Due to the parallel computational framework, the proposed distributed method
can significantly reduce the training time.

  3. 3.

The over-fitting problem of the local model can be mitigated by mutual
correction during the communication and training process.

The rest of the paper is organised as follows. Mathematical preliminaries
applied in this paper are summarised in Sec. II. The proposed distributed LSTM
method is described in Sec. III. Simulation results with discussion are
presented in Sec. IV. Finally, we conclude the work in Sec. V.

##  II Applied Methodologies

In this section, we recall methods regarding graph theory, consensus control
and LSTM. Let $\mathbb{R}^{n\times m}$ be the set of $n\times m$ real matrices
and the superscript $T$ means the transpose of real matrices. $I_{N}$ denotes
the identity matrix of dimension $N$ and $\mathbf{1}_{N}$ represents a column
vector with all entries being 1. $\mathbb{R}^{++}$ denotes the positive real
numbers. $\|\cdot\|_{2}$ represents the 2-norm of the argument.

###  II-A Graph Theory

Following [16], an undirected graph $\mathcal{G=(V,E)}$ can be used to
describe the communication topology among the local computing centres, where
$\mathcal{V}=\\{\nu_{1},\cdots,\nu_{N}\\}$ is the vertex set and
$\mathcal{E}\in\mathcal{V}\times\mathcal{V}$ is the edge set. The adjacency
matrix $\mathcal{A}=[a_{ij}]\in\mathbb{R}^{N\times N}$ of $\mathcal{G(V,E)}$
is an $N\times N$ matrix, such that $a_{ij}=1$ if
$(\nu_{j},\nu_{i})\in\mathcal{E}$ and $a_{ij}=0$ otherwise. Define the degree
matrix $D=\text{diag}\\{\sum_{j=1}^{N}a_{1j},\sum_{j=1}^{N}a_{2j},$
$\cdots,\sum_{j=1}^{N}a_{Nj}\\}$. A graph is connected if and only if every
pair of vertices can be connected by a path, namely, a sequence of edges. In
this paper, we assume that the graph is connected and undirected. The
Laplacian matrix related to $\mathcal{G(V,E,A)}$ is defined as
$\mathcal{L=D-A}$, i.e.,

$\mathcal{L}=\left\\{\begin{array}[]{l}l_{ij}=-a_{ij},i\neq j\\\
l_{ii}=\sum_{i\neq j}a_{ij}.\end{array}\right.$ (1)

When $\mathcal{G(V,E)}$ is a connected undirected graph, 0 is an eigenvalue of
Laplacian $\mathcal{L}$ with the eigenvector $\mathbf{1}_{N}$ and all the
other eigenvalues are positive. Then,

$\mathcal{L}1_{N}=0_{N},\ \ 1_{N}^{T}\mathcal{L}=0_{N}^{T}.$ (2)

In this paper, we assume that the connection between computing centres are
undirected since the communication line has no direction.

###  II-B Long Short-Term Memory

The LSTM is a refined model of Recurrent Neural Network (RNN), and it has the
advantage of avoiding exploding and vanishing gradient problem [17]. The basic
structure of an LSTM network is shown in Fig. 1.

Figure 1: Structure of LSTM.

The propagate steps of LSTM can be separated into four sub-steps: $forget\
gate$ step, $input\ gate$ step, $update$ step and $output$ step. The $forget\
gate$ step is made by a sigmoid layer which is to decide what information we
are going to forget. The inputs of this step are historical output $h_{t-1}$
and current state $x_{t}$, and it outputs a number within a range $(0,1)$.

$f_{t}=\sigma(W_{f}\cdot[h_{t-1},x_{t}]+b_{f})$ (3)

where $W_{f},b_{f}$ are the weights and bias of the LSTM forget gate layer,
and $\sigma$ denotes the sigmoid activation function.

The second step is composed of a sigmoid layer and a tanh layer. This sigmoid
layer is to decide what new information we are going to memorise, and the tanh
layer is to estimate the current cell state $\hat{C_{t}}$:

$\displaystyle j_{t}=$ $\displaystyle\sigma(W_{j}\cdot[h_{t-1},x_{t}]+b_{j})$
(4) $\displaystyle\hat{C_{t}}=$
$\displaystyle\tanh(W_{C}\cdot[h_{t-1},x_{t}]+b_{C})$ (5)

The third step is to combine the outputs of second step and update the cell
state:

$C_{t}=f_{t}\times C_{t-1}+j_{t}\times\hat{C_{t}}$ (6)

Finally, the LSTM yields the output controlled by the $output$ step, which is
based on the current cell state $C_{t}$, history output $h_{t-1}$ and current
input $x_{t}$. It uses a tanh activate function as a filter to push the cell
state between $-1$ and $1$, so that the $output$ step can only output the
parts we choose.

$\displaystyle o_{t}=$ $\displaystyle\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{o})$
(7) $\displaystyle h_{t}=$ $\displaystyle o_{t}\times\tanh(C_{t})$ (8)

There is the final output $y$ if end of the hidden layers:

$y=\textit{softmax}(W_{v}\cdot h_{t}+b_{v})$ (9)

##  III Distributed Long Short-Term Memory

In this section, we formulate a Distributed Long Short-Term Memory (DLSTM)
method for STLF by consensus-based approaches. It consists of a group of $N$
local LSTM models distributed over a connected graph, where each local
computing centre has its own local dataset and cannot be revealed to other
computing centres. The objective of the LSTM models is to minimise the
empirical loss over the entire data set, which is formulated as

$\min_{\Theta\in\mathbb{R}^{n}}\mathbb{E}(\mathcal{D},\Theta)=\sum_{i=1}^{N}\mathbb{E}_{i}(\mathcal{D}_{i},\Theta_{i})$
(10)

where $\mathcal{D}_{i}$ and $\Theta_{i}\in\mathbb{R}^{n}$ are the sub-dataset
and weight parameters of the $i$th local LSTM model agent, respectively. $n$
is the dimension of the weight matrix. It is noticed that the weight
parameters $\Theta_{i}$ includes all the parameters in LSTM models, include
$forget\ gate$, $input\ gate$, $update$ and $output$ steps:

$\Theta_{i}=[W_{f},b_{f},W_{j},b_{j},W_{C},b_{C},W_{o},b_{o}]^{T}$ (11)

In the traditional LSTM algorithm, the training process is to calculate the
gradient and update all the weights $\Theta$:

$\Theta^{t}=\Theta^{t-1}-\eta_{i}\nabla_{\Theta}\mathbb{E}(\mathcal{D},\Theta^{t-1})$
(12)

For the DLSTM algorithm, the gradient descent method is applied to optimise
the parameters of local LSTM models, and the distributed consensus algorithm
is applied to optimise the parameters between connected LSTM models. All the
communication and training processes only transfer the weights of different
models, which does not require any customer data, and therefore the privacy
can be protected. The procedure code of DLSTM is shown in Algorithm 1.

Algorithm 1 Distributed Long Short Term Memory.

1:procedure DLSTM(LSTM, Agents)

2: $\textit{LSTM}\leftarrow\text{Initial weights $\Theta_{i}$ for local LSTM
models}$

3: $\textit{Sites}\leftarrow\text{Array of Network pipes to local CA${}_{i}$}$

4: while Sites contains unused data do

5: //Start local training for each local site

6: for each  CAi$\in\textit{Sites}$ do

7: CAi$\leftarrow\text{Forward Propagation}$

8: CAi$\leftarrow\text{Gradient Descent}$ Calculation

9: CAi$\leftarrow\text{Back Propagation}$ with (14)

10: end for

11: //Consensus the weight of local LSTM models

12: while weights are different do

13: for each CAi $\in\textit{Sites}$ do

14: Consensus weight with (14)

15: end for

16: end while

17: $\textit{DLSTM}\leftarrow\text{Consensus weight}$

18: end while

19:end procedure

During each training step of the local computing agent, we add an
communication update step for each local LSTM model. Thus, the gradient-based
training of DLSTM can be formulated as the following sub-steps:

$\displaystyle\Phi_{i}^{t}$
$\displaystyle=\Theta_{i}^{t-1}-\eta_{i}\nabla_{\Theta}\mathbb{E}_{i}(\mathcal{D}_{i},\Theta_{i}^{t-1})$
(13) $\displaystyle\Theta_{i}^{t}$
$\displaystyle=\sum_{j\in\mathbb{N}}a_{ij}\Phi_{i}^{t}$ (14)

where $\Phi_{i}^{t}$ is the intermediate variable and $\eta_{i}\in[0,1]$ is
the learning rate of the local model. $a_{ij}$ is the element of Laplace
matrix $\mathcal{L}$ in (1). The updates of weights $\Theta_{i}$ include two
procedures, which make up the learning before consensus (LBC) algorithm. In
the first stage, the neural networks are trained independently with local sub-
dataset through the gradient descent algorithm (13). This stage only uses
local information, such as the weights of latest step $\Theta_{i}^{t-1}$ and
the gradient of the empirical loss
$\nabla_{\Theta}\mathbb{E}_{i}(\mathcal{D}_{i},\Theta_{i}^{t-1})$. In the
second stage, the weights of the neural networks are updated via consensus
algorithm, where neighbouring information $\Phi_{i}^{t}$ are applied. With
this learning process, all the agents are able to obtain the single and
optimal neural network model when $t$ tends to $\infty$ [18]. Similarly, the
equations (14) and (13) can be swapped to yield the consensus before learning
(CBL) algorithm as

$\displaystyle\Phi_{i}^{t}$
$\displaystyle=\sum_{j\in\mathbb{N}}a_{ij}\Theta_{i}^{t-1}$ (15)
$\displaystyle\Theta_{i}^{t}$
$\displaystyle=\Phi_{i}^{t}-\eta_{i}\nabla_{\Theta}\mathbb{E}_{i}(\mathcal{D}_{i},\Phi_{i}^{t})$
(16)

Note that the CBL algorithm is to consensus the gradients first, then update
the local model. In contrast, the LBC algorithm is to update the local model
first, then consensus the models. Theoretically, the gradients will converge
to $0$, and the model will become the optimal model with the training time
goes infinity. Therefore, there is not much difference between CBL and LBC
algorithms.

Based on the proposed distributed manner, each local LSTM model only needs to
communicate with its neighbours. Thus, the STLF model can be trained locally
and the multi-agent framework can reduce the computational and communication
cost, which can be further extended to large networks [19]. Moreover, the
distributed framework is more robust to single-point failures as long as the
communication network remains connected [15].

##  IV Case Study

###  IV-A Experiment and Model Setup

The case study is based on the historical load data, daily average temperature
data and holiday type data for the period 2016-2019 which are provided by the
GEFCom 2017 competition [20] and ISO New England [21]. The system topology
under consideration is the connection of four agents (data centres) as shown
in Fig. 2, each of them has a quarter of dataset. In addition, the iteration
times of consensus between each information connection is set as 20, which
makes the algorithm converge faster.

Figure 2: Communication Topology.

For STLF, the main influencing factors are historical load, temperature, date
type and other variables. These three types of data are selected as input
variables for the forecasting model. The load has a characteristic of a
24-hour periodic change, so that the power load value at the same time before
the forecasting day and two days before can be used as the input variable of
the model. In addition, considering the effects of temperature on the
electrical load, daily mean temperature after preprocessing can be used as
input variables for the LSTM. The input and output information designed in
this paper are summarised in Table I.

Table I: Input and output information.  input | input variables
---|---
1-7 | the electricity consumption of last week
8-14 | the average temperature of last week
15-21 | the day type of last week
22 | the electricity consumption at 2 days ago
23 | the average temperature at 2 days ago
24 | the day type at 2 days ago
25 | the actual load at 24 hours before
26 | the average temperature at 24 hours before
27 | the day type at 24 hours before
28 | the temperature of the forecast day
29 | the day type of the forecast day
output |  the predicted load value at time $t$ of the forecast day

Further, all raw data are pre-normalised to improve the accuracy of the
forecasting model:

$x_{i}(k)=\dfrac{x_{i}(k)-x_{i}^{min}}{x_{i}^{max}-x_{i}^{min}}$ (17)

where $x_{i}^{min}$ and $x_{i}^{max}$ are the minimum and maximum value of the
$i$-th original input data.

###  IV-B Evaluation Criteria and Error Analysis

The mean square error (MSE) and mean absolute percentage error (MAPE) are
adopted as the final evaluating indicators, which are shown as follow:

$\varepsilon_{MAPE}=\dfrac{1}{N}\sum_{i=1}^{N}\left|\dfrac{y_{i}-\hat{y_{i}}}{y_{i}}\right|\times
100\%,$ (18)
$\varepsilon_{MSE}=\dfrac{\sum_{i=1}^{N}(y_{i}-\hat{y_{i}})^{2}}{\sum_{i=1}^{N}(y_{i})^{2}},$
(19)

where $y_{i}$ is the actual load at period $i$; $N$ is the total length of
forecasting periods; and $\hat{y_{i}}$ is the forecast load at time $i$.

###  IV-C Results and Analysis

####  IV-C1 Case 1

This case verified the validity of the proposed DLSTM model. We use whole year
customer electricity consumption data as the training dataset, and the test
data is a dataset of a random week in next year. The results are shown in Fig.
3.

Figure 3: Short-term Load Forecasting Results.

The green line is the forecasting results of the proposed DLSTM method, and it
is evident that the proposed DLSTM has less prediction errors at most of the
data points than centralised LSTM method.

####  IV-C2 Case 2

In this case, we investigate the training performance of the proposed DLSTM
model. Fig. 4 depicts the validation errors during the training process among
4 distributed local DLSTM agents and centralised LSTM agent. Although each
DLSTM agent only has a quarter of the dataset, it shows comparable training
performance as the centralised LSTM method.

Figure 4: Distributed Training Performance.

In Fig. 4, we may clearly see that the converge speed of local computing LSTM
models are slower than centralised LSTM model, but they can achieve similar
accuracy after sufficient training epochs. Apart from the accuracy, the
training speed of the proposed DLSTM is only 412 seconds, which is much faster
than centralised LSTM (583 second). This is because that the dataset size of
local training agent is only a quarter of centralised LSTM agent. Therefore,
the proposed DLSTM algorithm can reduce the training time with an increasing
number of computing agents, and the aim of fast training can be achieved by
adding more computing agents if the load dataset continues to increase in the
future.

####  IV-C3 Case 3

In this case, we summarise different STLF models to illustrate the advantage
of the proposed model, and four typical state-of-the-art algorithms are chosen
for comparison: Support Vector Regression (SVR), Artificial Neural Network
(ANN), Deep Belief Network (DBN) and Distributed Artificial Neural Network
(DANN). The compared models are simulated based on the same dataset in this
paper. All the models are trained with the same training dataset, validation
dataset and test dataset. The results of different models are summarised in
Table II.

Table II: Comparison of STLF Models in Case 3. | MAPE | MAE | MSE
---|---|---|---
LSTM | 0.0150 | 201.746 | 70021.160
DLSTM | 0.0138 | 183.352 | 62095.324
SVR | 0.0152 | 202.246 | 73495.060
ANN | 0.0280 | 324.650 | 131273.362
DBN | 0.0206 | 144.326 | 44299.052
DANN | 0.0275 | 284.240 | 123363.703

From the results provided in Table. II, the DLSTM algorithm meets the
requirements of STLF and has better accuracy than other models. Comparing with
the distinct machine learning algorithms, the LSTM and DLSTM reveals better
performance and less prediction errors due to the time-sequence characteristic
of STLF problems. For distributed algorithms, we can see that the DANN and
DLSTM show the similar accuracy as ANN and LSTM, respectively. It can be seen
from the Table. II and Fig. 4, the proposed DLSTM method demonstrates similar
and even better accuracy than traditional single centralise LSTM algorithm
after around 50 training epochs.

##  V Conclusion and Future Work

This paper ascertains the effectiveness of using the distributed long short-
term memory models in short-term load forecasting. The multiple and time-
variable variations can be predicted by the proposed approach. The
decentralised protocol makes it possible to separate data sets and pre-train
models at an individual data centre, which simplifies the forecasting problem
and leads to more privacy results. In the application examples, the proposed
distributed long short-term memory model provided accurate forecast results
and faster training speed than other machine learning models. The distributed
model is simpler to train and tune than the centralised models, does not over-
fit and reduces variance due to the consensus weight of many subsets. In
future, we plan to apply trustworthy AI techniques, e.g. [22, 23, 24], to
assure the reliability and security of our method, as well as its
explainability.

## References

  * [1] S. N. Fallah, M. Ganjkhani, S. Shamshirband, and K.-w. Chau, “Computational intelligence on short-term load forecasting: A methodological overview,” _Energies_ , vol. 12, no. 3, p. 393, 2019.
  * [2] H. S. Hippert, C. E. Pedreira, and R. C. Souza, “Neural networks for short-term load forecasting: A review and evaluation,” _IEEE Transactions on power systems_ , vol. 16, no. 1, pp. 44–55, 2001.
  * [3] J. W. Taylor and P. E. McSharry, “Short-term load forecasting methods: An evaluation based on european data,” _IEEE Transactions on Power Systems_ , vol. 22, no. 4, pp. 2213–2219, 2007.
  * [4] A. D. Papalexopoulos and T. C. Hesterberg, “A regression-based approach to short-term system load forecasting,” _IEEE Transactions on power systems_ , vol. 5, no. 4, pp. 1535–1547, 1990.
  * [5] M. T. Hagan and S. M. Behr, “The time series approach to short term load forecasting,” _IEEE transactions on power systems_ , vol. 2, no. 3, pp. 785–791, 1987.
  * [6] W. Christiaanse, “Short-term load forecasting using general exponential smoothing,” _IEEE Transactions on Power Apparatus and Systems_ , no. 2, pp. 900–911, 1971.
  * [7] C.-N. Lu, H.-T. Wu, and S. Vemuri, “Neural network based short term load forecasting,” _IEEE Transactions on Power Systems_ , vol. 8, no. 1, pp. 336–342, 1993.
  * [8] S. Kiartzis, A. Bakirtzis, and V. Petridis, “Short-term load forecasting using neural networks,” _Electric Power Systems Research_ , vol. 33, no. 1, pp. 1–6, 1995.
  * [9] Z. Deng, B. Wang, Y. Xu, T. Xu, C. Liu, and Z. Zhu, “Multi-scale convolutional neural network with time-cognition for multi-step short-term load forecasting,” _IEEE Access_ , vol. 7, pp. 88 058–88 071, 2019.
  * [10] M. Alhussein, K. Aurangzeb, and S. I. Haider, “Hybrid cnn-lstm model for short-term individual household load forecasting,” _IEEE Access_ , vol. 8, pp. 180 544–180 557, 2020.
  * [11] X. Kong, C. Li, F. Zheng, and C. Wang, “Improved deep belief network for short-term load forecasting considering demand-side management,” _IEEE Transactions on Power Systems_ , vol. 35, no. 2, pp. 1531–1538, 2019.
  * [12] P. Voigt and A. Von dem Bussche, “The eu general data protection regulation (gdpr),” _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ , vol. 10, no. 3152676, pp. 10–5555, 2017.
  * [13] Sen. Markey, Edward J. [D-MA], “S.1214 - privacy bill of rights act,” https://www.congress.gov/bill/116th-congress/senate-bill/1214#, April 2019\.
  * [14] M. N. Fekri, K. Grolinger, and S. Mir, “Distributed load forecasting using smart meter data: Federated learning with recurrent neural networks,” _International Journal of Electrical Power & Energy Systems_, vol. 137, p. 107669, 2022.
  * [15] Y. Dong, Z. Dong, T. Zhao, Z. Li, and Z. Ding, “Short term load forecasting with markovian switching distributed deep belief networks,” _International Journal of Electrical Power & Energy Systems_, vol. 130, p. 106942, 2021.
  * [16] N. Biggs, N. L. Biggs, and E. N. Biggs, _Algebraic Graph Theory_. Cambridge university press, 1993, vol. 67.
  * [17] G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,” _Artificial Intelligence Review_ , vol. 53, no. 8, pp. 5929–5955, 2020.
  * [18] Z. Li, B. Liu, Z. Dong, and Z. Ding, “Cooperative training over networks via consensus-based algorithms,” in _2021 40th Chinese Control Conference (CCC)_. IEEE, 2021, pp. 5309–5314.
  * [19] Y. Dong, T. Zhao, and Z. Ding, “Demand-side management using a distributed initialisation-free optimisation in a smart grid,” _IET Renewable Power Generation_ , vol. 13, no. 9, pp. 1533–1543, 2019.
  * [20] T. Hong and S. Fan, “Probabilistic electric load forecasting: A tutorial review,” _Int. J. Forecast._ , vol. 32, no. 3, pp. 914–938, 2016.
  * [21] K. B. Sahay and M. M. Tripathi, “Day ahead hourly load forecast of pjm electricity market and iso new england market by using artificial neural network,” in _ISGT 2014_. IEEE, 2014, pp. 1–5.
  * [22] W. Huang, Y. Sun, X. Zhao, J. Sharp, W. Ruan, J. Meng, and X. Huang, “Coverage-Guided Testing for Recurrent Neural Networks,” _IEEE Transactions on Reliability_ , pp. 1–16, 2021.
  * [23] X. Zhao, A. Banks, J. Sharp, V. Robu, D. Flynn, M. Fisher, and X. Huang, “A Safety Framework for Critical Systems Utilising Deep Neural Networks,” in _Computer Safety, Reliability, and Security_ , ser. LNCS, vol. 12234. Cham: Springer, 2020, pp. 244–259.
  * [24] X. Zhao, W. Huang, X. Huang, V. Robu, and D. Flynn, “BayLIME: Bayesian local interpretable model-agnostic explanations,” in _UAI’21_ , vol. 161\. PMLR, 2021, pp. 887–896.