# Balancing Cost and Benefit with Tied-Multi Transformers

Raj Dabre Raphael Rubino Atsushi Fujita
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan
<EMAIL_ADDRESS>
###### Abstract

We propose and evaluate a novel procedure for training multiple Transformers
with tied parameters which compresses multiple models into one enabling the
dynamic choice of the number of encoder and decoder layers during decoding. In
sequence-to-sequence modeling, typically, the output of the last layer of the
$N$-layer encoder is fed to the $M$-layer decoder, and the output of the last
decoder layer is used to compute loss. Instead, our method computes a single
loss consisting of $N\times M$ losses, where each loss is computed from the
output of one of the $M$ decoder layers connected to one of the $N$ encoder
layers. Such a model subsumes $N\times M$ models with different number of
encoder and decoder layers, and can be used for decoding with fewer than the
maximum number of encoder and decoder layers. We then propose a mechanism to
choose a priori the number of encoder and decoder layers for faster decoding,
and also explore recurrent stacking of layers and knowledge distillation for
model compression. We present a cost-benefit analysis of applying the proposed
approaches for neural machine translation and show that they reduce decoding
costs while preserving translation quality.

##  1 Introduction

Neural networks for sequence-to-sequence modeling typically consist of an
encoder and a decoder coupled via an attention mechanism. Whereas the very
first deep models used stacked recurrent neural networks (RNN) (Sutskever et
al., 2014; Cho et al., 2014; Bahdanau et al., 2015) in the encoder and
decoder, the recent Transformer model (Vaswani et al., 2017) constitutes the
current state-of-the-art approach, owing to its better context modeling via
multi-head self- and cross-attentions.

Given an encoder-decoder architecture and its hyper-parameters, such as the
number of encoder and decoder layers, vocabulary sizes (in the case of text
based models), and hidden layers, the parameters of the model, i.e., matrices
and biases for non-linear transformations, are optimized by iteratively
updating them so that the loss for the training data is minimized. The hyper-
parameters can also be tuned, for instance, through maximizing the automatic
evaluation score on the development data. However, in general, it is highly
unlikely (or impossible) that a single optimized model suffices diverse cost-
benefit demands at the same time. For instance, in practical low-latency
scenarios, one may accept some performance drop for speed. However, a model
used with a subset of optimized parameters might perform badly. A single
optimized model cannot also guarantee the best performance for each individual
input. Although this is practically important, it has drawn only a little
attention. An existing solution for this problem is to host multiple models
simultaneously for flexible choice. However, this approach is not very
practical, because it requires an unreasonably large quantity of resources.
Furthermore, there are no well-established methods for selecting appropriate
models for each individual input prior to decoding.

(a) Multiple tied-layer vanilla models.

(b) Collapsing tied layers into one.

Figure 1: The general concept of multi-layer softmaxing for training multi-
layer neural models with an example of a 4-layer model. Figure 1(a) is a
depiction of our idea in the form of multiple vanilla models whose layers are
tied together. Figure 1(b) shows the result of collapsing all tied layers into
a single layer. The red lines indicate the flow of gradients and hence the
lowest layer in the stack receives the largest number of updates.

As a more effective solution, we consider training a single model that
subsumes multiple models which can be used for decoding with different hyper-
parameter settings depending on the input or on the latency requirements. In
this paper, we focus on the number of layers as an important hyper-parameter
that impacts both speed and quality of decoding, and propose a _multi-layer
softmaxing_ method, which trains multi-layer neural models referring to the
output of all layers during training. Conceptually, as illustrated in Figure
1, this method involves tying (sharing) the parameters of multiple models with
different number of layers and is not specific to particular types of multi-
layer neural models. On top of the above method, we consider to exploit the
model further. For saving decoding time, we design and evaluate mechanisms to
choose appropriate number of layers depending on the input prior to decoding.
As for further model compression, we leverage other orthogonal types of
parameter tying approaches, such as those reviewed in Section 2.

Despite the generality of our proposed method, in this paper, we focus on
encoder-decoder models with $N$ encoder and $M$ decoder layers, and compress
$N\times M$ models111Rather than casting the encoder-decoder model into a
single column model with ($N+M$) layers. by updating the model with a total of
$N\times M$ losses computed by softmaxing the output of each of the $M$
decoder layers, where it attends to the output of each of the $N$ encoder
layers. The number of parameters of the resultant encoder-decoder model is
equivalent to that of the most complex subsumed model with $N$ encoder and $M$
decoder layers. Yet, we can now perform faster decoding using a fewer number
of layers, given that shallower layers are better trained.

To evaluate our proposed method, we take the case study of neural machine
translation (NMT) (Cho et al., 2014; Bahdanau et al., 2015), using the
Transformer model (Vaswani et al., 2017), and demonstrate that a single model
with $N$ encoder and $M$ decoder layers trained by our method can be used for
flexibly decoding with fewer than $N$ and $M$ layers without appreciable
quality loss. We evaluate our proposed method on WMT18 English-to-German
translation task, and give a cost-benefit analysis for translation quality vs.
decoding speed.

The rest of the paper is organized as follows. Section 2 briefly reviews
related work for compressing neural models. Section 3 covers our method that
ties multiple models by softmaxing all encoder-decoder layer combinations.
Section 4 describes our efforts towards designing and evaluating a mechanism
for dynamically selecting encoder-decoder layer combinations prior to
decoding. Section 5 describes two orthogonal extensions to our model aiming at
further model compression and speeding-up of decoding. The paper ends with
Section 6 containing conclusion and future work.

##  2 Related Work

There are studies that exploit multiple layers simultaneously. Wang et al.
(2018) fused hidden representations of multiple layers in order to improve the
translation quality. Belinkov et al. (2017) and Dou et al. (2018) attempted to
identify which layer can generate useful representations for different natural
language processing tasks. Unlike them, we make all layers of the encoder and
decoder usable for decoding with any encoder-decoder layer combination. In
practical scenarios, we can save significant amounts of time by choosing
shallower encoder and decoder layers for inference.

Our method ties the parameters of multiple models, which is orthogonal to the
work that ties parameters between layers (Dabre and Fujita, 2019) and/or
between the encoder and decoder within a single model (Xia et al., 2019; Dabre
and Fujita, 2019). Parameter tying leads to compact models, but they usually
suffer from drops in inference quality. In this paper, we counter such drops
with knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016; Freitag
et al., 2017). This approach utilizes smoothed data or smoothed training
signals instead of the actual training data. A model with a large number of
parameters and high performance provides smoothed distributions that are then
used as labels for training small models instead of one-hot vectors.

As one of the aims in this work is model size reduction, it is related to a
growing body of work that addresses the computational requirement reduction.
Pruning of pre-trained models (See et al., 2016) makes it possible to discard
around 80% of the smallest weights of a model without deterioration in
inference quality, given it is re-trained with appropriate hyper-parameters
after pruning. Currently, most deep learning implementations use 32-bit
floating point representations, but 16-bit floating point representations
(Gupta et al., 2015; Ott et al., 2018) or aggressive binarization (Courbariaux
et al., 2017) can be alternatives. Compact models are usually faster to
decode; studies on quantization (Lin et al., 2016) and average attention
networks (Xiong et al., 2018) address this topic.

To the best of our knowledge, none of the above work has attempted to combine
multi-model parameter tying, knowledge distillation, and dynamic layer
selection for obtaining and exploiting highly-compressed and flexible deep
neural models.

##  3 Multi-Layer Softmaxing

###  3.1 Proposed Method

Figure 1 gives a simple overview of the concept of multi-layer softmaxing,
taking a generic model as an example. The rightmost 4-layer model takes an
input and passes it through 4 layers222We make no assumptions about the nature
of the layers. before a softmax layer to predict the output. Typically, one
would apply softmax to the 4th layer only, compute loss, and then back-
propagate gradients in order to update parameters. Instead, we propose to
apply softmax to each layer, aggregate the computed losses, and then perform
back-propagation. This enables us to choose any layer combination during
decoding instead of only the topmost layer.

Extending this to a multi-layer encoder-decoder model is straightforward. In
encoder-decoder models, the encoder comprises an embedding layer for the input
(source language for NMT) and $N$ stacked transformation layers. The decoder
consists of an embedding layer and a softmax layer for generating the output
(target language for NMT) along with $M$ stacked transformation layers. Let
$X$ be the input to the $N$-layer encoder, $Y$ the anticipated output of the
$M$-layer decoder as well as the input to the decoder (for training), and
$\hat{Y}$ the predicted output by the decoder. Algorithm 1 shows the pseudo-
code for our proposed method. The line 3 represents the process done by the
$i$-th encoder layer, $L^{enc}_{i}$, and the line 5 does the same for the
$j$-th decoder layer, $L^{dec}_{j}$. In simple words, we compute a loss using
the output of each of the $M$ decoder layers which in turn is computed using
the output of each of the $N$ encoder layers. In line 8, the $N\times M$
losses are aggregated333We averaged multiple losses in our experiment, but
there are a number of options, such as weighted averaging. before back-
propagation. Henceforth, we will refer to this as the _Tied-Multi model_.

1 $\mathit{enc}_{0}=X$;

2 for  _$i$ in 1 to $N$_ do

3 $\mathit{enc}_{i}=L^{enc}_{i}(\mathit{enc}_{i-1})$;

4 for  _$j$ in 1 to $M$_ do

5 $\mathit{dec}_{j}=L^{dec}_{j}(\mathit{dec}_{j-1},\mathit{enc}_{i})$;

6 $\hat{Y}=\mathit{softmax}(\mathit{dec}_{j})$;

7 $\mathit{loss}_{i,j}=\mathit{cross\\_entropy}(\hat{Y},Y)$;

8

9

10$\mathit{overall\\_loss}=\mathit{aggregate}(\mathit{loss}_{1,1},\ldots,\mathit{loss}_{N,M})$;

11 Back-propagate using $\mathit{overall\\_loss}$;

Algorithm 1 Training a tied-multi model

For a comparison, the vanilla model is formulated as follows:
$\mathit{dec}_{j}=L^{dec}_{j}(\mathit{dec}_{j-1},\mathit{enc}_{N})$,
$\hat{Y}=\mathit{softmax}(\mathit{dec}_{M})$, and
$\mathit{overall\\_loss}=\mathit{cross\\_entropy}(\hat{Y},Y)$.

| BLEU score | Decoding time (sec)
---|---|---
| 36 vanilla models | Single tied-multi model
$n{\backslash}m$ | 1 | 2 | 3 | 4 | 5 | 6 | 1 | 2 | 3 | 4 | 5 | 6 | 1 | 2 | 3 | 4 | 5 | 6
1 | 26.3 | 30.3 | 31.9 | 32.2 | 32.4 | 32.9 | 23.2 | 28.6 | 30.5 | 30.8 | 31.2 | 31.5 | 91.3 | 112.4 | 150.9 | 180.2 | 218.2 | 254.2
2 | 28.6 | 32.5 | 33.1 | 33.3 | 33.5 | 33.2 | 26.5 | 31.5 | 33.0 | 33.6 | 33.8 | 34.0 | 92.8 | 112.7 | 148.5 | 178.7 | 224.5 | 255.5
3 | 29.2 | 32.6 | 33.6 | 34.4 | 34.3 | 34.1 | 27.8 | 32.5 | 33.9 | 34.6 | 34.7 | 34.7 | 92.3 | 113.4 | 151.1 | 188.7 | 240.6 | 259.4
4 | 29.8 | 33.6 | 34.3 | 34.7 | 34.4 | 34.5 | 28.3 | 33.0 | 34.3 | 34.8 | 34.9 | 34.9 | 92.4 | 114.4 | 151.5 | 193.7 | 231.9 | 262.4
5 | 30.7 | 33.9 | 34.6 | 35.5 | 34.4 | 35.0 | 28.6 | 33.1 | 34.5 | 34.8 | 35.0 | 35.1 | 94.4 | 113.4 | 161.5 | 194.7 | 241.5 | 261.5
6 | 30.8 | 34.0 | 34.4 | 35.7 | 35.0 | 35.0 | 28.7 | 33.1 | 34.6 | 34.7 | 34.9 | 35.0 | 93.2 | 114.9 | 158.3 | 203.3 | 246.3 | 266.9

Table 1: BLEU scores of 36 separately trained vanilla models and our single
tied-multi model used with $n$ ($1\leq n\leq N$) encoder and $m$ ($1\leq m\leq
M$) decoder layers. One set of decoding times is also shown given the fact
that vanilla and our tied-multi models have identical shapes when $n$ and $m$
for encoder and decoder layers are specified.

###  3.2 Experimental Setup

We evaluated the following two types of models on both translation quality and
decoding speed.

Vanilla models:

36 vanilla models with 1 to 6 encoder and 1 to 6 decoder layers, each trained
referring only to the last layer for computing loss.

Tied-Multi model:

A single tied-multi model with $N=6$ encoder and $M=6$ decoder layers, trained
by our multi-layer softmaxing.

We experimented with the WMT18 English-to-German (En$\rightarrow$De)
translation task. We used all the parallel corpora available for WMT18, except
ParaCrawl corpus,444http://www.statmt.org/wmt18/translation-task.html
We excluded ParaCrawl following the instruction on the WMT18 website: “BLEU
score dropped by 1.0” for this task. consisting of 5.58M sentence pairs, as
the training data and 2,998 sentences in newstest2018 as test data. The
English and German sentences were pre-processed using the tokenizer.perl and
truecase.perl scripts in Moses.555https://github.com/moses-smt/mosesdecoder
The true-case models for English and German were trained on 10M sentences
randomly extracted from the monolingual data made available for the WMT18
translation task, using the train-truecaser.perl script available in Moses.

Our multi-layer softmaxing method was implemented on top of an open-source
toolkit of the Transformer model (Vaswani et al., 2017) in the version 1.6
branch of tensor2tensor.666https://github.com/tensorflow/tensor2tensor For
training, we used the default model settings corresponding to
transformer_base_single_gpu in the implementation, except what follows. We
used a shared sub-word vocabulary of 32k determined using the internal sub-
word segmenter of tensor2tensor and trained the models for 300k iterations. We
trained the vanilla models on 1 GPU and our tied-multi model on 2 GPUs with
batch size halved to ensure that both models see the same amount of training
data. We averaged the last 10 checkpoints saved every after 1k updates,
decoded the test sentences, fixing a beam size777One can realize faster
decoding by narrowing down the beam width. This approach is orthogonal to ours
and in this paper we do not insist which is superior to the other. of 4 and
length penalty of 0.6, and post-processed the decoded results using the
detokenizer.perl and detruecase.perl scripts in Moses.

We evaluated our models using the BLEU metric (Papineni et al., 2002)
implemented in sacreBLEU (Post, 2018).888https://github.com/mjpost/sacreBLEU
signature: BLEU+case.mixed+lang.en-de+numrefs.1
+smooth.exp+test.wmt18+tok.13a+version.1.3.7 We also present the time (in
seconds) consumed to translate the test set, which includes times for the
model instantiation, loading the checkpoint, sub-word splitting and indexing,
decoding, and sub-word de-indexing and merging, whereas times for
detokenization are not taken into account.

Note that we did not use any development data for two reasons. First, we train
all models for the same number of iterations.999This enables a fair
comparison, since it ensures that each model sees roughly the same number of
training examples. Second, we use checkpoint averaging before decoding, which
does not require a development set unlike early stopping.

###  3.3 Results

Table 1 summarizes the BLEU scores and the decoding times of all the models,
exhibiting the cost-benefit property of our tied-multi model in comparison
with the results of the corresponding 36 vanilla models.

Even though the objective function for the tied-multi model is substantially
more complex than the one for the vanilla model, when performing decoding with
the 6 encoder and 6 decoder layers, it achieved a BLEU score of 35.0, which is
approaching to the best BLEU score of 35.7 given by the vanilla model with 6
encoder and 4 decoder layers. Note that when using a single encoder layer
and/or a single decoder layer, the vanilla models gave significantly higher
BLEU score than the tied-multi model. However, when the number of layers is
increased, there is no significant difference between the two types of models.

Regarding the cost-benefit property of our tied-multi model, two points must
be noted:

  * •

BLEU score and decoding time increase only slightly, when we use more encoder
layers.

  * •

The bulk of the decoding time is consumed by the decoder, since it works in an
auto-regressive manner. We can substantially cut down decoding time by using
fewer decoder layers which does lead to sub-optimal translation quality.

One may argue that training a single vanilla model with optimal number of
encoder and decoder layers is enough. However, as discussed in Section 1, it
is impossible to know a priori which combination is the best. More
importantly, a single vanilla model cannot suffice diverse cost-benefit
demands and cannot guarantee the best translation for any input (see Section
4.1). Recall that we aim at a flexible model and that all the results in Table
1 have been obtained using a single tied-multi model, albeit using different
number of encoder and decoder layers for decoding.

We conducted an analysis from the perspective of training times and model
sizes, in comparison with vanilla models.

#### Training Time:

Given that all our models were trained for the same number of iterations, we
compared the training times between vanilla and tied-multi models. As a
reference, we use the vanilla model with 6 encoder and 6 decoder layers. The
total training time for all the 36 vanilla models was 25.5 times101010We
measured the collapsed time for a fair comparison, assuming that all vanilla
models were trained on a single GPU one after another, even though one may be
able to use multiple GPUs to train the 36 vanilla models in parallel. that of
the reference model. In contrast, the training time for our tied-multi model
was about 9.5 times that of the same reference model. Unexpectedly, training a
tied-multi model was much more computationally efficient than independently
training all the 36 vanilla models with different number of layers.

#### Model Size:

The number of parameters of our tied-multi model is exactly the same as the
vanilla model with $N$ encoder and $M$ decoder layers. If we train a set of
vanilla models with different numbers of encoder and decoder layers, we end up
with significantly more parameters. For instance, in case of $N=M=6$ in our
experiment, we have 25.2 times more parameters: a total of 4,607M for the 36
vanilla models against 183M for our tied-multi model. In Section 5, we discuss
the possibility of further model compression.

##  4 Dynamic Layer Selection

To better understand the nature of our proposed method, we analyzed the
distribution of oracle translations within 36 translations generated by each
of the vanilla and our tied-multi models. Having confirmed that a single
encoder-decoder layer combination cannot guarantee the best translation for
any input, we tackled an advanced problem: designing a mechanism for
dynamically selecting one layer combination prior to decoding.111111This is
the crucial difference from two post-decoding processes: translation quality
estimation (Specia et al., 2010) and $n$-best-list re-ranking (Kumar and
Byrne, 2004).

###  4.1 Decoding Behavior of Tied-Multi Model

Let $(n,m)$ be an encoder-decoder layer combination of a given model with $n$
encoder and $m$ decoder layers. The oracle layer combination for an input
sentence was determined by measuring the quality of the translation derived
from each layer combination. We used a reference-based metric, chrF (Popović,
2016), since it has been particularity designed to sentence-level translation
evaluation and was shown to have relatively high correlation with human
judgment of translation quality at sentence level for the English–German pair
(Ma et al., 2018). In cases where multiple combinations have the highest
score, we chose the fastest one following the overall trend of decoding time
(Table 1). Formally, we considered a combination $(n_{1},m_{1})$ is faster
than another combination $(n_{2},m_{2})$ if the following holds.

$\begin{split}&(n_{1},m_{1})<(n_{2},m_{2})\\\ &\equiv
m_{1}<m_{2}\vee(m_{1}=m_{2}\wedge n_{1}<n_{2}).\end{split}$ (1)

(a) 36 vanilla models.

(b) Single tied-multi model.

Figure 2: Distribution of oracle translations determined by chrF scores
between reference and each of the hypotheses derived from the 36 combinations
of encoder and decoder layers (2,998 sentences).

Figure 2 presents the distribution of oracle layer combinations for the
vanilla and our tied-multi models. A comparison between the two distributions
revealed that the shallower layer combinations in our tied-multi model often
generates better translations than deeper ones unlike the vanilla models,
despite the lower corpus-level BLEU scores. This sharp bias towards shallower
layer combinations suggests the potential reduction of decoding time by
dynamically selecting the layer combination per input sentence prior to
decoding, ideally without performance drop.

###  4.2 Method

We formalize the encoder-decoder layer combination selection with a supervised
learning approach where the objective is to minimize the following loss
function (2).

$\operatorname*{arg\,min}_{\theta}\frac{1}{|S|}\sum_{s^{i}\in
S}\mathcal{L}(f(s^{i};\theta),t^{i}_{k}),$ (2)

where $s^{i}$ is the $i$-th input sentence ($1\leq i\leq|S|$), $t^{i}_{k}$ is
the translation for $s^{i}$ derived from the $k$-th layer combination ($1\leq
k\leq K=N\times M$) among all $K$ possible combinations, $f$ is the model with
parameters $\theta$, and $\mathcal{L}$ is a loss function. Assuming that the
independence of target labels (layer combinations) for a given input sentence
allows for ties, the model is able to predict multiple layer combinations for
the same input sentence.

The model $f$ with parameters $\theta$ implemented in our experiment is a
multi-head self-attention neural network inspired by Vaswani et al. (2017).
The number of layers and attention heads are optimized during a hyper-
parameter search, while the feed-forward layer dimensionality is fixed to
2,048. Input sequences of tokens are mapped to their corresponding embeddings,
initialized by the embedding table of the tied-multi NMT model. Similarly to
BERT (Devlin et al., 2019), a specific token is appended to each input
sequence before being fed to the classifier. This token is finally fed during
the forward pass to the output linear layer for sentence classification. The
output linear layer has $K$ dimensions, allowing to output as many logits as
there are layer combinations in the tied-multi NMT model. Finally, a sigmoid
function outputs probabilities for each layer combination among the $K$
possible combinations.

The parameters $\theta$ of the model are learned using mini-batch stochastic
gradient descent with Nesterov momentum (Sutskever et al., 2013) and the loss
function $\mathcal{L}$, implemented as a weighted binary cross-entropy (BCE)
function detailed in (3) and averaged over the batch.

$\begin{split}&\mathcal{L}_{\text{BCE}^{i}_{k}}\\\
&=-w^{i}_{k}\left[\delta_{k}y^{i}_{k}\cdot\log\hat{y}^{i}_{k}+(1-y^{i}_{k})\cdot\log(1-\hat{y}^{i}_{k})\right],\end{split}$
(3)

where $y^{i}_{k}$ is the reference class of the $i$-th input sentence $s^{i}$,
$\hat{y}^{i}_{k}$ is the output of the network after the sigmoid layer given
$s^{i}$ and $\delta_{k}=(1-p(t_{k}))^{\alpha}$ is the weight given to the
$k$-th class based on class distribution prior knowledge. During our
experiment, we have found that the classifier tends to favor recall in
detriment to precision. To tackle this issue, we introduce another loss using
an approximation of the macro $F_{\beta}$ implemented following (4).

$\mathcal{L}_{F_{\beta}}=1-\left[(1+\beta^{2})\cdot\frac{P\cdot
R}{(\beta^{2}\cdot P)+R}\right],$ (4)

where $P=\mu/\sum_{k}\hat{y}^{i}_{k}$, $R=\mu/\sum_{k}y^{i}_{k}$, and
$\mu=\sum_{k}(\hat{y}^{i}_{k}\cdot y^{i}_{k})$.

The final loss function is the linear interpolation of
$\mathcal{L}_{\text{BCE}}$ averaged over the $K$ classes and
$\mathcal{L}_{F_{\beta}}$ with parameter $\lambda$:
$\lambda\times\mathcal{L}_{\text{BCE}}+(1-\lambda)\times\mathcal{L}_{F_{\beta}}$.
We tune $\alpha$, $\beta$, and $\lambda$ during the classifier hyper-parameter
search based on the validation loss.

###  4.3 Experiments

The layer combination classifier was trained on a subset of the tied-multi NMT
model training data presented in Section 3.2 containing 5.00M sentences,
whereas the remaining sentences compose a validation and a test set containing
approximately 200k sentences each. The two latter subsets were used for hyper-
parameter search and evaluation of the classifier, respectively. To allow for
comparison and reproducibility, the final evaluation of the proposed approach
in terms of translation quality and decoding speed were conducted on the
official WMT development (newstest2017, 3,004 sentences) and test
(newstest2018, 2,998 sentences) sets; the latter is the one also used in
Section 3.2.

The training, development, and test sets were translated with each layer
combination of the tied-multi NMT model. Each source sentence was thus aligned
with 36 translations whose quality were measured by the chrF metric. Because
several combinations can lead to the best score, the obtained dataset was
labeled with multiple classes (36 layer combinations) and multiple labels
(ties with regard to the metric). During inference, the ties were broken by
selecting the layer combination with the highest value given by the sigmoid
function, or backing-off to the deepest layer combination (6, 6) if no output
value reaches 0.5. This tie breaking method differs from the oracle layer
selection presented in Equation (1) and in Figure 2 which prioritizes
shallowest layer combinations. In this experiment, decoding time was measured
by processing one sentence at a time instead of batch decoding, the former
being slower compared to the latter, but leads to precise results. The
decoding times were 954s and 2,773s when using (1,1) and (6,6) layer
combinations, respectively. By selecting the fastest encoder-decoder layer
combinations according to an oracle, the decoding times went down to 1,918s
and 1,812s for the individual and tied-multi models, respectively. However,
our objective is to be faster than default setting, that is, where one would
choose (6,6) combination.

Classifier | Fine-tuning | Speed (s) | BLEU
---|---|---|---
Baseline (tied (6,6)) |  | 2,773 | 35.0
Oracle (tied) |  | 1,812 | 42.1
(#1) 8 layers, 8 heads | ✓ | 2,736 | 35.0
(#2) 2 layers, 4 heads | ✓ | 2,686 | 34.8
(#3) 2 layers, 4 heads |  | 2,645 | 34.7
(#4) 4 layers, 2 heads |  | 2,563 | 34.3

Table 2: Dynamic layer combination selection results in decoding speed (in
seconds, batch size of 1) and BLEU, including the baseline and oracle for the
WMT newstest2018 using the tied-multi model architecture.

Several classifiers were trained and evaluated on the WMT test set, with or
without fine-tuning on the WMT development set. Table 2 presents the results
in terms of corpus-level BLEU and decoding speed. Some classifiers maintain
the translation quality (top rows), whereas others show quality degradation
but further gain in decoding speed (bottom rows). The classification results
show that gains in decoding speed are possible with an a-priori decision for
which encoder-decoder combination to select, based on the information
contained in the source sentence only. However, no BLEU gain has so far been
observed, demonstrating a trade-off between decoding speed and translation
quality. Our best configuration for decoding speed (#4) reduced 210s but leads
to a 0.7 point BLEU degradation. On the other hand, when preserving the
translation quality compared to the baseline configuration (#1) we saved only
37s. The oracle layer combination can achieve substantial gains both in terms
of BLEU (7.1 points) and decoding speed (961s). These oracle results motivate
possible future work in layer combination prediction for the tied-multi NMT
model.

##  5 Further Model Compression

We examined the combination of our multi-layer softmaxing approach with
another parameter-tying method in neural networks, called _recurrent stacking_
(RS) (Dabre and Fujita, 2019), complemented by _sequence-level knowledge
distillation_ (Kim and Rush, 2016), a specific type of knowledge distillation
(Hinton et al., 2015). We demonstrate that these existing techniques help
reduce the number of parameters in our model even further.

|  | Tied-multi model | Tied-multi RS model
---|---|---|---
| $n{\backslash}m$ | 1 | 2 | 3 | 4 | 5 | 6 | 1 | 2 | 3 | 4 | 5 | 6
without distillation | 1 | 23.2 | 28.6 | 30.5 | 30.8 | 31.2 | 31.5 | 25.7 | 29.8 | 30.6 | 30.8 | 30.7 | 30.9
2 | 26.5 | 31.5 | 33.0 | 33.6 | 33.8 | 34.0 | 28.5 | 32.3 | 32.9 | 33.0 | 33.1 | 33.2
3 | 27.8 | 32.5 | 33.9 | 34.6 | 34.7 | 34.7 | 29.2 | 32.9 | 33.6 | 33.8 | 33.6 | 33.5
4 | 28.3 | 33.0 | 34.3 | 34.8 | 34.9 | 34.9 | 29.3 | 33.2 | 33.7 | 33.9 | 33.6 | 33.7
5 | 28.6 | 33.1 | 34.5 | 34.8 | 35.0 | 35.1 | 29.4 | 33.2 | 33.7 | 33.9 | 33.9 | 34.0
6 | 28.7 | 33.1 | 34.6 | 34.7 | 34.9 | 35.0 | 29.2 | 33.2 | 33.7 | 33.9 | 34.0 | 33.8
with distillation | 1 | 30.1 | 34.0 | 35.1 | 35.3 | 35.6 | 35.7 | 31.2 | 33.5 | 34.1 | 34.2 | 34.3 | 34.3
2 | 33.4 | 35.8 | 36.6 | 36.8 | 37.1 | 37.3 | 33.7 | 35.5 | 35.7 | 35.7 | 35.8 | 35.8
3 | 34.7 | 36.5 | 37.0 | 37.4 | 37.4 | 37.5 | 34.1 | 35.8 | 36.1 | 36.1 | 36.2 | 36.2
4 | 35.2 | 36.8 | 37.2 | 37.4 | 37.5 | 37.5 | 34.3 | 36.0 | 36.2 | 36.2 | 36.3 | 36.3
5 | 35.5 | 36.9 | 37.1 | 37.4 | 37.5 | 37.6 | 34.5 | 36.1 | 36.2 | 36.3 | 36.3 | 36.3
6 | 35.5 | 37.0 | 37.2 | 37.5 | 37.6 | 37.6 | 34.6 | 36.1 | 36.2 | 36.2 | 36.3 | 36.2

Table 3: BLEU scores of a total of four NMT models: the tied-multi model with
(left block) and without (center and right blocks) RS layers, each trained
with (top block) and without (bottom block) sequence distillation.

###  5.1 Distillation into a Recurrently Stacked Model

In Section 2, we have discussed several model compression methods orthogonal
to multi-layer softmaxing. Having already compressed $N\times M$ models with
our approach, we consider further compressing it using RS. However, models
that use RS layers tend to suffer from performance drops due to the large
reduction in the number of parameters. As a way of compensating the
performance drop, we apply sequence-level knowledge distillation.

First, we decode all source sentences in the training data to obtain a pseudo-
parallel corpus containing distillation target sequences. By forward-
translating the data, we create soft-targets for the child model which makes
learning easier and hence is able to mimic the behavior of the parent model.
Then, an RS child model is trained with multi-layer softmaxing on the
generated pseudo-parallel corpus. Among a variety of distillation techniques,
we chose the simplest one to show the impact that distillation can have in our
setting, leaving an extensive exploration of more complex methods for the
future.

###  5.2 Experiment

We conducted an experiment to show that RS and sequence distillation can lead
to an extremely compressed tied-multi model which no longer suffers from
performance drops. We compared the following four variations of our tied-multi
model trained with multi-layer softmaxing.

Tied-multi model:

A model that does not share the parameters across layers, trained on the
original parallel corpus.

Distilled tied-multi model:

The same model as above but trained on the pseudo-parallel corpus.

Tied-multi RS model:

A tied-multi model that uses RS layers, trained with the original parallel
corpus.

Distilled tied-multi RS model:

The same model as above but trained on the pseudo-parallel corpus.

Note that we incurred much higher cost for training the distilled models than
training models directly on the original parallel corpus. First, we trained 5
vanilla models with 6 encoder and 6 decoder layers, because the performance of
distilled models is affected by the quality of parent models, and NMT models
vary vastly in performance (around 2.0 BLEU) depending on parameter
initialization. We then decode the entire training data (5.58M sentences) with
the one121212Ensemble of multiple models (Freitag et al., 2017) is commonly
used for distillation, but we used a single model to save decoding time. with
the highest BLEU score on the newstest2017 (used in Section 4.3) in order to
generate pseudo-parallel corpus for sequence distillation. Nevertheless, we
consider that we can fairly compare the performance of the above four models,
since they were trained only once with a random parameter initialization,
without seeing the test set.

Table 3 gives the BLEU scores for all models. Comparing top-left and top-right
blocks of the table, we can see that the BLEU scores for RS models are higher
than their non-RS counterparts when using fewer than 3 decoder layers. This
shows the benefit of RS layers despite the large parameter reduction. However,
the reduction in parameters negatively affects (up to 1.3 BLEU points) when
decoding with more decoder layers, confirming the limitation of RS as
expected.

Comparing the scores of the top and bottom halves of the table, we can see
that distillation dramatically boosts the performance of the shallower encoder
and decoder layers. For instance, without distillation, the tied-multi model
gave a BLEU of 23.2 when decoding with 1 encoder and 1 decoder layers, but the
same layer combination reaches 30.1 BLEU through distillation. Given that RS
further improves performance using lower layers, the BLEU score increases to
31.2. As such, distillation enables decoding using fewer layers without
substantial drops in performance. Furthermore, the BLEU scores did not vary
significantly when the layers deeper than 3 were used, meaning that we might
as well train shallower models using distillation. The performance of our
final model, i.e., the distilled tied-multi RS model (bottom-right), was
significantly lower (up to 1.5 BLEU points) similarly to its non-distilled
counterpart. However, given that it outperforms our original tied-multi model
(top-left) in all the encoder-decoder layer combinations, we conclude that we
can obtain a substantially compressed model with better performance.

We now analyze model size and decoding speed resulted by applying RS and
knowledge distillation. Note that RS has no effect on training time because
the computational complexity is the same.

#### Model Size:

Table 4 gives the sizes of various models that we have trained and their ratio
with respect to the tied-multi model. Training vanilla and RS models with 36
different encoder-decoder layer combinations required 25.2 and 14.3 times the
number of parameters of a single tied-multi model, respectively. Although RS
led to some parameter reduction, combining RS with our tied-multi model
resulted a further compressed single model: 0.40 times that of the single
tied-multi model without RS. This model has 63.2 times and 36.0 times fewer
parameters than all the individual vanilla and RS models, respectively. Given
that knowledge distillation can reduce the performance drops due to RS (see
Table 3), we believe that combining it with this approach is an effective way
to compress a large number of models into one.

Model(s) | Parameters | Relative size
---|---|---
36 vanilla models | 4,608M | 25.16
Single tied-multi model | 183M | 1.00
36 RS models | 2,623M | 14.33
Single tied-multi RS model | 73M | 0.40

Table 4: Model sizes for different encoder-decoder layer combinations. The
relative size is calculated regarding the tied-multi model as a standard.
Similarly to “36 vanilla models,” “36 RS models” represents the total number
of parameters of all RS models.

#### Decoding Speed:

Although we do not give scores, we observed that greedy decoding is faster
than beam decoding but suffers from significantly reduced scores of around 2.0
BLEU. By using our distilled models, however, greedy decoding reduced the
scores only by 0.5 BLEU compared to beam decoding. This happens because we
have used translations generated by beam decoding as target sentences for
knowledge distillation, which has the ability to loosely distill beam search
behavior into greedy decoding behavior Kim and Rush (2016). For instance,
greedy decoding with the distilled tied-multi RS model with 2 encoder and 2
decoder layers resulted in a BLEU score of 35.0 in 66.1s. In comparison, beam
decoding with the tied-multi model without RS and distillation with 5 encoder
and 6 decoder layers led a BLEU score of 35.1 in 261.5s (Table 1), showing
that comparable translation quality is obtained with a factor of 4 in decoding
time when using RS and distillation. Even though we intended to minimize
performance drops besides the model compression, we obtained an unexpected
benefit in terms of faster decoding through greedy search without a
significant loss in translation quality.

##  6 Conclusion

In this paper, we have proposed a novel procedure for training encoder-decoder
models, where the softmax function is applied to the output of each of the $M$
decoder layers derived using the output of each of the $N$ encoder layers.
This compresses $N\times M$ models into a single model that can be used for
decoding with a variable number of encoder ($1\leq n\leq N$) and decoder
($1\leq m\leq M$) layers. This model can be used in different latency
scenarios and hence is highly versatile. We have made a cost-benefit analysis
of our method, taking NMT as a case study of encoder-decoder models. We have
proposed and evaluated two orthogonal extensions and show that we can (a)
dynamically choose layer combinations for slightly faster decoding and (b)
further compress models using recurrent stacking with knowledge distillation.

For further speed up in decoding as well as model compression, we plan to
combine our approach with other techniques, such as those mentioned in Section
2. Although we have only tested our idea for NMT, it should be applicable to
other tasks based on deep neural networks.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _Proceedings of the 3rd International Conference on Learning Representations_ , San Diego, USA.
  * Belinkov et al. (2017) Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017.  Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1–10, Taipei, Taiwan.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , pages 1724–1734, Doha, Qatar.
  * Courbariaux et al. (2017) Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2017.  Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1.  _CoRR_ , abs/1602.02830.
  * Dabre and Fujita (2019) Raj Dabre and Atsushi Fujita. 2019.  Recurrent stacking of layers for compact neural machine translation models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 6292–6299, Honolulu, USA.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional Transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, USA.
  * Dou et al. (2018) Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018.  Exploiting deep representations for neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4253–4262, Brussels, Belgium.
  * Freitag et al. (2017) Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. 2017.  Ensemble distillation for neural machine translation.  _CoRR_ , abs/1702.01802.
  * Gupta et al. (2015) Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015\.  Deep learning with limited numerical precision.  In _Proceedings of the 32nd International Conference on Machine Learning_ , pages 1737–1746, Lille, France.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  _CoRR_ , abs/1503.02531.
  * Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016.  Sequence-level knowledge distillation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1317–1327, Austin, USA.
  * Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004.  Minimum Bayes-risk decoding for statistical machine translation.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 169–176, Boston, USA.
  * Lin et al. (2016) Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. 2016.  Fixed point quantization of deep convolutional networks.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning_ , pages 2849–2858, New York, USA.
  * Ma et al. (2018) Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018.  Results of the WMT18 metrics shared task.  In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_ , pages 682–701, Brussels, Belgium.
  * Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018.  Scaling neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 1–9, Brussels, Belgium.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , pages 311–318, Philadelphia, USA.
  * Popović (2016) Maja Popović. 2016.  chrF deconstructed: $\beta$ parameters and $n$-gram weights.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 499–504, Berlin, Germany.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium.
  * See et al. (2016) Abigail See, Minh-Thang Luong, and Christopher D. Manning. 2016.  Compression of neural machine translation models via pruning.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 291–301, Berlin, Germany.
  * Specia et al. (2010) Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010.  Machine translation evaluation versus quality estimation.  _Machine Translation_ , 24(1):39–50.
  * Sutskever et al. (2013) Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013.  On the importance of initialization and momentum in deep learning.  In _Proceedings of the International Conference on Machine Learning_ , pages 1139–1147, Atlanta, USA.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Proceedings of the 27th Neural Information Processing Systems Conference_ , pages 3104–3112, Montréal, Canada.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 30th Neural Information Processing Systems Conference_ , pages 5998–6008, Long Beach, USA.
  * Wang et al. (2018) Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. 2018.  Multi-layer representation fusion for neural machine translation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3015–3026, Santa Fe, USA.
  * Xia et al. (2019) Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. 2019.  Tied Transformers: Neural machine translation with shared encoder and decoder.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 5466–5473, Honolulu, USA.
  * Xiong et al. (2018) Deyi Xiong, Biao Zhang, and Jinsong Su. 2018.  Accelerating neural Transformer via an average attention network.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Long Papers_ , pages 1789–1798, Melbourne, Australia.


# End-to-End Learning with Multiple Modalities for System-Optimised Renewables
Nowcasting

Rushil Vohra, Ali Rajaei, and Jochen L. Cremer  0000-0002-1944-546X
0000-0001-9284-5083 _Department of Electrical Sustainable Energy_
_Delft University of Technology_
Delft, The Netherlands
<EMAIL_ADDRESS>{A.Rajaei<EMAIL_ADDRESS>
###### Abstract

With the increasing penetration of renewable power sources such as wind and
solar, accurate short-term, nowcasting renewable power prediction is becoming
increasingly important. This paper investigates the multi-modal (MM) learning
and end-to-end (E2E) learning for nowcasting renewable power as an
intermediate to energy management systems. MM combines features from all-sky
imagery and meteorological sensor data as two modalities to predict renewable
power generation that otherwise could not be combined effectively. The
combined, predicted values are then input to a differentiable optimal power
flow (OPF) formulation simulating the energy management. For the first time,
MM is combined with E2E training of the model that minimises the expected
total system cost. The case study tests the proposed methodology on the real
sky and meteorological data from the Netherlands. In our study, the proposed
MM-E2E model reduced system cost by $30$% compared to uni-modal baselines.

###### Index Terms:

multi-modal learning, end-to-end learning, optimal power flow, power
forecasting.

##  I Introduction

In many regions, sustainable energy systems will rely heavily on photovoltaic
(PV) power, wind power, or both. These forms of energy generation are
inherently uncontrollable due to their dependence on the local weather.
Therefore, predicting energy generation is essential to ensure a balanced
electrical system. A shared goal of system operators and microgrid managers is
maintaining grid stability cost-effectively, where forecasting power can be
considered an intermediate step to managing the energy system.

Conventional numerical weather prediction (NWP) models and statistical methods
are for instance autoregressive moving average (ARMA) and autoregressive
integrated moving average (ARIMA). However, these NWP and statistical models
sometimes result in inaccurate forecasts, especially in the short term.
Applying machine learning (ML) methods for these predictions in recent years
aimed at lowering these inaccuracies. For example, models based on artificial
neural networks (ANNs) improved prediction accuracy using meteorological time
series input data based on input features such as humidity and air pressure
for forecasting outputs such as solar irradiance and wind speed [1, 2].
Authors in [3] developed a hybrid PV power short-term prediction method that
combines an ARIMA model with an ANN model and considers the different seasons
as model inputs. Reference [4] proposed an approach that considers
meteorological inputs and produces ensemble predictions. Beyond meteorological
data, [5] uses another type of ANN, a convolutional neural network (CNN), and
a long short and term memory (LSTM) module that takes sky images as input to
produce the expected PV power as output. Despite this impressive development
improving the accuracy over the years through machine learning, ML-based
approaches for PV and wind predictions do not consider heterogeneous input
data simultaneously in their predictions. As these data sources,
meteorological and sky images, contain different information for predicting PV
and wind output, their combination is promising.

Multi-modal (MM) learning combines time series data with imagery and improves
the prediction of severe short-term fluctuations that other ANN approaches may
miss [6]. MM learning combines different modalities of heterogenous data
streams (such as text and audio) so that features from either modality enhance
each other [7]. Combining these modalities means learning a common function
for all the modalities simultaneously to produce a single representation
vector. This ‘function’ can be an ANN. Three different ’fusion’ methods to
combine these modalities are early, late, and intermediate fusion[8]. Early
fusion combines raw or pre-processed data from different modalities, e.g., by
concatenation, before the input to an ML model. Late fusion combines the
predictions made from either modality, e.g., using a weighted average.
Finally, intermediate fusion implements a hidden layer, called as MM layer, in
the ANN. The engineer designs the ANN model architecture and selects this
layer’s position in the ANN. Recently, [6] showed that MM learning
successfully forecasted PV power outperforming baselines by a large margin.
Beyond power prediction, MM also improved energy load time series forecasting
[9].

End-to-end (E2E) learning models the task of the user in the training of the
ML model [10]. This learning differs from conventional learning of ML models
that uses a prepared (labelled) data set and only assumes a loss on the
training data. This assumption has a weakness of not considering the task for
what the trained ML model has used afterwards, which is what E2E learning
addresses. For example, an ML model trained for predicting PV or wind power in
power systems could have the final task of being used as input optimising an
energy management schedule of resource allocations [11]. In more detail, from
the perspective of a power system operator, the task can be modelled as a
mathematical optimisation problem with the objective to minimise the total
system operational cost and constraints that ensure the grid’s physical
security limits. Donti et al. [12] developed an E2E learning approach for
convex optimisations such as the optimal power flow (OPF) problem. E2E
learning considers then the optimisation, here an OPF, that can be modelled in
an ANN as a differentiable optimisation layer making use of the implicit
function theorem [13]. The ANN is then trained on the system’s true objective,
the economic cost [12].

This paper combines MM with E2E for the first time and applies our MM-E2E
technique to a MM PV and wind forecasting using an ANN that is trained E2E
based on the system’s cost arising from inaccurate PV and wind forecasts. Our
proposed MM ANN combines features with intermediate fusion from imagery and
time-series meteorological data to predict PV and wind generation.
Respectively, the predicted PV and wind generation are input to a
differentiable optimisation layer that determines the schedule to dispatch
power generations cost-efficiently and minimises their re-dispatch cost. The
two contributions are:

  1. (i)

MM-E2E combines the intermediate fusion of MM and E2E learning of a model for
PV and wind generation prediction in energy systems, leveraging their
advantages: MM maximises learnable information from heterogeneous data, and
E2E learns models for accurate predictions.

  2. (ii)

E2E learning for an energy system with PV and wind generation. The dispatching
and redispatching optimisation problem captures the ultimate system economic
cost caused by prediction inaccuracies. The ANN model learns to compensate for
the inaccuracies in the most cost-effective way.

The rest of this paper is structured as follows. Section II outlines the
design and formulation of the proposed MM-E2E model. The case study in Section
III presents the numerical results of applying the proposed framework on the
IEEE $6$-bus system, followed by a discussion. The paper is concluded in
Section IV.

##  II Methodology

This section describes the proposed MM-E2E learning approach for solar and
wind prediction along the workflow of three steps: pre-processing, MM learning
and E2E with optimal power flow as task optimisation. After pre-processing the
heterogeneous data sources, i.e., imagery and meteorological sensor data, the
proposed approach combines these data features with MM. Subsequently, E2E
learning trains the ANN-based model on the ultimate economic system cost
arising from prediction errors.

###  II-A Data Pre-processing

To reduce the computational times of image processing, the imagery dataset is
pre-processed in two ways. Firstly, the images are converted from colour scale
to grayscale, which may also benefit object recognition [14]. Secondly, the
resolution of the imagery is reduced from $1536\times 1536$ to $64\times 64$.
Third, as ANNs training converges faster with input feature values between $0$
and $1$, the image pixel values are min-max normalised [15]. Hence the
resulting images $i\in\Omega^{N}$ have the pixel features
$x_{i}^{Image}\in(0,1)^{64\times 64}$. The tensor $X$ combines many of such
images, e.g., $X^{Image}\in(0,1)^{64\times 64\times N}$ where $N$ is the
cardinality $|\Omega^{N}|=N$ of all (training) images $\Omega^{N}$. Also, min-
max normalises the meteorological data resulting in
$x_{i}^{Meteo}\in(0,1)^{F\times N}$ with the number of meteorological features
$F$. Finally, imagery and meteorological data samples were measured every $10$
minute to nowcast renewable power from $10$-minutes-recent data.

###  II-B Multi-Modal Learning: Intermediate Fusion of Features

Convolutional neural network (CNN) $f_{1}$ can predict PV or wind power
$\hat{y}_{i}\in\mathbb{R}_{\geq 0}^{2}$ exclusively from imagery
$x_{i}^{Image}$ and feedforward NN (FNN) $f_{2}$ is suitable for exclusively
using numerical (meteorological) data $x_{i}^{Meteo}$, respectively

$\displaystyle f_{1}:x_{i}^{Image}\rightarrow\hat{y}_{i}$ (1a) $\displaystyle
f_{2}:x_{i}^{Meteo}\rightarrow\hat{y}_{i}.$ (1b)

A typical CNN architecture contains stacks of convolutional layers with
activation functions, followed by a pooling layer and, optionally, a batch
normalisation layer [15]. Fully-connected (FC) layers with a specified
activation function are stacked ’on top’ to make the final predictions
$\hat{y}_{i}$.

To predict power combined from the two modalities of data, feature information
from either one must be considered in conjunction. MM-E2E via intermediate
fusion conjuncts these pieces of information: features $h_{1,i}$ are extracted
from the imagery using a CNN

$\tilde{f}_{1}:x_{i}^{Image}\rightarrow h_{1,i}$ (2)

and concatenated with features from the meteorological data

$h_{mm,i}={h_{1,i}}^{\frown}x_{i}^{Meteo}.$ (3)

This intermediate fusion of features was selected as the early fusion of input
data directly ${x_{i}^{Image}}^{\frown}x_{i}^{Meteo}$ is not suited. Fusing
early imagery and sensor data as modalities without any feature extraction is
not well suited as the sensor data cannot be directly concatenated with the
pixel values of an image, and different types of ANNs are suited for the
different data types. Another suggested approach for two modalities is
bilinear pooling [16] with the outer product

$h_{mm}=\begin{bmatrix}h_{1}\\\ 1\end{bmatrix}\otimes\begin{bmatrix}h_{2}\\\
1\end{bmatrix}=\begin{bmatrix}h_{1}&h_{1}\otimes h_{2}\\\
1&h_{2}\end{bmatrix},$ (4)

where the index $i$ was dropped for simplicity reasons for the two feature
tensors $h_{1}$ and $h_{2}$ that give the MM feature tensor $h_{mm}$, where
$h_{1,i}$ is the extracted feature tensor for the first modality, and
$h_{2,i}$ is the extracted feature tensor for the second. Subsequently,
another ANN

$f_{3}:h_{mm,i}\rightarrow\hat{y}_{i},$ (5)

learns predicting the PV and wind power levels
$\hat{y}_{i}=(\hat{P}^{PV}_{i},\hat{P}^{W}_{i})$ from the concatenated,
extracted features $h_{mm}$.

A possible disadvantage of intermediate fusion is that the approach may result
in a very large and imbalanced feature tensor. However, it has also been
argued that these imbalances in the tensor accelerate the convergence of
training the ANN. The case study, Section III-C compares these two approaches.

###  II-C Optimal Power Flow

After the ANN $f_{3}$ predicts renewable power generation $\hat{y}_{i}$, two
varying OPF optimisation problems are solved in a sequence minimising the
system cost and ensuring the physical constraints of the grid. The first
problem referred to as OPF-Schedule, receives the predicted renewable
generation $\hat{y}_{i}$ from the ANN $f_{3}$ as a parameter and aims to
minimise system cost by scheduling the dispatch of generators. However, during
the real-time operation, the predicted renewable generation may be different
from the true value, which may lead to energy imbalances in the system.
Therefore, the second optimisation problem, referred to as OPF-Redispatch, is
formulated to find the optimal redispatch of the generators that can balance
supply and demand with the lowest redispatch cost.

The transmission grid consists of a set of nodes $B$, and a set of lines $L$.
Without loss of generality, it is considered that each node $b\in B$ could be
connected to a conventional generator, load demand, PV and wind generation.
Therefore, ANN $f_{3}$ outputs the PV and wind power predictions at multiple
locations, and we expand
$\hat{y}_{i,b}=(\hat{P}^{PV}_{i,b},\hat{P}^{W}_{i,b})$ by the index $b$ for
the bus locations $\forall b\in B$. Then, we drop the index $i$ in the
following for simplicity reasons and continue only with $\hat{P}^{PV}_{b}$
($\hat{P}^{W}_{b}$). The OPF-Schedule assumes the DC power flow equations with
known load demand as follows

$\displaystyle\min$ $\displaystyle C^{sch}=\sum_{b\in
B}c_{b}^{g}P_{b}^{g}+\sum_{b\in B}\big{(}\gamma^{PV}_{b}\lambda^{PV}_{b}$ (6)
$\displaystyle+\gamma^{W}_{b}\lambda^{W}_{b}+\gamma_{b}(\lambda^{+}_{b}+\lambda^{-}_{b})\big{)}$
s.t. $\displaystyle
P^{g}_{b}-P^{d}_{b}+(\hat{P}^{PV}_{b}-\lambda^{PV}_{b})+(\hat{P}^{W}_{b}-\lambda^{W}_{b})$
(7) $\displaystyle\quad=\sum_{l\in
L_{b}}f_{l}+(\lambda^{+}_{b}-\lambda^{-}_{b})\quad\quad\forall b\in B$
$\displaystyle\underline{P^{g}_{b}}\leq
P_{b}^{g}\leq\overline{P^{g}_{b}}\quad\quad\forall b\in B$ (8) $\displaystyle
f_{l}=\frac{\delta_{i}-\delta_{j}}{x_{l}}\quad\quad\forall l\in L$ (9)
$\displaystyle-\overline{f_{l}}\leq f_{l}\leq\overline{f_{l}}\quad\quad\forall
l\in L$ (10) $\displaystyle 0\leq\lambda^{PV}_{b}\leq\hat{P}^{PV}_{b},\quad
0\leq\lambda^{W}_{b}\leq\hat{P}^{W}_{b}\quad\forall b\in B$ (11)

where the objective function in (6) minimises the operational dispatching cost
of the system denoted by $C^{sch}$, which consists of dispatching cost,
renewable curtailment cost, and a penalty term for infeasibility in the node
balance. In this regard, the conventional generator output and cost are
$P^{g}_{b}$ and $c^{g}_{b}$. Furthermore, the predicted PV (wind) generation
is $\hat{P}^{PV}_{b}$ ($\hat{P}^{W}_{b}$), while PV (wind) curtailment and
corresponding curtailment cost are $\lambda^{PV}_{b}$ ($\lambda^{W}_{b}$), and
$\gamma^{PV}_{b}$ ($\gamma^{W}_{b}$), respectively. Positive (negative) node
injection imbalance and its associated penalty are given by $\lambda^{+}_{b}$
($\lambda^{-}_{b}$) and $\gamma_{b}$. Furthermore, load demand and active line
flow are represented by $P^{d}_{b}$, and $f_{l}$, respectively. Eq. (7) models
the node balance, while the flow of line $l$ between adjacent nodes with the
reactance of $x_{l}$ is formulated in (9). Finally, the limits on generator
output, line flow, PV and wind curtailment are presented in (8), (10)-(11).
While the dispatch of generators $\hat{P}^{g}_{b}$ is determined based on the
predicted renewable generation $\hat{P}^{PV}_{b}$ and $\hat{P}^{W}_{b}$, the
true values of renewable generation $P^{PV}_{b}$ and $P^{W}_{b}$, might be
different, e.g.

$\displaystyle\hat{P}^{PV}_{b}\neq P^{PV}_{b}$ $\displaystyle\exists\,b\in B$
(12a) $\displaystyle\hat{P}^{W}_{b}\neq P^{W}_{b}$ $\displaystyle\exists\,b\in
B.$ (12b)

Therefore, redispatching accounts for this mismatch between the predicted and
actual renewable generation. In this context, the OPF-Redispatch problem is
formulated as below:

$\displaystyle\min$ $\displaystyle C^{rd}=\sum_{b\in
B}(c_{b}^{up}dP_{b}^{g+}+c_{b}^{dn}P_{b}^{g-})$ (13)
$\displaystyle+\gamma^{PV}_{b}\lambda^{PV}_{b}+\gamma^{W}_{b}\lambda^{W}_{b}+\sum_{b\in
B}\big{(}\gamma_{b}(\lambda^{+}_{b}+\lambda^{-}_{b})\big{)}$ s.t.
$\displaystyle(\hat{P}^{g}_{b}+dP_{b}^{g+}-dP_{b}^{g-})-P^{d}_{b}$ (14)
$\displaystyle+(P^{PV}_{b}-\lambda^{PV}_{b})+(P^{W}_{b}-\lambda^{W}_{b})$
$\displaystyle=\sum_{l\in
L_{b}}f_{l}+(\lambda^{+}_{b}-\lambda^{-}_{b})\quad\quad\forall b\in B$
$\displaystyle 0\leq dP_{b}^{g+}\leq\overline{dP_{b}^{g+}},\quad 0\leq
dP_{b}^{g-}\leq\overline{dP_{b}^{g-}}\quad\quad\forall b\in B$ (15)
$\displaystyle\underline{P^{g}_{b}}\leq\hat{P}_{b}^{g}+dP_{b}^{g+}-dP_{b}^{g-}\leq\overline{P^{g}_{b}}\quad\quad\forall
b\in B$ (16) $\displaystyle\quad\text{Eqs.
}\eqref{eq:branchflow}-\eqref{eq:lamPV},$

where the objective minimises the redispatching cost of the system, shown by
$C^{rd}$, which includes upward and downward ramping cost of generators, as
well as renewable and penalised nodal imbalances. In this regard, upward
(downward) redispatch of generators and its associated cost is represented by
$dP_{b}^{g+}$ ($dP_{b}^{g-}$), and $c_{b}^{up}$ ($c_{b}^{up}$), respectively.
The nodal balance constraint is shown in (14), while generator redispatch
constraints are presented in (15)-(16).

After solving OPF-Redispatch, the total system cost $C^{sys}$ can be
calculated as below:

$\displaystyle C^{sys}=C^{sch}+C^{rd}$ (17)

The total system cost is a function of renewable prediction accuracy, where
prediction errors lead to higher operational costs $C^{sys}$, where the root-
mean-squared error for PV and wind are Eq. (18) and Eq. (19) respectively.

$RMSE^{PV}=\sqrt{\frac{\sum_{b\in
B}(\hat{P}^{PV}_{b}-{P}^{PV}_{b})^{2}}{|B|}}$ (18)
$RMSE^{W}=\sqrt{\frac{\sum_{b\in B}(\hat{P}^{W}_{b}-{P}^{W}_{b})^{2}}{|B|}}$
(19)

###  II-D Proposed Multi-Modal End-to-End Learning Approach

The proposed MM-E2E learning approach trains the ANNs $\tilde{f_{1}}$ and
$f_{3}$ to minimise the ultimate system cost $C^{sys}$. Our approach combines
the features from the modalities $h_{1}$ and $x^{Meteo}_{i}$ in a single ANN
$f_{3}$ to perform renewable power prediction, which is then used as input for
OPF optimisation as the ultimate task. The proposed approach is shown in Fig.
1 and follows $5$ consecutive steps:

Figure 1: The proposed multi-modal end-to-end architecture (MM-E2E). The
extracted features from imagery are fused with the meteorological data to
predict renewable power ($\hat{y}$) by FC layers. The power prediction is then
used to solve the OPF problems, the output of which trains the ANN. Figure 2:
The OPF-Schedule and OPF-Redispatch minimise system costs.

  1. 1.

A feature extraction network $\tilde{f_{1}}$, CNN-fe is applied to an image
input $x^{Image}_{i}$ according to Eq. (2). The CNN-fe shown as part of Fig. 1
consists of convolutional, ReLU activation, and max pooling layers, followed
by an FC layer with ReLU activation and another with Sigmoid activation.
$\tilde{f_{1}}$outputs the feature vector $h_{1,i}$.

  2. 2.

The meteorological features $x^{Meteo}_{i}$ are fused with $h_{1,i}$ in a
joint representation using concatenation as in Eq. (3)

  3. 3.

Renewable power $\hat{y}_{i}$ is predicted from the fused feature vector,
$h_{mm,i}$, using a FNN layer $f_{3}$ as in Eq. (5), referred to as FNN-pred.

  4. 4.

The prediction $\hat{y}_{i}=(\hat{P}^{PV}_{i},\hat{P}^{W}_{i})$ is one of the
inputs to the OPFs as illustrated in Fig. 2. These OPFs consist of two
successively executed variations of OPFs (OPF-Schedule and OPF-Redispatch) as
in Sec. II-C. Both are integrated as differentiable parameterized optimization
layers in the neural network, where the solutions are differentiated
concerning the parameters $\hat{y}_{i}$ [13].

    1. 4.1)

As shown in Fig. 2, the renewable prediction
$\hat{y}_{i}=(\hat{P}^{PV}_{i},\hat{P}^{W}_{i})$ is the input to the optimal
dispatch schedule of the generators with OPF-Schedule.

    2. 4.2)

Then, if there exists any power imbalance due to a mismatch of the predicted
power ($\hat{P}^{PV}_{b},\hat{P}^{W}_{b})$ to the true powers
$(P^{PV}_{b},P^{W}_{b})$ according to Eq. (12a), the OPF-Redispatch problem is
solved to find the optimal redispatch of the generators.

  5. 5.

During training, the loss is considered as the total system cost of dispatch
and redispatch, given by Eq. (17). This loss is differentiated and the
gradients backpropagated to update the weights of all parameters defining
functions $\tilde{f_{1}}$ and ${f_{3}}$ of CNN-fe and FNN-pred, respectively.

Figure 3: The modified IEEE 6-bus system with PV and wind generation.

##  III Case Studies

###  III-A Settings and Test Network

####  III-A1 Power System

Fig. 3 shows the modified IEEE $6$-bus system with $3$ generator, $3$ load
demands, PV generation at bus $1$, and wind generation at bus $2$. Other
parameters used in OPF-Schedule and OPF-Redispatch optimisations are presented
in Appendix References. Below are details regarding load demand, PV
generation, and wind generation.

  * •

The load is modelled from the dynamic load profiles of $1.5$ million identical
households, normalised to IEEE $6$-bus values and distributed over $3$ buses
[17]. In this context, the maximum load demand of the considered data is
$293$MW.

  * •

PV generation is based on the LG NeON BiFacial 440W panel (with a panel area
of $2.2$ m2). Eq. (20) approximates the output of a PV farm with $N_{panels}$
panels of area $A_{panel}$, given their rated power $P_{rated}$ and the Global
Horizontal Irradiance (GHI).

$P_{PV}=\frac{GHI}{1000W/m^{2}}\times P_{Rated}\times N_{panels}$ (20)

Eq. (20) disregards losses in panel efficiency due to temperature. We assume a
farm of $250,000$ panels and corresponding rating of $110$MWp.

  * •

Wind generation is based on a farm of 36 Vestas V112-3.45MW turbines (modelled
on the existing OWEZ offshore farm), resulting in a rating of 124MWp. The
turbine power curve is approximated by a Tanh function, which can be found in
Appendix References.

Sky imagery data and meteorological sensor data (with $10$ minutes resolution)
are two modalities. For meteorological features, time, air temperature, cloud
opacity, relative humidity, wind direction, wind speed, precipitable water,
and surface pressure are considered (more details in Appendix References). GHI
is not included to simulate a context where local pyranometer data is not
available. The sky imagery was captured at the TU Delft campus. The dataset
consists of images captured at a minute frequency between 04:00 and 23:00 from
01 May $2021$ to $24$ February $2022$. Each image is in full colour, with a
resolution of $1536\times 1536$. An example of an image taken on $01$ May
$2021$ at 11:00 is provided in Fig. 4.

Figure 4: All sky images captured by the camera at 11:00 on 01 May 2021.

####  III-A2 Training and Testing

The data set contains $10,000$ data samples, where $25$% is saved for testing,
$67.5$% is used for training (total $N=6750$ training samples), and $7.5$% is
for validation. During training, early stopping is implemented if the
validation error does not decrease for 5 consecutive epochs. The AdamW
optimiser with a batch size of $64$ is used to train the ANN. Furthermore,
$10$ random train-test trials are considered for all case studies. The
separation of the validation and training split between trials is varied, as
is the mini-batch shuffling, while the never-before-seen test set is the same.
The plots and tables in each case study show the distribution of test set
predictions over the $10$ trials. Moreover, for sequential learning models
(not E2E models), the mean squared error

$MSE=\frac{1}{N}\sum_{i\in\Omega^{N}}(y_{i}-\hat{y}_{i})^{2}$ (21)

and mean absolute error
$MAE=\frac{1}{N}\sum_{i\in\Omega^{N}}|y_{i}-\hat{y}_{i}|$ were used as loss
functions.

####  III-A3 Network Architecture

Several studies were conducted to obtain a good ANN network architecture. The
studies focused on the ideal ANN depth, number of images used, image
resolution, etc. The CNN-base model contains $3$ sets of convolutional and
max. pooling layers, followed by $2$ FC layers; whereas the FNN-base) for
forecasting from meteorological consists of $4$ FC layers. Apart from the last
layer (which uses Sigmoid activation), all other layers use the ReLU
activation function. Details of these architectures are in Appendix
References. Moreover, these initial studies showed that using $3$ recent
images, combined with a pixel-wise average, produced better PV power
predictions by CNN-base than $1$ image. The same conclusion was found for the
number of recent meteorological samples, resulting in $24$ meteorological
inputs ($3$ for each of the $8$ variables). The initial results in Table I
show that power predictions with the meteorological data and FNN-base were far
more accurate than using imagery and CNN-base.

TABLE I: Comparison of predictions of PV and wind power from Imagery (using CNN-base) and Meteorological data (using FNN-base). | PV | Wind
---|---|---
| Imagery | Meteo. Data | Imagery | Meteo. Data
Average MSE | 0.00836 | 0.00177 | 0.00867 | 0.00011
STD MSE | 0.000682 | 0.000134 | 0.00131 | 0.00013

The relative benefit of meteorological data can be explained by that images do
not contain as many obvious and distinct features, such as wind speed and
humidity, as are directly present in the meteorological data.

####  III-A4 Investigated Models

Five models were investigated.

  1. 1.

UM-base1-Seq: this naive ensemble baseline combines the predictions from
imagery (using CNN-base) $f_{1}$ and meteorological data (using FNN-base)
$f_{2}$ from Eq. (1) as an average

$\hat{y}^{UM1}=0.5f_{1}(x_{i}^{Image})+0.5f_{2}(x_{i}^{Meteo}),$ (22)

exemplifying the simplest late fusion method to combine the two modalities.
The predicted power is then used to solve for system cost $C^{sys}$ in the
$6$-bus system.

  2. 2.

UM-base2-Seq: a linear regressor is fitted to combine the two power
predictions

$\hat{y}^{UM2}=c_{1}f_{1}(x_{i}^{Image})+c_{2}f_{2}(x_{i}^{Meteo}),$ (23)

which can be considered as a more ‘intelligent’ version of UM-base1-Seq as the
weights $c_{1}$ and $c_{2}$ are fitted.

  3. 3.

MM-Seq: the MM approach shown in Fig. 1 is trained by using the $MSE$ shown in
Eq. (21) as loss, so the trained ANN has the objective to accurately
predicting renewable power. This baseline represents current ML-based
approaches for PV and wind prediction.

  4. 4.

MM-E2E (proposed model): the MM approach is used to predict power and is
trained E2E based on the minimised $C^{sys}$.

  5. 5.

Perfect Forecast: simulates a scenario where the system cost is calculated
using the true PV or wind powers $(P^{PV}_{b},P^{W}_{b})$. This simulation,
therefore, calculates the minimum achievable system cost.

###  III-B Meteorological Feature Extraction

Here, two MM networks are compared in terms of their effectiveness at
predicting PV power. The first includes a network for extracting features from
both modalities separately (FNN for meteorological data and CNN for imagery),
before combining them, whereas the second only uses a network (CNN) to extract
features from imagery and then fuses them with the meteorological data. The
method used for fusion in both architectures for this study is concatenation.
Over $10$ times training, the first model achieves an average MSE of 0.00143
(0.00058), where the number in parentheses is the standard deviation, compared
to an MSE of 0.00129 (0.00031) for the second model. Despite the added
complexity of the first architecture (i.e., more layers), it attains a higher
error and variance in error. These results may be attributed to the fact that
there are few meteorological features, to begin with ($F=24$), thus, the added
feature extraction network only overfits the training data without revealing
new information from the features. This effect may be alleviated with much
larger sets of training data or the use of more previous instances of
meteorological features.

###  III-C Multi-Modal Joint Representation Method

This study investigates bilinear pooling (given in Eq. (4)) and concatenation
for feature fusion within the MM architecture. This fusion occurs, as shown in
Fig. 1, after feature extraction from the imagery modality. The ANN using
concatenation achieves an average MSE of 0.00109 (0.00024), compared to an MSE
of 0.00144 (0.00033) for bilinear pooling over these 10 trials. Bilinear
pooling includes all the features represented in the fused feature space of
concatenation. However, it results in quadratically larger feature space with
greater sparsity and more irrelevant features. Thus, the ANN may miss
important information amongst the vast feature space, potentially prompting
the need for feature selection. Feature selection for regression problems
reduces not only computational time but also improves predictive performance
by avoiding spurious correlations between irrelevant features and the target
(a form of overfitting).

TABLE II: Average and standard deviation of MSE and %RMSE on predicted PV power over 10 trials. | Persistence | UM-base1 | UM-base2 | MM
---|---|---|---|---
Average MSE | 0.0008 | 0.00294 | 0.00151 | 0.00109
STD (MSE) | 0 | 0.000157 | 0.000179 | 0.000242
Average %RMSE | 2.8% | 5.4% | 3.9% | 3.2%
STD (%RMSE) | 0 | 0.14% | 0.23% | 0.34%

###  III-D Comparison of Methods for Power Forecasting

This case study quantifies the utility of MM learning for power forecasting.
UM-base1, UM-base2, the proposed MM model, and a persistence forecast are
compared in terms of prediction error. The OPF problems, system cost and E2E
training are therefore not considered here as we seek to compare the
prediction performance of the models. The results are presented in Table II.
The prediction performance of UM-base1, UM-base2 and MM, in terms of %RMSE, is
in line with state-of-the-art ANN-based methods for predicting solar power
[18]. The persistence forecast outperforms the other $3$ methods consistently.
However, it must be noted that this is because it is predicting PV power
(analogous to GHI given Eq. (20)) using the GHI from 10 minutes prior, while
the ANN methods do not include recent GHI as a feature. UM-base1 severely
lacks performance compared to UM-base2, likely due to the significantly worse-
performing imagery modality being equally weighed with the sensor data.
Additionally, MM learning improves predictions compared to UM-base2. Two
reasons may explain this phenomenon. Firstly, UM-base2 has only two trainable
parameters (as given in Eq. (23)), while MM has thousands; this additional
complexity may aid prediction. Secondly, UM-base2 employs the latest possible
fusion, only combining the predicted GHI from either modality, while MM
employs intermediate fusion, concatenating features from meteorological data
and imagery. During training, this architecture extracted feature layers (CNN-
fe) to learn how the visual features enhance the sensor data for forecasting.

###  III-E E2E learning with PV and Wind Prediction

Figure 5: Excess system cost compared to power forecast error (MAE) in a
system with PV generation.

This study investigates the system costs minimisation by E2E learning against
the baselines for predicting PV power which are established already. The MM
learning model in Fig. 5 is shown to not only improve power output prediction
but also benefit power system operation in terms of system cost as both the MM
models outperform the UM ones. A conventional approach, represented by
UMbase2-Seq, results in a 32% cost increase compared to a perfect knowledge
scenario. $C^{sys}$ is only increased by 10% when using MM learning and
training sequentially. Much of this cost reduction is due to the intelligent
combination of MM features using an ANN. E2E learning also contributes a clear
reduction in cost (comparing the 10% excess cost from MM-Seq to 7% from
MM-E2E). In addition, E2E training enables a more consistently reduced cost,
shown by the lower variance in cost. This improved performance of MM-E2E can
be explained by the fact that E2E training provides the ANN with information
about the generation cost functions ($c_{b}^{up}$, $c_{b}^{dn}$) of the
various generators. Thus, for instances where perfect prediction is not
obtained, E2E training may enable the ANN to appropriately over- or
underestimate power such to reduce cost.

This same study can be conducted in a power system with PV and wind
generation. The proposed model is adjusted to ‘MM-PVWind-E2E’ to predict both
power sources by including an FNN-pred network which uses only meteorological
data for wind prediction. The results are in Fig. 6, where the error is
measured as the average of $MAE(\hat{P}_{g}^{PV},P_{g}^{PV})$ and
$MAE(\hat{P}_{g}^{W},P_{g}^{W})$.

Figure 6: Excess system cost compared to power forecast error (MAE) in a
system with PV and wind generation.

While E2E training results in lower system costs than sequential learning, the
power prediction error is significantly greater in the former. Note that the
standard deviation on system cost is lowest from MM-PVWind-E2E, while the
standard deviation on the power prediction is the largest for MM-PVWind-E2E.

###  III-F System Cost Functions

The results found in Figs. 5 and 6 are worth investigating as they both show
MM-E2E achieving the lowest costs but with disparate prediction error. We
investigate the cost function in PV and wind systems with a load of 145MW, and
PV and wind generation of $25$MW each. Fig. 7 shows the system cost $C^{sys}$
under regular and reduced (by $50$%) branch limits. Fig. 7(a) shows no global
minimum, but isolines along which identical costs can be achieved by
overestimating PV and underestimating wind, or vice-versa. Thus, system
balance may be maintained with no (or very low) redispatch costs while
incurring prediction errors. This behaviour explains Fig. 6, where relatively
large prediction errors are not detrimental to cost minimisation. Fig. 7(b)
shows possible local minima as branch limits are reduced and more constraints
are binding. E2E training in this context would enable the learning of a more
complex cost function, but may also cause inaccurate power predictions. Thus,
the optimisation objective should be carefully selected depending on the
problem context for E2E training.

(a)

(b)

Figure 7: Cost functions of a system with PV and wind generation, (a) under
regular, (b) and reduced (by 50%) branch limits.

###  III-G Discussion

The studies presented promising results regarding the application of MM and
E2E learning for economically optimised renewable power predictions. However,
some limitations of the methodology can be noted, which most importantly is
the computational time to train the models as it involves solving
optimisations. The computational time’s complexity to the training data and
power network sizes must be thoroughly analysed to quantify the approach’s
limitations. Furthermore, in future works, using Pearson correlation for
selecting features may improve power prediction. Including features from early
layers into the MM power prediction may also improve performance and mitigate
the vanishing gradients [7]. Testing the proposed MM-E2E architecture with
various cost functions, e.g., asymmetric and quadratic ones would simulate
more realistic conditions. Designing a tailored loss function, such as a
multi-task objective, may prevent converging to local minima (sometimes seen
with E2E learning). Further, considering the prediction relies mostly on time-
sequenced data (in both modalities), ML techniques such as LSTM-RNNs could
also improve predictions compared to only CNNs and FNNs. Finally, a more
thorough analysis with a wider set of OPF-type problems is recommended to
fully assess the proposed approach’s benefits.

##  IV Conclusion

This paper investigated two deep learning techniques, multi-modal (MM)
learning and end-to-end (E2E) learning, for photovoltaic and wind power
nowcasting in energy management systems. The features from two modalities
(all-sky imagery and time series meteorological data) are combined through MM
learning. The model is trained E2E on the total system cost from prediction
errors modelled as optimization problems. The proposed model architecture,
MM-E2E, is tested in different settings against uni-modal sequentially trained
convolutional and feedforward networks to understand the advantages and
limitations of the methodology. Our studies found that the proposed MM-E2E
model reduced the economic cost by impressively around 30%. However, we are
also cautious and discussed the limitation regarding the computational scaling
of the E2E approach, which we recommend as future works.

## References

  * [1] Y. Ren, P. Suganthan, and N. Srikanth, “Ensemble methods for wind and solar power forecasting—a state-of-the-art review,” _Renewable and Sustainable Energy Reviews_ , vol. 50, pp. 82–91, 2015.
  * [2] R. Ahmed, V. Sreeram, Y. Mishra, and M. D. Arif, “A review and evaluation of the state-of-the-art in PV solar power forecasting: Techniques and optimization,” _Renewable and Sustainable Energy Reviews_ , vol. 124, 2020\.
  * [3] E. Vrettos and C. Gehbauer, “A Hybrid Approach for Short-Term PV Power Forecasting in Predictive Control Applications,” in _2019 IEEE Milan PowerTech_ , 2019, pp. 1–6.
  * [4] S. Leva, M. Mussetta, A. Nespoli, and E. Ogliari, “PV power forecasting improvement by means of a selective ensemble approach,” in _2019 IEEE Milan PowerTech_ , 2019, pp. 1–5.
  * [5] J. Zhang, R. Verschae, S. Nobuhara, and J.-F. Lalonde, “Deep photovoltaic nowcasting,” _Solar Energy_ , vol. 176, pp. 267–276, 2018.
  * [6] Z. Li, K. Wang, C. Li, M. Zhao, and J. Cao, “Multimodal Deep Learning for Solar Irradiance Prediction,” in _2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData)_ , 2019, pp. 784–792.
  * [7] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 2, pp. 423–443, 2018.
  * [8] D. Lahat, T. Adalı, and C. Jutten, “Multimodal data fusion: An overview of methods, challenges, and prospects,” _Proceedings of the IEEE_ , vol. 103, pp. 1449–1477, 2015.
  * [9] Q. Xian and W. Liang, “A multi-modal time series intelligent prediction model,” in _Proceeding of 2021 International Conference on Wireless Communications, Networking and Applications_ , Z. Qian, M. Jabbar, and X. Li, Eds. Singapore: Springer Nature Singapore, 2022, pp. 1150–1157.
  * [10] P. Donti, B. Amos, and J. Z. Kolter, “Task-based end-to-end model learning in stochastic optimization,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [11] D. Wahdany, C. Schmitt, and J. L. Cremer, “More than accuracy: end-to-end wind power forecasting that optimises the energy system,” _Electric Power Systems Research_ , vol. 221, p. 109384, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0378779623002730
  * [12] P. L. Donti, D. Rolnick, and J. Z. Kolter, “DC3: A learning method for optimization with hard constraints,” in _International Conference on Learning Representations_ , 2021.
  * [13] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and Z. Kolter, “Differentiable convex optimization layers,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [14] H. M. Bui, M. Lech, E. Cheng, K. Neville, and I. S. Burnett, “Using grayscale images for object recognition with convolutional-recursive neural network,” _2016 IEEE 6th International Conference on Communications and Electronics, IEEE ICCE 2016_ , pp. 321–325, 9 2016.
  * [15] A. Geron, _Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow_. O’Reilly, 2019.
  * [16] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017, pp. 1103–1114.
  * [17] BDEW, “Standardlastprofile Strom,” 2017, data retrieved from World Development Indicators, https://www.bdew.de/energie/standardlastprofile-strom/.
  * [18] M. Q. Raza, M. Nadarajah, and C. Ekanayake, “On recent advances in PV output power forecast,” _Solar Energy_ , vol. 136, pp. 125–144, 10 2016.

Details of the parameters used in the DCOPF-Schedule and DCOPF-Redispatch
optimisations are given in Table III. The wind turbine power curve and its
approximation are depicted in Fig. 8. This work considers meteorological
sensor data and sky imagery data modalities. The meteorological features are
presented in Table VIII. Each one is measured at a temporal resolution of $10$
minutes. The baseline models resulting from initial case studies (CNN-base and
FNN-base) are described in Tables V and IV. The architectures of CNN-fe and
FNN-pred are given in Tables VI and VII.

TABLE III: Values of essential parameters used in the optimisations. Parameter | Value | Unit
---|---|---
$\underline{P^{g}_{1}},\underline{P^{g}_{2}},\underline{P^{g}_{3}}$ | 0 | MW
$\overline{P^{g}_{1}}$ | 200 | MW
$\overline{P^{g}_{2}}$ | 150 | MW
$\overline{P^{g}_{2}}$ | 180 | MW
$c_{1}^{g}$ | 12 | Euro/MW
$c_{2}^{g}$ | 10 | Euro/MW
$c_{3}^{g}$ | 8 | Euro/MW
$\overline{f_{l}}$ | 100 | MW
$L$ | 11 |
$B$ | 6 |
$\gamma_{b}$ | 100 | Euro/MW
$\gamma^{PV}_{b},\gamma^{W}_{b}$ | 0.1 | Euro/MW
Figure 8: Power curve of Vestas V112-3.45 turbine and a Tanh approximation. TABLE IV: FNN-base architecture. Layer | Type | Input Size | # Neurons | Activation
---|---|---|---|---
F1 | FC |  $n_{inputs}\times$1 | 64 | ReLU
F2 | FC | 64$\times$1 | 128 | ReLU
F3 | FC | 128$\times$1 | 64 | ReLU
F4 | FC | 64$\times$1 | 1 | Sigmoid
TABLE V: CNN-base architecture. Layer | Type | Input Size | Kernel | Stride | Padding | Activation
---|---|---|---|---|---|---
C1 | Conv. | 64$\times$64$\times$1 | 3$\times$3 | 2$\times$2 | 1$\times$1 | ReLU
P2 | Max. Pool. | 32$\times$32$\times$8 | 2$\times$2 | 2$\times$2 | 0$\times$0 | -
C3 | Conv. | 16$\times$16$\times$8 | 3$\times$3 | 1$\times$1 | 1$\times$1 | ReLU
P4 | Max. Pool. | 16$\times$16$\times$32 | 2$\times$2 | 2$\times$2 | 0$\times$0 | -
C5 | Conv. | 8$\times$8$\times$32 | 3$\times$3 | 1$\times$1 | 1$\times$1 | ReLU
P6 | Max. Pool. | 8$\times$8$\times$16 | 2$\times$2 | 2$\times$2 | 0$\times$0 | -
Flatten
Layer | Type | Input Size | # Neurons | - | - | Activation
F7 | FC | 256$\times$1 | 64 | - | - | ReLU
F8 | FC | 64$\times$1 | 1 | - | - | Sigmoid
TABLE VI: CNN-fe architecture. Layer | Type | Input Size | Kernel | Stride | Padding | Activation
---|---|---|---|---|---|---
C1 | Conv. | 64$\times$64$\times$1 | 3$\times$3 | 2$\times$2 | 1$\times$1 | ReLU
P2 | Max. Pool. | 32$\times$32$\times$8 | 2$\times$2 | 2$\times$2 | 0$\times$0 | -
C3 | Conv. | 16$\times$16$\times$8 | 3$\times$3 | 1$\times$1 | 1$\times$1 | ReLU
P4 | Max. Pool. | 16$\times$16$\times$32 | 2$\times$2 | 2$\times$2 | 0$\times$0 | -
C5 | Conv. | 8$\times$8$\times$32 | 3$\times$3 | 1$\times$1 | 1$\times$1 | ReLU
P6 | Max. Pool. | 8$\times$8$\times$16 | 2$\times$2 | 2$\times$2 | 0$\times$0 | -
Flatten
Layer | Type | Input Size | # Neurons | - | - | Activation
F7 | FC | 256$\times$1 | 64 | - | - | ReLU
F8 | FC | 64$\times$1 | 64 | - | - | Sigmoid
TABLE VII: FNN-pred architecture. Layer | Type | Input Size | # Neurons | Activation
---|---|---|---|---
F1 | FC | 88$\times$1 | 64 | ReLU
F2 | FC | 64$\times$1 | 128 | ReLU
F3 | FC | 128$\times$1 | 64 | ReLU
F4 | FC | 64$\times$1 | 1 | Sigmoid
TABLE VIII: Meteorological features used for predicting $P^{PV}$ or $P^{W}$. Meteo. Variable |  Description |  Unit
---|---|---
Time |  The moment of the day when the instance was measured |  seconds
Air temperature |  Measured at 2 meters above surface level |  $\degree$C
Cloud opacity |  The attenuation of incoming sunlight due to clouds. Varies from 0% (no cloud) to 100% (full attenuation of incoming sunlight) |  %
Relative humidity |  Measured at 2 meters above ground level. Relative humidity is the amount of water vapour as a percentage of the amount needed for saturation at the same temperature. A value of 50% means the air is 50% saturated |  %
Wind direction |  Measured at 10m altitude |  $\degree$
Wind speed |  Measured at 10m altitude |  m/s
Precipitable water |  A measure of the precipitable water of the entire atmospheric column |  kg/m2
Surface pressure |  Air pressure measured at sea level |  hPa