# Model Selection, Adaptation, and Combination for Transfer Learning in Wind
and Photovoltaic Power Forecasts

###### Abstract

There is recent interest in using model hubs, a collection of pre-trained
models, in computer vision tasks. To utilize the model hub, we first select a
source model and then adapt the model for the target to compensate for
differences. While there is yet limited research on model selection and
adaption for computer vision tasks, this holds even more for the field of
renewable power. At the same time, it is a crucial challenge to provide
forecasts for the increasing demand for power forecasts based on weather
features from a numerical weather prediction. We close these gaps by
conducting the first thorough experiment for model selection and adaptation
for transfer learning in renewable power forecast, adopting recent results
from the field of computer vision on 667 wind and photovoltaic parks. To the
best of our knowledge, this makes it the most extensive study for transfer
learning in renewable power forecasts reducing the computational effort and
improving the forecast error. Therefore, we adopt source models based on
target data from different seasons and limit the amount of training data. As
an extension of the current state of the art, we utilize a Bayesian linear
regression for forecasting the response based on features extracted from a
neural network. This approach outperforms the baseline with only seven days of
training data. We further show how combining multiple models through ensembles
can significantly improve the model selection and adaptation approach.

###### keywords:

Transfer Learning , Time Series , Renewable Energies , Temporal Convolutional
Neural Network , Ensembles , Wind and Photovoltaic Power.

[Size of input features.]dimFeatureD [The parameters of a linear
model.]parametersLinModelθ [All input features.]inputFeaturesX [All response
features.]outputFeaturesy [A single input vector.]inputFeatureVectorx [A
single response respective target.]outputFeatureVectory [A single response
prediction.]outputPredictionVector^y [The set of input
features.]setInputFeaturesX [The set of response features.]setOutputFeaturesY
[Input feature space.]inputFeatureSpaceX [Output feature
space.]outputFeatureSpaceY [The number of samples.]nSamplesN [Index of
samples.]indexSamplesn [Set of all real numbers.]realNumberR [Set of all non-
negative real numbers.]realNumberNonNegR^+ [Set of all positive real
numbers.]realNumberPosR_≥1 [Set of all natural numbers.]naturalNumberN [Set of
all non-negative natural numbers.]naturalNumberNonNegN^+ [Set of all positive
natural numbers.]naturalNumberPosN_≥1 [Normal distribution.]normalDistN [A
task.]taskT [The set of all tasks.]allTasksT [Similarity measure between two
task.]similarityMeasureS [Precision matrix of a linear model.]precisionMatrixS
[Index of a source model.]indexSourceModelm [The number of source
models.]numberSourceModelsM [The dimension of the target.]dimensionTargetI
[The index of the target task.]indexTargetTaskT [CSGE weight not
normalized.]csgeWeightUnnormalized¯w [CSGE weight
normalized.]csgeWeightNormalizedw

1\. Selection and 2. Adaption and 3. Combination Model $\mathcal{T}_{1}$ Model
$\mathcal{T}_{2}$ Model $\mathcal{T}_{M}$ Model $\mathcal{T}_{T}$ Forecast
Input and Response Features Model HubTarget Model…

With less than 90 days data, fine-tuning a source model is often
disadvantageous.

With less than 30 days data, any adaption of a source model is often
disadvantageous.

With more than 30 days data, an adaption through a Bayesian linear regression
is advantageous.

Results can be significantly improved through ensemble techniques.

##  1 Introduction

With the extension of volatile energy resources, such as wind and photovoltaic
(PV) parks, one fundamental problem is adding new parks to an operator’s
portfolio. The historical data for such a new (target) park is often limited.
At the same time, reliable forecasts are fundamental to assure grid stability
due to the weather dependency. However, typically there are numerous pre-
trained models from existing parks that we can utilize for such a forecasting
task [1]. Utilizing those pre-trained source models often increases the
forecast accuracy, reduces the computational effort, and reduces the carbon
emission for training a new model [2, 3]. Now, the question arises what is the
best way to use of this model hub of pre-trained models. The field of research
inductive transfer learning (ITL) provides methods for this problem [4].

1\. Selection and 2. Adaption and 3. Combination Model $\mathcal{T}_{1}$ Model
$\mathcal{T}_{2}$ Model $\mathcal{T}_{M}$ Model $\mathcal{T}_{T}$ Forecast
Input and Response Features Model HubTarget Model…

(a) Process of model selection, adaptation, and combination.

$0$$5$$10$$15$$20$$0$$0.2$$0.4$$0.6$$0.8$$1$ Wind Speed [$\frac{m}{s^{2}}$]
$\mathbb{E}[p(\setOutputFeatures_{1}|\setInputFeatures_{1},\parametersLinModel_{1})]$
$\mathbb{E}[p(\setOutputFeatures_{T}|\setInputFeatures_{T},\parametersLinModel_{T})]$
$\mathbb{E}[p(\setOutputFeatures_{2}|\setInputFeatures_{2},\parametersLinModel_{2})]$
Normalized Power [-]
$\similarityMeasure({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\mathcal{T}_{1}},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mathcal{T}_{T}})$
$\similarityMeasure({\color[rgb]{0,0.390625,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.390625,0}\mathcal{T}_{2}},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mathcal{T}_{T}})$

(b) Similarity between sources $\task_{1}$ and $\task_{2}$ with a target task
$\task_{T}$ for mean wind power predictions.

Figure 1: Diagrams illustrating the knowledge transfer based on a model hub.

Fig. 1a summarizes our proposed strategy. The first step is to select an
appropriate source model. Recently, [4, 5] showed that selecting an
appropriate source model for knowledge transfer for a target substantially
influences the test error for computer vision tasks. To select a source model,
consider, e.g., we have two source tasks $\task_{1}$ and $\task_{2}$ of a wind
park with model parameters $\parametersLinModel_{1}$ and
$\parametersLinModel_{2}$. These models have a set of input observations
$\setInputFeatures_{1}$ and $\setInputFeatures_{2}$ as well as the sets of
response values $\setOutputFeatures_{1}$ and $\setOutputFeatures_{2}$. Based
on this information, we want to select one of the models for knowledge
transfer for a target task $\task_{T}$ with parameters
$\parametersLinModel_{T}$ and its respective sets $\setInputFeatures_{T}$ and
$\setOutputFeatures_{T}$.

The diagram in Fig. 1b visualizes this problem for wind power forecasts. In
renewable power forecasts, we utilize weather forecasts, such as the wind
speed or radiation, from a so called numerical weather prediction (NWP) model.
These predicted weather features are the input $\setInputFeatures$ to machine
learning (ML) models, with parameters $\parametersLinModel$, predicting the
expected power generation
$\mathbb{E}[p(\setOutputFeatures|\setInputFeatures,\parametersLinModel)]$ in a
day-ahead forecasting task between $24$ and $48$ hours into the future. In the
diagram, we can observe that the similarity $\similarityMeasure$ depends on
the relation between the input feature wind and the power generated by a wind
park, i.e., for different wind speeds and models, we expect a different power
generation.

Once a model is selected, the second step adapts the source knowledge with the
limited target data with an adaptation strategy. Often this adaptation
strategy is fine-tuning the final layer of a neural network. Only with such an
adaptation can we make reliable and task-specific power forecasts for a new
target task of a new park with limited data. To the best of our knowledge,
different adaptation and selection strategies have been so far not considered
for ITL in the field of renewable power forecasts. We close this gap with this
article. Selecting and adapting a single source model from a model hub has the
disadvantage of neglecting knowledge from other source tasks that are
potentially beneficial. However, we can additionally combine models through
ensemble techniques. Combining knowledge can be seen as a mixture of
selection, where more relevant models are weighted higher, and adaptation,
where source models are weighted to fit the target. Since ITL has been
insufficiently studied so far for renewable power forecasts [6], especially
for day-ahead forecast horizons between $24$ and $48$ hours into the future,
we answer the following research questions:

######  Research Question 1.

What is an appropriate similarity measure for model selection for a new target
park from a model hub with pre-trained models?

######  Research Question 2.

What is the best adaptation strategy once a model is selected?

######  Research Question 3.

Compared to selecting and adapting a single model, are ensemble strategies
beneficial for combining knowledge?

To answer those research questions, we train a Bayesian extreme learning
machine (BELM), a multi-layer perceptron (MLP), and a temporal convolution
network (TCN) as source models on six datasets including $667$ distinct parks.
We consider the gradient boosting regression tree (GBRT) as a baseline that
outperforms physical models even with limited target data [7].

Based on these datasets and source models, our main contributions to answering
the research questions can be summarized as follows: First, we propose the
normalized root mean squared error (nRMSE), due to its strong correlation with
the forecasting error, as well as the marginal likelihood or evidence as a
selection strategy for question one. Therefore, we replace the last layer of a
neural network with a Bayesian linear regression (BLR), as proposed in [4] for
computer vision tasks. Second, we propose to utilize the BLR as a model to
adapt a source model to the target and answer question two. To ensure that we
find the best adaption strategy, we directly apply a source model on the
target and consider the state-of-the-art such as weight decay as well as
Bayesian tuning [4]. Considering the two selection strategies, we evaluate
$18$ combinations making it the most extensive evaluation for transfer
learning (TL) in renewable power forecasts. Third, due to the BLR as the
output layer, we propose combining multiple source models through Bayesian
model averaging (BMA) to answer question three. Finally, we propose the
coopetitive soft gating ensemble (CSGE) as ensemble technique for ITL. The
source code is open accessible111https://github.com/scribbler00/DEELEA,
accessed 2022-07-15.

The remainder of this article is structured as follows. Section 2 describes
related work. The following Section 3 introduces relevant definitions and
details the proposed approach. We describe the datasets and discuss the
experiment’s most essential findings in Section 4. In the final Section 5, we
summarize our work and provide insights for future work.

##  2 Related work

In the following section, we overview recent developments for TL and, more
specifically for ITL in computer vision that has been not considered for
renewable power forecasts. This review determines relevant techniques that we
consider for renewable power forecasts. Afterward, we summarize related work
for ITL on deterministic renewable power forecasts.

There are two crucial dimensions in ITL. The first is the model selection and
the second is the adaptation strategy. The authors of [4, 5] provide a study
on selection strategies for the field of computer vision. They utilize a BLR
replacing the final layer of a source model and train it through empirical
Bayes, also referred to as evidence approximation, on the target data. The
authors repeat this approach for each available model from the model hub.
Finally, they determine the similarity through the evidence of a source model
on the target. As an adaptation strategy, they proposed Bayesian tuning, which
regularizes the fine-tuning process by predictions from multiple sources.
These proposed selections and adaptations need to be considered and extended
for renewable energies. For instance, we can directly forecast through the BLR
and compare it with fine-tuning of the final layer. Often, the adaptation
through fine-tuning is regularized by a weight decay regarding zero [8].
However, this regularizer does not consider parameters originating from the
source model. Therefore, in [9] a deviation from a source model is penalized
by weight decay considering the source model parameters.

Compared to research areas like computer vision, there is limited research on
ITL for renewable power forecasts, compare [6]. There has been some work to
learn a transferable representation of the input utilizing autoencoders [10,
11, 12, 13]. While the principle approach of transferring an autoencoder for a
target is combinable with our approach, we argue that considering the
conditional distribution of the power forecast is more relevant for model
selection and combination. The data driven TL approaches presented in [14] and
[15] are outside the scope of this article.

Also, most of the current research on TL in renewable power forecasts focuses
on meteorological measurements as input features, see [11, 12, 14, 16, 17, 18,
19, 20, 21, 22]. These articles consider forecast horizons between ten seconds
and two hours. At the same time, larger forecast horizons, such as day-ahead
forecasts, are inherently more difficult as they utilize NWP as input features
and forecast errors increase with an increasing forecast horizon [23].

Most of the previously mentioned related work for TL in renewable energies is
treating power forecast not as a time series approach. Instead, the authors
treat it as a regression. At the same time, periodic influences from, e.g.,
the diurnal cycle, are well known. Therefore, the authors of [24] consider
recurrent networks and fine-tuning to achieve good results for an ultra-short-
term forecast horizon of PV. Additionally, the article [3] considers time
series methods in a multi-task learning (MTL) architecture.

The authors of [25] achieve improvements in day-ahead PV forecasts through
multi-target models. The idea of model combination is similar to our ensemble
approach. However, the authors of this article do not evaluate it in the
context of ITL. The study of [26] proposes an MTL strategy for Gaussian
processes to forecast PV targets. By clustering wind parks, a weighting scheme
provides predictions for a new park in [27]. In this article, no actual
historical power measurements are used for evaluation; instead, the authors
used synthetic data.

A number of articles apply MTL architectures for TL [3, 28, 29] in day-ahead
forecasts. The proposed task embedding in [3] and [28] for MLPs and
convolutional neural networks (CNNs), encodes task-specific information
through an embedding to learn latent similarities between tasks. The article
[3] is especially interesting as we have a similar experimental set-up.

However, the authors look at errors per season and results can be misleading
as an ITL approach should avoid catastrophic forgetting for all seasons. Also,
MTL architectures are due to their additional training complexity not common
in the industry, e.g., due to data pre-processing and training complexity.
Therefore, for the extension of renewable energies making the best use of
existing single-task models for ITL is essential.

Furthermore, none of the related work studies different model selection and
model adaptation strategies for neural network architectures. While there has
been some work on combining knowledge from multiple sites for PV through
ensemble-like strategies, the studies are insufficient as they do not consider
the amount of available data. Also, the expected power is solely based on a
characteristic curve, or authors consider solely PV or wind data. We close
these research gaps by providing an extensive study that overcomes those
limitations for day-ahead forecasts.

##  3 Proposed Methods

The following sections define the proposed model selection, model adaptation,
and model combination strategies. Beforehand we introduce the BLR as it is the
basic model for one selection and one adaption strategy.

###  3.1 Bayesian Linear Regression

The following definitions of a BLR makes use of the introductions in [30] and
[31]. In contrast to a deterministic perspective to learn the model weights of
a linear regression model, a Bayesian approach gives additional insights
through the posterior, especially when there is insufficient data [30] as for
TL. It helps in measuring task similarity for model selection and allows
assessing the quality of the model in terms of its uncertainty, as proposed in
[4]. We propose to utilize it also for an adaptation for renewable forecasts.
We achieve this by replacing the final layer of a neural network with a BLR,
training it with the target data, and making predictions for the target
afterward. This approach additionally allows combining models through BMA, see
Section 3.4. Finally, we can utilize it to learn a BELM.

Therefore, let us assume that the following equation details the posterior
distribution of such a linear model:

$\underbrace{p(\parametersLinModel|\setInputFeatures,\setOutputFeatures)}_{\text{posterior}}=\frac{\overbrace{p(Y|X,\parametersLinModel)}^{\text{likelihood}}\overbrace{p(\parametersLinModel}^{\text{prior}})}{\underbrace{p(\setOutputFeatures|\setInputFeatures)}_{\text{marginal
likelihood}}},$ (1)

where $\setInputFeatures=\\{\inputFeatureVector_{n}\\}_{n=1}^{n=\nSamples}$
and $\setOutputFeatures=\\{\outputFeatureVector_{n}\\}_{n=1}^{n=\nSamples}$
are the sets of observed input and response values with
$\nSamples\in\naturalNumberPos$ samples from a training dataset. In this
setting, a single feature vector
$\inputFeatureVector_{n}\in\realNumber^{\dimFeature}$ has
$\dimFeature\in\naturalNumberPos$ features and $\outputFeatureVector_{n}$ is
of size $\realNumber$. Then the likelihood
$p(\setOutputFeatures|\setInputFeatures,\parametersLinModel)$ describes how
well $\setInputFeatures$ and the weights
$\parametersLinModel\in\realNumber^{D}$ describe the response values. Through
the prior, we encode our initial beliefs about the model weights. The marginal
likelihood normalizes the posterior. Finally, after observing training data,
the posterior encodes what we know about the target.

To calculate the distributions of the posterior of a linear regression model
consider that we have a prior over the weights $\parametersLinModel$ with
$p(\parametersLinModel|\alpha)=\normalDist(\parametersLinModel|0,\alpha^{-1}\mathbf{I})$,
where $\alpha\in\realNumberNonNeg$ is the precision of the zero mean isotropic
Gaussian distribution. Note, choosing an isotropic Gaussian distribution for
the prior allows deriving a closed form solution that reduces computational
effort for calculating the mean and covariance matrix, as detailed in the
following.

Now consider that we have a target $\outputFeatures$ of size $1\times N$ and
$\inputFeatures$ is the $N\times\dimFeature$ design matrix, where each row
corresponds to the $n$-th observation. For multivariate problems, we can train
one model per response. In our case, $\inputFeatures$ are either weather
predictions from an NWP such as wind speed or radiation, random features from
an BELM, or features extracted from a neural network at the second last layer.
In the latter two cases the weather predictions are transformed through the
neural network or the BELM.

Finally, the posterior distribution is given by $\outputFeatures$,
$\inputFeatures$, and the noise precision parameter
$\beta\in\realNumberNonNeg$ through
$p(\parametersLinModel|\inputFeatures,\outputFeatures)=\normalDist(\parametersLinModel|\mathbf{m}_{N},\precisionMatrix_{N})$,
where

$\mathbf{m}_{N}=\beta\precisionMatrix_{N}\inputFeatures^{T}\outputFeatures\text{~{}and~{}}\precisionMatrix_{N}^{-1}=\alpha\mathbf{I}+\beta\inputFeatures^{T}\inputFeatures.$
(2)

In this setting, $\nSamples$ indicates the number of training samples used to
update our prior beliefs of the model weights. In most cases, we are
interested in predicting an unknown response $\outputFeatures_{*}$ based on
input $\inputFeatures_{*}$ from a (test) dataset not seen during the training
of the model. Therefore, the predictive posterior is defined by:

$\displaystyle p(\inputFeatures_{*}|\outputFeatures,\alpha,\beta)$
$\displaystyle=\int
p(\inputFeatures_{*}|\outputFeatures,\parametersLinModel)p(\parametersLinModel|\inputFeatures,\outputFeatures)\text{d}\parametersLinModel$
$\displaystyle=\normalDist(y_{*}|\inputFeatures_{*}\mathbf{m}_{N},\bm{\sigma}_{N}^{2}(\inputFeatures_{*})),$
(3)

where $\outputFeatures$ and $\inputFeatures$ are from the training set and the
posterior variance is given by

$\bm{\sigma}^{2}_{N}(\inputFeatures_{*})=\beta^{-1}+\inputFeatures^{T}_{*}\precisionMatrix_{N}\inputFeatures_{*}.$
(4)

###  3.2 Model Selection for Inductive Transfer Learning

Measuring task similarity between a target task and multiple source tasks is a
critical challenge in ITL [4] as it allows selecting an appropriate source
task for knowledge transfer. Ideally, a valid model selection avoids negative
transfer, so utilizing knowledge from the source model has a minor error than
training a target model from scratch.

$\mathcal{D}$Evidence$\mathcal{D}_{T}$$p(\mathcal{D}|M_{1})$$p(\mathcal{D}|M_{2})$

Figure 2: Bayesian model selection adapted from [31].

Before we formalize the concept of model selection in the context of TL we
will provide an intuition behind (Bayesian) model selection in a broader
sense. In [30], it is argued that a model selection (outside the context of
TL) approach should find a trade-off between model complexity and the fit for
the data. This trade-off is visualized from a Bayesian perspective in Fig. 2.
On the horizontal axis, the space of all possible datasets is given. The
evidence of a model for a given dataset $\mathcal{D}_{T}$ is given on the
vertical axis. In this case, consider that model $M_{2}$ is a larger model
with more parameters than model $M_{1}$ and, therefore, can express a larger
number of datasets. We can see that with the model evidence
$p(\mathcal{D}|M_{\indexSourceModel})$ we would favor the simpler model for
dataset $\mathcal{D}_{T}$ through the Bayesian perspective. The concept that a
Bayesian perspective on model selection favors the simpler model is also known
as Occam’s razor.

The general concept of model selection is also valid in the context of TL. We
aim to find a source model, from $m\in\\{1,\ldots,M\\}$ source models with
$M\in\mathbb{N}_{>1}$, that explains the limited target data $T$ best.
Consider two tasks
$\task_{1}=\\{\outputFeatureSpace,~{}P_{1}(\setOutputFeatures_{1}\mid\setInputFeatures_{1})\\}$
and
$\task_{2}=\\{\outputFeatureSpace,~{}P_{2}(\setOutputFeatures_{2}\mid\setInputFeatures_{2})\\}$,
where the tasks $\task_{1},\task_{2}\in\allTasks$ and $\allTasks$ is the set
of all possible tasks. The sets $\setOutputFeatures_{m}$ and
$\setInputFeatures_{m}$ are from the response space $\outputFeatureSpace$ and
feature space $\inputFeatureSpace$. By defining a similarity measure
$\similarityMeasure$ with
$\similarityMeasure\colon\allTasks\times\allTasks\to\realNumberPos$, the
mapping into a scalar allows making quantitative statements. For instance,
given two source tasks $\task_{1},\task_{2}$ and a target task $\task_{T}$ if
$\similarityMeasure\big{(}\task_{1},\task_{T}\big{)}>\similarityMeasure\big{(}\task_{2},\task_{T}\big{)}$
then $\task_{1}$ is more similar to the target $\task_{T}$ compared to
$\task_{2}$, which means, that a high value implicates a high similarity.
Respectively, we define dissimilarity by the inverse of a similarity measure.

Now the question arises what (similarity) measure and what kind of data should
be considered to select a source model from a model hub for a specific target.
One choice would be to measure similarity solely based on the input feature
space. However, the input feature space contains limited information on the
expected power generation, the response variable, in renewable power
generation. For example, different amounts of energy will be produced with the
same radiation for different solar modules. Consequently, we need to take the
response variable into account.

####  3.2.1 Evidence or Marginal Likelihood:

The authors of [4] utilize the marginal likelihood or evidence as a similarity
measure $\similarityMeasure$. For that purpose, the final layer of a (source)
neural network $\task_{m}$ is replaced by a BLR, where the priors $\alpha$ and
$\beta$ of this model are optimized through empirical Bayes [5, 30] on limited
target data. In this way, the source model acts as a feature extractor. The
marginal likelihood is then given by

$\displaystyle\similarityMeasure(\task_{T},\task_{m})$ $\displaystyle=\ln
p(\outputFeatures|\alpha,\beta)$
$\displaystyle=\frac{D}{2}\ln\alpha+\frac{N}{2}\ln\beta-E(\mathbf{m}_{N})-\frac{1}{2}\ln{|\mathbf{S}^{-1}_{N}|-\frac{N}{2}\ln{2\pi}},$
(5)

where $D$ is the number of features, e. g., defined by the dimension of the
second last layer of the neural network $m$ with $N$ samples and
$E(\mathbf{m}_{n})=\frac{\beta}{2}\cdot||\outputFeatures-\inputFeatures\mathbf{m}_{n}||+\frac{\alpha}{2}\mathbf{m}_{n}^{T}\mathbf{m}_{n}$.
This way, we consider features extracted from the source neural network of
task $\task_{m}$ and the response feature from the target $\task_{T}$. If we
do this for each source model $m$, we can calculate the marginal likelihood of
each source model on the target to calculate
$\similarityMeasure(\task_{m},\task_{T})$. We then select the model with the
most extensive evidence as the appropriate source model. We repeat this for
each dimension for multivariate problems and average the results [4].

While this approach is theoretically appealing and generalizes for a broad
number of problems, it has one drawback in the context of ITL: It is not
considering already learned weights from the final layer of a model. This
consideration is essential, as we often do not need to remove the final layer
to assure compatibility between a source and a target task in renewable
energies. At the same time, a pre-trained layer is usually beneficial in
contrast to a randomly initialized layer.

####  3.2.2 Normalized Root Mean-squared Error:

Respectively, we propose to directly measure the similarity through the nRMSE
based on the pre-trained layer of a source model by

$\similarityMeasure(\task_{T},\task_{m})^{-1}=\text{nRMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{i=N}{(y_{i}^{(T)}-\hat{y}_{i}^{(m)})^{2}}},$
(6)

where $y_{i}^{(T)}$ is the $i$-th normalized response from the target and
$\hat{y}_{i}^{(m)}$ is the prediction from source model $m$ on the target.
Note that in the context of renewable power forecasts, we normalize the
response $y_{i}$ by the nominal power to assure comparability of the error for
different parks. We can directly measure how well a source model performs on
the available target data to measure the similarity
$\similarityMeasure(\task_{m},\task_{T})$. Consequently, we can select the
source model with a lower nRMSE on, e.g., a validation error from the target
data.

###  3.3 Adaptation Strategies for Inductive Transfer Learning

Table 1 overviews all $18$ combinations of models and adaptation strategies.
As a simple TL model, we consider an online update of the posterior of the
BELM. Therefore, the posterior from a source model acts as a prior for the
target. Additionally, we evaluate directly applying a selected source model on
the target without any adaptation of a source model’s parameter.

We also consider two standard fine-tuning methods from the field of computer
vision. The first one is weight decay which penalizes the deviation of weights
from zero and weight decay source which penalizes a deviation from the source
model’s weights. Additionally, we examine Bayesian tunning as introduced in
[5].

Table 1: Overview of different combinations for models, selections, and adaptations. RM abbreviates the RMSE selection strategy, EV the selection through evidence, DI stands for directly applying the model, WD for fine-tuning through weight decay regarding the origin, WDS for a fine-tuning through weight decay regarding the source parameters, BT for fine-tuning with Bayesian tuning. Model | Selection | Adaptation | Abbreviation
---|---|---|---
Type | Strategy | Strategy |
MLP/TCN | RMSE [ours] | direct [ours] | MLP-/TCN-RM-DI
MLP/TCN | RMSE [ours] | weight decay [8] | MLP-/TCN-RM-WD
MLP/TCN | RMSE [ours] | weight decay source [9] | MLP-/TCN-RM-WDS
MLP/TCN | EVIDENCE [5] | direct [ours] | MLP-/TCN-EV-DI
MLP/TCN | EVIDENCE [5] | direct linear [ours] | MLP-/TCN-EV-DILI
MLP/TCN | EVIDENCE [5] | weight decay [8] | MLP-/TCN-EV-WD
MLP/TCN | EVIDENCE [5] | weight decay source [9] | MLP-/TCN-EV-WDS
MLP/TCN | EVIDENCE [5] | Bayesian tuning [5] | MLP-/TCN-EV-BT
BELM | RMSE [ours] | Online [ours] | BELM-RM
BELM | EVIDENCE [5] | Online [ours] | BELM-EV

The three latter adaption strategies can be considered as a type of
regularization. In general, this means that we add an additional penalty term
$L_{pen}$ to the loss function $L_{task}$ of a task through

$L=L_{task}+\lambda\cdot L_{pen},$ (7)

where $\lambda\in\mathcal{R}$ is a hyper-parameter for the amount of
regularization that is typically selected by hyper-parameter optimization.
$L_{task}$ is given by

$L_{task}=\frac{1}{\nSamples}\sum^{\nSamples}_{\indexSamples=1}{l(f(\inputFeatureVector_{\indexSamples},\bm{\theta}),\outputFeatureVector_{\indexSamples})},$
(8)

where $\parametersLinModel\in\realNumber^{p}$ and $p\in\naturalNumberPos$ is a
vector of the parameters we update, $\inputFeatureVector_{\indexSamples}$ is
the $\indexSamples$-th input vector with $\indexSamples\in\nSamples$ and
$\nSamples\in\naturalNumberPos$, and $\outputFeatureVector_{\indexSamples}$ is
the respective response. For simplicity, we consider a uni-variate response
here. For weight decay (WD) with respect to the origin [8], $L_{pen}$ is then
given by

$L_{WD}=\frac{1}{2}||\bm{\theta}||^{2}_{2}$ (9)

To penalizes a deviation from the source model, [9] proposes a weight decay
w.r.t. to source weights (WDS) given by

$L_{WDS}=\frac{1}{2}||\bm{\theta}-\bm{\theta}^{0}||^{2}_{2},$ (10)

where $\parametersLinModel^{0}\in\realNumber^{p}$ is the vector of parameters
from the source model before fine-tunning. Finally, in Bayesian tunning
$L_{pen}$ is given by [5]:

$L_{Bayesian}=\frac{1}{\nSamples}\sum^{\nSamples}_{\indexSamples=1}{\frac{1}{K}\sum^{K}_{k=1}}{(\frac{1}{M}\sum^{M}_{\indexSourceModel=1}{\inputFeatureVector_{\indexSourceModel,\indexSamples}^{T}\parametersLinModel_{\indexSourceModel,k}-\inputFeatureVector_{t,\indexSamples}^{T}\parametersLinModel_{t,k}})^{2}},$
(11)

where $\indexSamples\in\nSamples$ is the $\indexSamples$-th data sample,
$\indexSourceModel$ is the $\indexSourceModel$-th source model adapted with
BLR, $k$ is the $k$-th dimension of the response for example for different
forecast horizons. $\inputFeatureVector_{\indexSourceModel,k}$ are features
extracted from the $\indexSourceModel$-th source model,
$\inputFeatureVector_{t,\indexSamples}$ are the respective features extracted
from the target model $t$. $\parametersLinModel_{k,c}$ and
$\parametersLinModel_{t,c}$ are the mean vectors calculated by the BLR.

###  3.4 Model Combination for Inductive Transfer Learning

We discussed the model selection and adaptation strategies for a single source
model for a target. However, a single model might be prone to overfitting with
limited data. Combining source models through an ensemble reduces this risk.

####  3.4.1 Bayesian Model Averaging

We extend the concept of [4] so that instead of choosing a single model based
on the evidence, we combine models adapted through BLR by BMA. BMA is
theoretically appealing as it considers the predictive posterior [32] and
therefore considers the uncertainty of a model through

$p(\outputFeatures_{T*}|\outputFeatures_{T})=\sum_{i=1}^{i=M}p(\outputFeatures_{T*}|\outputFeatures_{T},\parametersLinModel_{M_{i}})p(\parametersLinModel_{M_{i}}|\outputFeatures_{T}).$
(12)

The prior probability $p(\parametersLinModel_{M_{i}}|\outputFeatures_{T})$
encodes our prior belief that a model $M_{i}$ is more likely to another one
for the target data set. For simplicity, we consider an equal prior for all
source models. Note that we have omitted the input here to simplify notations.
$p(\outputFeatures_{T*}|\outputFeatures_{T},\parametersLinModel_{M_{i}})$ is
the predictive posterior of a model $M_{i}$ given by Eq. 3.1, where, e.g, the
model results from the proposed direct linear adaptation strategy.

####  3.4.2 Coopetitive Soft Gating Ensemble:

We also propose to utilize the CSGE for model combination in the context of
ITL. The CSGE was initially introduced for renewable power forecast in [33].
The idea of the CSGE is to link the weights to the ensemble members’
performance, i.e., good source models are weighted stronger than weaker ones.
The CSGE characterizes the overall weight of a source model using three
aspects:

  * •

The global weight is defined by how well a source model performs with the
available training data on the target task.

  * •

The local weight is defined by how well a source model performs on the target
tasks for different areas in the feature space. For example, in the case of
wind, one model might perform well for low wind speeds, while another source
model might perform well for larger wind speeds on the target.

  * •

The forecast horizon-dependent weight is defined by how well a source model
performs for different lead times on the target task. Here, between $24$ and
$48$ hours into the future.

Compute Weights Normalize Weights Aggregate Forecasts Global Local Forecast
Horizon Source Model (1) Source Model (2) … Source Model (M)
$\inputFeatureVector_{t+k|t}$$\bar{\outputPredictionVector}_{t+k|t}$$\outputPredictionVector^{(1)}_{t+k|t}$$\outputPredictionVector^{(2)}_{t+k|t}$$\outputPredictionVector^{(\numberSourceModels)}_{t+k|t}$$\csgeWeightUnnormalized^{(1)}_{t+k|t}$$\csgeWeightUnnormalized^{(2)}_{t+k|t}$$\csgeWeightUnnormalized^{(\numberSourceModels)}_{t+k|t}$$\csgeWeightNormalized^{(1)}_{t+k|t}\cdot\outputPredictionVector^{(1)}_{t+k|t}$$\csgeWeightNormalized^{(2)}_{t+k|t}\cdot\outputPredictionVector^{(2)}_{t+k|t}$$\csgeWeightNormalized^{(\numberSourceModels)}_{t+k|t}\cdot\outputPredictionVector^{(\numberSourceModels)}_{t+k|t}$
… … …

Figure 3: The architecture of the CSGE. The source models’ predictions
$\outputPredictionVector^{(\indexSourceModel)}_{t+k|t}$ for the input
$\inputFeatureVector$ are passed to the CSGE. The ensemble member’s weights
are given by aggregating the respective global-and local- and forecast
horizon-dependent weights. The weights are normalized. The source models’
predictions are weighted and aggregated in the final step.

Fig. 3 provides an overview of the CSGE. The CSGE includes
$\numberSourceModels$ ensemble members, with
$\indexSourceModel\in\\{1,\ldots,\numberSourceModels\\}$. Each of these
ensemble members is a source model with a predictive function
$f_{\indexSourceModel}$. Each source model forecasts an univariate estimate
$\hat{y}^{(\indexSourceModel)}_{t+k|t}\in\mathbb{R}$ for the input
$\inputFeatureVector_{t+k|t}\in\mathbb{R}^{\dimFeature}$ of a target task
$\indexTargetTask$, we omit the subscript $\indexTargetTask$ for reasons of
clarity and comprehensibility. Let $\dimFeature$ be the dimension of the input
feature vector $\inputFeatureVector$. Then, $k$ denotes the forecast horizon,
denoted by the subscript, for the forecast origin $t$. For each prediction of
each source model, we compute an aggregated weight
$\csgeWeightNormalized^{(\indexSourceModel)}_{t+k|t}$.

The weight incorporating the global $w_{g}$, local $w_{l}$, and forecast
horizon-dependent weight $w_{h}$ for a single source model and lead time is
given by

$\displaystyle\csgeWeightUnnormalized^{(\indexSourceModel)}_{t+k|t}=\csgeWeightNormalized^{(\indexSourceModel)}_{g}\cdot\csgeWeightNormalized^{(\indexSourceModel,t)}_{l}\cdot\csgeWeightNormalized^{(\indexSourceModel,k)}_{h},$
(13)

where $\csgeWeightUnnormalized^{(\indexSourceModel)}_{t+k|t}$ is normalized to
sum up to one to calculate
$\csgeWeightNormalized^{(\indexSourceModel)}_{t+k|t}$.

To calculate the weights
$\csgeWeightNormalized^{(\indexSourceModel)}_{t+k|t}$, we utilize the
definition of the inverse similarity measurement $\similarityMeasure^{-1}$
from Section 3.2 and the coopetitive soft gating principle from Eq. 14.

$\displaystyle\varsigma^{{}^{\prime}}_{\eta}(\mathbf{\Phi},\phi)=\frac{\sum_{j=1}^{J}\mathbf{\Phi}_{j}}{\phi^{\eta}+\epsilon}$
(14)

By calculating the weighting through the inverse $\similarityMeasure^{-1}$,
here the nRMSE, we estimate how well a source model performs on the target.
Let us assume that $\mathbf{\Phi}\in\realNumber^{J}$ contains all
$J\in\naturalNumberPos$ estimates based on the nRMSE and
$\phi\in\mathbf{\Phi}$. Then, $\eta\geq 0$ depicts the amount of exponential
weighting and the small constant $\epsilon>0$ avoids division by zero. For
greater $\eta$, the CSGE tends to work as a gating ensemble, thereby
considering only a few source models. For smaller $\eta$ result in a weighting
ensemble. After calculating all weights from $\mathbf{\Phi}$ through Eq. 14,
we normalize the results to sum up to one estimating the final weights
$\csgeWeightNormalized^{(\indexSourceModel)}_{t+k|t}$. This approach is
repeated for each of the three weighting aspects as detailed in [33] and B.

##  4 Experimental Evaluation

In the following Section 4.1, we summarize the experimental setup. We conduct
experiments on six datasets with a total of $667$ parks. Due to the utilized
cross-validation, each park is once a target park. Thereby, we provide the
most extensive study for ITL for renewable power forecasts.

We evaluate models through the mean performance rank, calculated across parks
within a dataset, to show significant improvements against the baseline . In
the appendix, we provide additional results concerning the forecast error and
comparison to results from [3]. Section 4.2 provides the details of our first
experiment to answer research questions one and two. The second experiment in
Section 4.3 details our findings for research question three.

###  4.1 Overall Experimental Setup

The preprocessing of the data is aligned with [7, 3] to assure comparability
with the current state of the art. As source models, we considered a BELM,
MLP, and a TCN. To have a robust baseline that generalizes well with a limited
amount of data and is known to mitigate the effects of overfitting, we trained
a GBRT for each target task identical to [7].

####  4.1.1 Datasets:

We conducted all experiments for day-ahead forecasts, between $24$ and $48$
hours into the future. All datasets, summarized in Table 2, have NWP features
as inputs, e.g., wind speed, wind direction, air pressure, or radiation. For
all datasets, we align those weather forecasts with the historical power
measurements as the response for day-ahead forecasts. These input features are
weather forecasts from the European center for medium-range weather forecasts
(ECMWF) or the Icosahedral Nonhydrostatic-European Union (ICON-EU) weather
model.

Table 2: Overview of the evaluated datasets. Dataset | #parks | #features | train | mean | resolution | NWP
---|---|---|---|---|---|---
|  |  | samples | samples |  | model
PVOPEN  [3] | 21 | 47 | 6336 | 8424 | hourly | ECMWF
PVSYN  [7] | 114 | 20 | 30385 | 14920 | 15-min | ICON-EU
PVREAL | 42 | 25 | 58052 | 19344 | 15-min | ICON-EU
WINDOPEN  [3] | 45 | 13 | 27724 | 26636 | 15-min | ECMWF
WINDSYN  [7] | 260 | 29 | 33714 | 16678 | 15-min | ICON-EU
WINDREAL | 185 | 33 | 36129 | 12092 | 15-min | ICON-EU

In all datasets, we have varying amounts of input features, resolutions, and
different numbers of samples for training and testing. For instance, the
PVOPEN has $47$ features, where various manually engineered features take
seasonal patterns of the sun into account. In contrast, these manually
engineered features are not included in other datasets.

Also note that four datasets, the PVOPEN, WINDOPEN, WINDSYN, and PVSYN have
already been investigated, see e.g. [3, 7]. This is not the case for WINDREAL
and PVREAL. These two datasets are not publicly available. However, they are
the most realistic datasets due to their diversity. The WINDREAL dataset
consists of 99 different nominal capacities, 13 turbine manufacturers, and six
hub heights. All parks are located in Germany. PV power plants in the PVREAL
dataset have 31 different nominal capacities, ten tilt orientations, and nine
azimuth orientations and are also located in Germany. It is also important to
note that forecasting the expected power generation from wind parks is more
challenging than for PV parks. For additional insights on the challenges and
the datasets refer to the A.

Each dataset was split through five-fold cross-validation so that each park is
once a target task and four times a source park. We trained source models and
their hyperparameters on the training and validation data. We split the
training into the four seasons for training target models and limited the
training data to $7,14,30,60$ or $90$ days of training data, respectively. The
presented results are mean values for all tasks and seasons. This setup
assures that results are not biased by seasonality [23]. All input features of
all datasets are normalized. We normalized the historical power by the nominal
power to make errors comparable. We resampled all datasets to have a
$15$-minute resolution except the PVOPEN dataset, which we resampled for an
hourly resolution due to the low initial resolution. A predefined test set is
given for the WINDSYN and PVSYN datasets. In the case of the WINDOPEN and
PVOPEN, we used the first year’s data as training data and the remaining data
as test data, identical to [3]. Due to this diversity in the number of
historical power measurements for the PVREAL and WINDREAL datasets, $25\%$
randomly sampled days are considered test data. As each day is based on an
independent day ahead NWP forecasts, no information is leaked from the future
to the past [34]. We use $25\%$ of the remaining days for validation and the
rest for training.

####  4.1.2 Source Models

As pointed out earlier, due to the weather dependency for renewable power
forecasts, the input features of the models are themselves forecasts from the
NWP model. Respectively, we can directly utilize those to train, e.g., an MLP
to forecast the expected power of the next day. To optimize hyperparameters of
those models, we utilize a tree-structured Parzen sampler for $200$ samples on
the validation data. Details of the chosen hyperparameters are provided in the
C.

In total, we train four kinds of models. The trained BELM is particularly
interesting as a source model because it can directly measure similarity by
the evidence and has a linear increase in time for updating the model. We
train an MLP as it is common practice in the renewable power forecast industry
[3]. To account for cyclic behavior within the forecast, we also train a TCN
architecture, similar to [3]. To have a strong baseline that generalizes well
we trained a GBRT [7].

####  4.1.3 Evaluation Method

We calculated the error on the test dataset through the nRMSE through Eq. 6
for all combinations of seasons and available training data. For a given
dataset, season, and the number of days of training data, we calculated the
mean performance rank based on the nRMSE. We test for a significant
improvement compared to the baseline by the Wilcoxon test ($\alpha=0.01$)
across all parks within a dataset.

Table 3: Rank summary for all models, selections, and adaptation strategies on
the PV datasets, cf. Table 1. Only those within the top four ranks for a
dataset are included. GBRT is the baseline and all models are tested if the
forecasts error is significantly ($\alpha=0,01$) better ($\vee$), worse
($\wedge$), or not significantly different ($\diamond$). We conduct this
hypothesis test for all parks within a dataset for the given number of days
training data. The colors denote the respective rank. Blue indicates a smaller
(better) rank and red a higher (worse) rank. Table 4: Rank summary for all
source models, selections, and adaptation strategies on the wind datasets. Cf.
Tables 1 and 3. Table 5: Rank summary of ensembles on the PV datasets. The
best model, the TCN-EV-DILI, from the experiment in Section 4.2 is the
baseline. Cf. Tables 1 and 3. Table 6: Rank summary of ensembles on the wind
datasets. Cf. Table 5. Table 7: Mean nRMSE of ensembles on the PV datasets.
The best model, the TCN-EV-DILI, from the experiment in Section 4.2 is the
baseline. Cf. Tables 1 and 3. Table 8: Mean nRMSE of ensembles on the wind
datasets. Cf. Table 7.

###  4.2 Experiment on Model Selection and Model Adaptation

This section conducts an experiment to answer research questions one and two
simultaneously as a model selection technique can only be evaluated after the
adaptation:

######  Research Question 1.

What is an appropriate similarity measure for model selection for a new target
park from a model hub with pre-trained models?

######  Research Question 2.

What is the best adaptation strategy once a model is selected?

####  4.2.1 Findings Questions 1 & 2

Model selection and adaptation strategies highly influence each other. With
limited training data, between $7$ and $30$ days, selecting a model based on
the forecast error with no adaptation has one of the best results. Replacing
the final layer by a BLR is superior with additional data. None of the fine-
tuning methods is among the best models.

####  4.2.2 Experimental Setup

The source models are those detailed in Section 4.1.2. As adaptation
strategies, we consider those mentioned in Section 3.3. For fine-tuning, we
train for a single epoch and optimize hyperparameters through grid search on
$30\%$ of the available target data.

For the weight decay adaptation, we optimize seven logarithmically spaced
learning rates between $10^{-1}$ and $10^{-4}$, similar to [4]. We take seven
grid points for the amount of penalty $\lambda$ in the logarithmic space
between $10^{-6}$ to $10^{-3}$, similar to [4]. We use the same learning rate
for the Bayesian tunning and weight decay source. The amount of penalty
$\lambda$ for the $L_{\text{Bayesian}}$ loss is one of
$[0.1,0.25,0.5,1,2,4,8]$. $\lambda$ is one of $[1,0.1]$ for the weight decay
source. Note that during the training, we shuffle the data. For other
approaches, hyperparameter optimization is not required.

####  4.2.3 Detailed Findings

Results of the best techniques are summarized in Table 3 and 4. We only show
models appearing at least once within the top four ranks for a dataset. The
BELM is among the best models and outperforms the baseline up to $30$ days of
training data. With less or equal than $14$ days of training data, it seems
beneficial to directly utilize a model without any model adaptation. Starting
with $30$ days of training data utilizing a BLR trained on extracted features
from the source model and the historical power from the target is beneficial,
especially for WINDREAL and PVREAL. This effect occurs as features extracted
from a single model from a single prediction task do not generalize well
enough for other parks. Therefore a sufficient amount of data is required to
train the BLR to compensate for differences between a source and a target
park.

These two observations let us conclude with two critical considerations for
real-world applications. First, due to the learning procedure of the gradient
descent, there is a high risk of catastrophic forgetting that should be
avoided for model hubs in safety-critical areas such as renewable power
forecasts. Second, the BLR gives rise to optimal training due to the convex
optimization problem, which reduces the risk of catastrophic forgetting.
Notably, neither a weight decay nor the Bayesian tuning adaptation strategy is
within the best models in the evaluated scenarios. This observation is
surprising as this fine-tuning approach is common in various domains. However,
due to the source model’s training on a single park approach, there is a high
risk that even the best-selected source model causes catastrophic forgetting
as the model is too specific. For instance, catastrophic forgetting may appear
due to slightly different weather conditions or physical characteristics such
as the turbine type.

An additional study given in the C.8 shows that for fine-tuning techniques,
the evidence selection strategy is superior for the TCN model regardless of
the adaptation strategy. For the MLP, a selection through the nRMSE is
preferable. Most likely, the probabilistic approach of the evidence and,
therefore, the more comprehensive treatment of similarity better captures the
correlations between source and target for the convolutional layers in the
TCN. To update the weights of the final layer of the MLP the nRMSE is
sufficient.

###  4.3 Experiment on Model Combination

In this section, we conduct an experiment to answer research question three.

######  Research Question 3.

Compared to selecting and adapting a single model, are ensemble strategies
beneficial for combining knowledge?

####  4.3.1 Findings Research Question 3

Ensembles improve results from the previous experiment significantly. For more
straightforward problems, an approach utilizing BMA is preferable. For more
complex scenarios, an approach by the CSGE is superior.

####  4.3.2 Experimental Setup

The best model, the TCN-EV-DILI, from the experiment in Section 4.2 is the
baseline . For the BMA, we first update all source models based on available
target data as previously described. For the MLP and TCN source models we
replace the final layer through BLR model(s), as detailed for the direct
linear adaptation. After this adaptation for the target, each model provides a
predictive posterior distribtion according to Eq. 3.1 that is combined by BMA
with Eq. 12. We consider three variants for the BMA, one for each of the three
source model types.

For the CSGE, we calculate the global and forecast horizon-dependent error
based on the nRMSE. We estimate the local error through a k-nearest neighbor
approach. Therefore, we first reduce the dimension of the feature space
through principal component analysis (PCA) to two components. We consider
three neighbors within this reduced features space to estimate the local error
in the features space. The hyperparameter $\eta$ is selected as either one or
two through grid-search. In total, we consider six variants of the CSGE: Two
for the MLP and TCN model where the source models are not updated for the
target (CSGE-MLP-DI/CSGE-TCN-DI), two variants, where the final layer of the
MLP and TCN source models are updated through BLR (CSGE-MLP-DILI/CSGE-TCN-
DILI), and these two variants are extended, where we utilize the GBRT as an
additional source model (CSGE-MLP-DILI-GBRT/CSGE-TCN-DILI-GBRT).

####  4.3.3 Detailed Findings

Results are summarized in Tables 5 and 6. For the PV datasets, the best CSGE
variants outperform the baseline in almost all cases. At the same time, the
BMA achieves excellent results for the PVOPEN dataset and the PVREAL dataset
for up to 30 days of training data. With minimal data, less than 30 days, the
BMA is among the best for the wind datasets. With more training data, the CSGE
with TCN source models, where the final layer is replaced by BLR is the best.
We can also observe for these datasets that additionally considering the GBRT
as the source model improves the results.

This observation also shows the flexibility of the CSGE. Due to the
combination through the forecast error, we can combine arbitrary models. This
flexibility is not given by the proposed BMA approach. However, the BMA has
the advantage that probabilistic forecasts are provided, which are not the
focus of this article, but their evaluation is given in the appendix.

Another important consideration is that in almost all cases, the ensemble
techniques outperform the baseline, which is the best model from the previous
experiment. These results show that a single source model’s selection and
adaptation process is highly uncertain because the model may be too specific
for the target. Due to specific characteristics of a single model, for
example, the weather at the location or technical characteristics, selecting
and adapting a single source model for the target is challenging. In contrast,
the combination of several models probably has a balancing effect on
individual properties and improves the error significantly.

Besides the previous statistical discussion through the mean performance
ranking, we also need to include an analysis of the forecast error for real-
world implications. Therefore, the forecast error, by means of the mean nRMSE,
is summarized for this experiment in Tables 7 and 8. The best model from the
previous experiment is again the baseline.

In these tables, the error of the models decreases with increasing training
data amount for all six datasets. For the PV datasets, the best model has the
largest error for the PVREAL dataset. The best forecast error for this dataset
is only $9.8$ percent with seven days of training data. For the PVOPEN with
seven days of training data, the error is with $7$ percent error rate lower
than results from [3].

Also, for the WINDOPEN dataset, the best models have similar error rates,
between $16.3$ for seven days and $13.8$ percent for $90$ days of training
data, similar to the result in [3] The WINDSYN dataset has the largest errors,
between $16.7$ and $14.6$ percent, for the wind datasets.

Based on the analysis of the nRMSE, we can observe that even with a small
amount of training data, good up to excellent prediction quality can be
achieved. Furthermore, the mean nRMSE with more than 30 days often corresponds
to error rates with a whole year of training data [3, 7].

##  5 Conclusion and Future Work

We successfully evaluated several combinations of models, model selection,
adaptation strategies, and two combination strategies on six datasets. Our
study’s exhaustive evaluation is the most extensive evaluation for transfer
learning in renewable power forecasts on real-world datasets. We found that
fine-tuning the final layer of a neural network, a well-known strategy, does
not lead to convincing results in this setting. Instead, replacing the layer
with a Bayesian linear regression model trained with features extracted from
the source and limited power measurements from the target task yields one of
the best results, especially for a temporal convolutional neural network. This
result is best explained as source models in computer vision tasks are
typically trained on many variations, e.g., various classification tasks. In
contrast, renewable energy models are often trained on a single forecasting
task. This approach seems to generalize insufficiently for fine-tuning. We
suggest utilizing the forecast error with less than 30 days of training data
for source model selection; the evidence is recommended with additional data.
We also showed how combining models leads to further significant improvements
compared to considering a single model. The proposed coopetitive soft-gating
ensemble combines source models solely based on the error of the target. Our
suggestion to utilize the Bayesian model averaging as an ensemble strategy is
beneficial for minimal historical data. To overcome the shortcomings of fine-
tuning caused by limited data, we aim to augment the target data with
synthetic data in the future. Likewise, we will expand our analysis for multi-
task problems.

Acknowledgments This work results from the project TRANSFER (01IS20020B)
funded by BMBF (German Federal Ministry of Education and Research). We thank
Enercast GmBH for providing the PVREAL and WINDREAL datasets. We also thank
Marek Herde, Mohammad Wazed Ali, and David Meier for their valuable input.

## References

  * [1] J. Schreiber, Transfer Learning in the Field of Renewable Energies - A Transfer Learning Framework Providing Power Forecasts Throughout the Lifecycle of Wind Farms After Initial Connection to the Electrical Grid, in: Organic Computing - Doctoral Dissertation Colloquium, kassel university press GmbH, 2019, pp. 75–87.
  * [2] R. Schwartz, J. Dodge, N. A. Smith, et al., Green AI, CoRR (2019) 1–12ArXiv: 1907.10597.
  * [3] J. Schreiber, S. Vogt, B. Sick, Task Embedding Temporal Convolution Networks for Transfer Learning Problems in Renewable Power Time-Series Forecast, in: ECML, 2021, pp. 1–16.  doi:10.1007/978-3-030-86514-6_8.
  * [4] K. You, Y. Liu, J. Wang, et al., LogME: Practical Assessment of Pre-trained Models for Transfer Learning, in: ICML, 2021, pp. 12133–12143.  arXiv:2102.11005.
  * [5] K. You, Y. Liu, J. Wang, et al., Ranking and Tuning Pre-trained Models: A New Paradigm of Exploiting Model Hubs, CoRR arXiv:2110.10545 (2021) 1–45.  arXiv:2110.10545, doi:10.48550/arXiv.2110.10545.
  * [6] G. Alkhayat, R. Mehmood, A review and taxonomy of wind and solar energy forecasting methods based on deep learning, Energy and AI 4 (2021) 100060.  doi:10.1016/j.egyai.2021.100060.
  * [7] S. Vogt, J. Schreiber, Synthetic Photovoltaic and Wind Power Forecasting Data, CoRR arXiv:2204.00411 (2022).
  * [8] H. Li, P. Chaudhari, H. Yang, et al., Rethinking the Hyperparameters for Fine-tuning, in: ICLR, 2020, pp. 1–20.  arXiv:2002.11770.
URL http://arxiv.org/abs/2002.11770

  * [9] X. Li, Y. Grandvalet, F. Davoine, Explicit inductive bias for transfer learning with convolutional networks, in: ICML, Vol. 6, 2018, pp. 4408–4419.  arXiv:1802.01483.
  * [10] A. S. Qureshi, A. Khan, Adaptive transfer learning in deep neural networks: Wind power prediction using knowledge transfer from region to region and between different task domains, Computational Intelligence 35 (4) (2019) 1088–1112.  doi:10.1111/coin.12236.
  * [11] X. Liu, Z. Cao, Z. Zhang, Short-term predictions of multiple wind turbine power outputs based on deep neural networks with transfer learning, Energy 217 (2021) 119356.  doi:10.1016/j.energy.2020.119356.
  * [12] Y. Ju, J. Li, G. Sun, Ultra-Short-Term Photovoltaic Power Prediction Based on Self-Attention Mechanism and Multi-Task Learning, IEEE Access 8 (2020) 44821–44829.  doi:10.1109/access.2020.2978635.
  * [13] J. Henze, J. Schreiber, B. Sick, Representation Learning in Power Time Series Forecasting, in: Deep Learning: Algorithms and Applications, Springer, Cham, 2020, pp. 67–101.  doi:10.1007/978-3-030-31760-7_3.
  * [14] L. Cao, L. Wang, C. Huang, X. Luo, J.-H. Wang, A Transfer Learning Strategy for Short-term Wind Power Forecasting, in: Chinese Automation Congress, 2018, pp. 3070–3075.  doi:10.1016/j.renene.2015.06.034.
  * [15] L. Cai, J. Gu, J. Ma, et al., Probabilistic wind power forecasting approach via instance-based transfer learning embedded gradient boosting decision trees, Energies 12 (1) (2019) 159.  doi:10.3390/en12010159.
  * [16] Y. Liu, J. Wang, Transfer learning based multi-layer extreme learning machine for probabilistic wind power forecasting, Applied Energy 312 (2022) 118729.  doi:10.1016/J.APENERGY.2022.118729.
  * [17] J. Chen, Q. Zhu, H. Li, L. Zhu, D. Shi, Y. Li, X. Duan, Y. Liu, Learning Heterogeneous Features Jointly: A Deep End-to-End Framework for Multi-Step Short-Term Wind Power Prediction, IEEE Transactions on Sustainable Energy 11 (3) (2020) 1761–1772.  doi:10.1109/TSTE.2019.2940590.
  * [18] H. Sheng, B. Ray, J. Shao, D. Lasantha, N. Das, Generalization of solar power yield modelling using knowledge transfer, Expert Systems with Applications (2022) 116992doi:10.1016/J.ESWA.2022.116992.
  * [19] H. Yin, Z. Ou, J. Fu, Y. Cai, S. Chen, A. Meng, A novel transfer learning approach for wind power prediction based on a serio-parallel deep learning architecture, Energy 234 (2021) 121271.  doi:10.1016/J.ENERGY.2021.121271.
  * [20] M. Khan, M. R. Naeem, E. A. Al-Ammar, W. Ko, H. Vettikalladi, I. Ahmad, Power Forecasting of Regional Wind Farms via Variational Auto-Encoder and Deep Hybrid Transfer Learning, Electronics 11 (2) (2022) 206.  doi:10.3390/ELECTRONICS11020206.
  * [21] G. Almonacid-Olleros, G. Almonacid, D. Gil, J. Medina-Quero, Evaluation of Transfer Learning and Fine-Tuning to Nowcast Energy Generation of Photovoltaic Systems in Different Climates, Sustainability 14 (5) (2022) 3092\.  doi:10.3390/SU14053092.
  * [22] C. Yan, Y. Pan, C. L. Archer, A general method to estimate wind farm power using artificial neural networks, Wind Energy 22 (11) (2019) 1421–1432.  doi:10.1002/WE.2379.
  * [23] J. Schreiber, A. Buschin, B. Sick, Influences in Forecast Errors for Wind and Photovoltaic Power: A Study on Machine Learning Models, in: INFORMATIK 2019, GI e.V., 2019, pp. 585–598.  doi:10.18420/inf2019\\_74.
  * [24] S. Zhou, L. Zhou, M. Mao, et al., Transfer learning for photovoltaic power forecasting with long short-term memory neural network, in: International Conference on Big Data and Smart Computing (BigComp), 2020, pp. 125–132.  doi:10.1109/BigComp48618.2020.00-87.
  * [25] M. Ceci, R. Corizzo, F. Fumarola, et al., Predictive Modeling of PV Energy Production: How to Set Up the Learning Task for a Better Prediction?, IEEE TIL 13 (3) (2017) 956–966.  doi:10.1109/TII.2016.2604758.
  * [26] T. Shireen, C. Shao, H. Wang, et al., Iterative multi-task learning for time-series modeling of solar panel PV outputs, Applied Energy 212 (2018) 654–662.  doi:10.1016/j.apenergy.2017.12.058.
  * [27] S. Tasnim, A. Rahman, A. Oo, et al., Wind power prediction in new stations based on knowledge of existing Stations: A cluster based multi source domain adaptation approach, Knowledge-Based Systems 145 (2018) 15–24.  doi:10.1016/j.knosys.2017.12.036.
  * [28] S. Vogt, A. Braun, J. Dobschinski, et al., Wind Power Forecasting Based on Deep Neural Networks and Transfer Learning, in: Wind Integration Workshop, Vol. 18, 2019, p. 8.
  * [29] J. Schreiber, B. Sick, Emerging Relation Network and Task Embedding for Multi-Task Regression Problems, in: ICPR, 2020, pp. 2663–2670.  doi:10.1109/ICPR48806.2021.9412476.
  * [30] C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
  * [31] M. Borthwick, Math for Machine Learning, Cambridge University, 2019.  doi:10.1515/9781400843909-001.
  * [32] J. A. Hoeting, D. Madigan, A. E. Raftery, et al., Bayesian model averaging: a tutorial, MathSciNet 14 (4) (1999) 382–417.  doi:10.1214/SS/1009212519.
  * [33] A. Gensler, B. Sick, A multi-scheme ensemble using coopetitive soft gating with application to power forecasting for renewable energy generation, CoRR arXiv:1803.06344 (2018).
  * [34] A. Gensler, Wind power ensemble forecasting, Ph.D. thesis, University of Kassel (2018).
  * [35] S. Deist, M. Bieshaar, J. Schreiber, A. Gensler, B. Sick, Coopetitive soft gating ensemble, in: Workshop on Self-Improving System Integration (SISSY), Trento, Italy, 2018, pp. 190–197.
  * [36] M. Bieshaar, Cooperative intention detection using machine learning. Advanced cyclist protection in the context of automated driving, kassel university press, 2021.
  * [37] A. Gensler, B. Sick, Forecasting wind power - an ensemble technique with gradual coopetitive weighting based on weather situation, in: IJCNN, Vancouver, BC, 2016, pp. 4976–4984.
  * [38] S. R. Sain, V. N. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag New York, 2006.  doi:10.1007/978-1-4757-3264-1.
  * [39] F. Van Hulle, J. O. Tande, K. Uhlen, et al., Further Developing Europe’s Power Market for Large Scale Integration of Wind Power - Equivalent Wind Power Curves, European Wind Energy Association (EWEA) (2008).
  * [40] J. Schreiber, M. Siefert, K. Winter, A. Wessel, R. Fritz, G. Good, A. Schella, J. Muraschko, S. Staedler, Abschlussbericht Projekt Prophesy : Prognoseunsicherheiten von Windenergie und Photovoltaik in zukünftigen Stromversorgungssysteme, German National Library of Science and Technology (2020) 168doi:10.2314/KXP:176026797X.
  * [41] A. Gensler, B. Sick, Probabilistic wind power forecasting: A multi-scheme ensemble technique with gradual coopetitive soft gating, in: SSCI, Honolulu, HI, 2017, pp. 1–10.

##  Appendix A Data

The following section details challenges with the different datasets. We also
include a description on the overall process for generating power forecasts
for a wind or photovoltaic (PV) park. Fig. A.1 summarizes this process.

Figure A.1: Overview of renewable power forecast process.

Due to the weather dependency of renewable power plants, we require weather
predictions from so called numerical weather prediction (NWP) models. The NWP
model has input from sensors that approximate the current weather situation.
Based on the last sensory data, a so-called model run is calculated. This
model run is, e.g., 0.00 am. Due to complex and manifold stochastic
differential equations involved in predicting the weather, such a model run
typically requires about six hours until it is finished. Afterward, the NWP
provides forecasts, e.g., up to $72$ hours into the future. In our case, we
are interested in so-called day-ahead power forecasts based on weather
forecasts between $24$ and $48$ hours into the future. Based on these weather
forecasts and historical power measurements, we can train a machine learning
(ML) model to predict the expected power generation for day-ahead forecasting
problems.

However, due to the dependency of renewable power forecasts on weather
forecasts as input features, substantial uncertainty is associated with these
forecasts making it a challenging problem. At the same time, weather forecasts
are valid for larger grid sizes, e.g., three kilometers and a mismatch between
these grids and the location of a wind or PV causes additional uncertainty in
the power forecasts. These mismatches and the non-linearity of the forecasting
problem are visually accessible in Fig. A.2. We can observe various mismatches
between the predicted wind speed (or radiation) with historical power
measurements. For instance, we can observe (outliers) where a large amount of
power is generated for low values of those features. Those mismatches are also
visually accessible in the time-series plots in Fig. A.3 and A.4. This
observation indicates that the weather forecasts were wrong. Various examples
are also present where we observe a large value of wind speed or radiation,
but no or little power is generated. A wrong weather forecast can cause this
problem. However, often, it is associated with regular interventions. For
instance, in some regions in Germany, wind turbines need to limit the rotation
speed during the night. Also, there is a large portion of feed-in management
interventions in Germany. These interventions are used to stabilize the
electrical grid. A typical pattern for those interventions is given in Fig.
A.4c. We can observe an initial large power production associated with large
wind speed values. At this point, the power generation drops to zero, while
the wind speed remains high. Such a sudden drop is typically associated with
feed-in management interventions. As those interventions depend on the power
grid’s state, we typically have no information about such drops, making the
forecasting problem even more challenging.

(a) PVSYN datastet. (b) WINDSYN datastet.

(c) PVOPEN datastet. (d) WINDOPEN datastet.

(e) PVREAL datastet. (f) WINDREAL datastet.

Figure A.2: Scatter plots of most relevant features for power forecasts from
day-ahead weather forecasts and the historical power measurements for an
exemplary park from all datasets.

(a) PVSYN dataset. (b) PVOPEN dataset.

(c) PVREAL dataset.

Figure A.3: Timeseries plots of PV datasets.

(a) WINDSYN dataset. (b) WINDOPEN dataset.

(c) WINDREAL dataset.

Figure A.4: Timeseries plots of wind datasets.

##  Appendix B Method

The following sections detail proposed methods and utilized approaches.

###  B.1 Coopetitive Soft Gating Ensemble for Model Combination

To adapt source models from a model hub for a target, it is essential that a
learning strategy only adapts relevant knowledge for the target. For instance,
a learning strategy should assure that a model is not adapted to seasonal
specific behavior when only limited data of a specific season is available.
One way to achieve this is through ensembles. The Bayesian model averaging
(BMA) ensemble, for instance, allows combining source models by their marginal
likelihood on the target. However, in practice, Bayesian models are often not
available. While the BMA combines models through the posterior, we propose to
utilize the coopetitive soft gating ensemble (CSGE), which combines ensemble
members based on the error scores. Recently, the CSGE has proven successful in
renewable power [35, 34, 33] and trajectory forecasts [36]. The following
section details this method in the context of inductive transfer learning
(ITL).

Ensemble approaches are categorized as either weighting or gating strategies
to combine ensemble members. In the case of a gating strategy, a single
ensemble member provides a prediction and other source models are neglected.
In the case of a weighting strategy, several ensemble members are combined.
The CSGE incorporates both principles through the soft gating principle [37].
The idea of the CSGE is to link the weights to the ensemble members’
performance, i.e., good source models are weighted stronger than weaker ones.
The overall weight of a source model for the target task is characterized
using three aspects:

  * •

The global weight is defined by how well a source model performs with the
available training data on the target task.

  * •

The local weight is defined by how well a source model performs on the target
tasks for different areas in the feature space. For example, in the case of
wind, one model might performs well for low wind speeds, while another source
model might performs well for larger wind speeds on the target.

  * •

The forecast horizon-dependent weight is defined by how well a source model
performs for different lead times on the target task.

Compute Weights Normalize Weights Aggregate Forecasts Global Local Forecast
Horizon Source Model (1) Source Model (2) … Source Model (M)
$\inputFeatureVector_{t+k|t}$$\bar{\outputPredictionVector}_{t+k|t}$$\outputPredictionVector^{(1)}_{t+k|t}$$\outputPredictionVector^{(2)}_{t+k|t}$$\outputPredictionVector^{(\numberSourceModels)}_{t+k|t}$$\csgeWeightUnnormalized^{(1)}_{t+k|t}$$\csgeWeightUnnormalized^{(2)}_{t+k|t}$$\csgeWeightUnnormalized^{(\numberSourceModels)}_{t+k|t}$$\csgeWeightNormalized^{(1)}_{t+k|t}\cdot\outputPredictionVector^{(1)}_{t+k|t}$$\csgeWeightNormalized^{(2)}_{t+k|t}\cdot\outputPredictionVector^{(2)}_{t+k|t}$$\csgeWeightNormalized^{(\numberSourceModels)}_{t+k|t}\cdot\outputPredictionVector^{(\numberSourceModels)}_{t+k|t}$
… … …

Figure B.1: The architecture of the CSGE. The source models’ predictions
$\outputPredictionVector^{(\indexSourceModel)}_{t+k|t}$ for the input
$\inputFeatureVector$ are passed to the CSGE. The ensemble member’s weights
are given by the aggregation of the respective global-, local- and forecast
horizon-dependent weights. The weights are normalized. In the final step, the
source models’ predictions are weighted and aggregated.

Fig. 3 provides an overview of the CSGE. In the following, we treat ensemble
members equivalently to source models from a model hub. The CSGE includes
$\numberSourceModels$ ensemble members, with
$\indexSourceModel\in\\{1,...,\numberSourceModels\\}$. Each of these ensemble
members is a source model with a predictive function $f_{m}$. Each source
model forecasts uni-variate estimates
$\outputPredictionVector^{(\indexSourceModel)}_{t+k|t}\in\mathbb{R}$ for the
input $\inputFeatureVector_{t+k|t}\in\mathbb{R}^{\dimFeature}$ of a target
task $T$, we omit the subscript $T$ for reasons of clarity and
comprehensibility. Let $\dimFeature$ be the dimension of the input feature
vector $\inputFeatureVector_{t+k|t}$. Then, $k$ denotes the forecast horizon,
denoted by the subscript, for the forecast origin $t$. Furthermore, let
$\outputPredictionVector^{(\indexSourceModel)}_{t+k|t}$ the
$\indexSourceModel$-th source model’s prediction. For each prediction of each
source model, we compute an aggregated weight
$\csgeWeightNormalized^{(m)}_{t+k|t}$, based on the global, local, and
forecast horizon-dependent weighting. Finally, the weight and the prediction
of each model for each forecast horizon are aggregated as a final prediction
through

$\displaystyle\bar{\outputPredictionVector}_{t+k|t}=\sum^{\numberSourceModels}_{\indexSourceModel=1}\csgeWeightNormalized^{(\indexSourceModel)}_{t+k|t}\cdot\outputPredictionVector^{(\indexSourceModel)}_{t+k|t},$
(B.1)

with the following constraints

$\displaystyle\sum_{\indexSourceModel=1}^{\numberSourceModels}w^{(\indexSourceModel)}_{t|t+k}=1\qquad\forall
k\in\naturalNumberPos.$ (B.2)

The constraints ensure that the weights of all $\numberSourceModels$ ensemble
members, where $\csgeWeightNormalized^{(\indexSourceModel)}_{t|t+k}$ denotes
the weight of the $m$-th ensemble member’s prediction with lead time $k$, are
properly normalized. The weight incorporating the global, local, and forecast
horizon-dependent weight, for a single source model and lead time, is given by

$\displaystyle\bar{\csgeWeightNormalized}^{(m)}_{t+k|t}=\csgeWeightNormalized^{(m)}_{g}\cdot\csgeWeightNormalized^{(m,t)}_{l}\cdot\csgeWeightNormalized^{(m,k)}_{h}.$
(B.3)

We achieve a normalization of $\bar{\csgeWeightNormalized}^{(m)}_{t+k|t}$
through the division of the sum of all weights for all source models and lead
times:

$\displaystyle\csgeWeightNormalized^{(m)}_{t+k|t}=\bar{\csgeWeightNormalized}^{(m)}_{t+k|t}/\sum_{\tilde{m}=1}^{M}\bar{\csgeWeightNormalized}^{(\tilde{m})}_{t+k|t}.$
(B.4)

####  B.1.1 Soft Gating Principle

The soft gating principle to calculate a source model’s weight has several
theoretical advantages in the context of an ensemble for a target task from a
model hub. These advantages are all due to the compromise between the
weighting and the gating principle of the CSGE. On the one hand, we want
gating, as properly many source models do not apply to a target task and
should therefore not be considered. On the other hand, those applicable source
models should be combined through weighting. In a sense, this is comparable to
an BMA approach, where models with a low posterior are neglected (gated),
while those with a larger posterior get a larger weight.

To calculate the weights
$\csgeWeightNormalized^{(\indexSourceModel)}_{t+k|t}$, we utilize the
definition of the inverse similarity measurement $\similarityMeasure^{-1}$
from Section 3.2 and the coopetitive soft gating principle from Eq. B.5. By
calculating the weighting through the inverse $\similarityMeasure^{-1}$, here
the normalized root mean squared error (nRMSE), we directly measure how well a
source model performs on the target. Therefore, let us assume that
$\mathbf{\Phi}\in\realNumber^{J}$ contains all $J\in\naturalNumberPos$
estimates based on the nRMSE and $\phi$ is an arbitrary element from
$\mathbf{\Phi}$. Then, $\eta\geq 0$ depicts the amount of exponential
weighting and the small constant $\epsilon>0$ avoids division by zero. For
greater $\eta$, the CSGE tends to work as a gating ensemble and, thereby,
considering only a few source models. For smaller $\eta$ result in a weighting
ensemble. After calculating all weights from $\mathbf{\Phi}$ through Eq. 14,
we normalize the results to sum up to one estimating the final weights.

$\displaystyle\varsigma^{{}^{\prime}}_{\eta}(\mathbf{\Phi},\phi)=\frac{\sum_{j=1}^{J}\mathbf{\Phi}_{j}}{\phi^{\eta}+\epsilon},$
(B.5)

Finally, to assure that calculated weights for each ensemble member are
normalized, we use the function
$\varsigma_{\eta}:(\mathbb{R}^{+})^{M+1}\rightarrow[0,1]$ with

$\displaystyle\varsigma_{\eta}(\mathbf{\Phi},\phi)=\frac{\varsigma^{{}^{\prime}}_{\eta}(\mathbf{\Phi},\phi,)}{\sum_{m=1}^{M}\varsigma^{{}^{\prime}}_{\eta}(\mathbf{\Phi},\phi_{m})}$
(B.6)

####  B.1.2 Global Weighting

The global weight expresses the overall performance of the different source
models on the target based on the available training data. The average
$r^{(\indexSourceModel)}\in\realNumberPos$ of a single source model is given
by

$\displaystyle r^{(\indexSourceModel)}=$
$\displaystyle\,\frac{1}{\nSamples}\cdot\sum_{\indexSamples=1}^{\nSamples}\similarityMeasure^{-1}(\outputPredictionVector^{(\indexSourceModel)}_{\indexSamples},o_{\indexSamples}),$
(B.7)

where $\nSamples$ refers to the number of available training data,
$\similarityMeasure^{-1}$ is the inverse similarity for the prediction
$\outputPredictionVector^{(\indexSourceModel)}_{\indexSamples}$ and the
respective observations $o_{\indexSamples}$. Then, the set of all errors, from
all source models, is given by
$R=\,(r^{(1)},\ldots,r^{(\indexSourceModel)},\ldots,r^{(\numberSourceModels)})$.
This set allows us to calculate the global weight by

$\displaystyle w_{g}^{(m)}=\varsigma_{\eta_{g}}(R,r^{(m)}),$ (B.8)

with the hyperparameter $\eta_{g}$ for the soft gating formula.

####  B.1.3 Local Weighting

The local weighting expresses how well a source model performs for different
areas in the feature space. For instance, one source model might perform well
on the target for cloudy days while another performs well for sunny days. To
calculate an expectation of the error under certain conditions in the input
features $\inputFeatureVector$, we train a regression model for each source
model based on the available training data. Once the regression model is
trained, this function with
$L^{(\indexSourceModel)}:\realNumber^{\dimFeature}\rightarrow\realNumberNonNeg$
lets us predict the expected error $\hat{q}^{(m)}_{t+k|t}$ for each ensemble
member and each $k$ with

$\displaystyle\hat{q}^{(m)}_{t+k|t}=L^{(m)}(\inputFeatureVector_{t+k|t}).$
(B.9)

Similar, to the global weighting, the set of all local errors

$Q=(\hat{q}^{(1)}_{t+k_{\text{min}}|t},\ldots,\hat{q}^{(m)}_{t+k|t},\ldots,\hat{q}^{(M)}_{t+k_{\text{max}}|t}),$

$k_{\text{min}}$ refers to the smallest forecast horizon and $k_{\text{max}}$
refers to the maximum forecast horizon, lets us calculate the local weight
$w_{l}^{(j)}$ with

$\displaystyle w_{l}^{(t,k)}=\varsigma_{\eta_{l}}(Q,\hat{q}^{(m)}_{t+k|t}),$
(B.10)

where $\eta_{l}$ is the hyperparameter of the soft gating formula for the
local weighting.

####  B.1.4 Forecast horizon-Dependent Weighting

The time-dependent weighting considers that ensemble members may perform
differently for different forecast horizons $k$, e.g., some source models
might be better in the short-term while others are better in the long term.
Similar to global weighting, we calculate this weighting during training. But
instead of averaging across all training samples, we calculate the mean with

$\displaystyle p^{(\indexSourceModel)}_{k}=$
$\displaystyle\,\frac{1}{\nSamples_{k}}\cdot\sum_{\indexSamples_{k}=1}^{\nSamples_{k}}\similarityMeasure^{-1}(\outputPredictionVector^{(\indexSourceModel)}_{\indexSamples_{k}},o_{\indexSamples_{k}})$
(B.11)

to obtain the forecast horizon-dependent error $p^{(\indexSourceModel,k)}$,
where $\nSamples_{k}$ refers to the number of samples for a forecast horizon
$k$. All errors of all ensemble member are the combined through the set
$P_{k}=\,(p_{k}^{(1)},\ldots,p_{k}^{(\indexSourceModel)},\ldots,p_{k}^{(\numberSourceModels)})$
to calculate the time-dependent weight through

$\displaystyle w_{lt}^{(\indexSourceModel,k)}=$
$\displaystyle\,\varsigma_{\eta_{lt}}(P_{k},p_{k}^{(m)}).$ (B.12)

The forecast horizon-dependent hyperparameter is here denoted with
$\eta_{lt}$.

##  Appendix C Experiment

In the following, we detail our hyper-parameters for model training, compare
them to a physical model for the REAL datasets, provide additional evaluations
and example forecasts for the most promising methods.

###  C.1 Hyperparameter Tuning of Multi-Layer Perceptron and Temporal
Convolution Network Source Models

As source models for the model hub, we utilize an multi-layer perceptron (MLP)
and a temporal convolution network (TCN). The layers’ initialization strategy
is the default one of the utilized libraries
(pytorch111https://pytorch.org/docs/stable/index.html, accessed 2021-02-28,
fastai222https://docs.fast.ai/, accessed 2021-02-28). For all source models,
we conducted a hyperparameter search through an tree-structured Parzen sampler
for $200$ samples through the optuna333https://optuna.org/, accessed
2021-02-28 library. The best hyperparameters are selected based on the
smallest nRMSE on the validation dataset. We initially transform the model’s
number of input features in a higher dimension through a factor
$k\in\\{1,\ldots,20\\}$. Note that initially transforming the input features
in a higher-dimensional space typically improves the performance [38].
Afterward, the features are reduced by $50$ percent in each layer to a minimum
of $11$ before the final output. The final two layers of all networks have
sizes $3$ and $1$. The learning rate of the adam optimizer is sampled from a
logarithmic uniform distribution
$\mathcal{U}(\ln(${10}^{-6}\text{\,}\mathrm{)}$,\ln(${10}^{-1}\text{\,}\mathrm{)}$)$.
For each model, we sample the number of training epochs from the set
$\\{10,\ldots,100\\}$. The dropout is sampled from $\mathcal{U}(0,0.9)$ and
the weight decay from $\\{0.0,0.1,0.2,0.4\\}$. For the MLP the batch size is
sampled from the set $\\{1024,2048,4096\\}$. In the case of the TCN network
architecture, these batch sizes are divided by the number of samples per day,
$24$ for the PVOPEN dataset and $96$ for the remaining datasets.

###  C.2 Hyperparameter Tuning Bayesian Extreme Learning Machine Source Models

The Bayesian extreme learning machine (BELM) is optimized by sampling $200$
times through the tree-structured Parzen sampler and the best parameters are
selected by the largest evidence on the validation data. The number of hidden
neurons is from $\mathcal{U}(10,1000)$ to generate the random features. As an
activation function for the random features, one of Rectified Linear Unit
(ReLU), sigmoid, ReLU and sigmoid is sampled. In the latter case, the results
of the two activation functions are concatenated to generate a maximum of
$2000$ random features. Additionally to these random features, we give the
hyperparameter tuning the option to include the original NWP features. Alpha
and beta of the Bayesian linear regression are optimized through empirical
Bayes [30].

###  C.3 Gradient Boosting Regression Tree Target Model

To have a strong baseline that generalizes well with the limited amount of
data and is known to mitigate the effects of overfitting, we train a gradient
boosting regression tree (GBRT) for each target model. We optimize the GBRT
for each park, season, and number of training data through a grid search by
three-fold cross-validation. Thereby, the learning rate is evaluated at
$[${10}^{-6}\text{\,}\mathrm{,}$$3.1\text{\times}{10}^{-6}$,\ldots,$3.1\text{\times}{10}^{-1}$,1]$.
The number of estimators is $300$ and the maximum depth is one of $[2,4,6,8]$.
Other values are the default ones of scikit-learn444https://scikit-learn.org/,
accessed 2022-02-28.

###  C.4 Target Training Extreme Learning Machine

: For the adaptation of the BELM model, we assume that given an appropriate
model selection strategy, the prior $\alpha$ and $\beta$ of the Bayesian
linear regression is also valid for the target. In principle, this is a
relatively strong assumption. While the aleatoric uncertainty might be the
same between a source and a target, the knowledge of the source model for the
target is low and therefore, the epistemic uncertainty should be high. We
argue that the problem is reduced as we update the mean vector and the
precision matrix, given the available target data through Eq. 2, where the
previous posterior, from the source model, acts as prior for the target.
Another interesting consideration is that the insufficient evidence of the
source model on the target requires that an update of the source model needs a
large amount of target data which contracts to the training from the source
model. Therefore, it reduces the risk of catastrophic forgetting.

We first update all source models by the given target data to select a source
model. We train and select the source model with all the available data for
the evidence selection strategy. For the nRMSE selection, we use $70\%$ of the
data for training and the remaining data for the selection.

###  C.5 Target training of Multi-Layer Perceptron and Temporal Convolution
Network

As pointed out earlier, we deploy different model selection and adaptation
strategies, as summarized in Table 1. In case we fine-tune a source model,
e.g., through weight decay, we conduct a grid search for different
hyperparameters and select the best one based on the lowest nRMSE through
$30\%$ of the training data as a validation dataset. The learning rate and the
amount of penalty $\lambda$ are optimized for all adaptation strategies, as
detailed earlier in this section. For all optimizations, we use the stochastic
gradient descent algorithm without momentum as suggested in [8] for transfer
learning (TL). For the Bayesian tunning, we select the ten most relevant
models selected through the evidence for regularization. In all cases, the
batch size is selected to have about ten iterations within an epoch and we
train for a single epoch to limit the amount of training time.

For the direct adaptation, we utilize the source model without any adaptation.
In the case of the direct linear adaptation, we replace the final layer
through a Bayesian linear regression (BLR) model(s). These BLRs are trained
through empirical Bayes. For the TCN source model, we train one BLR for each
forecast horizon based on the extracted features from the source model from a
specific forecast horizon and its response.

###  C.6 Comparison to Physical Models

As pointed out in Section 4 the WINDSYN , PVSYN , WINDOPEN , PVOPEN have been
previously mentioned in [3] and [7]. The WINDREAL and PVREAL have not been
mentioned previously. Therefore, to assure that the GBRT baseline is superior
to physical models that are commonly utilized for parks with limited data, we
detail a comparison in this section. Generally, the experimental set-up is as
detailed in Section 4. Hyper-parameters of the GBRT model are detailed in
Section C.3. For the WINDREAL dataset, we compare the GBRT results for the
different amounts of training data to the empirical McLean power curve [39].
This power curve is well known and considers typical influences by different
terrains. For the PVREAL dataset we created forecasts through the
PVLib555https://pvlib-python.readthedocs.io/, accessed 2022-02-28. For both
datasets, it is not feasible to utilize more specific physical models, as
details on the physical characteristics are not given.

Fig. C.1 compares the physical models to the GBRT model by means of the skill,
for the WINDREAL dataset, given by

$\text{Skill}=\text{nRMSE}_{gbrt}-\text{nRMSE}_{phy},$ (C.1)

where $\text{nRMSE}_{gbrt}$ is the nRMSE of the GBRT model and
$\text{nRMSE}_{phy}$ of the physical model, respectively. We can observe that
the GBRT leads to improvements for the WINDREAL dataset starting with $30$
days of training data. With more than $30$ days of training data, there is a
substantial improvement compared to all types of the McLean power curve. The
mean nRMSE for the WINDREAL dataset is given in Table C.1. In comparison to
the analysis of forecast errors in [40], we observe a similarly mean nRMSE
with $30$ days of training data. Note that the experimental set-up in [40]
utilizes more than $365$ days of training data. Results for the PVREAL dataset
are given Fig. C.2 and Table C.2. Substantial improvements of the GBRT model
over the physical PV model is given by more than seven days. The mean forecast
errors are similar to those in [40] between $30$ and $60$ days of training
data. Due to forecast errors being similar to results in [40] with limited
data, we can conclude that the GBRT is a strong baseline for ITL for the
WINDREAL and PVREAL dataset.

Figure C.1: Comparison of physical model and the GBRT model for the WINDREAL dataset for different amounts of number of days training data through the skill. Values below zero indicate an improvement of the GBRT over the physical model. Figure C.2: Comparison of physical model and the GBRT model for the PVREAL dataset for different amounts of number of days training data through the skill. Values below zero indicate an improvement of the GBRT over the physical model. Table C.1: Mean nRMSE values for the GBRT model and McLean power curves for the WINDREAL dataset. Best values for the number of available training data in days is highlighted in bold. | GBRT | McLean- | McLean- | McLean- | McLean-
---|---|---|---|---|---
|  | Lowland | LowlandStallRegulated | Offshore | Upland
#Days |  |  |  |  |
7 | 0.190 | 0.155 | 0.154 | 0.156 | 0.152
14 | 0.164 | 0.155 | 0.154 | 0.156 | 0.152
30 | 0.149 | 0.155 | 0.154 | 0.156 | 0.152
60 | 0.140 | 0.157 | 0.156 | 0.158 | 0.154
90 | 0.137 | 0.157 | 0.155 | 0.157 | 0.153
365 | 0.122 | 0.159 | 0.157 | 0.160 | 0.155
Table C.2: Mean nRMSE values for the GBRT and physical model for the PVREAL dataset. Best values for the number of available training data in days is highlighted in bold. | GBRT | PVPhysical
---|---|---
#Days |  |
7 | 0.133 | 0.137
14 | 0.120 | 0.137
30 | 0.112 | 0.137
60 | 0.107 | 0.137
90 | 0.105 | 0.137
365 | 0.091 | 0.137

###  C.7 Mean Forecast Errors

Fig. C.3 summarizes the mean nRMSE values for the model selection and
adaptation experiment for all six datasets. For clarity of the plots, we show
only the most relevant models. All models decrease with increasing available
training data for all six datasets. For the wind datasets, the WINDSYN dataset
has the largest errors. For the PV datasets, the PVREAL dataset has the
largest error. Except for the PVREAL dataset, the GBRT baseline has a similar
nRMSE with $14$ days of training data, showing that this model is a strong
baseline . With increasing training samples, all models, selection strategies,
and adaptation strategies converge to a similar minimum. The difference
between the best and the worse model can be up to five percent for seven days
of training data. With $90$ days of training data, the difference is between
one and two percent. Models without any adaptation, the direct adaptation
strategy, start with a low forecast error but converge to a plateau in most
cases with $30$ days of training samples. In contrast, the models with
adaptation strategies start with a larger mean forecast error but improve upon
the direct adaptation with equal or more than $30$ days. We can also observe
that fine-tuning adaptation strategies require more data to achieve good
results. This observation is potentially caused by the problem that knowledge
from a single source task is too specific and causes catastrophic forgetting
in various cases.

The improvements of the mean nRMSE through model combination is summarized in
Fig. C.4. The baseline is the TCN-EV-DILI model from the previous experiment.
Compared to the ensemble techniques results, the baseline results vary
depending on the dataset and the amount of available training data.
Nonetheless, often the baseline has at least the fifth-most minor forecast
error. Comparing the worst models of the two experiments, we can observe
improvements of more than two percent and at least one percent in most cases.
The forecast errors between the different ensemble techniques are less than
the differences in techniques in the previous experiment.

Figure C.3: Mean nRMSE of all folds for most relevant techniques for the model
selection and adaptation experiment. Figure C.4: Mean nRMSE of all folds for
the model combination experiment.

An interesting comparison of our best ensemble techniques to the state-of-the-
art for the WINDOPEN and PVOPEN datasets are in Table C.3. The authors of
these results [3] utilize an multi-task learning (MTL) approach. As also
explained in Section 2, MTL approaches are not yet common in the industry, and
therefore it is necessary to make the best use of existing single-task
learning (STL) models. The table shows that the error is in a similar range
for the WINDOPEN and PVOPEN datasets. For the PVOPEN dataset, we have better
results, while with more than $30$ days of training data, the MTL approach is
better for the WINDOPEN dataset. However, note that the table compares our
mean nRMSE from all seasons to values from results for the spring seasons
given in [3], as other values are not given. Nonetheless, the similarity of
the forecast errors suggests that our approach leads to similar good results.

Table C.3: Results of our best ensemble techniques, given in parenthesis, in comparison to the current state of the art [3] in inductive transfer learning for the WINDOPEN and PVOPEN datasets. The authors of [3] utilize a multi-task learning approach and mean nRMSE values from the spring season in [3] are compared to the mean values across all seasons from our experiments. $\downarrow$ indicates that our best ensemble model has a smaller mean nRMSE and $\uparrow$ indicates that our best ensemble technique has a worse error.  DataType #Days | 7 | 14 | 30 | 60 | 90
---|---|---|---|---|---
PVOPEN | 0.096 | 0.083 (0.074)$\downarrow$ | 0.078 | 0.076 | 0.077
| (0.075) $\downarrow$ | (0.074) $\downarrow$ | (0.073) $\downarrow$ | (0.072) $\downarrow$ | (0.071) $\downarrow$
WINDOPEN | 0.161 | 0.173 | 0.152 | 0.141 | 0.137
| (0.164) $\uparrow$ | (0.160) $\downarrow$ | (0.167) $\uparrow$ | (0.148) $\uparrow$ | (0.143) $\uparrow$

###  C.8 Model Selection and Adaptation

Previously in Section 4, we discussed the best model selection and adaptation
strategies through the mean rank. Table C.4 and C.5 lists those combination,
from the overview in Table 1, previously not discussed. We additionally
include the TCN-EV-WD model, as it allows a better comparison across all
evaluated fine-tuning techniques. We calculate the mean rank separate from the
best models to better quantify the quality within the worse models. Again, we
test for a significant improvement compared to the GBRT baseline . We evaluate
the influence of available training data on the different strategies.

For all datasets, we observe that TCN based models have a smaller rank than
MLP based models and adaptations. For both datasets, only one model is
included that is not considering fine-tuning, the MLP-EV-DI model, which is
not adapted to the target. This model is the best for PV datasets with limited
data for up to $14$ days. Additional results for the PV datasets can be
summarized as follows:

  * •

The MLP model has the best results when the weight decay source adaptation
strategy is utilized for PV datasets. A weight decay concerning the origin has
a larger rank in most cases.

  * •

For the PVOPEN dataset, models are not significantly better than the baseline
with more than $14$ days of training data.

  * •

For PVSYN and PVREAL , the TCN model with the evidence selection strategy and
weight decay source adaptation leads to the best significant improvements.

  * •

The Bayesian tuning is among the best for the TCN model but leads to one of
the worst results for the MLP model.

The results for the wind datasets can be summarized as follows

  * •

For the MLP and the selection through nRMSE there is no clear winner of the
adaptation strategies.

  * •

For the MLP and the selection through nRMSE with limited data, the weight
decay source is preferable; with more data, the weight decay leads to better
results.

  * •

For the MLP and the adaptation through Bayesian tunning, results are worse
than those from other selection and adaptation strategies.

  * •

For the TCN, the Bayesian tuning has the best results.

  * •

The second best technique for the TCN is selecting through the evidence and a
weight decay with respect to the origin.

  * •

No model, selection, and adaptation is superior to the baseline with $90$ days
of training data.

Generally, we can observe that for the MLP model, a selection through the
nRMSE is preferable over the evidence, regardless of the fine-tuning
technique. The opposite is the case for the TCN model. Here, we observe that
in most cases, it is preferable to select a source model based on the evidence
instead of the nRMSE This observation suggests that the nRMSE correlates
better with the transferability of the final layer than for the evidence for
the MLP. For the TCN, the final layer is a residual block and the evidence
better captures this complexity.

Table C.4: Rank summary for different source models, selections, and
adaptation strategies on the PV datasets. Only those that are not within the
top four ranks for a dataset are included. GBRT is the baseline and all models
are tested if the forecasts error is significantly better ($\vee$), worse
($\wedge$), or not significantly different ($\diamond$). EV indicates the
model selection by the evidence, RM the one by the nRMSE. DI shows that a
source model is directly applied to the target without any adaptation. DILI
indicates the adaptation though the BLR, WD indicates fine-tuning regularized
by weight decay w.r.t. the origin, WDS indicates fine-tuning regularized by
weight decay w.r.t. the source models parameter, and BT indicates Bayesian
tunning. Table C.5: Rank summary for the wind datasets. Compare Table C.4.

###  C.9 Probabilistic Evaluation of Models

In the following, we list results for probabilistic forecasts that are
utilizable through BLR. The evaluation include MLP-EV-DILI and the TCN-EV-DILI
models. We also include the combination of those models through the BMA.
Finally, we consider the BELM where the source model is either selected
through the nRMSE and or the evidence. The latter is the baseline . All models
are evaluated through the continuous ranked probability score (CRPS) given by:

$CRPS(F,y)=\int_{\mathbb{R}}(F(x)-\mathbb{H}(x\geq y))^{2}dx,$ (C.2)

where $y\in\mathcal{R}$ are observations, $F$ is the predicted cumulative
function, and $\mathbb{H}$ is the Heaviside step function. Similar to the
previous experiments, the mean ranks of the CRPS for all models is given in
Table C.6 and C.7. The mean CRPS values are given in Table C.8 and C.9. For
the PVSYN and PVREAL datasets, the TCN-EV-DILI has the best results with more
than $14$ days of training data improving the baseline significantly. In case
of the PVOPEN dataset, the BMABELM has the best rank in most cases.

With less than $60$ days of limited data, the ensemble technique BMATCN has
the best results for all wind datasets. Through additional data, again, the
TCN-EV-DILI has the best results. This result holds for all datasets and the
different number of training data for the wind dataset, except for one case.

The mean CRPS for the WINDOPEN dataset are worse than those presented [41] by
$8.2$ percent. Even though results from [41] use about six times more data
compared to our experiment with $90$ days of training data, this observation
suggestion that additional evaluations need to be conducted in future work.

Table C.6: Rank summary of the CRPS for different ensemble types on the PV
datasets. BELM-EV is the baseline. DI and DILI are the direct and direct
linear adaptation as ensemble members. BMA models are always updated on the
target by the BLR. Compare also Table 3. Table C.7: Rank summary of the CRPS
for different ensemble types on the WIND datasets. BELM-EV is the baseline. DI
and DILI are the direct and direct linear adaptation as ensemble members. BMA
models are always updated on the target by the BLR. Compare also Table 3.
Table C.8: Mean CRPS values for different ensemble types on the PV datasets.
BELM-EV is the baseline. DI and DILI are the direct and direct linear
adaptation as ensemble members. BMA models are always updated on the target by
the BLR. Compare also Table 3. Table C.9: Mean CRPS values for different
ensemble types on the PV datasets. BELM-EV is the baseline. DI and DILI are
the direct and direct linear adaptation as ensemble members. BMA models are
always updated on the target by the BLR. Compare also Table 3.

###  C.10 Deterministic and Probabilistic Forecasts

The following figures give an impression of the different forecasting
techniques. All plots include power generation and forecasts from $14$ days
from the real-world datasets. We include the standard deviation as a
confidence interval for the probabilistic forecasts. Often these probabilistic
forecasts have a large uncertainty when the power generation is close to zero.
The PV dataset shows the sun’s diurnal cycle in the form of a Gaussian curve.
Such a periodic pattern is not present in the wind dataset.

Figure C.5: Example of a deterministic forecast for a PV Park. Figure C.6:
Example of a probabilistic forecast from a single model for a PV Park. Figure
C.7: Example of a probabilistic forecast based on an ensemble for a PV Park.
Figure C.8: Example of a deterministic forecast for a wind Park. Figure C.9:
Example of a probabilistic forecast from a single model for a wind Park.
Figure C.10: Example of a probabilistic forecast based on an ensemble for a
wind Park.