# Comment on the relation between the velocity- and position-Verlet
integrators

Liyan Ni  Institute of Frontier Chemistry, School of Chemistry and Chemical
Engineering, Shandong University, Qingdao, 266237, P. R. China  Qingdao
Institute for Theoretical and Computational Sciences (QiTCS), Shandong
University, Qingdao, 266237, P. R. China  Zhonghan Hu<EMAIL_ADDRESS>Institute of Frontier Chemistry, School of Chemistry and Chemical Engineering,
Shandong University, Qingdao, 266237, P. R. China  Qingdao Institute for
Theoretical and Computational Sciences (QiTCS), Shandong University, Qingdao,
266237, P. R. China

About three decades ago, Tuckerman et al. suggested applying the Trotter
factorization of the Liouville propagator to systematically generate time-
reversible symplectic molecular dynamics integratorsTuckerman  _et al._
(1992). Their formulation has been widely adapted to producing novel
algorithms for simulations with or without an external coupling to thermo- and
baro-stats (e.g. Refs.Li  _et al._ (2017); Frenkel and Smit (2023) and
references therein). In the original workTuckerman  _et al._ (1992), they
showed how to use the Trotter factorization to easily derive the well known
velocity-Verlet integratorSwope  _et al._ (1982), which updates the position
and velocity at the time $t=n\tau$, $(x(n\tau),v(n\tau))=(x_{n},v_{n})$ with
the time-interval $\tau$ fixed and $n\geqslant 0$ an integer, according to
(Ref.Tuckerman  _et al._ (1992), Eq.(2.18))

$\left\\{\begin{aligned} \displaystyle
x_{n+1}&=x_{n}+\tau\,v_{n}+\dfrac{\tau^{2}}{2}f(x_{n})\\\
v_{n+1}&=v_{n}+\dfrac{\tau}{2}\left[f(x_{n})+f(x_{n+1})\right]\end{aligned}\right.,$
(1)

where $f(x)$ is the position-dependent force reduced by mass. Interestingly,
the Trotter factorization was also used to derive an entirely new integrator,
which they named the position-Verlet integrator, updating the position and
velocity, $(y(n\tau),w(n\tau))=(y_{n},w_{n})$, according to (Ref.Tuckerman
_et al._ (1992), Eq.(2.22))

$\left\\{\begin{aligned}
y_{n+1}&=y_{n}+\tau\,w_{n}+\dfrac{\tau^{2}}{2}f(y_{n}+w_{n}\tau/2)\\\
w_{n+1}&=w_{n}+\tau\,f(y_{n}+w_{n}\tau/2)\end{aligned}\right..$ (2)

Eq. (2) differs from Eq. (1) in that the force is always evaluated at the
position of a half time-step: $y_{n}+w_{n}\tau/2$.

By setting identical initial conditions: $x_{0}=y_{0}$ and $v_{0}=w_{0}$ for
either the Lennard-Jones fluidTuckerman  _et al._ (1992) or the harmonic
oscillator giving the linear force: $f(x)=-x$Tuckerman  _et al._ (1993), it
was demonstrated numerically that the two integrators, Eqs. (1) and (2)
outputted distinct trajectories, as expected. However, ToxvaerdToxvaerd
(1993); not  argued that both integrators reduce to the Verlet algorithm
(Eq.(7) of ref.Toxvaerd (1993))

$r_{n+2}=2r_{n+1}-r_{n}+\tau^{2}f(r_{n+1}),$ (3)

which, by discretizing time, approximates the Newton’s equation of motion
governing the continuous evolution of the position $r$ only. In the viewpoint
of Ref.Toxvaerd (1993), all formulations, i.e., Eqs. (1) to (3) in our
notation, generate discrete trajectories that identically follow the exact
time evolution of a slight perturbed HamiltonianYoshida (1992). Since no
explicit expression for any nontrival perturbed Hamiltonian was available at
that time, it was unknown to what extent this viewpoint remains valid although
it appears to be certainly informative. Fortunately, the complete and explicit
perturbed Hamiltonian for any discrete dynamics of the harmonic oscillator has
now been solved exactlyNi  _et al._ (2024) and thus allows a direct evaluation
of the relation between Eq. (1) and (2) in terms of their continuous
Hamiltonian representations. As opposed to the view of Ref.Toxvaerd (1993)
that the difference between the velocity- and position-Verlet integrators is
“only a question of notation”, we now clarify the relation between the two
integrators in the following.

i) For a system with a constant $f(x)$ independent of $x$, it is obvious that
Eqs. (1) and (2) produce the same trajectories for both velocity and position.
The discrete trajectories always overlap the exact solutions to the Newton’s
equation of motion.

ii) For the harmonic oscillator with $f(x)=-x$, both Eqs. (1) and (2) reduce
to Eq. (3) for the discrete evolution of the position or velocity. The
Hamiltonians of the two integrators differ but yield the identical second-
order differential equation governing the evolution of position or velocity
alone. Once the input initial conditions are set to be: $x_{0}=y_{0}$ and
$v_{0}=(1-\tau^{2}/4)w_{0}$, the outputs always satisfy simple relations:
$x_{n}=y_{n}$ and $v_{n}=(1-\tau^{2}/4)w_{n}$ for $n\geqslant 1$, as shown in
Tab. 1. Similar relations exist whenever the velocity rather than position is
focused. The existence of such simple relations stems from the fact that the
perturbed Hamiltonians remain “harmonic”, i.e., linear combinations of the
squares of the generalized coordinate $q$ and momentum $p$.

Table 1: The first several steps accurate up to at least six digits generated by the velocity-Verlet ($x_{n}$ and $v_{n}$) and position-Verlet ($y_{n}$ and $w_{n}$) integrators subject to $\tau=0.2$, $x_{0}=y_{0}=0$, and $x_{1}=y_{1}=0.1$ for the harmonic and anharmonic systems. | $f(x)=-x$ | $f(x)=-x^{3}$
---|---|---
n | $x_{n},y_{n}$ | $v_{n},0.99w_{n}$ | $x_{n}$ | $v_{n}$ | $y_{n}$ | $w_{n}$
0 | 0 | 0.5 | 0 | 0.5 | 0 | 0.500013
1 | 0.1 | 0.49 | 0.1 | 0.4999 | 0.1 | 0.499987
2 | 0.196 | 0.4604 | 0.19996 | 0.499000 | 0.199930 | 0.499313
3 | 0.28416 | 0.412384 | 0.299600 | 0.495512 | 0.299481 | 0.496193

iii) For a general system described by a complex force field, while the
velocity-Verlet integrator still identifies with the Verlet integrator, the
position-Verlet integrator does not. Tab. 1 also numerically demonstrates that
no simple relation between the position- and velocity-Verlet integrators is
found for an anharmonic oscillator.

For the harmonic oscillator, the perturbed Hamiltonians corresponding to the
discrete motion in Eq. (1) at the condition of $0<\tau<2$, readNi  _et al._
(2024)

${\cal H}_{\rm
v}(q,p,\tau|m)=\zeta_{m}\left[(1-\tau^{2}/4)q^{2}+p^{2}\right],$ (4)

where $m$ is an arbitrary integer and the $q$\- and $p$-independent
coefficient $\zeta_{m}=2(m\pi+{\rm asin}(\tau/2))/(\tau\sqrt{4-\tau^{2}})$.
Given $q_{0}=x_{0}$ and $p_{0}=v_{0}$, the discrete points, $(x_{n},v_{n})$
generated by Eq. (1) always lie on the continuous trajectory produced by
Hamilton’s canonical equations of motion:

$\frac{dq(t)}{dt}\equiv\dot{q}=\dfrac{\partial{\cal H}}{\partial
p};\quad\frac{dp(t)}{dt}\equiv\dot{p}=-\dfrac{\partial{\cal H}}{\partial q},$
(5)

that is, the solutions satisfy $q(t=n\tau)=x_{n}$ and $p(t=n\tau)=v_{n}$. The
Hamiltonian representations are surprisingly not unique, which physically
states that there are multiple continuous trajectories intersecting at all the
discrete points, as shown in Fig.1 of Ref.Ni  _et al._ (2024). On the other
hand, the corresponding perturbed Hamiltonians for the position-Verlet
integrator readNi  _et al._ (2024) instead

${\cal H}_{\rm
p}(q,p,\tau|m)=\zeta_{m}\left[q^{2}+(1-\tau^{2}/4)p^{2}\right].$ (6)

Therefore, the continuous evolution of $(q,p)$ in the phase phase follows
different perturbed Hamiltonians. However, the Hamiltonians of Eqs. (4) and
(6) remain linear combinations of $q^{2}$ and $p^{2}$ with modified
coefficients. Consequently, Eq. (5), combined with either Eq.(4) or Eq. (6),
yields the identical second-order differential equation for $q$ and $p$:

$\ddot{q}=-4\zeta_{m}^{2}(1-\tau^{2}/4)q;\quad\ddot{p}=-4\zeta_{m}^{2}(1-\tau^{2}/4)p.$
(7)

The continuous propagation of $q$ or $p$ alone remains the same for the two
integrators. Thus, if the trajectory of the position is focused by choosing
initial conditions: $q(0)=x_{0}=y_{0}$ and
$\dot{q}(0)=2\zeta_{m}v_{0}=2\zeta_{m}(1-\tau^{2}/4)w_{0}$, the simple
relations hold at all later times:

$x_{n}=y_{n}=q(n\tau);\,v_{n}=(1-\tau^{2}/4)w_{n}=\frac{\dot{q}(n\tau)}{2\zeta_{m}},$
(8)

for $n=0,1,2,\cdots$. Otherwise, focusing on the trajectory of the velocity
gives

$(1-\tau^{2}/4)x_{n}=y_{n}=-\frac{\dot{p}(n\tau)}{2\zeta_{m}};\,v_{n}=w_{n}=p(n\tau).$
(9)

The validity of Eqs. (8) and (9) can be directly confirmed by Eqs. (1) and (2)
subject to $f(x)=-x$. In fact, for the harmonic oscillator, Eqs. (1) and (2)
with either the velocity or position eliminated, identically reduce to the
Verlet form of Eq. (3) with $r$ interpreted correspondingly as either the
position or velocity.

However, no simple relation between the two integrators can be found for any
complex system producing nonlinear forces. For such a general system, the
position-Verlet integrator differs from the velocity-Verlet integrator in that
it does not reduce to the Verlet form of Eq. (3) any more. The evolution
derived from Eq. (2) reads alternatively

$y_{n+2}=2y_{n+1}-y_{n}+\tau^{2}\\\
\left[f(y_{n}+\tau\,w_{n}/2)+f(y_{n+1}+\tau\,w_{n+1}/2)\right]/2,$ (10)

where the last term is not equal to $f(y_{n+1})$ in general unless a linear
force is considered. Instead of evaluating the forces of Eq. (10) at half time
steps, Eqs. (4) and (5) of Ref.Toxvaerd (1993) incorrectly deduced the
evolution of the position from the force at the full time step. The difference
between the two integrators for an anharmonic oscillator can be demonstrated
numerically. Tab. 1 clearly shows that, even if the initial velocities of the
two integrators are chosen such that the first two positions match, difference
appears soon at later times for the anharmonic oscillator giving
$f(x)=-x^{3}$.

We appreciate the discussion with Søren Toxvaerd although we disagree with
each other. This work was supported by NSFC (Grant No. 22273047).

## References

  * Tuckerman  _et al._ (1992) M. Tuckerman, B. J. Berne, and G. J. Martyna, “Reversible multiple time scale molecular dynamics,” J. Chem. Phys. 97, 1990–2001 (1992).
  * Li  _et al._ (2017) D. Z. Li, X. Han, Y. C. Chai, C. Wang, Z. J. Zhang, Z. F. Chen, J. Liu, and J. S. Shao, “Stationary state distribution and efficiency analysis of the langevin equation via real or virtual dynamics,” J. Chem. Phys. 147, 184104 (2017).
  * Frenkel and Smit (2023) Daan Frenkel and Berend Smit,  _Understanding Molecular Simulation: From Algorithms to Applications_ , 3rd ed. (Academic Press, Inc., San Diego, 2023).
  * Swope  _et al._ (1982) W. C. Swope, H. C. Andersen, P. H. Berens, and K. R. Wilson, “A computer simulation method for the calculation of equilibrium constants for the formation of physical clusters of molecules: Application to small water clusters,” J. Chem. Phys. 76, 637–649 (1982).
  * Tuckerman  _et al._ (1993) M. Tuckerman, B. J. Berne, and G. J. Martyna, “Reply to comment on: Reversible multiple time scale molecular dynamics,” J. Chem. Phys. 99, 2278–2279 (1993).
  * Toxvaerd (1993) S. Toxvaerd, “Comment on: Reversible multiple time scale molecular dynamics,” J. Chem. Phys. 99, 2277–2277 (1993).
  * (7) In a private communication with Prof. Søren Toxvaerd in May, 2021, Toxvaerd insisted that his commentToxvaerd (1993) was correct.
  * Yoshida (1992) H. Yoshida, “Symplectic integrators for hamiltonian systems: basic theory,” Symposium-International Astronomical Union 152, 407–411 (1992).
  * Ni  _et al._ (2024) L. Ni, Y. Zhao, and Z. Hu, “Non-unique hamiltonians for discrete symplectic dynamics,” J. Chem. Phys. 161, DOI: 10.1063/5.0221014 (2024).


Short-term load forecasting (STLF) is crucial for the daily operation of power grids. However, the non-linearity, non-stationarity, and randomness characterizing electricity demand time series renders STLF a challenging task. Various forecasting approaches have been proposed for improving STLF, including neural network (NN) models which are trained using data from multiple electricity demand series that may not necessary include the target series. In the present study, we investigate the performance of this special case of STLF, called transfer learning (TL), by considering a set of 27 time series that represent the national day-ahead electricity demand of indicative European countries. We employ a popular and easy-to-implement NN model and perform a clustering analysis to identify similar patterns among the series and assist TL. In this context, two different TL approaches, with and without the clustering step, are compiled and compared against each other as well as a typical NN training setup. Our results demonstrate that TL can outperform the conventional approach, especially when clustering techniques are considered.

ADAMADAptive Moment estimation algorithm
DLDeep Learning

MAPEMean Absolute Percentage Error
MLMachine Learning
MLOpsMachine Learning Operations
MLPMulti-Layer Perceptron
NNNeural Network
RMSERoot Mean Squared Error
sNaiveseasonal Naive
STLFShort-Term Load Forecasting
TLTransfer Learning
TPETree-structured Parzen Estimator
TSOTransmission System Operator

§ INTRODUCTION

§.§ Background

From generation to transmission and distribution, the operation of electricity systems has to be carefully designed and planned. In this context, load forecasting is critical to properly control the operation of electricity systems and support decisions about fuel allocation, personnel management, and voltage regulation, among others. The negative implications of forecast inaccuracies in such settings have been widely explored and discussed in the literature [Feinberg Eugene A. and Genethliou Dora, 2005, Alfares and Nazeeruddin, 2002, Hahn et al., 2009]. Therefore, there is an increasing need for highly accurate load forecasting models that has resulted in numerous forecasting approaches through the years.

The present study focuses on short-term load forecasting (STLF) which is most relevant for the day-to-day operation of electricity systems but also supports decisions of utilities and companies participating in electricity markets. STLF involves day- to week-ahead forecasts, while the resolution of said forecasts ranges from 15 minutes to one hour. STLF is considered a challenging task as electricity load time series are characterized by non-linearity, non-stationarity, randomness, and multiple seasonal patterns [Pelekis et al., 2023]. This is because electrical power demand originates from various electrical loads that, in turn, depend on numerous external variables, including weather and calendar factors. Additionally, the necessity for higher penetration of renewable energy sources urges for more accurate forecasts that effectively adapt to the flexibility and demand response requirements posed within transmission and distribution systems [Pelekis et al., 2023]. As a result, STLF has been investigated in a variety of recent research studies [Bahrami et al., 2021] and projects [Karakolis et al., 2022, Wehrmeister et al., 2022, Pelekis et al., 2023].

Conventional STLF approaches involve the selection of a proper forecasting model and its training using historical data of the target time series (e.g. national electricity demand), often accompanied with key explanatory variables [Moghaddas-Tafreshi and Farhadi, 2008, Cui and Peng, 2015, Haben et al., 2019, Pelekis et al., 2022]. Although such models are theoretically focused on a single time series, in practice, they may also be capable of accurately forecasting other series, provided that said series exhibit a sufficient level of similarity with the ones used for originally training the models. This process of "transferring" the knowledge gained from solving a certain learning problem (source task) to another, usually related problem (target task) [Torrey and Shavlik, 2010], is commonly referred in the literature as transfer learning (TL). The fact that TL enables the development of reusable, generalized models that can be used directly and with a minimal cost for forecasting multiple new series, has popularized the research and development on TL. This has been particularly true in applications that involve the utilization of computationally expensive models, such as deep neural networks (NNs), or the prediction of relatively small data sets. In the first case, TL can be exploited to democratize the use of sophisticated models by making them widely available to the public, while in the second case to support forecasting tasks where limited data availability would have introduced several modeling challenges. Effectively, the application of TL in the STLF domain offers the opportunity to generate accurate forecasts using the concept of pre-trained, global forecasting models [Semenoglou et al., 2021] that, with little or no retraining, can be adapted to the needs of a previously unseen (not included in the training set – zero-shot forecasting) or slightly seen (included in the train set among many other time series – few shot forecasting) time series.

Although the research done on TL in STLF applications has been relatively limited, the concept of TL has been extensively studied in the fields of computer vision and image recognition [Chang et al., 2017, Gopalakrishnan et al., 2017, Kentsch et al., 2020]. The main objective of TL in such settings has been the transfer of existing knowledge, representations, or patterns learned from one context and applying that knowledge to improve the performance in a different context [Zhuang et al., 2021]. In contrast to traditional machine learning (ML) techniques, which require past (training) and future (testing) data to have the same domain, TL allows learning from different tasks, domains, or feature spaces. This element of TL offers immense flexibility when it comes to the source, nature, and context of the source/target data. There are various TL approaches [Ribani and Marengoni, 2019] that can be distinct based on the nature of knowledge that is being transferred, namely (a) instance transfer, (b) feature representation transfer, (c) parameter transfer, and (d) relational knowledge transfer. Depending on the approach, different techniques have

been proposed in the literature: from simply reusing data from the source domain as part of the target domain in a process called "re-weighing" (instance transfer) [Pan and Yang, 2010] to more complicated ones, such as "warm-start" (parameter transfer) [Künzel et al., 2018, Shafahi et al., 2019, Mitra et al., 2022, Gunduz et al., 2023], "freezing/fine-tuning" (feature representation) [Shafahi et al., 2019, Mitra et al., 2022],

and "head replacement" (parameter transfer) [Gao and Mosalam, 2018]. Note that TL techniques are not mutually exclusive. In fact, they can be blended, addressing different aspects of the forecasting problem.

Given the benefits of TL and the limited attention it has attracted in the area of STLF, this study focuses on TL approaches for day-ahead forecasting on 27 net aggregated load time series that are available at the ENTSO-E transparency platform [ENTSO-E, 2022]. Our main objective is to investigate the potential accuracy improvements that transfer of knowledge can offer to conventional STLF NN models currently used by transmission system operators (TSOs) in Europe for forecasting  the national day-ahead electricity demand, especially within countries that share similar load patterns. This is of high interest for electrical power and energy system stakeholders as it can enable better regulating the balance between electricity production and consumption, reduce operational costs, and enhance the safety and robustness of the systems.

We apply TL by considering an approach which combines the warm-start and fine-tuning techniques. The former technique involves copying the weights and biases from the source task model and using them to initialize the weights of the target task's models, while the latter adjusting said weights and biases on the target task. Following this approach, the time and target data needed to train the respective forecasting  models are significantly decreased, since a significant part of the solution has already been established from the source task's solution. To assess the performance of the proposed TL approach, we consider three different experimental setups. In the first setup, the forecasting models are optimized, trained, and evaluated for each country individually, i.e. in a conventional fashion and without using any TL operation, thus serving as a baseline. The second setup involves transferring information from a model trained on the 26 countries of the data set to a model tasked to generate forecasts for the respective remaining country. Finally, the third setup applies the same TL approach to the above on subsets of the data set that have been constructed using a clustering algorithm so that information is exchanged within countries of similar electricity demand patterns.

§.§ Related work

There has been abundant research on NNs and their variations on energy forecasting applications. Multi-layer perceptron (MLP) architectures are  frequently employed for solving STLF problems  [Ho et al., 1992, Kandil et al., 2006, Hayati and Shirvany, 2007, Arvanitidis et al., 2021], especially when it comes to new searching algorithms for model training [Ho et al., 1992, Mishra and Patra, 2008, Amjady and Keynia, 2011]. Graphical Neural Networks (GNNs) have also been utilized [Wu and Lin, 2022] to examine the spatial correlation that data sources have with each other, while Residual Neural Networks (ResNet) models [Zhang et al., 2021] have been proposed as an organic way for mapping complex relationships between energy loads of different regions. Feed-forward NNs have also been the topic of research concerning hyperparameter tuning and optimization [Hernandez et al., 2013, Farsi et al., 2021, Lee and Rhee, 2021], aiming to increase forecast accuracy and decrease computational costs of said models. Additionally, studies like those of De Felice Matteo and Yao Xin, 2011 and Vesa Andreea Valeria et al., 2020 have introduced ensembling techniques as a means of reducing parameter uncertainty in NN models, leading to more consistent load forecasts.

Focusing on deep learning (DL) NNs for STLF, recent research has put a particular emphasis on more sophisticated model architectures, such as recurrent neural networks (RNNs) and long short-term memory (LSTM) networks [Zheng et al., 2017, Bouktif et al., 2018, Kwon Bo-Sungand Park et al., 2020]. Many variants of RNNs and hybrid DL models [Sajjad et al., 2020, Rafi et al., 2021] have been presented in several studies, providing forecasts concerning energy consumption at regional, national, or even building level [Wang et al., 2019, Memarzadeh and Keynia, 2021, Yuan et al., 2021, Lee and Cho, 2022]. Similarly, advanced NN architectures have gained popularity [Pelekis et al., 2022, Zhao Wenhui et al., 2022, Pelekis et al., 2023], introducing innovative deep STLF approaches based on feed-forward [Oreshkin et al., 2020, Singh et al., 2022], convolutional [Yin and Xie, 2021], or transformer [Huy et al., 2022, Giacomazzi et al., 2023] setups.

Among the various forecasting approaches used, TL has been a key topic of research, with applications in multiple sectors. There is an extensive review done by Iman et al., 2023, presenting studies involving TL in various topics including, but not limited to medical imaging, psychology, natural language processing (NLP) and quantum mechanics. TL techniques have also been employed to solve problems in the energy sector. Following the trend of aforementioned complicated architectures, models such as LSTM, ResNet, and TCN have been used to process data from multiple meters at regional [Jung et al., 2020], national [Lee and Rhee, 2021, Cai et al., 2020, Abdulrahman et al., 2022], and international level [Zhao et al., 2023, Zhang et al., 2023],

all achieving considerable forecasting performance. Also in the context of TL, clustering techniques have been employed, aiming to identify distribution nodes that have similar trends based on the energy consumption of smart grids [Syed et al., 2023, Campos et al., 2023]. Ultimately, ensemble techniques have become popular in the context of TL [Tan et al., 2020, Wu and Lin, 2022], once more demonstrating the importance of NN ensembles within time series forecasting applications.

Among the numerous comparative studies related to DL architectures [Hu et al., 2019, Ma et al., 2020, Khalil et al., 2021], we can observe cases where MLP architectures achieve higher accuracy than other NN models [Feng et al., 2020], as well as studies demonstrating that MLPs, aided by TL, perform better than LSTM architectures [Khalil et al., 2021]. Another study [Lee and Rhee, 2021] compares TL and meta-learning techniques on national data sets of Korea and Portugal, respectively, highlighting the importance of hyperparameter optimization in the process. Additionally, depending on the nature of the use case [Chen et al., 2020, Chen et al., 2020], MLPs can be preferred over a recurrent neural network, including LSTMs. Considering these findings, alongside the numerous applications of MLP-based TL in the energy sector [Demianenko and De Gaetani, 2021, Kazmi et al., 2019, Jung et al., 2020, Wu Di et al., 2020], MLPs clearly demonstrate their usefulness as a base model within the contemporary TL landscape.

§.§ Contribution

This study aims to investigate the value added by TL to STLF at an inter-country level when applied to forecast the national aggregated electricity load of different countries. In this regard, the contribution of our study is summarized as follows:

* While previous research has examined TL for STLF using national [Lee and Rhee, 2021, Cai et al., 2020, Abdulrahman et al., 2022] or small-scale international transfer [Zhao et al., 2023, Zhang et al., 2023], the present study, extends the scale of international TL to a pan-European level (27 EU countries) using load consumption data sets available at ENTSO-E [ENTSO-E, 2022]. We argue that global forecasting models can be used as a starting point to provide more accurate load forecasts by taking advantage of knowledge transfer opportunities among different, yet possibly similar electricity demand time series. This is a major contribution to the electrical power and energy system domain that can be of significant interest for TSOs and utilities, both at European and international level.

* While many researchers have chosen to experiment with DL architectures [Wu and Lin, 2022, Cai et al., 2020, Ho et al., 1992, Mishra and Patra, 2008, Amjady and Keynia, 2011], in this work we focus on MLPs, a relatively simple NN architecture that is well-established in the time series forecasting domain and has been proven to often outperform more sophisticated models. Apart from being faster to compute and easier to implement, MLPs allow for greater flexibility when it comes to parameter adjusting and re-training, a feature that is particularly useful in the context of TL.

* Different countries may exhibit different electricity demand patterns due to key latent variables (e.g. geography, population size, and economic state). Therefore, we introduce a time series clustering approach with the objective to identify countries of similar demand patterns and facilitate TL. Although similar clustering approaches have been examined in the literature [Syed et al., 2023, Campos et al., 2023], these were applied on low voltage distribution and building level data rather than country level demand time series. To this end, we introduce an additional TL setup in our study, where we apply TL among the countries of a given cluster rather than the complete set of countries available in the data set.

§.§ Structure of the paper

The rest of the paper is structured as follows. Section <ref> covers the methodological steps of our study including the following stages: (i) data pre-processing (sanitization of missing values, duplicates, and outliers), (ii) an exploratory data analysis that provides useful insights about the data set, (ii) the clustering process used to form groups of countries with similar load patterns, (iv) the selected model architecture, (v) the TL methodology and the respective examined setups, and (vi) the model training, validation, and evaluation procedures. Section <ref> follows with the results derived from the investigated TL setups, followed by a brief discussion and meta-analysis of the results. Finally, in section <ref> we conclude our work and present future perspectives.

§ METHODOLOGY

To effectively address our TL problem, the experimental process took place using an automated machine learning operations (MLOps) pipeline developed with MLflow [Alla and Adari, 2021], building up to the one described by Pelekis et al., 2022. The methodological steps of our approach are structured as described in the following sections.

§.§ Data collection and curation

The data set used in the present study contains the time series of the national net aggregated electricity demand of 27 different European countries [The data set includes the electricity demands of all EU countries at an hourly resolution, excluding Luxembourg and Cyprus, with the addition of Switzerland and Norway]. The data set is available online and was obtained by the ENTSO-E transparency platform [ENTSO-E, 2022]. The time series, containing a total of 1,577,954 data points, span from 2015 up to 2021, as this was the most recent version of the data collection containing full calendar years at the time when the experiments were carried out.

Prior to model training, data wrangling had to be carried out on the electrical load time series. To do so, we extended the methodology proposed in [Pelekis et al., 2023] to all countries included in the data set. Specifically we proceeded with the following operations:

* removal of duplicate entries: data may become duplicated as a result of storage or measurement errors, and therefore cleaning our data from such duplicates is necessary to avoid costly mistakes (e.g skew prediction results)

* removal of outliers: After calculating the mean and standard deviation of the load for each month in the data set, we removed any values that arithmetically deviated from the mean more than a certain amount, aiming to exclude any outliers. More specifically, we set the maximum distance from the mean as 4.5 times the standard deviation, so as to maintain data credibility and remove only extreme outliers. This process detected 233 different outliers, with many of them forming groups in segments of our data set as shown in Fig. <ref> and <ref>.

* conversion from native UTC to local time of each country: Electricity demand time series can exhibit different patterns, depending on the time of the day (e.g due to daylight hours). To accommodate that and align the inter-country time series patterns through time, for each time series we inferred the timezone based on the country it referred to, changing it to local time rather than UTC which was the initial one.

* missing data imputation: In the cases where imputation was required, we employed a hybrid method between weighted average of historical data and simple linear interpolation, similar to Peppanen et al., 2016, Pelekis et al., 2023. The weights of each method depend exponentially on the distance of the missing value in question from the nearest timestamp that has a value according to the formula:
$$w = e^{a \times d_i}, r = w \times L + (1 - w) \times H$$

where $\alpha$ (empirically set to 0.3) is a positive weight parameter that depicts how rapidly the weight of simple interpolation drops as the distance to the closest non-empty value increases, $d_i$ is the (positive) distance (in samples) to the closest (preceding or succeeding) available sample, $L$ is the imputation suggestion by linear interpolation, $H$ is the imputation suggestion for historical data, and $r$ is the resulting imputed value. Therefore, as $d_i$ increases, the contribution of the linear interpolation suggestion decreases and the contribution of the historical data suggestion increases exponentially. Said algorithm resulted to the imputation of 13,290 data points within our data set, while an illustrative example of the process is demonstrated in Fig. <ref> and <ref>.

Extreme outliers at random data points (Denmark)

Outliers forming grouping at specific data points (Norway)

Imputation matching complicate pattern (France)

Imputation on multiple close proximity data points (Greece)

Visual representation of the outlier removal and missing value imputation procedures that were performed during the pre-processing of the data set.

§.§ Load profiling

Proceeding with load profiling, we initially calculated the average load profiles of the entire load time series data set, grouped by and aggregated on selected time units (hour, weekday, and month). Specifically, Fig. <ref> illustrates the graphs of the daily (Fig.<ref>), weekly (Fig.<ref>) and yearly (Fig.<ref>) load profiles of the countries of the data set. Subsequently, several key patterns can be noted with respect to our data, as follows:

* daily load profile: In general, most countries exhibit a steep increase starting from sunrise, reaching a peak at noon, with a small decrease during the noon break time, followed by a steady decrease towards night hours (fig. <ref>). This can be attributed to the increased energy demand during working hours, and confirmed by the decrease during noon break/lunch time where working activity is decreased. However, several countries (e.g Switzerland, France) exhibit differentiating patterns leading to the need for further investigation.

* weekly load profile: we observe a steady energy demand during the working days and a steep decrease as we reach the weekend days (fig. <ref>). This is attributed to the fact that commercial and industrial activity is decreased at the end of the week (Saturday and Sunday).

* yearly load profile: For the vast majority of countries, the summer months have on average the lowest energy requirements and they steadily increase as we move towards winter. After that, the energy demand follows the reverse pattern: steadily decreasing till it reaches the lowest again at summer (Fig. <ref>). This behavior can be attributed to the increased demand of heating loads during the winter. However, the opposite trend can be observed for southern European countries with much warmer climates (Greece, Spain, Italy, Croatia, and Portugal) which demonstrate higher energy load during summer months due to the increased cooling demand.

Taking the above into consideration, it becomes evident that most countries follow common patterns during the span of any time profile given. However, as several differences and sub-classifications can be observed – especially within the yearly (Fig.  <ref>) and daily profiles (Fig. <ref>) – we are prompted to compare their energy loads and seek further classifications and dissimilarities through the clustering procedure of the following section.

Average daily load profile (hourly resolution)

Average weekly load profile (daily resolution)

Average yearly load profile (monthly resolution)

Average load profiles of the entire load time series data set aggregated by selected time intervals (hour, weekday, and month) for the time period of 2015 to 2021.

§.§ Clustering

Motivated by the insights of the previous section regarding the inter-country load profile patterns, we are prompted towards grouping countries together in clusters that will act as source domains during TL aiming at higher forecasting performance. A hierarchical clustering approach using Ward's linkage has been identified as the best option for clustering functional data that exhibit periodic trends [Ferreira and Hitchcock, 2009, Vijaya et al., 2019]. To measure the distance between clusters (and by extension countries) our approach has been focused on the countries' timely load profiles. In this direction, each country is represented as a vector consisting of 4 sub-vectors, each one containing its daily, weekly, and yearly profiles, as depicted in <ref>. The vectors have been normalized to (i) ensure a clustering process of high quality that only depends on load shapes rather than magnitudes, as the latter are highly variable among countries, and (ii) enhance the computational efficiency of the algorithm. Note that longer-term load profiles have been ignored as we focus on STLF (the contribution of such profiles is negligible for the examined forecasting horizon).

An illustration of the derived load profile vectors containing daily and yearly profiles for each country of the data set.

The results of the clustering process are illustrated in Fig. <ref>, where we can observe four basic clusters: (i) Mediterranean countries (cluster 1), (ii) central European countries (cluster 2), (iii) Eastern-European countries (cluster 3), (iv) Scandinavian and Baltic countries (cluster 4). Given these results, that imply certain correlation among the countries of the same cluster, we additionally come up with several geographical and socio-economic facts that can be used to validate the results of the derived grouping by the clustering algorithm:

* Baltic states (cluster 4): Estonia, Latvia, and Lithuania share historical ties and experiences among the Baltic states, particularly their time under Soviet authority. Since attaining their independence, they have worked to improve their collaboration.

* Mediterranean sea (cluster 1): Spain, Greece, Italy, and Croatia share many connections due to their unique geographical position. To develop a more integrated energy market in the Mediterranean region, many of these nations work together on electricity and gas interconnections. They seek to increase cross-border gas and electricity trade to guarantee a steady supply of energy.

* Visegrad Group (cluster 2): Poland, Slovakia, and Hungary have made significant progress at building transnational energy pipelines, electrical connections, and transportation networks.

* Benelux (cluster 2): Belgium and the Netherlands share a commitment for free trade as well as a history of economic cooperation, while they are also renowned for their close proximity and highly advanced logistics and transportation networks.

* Scandinavia (cluster 4): Sweden, Norway, and Denmark adopt a similar social structure, close proximity to one another, and a comparable cultural history. They work together on a variety of local concerns and have close trading relations (energy policies, resource management, and development of RES, among others).

The aforementioned classifications and current affairs imply that the clustering algorithm has sufficiently distinguished among the observed load patterns of the European countries involved in our study.

Dendrogram and related choropleth based on clustering done on daily, weekly, and yearly profiles of the countries.

§.§ Selected model

Regarding the selected model, we have opted for the MLP for several reasons. Compared to other popular NN architectures, MLPs are relatively simpler, more flexible, and similarly skillful in learning how to map inputs to outputs within a wide range of forecasting applications. Their structure allows to examine them with high precision and they do not require the time-consuming network design search process needed for deeper and more state-of-the-art architectures, therefore allowing for the rapid deployment of the experimental approach and interpretable comparisons. This trait also results in a more straight-forward application of TL techniques, enabling the execution of a wider variety of tests compared to larger, and usually computationally intensive architectures.

The MLP consists exclusively of perceptrons (or neurons), which are organized in one or more layers. Each perceptron of a layer is fed with the outputs of each perceptron of the previous layer and feeds, in turn, each perceptron of the next layer (forward pass). Equation <ref> is used to calculate a given neuron's output.

\begin{equation}
y = f(\sum_{i=1}^{n} (w_ix_i) + b)
\label{eq:00}
\end{equation}

The weighted (with weights $w_i$) sum of the outputs ($x_i$) of all the neurons in the previous layer plus a bias term ($b$) is computed first. Then, the activation function $f$ is applied to this result. The purpose of this function is to enable the model to capture non-linear relationships among the inputs (lags in our case) and outputs (forecasts). Various activation functions have been used in literature, such as the sigmoid, hyperbolic tangent (tanh), and sigmoid linear units (SiLU), as well as the rectified linear unit (ReLU), with the latter being among the most popular (see Eq. <ref>).

\begin{equation}
f(x) = \frac{1}{1 + e^{-x} }
\label{eq:01}
\end{equation}

During the training process of a MLP, the back-propagation algorithm is used. The weights and biases of the MLP are iteratively updated with the use of optimizers to ensure the optimal values for its weights and biases. Among multiple optimizer options (e.g. Gradient Descent, Adagrad, Adadeltra) our choice was the adaptive moment estimation algorithm (ADAM) since it is one of the most widely adopted options in current research.

§.§ Transfer learning setup

We define TL as the extension of ML that uses knowledge previously gained from solving a task, in order to improve performance on a subsequent task ($T$). Given the popular study by Pan and Yang, 2010 to define TL, we first need to formalize the following:

Let $D$ be a domain that includes: (i) a feature space ($X$) and (ii) a marginal probability distribution $P(X)$ for said $X$. Each task $T$ (denoted by $T = \{Y, f(\cdot)\}$) consists of: (i) a label space $Y$ and (ii) a function $f(\cdot)$, which is unobserved and can only be inferred from the training data ($\{x_i, y_i\}\ |\ x_i \in X$ and $y_i \in Y$), and $f(.)$ is used to predict the label $f(x)$ of a new instance $x$. A general-purpose definition of TL is stated as follows:

Definition 1: "Given a source domain $D_S$ and a learning task $T_S$, a target domain $D_T$ and learning task $T_T$, TL aims to help improve the learning of the target predictive function $f(\cdot)$ in $D_T$ using the knowledge in $D_S$ and $T_S$, where $D_S \neq D_T$ or $T_S \neq T_T$ ."

Aiming to adapt this definition to our TL use case, we can come up with the following statements for each one of our TL experiments:

* Domains ($D_S/D_T$): The feature space is the sequence of historical values used to train the model. Specifically, the length of this sequence directly depends on the optimal look-back window ($l$) of each trained NN. Mathematically, this leads to the following source and target feature spaces respectively: $X_S,X_T \in \mathbb{R}^l$). Therefore, the source and target feature spaces are the same within the TL workflow of a given experiment. In addition, the source and target marginal probability distributions for the predictor variable (future energy demand) are different ($P_S(X) \neq P_T(X)$) within the same experiment, since they depend on the countries included during the training (source training) and fine-tuning (target training) procedures. Given the above, the source and target domains differ within our TL studies ($D_S \neq D_T$).

* Tasks ($T_S/T_T$): The label space consists of the range of the possible energy demand forecasts, which is directly dependent on the forecast horizon used. The forecast horizon is constant (24 data points) within our day-ahead forecasting setting, causing source and target label spaces to be the same ($Y_S,Y_T \in \mathbb{R}^{24}$). With regards to the $f(\cdot)$ function, it corresponds to our trained NN, whose parameters (weights and biases) are used for the knowledge transfer procedure. Since $f(\cdot)$ is determined by the provided training data, and each experiment contains a different set of countries, thereby source and target $f(\cdot)$ functions are different ($f_S(\cdot) \neq f_T(\cdot)$). Given the above, the source and target tasks differ within our TL studies ($T_S \neq T_T$).

Considering these factors, our TL setups fall within the scope of homogeneous ($Y_S=Y_T$ and $X_S=X_T$) inductive transfer learning [Weiss et al., 2016, Zhuang et al., 2021], which can be defined as follows:

Definition 2 (Inductive Transfer Learning). "Given a source domain $D_S$ and a learning task $T_S$, a target domain $D_T$ and learning task $T_T$, inductive transfer learning aims to help improve the learning of the target predictive function $f(\cdot)$ in $D_T$ using the knowledge in $D_S$ and $T_S$, where $T_S \neq T_T$"

Regarding the specifics of TL sub-techniques, within our study, we apply warm-start paired with fine-tuning. Warm-start utilizes previously trained weights or parameters. Specifically, warm-start refers to models being trained on the source domain/task and their parameters being subsequently used as initial values for the target model's parameters (parameter transfer). Note that warm-start applies only an initialization of all model's parameters, in contrast to head replacement which initializes all but the final layers. Warm-start initially involves copying the entire solution from the source task to the target task. This is where fine-tuning comes into play to adjust our source task's solution to the target task. Taking into consideration that: (i) normally NNs' weights and biases are typically initialized using random values, which doesn't offer any initial advantageous/disadvantageous bias for the parameters regarding the target task; and that (ii) the source and target are similar in nature (task, domain, or feature space); the warm-start technique allows to start training the target models from a beneficial ("warmer") position on the loss surface, leading to faster convergence and decreased training time, as the optimal solution has already been significantly approached during the training of the source model.

Aiming to evaluate the added value of TL with or without clustering on the day-ahead national load forecasting of European countries, 3 main setups were implemented to support our comparisons:

* Baseline: We model each country included in the data set individually. In this respect, each country has a unique, individual forecasting model which is trained, optimized, and tested using historical data from said country alone. These models are used as a baseline to evaluate the potential improvements of the two TL setups described below.

* All-but-One (AbO): A TL setup according to which, given a certain country, a model is first pre-trained on the data of all the other countries included in the data set and then fine-tuned using the data of the selected country. As shown in <ref>, our data set contains data from 27 different countries. Therefore, according to this setup, we perform experiments equal to the number of countries, where each time a different country is set as the target domain, while the rest 27 are set as the source domain. A model ($f_S$) is trained in the source domain and its parameters are used (via the warm-start technique) to develop a new model ($f_S$) in order to generate forecasts in the target domain.

* Cluster-but-One (CbO): As stated in section <ref>, countries with a similar geographical, climatic, and socioeconomic characteristics may also share similarities in their electricity demand. In this context, this TL setup is developed to examine the performance of TL between countries pertaining to the same cluster. The approach is similar to the AbO setup, with the exception that in a given country's experiment, the source domain comprises countries that belong in the same cluster, rather than the entire data set.

For the purpose of completeness and objective assessment, we also introduce a seasonal naive model of weekly seasonality, namely seasonal naive with a look-back window of 168 values sNaive(168). This naive approach uses the observed values of the same day of the previous week as day-ahead forecasts for each of the countries and is expected to produce moderate results (daily and weekly seasonalities are incorporated), thus supporting the objective benchmarking of the TL and baseline approaches [Pelekis et al., 2023].

An illustration of the separation process of source and target domains for each experiment in the AbO setup. Each row of cells represents a separate execution of the entire ML pipeline. Source countries on each row are depicted in green, while the corresponding target domain country is depicted in blue. The same approach has been followed in the CbO setup as well, with the exception that this separation takes place internally within the countries of each cluster, excluding those that do not pertain to it.

§.§ Model training and validation pipeline

In section <ref> we described the MLP architecture selected for our models. As mentioned, during the model training and validation procedure, all models have been optimized to minimize the L2 loss, using the ADAM optimizer. Additionally, early stopping was applied with a patience of 10 epochs.

Our data set contains data from 27 different countries. Therefore,  depending on the setup and target domain, each model might be subject to different (but mostly overlapping) data points. In this context, the training process can be summarized as follows:

* each model is trained on a train set spanning from 2015 to 2019, that is $N_s$ - $N_t$ unique data points, where $N_s$ is the number of observations pertaining to the countries of the source domain and $N_t$ is the number of observations belonging to the target country. Note that $N_s$ can vary depending on (i) the historical data availability of each country and (ii) the TL setup. For example, the AbO setup results to much larger training data sets than CbO. The $N_t$ variable is affected by the fact that not all countries' observations date back to 2015, which leads to variant population size among target domains. This is not the case for validation and test sets, as all time series have available observations for the years 2020 and 2021.

* each model is optimized on the validation set (year 2020: 239,018 unique data points) to identify appropriate hyperparameter values.

* each model is evaluated on the remaining, and previously unseen test set (year 2021: 243,541 unique data points) without retraining the model on the full data set (union of the training and validation sets).

Note that the aforementioned data set splitting process allowed the inclusion of complete calendar years within each data set, thus maintaining the symmetry of seasonal patterns.

For each experiment, the hyperparameters of the MLPs were tuned using a multi-dimensional grid of hyperparameter values and more specifically the tree-structured Parzen estimator (TPE) optimization approach, as described by Bergstra et al., 2011 and implemented in Python (version 3.8) programming language in Optuna optimization library (version 2.10) [Akiba et al., 2019], similar to Pelekis et al., 2023. Additionally, the successive halving algorithm – a combination of random search and early stopping – was internally applied as a pruning criterion for TPE trials. A crucial hyperparameter that is common among all architectures, is the look-back window ($l$), namely the number of historical time series values (lags) that the model looks back at when being trained to produce forecasts. The $l$ directly determines the input layer size of the trained neural network. The specific details of the optimization process for each architecture are presented in Tab. <ref>. A total of 100 trials were executed for the optimization of each NN architecture.

Following the identification of the best model architecture in terms of hyperparameter values, 20 separate networks sharing the same hyperparameter values but different pseudo-random initialization of neural weights were trained for each NN architecture. Then, an ensemble of these models was used to produce the final forecasts. To improve the robustness of the results, the final forecast of our ensemble model was derived as the average of the predictions of all models. The ensemble method was performed using Ensemble-PyTorch (version 0.1.9) [82], a unified framework for ensembling in PyTorch, with a user-friendly API for ensemble training/evaluation, as well as high efficiency in training with parallelism.

During the TL step, hyperparameter tuning was bypassed, as the hyperparameters and trainable parameters of the given source model were passed, as were, for the initialization of the target model (warm-start technique). The re-training of the target model followed, being fine-tuned until its validation loss plateaued out (early stopping).

With respect to computing resources, the training, validation, and testing of the models was executed on an Ubuntu 22.04 virtual machine with an NVIDIA Tesla V100 GPU, 32 CPU cores, and 64GB of RAM.

The hyperparameters optimized per NN architecture and the respective search spaces.

1lArguments     1lValue Range     1lType

number of layers      {2, ... ,6}     Discrete

layer sizes     {128, 256, 512, 1024, 2048}     Discrete

$l$      {168, 336, 504, 672}     Discrete

forecast horizon     24     Fixed

learning rate      {$1e^{-5}$, ... , $1e^{-4}$}     Continuous

batch size     {256, 512, 1024}     Discrete

§.§ Model evaluation

Various forecasting performance measures are common in the literature for evaluating forecasting accuracy, such as the mean absolute error (MAE), the mean squared error (MSE), the mean absolute percentage error (MAPE), the root mean square error (RMSE), the symmetric mean absolute percentage error (sMAPE), and the mean absolute scaled error (MASE) Hyndman and Koehler, 2006. Within the present study, MAPE was selected as it is a widely accepted choice in STLF applications and results to interpretable measurements. Consequently, MAPE served as the objective function during the hyperparameter optimization process and as the main evaluation measure. MAPE is defined as follows:

\begin{equation}
MAPE=\frac{1}{m} \sum_{i=1}^{m}\left|\frac{Y_{t}-F_{t}}{Y_{t}}\right| \cdot 100(\%),
\label{eq:1}
\end{equation}

where $m$ represents the number of samples, while $Y_t$ and $F_t$ stand for the actual values and the forecasts at time $t$, respectively.

§ RESULTS AND DISCUSSION

Proceeding to the results of our experiments, the final performance of each architecture is displayed in Fig. <ref> and <ref> (derived from Tab. <ref> and <ref>, respectively), taking into account the ensembling modeling technique that followed the hyperparameter optimization process. More details on the optimal hyperparameters and training times of the best performing models can be found in Tab. <ref> of the appendix <ref>. As an initial observation, sNaive(168) is heavily outperformed at all cases by all NN models (AbO, CbO, baseline). This is but an expected outcome, since naive models do not entail any kind of learning process; they only repeat past values of the time series. In fact, this model was selected as a hard baseline, helping to ensure the validity of our setups rather than to actually benchmark them.

Focusing on the effectiveness of TL, Fig. <ref> demonstrates that within each cluster, at least one of 2 TL setups, achieve, on average, a lower forecasting error than the baseline. Therefore, it becomes apparent that TL, as a whole, outperforms at all cases the traditional NN approach that is represented by our baseline model. Note here that the TL approaches, in total, reduce said forecasting error, on average, by 0.34% with respect to MAPE, as shown in Tab. <ref>. Additionally, note that TL enables the advantageous initialization of target models from their respective source models, leading to much faster convergence and therefore a significant decrease in the target pipeline's (fine-tuning) execution compared to the respective baseline model execution. Said decrease in execution time ranges from 12.09% up to 66.87% (average 47%) depending on the experiment (target country). More details regarding the exact training times per experiment can be sought in in Tab. <ref> of the appendix <ref>.

Considering the analytical results of Tab. <ref> and Fig. <ref>, it is notable that the AbO setup results to an average error reduction of 0.16% compared to the baseline in terms of absolute MAPE percentage. Therefore, this setup, outperforms the baseline on average, even if this is not the case for all the countries in our data set. In the same direction, CbO leads to a higher average error reduction of 0.24% compared to the baseline, therefore confirming the value added by the clustering scheme proposed within our study.

Accuracy (MAPE) of the examined forecasting approaches (baseline, AbO, CbO) alongside the naive model sNaive(168). The last row lists the average improvement of each model compared to the baseline model. The last column contains the MAPE of the best performing model alongside its name. Note here that AbO or CbO are the only alternating values demonstrating that there is always a TL setup that outperforms the rest. In this context, the last cell of the table corresponds to the average improvement of TL (0.34 %), in general, compared to the NN baseline model.

Country     Cluster No.     5cMAPE %

Baseline     AbO     CbO     sNaive(168)     Best model

Italy      1     2.72     2.37      2.47     5.37     2.37 (AbO)

Croatia     1     3.35     3.00      2.86     6.13     2.86 (CbO)

Spain     1     2.00     1.95     2.28     4.51     1.95 (AbO)

Greece     1     3.66     3.60      3.39     7.97     3.39 (CbO)

Serbia     2     2.82     3.27      2.46     6.41     2.46 (CbO)

Portugal     2     2.24     2.79      2.23     4.31     2.23 (CbO)

Belgium     2     2.55     2.56      2.50     4.58     2.50 (CbO)

Ireland     2     2.15     2.03      2.09     3.32     2.03 (AbO)

Netherlands     2     4.21      4.16     4.26     5.59     4.16 (AbO)

France      2     4.53     2.31      2.22     7.15     2.22 (CbO)

Romania      2     2.54     2.09      2.33     4.36     2.09 (AbO)

Bulgaria     2     2.80     2.67      3.32     6.87     2.67 (AbO)

Finland      2     2.26      2.16      2.08     5.75     2.08 (CbO)

Hungary     3     2.96     3.26      2.88     5.64     2.88 (CbO)

Germany     3     2.76     3.17      2.42      4.26     2.42 (CbO)

Slovakia     3     2.06     1.94      2.17     3.95     1.94 (AbO)

Austria     3     3.07      3.02      3.04     5.32     3.02 (AbO)

Slovenia     3     3.56     3.58      3.49      6.44     3.49 (CbO)

Poland     3     2.18     2.40      2.05      4.39     2.05(CbO)

Lithuania     3     2.77     2.47      2.35     4.95     2.35 (CbO)

Switzerland     4     4.71     4.01      3.97     6.25     3.97 (CbO)

Norway     4     2.37     2.33      2.06     5.56     2.33 (CbO)

Denmark     4     3.87     2.80      2.79     5.46     2.79 (CbO)

Estonia     4     3.52     3.36      3.72     6.94     3.36 (AbO)

Czechia     4     2.21      1.83      1.96      4.97     1.83 (AbO)

Latvia     4     2.31     2.11      2.23     4.29     2.11(AbO)

Sweden     4     3.13     2.84     3.19     6.85     2.84 (AbO)

Average Improvement          -     0.18     0.24     -1.8     0.34 (TL)

Side-by-side barplots depicting the MAPE(%) for each TL setup and baseline for each country

Average MAPE(%) per cluster and training setup

1lCluster no.
1lBest setup

1     2.93     2.73     2.75     5.99     AbO

2     2.85     2.67     2.61     5.37     CbO

3     2.77     2.83     2.63     4.99     CbO

4     3.16     2.75     2.85     5.75     AbO

Side-by-side barplots depicting the average MAPE(%) for each TL setup and the baseline for each cluster

§.§ A note on the best-performing TL setup

With respect to CbO, its average improvement (0.24%) compared to the baseline is higher than AbO (0.18%) and therefore has already been considered as the best performing TL setup. To further reinforce this statement, it can be noted that CbO also holds the majority of countries where it achieved better results than its TL competitor (AbO). More specifically, among the 27 European countries that were examined in total, 12 ($44.5$%) of them demonstrate better results with the AbO setup compared to CbO, while the opposite being true for the other 15 ($55.5$%). This outcome can be attributed to the stratified training data selection proposed by our clustering methodology. In this direction, by providing a specific, highly correlated group of countries as its training set, the target model can take advantage of the similarities among countries and transfer the required knowledge in an optimal fashion. Despite the fact that CbO proved to be quite effective in most of the countries, the shortcomings of said method cannot be neglected. Countries like Spain, Romania, and Bulgaria exhibit reduced performance with CbO compared to the respective baseline MLP.

Focusing on Tab. <ref> and Fig. <ref>, an interesting observation lies in the fact that cluster 3 is the only case where AbO is significantly outperformed, even by the baseline model. This finding further validates the superiority of the clustering scheme, confirming that CbO is the most consistent TL setup. Regarding the poor performance of AbO in cluster 3, it should be noted however that its forecast error is not always worse than the baseline NN, being higher mainly in four (Germany, Poland, and Hungary) of the seven countries included in the cluster.

§.§ A note on selecting the optimal TL setup based on the target country

Another interesting discussion topic relates to the selection between AbO and CbO depending on the target country of interest. Our results suggest that said selection should be evaluated individually per use case, and therefore specific caution is advised during the selection process. However, a good indicator for clustering could, by definition, be the degree of similarity within the energy demand patterns observed among the countries of each cluster. This can be further supported by the significant overlap between countries, where clustering was the most effective for countries that indeed have several interconnections with each other. In this direction, several interesting insights can be extracted from the official ENTSO-E transmission system map
[84], as follows:

* Cluster 1: Croatia, Italy and Greece have made a lot of progress developing energy interconnections between them. There is an established 380-400 kV Transmission Line between Redipuglia (Italy) and Melina (Croatia) as well as a 400KV HVDC link between Galatina (Italy) and Arachthos (Greece)

* Cluster 2: Belgium and the Netherlands are closely energy related and have created transmission lines between  and Van Eyck–Maasbracht interconnection. Additionally, France and Belgium have also developed cross-border interconnections

between Mastaing-Avelgem and Avelin-Avelgem

* Cluster 3: Austria is connected with several interconnections with Hungrary, Germany, Slovenia and Slovakia (e.g Kainarkdal-Cirkovce, Silz-Oberbrumm etc). Poland and Lithuania are also connected through various electricity interconnections, including the LitPol Link: a high-voltage direct current (HVDC) interconnection that allows for electricity exchange between the two countries.

* Cluster 4: Sweden has a strong energy connection with Norway, primarily through hydroelectric power generation

as well as Denmark through electricity transmission lines and substations. Estonia and Latvia are also connected through electricity interconnections (e.g Valmiera-Trisguliina), promoting regional energy cooperation and grid stability.

An interesting observation lies in cluster 1. Croatia, Italy, and Greece have developed energy interconnections with each other, while, on the contrary, Spain has not – at least as the time of writing this paper. In this context, Spain is the only country in its cluster where CbO was outperformed by the NN baseline (and AbO by extension). Despite the appeal of this theory, there are a few exceptions, such as the case of Sweden which did not respond as expected to the CbO approach.

§ CONCLUSIONS AND FUTURE RESEARCH

This study conducted a comparative analysis of TL approaches used for day-ahead national net aggregated electricity load forecasting. A feed-forward NN architecture was selected to implement the baseline forecasting models, followed by proper hyperparameter tuning and ensembling. The warm-start technique, accompanied by fine-tuning, was employed as a widely used and streamlined TL technique. Within our initial approach (AbO), we examined the value of straight-forward TL. We developed a separate model for each country of the data set, aiming at forecasting its energy load (target domain). Each model draw its initial parameters' values from another unique model, pre-trained using the data from the rest of the countries included in the data set (source domain). Moreover, motivated by an exploratory data analysis, which revealed a certain correlation among the energy load patterns of different countries – due to similarities in their geographical and socioeconomic conditions – we investigated if the identified correlations could be exploited to further boost forecasting performance. In this direction, we developed a hierarchical clustering algorithm to derive clusters of countries and then used them to form the proper source domains for the TL pipeline by pre-training the respective global model using only the time series of the countries pertaining to the same cluster with the target country (Cb0 setup).

Following our experiments, the results suggest that at least one of the examined TL approaches (AbO or CbO) outperforms the baseline NN model. Specifically, the clustering-based TL pipeline (CbO) outperforms the rest of the NN models, improving the average MAPE by 0.18% when compared to AbO and 0.24% when compared to the baseline. Despite its benefits, CbO often exhibited worse performance, especially for countries with outlying socioeconomic profiles and/or load patterns and magnitudes. In such cases, the AbO model is recommended to counter the negative bias that clustering entails.

The methodology and results of our study can be used by researchers and forecasting practitioners, alongside certain stakeholders of the energy sector, such as transmission and distribution system operators, to develop unified TL-based models for previously unseen national load time series. Said models combine both increased accuracy and an average training acceleration of 47% compared to their respective, conventional NN approach.

Regarding future perspectives, it would be of high interest to explore the clusters and investigate their pertaining countries more thoroughly with respect to political and socioeconomic factors, such as energy policies, energy providers, and cross-country grid interconnections. Such an analysis could allow the selection of more appropriate clustering algorithms or even the manual classification of countries aiming to an optimized CbO performance. Furthermore, a possible future next step of this research could involve the experimentation and TL approach with more complex NN architectures as baseline models, such as LSTM, CNN, or transformers. With respect to the specifics of TL, the use of additional TL techniques (e.g freezing, head replacement/feature extraction) is also highly recommended in an attempt optimize the TL pipeline, leading to further increase of both the accuracy and computational efficiency of the models. With respect to available data sets, further performance increase could be potentially achieved by (i) enriching the utilized electricity demand data set with non-European countries and (ii) incorporating weather forecasts or energy price fluctuations within the training procedure. Finally, future research could involve the comparison of global forecasting models with the existing TL models (AbO, CbO), alongside their zero-shot versions, i.e. approaches that omit the fine-tuning stage. Such a comparison would lead to interesting results regarding the cases where (a) target data sets are (global) or are not (TL) available during the model training process and (b) fine-tuning is hard to conduct after handing the pre-trained model (zero-shot TL case).

§ ACKNOWLEDGMENT

This work has been funded by the European Union’s Horizon 2020 research and innovation program under the I-NERGY project, grant agreement No. 101016508. Additionally, the HPC resources utilized for training and optimizing the required ML models in this study have been provided by the EGI-ACE project, which also receives funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 101017567.

[Feinberg Eugene A. and Genethliou Dora, 2005]
Feinberg Eugene A. and Genethliou Dora.
Load Forecasting.
In Momoh James Chow Joe H., Wu Felix F., editor, Applied
Mathematics for Restructured Electric Power Systems: Optimization, Control,
and Computational Intelligence, pages 269–285. Springer US, Boston, MA,
ISBN 978-0-387-23471-7.
URL <https://doi.org/10.1007/0-387-23471-3_12>.

[Alfares and Nazeeruddin, 2002]
Hesham K Alfares and Mohammad Nazeeruddin.
Electric load forecasting: Literature survey and classification of
International Journal of Systems Science, 330
(1):0 23–34, 2002.
URL <https://doi.org/10.1080/00207720110067421>.

[Hahn et al., 2009]
Heiko Hahn, Silja Meyer-Nieberg, and Stefan Pickl.
Electric load forecasting methods: Tools for decision making.
European Journal of Operational Research, 1990
(3):0 902–907, 12 2009.
ISSN 0377-2217.

[Pelekis et al., 2023]
Sotiris Pelekis, Ioannis-Konstantinos Seisopoulos, Evangelos Spiliotis,
Theodosios Pountridis, Evangelos Karakolis, Spiros Mouzakitis, and Dimitris
A comparative assessment of deep learning models for day-ahead load
forecasting: Investigating key accuracy drivers.
Sustainable Energy, Grids and Networks, 36:0 101171,
12 2023a.
ISSN 2352-4677.

[Pelekis et al., 2023]
Sotiris Pelekis, Angelos Pipergias, Evangelos Karakolis, Spiros Mouzakitis,
Francesca Santori, Mohammad Ghoreishi, and Dimitris Askounis.
Targeted demand response for flexible energy communities using
clustering techniques.
Sustainable Energy, Grids and Networks, 36:0 101134,
12 2023b.
ISSN 2352-4677.

[Bahrami et al., 2021]
Shahab Bahrami, Yu Christine Chen, and Vincent W.S. Wong.
Deep Reinforcement Learning for Demand Response in Distribution
IEEE Transactions on Smart Grid, 120 (2):0
1496–1506, 3 2021.
ISSN 19493061.

[Karakolis et al., 2022]
Evangelos Karakolis, Sotiris Pelekis, Spiros Mouzakitis, Ourania Markaki,
Katerina Papapostolou, Giorgos Korbakis, and John Psarras.
Artificial Intelligence for Next Generation Energy Services Across
Europe - The I-NERGY Project.
In ES 2021 : 19th International Conference e-Society 2021,
pages 61–68, Lisbon, 2022.
ISBN 978-989-8704-38-2.

[Wehrmeister et al., 2022]
Katharina Aurelia Wehrmeister, Efthimios Bothos, Vangelis Marinakis, Babis
Magoutas, Alexander Pastor, Leonardo Carreras, and Antonello Monti.
The BD4NRG Reference Architecture for Big Data Driven Energy
13th International Conference on Information, Intelligence,
Systems and Applications, IISA 2022, 2022.

[Pelekis et al., 2023]
Sotiris Pelekis, Evangelos Karakolis, Theodosios Pountridis, George Kormpakis,
George Lampropoulos, Spiros Mouzakits, and Dimitris Askounis.
DeepTSF: Codeless machine learning operations for time series
arXiv, 7 2023c.
URL <https://arxiv.org/abs/2308.00709>.

[Moghaddas-Tafreshi and Farhadi, 2008]
S. M. Moghaddas-Tafreshi and Mahdi Farhadi.
A linear regression-based study for temperature sensitivity analysis
of iran electrical load.
In Proceedings of the IEEE International Conference on
Industrial Technology, 2008.
ISBN 9781424417063.

[Cui and Peng, 2015]
Herui Cui and Xu Peng.
Short-Term City Electric Load Forecasting with Considering
Temperature Effects: An Improved ARIMAX Model.
Mathematical Problems in Engineering, 2015, 2015.
ISSN 15635147.

[Haben et al., 2019]
Stephen Haben, Georgios Giasemidis, Florian Ziel, and Siddharth Arora.
Short term load forecasting and the effect of temperature at the low
voltage level.
International Journal of Forecasting, 350
(4):0 1469–1484, 10 2019.
ISSN 0169-2070.

[Pelekis et al., 2022]
Sotiris Pelekis, Evangelos Karakolis, Francisco Silva, Vasileios Schoinas,
Spiros Mouzakitis, Georgios Kormpakis, Nuno Amaro, and John Psarras.
In Search of Deep Learning Architectures for Load Forecasting: A
Comparative Analysis and the Impact of the Covid-19 Pandemic on Model
In 2022 13th International Conference on Information,
Intelligence, Systems and Applications (IISA), pages 1–8. IEEE, 7 2022.
ISBN 978-1-6654-6390-4.

[Torrey and Shavlik, 2010]
Lisa Torrey and Jude Shavlik.
Transfer Learning.
Handbook of Research on Machine Learning Applications and
Trends: Algorithms, Methods, and Techniques, pages 242–264, 1 2010.
URL <https://www.igi-global.com/chapter/transfer-learning/36988

[Semenoglou et al., 2021]
Artemios Anargyros Semenoglou, Evangelos Spiliotis, Spyros Makridakis, and
Vassilios Assimakopoulos.
Investigating the accuracy of cross-learning time series forecasting
International Journal of Forecasting, 370
(3):0 1072–1084, 7 2021.
ISSN 0169-2070.

[Chang et al., 2017]
Jongwon Chang, Jisang Yu, Taehwa Han, Hyuk-jae Chang, and Eunjeong Park.
A method for classifying medical images using transfer learning: A
pilot study on histopathology of breast cancer.
In 2017 IEEE 19th international conference on e-health
networking, applications and services (Healthcom), pages 1–4, 2017.

[Gopalakrishnan et al., 2017]
Kasthurirangan Gopalakrishnan, Siddhartha K Khaitan, Alok Choudhary, and Ankit
Deep convolutional neural networks with transfer learning for
computer vision-based data-driven pavement distress detection.
Construction and building materials, 157:0 322–330,

[Kentsch et al., 2020]
Sarah Kentsch, Maximo Larry Lopez Caceres, Daniel Serrano, Ferran Roure, and
Yago Diez.
Computer vision and deep learning techniques for the analysis of
drone-acquired forest images, a transfer learning study.
Remote Sensing, 120 (8):0 1287, 2020.

[Zhuang et al., 2021]
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui
Xiong, and Qing He.
A Comprehensive Survey on Transfer Learning.
Proceedings of the IEEE, 1090 (1):0 43–76, 1
ISSN 15582256.

[Ribani and Marengoni, 2019]
Ricardo Ribani and Mauricio Marengoni.
A Survey of Transfer Learning for Convolutional Neural Networks.
Proceedings - 32nd Conference on Graphics, Patterns and Images
Tutorials, SIBGRAPI-T 2019, pages 47–57, 10 2019.

[Pan and Yang, 2010]
Sinno Jialin Pan and Qiang Yang.
A Survey on Transfer Learning.
IEEE Transactions on Knowledge and Data Engineering,
220 (10):0 1345–1359, 2010.

[Künzel et al., 2018]
Sören R. Künzel, Bradly C. Stadie, Nikita Vemuri, Varsha Ramakrishnan,
Jasjeet S. Sekhon, and Pieter Abbeel.
Transfer Learning for Estimating Causal Effects using Neural
8 2018.
URL <https://arxiv.org/abs/1808.07804v1>.

[Shafahi et al., 2019]
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Cristoph Studer, David
Jacobs, and Tom Goldstein.
Adversarially robust transfer learning.
8th International Conference on Learning Representations, ICLR
2020, 5 2019.
URL <https://arxiv.org/abs/1905.08232v2>.

[Mitra et al., 2022]
Saptashwa Mitra, Menuka Warushavithana, Mazdak Arabi, Jay Breidt, Sangmi
Pallickara, and Shrideep Pallickara.
Alleviating Resource Requirements for Spatial Deep Learning
Proceedings - 22nd IEEE/ACM International Symposium on Cluster,
Cloud and Internet Computing, CCGrid 2022, pages 452–462, 2022.

[Gunduz et al., 2023]
Salih Gunduz, Umut Ugurlu, and Ilkay Oksuz.
Transfer learning for electricity price forecasting.
Sustainable Energy, Grids and Networks, 34:0 100996, 6
ISSN 2352-4677.

[Gao and Mosalam, 2018]
Yuqing Gao and Khalid M. Mosalam.
Deep Transfer Learning for Image-Based Structural Damage
Computer-Aided Civil and Infrastructure Engineering,
330 (9):0 748–768, 9 2018.
ISSN 1467-8667.
URL <https://onlinelibrary.wiley.com/doi/full/10.1111/mice.12363

[ENTSO-E, 2022]
ENTSO-E Transparency Platform, 2022a.
URL <https://transparency.entsoe.eu/>.

[Ho et al., 1992]
K.L. Ho, Y.-Y. Hsu, and C.-C. Yang.
Short term load forecasting using a multilayer neural network with
an adaptive learning algorithm.
IEEE Transactions on Power Systems, 70 (1):0
141–149, 1992.
ISSN 08858950.

[Kandil et al., 2006]
Nahi Kandil, René Wamkeue, Maarouf Saad, and Semaan Georges.
An efficient approach for short term load forecasting using
artificial neural networks.
International Journal of Electrical Power & Energy Systems,
280 (8):0 525–530, 10 2006.
ISSN 01420615.

[Hayati and Shirvany, 2007]
Mohsen Hayati and Yazdan Shirvany.
Artificial neural network approach for short term load forecasting
for Illam region.
World Academy of Science, Engineering and Technology,
28:0 280–284, 2007.

[Arvanitidis et al., 2021]
Athanasios Ioannis Arvanitidis, Dimitrios Bargiotas, Aspassia Daskalopulu,
Vasileios M. Laitsos, and Lefteri H. Tsoukalas.
Enhanced Short-Term Load Forecasting Using Artificial Neural
Energies, 140 (22):0 7788, 11 2021.
ISSN 1996-1073.

[Mishra and Patra, 2008]
Sanjib Mishra and Sarat Kumar Patra.
Short term load forecasting using neural network trained with
genetic algorithm & particle swarm optimization.
In Proceedings - 1st International Conference on Emerging
Trends in Engineering and Technology, ICETET 2008, pages 606–611, 2008.
ISBN 9780769532677.

[Amjady and Keynia, 2011]
Nima Amjady and Farshid Keynia.
A New Neural Network Approach to Short Term Load Forecasting of
Electrical Power Systems.
Energies, 40 (3):0 488–503, 2011.
ISSN 1996-1073.
URL <https://www.mdpi.com/1996-1073/4/3/488>.

[Wu and Lin, 2022]
Di Wu and Weixuan Lin.
Efficient Residential Electric Load Forecasting via Transfer
Learning and Graph Neural Networks.
IEEE Transactions on Smart Grid, 2022.
ISSN 19493061.

[Zhang et al., 2021]
Zhenyuan Zhang, Pengfei Zhao, Peng Wang, and Wei Jen Lee.
Transfer Learning Featured Combining Short-Term Load Forecast with
Small-Sample Conditions.
Conference Record - IAS Annual Meeting (IEEE Industry
Applications Society), 2021-October, 2021.
ISSN 01972618.

[Hernandez et al., 2013]
Luis Hernandez, Carlos Baladron, Javier Aguiar, Belen Carro, Antonio
Sanchez-Esguevillas, Jaime Lloret, David Chinarro, Jorge Gomez-Sanz, and
Diane Cook.
A multi-agent system architecture for smart grid management and
forecasting of energy demand in virtual power plants.
IEEE Communications Magazine, 510 (1):0
106–113, 2013.
ISSN 01636804.

[Farsi et al., 2021]
Behnam Farsi, Manar Amayri, Nizar Bouguila, and Ursula Eicker.
On short-term load forecasting using machine learning techniques and
a novel parallel deep LSTM-CNN approach.
IEEE Access, 9:0 31191–31212, 2021.
ISSN 21693536.

[Lee and Rhee, 2021]
Eunjung Lee and Wonjong Rhee.
Individualized short-term electric load forecasting with deep neural
network based transfer learning and meta learning.
IEEE Access, 9:0 15413–15425, 2021.
ISSN 21693536.

[De Felice Matteo and Yao Xin, 2011]
De Felice Matteo and Yao Xin.
Short-Term Load Forecasting with Neural Network Ensembles: A
Comparative Study [Application Notes].
IEEE Computational Intelligence Magazine, 60
(3):0 47–56, 8 2011.
ISSN 1556-603X.

[Vesa Andreea Valeria et al., 2020]
Vesa Andreea Valeria, Cioara Tudor, Anghel Ionut, Antal Marcel, Pop
Claudia, Salomie Ioan, and Dadarlat Vasile Teodor.
Energy flexibility prediction for data center engagement in demand
response programs.
Sustainability, 120 (4):0 1417, 2020.

[Zheng et al., 2017]
Huiting Zheng, Jiabin Yuan, and Long Chen.
Short-Term Load Forecasting Using EMD-LSTM Neural Networks with a
Xgboost Algorithm for Feature Importance Evaluation.
Energies, 100 (8), 2017.
ISSN 1996-1073.
URL <https://www.mdpi.com/1996-1073/10/8/1168>.

[Bouktif et al., 2018]
Salah Bouktif, Ali Fiaz, Ali Ouni, and Mohamed Adel Serhani.
Optimal Deep Learning LSTM Model for Electric Load Forecasting using
Feature Selection and Genetic Algorithm: Comparison with Machine Learning
Approaches †.
Energies, 110 (7), 2018.
ISSN 1996-1073.
URL <https://www.mdpi.com/1996-1073/11/7/1636>.

[Kwon Bo-Sungand Park et al., 2020]
Kwon Bo-Sungand Park, Rae-Junand Song, and Kyung-Bin.
Short-Term Load Forecasting Based on Deep Neural Networks Using LSTM
Journal of Electrical Engineering & Technology, 150
(4):0 1501–1509, 7 2020.
URL <https://doi.org/10.1007/s42835-020-00424-7>.

[Sajjad et al., 2020]
Muhammad Sajjad, Zulfiqar Ahmad Khan, Amin Ullah, Tanveer Hussain, Waseem
Ullah, Mi Young Lee, and Sung Wook Baik.
A Novel CNN-GRU-Based Hybrid Approach for Short-Term Residential
Load Forecasting.
IEEE Access, 8:0 143759–143768, 2020.
ISSN 21693536.

[Rafi et al., 2021]
Shafiul Hasan Rafi, Nahid Al-Masood, Shohana Rahman Deeba, and Eklas Hossain.
A short-term load forecasting method using integrated CNN and LSTM
IEEE Access, 9:0 32436–32448, 2021.
ISSN 21693536.

[Wang et al., 2019]
Xin Wang, Fang Fang, Xiaoning Zhang, Yajuan Liu, Le Wei, and Yang Shi.
LSTM-based Short-term Load Forecasting for Building Electricity
In 2019 IEEE 28th International Symposium on Industrial
Electronics (ISIE), pages 1418–1423, 2019.

[Memarzadeh and Keynia, 2021]
Gholamreza Memarzadeh and Farshid Keynia.
Short-term electricity load and price forecasting by a new optimal
LSTM-NN based prediction algorithm.
Electric Power Systems Research, 192:0 106995, 2021.
ISSN 0378-7796.

[Yuan et al., 2021]
Tzu-Lun Yuan, Dian-Sheng Jiang, Shih-Yun Huang, Yuan-Yu Hsu, Hung-Chih Yeh,
Mong-Na Lo Huang, and Chan-Nan Lu.
Recurrent Neural Network Based Short-Term Load Forecast with Spline
Bases and Real-Time Adaptation.
Applied Sciences, 110 (13), 2021.
ISSN 2076-3417.
URL <https://www.mdpi.com/2076-3417/11/13/5930>.

[Lee and Cho, 2022]
Juyong Lee and Youngsang Cho.
National-scale electricity peak load forecasting: Traditional,
machine learning, or hybrid model?
Energy, 239:0 122366, 2022.
ISSN 0360-5442.

[Zhao Wenhui et al., 2022]
Zhao Wenhui, Li Tong, Xu Danyang, and Wang Zhaohua.
A global forecasting method of heterogeneous household short-term
load based on pre-trained autoencoder and deep-LSTM model.
Annals of Operations Research, 12 2022.
ISSN 1572-9338.
URL <https://doi.org/10.1007/s10479-022-05070-y>.

[Oreshkin et al., 2020]
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio.
N-BEATS: Neural basis expansion analysis for interpretable time
series forecasting.
In International Conference on Learning Representations, 2020.
URL <https://openreview.net/forum?id=r1ecqn4YwB>.

[Singh et al., 2022]
Neelesh Pratap Singh, Aniket Ramendrakumar Joshi, and Mahamad Nabab Alam.
Short-Term Forecasting in Smart Electric Grid Using N-BEATS.
ICPC2T 2022 - 2nd International Conference on Power, Control
and Computing Technologies, Proceedings, 2022.

[Yin and Xie, 2021]
Linfei Yin and Jiaxing Xie.
Multi-temporal-spatial-scale temporal convolution network for
short-term load forecasting of power systems.
Applied Energy, 283:0 116328, 2 2021.
ISSN 0306-2619.

[Huy et al., 2022]
Pham Canh Huy, Nguyen Quoc Minh, Nguyen Dang Tien, and Tao Thi Quynh Anh.
Short-Term Electricity Load Forecasting Based on Temporal Fusion
Transformer Model.
IEEE Access, 10:0 106296–106304, 2022.
ISSN 21693536.

[Giacomazzi et al., 2023]
Elena Giacomazzi, Felix Haag, and Konstantin Hopf.
Short-Term Electricity Load Forecasting Using the Temporal Fusion
Transformer: Effect of Grid Hierarchies and Data Sources.
arXiv, 5 2023.

[Iman et al., 2023]
Mohammadreza Iman, Hamid Reza Arabnia, and Khaled Rasheed.
A Review of Deep Transfer Learning and Recent Advancements.
Technologies 2023, Vol. 11, Page 40, 110 (2):0
40, 3 2023.
ISSN 2227-7080.
URL <https://www.mdpi.com/2227-7080/11/2/40/htm

[Jung et al., 2020]
Seung-Min Jung, Sungwoo Park, Seung-Won Jung, and Eenjun Hwang.
Monthly Electric Load Forecasting Using Transfer Learning for Smart
Sustainability, 120 (16), 2020.
ISSN 2071-1050.
URL <https://www.mdpi.com/2071-1050/12/16/6364>.

[Cai et al., 2020]
Long Cai, Jie Gu, and Zhijian Jin.
Two-Layer Transfer-Learning-Based Architecture for Short-Term Load
IEEE Transactions on Industrial Informatics, 160
(3):0 1722–1732, 2020.

[Abdulrahman et al., 2022]
Mustapha Lawal Abdulrahman, Abdusalam Ya u. Gital, Kabiru Musa Ibrahim,
Fatima Umar Zambuk, Ibrahim Muhammad Umar, and Zahraddeen Ismail Yakubu.
Predicting Electricity Consumption in Residential Building's Using
Deep Transfer Learning with Attention-LSTM.
2022 International Mobile and Embedded Technology Conference,
MECON 2022, pages 540–546, 2022.

[Zhao et al., 2023]
Pengfei Zhao, Di Cao, Yanbo Wang, Zhe Chen, and Weihao Hu.
Gaussian Process-Aided Transfer Learning for Probabilistic Load
Forecasting Against Anomalous Events.
IEEE Transactions on Power Systems, 380 (3):0
2962–2965, 5 2023.
ISSN 15580679.

[Zhang et al., 2023]
Zhenhao Zhang, Jiefeng Liu, Senshen Pang, Mingchen Shi, Hui Hwang Goh, Yiyi
Zhang, and Dongdong Zhang.
General short-term load forecasting based on multi-task temporal
convolutional network in COVID-19.
International Journal of Electrical Power & Energy Systems,
147:0 108811, 5 2023.
ISSN 0142-0615.

[Syed et al., 2023]
Dabeeruddin Syed, Ameema Zainab, Shady S. Refaat, Haitham Abu-Rub, Othmane
Bouhali, Ali Ghrayeb, Mahdi Houchati, and Santiago Bañales.
Inductive Transfer and Deep Neural Network Learning-Based
Cross-Model Method for Short-Term Load Forecasting in Smarts Grids.
IEEE Canadian Journal of Electrical and Computer Engineering,
460 (2):0 157–169, 3 2023.
ISSN 26941783.

[Campos et al., 2023]
Enrique Mármol Campos, Aurora González Vidal, José L. Hernández Ramos,
and Antonio Skarmeta.
Federated Transfer Learning for Energy Efficiency in Smart
IEEE INFOCOM 2023 - Conference on Computer Communications
Workshops, INFOCOM WKSHPS 2023, 2023.

[Tan et al., 2020]
Mao Tan, Siping Yuan, Shuaihu Li, Yongxin Su, Hui Li, and Feng He He.
Ultra-Short-Term Industrial Power Demand Forecasting Using LSTM
Based Hybrid Ensemble Learning.
IEEE Transactions on Power Systems, 350 (4):0
2937–2948, 7 2020.
ISSN 15580679.

[Hu et al., 2019]
Weizheng Hu, Yong Luo, Zongqing Lu, and Yonggang Wen.
Heterogeneous transfer learning for thermal comfort modeling.
BuildSys 2019 - Proceedings of the 6th ACM International
Conference on Systems for Energy-Efficient Buildings, Cities, and
Transportation, pages 61–70, 11 2019.
URL <https://dl.acm.org/doi/10.1145/3360322.3360843>.

[Ma et al., 2020]
Jun Ma, Jack C.P. Cheng, Feifeng Jiang, Weiwei Chen, Mingzhu Wang, and Chong
A bi-directional missing data imputation scheme based on LSTM and
transfer learning for building energy data.
Energy and Buildings, 216:0 109941, 6 2020.
ISSN 0378-7788.

[Khalil et al., 2021]
Mohamad Khalil, Stephen McGough, Zoya Pourmirza, Mehdi Pazhoohesh, and Sara
Transfer Learning Approach for Occupancy Prediction in Smart
2021 12th International Renewable Engineering Conference, IREC
2021, 4 2021.

[Feng et al., 2020]
Cong Feng, Ali Mehmani, and Jie Zhang.
Deep Learning-Based Real-Time Building Occupancy Detection Using AMI
IEEE Transactions on Smart Grid, 110 (5):0
4490–4501, 9 2020.
ISSN 19493061.

[Chen et al., 2020]
Yujiao Chen, Yang Zheng, and Holly Samuelson.
Fast Adaptation of Thermal Dynamics Model for Predictive Control of
HVAC and Natural Ventilation Using Transfer Learning with Deep Neural
Proceedings of the American Control Conference,
2020-July:0 2345–2350, 7 2020a.
ISSN 07431619.

[Chen et al., 2020]
Yujiao Chen, Zheming Tong, Yang Zheng, Holly Samuelson, and Leslie Norford.
Transfer learning with deep neural networks for model predictive
control of HVAC and natural ventilation in smart buildings.
Journal of Cleaner Production, 254:0 119866, 5
ISSN 0959-6526.

[Demianenko and De Gaetani, 2021]
Mikhail Demianenko and Carlo Iapige De Gaetani.
A Procedure for Automating Energy Analyses in the BIM Context
Exploiting Artificial Neural Networks and Transfer Learning Technique.
Energies 2021, Vol. 14, Page 2956, 140 (10):0
2956, 5 2021.
ISSN 1996-1073.
URL <https://www.mdpi.com/1996-1073/14/10/2956/htm

[Kazmi et al., 2019]
Hussain Kazmi, Johan Suykens, and Johan Driesen.
Large-scale transfer learning for data-driven modelling of hot water
Building Simulation Conference Proceedings, 4:0
2611–2618, 2019.
ISSN 25222708.

[Wu Di et al., 2020]
Wu Di, Wang Boyu, Precup Doina, and Boulet Benoit.
Multiple Kernel Learning-Based Transfer Regression for Electric Load
IEEE Transactions on Smart Grid, 110 (2):0
1183–1192, 3 2020.
ISSN 19493061.

[ENTSO-E, 2022]
ENTSO-E Website, 2022b.
URL <https://www.entsoe.eu/>.

[Alla and Adari, 2021]
Sridhar Alla and Suman Kalyan Adari.
Beginning MLOps with MLFlow.
Apress, 2021.

[Peppanen et al., 2016]
Jouni Peppanen, Xiaochen Zhang, Santiago Grijalva, and Matthew J. Reno.
Handling bad or missing smart meter data through advanced data
2016 IEEE Power and Energy Society Innovative Smart Grid
Technologies Conference, ISGT 2016, 12 2016.

[Ferreira and Hitchcock, 2009]
Laura Ferreira and David B Hitchcock.
A Comparison of Hierarchical Methods for Clustering Functional
http://dx.doi.org/10.1080/03610910903168603, 380
(9):0 1925–1949, 7 2009.
ISSN 03610918.

[Vijaya et al., 2019]
V Vijaya, Shweta Sharma, and Neha Batra.
Comparative Study of Single Linkage, Complete Linkage, and Ward
Method of Agglomerative Clustering.
Proceedings of the International Conference on Machine
Learning, Big Data, Cloud and Parallel Computing: Trends, Prespectives and
Prospects, COMITCon 2019, pages 568–573, 7 2019.

[Weiss et al., 2016]
Karl Weiss, Taghi M. Khoshgoftaar, and Ding Ding Wang.
A survey of transfer learning.
Journal of Big Data, 30 (1):0 1–40, 12 2016.
ISSN 21961115.

[Bergstra et al., 2011]
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.
Algorithms for Hyper-Parameter Optimization.
Advances in Neural Information Processing Systems, 24, 2011.

[Akiba et al., 2019]
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.
Optuna: A Next-generation Hyperparameter Optimization Framework.
In Proceedings of the ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pages 2623–2631. Association for
Computing Machinery, 7 2019.
ISBN 9781450362016.

[82]
Ensemble PyTorch.
Ensemble PyTorch | Documentation.
URL <https://ensemble-pytorch.readthedocs.io/en/latest/>.

[Hyndman and Koehler, 2006]
Rob J. Hyndman and Anne B. Koehler.
Another look at measures of forecast accuracy.
International Journal of Forecasting, 220
(4):0 679–688, 10 2006.
ISSN 01692070.

[84]
Grid Map.
URL <https://www.entsoe.eu/data/map/>.

§ TABLES

Training time and hyperparameters of the best-performing setups in terms of MAPE. Each line of table represents a country, containing (a) the name of its best performing setup (b) the training time of the target model (fine-tuning process) (c) the training time of the baseline model (d) hyperparameters of the best model.
width=, totalheight=, keepaspectratio

Country     Best performing setup     3c|Pipeline duration (minutes)     5cBest hyperparameters

source     target     baseline     lookback window     learning rate     layer number     layer sizes     batch size

Italy            AbO     402        20      51     168     0.000206     4     {1024,256,2048,1024}       1024

Croatia          CbO     90       17      40     168     0.000162     3     {256,256,1024}             256

Spain            AbO     402      16      50     168     0.000224     2     {256,2048}                 512

Greece           CbO     72       24      58     504     0.000313     5     {512,256,512,128,2048}     1024

Serbia           CbO     126       21      38     168     0.000617     3     {2048,1024,1024}           256

Portugal         CbO     168      29        44     672     0.000541     4     {1024,2048,512,1024}       512

Belgium          CbO     114      21        47       168     0.000839     2     {128,512}                  256

Ireland          AbO     546      29      33     336     0.000116     4     {512,256,512,2048}         256

Netherlands     AbO      492      23      55     168     0.000511     3     {1024,2048,128}            256

France          CbO      138       25      46     168     0.000428     3     {512,2048,2048}            256

Romania         AbO      534      25      46     336     0.000213     3     {512,256,256}              256

Bulgaria        AbO      1134        32      47     336     0.000169     4     {2048,256,1024,256}        256

Finland         CbO      156      23      46     168     0.000307     3     {512,1024,256}             256

Hungary         CbO      132       26      46     168     0.000928     4     {128,512,512,128}          256

Germany         CbO     156       14       31     504     0.000810     4     {128,256,128,2048}         256

Slovakia        AbO     600       26      66       168     0.000130     3     {2048,256,512}             256

Austria         AbO     348       19      49       168     0.000644     4     {2048,128,512,256}         512

Slovenia        CbO     168       29        36     168     0.000385     5     {512,256,128,128,128}      256

Poland          CbO     150       25      54     504     0.000368     4     {128,2048,1024,256}        512

Lithuania       CbO     174       25      32     504     0.000240     2     {1024,1024}                256

Switzerland     CbO     162       28      54     336     0.000136     5     {1024,256,2048,512,512}    256

Norway          CbO     150       22      51     168     0.000188     2     {128,1024}                 256

Denmark         CbO     126         20      44     168     0.000140     3     {2048,1024,512}            512

Estonia         AbO     546       25      58       336     0.000117     4     {512,1024,512,2048}        256

Czechia         AbO     498       24      44     336     0.000165     2     {1024,2048}                256

Latvia          AbO     1584       40      46     168     0.000124     4     {1024,256,2048,512}        256

Sweden          AbO     570       30      60       168     0.000278     4     {256,512,256,128}          256