# Structural hierarchical learning for energy networks

Julien Leprince  Technical University of Eindhoven, 5 Groene Loper, Eindhoven
5600 MB, the Netherlands<EMAIL_ADDRESS>Waqas Khan<EMAIL_ADDRESS>Henrik
Madsen  Technical University of Denmark, Building 303B Matematiktorvet, Lyngby
2800, Denmark<EMAIL_ADDRESS>Jan Kloppenborg Møller<EMAIL_ADDRESS>Wim Zeiler
<EMAIL_ADDRESS>
###### Abstract

Many sectors nowadays require accurate and coherent predictions across their
organization to effectively operate. Otherwise, decision-makers would be
planning using disparate views of the future, resulting in inconsistent
decisions across their sectors. To secure coherency across hierarchies, recent
research has put forward hierarchical learning, a coherency-informed
hierarchical regressor leveraging the power of machine learning thanks to a
custom loss function founded on optimal reconciliation methods. While
promising potentials were outlined, results exhibited discordant performances
in which coherency information only improved hierarchical forecasts in one
setting. This work proposes to tackle these obstacles by investigating custom
neural network designs inspired by the topological structures of hierarchies.
Results unveil that, in a data-limited setting, structural models with fewer
connections perform overall best and demonstrate the value brought by
coherency information in both accuracy and coherency forecasting performances,
provided individual forecasts were generated within reasonable accuracy
limits. Overall, this work expands and improves hierarchical learning methods
thanks to a structurally-scaled learning mechanism extension coupled with
tailored network designs, producing a resourceful, data-efficient, and
information-rich learning process.

###### keywords:

Deep learning, Hierarchical forecasting, Coherency, Constrained learning,
Smart grids

††journal: International Journal of Forecasting

##  1 Introduction

Optimal decision-making necessitates an accurate estimation of the future in
order to effectively perform. This is true in all domains. However, in many
cases, accurate forecasts alone do not suffice. In particular, systems
organized over multiple abstraction levels principally require their
predictions to be consistent across all considered layers, not to result in
unaligned decisions [1]. This phenomenon has become a concern in numerous
domains encompassing retail [2], stock management [3], tourism [4, 5] as well
as smart energy management [6, 7]. The latter illustrates the situation quite
well while being at the center of current preoccupations toward decarbonizing
existing energy systems [8, 9]. Typically, energy networks are composed of
multiple abstraction levels, divided into groups that further subdivide into
groups of groups over multiple layers, referred to as hierarchies or trees.
Such structures possess varying scales of information, which are commonly
employed disparately to produce element-specific forecasts across the network
elements, so-called independent or base forecasts [5]. Further, energy network
operators now profit from high-frequency measurements at multiple levels of
the system, backing accurate predictions across different aggregations, i.e.,
with per seconds to yearly resolutions across sub-meters to regional
aggregated information [6, 10, 11]. However, the diversity of these models
inevitably produces inconsistencies across aggregation levels, i.e., lower-
level forecasts may not add up to higher-level ones and vice-versa [12]. The
resulting challenge decision-makers are confronted with is obtaining coherent
forecasts across disparate levels of the system. A hierarchy is defined as
coherent when its values at the disaggregate and aggregate scales are equal
when brought to the same level [4]. Optimal decision-making consequently
necessitates not only accurate but coherent forecasts in order to effectively
perform.

###  1.1 Securing coherency

In this context, data-driven approaches have been proposed to produce coherent
predictions from disconnected, and presumably inconsistent, base forecasts [5,
13], referred to as reconciliation [14]. The process exploits linear balancing
equations from covariance compositions inherent to hierarchical time series to
optimally re-adjust coherency mismatches. Prevalent techniques encompass
elemental bottom-up and top-down approaches, as well as trace minimization
from optimal combinations referred to as generalized least-squares.

Bottom-up reconciliation consists in aggregating together across the hierarchy
the base forecasts of the very bottom level of the hierarchy, i.e., the tree
leaves, thus enforcing its coherency [15]. The approach benefits from detailed
information on the most disaggregate levels of the tree, avoiding subsequent
information loss from aggregation [5]. It is, however, poorly suited to time
series displaying low signal-to-noise ratios as bottom-up aggregation would be
unlikely to provide accurate forecasts across the upper levels of the tree
[13].

Top-down reconciliation, on the other hand, proceeds to disaggregate and
distribute the top-level forecast of the hierarchy, i.e., tree root, either
from historical [16] or forecasted [5] proportions of the data. The method
notably produces accurate predictions for higher aggregation levels of the
tree, leveraging clear uncovered trends from aggregated time series, while
being valuable for low-count data. Yet, as a result of aggregation, large
quantities of information cannot be exploited, such as temporal dynamics [5].
The approach also relies on a unique top-level model, thus creating higher
performance risks originating from eventual model inaccuracies or
misspecification [17].

Optimal reconciliation subsequently emerged as a solution to exploit the full
extent of available information across a given hierarchy. It exploits
correlations and covariances present in hierarchical time series, linearly
reconciling base forecasts towards coherency with varied interactions between
tree levels. However, covariance structures of hierarchies are challenging to
estimate from base forecasts, such that even with high-frequency data
available, assumptions on its form must be made [18]. In this context,
ordinary (OLS) [5, 13], weighted (WLS) [19], and generalized least-square
(GLS) [20] estimators were proposed providing varying approximations of the
covariance matrix, namely identity, variance, and covariance respectively. The
approaches demonstrated improved results compared to other commonly adopted
technics while the generalized least-square formulation guaranteed reconciled
forecasts to be, in mean or in sample, as good as their original base
performance.

###  1.2 From hierarchical forecasting to hierarchical learning

Hierarchical forecasting thus emerged as a solution providing coherent
predictions within an established hierarchical structure. Typically, (base)
forecasts are first estimated across the tree in a disjointed manner. This
allows models to be hierarchically-tailored to the considered element of the
hierarchy while exploiting available node-specific information [4]. Then,
forecasts are linearly combined (reconciled), securing base forecast coherency
from structural hierarchical information. This phase brings formerly detached
elements together by sharing targeted information between aggregation levels
to generate coherent predictions.

Undertaking these two phases separately, however, engenders two noteworthy
shortcomings; it inherently deprives similar forecasting models from (i) the
benefits of data information (learning) transfer across models, as well as
(ii) structural hierarchical information together with coherency requirements
which cannot be exploited by forecasting models despite their valuable
information.

Thereby, recent work suggested bridging forecasting with the coherency
requirements of the reconciliation phase [21]. The approach proposes (i) a
unique hierarchical forecasting model, providing a global overview of
information across the hierarchy to the regressor, while (ii) including
coherency requirements within its learning process. Notably, it puts forward a
custom loss function, embedding optimal reconciliation information in the
learning process of deep neural networks founded on established field
taxonomy. Overall, the work expanded and united traditionally disjointed
methods together unveiling a novel generation of forecasting regressors:
hierarchical learners.

While the work portrays a pioneering and promising method for hierarchical
learning, results, however, displayed disparate performances, in which
coherency learning was found to improve forecasting scores in simply one
setting. The analysis particularly unveiled two important challenges the
approach struggled with; laborious learning and faulty coherent learning from
scaled trees. Indeed, while unifying hierarchical predictions under one
machine-learning model fuels the regressor with a rich data-learning process,
the consequent size of the produced model renders the learning process of the
algorithm arduous. The produced number of weights to update possesses a
$n\cdot m$ relationship between layers of size $n$ and $m$, which
exponentially explodes as the hierarchy grows larger. Additionally, estimated
forecasts of large hierarchical multi-output regressors might produce either
conflicting or highly correlated outputs, that are challenging to capture from
limited data. Further, hierarchical-coherent learning exhibited faulty
performances in certain settings, where top-level forecasts mirrored their
expected values in the negative domain. The identified undesirable result was
traced back to the normalization of hierarchical time-series required for
adequate learning, which was shown to irregularly bias the coherency learning
process.

In light of this, tailored designs need to be examined to exploit the full
potential of hierarchical learning methods.

###  1.3 Motivation

This paper intends to address identified complications by investigating custom
neural network designs mirroring the structural dimensions of hierarchies.
Namely, we propose first to divide large hierarchical models into separate
partitions, thus effectively reducing the number of weights to learn for a
more data-efficient training process while producing distinct model components
tailored to node-specific elements. Second, we connect separated model
partitions leveraging established structural hierarchical designs, hence
incorporating structural information accurately into the model layout while
allowing separated models to share specific information with one another.
Finally, leveraging batch normalization within the design of neural networks,
we introduce a learning process robust to the scale differences endowed with
hierarchies, avoiding a priori tree normalization and its identified
subsequent biased coherency learning. Contributions of this work can ergo be
summarized as three-fold;

  1. 1.

We propose to expand promising hierarchical learning regressors by putting
forward a novel structural hierarchical learning method. Founded on tailored
network designs, the approach exploits topological hierarchical information
from trees targeted to support a resourceful, data-efficient, and information-
rich learning process.

  2. 2.

We formally evaluate the relative performance brought by the addition of the
coherency requirement across all examined designs, thus clearly establishing
the realized value of coherent hierarchical learning.

  3. 3.

Finally, we open-source all developed implementations backing effective
knowledge dissemination and allowing full research reproducibility under a
public GitHub repository
111https://github.com/JulienLeprince/structuralhierarchicallearning.

With these contributions, this work adequately addresses the exposed
challenges of hierarchical learning. By putting forward tailored, ingenious
architectures of neural networks it effectively reduces hierarchical model
complexities while serving improved and coherency-aware hierarchical
forecasts. The work notably serves domains such as stock management, retail,
business analytics, and distribution networks, by providing improved and more
consistent forecasts across all levels of considered hierarchies.

The rest of this paper is organized as follows; the mathematical foundations
for the reconciliation of hierarchical time-series is first presented along
with hierarchical learning regressors and their structural extension in Sec.
2. Next, the implementation specifics related to the machine learning design
and evaluated case study are exposed under Sec. 3, followed by its results in
Sec. 4. Section 5 concludes the article.

##  2 Structural hierarchical learning

Coherency requirements and reconciliation technics feeding hierarchical
learning regressors are first presented, serving as a foundation on which
topologically-tailored model designs can be built.

###  2.1 Hierarchies and coherency

Let us introduce the simple hierarchy of Fig. 1 to illustrate the requirements
for coherency. Every element (node) of the hierarchy can be labeled as
$y_{kj}$, where subscripts k and j stand for the aggregation-level and node
observations respectively. We define $k_{1}=m$ as the most aggregate level of
the hierarchy (tree root), i.e., node $y_{61}$, and $k_{3}=1$ as the most
disaggregate level (tree leaves), i.e., nodes $y_{1j}$ where $j\in[1:m]$. In
such a setting, two important components must be considered; the number of
leaves, denoted as m, and the total number of nodes on the tree n. Here $n=9$
and $m=6$.

Figure 1: A two-level hierarchical tree diagram.

Stacking all hierarchical elements in a n-dimensional vector
$\boldsymbol{y}=(y_{61},y_{31},y_{32},y_{11},y_{12},y_{13},y_{14},y_{15},y_{16})^{T}$,
and bottom-level observations in an m-dimensional vector
$\boldsymbol{b}=(y_{11},y_{12},y_{13},y_{14},y_{15},y_{16})^{T}$, we can write

$\displaystyle\boldsymbol{y}=S\boldsymbol{b}\text{,}$ (1)

where $S$ is the summation matrix, here expressed as

$\displaystyle S=\begin{bmatrix}1&1&1&1&1&1\\\ 1&1&1&0&0&0\\\ 0&0&0&1&1&1\\\
\lx@intercol\hfil I_{m}\hfil\lx@intercol\\\ \end{bmatrix}\text{,}$ (2)

which is of dimension $n\times m$, and $I_{m}$ is an identity matrix of size
$m$. $S$ encapsulates the hierarchical structure of the tree, from which the
complete hierarchy $\boldsymbol{y}$ can be obtained given the tree leaves
$\boldsymbol{b}$. Consider how $S$ captures the coherency requirements within
the hierarchy, included here as the linear summations of the bottom-level
observations. By introducing a matrix

$\displaystyle G=\big{[}0_{m\hskip 1.42262pt\times\hskip 1.42262pt(n-m)}\hskip
1.42262pt|\hskip 1.42262ptI_{m}\big{]}\text{,}$ (3)

of order $m\hskip 1.42262pt\times\hskip 1.42262ptn$ that extracts the $m$
bottom-level forecasts, the reconciliation constraint is formulated as

$\displaystyle\tilde{\boldsymbol{y}}=SG\tilde{\boldsymbol{y}}\text{.}$ (4)

Reconciliation is necessary when base forecasts $\hat{\boldsymbol{y}}$ do not
satisfy this constraint [1]. In such situations, Eq. (4) becomes
$\tilde{\boldsymbol{y}}=SG\hat{\boldsymbol{y}}$, where $G$ maps the base
forecasts into the reconciled tree-leaves and $S$ sums these up to a set of
coherent forecasts $\tilde{\boldsymbol{y}}$. $SG$ can thus be thought of as a
reconciliation matrix taking the incoherent base forecasts as input and
reconciling them to $\tilde{\boldsymbol{y}}$. This formulation, however, is
sub-optimal as $G$, as defined in Eq. (3), only considers information from a
single level.

To include the exploitation of all aggregation levels in an optimal manner,
Hyndman et al. [13] and later, Van Erven and Cugliari [22] and Athanasopoulos
et al. [23] formulated the reconciliation problem, as linear regression
models. Exploiting a defined hierarchical structure, reconciled forecasts are
found employing the generalized least-squares problem:

minimize $\displaystyle\hskip
14.22636pt\big{(}\tilde{\boldsymbol{y}}-\hat{\boldsymbol{y}}\big{)}^{T}\Sigma^{-1}\big{(}\tilde{\boldsymbol{y}}-\hat{\boldsymbol{y}}\big{)}\text{,}$
(5) subject to $\displaystyle\hskip
14.22636pt\tilde{\boldsymbol{y}}=SG\tilde{\boldsymbol{y}}\text{,}$

where $\tilde{\boldsymbol{y}}\in\mathbb{R}^{n}$ is the decision variable of
the optimization problem and $S\in\mathbb{R}^{n\times m}$ and
$G\in\mathbb{R}^{m\times n}$ are constant matrices defined by the structure of
the hierarchy. The parameter $\Sigma\in\mathbb{R}^{n\times n}$ is the positive
definite covariance matrix of the coherency errors
$\varepsilon=\tilde{\boldsymbol{y}}-\hat{\boldsymbol{y}}$, which are assumed
to be multivariate Gaussian and unbiased, i.e., with zero mean.

If $\Sigma$ were known, the solution to (5) would be given by the generalized
least-squares (GLS) estimator

$\displaystyle\tilde{\boldsymbol{y}}=S\big{(}S^{T}\Sigma^{-1}S\big{)}^{-1}S^{T}\Sigma^{-1}\hat{\boldsymbol{y}}\text{,}$
(6)

which has been employed in close to all notable hierarchical forecasting works
over the last years [1, 4, 5, 6, 12, 13, 20, 23]. The precision matrix
$\Sigma^{-1}$ is used to scale discrepancies from the base forecasts and can
also be referred to as the weight matrix. The recurrent challenge in
estimating $\Sigma^{-1}$ originates from its dimension $n\times n$ which can
potentially become very large.

Varying approximations of the covariance matrix were consequently put forward
in previous work, including identity (ordinary least-square), structural [23],
variance [1, 23], Markov [1], and spectral [24] scaling, along with shrank
covariance estimations [1, 20]. All of these allow distinct information
sharing between elements of the hierarchy serving optimal reconciliation in
differing settings.

###  2.2 Hierarchical learning

The more recently proposed hierarchical learning method [21] intended to unify
hierarchical forecasts together while exploiting the coherency requirement of
produced predictions. The approach builds on the presented formulations of
optimal reconciliation, see Eq. (6), included in machine learning processes
thanks to a custom loss function.

To propose a regressor that is robust to scale differences and avoids
normalization requirements and its subsequent identified faulty-coherency
learning [21], we leverage the Mean Structurally-Scaled Square Error (MS3E)
defined in Ref. [21] for the definition of the learning loss function.
Structurally scaled errors must, indeed, be considered in hierarchical
forecasting as produced predictions will possess major scale differences
inherent to hierarchical structures. Designing hierarchical loss functions
notably differentiates itself in this way from the scaled multi-output
regressors of Ref. [21]. To avoid a biased learning process favoring the top
levels of the aggregation, where predicted values possess larger magnitudes,
such structural scale differences must be adjusted. To consider this, we
employ structural scaling leveraging the aggregation-level vector

$\displaystyle\boldsymbol{\kappa}_{str}=S\boldsymbol{1}_{m}\text{.}$ (7)

For example, for the hierarchy illustrated in Fig. 1,
$\boldsymbol{\kappa}_{str}$ = (6, 3, 3, 1, 1, 1, 1, 1, 1). The structural-
hierarchical loss function $\mathcal{L}^{h}$ can then be expressed as

$\displaystyle\mathcal{L}^{sh}\big{(}\mathcal{Y},\widehat{\mathcal{Y}}|\Theta\big{)}=\frac{1}{T}\sum^{T}_{t=1}\Big{(}(\boldsymbol{y}_{t}-\hat{\boldsymbol{y}}_{t})\oslash\boldsymbol{\kappa}_{str}\Big{)}^{2}\text{.}$
(8)

where $T$ is the number of time-steps, $\mathcal{L}^{sh}$ denotes the mean
structurally-scaled square loss function between the predicted independent
base forecast set $\widehat{\mathcal{Y}}$ subject to a set of parameters
$\Theta$ and a set of observed values $\mathcal{Y}$. The operator $\oslash$ is
a Hadamard division. It should be noted that while the proposed hierarchical
forecasting loss function employs the MS3E metric, any other accuracy metric
can be exploited provided they use structurally scaled error
$\boldsymbol{e}^{str}_{t}=(\boldsymbol{y}_{t}-\hat{\boldsymbol{y}}_{t})\oslash\boldsymbol{\kappa}_{str}$
as error reference.

The structural-coherency loss function is formulated as the structurally
scaled differences between predicted values $\hat{\boldsymbol{y}}$ and their
reconciled counterpart $\tilde{\boldsymbol{y}}$, following the reconciliation
product of Eq. (6). The coherency error $\boldsymbol{e}^{coh}_{t}$ and
subsequent scaled loss function $\mathcal{L}^{sc}$ can consequently be
expressed as

$\displaystyle\boldsymbol{e}^{coh}_{t}=\hat{\boldsymbol{y}}_{t}-S\big{(}S^{T}\Sigma^{-1}S\big{)}^{-1}S^{T}\Sigma^{-1}\hat{\boldsymbol{y}}_{t}\text{,}$
(9)
$\displaystyle\mathcal{L}^{sc}\big{(}\mathcal{Y},\widehat{\mathcal{Y}}|\Theta\big{)}=\frac{1}{T}\sum^{T}_{t=1}\Big{(}\boldsymbol{e}^{coh}_{t}\oslash\boldsymbol{\kappa}_{str}\Big{)}^{2}\text{.}$
(10)

Both structural accuracy and coherency losses are then combined together
forming the structural-hierarchical-coherent loss function
$\mathcal{L}^{shc}$,

$\displaystyle\mathcal{L}^{shc}\big{(}\mathcal{Y},\widehat{\mathcal{Y}}|\Theta\big{)}=\alpha\mathcal{L}^{sh}_{t}+(1-\alpha)\mathcal{L}^{sc}_{t}\text{,}$
(11)

where $\alpha\in[0,1]$ weights the structural hierarchical loss against the
structural coherency loss. This avoids the over-adjustment of weights during
the training of the regressor due to the addition of the coherency loss to the
loss function. We typically set $\alpha$ to 0.75 for structural hierarchical
learning.

###  2.3 Structural learning

Hierarchical learning regressors are here tailored to exploit topological
structures of hierarchies, allowing the design of bespoke structural
hierarchical models. This is accomplished by manipulating two distinct
characteristics of machine learning models, namely, neuron partitions and
network weights connecting partitions together. This work proposes to
ingeniously shape both features echoing common hierarchical attributes.

####  2.3.1 Partitioning hierarchical models

We begin by defining a single-task fully connected deep neural network of 3
hidden layers and 2 inputs, as illustrated by the (a) base model of Fig. 2.

Figure 2: Example illustrations of (a) base single-task fully-connected deep
neural network, composed 3 hidden layers and 2 inputs, followed by four multi-
task stacked variations of the former model where $y_{1}$ and $y_{2}$ relate
to parent and child nodes of a larger hierarchical ensemble respectively. The
multi-task models encompass (b) disconnected, (c) fully-connected, (d) bottom-
up, and (e) top-down partition links, creating a deep neural network, composed
of 3 similar hidden layers, 4 inputs, and 2 outputs.

To best conceptually illustrate network partitions, we propose to compare two
stacked variations of said single-task regressor (a), thus forming the multi-
task regressors (b) and (c). While both models possess similar number of
layers, neurons, inputs, and outputs, the weights connecting these elements
into a common model are here notably different. In model (c) all neurons
composing the hidden layers are connected to their preceding and succeeding
elements, i.e., input, hidden layer, or output, while in model (b) two
distinct partitions can be recognized formed by their parent regressor (a).
Naturally, in a common prediction task, it would not be considered valuable to
assemble two disconnected models together to form model (b). There would be no
performance or computational gains to expect from such a setup compared to
employing two independent models (a) instead for instance. In a hierarchical
forecasting setting, however, it becomes on the contrary rather beneficial to
gather produced outputs under the hood of one model. Indeed, in this way
coherency requirements of produced predictions can be exploited by leveraging
the coherency loss function. Additionally, it becomes clear that producing
specific partitions over a defined neural network considerably reduces the
number of weights to update. Taking, again, the illustrated example of Fig. 2,
model (c) possesses a total of 36 weights to update, while model (b) has only
20. This significant reduction in the number of weights to learn can serve two
desirable outcomes. First, fewer weights to learn implies fewer iterations
required to calibrate the model and subsequently fewer data instances, an
entity that is typically expensive to gather in large and qualitative
quantities. And, second, it can effectively support targeted single-task
learning by helping isolate eventual conflicting multi-output predictions from
one another in the model design.

We propose three partition variations, serving as alternatives to the fully-
connected hierarchical model of Ref. [21], echoing established tree
topologies, namely, fully-disconnected, leaf-connected, and k-level
partitions, which we visually introduce in Fig. 3 as vertical lattices. Fully
disconnected partitioning intends on mirroring independent, or base,
forecasting by considering each tree node as a disjointed partition of the
neural network. Similarly to the (b) model of Fig. 2, tree-node partitions are
stacked together to form a larger, hierarchical model. We refer to this
partitioning as tree. Leaf-connected partitioning, referred to as cutree,
suggests linking leaf elements with identical parents together while
considering the rest of the tree nodes as independent partitions. This setup
is particularly interesting for hierarchies built from time-series clustering,
where leaf elements typically display similar dynamics. Finally, k-level
partitioning proposes to group hierarchical elements possessing alike
aggregation levels into common partitions. The advantage of this approach is
granting time series with similar levels of aggregation, and subsequent
signal-to-noise ratios, a shared partition to capture these, possibly similar,
dynamics. We refer to this partitioning as klvl.

####  2.3.2 Creating topological bridges

While leveraging distinct node-specific models in hierarchical forecasting is
profitable, it becomes relevant to create bridges between separated layers,
allowing targeted and effective learning across defined hierarchical modeling
partitions. Two topological bridges have already been presented, namely
disconnected partitioning, or disc, and fully connected ones.

To complement these we propose two well-established hierarchical connections,
namely, (d) bottom-up and (e) top-down illustrated in Fig. 2, to bridge the
disconnected partitions of $y_{1}$ and $y_{2}$ together. We here assume
$y_{1}$ and $y_{2}$ to be parent and child nodes within a larger hierarchical
ensemble respectively. The bottom-up setup connects the lower levels of the
hierarchy with their closest higher-level elements. This implies creating
neuron links, or weights, between each element of lower-level partitions,
$y_{2}$, and their next respective parent partition, $y_{1}$. We refer to this
partitioning as bu. Top-down topological bridging works in the opposite way.
It creates connections between higher-level elements of the hierarchy and
their associated children, i.e., here connecting $y_{1}$ partition elements to
its child $y_{2}$. This partitioning is thereafter mentioned as td. Lastly, a
subsequent topological bridge combining the above methods is put forward,
i.e., bottom-up and top-down, or butd.

Figure 3 visually summarizes all of the introduced partitions and topological
bridges, exemplified on a two-level hierarchy.

Figure 3: Schematic illustration of introduced model partitions (vertical) and
topological bridges (horizontal) on a two-level hierarchy

##  3 Implementation

This section details the implementation-related details of our study, namely,
the considered hierarchical time-series and predictive-learning setup.

###  3.1 Hierarchical time-series

Our study considers a open large data set of building smart-meter
measurements, namely the Building Data Genome project 2 (BDG2) [25]. This open
set was selected to allow reproducibility of our method while putting forward
an initial benchmark for hierarchical forecasting in the building sector. The
BDG2 includes 3053 energy meters from 1636 non-residential buildings grouped
by site located in Europe and, principally, North America. The set covers two
full years (2016–2017) at an hourly resolution with multi-meter building
measurements paired with site weather data. Due to the heavy computational
workloads arising from hierarchical forecasts, we select the Fox site of the
BDG2 for the remainder of this study. The site is composed of 133 meter-
readings, which is deemed large enough to appreciate the aggregation of large
numbers of building electrical loads, while not developing into too
computational burdensome forecasting models.

Figure 4: Spatial tree structure of the Fox site of the BDG2 dataset

A spatial hierarchical structure is defined from the hierarchical clustering
of the prediction target time series, i.e., electricity demand. Buildings are
consequently clustered employing the Ward variance minimization algorithm
[26], which iteratively groups similar buildings together by minimizing the
variance of the euclidean distances between their electrical loads, resulting
in a large hierarchical structure of 265 nodes. The obtained hierarchy is
reduced in size by cutting the tree using a defined distance threshold over
visual inspection of the derived dendrogram. In this way, hierarchical
structures located below the threshold will be clustered together, thus,
effectively reducing the number of nodes to 140. Figure 4 illustrates the
attained reduced tree of the Fox site, resulting in a power distribution-like
structure, where multiple buildings are typically connected to similar buses
over the low-voltage grid.

###  3.2 Model learning setup

We proceed to resample the time-series to hourly intervals. Time-series with
cumulative missing values larger than 2 hours are removed, and smaller gaps
are interpolated via a moving average using a window size of 8 hours. We set
the prediction horizon to one time-step (hour) ahead.

####  3.2.1 Feature engineering

Features are selected based on their Maximum Information Coefficient (MIC)
[27] computed in relation to the electrical load learning target. MIC is a
powerful indicator that captured a wide range of associations both functional
and not while providing a score that roughly equals the coefficient of
determination (R2) of the data relative to the regression function. It ranges
between values of 0 and 1, where 0 implies statistical independence and 1 a
completely noiseless relationship. The advantage of using MIC for feature
engineering over the more commonly employed pearson correlation indicator [28]
is that it captures non-linear relationships present in the data, which deep-
learning models are popularly capable of detecting. We retain features
exhibiting MIC values higher than 0.25, as electric loads can typically become
quite volatile and impede MIC values with noise.

Additionally, to feed the learner with the most relevant historical
information of the predicted target, we select the 3 top auto-correlation
values per temporal aggregation level above 0.25 as model input features. If
no target auto-correlation value is above 0.25, we consider the most recent
historical information, i.e., $t$ where $t+1$ is the predicted time-step and
the forecast horizon is 1 hour.

Both MIC and autocorrelation selection thresholds are settings that should
typically be included in the hyper-parameter optimization of the model
validation phase. This work considers the tuning of these thresholds, however,
to lay outside its scope as these rapidly become excessively burdensome.

####  3.2.2 Data partitioning and covariance matrix estimation

Training and testing sets are constructed employing a times-series cross-
validator of the sklearn package [29], i.e., TimeSeriesSplit, with equal test-
size in a rolling window setup. The data set is divided into 10 batches, over
which the model is trained and tested from according sets. We employ the
heterogeneous variance approximation of the covariance matrix, which displayed
good prediction performances in Ref. [21]. The heterogeneous variance includes
separate variance estimates for each node. With the example hierarchy of Fig.
1 this gives
$\Sigma_{hvar}=\text{diag}({\sigma}^{2}_{61},{\sigma}^{2}_{31},{\sigma}^{2}_{32},{\sigma}^{2}_{11},\ldots,{\sigma}^{2}_{17})$.
The covariance matrix is recursively estimated in the test sets. For the first
batch training, we employ the identity covariance estimate $id$ as no
forecasts are yet available. Each batch training $i$ then comes with a new
covariance matrix estimate $\Sigma_{i}$ that is employed in the coherency loss
function of the next training set $i+1$. This setup echoes the adaptive
covariance matrix estimation proposed by [7] employed for temporal
hierarchies, anchored here quite organically in the learning process of neural
networks.

####  3.2.3 Regressor design

To build onto the design of Ref. [21], we pursue the implementation of
structural hierarchical regressors with deep neural networks, here designed
according to three predominant features: partition widths, sequential layer
depth, and topological bridges (weights).

Each partition is designed as a series of sequential layers decreasing
proportionally in size, from the defined input layer width $w_{1p}$ to the
desired output dimension $w_{D}$, such that

$\displaystyle w_{ip}=(w_{1p}-w_{Dp})\cdot\frac{i}{D}\text{ ,}$ (12)

defines a partition’s width in function of its design depth $D$. The
subscripts $i$ and $p$ stand for the sequential layer depth and sequential
layer index respectively where $i\in[1,...,D]$ and $p\in[1,...,P]$.
Aggregating the partitions together in the regression model then produces $n$
forecasts, ensuring that $\sum_{p}w_{Dp}=n$. We select the aggregated number
of features per partition as the input layer width $w_{1p}$.

Between each sequential partition, we further introduce batch normalization
and dropouts, both serving different purposes. Batch normalization is a
technique to standardize activations in intermediate layers of deep neural
networks across mini-batches. It has demonstrated improved accuracies and
faster convergences due to its stabilization of the learning process [30].
Additionally, introducing batch normalization allows the in and outputs of the
regression model to remain unscaled, thus retaining the hierarchical structure
of the coherency-loss function. This is an essential design improvement from
Ref. [21], which allows the tackling of observed faulty-coherency learning
engendered from scaled trees. Dropout is a technique introduced by N.
Srivastava et al. [31] designed to prevent overfitting by combining
exponential numbers of combinations of neural network architectures
efficiently. The term “dropout” refers to dropping out units of a neural
network. Dropped-out units are removed from the network, along with all their
incoming and outgoing connections, thus producing a thinned network. In
essence, dropout simulates model assembling without creating multiple networks
[32] while increasing convergence time.

Topological bridges are then established between neurons of initially
disconnected partitions following the presented connections of Sec. 2.3.2,
namely disconnected (disc), bottom-up (bu), top-down (td), and bottom-up top-
down (butd).

The optimal number of layers of the model is selected heuristically based on
prediction performances while increasing step-wise the network’s depths
starting from shallow 1-layer perceptrons. This allows the selected
architecture to serve an ”as simple as possible yet as complex as necessary”
design. We select similar hyper-parameters as Ref. [21], resulting in 3
layers, leveraging sigmoid activation functions and dropout ratios of 0.2 on
all but the last layer favoring a linear activation and no dropouts. The
retained $\alpha$ coefficient value is 0.75. The presented models of Sec. 2
were implemented in Python using the TensorFlow package [33].

##  4 Results and discussion

We describe and discuss the outcome of the implementation here. In particular,
we evaluate the accuracy and coherency of the forecasted building loads
outlined in varying heatmaps allowing insights into the performances of the
forecast across the tree and forecasting methods. The improvement ratio
brought by the coherency loss function is also highlighted both for accuracy
and coherency forecast performances.

###  4.1 Forecast accuracy

Figure 5: Heatmap of the accuracy performance in Root Mean Structurally-Scaled
Square Error (RMS3E) across tree nodes and forecasting methods. Structural
hierarchical forecasting methods described against the y-axis are designated
by their respective partition, topological bridge, and considered loss
function, and are here sorted according to their forecasted accuracy.

Figure 5 presents the forecasted accuracy of all evaluated methods over the
tree nodes, sorted by their performances across the overall hierarchy. Both
extreme values of the heatmap present the tree partitioning with bottom-up
(bu) connections and structural hierarchical-coherent loss function (shc) as
the better performer across the forecasting methods, while the k-level
partitions with top-down (td) topological bridges and shc loss function
performs the worst, by an impressive 8 order of magnitude RMS3E difference.
The notable better performers possess RMS3Es ranging from 42 to 100 kWh and
all bear tree partitions that are either bridged in a disconnected (disc) or
bu fashion. These two leading contenders each perform best with the inclusion
of the coherency requirement in the loss function, i.e., shc versus sh. On the
other end of the heatmap, we can regroup flawed performers ranging from 3.4e5
to 2.1e8 kWh RMS3E. The structural characteristics of these networks display
k-level, cut-tree, and full partitions coupled to varying topological bridges,
mostly td and bottom-up-top-down (butd).

The tendencies that can be extracted from Figure 5 expose that (i) structural
models with fewer connections perform overall better than models with larger
numbers of connections, and (ii) that within good performers, the inclusion of
coherency information in the loss function improves the performance of the
overall accuracy of the forecast.

Indeed, considering the number of connections per topological-design places
the tree partition as the one with the least amount of connections, followed
by cut-tree, k-level, and full partitions. Inter-partition connections follow
the logical increasing ordering of disconnected, bu/td, and butd. It is
consequently observed that tree partitions perform best as they result in
narrower layers compared to cut-tree and k-level ones. However, the least
connected model design, tree-disc, stands as the second best performer, thus
demonstrating that some amount of information exchange between hierarchical
layers, here bu, is valuable for the performance of the forecast. The flawed
performers exhibit similar inclinations, where k-level partitions perform
overall worse than cut-tree ones, which disregard the connections between
leaves of dissimilar parents, thus cutting down their numbers. Then, the
inclusion of coherency information in the learning mechanism of the regressor
produces improved forecasts for the better half of the models, with the
exception of a few cases, namely td and butd trees. This will be further
discussed under Sec. 4.3.

It can be noticed that the td connections systematically perform much worse,
by at least a RMS3E order of magnitude, than their bu counterparts, i.e.,
within similar layer partitionings and loss functions. The only exception that
ignores this observation is the cut-tree partitioning with sh loss. This
poorer performance of the td connection also seems to negatively impact the
performance of its derivative butd. In turn, the butd linkage exclusively
performs worse than its bu setup, in similar neural network designs. This
topological bridge design, indeed, suffers from the influence of meager td
performances coupled with greater numbers of weights to learn, in a data-
limited setting.

Lastly, a few peculiar cases seem to produce results that deviate from
observable trends. The fully connected model, although possessing a larger
amount of weights by design is surprisingly not amongst the worst performers.
It also displays a much more uniform forecasting performance across its
hierarchy than its neighboring k-level or cut-tree models. Both observations
can be explained by the fact that it possesses a number of connections in a
similar order to k-level and cut-tree partitions while profiting from a more
uniform design. This allows the dropout layer to reduce the network in an
unconstrained manner, thus functioning under optimal conditions. Another
peculiar behavior can be examined under the tree td with shc loss which
displays few, but impacting, poor performances across its hierarchy, thus
negatively affecting its mean accuracy performance.

###  4.2 Forecast coherency

Figure 6: Heatmap of the accuracy and coherency Root Mean Structurally-Scaled
Square Error (RMS3E) across the forecasting methods. Structural hierarchical
forecasting methods described against the y-axis are designated by their
respective partition, topological bridge, and considered loss function, and
are here sorted according to their forecasted coherency.

While information exchange across a hierarchy in a forecasting setting has
demonstrated accuracy gain potentials, the coherency improvements of the
produced hierarchical time series must be evaluated. Figure 6 subsequently
presents the coherency RMS3E, as defined by Eq. (10), sorted across evaluated
hierarchical model designs. The coherency errors can be compared to their
associated accuracy biases, thus providing a complete overview of a method’s
performance.

The models producing the most coherent forecast range between 5.2e2 and 9.7e2
kWh RMS3E and all benefit from tree partitions, either connected in a bu,
butd, or disc fashion, by decreasing order of performance respectively. These
top coherency performers also relate to top accuracy ones, with the exception
of the tree-butd model. While both tree-bu and tree-butd produce slightly more
coherent forecasts without the inclusion of coherency information in their
loss functions, i.e., sh, compared to their coherent counterpart, shc, the
coherency MS3Es are fairly similar, and theses differences can here be
neglected.

The most incoherent forecasts are here produced by models with k-level
partitions and td, butd, or bu linkages, ranging between 1e19 and 1.3e7 kWh
RMS3Es respectively. Models including coherency information in their learning
process here also display poorer coherency performances but are, however,
associated with extremely poor accuracy performances.

A surprising observation showcases the fully connected model, full, and cut-
tree-butd with coherency losses as some of the better coherency performers, in
spite of their poor accuracies and large number of weights to learn.
Generally, however, coherency performances display similar tendencies as their
associated accuracy ones.

###  4.3 Coherency information value

(a)

(b)

Figure 7: Value ratio brought by the coherency loss function for (a) the
accuracy, and (b) coherency performances of the forecast.

Finally, to formally investigate the value brought by coherency information in
the learning process of structural-hierarchical models, we evaluate the
relative performance ratio between sh and shc loss functions of similar
models. The improvement ratios for accuracy, acc, and coherency, coh, are
defined as

$\displaystyle
r_{acc}=\frac{\mathcal{L}^{sh}-\mathcal{L}^{sh}_{coh}}{\mathcal{L}^{sh}}\text{
,}$ (13) $\displaystyle
r_{coh}=\frac{\mathcal{L}^{sc}-\mathcal{L}^{sc}_{coh}}{\mathcal{L}^{sc}}\text{
,}$ (14)

where $r$ is the improvement ratio defined by the difference between
structural-hierarchical losses $\mathcal{L}^{sh}$ or structural coherent ones
$\mathcal{L}^{sc}$ and their respective counterparts with the inclusion of
coherency in the learning mechanism of the model, i.e., Eq. (11). This
difference is then normalized by the reference loss, which does not consider
coherency information in its loss function, i.e., Eq. (8). As such, positive
improvement ratios relate to a performance improvement brought by coherency
knowledge, whereas negative ratios point to performance regressions. This
echoes the relative root mean square error (RRMSE) [23] evaluation metric
typically employed to estimate the value brought by a reconciliation approach
to a base forecast. The main difference in this setting is that instead of a
common base forecast, we consider the structural-hierarchical forecast
performance from each individual model architecture. This allows a relative
performance evaluation per model architecture of the inclusion of coherency
information in the learning process of the regressors.

Figure 7 presents the improvement ratios categorized by their network design
characteristics, i.e., per partition and topological bridge arrangement. Both
accuracy and coherence improvements brought by the coherency loss only display
four cases of performance regression, three of which are similar: k-level-bu,
k-level-td, and tree-butd.

Models k-level bu and td are extreme poor performers both in accuracy and
coherency and can thus be disregarded in the remainder of the examination,
together with k-level butd.

The tree-butd and tree-bu designs are the two best coherency performers, with
equivalent 5.2e2 to 5.7e2 kWh RMS3Es. The accuracy of the tree-butd model,
however, is more modest, with 1.4e4 and 1.9e4 kWh RMS3E for sh and shc
corresponding losses. Due to the equivalent, top-performing coherencies of
these models, their obtained negative coherency improvement ratios can thus be
considered null and consequently disregarded. The remaining network
characteristics all exhibit improved coherency forecasts thanks to the
inclusion of the coherency loss in their learning procedure. This significant
finding places structural hierarchical coherent learning as a valuable method,
bringing forecasts one step closer to coherency, prior to reconciliation.

Regarding the accuracy improvements brought by the coherency loss, the tree-td
design demonstrates an interesting behavior where coherency is improved but
accuracy deteriorates. By looking further into the accuracy performance of
this method in Fig. 5, it was noted that the model produced overall good
accuracies across its nodes with the exception of a few extreme cases, which
significantly impact the overall performance of the forecast. As such, the shc
loss function consequently pushes the forecast to a more coherent outcome than
its sh equivalent, at the cost of a poorer accuracy across the hierarchy. A
similar, but less pronounced, outcome can be observed for the tree-butd
network, which maintains a similar coherency score but tapers its accuracy by
adjusting fewer excessive forecasts. The coherency value investigation
consequently allows us to claim that coherency knowledge improves the accuracy
of produced hierarchical forecasts provided individual forecasts are generated
within reasonable accuracy limits.

##  5 Conclusion

To secure coherent forecasts across hierarchical structures, recent research
proposed hierarchical learning as a pioneer solution bridging the two formerly
distinct phases that are forecasting and reconciliation. While the work
depicted promising potential, results displayed disparate performances where
coherency information was only found to improve forecasting performances in
one setting. Additionally, the method suffered from two prevailing challenges,
namely, an arduous learning process and faulty coherent learning as a result
of input hierarchical time-series normalization.

This work addresses both complications by investigating custom neural network
designs echoing the structural topologies of hierarchies. The approach notably
exploits layer partitions producing distinct model components tailored to
node-specific elements, while sharing specific information across the model
from varying topological bridges resulting in 13 different model
architectures. Batch normalization is notably included between layers of the
model, providing structural-scale robustness to the learning process, while
exempting input hierarchical time series from prior normalization. We
investigate all designs under two novel structurally-scaled learning
functions, i.e., structural-hierarchical loss and structural hierarchical-
coherent loss, leveraging the mean structurally-scaled square error (MS3E)
[21], and subsequently entitle our approach structural hierarchical learning.

The varying neural network designs are evaluated over the accuracy and
coherency performances of their produced forecasts from real-world
measurements extracted from the Building Data Genome project 2 (BDG2) [25].
Models with tree partitionings notably performed best, particularly coupled to
bottom-up and disconnected topological bridges, for both structural-
hierarchical and structural hierarchical-coherent losses. Links between the
performance of a model and its network topology specifically revealed that (i)
structural models with fewer connections performed overall better than models
with larger numbers of connections, and (ii) that the inclusion of coherency
information in the loss function improved both the accuracy and coherency
performances of forecasts, provided individual forecasts were generated within
reasonable accuracy limits.

This work consequently confirms the value potential brought by coherency
information in structural hierarchical regressors and places structural
hierarchical learning as a successful hierarchical-forecasting method,
bringing forecasts one step closer to coherency, prior to reconciliation. By
putting forward tailored, ingenious architectures of neural networks we
effectively reduced hierarchical model complexities while serving advanced and
coherency-aware hierarchical forecasts. The approach could notably support
domains such as retail, stock management, and distribution networks, thanks to
improved and more consistent predictions across all levels of considered
hierarchies.

Finally, to encourage knowledge dissemination we render our work fully
replicable by open-sourcing all developed python implementations under the
public GitHub repository:
https://github.com/JulienLeprince/structuralhierarchicallearning.

##  6 CRediT authorship contribution statement

Julien Leprince: Conceptualization, Methodology, Software, Data curation,
Formal analysis, Visualization, Writing - original draft, review and editing.
Waqas Khan: Conceptualization, Methodology, Writing - review and editing.
Henrik Madsen: Methodology, Supervision, Validation, Writing - review and
editing. Jan Kloppenborg Møller: Methodology, Supervision, Validation, Writing
- review and editing. Wim Zeiler: Supervision, Funding acquisition.

All authors have read and agreed to the published version of the manuscript.

##  7 Acknowledgments

This work is funded by the Dutch Research Council (NWO), in the context of the
call for Energy System Integration & Big Data (ESI-bida). Additionally,
funding related to SEM4Cities (Innovation Fund Denmark, No. 0143-0004) is
gratefully acknowledged.

## References

  * [1] P. Nystrup, E. Lindstrøm, P. Pinson, H. Madsen, Temporal hierarchies with autocorrelation for load forecasting, European Journal of Operational Research 280 (3) (2020) 876–888.
  * [2] M. Kremer, E. Siemsen, D. J. Thomas, The sum and its parts: Judgmental hierarchical forecasting, Management Science 62 (9) (2016) 2745–2764.
  * [3] E. Spiliotis, M. Abolghasemi, R. J. Hyndman, F. Petropoulos, V. Assimakopoulos, Hierarchical forecast reconciliation with machine learning, Applied Soft Computing 112 (2021) 107756.
  * [4] N. Kourentzes, G. Athanasopoulos, Cross-temporal coherent forecasts for australian tourism, Annals of Tourism Research 75 (2019) 393–409.
  * [5] G. Athanasopoulos, R. A. Ahmed, R. J. Hyndman, Hierarchical forecasts for australian domestic tourism, International Journal of Forecasting 25 (1) (2009) 146–166.
  * [6] S. B. Taieb, J. W. Taylor, R. J. Hyndman, Hierarchical probabilistic forecasting of electricity demand with smart meter data, Journal of the American Statistical Association 116 (533) (2021) 27–43.
  * [7] H. G. Bergsteinsson, J. K. Møller, P. Nystrup, Ó. P. Pálsson, D. Guericke, H. Madsen, Heat load forecasting using adaptive temporal hierarchies, Applied Energy 292 (2021) 116872.
  * [8] W. Gu, Z. Wu, R. Bo, W. Liu, G. Zhou, W. Chen, Z. Wu, Modeling, planning and optimal energy management of combined cooling, heating and power microgrid: A review, International Journal of Electrical Power & Energy Systems 54 (2014) 26–37.
  * [9] A. Panday, H. O. Bansal, A review of optimal energy management strategies for hybrid electric vehicle, International Journal of Vehicular Technology 2014 (2014).
  * [10] T. Ahmad, H. Zhang, B. Yan, A review on renewable energy and electricity requirement forecasting models for smart grid and buildings, Sustainable Cities and Society 55 (2020) 102052.
  * [11] J. Peng, A. Kimmig, Z. Niu, J. Wang, X. Liu, J. Ovtcharova, A flexible potential-flow model based high resolution spatiotemporal energy demand forecasting framework, Applied Energy 299 (2021) 117321.
  * [12] E. Spiliotis, F. Petropoulos, N. Kourentzes, V. Assimakopoulos, Cross-temporal aggregation: Improving the forecast accuracy of hierarchical electricity consumption, Applied Energy 261 (2020) 114339.
  * [13] R. J. Hyndman, R. A. Ahmed, G. Athanasopoulos, H. L. Shang, Optimal combination forecasts for hierarchical time series, Computational statistics & data analysis 55 (9) (2011) 2579–2589.
  * [14] M. Weale, The reconciliation of values, volumes and prices in the national accounts, Journal of the Royal Statistical Society: Series A (Statistics in Society) 151 (1) (1988) 211–221.
  * [15] J. B. Edwards, G. H. Orcutt, Should aggregation prior to estimation be the rule?, The Review of Economics and Statistics (1969) 409–420.
  * [16] Y. Grunfeld, Z. Griliches, Is aggregation necessarily bad?, The Review of Economics and Statistics (1960) 1–13.
  * [17] N. Kourentzes, D. Barrow, F. Petropoulos, Another look at forecast selection and combination: Evidence from forecast pooling, International Journal of Production Economics 209 (2019) 226–235.
  * [18] J. Bien, F. Bunea, L. Xiao, Convex banding of the covariance matrix, Journal of the American Statistical Association 111 (514) (2016) 834–845, pMID: 28042189.  arXiv:https://doi.org/10.1080/01621459.2015.1058265, doi:10.1080/01621459.2015.1058265.
URL https://doi.org/10.1080/01621459.2015.1058265

  * [19] R. J. Hyndman, A. J. Lee, E. Wang, Fast computation of reconciled forecasts for hierarchical and grouped time series, Computational Statistics & Data Analysis 97 (2016) 16–32.  doi:https://doi.org/10.1016/j.csda.2015.11.007.
URL https://www.sciencedirect.com/science/article/pii/S016794731500290X

  * [20] S. L. Wickramasuriya, G. Athanasopoulos, R. J. Hyndman, Optimal forecast reconciliation for hierarchical and grouped time series through trace minimization, Journal of the American Statistical Association 114 (526) (2019) 804–819.
  * [21] J. Leprince, H. Madsen, J. K. Møller, W. Zeiler, Hierarchical learning, forecasting coherent spatio-temporal individual and aggregated building loads (2023).  doi:10.48550/ARXIV.2301.12967.
URL https://arxiv.org/abs/2301.12967

  * [22] T. van Erven, J. Cugliari, Game-theoretically optimal reconciliation of contemporaneous hierarchical time series forecasts, in: A. Antoniadis, J.-M. Poggi, X. Brossat (Eds.), Modeling and Stochastic Learning for Forecasting in High Dimensions, Springer International Publishing, Cham, 2015, pp. 297–317.
  * [23] G. Athanasopoulos, R. J. Hyndman, N. Kourentzes, F. Petropoulos, Forecasting with temporal hierarchies, European Journal of Operational Research 262 (1) (2017) 60–74.  doi:https://doi.org/10.1016/j.ejor.2017.02.046.
URL https://www.sciencedirect.com/science/article/pii/S0377221717301911

  * [24] P. Nystrup, E. Lindström, J. K. Møller, H. Madsen, Dimensionality reduction in forecasting with temporal hierarchies, International Journal of Forecasting 37 (3) (2021) 1127–1146.
  * [25] C. Miller, A. Kathirgamanathan, B. Picchetti, P. Arjunan, J. Y. Park, Z. Nagy, P. Raftery, B. W. Hobson, Z. Shi, F. Meggers, the building data genome project 2, energy meter data from the ashrae great energy predictor iii competition, Scientific data 7 (1) (2020) 1–13.
  * [26] D. Müllner, Modern hierarchical, agglomerative clustering algorithms, arXiv preprint arXiv:1109.2378 (2011).
  * [27] D. N. Reshef, Y. A. Reshef, H. K. Finucane, S. R. Grossman, G. McVean, P. J. Turnbaugh, E. S. Lander, M. Mitzenmacher, P. C. Sabeti, Detecting novel associations in large data sets, science 334 (6062) (2011) 1518–1524.
  * [28] C. Miller, What’s in the box?! towards explainable machine learning applied to non-residential building smart meter classification, Energy and Buildings 199 (2019) 523–536.
  * [29] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.
  * [30] N. Bjorck, C. P. Gomes, B. Selman, K. Q. Weinberger, Understanding batch normalization, Advances in neural information processing systems 31 (2018).
  * [31] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research 15 (1) (2014) 1929–1958.
  * [32] C. Garbin, X. Zhu, O. Marques, Dropout vs. batch normalization: an empirical study of their impact to deep learning, Multimedia Tools and Applications 79 (19) (2020) 12777–12815.
  * [33] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow: Large-scale machine learning on heterogeneous systems, software available from tensorflow.org (2015).
URL https://www.tensorflow.org/