11institutetext:  Institute of Research and Innovation in Bioengineering,
Universitat Politècnica de València, Valencia, Spain (11email:
<EMAIL_ADDRESS>
22institutetext: Department of Pathology, Stavanger University Hospital,
Stavanger, Norway
33institutetext: Department of Chemistry, Bioscience and Environmental
Engineering, University of Stavanger, Stavanger, Norway 44institutetext:
Institute of Transport and Territory, Universitat Politècnica de València,
Valencia, Spain

# Challenging mitosis detection algorithms:
Global labels allow centroid localization††thanks: The work of C. Fernández-
Martín and U. Kiraz was funded from the Horizon 2020 of European Union
research and innovation programme under the Marie Sklodowska Curie grant
agreement No 860627 (CLARIFY Project). The work of Sandra Morales has been co-
funded by the Universitat Politècnica de València through the program
PAID-10-20. This work was partially funded by GVA through project
PROMETEO/2019/109.

Claudio Fernandez-Martín 11 Umay Kiraz 22 3 3 Julio Silva-Rodríguez 44 Sandra
Morales 11 Emiel Janssen 22 3 3 Valery Naranjo 11

###### Abstract

Mitotic activity is a crucial proliferation biomarker for the diagnosis and
prognosis of different types of cancers. Nevertheless, mitosis counting is a
cumbersome process for pathologists, prone to low reproducibility, due to the
large size of augmented biopsy slides, the low density of mitotic cells, and
pattern heterogeneity. To improve reproducibility, deep learning methods have
been proposed in the last years using convolutional neural networks. However,
these methods have been hindered by the process of data labelling, which
usually solely consist of the mitosis centroids. Therefore, current literature
proposes complex algorithms with multiple stages to refine the labels at pixel
level, and to reduce the number of false positives. In this work, we propose
to avoid complex scenarios, and we perform the localization task in a weakly
supervised manner, using only image-level labels on patches. The results
obtained on the publicly available TUPAC16 dataset are competitive with state-
of-the-art methods, using only one training phase. Our method achieves an
F1-score of $0.729$ and challenges the efficiency of previous methods, which
required multiple stages and strong mitosis location information.

###### Keywords:

Mitosis detection Weak labels Histology Digital pathology.

##  1 Introduction

In digital pathology, mitosis counting is one of the most important tasks in
the histopathological clinical practice. In the case of breast cancer, the
mitotic activity index (MAI) is considered one of the strongest proliferation-
associated prognostic factors [1]. However, mitosis counting is a laborious
and time-consuming task due to the large size of the Hematoxylin and Eosin
(H&E) slides under a microscope, and the low occurrence of mitotic figures. In
addition, the large heterogeneity of patterns and similarity between mitotic
and non-mitotic cells (see Figure 1) makes this task highly variable among
clinical experts [2], which hinders its reproducibility.

(a)  Mitotic cells

(b)  Non-mitotic cells

Figure 1: Visual illustration of the morphological heterogeneity and the
challenge of differentiating patterns between mitotic and non-mitotic cells,
extracted from TUPAC16 [3].

In the last years, the advent of modern deep learning algorithms has emerged
as a possible solution to bring objectivity and reproducibility to the
challenge of mitosis localization. Deep learning using convolutional neural
networks (CNNs) has reached remarkable results in a wide range of applications
under the supervised learning paradigm. Nevertheless, it requires a reasonable
amount of carefully-labeled data to perform properly. In the case of mitosis
localization, this is a tedious process, which usually is repeated by
different pathologists to reach consensus labels. Since delineating at pixel-
level individual mitotic cells is an unfeasible task, the reference datasets
normally contain centroid-based labels [3] or inexact pixel-level annotations
[4]. Because of this, previous works to automate the mitosis localization
process have struggled to match the available labels to the use of
segmentation or object detection CNNs, which are typically used localization
tasks. Contrary to this line of work, we propose to make use of inherent
spatial localization capacity of CNNs in image-level classification tasks [5],
without the need to resort to an exact localization of the mitotic cell inside
the region of interest. Our main contributions are summarized as follows:

  * $\mathbin{\vbox{\hbox{\scalebox{0.5}{$\bullet$}}}}$

A CNN for weakly supervised segmentation of mitotic figures on H&E patches
using image-level labels.

  * $\mathbin{\vbox{\hbox{\scalebox{0.5}{$\bullet$}}}}$

Concretely, training is driven by maximum aggregation of instance-level
predictions.

  * $\mathbin{\vbox{\hbox{\scalebox{0.5}{$\bullet$}}}}$

Comprehensive experiments demonstrate the competitive performance on the
popular TUPAC16 dataset, using a single-phase pipeline without requiring the
exact localization information for training our model.

##  2 Related Work

###  2.1 Mitosis detection

Mitosis localization algorithms using CNNs deal with labels in the form of
centroid annotations, or inexact pixel-level delineation of the mitotic cell.
In that sense, Li et al. [6] propose a novel concentric loss to move from
centroid labels to pixel-level segmentation using the pixels surrounding
certain radius of the centroid. Other works focus on leveraging cell-level
predictions using multi-phase pipelines [7, 8, 9, 10, 11, 12]. For instance,
Sohail et al. [12] propose a complex multi-phase pipeline that includes
pseudolabeling centroid-labelled mitosis via previously trained Mask R-CNNs.
Also, Nateghi et al. use multiple training stages to refine the false positive
detection using hard-negative mining via stain priors, or prediction
uncertainty. In contrast to these works, we study how training a CNN at the
image-level for a classification task also allows the precise location of
mitotic cells without using any localization information, shape or stain
priors, or multi-phase refinement pipelines.

###  2.2 Weakly supervised segmentation

Weakly supervised segmentation (WSS) aims to leverage pixel-level localization
using global (a.k.a image-level) labels during training. According to [13],
WSS methods use fully-convolutional CNNs with an aggregation function that
merges all the spatial information into one value, that serves as global
prediction [14]. This output is then used to compute the loss function, and
drives the network optimization. Different strategies include the aggregation
of spatial features (embedding-based) or pixel-level predictions (instance-
based). Finally, the probability maps before aggregation operation are used as
segmentation predictions. Lately, these segmentation maps are refined to
incorporate self-supervised learning pipelines [15] or uncertainty proxies
[16], among others.

##  3 Methods

An overview of our proposed method is depicted in Figure 2. In the following,
we describe the problem formulation, and each of the proposed components.

Figure 2: Overview of the proposed method for mitosis localization.

#### Problem Formulation

In the paradigm of weakly supervised segmentation (WSS), the training set is
composed of images $\\{x_{n}\\}_{n=1}^{N}$, whose binary label
$\\{Y_{n}^{k}\\}_{n=1}^{N}$, such that, $Y_{n}^{k}=\\{0,1\\}$ is known, and
defines if a category $k$ is present within the image. Also, each positive
image has pixel-level labels $y_{n,i}$ for each $i$ pixel in the image, but
they remain unknown during training. Further, we denote $Y_{n}^{k}$ as $Y_{n}$
for simplicity, since one unique class is taken into account, and we assume
image index $n$.

#### Instance-based WSS

In this work, we aim to train a CNN capable of locating positive mitosis
during inference, while being trained only with image-level labels. To do so,
we make use of an instance-based weakly supervised learning strategy. Let us
denote a CNN model,
$f_{\boldsymbol{\theta}}(\cdot):\mathcal{X}\rightarrow\mathcal{H}^{K}$,
parameterized by $\boldsymbol{\theta}$, which processes instances
$x\in\mathcal{X}$ to output sigmoid-activated instance-level probabilities,
$h_{i}$, such that $h_{i}\in[0,1]$. Also, we use a parameter free aggregation
function, $f_{a}(\cdot)$, in charge of combining the pixel-level scores into
one global output $H$, such that $H=f_{a}(f_{\boldsymbol{\theta}}({x}))$.
Then, the optimization of $\boldsymbol{\theta}$ is driven by the minimization
of cross entropy loss between reference and predicted image-level score.

$\mathcal{L}_{ce}=Ylog(H)+(1-Y)log(1-H)$ (1)

In this work, we propose to use the maximum operation as aggregation function,
$f_{a}(\cdot)$. Although this aggregation only backpropagates gradients
through the maximum-activated spatial regions, this effect produces that only
very discriminative cells will be classified as mitosis, which avoids false
positive predictions.

#### Inference

During inference, pixel-level predictions are inferred using the pixel-level
predictions given by the trained CNN, $y_{i}=f_{\boldsymbol{\theta}}(x)$. The
probability maps are resized to the original image dimensions by bi-linear
interpolation. Then, sigmoid scores are converted to a binary mask by applying
a threshold to the probability maps. Concretely, the threshold is obtained
from the operative point of the ROC curve between image-level predictions and
references. Finally, a centroid is assigned to each element in the mask, to be
located as a mitosis.

##  4 Experimental setting

###  4.1 Datasets

The experiments described in this work are carried out using the popular 2016
TUmor Proliferation Assessment Challenge (TUPAC16) dataset [3]. TUPAC16 is
publically available and is composed of $73$ breast cancer whole slide images
from two different institutions. In particular, the auxiliary mitosis dataset
contains $1552$ processed regions of interest at $40\times$ magnification,
with centroid-labelled mitosis by consensus of expert pathologists. Following
relevant literature in [6], we extracted patches of size $500$ pixels from the
regions of interest for computational efficiency. The dataset is divided into
patient-level training, validation, and testing cohorts in a similar fashion
to prior literature [6].

###  4.2 Metrics

We use standard metrics for mitosis localization evaluation. First, the model
is optimized using only global image-level labels, by means of the accuracy,
AUC, and F1-score. Then, the comparison with state-of-the-art methods on
mitosis detection is assessed using the standard criteria of mitosis detection
contests [12]. A detected mitosis is considered true if it is located at most
30 pixels from an annotated mitosis. Under this criteria, precision, recall
and F1-score are computed.

###  4.3 Implementation details

The proposed method is trained using ResNet-18 [17] convolutional blocks as a
backbone. Concretely, the first $3$ blocks pre-trained on ImageNet are used as
feature extractor, which are retrained for the mitosis detection task. We
trained this architecture during $40$ epochs to optimize Eq. 1 using a batch
size of $32$ images and a learning rate of $0.0001$. In order to deal with
class imbalance, the images are sampled homogeneously according to its class
in each epoch. Also, color normalization and augmentation techniques are
employed to increase robustness against stain variations and artifacts in the
digitized slides. Images are color-normalized using the stain normalization
method of Macenko et al. [18], and data augmentation is included during
training using spatial translations, rotations, and blurring.

##  5 Results

###  5.1 Comparison to literature

The quantitative results obtained by the proposed method for mitosis
localization on the test cohort are presented in Table 1. Also, we include
results reported in previous literature on the TUPAC16 dataset. The proposed
weakly-supervised method reaches an F1-score value of 0.729, which is
comparable to prior literature without accessing to any supervision regarding
the exact location of the mitosis in the image. It should be noted that, in
addition, the best previous methods use additional training data, and require
multiple stages of label refinement. In contrast, the proposed method uses
only one training cycle. Moreover, the proposed approach obtains the best
precision on mitosis localization that only use one training phase. This could
be due to maximum aggregation, which propagates gradients only in those
regions that are highly discriminating.

Method | Precision | Recall | F-score |  | Multiple
---
phases
| Location
---
supervision
| External
---
data
Paeng et al. (2017) [7] | - | - | 0.652 | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$ |
Zerhouni et al. (2017) [8] | 0.675 | 0.623 | 0.648 | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$ |
Akram et al. (2018) [9] | 0.613 | 0.671 | 0.640 | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$
Li et al. (2019) [6] | 0.64 | 0.70 | 0.669 |  | $\color[rgb]{0,0,0}\times$ |
Wahab et al. (2019) [10] | 0.770 | 0.660 | 0.713 | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$ |
Mahmood et al. (2020) [19] | 0.641 | 0.642 | 0.642 | $\color[rgb]{0,0,0}\times$ |  |
Nateghi et al. (2021) [11] | 0.764 | 0.714 | 0.738 | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$ |
Sohail et al. (2021) [12] | 0.710 | 0.760 | 0.750 | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$ | $\color[rgb]{0,0,0}\times$
Proposed | 0.739 | 0.720 | 0.729 |  |  |

Table 1: Performance comparison of the proposed model with existing methods on
test subset of TUPAC16 auxiliary dataset.

###  5.2 Ablation experiments

In the following, we depict ablation experiments to motivate the choice of the
different components of the proposed method.

####  Weakly Supervised Setting.

First, we study the configuration of the WSS model architecture. To do so, we
explore the most outstanding configurations. First, embedding-based approaches
that aggregate spatial features before the classification layers, and
instance-based approaches that apply the classification layer spatially. Also,
we use different aggregation methods, such as mean and max operations, and the
trainable attentionMIL mechanism [13]. Results are presented in Table 2. The
figures of merit show that, although all methods reach similar results at
image-level, only the instance-based with maximum aggregation performs
properly on mitosis localization, since it is the only method that penalizes
false positive localization during training.

Configuration |  | F-score
---
image-level
| F-score
---
localization
embedding - mean | 0.762 | 0.134
embedding - max | 0.772 | 0.234
attentionMIL [13] | 0.768 | 0.014
instance - mean | 0.753 | 0.004
instance - max | 0.761 | 0.729
Table 2: Performance comparison of the different configurations of the WSS
proposed model, in terms of aggregation strategies. Results are presented for
mitosis localization and image-level classification.

####  On the importance of the feature complexity.

Convolutional neural networks combine stacked convolutional and pooling
operations, which merge spatial information. Thus, later layers in CNNs
extract high-level features with complex shapes, and low spatial resolution.
Although CNNs for classification tasks usually benefit from deep structures,
we observed that spatial resolution and low-level features are vital for
mitosis localization, as shown in Figure 3. For that reason, we used only 3
residual blocks of ResNet-18 architecture for the proposed method.

Figure 3: Ablation study on the number of residual blocks used for feature
extraction. Metric presented for mitosis localization.

###  5.3 Qualitative evaluation

Finally, we present visual results of the proposed method performance on the
test subset in Figure 4. In particular, correct detections of mitotic cells
(true positives), cells wrongly classified as mitosis (false positives) and
non-detected mitosis (false negatives) are shown in green, yellow and blue
colors, respectively. Visual results show a promising performance of the
proposed method, with false positive classifications occur with irregularly-
shaped non-mitotic cells.

Figure 4: Qualitative evaluation of the proposed method for mitosis
localization. Green: true positive; Blue: false negative; Yellow: false
positive.

##  6 Conclusions

In this work, we have presented a deep learning model for weakly supervised
mitosis location on H&E histology images. In particular, the model is composed
of a narrow CNN backbone that leverages pixel-level predictions. Then, those
predictions are grouped into an image-level score using maximum aggregation,
that serves as proxy for CNN training via global labels. Thanks to the maximum
operation, that only focus on very discriminative cells, obtained results have
very few false positive predictions, and reaching a precision of $0.739$ and
an F-score of $0.729$ on TUPAC16 dataset. The proposed approach, yet simple,
reaches competitive performance in comparison to previous literature, without
requiring any information of mitosis localization in the image during
training. This calls into question the efficiency of other approaches, which
require this location information, and resort to multiple phases of training
to refine centroid-based labels and to alleviate false positive predictions.
Further research could complement the proposed setting to take into account
uncertainties on predicted mitoses, and to incorporate location information
using a soft, constrained formulation.

## References

  * [1] Jan P.A. Baak, Paul J. van Diest, Feja J. Voorhorst, Elsken van der Wall, Louk V.A.M. Beex, Jan B. Vermorken, and Emiel A.M. Janssen,  “Prospective multicenter validation of the independent prognostic value of the mitotic activity index in lymph node–negative breast cancer patients younger than 55 years,”  Journal of Clinical Oncology, vol. 23, no. 25, 2005.
  * [2] Joann G. Elmore, Gary M. Longton, Patricia A. Carney, Berta M. Geller, Tracy Onega, Anna N. A. Tosteson, Heidi D. Nelson, Margaret S. Pepe, Kimberly H. Allison, Stuart J. Schnitt, Frances P. O’Malley, and Donald L. Weaver,  “Diagnostic Concordance Among Pathologists Interpreting Breast Biopsy Specimens,”  JAMA, vol. 313, no. 11, pp. 1122–1132, 03 2015.
  * [3] Mitko Veta, Yujing J. Heng, Nikolas Stathonikos, Babak Ehteshami Bejnordi, Francisco Beca, Thomas Wollmann, Karl Rohr, Manan A. Shah, Dayong Wang, and Mikael et al. Rousson,  “Predicting breast tumor proliferation from whole-slide images: The tupac16 challenge,”  Medical Image Analysis, vol. 54, pp. 111–121, 2019.
  * [4] Ludovic Roux, Daniel Racoceanu, Nicolas Loménie, Maria Kulikova, Humayun Irshad, Jacques Klossa, Frédérique Capron, Catherine Genestie, Gilles Le Naour, and Metin N Gurcan,  “Mitosis detection in breast cancer histological images an icpr 2012 contest,”  Journal of Pathology Informatics, vol. 4, no. 1, 2013.
  * [5] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic,  “Learning and transferring mid-level image representations using convolutional neural networks,”  in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [6] Chao Li, Xinggang Wang, Wenyu Liu, Longin Jan Latecki, Bo Wang, and Junzhou Huang,  “Weakly supervised mitosis detection in breast histopathology images using concentric loss,”  Medical Image Analysis, vol. 53, pp. 165–178, 2019.
  * [7] Kyunghyun Paeng, Sangheum Hwang, Sunggyun Park, and Minsoo Kim,  “A unified framework for tumor proliferation score prediction in breast histopathology,” 2016.
  * [8] Erwan Zerhouni, David Lanyi, Matheus Viana, and Maria Gabrani,  “Wide residual networks for mitosis detection,”  2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), 2017.
  * [9] Saad Ullah Akram, Talha Qaiser, Simon Graham, Juho Kannala, Janne Heikkilä, and Nasir Rajpoot,  “Leveraging unlabeled whole-slide-images for mitosis detection,” 2018\.
  * [10] Noorul Wahab, Asifullah Khan, and Yeon Soo Lee,  “Transfer learning based deep cnn for segmentation and detection of mitoses in breast cancer histopathological images,”  Microscopy, vol. 68, no. 3, pp. 216–233, 2019.
  * [11] Ramin Nateghi, Habibollah Danyali, and Mohammad Sadegh Helfroush,  “A deep learning approach for mitosis detection: Application in tumor proliferation prediction from whole slide images,”  Artificial Intelligence in Medicine, vol. 114, pp. 102048, 2021\.
  * [12] Anabia Sohail, Asifullah Khan, Noorul Wahab, Aneela Zameer, and Saranjam Khan,  “A multi-phase deep cnn based mitosis detection framework for breast cancer histopathological images,”  Scientific Reports, vol. 11, no. 1, 2021.
  * [13] Maximilian Ilse, Jakub M. Tomczak, and Max Welling,  “Attention-based deep multiple instance learning,”  in 35th International Conference on Machine Learning (ICML), 2018\.
  * [14] Julio Silva-Rodríguez, Adrián Colomer, and Valery Naranjo,  “Weglenet: A weakly-supervised convolutional neural network for the semantic segmentation of gleason grades in prostate histology images,”  Computerized Medical Imaging and Graphics, vol. 88, pp. 101846, 2021\.
  * [15] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen,  “Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation,”  in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [16] Soufiane Belharbi, Jérôme Rony, Jose Dolz, Ismail Ben Ayed, Luke Mccaffrey, and Eric Granger,  “Deep interpretable classification and weakly-supervised segmentation of histology images via max-min uncertainty,”  IEEE Transactions on Medical Imaging, vol. 41, no. 3, pp. 702–714, 2022.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [18] Marc Macenko, Marc Niethammer, J. S. Marron, David Borland, John T. Woosley, Xiaojun Guan, Charles Schmitt, and Nancy E. Thomas,  “A method for normalizing histology slides for quantitative analysis,”  in 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2009, pp. 1107–1110.
  * [19] Tahir Mahmood, Muhammad Arsalan, Muhammad Owais, Min Beom Lee, and Kang Ryoung Park,  “Artificial intelligence-based mitosis detection in breast cancer histopathology images using faster r-cnn and deep cnns,”  Journal of Clinical Medicine, vol. 9, no. 3, pp. 749, 2020.


mode = titleDR for residential building heating: Effective MCTS control based
on PiNN

[orcid=0009-0005-7904-099X] Conceptualization, Methodology, Software, Formal
analysis, Investigation, Visualization, Writing - Original Draft

[orcid=0000-0002-1451-397X] Software, Conceptualization, Writing - Review &
Editing

[orcid=0009-0006-6116-1483] Supervision, Conceptualization, Writing - Review &
Editing

[orcid=0000-0003-2707-4176] Supervision, Funding acquisition, Writing - Review
& Editing

1]organization=IDLab Ghent university – imec, addressline=Technologiepark
Zwijnaarde 126, postcode=9052, postcodesep=, city=Gent, country=Belgium
2]organization=Beepop, country=Belgium

# Demand response for residential building heating: Effective Monte Carlo Tree
Search control based on physics-informed neural networks

Fabio Pavirani<EMAIL_ADDRESS>Gargya Gokhale  Bert Claessens  Chris
Develder  [  [

(June 2023)

###### Abstract

Controlling energy consumption in buildings through demand response (DR) has
become increasingly important to reduce global carbon emissions and limit
climate change. In this paper, we specifically focus on controlling the
heating system of a residential building to optimize its energy consumption
while respecting user’s thermal comfort. Recent works in this area have mainly
focused on either model-based control, e.g., model predictive control (MPC),
or model-free reinforcement learning (RL) to implement practical DR
algorithms. A specific RL method that recently has achieved impressive success
in domains such as board games (go, chess) is Monte Carlo Tree Search (MCTS).
Yet, for building control it has remained largely unexplored. Thus, we study
MCTS specifically for building demand response. Its natural structure allows a
flexible optimization that implicitly integrate exogenous constraints (as
opposed, for example, to conventional RL solutions), making MCTS a promising
candidate for DR control problems. We demonstrate how to improve MCTS control
performance by incorporating a Physics-informed Neural Network (PiNN) model
for its underlying thermal state prediction, as opposed to traditional purely
data-driven Black-Box approaches. Our MCTS implementation aligned with a PiNN
model is able to obtain a 3% increment of the obtained reward compared to a
rule-based controller; leading to a 10% cost reduction and 35% reduction on
temperature difference with the desired one when applied to an artificial
price profile. We further implemented a Deep Learning layer into the Monte
Carlo Tree Search technique using a neural network that leads the tree search
through more optimal nodes. We then compared this addition with its Vanilla
version, showing the improvement in computational cost required.

###### keywords:

Building Control Demand Response Monte Carlo Tree Search Physics informed
Neural Network Machine Learning Thermal dynamics modeling

##  1 Introduction

Even though recently the energy sector already underwent a rather drastic
transformation (e.g., via the increased penetration of renewable energy
sources, RES), given the urgent need to combat climate changes, it is pressed
even further to optimize energy consumption and reduce carbon emission. The
rising contribution of RES leads to higher uncertainty, calling for Demand
Response (DR) control to stabilize the grid. On top of that, a substantial
part of energy consumption stems from the residential sector: in the EU, it
represented 26.1% of the total energy in 2018 [1]. Of that residential energy,
63.6% is used for heating [1]. This high portion of energy consumed in the
residential heating domain, combined with the uncertainty given by RES,
implies a great potential for flexible and effective controllers that can
manage varying constraints.

To realize sequential residential heating control, two main strategies have
been followed [2]: either  (i) model-based controllers, or  (ii) purely data-
driven controllers.  For i, typically Model Predictive Control (MPC) is
adopted, where a model of the building’s thermal dynamics is integrated into
an optimization problem. This optimization problem is solved to obtain or
approximate an optimal control decision. Model-based techniques, although very
efficient, are highly dependent on the model used and can be difficult to
scale over different houses [3]. On the other hand, regarding purely data-
driven controllers, one of the most used techniques is Reinforcement Learning
RL, which is more flexible and scalable towards different households. However,
it requires a large amount of data obtained by interacting with the building
[4], which is usually hard to obtain. A solution to this problem is to develop
a simulator for the RL agent to interact with, adding once again more
importance to the modeling task. Despite their promising results in producing
effective control policies, several challenges regarding RL algorithms in the
Building Energy Management (BEM) domain are yet to be solved, such as their
lack of interpretability and safety [4], as well as their flexibility when
dealing with exogenous constraints, e.g., user temperature bounds or grid
balancing signals. To tackle these problems, inspired by recent Monte Carlo
Tree Search (MCTS) algorithms such as MuZero [5], we present an RL framework
for energy cost and thermal comfort optimization that is able to directly
manage varying exogenous constraints while also producing flexible control
actions.

To model the thermal dynamics of a building, we developed a Physics informed
encoder-based Neural Network forecaster, expanding the PhysNet presented by
Gokhale et al. [6] to a multi-step prediction model. Then, using the model as
a simulator, we developed an MCTS algorithm that optimizes energy cost while
following user-defined constraints. The main hypotheses that we will verify
through this paper are the following:

  1. H1:

Our extension to the PhysNet architecture (Sections 3.2 and 4.2) improves
forecasting performance compared to purely data-driven models when used in
multi-step forecasting problems of a residential household thermal dynamics
(Section 5.1);

  2. H2:

Our Physics-informed Neural Networks model can substantially improve the
control performance of an MCTS algorithm compared to Black-Box models (Section
5.2);

  3. H3:

Adding a Deep Learning (DL) layer to MCTS techniques to guide the tree search
(Section 4.4) can optimize the computational effort required when applied to
the BEM residential domain (Section 5.3).

##  2 Literature review

Below we summarize previous work relevant to this paper, focusing in
particular on thermal modeling and Demand Response control of buildings.

###  2.1 Thermal dynamics modeling of buildings

Creating an accurate and scalable model that describes the thermal dynamics of
a building is crucial to obtain a sequential controller. These models can be
divided into three main categories: white-box models, Black-Box models and
hybrid models [7, 8, 9, 10, 11, 12, 13, 14]. White-box models (or first
principle models) are typically physics-based methods that model the system
using Ordinary Differential Equations [15, 16]. Despite their accuracy, the
main drawback of white-box models is the complexity of the physical model
developed [8] that obstructs scalability. Conversely, Black-Box methods try to
model the thermal dynamics of a building by learning from historical data.
Yet, Black-Box models such as Artificial Neural Networks (ANN) suffer from
generalization issues [17] and hence need a large and complete dataset, which
often is hard to obtain [18].

####  2.1.1 Hybrid models

Hybrid models (or gray-box models) combine both Black-Box and white-box
models, trying to solve their respective limitations while exploiting their
good features. The idea is to use generic and usually manageable physical
equations together with data-based training to obtain a simple yet accurate
model of the building [8, 19, 20]. Because of their simplicity, linear RC
models are often used for this [21, 22].

####  2.1.2 Physics-informed neural networks

One particular subfamily of hybrid models is based on Physics-informed Neural
Networks (PiNN) [23], where an ANN that incorporates physical prior knowledge
into the prediction process is used. The idea of adapting the Neural Network
(NN) structure and its training process to a specific kind of problem is not
new in the DL literature. The most successful architectures that are used for
this include Long Short-Term Memory modules [24], Convolutional Neural
Networks [25] and Triplet Neural Network loss [26]. For modeling the thermal
dynamics of buildings, different types of PiNNs have already been applied [6,
27, 18]. Gokhale et al. [6] modified the loss function of an encoder-based NN
to guide it toward physical adherence of the latent state. Drgoňa et al. [27]
modeled the state, input, disturbance, and output matrices of a classical
linear equation using four different NNs and enforced physical constraints on
them. Di Natale et al. [18] modeled the NN architecture to enforce physical
consistency on its prediction.

This subfamily of models is particularly relevant for the problem of optimal
control action search thanks to their ability to provide accurate non-linear
system correlations related to the well-studied ANN methods, while also
providing some level of physical consistency through the model predictions.
Last, PiNN has shown promising results regarding their sample efficiency,
requiring less data for acceptable results compared to purely Black-Box models
[6]. All of these features are crucial when training an agent for building
demand response control.

###  2.2 Demand response control of buildings

In the last decades, an increasing amount of work has focused on Demand
Response [2]. With this term we refer to the problem of obtaining control
policies that react to external signals (e.g., a cost profile, the desired
temperature). These control algorithms can be roughly classified into two main
categories: Model Predictive Control (MPC) and Reinforcement Learning (RL)
approaches.

####  2.2.1 Model Predictive Control

With MPC, an optimization problem is solved (or approximated) for each control
action in a receding horizon approach. This kind of technique is model-biased
and requires an accurate model to get satisfactory results. As outlined in
Section 2.1, models can be either White-box, Black-box, or Gray-box, each with
their respective benefits and drawbacks. All three approaches have been
applied successfully in real applications [28]. A more detailed list and
review of different MPC techniques can be found in [29]. Although White-box
MPC models have been shown to be promising, their application needs a non-
negligible amount of expert knowledge about the problem configuration, making
it difficult to generalize to different situations. For this reason, data-
driven MPC algorithms have received increasing attention. Among these, Deep
Learning MPC methods are the most popular [30, 31, 32, 33]. Compared to linear
models that can be readily solved using commercial solvers [34], non-linear
models like ANN may be more complicated to use in optimization problems,
adding complexity to ANN-based MPC algorithms and limiting their application.

####  2.2.2 Reinforcement Learning

Another family of data-driven algorithms that showed promising results in the
last decade is Reinforcement Learning. RL algorithms formalize the system to
control as a Markovian Decision Process (MDP) and interact with it to learn an
adequate policy that tries to maximize the expected reward obtained from the
environment. Such RL controllers have been applied successfully to several
problems in the BEM domain [35, 36, 37, 38, 39]. However, multiple challenges
arise with RL techniques. For example, RL algorithms usually require a large
amount of data for training to reach acceptable performance. Also, most RL
algorithms used in the BEM environment are model-free [3], which often
obstructs an effective planning approach that considers dynamic constraints.

####  2.2.3 Monte Carlo Tree Search

Monte Carlo Tree Search (MCTS) can be a solution to the problems described
above. MCTS has already been applied as a DR planning method for balancing the
electrical grid [40, 41, 42, 43]. For residential DR with a heating system,
Kiljander et al. [44] used MCTS, modeling the thermal dynamic of a household
using a Feed Forward Neural Network (FFNN), and using it as a simulator with a
standard MCTS algorithm to obtain planning decisions evaluated in a real
house. We follow a similar approach by considering a PiNN architecture to
model the household and adding a DL layer to an MCTS algorithm inspired by the
recent success of Silver et al. [45, 46]. Using an MCTS-based controller
enables to implicitly manage future exogenous constraints by trimming off
actions that would lead to violating them. Also, by using hybrid models as a
simulator instead of a Black-Box one, more reliable results are obtained while
retaining the scalability of Black-Box models.

In this paper, we want to evaluate MCTS control ability in a residential
heater DR problem. MCTS’s structure enables a planning framework that can
simplify constraints management, making the algorithm a promising candidate
for DR applications. The solution methodology is described in Section 4.4.

##  3 Mathematical formulation

###  3.1 Markov Decision Process

We model the sequential control problem of heating a household as a Markov
Decision Process (MDP) [47]. An MDP is a mathematical structure described by
the tuple $(\mathrm{X},\mathrm{U},f,\rho)$ where $\mathrm{X}$ is the state
space, $\mathrm{U}$ is the action space,
$f:\mathrm{X}\times\mathrm{U}\times\mathrm{X}\rightarrow[0,1]$ is the state
transition probability function and
${\rho:\mathrm{X}\times\mathrm{U}\times\mathrm{X}\rightarrow\mathbb{R}}$ is
the reward function. Through this mathematical structure, we can model the
control problem as an optimization problem to find the control policy
$\pi:\mathrm{X}\rightarrow\mathrm{U}$ that maximizes the expected reward
obtained from the environment. Formally, given a policy $\pi$, we consider the
definition of its Q-function [47]: comment

$Q^{\pi}(x,u)\doteq\mathbb{E}_{x^{\prime}\sim
f(x,u,\cdot)}\left[\rho(x,u,x^{\prime})+\gamma R^{\pi}(x^{\prime})\right],$

where:

$R^{\pi}(x_{0})\doteq\lim_{T\rightarrow+\infty}\mathbb{E}_{x_{t+1}\sim
f(x,u,\cdot)}\left[{\sum_{t=0}^{T}{\gamma^{t}\rho(x_{t},\pi(x_{t}),x_{t+1})}}\right].$

We then consider the definition of the optimal Q-function [47], that is:

$Q^{*}(x,u)\doteq\max_{\pi}{Q^{\pi}(x,u)}\;.$

The optimization problem is then to find an optimal policy $\pi^{*}$ [47] such
that:

$\pi^{*}(x)\in\arg\max_{u}{Q^{*}(x,u)}\;;\;\forall x\in\mathrm{X}\;.$

For simplicity of notation, in this work we consider the set $\mathrm{W}$ that
contains the influence that adds stochasticity into the system dynamics. This
set will be used in the next sections and will help have a more readable
formulation of the problem.

MDPs can be classified into two distinct categories: Fully Observable Markov
Decision Process (FOMDP) and Partially Observable Markov Decision Process
(POMDP). With FOMDP, the states of the system $x\in\mathrm{X}$ are fully
observable by the policy $\pi$. Conversely, in POMDP only a subset of the
state space $\mathrm{X}^{\text{obs}}\subset\mathrm{X}$ is observable by the
policy: $\pi:\mathrm{X}^{\text{obs}}\rightarrow\mathrm{U}$. More details
follow in Section 4.

###  3.2 Physics-informed Neural Network

Considering a POMDP as described in Section 3.1, we use a PiNN architecture to
forecast the next observable states of a system
$x^{\text{obs}}\in\mathrm{X}^{\text{obs}}$ by inserting physical equations
into the learning loss function of the ANN. We follow the PhysNet structure
proposed by Gokhale et al. [6] by deploying an encoder-based neural network.
PhysNet is specifically built for projecting a time-series of $d\in\mathbb{N}$
historical observable states
${x^{\text{obs}}_{t-d:t}\doteq(x^{\text{obs}}_{t-d},\dots,x^{\text{obs}}_{t})}$
into a compact hidden state $z_{t}$, while training the ANN such that these
latent values adhere to certain physical values. The obtained hidden state
$z_{t}$, together with the current observable state $x^{\text{obs}}_{t}$, the
current action $u_{t}\in\mathrm{U}$ and the exogenous state
$w_{t}\in\mathrm{W}$ are processed by a separate predictor component that will
return the next observable state $x^{\text{obs}}_{t+1}$ prediction. The
overall structure of the architecture is captured in Fig. 1.

Figure 1: Architecture of the PiNN. Yellow nodes represent past observations,
green nodes are the current observations, the pink nodes represent the current
exogenous state and action, the blue node is the hidden state and the red
nodes are the next predicted observations

Further details about the architecture training procedure and our contribution
to the model are presented in Section 4.2.

###  3.3 Monte Carlo Tree Search

MCTS models the states of the environment as nodes and the actions as edges in
a tree representing the possible scenarios that the current state may evolve
into. We will consider deterministic environments with the assumption that a
generative model of the MDP is available. To differentiate these deterministic
and simulable MDPs, we will adopt the following notation: we will refer to the
state space of the environment as $\tilde{\mathrm{X}}$, the action space as
$\tilde{\mathrm{U}}$, the transition function as
$\tilde{f}:\tilde{\mathrm{X}}\times\tilde{\mathrm{U}}\rightarrow\tilde{\mathrm{X}}$
and the reward function as
${\tilde{\rho}:\tilde{\mathrm{X}}\times\tilde{\mathrm{U}}\rightarrow\mathbb{R}}$.
One of the simplest and most effective versions of the algorithm is UCT [48]
and can be described in 3 different phases that are repeated iteratively:

  1. 1.

Selection: Starting from a root node $\tilde{x}^{0}\in\tilde{\mathrm{X}}$, the
tree is traversed by selecting, at each encountered node $\tilde{x}^{k}$, the
action ${\tilde{u}^{k}\in\tilde{\mathrm{U}}}\;$ (${k\in\mathbb{N}}$, where $k$
represent the depth of the node) until a leaf node is reached through action
$\tilde{u}^{l-1}$. The selection of each action in this phase is based on
scoring as described in Eq. 1 [48]:

$\displaystyle\tilde{u}^{k}=\arg\max_{\tilde{u}\in\tilde{\mathrm{U}}^{k}}{\biggl{\\{}\underbrace{Q(\tilde{x}^{k},\tilde{u})}_{\text{Value
score}}+\underbrace{2\,C_{p}\sqrt{\frac{\ln{N(\tilde{x}^{k})}}{N(\tilde{x}^{k},\tilde{u})}}}_{\text{Exploration
score}}\biggl{\\}}}$ (1) $\displaystyle;\forall k\in 0,1,\dots,l-1$

where $N(\tilde{x})$ is the number of time the node $\tilde{x}$ has been
reached following an action selected in Eq. 1, $N(\tilde{x},\tilde{u})$ is the
number of times the action $\tilde{u}$ has been selected in equation Eq. 1
from state $\tilde{x}$, $Q(\tilde{x},\tilde{u})$ is a state-action value
function that estimates the expected future rewards by executing the action
$\tilde{u}$ from state $\tilde{x}$ and $C_{p}$ is a hyperparameter used to
balance the exploration-exploitation trade-off.

  2. 2.

Expansion and evaluation: Once a leaf node $\tilde{x}^{l}$ is reached, if the
state is not terminal, the leaf gets expanded by adding a new node. The node
value then gets evaluated by performing a rollout that estimates the expected
reward obtainable in the future.

  3. 3.

Backpropagation: Finally, the statistical information about the rewards
obtained in the tree traversal gets propagated back to the root node by
updating each node’s Q value.

These phases are repeated multiple times. At the end of the search, the best
action is selected based on the optimal value approximation obtained. It has
been proven that this search converges to the optimal solution [48].

New implementations have been made in the last decades by, for example,
changing the UCT score formula Eq. 1 [49] and adding NNs for value and policy
estimation in the tree search [50, 45, 46, 5]. Full details of our
implementation of MCTS are described in Section 4.4.

##  4 Application framework

###  4.1 Residential heating as an MDP

We model the control problem of heating a household as a stochastic POMDP to
optimize the energy cost of the heating component while staying close to the
desired temperature set by the users $T_{\text{r\\_set}}$.

As before, with $t$ we note the time-step index of each value. We define an
observable state at a time-step $t$ as:
${x^{\text{obs}}_{t}=(T_{\text{r},t},u_{\text{phys},t-1})\in\mathrm{X}^{\text{obs}}}$
where $T_{\text{r},t}$ is the room temperature of the building we wish to
control and $u_{\text{phys},t-1}$ is the energy consumed by the heating system
in the previous time-step. The action $u\in\mathrm{U}$ is a continuous single
value indicating the control signal for the heating component. The exogenous
influences are defined as ${w_{t}=(\tau_{t},T_{\text{a},t})}$ where
$\tau_{t}\in[0,24]$ indicates the hour of the day and $T_{\text{a},t}$ is the
outdoor temperature. The reward function is defined as:

$\displaystyle\rho(x_{t},u_{t})\doteq$
$\displaystyle\underbrace{-\,u_{\text{phys},t}\,\lambda_{t}}_{\text{Cost
optimization}}$ (2)
$\displaystyle\underbrace{-\,(\,T_{\text{r\\_set},t}-T_{\text{r},t+1}\,)^{+}\,c_{1}-(\,T_{\text{r},t+1}-T_{\text{r\\_set},t}\,)^{+}\,c_{2}}_{\text{Thermal
Comfort optimization}}\;.$

where $\lambda_{t}$ is the energy price at time-step $t$,
$T_{\text{r\\_set},t}$ is the desired temperature set by the users and
$c_{1},c_{2}$ are hyperparameters used to balance the energy cost objective
with the user constraints. The thermal comfort optimization uses an
asymmetrical form ($c1>c2$) to avoid excessive penalization for pre-heating
the room. Last, the transition function $f$ describes the thermal dynamics of
the building. Because these dynamics are hard to fully describe
mathematically, we approximate them: we consider a deterministic FOMDP
approximating the POMDP described above by using a PiNN that represents the
state transition function.

###  4.2 PhysNet as a forecaster

To model the building’s thermal dynamics, we use a PhysNet architecture as
described in Section 3.2. We expanded the model to obtain a multi time-step
forecast by deploying a sequential, autoregressive, multi-loss training loop
with windowed inference. Following [6], we considered the 2R2C model [22]
shown in Fig. 2 to infuse physical knowledge into the encoder training.

Figure 2: RC model used in the encoder loss [22].

The physical equations obtained are then:

$\displaystyle\begin{bmatrix}\dot{T_{\text{r}}}\\\
\dot{T_{\text{m}}}\end{bmatrix}=\begin{bmatrix}-\left(\frac{1}{C_{\text{r}}R_{\text{ra}}}+\frac{1}{C_{\text{r}}R_{\text{rm}}}\right)&\frac{1}{C_{\text{r}}R_{\text{rm}}}\\\\[7.0pt]
\frac{1}{C_{\text{m}}R_{\text{rm}}}&-\frac{1}{C_{\text{m}}R_{\text{rm}}}\end{bmatrix}\cdot\begin{bmatrix}T_{\text{r}}\\\
T_{\text{m}}\end{bmatrix}+$
$\displaystyle+\begin{bmatrix}\frac{c_{\text{p}}T_{\text{s}}}{C_{\text{r}}}\\\\[5.0pt]
0\end{bmatrix}\cdot\dot{u}+\begin{bmatrix}\frac{1}{C_{\text{r}}R_{\text{ra}}}&\frac{\gamma}{C_{\text{r}}}&\frac{1}{C_{\text{r}}}\\\\[5.0pt]
0&0&0\end{bmatrix}\cdot\begin{bmatrix}T_{\text{a}}\\\ G\\\
I_{\text{g}}\end{bmatrix}\;.$ (3)

Here, $T_{\text{m}}$ indicates the building mass temperature. The rest of the
symbols follow the notation used in Vrettos et al. [22]. The NN structure is
designed to predict the next observable state for each prediction. The output
of the predictor is then used as input for the next predictions in an
autoregressive loop to extend the prediction horizon up to a desired number of
time-steps $h\in\mathbb{N}^{+}$. To further incentivize the model’s
generalization ability, the predictor is trained to forecast the room
temperature difference between the current observable state and the next one.
To train the PhysNet, two different losses are integrated into the training
loop: one regression loss responsible for the observable states prediction,
and one physics loss responsible for the adherence of the latent space with
the thermal mass physical values:

$\mathcal{L}=\mathcal{L}_{\text{reg}}+\mathcal{L}_{\text{phys}}\;.$

The regression loss is an L2 loss between the predicted observable states and
the measured ones:

${\mathcal{L}_{\text{reg}}\doteq\sum_{t=1}^{N}{\left(\hat{x}^{\text{obs}}_{t+1:t+h}-x^{\text{obs}}_{t+1:t+h}\right)}^{2}}\;.$

The physics loss is used to train the hidden state prediction. It consists of
another L2 loss that compares the predicted mass temperatures vector with a
target vector $\mathbf{\hat{T}}_{\text{m}}^{t}$. To better understand the
training process of the encoder, we introduce the following notations:

  * •

We define $\mathbf{T}_{\text{m}}^{t}\in\mathbb{R}^{h}$ as the vector
containing the $h$ autoregressive predictions of the mass temperature (i.e.,
of the hidden state) when starting the prediction from the time-step $t$;

  * •

We define $\mathbf{T}_{\text{r}}^{t}\in\mathbb{R}^{h}$ as the vector
containing the $h$ measured room temperatures starting from the time-step $t$.

The target value for the encoder training when starting from the time-step $t$
is then defined by applying Section 4.2 as:

$\mathbf{\hat{T}_{m}}^{t}=\mathbf{T}_{\text{m}}^{t-1}+\frac{\Delta_{t}}{C_{\text{m}}R_{\text{rm}}}\left(\mathbf{T}_{\text{r}}^{t-1}-\mathbf{T}_{\text{m}}^{t-1}\right)\;,$
(4)

where $\Delta_{t}$ is the duration of a time-step. The loss function to be
minimized for the predictions starting from time-step $t$ will then be:

${\mathcal{L}_{\text{phys}}\doteq\sum_{t=1}^{N}{\left(\mathbf{\hat{T}}_{\text{m}}^{t}-\mathbf{T}_{\text{m}}^{t}\right)}^{2}}\;.$

To study the impact of the PiNN component, we also perform a Black-Box
ablation that omits $\mathcal{L}_{\text{phys}}$. More details about the neural
networks hyperparameters are in Appendix A.

###  4.3 Control-oriented modeling

We design a second MDP that will approximate the real household and will
interact with the MCTS agent to gain optimal control actions. In particular:
we design a FOMDP with deterministic dynamics described by the PhysNet
introduced in Section 4.2, i.e., the exogenous information gets forecasted for
a fixed horizon $h$ to enable a deterministic simulation provided by the PiNN
model. To differentiate it from the real household POMDP, we indicate this new
environment with a different notation:
$(\tilde{\mathrm{X}},\tilde{\mathrm{U}},\tilde{f},\tilde{\rho})$, consistent
with what we introduced in Section 3.3. The state at certain time-step $t$ is
a continuous vector defined as:

$\tilde{\mathbf{x}}_{t}\doteq(\tau_{t},\;T_{\text{r},t},\;T_{\text{m},t})\in\tilde{\mathrm{X}}$
(5)

where: $\tau_{t}\in[0,24[$ is the time of the day,
$T_{\text{r},t}\in\mathbb{R}$ is the room temperature and
$T_{\text{m},t}\in\mathbb{R}$ is the (predicted) mass temperature of the
building. Because the PhysNet model requires a series of $d$ past states to
obtain the predicted mass temperature, we define an augmented state as:

$\tilde{\mathbf{x}}_{t}^{\text{aug}}\doteq(T_{\text{r},t-d:t-1},u_{\text{phys},t-d-1:t-1})$
(6)

The action space is discrete and single-dimensional
$\tilde{\mathrm{U}}\subset\mathrm{U}$, representing the heating system control
action. The transition function $\tilde{f}$ is a PiNN as described in Section
4.2. Last, the reward function $\tilde{\rho}$ is defined, similarly to the
original one introduced in Eq. 2, as:

$\displaystyle\tilde{\rho}(\tilde{x}_{t},\tilde{u}_{t})\doteq$
$\displaystyle\underbrace{-\,u_{\text{phys},t}\,\lambda_{t}}_{\text{Cost
optimization}}$ (7)
$\displaystyle\underbrace{-\,(\,T_{\text{r\\_set},t}-T_{\text{r},t+1}\,)^{+}\,c_{1}-(\,T_{\text{r},t+1}-T_{\text{r\\_set},t}\,)^{+}\,c_{2}}_{\text{Thermal
Comfort optimization}}$

This structure provides the capability to simulate future states — which
approximate the real dynamics through using a PiNN — enabling the deployment
of an MCTS algorithm.

###  4.4 Applied MCTS algorithms

We now describe the details of our MCTS algorithm, strongly inspired by the
work of [46, 5]. Similarly to Section 3.3, the algorithm consists of a
sequence of 3 phases repeated iteratively:

  1. 1.

Selection: Starting from a root node $\tilde{x}^{0}\in\tilde{\mathrm{X}}$, the
tree is traversed by selecting the action
${\tilde{u}^{k}\in\tilde{\mathrm{U}}^{k}\;;k\in\mathbb{N}}$ until a leaf node
$\tilde{x}^{l}$ is reached. We use a dynamic action space
${\tilde{\mathrm{U}}^{k}\subseteq\tilde{\mathrm{U}}\;;\;\forall
k\in\mathbb{N}}$ that represents the allowed actions from state
$\tilde{x}^{k}$ with respect to the environment-based constraints. The
selection of each action in this phase is based on:

$\displaystyle\tilde{u}^{k}=\arg\max_{\tilde{u}\in\tilde{\mathrm{U}}^{k}}{\biggl{\\{}\underbrace{\tilde{Q}(\tilde{x}^{k},\tilde{u})}_{\text{Value
score}}+\underbrace{\alpha\frac{\sqrt{N(\tilde{x}^{k})}}{1+N(\tilde{x}^{k},\tilde{u})}}_{\text{Exploration
score}}\biggl{\\}}};$ (8) $\displaystyle\forall k\in 0,1,\dots,l-1$

where $\alpha$ is a hyperparameter used to balance the exploration-
exploitation trade-off. In our experiments, we set $\alpha=1$, unless stated
otherwise.

  2. 2.

Expansion: Once a leaf node $\tilde{x}^{l}$ is reached, if its depth does not
exceed a fixed maximum value, the leaf gets expanded by adding new nodes, one
for each possible action: $\tilde{\mathrm{U}}^{l}\subseteq\tilde{\mathrm{U}}$.

  3. 3.

Backpropagation: After the expansion phase, statistical information about the
reward obtained in the selected trajectory gets backpropagated until the root
node. In particular, starting from the leaf node, the $\tilde{Q}$ function
gets updated as:

$\displaystyle\tilde{Q}(\tilde{x}^{k},\tilde{u}^{k})=\frac{N(\tilde{x}^{k},\tilde{u}^{k})\tilde{Q}(\tilde{x}^{k},\tilde{u}^{k})+\frac{G^{k+1}}{(l-k)}}{N(\tilde{x}^{k},\tilde{u}^{k})+1};$
(9) $\displaystyle\forall k=l-1,\dots,0$

where $\tilde{Q}(\tilde{x}^{k},\tilde{x}^{k})$ is initialized as
$\tilde{\rho}(\tilde{x}^{k},\tilde{u}^{k})$ and $G^{k}$ is the discounted
accumulated reward defined as:

$\begin{cases}G^{l}=\tilde{\rho}(\tilde{x}^{l-1},\tilde{u}^{l-1})\\\
G^{k}=\tilde{\rho}(\tilde{x}^{k-1},\tilde{u}^{k-1})+\gamma G^{k+1}\;;\forall
k=l-1,\dots,1\end{cases}$ (10)

where $\gamma$ is the discount value.

It is important to note that $\tilde{Q}(\tilde{x},\tilde{u})$ is not intended
as an expected cumulative reward of performing action $\tilde{u}$ from state
$\tilde{x}$ (i.e., the interpretation of a Q function in classical RL). In
fact; $\tilde{Q}$ is the approximation of the expected average reward,
conversely to the classical RL interpretation of a Q function that makes use
of the Bellman equations.

The rewards $\tilde{\rho}(\tilde{x},\tilde{u})$ considered in this algorithm
are normalized between $0$ and $1$, ensuring that every $\tilde{Q}$ value in
Eq. 9 is also contained in $[0,1]$. At the end of these repeated phases, the
optimal action is obtained by considering the most selected edge of the tree
during the Selection phase (Eq. 8). We will refer to this algorithm as Vanilla
MCTS. To further evaluate MCTS techniques in the BEM domain, we also deployed
a more advanced version that uses an NN to lead the tree search through more
optimal nodes, similarly to what was proposed in [46]. The algorithm follows
the same structure as described above and changes Eq. 8 to:

$\displaystyle\tilde{u}^{k}=\arg\max_{\tilde{u}\in\tilde{\mathrm{U}}^{k}}{\biggl{\\{}\underbrace{\tilde{Q}(\tilde{x}^{k},\tilde{u})}_{\text{Value
score}}+P(\tilde{x}^{k},\tilde{u})\underbrace{\alpha\frac{\sqrt{N(\tilde{x}^{k})}}{1+N(\tilde{x}^{k},\tilde{u}^{k})}}_{\text{Exploration
score}}\biggl{\\}}}\,;$ (11) $\displaystyle\forall k\in 0,1,\dots,l-1$

Where $P(\tilde{x},\tilde{u})$ is a prior probability value associated to
action $\tilde{u}$ from state $\tilde{x}$ and approximated by a neural network
$p(\tilde{x})$. The neural network is trained offline by using sampled
trajectories obtained with the Vanilla MCTS algorithm, considering the FOMDP
simulator for both the search and evaluation environment. We will refer to
this algorithm as AlphaZero MCTS, following the name of the original work that
inspired us [46]. When using AlphaZero MCTS, we fixed $\alpha=3.5$. More
details about the ANN structure are in Appendix A.

###  4.5 Experiments set up

To train and evaluate our PiNN architecture, we used a benchmark model
developed by Blum et al. [51]. We consider their single-zone Hydronic Heat
Pump. In particular, we used BOPTEST-Gym: an Open-AI-Gym implementation
thereof developed by Arroyo et al. [52]. In this set-up, the action space
consists of the modulation signal of the heat pump, that is:
$\mathrm{U}\doteq[0,1]$.

To evaluate the Physnet forecasting ability, we used BOPTEST-Gym to generate a
time-series of data by simulating the heat pump usage with a continuous
controller (details in Appendix B). The time-series data is being simulated
starting by the first of January and is described in Section 4.5. We use a
time-step of 30 minutes. We add cumulative noise to the outside temperature to
emulate the prediction error of a weather forecaster tool (see Appendix C for
more details).

Regarding the MCTS techniques evaluation, we first initialized the BOPTEST-Gym
Environment and simulated 10 days with a discrete rule-based controller
(details in Appendix B). The data obtained by the simulations are then used to
train the PhysNet (or the Black-Box) model.

Table 1: Description of the data used
Data name | Description | Value domain
---|---|---
$\tau$ | Hour of the day | $[0,24[$
$T_{\text{r}}$ | Indoor temperature [°C] | $[15,25]$
$T_{\text{a}}$ | Outside temperature measured in a dry bulb [°C] | $[-10,20]$
$u$ | Heat pump modulating signal for the compressor speed | $[0,1]$
$u_{\text{phys}}$ | Heat pump electrical power consumed [W] | $[0,4000]$
$\lambda$ | Belgian day-ahead energy prices as determined by the BELPEX electricity market [€/kWh] | $[-0.4,0.4]$

We then used our trained networks to build the tree search of the algorithms
described in Section 4.4. When the tree search is completed, the best action
is then applied to the BOPTEST-Gym Environment, which provides the new state
values to initialize another simulated environment. The process is then
repeated as needed. The simulator’s NNs are retrained after every day of
evaluation by adding the newly obtained data into the training dataset. In our
case, the algorithms are evaluated for a consecutive period of 11 days. The
whole controller structure is illustrated in Fig. 3.

Figure 3: Controller structure. The simulator first gets trained with
historical data. It is then used to build a tree search. The optimal action is
then deployed in the real case, which updates the states of the simulator.
These steps are repeated as needed.

Because the MCTS algorithm works with discrete actions, we considered 5
equally spaced actions over the entire action space:
$\tilde{\mathrm{U}}\subset\mathrm{U}\doteq[0,1]$. To further analyze the MCTS
algorithm’s potential, we consider two different price situations: a realistic
one with real-world price data; and a synthetic one, with an artificially
built squared wave price data.

###  4.6 Constraints management

When using MCTS, we assume a backup controller that forces the heat pump to
activate (respectively turn off) when the room temperature falls below the
desired one by $\Delta_{\text{T}}^{-}$ (respectively exceeds it by
$\Delta_{\text{T}}^{+}$). The values and the relations described above are
easily changeable based on the use case. In the MCTS framework, this backup
controller translates to trimming the action edges, i.e., only keeping the
actions as enforced by the $\Delta_{T}^{\pm}$ bounds. It is worth noting that
such backup constraints can be picked with total freedom based on the
available state values, so more complex constraints as opposed to just
clipping could be integrated.

##  5 Results

In this section we aim to answer the following research questions:

  1. Q1:

Is the PhysNet model more accurate and data-efficient compared to a Black-Box
model? (Section 5.1)

  2. Q2:

Is the PhysNet model able to provide better control actions when used in a
Vanilla MCTS algorithm compared to a Black-Box model? (Section 5.2)

  3. Q3:

What computational benefits can we obtain by a more targeted tree search
exploration (i.e., using AlphaZero MCTS instead of Vanilla MCTS)? (Section
5.3)

We present a sequence of three experiments that will answer these questions.

###  5.1 Forecasting results

We first compare the forecasting results of the PiNN model to its Black-Box
counterpart to analyze the overall prediction metrics of both models and
investigate their interpretability. We generated training sets of 2, 5, and 24
days using the BOPTEST framework, with a prediction horizon of 3, 6, and 12
hours. Each trained model is then evaluated on a test set of 6 days. For each,
we repeat the training process for 100 times to check the training stability.
Using the Mean Absolute Error (MAE) of the predictions in the test set, Fig. 4
plots the median value and the interquartile range of the results.

Figure 4: Prediction results of the PhysNet architecture and its Black-Box
counterpart with different training sizes and prediction horizons.

We observe that the PhysNet model is more data efficient than the Black-Box
model, confirming that infusing physical knowledge helps, for a small dataset,
the temperature predictions. Yet, in terms of energy consumption prediction,
the two models’ performance is virtually the same. In both the temperature and
energy consumption predictions, the results converge to the same values when
provided with a large enough training dataset. Last, because the mass
temperature of a building is difficult to measure, we cannot quantitatively
evaluate the hidden state prediction of the PhysNet model with a numerical
metric. Therefore, we empirically validate that the results follow an expected
behavior. A plot of a whole prediction horizon of 12 hours in the test set is
shown in Fig. 5. From a graphical analysis, we observe that the predicted
$T_{\text{m}}$ is varying with a higher delay compared to $T_{\text{r}}$,
similarly to a physical value subject to a higher inertia. Even though we
cannot validate the accuracy of such predictions, we can acknowledge that they
follow an expected behavior, i.e., the mass temperature acts as a thermal
storage that slowly releases (or absorbs) heat to (or from) the room.

Figure 5: PiNN predictions with a horizon of 12 hours.

###  5.2 PiNN control benefits

To further assess the PiNN model’s benefits from a control perspective, we
benchmark the cumulative reward obtained by the Vanilla MCTS algorithm when
aligned with the PhysNet. We also considered a bang-bang rule-based controller
as a baseline to further benchmark our results. Details of the bang-bang rule-
based controller can be found in Appendix B. The reward is normalized between
0 and 1 for each time-step. We use different configurations of the MCTS
algorithm’s hyperparameters for each model, starting from a fast evaluation
(low number of simulations and low tree depth) to a slower but more accurate
evaluation (higher number of simulations and higher tree depth). The results
are shown in Fig. 6.

Figure 6: Average daily rewards obtained by the Vanilla MCTS when applied to
11 test days with a PhysNet and a Black-Box simulator.

We clearly see that for both price profiles, a substantial improvement of the
MCTS algorithm is achieved when using the PiNN model vs. the purely data-
driven Black-Box one. We hypothesize that this is due to the higher physical
consistency of the PiNN model when used in situations that are not seen in the
training data, as previously observed by Di Natale et al. [18]. Last, in Fig.
7 we show two different policies applied to the same day.

Figure 7: Action sequence selected by the rule-based bang-bang policy and the
PhysNet-based Vanilla MCTS respectively. The two policies are considered in
the same days. The MCTS policy is able to remain closer to the desired
temperature while heating the room at lower price slots; resulting in a higher
reward compared to the rule-based controller.

There we can observe how the MCTS controller is able to better follow to the
desired temperatures (as observable, e.g., at time $\sim 20$h), while using
the heater at times with lower energy cost, leading to higher rewards compared
to the rule-based controller.

###  5.3 DL-infused MCTS results

A drawback of the Vanilla MCTS algorithm is its slow execution time for each
control action. As already shown for different problems [46, 5], the addition
of prior knowledge into the tree search can increase its performance in terms
of higher rewards at lower computational time. For this reason, we implemented
the AlphaZero MCTS algorithm as described in Section 4.4. The NN used in Eq.
11 is trained offline at the end of each day by using simulated samples
obtained with data from previous days. These samples are obtained by using the
Vanilla MCTS algorithm and by searching and evaluating on the same simulated
environment. Since training is performed offline, we choose a high enough
number of tree traversals to sample our simulated trajectories. We then
compared the results of the Vanilla MCTS algorithm with this AlphaZero MCTS
algorithm. Figure 8 shows the daily average cumulative reward obtained using
the PiNN model. The results show a noticeable improvement in the obtained
rewards for the AlphaZero MCTS algorithm compared to its Vanilla version,
especially when a low number of tree traversals is used. Note that for real-
world applications there needs to be a balance between performance and
computational effort required. The results presented in Fig. 8 demonstrate
that the AlphaZero MCTS controller can efficiently find valid control actions
with a lower number of simulations, leading to a controller that can be
(potentially) deployed in real-world buildings in a scalable manner.

Figure 8: Average daily rewards obtained by the Vanilla MCTS and AlphaZero
MCTS when applied to 11 test days with a PhysNet simulator.

##  6 Conclusions

In this paper, we presented an MCTS algorithm with a PiNN applied to a
residential heating Demand Response problem. We showed the control benefits of
the PiNN simulator compared to a Black-Box one. We then used a Deep Learning
infused MCTS algorithm inspired by AlphaZero to evaluate its efficiency
benefits compared to a more standard version, showing promising results for
the AlphaZero-based technique. Thanks to its flexibility to incorporate state-
dependent constraints on the allowed actions, we believe MCTS can be a valid
technique for Demand Response control of a residential heater. The results
show that MCTS control performance is remarkable, especially when a physically
infused model is used.

Following this work, different research directions can be made to further
explore MCTS in this scope. To list a few that we consider the most relevant:
a complete comparison between the MCTS techniques and state-of-the-art control
algorithms would further clarify the potential of these methods. Further
refining of the MCTS algorithm based on recent evolutions in that field (e.g.,
MuZero [5]) is a direction of future work. Also, in settings where the
constraints considered vary dynamically over time, MCTS could be even more
beneficial. Future works will explore this aspect in more details. Finally,
while this paper established the potential of advanced MCTS techniques for
heat pump control, further validation in real-world deployments should be
carried out.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgments and funding

This work was supported by the European Union’s Horizon 2020 research and
innovation programme under the projects BRIGHT (grant agreement no. 957816)
and BIGG (grant agreement no. 957047).

We also thank Marie-Sophie Verwee for her technical support in the deployment
of our work.

##  Appendix A Hyperparameters

The PhysNet network is composed of two separate Feed Forward Neural Networks
(FFNNs), the encoder and the predictor. The encoder has a single hidden layer
composed of 32 neurons activated with a ReLU function. Its output layer is
activated with a tanh function. The predictor has a single hidden layer
composed of 64 neurons activated with a ReLU function. The output of the
predictor layer has two different activation functions: tanh for the
temperature, and ReLU for the energy consumption. The past observations depth
value $d$ is fixed to 24 time-steps (12 hours). All the neural network
parameters are trained with an Adam optimizer. The prior policy neural network
$p(\tilde{x})$ of the AlphaZero algorithm is a FFNN with 2 hidden layers of 64
and 32 neurons activated with a ReLU function. Its final output gets activated
through a softmax function and the whole network gets trained with a Cross-
Entropy loss and an Adam optimizer. All the neural networks used in this paper
were implemented using PyTorch 2.0 [53].

##  Appendix B Rule Based Controllers

The rule-based bang-bang controller referred to in this work heats the room
whenever its temperature drops below the desired one. Mathematically, its
policy is defined as:

$u_{t}=\begin{cases}1&\text{if}\;\;T_{\text{r},t}<T_{\text{r\\_set},t}\\\
0&\text{otherwise}\end{cases}$ (12)

The discrete rule-based controller is defined as:

$u_{t}=\begin{cases}0&\text{if}\;\;T_{\text{r},t}>T_{\text{r\\_set},t}\\\
0.25&\text{else if}\;\;T_{\text{r},t}>T_{\text{r\\_set},t}-0.05\\\
0.5&\text{else if}\;\;T_{\text{r},t}>T_{\text{r\\_set},t}-0.15\\\
0.75&\text{else if}\;\;T_{\text{r},t}>T_{\text{r\\_set},t}-0.25\\\
1&\text{otherwise}\end{cases}$ (13)

Finally, the continuous controller used to generate data in Section 5.1 is
defined as:

$u_{t}=\min\left(2(T_{\text{r},t}-T_{\text{r\\_set},t}),\;1\right)$

##  Appendix C Cumulative noise

To evaluate our experiments in a more realistic manner, we added random noise
to the future outside temperature used in the forecast and in the control
problems. For each time-step, we randomly chose a binary value to determine
the sign of the added noise (positive or negative). We then sampled and added
random noise from a normal distribution to the outside temperature. The noise
added gets cumulated while considering values more far into the future. Each
addition get multiplied with a $\sigma$ (or $-\sigma$) hyperparameter that
determines the magnitude of the added noise

## References

  * Eurostat [2020] Eurostat, 2020.  Energy, transport and environment statistics — 2020 edition.
  * Stoffel et al. [2023] Stoffel, P., Maier, L., Kümpel, A., Schreiber, T., Müller, D., 2023.  Evaluation of advanced control strategies for building energy systems.  Energy and Buildings 280, 112709.
  * Wang and Hong [2020] Wang, Z., Hong, T., 2020.  Reinforcement learning for building controls: The opportunities and challenges.  Applied Energy 269, 115036.
  * Nagy et al. [2023] Nagy, Z., Henze, G., Dey, S., Arroyo, J., Helsen, L., Zhang, X., Chen, B., Amasyali, K., Kurte, K., Zamzam, A., et al., 2023.  Ten questions concerning reinforcement learning for building energy management.  Building and Environment , 110435.
  * Schrittwieser et al. [2020] Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al., 2020.  Mastering atari, go, chess and shogi by planning with a learned model.  Nature 588, 604–609.
  * Gokhale et al. [2022] Gokhale, G., Claessens, B., Develder, C., 2022.  Physics informed neural networks for control oriented thermal modeling of buildings.  Applied Energy 314, 118852.
  * Boodi et al. [2018] Boodi, A., Beddiar, K., Benamour, M., Amirat, Y., Benbouzid, M., 2018.  Intelligent systems for building energy and occupant comfort optimization: A state of the art review and recommendations.  Energies 11, 2604.
  * Afroz et al. [2018] Afroz, Z., Shafiullah, G., Urmee, T., Higgins, G., 2018.  Modeling techniques used in building hvac control systems: A review.  Renewable and sustainable energy reviews 83, 64–84.
  * Homod [2013] Homod, R.Z., 2013.  Review on the hvac system modeling types and the shortcomings of their application.  Journal of Energy 2013.
  * Foucquier et al. [2013] Foucquier, A., Robert, S., Suard, F., Stéphan, L., Jay, A., 2013.  State of the art in building modelling and energy performances prediction: A review.  Renewable and Sustainable Energy Reviews 23, 272–288.
  * Li and Wen [2014] Li, X., Wen, J., 2014.  Review of building energy modeling for control and operation.  Renewable and Sustainable Energy Reviews 37, 517–537.
  * Deb et al. [2017] Deb, C., Zhang, F., Yang, J., Lee, S.E., Shah, K.W., 2017.  A review on time series forecasting techniques for building energy consumption.  Renewable and Sustainable Energy Reviews 74, 902–924.
  * Bourdeau et al. [2019] Bourdeau, M., qiang Zhai, X., Nefzaoui, E., Guo, X., Chatellier, P., 2019.  Modeling and forecasting building energy consumption: A review of data-driven techniques.  Sustainable Cities and Society 48, 101533.
  * Ali et al. [2021] Ali, U., Shamsi, M.H., Hoare, C., Mangina, E., O’Donnell, J., 2021.  Review of urban building energy modeling (ubem) approaches, methods and tools using qualitative and quantitative analysis.  Energy and buildings 246, 111073.
  * Wetter et al. [2014] Wetter, M., Zuo, W., Nouidui, T.S., Pang, X., 2014.  Modelica buildings library.  Journal of Building Performance Simulation 7, 253–270.
  * Crawley et al. [2001] Crawley, D.B., Lawrie, L.K., Winkelmann, F.C., Buhl, W.F., Huang, Y.J., Pedersen, C.O., Strand, R.K., Liesen, R.J., Fisher, D.E., Witte, M.J., et al., 2001.  Energyplus: creating a new-generation building energy simulation program.  Energy and buildings 33, 319–331.
  * Szegedy et al. [2013] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R., 2013.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199 .
  * Di Natale et al. [2022] Di Natale, L., Svetozarevic, B., Heer, P., Jones, C.N., 2022.  Physically consistent neural networks for building thermal modeling: theory and analysis.  Applied Energy 325, 119806.
  * Afram and Janabi-Sharifi [2015] Afram, A., Janabi-Sharifi, F., 2015.  Gray-box modeling and validation of residential hvac system for control system design.  Applied Energy 137, 134–150.
  * Matthiss et al. [2023] Matthiss, B., Azzam, A., Binder, J., 2023.  Thermal building models for energy management systems, in: 2023 IEEE International Conference on Environment and Electrical Engineering and 2023 IEEE Industrial and Commercial Power Systems Europe (EEEIC/I&CPS Europe), IEEE. pp. 1–6.
  * Bacher and Madsen [2011] Bacher, P., Madsen, H., 2011.  Identifying suitable models for the heat dynamics of buildings.  Energy and buildings 43, 1511–1522.
  * Vrettos et al. [2016] Vrettos, E., Kara, E.C., MacDonald, J., Andersson, G., Callaway, D.S., 2016.  Experimental demonstration of frequency regulation by commercial buildings—part i: Modeling and hierarchical control design.  IEEE Transactions on Smart Grid 9, 3213–3223.
  * Raissi et al. [2019] Raissi, M., Perdikaris, P., Karniadakis, G.E., 2019.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational physics 378, 686–707.
  * Hochreiter and Schmidhuber [1997] Hochreiter, S., Schmidhuber, J., 1997.  Long short-term memory.  Neural computation 9, 1735–1780.
  * LeCun et al. [1998] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE 86, 2278–2324.
  * Schroff et al. [2015] Schroff, F., Kalenichenko, D., Philbin, J., 2015.  Facenet: A unified embedding for face recognition and clustering, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823.
  * Drgoňa et al. [2021] Drgoňa, J., Tuor, A.R., Chandan, V., Vrabie, D.L., 2021.  Physics-constrained deep learning of multi-zone building thermal dynamics.  Energy and Buildings 243, 110992.
  * Drgoňa et al. [2020] Drgoňa, J., Arroyo, J., Figueroa, I.C., Blum, D., Arendt, K., Kim, D., Ollé, E.P., Oravec, J., Wetter, M., Vrabie, D.L., et al., 2020.  All you need to know about model predictive control for buildings.  Annual Reviews in Control 50, 190–232.
  * Yao and Shekhar [2021] Yao, Y., Shekhar, D.K., 2021.  State of the art review on model predictive control (mpc) in heating ventilation and air-conditioning (hvac) field.  Building and Environment 200, 107952.
  * Afram et al. [2017] Afram, A., Janabi-Sharifi, F., Fung, A.S., Raahemifar, K., 2017.  Artificial neural network (ann) based model predictive control (mpc) and optimization of hvac systems: A state of the art review and case study of a residential hvac system.  Energy and Buildings 141, 96–113.
  * Lee et al. [2015] Lee, Y.M., Horesh, R., Liberti, L., 2015.  Optimal hvac control as demand response with on-site energy storage and generation system.  Energy Procedia 78, 2106–2111.
  * Reynolds et al. [2018] Reynolds, J., Rezgui, Y., Kwan, A., Piriou, S., 2018.  A zone-level, building energy optimisation combining an artificial neural network, a genetic algorithm, and model predictive control.  Energy 151, 729–739.
  * Chen et al. [2018] Chen, J., Augenbroe, G., Song, X., 2018.  Lighted-weighted model predictive control for hybrid ventilation operation based on clusters of neural network models.  Automation in Construction 89, 250–265.
  * Ławryńczuk [2014] Ławryńczuk, M., 2014.  Computationally efficient model predictive control algorithms.  A Neural Network Approach, Studies in Systems, Decision and Control 3.
  * Brandi et al. [2020] Brandi, S., Piscitelli, M.S., Martellacci, M., Capozzoli, A., 2020.  Deep reinforcement learning to optimise indoor temperature control and heating energy consumption in buildings.  Energy and Buildings 224, 110225.
  * Wei et al. [2017] Wei, T., Wang, Y., Zhu, Q., 2017.  Deep reinforcement learning for building hvac control, in: Proceedings of the 54th annual design automation conference 2017, pp. 1–6.
  * Patyn et al. [2018] Patyn, C., Ruelens, F., Deconinck, G., 2018.  Comparing neural architectures for demand response through model-free reinforcement learning for heat pump control, in: 2018 IEEE international energy conference (ENERGYCON), IEEE. pp. 1–6.
  * Nagy et al. [2018] Nagy, A., Kazmi, H., Cheaib, F., Driesen, J., 2018.  Deep reinforcement learning for optimal control of space heating.  arXiv preprint arXiv:1805.03777 .
  * Jiang et al. [2021] Jiang, Z., Risbeck, M.J., Ramamurti, V., Murugesan, S., Amores, J., Zhang, C., Lee, Y.M., Drees, K.H., 2021.  Building hvac control with reinforcement learning for reduction of energy cost and demand charge.  Energy and Buildings 239, 110833.
  * Wijaya et al. [2013] Wijaya, T.K., Papaioannou, T.G., Liu, X., Aberer, K., 2013.  Effective consumption scheduling for demand-side management in the smart grid using non-uniform participation rate, in: 2013 Sustainable Internet and ICT for Sustainability (SustainIT), IEEE. pp. 1–8.
  * Galván-López et al. [2014] Galván-López, E., Harris, C., Trujillo, L., Rodriguez-Vazquez, K., Clarke, S., Cahill, V., 2014.  Autonomous demand-side management system based on monte carlo tree search, in: 2014 IEEE International Energy Conference (ENERGYCON), IEEE. pp. 1263–1270.
  * Golpayegani et al. [2015] Golpayegani, F., Dusparic, I., Clarke, S., 2015.  Collaborative, parallel monte carlo tree search for autonomous electricity demand management, in: 2015 Sustainable Internet and ICT for Sustainability (SustainIT), IEEE. pp. 1–8.
  * Golpayegani et al. [2016] Golpayegani, F., Dusparic, I., Taylor, A., Clarke, S., 2016.  Multi-agent collaboration for conflict management in residential demand response.  Computer Communications 96, 63–72.
  * Kiljander et al. [2021] Kiljander, J., Sarala, R., Rehu, J., Pakkala, D., Pääkkönen, P., Takalo-Mattila, J., Känsälä, K., 2021.  Intelligent consumer flexibility management with neural network-based planning and control.  IEEE Access 9, 40755–40767.
  * Silver et al. [2017] Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al., 2017.  Mastering the game of go without human knowledge.  nature 550, 354–359.
  * Silver et al. [2018] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al., 2018.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science 362, 1140–1144.
  * Busoniu et al. [2017] Busoniu, L., Babuska, R., De Schutter, B., Ernst, D., 2017.  Reinforcement learning and dynamic programming using function approximators.  CRC press.
  * Kocsis and Szepesvári [2006] Kocsis, L., Szepesvári, C., 2006.  Bandit based monte-carlo planning, in: European conference on machine learning, Springer. pp. 282–293.
  * Rosin [2011] Rosin, C.D., 2011.  Multi-armed bandits with episode context.  Annals of Mathematics and Artificial Intelligence 61, 203–230.
  * Silver et al. [2016] Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al., 2016.  Mastering the game of go with deep neural networks and tree search.  nature 529, 484–489.
  * Blum et al. [2021] Blum, D., Arroyo, J., Huang, S., Drgoňa, J., Jorissen, F., Walnum, H.T., Chen, Y., Benne, K., Vrabie, D., Wetter, M., et al., 2021.  Building optimization testing framework (boptest) for simulation-based benchmarking of control strategies in buildings.  Journal of Building Performance Simulation 14, 586–610.
  * Arroyo et al. [2021] Arroyo, J., Manna, C., Spiessens, F., Helsen, L., 2021.  An open-ai gym environment for the building optimization testing (boptest) framework, in: Building Simulation 2021, IBPSA. pp. 175–182.
  * Paszke et al. [2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S., 2019.  Pytorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems 32. Curran Associates, Inc., pp. 8024–8035.