<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Semi-analytical Industrial Cooling System Model for Reinforcement Learning

Yuri Chervonyi  Equal contributions  DeepMind  Praneet Dutta  Equal
contributions  DeepMind  Piotr Trochim  Work done at DeepMind  Octavian Voicu
DeepMind  Cosmin Paduraru  DeepMind  Crystal Qian  Work done at DeepMind
Google  Emre Karagozler  DeepMind  Jared Quincy Davis  DeepMind  Richard
Chippendale  Work done at COMSOL  Gautam Bajaj  Google  Sims Witherspoon
DeepMind  Jerry Luo  DeepMind

###### Abstract

We present a hybrid industrial cooling system model that embeds analytical
solutions within a multi-physics simulation. This model is designed for
reinforcement learning (RL) applications and balances simplicity with
simulation fidelity and interpretability. The model’s fidelity is evaluated
against real world data from a large scale cooling system. This is followed by
a case study illustrating how the model can be used for RL research. For this,
we develop an industrial task suite that allows specifying different problem
settings and levels of complexity, and use it to evaluate the performance of
different RL algorithms.

##  1 Introduction

###  1.1 Background and Motivation

Industrial systems account for 54% of global energy usage [6] and 24% of
global net anthropogenic Greenhouse Gas (GHG) emissions. The latter percentage
rises to 34% if indirect emissions from energy are included, which would make
industrial systems the highest emitting sector [35]. Due to increasing global
demand for the products and services enabled by industrial systems, emissions
from this sector will continue to rise [26]. However, there is strong evidence
that interventions such as reduction in energy use per unit of output [38],
lightweight designs and extended product lifetimes can facilitate critical
emissions reductions across industrial systems [21]. Yet, optimizing
industrial systems is not straightforward; subsectors such as metals,
chemicals, waste and cement require customized approaches accounting for
different materials, processes and facility configurations.

Recent work has shown that reinforcement learning can be leveraged to
efficiently control and optimize industrial processes (e.g. see [37, 39]).
Among these processes, cooling/heating has been the main focus for many
applied researchers due its wide usage in residential and commercial
buildings, automobiles, industrial plant machinery, nuclear reactors, and many
other types of machinery. Examples of applying RL in real cooling/heating
systems include reducing energy consumption for cooling Google data centers by
40% [2] and controlling the HVAC systems, hot water tank and heat pumps in
residential [23] and industrial facilities [12, 40, 20].

However, applying RL in live facilities comes with technical challenges, such
as:

  * •

Limited exploration: During exploration, the agent must act cautiously to
avoid real-world consequences such as overheating a critical facility. The
agent cannot explore all combinations of the state and action space, resulting
in low-variance data. Furthermore, explore-exploit schedules must be
negotiated with the facility, often with the motivation for the agent to
exploit sooner rather than later.

  * •

Generalization: Although many industrial facilities are similar, policies
trained on one facility may not generalize to another facility due to slight
differences in the input feature space. Solutions are not robust to different
equipment conditions, weather patterns, and other sources of variance, as
training is limited to experiences with a singular system during a fixed
amount of real time interactions.

  * •

Sample efficiency: In live facilities, the agent interacts with the
environment and receives information at coarse frequencies such as 5 minutes
or an hour; this limits the amount of data that can be used for training and
also slows the process of validating existing models. From an algorithmic
perspective, we are constrained to approaching this problem with sample-
efficient algorithms.

  * •

Infrastructure and stakeholder management: Interacting with a live facility
requires significant additional engineering effort, including developing a
shared data schematic and data transfer API with the facility partner, writing
custom logic to integrate facility-specific features, and ensuring reliability
and availability during deployment.

With these challenges in mind, we present a novel semi-analytical simulation
model of an industrial cooling facility, developed specifically for
reinforcement learning applications. Interacting with a simulator rather than
a live facility greatly increases the amount of available data and allows the
agent to explore a broader action space safely. We provide a set of analytical
equations as well as the description of how to embed them into a multi-physics
simulation software. We hope that this will be a useful blueprint for
practitioners who want to improve industrial cooling system efficiency using
RL.

In order to allow us to create agents that generalize well, we also provide
the high level design for an Industrial Task Suite (ITS) that allows us to
specify different RL tasks based on the simulator. ITS can be used to generate
a curriculum of scenarios and defines highly parameterized facility
configurations to enable flexible experimentation and the development of
generalizable agents.

###  1.2 Related Work

####  1.2.1 Simulation

There are several tools offered for developing simulations for optimizing
industrial systems and processes. Among such tools, EnergyPlus [14] is a
popular open-source library for whole-building energy simulations. It is used
by by many researchers, especially for the RL applications [13, 30, 20].
Although it provides full-featured functionality, there are fidelity
limitations that can be crucial when simulating real-world environments [25,
32].

####  1.2.2 Optimization Research

Work in the domain of simulation-enabled industrial system optimization
research can broadly be segmented into those primarily addressing questions
pertaining to data, infrastructure, and methods. Nweye et al. [33] has probed
challenges around safety-critical building control and offline training,
arguing that the sequence of operations policy used to generate offline
trajectory data has a dramatic effect on the downstream performance of the
associated offline RL agents. With regards to infrastructure, [20] proposes a
scheme for weaving together popular free and open source tools to create
common building control research infrastructure and environments. It
illustrates a scheme for framing HVAC control as a Deep RL problem using
OpenAI gym [10], EnergyPlus [14], and Ray RLlib [27]. The authors overlay
real-world historical weather data from Toronto, Canada, and simulate a year’s
worth of weather dynamics, producing ablations to show the impact of weather,
and energy-temperature penalty coefficients. In terms of the methods, many
approaches have been proposed for energy savings and temperature control. For
example, Gao et al. [17] explored the use of a Deep Deterministic Policy
Gradients algorithm (DDPG) [28]; Zhang et al. [40] considered the Asynchronous
Advantage Actor Critic (A3C) [29]. Other adjacent works explore applications
of deep RL in other industrial control constrained optimization settings, such
as assembly line control [37] and combustion control [39], proposing parallel
agent training schemes to optimize throughput and mitigate simulation step-
time limitations.

###  1.3 Summary of Results

Our main contributions include:

  * •

Equations and design for our Semi-analytical Industrial Cooling System Model,
a simulation of a large cooling facility that combines analytical solutions
with a multi-physics simulation software (Section 2.2). The model’s fidelity
is verified against real-world data from a large commercial cooling facility
in the United States (Section 2.3).

  * •

Design for the Industrial Task Suite (ITS), a framework for RL research in the
domain of industrial systems (Section 3.1). Although we primary use it for the
industrial cooling applications, several other use-cases are presented in
Appendix B. ITS is designed to allow more granular configurability of
environments compared to the available task suites (such as the DeepMind
Control Suite [36]); specifically, allowing the combination of independently-
defined noise, constraints, scenarios, tasks and simulators for modelling
industrial problems.

  * •

Tasks for industrial cooling systems categorized by their perceived complexity
(Section 3.2). We also show how to define other new tasks using the
capabilities of the ITS.

  * •

RL Agents Benchmarks on these tasks (Section 3.4).

##  2 Industrial Cooling System Simulation

###  2.1 Chilled Water Plant System Overview

A cooling system is an apparatus employed to keep the temperature of a
structure or device from exceeding limits imposed by needs of safety and
efficiency [1]. Cooling systems use air or a liquid as a medium to absorb
heat. Many applications use water because it has a high boiling point and
specific heat. In the following sections, we focus on chilled water plants,
cooling systems that leverage water-cooled pipelines.

A chilled water plant system is depicted on Figure 1. These systems are
typically divided into two broader halves. The chilled water side extracts
heat from the building and is responsible for cooling the rooms, keeping them
within the desired temperature range. The condenser water side interacts with
the outside environment and, in turn, extracts heat from the chilled water
side. The flow of water is regulated through pumps on both sides. No water
mass is exchanged between the loops, being restricted to heat transfer via
conduction and convection. The cooling process can be achieved in the cooling
tower by the latent heat of evaporation and the chillers via the vapor
compression cycle.

Figure 1: A high level overview of a typical chilled water plant system. The
main components include one or several chillers, a heat exchanger (HEX),
pumps, cooling towers. Building load is heat transferred through the building
envelope (walls, roof, floor, windows, doors etc.) and heat generated by
occupants, equipment, and lights.

Chilled water plant systems use evaporative cooling, in which highly energetic
water molecules are evaporated from the surface, resulting in reduced
temperature for the remaining water [19]. There are two types of temperature
that are important in these cooling systems: dry bulb temperature and wet bulb
temperature. Dry bulb temperature is measured by a regular thermometer, by
exposing it to the stream of air while shielding it from radiation and
moisture. Wet bulb temperature is the lowest possible temperature of water
that can be achieved just by the evaporation of water. This can be measured by
a thermometer covered in the water-soaked cloth (wet-bulb thermometer), over
which air is passed. Dry bulb temperature and wet bulb temperature can be used
to calculate the relative humidity of air.

The rate of cooling is controlled by setting various system parameters known
as setpoints, such as temperature, pressure, fluid flow rates etc. Industrial
cooling systems are typically controlled by a Building Management System
(BMS), a heuristic-based policy that operates on these setpoints to keep the
building at a safe and comfortable desired temperature. A low level
proportional–integral–derivative (PID) controller changes an actuator
parameter to minimize the delta between this desired setpoint recommendation
and a measured sensor reading (see Figure 2). This setup, controlling
setpoints for multi-objective optimization subject to constraints, lends
itself well to reinforcement-learning approaches.

Figure 2: The detailed overview of a chilled water plant system, including the
most important setpoints that can be controlled to achieve the required
temperature. Here we see multiple chillers arranged in parallel to handle
various external conditions. Unlike a system presented on Figure 1, this
design does not include a heat exchanger.

###  2.2 Semi-Analytical Simulation Model

We leverage a multi-physics simulator software (COMSOL) for modeling fluid and
heat transfer across the system111This setup could be replicated by any
applicable fluid dynamics simulator. The main components of our cooling system
are (see Figure 1):

  * •

Chillers.

  * •

Cooling Towers.

  * •

Chilled Water Pump Bank.

  * •

Condenser Water Pump Bank.

  * •

Air Handling Unit.

  * •

Heat Exchanger.

  * •

PID controllers.

  * •

Valves, bends, friction loss parameters of the systems.

Our design goal is to balance simplicity with simulation fidelity and
interpretability. We leverage a simplified pipe-flow module for modeling the
fluid dynamics, which turns the meshed finite element analysis into a 2D
curve, improving computational efficiency for 3D pipes with a finite diameter.
We aim to model a large-scale industrial system where this approximation would
be a reasonable assumption. Specifically, the pipe length to diameter ratio is
large enough for the flow inside each pipe segment to be considered fully-
developed. We utilize additional libraries for modeling the heat exchanger and
PID controls, which controls components such as the temperature (via the
compressor), pressure and flow rate (via the pumps) across the fluid loops.
External weather conditions such as the dry bulb temperature, wet bulb
temperature, relative humidity and the building load profile form a critical
aspect of the system; we simulate these using public historical weather
databases. Finally, chillers, cooling towers and pump banks are described via
analytical solutions that have been verified using data from a real-world
facility. Next, we discuss these solutions in detail.

##### Chiller and the improved Gordon-Ng model

Figure 3: Vapor-compression cycle scheme used by a chiller. $T$s denote
temperatures and $Q$s denote heat. For the details see the description under
Eq. 4. This diagram is adapted from [16].

For the chiller model we start with [16], where the canonical Gordon-Ng model
[18] was simplified and improved. In this approximation, the chiller is
represented via a simple vapor-compression cycle (see Figure 3). The laws of
thermodynamics applied to this cycle under several assumptions, such as steady
state, ignoring losses, etc (for the details see [16]), give:

$\displaystyle W_{comp}=\frac{-DQ_{ev}^{2}-CQ_{ev}-BQ_{ev}+A}{DQ_{ev}+C},$ (1)
$\displaystyle Q_{ev}-Q_{c}+W_{comp}+Q_{LT}=0,$ (2) $\displaystyle
Q_{ev}=C_{ch}(T_{chi}-T_{cho}),$ (3) $\displaystyle
Q_{c}=C_{cw}(T_{ci}-T_{co}).$ (4)

Here $W_{comp}$ is the chiller compressor power. $Q_{ev}$ is the cooling load,
the heat removed from the building through water flow. $Q_{c}$ is the removed
heat in the condenser. $Q_{LT}$ is a total heat loss. $T_{chi}$ and $T_{cho}$
are the evaporator inlet and outlet temperatures. $T_{ci}$ and $T_{co}$ are
the condenser inlet and outlet temperatures. $C_{ch}=\dot{m}_{ch}c_{ch}$ is
the thermal capacitance of cooling water ($kW/K$) with $\dot{m}_{ch}$ as a
mass flow rate ($kg/s$) and $c_{ch}$ water specific heat capacity. $A,B,C,D$
are constants that encode various other quantities, the exact expressions for
them can be found in [16]. As suggested in [16], one can estimate values for
these constants upon actual measured values for the chiller.

If we neglect all heat losses $Q_{LT}$, Eqs. (1)-(4) become a system of four
equations with four unknowns: $Q_{ev},Q_{c},T_{cho},T_{co}$. Here, the agent
provides a recommendation for the chiller leaving temperature $T_{cho}$. Based
on this, a low-level PID controller adjusts the compressor speed, hence
affecting the compressor power $W_{comp}$ parameter. This adjustment is made
to minimize the error between the current sensor reading for $T_{cho}$ with
the recommendation provided. Analytically, we are now able to compute the
ground truth output temperatures $T_{cho},T_{co}$ with auxiliary variables
$Q_{ev},Q_{c}$. Since Eq. (1) is quadratic, in order to compute $Q_{ev},Q_{c}$
one needs to select a square root branch. Matching the solution to the read
data shows that one must pick the positive branch, this gives the final
result:

$\displaystyle T_{cho}=T_{chi}-\frac{Q_{ev}}{C_{ch}},\quad
T_{co}=T_{ci}-\frac{Q_{c}}{C_{cw}},$ (5) $\displaystyle
Q_{ev}=\frac{-b+R}{2D},\quad Q_{c}=Q_{ev}+W,$ (6) $\displaystyle
b=C+B+DW,\quad R=\sqrt{b^{2}-D(CW-A)}.$ (7)

Figure 4 shows a qualitative match between the chiller power computed via Eq.
(1) and the observed data from a real facility. We see that the analytical
model reproduces the observed power trends observed in the real facility. A
precise match can be obtained by tuning $A,B,C,D$ coefficients (as done in
[16]).

Figure 4: Comparison of the analytical model for a chiller power Eq.(1) and
the real observed data (x-axis represents days of a month). While we see a
qualitative agreement, the precise match is obtained by tuning the
coefficients $A,B,C,D$ (as done in [16]).

##### Cooling tower

Using the laws of thermodynamics, it was shown in [11] that the temperature of
the water leaving the cooling tower can be computed as:

$T_{co}=T_{ci}-(T_{ci}-T_{wb})(1-\mathrm{exp}[c_{8}(P_{pump})^{c_{9}}(P_{fan})^{c_{10}}]),$
(8)

where $T_{ci}$ is the temperature of the water entering the cooling tower,
$T_{wb}$ is the wet-bulb temperature, $P_{pump}$ is the condenser water pump
frequency (speed), $P_{fan}$ is the cooling tower fan frequency (speed). As
the valve position on cooling water pipe and the resistance characteristics of
the cooling water pipe unchanged, the condenser water pump flow is
proportional to the water pump frequency. Similarly, the cooling tower air
flow and fan power are also consistent with the law. The condenser water flow
rate $m_{c}$, the condenser water pump power consumption $W_{pump}$, the
cooling tower air flow $m_{a}$ and the cooling tower fan power consumption
$W_{fan}$ are calculated [11] by:

$\displaystyle m_{c}=c_{11}P_{pump},\quad W_{pump}=c_{12}(P_{pump})^{3},$ (9)
$\displaystyle m_{a}=c_{13}P_{fan},\quad W_{fan}=c_{14}(P_{fan})^{3}.$ (10)

The coefficients $c_{i}$ are set by fitting the observed data (this can be
done via a regression as described in[16]). Figure 5 illustrates the matching
dynamics between equations for the fan/pump power and the observed data.

(a) Pump power.

(b) Fan power.

Figure 5: Verification of equations for a pump (9) and a fan (10) power
(x-axis represents times of day). Red lines represent computed values and blue
lines - measured.

##### Multiple cooling towers

Real-world cooling facilities usually have multiple cooling towers. In order
to describe a real facility we need to extend Eqs.(9), (10) to multiple
cooling towers. To do so, we observe that the pumps create a force to push the
water, and that there are several obstructions that influence the flow. When
the number of chillers and pumps are the same, we get a perfect linear
relationship; when there are more pumps, the chillers’ flow is weaker due to
resistance. To account for this, we can introduce another term in the first
equation in Eq. (9), which is proportional to the difference between the
number of chillers and pumps:

$m_{c}=\sum_{i}^{N}P_{pump}^{(i)}(a_{1}-a_{2}(N_{pumps}-N_{chillers})).$ (11)

Note that the number of fans might have a small effect on this as well. It
turns out that this effect is negligible, as the suggested equation holds in
the observed data (see Figure 6).

(a) Short timescale (x-axis represents times of day).

(b) Long timescale (x-axis represents days).

Figure 6: Verification of Eq.(11) for the condenser water flow rate under
different circumstances. The second row on each figure shows a good agreement
between computed vs observed flow rates for both (a) short and (b) long time
scales. Figures in other rows show that the good agreement is observed for
various situations including different number of chillers, fans and pumps.
Here we have three pumps labeled as pump5, pump6, pump7.

A further analysis shows that Eq.(8) can be easily extended for multiple
pumps/fans via a simple replacement $P\to\sum_{i}P^{(i)}$:

$T_{co}=T_{ci}-(T_{ci}-T_{wb})\left(1-\mathrm{exp}\left[c_{8}\left(\sum_{i}P_{pump}^{(i)}\right)^{c_{9}}\left(\sum_{j}P_{fan}^{(j)}\right)^{c_{10}}\right]\right).$
(12)

Now, we apply the above equations to the chiller plant depicted at Figure 2.
We need to use inverse variations on the above equations, such that the inputs
are the setpoints (plus other measured variables), and the outputs are the
condenser water pump power ($W_{pump}$) and the fan power ($W_{fan}$). In our
chiller plant, all pumps (and fans) are run with the same frequencies, so re-
arranging (11), (10), (12) gives:

$\displaystyle
P_{pump}^{(i)}=P_{pump}=\frac{m_{c}}{c_{11}N_{pumps}(a_{1}-a_{2}(N_{pumps}-N_{chillers}))},$
(13) $\displaystyle W_{pump}^{(i)}=c_{12}(P_{pump})^{3},\quad
i\in[1,\cdots,N_{pumps}].$ (14) $\displaystyle
P_{fan}^{(j)}=P_{fan}=\frac{1}{N_{fans}}\left(\frac{1}{c_{8}(N_{pumps}P_{pump})^{c_{9}}}\log\left[1+\frac{T_{co}-T_{ci}}{T_{ci}-T_{wb}}\right]\right)^{1/c_{10}},$
(15) $\displaystyle W_{fan}^{(j)}=c_{14}(P_{fan})^{3},\quad
j\in[1,\cdots,N_{fans}].$ (16)

##### Integrating Analytical Solutions into Multi-physics Software

As described in the beginning of this section, we use a commercially available
multi-physics software for modeling the fluid dynamics. This offers the
flexibility to consider a wide variety of different physical
processes/applications that can be solved as transient or steady state
problems. In our case this software allows to define source and sinks for the
physical parameters such as temperature, fluid velocity and pressure. Given
the transient simulation, we extract the measured parameter values at every
simulation step. Hence, we leverage the simulation state variables which serve
as a subset of input parameters for the analytical equations described above.

For example, in case of the Gordon-Ng chiller model, we wish to compute the
temperature gradient at both the evaporator and the condenser side (see Figure
3). We consider the measured chiller inlet temperatures and mass flow rate at
each of these ends. These are measured via "sensor probes" attached at the
inlet of the chiller. At every simulation step, these readings are fed into
the system of equations (5)-(7) and the output temperature values are
computed. We assume a negligible fluid flow, pressure drop across the chiller
cross section (relative to that across the facility). Next, we define a fluid
flow source, which acts as the chiller outlet. The incoming water temperature
is also provided by the output of the Gordon-Ng model (5)-(7). We constrain
this outlet incoming differential pressure 222The differential pressure is
defined as the difference in pressure between two points in the water pipes.
In the case of the chilled water system, the differential pressure at any
point is measured relative to the pressure at the chilled water pump inlet.
and mass flow rate to that of the inlet measured values. This process is
repeated for all chillers.

The same technique is used for the cooling tower and pump bank analytical
models. The former only computes a temperature gradient similar to the
chiller, and the pump bank computes the differential pressure.

###  2.3 Simulation Fidelity

In the previous section, we summarized analytical solutions used as a part of
our simulation and demonstrated their validity against real-world data. In
this section we provide some additional fidelity checks for the entire
simulation, which combines the analytical solutions with simulation software
libraries as described above.

We start with simple demonstrations in Figure 7. First, keeping all setpoints
fixed, we vary the number of chillers. As expected, turning on more chillers
decreases the system’s temperature and turning off chillers have the opposite
effect (see Figure 7(a)). Second, we vary the number of cooling towers. Again,
as expected, turning on more cooling towers decreases the system’s temperature
and other way around - turning off cooling towers increases the temperature
(see Figure 7(b)). As a slightly different example, we fix the chiller leaving
temperature setpoint and increase the dry bulb temperature, which represents
the external weather conditions. As expected, we observe (Figure 7(c)) an
increase in the compressor power required to maintain the chiller temperature.
After leaving the air handling unit (AHU), the water gets progressively warmer
due to the greater heat generated from the building (which is modeled as the
increase in the dry bulb temperature). We also see the intermediate
transitions in the temperature as the PID controllers try to control the
chiller power. After a rapid transition due to the increase in dry bulb
temperature, the PIDs react by reducing the power.

(a) Changing the number of chillers.

(b) Changing the number of cooling towers.

(c) Increasing the dry bulb temperature.

Figure 7: High level simulation fidelity check. Increasing the number of
chillers (a) or (b) number of cooling towers decreases the overall temperature
in the system. Similarly, in (c) an increase in the external temperature,
increases the chiller power consumption.

Next, we demonstrate that our simulation exhibits a more complex "emergent"
behavior. Since chillers are usually the most energy-intensive components of
any cooling facility, it is important to efficiently control how many chillers
are used at any given time. The domain experts know that depending on external
conditions, there are regimes when it is more energy-efficient to use two
chillers even though a single chiller can achieve the desired temperature. To
show that our model encodes this information, we plot the total power used by
all chillers as a function of dry bulb temperature. For each value of the dry
bulb temperature we show the total power used by different number of chillers
(see Figure 8(a)). This needs to be compared to the real world data. To do so,
we display the total power used by all chillers as a function of load (in our
simulated environment dry-bulb temperature is a proxy for the load). For each
value of the load we show the total power used by different number of
chillers. For low values, one chiller always consumes the lowest amount of
energy. However, for higher values two chillers are more energy efficient. We
observe the quantitative match between the simulated and observed transitions
from real world data.

(a) Simulated data.

(b) Real data.

Figure 8: Total power consumed by all chillers as a function of dry-bulb
temperature (load). We see that for low dry bulb temperatures (load) it is
always more energy-efficient to use one chiller. Two chillers become more
efficient for high values of the dry-bulb temperature (load).

##  3 Reinforcement Learning Applications

To demonstrate how our industrial cooling system model (Section 2) can be used
for reinforcement learning applications, in this section we introduce the
Industrial Task Suite, define several tasks for the model, and use these tasks
to evaluate the performance of different RL algorithms.

###  3.1 Industrial Task Suite

We begin with the Industrial Task Suite, a flexible framework for defining
industrial control problems within a reinforcement learning context. It allows
to create environments and tasks to train and evaluate artificial controllers
safely and efficiently, subject to various real world conditions, such as
sensor imperfections and white noise. ITS is in-line with the design of the
DeepMind Control Suite [36] and Real-World Reinforcement Learning (RWRL)
Challenge Framework [15], while providing more granular configurability of
environments. Balancing ease-of-use and customization, the task suite only
requires a simulation (model) and a task to define a RL problem (Figure 9),
and yet offers a highly parameterized facility configuration, which provides
industrial equipment components with customizable capacities, amounts and
connection topology. The task suite also supports specific scenarios such as
sensor noise/drift, weather variations, and varying initial conditions to
mimic specific real-world operating regimes. A diversity of such "frozen"
scenarios could serve as a test bed for evaluating performance and
benchmarking against existing baselines.

Figure 9: The Industrial Task Suite design diagram. The Environment class
orchestrates the main components: simulator, noise generator, task, constraint
evaluator and agent. It takes a configuration and uses it to initialize a
simulation loaded from the simulator database. It then starts relaying the
traffic between the agent and the simulator.

The core Environment class encapsulates and orchestrates the following
components: simulator, noise generator, task, constraint evaluator and agent
(see Figure 9). It takes a configuration from the database and uses it to
initialize a simulation. Then Environment applies one or several implemented
scenarios and defines the constraints. It uses a pre-defined task to compute
rewards and expand the observations space with additional information. It then
starts relaying the traffic between the agent and the simulation. While doing
so, it injects noise into the exchanged data, evaluates the constraints and
calculates the rewards.

Environment converts the inbound actions sent by an agent into simulation
control signals, and subsequently convert the outbound simulation measurements
into observations. Given a broad range of possible control/measurement specs,
the converters pad the values to a defined maximum range and append masks to
the observation to inform the agent about the differences in the
configuration. In order to provide access to all of the available information,
the facility configuration and the spec definition is also added to the
observation dictionary. Should that information be obsolete for a particular
agent architecture, it can be filtered out in an environment wrapper.

Noise is added to the control signal and to the observation respectively. The
noise simulates sensor/pipeline imperfections, and in effect diversifies the
distribution of simulated states (control noise) and observations (observation
noise). At each episode step, in addition to sending the control signal,
Environment also changes the values of the selected configuration parameters
according to pre-defined scenarios. This allows simulating the changing
environment conditions and other environmental instabilities.

A more detailed overview of ITS’s main components along with some
implementation details are given in Appendix A.

###  3.2 Tasks

The goal of an AI-powered system that operates a cooling facility is to
minimize the energy consumption while meeting a set of constraints. For
industrial cooling systems these constraints are defined to ensure that
buildings are comfortable for the people inside and to prevent the equipment
breakage. To efficiently train RL agents within the ITS framework, we consider
a curriculum learning based approach [9] (for a recent review see [34]).
Namely, we define various tasks categorized by their perceived complexity and
teach agents to solve them sequentially. Eventually, we will have an agent
that knows how to solve all the tasks and operate the cooling facility.

We start with action (Table 1) and observation (Table 2) space defined for our
simulator (Section 2). All values are selected based on a typical industrial
cooling facility configuration.

Action | Unit | Type | Default | Minimum | Maximum
---|---|---|---|---|---
# Chillers Enabled | - | Integer | 1 | 0 | 3
# Chilled Water Pumps | - | Integer | 1 | 1 | 3
# Condenser Water Pumps | - | Integer | 1 | 1 | 3
Chiller Leaving Temperature | deg F | Continuous | 48 | 40 | 75
Cooling Tower Return Water Temperature | deg F | Continuous | 55 | 32 | 90
Condenser Water Pump Flow rate | kg/s | Continuous | 50 | 10 | 200
Distributional Differential Pressure | psi | Continuous | 15 | 0.1 | 50
# Free cooling Heat Exchangers | - | Integer | 1 | 1 | 3

Table 1: Action space used in our experiments. Observation | Type | Unit
---|---|---
Building load | Continuous | kW
Dry bulb temperature | Continuous | deg F
Wet bulb temperature | Continuous | deg F
Relative humidity | Continuous | -
Condenser leaving temperature (per chiller) | Continuous | deg F
Chilled water flow rate (per chiller) | Continuous | kg/s
Compressor power (per chiller) | Continuous | kW
Cooling tower fan power | Continuous | kW
Condenser water pump bank power | Continuous | kW
Chilled water pump bank power | Continuous | kW
Chiller bank leaving temperature | Continuous | deg F
Chilled water supply temperature | Continuous | deg F
Chilled water return temperature | Continuous | deg F
# of chillers enabled | Integer | -
Table 2: Observation space used in our experiments.

To mimic the industrial setting, the constraints are split into soft and hard.
A hard constraint violation terminates the episode (the agent gets kicked out
of the environment), which incentivizes the agent to respect the constraint.
Soft constraints can be added to the reward function via additional terms (for
more see Section 3.3). Additionally, any constraint violation is communicated
to the agent as an observation. For more about constraints see Section A.4.

Next, we list several tasks sorted by difficulty and provide (if known)
corresponding baseline and the optimal policies.

##### Easy

Unconstrained number of chillers control Controls: number of chillers (1D
integer). Baseline policy: use 0 chillers. Optimal policy: use 0 chillers.
This is a very simple task, where we do not impose any constraints and not
worry about any temperatures in the system. Ultimately, controlling the number
of chillers is the most important aspect of a cooling facility control,
because chillers are the most power-consuming components.

Constrained number of chillers control Controls: number of chillers (1D
integer). Hard constraint: use only a subset of available chillers, for
example, if there are 3 available chillers, the agent is allowed to use only 1
or 2 chillers. Baseline policy: use 1 chiller. Optimal policy: use 1 chiller.
This is a slightly more complicated chiller control task, where we still do
not worry about the temperatures, but impose the constraint on the number of
chillers. Via this task we explore whether the agent can learn to satisfy hard
constraints.

Chiller temperature control Controls: chiller temperature (1D float). Baseline
policy: use the highest possible chiller temperature. Optimal policy: use the
highest possible chiller temperature. In a typical cooling facility, along
with the number of chillers, one can control individual chiller temperatures.
With this task we investigate whether the agent can learn to effectively
control them.

##### Medium

Constrained number of chillers control with an additional constraint on the
chilled water supply temperature Controls: number of chillers (1D integer).
Hard constraint: use only a subset of available chillers and constraint on the
range of chilled water supply temperature. Scenario: Randomized dry-bulb
temperature. Baseline policy: use 1 chiller. Optimal policy: depending on the
external conditions use 1 or 2 chillers. This task is closer to real facility
requirements; the agent controls the number of chillers while keeping the
chilled water supply temperature within a given range, which directly
translates to the overall temperature in a building and its occupants’
comfort.

Number of chillers and condenser temperature control with an additional
constraint on the chilled water supply temperature Controls: number of
chillers (1D integer) and condenser temperature (1D float). Hard constraint:
constraint on the range of chilled water supply temperature. In this task, in
addition to the number of chiller control, the agent controls the condenser
temperature while keeping the chilled water supply temperature within a given
range.

##### Hard

Full cooling system control Controls: All actions defined in Table 1. Hard
constraint: constraint on the range of chilled water supply temperature. In
this task the agent controls all available cooling facility setpoints while
keeping the chilled water supply temperature within a given range.

##### Other tasks

For a fixed simulator the new tasks can be constructed by using one or a
combination of several possible ways:

  * •

Selecting a task objective (for example, minimize the power consumption).

  * •

Adding a scenario (for example, changing weather).

  * •

Selecting a subset of controls (for example, only control the number of
chillers).

  * •

Adding noise to controls.

  * •

Adding noise to measurements.

  * •

Imposing constraints.

###  3.3 Reward Function

As discussed in Section 3.2, we split constraints into soft and hard, where a
hard constraint violation terminates the episode, and soft constraints are
added to a reward function via additional terms. Moreover, any constraint
violation is communicated to the agent as an observation. The base reward
function (without soft constrains) should satisfy the following requirements:

  1. 1.

Inverse relationship between power and reward (more power - less reward).

  2. 2.

Maximum reward = 1 (when the power is 0, which is the best case scenario, the
reward is 1).

  3. 3.

Reward function is sensitive to a change in the relevant range of power (this
depends on the cooling system configuration).

Figure 10: Base reward function for power optimization tasks - Eq. (17). The
normalization constant, $\alpha$, allows to make the reward function sensitive
to the power change in the relevant range of values. In this example, when
$\alpha=1$ the reward function has nearly the same value for the entire
10-3000 kWatt range. $\alpha=1000$ provides a good gradual reward change from
10 to 3000 kWatts.

The following simple function satisfies all the requirements:

$r=\frac{1}{W/\alpha+1}.$ (17)

Here $W$ is the power consumed by the system and $\alpha$ is a normalization
constant. $1$ in the denominator ensures that $r(0)=1$ and the normalization
constant ($\alpha$) makes the function sensitive to the relevant range of
power. For example, for industrial chillers $\alpha\sim 1000$ in the SI units
(kWatt).

###  3.4 Experiments and Results

We now investigate whether the common deep RL algorithms can be used for the
industrial cooling facility control by benchmarking these algorithms on some
of the tasks defined in Section 3.2. We consider DMPO [3, 7], DDPG [28], D4PG
[8], and use their ACME implementations [22]. For some of the tasks we know
the optimal policy and will see if the RL agents are able to learn it.

For all agents we employ the continuous action space normalized to [-1, 1]. If
controls are integer-valued we quantize the agent’s continuous actions333This
is done for simplicity since the agents we consider are designed for
continuous actions.. Each agent was run 3 times per task using different
seeds. For all tasks we use episodes of length 10. Figure 11 shows performance
on individual tasks, where the lines denote the mean return and the shaded
regions denote standard deviation across seeds. For easy tasks we use 10
actors per agent and for the medium task - 100. We used the same
hyperparameters across all tasks (i.e. so that nothing is tuned per-task). Due
to a slow simulator, we had to use mini-batches of size 4 across all tasks,
and frequently update the actor networks - every 10 actor steps.

The summary of results is presented on Figure 11. For our experiments we
consider tasks for controlling the number of chillers and the chiller
temperature. As motivated in Section 3.2, these are the most important
setpoints for an effective cooling facility control. For the number of
chillers control, we start with a very simple binary configuration: the agent
can select 0 or 1 chiller. All agents learn this trivial task very fast (in
$\sim$100 training steps). Next, we use the configuration with 4 states: the
agent can choose between 0, 1, 2, and 3 chillers. And again all agents learned
this task easily, although DMPO took a bit longer while also experiencing a
few spikes in the mean return across different seeds. For the chiller
temperature control, DDPG converges to the optimal policy very fast, while
D4PG converges to a sub-optimal policy across all seeds. DMPO does learn the
optimal policy, but again it converges slower than DDPG. Finally, we consider
a medium difficulty task: the agent controls the number of chillers (0, 1, 2,
3 chillers are available), but if it selects 0 or 3 chillers the episode
terminates; there is also additional constraint on the chiller water supply
temperature, which directly maps to the overall building temperature. In this
case DMPO is the only agent to learn the optimal policy. Overall, we see that
DMPO performs the best, as it learns the optimal policies for all four tasks.
Next, we provide more details on the architectures for each agent.

(a) Easy tasks.

(b) Medium task.

Figure 11: Comparison of DMPO, DDPG and D4PG on the tasks defined for our
simulated cooling system environment. We consider three easy and one medium
tasks defined in Section 3.2. Easy tasks include: unconstrained chiller
control (a binary 0-1 chillers and non-binary 0-3 chillers) and a chiller
temperature control. For the medium task we use the constrained number of
chiller with an additional constraint on the chilled water temperature. Note
that for each task we run an experiment with three seeds and report the mean
return and uncertainty (shaded area). Interestingly, DMPO is the only agent
that learned the optimal policy for all the tasks (for optimal policies and
max returns see Section 3.2).

##### DMPO

Both policy and critic networks are LayerNorm MLPs - they are MLPs with ELU
non-linearities preceded by a linear layer followed by a LayerNorm and $\tanh$
activation: $Linear\to LayerNorm\to tanh\to Linear\to ELU\to Linear\to
ELU\to\cdots$. The policy network had three layers of [256, 256, 256] units
respectively. Actions are sampled from a multivariate Gaussian with diagonal
covariance, parameterized by initial and minimum scale parameters for which we
use the default values of 0.3 and $10^{-6}$ respectively. The critic network
had three layers of [512, 512, 256] units, the output of the last layer is
concatenated with the actions and passed into the discrete valued head that
returns the categorical distribution, using 51 categories spaced evenly across
[-15, 15].

##### DDPG

Just like in case of DMPO both policy and critic networks are LayerNorm MLPs
where we use the same hyperparameters. The policy network is the LayerNorm MLP
with [256, 256, 256] units. The critic network concatenates observations and
actions and passes them into the LayerNorm MLP with [512, 512, 256] units,
which go into another Linear Layer with output dimension 1.

##### D4PG

For D4PG we use the critic network from DDPG and the policy networks from DMPO
with the same hyperparameters.

##  4 Limitations

Some limitations of our model and the Industrial Task Suite include:

  * •

Simulation Speed: The simulation software solves complex ordinary differential
equations at every step. Even with our hybrid analytical model, this results
in a magnitude of $\sim 10$ seconds/step with each step simulating a real
world 5 minute time horizon. With the current setup, this doesn’t match the
high frequency iterations of existing reinforcement learning environments. A
next step would involve optimizing the environment setup and the solver for
faster simulated transitions.

  * •

Simulation Fidelity The simulator fidelity checks described in this paper are
not exhaustive; more work is required to ensure its accuracy.

  * •

Sim2Real Validation: We have designed and implemented the Industrial Task
Suite to enable domain randomization, a technique that randomizes the
simulator parameters to expose the agent to a diverse set of environments in
the training phase. Additionally, it provides access to a wide diversity of
real world scenarios. However, it remains a future work to develop a Sim2real
adapter to integrate this with a real world facility.

The modular Industrial Task Suite, as well as the hybrid simulation model
design, have been described in detail. While this should be descriptive to
provide a high level understanding of the contribution, releasing the codebase
remains out of scope for this work.

##  5 Conclusions

Designing and evaluating accurate whole-facility industrial cooling models is
a challenging optimization problem. In this paper, we presented one possible
approach by combining analytical solutions with a multi-physics simulation
software. We showed that it empirically matches key properties of real-world
facility behavior.

We extended and demonstrated the usability of this model by providing the
design for an Industrial Task Suite, which allows custom RL problem
configurations using our model or any other physical control simulator. We
benchmarked RL agents using this task suite and identified DMPO to be a well-
performing candidate for the problem of cooling system control, as it found
the optimal policies for all four given tasks. Providing this model and
framework, we hope to make applying RL to the real-world problem of optimizing
industrial controls more accessible.

Appendix

##  Appendix A Industrial Task Suite Components

In this section we will briefly describe all components of the task suite
defined in Section 3.1 and provide some implementation details.

###  A.1 Environment

Environment extends dm_env.Environment, an abstract base class for RL
environments defined in The DeepMind RL Environment API [31]. Its main job is
to comprise and orchestrate all components (see Figure 9).

⬇

class Environment(dm_env.Environment):

def __init__(self,

simulator: SimulatorInterface,

task: TaskInterface,

evaluator: Evaluator,

initial_conditions: Mapping[str, Any],

initial_conditions_noise: Mapping[str,float],

controls: Collection[str],

controls_noise: Mapping[str, float],

measurements: Collection[str],

measurements_noise: Mapping[str, float]

...)

self._simulator = simulator

self._task = task

self._evaluator = evaluator

...

self._ic = initial_conditions

self._ic_noise = initial_conditions_noise

...

self._controls_noise = controls_noise

...

@property

def action_spec(self):

@property

def observation_spec(self):

def reset(self):

# Add noise to initial conditions.

noisy_cfg = {}

for ic_id, ic_value in self._ic.items():

noisy_cfg[ic_id] = ic_value + self._ic_noise[ic_id]

# Reset the simulator with the new noisy initial conditions

timestep = self._simulator.reset(noisy_cfg)

# Add noise to measurements and evaluate constraints

core_obs, _ = process_timestep(timestep, self._evaluator)

# Reset task.

task_obs = self._task.reset(noisy_cfg, core_obs)

observation = dict(**core_obs, **task_obs)

return dm_env.restart(observation)

def step(self, action):

# Add noise to controls (actions)

for action_id in action:

action[action_id] += self._controls_noise[action_id]

timestep = self._simulator.step(action)

# Add noise to measurements and evaluate constraints

core_obs, hard_constraint_violation = process_timestep(timestep,
self._evaluator)

# Compute reward and extend observation

reward, task_obs = self._task.calculate_reward_and_observations(

core_obs)

observation = dict(**core_obs, **task_obs)

if timestep.is_last or hard_constraint_violation:

return dm_env.termination(reward, observation)

else:

return dm_env.transition(reward, observation)

###  A.2 Tasks

Tasks are designed to exercise orthogonal areas of competence, but at the same
time work with any configuration sampled from the defined distribution. In
addition to that, the shared component framework allows testing out certain
policies on the entire range of tasks. From the implementation point of view
tasks define and compute rewards as well as provide additional observations to
the agent.

⬇

class TaskInterface:

@property

def observation_spec(self):

def reset(self, configuration):

def calculate_reward_and_observations(observation):

###  A.3 Simulator

A simulator is the ground-truth model of the system dynamics. It is configured
at the beginning of each new episode. That is the moment when a new
configuration will be sampled. This means that at each new episode, a new
simulation might be selected as well. Once the simulator is configured it
starts taking controls from the agent at every agent step and returns the
results in the form of measurements, which are a subset of the simulation
state.

⬇

class SimulatorInterface:

@property

def measurement_spec(self):

def reset(self, configuration):

def step(self, controls):

###  A.4 Constraints

Constraints are defined in a configuration file. In the industrial setting the
agent gets kicked out of the system, when it violates some constraints, to
mimic this we divide constraints into hard and soft. Violation of a hard
constraint automatically ends an episode (equivalent to the agent being kicked
out of the system). Violation of any of these constraints is communicated to
the agent as an observation. Constraints are evaluated on every step - after
receiving an agent’s action, and subsequently after receiving an updated
simulation measurements. The constraint evaluator is used in both cases to
validate them.

Figure 12: Evaluation of the constraint ranges for a single value.

Hard constraints can be expressed by a single range of values. Soft
constraints define ranges bounded by the hard constraint upper and lower
limits. This can be expressed with a custom structure that defines four values
for each of the limits.

⬇

class Constraint:

simulation_parameter_name: str

hard_upper: float

soft_upper: float

soft_lower: float

hard_lower: float

The values should meet the following criteria hard_lower <= soft_lower <
soft_upper <= hard_upper.

###  A.5 Scenarios

Scenarios add another configuration dimension to the environment. Each
scenario describes a trajectory that modifies a set of simulation
configuration parameters. Along with noise, scenarios make environment more
realistic. Below we list a set of scenarios each representing a challenge we
observed in the real world industrial systems.

  * •

Frozen sensors: The scenario freezes the values of selected sensors for a
random amount of time. A dedicated implementation of the measurement noise
handles this. Its purpose is to test an agent’s resilience to missing
information.

  * •

Sensor drift: The scenario introduces a temporally correlated noise into the
selected measurement components. The components are selected at random at the
beginning of each episode. Its purpose is to test an agent’s resilience to
partially false information.

  * •

Frozen controls: The scenario freezes the values of selected controls for a
random amount of time. A dedicated implementation of the control noise handles
this.Its purpose is to test an agent’s ability to detect when a selected
policy fails and adapt by switching to an alternative.

  * •

Simulation dynamics non-stationarity: The scenario uses a set of configuration
trajectories to modify the selected simulation configuration parameters over
the course of an episode. Configuration trajectories produce changes to
selected parameters, which the task adds to their baseline values and
subsequently passes to the simulation. Control tensor is used for that
purpose. Absolute configuration values are used so that limits can be
observed. The limits are stored in the configuration definition. Its purpose
is to test an agent’s resilience to ever changing environmental conditions,
for example, weather change and other variables outside the domain of agent’s
control.

##  Appendix B Other Industrial Task Suite Applications

The modularity of the industrial task suite and the tools developed around the
cooling system simulation model allow the easy implementation of other control
problems of interest, which might have very different physics and task
definitions.

###  B.1 Brushless DC motor

Direct control electrical machines is a long standing research problem for
industrial control systems, and using reinforcement learning based control
approaches offers potential to improve on the state-of-the-art performance of
existing controllers.

In order to explore research into this area, we have implemented a simulation
model for a Brushless DC (BLDC) electrical motor using COMSOL. The simulation
model is a 2D axial cross-section of a 3-phase electrical motor, with a
magnetized rotary core (see Figure 13). The agent observes the angular
position and velocity of the rotor, and controls the currents being applied to
each of the three phase coils. The simulation also allows us to visualize
(Figure 13-2) the magnetic field intensity of the simulated motor.

Figure 13: BLDC motor simulation environment captured as stills from video
[5]. The agent sets actions in the form of three phase currents (top left), to
control the motor. (Bottom Center) Target position is shown with a red arrow
and the current position is shown with a green arrow.

We have successfully trained a DMPO agent [7, 3] to control the motor in a
generic trajectory tracking task. A random target position trajectory is
created and sent to the agent; and a reward is calculated based on the
difference between the actual and target positions, with reward = 1 when they
are equal. The agent also uses a secondary objective to reduce the size of
actions ("phase currents"), to minimize power.

###  B.2 Airfoil

Another example domain with complex physics is fluid flow. Flight control
systems aim to achieve lift and stability while countering non-linear and
time-varying physics [24]. This domain provides a challenging test domain for
our agents and our methods for providing and soft/hard constraints to achieve
stability.

We have implemented an airfoil under flow to control the altitude. The agent
observes the current altitude, a target altitude and current pitch of the
airfoil in addition to the lift force generated, and controls the rate of
change of the pitch. We have implemented a DMPO agent to learn a control
policy that achieves a target altitude. We have successfully demonstrated this
policy in the following video [4].

## References

  * [1] Britannica, t. editors of encyclopaedia (2016, february 16). cooling system. encyclopedia britannica.  https://www.britannica.com/technology/cooling-system.
  * [2] Deepmind ai reduces energy used for cooling google data centers by 40  https://blog.google/outreach-initiatives/environment/deepmind-ai-reduces-energy-used-for/.  Accessed: 2022-07-26.
  * [3] Distributional maximum a posteriori policy optimization (dmpo).  https://github.com/deepmind/acme/tree/master/acme/agents/tf/dmpo.  Accessed: 2022-07-11.
  * [4] Dmpo agent learns to control a airfoil under fluid flow.  https://youtu.be/fGRDnwYYJnw.  Accessed: 2022-07-12.
  * [5] Dmpo agent learns to control a bldc motor to achieve "constant speed" task.  https://youtu.be/dR2cT8cRTlw.  Accessed: 2022-07-12.
  * [6] Industrial sector energy consumption.  https://www.eia.gov/outlooks/ieo/pdf/industrial.pdf.
  * [7] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller.  Maximum a posteriori policy optimisation.  arXiv preprint arXiv:1806.06920, 2018.
  * [8] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap.  Distributed distributional deterministic policy gradients.  arXiv preprint arXiv:1804.08617, 2018.
  * [9] Y. Bengio, J. Louradour, R. Collobert, and J. Weston.  Curriculum learning.  In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
  * [10] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba.  Openai gym.  arXiv preprint arXiv:1606.01540, 2016.
  * [11] Y. Cao, L. Huang, Z. Cui, and J. Liu.  The optimal operation of cooling tower systems with variable-frequency control.  In IOP Conference Series: Earth and Environmental Science, volume 113, page 012085. IOP Publishing, 2018.
  * [12] B. Chen, Z. Cai, and M. Bergés.  Gnu-rl: A precocial reinforcement learning solution for building hvac control using a differentiable mpc policy.  In Proceedings of the 6th ACM international conference on systems for energy-efficient buildings, cities, and transportation, pages 316–325, 2019.
  * [13] Y. Chen, Y. Shi, and B. Zhang.  Optimal control via neural networks: A convex approach.  arXiv preprint arXiv:1805.11835, 2018.
  * [14] D. B. Crawley, L. K. Lawrie, F. C. Winkelmann, W. F. Buhl, Y. J. Huang, C. O. Pedersen, R. K. Strand, R. J. Liesen, D. E. Fisher, M. J. Witte, et al.  Energyplus: creating a new-generation building energy simulation program.  Energy and buildings, 33(4):319–331, 2001.
  * [15] G. Dulac-Arnold, D. Mankowitz, and T. Hester.  Challenges of real-world reinforcement learning.  arXiv preprint arXiv:1904.12901, 2019.
  * [16] B. Foliaco, A. Bula, and P. Coombes.  Improving the gordon-ng model and analyzing thermodynamic parameters to evaluate performance in a water-cooled centrifugal chiller.  Energies, 13(9):2135, 2020.
  * [17] G. Gao, J. Li, and Y. Wen.  Energy-efficient thermal comfort control in smart buildings via deep reinforcement learning.  arXiv preprint arXiv:1901.04693, 2019.
  * [18] J. Gordon, K. C. Ng, and H. T. Chua.  Centrifugal chillers: thermodynamic modelling and a diagnostic case study.  International Journal of refrigeration, 18(4):253–257, 1995.
  * [19] M. A. Hanif, F. Nadeem, R. Tariq, and U. Rashid.  Renewable and Alternative Energy Resources.  Academic Press, 2022.
  * [20] V. Hanumaiah and S. Genc.  Distributed multi-agent deep reinforcement learning framework for whole-building hvac control.  arXiv preprint arXiv:2110.13450, 2021.
  * [21] E. G. Hertwich, S. Ali, L. Ciacci, T. Fishman, N. Heeren, E. Masanet, F. N. Asghari, E. Olivetti, S. Pauliuk, Q. Tu, et al.  Material efficiency strategies to reducing greenhouse gas emissions associated with buildings, vehicles, and electronics—a review.  Environmental Research Letters, 14(4):043004, 2019.
  * [22] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas.  Acme: A research framework for distributed reinforcement learning.  arXiv preprint arXiv:2006.00979, 2020.
  * [23] H. Kazmi, F. Mehmood, S. Lodeweyckx, and J. Driesen.  Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems.  Energy, 144:159–168, feb 2018.
  * [24] B. S. Kim and A. J. Calise.  Nonlinear flight control using neural networks.  Journal of Guidance, Control, and Dynamics, 20(1):26–33, 1997.
  * [25] D.-W. Kim and C.-S. Park.  Difficulties and limitations in performance simulation of a double skin façade with energyplus.  Energy and Buildings, 43(12):3635–3645, 2011.
  * [26] F. Krausmann, C. Lauk, W. Haas, and D. Wiedenhofer.  From resource extraction to outflows of wastes and emissions: The socioeconomic metabolism of the global economy, 1900–2015.  Global Environmental Change, 52:131–140, 2018.
  * [27] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. Jordan, and I. Stoica.  Rllib: Abstractions for distributed reinforcement learning.  In International Conference on Machine Learning, pages 3053–3062. PMLR, 2018.
  * [28] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [29] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  * [30] T. Moriyama, G. D. Magistris, M. Tatsubori, T.-H. Pham, A. Munawar, and R. Tachibana.  Reinforcement learning testbed for power-consumption optimization.  In Asian simulation conference, pages 45–59. Springer, 2018.
  * [31] A. Muldal, Y. Doron, J. Aslanides, T. Harley, T. Ward, and S. Liu.  dm_env: A python interface for reinforcement learning environments, 2019\.
  * [32] S.-H. Mun, J. Kang, Y. Kwak, Y.-S. Jeong, S.-M. Lee, and J.-H. Huh.  Limitations of energyplus in analyzing energy performance of semi-transparent photovoltaic modules.  Case Studies in Thermal Engineering, 22:100765, 2020.
  * [33] Z. Nagy and K. Nweye.  Real-world challenges for reinforcement learning in building control.  arXiv preprint arXiv:2112.06127, 2021.
  * [34] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone.  Curriculum learning for reinforcement learning domains: A framework and survey.  arXiv preprint arXiv:2003.04960, 2020.
  * [35] H. O. Portner, D. C. Roberts, H. Adams, C. Adler, P. Aldunce, E. Ali, R. A. Begum, R. Betts, R. B. Kerr, R. Biesbroek, et al.  Climate change 2022: impacts, adaptation and vulnerability.  2022\.
  * [36] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al.  Deepmind control suite.  arXiv preprint arXiv:1801.00690, 2018.
  * [37] A. Tortorelli, M. Imran, F. Delli Priscoli, and F. Liberati.  A parallel deep reinforcement learning framework for controlling industrial assembly lines.  Electronics, 11(4):539, 2022.
  * [38] J. Wang, J. F. Rodrigues, M. Hu, P. Behrens, and A. Tukker.  The evolution of chinese industrial co2 emissions 2000–2050: A review and meta-analysis of historical drivers, projections and policy goals.  Renewable and Sustainable Energy Reviews, 116:109433, 2019.
  * [39] X. Zhan, H. Xu, Y. Zhang, Y. Huo, X. Zhu, H. Yin, and Y. Zheng.  Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning.  arXiv preprint arXiv:2102.11492, 2, 2021.
  * [40] Z. Zhang, A. Chong, Y. Pan, C. Zhang, and K. P. Lam.  Whole building energy model for hvac optimal control: A practical framework based on deep reinforcement learning.  Energy and Buildings, 199:472–490, 2019.