# Enforcing Policy Feasibility Constraints through Differentiable Projection
for Energy Optimization

Bingqing Chen  1234-5678-9012 Carnegie Mellon University5000 Forbes
AvenuePittsburghPAUSA<EMAIL_ADDRESS>,  Priya Donti  1234-5678-9012
Carnegie Mellon University5000 Forbes AvenuePittsburghPAUSA<EMAIL_ADDRESS>,  Kyri Baker  1234-5678-9012 University of Colorado, BoulderBoulderCOUSA
<EMAIL_ADDRESS>,  J. Zico Kolter  1234-5678-9012 Carnegie Mellon
University5000 Forbes AvenuePittsburghPAUSA<EMAIL_ADDRESS>and  Mario
Bergés  0003-2948-9236 Carnegie Mellon University5000 Forbes
AvenuePittsburghPAUSA<EMAIL_ADDRESS>
(2021)

###### Abstract.

While reinforcement learning (RL) is gaining popularity in energy systems
control, its real-world applications are limited due to the fact that the
actions from learned policies may not satisfy functional requirements or be
feasible for the underlying physical system. In this work, we propose
PROjected Feasibility (PROF), a method to enforce convex operational
constraints within neural policies. Specifically, we incorporate a
differentiable projection layer within a neural network-based policy to
enforce that all learned actions are feasible. We then update the policy end-
to-end by propagating gradients through this differentiable projection layer,
making the policy cognizant of the operational constraints. We demonstrate our
method on two applications: energy-efficient building operation and inverter
control. In the building operation setting, we show that PROF maintains
thermal comfort requirements while improving energy efficiency by 4% over
state-of-the-art methods. In the inverter control setting, PROF perfectly
satisfies voltage constraints on the IEEE 37-bus feeder system, as it learns
to curtail as little renewable energy as possible within its safety set.

safe reinforcement learning, implicit layers, differentiable optimization,
inverter control, smart building

††copyright: acmcopyright††journalyear: 2021††copyright:
rightsretained††conference: The Twelfth ACM International Conference on Future
Energy Systems; June 28-July 2, 2021; Virtual Event, Italy††booktitle: The
Twelfth ACM International Conference on Future Energy Systems (e-Energy ’21),
June 28-July 2, 2021, Virtual Event, Italy††doi:
10.1145/3447555.3464874††isbn: 978-1-4503-8333-2/21/06††ccs: Hardware Smart
grid††ccs: Hardware Energy generation and storage††ccs: Computing
methodologies Reinforcement learning

##  1\. Introduction

There has been increasing interest in using learning-based methods such as
reinforcement learning (RL) for applications in energy systems control.
However, a fundamental challenge with many of these methods is that they do
not respect the physical constraints or functional requirements associated
with the systems in which they operate. Therefore, there have been many calls
for embedding safety guarantees into learning-based methods in the context of
energy systems applications (Zhang et al., 2019; Glavic, 2019; Dobbe et al.,
2020).

One common proposal to address this challenge is to provide machine learning
methods with “soft penalties” to encourage them to learn feasible solutions.
For instance, the authors of (Zhang and Lam, 2018; Chen et al., 2019)
incentivize their RL-based building HVAC controller to satisfy thermal comfort
constraints by adding a constraint violation penalty to the reward function.
While such approaches often involve tuning some weight on the penalty term,
recent work has proposed more theoretically-grounded approaches to choosing
these weights; for instance, in the setting of approximating AC optimal power
flow, the authors of (Fioretto et al., 2020; Chatzos et al., 2020) interpret
the weight on their constraint violation penalty as a dual variable, and learn
it via primal-dual updates. Gupta et al. (2020) adopt a similar approach in an
inverter control problem. However, a challenge with these types of “soft
penalty” methods in general is that while they _incentivize_ feasibility, they
do not strictly _enforce_ it, which is potentially untenable in safety-
critical applications.

Figure 1. The PROF framework. Our policy consists of a neural network followed
by a differentiable projection onto a convexified set of operational
constraints, $\hat{\mathcal{C}}_{k}$ (which is constructed via an approximate
model, $\hat{f}_{k}$, of the environment). The differentiable projection layer
enforces the constraints in the forward pass, and induces policy gradients
that make the neural network cognizant of the constraints in its learning.

Given this limitation, a second class of approaches has aimed to strictly
enforce operational constraints. For instance, in some cases, the outputs of a
machine learning algorithm can be clipped post-hoc in order to make them
feasible. However, a challenge is that such post-hoc corrections are not taken
into account during the learning process, potentially negatively impacting
overall performance. More recent approaches based in deep learning have
therefore aimed to enforce simple classes of constraints in a way that _can_
be taken into account during learning; for instance, Zamzam and Baker (2020)
train a neural network to approximate AC optimal power flow (OPF), and enforce
box constraints on certain variables via sigmoid activations in the last layer
of the neural network. In general, however, existing approaches have only been
able to accommodate _simple_ sets of constraints, prompting a need for methods
that can incorporate broader classes of constraints.

In this work, we propose a method to enforce general convex constraints into
RL-based controllers in a way that can be taken into account during the
learning process. In particular, we construct a neural network-based policy
that culminates in a _projection_ onto a set of constraints characterized by
the underlying system. While the “true” constraints associated with the system
may be somewhat complex, we observe that simple, approximate physical models
are often available for many systems of interest, allowing us to specify
convex approximations to the relevant constraints. The projections onto these
(approximate) sets can thus be characterized as convex optimization problems,
allowing us to leverage recent developments in differentiable convex
optimization (Amos and Kolter, 2017; Agrawal et al., 2019) to train our neural
network and projection _end-to-end_ using standard RL methods. The result is a
powerful neural policy that can flexibly optimize performance on the true
underlying dynamics, while still satisfying the specified constraints.

We demonstrate our PROjected Feasibility approach, PROF, on two settings of
interest. Specifically, we explore a building operation setting in which the
goal is to reduce energy consumption during the heating season, while ensuring
the satisfaction of thermal comfort constraints. We additionally explore an
inverter control setting where the goal is to mitigate curtailment, while
satisfying inverter operational constraints and nodal voltage bounds. In both
settings, we find that our controller achieves good performance with respect
to the control objective, while ensuring that relevant operational constraints
are satisfied.

To summarize, our key contributions are as follows:

  * •

A framework for incorporating convex constraints. We propose a projection-
based method to flexibly enforce convex constraints within neural policies (as
summarized in Figure 1). By examining the gradient fields of the
differentiable projection layer, we recommend the incorporation of an
auxiliary loss for more robust results. We also show in an ablation study
(Section 5.3) that propagating gradients through the differentiable projection
layer is indeed conducive to policy learning.

  * •

Demonstration on building control. In the building control setting, we show
that PROF further improves energy efficiency by 10% and 4%, respectively,
compared to the best-performing RL agents in (Zhang and Lam, 2018) and (Chen
et al., 2019). By using a locally-linear assumption to approximate the
building thermodynamics and thereby formulating the constraints as a polytope
(Zhao et al., 2017; Chen et al., 2020a), we largely maintain the temperature
within the deadband, except when the control is saturated.

  * •

Demonstration on inverter control. In the inverter control setting, PROF
satisfies the voltage constraints 100% of the time over more than half a
million time steps (1 week at one second per time step), with a randomly
initialized neural network, compared to 22% over-voltage violations incurred
by a Volt/Var control strategy. With respect to the objective of minimizing
renewable generation curtailment, PROF performs as well as possible within its
conservative safety set after learning safely for a day.

##  2\. Related Work

Our approach relies on recent developments in implicit neural network layers,
and is thematically similar to several recent works in safe RL. We briefly
discuss these topics, and refer interested readers to (Dobbe et al., 2020;
Zhang et al., 2019; Glavic, 2019; Rolnick et al., 2019; Drgoňa et al., 2020)
for comprehensive reviews of relevant work in power and energy systems
application domains.

##### Implicit layers

A neural network can be viewed as a composition of functions, or _layers_ ,
with parameters that can be adjusted to improve performance on some task.
While many of the layers commonly used within neural networks (e.g.,
convolutions or sigmoid functions) represent _explicit_ functions that provide
a direct mapping between inputs and outputs, there has recently been a great
deal of interest in expanding the set of commonly-used layers to include those
representing _implicit_ functions (Kolter et al., 2020). This has included the
creation of layers capturing optimization problems (Amos and Kolter, 2017;
Djolonga and Krause, 2017; Tschiatschek et al., 2018; Wang et al., 2019;
Agrawal et al., 2019; Gould et al., 2019), physical equations (de Avila
Belbute-Peres et al., 2018; Chen et al., 2018; Greydanus et al., 2019),
sequence modeling processes (Bai et al., 2019), and games (Ling et al., 2018).
In this work, we leverage advances in differentiable optimization in
particular, namely by incorporating a differentiable convex optimization layer
into our neural policy in order to project proposed control actions onto the
feasible set of constraints.

##### Safe reinforcement learning

While (deep) RL methods in general lack safety or stability guarantees, there
has been recent interest in learning RL-based controllers that attempt to
maintain some notion of safety during training and/or inference – e.g., to
satisfy physical constraints or avoid particularly negative outcomes (Garcıa
and Fernández, 2015). These include methods that aim to determine “safe”
regions of the state space by making smoothness assumptions on the underlying
dynamics (Wachi et al., 2018; Berkenkamp et al., 2017; Turchetta et al., 2016;
Akametalu et al., 2014), methods that combine concepts from RL and control
theory (Morimoto and Doya, 2005; Luo et al., 2014; Pinto et al., 2017; Chang
et al., 2019; Han et al., 2019; Zhang et al., 2020; Donti et al., 2021),
approaches based on formal verification logics (Hunt et al., 2020; Hasanbeig
et al., 2020; Fulton and Platzer, 2019), and methods that aim to bound some
(discounted) cost function characterizing violations of safety constraints
(Yang et al., 2020; Taleghan and Dietterich, 2018; Achiam et al., 2017;
Altman, 1999). While the particular notion of “safety” considered varies
between settings, relevantly to the present work, several of these prior works
employ some form of differentiable projection within the loop of deep RL. For
instance, within the context of constrained Markov decision processes
(C-MDPs), Yang et al. (2020) project neural network-based policies onto a
linearly-constrained set of policies with bounded cumulative discounted cost.
In the context of asymptotic stability, Donti et al. (2021) project the
actions output by their controller onto a convex set of actions satisfying
stability specifications obtained via robust control. In the setting of
robotic motion planning, Pham et al. (2018) project actions onto a linear set
of robotic operational constraints, and apply separate updates to the neural
network based on both pre-projection and post-projection actions. Similarly to
this prior work, our approach employs differentiable projections within a
neural network policy to enforce operational constraints over some planning
horizon.

##  3\. Preliminaries

We now present background on technical concepts used by PROF, namely
reinforcement learning and differentiable projection layers.

###  3.1. Reinforcement Learning

The goal of RL is to learn an optimal control policy through direct
interaction with the environment. The problem is usually formulated as a
Markov decision process (MDP). At each time step $k$, the agent selects an
action $u_{k}$ given the current state $x_{k}$, using its policy
$\pi_{\theta}$ (Equation 1). In many modern RL techniques, the policy is
commonly represented by a neural network parameterized by $\theta$. When the
agent takes the action $u$, the state transitions to $x^{\prime}$ based on the
system dynamics $f$ (Equation 2), and the agent receives a reward $r_{k}$ (or
equivalently, incurs a cost $c_{k}=-r_{k}$).

(1) $\displaystyle u$ $\displaystyle\sim\pi_{\theta}(u_{k}|x_{k}),$ (2)
$\displaystyle x^{\prime}$ $\displaystyle\sim f(x_{k},u_{k}).$

RL algorithms optimize for a policy that maximizes the expected cumulative
reward, or equivalently, minimizes the expected cumulative cost, where
$\gamma$ is a temporal discount factor:

(3)
$\theta^{\star}=\arg\max_{\theta}{\mathbb{E}_{\pi_{\theta}}}\left[\sum_{l=0}^{\infty}\gamma^{l}r_{k+l}\right]=\arg\min_{\theta}{\mathbb{E}_{\pi_{\theta}}}\left[\sum_{l=0}^{\infty}\gamma^{l}c_{k+l}\right].$

To simplify notation, we will denote the expected cumulative cost as
$J(\theta)$, i.e.,

(4)
$J(\theta)=\mathbb{E}_{\pi_{\theta}}\left[\sum_{l=0}^{\infty}\gamma^{l}c_{k+l}\right].$

There are three general approaches to RL, namely value-based methods, policy
gradient methods, and actor-critic methods. Value-based methods, e.g.,
Q-learning and its variants, update the value function of state-action pairs
using the Bellman equation and take the action that maximizes the value of an
action selection policy (the Q function) through exploration. Policy gradient
methods, e.g., Proximal Policy Optimization (PPO) (Schulman et al., 2017),
directly search for an optimal policy $\pi_{\theta}^{\star}$ using estimates
of policy gradients. Denoting the policy gradient as
$g:=\nabla_{\theta}J(\theta)$, the core idea of policy gradient algorithms is
that they update $\theta$ based on an estimate, $\hat{g}$, of the gradient,
i.e.,

(5) $\theta\xleftarrow{}\theta-\alpha\hat{g}$

for some learning rate $\alpha$. Different algorithms vary in how they obtain
$\hat{g}$. For instance, the learning objective for PPO, which we use in our
building control experiment (Section 5), is given by the following equation,
where $\hat{A}_{t}$ is the generalized advantage estimate that can be
estimated via any of the estimators in (Schulman et al., 2015):

(6) $\displaystyle J_{\text{PPO}}(\theta)$
$\displaystyle=\hat{\mathbb{E}}_{k}\left[\min(w_{k}(\theta)\hat{A}_{k},\text{clip}(w_{k}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{k})\right],$
$\displaystyle w_{k}(\theta)$
$\displaystyle=\frac{\pi_{\theta}(u_{k}|x_{k})}{\pi_{\theta_{old}}(u_{k}|x_{k})},$

and the estimate $\hat{g}$ is constructed based on this learning objective.
Actor-critic methods, e.g., Advantage Actor-Critic (A2C), are hybrids of the
value-based and policy gradient approaches, using a policy network to select
actions (the actor) and a value network to evaluate the action (the critic).

###  3.2. Differentiable Projection Layers

As previously described, a neural network is a composition of parameterized
functions (_layers_) whose parameters are adjusted during training via
_backpropagation_ (a class of gradient-based methods). Any function can be
incorporated into a neural network as a layer provided that it satisfies two
main conditions. The first condition is that it must have a _forward
procedure_ to map from inputs to outputs (i.e., do _inference_). The second is
that it must have a _backwards procedure_ to compute gradients of the outputs
with respect to the inputs and function parameters, in order to enable
backpropagation.

With that in mind, consider the $L_{2}$-norm projection
$\mathcal{P}_{\mathcal{C}}:\mathbb{R}^{n}\to\mathcal{C}$ that maps from some
point in $\hat{u}\in\mathbb{R}^{n}$ to its closest point in some constraint
set $\mathcal{C}\subseteq\mathbb{R}^{n}$ as follows:

(7)
$\mathcal{P}_{\mathcal{C}}(\hat{u})=\operatorname*{argmin}_{u\in\mathcal{C}}\frac{1}{2}\|u-\hat{u}\|_{2}^{2}.$

In cases where $\mathcal{C}$ is convex, Equation 7 is a convex optimization
problem. The forward procedure of this operation can then be implemented by
simply solving the optimization problem, e.g., using standard convex
optimization solvers. Perhaps less evidently, it is also possible to construct
a backwards procedure for this problem by using the implicit function theorem
(Krantz and Parks, 2012), as described in previous work (e.g., (Amos and
Kolter, 2017; Agrawal et al., 2019)).

As an example, consider the case where $\mathcal{C}$ characterizes linear
constraints, i.e., $\mathcal{C}\equiv\\{u:Au=b,Gu\leq h\\}$ for some
$A\in\mathbb{R}^{n_{\text{eq}}\times n}$, $b\in\mathbb{R}^{n_{\text{eq}}}$,
$G\in\mathbb{R}^{n_{\text{ineq}}\times n}$, and
$h\in\mathbb{R}^{n_{\text{ineq}}}$. It is then possible to efficiently compute
gradients through Equation 7 by implicitly differentiating through its KKT
conditions, i.e., conditions that are necessary and sufficient to describe its
optimal solutions. In particular, as described in (Amos and Kolter, 2017), the
KKT conditions for stationarity, primal feasibility, and complementary
slackness for this case are given by

(8)
$\begin{split}u^{\star}-\hat{u}+A^{T}\nu^{\star}+G^{T}\lambda^{\star}&=0\\\
Au^{\star}-b&=0\\\
\mathrm{diag}(\lambda^{\star})(Gu^{\star}-h)&=0,\end{split}$

where $u^{\star}$, $\lambda^{\star}$, and $\nu^{\star}$ are the optimal primal
and dual solutions. By the implicit function theorem, we can then take
derivatives through these conditions at the optimum in order to obtain
relevant gradients. Specifically, the total differentials of these KKT
conditions are given by

(9)
$\begin{split}\mathsf{d}u-\mathsf{d}\hat{u}+\mathsf{d}A^{T}\nu^{\star}+A^{T}\mathsf{d}\nu+\mathsf{d}G^{T}\lambda^{\star}+G^{T}\mathsf{d}\lambda&=0\\\
\mathsf{d}Au^{\star}+A\mathsf{d}u-\mathsf{d}b&=0\\\
\mathrm{diag}(Gu^{\star}-h)\mathsf{d}\lambda+\mathrm{diag}(\lambda^{\star})(\mathsf{d}Gu^{\star}+G\mathsf{d}z-\mathsf{d}h)&=0.\end{split}$

As described in (Amos and Kolter, 2017), these equations can then be
rearranged to solve for the Jacobians of any of the solution variables
$u^{\star},\lambda^{\star},\nu^{\star}$ with respect to any of the problem
parameters $\hat{u},A,b,G,h$ (or, in practice, to solve directly for these
Jacobians’ left matrix-vector product with some backward pass vector, in order
to reduce space complexity).

While the above example is for the case of a linearly-constrained projection
operation, these kinds of gradients can be computed for convex projection
problems in general. For instance, Donti et al. (2021) compute gradients
through a projection onto a second order cone by differentiating through the
fixed point equations of their solver, and Agrawal et al. (2019) provide a
method and library for differentiable disciplined convex programs. A key
benefit of using these kinds of projection layers for constraint enforcement
is that they allow gradients through the enforcement procedure to flow back to
the neural network, thereby informing the parameter updates of this network
during training.

##  4\. Approach

We now describe PROF, which incorporates differentiable projections onto
convex(ified) sets of operational constraints within a neural policy.

###  4.1. Problem Formulation

Consider a discrete-time dynamical system

(10) $x_{k+1}=f(x_{k},u_{k},w_{k}),$

where $x_{k}\in\mathbb{R}^{s}$ is the state at time $k$,
$u_{k}\in\mathbb{R}^{a}$ is the control input, $w_{k}\in\mathbb{R}^{d}$ is an
uncontrollable disturbance (which we assume to be observable), and
$f:\mathbb{R}^{s}\times\mathbb{R}^{a}\times\mathbb{R}^{d}\to\mathbb{R}^{s}$
denotes the system dynamics. Letting $\mathcal{X}_{k}$ and $\mathcal{U}_{k}$
denote the allowable state and action space, respectively, we can define the
set of all feasible actions over the planning horizon $T$ as
$\mathcal{C}_{k}$, where

(11)
$\mathcal{C}_{k}=\left\\{u_{k:k+T-1}\;\Bigg{|}\begin{array}[]{l}x_{i+1}=f(x_{i},u_{i},w_{i}),\\\
\;x_{i}\in\mathcal{X}_{i},\;u_{i}\in\mathcal{U}_{i}\end{array}\;\;\forall
i\in\\{k,...,k+T-1\\}\right\\}.$

Our goal is then to learn a policy that optimizes the control objective, $J$,
while enforcing the operational constraints. To simplify notation, we denote
$\mathbf{u}=u_{k:k+T-1}$. In the case of a deterministic policy, i.e.,
$\mathbf{u}=\pi_{\theta}$, the learning problem is simply

(12)
$\displaystyle\min_{\theta}\;\;J(\theta)\quad\textrm{s.t.}\;\;\pi_{\theta}\in\mathcal{C}_{k}.$

In the case of a stochastic policy, e.g.
$\mathbf{u}\sim\mathcal{N}(\boldsymbol{\mu},\mathrm{diag}(\boldsymbol{\sigma}^{2})),\;[\boldsymbol{\mu},\boldsymbol{\sigma}]=\pi_{\theta}(x_{k})$,
we can write the problem as

(13)
$\displaystyle\min_{\theta}\;\;J(\theta)\quad\textrm{s.t.}\;\;\mathbf{u},\boldsymbol{\mu}\in\mathcal{C}_{k}.$

In this case, it is necessary to sample actions around $\boldsymbol{\mu}$ in
order to estimate policy gradients. At the same time the actions sampled from
$\pi_{\theta}$ might fall outside of $\mathcal{C}_{k}$. Thus, we enforce that
both $\boldsymbol{\mu}$ and the sample action $\mathbf{u}$ satisfy the
constraints.

###  4.2. Approximate Convex Constraints

In practice, there are two key challenges inherent in solving Equations 12–13
as written. The first is that the disturbances $w_{i}$ are not known ahead of
time, meaning that the optimization problem must be solved under uncertainty.
One approach to addressing this, from the field of robust control (Zhou and
Doyle, 1998), involves constructing an uncertainty set over the disturbance,
and then optimizing for worst-case or expected cost under this uncertainty
set. Here, we simply assume a predictive model of the disturbances is
available. (By re-planning frequently, we observe that the prediction errors
have limited empirical impact on performance in the two applications we
study.) We will use the notation $\hat{w}_{k}$ to denote our forecast of the
disturbance if $k$ is a future time step, and the true value of the
disturbance if $k$ is the present or a prior time step.

The second challenge pertains to the form of the set $\mathcal{C}_{k}$, which
may be poorly structured or otherwise difficult to optimize over. In
particular, our framework relies on obtaining convex approximations to the
constraints in order to enable differentiable projections (see Section 3.2).
Fortunately, for many energy systems applications, some approximate model
$\hat{f}_{k}$ is often available based on domain knowledge that allows
$\mathcal{C}_{k}$ to be approximated as a convex set, despite the complex
nature of the true dynamical system.

Thus, letting $\hat{f}_{i}$ denote our approximations of the dynamics and
$\hat{w}_{i}$ denote the (forecast or known) disturbance at each
$i=k,\ldots,k+T-1$, we define our approximate convex constraint set as

(14)
$\hat{\mathcal{C}}_{k}=\left\\{u_{k:k+T-1}\;\Bigg{|}\begin{array}[]{l}x_{i+1}=\hat{f}_{i}(x_{i},u_{i},\hat{w}_{i}),\\\
\;x_{i}\in\mathcal{X}_{i},\;u_{i}\in\mathcal{U}_{i}\end{array}\;\;\forall
i\in\\{k,...,k+T-1\\}\right\\}.$

We note that $f$ and $w$ are approximated _solely_ for the purposes of
constructing approximate constraint sets, and are not used otherwise during
training and inference (i.e., our neural policy interacts with the _true_
dynamics and disturbances during training and inference).

Algorithm 1 PROF

1:procedure main(env, $J$) _// input: environment, control objective_

2: init neural network $\hat{\pi}_{\theta}$, replay memory $\mathcal{M}$

3: specify RL algorithm $\mathcal{A}$, batch size $M$, update interval $K$

4: specify planning horizon $T$

5:  _// online execution_

6: for $k=1,\ldots$ do

7: observe state $x_{k}$

8: predict future disturbances $\hat{w}_{k:k+T-1}$

9: construct constraint set $\hat{\mathcal{C}}_{k}$, policy
$\pi_{\theta}=\mathcal{P}_{\hat{\mathcal{C}}_{k}}\circ\hat{\pi}_{\theta}$

10: compute $u_{k}$ = inference(${\pi}_{\theta}$, $x_{k}$, $T$)

11: execute action env.step($u_{k}$)

12: save memory.append($x_{k}$, $u_{k}$, $\hat{w}_{k:k+T-1}$)

13:  _// update policy every $K$ time steps_

14: if $\bmod(k,K)=0$ then

15: $\hat{\pi}_{\theta}$ = train($\hat{\pi}_{\theta}$, $J$, $\mathcal{M}$,
$\mathcal{A}$)

16: end if

17: end for

18:end procedure

19:

20:procedure inference($\pi_{\theta}$, $x_{k}$, $T$)

21:  _// input: neural policy, current state, planning horizon_

22: select action $u_{k:k+T-1}\sim\pi_{\theta}$ _// only return the current
action; replan at each time step_

23: return $u_{k}$

24:end procedure

25:

26:procedure train($\hat{\pi}_{\theta}$, $J$, $\mathcal{M}$, $\mathcal{A}$)

27:  _// input: neural policy, objective, replay memory, RL algorithm_

28: init $\mathcal{L}(\theta)$ = 0

29: for $i=1,\ldots,M$ do

30: sample $x,u,w\sim\mathcal{M}$

31: construct constraint set $\hat{\mathcal{C}}_{k}$, policy
$\pi_{\theta}=\mathcal{P}_{\hat{\mathcal{C}}_{k}}\circ\hat{\pi}_{\theta}$

32: compute training loss
$\mathcal{L}(\theta)\mathbin{{+}{=}}J(\theta)+\lambda\|\pi_{\theta}(x)-\hat{\pi}_{\theta}(x)\|_{2}^{2}$

33: end for

34: train $\hat{\pi}_{\theta}$ via $\mathcal{A}$ to minimize $\mathcal{L}$

35: return $\hat{\pi}_{\theta}$

36:end procedure

###  4.3. Policy Optimization

Let $\hat{\pi}_{\theta}$ be any (e.g., fully-connected or recurrent) neural
network parameterized by $\theta$. Our policy entails passing the output from
the neural network to the differentiable projection layer
$\mathcal{P}_{\hat{\mathcal{C}}_{k}}$ characterized by the approximate
constraints, which enforces that the resultant action is feasible with respect
to these constraints. The overall (differentiable) neural policy is then given
by

(15)
$\pi_{\theta}(x_{k})=\mathcal{P}_{\hat{\mathcal{C}_{k}}}\circ\hat{\pi}_{\theta}(x_{k}).$

The key benefit of embedding a differentiable projection into our policy is
that it enforces constraints in a way that is visible to the neural network
during learning. In this work, we implement the differentiable projection
using the cvxpylayers library (Agrawal et al., 2019).

We construct the following loss function, which is a weighted sum of the
control objective $J$ and an auxiliary loss term to be explained shortly in
this section. $\lambda>0$ is a hyperparameter.

(16)
$\mathcal{L}(\theta,x_{k})=J(\theta)+\lambda\|\pi_{\theta}(x_{k})-\hat{\pi}_{\theta}(x_{k})\|_{2}^{2}.$

We then train our policy (Equation 15) to minimize this cost using standard
approaches in deep reinforcement learning. The full algorithm is presented in
Algorithm 1.

Figure 2. Illustrative example of gradients from the differentiable projection
layer. $u^{\bullet}$ and $u^{\star}$ denote unique optimal actions minimizing
some convex control objective $J$ in the unconstrained and constrained
settings, respectively; $\nabla_{\hat{\pi}}\|\pi-u^{\bullet}\|_{2}^{2}$ is
thus a proxy for $\nabla_{\hat{\pi}}J$. (a) $u^{\bullet}\not\in\mathcal{C}$.
The gradients $\nabla_{\hat{\pi}}J$ point towards $u^{\star}$ as desired, such
that $\pi=\mathcal{P}_{\hat{\mathcal{C}}}\circ\hat{\pi}$ will reach this
optimal point. (b) $u^{\bullet}=u^{\star}$ on the interior of $\mathcal{C}$.
The gradients $\nabla_{\hat{\pi}}J$ do not cause $\hat{\pi}$ (or its
projection) to update towards the interior. Adding a weighted auxiliary loss
term, e.g., $\|\pi-\hat{\pi}\|$, can help direct updates towards the interior.

####  4.3.1. Visualization of gradient fields.

To provide more intuition on the differentiable projection layer and our cost
function, we visualize the gradient fields in a hypothetical example with a
deterministic policy and a planning horizon of $T=1$. Specifically, for the
purposes of illustration, let $u^{\bullet}$ and $u^{\star}$ denote unique
optimal actions minimizing some convex control cost $J$ in the unconstrained
and constrained settings, respectively:

$\displaystyle u^{\bullet}\sim\pi_{\theta^{\bullet}};$
$\displaystyle\theta^{\bullet}=\arg\min_{\theta}\;\;J(\theta)$ $\displaystyle
u^{\star}\sim\pi_{\theta^{\star}};$
$\displaystyle\theta^{\star}=\arg\min_{\theta}\;\;J(\theta)\quad\textrm{s.t.}\;\;u\in\mathcal{C}_{k}.$

In Figure 2, we then plot the gradient fields in two cases: (a)
$u^{\bullet}\not\in\mathcal{C}_{k}$, and (b) $u^{\bullet}\in\mathcal{C}_{k}$.
Note that $u^{\bullet}$ and $u^{\star}$ are assumed to be known here for
illustrative purposes only, and are not known during training.

In particular, we plot the gradients (black arrows) of
$\|u^{\bullet}-\mathcal{P}_{\mathcal{C}_{k}}\circ\hat{\pi}\|_{2}^{2}$ with
respect to the output of the neural network $\hat{\pi}$. These indicate the
direction in which the neural network would be incentivized to update in order
to minimize the system cost. If no differentiable projection were embedded
within the policy, all the gradients would point towards $u^{\bullet}$ without
regard for the constraints. Instead, in the case of
$u^{\bullet}\not\in\mathcal{C}_{k}$ (Figure 2a), the gradients through the
differentiable projection layer point towards $u^{\star}$ instead of
$u^{\bullet}$. More specifically, if
$\hat{\pi}_{\theta}(x_{k})\in\mathcal{C}_{k}$, then the projection layer is
simply the identity, and the gradients point directly towards $u^{\star}$;
otherwise, the gradients point along the boundary of $\mathcal{C}_{k}$ in the
direction of $u^{\star}$.

This case is of particular interest, as in many practical applications some
operational constraint will be binding. As a concrete example, the ultimate
energy-saving strategy for building operations is to keep all mechanical
systems off (i.e., $u^{\bullet}=0$), which obviously violates occupants’
comfort requirements and is outside the set of allowable actions (i.e.,
$u^{\bullet}\not\in\mathcal{C}_{k}$). Thus, the problem is to find a policy
that uses the mechanical system as little as possible without violating
comfort requirements. Given the common case where the control objective is
convex, this then lies on the boundary of the constraint set (i.e.,
$u^{\star}=\mathcal{P}_{\mathcal{C}_{k}}\circ u^{\bullet}$).

We also depict the case where the solution of the unconstrained problem
already satisfies the constraints, i.e., $u^{\bullet}=u^{\star}\in C_{k}$
(Figure 2b). If this is generally the case for a particular application, we
note that a constraint enforcement approach (ours or otherwise) is likely not
needed, and indeed utilizing gradients through the projection layer may
actually degrade performance. Specifically, if
$\hat{\pi}_{\theta}(x_{k})\not\in\mathcal{C}_{k}$, the gradients do not point
towards the interior of the constraint set, meaning that
$\pi_{\theta}(x_{k})=\mathcal{P}_{\mathcal{C}_{k}}\circ\hat{\pi}_{\theta}(x_{k})$
will lie on the boundary of the constraints despite the optimal solution being
in the interior. This can be amended by augmenting the loss function with a
(weighted) auxiliary term such as
$\|\pi_{\theta}(x_{k})-\hat{\pi}_{\theta}(x_{k})\|_{2}^{2}$ whose gradients
(blue arrows) point towards the interior.

It may not be known a priori whether or not $u^{\bullet}$ is in the constraint
set in general or at any given time, except when domain experts are fully
clear on the structure of the solutions for specific applications. In
particular, $\mathcal{C}_{k}$ is time-varying, making it difficult to know for
sure whether or not the constraints will indeed be binding at any given time.
For robustness, we therefore recommend incorporating the auxiliary loss
$\|\pi_{\theta}(x_{k})-\hat{\pi}_{\theta}(x_{k})\|_{2}^{2}$ within the RL
training cost, unless it is known from domain knowledge that the constraints
will certainly be active. As such, we formulate the training cost function as
previously given in Equation 16.

##  5\. Experiment 1: Energy-efficient Building Operation

There is significant potential to save energy through more efficient building
operation. Buildings account for about 40% of the total energy consumption in
the United States, and it is estimated that up to 30% of that energy usage may
be reduced through advanced sensing and control strategies (Fernandez et al.,
2017). However, this potential is largely untapped, as the heterogeneous
nature of building environments limits the ability of control strategies
developed for one building to scale to others (Chen et al., 2019). RL can
address this challenge by adapting to individual buildings by directly
interacting with the environment.

The most important constraint in building operation is to maintain a
satisfactory level of comfort for occupants, while minimizing energy
consumption. It is common in the RL-based building control literature to
penalize thermal comfort violations (Zhang and Lam, 2018; Chen et al., 2019),
which incentivizes but does not guarantee the satisfaction of these comfort
requirements. In comparison, our proposed neural policy can largely maintain
temperature within the specified comfortable range, except when the control is
saturated.

We evaluate our policy in the same simulation testbed as (Zhang and Lam, 2018;
Chen et al., 2019), following the same experimental setup as (Chen et al.,
2019). Specifically, we first pre-train the neural policy by imitating a
proportional-controller (P-controller). We then evaluate and further train our
agent in the simulation environment, using a different sequence of weather
data.

###  5.1. Problem Description

##### Simulation testbed

We utilize an EnergyPlus (E+) model of a 600m2 multi-functional space (Figure
3(a)), based on the Intelligent Workplace (IW) on Carnegie Mellon University
(CMU) campus, located in Pittsburgh, PA, USA. The system of interest is the
water-based radiant heating system, of which a schematic is provided in Figure
3(b). In this experiment, we control the supply water temperature so as to
maintain the state variable, i.e., the zone temperature, within a comfortable
range during the heating season. In the existing control, the supply water
(SW) is maintained at a constant flow rate, and its temperature is managed by
a P-controller. For more information on the simulation testbed, refer to
(Zhang and Lam, 2018).

##### Approximate system model

We approximate the environment as a linear system as follows:

(17) $x_{k+1}\approx\hat{f}(x_{k},u_{k},w_{k})=Ax_{k}+B_{u}u_{k}+B_{d}w_{k},$

where $x_{k}$ represents the zone temperature and $u_{k}$ represents the
supply water temperature. $w_{k}$ includes distributions from weather and
occupancy. While building thermodynamics are fundamentally nonlinear, the
locally-linear assumption works well for many control inputs (Privara et al.,
2013). We identify the approximate model parameters $A$, $B_{u}$, and $B_{d}$
with prediction error minimization (Privara et al., 2013) on the same data
used to pre-train the RL agent (see Section 5.2). The root mean squared error
(RMSE) of this model on a unseen test set is 0.14oC.

##### Objective

Since our goal is to minimize energy consumption, we define the control cost
at each time step as the agent’s control action, i.e. supply water
temperature, which is linearly proportional to the heating demand, i.e.,
$c_{k}=u_{k}.$

In contrast to the objectives in (Zhang and Lam, 2018; Chen et al., 2019),
which are defined as weighted sum of energy cost and some penalty on thermal
comfort violations, we consider the thermal comfort requirement as hard
constraints, in the form of Equation 13.

##### Constraints

To maintain a satisfactory comfort level, we require the zone temperature to
be within a deadband $\mathcal{X}=\\{x\;|\;21.9\textsuperscript{o}C\leq x\leq
25.5\textsuperscript{o}C\\}$ when the building is occupied, based on the
building code requirement of 10% Predicted Percentage of Dissatisfied (PPD)
(Fanger, 1986). We allow for a wider temperature range during unoccupied
hours. For the action, the allowable range of supply water temperature for the
physical system is $\mathcal{U}=\\{u\;|\;20\textsuperscript{o}C\leq u\leq
65\textsuperscript{o}C\\}$.

While it may appear from this description that we have only simple box
constraints on both the state and action, we highlight the fact that actions
are coupled over time through the building thermodynamics (Zhao et al., 2017).
More concretely, a future state depends on all past actions. Thus, a box
constraint on $x_{k+l+1}$ is in fact a constraint on $u_{k:k+l}$. In this
case, assuming $\hat{f}$ to be a linear system, $\hat{\mathcal{C}}_{k}$ is
then a set of linear inequalities, which can be geometrically interpreted as a
polytope.222A polytope can be characterized as a set
$\mathcal{S}=\\{x\in\mathbb{R}^{n}|Ax\leq b\\}$. We refer interested readers
to (Chen et al., 2020a; Zhao et al., 2017) for more details on this
formulation. In fact, it was experimentally demonstrated in (Chen et al.,
2020a) that projecting actions onto the polytope constructed with an
approximate linear model was sufficient to maintain temperature within the
deadband in a real-world residential household (though (Chen et al., 2020a)
did not then differentiate through this projection).

##### Control time step

The EnergyPlus model has a 5-minute simulation time step. Following (Zhang and
Lam, 2018; Chen et al., 2019), we use a 15-min control time step (i.e., each
action is repeated 3 times) and a planning horizon of $T=12$ (i.e., a 3 hour
look-ahead).

(a) Geometric view

(b) System schematic

Figure 3. Building simulation testbed (reproduced from (Chen et al., 2019)).

###  5.2. Implementation Details

##### Offline pre-training

We pre-train a long short-term memory (LSTM) recurrent policy (without a
subsequent projection) by imitating a P-controller operating under the Typical
Meteorological Year 3 (TMY3) (Wilcox and Marion, 2008) weather sequence, from
Jan. 1 to Mar. 31. We min-max normalize all of the state, action, and
disturbance, and use a learning rate of $10^{-3}$. Specifically, we use the
pre-trained weights after training on the expert demonstrations for 20 epochs
following the same procedures as (Chen et al., 2020b). We refer readers to
(Chen et al., 2020b) for more details on the neural network architecture,
training procedures, loss, and performance evaluation.

##### Online policy learning

We optimize the policy with PPO (Schulman et al., 2017) over the weather
sequence in 2017 from Jan. 1 to Mar. 31. We use $\lambda=10$ (see Equation
16), a learning rate of $5\times 10^{-4}$, and RMSprop (Tieleman and Hinton,
2012) as the optimizer333The code is available at
https://github.com/INFERLab/PROF.. We update the policy every four days, by
iterating over those samples for 8 epochs with a batch size of 32. For
hyperparameters, we use a temporal discount rate of $\gamma$ = 0.9, $\epsilon$
= 0.2 (see Equation 6), and a Gaussian policy (see Equation 13) with $\sigma$
linearly decreased from 0.1 to 0.01.

###  5.3. Results

(a) The differentiable projection layer enforces preheating behavior to ensure
deadband constraints are never violated, even though this behavior is not
present in the expert demonstrations.

(b) The agent has found a more energy-efficient control strategy by
maintaining temperature at the lower end of the deadband.

Figure 4. Behavior of our proposed agent (a) at the onset of deployment, with
pre-trained weights based on expert demonstrations and (b) after a month of
interacting with and training on the environment.

After pre-training on expert demonstrations from the baseline P-controller,
our agent directly operated the simulation testbed based on actual weather
sequences in Pittsburgh from Jan. 1 to Mar. 31 in 2017. Figure 4(a) shows the
behavior of our agent at the onset of deployment over a 3-day period. The
baseline P-controller reactively turns on heating when the environment
switches from unoccupied to occupied, which results in thermal comfort
violations in the mornings. In comparison, PROF preheats the environment such
that the environment is already at a comfortable temperature when occupants
arrive in the morning. Notably, the differentiable projection layer manages to
enforce this preheating behavior despite this behavior not being present in
the expert demonstrations.

Figure 4(b) shows the behavior of our agent in comparison with Gnu-RL (Chen et
al., 2019), having interacted with and trained on the environment for a month.
Gnu-RL is updated via PPO, similarly to the current work, and incorporates
domain knowledge on system dynamics. In comparison to Gnu-RL (Chen et al.,
2019), which ends up trying to maintain temperature at the setpoint, PROF
learns an energy-saving behavior by maintaining the temperature at the lower
end of the deadband. This explains the further energy savings compared with
Gnu-RL (Chen et al., 2019). However, we also notice that the temperature
requirement may be violated on cold mornings. This happens when the control
action is saturated, i.e., full heating over the 3-hour planning horizon is
not sufficient to bring temperature back to the comfortable range. (In
principle, even these constraint violations could be mitigated by increasing
the length of the planning horizon.)

Table 1 summarizes the performance of our agent with comparison to the RL
agents in (Zhang and Lam, 2018; Chen et al., 2019). Our proposed agent
(averaged over 5 random seeds) saves 10% and 4% energy compared to the best-
performing agents in (Zhang and Lam, 2018) and (Chen et al., 2019),
respectively.

Table 1. Performance comparison. Our method saves energy while incurring minimal comfort violations.  | Heating | PPD
---|---|---
| Demand | Mean | SD
| (kW) | (%) | (%)
Existing P-Controller (Zhang and Lam, 2018) | 43709 | 9.45 | 5.59
Agent #6 (Zhang and Lam, 2018) | 37131 | 11.71 | 3.76
Baseline P-Controller (Chen et al., 2019) | 35792 | 9.71 | 6.87
Gnu-RL (Chen et al., 2019) | 34687 | 9.56 | 6.39
LSTM & Clip + No Update | 37938 | 8.55 | 3.39
LSTM & Clip | 36068 $\pm$ 2187 | 9.18 $\pm$ 0.67 | 3.49
PROF (ours) |  33271 $\pm$ 1862 | 9.68 $\pm$ 0.48 | 3.66

We also compare our method to two ablations: (1) LSTM & Clip + No Update,
which uses the same pre-trained weights and the projection layer to enforce
feasible actions, but does not update the policy, and (2) LSTM & Clip, which
uses the same pre-trained weights and the projection layer to enforce feasible
actions during inference, but does not propagate gradients through the
differentiable projection layer in the policy updates. We find that LSTM &
Clip slightly improves upon LSTM & Clip + No Update, but is less performant
compared to PROF. This affirms our hypothesis that the gradients through the
differentiable projection layer are cognizant of the constraints and are thus
conducive to policy learning.

##  6\. Experiment 2: Inverter Control

Distributed energy resources (DERs), e.g., solar photovoltaic (PV) panels and
energy storage, are becoming increasingly prevalent in an effort to curb
carbon dioxide emissions and combat climate change. However, DERs interfacing
with the power grid via power electronics, such as inverters, also introduce
unintended challenges for grid operators. For instance, over-voltages have
become a common occurrence in areas with high renewable penetration (Stringer
et al., 2020), and power electronics-interfaced generation has low-inertia and
requires active control at much faster timescales compared to traditional
synchronous machines (Milano et al., 2018).

To alleviate these issues, IEEE standard 1547.8-2018 (Basso, 2014) recommends
a Volt/Var control strategy in which the reactive power contribution of an
inverter is based on local voltage measurements. As will be clear in our
empirical evaluation, this network-agnostic heuristic based on local
information alleviates, but does not avoid, over-voltage issues. Given that
the optimal solution needs to be obtained at the system-level and that the
problem needs to be solved at very short timescales, a common paradigm is to
address the problem in a quasi-static fashion (Jalali et al., 2019) adopted in
works such as (Baker et al., 2017; Jalali et al., 2019; Gupta et al., 2020),
where one chooses a policy over the next time period, e.g., 15 minutes-1 hour,
and uses the policy without update for fast inference. In this work, we adopt
the same paradigm and consider real-time control on a 1-second timescale of
both active (P) and reactive (Q) power setpoints at each inverter.

We envision that a neural policy can learn from its prior experiences, in
contrast to the traditional fit-and-forget approach (Dobbe et al., 2020), and
is capable of making decisions faster compared to solving optimization
problems. Our primary contribution compared to existing work is the ability to
enforce physical constraints within the neural network. In fact, we
successfully enforce voltage constraints 100% of the time with a randomly
initialized neural network, over more than half a million time-steps (i.e., 1
week with a one-second time step). The assumed control and communication
scheme is consistent with the new definitions for smart inverter capabilities
under IEEE standard 1547.1-2020 (IEEE, 2020).

###  6.1. Problem Description

The problem we are considering here is to control active and reactive power
setpoints at each inverter in order to maximize utilization (i.e., minimize
curtailment) of renewable generation, while satisfying the maximum and minimum
grid voltage requirements. Here, we first define the considered test case and
input data, and describe the model of the network. We refer readers to (Baker
et al., 2017) for more details on the problem set-up.

##### IEEE 37-bus test case

We evaluate our method on the IEEE 37-bus distribution test system (IEEE,
[n.d.]), with 21 solar PV systems indicated by green rectangles in Figure 5.
We utilize a balanced, single-phase equivalent of the system, and simulate the
nonlinear AC power flows using PYPOWER (R. D. Zimmerman, C. E. Murillo-
Sanchez, and R. J. Thomas, 2011). For the simulation, the solar generation and
loads are based on 1-second solar irradiance and load data collected from a
neighborhood in Rancho Cordova, CA (Bank and Hambrick, 2013) over a period of
one week (604800 samples).

Figure 5. IEEE 37-bus feeder system, where the solar PV systems are indicated
by green rectangles.

##### Approximate system model.

Denote the number of buses, excluding the slack bus (e.g., the distribution
substation), as $N$, the net active and the reactive power as
$\mathbf{p}\in\mathbb{R}^{N}$ and $\mathbf{q}\in\mathbb{R}^{N}$, and the
voltage at all buses as $\mathbf{v}\in\mathbb{R}^{N}$. We linearize the AC
power flow equations around the flat voltage solution, i.e.
$\mathbf{\bar{v}}=\mathbbm{1}$, using the method in (Bolognani and Dörfler,
2015). The reference active and reactive power corresponding to
$\mathbf{\bar{v}}=\mathbbm{1}$ is denoted as $\mathbf{\bar{p}}$ and
$\mathbf{\bar{q}}$. The linearized grid model, $\hat{f}$, is given by Equation
18, where $\mathbf{R},\;\mathbf{B}\in\mathbb{R}^{N\times N}$ represent system-
dependent network parameters that can be either estimated from linearization
(e.g., (Bolognani and Dörfler, 2015)) or data-driven methods:

(18) $\displaystyle\mathbf{v}\approx\hat{f}(\mathbf{p},\mathbf{q})$
$\displaystyle=\mathbf{\bar{v}}+\mathbf{R}(\mathbf{p-\bar{p}})+\mathbf{B}(\mathbf{q-\bar{q}})$
$\displaystyle=\mathbf{\bar{v}}+\underbrace{[\mathbf{R},\mathbf{B}]}_{\mathbf{H}}\underbrace{\begin{bmatrix}\mathbf{p-\bar{p}},\\\
\mathbf{q-\bar{q}}\end{bmatrix}}_{\mathbf{u}}.$

A notable advantage of the method in (Bolognani and Dörfler, 2015) is that the
resulting model has bounded error with respect to the true dynamics. By
incorporating the error bound when constructing the safety set, the safety set
is guaranteed to be a conservative under-approximation of the true safety set,
and thus allow us to satisfy voltage constraints 100% of the time.

##### Policy

Our policy takes as input the voltage from the previous time-step, load, and
generation at all the buses, and outputs active and reactive power setpoints
at each inverter. (This is a deterministic policy; see Equation 12.) Note that
while the grid model (Equation 18) contains all $N$ buses, only those with
inverters are controllable.

Our neural architecture is similar to the one used in (Gupta et al., 2020),
which consists of a utility-level network, and inverter-level networks for
individual inverters. The utility-level network collects information from all
nodes, and broadcasts an intermediate representation to all inverter-level
networks. Using this information along with its local observations, each
inverter makes its local control decisions, which are then projected onto the
constraints (discussed below).

Figure 6. PROF satisfies voltage constraints throughout the experiment, and
learns to minimize curtailment as well as possible within its conservative
safety set, $\hat{C}_{k}$, after learning safely for a day.

##### Objective

The objective is to minimize the curtailment of solar generation, or
equivalently to maximize the utilization of the available solar power,
$p_{av}$. Specifically, letting $\mathcal{I}$ denote the set of buses with
inverters, the objective is

(19)
$J(\theta)=\min_{\mathbf{p}_{\mathcal{I}},\mathbf{q}_{\mathcal{I}}}\sum_{i\in\mathcal{I}}[p_{av,i}-p_{i}]_{+},\quad\text{where}\begin{bmatrix}\mathbf{p}_{\mathcal{I}}&\mathbf{q}_{\mathcal{I}}\end{bmatrix}=\pi_{\theta}$

##### Constraints

For an individual inverter, $i$, with rated power $s_{i}$ and an available
power (from available solar generation) $p_{av,i}$, the feasible action space
is

$\mathcal{U}_{i}(k)=\\{(p_{i},q_{i}):0\leq p_{i}\leq
p_{av,i}(k),p^{2}_{i}+q^{2}_{i}\leq s^{2}_{i}\\}$
$\mathcal{U}(k):=\mathcal{U}_{1}(k)\times\dots\times\mathcal{U}_{|\mathcal{I}|}(k).$

At the same time, the voltage at each bus should remain between
$0.95$-$1.05~{}p.u.$ The primary challenge of satisfying voltage constraints
is that the voltage at each bus depends on actions of neighboring nodes, i.e.

$\mathcal{X}=\\{v\;|\;0.95\times\mathbbm{1}\leq\mathbf{v}\approx\mathbf{\bar{v}}+\mathbf{H}\mathbf{u}\leq
1.05\times\mathbbm{1}\\},$

where the sparsity pattern of $H$ is characterized by the admittance matrix.
We jointly project actions from all inverters at each time step $k$ onto the
constraints $\mathcal{U}(k)\cap\mathcal{X}$.

###  6.2. Implementation Details

We evaluate PROF by executing it over the 1-week dataset (at 1 second) once.
Similarly to other quasi-static approaches, we update the policy every
15-minutes. Similarly to (Gupta et al., 2020), we optimize the neural policy
with stochastic samples by directly differentiating through the objective
(Equation 19) and the linearized grid model (Equation 18). However our method
differs in that Gupta et al. (2020) characterized the constraints as a
regularization term, and learned the policy via primal-dual updates. We
incorporate the constraints directly via the differentiable projection layer
and thus guarantee constraint satisfaction.

We use $\lambda$=10 (see Equation 16), a learning rate of $10^{-3}$, and
RMSprop (Tieleman and Hinton, 2012) as the optimizer. At every 15 minutes, we
sample 16 batches of data with size of 64 from the replay memory. We keep a
replay memory size of 86400, i.e., samples from the previous day. For the both
the utility-level network and the inverter-level network, we use fully-
connected layers with ReLU activations. The utility-level network has hidden
layer sizes (256, 128, 64), and each inverter-level network has hidden layer
sizes (16, 4) and outputs active and reactive power. On top of the neural
network, we implement the differentiable projection layer, following the
constraints described in Section 6.1.

We compare our methods to three baselines, (1) a Volt/Var strategy following
IEEE 1547.8 (Basso, 2014), (2) the optimal solution with respect to the
linearized grid model, and (3) the optimal solution with respect to the true
AC power flow equations.

###  6.3. Results

The performance of PROF in comparison to the three baselines is summarized in
Figure 6. For clarity, we only show the maximum voltage over all buses; under-
voltage is not a concern for this particular test case.

We see that the Volt/Var strategy violates voltage constraints 22.3% of time,
mostly around noon when the solar generation is high and there is a surplus of
energy. Since the Volt/Var baseline does not adjust active power, there is no
curtailment.

In comparison, PROF satisfies the voltage constraints throughout the
experiment, even with a randomly initialized neural policy. While PROF
performs poorly on the first morning, it quickly improves its policy. In fact,
the behavior of PROF is barely distinguishable from the optimal solution with
respect to the linearized grid model, after learning safely for a day. This
implies that PROF learned to control inverters as well as possible given its
approximate model, which constructs a conservative under-approximation of the
true safety set.

The optimal baseline with respect to the true AC power flow equations
unsurprisingly achieves the best performance with respect to minimizing
curtailment, as it can push the maximum voltage to the allowable limit in
order to maximally reduce the amount of curtailed energy. However, inverter
control is a task that requires near real-time inputs, and we find that
running this baseline can be prohibitively slow. Specifically, we evaluate the
computation time of different operations by averaging over 1000 randomly
sampled problems from our dataset on a personal laptop. For PROF, on average,
a forward pass in the neural network (excluding the projection layer) took 4.5
ms and the differentiable projection operation took 8.6 ms. The computation
cost of the differentiable projection could be further reduced by using
customized projection solvers such as the ones in (Amos and Kolter, 2017;
Donti et al., 2021) that avoid the “canonicalization” costs introduced by
general-purpose solvers such as the one we use (Agrawal et al., 2019). In
comparison, solving the optimization baseline with respect to the true AC
power flow equations took 1.02s on the same machine, which is even longer than
the 1s control time-step.

##  7\. Discussion and Conclusions

In this work, we have presented a method, PROF, for integrating convex
operational constraints into neural network policies for energy systems
applications. In particular, we propose a policy that entails passing the
output of a neural network to a differentiable projection layer, which
enforces a convex approximation of the operational constraints. These convex
constraint sets are obtained using approximate models of the system dynamics,
which can be fit using system data and/or constructed using domain knowledge.
We can then train the resultant neural policy via standard RL algorithms,
using an augmented cost function designed to effect desirable policy
gradients. The result is that our neural policy is cognizant of relevant
operational constraints during learning, enhancing overall performance.

We find in both the building energy optimization and inverter control settings
that PROF successfully enforces relevant constraints while improving
performance on the control objective. In particular, in the building thermal
control setting, we find that our approach achieves a 4% energy savings over
the state of the art while largely maintaining the temperature within the
deadband. In the inverter control setting, our method perfectly satisfies the
voltage constraints over more than half a million time steps, while learning
to minimize curtailment as much as possible within the safety set.

While these results demonstrate the promise of our method, a key limitation is
in its computational cost. In particular, computing a projection during every
forward pass of training and inference is decidedly more expensive than
running a “standard” neural network. A fruitful area for future work – both in
the context of our method, and in the context of research in differentiable
optimization layers as a whole – may be to improve the speed of such
differentiable projection layers. For instance, this might entail developing
special-purpose differentiable solvers (Amos and Kolter, 2017; Donti et al.,
2021) for optimization problems commonly encountered in energy systems
applications, developing approximate solvers that do not rely on obtaining
optimal solutions in order to compute reasonable gradients, or employing
cheaper projection schemes such as $\alpha$-projection (Shah et al., 2020)
where possible.

Additionally, the success of our method (and many other constraint enforcement
methods) depends fundamentally on the quality of the approximate model used to
characterize the constraint sets. In particular, this determines the extent to
which the resultant approximate constraint sets are a good representation of
the true operational constraints. While we were able to employ reasonably
high-quality approximation schemes in the context of this work, future work on
safely updating the models or the constraint sets directly (Fisac et al.,
2019) may greatly improve the quality of the solutions.

More generally, while our work highlights one approach to enforcing physical
constraints within learning-based methods, we believe this is only the start
of a broader conversation on closely integrating domain knowledge and control
constraints into learning-based methods. In particular, strictly enforcing
physical constraints will be paramount to the real-world success of these
methods in energy systems contexts, and we hope that our paper will serve to
spark further inquiry into this important line of work.

##  8\. Acknowledgments

This material is based, in part, on work supported by Carnegie Mellon
University’s College of Engineering Moonshot Award for Autonomous Technologies
for Livability and Sustainability (ATLAS). This work was also supported by the
U.S. Department of Energy Computational Science Graduate Fellowship (DE-
FG02-97ER25308), the Center for Climate and Energy Decision Making through a
cooperative agreement between the National Science Foundation and Carnegie
Mellon University (SES-00949710), the Computational Sustainability Network,
and the Bosch Center for AI. The work of K. Baker is supported by the National
Science Foundation CAREER award 2041835.

## References

  * (1)
  * Achiam et al. (2017) Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017\.  Constrained Policy Optimization. In _Proceedings of the 34th International Conference on Machine Learning_.
  * Agrawal et al. (2019) Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. 2019\.  Differentiable Convex Optimization Layers. In _Advances in Neural Information Processing Systems_. 9558–9570.
  * Akametalu et al. (2014) Anayo K. Akametalu, Shahab Kaynama, Jaime F. Fisac, Melanie Nicole Zeilinger, Jeremy H. Gillula, and Claire J. Tomlin. 2014.  Reachability-based safe learning with Gaussian processes. In _53rd IEEE Conference on Decision and Control, CDC 2014_.
  * Altman (1999) Eitan Altman. 1999\.  _Constrained Markov Decision Processes_. Vol. 7.  CRC Press.
  * Amos and Kolter (2017) Brandon Amos and J Zico Kolter. 2017.  OptNet: Differentiable Optimization as a Layer in Neural Networks. In _Proceedings of the 34th International Conference on Machine Learning_. 136–145.
  * Bai et al. (2019) Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2019\.  Deep equilibrium models.  _arXiv preprint arXiv:1909.01377_ (2019).
  * Baker et al. (2017) Kyri Baker, Andrey Bernstein, Emiliano Dall’Anese, and Changhong Zhao. 2017.  Network-cognizant voltage droop control for distribution grids.  _IEEE Transactions on Power Systems_ 33, 2 (2017), 2098–2108.
  * Bank and Hambrick (2013) J. Bank and J. Hambrick. 2013.  Development of a high resolution, real time, distribution-level metering system and associated visualization modeling, and data analysis functions.  (2013).  National Renewable Energy Laboratory, Tech. Rep. NREL/TP-5500-56610.
  * Basso (2014) T.S. Basso. 2014\.  In _IEEE 1547 and 2030 Standards for Distributed Energy Resources Interconnection and Interoperability with the Electricity Grid_. National Renewable Energy Laboratory.
  * Berkenkamp et al. (2017) Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. 2017.  Safe Model-based Reinforcement Learning with Stability Guarantees. In _Advances in Neural Information Processing Systems_.
  * Bolognani and Dörfler (2015) Saverio Bolognani and Florian Dörfler. 2015.  Fast power system analysis via implicit linearization of the power flow manifold. In _2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton)_. IEEE, 402–409.
  * Chang et al. (2019) Ya-Chien Chang, Nima Roohi, and Sicun Gao. 2019.  Neural Lyapunov Control. In _Advances in Neural Information Processing Systems_. 3245–3254.
  * Chatzos et al. (2020) Minas Chatzos, Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. 2020.  High-Fidelity Machine Learning Approximations of Large-Scale Optimal Power Flow.  _arXiv preprint arXiv:2006.16356_ (2020).
  * Chen et al. (2019) Bingqing Chen, Zicheng Cai, and Mario Bergés. 2019\.  Gnu-RL: A precocial reinforcement learning solution for building HVAC control using a Differentiable MPC policy. In _Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation_. 316–325.
  * Chen et al. (2020a) Bingqing Chen, Jonathan Francis, Marco Pritoni, Soummya Kar, and Mario Berg’es. 2020a.  COHORT: Coordination of Heterogeneous Thermostatically Controlled Loads for Demand Flexibility. In _Proceedings of the 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation_. 31–40.
  * Chen et al. (2020b) Bingqing Chen, Ming Jin, Zhe Wang, Tianzhen Hong, and Mario Bergés. 2020b.  Towards Off-policy Evaluation as a Prerequisite for Real-world Reinforcement Learning in Building Control. In _Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities_. 52–56.
  * Chen et al. (2018) Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018.  Neural ordinary differential equations. In _Advances in neural information processing systems_. 6571–6583.
  * de Avila Belbute-Peres et al. (2018) Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. 2018\.  End-to-end differentiable physics for learning and control. In _Advances in Neural Information Processing Systems_. 7178–7189.
  * Djolonga and Krause (2017) Josip Djolonga and Andreas Krause. 2017.  Differentiable Learning of Submodular Models. In _Advances in Neural Information Processing Systems_. 1013–1023.
  * Dobbe et al. (2020) Roel Dobbe, Patricia Hidalgo-Gonzalez, Stavros Karagiannopoulos, Rodrigo Henriquez-Auba, Gabriela Hug, Duncan S Callaway, and Claire J Tomlin. 2020.  Learning to control in power systems: Design and analysis guidelines for concrete safety problems.  _Electric Power Systems Research_ 189 (2020), 106615.
  * Donti et al. (2021) Priya L Donti, Melrose Roderick, Mahyar Fazlyab, and J Zico Kolter. 2021.  Enforcing robust control guarantees within neural network policies. In _International Conference on Learning Representations_.
  * Drgoňa et al. (2020) Ján Drgoňa, Javier Arroyo, Iago Cupeiro Figueroa, David Blum, Krzysztof Arendt, Donghun Kim, Enric Perarnau Ollé, Juraj Oravec, Michael Wetter, Draguna L Vrabie, et al. 2020\.  All you need to know about model predictive control for buildings.  _Annual Reviews in Control_ (2020).
  * Fanger (1986) PO Fanger. 1986\.  Thermal environment—Human requirements.  _Environmentalist_ 6, 4 (1986), 275–278.
  * Fernandez et al. (2017) Nicholas EP Fernandez, Srinivas Katipamula, Weimin Wang, YuLong Xie, Mingjie Zhao, and Charles D Corbin. 2017.  _Impacts of commercial building controls on energy savings and peak load reduction_.  Technical Report. Pacific Northwest National Lab.(PNNL), Richland, WA (United States).
  * Fioretto et al. (2020) Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. 2020.  Predicting AC optimal power flows: Combining deep learning and lagrangian dual methods. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 630–637.
  * Fisac et al. (2019) Jaime F Fisac, Neil F Lugovoy, Vicenç Rubies-Royo, Shromona Ghosh, and Claire J Tomlin. 2019\.  Bridging hamilton-jacobi safety analysis and reinforcement learning. In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 8550–8556.
  * Fulton and Platzer (2019) Nathan Fulton and André Platzer. 2019.  Verifiably safe off-model reinforcement learning. In _International Conference on Tools and Algorithms for the Construction and Analysis of Systems_. Springer, 413–430.
  * Garcıa and Fernández (2015) Javier Garcıa and Fernando Fernández. 2015.  A comprehensive survey on safe reinforcement learning.  _Journal of Machine Learning Research_ 16, 1 (2015), 1437–1480.
  * Glavic (2019) Mevludin Glavic. 2019\.  (Deep) Reinforcement learning for electric power system control and related problems: A short review and perspectives.  _Annual Reviews in Control_ 48 (2019), 22–35.
  * Gould et al. (2019) Stephen Gould, Richard Hartley, and Dylan Campbell. 2019\.  Deep declarative networks: A new hope.  _arXiv preprint arXiv:1909.04866_ (2019).
  * Greydanus et al. (2019) Samuel Greydanus, Misko Dzamba, and Jason Yosinski. 2019\.  Hamiltonian neural networks. In _Advances in Neural Information Processing Systems_. 15379–15389.
  * Gupta et al. (2020) Sarthak Gupta, Vassilis Kekatos, and Ming Jin. 2020.  Deep Learning for Reactive Power Control of Smart Inverters under Communication Constraints. In _2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)_. IEEE, 1–6.
  * Han et al. (2019) Minghao Han, Yuan Tian, Lixian Zhang, Jun Wang, and Wei Pan. 2019.  ${H}_{\infty}$ Model-free Reinforcement Learning with Robust Stability Guarantee.  _CoRR_ (2019).
  * Hasanbeig et al. (2020) Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. 2020\.  Deep reinforcement learning with temporal logics. In _International Conference on Formal Modeling and Analysis of Timed Systems_. Springer, 1–22.
  * Hunt et al. (2020) Nathan Hunt, Nathan Fulton, Sara Magliacane, Nghia Hoang, Subhro Das, and Armando Solar-Lezama. 2020\.  Verifiably safe exploration for end-to-end reinforcement learning.  _arXiv preprint arXiv:2007.01223_ (2020).
  * IEEE ([n.d.]) IEEE. [n.d.].  37 node distribution test feeder.  https://ewh.ieee.org/soc/pes/dsacom/testfeeders/.  Online.
  * IEEE (2020) IEEE. 2020.  IEEE Standard Conformance Test Procedures for Equipment Interconnecting Distributed Energy Resources with Electric Power Systems and Associated Interfaces.  _IEEE Std 1547.1-2020_ (2020), 1–282.  https://doi.org/10.1109/IEEESTD.2020.9097534
  * Jalali et al. (2019) Mana Jalali, Vassilis Kekatos, Nikolaos Gatsis, and Deepjyoti Deka. 2019.  Designing reactive power control rules for smart inverters using support vector machines.  _IEEE Transactions on Smart Grid_ 11, 2 (2019), 1759–1770.
  * Kolter et al. (2020) Zico Kolter, David Duvenaud, and Matthew Johnson. 2020\.  Tutorial: Deep Implicit Layers - Neural ODEs, Deep Equilibirum Models, and Beyond.  http://implicit-layers-tutorial.org/.
  * Krantz and Parks (2012) Steven G Krantz and Harold R Parks. 2012.  _The implicit function theorem: history, theory, and applications_.  Springer Science & Business Media.
  * Ling et al. (2018) Chun Kai Ling, Fei Fang, and J Zico Kolter. 2018.  What game are we playing? end-to-end learning in normal and extensive form games.  _arXiv preprint arXiv:1805.02777_ (2018).
  * Luo et al. (2014) Biao Luo, Huai-Ning Wu, and Tingwen Huang. 2014.  Off-Policy Reinforcement Learning for ${H}_{\infty}$ Control Design.  _IEEE Transactions on Cybernetics_ 45, 1 (2014), 65–76.
  * Milano et al. (2018) Federico Milano, Florian Dörfler, Gabriela Hug, David J Hill, and Gregor Verbič. 2018\.  Foundations and challenges of low-inertia systems. In _2018 Power Systems Computation Conference (PSCC)_. IEEE, 1–25.
  * Morimoto and Doya (2005) Jun Morimoto and Kenji Doya. 2005.  Robust Reinforcement Learning.  _Neural Computation_ 17, 2 (2005), 335–359.
  * Pham et al. (2018) Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. 2018\.  Optlayer-practical constrained optimization for deep reinforcement learning in the real world. In _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 6236–6243.
  * Pinto et al. (2017) Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. 2017.  Robust Adversarial Reinforcement Learning. In _Proceedings of the 34th International Conference on Machine Learning_. JMLR. org, 2817–2826.
  * Privara et al. (2013) Samuel Privara, Jiří Cigler, Zdeněk Váňa, Frauke Oldewurtel, Carina Sagerschnig, and Eva Žáčeková. 2013\.  Building modeling as a crucial part for building predictive control.  _Energy and Buildings_ 56 (2013), 8–22.
  * R. D. Zimmerman, C. E. Murillo-Sanchez, and R. J. Thomas (2011) R. D. Zimmerman, C. E. Murillo-Sanchez, and R. J. Thomas. 2011.  MATPOWER: Steady-State Operations, Planning and Analysis Tools for Power Systems Research and Education.  _IEEE Transactions on Power Systems_ 26, 1 (2011), 12–19.
  * Rolnick et al. (2019) David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. 2019\.  Tackling Climate Change with Machine Learning.  _arXiv preprint arXiv:1906.05433_ (2019).
  * Schulman et al. (2015) John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ (2015).
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal Policy Optimization Algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * Shah et al. (2020) Sanket Shah, Sinha Arunesh, Varakantham Pradeep, Perrault Andrew, and Tambe Milind. 2020\.  Solving online threat screening games using constrained action space reinforcement learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 2226–2235.
  * Stringer et al. (2020) N. Stringer, A. Bruce, I. MacGill, N. Haghdadi, P. Kilby, J. Mills, T. Veijalainen, M. Armitage, and N. Wilmot. 2020.  Consumer-Led Transition: Australia’s World-Leading Distributed Energy Resource Integration Efforts.  _IEEE Power and Energy Magazine_ 18, 6 (2020), 20–36.  https://doi.org/10.1109/MPE.2020.3014720
  * Taleghan and Dietterich (2018) Majid Alkaee Taleghan and Thomas G. Dietterich. 2018.  Efficient Exploration for Constrained MDPs. In _2018 AAAI Spring Symposia_.
  * Tieleman and Hinton (2012) Tijmen Tieleman and Geoffrey Hinton. 2012.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural networks for machine learning_ 4, 2 (2012), 26–31.
  * Tschiatschek et al. (2018) Sebastian Tschiatschek, Aytunc Sahin, and Andreas Krause. 2018\.  Differentiable Submodular Maximization. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_. 2731–2738.
  * Turchetta et al. (2016) Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. 2016\.  Safe Exploration in Finite Markov Decision Processes with Gaussian Processes. In _Advances in Neural Information Processing Systems_.
  * Wachi et al. (2018) Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. 2018\.  Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Wang et al. (2019) Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. 2019\.  SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In _Proceedings of the 36th International Conference on Machine Learning_. 6545–6554.
  * Wilcox and Marion (2008) Stephen Wilcox and William Marion. 2008.  _Users manual for TMY3 data sets_.  National Renewable Energy Laboratory Golden, CO.
  * Yang et al. (2020) Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. 2020.  Projection-based constrained policy optimization.  _arXiv preprint arXiv:2010.03152_ (2020).
  * Zamzam and Baker (2020) Ahmed S Zamzam and Kyri Baker. 2020.  Learning optimal solutions for extremely fast AC optimal power flow. In _2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)_. IEEE, 1–6.
  * Zhang et al. (2020) Kaiqing Zhang, Bin Hu, and Tamer Basar. 2020.  Policy Optimization for $\mathcal{H}_{2}$ Linear Control with $\mathcal{H}_{\infty}$ Robustness Guarantee: Implicit Regularization and Global Convergence. In _Learning for Dynamics and Control_. PMLR, 179–190.
  * Zhang and Lam (2018) Zhiang Zhang and Khee Poh Lam. 2018.  Practical Implementation and Evaluation of Deep Reinforcement Learning Control for a Radiant Heating System. In _Proceedings of the 5th Conference on Systems for Built Environments_ (Shenzen, China) _(BuildSys ’18)_. ACM, New York, NY, USA, 148–157.
  * Zhang et al. (2019) Zidong Zhang, Dongxia Zhang, and Robert C Qiu. 2019\.  Deep reinforcement learning for power system applications: An overview.  _CSEE Journal of Power and Energy Systems_ 6, 1 (2019), 213–225.
  * Zhao et al. (2017) Lin Zhao, Wei Zhang, He Hao, and Karanjit Kalsi. 2017\.  A geometric approach to aggregate flexibility modeling of thermostatically controlled loads.  _IEEE Transactions on Power Systems_ 32, 6 (2017), 4721–4731.
  * Zhou and Doyle (1998) Kemin Zhou and John Comstock Doyle. 1998.  _Essentials of Robust Control_. Vol. 104.  Prentice hall Upper Saddle River, NJ.


* [5] “AToMPM Project Homepage”, http://www-ens.iro.umontreal.ca/~syriani/atompm/atompm.htm
  * [6] Thomas Beyhl, Regina Hebig and Holger Giese  “A Model Management Framework for Maintaining Traceability Links”  In _Software Engineering 2013 Workshopband_ P-215, Lecture Notes in Informatics (LNI)  Aachen: Gesellschaft für Informatik (GI), 2013, pp. 453–457
  * [7] Thomas Beyhl, Dominique Blouin, Holger Giese and Leen Lambers  “On the operationalization of graph queries with generalized discrimination networks”  In _International Conference on Graph Transformation_ , 2016, pp. 170–186  Springer
  * [8] Jean Bézivin, Frédéric Jouault and Patrick Valduriez  “On the Need for Megamodels”  In _Proceedings of the OOPSLA/GPCE: Best Practices for Model-Driven Software Development workshop, 19th Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications_ , 2004  URL: http://atlanmod.emn.fr/www/papers/OOPSLA04/bezivin-megamodel.pdf
  * [9] Jean Bézivin, Frédéric Jouault, Peter Rosenthal and Patrick Valduriez  “Modeling in the Large and Modeling in the Small”  In _Model Driven Architecture_ 3599/2005, Lecture Notes in Computer Science (LNCS)  Springer-Verlag, 2005, pp. 33–46  URL: http://dx.doi.org/10.1007/11538097_3
  * [10] Jean Bézivin, Frédéric Jouault, Peter Rosenthal and Patrick Valduriez  “Modeling in the large and modeling in the small”  In _Model Driven Architecture_ Springer, 2004, pp. 33–46
  * [11] Xavier Blanc, Alix Mougenot, Isabelle Mounier and Tom Mens  “Incremental Detection of Model Inconsistencies Based on Model Operations”  In _CAiSE ’09: Proceedings of the 21st International Conference on Advanced Information Systems Engineering, Amsterdam, The Netherlands_ 5565/2009 Berlin, Heidelberg: Springer Verlag, 2009, pp. 32–46
  * [12] Arnaud Blouin, Benoit Combemale, Benoît Baudry and Olivier Beaudoux  “Kompren: modeling and generating model slicers”  In _Software & Systems Modeling_ 14.1 Springer Berlin Heidelberg, 2015, pp. 321–337  URL: http://dx.doi.org/10.1007/s10270-012-0300-x
  * [13] Dominique Blouin, Yvan Eustache and Jean-Philippe Diguet  “Extensible Global Model Management with Meta-model Subsets and Model Synchronization”  In _Proceedings of the 2nd International Workshop on The Globalization of Modeling Languages co-located with ACM/IEEE 17th International Conference on Model Driven Engineering Languages and Systems, GEMOC@Models 2014, Valencia, -_ , 2014, pp. 43–52  URL: http://ceur-ws.org/Vol-1236/paper-07.pdf
  * [14] Dominique Blouin, Gilberto Ochoa Ruiz, Yvan Eustache and Jean-Philippe Diguet  “Kaolin: a System-level AADL Tool for FPGA Design Reuse, Upgrade and Migration”  In _NASA/ESA International Conference on Adaptive Hardware and Systems (AHS)_ , 2015
  * [15] Mark Boddy et al.  “FUSED: A Tool Integration Framework for Collaborative System Engineering”  In _Analytic Virtual Integration of Cyber-Physical Systems Workshop_ , 2011
  * [16] Hugo Bruneliere, Erik Burger, Jordi Cabot and Manuel Wimmer  “A feature-based survey of model view approaches”  In _Software & Systems Modeling_ 18.3 Springer, 2019, pp. 1931–1952
  * [17] Hugo Bruneliere, Jokin Garcia Perez, Manuel Wimmer and Jordi Cabot  “EMF Views: A View Mechanism for Integrating Heterogeneous Models”  In _Conceptual Modeling - 34th International Conference, ER 2015, Stockholm Sweden, October 19-22, 2015, Proceedings_ , 2015, pp. 317–325  URL: http://dx.doi.org/10.1007/978-3-319-25264-3_23
  * [18] Hugo Bruneliere, Jordi Cabot, Frédéric Jouault and Frédéric Madiot  “MoDisco: a generic and extensible framework for model driven reverse engineering”  In _Proceedings of the IEEE/ACM international conference on Automated software engineering_ , 2010  DOI: 10.1145/1858996.1859032
  * [19] Jordi Cabot and Ernest Teniente  “Incremental Evaluation of OCL Constraints”  In _CAiSE’06: 18th International Conference on Advanced Information Systems Engineering, Luxembourg, Luxembourg_ 4001/2006, Lecture Notes in Computer Science (LNCS)  Springer Verlag, 2006, pp. 81–95  URL: http://dx.doi.org/10.1007/11767138_7
  * [20] “CoEST Project Homepage”, http://www.coest.org/
  * [21] “Composite EMF Models Project Homepage”, http://www.mathematik.uni-marburg.de/~swt/compoemf/
  * [22] Csaba Debreceni et al.  “Query-driven Incremental Synchronization of View Models”  In _Proceedings of the 2Nd Workshop on View-Based, Aspect-Oriented and Orthographic Software Modelling_ , VAO ’14  New York, NY, USA: ACM, 2014, pp. 31:31–31:38  URL: http://doi.acm.org/10.1145/2631675.2631677
  * [23] Alexander Egyed  “Instant Consistency Checking for the UML”  In _ICSE ’06: Proceedings of the 28th International Conference on Software Engineering_ , 2006, pp. 381–390
  * [24] Hartmut Ehrig, Karsten Ehrig, Ulrike Prange and Gabriele Taentzer  “Fundamentals of algebraic graph transformation”, EATCS  Springer, 2006  DOI: 10.1007/3-540-31188-2
  * [25] “EMF-IncQuery Project Homepage”, https://www.eclipse.org/incquery/
  * [26] “EMF: Eclipse Modeling Framework” Last accessed 25 November 2022, https://www.eclipse.org/modeling/emf/
  * [27] “EMF Views Project Homepage”, http://atlanmod.github.io/emfviews/
  * [28] “Epsilon Project Homepage”, http://eclipse.org/epsilon/
  * [29] Anne Etien, Alexis Muller, Thomas Legrand and Xavier Blanc  “Combining Independent Model Transformations”  In _Proceedings of the 2010 ACM Symposium on Applied Computing_ , SAC ’10  New York, NY, USA: ACM, 2010, pp. 2237–2243  URL: http://doi.acm.org/10.1145/1774088.1774557
  * [30] “Extended Generalized Discrimination Networks Evaluation Artifacts” Last accessed 25 November 2022, https://github.com/hpi-sam/Extended-GDNs
  * [31] Jean-Marie Favre  “Foundations of Model (Driven) (Reverse) Engineering – Episode I: Story of The Fidus Papyrus and the Solarus”  In _Post-Proceedings of Dagstuhl Seminar on Model Driven Reverse Engineering_ , 2004
  * [32] Jean-Marie Favre, Ralf Lämmel and Andrei Varanovich  “Modeling the linguistic architecture of software products”  In _Proceedings of the 15th international conference on Model Driven Engineering Languages and Systems_ , MODELS’12  Berlin, Heidelberg: Springer-Verlag, 2012, pp. 151–167  URL: http://dx.doi.org/10.1007/978-3-642-33666-9_11
  * [33] Franck Fleurey, Beno t Baudry, Robert France and Sudipto Ghosh  “Models in Software Engineering”  Berlin, Heidelberg: Springer-Verlag, 2008, pp. 7–15  URL: http://dx.doi.org/10.1007/978-3-540-69073-3_2
  * [34] Charles L Forgy  “Rete: A fast algorithm for the many pattern/many object pattern match problem”  In _Readings in Artificial Intelligence and Databases_ Elsevier, 1989, pp. 547–559
  * [35] “FUSED Project Homepage”, http://www.adventiumlabs.com/
  * [36] Erich Gamma et al.  “Design patterns: elements of reusable object-oriented software”  Pearson Deutschland GmbH, 1995
  * [37] “Gaspard2 Project Homepage”, http://www.lifl.fr/west/gaspard/
  * [38] Holger Giese, Stefan Neumann and Stephan Hildebrandt  “Model Synchronization at Work: Keeping SysML and AUTOSAR Models Consistent”  In _Graph Transformations and Model Driven Enginering - Essays Dedicated to Manfred Nagl on the Occasion of his 65th Birthday_ 5765, Lecture Notes in Computer Science  Springer Berlin / Heidelberg, 2010, pp. 555–579
  * [39] Holger Giese and Robert Wagner  “From model transformation to incremental bidirectional model synchronization”  In _Software & Systems Modeling_ 8.1 Springer, 2009, pp. 21–43
  * [40] Holger Giese, Stefan Neumann, Oliver Niggemann and Bernhard Schätz  “Model-Based Integration”  In _Model-Based Engineering of Embedded Real-Time Systems - International Dagstuhl Workshop, Dagstuhl Castle, Germany, November 4-9, 2007. Revised Selected Papers_ 6100, Lecture Notes in Computer Science  Springer, 2011, pp. 17–54  URL: http://www.springerlink.com/content/p44512248g5u6717/
  * [41] Joshua Gleitze, Heiko Klare and Erik Burger  “Finding a universal execution strategy for model transformation networks”  In _International Conference on Fundamental Approaches to Software Engineering_ , 2021, pp. 87–107  Springer, Cham
  * [42] “GME Project Homepage”, http://www.isis.vanderbilt.edu/projects/gme/
  * [43] Iris Groher, Alexander Reder and Alexander Egyed  “Incremental Consistency Checking of Dynamic Constraints”  In _Fundamental Approaches to Software Engineering_ 6013, Lecture Notes in Computer Science  Springer Berlin Heidelberg, 2010, pp. 203–217  URL: http://dx.doi.org/10.1007/978-3-642-12029-9_15
  * [44] Annegret Habel and Karl-Heinz Pennemann  “Correctness of high-level transformation systems relative to nested conditions”  In _Mathematical Structures in Computer Science_ 19.2 Cambridge University Press, 2009, pp. 245–296
  * [45] Eric N Hanson, Sreenath Bodagala and Ullas Chadaga  “Trigger condition testing and view maintenance using optimized discrimination networks”  In _IEEE Transactions on Knowledge and Data Engineering_ 14.2 IEEE, 2002, pp. 261–280
  * [46] Regina Hebig, Andreas Seibel and Holger Giese  “On the Unification of Megamodels”  In _Proceedings of the 4th International Workshop on Multi-Paradigm Modeling (MPM 2010)_ 42, Electronic Communications of the EASST, 2011  URL: http://journal.ub.tu-berlin.de/eceasst/article/view/704/713
  * [47] Reiko Heckel, Jochen Malte Küster and Gabriele Taentzer  “Confluence of typed attributed graph transformation systems”  In _International Conference on Graph Transformation_ , 2002, pp. 161–176  Springer
  * [48] Sebastian J.I. Herzig, Ahsan Qamar and Christiaan J.J. Paredis  “An Approach to Identifying Inconsistencies in Model-based Systems Engineering” 2014 Conference on Systems Engineering Research  In _Procedia Computer Science_ 28.0, 2014, pp. 354–362  DOI: http://dx.doi.org/10.1016/j.procs.2014.03.044
  * [49] Sebastian JI Herzig and Christiaan JJ Paredis  “Bayesian Reasoning Over Models”  In _11th Workshop on Model Driven Engineering, Verification and Validation MoDeVVa 2014_ , 2014, pp. 69
  * [50] Anders Hessellund and Andrzej Wasowski  “Interfaces and Metainterfaces for Models and Metamodels”  In _Model Driven Engineering Languages and Systems_ 5301, Lecture Notes in Computer Science  Springer Berlin Heidelberg, 2008, pp. 401–415  URL: http://dx.doi.org/10.1007/978-3-540-87875-9_29
  * [51] Jendrik Johannes and Uwe Assmann  “Concern-Based (de)composition of Model-Driven Software Development Processes” 10.1007/978-3-642-16129-2_5  In _Model Driven Engineering Languages and Systems_ 6395, Lecture Notes in Computer Science  Springer Berlin / Heidelberg, 2010, pp. 47–62  URL: http://dx.doi.org/10.1007/978-3-642-16129-2%5C_5
  * [52] Stefan Jurack and Gabriele Taentzer  “A Component Concept for Typed Graphs with Inheritance and Containment Structures”  In _Graph Transformations - 5th International Conference, ICGT 2010 Enschede, The Netherlands, September 27 - - October 2, 2010. Proceedings_ , 2010, pp. 187–202  URL: http://dx.doi.org/10.1007/978-3-642-15928-2_13
  * [53] Heiko Klare et al.  “Enabling consistency in view-based system development—the vitruvius approach”  In _Journal of Systems and Software_ 171 Elsevier, 2021, pp. 110815
  * [54] “Kompose Project Homepage”, http://www.kermeta.org/mdk/kompose
  * [55] “Kompren Project Homepage”, http://people.irisa.fr/Arnaud.Blouin/software_kompren.html
  * [56] Harald König and Zinovy Diskin  “Efficient consistency checking of interrelated models”  In _European Conference on Modelling Foundations and Applications_ , 2017, pp. 161–178  Springer
  * [57] D. Langsweirdt, N. Boucke and Yolande Berbers  “Architecture-Driven Development of Embedded Systems with ACOL”  In _Object/Component/Service-Oriented Real-Time Distributed Computing Workshops (ISORCW), 2010 13th IEEE International Symposium on_ , 2010, pp. 138–144
  * [58] Henrik Lochmann and Anders Hessellund  “An Integrated View on Modeling with Multiple Domain-Specific Languages”  In _Proceedings of the IASTED International Conference Software Engineering (SE 2009)_ ACTA Press, 2009, pp. 1–10
  * [59] Levi Lúcio et al.  “FTG+PM: An Integrated Framework for Investigating Model Transformation Chains”  In _SDL 2013: Model-Driven Dependability Engineering_ 7916, Lecture Notes in Computer Science  Springer Berlin Heidelberg, 2013, pp. 182–202  URL: http://dx.doi.org/10.1007/978-3-642-38911-5_11
  * [60] “ModeSoC Project Homepage”, http://www.reuseware.org/index.php/Reuseware_ModelSoC
  * [61] “MoTE Project Homepage”, http://www.mdelab.org/mdelab-projects/mote-a-tgg-based-model-transformation-engine/
  * [62] Object Management Group  “Object Constraint Language Specification” Last accessed 25 November 2022, https://www.omg.org/spec/OCL/
  * [63] “OSLC Project Homepage”, http://open-services.net/
  * [64] Richard F. Paige et al.  “The Design of a Conceptual Framework and Technical Infrastructure for Model Management Language Engineering”  In _Engineering of Complex Computer Systems, 2009 14th IEEE International Conference on_ , 2009, pp. 162–171
  * [65] Cosmina Cristina Raţiu, Wesley KG Assunção, Rainer Haas and Alexander Egyed  “Reactive links across multi-domain engineering models”  In _Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems_ , 2022, pp. 76–86
  * [66] José E. Rivera et al.  “Orchestrating ATL Model Transformations”  In _In Proc. of MtATL 2009: 1st International Workshop on Model Transformation with ATL_ , 2009, pp. 34–46  URL: http://www.lcc.uma.es/~av/Publicaciones/09/MtATL2009.pdf
  * [67] Andy Schürr  “Specification of graph translators with triple graph grammars”  In _Proc. of the $20^{t}h$ International Workshop on Graph-Theoretic Concepts in Computer Science_ 903, Lecture Notes in Computer Science  Herrsching, Germany: Spinger Verlag, 1994, pp. 151–163
  * [68] Andreas Seibel, Regina Hebig and Holger Giese  “Traceability in Model-Driven Engineering: Efficient and Scalable Traceability Maintenance”  In _Software and Systems Traceability_ Springer London, 2012, pp. 215–240
  * [69] Andreas Seibel, Stefan Neumann and Holger Giese  “Dynamic hierarchical mega models: comprehensive traceability and its efficient maintenance”  In _Software & Systems Modeling_ 9.4 Springer, 2010, pp. 493–528
  * [70] Andreas Seibel, Stefan Neumann and Holger Giese  “Dynamic hierarchical mega models: comprehensive traceability and its efficient maintenance”  In _Software and Systems Modeling_ 9.4, 2010, pp. 493–528  URL: http://dx.doi.org/10.1007/s10270-009-0146-z
  * [71] Andreas Seibel, Regina Hebig, Stefan Neumann and Holger Giese  “A Dedicated Language for Context Composition and Execution of True Black-Box Model Transformations”  In _4th International Conference on Software Language Engineering (SLE 2011) , Braga, Portugal_ , 2011
  * [72] Gabor Simko et al.  “Foundation for model integration: Semantic backplane”  In _ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference_ , 2012, pp. 1077–1086  American Society of Mechanical Engineers
  * [73] “SmartEMF Project Homepage”, http://www.itu.dk/~hessellund/smartemf/
  * [74] Perdita Stevens  “Connecting software build with maintaining consistency between models: towards sound, optimal, and flexible building from megamodels”  In _Software and Systems Modeling_ 19.4 Springer, 2020, pp. 935–958
  * [75] Perdita Stevens  “Maintaining consistency in networks of models: bidirectional transformations in the large”  In _Software and Systems Modeling_ 19.1 Springer, 2020, pp. 39–65
  * [76] Zoltan Ujhelyi et al.  “EMF-IncQuery: An integrated development environment for live model queries” Fifth issue of Experimental Software and Toolkits (EST): A special issue on Academics Modelling with Eclipse (ACME2012)  In _Science of Computer Programming_ 98, Part 1, 2015, pp. 80–99
  * [77] Hans Vangheluwe, Juan Lara and Pieter J Mosterman  “An introduction to multiparadigm modelling and simulation”  In _Proceedings of the AIS2002 Conference (2002)_ , 2002
  * [78] Dániel Varró et al.  “Road to a reactive and incremental model transformation platform: three generations of the VIATRA framework”  In _Software & Systems Modeling_ 15.3 Springer, 2016, pp. 609–629
  * [79] Andrés Vignaga, Frédéric Jouault, MaríaCecilia Bastarrica and Hugo Brunelière  “Typing artifacts in megamodeling”  In _Software & Systems Modeling_ 12 Springer-Verlag, 2013, pp. 105–119  URL: http://dx.doi.org/10.1007/s10270-011-0191-2
  * [80] “Wires Project Homepage”, {http://atenea.lcc.uma.es/index.php/Main_Page/Resources/Wires*}