# Multi-agent Deep Reinforcement Learning for Distributed Load Restoration

Linh Vu, Tuyen Vu,
Thanh Long Vu, and Anurag Srivastava

###### Abstract

This paper addresses the load restoration problem after power outage events.
Our primary proposed methodology is using multi-agent deep reinforcement
learning to optimize the load restoration process in distribution systems,
modeled as networked microgrids, via determining the optimal operational
sequence of circuit breakers (switches). An innovative invalid action masking
technique is incorporated into the multi-agent method to handle both the
physical constraints in the restoration process and the curse of
dimensionality as the action space of operational decisions grows
exponentially with the number of circuit breakers. The features of our
proposed method include centralized training for multi-agents to overcome non-
stationary environment problems, decentralized execution to ease the
deployment, and zero constraint violations to prevent harmful actions. Our
simulations are performed in OpenDSS and Python environments to demonstrate
the effectiveness of the proposed approach using the IEEE 13, 123, and
8500-node distribution test feeders. The results show that the proposed
algorithm can achieve a significantly better learning curve and stability than
the conventional methods.

###### Index Terms:

Deep reinforcement learning, invalid action masking, distribution systems,
networked microgrids, multi-agent systems, load restoration, OpenDSS.

## Nomenclature

$P_{ij}$

Rated power of load $j$ (downstream of switch $i$)

$w_{ij}$

Weighting factor of load $j$ (downstream of switch $i$)

$n_{i}$

Binary variable, represents the status of switch $i$ (0 means switch $i$ is
open and 1 means switch $i$ is close)

$V_{u}$

Voltage at bus $u$

$p_{u}$

Active power injected at bus $u$

$q_{u}$

Reactive power injected at bus $u$

$s_{u}$

Apparent power injected at bus $u$

$s_{mn}$

Apparent power from bus $m$ to bus $n$

$o$

Current observation of the system

$o^{\prime}$

Next observation of the system

$r$

Current reward

$a_{i}$

Current action of agent $i$

$A_{i}$

Action space of agent $i$

a

Joint action of all agents

$m$

Invalid action masking information

$\gamma$

Discounted value

$\alpha$

Learning rate

$\epsilon$

Epsilon value

$\lambda$

Decay value

$Q(s,a)$

Q value of ANN model

$\theta$

Weights and biases of ANN model.

##  I Introduction

In recent years, weather patterns have varied unpredictably. Unfortunately,
the power systems are not adequately prepared to face extreme weather events.
For example, the Texas freeze in February 2021 impacted 4.5 million power
users and resulted in $130 billion in economic damages [1]; Hurricane Isaias
in August 2020 affected more than 6.4 million customers; the California 2019
Power Shutoffs in October 2019 impacted 1.6 million customers and the New
England Storm in October 2019 affected 1.25 million customers. In certain
worst-case scenarios, it can take an extended period of time for power to be
restored through the utility grid and made available to customers.

A microgrid is one promising solution that can produce immediate power to
specific customers in the event of a utility power outage. After a fault,
microgrids can switch from grid-connected mode to islanded mode and operate
independently. However, since the power from microgrids is limited, they
cannot provide a full power supply for all customers, so critical loads must
be identified and prioritized. In order to transfer power only to critical
loads, it is vital to determine the optimal sequence to regulate the circuit
breakers. Solving the optimal pattern of circuit breaker usage is referred to
as the load restoration problem. Load restoration is a complex problem as its
properties involve multi-objective, non-linear, and combinatorial selections.
To tackle the load restoration problem, several approaches have been
investigated. Recent studies on load restoration may be divided into two
categories: optimization-based and machine learning-based methods.

Optimization-based approach: The Spanning Tree Search approach is presented in
[2], in which the objective function is formulated to minimize the number of
switching actions and maximize the restored power. Although this formulation
has reduced the operating stress on the switches, it failed to address the
essential aspect of load priority. The authors in [3] have overcome this
limitation by reformulating the problem into two objective functions to
integrate the weighting of loads and minimize the voltage variation of the
power network. Furthermore, a multi-stage restoration is proposed in [4],
where load priority, number of switching actions, and voltage variation are
combined into the objective function. In order to formulate the problem where
some switching variables are represented as integers and others as continuous
variables, further optimization-based approaches employing mixed-integer
linear programming (MILP) methods are described in [5, 6, 7, 8] or mixed-
integer non-linear programming (MINLP) methods in [9, 10, 11]. The results
from these papers using mixed-integer methods have indicated that a globally
optimal solution can be achieved, but the computation time presents a
significant challenge, especially for real-time decision-making. In [12], the
authors presented a novel formulation that has reduced variables, constraints,
and computation time. Other aspects of load restoration problems have also
been investigated in [13, 14, 15] that include repair time, renewable energy
availability, and the optimal path for sending crews to the fault fields.

Learning-based approach: The optimization-based methods are primarily
restricted in deterministic environments or bounded uncertainty conditions,
whereas often there are uncertainties in the data and model that are not
bounded and must be taken into account, such as load randomness, cold load
pickup, imperfect renewable forecast, and mobile energy resources [16, 17, 18,
19]. Moreover, optimization-based methods mainly produce the final
configuration topology, largely omitting the sequence needed to regulate the
switches [20, 21]. Furthermore, optimization-based approaches have an
additional disadvantage due to the non-convex nature of power flow constraints
and the requirement for a complete and accurate mathematical model of the
environment. As demonstrated in [22], the computing complexity of classical
MILP formulations can make them infeasible to deploy. Since the learning-based
approach uses trained models to make decisions, the computational time can be
significantly reduced because it only involves feed-forward calculations to
make decisions. To take advantage of learning-based approach and to overcome
such limitations of optimization-based approach, reinforcement learning (RL)
has recently been gaining attention. The authors in [23, 24, 25, 26]
introduced Q-learning, an off-policy, model-free reinforcement learning
algorithm. The idea of Q-learning is to create a finite table of states and
actions, then seek the best action given the current system states. Although
this method can ensure policy convergence and an optimal solution [27], it is
very burdensome to use at a large scale on complex power systems since the
number of states and actions will grow exponentially as the number of switches
increases. To address this drawback, a Deep Q-learning approach is proposed in
[28]. Instead of forming a table to store state-action pairings, Deep
Q-learning utilizes an artificial neural network (ANN) as a functional
approximation of state-action pairs and trains the Deep Q-networks to generate
the optimal decision. ANNs are also used as one of their algorithms’
components in [29, 19], where two ANNs known as the critic network and actor
network are implemented in the Deep Deterministic Policy Gradient (DDPG) and
Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithms. Those
algorithm uses a critic network to estimate the action-value function and an
actor network to learn the optimal policy of the load restoration agent. To
deal with the uncertainty of available power from renewable energy during the
restoration process, the authors in [30] developed a reinforcement learning
algorithm based on evolution strategies. Furthermore, work in [31] integrated
the asynchronous information of available generator power resources into the
DQN agent in the training phase.

Multi-agent learning: Although the above-mentioned reinforcement learning
algorithms have been carried out to solve the load restoration problem, they
are performed in a centralized manner, where only one centralized agent
interacts with the environment at each time step. However, such a centralized
machine learning-based control paradigm requires communication links from the
central controller to all remote locations. This can potentially incur
communication failures or cyberattacks on the controller and communication
links. Also, participants need to upload the necessary information to the
controller. Thus some private information may be at risk of leaks. Most
critically, training the centralized RL on large-scale systems with thousands
of nodes would require a prohibitively long training time to train an
excessive number of neural network parameters. Finally, many problems in real-
world environments require more than one agent to be fully solved; examples
include unmanned aerial vehicles [32], traffic flow [33], and more [34, 35,
36, 37, 38]. In [39], the author shows that by using a multi-agent
reinforcement learning (MARL) platform, the hunters (agents) can collaborate
to capture prey faster than a single agent by sharing their learned knowledge.
According to [40], authors emphasize that MARL can accelerate the learning
process since the agents not only learn from the environment but also from
each other. In [41], the multi-agent concept was firstly leveraged to restore
service in the distribution systems by treating different types of electrical
devices (such as load, aggregator, distributed generator, and switch) as
individual agents. These agents could communicate with each other to implement
a heuristic rule-based algorithm. However, this approach focused primarily on
the communication aspect between agents and did not utilize learning-based
techniques. In contrast, the work presented in [42] utilized deep
reinforcement learning to enhance power system resilience by using the shunt
reactive power compensators as control objectives. To control the status of
circuit breakers, another approach used in [43] was a graph-based method,
where each agent must depart from a depot and travel sequentially along the
energization path based on features from graphs. However, this paper’s
limitation was that the agents had limited autonomy in deciding their actions.

In this paper, we adopted the advantages of a deep MARL platform to restore
critical loads in distribution systems that can be modeled as networked
microgrids. In a mixed ownership environment, each microgrid can belong to
either a utility or non-utility owner, and hence, no centralized controller is
authorized to control all microgrids. In this context, it is natural to adopt
the multi-agent framework, in which each microgrid has one reinforcement
learning agent. These RL agents in networked microgrids can aggregate their
information in a centralized training setting to learn optimal control
policies. The collaboration of multi-agents consists of: (1) sharing actions
taken by other agents, (2) sharing invalid actions to avoid, and (3) sharing
the reward they contribute to the global reward function.
The following are the paper’s main contributions:

  * •

To the best of our knowledge, this is the first time a multi-agent deep
reinforcement learning algorithm has been developed and adopted to solve the
load restoration problem.

  * •

The paper proposes a novel enhancement into the MARL of an invalid action
masking technique to improve the learning performance and convergence against
physical constraint violations to ensure safe load restoration.

  * •

The paper compares the convergence curves and optimal results between single-
agent and multi-agent RL and also provides a comprehensive analysis of the
importance of the invalid action masking technique.

The remainder of this paper is organized as follows. Section II is divided
into four subsections. We formulated the load restoration problem as an
optimization in II-A. Then a multi-agent DQN framework is introduced to
control multiple microgrids inside a complex power system in II-B. II-C
describes the epsilon-greedy method to balance exploration and exploitation.
Lastly, II-D presents the invalid action masking technique and the main
algorithm in detail. Section III shows the experimental results using IEEE
standard test feeders.

##  II Multi-agent Reinforcement Learning for Load Restoration

The power system’s capability to resist and minimize the magnitude or duration
of extreme events, including the ability to predict, absorb, adapt to, or
quickly recover from such catastrophes, is called resilience[44]. Several
publications [45, 46, 47, 48, 49] defined the resilience curve to evaluate the
performance of the power system during a disaster. Illustrated in Fig. 1, the
resilience level is used to measure the ability of a power system under an
extreme event. This process can be divided into several stages. In this paper,
our proposed algorithm aims to focus on the recoverability stage represented
as $t_{4}\ \text{to}\ t_{5}$, where the proposed algorithm will work on
maximizing the critical loads within the available power limitations.

Figure 1: The concept of resiliency curve during an extreme event.[50]

###  II-A Load Restoration Problem

During the restoration period, power channels from the utility grid to
consumers are disrupted, requiring microgrid systems as alternate sources to
meet the power demand of critical loads. These power paths are defined by the
status of circuit breakers, which were installed in the power systems to
manage the power supply to downstream loads.

The primary objective function: |l|[3] n_i∑n_i ∑P_ijw_ijeq:2010 ∑P_loss+∑n_i
∑P_ij≤∑P_geneq:2011 V^min≤V_u≤V^maxeq:2012 p_min≤p_u≤p_maxeq:2013
q_min≤q_u≤q_maxeq:2014 p^2_mn+q^2_mn≤s^2_mn.eq:2015

The main objective function is formulated in the form of optimization-based
equations (LABEL:eq:2010), where the purpose is to find the maximum value of
the restored load. Several safe constraints of the physical power system are
satisfied by (LABEL:eq:2011 \- LABEL:eq:2015) during the restoration process.
In (LABEL:eq:2010), $P_{ij}$ and $w_{ij}$ are rated power and weighting factor
(that represents the load’s priority) of the downstream load $j$ of the
circuit breaker $i$, respectively. Binary variable $n_{i}$ represents the
status of circuit breakers $i$ ($0$ means circuit breaker is open, and vice
versa). Constraint (LABEL:eq:2011) ensures the power balance between
generators and loads. Constraint (LABEL:eq:2012) guarantees the operating
voltage at each bus to be within a safe range of the nominal $V_{i}$ value.
Constraint (LABEL:eq:2013 \- LABEL:eq:2014) represents the active power
($p_{u}$) and reactive power ($q_{u}$) output limits for each local generator.
Finally, equality constraint (LABEL:eq:2015) represents the sending-end
complex power from bus $m$ to bus $n$.

###  II-B Multi-agent DQN Formulation for Load Restoration

Based on the problem formulation in II-A, this paper proposes a multi-agent
deep reinforcement learning framework for load restoration. We consider that a
distribution power system can be divided into multiple microgrids. The
division depends on the total rated power of loads, the number of switches,
and the available power of generators. In this framework, each agent is in
charge of a distinctive microgrid and is responsible for the status of all
switches inside that microgrid.

Figure 2: The overall diagram of multi-agent DRL framework.

This multi-agent algorithm is an enhancement based on single-agent deep
reinforcement learning algorithms [31]. We formulate the load restoration
problem as a Markov decision process (MDP) [51]. Fig. 2 illustrates the
overall diagram of the multi-agent framework. The proposed algorithm uses $n$
agents ($n\geq 2$), where each agent is an artificial neural network model, to
learn from the environment and make decisions. In the centralized training
phase, each agent chooses their own action simultaneously (i.e., agent 1 sends
action $a_{1}$, agent 2 sends action $a_{2}$, etc.) at the beginning of every
time-step from their action spaces. The epsilon-greedy policy (II-C) is
utilized for decision-making by each agent, which allows for either random
selection or following their current policies $a_{i,t}\sim\pi(\cdot|s_{i,t})$.
These individual actions are then aggregated into a joint action denoted as
$\textbf{a}\triangleq[a_{1},\dots,a_{n}]$, as described in [52]. The
environment receives the joint action and deploys the corresponding actions to
each microgrid, resulting in a transition of the system’s state to the next
state $s_{t+1}\sim\rho(s_{t+1}|s_{t},\textbf{a}_{t})$. Each agent then
receives an immediate scalar reward, and the resulting system state
(observation ($o$), reward ($r$), action masking ($m$)) is provided as
feedback to the agents.

After each training time-step, each agent stores a tuple information of
observation, reward, action, and next observation
($o^{t},r^{t},a^{t},o^{t+1}$) to accumulate the knowledge. The agents repeat
this process multiple times until they have sufficient information to tackle
the problem. Below are detailed descriptions of the environment, action space,
observation, and reward function for problem formulation.

Environment: In this framework, the environment is the distribution system,
which is divided into multiple microgrids as shown in Fig. 2. Controllable
circuit breakers are installed for test feeders to maneuver the power supply
paths to the loads. Each time a circuit break is activated, the states of the
system will be changed accordingly. Therefore, the power flow problem needs to
be solved to determine the states of the system following the deployment of an
action and to guarantee that all constraints will be met. Any power system
simulation tools such as OpenDSS [53], MATLAB-Simulink, or MATPOWER [54] can
be used to simulate the environment.

Action space: The action at each time-step of an agent refers to turning on or
off one circuit breaker inside its microgrid. In this paper, we assume that
circuit breakers can be monitored and operated ideally following the commands
from the controller. Therefore, the control action of agent 1 can be
represented as an array of binary values $a_{1}=[cb1_{on}\ cb1_{off}\
cb2_{on}\ cb2_{off}\dots cbn_{on}\ cbn_{off}]$, where $n$ is the number of
circuit breakers in microgrid 1. For instance, agent 1 selects to turn on
circuit breaker 2, then $a_{1}=[0\ 0\ 1\ 0\ \dots\ 0\ 0]$. Note that each
agent will have a different size of action space depending on the number of
circuit breakers (switches) inside its microgrid. The joint action is a
combination of actions sending from all agents
($\textit{{a}}\triangleq[a_{1},\dots,a_{n}]$) to the environment.

Observation: In this framework, each agent is able to access its local
observation at every time step as shown in Fig. 2. This observation consists
of the status of all circuit breakers within its zone and can be represented
as an array of binary values, denoted as $\textit{{o}}=[o_{1}\ o_{2}\ \dots\
o_{m}]$, where $m$ represents the number of circuit breakers under the agent’s
responsibility. Note that when $o_{1}=1$, it indicates that circuit breaker 1
is in a closed state, and conversely, when $o_{1}=0$, it means that circuit
breaker 1 is in an open state.

Reward function: In this framework, all the agents will share the same global
reward function, which is shown in (1). Similar to the main objective function
in LABEL:eq:2010, this reward function is designed to encourage the agents
trying to maximize the restored power of loads. It should be noted that we
normalized the reward function (re-scale to between 0 and 1) in order to
stabilize the training.

$\displaystyle r=\frac{\sum n_{i}\sum P_{ij}w_{ij}}{\sum P_{ij}}.$ (1)

Fig. 3 goes into detail about the Deep Q-network architecture of an agent
within a multi-agent framework. The input of ANN is the current observation of
this agent, which is the status of all the circuit breakers within its
microgrid, while the output is Q-values for selecting the available actions.
The output of ANN has its number depends on the number of circuit breakers
that the agent manages and represents the benefit of selecting the actions.
The hidden layers are designed depending on the complexity of the problem to
store the learned policies.

This paper incorporates the experience replay technique to break harmful
correlations and enable agents to learn more effectively by recalling rare
occurrences. Specifically, each agent has its own experience replay data
during training, consisting of multiple tuples (observation, reward, action,
next observation) learned during exploration and stored for each agent. To
utilize this data for training, the training process randomly samples small
mini-batches from the larger experience replay dataset, as shown in Fig. 2.

Figure 3: An example of an agent’s ANN structure within a multi-agent
framework. Figure 4: The training process of an agent within a multi-agent
framework.

Fig. 4 shows the overall diagram of the training process. The RL agents
collect data from the trial and error process until they perform sufficient
numbers of actions in the OpenDSS environment. For the training process, two
ANNs are used for each agent, namely the main ANN and target ANN. They have
the same initialization and the weights of the main ANN will be updated
periodically into the target ANN. According to [55, 56], the implementation of
two ANNs for agents helps slow down learning, strengthens the algorithm, and
improves the effectiveness of agent environment exploration.

Based on the current reward and the next observation, we can calculate the
maximum value in the future that the agent can take after choosing the current
action at time step $t$ using (2). Value $\gamma$ is called the discounted
value, and it is used to decide the importance of rewards in the distant
future relative to those in the immediate future.

$\displaystyle y^{t}=r^{t}+\gamma\max_{a}Q(o^{t+1},\textbf{a})$ (2)

Equation (3) is used to update the $Q_{predicted}(s,a)$ value during the
training to track the $Q_{target}(s,a)$ value gradually. Value $\alpha$ is
known as learning rate value to decide the changing rate of $Q_{target}(s,a)$
compared to $Q(s,a)_{predicted}$.

$\displaystyle Q_{target}(s,a)=(1-\alpha)Q(s,a)_{predicted}+\alpha y^{t}$ (3)

To train the model, a loss function is defined as shown in (4) to perform
gradient descent steps.

$\displaystyle L(\theta)=\mathbb{E}[(Q_{target}(s,a)-Q_{predicted}(s,a))^{2}]$
(4)

This equation represents the squared difference between the $Q_{target}(s,a)$
and the $Q_{predicted}(s,a)$.

For each individual tuple of the mini-batch, we can calculate a pair of input
data as shown in Fig. 4, which is the state of the system and output data.
Based on the input and target data, the model can be trained to fit the data
as the Q-value will be updated during the exploration process.

In the decentralized execution phase, the agents utilize the trained models to
make decisions based on their local observations without the invalid mask
technique. This approach offers the added benefit of enhancing data privacy
and security since the data is processed locally on each networked-microgrid
and is not transmitted to a central controller. This is particularly crucial
in scenarios where sensitive data is at risk of being compromised by
cyberattacks.

###  II-C Epsilon-greedy Method

The epsilon-greedy approach [57] is selected to balance exploration and
exploitation over the learning course. This method is motivated by the fact
that during the initial training phase, agents lack information about the
environment and must prioritize exploration over exploitation. Once the agents
have gained sufficient knowledge through training the ANN models, they can
rely on that knowledge to inform their decision-making.

At the beginning of the algorithm, the epsilon value is initialized to 1. To
gradually decrease the epsilon value $\epsilon$ to a minimum value of
$\epsilon_{min}$ over episodes, (5) is utilized. The decay value ($\lambda$)
determines the rate of reduction, while $\epsilon_{max}$ and $\epsilon_{min}$
define the upper and lower limits of epsilon, respectively.

$\displaystyle\epsilon=\epsilon_{min}+(\epsilon_{max}-\epsilon_{min})e^{-\lambda\text{episode}}$
(5)

###  II-D Invalid Action Masking Technique

One of the biggest challenges in solving the load restoration problem is that
we both need to maximize the main objective function as well as ensure all
power flow constraints (LABEL:eq:2013 \- LABEL:eq:2015) are satisfied at all
times.

The traditional approach is to put a large negative reward into the reward
function each time the agents make a decision that would violate any
constraint as described in [58, 59].

The algorithm can perform well with this approach for a small distribution
system with a modest number of switches. In this case, the reward function can
be written as in (6).

$\displaystyle r=\begin{cases}\sum n_{i}\sum P_{ij}w_{ij}&\text{All
constraints satisfy}\\\ M&\text{Any constraint violates}\end{cases}$ (6)

With M being a large negative value, agents are warned not to choose
undesirable actions. One drawback of this approach is that M will vary
depending on the particular problem. In most cases, M needs to be well-
designed based on experience. This approach becomes even more complex when the
system at some certain states has a very small number of actions that can
satisfy all the constraints, cost the agent a huge amount of time to learn, or
even fail to learn [60]. As demonstrated in [61], DOTA2 is an example of an
environment where agents are presented with up to 1,837,080 actions to choose
from their action spaces at each time-step. However, most of these actions are
irrelevant to the task at hand, resulting in a significant amount of wasted
time if agents attempt to learn them all [62]. Therefore, it’s crucial to
identify which actions should be ignored in order to optimize the learning
process.

To deal with this type of problem, AlphaStar and OpenAI 5 proposed an invalid
action masking technique in their research framework [63, 64], which is a
trick to mask out meaningless actions in the full action space of each agent.

Figure 5: Visualization of action masking.

Fig. 5 shows the mechanism of the invalid action masking. With a given state
as the input to each agent’s ANN, the ANN will output the Q-value of picking
actions corresponding to each circuit breaker turning on or off. The deep
Q-learning algorithm uses the $argmax$ function to choose the action that
raises the highest reward value across the available actions. The basic
principle behind action masking is that it will obstruct this process by
extracting invalid action information from the environment at each time-step
and creating a mask that will inform the agents which one they should not take
at that time. In [65], the authors presented the theory of action masking and
how to implement the action masking technique with the Proximal Policy
Optimization (PPO) algorithm, which is a gradient-based policy algorithm by
renormalizing the distribution of output Q-value after removing the action
that would violate the constraints. In this study, we implemented invalid
action masking by using the deep Q-learning method, which is a value-based
policy technique, by assigning a big negative value ($-\infty$) to actions
that violate the environment and then reselecting the action with the highest
value after removing the first highest one.

Because our challenge involves a multi-agent system, a joint action should be
considered to send to the environment in order for each agent to know if their
actions would break the constraints or not. In case the joint action violates
the environment constraints, a random agent will be selected to remove its
highest action value and reselect the action from the agents again. In the
next action selection, this would help reduce the restored power as well as
the probability of violating the constraints. This process will be repeated
until the agents choose a joint action that will not violate any constraint.

The above sections have introduced the essential keys of the algorithm. The
detailed version of Algorithm 1 describes distributed DQL approach for load
restoration.

1Initialize total number of agents $m$;

2 Initialize $\epsilon_{0},\ \epsilon_{max},\ \epsilon_{min},\ \lambda$;

3 Initialize experience replay size and mini-batch size;

4 Initialize training episode $n_{0},\ n_{max}$;

5 Initialize total training reward $R$;

6 Initialize step to update models $k_{0},\ k_{max}$;

7 Initialize step within an episode $u_{o},\ u_{max}$;

8 repeat

9 Initialize the main model for agent $i$;

10 Initialize the target model for agent $i$;

11 Copy weights from the main model agent $i$ to the target model agent $i$;

12 Initialize replay experience tuple for agent $i$;

13

14until  __m agents are initialized__ ;

15for  _$n_{0}=0$ to $n_{max}$_ do

$R=0$; $\triangleright$ Reset episode reward

16 Reset the OpenDSS Environment;

17 while  _$u_{0}\leq u_{max}$_ do

18 $u_{0}\mathrel{+}=1$; $k_{0}\mathrel{+}=1$;

19 Randomly generate $\epsilon_{i}\in(0,1)$;

20 if  _$\epsilon_{i}\leq\epsilon_{0}$_ then

21 repeat

22 Randomly select action $a_{i}$ for agent $i$ ($a_{i}\in A_{i}$);

23 Joint actions from agents;

24

25 until  __$m$ agents decide their actions__;

26 while  _Invalid action masking == False_ do

27 Jump to step (1)

28

29 else

30 repeat

31 Select action $a_{i}$ for agent $i$ ($a_{i}\in A_{i})$ based on their
current policies;

32 Joint actions from agents;

33

34 until  __$m$ agents decide their actions__;

35 while  _Invalid action masking == False_ do

36 Randomly select an agent $j\ (j\in[1,m])$;

37 Assign the maximum Q-value of agent $j$’s current output to $-\infty$;

38 Jump to step (1) with modified agent $j$’s output;

39

40

41 OpenDSS deploys joint action then returns new observation ($o^{\prime}$)
and immediate reward ($r$);

42 $R\mathrel{+}=r$;

43 repeat

44 Append $(o,a_{i},r,o^{\prime})$ into experience replay tuple of agent $i$;

45

46 until  __$m$ agents are appended__;

47 $o\leftarrow o^{\prime}$;

48 Train main models using experience replay (2$-$3);

49 if  _$(k_{o}\bmod k_{max})==0$_ then

50 Copy weights from main model to target model of each agent;

51

52

53 Reduce $\epsilon_{0}$ using (5);

Algorithm 1 Distributed deep reinforcement learning algorithm for load
restoration

##  III Case Studies

The algorithm is verified by using an Intel(R) Core(TM) i7-10700 CPU (2.90 GHz
processor and 16.0 Gb RAM). To validate the scalability of the algorithm, we
investigate three systems of varying sizes: small (IEEE 13-node), medium (IEEE
123-node), and large (IEEE 8500-node) systems. The algorithm is coded in a
Python environment and communicated with OpenDSS software using the "py-dss-
interface" package [66].

###  III-A IEEE 13 Nodes

Figure 6: 13-node networked microgrids configuration.

In Fig. 6, the distribution system is divided into two microgrids
corresponding to two agents. Microgrid 1 has four loads
$L_{MG1}=[230,170,400,200]$ kW and microgrid 2 has five loads
$L_{MG2}=[170,128,1150,170,843]$ kW. During normal operation, the system needs
to provide 3461 kW to serve the full power of loads. Three solar power sources
with a total capacity of 2600 kW are installed on the distribution system.
After a major power outage, we assume that all circuit breakers are in the
open state, and the algorithm needs to find a sequence of turning them on and
off to maximize the restored power while still satisfying physical
constraints.

The algorithm trains for 500 episodes and returns a cumulative reward curve
shown in Fig. 7. The cumulative reward represents the amount of reward that
agents can achieve during each episode over the learning course. As can be
seen, its value gradually increases until it reaches 12, which is the highest
reward they could earn. After 300 episodes, the cumulative reward curve
converges and the agents keep their same optimal policies.

Figure 7: Training curve of multi-agent RL IEEE 13-node system.

Fig. 8 shows the testing result after training. The sequential configuration
of circuit breakers is to restore loads 3, 7, 9, and 2 with the restored power
of 2563 kW. As the maximum available generator is 2600 kW, the models, after
being trained, can achieve up to 98.6% of restored power after three steps.

Figure 8: Testing of two-agent RL IEEE 13-node system.

###  III-B IEEE 123 Nodes

The algorithm is then expanded to validate the medium-size system of the IEEE
123-node distribution test feeder, as shown in Fig. 9. In this case, the
distribution system is divided into five microgrids corresponding to five
agents. There are a total of 26 circuit breakers distributed throughout the
system (10 circuit breakers in microgrid 1, 5 circuit breakers in microgrid 2,
3 circuit breakers in microgrid 3, 3 circuit breakers in microgrid 4, and 5
circuit breakers in microgrid 5) to manage 3025 kW power of loads. Note that
the number of circuit breakers in each microgrid is different depending on the
number of loads and total power of each microgrid. A total of five renewable
energy sources with a maximum capacity of 2400 kW are installed to provide
power to the system during the outage.

Figure 9: 123-node networked microgrids configuration.

The training curve of the IEEE 123-node system is shown in Fig. 10. As can be
seen, the cumulative reward converges (training curve) after 1500 episodes.
Similar to the IEEE 13-node system’s training curve, the cumulative reward
increases as the agents explore and gain knowledge of the environment. The
highest reward they can achieve in this setting is 37, with 7% standard
deviation. Using the trained models can help to recover up to 2305 kW of
restored power, which is equivalent to 96.04% of the 2400 kW total available
power.

Figure 10: Training curve of multi-agent RL IEEE 123-node system.

###  III-C IEEE 8500 Nodes

The last case study is conducted with an IEEE 8500-node distribution test
feeder, as shown in Fig. 11. The distribution system is divided into ten
microgrids with a total load of 4090 kW power. Multiple solar energy panels
are located along the system with a total capacity of 2400 kW. The total
number of circuit breakers being used in this system is 77, which is a very
large number. In this case, we tried to use a single agent to make decisions;
however, with the total action space being $2^{77}$, a single agent failed to
address the problem.

Figure 11: 8500-node networked microgrids configuration. Figure 12: Training
curve of multi-agent RL IEEE 8500-node system.

Fig. 12 shows the cumulative reward result after training ten agents in 2000
episodes. As can be seen, the cumulative reward starts from 63.75 and
gradually increases to 86.25, which is the optimal value of this setting. Fig.
13 shows the testing result for a total restoration of 2087.02 kW, which is
equivalent to 86.96% of the 2400 kW available power. Note that the available
power from generators needs to supply for both restored loads and losses in
the system.

Figure 13: Testing of ten-agent RL IEEE 8500-node system.

###  III-D Comparative Analyses

####  III-D1 Single-agent and multi-agent load restoration

In this subsection, in order to fairly compare single-agent and multi-agent
frameworks, all the common hyperparameters (ANN structure, mini-batch size,
training episodes, etc.) are the same.

Figure 14: 13-node single agent with and without action masking.

Fig. 14 shows the IEEE 13-node system results. For this small-size system, we
can see that both frameworks can converge and find the optimal solution.
Interestingly, the multi-agent can achieve better load pickup in each step,
and obtain the optimal value one step earlier than a single agent as shown in
Fig. 15. The underlying reason is that the multi-agent framework enables
better flexibility where each agent can control a small number of loads to be
picked up in its area, and hence, it can quickly find the load to match the
generation limit in its area. Whereas, the single agent has to control a much
bigger number of loads, and thus, it needs more time to find the optimal
solution.

Figure 15: Testing results of two IEEE 13-node systems.

####  III-D2 With and without invalid action masking technique

The next comparison is carried out to analyze the effectiveness of invalid
action masking in terms of dealing with safety constraints in reinforcement
learning. In the first case, the IEEE 13-node system is used to apply for
single agent with and without invalid action masking. As can be seen in Fig.
16, the single agent with the invalid action masking technique can achieve the
optimal reward faster (which can be clearly observed by the gap between the
two curves) and has 9.75% higher total cumulative reward than one without
action masking (pure single agent). Note that the standard deviation of a
single agent with invalid action masking is 57.32% smaller than a pure single
agent because the pure single agent fails to learn meaningful actions and
easily drops the training reward value as it violates the power flow
constraints.

Figure 16: 13-node single agent with and without invalid action masking.

In the second case, we expand the single-agent approach to an IEEE 123-node
system. As can be seen from the training curve shown in Fig. 17, although both
approaches still can converge after 1500 episodes, result for the approach
without invalid action masking can only reach a sub-optimal result (30)
compared to the invalid action masking approach (37). In terms of standard
deviation, the invalid action masking agent also has a lower value of 66%,
indicating that under the same settings, the invalid action masking technique
definitely helps reduce exploration time and stabilize the learning process.

In the third scenario, we test the performance of the invalid action masking
technique for a deep MARL framework using IEEE 123-node system. Fig. 18
depicts the critical importance of invalid action masking when the problem is
scaled up. As can be seen, the learning curve of five pure agents starts at
the value of 22 and doesn’t increase during the training phase. This implies
those agents cannot find any good policies given the fact that all other
settings are the same compared to the one with invalid action masking. The
high value of standard variance also dictates that the pure agents keep
violating the constraints even at the end of the training phase.

In the final scenario, we utilize the deep MARL framework that was previously
applied to the IEEE 123-node system and expand it to the IEEE 8500-node system
for comparison. As depicted in Fig. 19, for the implementation of ten agents
without invalid action masking, they faced difficulties collaborating and
violated the physical constraints. This led to an unconverged policy, unlike
the scenario where invalid action masking was used with the same settings.

Table I provides a comprehensive comparison of single-agent and multi-agent
approaches in terms of solution time and restored load under different system
sizes. The results demonstrate the unique effectiveness of our proposed
approach in improving the stability of learning convergence and the cumulative
reward in large-scale settings.

Figure 17: 123-node single agent with and without invalid action masking.
Figure 18: 123-node multi agent with and without invalid action masking.
Figure 19: 8500-node multi agent with and without invalid action masking.
TABLE I: Comparison of Solution Time and Restored Load With and Without
Invalid Action Masking. NC: Not converged

|  | Solution
---
Time (s)
| Restored
---
Load (%)
| Single
---
Agent
| Multi
---
Agent
| Single
---
Agent
| Multi
---
Agent
IEEE 13-node | Without Mask | 0.43 | 0.35 | 98.58 | 98.58
With Mask | 0.41 | 0.30 | 98.58 | 98.58
IEEE 123-node | Without Mask | 2.43 | NC | 91.88 | NC
With Mask | 2.17 | 1.00 | 96.04 | 96.04
IEEE 8500-node | Without Mask | NC | NC | NC | NC
With Mask | 12.27 | 2.26 | 64.71 | 86.96

####  III-D3 Limitations

Although our experimental results have demonstrated promising performance,
there are some limitations that need to be further investigated. (1) Since the
input and output of the model after training are fixed, the model cannot be
reused in case additional circuit breakers are installed that modify the
structure of the distribution system. (2) Although the algorithm can execute
the testing configuration of circuit breakers almost immediately, the training
process may take a long time because agents need sufficient time to explore
the environment, and the training time increases as the number of agents
grows. This training time drawback may be reduced by leveraging the parallel
training feature that most reinforcement learning platforms provide [67, 68,
69]. Since reinforcement learning emerged only recently for addressing
problems in power systems, those limitations can become subjects for further
study.

##  IV Conclusion

In this work, we proposed a multi-agent deep reinforcement learning approach
to improve the resilience of distribution systems after a power outage. The
load restoration agents were trained offline in a centralized way to have
better information. Then the trained agents were used at execution time in a
decentralized manner. The invalid action masking technique was used to
guarantee the power flow constraints both in the training and testing phases.
The experimental results show that agents can collaborate to achieve the
highest reward and the learning curve is stable during the training phase. Two
major disadvantages were pointed out, namely generalization and adaptability;
those issues will be our follow-up research.

## References

  * [1] J. W. Busby, K. Baker, M. D. Bazilian, A. Q. Gilbert, E. Grubert, V. Rai, J. D. Rhodes, S. Shidore, C. A. Smith, and M. E. Webber, “Cascading risks: Understanding the 2021 winter blackout in texas,” _Energy Research & Social Science_, vol. 77, p. 102106, 2021.
  * [2] J. Li, X.-Y. Ma, C.-C. Liu, and K. P. Schneider, “Distribution system restoration with microgrids using spanning tree search,” _IEEE Transactions on Power Systems_ , vol. 29, no. 6, pp. 3021–3029, 2014.
  * [3] H. Gao, Y. Chen, Y. Xu, and C.-C. Liu, “Resilience-oriented critical load restoration using microgrids in distribution systems,” _IEEE Transactions on Smart Grid_ , vol. 7, no. 6, pp. 2837–2848, 2016.
  * [4] F. Wang, C. Chen, C. Li, Y. Cao, Y. Li, B. Zhou, and X. Dong, “A multi-stage restoration method for medium-voltage distribution system with dgs,” _IEEE Transactions on Smart Grid_ , vol. 8, no. 6, pp. 2627–2636, 2017.
  * [5] A. Arif, S. Ma, Z. Wang, J. Wang, S. M. Ryan, and C. Chen, “Optimizing service restoration in distribution systems with uncertain repair time and demand,” _IEEE Transactions on Power Systems_ , vol. 33, no. 6, pp. 6828–6838, 2018\.
  * [6] A. Arif, B. Cui, and Z. Wang, “Switching device-cognizant sequential distribution system restoration,” _IEEE Transactions on Power Systems_ , vol. 37, no. 1, pp. 317–329, 2022.
  * [7] K. Zou, G. Mohy-ud din, A. P. Agalgaonkar, K. M. Muttaqi, and S. Perera, “Distribution system restoration with renewable resources for reliability improvement under system uncertainties,” _IEEE Transactions on Industrial Electronics_ , vol. 67, no. 10, pp. 8438–8449, 2020.
  * [8] Z. Wang, C. Shen, Y. Xu, F. Liu, X. Wu, and C.-C. Liu, “Risk-limiting load restoration for resilience enhancement with intermittent energy resources,” _IEEE Transactions on Smart Grid_ , vol. 10, no. 3, pp. 2507–2522, 2019.
  * [9] J. C. López, J. F. Franco, M. J. Rider, and R. Romero, “Optimal restoration/maintenance switching sequence of unbalanced three-phase distribution systems,” _IEEE Transactions on Smart Grid_ , vol. 9, no. 6, pp. 6058–6068, 2018.
  * [10] F. Shen, Q. Wu, J. Zhao, W. Wei, N. D. Hatziargyriou, and F. Liu, “Distributed risk-limiting load restoration in unbalanced distribution systems with networked microgrids,” _IEEE Transactions on Smart Grid_ , vol. 11, no. 6, pp. 4574–4586, 2020.
  * [11] S. Khushalani, J. M. Solanki, and N. N. Schulz, “Optimized restoration of unbalanced distribution systems,” _IEEE Transactions on Power Systems_ , vol. 22, no. 2, pp. 624–630, 2007.
  * [12] M. Ott, M. AlMuhaini, and M. Khalid, “A milp-based restoration technique for multi-microgrid distribution systems,” _IEEE Access_ , vol. 7, pp. 136 801–136 811, 2019.
  * [13] B. Chen, Z. Ye, C. Chen, and J. Wang, “Toward a milp modeling framework for distribution system restoration,” _IEEE Transactions on Power Systems_ , vol. 34, no. 3, pp. 1749–1760, 2019.
  * [14] J. Li, Y. Xu, and Y. Wang, “Critical load restoration method for unbalanced distribution systems based on linear approximation,” _2018 IEEE 2nd International Electrical and Energy Conference (CIEEC)_ , pp. 70–75, 2018.
  * [15] G. Patsakis, D. Rajan, I. Aravena, and S. Oren, “Strong mixed-integer formulations for power system islanding and restoration,” _IEEE Transactions on Power Systems_ , vol. 34, no. 6, pp. 4880–4888, 2019.
  * [16] J. Zhao and H. Wang, “Optimization of uncertainty load restoration strategy based on assessment of conditional value-at-risk,” in _2016 IEEE PES Asia-Pacific Power and Energy Engineering Conference (APPEEC)_ , 2016, pp. 115–119.
  * [17] Y. L. Li, W. Sun, W. Yin, S. Lei, and Y. Hou, “Restoration strategy for active distribution systems considering endogenous uncertainty in cold load pickup,” _IEEE Transactions on Smart Grid_ , vol. 13, no. 4, pp. 2690–2702, 2022.
  * [18] X. Zhang, A. T. Eseye, B. Knueven, W. Liu, M. Reynolds, and W. Jones, “Curriculum-based reinforcement learning for distribution system critical load restoration,” _IEEE Transactions on Power Systems_ , pp. 1–13, 2022\.
  * [19] S. Yao, J. Gu, H. Zhang, P. Wang, X. Liu, and T. Zhao, “Resilient load restoration in microgrids considering mobile energy storage fleets: A deep reinforcement learning approach,” _2020 IEEE Power & Energy Society General Meeting (PESGM)_, pp. 1–5, 2020.
  * [20] S. Das, S. Bose, S. Pal, N. N. Schulz, C. M. Scoglio, and B. Natarajan, “Dynamic reconfiguration of shipboard power systems using reinforcement learning,” _IEEE Transactions on Power Systems_ , vol. 28, no. 2, pp. 669–676, 2013.
  * [21] S. Ghasemi, A. Khodabakhshian, and R.-a. Hooshmand, “New multi-stage restoration method for distribution networks with dgs,” _IET Generation, Transmission & Distribution_, vol. 13, no. 1, pp. 55–63, 2019. [Online]. Available: https://ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/iet-gtd.2018.5624
  * [22] Y. Jiang, S. Chen, C.-C. Liu, W. Sun, X. Luo, S. Liu, N. Bhatt, S. Uppalapati, and D. Forcum, “Blackstart capability planning for power system restoration,” _International Journal of Electrical Power & Energy Systems_, vol. 86, pp. 127–137, 2017.
  * [23] S. Pal, S. Bose, S. Das, C. Scoglio, B. Natarajan, and N. Schulz, “Shipboard power system reconfiguration using reinforcement learning,” _North American Power Symposium 2010_ , pp. 1–7, 2010.
  * [24] L. R. Ferreira, A. R. Aoki, and G. Lambert-Torres, “A reinforcement learning approach to solve service restoration and load management simultaneously for distribution networks,” _IEEE Access_ , vol. 7, pp. 145 978–145 987, 2019\.
  * [25] D. Ye, M. Zhang, and D. Sutanto, “A hybrid multiagent framework with q-learning for power grid systems restoration,” _IEEE Transactions on Power Systems_ , vol. 26, no. 4, pp. 2434–2441, 2011.
  * [26] M. J. Ghorbani, M. A. Choudhry, and A. Feliachi, “A multiagent design for power distribution systems automation,” _IEEE Transactions on Smart Grid_ , vol. 7, no. 1, pp. 329–339, 2016.
  * [27] F. S. Melo, “Convergence of q-learning: A simple proof,” _Institute Of Systems and Robotics, Tech. Rep_ , pp. 1–4, 2001.
  * [28] M. Gautam, N. Bhusal, and M. Benidris, “A deep reinforcement learning-based approach to post-disaster routing of movable energy resources,” in _2022 IEEE Industry Applications Society Annual Meeting (IAS)_ , 2022, pp. 1–6.
  * [29] Y. Du and D. Wu, “Deep reinforcement learning from demonstrations to assist service restoration in islanded microgrids,” _IEEE Transactions on Sustainable Energy_ , vol. 13, no. 2, pp. 1062–1072, 2022.
  * [30] X. Zhang, A. T. Eseye, B. Knueven, and W. Jones, “Restoring distribution system under renewable uncertainty using reinforcement learning,” in _2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)_ , 2020, pp. 1–6.
  * [31] J. C. Bedoya, Y. Wang, and C.-C. Liu, “Distribution system resilience under asynchronous information using deep reinforcement learning,” _IEEE Transactions on Power Systems_ , vol. 36, no. 5, pp. 4235–4245, 2021.
  * [32] Z. Jiandong, Y. Qiming, S. Guoqing, L. Yi, and W. Yong, “Uav cooperative air combat maneuver decision based on multi-agent reinforcement learning,” _Journal of Systems Engineering and Electronics_ , vol. 32, no. 6, pp. 1421–1438, 2021.
  * [33] T. Chu, J. Wang, L. Codecà, and Z. Li, “Multi-agent deep reinforcement learning for large-scale traffic signal control,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 3, pp. 1086–1095, 2020.
  * [34] V.-H. Bui, T.-T. Nguyen, and H.-M. Kim, “Distributed operation of wind farm for maximizing output power: A multi-agent deep reinforcement learning approach,” _IEEE Access_ , vol. 8, pp. 173 136–173 146, 2020.
  * [35] M. Jiang, T. Hai, Z. Pan, H. Wang, Y. Jia, and C. Deng, “Multi-agent deep reinforcement learning for multi-object tracker,” _IEEE Access_ , vol. 7, pp. 32 400–32 407, 2019.
  * [36] T. Qian, C. Shao, X. Li, X. Wang, Z. Chen, and M. Shahidehpour, “Multi-agent deep reinforcement learning method for ev charging station game,” _IEEE Transactions on Power Systems_ , vol. 37, no. 3, pp. 1682–1694, 2022.
  * [37] X. Liao, W. Li, Q. Xu, X. Wang, B. Jin, X. Zhang, Y. Wang, and Y. Zhang, “Iteratively-refined interactive 3d medical image segmentation with multi-agent reinforcement learning,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 9391–9399.
  * [38] H. Allioui, M. A. Mohammed, N. Benameur, B. Al-Khateeb, K. H. Abdulkareem, B. Garcia-Zapirain, R. Damaševičius, and R. Maskeliūnas, “A multi-agent deep reinforcement learning approach for enhancement of covid-19 ct image segmentation,” _Journal of Personalized Medicine_ , vol. 12, no. 2, 2022\.
  * [39] M. Tan, “Multi-agent reinforcement learning: Independent versus cooperative agents,” in _Proceedings of the Tenth International Conference on International Conference on Machine Learning_ , ser. ICML’93. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1993, p. 330–337.
  * [40] S. Iqbal and F. Sha, “Actor-attention-critic for multi-agent reinforcement learning,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 2961–2970.
  * [41] A. Sharma, D. Srinivasan, and A. Trivedi, “A decentralized multiagent system approach for service restoration using dg islanding,” _IEEE Transactions on Smart Grid_ , vol. 6, no. 6, pp. 2784–2793, 2015.
  * [42] M. Kamruzzaman, J. Duan, D. Shi, and M. Benidris, “A deep reinforcement learning-based multi-agent framework to enhance power system resilience using shunt resources,” _IEEE Transactions on Power Systems_ , vol. 36, no. 6, pp. 5525–5536, 2021.
  * [43] T. Zhao and J. Wang, “Learning sequential distribution system restoration via graph-reinforcement learning,” _IEEE Transactions on Power Systems_ , vol. 37, no. 2, pp. 1601–1611, 2022.
  * [44] H. Chen, F. S. Bresler, M. E. Bryson, K. Seiler, and J. Monken, “Toward bulk power system resilience: Approaches for regional transmission operators,” _IEEE Power and Energy Magazine_ , vol. 18, no. 4, pp. 20–30, 2020.
  * [45] P. Cicilio, D. Glennon, A. Mate, A. Barnes, V. Chalishazar, E. Cotilla-Sanchez, B. Vaagensmith, J. Gentle, C. Rieger, R. Wies, and M. Kapourchali, “Resilience in an evolving electrical grid,” _Energies_ , vol. 14, p. 694, 01 2021.
  * [46] J. Lu, J. Guo, Z. Jian, Y. Yang, and W. Tang, “Resilience assessment and its enhancement in tackling adverse impact of ice disasters for power transmission systems,” _Energies_ , vol. 11, p. 2272, 08 2018.
  * [47] F. H. Jufri, V. Widiputra, and J. Jung, “State-of-the-art review on power grid resilience to extreme weather events: Definitions, frameworks, quantitative assessment methodologies, and enhancement strategies,” _Applied Energy_ , vol. 239, pp. 1049–1065, 2019.
  * [48] Z. Bie, Y. Lin, G. Li, and F. Li, “Battling the extreme: A study on the power system resilience,” _Proceedings of the IEEE_ , vol. PP, pp. 1–14, 04 2017\.
  * [49] M. Panteli and P. Mancarella, “The grid: Stronger, bigger, smarter?: Presenting a conceptual framework of power system resilience,” _Power and Energy Magazine, IEEE_ , vol. 13, pp. 58–66, 06 2015.
  * [50] T. V. Vu, B. L. Nguyen, Z. Cheng, M.-Y. Chow, and B. Zhang, “Cyber-physical microgrids: Toward future resilient communities,” _IEEE Industrial Electronics Magazine_ , vol. 14, no. 3, pp. 4–17, 2020.
  * [51] M. Littman, “Markov decision processes,” in _International Encyclopedia of the Social & Behavioral Sciences_, N. J. Smelser and P. B. Baltes, Eds. Oxford: Pergamon, 2001, pp. 9240–9242.
  * [52] L. Buşoniu, R. Babuška, and B. De Schutter, _Multi-agent Reinforcement Learning: An Overview_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 183–221. [Online]. Available: https://doi.org/10.1007/978-3-642-14435-6_7
  * [53] R. C. Dugan and D. Montenegro, “The open distribution system simulator (opendss): Reference guide,” _Electric Power Research Institute (EPRI)_ , 2018.
  * [54] R. D. Zimmerman, C. E. Murillo-Sánchez, and R. J. Thomas, “Matpower: Steady-state operations, planning, and analysis tools for power systems research and education,” _IEEE Transactions on Power Systems_ , vol. 26, no. 1, pp. 12–19, 2011.
  * [55] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, pp. 529–533, 2015.
  * [56] A. Piché, V. Thomas, J. Marino, G. M. Marconi, C. Pal, and M. E. Khan, “Beyond target networks: Improving deep $q$-learning with functional regularization,” 2021.
  * [57] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013.
  * [58] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. Cambridge, MA, USA: A Bradford Book, 2018.
  * [59] H. Yoo, V. M. Zavala, and J. H. Lee, “A dynamic penalty function approach for constraint-handling in reinforcement learning,” _IFAC-PapersOnLine_ , vol. 54, no. 3, pp. 487–491, 2021, 16th IFAC Symposium on Advanced Control of Chemical Processes ADCHEM 2021.
  * [60] S. Huang and S. Ontañón, “Comparing observation and action representations for deep reinforcement learning in urts,” 2019.
  * [61] OpenAI, C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, F. Wolski, and S. Zhang, “Dota 2 with large scale deep reinforcement learning,” 2019. [Online]. Available: https://arxiv.org/abs/1912.06680
  * [62] S. Huang, S. Ontañón, C. Bamford, and L. Grela, “Gym-µrts: Toward affordable full game real-time strategy games research with deep reinforcement learning,” in _2021 IEEE Conference on Games (CoG)_ , 2021, pp. 1–8.
  * [63] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas, “Acme: A research framework for distributed reinforcement learning,” _arXiv preprint arXiv:2006.00979_ , 2020\.
  * [64] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. P. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver, “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” _Nature_ , pp. 1–5, 2019.
  * [65] S. Huang and S. Ontañón, “A closer look at invalid action masking in policy gradient algorithms,” _The International FLAIRS Conference Proceedings_ , vol. 35, may 2022.
  * [66] P. Radatz, E. Viana, and P. Londero, “Py_dss_interface 1.0.2 documentation.” [Online]. Available: https://py-dss-interface.readthedocs.io/
  * [67] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. Jordan, and I. Stoica, “RLlib: Abstractions for distributed reinforcement learning,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 3053–3062.
  * [68] J. K. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sulivan, L. Santos, R. Perez, C. Horsch, C. Dieffendahl, N. L. Williams, Y. Lokesh, R. Sullivan, and P. Ravi, “Pettingzoo: Gym for multi-agent reinforcement learning,” _arXiv preprint arXiv:2009.14471_ , 2020.
  * [69] J. Weng, M. Lin, S. Huang, B. Liu, D. Makoviichuk, V. Makoviychuk, Z. Liu, Y. Song, T. Luo, Y. Jiang, Z. Xu, and S. Yan, “Envpool: A highly parallel reinforcement learning environment execution engine,” 2022.