# A Hierarchical Deep Actor-Critic Learning Method for Joint Distribution
System State Estimation

Yuxuan Yuan, Kaveh Dehghanpour, Zhaoyu Wang, and Fankun Bu This work is
supported by the U.S. Department of Energy Office of Electricity Delivery and
Energy Reliability under DE-OE0000875 (Corresponding author: Zhaoyu Wang.) Y.
Yuan, K. Dehghanpour, Z. Wang, and F. Bu are with the Department of Electrical
and Computer Engineering, Iowa State University, Ames, IA 50011 USA (e-mail:
<EMAIL_ADDRESS>wzy@iastate.edu).

###### Abstract

Due to increasing penetration of volatile distributed photovoltaic (PV)
resources, real-time monitoring of customers at the grid-edge has become a
critical task. However, this requires solving the distribution system state
estimation (DSSE) jointly for both primary and secondary levels of
distribution grids, which is computationally complex and lacks scalability to
large systems. To achieve near real-time solutions for DSSE, we present a
novel hierarchical reinforcement learning-aided framework: at the first layer,
a weighted least squares (WLS) algorithm solves the DSSE over primary medium-
voltage feeders; at the second layer, deep actor-critic (A-C) modules are
trained for each secondary transformer using measurement residuals to estimate
the states of low-voltage circuits and capture the impact of PVs at the grid-
edge. While the A-C parameter learning process takes place offline, the
trained A-C modules are deployed online for fast secondary grid state
estimation; this is the key factor in scalability and computational efficiency
of the framework. To maintain monitoring accuracy, the two levels exchange
boundary information with each other at the secondary nodes, including
transformer voltages (first layer to second layer) and active/reactive total
power injection (second layer to first layer). This interactive information
passing strategy results in a closed-loop structure that is able to track
optimal solutions at both layers in few iterations. Moreover, our model can
handle the topology changes using the Jacobian matrices of the first layer. We
have performed numerical experiments using real utility data and feeder models
to verify the performance of the proposed framework.

###### Index Terms:

Actor-critic method, joint distribution system state estimation, distributed
PV generation, secondary distribution network

##  I Introduction

As more stochastic customer-owned distributed resources, such as photovoltaic
(PV) power generators, are connected to low-voltage (LV) secondary
distribution grids, an urgent need grows for accurate system monitoring [1].
Specifically, topological details of secondary networks and the real-time
measurements of customers have to be incorporated into distribution system
state estimation (DSSE) to accurately capture voltage fluctuations across LV
systems and quantify the impacts of these variations on medium-voltage (MV)
primary distribution feeders. Recent years have seen a rapid growth in the
deployment of smart meters, providing a good opportunity to achieve this [2].

###  I-A Literature Review and Challenges

Most existing works have provided DSSE solutions only in a disjoint manner
(i.e., by decoupling primary and secondary networks); these works can be
roughly categorized into two general groups: (1) Primary Grid DSSE: various
DSSE methods have been provided for MV primary distribution feeders, while
aggregating all LV resources at the secondary transformers and disregarding
the secondary grid topology and parameters [3, 4, 5, 6, 7, 8, 9, 10]. The
basic approach is to compensate for lack of a detailed secondary model in DSSE
by estimating LV network losses, which can then be used as pseudo-measurements
to revise measurement aggregation [11]. (2) Secondary Grid DSSE: Another group
of papers have explored DSSE techniques for LV secondary networks while
simplifying primary MV feeders [12, 13, 14, 15, 16, 17, 18]. Here, the primary
feeder has been generally modeled as a constant voltage source to which the
secondary network is connected to.

Figure 1: Reinforcement learning-aided hierarchical DSSE framework.

Due to their disjoint approaches towards system monitoring, previous works in
both groups can fail to accurately capture the potential mutual impacts of LV
and MV networks on each other; furthermore, the mutual impacts of several
neighboring secondary networks connected to the same primary feeder have not
been quantified. Disjoint DSSE solvers become untenable and less accurate as
conventional distribution systems move towards more active grids with higher
penetration of renewable resources that can cause multi-directional power flow
across the grid. Under this new situation, previous modeling assumptions, such
as constant voltage level in primary feeders, can become too strong.
Furthermore, the impact of secondary network topology on voltage fluctuations
at the grid-edge can no longer be ignored. To address these shortcomings of
previous works, it is imperative to devise a DSSE solution that is able to
jointly monitor primary and secondary networks. Few recent papers [19, 20]
have proposed distributed multi-level architectures for performing DSSE at LV
and MV levels. However, in these cases the DSSE algorithms only have an open-
loop one-directional flow of information from secondary to primary feeders
which can fail to accurately capture the mutual impacts of LV-MV and LV-LV
networks on each other, as the distribution grids become more active.

However, a joint approach towards DSSE faces several challenges: (1) a
traditional weighted least squares (WLS) DSSE solver [2] over a unified model
of all primary and secondary circuits can lead to computational blow-up due to
the extremely large size of joint primary-secondary systems, especially for
urban systems. Traditional state estimation methods can take considerable time
delays in practical systems, which does not truly reflect the current states
[21]. This lack of scalability contributes to unacceptable time delays in
obtaining system states and hinders online monitoring of distribution grids.
(2) Accurate parameters of secondary networks, such as branch impedance
values, can be unknown or highly uncertain in practice [22]. This limits the
ability of conventional WLS-based solvers to obtain reliable power flow-based
Jacobian matrices. (3) Primary and secondary networks have distinct parametric
characteristics. Compared to MV systems, the LV networks have higher R/X
values and typical branch impedance levels. This characteristic difference
between primary and secondary systems can lead to severe ill-conditioning of
traditional joint DSSE solvers [23].

###  I-B Overall Structure of the Proposed Hierarchical Joint DSSE Framework

To tackle these challenges, in this paper we have proposed a hierarchical
reinforcement learning-aided framework for joint DSSE over primary and
secondary distribution systems using smart meter data, as shown in Fig. 1.
This framework consists of two layers: at the first layer, a WLS-based branch
current state estimation (BCSE) algorithm is performed over the primary feeder
to obtain the states of the MV distribution network, i.e., real/imaginary
branch currents. At this layer, all the secondary circuits are treated as
aggregated nodes with net equivalent active/reactive power injections provided
by the second layer of the hierarchy. Since the WLS is performed only over the
primary feeder, it is computationally efficient. After obtaining the states of
the primary feeder, the solver passes down the estimated secondary transformer
nodal voltages to the second layer of the hierarchy.

At the second layer, a deep actor-critic (A-C) module [24] is trained for each
LV network of secondary transformers. The basic idea behind the A-C approach
is to estimate the states of secondary networks (i.e., secondary branch
currents) to minimize the residuals of customer smart meter voltage
measurements. Unlike WLS, the A-C modules leverage their past experiences to
adaptively improve their future performance and generalize to unseen
situations. Thus, the training process takes place offline and the A-C modules
are employed online to estimate network states. Thanks to the neural network
implementation of the A-C model, the online computation cost is several orders
of magnitude lower than that of the WLS method. Furthermore, to estimate the
secondary network states the A-C modules receive several inputs, including the
latest WLS-based estimated secondary transformers’ terminal voltages generated
at the first layer and smart meter measurement data. Unlike the previous
supervised learning-based works that only use the customer power data as the
input of the learning model [25], the voltage estimation at the service
transformer is included as the input to capture the mutual impacts of LV and
MV networks. The outputs of the second layer of the hierarchy, which are
passed back to the first layer, are the net injected active/reactive powers to
the primary feeder for each secondary transformer. These outputs are easily
determined using the A-C-based estimated states of secondary circuits. Hence,
the interaction between the two layers of the joint DSSE takes place at the
secondary nodes, where nodal voltage flows from the first layer to the second
layer and active/reactive power injections are passed in reverse. At each
iteration of this closed-loop interaction, each layer revises the states of
the network in response to the received inputs from the other layer.

Compared to previous works, our joint DSSE framework accurately quantifies the
mutual impacts of primary-secondary networks, and secondary-secondary networks
on each other, which is of practical importance in grids with high solar
penetration. Furthermore, using the proposed A-C method, utilities can achieve
a considerable speed-up in solving the joint DSSE in large-scale grids which
allows them to monitor the whole system in near real-time. The distributed
nature of the proposed framework allows for allocating the computational
burdens of DSSE among multiple A-C modules, which further reduces the
computation time. Moreover, compared to the traditional WLS-based method, our
learning-based framework eliminates the need for pseudo-measurements to avoid
the additional imputation error. Note that several previous papers have used
supervised learning models to fully replace optimization solutions for
estimating system states [25, 26]. However, instead of approximating the joint
distribution system state estimation directly with machine learning
techniques, our method only approximates the secondary-level estimation
process. Given that the secondary transformers are generally equipped with
protection devices, the topology of the secondary distribution systems are
constant111When an outage happens in a radial system, a protective device
isolates the faulted area along with the loads downstream of the fault
location (i.e., the whole secondary distribution systems. Hence, the proposed
method is able to handle the topology changes at the MV primary distribution
networks using the Jacobin matrices in the first layer. Another advantage of
the proposed hierarchical structure is that the A-C technique allows for
explicit learning of the uncertainty of LV networks’ states through parametric
probabilistic policy functions. This not only helps operators to compensate
for lack of accurate knowledge of secondary networks, but will also be
integrated into the WLS formulation of the primary feeder to enhance overall
monitoring accuracy.

Figure 2: Layer II: A-C-based DSSE for secondary circuits

The rest of the paper is organized as follows: in Section II, the technical
details of the proposed hierarchical joint DSSE are presented. In Section III,
the numerical results have been analyzed to verify the performance of the
joint DSSE method. In Section IV, paper conclusions are presented.

##  II Deep Actor-Critic Strategy for Joint DSSE

The goal of the proposed method is to provide an efficient model for improving
distribution system situational awareness. In general, our joint DSSE model
consists of two parts: an optimization-based solution that infers the system
states of the primary-level network, and a learning-based method that
estimates the customer-level states and provides a feedback to the first
model. In the following, we describe the proposed model in detail.

###  II-A Primary Network BCSE

At the first layer of the hierarchical joint DSSE, a WLS-based BCSE algorithm
is performed over the MV network to minimize the sum of squared residuals
($J$) [27]:

$\begin{split}\min_{\boldsymbol{x_{p}}}J&=(\boldsymbol{z_{p}}-\boldsymbol{h}(\boldsymbol{x_{p}}))^{\top}W(\boldsymbol{z_{p}}-\boldsymbol{h}(\boldsymbol{x_{p}}))\\\
&s.t.\ \ \ \boldsymbol{z_{p}}=\left[\begin{array}[]{c}\boldsymbol{z_{MV}}\\\
\boldsymbol{\hat{p}_{s}}\\\ \boldsymbol{\hat{q}_{s}}\end{array}\right]\\\
&W=\left[\begin{array}[]{ccc}W_{MV}&\boldsymbol{0}&\boldsymbol{0}\\\
\boldsymbol{0}&W_{p_{s}}&\boldsymbol{0}\\\
\boldsymbol{0}&\boldsymbol{0}&W_{q_{s}}\end{array}\right]\end{split}$ (1)

where, $\boldsymbol{x_{p}}$ is a vector denoting the primary network states,
including real/imaginary branch current values, $\boldsymbol{z_{p}}$ is a
vector containing the MV network sensor measurements ($\boldsymbol{z_{MV}}$),
including SCADA and $\mu$PMUs, and the estimated total active/reactive power
injections of secondary transformers ($\boldsymbol{\hat{p}_{s}}$,
$\boldsymbol{\hat{q}_{s}}$) that are provided by the second layer of the
hierarchy. $\boldsymbol{h}$ is the primary network measurement function which
maps state values to measurements. Furthermore, $W$ is a weight matrix that
represents the solver’s confidence level in each element of
$\boldsymbol{z_{p}}$. $W$ consists of sub-matrces $W_{MV}$, $W_{p_{s}}$, and
$W_{q_{s}}$ corresponding to $\boldsymbol{z_{MV}}$,
$\boldsymbol{\hat{p}_{s}}$, and $\boldsymbol{\hat{q}_{s}}$, respectively.
Here, $W_{MV}$ is determined by the nominal accuracy levels of MV network
sensors, e.g., the weight assigned to the measurements received from a
specific sensor is selected as the inverse of measurement error variance for
that sensor [27]. The elements of $W_{p_{s}}$, and $W_{q_{s}}$ are determined
by the estimated uncertainty of the secondary network states as elaborated in
Section II-B.

Given the formulation (1), the WLS-based solver performs the following steps
to estimate the states of the primary network:

  * •

Step I: Receive the latest values of $\boldsymbol{\hat{p}_{s}}$,
$\boldsymbol{\hat{q}_{s}}$, $W_{p_{s}}$, and $W_{q_{s}}$ from the second layer
of the hierarchy (see Section II-B).

  * •

Step II: Random state initialization ($\boldsymbol{x_{p}}[0]$, $k\leftarrow
1$).

  * •

Step III: At iteration $k$, update the measurement function Jacobian matrix,
$H$:

$H=\frac{\partial\boldsymbol{h}(\boldsymbol{x_{p}}[k-1])}{\partial\boldsymbol{x_{p}}}$
(2)

The elements of the Jacobian matrix for the BCSE method can be easily obtained
for arbitrary feeders with known topology according to previous works [27].

  * •

Step IV: Update the gain matrix, $G$:

$G(x)=H^{\top}(\boldsymbol{x_{p}}[k-1])WH(\boldsymbol{x_{p}}[k-1])$ (3)

  * •

Step V: Update the state values using the gain and Jacobian matrices to reduce
measurement residuals:

$\boldsymbol{x_{p}}[k]=\boldsymbol{x_{p}}[k-1]+G^{-1}H^{\top}W(\boldsymbol{z_{p}}-\boldsymbol{h}(\boldsymbol{x_{p}}[k-1]))$
(4)

  * •

Step VI: $k\leftarrow k+1$; go back to Step III until convergence, i.e.,
$||\boldsymbol{x_{p}}[k]-\boldsymbol{x_{p}}[k-1]||\leq\delta$, with $\delta$
being a user-defined threshold.

  * •

Step VII: Given the estimated values of the branches, perform a forward sweep
[27] to obtain the voltages of secondary transformers throughout the network.
Pass down the estimated voltage of the $n$’th secondary transformer ($V_{n}$)
to the corresponding A-C module in the second layer of the joint DSSE
hierarchy.

Compared to traditional state estimation solutions that use node voltages,
BCSE adopts branch current as state variables, which is a more natural way of
DSSE formulation for distribution systems [2]. The simplification of the
measurement functions helps improve the computation speed and memory usage.
Therefore, BCSE is more suitable for large-scale distribution grids.

###  II-B Reinforcement Learning-Aided State Estimation for Secondary Networks

The computational complexity of conventional WLS technique is determined by
the matrix inversion, which induces a complexity of $O(N^{3})$. Thus, running
a BCSE algorithm over the whole primary and secondary networks at the same
time is a computationally intensive task, especially for large-scale urban
systems. To solve this challenge, the second layer of the hierarchy is
designed with the objective of simplifying and speeding-up the joint DSSE
process to achieve near real-time monitoring. Inspired by the recent success
of machine learning techniques, we have leveraged a reinforcement learning
technique, A-C method. Accordingly, an A-C module is trained for each
secondary transformer as shown in Fig. 2. This module consists of two deep
learning components that are trained cooperatively: (1) the actor represents
the secondary state estimation policy function ($\pi_{n}$), which receives
external inputs for the $n$’th secondary circuit, including the smart meter
voltage/energy measurements ($\boldsymbol{z}_{s,n}$), and the estimated
transformer voltage from the first layer ($V_{n}$), and maps them to secondary
states, $\boldsymbol{x}_{s,n}$. Here, $\boldsymbol{x}_{s,n}$ are the
real/imaginary components of secondary circuit branch currents. To account for
the uncertainty of the renewable resources in the state estimation process,
this mapping is represented as a $D_{n}$-dimensional parametric multi-variate
Gaussian probability distribution function, where
$\boldsymbol{x}_{s,n}\in\mathbb{R}^{D_{n}}$:

$\begin{split}&\boldsymbol{x}_{s,n}\sim\pi_{n}(\boldsymbol{\mu_{n}},\Sigma_{n})\\\
&=\frac{1}{\sqrt{|\Sigma_{n}|(2\pi)^{D_{n}}}}e^{-\frac{1}{2}(\boldsymbol{x}_{s,n}-\boldsymbol{\mu_{n}})^{\top}\Sigma_{n}^{-1}(\boldsymbol{x}_{s,n}-\boldsymbol{\mu_{n}})}\end{split}$
(5)

where, $\boldsymbol{c}_{n}=[\boldsymbol{z}_{s,n}\ V_{n}]$, and
$\boldsymbol{\mu_{n}}$ and $\Sigma_{n}$ are the $n$’th secondary circuit state
mean vector and covariance matrix, respectively. In this paper, these two
statistical factors are parameterized using two deep neural networks (DNNs),
$\mathcal{A}_{\mu}$ and $\mathcal{A}_{\Sigma}$, with parameters
$\boldsymbol{\theta_{n}}$ and $\boldsymbol{\gamma_{n}}$:

$\boldsymbol{\mu}_{n}=\mathcal{A}_{\mu}(\boldsymbol{c}_{n}|\boldsymbol{\theta_{n}})$
(6)
$\Sigma_{n}=\mathcal{A}_{\Sigma}(\boldsymbol{c}_{n}|\boldsymbol{\gamma_{n}})$
(7)

Basically, parameters $\boldsymbol{\theta_{n}}$ and $\boldsymbol{\gamma_{n}}$
are the weight and biases assigned to the synapses in the DNNs, which need to
be learned. This enables the operator to accurately quantify, not only the
expected value of the secondary circuit states, but also their uncertainty,
which is a critical element in grids with high renewable penetration. (2) The
critic is a DNN denoted by $\mathcal{C}$ with parameters
$\boldsymbol{\alpha_{n}}$ for the $n$’th circuit, which quantifies how well
the actor is performing. In our problem, the critic tries to predict the
secondary network DSSE residuals based on the inputs to the second layer:

$\hat{r}_{n}=\mathcal{C}(\boldsymbol{c}_{n}|\boldsymbol{\alpha_{n}})$ (8)

where, $\hat{r}_{n}$ represents the approximate residuals; ideally, if the
critic has perfect performance, then, $\hat{r}_{n}=r_{n}$, meaning that the
predicted residuals are equal to the realized measurement measurement
residuals $r_{n}$.

Given the defined A-C modules, the computational process at the second layer
of the hierarchy consists of an online state estimation stage (A), which is
performed jointly with the first layer, and an offline parameter update stage
(B), which is confined to the second layer alone:

  * •

Stage A - [Online Joint DSSE]

  * •

Step A-I: Input the learned A-C parameters $\boldsymbol{\theta_{n}}$,
$\boldsymbol{\gamma_{n}}$, and $\boldsymbol{\alpha_{n}}$.

  * •

Step A-II: Receive the updated $V_{n}$ from the first layer, and construct the
external input vector, $\boldsymbol{c}_{n}$.

  * •

Step A-III: Construct the policy function $\pi_{n}$, according to (5), using
parameters $\boldsymbol{\theta_{n}}$ and $\boldsymbol{\gamma_{n}}$ and
external inputs $\boldsymbol{c}_{n}$.

  * •

Step A-IV: Sample secondary circuit states in real-time using the constructed
policy function, $\boldsymbol{x}_{s,n}\leftarrow\pi_{n}$.

  * •

Step A-V: Use generated states to perform a forward sweep [27] over the
secondary circuit to obtain the net active/reactive power injections at the
transformer node, $\hat{p}_{s,n}$ and $\hat{q}_{s,n}$, as follows:

$\hat{p}_{s,n}=V_{n}I_{Re,n}$ (9) $\hat{q}_{s,n}=V_{n}I_{Im,n}$ (10)

where, $I_{Re,n}\in\boldsymbol{x}_{s,n}$ and $I_{Im,n}\in\boldsymbol{x}_{s,n}$
are the estimated net real and imaginary components of $n$’th secondary
transformer.

  * •

Step A-VI: To construct $W_{p_{s}}$ and $W_{q_{s}}$, the variances of
$\hat{p}_{s,n}$ and $\hat{q}_{s,n}$ need to be obtained. Noting that the
uncertainty of LV circuits states are explicitly quantified by the covariance
matrix of the policy function, $\pi_{n}$, we have:

$\sigma^{2}_{p_{s,n}}=(V_{n})^{2}\Sigma_{I_{Re,n}}$ (11)
$\sigma^{2}_{q_{s,n}}=(V_{n})^{2}\Sigma_{I_{Im,n}}$ (12)

where, $\sigma^{2}_{p_{s,n}}$ and $\sigma^{2}_{q_{s,n}}$ are the variances of
the net active and reactive power for the $n$’th LV system, and
$\Sigma_{I_{Re,n}}$ and $\Sigma_{I_{Im,n}}$ are components of $\Sigma_{n}$
corresponding to the states $I_{Re,n}$ and $I_{Im,n}$, respectively. These
variables are determined using
$\mathcal{A}_{\Sigma}(\boldsymbol{c}_{n}|\boldsymbol{\gamma_{n}})$. Therefore,
the weights assigned to $p_{s,n}$ and $q_{s,n}$ in the WLS-based solver of
layer I are equal to $\sigma^{-2}_{p_{s,n}}$ and $\sigma^{-2}_{q_{s,n}}$,
respectively.

  * •

Step A-VII: Pass the net active/reactive power injection of all secondary
transformers to the first layer of the joint DSSE framework,
$\boldsymbol{\hat{p}_{s}}=[\hat{p}_{s,1},...,\hat{p}_{s,N}]$ and
$\boldsymbol{\hat{q}_{s}}=[\hat{q}_{s,1},...,\hat{q}_{s,N}]$. Go back to Step
A-II until $V_{n}$ is stabilized.

Figure 3: Temporal function of the proposed hierarchical joint DSSE.

  * •

Stage B - [Offline A-C Parameter Update]

  * •

Step B-I: After the online process has converged, re-sample states using the
latest policy function,
$\boldsymbol{x}_{s,n}\leftarrow\pi_{n}+\boldsymbol{u_{n}}$, where
$\boldsymbol{u_{n}}$ is a exploratory perturbation generated using a zero-mean
uniform distribution. This perturbation allows the A-C module to actively
search for potential improvements in the learned policy and escape local
minimums.

  * •

Step B-II: Estimate the secondary DSSE residuals from the critic, using
$\boldsymbol{c}_{n}$ and DNN parameters $\boldsymbol{\alpha_{n}}$, according
to (8).

  * •

Step B-III: Use generated state sample and the latest value of $V_{n}$ from
Step A-VII, to perform a forward sweep over the secondary circuit to obtain
the estimated voltages; use the estimated nodal voltages to obtain the
realized residual, $r_{n}$.

  * •

Step B-IV: Obtain the temporal difference error (TDE),
$\delta_{n}=r_{n}-\hat{r}_{n}$, and use it to update the parameters of the
critic:

$\boldsymbol{\alpha_{n}}\leftarrow\boldsymbol{\alpha_{n}}+l_{c}\delta_{n}\nabla_{\boldsymbol{\alpha_{n}}}\mathcal{C}(\boldsymbol{c}_{n})$
(13)

where, $l_{c}$ is a learning rate, and
$\nabla_{\boldsymbol{\alpha_{n}}}\mathcal{C}$ is the gradient of the critic
DNN with respect to its parameters. This computation is performed using back-
propagation over the DNN [24].

  * •

Step B-V: Update the parameters of the actor, using the TDE:

$\boldsymbol{\theta_{n}}\leftarrow\boldsymbol{\theta_{n}}+l_{a}\delta_{n}\boldsymbol{u_{n}}\nabla_{\boldsymbol{\theta_{n}}}\pi_{n}(\boldsymbol{c}_{n})$
(14)
$\boldsymbol{\gamma_{n}}\leftarrow\boldsymbol{\gamma_{n}}+l_{a}\delta_{n}\boldsymbol{u_{n}}\nabla_{\boldsymbol{\gamma_{n}}}\pi_{n}(\boldsymbol{c}_{n})$
(15)

with $l_{a}$ denoting the rate of policy learning. To obtain the gradient of
policy function with respect to DNN parameters,
$[\boldsymbol{\theta_{n}},\boldsymbol{\gamma_{n}}]$, chain rule is applied to
the two sets of parameters separately:

$\nabla_{\boldsymbol{\theta_{n}}}\pi(\boldsymbol{c}_{n})=\frac{\Sigma_{n}^{-1}(\boldsymbol{x}_{s,n}-\boldsymbol{\mu}_{n})}{\sqrt{|\Sigma_{n}|(2\pi)^{D_{n}}}}e^{-\frac{M}{2}}\nabla_{\boldsymbol{\theta_{n}}}\mathcal{A}_{\mu}(\boldsymbol{c}_{n})$
(16)
$\begin{split}&\nabla_{\boldsymbol{\gamma_{n}}}\pi(\boldsymbol{c}_{n})=\\\
&\frac{-\Sigma_{n}^{-1}(I-(\boldsymbol{x}_{s,n}-\boldsymbol{\mu}_{n})(\boldsymbol{x}_{s,n}-\boldsymbol{\mu}_{n})^{\top}\Sigma_{n}^{-1})e^{-\frac{M}{2}}}{2\sqrt{|\Sigma_{n}|(2\pi)^{D_{n}}}}\nabla_{\boldsymbol{\gamma_{n}}}\mathcal{A}_{\Sigma}(\boldsymbol{c}_{n})\end{split}$
(17)

where,
$M=(\boldsymbol{c_{n}}-\boldsymbol{\mu}_{n})^{\top}\Sigma_{n}^{-1}(\boldsymbol{c_{n}}-\boldsymbol{\mu}_{n})$
is an auxiliary matrix. Note that
$\nabla_{\boldsymbol{\theta_{n}}}\mathcal{A}_{\mu}$ and
$\nabla_{\boldsymbol{\gamma_{n}}}\mathcal{A}_{\Sigma}$ in (16) and (17) are
obtained using back-propagation over the two DNNs of the actor.

  * •

Step B-VI: Move to the next time-step; go back to Step A-I.

Fig. 3 shows the temporal functionality of the proposed A-C method. As can be
seen, the parameters of DNNs are updated and replaced across time steps, while
on the other hand, the bi-layer estimation takes place at each time step given
the latest values of parameters. This enables the hierarchical framework to
adapt to changes in the feeder across time, while offering fast real-time
monitoring capability to utilities.

Figure 4: Test feeder topology and secondary network examples

###  II-C Convergence Analysis

The two layers of our model continuously exchange boundary information,
including transformer voltages (first layer to second layer) and
active/reactive total power injection (second layer to first layer). A major
challenge in this model is to ensure the convergence of system monitoring,
especially at the earlier stage of training when unreliable estimates
generated by A-C modules may cause numerical instability for WLS. To avoid
this, we have designed a confidence weight-based strategy. The basic idea is
to integrate the TDE from the second layer (i.e., A-C modules) into the
confidence matrix of the first layer (i.e., WLS). The TDE is able to measure
how well the DNNs infer system states over time, which is a good metric for
determining the reliability of the estimated secondary network states.
Therefore, the A-C modules with lower TDE will receive higher confidence
weights at the WLS. It should be noted that the A-C modules will be pre-
trained using historical data, which further reduces the risk of numerical
instability during online estimation.

##  III Numerical Results

Figure 5: Comparison of estimated three-phase system states and real values.

(a) Voltage magnitude component error (b) Voltage phase component error

Figure 6: State estimation performance using the proposed reinforcement
learning-aided hierarchical DSSE model.

The proposed joint DSSE framework is verified using real SCADA/smart meter
data and MV-LV network OpenDSS models from an utility partner in the U.S. The
data includes customers’ energy/voltage measurements at the secondary
networks, and total primary feeder active/reactive power and substation
voltages. More details on the data are available online [28]. Since we have
real smart meter data for each customer, the transformer-level load
consumption is aggregated based on the corresponding customer-level smart
meter data. The test system consists of a 60-node 13.8 kV primary feeder and
44 secondary circuits with a total number of 238 customers, as shown in Fig.
4. In this figure, we have shown the topology of the primary feeder and two
exemplary secondary networks. Distributed solar resources are added to the
secondary networks to capture the impact of uncertain renewable resources on
DSSE. The penetration level of renewable power is $50\%$ with respect to the
long-term average peak load. The solar power data is adopted from [29]. To
validate our hierarchical reinforcement learning-aided DSSE framework, we have
assumed that $10\%$ of customers in this feeder have smart meters. To build
the training dataset for A-C method, the consumption data for the rest of
unobservable customers are randomly generated. The corresponding system states
are obtained using the OpenDSS. In DSSE, the maximum error values for the real
measurements is $3\%$. In this work, the hyperparameter set of the A-C modules
are calibrated by using the random search strategy [30]. As a result, the
three DNNs, $\mathcal{A}_{\mu}$, $\mathcal{A}_{\Sigma}$, and $\mathcal{C}$,
consist of 3 hidden layers of 10 neurons. The learning rates of actor and
critic, $l_{a}$ and $l_{c}$, are selected as 0.01.

The A-C module is trained for various secondary networks in parallel based on
the historical/simulation data222When the system is fully observable (i.e.,
smart meters are installed for each customer), we use the historical data to
train our model. If the system is partially observable, a parametric model is
used to learn the underlying marginal distribution of the customer load
consumption and then generate a training set based on this distribution. The
OpenDSS software is leveraged to obtain the system states. and tested using
the new data inquiry. After model training, Fig. 5 compares the estimated
primary-level distribution system states (i.e., branch current real and
imaginary parts) with the actual state values using the proposed method at a
specific time point. As is demonstrated in the figure, the outcome of our
method closely follows the underlying states. It should be noted that our test
network is a three-phase unbalanced distribution system and the phase
connections of customers are known. Furthermore, to validate the average
performance of the proposed method, we have tested our method over a long-term
period (more than 1500 time points). The error distribution of state
estimation is shown in Fig. 6. The Mean Absolute Percentage Error (MAPE)
criterion is used here to evaluate the accuracy of state estimation:

$M=\frac{100\%}{n_{s}}\sum_{t=1}^{n_{s}}|\frac{\hat{A(t)}-A(t)}{\hat{A(t)}}|$
(18)

where, $\hat{A(t)}$ and $A(t)$ are the actual state value and the estimated
value. As is demonstrated in these figures, the estimation errors for voltage
magnitude and phase angle are $1.1\%$ and $0.26\%$, respectively. These
results corroborate the satisfactory performance of the proposed model over
real data.

Figure 7: Sensitivity analysis: quantifying the impact of observability on
estimation accuracy.

(a) Probability density function of online action selection time (b)
Cumulative distribution function of online action selection time

Figure 8: Statistical results of online action selection time.

Even though the learning-based method can eliminate the need for pseudo-
measurement generation, the system observability (i.e., smart meter
penetration) still impacts its performance. To represent the sensitivity of
the state estimation accuracy to the system observability, Fig. 7 shows the
secondary-level state estimation accuracy of the proposed method under various
smart meter penetration levels. Moreover, in this figure, the accuracy of a
previous learning-based method is compared with our solution [25]. It can be
observed that the estimation accuracy decreases as the percentage of smart
meter penetration decreases due to the information loss. Thanks to its
hierarchical nature, our method outperforms the existing learning method at
all observability levels.

To ensure that the proposed method can provide real-time monitoring in
practice, we have tracked the computation time. Note that the case study is
conducted on a standard PC with an Intel(R) Xeon(R) CPU running at 3.70 GHz
and with 32.0 GB of RAM. Fig. 8 presents the computation time distribution of
the online action selection of A-C modules. Considering the uncertainty of the
computation speed, $3500$ Monte Carlo simulations have been performed. As
shown in the figure, the majority of online action time are concentrated
around $0.02$ second. Moreover, based on the cumulative distribution function
of online action time, almost $90\%$ of simulations have online action time
below $0.024$ seconds, thus ensuring real-time system monitoring. Moreover,
the computation time of the whole hierarchical framework is tested and
compared to the traditional DSSE method [31]. Fig. 9 shows the computation
time distributions of our proposed method and an existing monitoring model
[31] over a 60-node distribution network. In this case, our framework is able
to significantly improve the computation time by an average factor of $6$
times. It should be noted that our test system is a middle-size rural
distribution feeder that has a limited number of customers. Since the
computation burden of the optimization method grows exponentially, our
method’s improvements in computation time would be higher in large-scale urban
systems.

Figure 9: Computation time comparisons (the proposed method versus a previous
method).

##  IV Conclusion

In this paper, we have presented a reinforcement learning-aided hierarchical
DSSE solution to jointly monitor the primary and secondary networks. Compared
to previous works, the proposed solution is scalable to large grids and can
accurately capture the impact of volatile grid-edge renewable resources on
system states. Our model enables fast online estimation of secondary network
states, while allowing for offline evaluation and updates of DNNs. The
hierarchical joint DSSE method has been tested using real smart meter data and
models of distribution grids. It is observed that after the estimation policy
function is fully learned, the proposed method can accurately estimate the
primary and secondary system states. Moreover, the results show that this
solution is able to outperform previous monitoring methods in terms of
estimation accuracy and computation time.

## References

  * [1] V. Zamani and M. Baran, “Feeder monitoring for Volt/VAR control in distribution systems,” _In IEEE PES General Meeting (Conference & Exposition)_, pp. 1–5, Jul. 2014.
  * [2] K. Dehghanpour, Z. Wang, J. Wang, Y. Yuan, and F. Bu, “A survey on state estimation techniques and challenges in smart distribution systems,” _IEEE Trans. Smart Grid_ , vol. 10, no. 2, pp. 2312–2322, Mar. 2019.
  * [3] P. A. Pegoraro, A. Angioni, M. Pau, A. Monti, C. Muscas, F. Ponci, and S. Sulis, “Bayesian approach for distribution system state estimation with non-Gaussian uncertainty models,” _IEEE Trans. Instrum. Meas._ , vol. 66, no. 11, pp. 2957–2966, Nov. 2017.
  * [4] M. Pau, F. Ponci, A. Monti, S. Sulis, C. Muscas, and P. A. Pegoraro, “An efficient and accurate solution for distribution system state estimation with multiarea architecture,” _IEEE Trans. Instrum. Meas._ , vol. 66, no. 5, pp. 910–919, May 2017.
  * [5] J. Wu, Y. He, , and N. Jenkins, “A robust state estimator for medium voltage distribution networks,” _IEEE Trans. Power Syst._ , vol. 28, no. 2, pp. 1008––1016, May 2013.
  * [6] F. Therrien, I. Kocar, and J. Jatskevich, “A unified distribution system state estimator using the concept of augmented matrices,” _IEEE Trans. Power Syst._ , vol. 28, no. 3, pp. 3390––3400, Aug. 2013.
  * [7] A. Gomez-Exposito, C. Gomez-Quiles, and I. Dzafic, “State estimation in two time scales for smart distribution systems,” _IEEE Trans. Smart Grid_ , vol. 6, no. 1, pp. 421––430, Jan. 2015.
  * [8] B. P. Hayes, J. K. Gruber, and M. Prodanovic, “A closed-loop state estimation tool for MV network monitoring and operation,” _IEEE Trans. Smart Grid_ , vol. 6, no. 4, pp. 2116–2125, Jul. 2015.
  * [9] A. Al-Wakeel, J. Wu, and N. Jenkins, “State estimation of medium voltage distribution networks using smart meter measurements,” _Applied Energy_ , vol. 184, pp. 207––218, Dec. 2016.
  * [10] Y. Zhang, J. Wang, and Z. Li, “Interval state estimation with uncertainty of distributed generation and line parameters in unbalanced distribution systems,” _IEEE Trans. Power Syst._ , pp. 1–1, 2012.
  * [11] D. A. Haughton and G. T. Heydt, “A linear state estimation formulation for smart distribution systems,” _IEEE Trans. Power Syst._ , vol. 28, no. 2, pp. 1187––1195, May 2013.
  * [12] A. Angioni, T. Schlosser, F. Ponci, and A. Monti, “Impact of pseudo-measurements from new power profiles on state estimation in low-voltage grids,” _IEEE Trans. Instrum. Meas._ , vol. 65, no. 1, pp. 70–77, Jan. 2016.
  * [13] M. Huang, Z. Wei, M. Pau, F. Ponci, and G. Sun, “Interval state estimation for low-voltage distribution systems based on smart meter data,” _IEEE Trans. Instrum. Meas._ , vol. 68, no. 9, pp. 3090–3099, Sep. 2019.
  * [14] A. Abdel-Majeed and M. Braun, “Low voltage system state estimation using smart meters,” _In 47th International Universities Power Engineering Conference (UPEC)_ , pp. 1–6, Sep. 2012.
  * [15] A. Mutanen, S. Repo, P. Järventausta, A. Löf, and D. D. Giustina, “Testing low voltage network state estimation in RTDS environment,” _In IEEE PES ISGT Europe_ , pp. 1–5, Oct. 2013.
  * [16] M. Pertl, K. Heussen, O. Gehrke, and M. Rezkalla, “Voltage estimation in active distribution grids using neural networks,” _In IEEE Power and Energy Society General Meeting (PESGM)_ , pp. 1–5, Jul. 2016.
  * [17] R. Bessa, G. Sampaio, V. Miranda, and J. Pereira, “Probabilistic low-voltage state estimation using analog-search techniques,” _In Power Systems Computation Conference (PSCC)_ , pp. 1–7, Jun. 2018.
  * [18] D. Waeresch, R. Brandalik, W. H. Wellssow, J. Jordan, R. Bischler, and N. Schneider, “Linear state estimation in low voltage grids based on smart meter data,” _In IEEE Eindhoven PowerTech_ , pp. 1–6, Jun. 2015.
  * [19] M. Pau, E. Patti, L. Barbierato, A. Estebsari, E. Pons, F. Ponci, and A. Monti, “A cloud-based smart metering infrastructure for distribution grid services and automation,” _Sustainable Energy, Grids and Networks_ , vol. 15, pp. 14–25, Sep. 2018.
  * [20] ——, “Design and accuracy analysis of multilevel state estimation based on smart metering infrastructure,” _IEEE Trans. Instrum. Meas._ , vol. 68, no. 11, pp. 4300–4312, Nov. 2019.
  * [21] Y. Weng, R. Negi, C. Faloutsos, and M. D. Ilić, “Robust data-driven state estimation for smart grid,” _IEEE Trans. Smart Grid_ , vol. 8, no. 4, pp. 1956–1967, Jul. 2017.
  * [22] J. Peppanen, C. Rocha, J. A. Taylor, and R. C. Dugan, “Secondary low-voltage circuit models-how good is good enough?” _IEEE Trans. Ind. Appl._ , vol. 54, no. 1, pp. 150–159, Jan. 2018.
  * [23] A. Abdel-Majeed and M. Braun, “Low voltage system state estimation using smart meters,” _In 2012 47th International Universities Power Engineering Conference (UPEC)_ , pp. 1–6, 2012.
  * [24] I. Grondman, M. Vaandrager, L. Busoniu, R. Babuska, and E. Schuitema, “Efficient model learning methods for actor-critic control,” _IEEE Trans. Syst., Man, Cybern. B, Cybern_ , vol. 42, no. 3, pp. 591–602, Jun. 2012\.
  * [25] K. R. Mestav, J. Luengo-Rozas, and L. Tong, “Bayesian state estimation for unobservable distribution systems via deep learning,” _IEEE Transactions on Power Systems_ , vol. 34, no. 6, pp. 4910–4920, 2019.
  * [26] W. Zhou, O. Ardakanian, H. Zhang, and Y. Yuan, “Bayesian learning-based harmonic state estimation in distribution systems with smart meter and dpmu data,” _IEEE Transactions on Smart Grid_ , vol. 11, no. 1, pp. 832–845, 2020.
  * [27] M. E. Baran and A. W. Kelley, “A branch-current-based state estimation method for distribution systems,” _IEEE Trans. Power Syst._ , vol. 10, no. 1, pp. 483–491, Feb. 1995.
  * [28] F. Bu, Y. Yuan, Z. Wang, K. Dehghanpour, and A. Kimber. (2019) A time-series distribution test system based on real utility data. [Online]. Available: https://arxiv.org/abs/1906.04078.
  * [29] C. Holcomb. (2012) Pecan street inc.: A test-bed for nilm. [Online]. Available: https://www.pecanstreet.org/
  * [30] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” _Journal of Machine Learning Research_ , vol. 13, pp. 281–305, Feb. 2012\.
  * [31] H. Wang and N. N. Schulz, “A revised branch current-based distribution system state estimation algorithm and meter placement impact,” _IEEE Trans. Power Syst._ , vol. 19, no. 1, pp. 207–213, Feb. 2004.