# A Broad-persistent Advising Approach for Deep Interactive Reinforcement
Learning in Robotic Environments

Hung Son Nguyen, Francisco Cruz, and Richard Dazeley The authors are with the
School of Information Technology, Deakin University, Geelong, Australia
(e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep Reinforcement Learning (DeepRL) methods have been widely used in robotics
to learn about the environment and acquire behaviors autonomously. Deep
Interactive Reinforcement Learning (DeepIRL) includes interactive feedback
from an external trainer or expert giving advice to help learners choosing
actions to speed up the learning process. However, current research has been
limited to interactions that offer actionable advice to only the current state
of the agent. Additionally, the information is discarded by the agent after a
single use that causes a duplicate process at the same state for a revisit. In
this paper, we present Broad-persistent Advising (BPA), a broad-persistent
advising approach that retains and reuses the processed information. It not
only helps trainers to give more general advice relevant to similar states
instead of only the current state but also allows the agent to speed up the
learning process. We test the proposed approach in two continuous robotic
scenarios, namely, a cart pole balancing task and a simulated robot navigation
task. The obtained results show that the performance of the agent using BPA
improves while keeping the number of interactions required for the trainer in
comparison to the DeepIRL approach.

###### Index Terms:

Reinforcement learning, Deep reinforcement learning, Interactive reinforcement
learning, Persistent advice, Broad-persistent Advising

##  I Introduction

Robot development has achieved big steps of improvement and gains more
attention in recent years. This success does not only come from industrial
areas where robots are gradually replacing humans [1], but also known in the
domestic areas. Their presence in domestic environments is still limited,
mainly due to the presence of many dynamic variables [2] and safety
requirements [3]. Intelligence robots in the future should be able to know and
detect users, learn action objects, select opportunities, and learn to behave
in domestic scenarios. To successfully perform these complex tasks, robots
face many challenges such as pattern recognition, navigation and object
manipulation all in different environmental conditions. That is, robots in the
domestic environment need to be able to continuously acquire and learn new
skills.

Reinforcement Learning (RL)is a method used for a robot controller in order to
learn optimal policy through interaction with the environment, through trial
and error [4]. The use of RL in previous results shows that there is great
potential for using RL in robots [2]. Especially, Deep Reinforcement Learning
(DeepRL)has also achieved promising results in manipulation skills [5, 6], and
on how to grasp as well as legged locomotion [7]. However, there is an open
issue relating to the performance in the RL and DeepRL algorithms, which is
the excessive time and resources required by the agent to achieve acceptable
outcomes [8, 9]. The larger and complex the state space is, the more
computational costs will be spent to find the optimal policy.

Among of different approaches to speed up this process, there is one promising
method named Interactive Reinforcement Learning (IRL)that can improve
convergence speed and has showed its feasibility. IRL allows a trainer to give
advice or evaluate a learning agent’s behaviour [10], helps the agent to shape
exploration policy and reducing search space in the early stages. Combining
IRL with DeepRL gives a model of Deep Interactive Reinforcement Learning
(DeepIRL)which can be used in continuous space with improved learning speed
[11]. However, current techniques using DeepIRL allow trainers to evaluate or
recommend actions based only on the current state of the environment. The
advice from the trainer have discarded causes by the agent after a single use,
which leads to duplicate process at the same state for reuse.

This work introduces the Broad-persistent Advising (BPA)approach for DeepIRL
to provide the agent a method for information retention and reuse of previous
advice from a trainer. This approach includes two components: generalisation
and persistence. In this article, we used $k$-means algorithm and
Probabilistic Policy Reuse (PPR)for each component, respectively. Agents using
the BPA approach have better results than their non-using counterparts while
keeping the number of interactions required for the trainer.

##  II Related works

###  II-A Deep reinforcement learning

Reinforcement Learning (RL)is a branch of machine learning in which
artificially intelligent agents learn behaviours by interacting with their
surroundings [4]. Reinforcement learning tools learn through trial and error
by repeatedly interacting with the surrounding environment and learning which
actions do and which actions will not produce the expected results.

RL is appropriate for studying tasks that may be modelled as Markov Decision
Processes (MDP)[4]. An MDP is specified by the tuple ($S$,$A$,$T$,$R$,
$\gamma$), where $S$ is a finite set of states in the environment, $A$ is a
set of actions available in each state, $T$ is the transition function $T$ :
$S_{n}\times A\rightarrow S_{n+1}$, $R$ is the reward function $R$ : $S\times
A\rightarrow R$, and $\gamma$ is a discount factor which is $0\leq\gamma\leq
1$

In the RL setup, a machine learning algorithm-controlled agent observes a
state $s_{t}$ from its environment at timestep $t$. In state $s_{t}$, the
agent communicates with the environment by performing action $a_{t}$. Then the
agent moves to a new state $s_{t+1}$ and receive reward $r_{t+1}$ as feedback
from environment based on the previous state and the chosen action. Therefore,
the reward collected by policy $\pi$ at timestep $t$ is shown in Equation (1)

$r_{t}+\gamma r_{t+1}+\gamma^{2}r_{t+2}+...=\sum_{k=0}\gamma^{k}r_{t+k}$ (1)

where $r_{t}$ is the reward at timestep $t$. The discount rate $\gamma$ stands
for the importance of rewards in the future. The agent’s aim is to find out a
policy $\pi$ that maximises anticipated profit (reward).

In conventional RL algorithms, most of the time, are only considered MDP with
discrete states and actions space. However, in many real-world applications,
the state space is not really discrete, but rather a continuous domain [12,
13]. Therefore, to be usable in the continuous state space, neural networks
are also considered as function approximators that are especially useful in RL
when the state space or action space is too broad to fully comprehend [14].
Neural nets can discover ways to map states to values in this way. When the
problem state space is too big or considered as continuous space, we cannot
use a lookup table to store and update all possible states and actions. In
that case, one alternative is to train a neural network with samples from the
state and the environment and expect them to predict the value of the next
action as our target in RL. More formally, we use a neural network to
approximate the optimal action-value function which is the maximum sum of
rewards in Equation (2)

$Q^{*}(s_{t},a_{t})=\max_{\pi}\mathbb{E}[\sum_{0}\gamma^{k}r_{t+k}|s_{t}=s,a_{t}=a,\pi])$
(2)

A vast variety of recent advanced robot applications have been accomplished
using deep reinforcement learning to teach agent complex activities including
cube play [15], ambidextrous robot gripping [5], categorized objects [11] and
cleaning table task [16]. For instance, Cruz et al. [16] used an associative
neural architecture to learn the available action possibilities of agents with
the objects in the current context. Levine et al. [17] proposed a learning-
based approach to hand-eye coordination for robotic grasping from monocular
images using a large Convolution Neural Network (CNN) to learn the way to
grasp objects.

###  II-B Reinforcement learning with interactive feedback

In Interactive Reinforcement Learning (IRL), there is an external trainer
involved in the agent’s learning process [10]. Figure 1 depicts the IRL
solution, which includes a advisor who observes the learning process and
offers guidance on the way to improve decision-making [18]. The advisor can be
an expert human, or an artificial agent.

Figure 1: At state $s_{t}$, the agent performs action $a_{t}$ and obtaining
reward $r_{t+1}$ and reaching the next state $s_{t+1}$ in conventional RL. In
IRL model, the involvement of an external trainer gives the agent more options
to chose which action to perform next in the following iteration.

Adaptive agent behaviour is needed in domestic environments. IRL enables a
parent-like tutor to facilitate learning by providing useful guidance in some
particular situation, allowing the apprenticeship process to be accelerated
[19]. In contrast to an agent exploring completely autonomously, this makes
for a smaller search space and hence quicker learning of the mission [20].

When operating alone, the next step is chosen by selecting the better known
action at the current time, defined by the highest state-action pair. While
IRL accelerates the learning process by incorporating additional guidance into
the apprenticeship loop. Using IRL, a trainer with prior experience of the
target goal is required [21].

There is a difference between the two main methods dedicated to feedback
learning: reward shaping and policy shaping. While in the reward shaping,
external trainers can assess the quality of the actions performed by the RL
agent, as good or bad [21]. Using policy shaping, the actions proposed by the
RL agent can be replaced by more appropriate actions selected by the external
trainer before implementation [22].

An open problem that can significantly affect the agent’s performance is
inaccurate advice from the trainer [10], since lack of accuracy and repetitive
mistakes will result in a longer training time. Human advice, on the other
hand, is not 100% correct [23]. When an advisor gives so much guidance, the
agent will have limited experience in exploration because the trainer makes
almost all of the decisions [24]. To address the problem, a prior study [25]
applied to the agent a strategy of discarding or refusing advice after an
amount of time, endowing an agent with the ability to work with potentially
incorrect information.

##  III A broad-persistent advising approach

In this section, we give more details about the proposed Broad-persistent
Advising (BPA)approach that includes a generalisation model along with a
persistent approach. These details are described next.

###  III-A Persistent advice

A recent study [25] suggests a permanent agent that records each interaction
and the circumstances around particular states. The actions are re-picked when
the conditions are met again in the future. As a consequence, the
recommendations from the advisor are used more effectively, and the agent’s
performance improves. Furthermore, as the training step is no need to provide
advice for each repeated state, less interaction with the advisor is required.
However, in this experiment, we limit the research to keeping the same number
of interactions to the trainer to investigate the effectiveness of BPA
approach in continuous domain.

As aforementioned, there is an issue relating to inaccurate advice. After a
certain amount of time, a mechanism for discarding or ignoring advice is
needed. Probabilistic Policy Reuse (PPR)is a strategy for improving RL agents
that use advice [26]. Where various exploration policies are available, PPR
uses probabilistic bias to decide which one to choose, with the intention of
balancing between random exploration, the use of a guideline policy, and the
use of the existing policy.

Figure 2 denotes an example of IRL using PPR. The advising user has the
opportunity to engage with the agent at each time point. When there is an
interaction, the model is updated. At the time advice is firstly recommended,
it is assumed that the agent will carry it out the suggested action,
regardless of the setting of PPR. PPR is used the time step when the agent did
not receive advice from the trainer, which flow is denoted by red arrows.
First, the agent’s policy is examined to see whether any advice is applicable
to the existing state. If the current policy suggests an action, the action is
taken with the determined by the PPR selection policy.

PPR is used where an agent chooses an action in a time step where the user has
not recommended a prior action, which is denoted by red arrows. First, the
agent’s policy is examined to see whether any advice is applicable to the
existing state. If the current policy suggests a action, the action is taken
with the determined by the PPR selection policy.

Figure 2: Process flow of an interactive reinforcement learning agent using
PPR system. After receiving advice from the trainer, the agent will store the
action into the PPR model. In the iteration not receiving advice from the
trainer, the agent will check and reuse the old advice (red arrows) from the
past.

###  III-B Broad advice

To use PPR, we need a system to store the used pairs of state-action. When the
agent arrives at a certain state at a time step, agents using PPR need to
check with the system if this state has been suggested by the trainer in the
past. If there is advice in the memory of the model, the agent can use the
option to reuse the action. However, there is a problem when using PPR in
infinite domains. We cannot build a system that stores state-action pairs with
infinite state values. In addition, when the amount of state becomes too large
in space, which is equivalent to infinity, the possibility that agents revisit
exactly the same state will be very small. Therefore, building this model will
become cumbersome and inefficient in large spaces.

BPA includes a model for clustering states and then building a system for
cluster-action pairs instead of traditional state and action pairs. The
proposed model is shown in Figure 3. When the agent receives current state
information from the environment and it does not receive any advice from the
trainer, the agent will use PPR by injecting the state into the generalisation
model and defining its cluster. Then proceed to consider whether any advice
pertains to the current cluster. If there is an action recommend in the past,
the agent can reuse it with the PPR selection probability, or use default
action as $\epsilon$-greedy.

Figure 3: Broad advice transform continuous states into finite clusters.
Hence, the state-action pair becomes cluster-action used in the PPR model.

The generalisation model we use in this paper is the $k$-means algorithm.
$k$-means is one of the most popular clustering methods [27]. $k$-means is
simple to implement, and its complexity scales well with a higher number of
data. However, the user must decide on the number of clusters beforehand [28].
We used the elbow technique to specify the number of clusters [29]. It is the
visual graphic approach that was generated from the Sum Square Error (SSE)
computation. This technique is based on the idea that the number of clusters
should be chosen so that adding another cluster does not cause significantly
improved modeling. The early clusters will provide a lot of information, but
at some point, the marginal gain will drop drastically, giving the graph an
angle. At this angle, the correct $k$-number of clusters is determined, thus
called ”elbow criteria”.

##  IV Experimental environments

###  IV-A Cart pole gym environment

The deep reinforcement learning environment is implemented using the well-
known library of AI gym environments [30]. First, we build the Cart Pole
environment. In this environment, there is a pole that is attached to the
cart. The carriage can move by applying force to the left or right. The
purpose of this problem is to prolong the time while avoiding the pole falling
down. The terminal condition is that the pole deviates more than 15 degrees
from the vertical or the wagon moves 2.4 units from the center. The cart pole
MDP is defined as follow:

  * •

State: The state vector has a continuous representation with four attribute
which represent for cart position, cart velocity, pole angle and pole
velocity.

  * •

Action: The cart can perform two actions on the track: go to left or right.

  * •

Reward function: As long as the agent holds the pole in a vertical position, a
reward equal to 1 is awarded, and if it drops, or goes beyond the boundaries
of the track, the reward is equal to 0.

Figure 4 below denotes a graphic of the Cart Pole in the AI-gym environment.

Figure 4: A graphical representation of the Cart Pole environment. The goal is
to keep the pole balanced while applying forces to the carriage. The terminal
condition is that the pole deviates more than 15 degrees from the vertical or
the wagon moves 2.4 units from the center.

###  IV-B Domestic robot environment

Additionally, we also build an environment for domestic robots using Webots,
given the overall good performance shown previously [31]. In this environment,
the goal is to train the robot to go from the initial position to the target
position. Figure 5 denotes a graphic of our experimental environment in
Webots.

Figure 5: A example of Webots environment with initial position and final
position. The robot has goals to go from initial position to final position
while avoiding obstacles. The robot will be returned to its initial position
after any collision.

The robot is equipped with distance sensors on its left and right eyes. The
robot is completely unaware of its current position in the environment. The
robot can only choose one of three actions: go straight at 3m/s, turn left, or
turn right. At each step, the robot will be deducted 0.1 points if it uses the
action of turning left or right, no points will be deducted if it chooses to
go straight. This is to optimize the robot’s straight movement and avoid the
robot running in circles by turning left or right continuously. The robot is
equipped with a few touch sensors next to it, to detect the collision with the
environment. The robot will be returned to its initial position and receive
100 penalty points every time it collides on the way. The robot does not know
where the touch sensor is located relative to itself, the only information it
receives is whether it is a collision with obstacles or not. When the robot
goes to the finish position located in the lower right corner of the
environment, the robot is considered to complete the task and be rewarded with
1000 points.

To decide on the next action the robot will choose, the robot’s supervisor
will use the image taken from the top of the environment to enter the
Convolutional Neural Network (CNN)system to decide. The CNN system built in
this environment will be a system whose input is 64x64 image RGB channels.
This architecture is inspired by similar networks used in other DeepRL works
[11, 32]. In more detail, we use 4 kernels with size 8x8. The second layer is
8 kernels with size 4x4, and the last layer convolution is 16 kernels with
size 2x2. Following each convolution network layer is a 2x2 max-pooling layer.
Finally, there is a flatten and dense layer with 256 neurons fully connected
with the output layer. The network architecture is described in Figure 6.

Figure 6: CNN architecture with 64x64 RGB image as input, group of three
convolution layers, three max-pooling layers, two dense full connected layers,
and a softmax function at the output

The environment MDP is defined as follow:

  * •

State: RGB image size 64x64 taken from the top of the environment.

  * •

Action: Three actions: go straight at 3m/s, turn left, or turn right

  * •

Reward function: Turn left, right: -0.1; Go straight: 0; Collision: -100;
Reach to final position: 1000

###  IV-C Interactive feedback

While the interactive agent’s human-related approach to learning is one of its
greatest strengths, it may also be its greatest weakness [33, 34]. Advice with
good accuracy given in the proper time will help the agent a lot in speeding
up the speed of finding the optimal solution. However, in the case when the
agent only gives advice with low accuracy and in high frequency, that not only
does not help the agent but also brings it to a dead road and is much more
time-consuming than no interaction situation. Furthermore, human experiments
are costly, time-consuming, have problems of repeatability, and can be
difficult to recruit volunteers. Therefore, during the early stages of the
agent, we suggested that simulating human interactions would be much more
convenient.

To compare agent performance, information about interactions, agent steps,
rewards, and interactions are recorded. To identify the efficiency of BPA, we
need to test the experiment with three cases: No interactive action,
interactive actions without BPA, and interactive actions with BPA.

In addition, each use case of the simulated user will have different advice’s
accuracy and frequency. Frequency is the availability of the interaction of
the advisor at the given time step. The higher frequency, the advisor has more
rate for giving advice to the agent. Accuracy is a measure of the precision of
advice provided by an advisor. When the advisor’s accuracy is high, the action
would be proposed precisely as the advisor’s knowledge. On the contrary, the
action proposed is different from the advisor’s knowledge. The frequency and
accuracy of the real-world simulation were simulated using data from a human
test [23] and the article [25] which is described in Table I. The value of
frequency and accuracy of advice is beyond the scope of this study.

Each use case of the simulated user will have different advice’s accuracy and
frequency. Accuracy is a measure of the precision of advice provided by an
advisor. When the advisor’s precision is high, the action would be proposed
precisely as the advisor’s knowledge of the environment. On the contrary, the
advisor would propose not optimal action based on how it knows about the
environment. Frequency is the availability of the interaction of the advisor
at the given time step. The higher frequency, the advisor has more rate for
giving advice to the agent. Accuracy is a measure of the precision of advice
provided by an advisor. When the advisor’s accuracy is high, the action would
be proposed precisely as the advisor’s knowledge. On the contrary, the action
proposed is different from the advisor’s knowledge. Accuracy and frequency of
three kinds of agents are used with value described in Table I. Optimistic
simulated agents have 100% accurate advice and always provide advice on every
time step. Realistic simulated agents use accuracy and frequency value from
results in a human trial [23, 25]. The pessimistic value of frequency is 0%,
however, it works the same as in the case without interactive feedback.
Therefore, we use half of the realistic value for the case with the least
interaction of the advisor. The accuracy and frequency value advice is beyond
the scope of this study.

TABLE I: The three simulated users designed for the experiments. These users will are not intended to be compared against each other, rather than comparing with persistent counterpart Agent |  Frequency |  Accuracy
---|---|---
Pessimistic Advisor |  23.658% |  47.435%
Realistic Advisor |  47.316% |  94.87%
Optimistic Advisor |  100% |  100%

The corresponding frequency and accuracy values above are pessimistic value,
realistic value and optimistic value, respectively. The pessimistic value of
frequency is 0%, however it works the same as in the case without interactive
feedback. Therefore, we use half of the realistic value as the case with the
least interaction of the advisor.

Experiments are performed for each case and the indicator of how accumulated
reward can achieve the optimal policy will be recorded to compare the result
between many approaches. The more reward the agent takes, the better result of
the method is.

###  IV-D Generalise model and probabilistic policy reuse

Next, we demonstrate the use of and broad advice and persistent advice using
Probabilistic Policy Reuse (PPR). The flow of using PPR is depicted as shown
in Figure 7. Initially, the agent reuses the action using PPR with a certain
chance if the current state has been recommended by the trainer in the past.
At the current work, we use chance value at 80% which is also used in previous
research [25]. This probability decreases by 5% for each step. With the
remaining 20%, the greedy action policy is selected.

Figure 7: Flow of using broad-persistent advising. The agent will reuse
previously obtained advice with 80% chance (decays over time) and perform its
exploration policy for the remaining change (20%).

Algorithm 1 shows the process flow for selecting an action using BPA approach
to assist a learning agent.

Algorithm 1 Interactive reinforcement learning with a BPA

1:Built $k$-means model with states from trainer

2:Initialize environment selecting $s_{t}$

3:for all (episodes) do

4: repeat

5: if (have feedback) then

6: Get recommend action $a_{t}$

7: Get cluster $c_{t}$ by using $k$-means

8: Add pair ($c_{t}$, $a_{t}$) to ppr

9: else

10: if ($rand(0,1)<\epsilon$) then

11: Get $c_{t}$ by using $k$-means

12: if ($c_{t}$ is available in ppr) then

13: Get $a_{t}$ is reuse action from ppr

14: else

15: Random action $c_{t}$ from environment

16: end if

17: else

18: Choose action $a_{t}$ using $\pi$

19: end if

20: end if

21: Perform action $a_{t}$

22: Observe next state $s_{t+1}$

23: until ($s$ is terminal)

24: Update policy $\pi$

25:end for

The model will be tested with the following agent listed below:

  * •

Baseline Reinforcement Learning: The model will be trained basically and
collect information from the environment without using any interactive
feedback or evaluation from the trainer. It is used as a benchmark.

  * •

Non-persistent Reinforcement Learning: The agent is assisted by multiple type
of users with in mentioned before in Table I. After taking recommendation from
trainer and execute the action, the agent will discard the advice. When the
agent come to the similar state again in the future, it cannot recall the
previous recommendation and performs an $\epsilon$-greedy action instead.

  * •

Persistent Reinforcement Learning: This agent is supported by a trainer and
PPR system. The trainer can suggest an action in each time step for the agent
to take. If recommended, the learning agent will perform on that time step and
retain the recommendation for reusing when it visits the similar state in the
future. When an agent accesses similar state it has previously suggested, it
will perform that action with the probability determined by the PPR action
selection rate.

##  V Results

###  V-A Cart pole domain

In this section, we proceed to display the results three types of agents
proposed above, including: baseline RL for bench marking, non-persistent RL,
persistent RL. For agents of the type non-persistent RL and persistent RL we
conduct tests on different frequency and accuracy of feedback, called
optimistic user, realistic user and pessimistic user. The method for all the
agents are tested with the same hyper-parameters as follows: initial value of
$\epsilon$ = 1, $\epsilon$ decay rate of 0.99, learning rate $\alpha$ = 0.01,
and discount factor $\gamma$ = 0.99 during 500 episodes. To better display, we
computed the average value of the last 100 rewards instead of the current
episode reward. We inspired the idea done on this article with the same result
from cart pole environment [35].

The results obtained are shown in Figure 8. Optimistic, realistic, and
pessimistic agents are run for five times and are represented by red, green,
and blue lines respectively. The shaded area indicates the standard deviation
of the agent’s reward after multiple training. Overall, all interactive agents
outperformed the autonomous one (baseline RL in yellow), except the
pessimistic agents. Agents which receive advice from the instructor make fewer
mistakes, especially in the early stages of the learning process, and can
learn the task in fewer episodes. However, in this work we want to compare
pairs of non-persistent and persistent agents with the same style delivered
advice to verify if the BPA approach implementation is indeed effective.

(a) Optimistic agents

(b) Realistic agents

(c) Pessimistic agents

Figure 8: The comparison for persistent agents, non-persistent agents, and
baseline each kind of agent in deep reinforcement learning built with cart
pole environment. The shaded area indicates the deviation between the minimum
and maximum values of the agent value after multiple training.

The agents assisted by optimistic achieved the maximum score of DeepIRL
algorithms at very early after a few episodes. Because the trainer always
makes decisions for the agent (100%), and this decision is absolutely correct
(100%). In this experiment, the agents did not even have any chance to make
their own decisions or use PPR, the trainer made all decision.

On the contrary, agents supported by pessimistic users have different results,
but in fact, neither of them can solve the problem. On many runs in both non-
persistent and persistent cases, the agent failed to achieve convergence. Both
cases are considered worse than the baseline. This can be explained because
the accuracy of advice for pessimistic agents is only 23.658%

On the graph of the agents being helped by realistic trainers, we can see that
using PPR produces slightly better results than the non-using counterpart.
Persistent agents not only have a better initial reward but also can achieve
convergence results 100 episodes earlier than non-persistent agents. This
difference in learning rates is due to the fact that the agent retains and
reuses advice. In this experiment, the realistic agent has a 47.3% chance to
interact with the trainer and the agent will withhold or not withhold the
advice from the trainer depending on whether it is a persistent agent or non-
persistent agent. However, the persistent agent will retain and reuse the
advice with an 80% probability (decreasing over time) for any state in which
it has received the advice in the past. As long as the stored advice is
accurate enough, the persistent agent will learn faster because they use the
advice more often.

In this experiment, we focused on implementing persistent advice in a
continuous environment. The ratio of the number of interactions with our
trainers remains the same: for example 47.316% with the realistic agent. When
receiving advice from the trainer, the agent will always prioritise executing
this recommended action. Therefore, the number of interactions using the BPA
method is equivalent to not using it. Table II shows the average number and
percentage of interactions that occurred for each agent. Both non-persistent
and persistent agents use the interaction rate according to Table I. We can
see that the number of interactions of every pair of optimistic, realistic and
pessimistic agents are similar in the experiment.

TABLE II: The average number of interactions in experiment for each kind of agent, and the percent compare with total steps taken Agent | Interaction
---|---
| Non-persistent | Persistent
Optimistic Advisor | 99796 (100%) | 99846 (100%)
Realistic Advisor | 40976 (47.15%) | 41832 (47.1%)
Pessimistic Advisor | 18034 (23.62%) | 16685 (23.76%)

Figure 9(a) shows a graph using the elbow method to specify the number of
clusters as a parameter of $k$-means, using the data of 50000 states, the same
number as the experiment of cart pole, in the actual running environment. The
elbow method shows the best value for using k at the value 3. Figure 9(b)
displays the data distribution in cart position and cart velocity attributes
axes at the value k = 3.

(a) Elbow method

(b) Distribution

Figure 9: Total of squared distance for value k from 1-9 and distribution for
50000 states with value k = 3 at cart position and cart velocity attributes

###  V-B Webots domain

In this scenario, we focus only to examine the results for the realistic
agent, because this can be transferred to the real-world scenarios in a more
rational manner. The method is tested with the following hyperparameters:
initial value of $\epsilon$ = 1, $\epsilon$ decay rate of 0.99, learning rate
$\alpha$ = 0.01, and discount factor $\gamma$ = 0.99 during 500 episodes. We
use average value of the last 100 rewards instead of the current reward only.

The results obtained are shown in Figure 10. Non-persistent RL agent is shown
by a green dashed line while persistent RL agent is shown by green solid line.
Baseline RL is drawn with yellow line used for bench marking. Similar to the
cart pole environment, both agents supported by the trainer, regardless of
whether or not they used PPR, obtain better results than baseline RL. Then,
the persistent agent achieves convergence results slightly earlier than its
non-persistent counterpart. The trainer’s accuracy and frequency feedback are
used the same as in the cart pole environment, so the results are reflected
for use in the domestic robot environment as well and, not just the ideal
hypothetical environment like cart pole in AI gym.

Figure 10: Result for deep reinforcement learning with autonomous agent, non-
persistent agent and persistent agent built with Webots domestic robot
environment.

Table III shows the average number and percentage of interactions that
occurred for each agent. We can see that the number of interactions is similar
in the experiment.

Agent | Interaction
---|---
| Non-persistent | Persistent
Realistic Advisor | 9077(47.64%) | 8241(47.18%)
TABLE III: The average number of interactions in experiment and the percent
compare with total steps taken for realistic agent in Webots environment

Figure 11(a) shows a graph using the elbow method to specify the number of
clusters as a parameter of $k$-means, using the data of 50000 states in the
actual running environment. The elbow method shows the best value for using k
at the value 4. Figure 11(b) displays the data distribution in two axes of
distance sensor value at the value k = 4.

(a) Elbow method

(b) Distribution

Figure 11: Total of squared distance for value k from 1-9 and distribution for
50000 states with value k = 4 at two value attributes of distance sensor.

##  VI Conclusion & future work

In this work, we proposed BPA, a broad-persistent advising approach to
implement the use of PPR and generalised advice in continuous-state
environments. Moreover, we also performed a comparison between autonomous
DeepRL, DeepIRL without BPA approach and DeepIRL with BPA approach. Two
environments were tested to investigate the impact that BPA approach achieve
the measured performance.

Overall, results obtained show that the BPA approach with $k$-means as a
generalise model and PPR as a model of persistence performed slightly faster
when the advice is withheld. The more accuracy for the advice and the longer
time to retain it increase learning speed significantly. Our research shows
that the first step for applying PPR in a continuous state-space environment
is feasible and effective. In addition, $k$-means used as broad advice
inherits advantages based on its characteristic: runs fast, and scales very
well over a large state space. Therefore, it is very suitable for the model to
run in a real-world environment.

As future work, we are planning to further investigate the number of clusters
of the $k$-means model which is obtained using the elbow method. Currently, it
is quite small and, therefore, BPA is implemented to remember advice within a
few steps. If agents try to keep in a long time with a small number of
clusters will effectively affect the accuracy of the model suggested by the
PPR. The agent may find difficulties to achieve convergence results with many
incorrect suggested actions. A more in-depth survey of the generalisation
model is needed to get the best results for using PPR. The accuracy of the
generalisation model greatly affects the speed and convergence of the IRL
model. Additionally, we suggest reducing the number of interactions with the
trainer by reusing the action in the persistent model more often. When the
agent reaches a new state that is already in memory, the agent reuses the
recommended action immediately without interacting with the trainer. However,
this should only be done when we have a good enough generalisation model. In
addition, we plan to transfer and test the proposed approach on a real-world
scenario with human and robot interaction.

## References

  * [1] E. Dahlin, “Are Robots Stealing Our Jobs?” _Socius: Sociological Research for a Dynamic World_ , vol. 5, p. 237802311984624, 2019.
  * [2] F. Cruz, P. Wuppen, A. Fazrie, C. Weber, and S. Wermter, “Action Selection Methods in a Robotic Reinforcement Learning Scenario,” _2018 IEEE Latin American Conference on Computational Intelligence, LA-CCI 2018_ , 2019.
  * [3] T. S. Tadele, T. De Vries, and S. Stramigioli, “The safety of domestic robotics: A survey of various safety-related publications,” _IEEE Robotics and Automation Magazine_ , vol. 21, no. 3, pp. 134–142, 2014.
  * [4] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT Press, 2018.
  * [5] C. Wang, Q. Zhang, Q. Tian, S. Li, X. Wang, D. Lane, Y. Petillot, and S. Wang, “Learning mobile manipulation through deep reinforcement learning,” _Sensors (Switzerland)_ , vol. 20, no. 3, pp. 1–18, 2020.
  * [6] H. Nguyen and H. La, “Review of Deep Reinforcement Learning for Robot Manipulation,” _Proceedings - 3rd IEEE International Conference on Robotic Computing, IRC 2019_ , no. February, pp. 590–595, 2019.
  * [7] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to train your robot with deep reinforcement learning: lessons we have learned,” _International Journal of Robotics Research_ , pp. 1–24, 2021\.
  * [8] C. Millán, B. Fernandes, and F. Cruz, “Human feedback in continuous actor-critic reinforcement learning,” in _Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning ESANN_. ESANN, 2019, pp. 661–666.
  * [9] A. Ayala, C. Henríquez, and F. Cruz, “Reinforcement learning using continuous states and interactive feedback,” _ACM International Conference Proceeding Series_ , 2019.
  * [10] F. Cruz, S. Magg, C. Weber, and S. Wermter, “Training Agents With Interactive Reinforcement Learning and Contextual Affordances,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 8, no. 4, pp. 271–284, 2016.
  * [11] I. Moreira, J. Rivas, F. Cruz, R. Dazeley, A. Ayala, and B. Fernandes, “Deep reinforcement learning with interactive feedback in a human-robot environment,” _Applied Sciences (Switzerland)_ , vol. 10, no. 16, 2020.
  * [12] G. Dulac-Arnold, D. Mankowitz, and T. Hester, “Challenges of real-world reinforcement learning,” _arXiv preprint arXiv:1904.12901_ , 2019.
  * [13] C. C. Millán-Arias, B. J. Fernandes, F. Cruz, R. Dazeley, and S. Fernandes, “A robust approach for continuous interactive actor-critic algorithms,” _IEEE Access_ , vol. 9, pp. 104 242–104 260, 2021.
  * [14] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [15] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas _et al._ , “Solving rubik’s cube with a robot hand,” _arXiv preprint arXiv:1910.07113_ , 2019.
  * [16] F. Cruz, G. I. Parisi, and S. Wermter, “Learning contextual affordances with an associative neural architecture.” in _ESANN_ , 2016.
  * [17] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” _The International Journal of Robotics Research_ , vol. 37, no. 4-5, pp. 421–436, 2018.
  * [18] W. B. Knox and P. Stone, “Interactively shaping agents via human reinforcement: The tamer framework,” in _Proceedings of the fifth international conference on Knowledge capture_ , 2009, pp. 9–16.
  * [19] A. Bignold, F. Cruz, M. E. Taylor, T. Brys, R. Dazeley, P. Vamplew, and C. Foale, “A conceptual framework for externally-influenced agents: An assisted reinforcement learning review,” _Journal of Ambient Intelligence and Humanized Computing_ , pp. 1–24, 2021.
  * [20] F. Cruz, P. Wüppen, S. Magg, A. Fazrie, and S. Wermter, “Agent-advising approaches in an interactive reinforcement learning scenario,” in _2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_. IEEE, 2017, pp. 209–214.
  * [21] A. L. Thomaz, G. Hoffman, and C. Breazeal, “Real-time interactive reinforcement learning for robots,” in _AAAI 2005 workshop on human comprehensible machine learning_ , 2005.
  * [22] T. Cederborg, I. Grover, C. L. Isbell, and A. L. Thomaz, “Policy shaping with human teachers,” in _Twenty-Fourth International Joint Conference on Artificial Intelligence_ , 2015.
  * [23] A. Bignold, F. Cruz, R. Dazeley, P. Vamplew, and C. Foale, “Human engagement providing evaluative and informative advice for interactive reinforcement learning,” _arXiv preprint arXiv:2009.09575_ , 2020.
  * [24] M. E. Taylor, N. Carboni, A. Fachantidis, I. Vlahavas, and L. Torrey, “Reinforcement learning agents providing advice in complex video games,” _Connection Science_ , vol. 26, no. 1, pp. 45–63, 2014. [Online]. Available: https://doi.org/10.1080/09540091.2014.885279
  * [25] A. Bignold, F. Cruz, R. Dazeley, P. Vamplew, and C. Foale, “Persistent rule-based interactive reinforcement learning,” 2021.
  * [26] F. Fernández and M. Veloso, “Probabilistic policy reuse in a reinforcement learning agent,” in _Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems_ , 2006, pp. 720–727.
  * [27] G. Y. Park, H. Kim, H. W. Jeong, and H. Y. Youn, “A novel cluster head selection method based on k-means algorithm for energy efficient wireless sensor network,” in _2013 27th international conference on advanced information networking and applications workshops_. IEEE, 2013, pp. 910–915.
  * [28] T. S. Madhulatha, “An overview on clustering methods,” _arXiv preprint arXiv:1205.1117_ , 2012.
  * [29] H. Humaira and R. Rasyidah, “Determining the appropiate cluster number using elbow method for k-means algorithm,” _WMA-2, January_ , 2020.
  * [30] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [31] A. Ayala, F. Cruz, D. Campos, R. Rubio, B. Fernandes, and R. Dazeley, “A comparison of humanoid robot simulators: A quantitative approach,” in _2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_. IEEE, 2020, pp. 1–6.
  * [32] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [33] A. Bignold, F. Cruz, R. Dazeley, P. Vamplew, and C. Foale, “An evaluation methodology for interactive reinforcement learning with simulated users,” _Biomimetics_ , vol. 6, no. 1, p. 13, 2021.
  * [34] F. Cruz, S. Magg, Y. Nagai, and S. Wermter, “Improving interactive reinforcement learning: What makes a good teacher?” _Connection Science_ , vol. 30, no. 3, pp. 306–325, 2018. [Online]. Available: https://doi.org/10.1080/09540091.2018.1443318
  * [35] S. Kumar, “Balancing a cartpole system with reinforcement learning–a tutorial,” _arXiv preprint arXiv:2006.04938_ , 2020.

|  Hung Son Nguyen received the Bachelor of Science degree in field of
information technology and the Master of Science degree in computer science
from the University of Science - Vietnam National University, Ho Chi Minh,
Vietnam in 2010 and 2016, respectively. He is currently pursuing a master’s
degree in artificial intelligence at School of IT, Deakin University, Geelong,
Victoria, Australia from 2020. His current research interests include computer
vision, artificial neural networks, deep learning, reinforcement learning and
human-robot interaction.
---|---
|  Francisco Cruz received the bachelor’s degree in engineering and the
master’s degree in computer engineering from the University of Santiago,
Chile, in 2004 and 2006, respectively, and the Ph.D. degree from the
University of Hamburg, Germany, in 2017, working in developmental robotics
focused on interactive reinforcement learning. In 2015, he was a Visiting
Researcher with the Emergent Robotics Laboratory, Osaka University. He joined
the Engineering School, Universidad Central de Chile, as a Research and
Teaching Associate, in 2017, and the School of IT, Deakin University, as a
Research Fellow, in 2019. His current research interests include reinforcement
learning, explainable artificial intelligence, human–robot interaction,
artificial neural networks, and psychologically and bio-inspired models.
---|---
|  Richard Dazaley received the Ph.D. degree in computer science from the
University of Tasmania, in 2007. He is currently an Associate Professor of
computer science with Deakin University, Geelong, where he is also the Deputy
Leader of the Machine Intelligence Lab. He is widely recognized for his
pioneering work in multiobjective reinforcement learning publishing several
highly cited papers that helped established the field. He has also organized
multiple workshops in the field. More recently, he has been interested in
applying reinforcement learning and multiobjective principles in interactive,
safe, explainable, and secure systems. He is also interested in prudence
analysis, natural language processing, data analytics, and security. He has
received multiple awards including the Australian Educator of the Year from
the Australian Computer Society, in 2016
---|---


# Information Structures in AC/DC Grids

Josh A. Taylor J. A. Taylor is with the Department of Electrical and Computer
Engineering, New Jersey Institute of Technology, Newark, New Jersey, 07102,
United States<EMAIL_ADDRESS>
###### Abstract

The converters in an AC/DC grid form actuated boundaries between the AC and DC
subgrids. We show how in both simple linear and balanced dq-frame models, the
states on either side of these boundaries are coupled only by control inputs.
This topological property imparts all AC/DC grids with poset-causal
information structures. A practical benefit is that certain decentralized
control problems that are hard in general are tractable for poset-causal
systems. We also show that special cases like multi-terminal DC grids can have
coordinated and leader-follower information structures.

###### Index Terms:

AC/DC grid; information structure; poset causality; decentralized control;
multi-terminal direct current grid

##  I Introduction

AC/DC grids consist of several AC and DC subgrids interfaced by power
electronic converters. For example, in a point-to-point DC link, the DC
subgrid is a single line, which is interfaced with one or two AC subgrids
through a pair of converters. Multi-terminal DC (MTDC) grids interconnect
several AC subgrids through voltage-sourced converters (VSCs) [1]. In any
AC/DC grid, the converters comprise actuated boundaries between the AC and DC
subgrids. In this paper, we study how this topological property determines the
information structure of an AC/DC grid.

The information structure of a dynamical system encodes which states influence
which other states. It is natural to think in terms of subsystems—the
information structure of an AC/DC grid specifies how each AC or DC subgrid
influences the others. The main result of this paper is that AC/DC grids have
poset-causal information structures [2]. This is because the structure of the
coupling between the subgrids can be chosen to be a directed acyclic graph
(DAG). One reason this is useful is that poset-causal systems admit tractable,
optimal decentralized controllers [2]. Coordinated systems [3] and leader-
follower systems [4] are successive special cases that admit even simpler
decentralized controllers. An interesting question is whether these
information structures make AC/DC grids amenable to other specialized tools,
e.g., via controllability or observability [5, 6].

To date, few studies have made explicit use of the network structure of
general AC/DC grids. There are a handful of papers that design distributed
controllers for MTDC grids [7, 8, 9]. Reference [10] relates the
controllability of systems with point-to-point DC links to the effective
reactance of the AC grid. Reference [11] designs local controllers for general
AC/DC grids; whereas we focus on information structure, they focus on specific
control objectives and stability. The most closely related paper to the
present is the author’s prior work [12], which showed that DC-segmented power
systems are poset-causal.

Our original results are as follows. The poset-causality of an AC/DC grid
depends on how each VSC partitions the states of the AC and DC subgrids on
either side; we define this precisely in Section III-B. We see that this
partitioning occurs in a simple linear model in Section IV. In Section V, we
show how different physical approximations each lead to partitioning in a
standard dq-frame model. In Section VI, we show that in both the linear and
dq-frame models, the partitions can be chosen so as to yield poset-causality;
this is due to the fact that an undirected graph always has an acyclic
orientation [13]. We also show that a single DC subgrid connected to multiple
AC subgrids, i.e., an MTDC grid, is a coordinated system; and a single AC
subgrid connected to a single DC subgrid is a leader-follower system. In
Section VII-B, we use concepts from [3] to design a decentralized controller
for an MTDC grid with an additional point-to-point DC link between two of the
AC subgrids.

##  II Poset-causality

A poset $\Psi$ is made up of a set P and a binary relation $\preceq$ [14]. The
following properties hold for all $a,b,c\in\textrm{P}$.

  * •

Reflexivity: $a\preceq a$;

  * •

Antisymmetry: $a\preceq b~{}\textrm{and}~{}b\preceq a\Rightarrow a=b$;

  * •

Transitivity: $a\preceq b~{}\textrm{and}~{}b\preceq c\Rightarrow a\preceq c$.

The following two results relate graphs and posets.

###### Result 1

Every DAG specifies a unique poset [15].

###### Result 2

Given a simple undirected graph, we can choose the directions of its edges so
that the resulting directed graph is acyclic [13]. This is called an acyclic
orientation.

Given $a\in\textrm{P}$, we denote the set of upstream elements $\uparrow
a=\\{b\in\textrm{P}\;|\;b\preceq a\\}$. The relations between elements of a
poset can be encoded in a function
$\sigma:\textrm{P}\times\textrm{P}\to\mathbb{R}$ such that $\sigma(a,b)=0$
when $a\not\preceq b$. Its incidence algebra, $\Psi$, is the set of all such
functions. If P has a finite number of elements, then for any $\sigma\in
I(\Psi)$, there is a matrix $M$ for which $M(j,i)=\sigma(g(i),g(j))$, where
$g:\mathbb{N}\to\textrm{P}$ maps row and column indices to elements of P. With
a slight abuse of notation we write $M\in I(\Psi)$.

###  II-A Nonlinear systems

Consider the system

$\displaystyle\dot{x}=f(x,u),\quad z=h(x,u),$ (1)

where $x\in\mathbb{R}^{n}$, $u\in\mathbb{R}^{m}$, and $z$ are states, inputs,
and outputs.

Suppose the system consist of $p$ subsystems. We partition $x$ into
$[x_{1};x_{2};\dots;x_{p}]$, where $x_{i}\in\mathbb{R}^{n_{i}}$ and
$\sum_{i}n_{i}=n$. $x_{i}$ are the states of subsystem $i$. We similarly
partition the inputs $u\in\mathbb{R}^{m}$ into $[u_{1};u_{2};\dots;u_{p}]$,
where $u_{i}\in\mathbb{R}^{m_{i}}$ and $\sum_{i}m_{i}=m$.

Suppose that we have a poset, $\Psi=(\textrm{P},\preceq)$, and that its
elements are the subsystems, $\textrm{P}=\\{1,...,p\\}$. System (1) is poset-
causal if we can write it as

$\displaystyle\dot{x}_{i}=f_{i}\left((x_{j},u_{j})_{j\in\uparrow
i}\right),\quad z_{i}=h_{i}\left((x_{j},u_{j})_{j\in\uparrow i}\right),\quad
i\in\textrm{P}.$

Intuitively, each subsystem’s state and output depend only on upstream
subsystems.

###  II-B LTI systems

Consider the LTI system

$\displaystyle\dot{x}=Ax+Bu+Fw,\quad z=Cx+Du,$ (2)

where $w$ is a disturbance. We assume $C^{\top}D=0$, $C^{\top}C$ is positive
semidefinite, $D^{\top}D$ is positive definite, and $F$ is block diagonal.

We write the matrix $A$ as $[A_{ij}]_{i,j\in\\{1,\cdots,p\\}}$, where $A_{ij}$
is the block indexed by the $i^{\textrm{th}}$ and $j^{\textrm{th}}$ partitions
of $x$. Matrices $B,F,C$ and $D$ can be similarly organized into blocks.

Consider a poset $\Psi$, and suppose $\sigma\in I(\Psi)$. The matrix $A$
belongs to the block incidence algebra $I_{A}(\Psi)$ if $A_{ji}=\bm{0}$
whenever $\sigma(g(i),g(j))=0$, where $\bm{0}$ is the appropriately sized
matrix of zeroes. A similar definition holds for $I_{B}(\Psi)$.

Let $\mathcal{A}=(sI-A)^{-1}$, and define the transfer matrices

$P_{11}=C\mathcal{A}F,\;P_{12}=C\mathcal{A}B+D,\;P_{21}=\mathcal{A}F,\;P_{22}=\mathcal{A}B.$

We can express LTI system (2) as

$\displaystyle z=P_{11}w+P_{12}u,\quad x=P_{21}w+P_{22}u.$

System (2) is poset-causal if $P_{22}\in I_{P_{22}}(\Psi)$ for some poset
$\Psi$. The following result from [2] directly links poset causality to the
system matrices.

###### Result 3

If $A\in I_{A}(\Psi)$ and $B\in I_{B}(\Psi)$, then $P_{22}\in
I_{P_{22}}(\Psi)$.

In other words, (2) is poset-causal if $A$ and $B$ are in the block incidence
algebra of a poset.

Given a controller $u=Kx$, the transfer matrix from the disturbance, $w$, to
output, $z$, is

$\displaystyle T_{zw}=P_{11}+P_{12}K(I-P_{22}K)^{-1}P_{21}.$ (3)

Reference [2] solves the problem of minimizing the $\mathcal{H}_{2}$ norm of
$T_{zw}$ over $K\in I_{K}(\Psi)$, i.e., controllers that are decentralized
because they are in the same poset as (2). We describe this further in Section
VII-A.

###  II-C Special cases

The following are special cases of poset-causal systems. We refer the reader
to [3] for a more detailed summary.

####  II-C1 Hierarchical systems

A system is hierarchical if its graph is a single, directed tree [16].

####  II-C2 Coordinated systems

Coordinated systems are hierarchical systems of depth one [3]. They consist of
a coordinator, $c\in\textrm{P}$, and subsystems, $i\in\downarrow c\setminus
c$, for which $\downarrow i=i$.

####  II-C3 Leader-follower systems

Leader-follower systems are coordinated systems with a single subsystem [4,
17].

##  III Network modeling

###  III-A Graph structure

Let $\mathcal{N}^{\textrm{A}}$ and $\mathcal{N}^{\textrm{D}}$ be buses in the
AC and DC parts of the network. Let $\mathcal{E}^{\textrm{A}}$ be the set of
AC lines. If $ij\in\mathcal{E}^{\textrm{A}}$, then $i$ and
$j\in\mathcal{N}^{\textrm{A}}$. Similarly, let $\mathcal{E}^{\textrm{D}}$ be
the set of DC lines. Let $\mathcal{C}$ be the set of converters. If
$ij\in\mathcal{C}$, the either $i\in\mathcal{N}^{\textrm{A}}$ and
$j\in\mathcal{N}^{\textrm{D}}$ or vice versa. The graphs
$\left(\mathcal{N}^{\textrm{A}},\mathcal{E}^{\textrm{A}}\right)$ and
$\left(\mathcal{N}^{\textrm{D}},\mathcal{E}^{\textrm{D}}\right)$ are
undirected. The graph
$\left(\mathcal{N}^{\textrm{A}}\cup\mathcal{N}^{\textrm{D}},\mathcal{C}\right)$
is directed, and if $ij\in\mathcal{C}$, then $ji\notin\mathcal{C}$.

Suppose that there are $m^{\textrm{A}}$ and $m^{\textrm{D}}$ connected AC and
DC subgraphs in
$\left(\mathcal{N}^{\textrm{A}},\mathcal{E}^{\textrm{A}}\right)$ and
$\left(\mathcal{N}^{\textrm{D}},\mathcal{E}^{\textrm{D}}\right)$. Let
$\left(\mathcal{N}^{\textrm{A}}_{k},\mathcal{E}^{\textrm{A}}_{k}\right)$ and
$\left(\mathcal{N}^{\textrm{D}}_{l},\mathcal{E}^{\textrm{D}}_{l}\right)$ be
the $k^{\textrm{th}}$ and $l^{\textrm{th}}$ such subgraphs for
$k=1,...,m^{\textrm{A}}$ and $l=1,...,m^{\textrm{D}}$. Define the mapping
$\mathcal{M}^{\textrm{A}}$ such that if $i\in\mathcal{N}_{k}^{\textrm{A}}$,
$\mathcal{M}^{\textrm{A}}(i)=k$; $\mathcal{M}^{\textrm{A}}$ identifies which
AC subgraph each bus belongs to. Similarly, define $\mathcal{M}^{\textrm{D}}$
such that if $i\in\mathcal{N}_{k}^{\textrm{D}}$,
$\mathcal{M}^{\textrm{D}}(i)=k$.

Let $\left(\mathcal{P},\mathcal{G}\right)$ be a simple directed graph such
that if $k$ and $l\in\mathcal{P}$ and $kl\in\mathcal{G}$, then there exists
$i\in\mathcal{N}_{k}^{\textrm{A}}$ and $j\in\mathcal{N}_{l}^{\textrm{D}}$ (or
$i\in\mathcal{N}_{l}^{\textrm{A}}$ and $j\in\mathcal{N}_{k}^{\textrm{D}}$)
such that $ij\in\mathcal{C}$. Observe that
$\left(\mathcal{P},\mathcal{G}\right)$ is bipartite. This is because an AC
subgrid can only be converter-connected to DC subgrids, and vice versa.

The directions of the edges in $\mathcal{G}$ are determined by those of
$\mathcal{C}$. We choose the directions of the edges in $\mathcal{C}$ such
that $\left(\mathcal{P},\mathcal{G}\right)$ is a DAG; we can always do so
because, as stated in Result 2, every undirected graph has at least one
acyclic orientation. $\left(\mathcal{P},\mathcal{G}\right)$ specifies a unique
poset, which we denote $\Phi=(\mathcal{P},\preceq)$; note that the nodes in
$\mathcal{P}$ are now also the elements of the poset. Observe that if
$\mathcal{M}(i)\preceq\mathcal{M}(j)$, either $\mathcal{M}(i)=\mathcal{M}(j)$
or there is a path from $i$ to $j$ through $\mathcal{E}$.

###  III-B State partitions

In the models we present later in Sections IV and V, a converter either
partially or fully decouples the AC and DC states on either side. In this
section, we specify how this decoupling determines the direction of each
converter in the set $\mathcal{C}$.

Each state is associated with a node or an edge in the system’s graph. Suppose
$x$ is the vector of AC states and $x_{i}$ the subvector associated with AC
bus $i\in\mathcal{N}^{\textrm{A}}$. Let $y$ similarly be the vector of DC
states and $y_{j}$ the subvector for $j\in\mathcal{N}^{\textrm{D}}$.

######  Definition 1 (One-way partition)

Let $i\in\mathcal{N}^{\textrm{A}}$ and $j\in\mathcal{N}^{\textrm{D}}$ be
connected by a converter. The converter partitions the state one way if one of
the following are true.

  * •

The evolution of $x_{i}$ does not depend on $y_{j}$. In this case
$ij\in\mathcal{C}$.

  * •

The evolution of $y_{j}$ does not depend on $x_{i}$. In this case
$ji\in\mathcal{C}$.

In this manner, the direction of a converter in $\mathcal{C}$ encodes the
direction that physical information flows through it.

######  Definition 2 (Full partition)

Let $i\in\mathcal{N}^{\textrm{A}}$, and $j\in\mathcal{N}^{\textrm{D}}$ be
connected by a converter. The converter fully partitions the state if the
evolution of $x_{i}$ does not depend on $y_{j}$, and vice versa. In this case
we may choose whether $ij$ or $ji\in\mathcal{C}$.

A pair of one-way partitions in both directions forms a full partition. In the
next two sections, we present several models in which the converter partitions
the state.

A converter’s direction affects its control structure. Consider a converter
between buses $i\in\mathcal{N}^{\textrm{A}}$ and
$j\in\mathcal{N}^{\textrm{D}}$. Assume $ij\in\mathcal{C}$, either due to a
one-way partition or because we have chosen this direction, and let $z_{ij}$
be the vector of control variables. We will regard the converter as a part of
AC subgrid $\mathcal{M}^{\textrm{A}}(i)$, and not DC subgrid
$\mathcal{M}^{\textrm{D}}(j)$. We are in effect associating $z_{ij}$ with bus
$i$, so that it influences bus $j$ and not vice versa; this implies
$\mathcal{M}^{\textrm{A}}(i)\prec\mathcal{M}^{\textrm{D}}(j)$. If poset
causality is to be preserved, then $z_{ij}$ can only receive feedback based on
states in subgrid $\mathcal{M}^{\textrm{A}}(i)$ (or further upstream in the
poset). If $z_{ij}$ were to receive feedback from a state in subgrid
$\mathcal{M}^{\textrm{D}}(j)$, information would flow from $j$ to $i$,
violating the partition.

##  IV Linear model

In this simple linear model, the converters are represented by controllable
current transfers. We use the standard linear power flow approximation to
model the AC part of the system, which contains most of the generation and
load, and model the DC part of the system as a linear circuit.

The converters fully partition the state in this model, and hence can have
either direction. A given converter $ij\in{\mathcal{C}}$,
$i\in\mathcal{N}^{\textrm{A}}$, and $j\in\mathcal{N}^{\textrm{D}}$, has
control input $\zeta_{ij}$, the current it injects on the DC side. Let
$\hat{v}_{j}$ be the constant nominal voltage on the DC side and $p_{ij}$ the
power on the AC side. Then, assuming a lossless converter as in [9],
conservation of power gives $p_{ij}=\hat{v}_{j}\zeta_{ij}$.

The AC states are the voltage angles, $\theta_{i}$, and frequencies,
$\omega_{i}$, for $i\in\mathcal{N}^{\textrm{A}}$. The dynamics at bus
$i\in\mathcal{N}^{\textrm{A}}$ are

$\displaystyle\dot{\theta}_{i}$ $\displaystyle=\omega_{i}$ (4a) $\displaystyle
J_{i}\dot{\omega}_{i}$
$\displaystyle=P_{i}-D_{i}\omega_{i}-\sum_{j:ij\in\mathcal{E}^{\textrm{A}}}B_{ij}(\theta_{i}-\theta_{j})$
$\displaystyle\quad\quad+\left\\{\begin{array}[]{ll}-\hat{v}_{j}\zeta_{ij}&\textrm{if
}ij\in\mathcal{C}\textrm{ for some }j\\\ \hat{v}_{j}\zeta_{ji}&\textrm{if
}ji\in\mathcal{C}\textrm{ for some }j\\\
0&\textrm{otherwise},\end{array}\right.$ (4e) where $P_{i}$ is the generation
or load, $J_{i}$ and $D_{i}$ are the rotor inertia and damping, and $B_{ij}$
is the line susceptance.

The DC states are the voltages, $v_{i}$ for $i\in\mathcal{N}^{\textrm{D}}$,
and currents, $i_{ij}$ for $ij\in\mathcal{E}^{\textrm{D}}$. The dynamics at
bus $i\in\mathcal{N}^{\textrm{D}}$ are

$\displaystyle C_{i}\dot{v}_{i}$
$\displaystyle=\sum_{j:ij\in\mathcal{E}^{\textrm{D}}}i_{ij}+\left\\{\begin{array}[]{ll}-\zeta_{ij}&\textrm{if
}ij\in\mathcal{C}\textrm{ for some }j\\\ \zeta_{ji}&\textrm{if
}ji\in\mathcal{C}\textrm{ for some }j\\\
0&\textrm{otherwise},\end{array}\right.$ (4i)

where $C_{i}$ is the bus’s capacitance. The dynamics of line
$ij\in\mathcal{E}^{\textrm{D}}$ are

$\displaystyle L_{ij}\dot{i}_{ij}=v_{i}-v_{j}-R_{ij}i_{ij},$ (4j)

where $L_{ij}$ and $R_{ij}$ are the line’s inductance and resistance.

Given a converter between $i\in\mathcal{N}^{\textrm{A}}$ and
$j\in\mathcal{N}^{\textrm{D}}$, the control variable $\zeta_{ij}$ fully
partitions the states on either side—the evolutions of $\theta_{i}$ and
$\omega_{i}$ do not depend on $v_{j}$ or $i_{jk}$,
$jk\in\mathcal{E}^{\textrm{D}}$. We may thus choose whether $ij$ or
$ji\in\mathcal{C}$. If we choose, say, $ij\in\mathcal{C}$, then we are
associating $\zeta_{ij}$ with bus $i$, so that it influences bus $j$ and not
vice versa.

##  V Nonlinear dq-frame model

We now describe the standard balanced dq-frame model, for which we follow the
presentation of Chapter 17 in [18]. We do not explicitly model the AC and DC
parts of the grid, which could be simple as in Section IV or as complicated as
desired.

Consider a VSC between buses $i\in\mathcal{N}^{\textrm{A}}$ and
$j\in\mathcal{N}^{\textrm{D}}$. The control inputs are $m_{ij}^{\textrm{d}}$
and $m_{ij}^{\textrm{q}}$, the d and q components of the averaged switching
signal.

The AC-side converter states are the currents, $i_{ij}^{\textrm{d}}$ and
$i_{ij}^{\textrm{q}}$. The component voltages at AC bus $i$ are
$v_{i}^{\textrm{d}}$ and $v_{i}^{\textrm{q}}$. Let $L_{ij}$ and $R_{ij}$ be
the combined inductance and resistance of the converter’s transformer and
filter. The dynamics of the AC-side converter currents are

$\displaystyle L_{ij}\dot{i}_{ij}^{\textrm{d}}$
$\displaystyle=v_{i}^{\textrm{d}}+\omega
L_{ij}i_{ij}^{\textrm{q}}-R_{ij}i_{ij}^{\textrm{d}}-\frac{1}{2}v_{j}^{\textrm{D}}m_{ij}^{\textrm{d}}$
(5a) $\displaystyle L_{ij}\dot{i}_{ij}^{\textrm{q}}$
$\displaystyle=v_{i}^{\textrm{q}}+\omega
L_{ij}i_{ij}^{\textrm{d}}-R_{ij}i_{ij}^{\textrm{q}}-\frac{1}{2}v_{j}^{\textrm{D}}m_{ij}^{\textrm{q}},$
(5b)

where $v_{j}^{\textrm{D}}$ is the DC-side capacitor voltage.

Let $C_{ij}$ be the DC-side capacitance. Let $\zeta_{ij}$ be DC-side converter
current, and $i_{ij}$ be the current from the converter to the DC node $j$.
The dynamics are

$\displaystyle C_{ij}\dot{v}_{j}^{\textrm{D}}$
$\displaystyle=\zeta_{ij}-i_{ij},$ (6a) where $\displaystyle\zeta_{ij}$
$\displaystyle=\frac{3}{4}\left(i_{ij}^{\textrm{d}}m_{ij}^{\textrm{d}}+i_{ij}^{\textrm{q}}m_{ij}^{\textrm{q}}\right).$
(6b)

The DC-side power is given by

$\displaystyle P^{\textrm{D}}_{ij}=v_{j}^{\textrm{D}}\zeta_{ij}.$

The real and reactive powers on the AC side are given by

$\displaystyle
P^{\textrm{A}}_{ij}=\frac{3}{4}\left(v_{i}^{\textrm{d}}i_{ij}^{\textrm{d}}+v_{i}^{\textrm{q}}i_{ij}^{\textrm{q}}\right),\quad
Q^{\textrm{A}}_{ij}=\frac{3}{4}\left(-v_{i}^{\textrm{d}}i_{ij}^{\textrm{q}}+v_{i}^{\textrm{q}}i_{ij}^{\textrm{d}}\right).$

Conservation of power implies $P^{\textrm{D}}_{ij}=P^{\textrm{A}}_{ij}$ (on
average).

As written, the states are not partitioned because the DC voltage,
$v_{j}^{\textrm{D}}$, influences the AC currents in (5), and the AC currents,
$i_{ij}^{\textrm{d}}$ and $i_{ij}^{\textrm{q}}$, influence
$v_{j}^{\textrm{D}}$ through (6).

In the following subsections, we obtain partitions in several ways.

  * •

In Section V-A, redefining the control inputs leads to one-way partitions.

  * •

In Section V-B, holding voltage constant leads to one-way partitions.

  * •

In Section V-C, we combine one-way partitions to obtain full partitions.

  * •

In Section V-D, we assume an inner current controller is fast enough that the
converter currents are in steady state, which is standard practice today. The
currents are always equal to their setpoints, and thus become control inputs.
This fully partitions the AC and DC states.

###  V-A Partitioning via substitution

We can obtain one-way partitions simply by redefining the control inputs. The
first below is standard (see Section 17.7.3 in [18]), and the latter new.

####  V-A1 A standard substitution

To decouple the d and q currents, define the new control inputs

$\displaystyle\beta^{\textrm{d}}_{ij}=2\frac{m^{\textrm{d}}_{ij}-\omega
L_{ij}i^{\textrm{q}}_{ij}}{v_{j}^{\textrm{D}}},\quad\beta^{\textrm{q}}_{ij}=2\frac{m^{\textrm{q}}_{ij}-\omega
L_{ij}i^{\textrm{d}}_{ij}}{v_{j}^{\textrm{D}}}.$ (7)

Substituting, (5) becomes

$\displaystyle L_{ij}\dot{i}_{ij}^{\textrm{d}}$
$\displaystyle=v_{i}^{\textrm{d}}-R_{ij}i_{ij}^{\textrm{d}}-\beta_{ij}^{\textrm{d}}$
(8a) $\displaystyle L_{ij}\dot{i}_{ij}^{\textrm{q}}$
$\displaystyle=v_{i}^{\textrm{q}}-R_{ij}i_{ij}^{\textrm{q}}-\beta_{ij}^{\textrm{q}}.$
(8b)

The AC-side converter currents now depend on no DC-side states. The DC voltage
still depends on $i_{ij}^{\textrm{d}}$ and $i_{ij}^{\textrm{q}}$. With only
this substitution, the direction of the corresponding edge is from $i$ to $j$,
i.e., $ij\in\mathcal{C}$.

####  V-A2 Another substitution

Set

$\displaystyle\rho_{ij}^{\textrm{d}}=i_{ij}^{\textrm{d}}m_{ij}^{\textrm{d}},\quad\rho_{ij}^{\textrm{q}}=i_{ij}^{\textrm{q}}m_{ij}^{\textrm{q}}.$
(9)

Substituting, (5) becomes

$\displaystyle L_{ij}\dot{i}_{ij}^{\textrm{d}}$
$\displaystyle=v_{i}^{\textrm{d}}+\omega
L_{ij}i_{ij}^{\textrm{q}}-R_{ij}i_{ij}^{\textrm{d}}-\frac{1}{2i_{ij}^{\textrm{d}}}v_{j}^{\textrm{D}}\rho_{ij}^{\textrm{d}}$
(10a) $\displaystyle L_{ij}\dot{i}_{ij}^{\textrm{q}}$
$\displaystyle=v_{i}^{\textrm{q}}+\omega
L_{ij}i_{ij}^{\textrm{d}}-R_{ij}i_{ij}^{\textrm{q}}-\frac{1}{2i_{ij}^{\textrm{q}}}v_{j}^{\textrm{D}}\rho_{ij}^{\textrm{q}},$
(10b)

and (6b) becomes

$\displaystyle\zeta_{ij}$
$\displaystyle=\frac{3}{4}\left(\rho_{ij}^{\textrm{d}}+\rho_{ij}^{\textrm{q}}\right).$
(11)

The DC voltage, $v_{j}^{\textrm{D}}$, still influences the AC-side states
through (10). Because $\zeta_{ij}$ now only depends on the control inputs, the
AC-side states do not influence the DC voltage. The direction of the edge is
therefore from $j$ to $i$, i.e., $ji\in\mathcal{C}$. We remark that unlike in
Section V-A1, this substitution serves no purpose beyond creating a one-way
partition.

###  V-B Partitioning via constant voltage

We now show how approximating either the AC or DC voltage as constant leads to
a one-way partition.

####  V-B1 Tightly regulated DC voltage

The converter’s DC voltage is usually tightly regulated (see Section 17.7.2 in
[18]). We may thus set $\dot{v}_{j}^{\textrm{D}}=0$, so that
$v_{j}^{\textrm{D}}$ is constant and $i_{ij}=\zeta_{ij}$. This partitions the
state one way in that the AC-side currents are directly coupled to the DC bus
states, but no DC-side states affect the AC-side currents. Under this
approximation, the direction of the corresponding edge is from $i$ to $j$,
i.e., $ij\in\mathcal{C}$.

####  V-B2 Tightly regulated AC voltages

The converter’s AC voltages are also usually tightly regulated (see Section
17.7.1 in [18]). We represent this by setting $v_{i}^{\textrm{q}}=0$ and
assuming $v_{i}^{\textrm{d}}$ is constant, which means the AC-side voltage is
on the d axis. Under this approximation, the AC-side currents,
$i_{ij}^{\textrm{d}}$ and $i_{ij}^{\textrm{q}}$, have no dependence on the AC
bus voltages. $i_{ij}^{\textrm{d}}$ and $i_{ij}^{\textrm{q}}$ do still depend
on the DC-side voltage, $v_{j}^{\textrm{D}}$. Therefore, the direction of the
corresponding edge is from $j$ to $i$, i.e., $ji\in\mathcal{C}$.

###  V-C Combining one-way partitions

We can obtain a full partition by combining one-way partitions in several
different ways.

  * •

Assume tightly regulated AC and DC voltages. The remaining converter states,
$i_{ij}^{\textrm{d}}$ and $i_{ij}^{\textrm{q}}$, depend only on the converter
controls, $m_{ij}^{\textrm{d}}$ and $m_{ij}^{\textrm{q}}$, and not on any AC
or DC grid states.

  * •

Make the substitution in Section V-A1 and assume tightly regulated AC
voltages. Examining (8), $i_{ij}^{\textrm{d}}$ and $i_{ij}^{\textrm{q}}$
depend only $\beta_{ij}^{\textrm{d}}$ and $\beta_{ij}^{\textrm{q}}$, and not
on any AC or DC grid states.

  * •

Make the substitution in Section V-A2 and assume tightly regulated DC
voltages. Now (10) depends on $\rho_{ij}^{\textrm{d}}$ and
$\rho_{ij}^{\textrm{q}}$, but not any DC-side states. The only input from the
converter to the DC side is $\zeta_{ij}$, which depends only on the control
variables through (11).

In the first two cases, the full partition is through $i_{ij}^{\textrm{d}}$
and $i_{ij}^{\textrm{q}}$, which depend only on the converter control inputs.
In the third, the full partition is through $\zeta_{ij}$ in that the states on
either side are uncoupled. In all cases, we may choose the converter’s
direction, i.e., whether $ij$ or $ji\in\mathcal{C}$.

###  V-D Partitioning via timescale separation

The converter currents are usually controlled locally on a faster timescale
(see Section 17.8 in [18]). We may assume they are in steady state, and
therefore always at their setpoints. The control inputs are thus
$i_{ij}^{\textrm{d}}$ and $i_{ij}^{\textrm{q}}$. Because there is no other
coupling across the converter, the converter fully partitions the states under
this assumption.

Today, it is common to have a local PID loop use $i_{ij}^{\textrm{q}}$ to
regulate reactive power, and in turn the AC voltage magnitude.
$i_{ij}^{\textrm{d}}$ is then used to control either the converter’s power
transfer or DC-side voltage. Each case entails a feedback loop with either AC-
or DC-side states. As a result, the converter no longer fully partitions the
states, but only one way. A converter’s local control loops, if it has any,
thus dictate its direction in $\mathcal{C}$.

Let $i\in\mathcal{N}^{\textrm{A}}$, $j\in\mathcal{N}^{\textrm{D}}$, and assume
the converter fully partitions the state. The above control loops specify the
converter’s direction as follows.

  * •

If reactive power is regulated by $i_{ij}^{\textrm{q}}$, then
$ij\in\mathcal{C}$. This is because information about reactive power is on the
AC-side (see Section 17.8.1 in [18]).

  * •

If the power transfer is regulated by $i_{ij}^{\textrm{d}}$, then
$ji\in\mathcal{C}$. This is because information about power transfer is taken
on the DC side (see Section 17.8.2 in [18]). In principle, this information
could be ontained on the AC side, e.g., via real power, in which case
$ij\in\mathcal{C}$.

  * •

If DC voltage is regulated by $i_{ij}^{\textrm{d}}$, then $ji\in\mathcal{C}$.
This is because information about the DC voltage is on the DC side (see
Section 17.8.3 in [18]).

###  V-E Discussion

There are a number of ways to obtain one-way and full partitions in the dq-
frame model. Figure 1 illustrates how each modification disables couplings
between the states, and the new couplings induced by the local controllers in
Section V-D.

$v_{i}^{\textrm{d}}$$v_{i}^{\textrm{q}}$$i_{ij}^{\textrm{d}}$$i_{ij}^{\textrm{q}}$$v_{j}^{\textrm{D}}$$i_{ij}$Substitution
(7)Timescale Constant $v_{i}^{\textrm{d}}$Timescale $v_{i}^{\textrm{q}}=0$
Constant $v_{j}^{\textrm{D}}$ or Sub. (7) Timescale or Sub. (9) Constant
$v_{j}^{\textrm{D}}$ or Sub. (7) Timescale or Sub. (9) Constant
$v_{j}^{\textrm{D}}$Regulate DC voltageRegulate AC voltageRegulate AC
voltageRegulate power Figure 1: The nodes in the above graph are states in the
dq-frame model of converter $ij\in\mathcal{C}$, (5)-(6), and at the adjacent
buses. Solid lines represent physical couplings between states. The text and
its location indicates the coupling direction disabled by each modification;
e.g., fixing $v_{i}^{\textrm{d}}$ eliminates the coupling from
$v_{i}^{\textrm{d}}$ to $i_{ij}^{\textrm{d}}$. The dashed lines represent the
one-way couplings induced by local control loops; e.g., regulating DC voltage
makes $i_{ij}^{\textrm{d}}$ depend on $v_{j}^{\textrm{D}}$.

Our analysis aligns with the intuition that converters dynamically decouple
the systems on either side, especially on slower timescales. In this paper, we
seek to orient the converters so that $(\mathcal{P},\mathcal{G})$ is a DAG,
and therefore the system model is poset-causal. The following example looks at
how the control loops in Section V-D restrict the number of acyclic
orientations.

######  Example 1 (Point-to-point DC link)

Consider a single DC line with converters and both ends. A common
configuration is for one converter to regulate the DC voltage, and the other
the power transfer [19].

There are two DC and two AC buses. Assume also that the two AC buses are
connected by an AC line. We have $\mathcal{N}^{\textrm{A}}=\\{1,4\\}$,
$\mathcal{N}^{\textrm{D}}=\\{2,3\\}$, $\mathcal{E}^{\textrm{A}}=\\{14\\}$, and
$\mathcal{E}^{\textrm{D}}=\\{23\\}$. We must choose directions for the
converters between buses 1 and 2 and between buses 3 and 4. If
$\mathcal{C}=\\{12,34\\}$ or $\mathcal{C}=\\{21,43\\}$, the system is not
poset-causal. The system is poset-causal if $\mathcal{C}=\\{12,43\\}$ or
$\mathcal{C}=\\{21,34\\}$; we assume the latter, as in Figure 2. Because there
are only two subgrids, this is also a leader-follower system, as described in
Section 2. Here the DC subgrid is the leader and the AC subgrid the follower.

Bus $1$Bus $4$VSCVSCBus $2$Bus $3$AC lineDC line Figure 2: A point-to-point DC
link. The converter directions make the system poset-causal.

Assume timescale separation as in Section V-D, and that the converter current
setpoints depend on AC- or DC-side states via local control loops. The
following pair of controllers (and vice versa) is compatible with the
converter directions we’ve chosen.

  * •

Converter 21 regulates the DC voltage at bus 2.

  * •

Converter 34 regulates the power transfer by feeding back the power at bus 3.

Unfortunately, if either converter regulates its AC-side reactive power (or
voltage), it must also feed back AC-side states, eliminating a one-way
partition and hence the model’s poset-causality. As reactive power mostly
affects local voltages, it is reasonable to omit this control loop when
focusing on system-level questions. $\triangle$

There are many choices leading to poset-causality. If interested in moving
energy through large grids, then modeling the converter as a controllable
current/power transfer as in Section IV may be adequate. The dq-frame model in
Section V is appropriate if reactive power or AC voltage dynamics are
important. In this case, the timescale approximation of Section V-D is
relatively simple and aligned with current practice. These models are
representative but not comprehensive; e.g., a dq0-frame model would be
appropriate for unbalanced AC grids, and a model with positive and negative
sequences can capture other converter control strategies. Analyzing
partitioning and information structures in such settings is a topic of future
work.

##  VI Information structures

We now discuss the poset-causality of the systems in Sections IV and V. The
results are essentially more general versions of Lemma 1 in [12], which
established the poset-causality of a linearized model with only point-to-point
DC links.

###### Lemma 1

System (4) is poset-causal if $(\mathcal{P},\mathcal{G})$ is a DAG.

###### Proof:

Assume $(\mathcal{P},\mathcal{G})$ is a DAG. By Result 1, it specifies a
poset, $\Psi$. The $A$ matrix is block diagonal, with each block corresponding
to either an AC or DC subgrid. This implies $A\in\mathcal{I}_{A}(\Psi)$. We
can also see by inspection that $B\in\mathcal{I}_{B}(\Psi)$. By Result 3,
system (4) is poset-causal. ∎

Result 2 says that we can always choose the directions of the converters in
$\mathcal{C}$ so that $(\mathcal{P},\mathcal{G})$ is acyclic. This means that
any AC/DC grid has at least one poset-causal representation. The total number
of acyclic orientations is $\left|\mathcal{X}(-1)\right|$, where $\mathcal{X}$
is the chromatic polynomial of the underlying undirected graph [20]. Figure 3
shows an AC/DC grid with converter directions chosen to yield poset-causality.

AC 1DC 1DC 2DC 5DC 3AC 3DC 4AC 4 Figure 3: The nodes are AC and DC subgrids.
Each edge is a VSC in $\mathcal{C}$, the directions of which make
$(\mathcal{P},\mathcal{G})$ acyclic. The total number of such acyclic
orientations for this system is $\left|\mathcal{X}(-1)\right|=392$.

As is, the dq-frame model in Section V is not poset-causal. We can make it
poset-causal with the modifications from Sections V-A to V-D, which, by
partitioning the states, determine the possible directions of the converters
in $(\mathcal{P},\mathcal{G})$. For example, if we assume the currents are in
steady state as in Section V-D, then the converters fully partition the
states, and we can pick the direction of each one. If we choose converter
directions so that $(\mathcal{P},\mathcal{G})$ is a DAG, then the dq-frame
model is poset-causal. We eschew a formal proof because the argument is
similar to that of Lemma 1, but more tedious.

###### Corollary 1

System (4) is a coordinated system if it consists of either

  * •

one AC and multiple DC subgrids, or

  * •

one DC and multiple AC subgrids.

The latter case might be referred to as an MTDC grid. Observe that if the
converters are pointed out of the single AC (DC) grid, then it is the
coordinator, and the DC (AC) subgrids are the subsystems. If the converters
are pointed into the single AC (DC) grid, then it is the subsystem, and the DC
(AC) subgrids are together the coordinator. As we’ll see in the example in
Section VII-B, the latter setup is of more practical interest because the DC
(AC) subgrids can each use local control, and only the single AC (DC) grid
must use feedback over the whole system.

###### Corollary 2

System (4) is a leader-follower system if it consists of one AC and one DC
subgrid.

These results similarly extend to the dq-frame model as well. Note that other
choices of subsystems can lead to different information structures. For
example, even in a very complicated AC/DC grid, one can obtain a leader-
follower system by only partitioning along a single boundary of converters.

##  VII Example

In this example, we illustrate how an AC/DC grid’s information structure can
guide the design of a decentralized controller.

###  VII-A Decentralized control

Decentralized control is intractable for general LTI systems [21, 22]. If the
system is also poset-causal, optimal decentralized control is only slightly
more complicated than the centralized regulator. Consider the following
problem: minimize the $\mathcal{H}_{2}$ norm of $T_{zw}$, as defined in (3),
subject to the communication constraint $K\in I_{K}(\Psi)$. Reference [2]
constructed the optimal solution to this problem in terms of a nested family
of Riccati equations.

The controller in [2] is dynamic in that new states are introduced, which
serve as estimates of downstream states. This might be seen as an undesirable
complication. Reference [3] constructs suboptimal static controllers for
coordinated LTI systems. In short, one first constructs a local controller for
the coordinator system, and then controllers for the subsystems that use
feedback from the local and coordinator subsystems. We apply this perspective
in this example.

###  VII-B Test system

We construct a decentralized controller for a stylized test system based on
the MTDC grid in [9]. We use the linear model of Section IV. The system
consists of the following parts.

  * •

An MTDC grid with ten lines and six VSCs, with parameters from [9]. Each line
is represented by a resistance and inductance, and each VSC an identical
capacitance.

  * •

An AC subgrid is attached to each of the six VSCs of the MTDC grid. Each AC
subgrid is approximated as a single inertia with normalized moment $10$
MWs/MVA and damping coefficient $0.1$ pu. The AC subgrids have no generator
inputs or other controls.

  * •

A point-to-point DC link between AC subgrids 1 and 6. It consists of two VSCs,
each with the same capacitance as those in the MTDC grid, and a DC line with
parameters from row two of Table I in [9].

The states are the six AC subgrid frequencies, the inductor currents of the
ten DC lines in the MTDC grid and the DC line between AC 1 and AC 6, and the
capacitor voltages of the eight VSCs’ DC-side terminals. The only control
inputs are the current/power transfers through the eight VSCs. The real power
on each VSC’s AC side is equal to the DC current times the nominal voltage,
which is 1 pu.

The system is shown in Figure 4. Our goal is to design a controller in which
VSCs 7 and 8 use local feedback and the six VSCs of the MTDC grid system-wide
feedback. We hence view this as a leader-follower system, a special case of
poset causality described in Section II-C3. The leader consists of AC 1, AC 6,
VSC 7, VSC 8, and the DC line between. The follower consists of the other four
AC subgrids, the MTDC grid, and VSCs 2 through 5. The boundary between the two
systems consists of VSCs 1 and 6, which are oriented toward the MTDC grid. The
other six VSCs do not need directions because they are within either the
leader or follower subsystem.

MTDC gridVSC 2VSC 5VSC 3VSC 4VSC 1VSC 6AC 2AC 3AC 4AC 5AC 1AC 6VSC 7VSC 8
Figure 4: The test system. AC 1 through AC 6 are subgrids modeled as single
inertias. The top and bottom boxes are the follower and leader systems,
respectively, and the two intermediary VSCs form their boundary.

The optimal decentralized controller for general leader-follow systems is
derived in [17]. It is a special case of that for poset-causal systems [2],
and similarly introduces new estimator states. We instead take a simpler
approach motivated by [3].

  * •

We first solve for the optimal regulator for the leader system and close the
loop.

  * •

We then solve for the optimal regulator for the full system, which consists of
the follower system and the locally controlled leader system.

The resulting controller uses only feedback from the leader system for VSCs 7
and 8, and feedback from the full system for VSCs 1 through 6.

We remark that we could have used a different information structure to design
a different decentralized controller for this system. For example, we could
have swapped leader and follower roles, or treated each VSC plus local AC
subgrid as a separate subsystem and implemented the poset-causal controller of
[2]. In the latter case, there would be multiple posets to choose from, each
corresponding to a different acyclic orientation of the graph of converters
and AC and DC subgrids.

###  VII-C Simulations

All computations were performed in Python using NumPy [23] and SciPy [24]. The
figure was made with Matplotlib [25].

The controllers seek to drive the system to zero. The cost coefficients for
all state and control variables were one. The initial frequency deviations of
AC 1, AC 4, and AC 6 were $\omega_{1}^{0}=-0.01$, $\omega_{4}^{0}=-0.01$, and
$\omega_{6}^{0}=0.02$ pu. All other states began at zero. We chose these
initial conditions because the DC lines primarily move energy between AC
subgrids. Had the initial frequencies not summed to zero, the system would
have taken considerably longer to settle because the dampings and resistances
dissipate little energy. In a more realistic setup, generator controls would
provide power balancing and damping. Also note that the system does not
oscillate because each AC subgrid was modeled as a single inertia.

The performances of the centralized and decentralized controllers were
similar, with respective closed loop $\mathcal{H}_{2}$ norms 9.166 and 9.172.
We only plot frequencies under the latter because the two look roughly the
same. The top plot of Figure 5 shows how much more quickly the controlled
system returns to the origin. This is because the converters seek to balance
deviations rather than letting them damp out within each subgrid. Subgrids
without initial frequency deviations remain at zero without control. With
control, energy passes through their inertias, resulting in frequency
deviations roughly an order of magnitude smaller than those shown.

Figure 5: The top plot shows the frequencies in AC 1 and AC 6 without control
($O$) and under the leader-follower controller ($D$). The middle plot shows
the voltages at the terminal of VSC 7 under the leader follower ($D$) and
centralized ($C$) controllers. The bottom plot shows the current through the
DC line between VSCs 7 and 8 under the leader follower ($D$) and centralized
($C$) controllers.

The middle and bottom plots of Figure 5 show the voltage of VSC 7 and the
current from VSC 7 to VSC 8 under the decentralized and centralized
controllers. Note that both undergo transients in first 0.002 seconds that
take them from zero to the starting value seen on the left side of each plot.
Their deviations from zero are larger under the decentralized controller. This
is because the leader controller, unaware of the rest of the system, expends
more control effort in VSC 7 and VSC 8 than is optimal, resulting in a
slightly higher $\mathcal{H}_{2}$ norm.

##  VIII Conclusion

Converters form actuated boundaries between AC and DC subgrids. We have shown
that for both simple linear and dq-frame models, this topological property
imparts all AC/DC grids with poset-causal information structures, and that
special cases can have coordinated and leader-follower information structures.
In a stylized example, we saw how this structure can inform the design of
decentralized controllers.

The topological structure of AC/DC grids is obvious, and it is natural to ask
how else it might be useful. Do poset-causality and other information
structures enable other tools or analyses, e.g., via controllability or
observability [5, 6], and do AC/DC grids possess other useful structural
properties? If so, graph theoretic notions like treewidth and bipartiteness
might aid in analysis, e.g., in characterizing the optimal acyclic
orientation.

More concrete directions include conducting similar analyses for other power
electronic interfaces like cycloinverters between AC subgrids; understanding
the interplay of information structure with stability and control objectives;
and making use of poset-causality without only considering controllers with
the same information structure, e.g., allowing bidirectional communication
between neighboring subgrids.

## References

  * [1] D. Van Hertem, O. Gomis-Bellmunt, and J. Liang, _HVDC grids_. Wiley Online Library, 2016.
  * [2] P. Shah and P. Parrilo, “$\mathcal{H}_{2}$-optimal decentralized control over posets: A state-space solution for state-feedback,” _Automatic Control, IEEE Transactions on_ , vol. 58, no. 12, pp. 3084–3096, Dec. 2013.
  * [3] P. L. Kempker, A. C. Ran, and J. H. van Schuppen, “LQ control for coordinated linear systems,” _IEEE Transactions on Automatic Control_ , vol. 59, no. 4, pp. 851–862, 2013.
  * [4] J. Cruz, “Leader-follower strategies for multilevel systems,” _IEEE Transactions on Automatic Control_ , vol. 23, no. 2, pp. 244–255, 1978.
  * [5] P. L. Kempker, A. C. Ran, and J. H. van Schuppen, “Controllability and observability of coordinated linear systems,” _Linear algebra and its applications_ , vol. 437, no. 1, pp. 121–167, 2012.
  * [6] S. ter Horst and J. Zeelie, “Realization theory for poset-causal systems: controllability, observability and duality,” _Mathematics of Control, Signals, and Systems_ , vol. 33, pp. 197–236, 2021.
  * [7] B. K. Johnson, R. H. Lasseter, F. L. Alvarado, and R. Adapa, “Expandable multiterminal DC systems based on voltage droop,” _IEEE Transactions on Power Delivery_ , vol. 8, no. 4, pp. 1926–1932, 1993.
  * [8] J. Beerten, S. Cole, and R. Belmans, “Modeling of multi-terminal VSC HVDC systems with distributed DC voltage control,” _IEEE Transactions on Power Systems_ , vol. 29, no. 1, pp. 34–42, 2013.
  * [9] M. Andreasson, D. V. Dimarogonas, H. Sandberg, and K. H. Johansson, “Distributed controllers for multiterminal HVDC transmission systems,” _IEEE Transactions on Control of Network Systems_ , vol. 4, no. 3, pp. 564–574, 2017.
  * [10] M. Pirani and J. Taylor, “Controllability of AC power networks with DC lines,” _Power Systems, IEEE Transactions on_ , vol. 36, no. 2, pp. 1649–1651, 2021.
  * [11] I. Subotić and D. Groß, “Power-balancing dual-port grid-forming power converter control for renewable integration and hybrid ac/dc power systems,” _IEEE Transactions on Control of Network Systems_ , vol. 9, no. 4, pp. 1949–1961, 2022.
  * [12] B. Vellaboyana and J. Taylor, “Optimal decentralized control of DC-segmented power systems,” _Automatic Control, IEEE Transactions on_ , vol. 63, no. 10, pp. 3616–3622, Oct 2018.
  * [13] C. D. Godsil and G. Royle, _Algebraic graph theory_ , ser. Graduate Texts in Mathematics. Springer, 2001, vol. 207\.
  * [14] M. Aigner, _Combinatorial theory_. Springer Science & Business Media, 2012.
  * [15] J. G. Oxley, _Matroid theory_ , 2nd ed. Oxford University Press, 2011.
  * [16] W. Findeisen, F. N. Bailey, M. Brdys, K. Malinowski, P. Tatjewski, and A. Wozniak, _Control and coordination in hierarchical systems_. John Wiley & Sons, 1980.
  * [17] J. Swigart and S. Lall, “An explicit state-space solution for a decentralized two-player optimal linear-quadratic regulator,” in _American Control Conference_ , 2010, pp. 6385–6390.
  * [18] D. Jovcic, _High voltage direct current transmission: converters, systems and DC grids_. John Wiley & Sons, 2019\.
  * [19] A. Yazdani and R. Iravani, _Voltage-Sourced Converters in Power Systems_. Wiley - IEEE Press, 2010.
  * [20] R. P. Stanley, “Acyclic orientations of graphs,” _Discrete Mathematics_ , vol. 5, no. 2, pp. 171 – 178, 1973.
  * [21] N. Sandell, P. Varaiya, M. Athans, and M. Safonov, “Survey of decentralized control methods for large scale systems,” _Automatic Control, IEEE Transactions on_ , vol. 23, no. 2, pp. 108–128, 1978.
  * [22] V. D. Blondel and J. N. Tsitsiklis, “A survey of computational complexity results in systems and control,” _Automatica_ , vol. 36, no. 9, pp. 1249 – 1274, 2000.
  * [23] C. Harris, K. Millman, S. van der Walt _et al._ , “Array programming with NumPy,” _Nature_ , vol. 585, no. 7825, pp. 357–362, 2020.
  * [24] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright _et al._ , “Scipy 1.0: fundamental algorithms for scientific computing in Python,” _Nature Methods_ , vol. 17, no. 3, pp. 261–272, 2020.
  * [25] J. D. Hunter, “Matplotlib: A 2D graphics environment,” _Computing in science & engineering_, vol. 9, no. 03, pp. 90–95, 2007.