# On the Stability of Networked Nonlinear Imaginary Systems with Applications
to Electrical Power Systems

Yijun Chen  Kanghong Shi  Ian R. Petersen  Elizabeth L. Ratnam  The School of
Engineering, The Australian National University, Canberra, Australia, emails:
{yijun.chen, kanghong.shi, ian.petersen<EMAIL_ADDRESS>
###### Abstract

In the transition to achieving net zero emissions, it has been suggested that
a substantial expansion of electric power grids will be necessary to support
emerging renewable energy zones. In this paper, we propose employing battery-
based feedback control and nonlinear negative imaginary (NI) systems theory to
reduce the need for such expansion. By formulating a novel Luré-Postnikov-like
Lyapunov function, stability results are presented for the feedback
interconnection of two single nonlinear NI systems, while output feedback
consensus results are established for the feedback interconnection of two
networked nonlinear NI systems based on a network topology. This theoretical
framework underpins our design of battery-based control in power transmission
systems. We demonstrate that the power grid can be gradually transitioned into
the proposed NI systems, one transmission line at a time.

###### keywords:

Nonlinear negative imaginary systems, electrical power systems, stability,
consensus

††thanks: This work was supported by the Australian Research Council under
grants DP230102443 and LP210200473.

, , ,

##  1 introduction

To enable the energy transition towards net zero power systems, it has been
proposed that a significant expansion of the transmission grid will be
necessary to accommodate the emergence of renewable energy zones (Australian
Energy Market Operator, 2023). In this paper, we propose employing battery-
based feedback control and nonlinear negative imaginary (NI) systems theory
(Petersen and Lanzon, 2010; Lanzon and Petersen, 2008; Shi et al., 2023) to
mitigate the need for such expansion, thereby facilitating fuller utilization
of existing grid infrastructure.

In recent years, advancements in battery technologies have led to the
widespread adoption of rechargeable batteries in electric vehicles, large grid
storage batteries, and domestic solar-powered batteries (Tran et al., 2019;
Borenstein, 2022). Apart from supporting local power management, this
transformative development also presents the possibility for active
participation in frequency and power flow regulation, as well as ensuring
transient stability within power systems. In pursuit of this goal, this paper
provides a systematic method to design feedback controllers based on the use
of battery-based actuators to synchronize the grid frequency across the
transmission network, regulate power flows, and ensure bulk system transient
stability.

Real-time angle measurements hold potential for future power systems, as
discussed by (Višić et al., 2020). NI systems theory can be utilized to
guarantee system stability, enhance system robustness, and address consensus
problems. Accordingly, the authors in (Chen et al., 2023a) propose angle
feedback linearization control using linear NI systems theory to enhance the
transient stability of power transmission systems. However, the method
proposed in (Chen et al., 2023a) directly cancels out the nonlinear properties
of power flow along transmission lines. In contrast, the work of (Chen et al.,
2023b) proposed nonlinear angle feedback control for a single-machine infinite
bus system. Although the method in (Chen et al., 2023b) uses nonlinear
techniques, the authors consider the more simple case of a single generator
bus connected to an infinite bus, ruling out its application to the more
realistic case of an interconnected transmission network.

In this paper, we propose a networked control framework for power transmission
systems that retains the nonlinear properties of power flow, which is
specifically designed to accommodate an interconnected transmission network.
From a technical standpoint, we constructed a novel candidate Lyapunov
function, fashioned in the Luré-Postnikov form (Haddad and Chellaboina, 2008;
Hill and Chong, 1989; Hill and Bergen, 1982; Bergen and Hill, 1981), intended
for stability proofs for the feedback interconnection of two single nonlinear
NI systems and output consensus proofs for the feedback interconnection of two
networked nonlinear NI systems based on the network structure. Our
contributions to the area of power transmission systems are two-fold: 1) in
the absence of controllers, we show that existing power transmission systems
are resilient — that is, resilient power transmission systems synchronize bus
frequencies and regulate power flows when initial angle deviations are within
a suitable domain; 2) we present a networked control framework using real-time
angle sensors and large-scale batteries to enhance the transient stability of
power transmission systems, where the control can be realized one transmission
line at a time. In order to achieve these results, we extend the networked
nonlinear NI theory of (Shi et al., 2023) to allow for nonlinear direct
feedthrough terms and Luré-Postnikov type Lyapunov functions.

This paper is organized as follows. Section 2 provides definitions for NI
systems, and stability results for the feedback interconnection of two single
nonlinear NI systems. In Section 3, a networked setting is considered, and
output feedback consensus is proved for two networked nonlinear NI systems. In
Section 4, we present an application to power transmission systems.

##  2 An Initial Stability Result

In this section, we present definitions for nonlinear NI systems and include
our stability results for the feedback interconnection of two single nonlinear
NI systems.

###  2.1 Definitions of Nonlinear NI Systems

Consider a multiple-input multiple-output (MIMO) nonlinear system with the
following state-space model:

$\displaystyle\dot{x}$ $\displaystyle=f(x,u),$ (1a) $\displaystyle y$
$\displaystyle=h(x)+g(u),$ (1b)

where $x\in\mathbb{R}^{n}$ is the state, $u\in\mathbb{R}^{m}$ is the input,
$y\in\mathbb{R}^{m}$ is the output,
$f:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{n}$ is a Lipschitz
continuous function, and $h:\mathbb{R}^{n}\to\mathbb{R}^{m}$ is a class C1
function. We impose Assumption 1 on the input function $g(u)$. As a special
case, we consider a static system $y=g(u)$ of the system (1). In general,
nonlinear controllers are dynamic unless otherwise stated for the special
static case.

###### Assumption 1

The input function $g(u)$ is independent in each input channel, such that

$g(u)=[g^{1}(u^{1}),\dots,g^{m}(u^{m})]^{\top},$ (2)

where each $g^{k}(u^{k})$ is a class $C^{1}$ function with the superscript
$k\in\\{1,2,\dots,m\\}$ representing the $k$th element of the input $u$.
Moreover, $g(0)=0.$

In this paper, nonlinear generalizations of standard properties for linear
systems are assumed as done in (Shi et al., 2023). The following Assumption 2
is an observability assumption, while Assumption 3 requires all system inputs
to have an effect on the system dynamics.

###### Assumption 2

Over any time interval $[t_{a},t_{b}]$ where $t_{b}>t_{a}$, $h(x)$ remains
constant if and only if $x$ remains constant; i.e., $\dot{h}(x)\equiv
0\iff\dot{x}\equiv 0$. Moreover, $h(x)\equiv 0\iff x\equiv 0$.

###### Assumption 3

Over any time interval $[t_{a},t_{b}]$ where $t_{b}>t_{a}$, $x$ remains
constant only if $u$ remains constant; i.e.,
$x\equiv\overline{x}{\implies}u\equiv\overline{u}$. Moreover, $x\equiv
0{\implies}u\equiv 0$.

Next, we define the negative imaginary (NI) property, and we include the
definition of output strictly negative imaginary (OSNI) property — tailored
for nonlinear MIMO systems.

###### Definition 1

The system (1) is said to be NI if there exists a positive semidefinite
storage function $V:\mathbb{R}^{n}\to\mathbb{R}$ of class C1 such that for any
locally integrable input $u$ and solution $x$ to (1a), then

$\displaystyle\dot{V}(x)$ $\displaystyle\leq u^{\top}\dot{h}(x),$ (3)

for all $t\geq 0$.

###### Definition 2

The system (1) is said to be OSNI if there exists a positive semidefinite
storage function $V:\mathbb{R}^{n}\to\mathbb{R}$ of class C1 and a scalar
$\epsilon>0$ such that for any locally integrable input $u$ and solution $x$
to (1a), then

$\displaystyle\dot{V}(x)$ $\displaystyle\leq
u^{\top}\dot{h}(x)-\epsilon\|\dot{h}(x)\|^{2},$ (4)

for all $t\geq 0$. Here, $\epsilon$ measures the degree of output strictness.

###  2.2 Stability of NI Systems

In what follows, we present some initial stability results for the positive
feedback interconnection of an NI system and an OSNI system.

Consider a nonlinear system $H_{p}$:

$\displaystyle H_{p}:\quad\dot{x}_{p}$ $\displaystyle=f_{p}(x_{p},u_{p}),$
(5a) $\displaystyle y_{p}$ $\displaystyle=h_{p}(x_{p}),$ (5b)

where $x_{p}\in\mathbb{R}^{n_{p}}$ is the state, $u_{p}\in\mathbb{R}^{m}$ is
the input, $y_{p}\in\mathbb{R}^{m}$ is the output,
$f_{p}:\mathbb{R}^{n_{p}}\times\mathbb{R}^{m}\to\mathbb{R}^{n_{p}}$ is a
Lipschitz continuous function, and $h_{p}:\mathbb{R}^{n_{p}}\to\mathbb{R}^{m}$
is a class C1 function. The subscript “p” implies that this system plays the
role of a plant. Also, we assume $f_{p}(0,0)=0$ and $h_{p}(0)=0$.

###### Assumption 4

For a system $H_{p}$ with a constant input $\overline{u}_{p}$ which results in
a constant state $\overline{x}_{p}$ and a constant output $\overline{y}_{p}$,
then $\overline{u}_{p}^{\top}\overline{y}_{p}\geq 0.$

Also consider a nonlinear system $H_{c}$:

$\displaystyle H_{c}:\quad\dot{x}_{c}$ $\displaystyle=f_{c}(x_{c},u_{c}),$
(6a) $\displaystyle y_{c}$ $\displaystyle=h_{c}(x_{c})+g_{c}(u_{c}),$ (6b)

where $x_{c}\in\mathbb{R}^{n_{c}}$ is the state, $u_{c}\in\mathbb{R}^{m}$ is
the input, $y_{c}\in\mathbb{R}^{m}$ is the output,
$f_{c}:\mathbb{R}^{n_{c}}\times\mathbb{R}^{m}\to\mathbb{R}^{n_{c}}$ is a
Lipschitz continuous function, and $h_{c}:\mathbb{R}^{n_{c}}\to\mathbb{R}^{m}$
is a class C1 function. Assumption 1 is assumed for the input function
$g_{c}(u_{c})$. The subscript “c” indicates that this system serves as a
controller. We allow for the special case of a static system
$y_{c}=g_{c}(u_{c})$, that is, a special case of the system (6) which is NI
according to Definition 1 with storage function $V_{c}=0$. Accordingly, we
assume $f_{c}(0,0)=0$ and $h_{c}(0)=0$.

###### Assumption 5

For a system $H_{c}$ with a constant input $\overline{u}_{c}$ which results in
a constant output $\overline{y}_{c}$, then
$\overline{u}_{c}^{\top}\overline{y}_{c}\leq-\gamma_{c}\|\overline{u}_{c}\|^{2}$
with $\gamma_{c}>0.$

Figure 1: The feedback interconnection of the nonlinear system $H_{p}$ and the
nonlinear system $H_{c}$.

Further, consider the feedback interconnection of the nonlinear system $H_{p}$
and the nonlinear system $H_{c}$ as shown in Fig. 1. The relationship between
the inputs and the outputs are described by

$\displaystyle y_{p}\equiv h_{p}(x_{p})\equiv u_{c},$ (7a) $\displaystyle
y_{c}\equiv h_{c}(x_{c})+g_{c}(u_{c})\equiv u_{p}.$ (7b)

According to (7a) and (7b), we obtain the following equation

$u_{p}^{\top}y_{p}\equiv u_{c}^{\top}y_{c}.$ (8)

For the feedback system $(H_{p},H_{c})$ in Fig. 1, we define a candidate of
Luré-Postnikov-like Lyapunov function (Haddad and Chellaboina, 2008) as

$\displaystyle W(x_{p},x_{c})=$ $\displaystyle\
V_{p}(x_{p})+V_{c}(x_{c})-h_{p}(x_{p})^{\top}h_{c}(x_{c})$
$\displaystyle-\sum_{k=1}^{k=m}\int_{0}^{h_{p}^{k}(x_{p})}g_{c}^{k}(\xi^{k})d\xi^{k}.$
(9)

###### Assumption 6

There exists an open domain
$\mathcal{D}\subset\mathbb{R}^{n_{p}}\times\mathbb{R}^{n_{c}}$ containing the
origin such that the candidate Lyapunov function (2.2) is positive definite on
$\mathcal{D}$.

The following theorems establish stability results for the feedback system
$(H_{p},H_{c})$.

###### Theorem 1

Consider a nonlinear OSNI plant $H_{p}$ satisfying Assumptions 2, 3, and 4.
Also, consider a nonlinear controller $H_{c}$, either being a nonlinear static
controller or a nonlinear dynamic NI controller. Suppose Assumptions 1 and 5
hold for the nonlinear static controller, and Assumptions 1, 2, and 5 hold for
the nonlinear dynamic NI controller. Further, consider the feedback system
$(H_{p},H_{c})$ as depicted in Fig 1. Also, suppose Assumption 6 holds for the
feedback system $(H_{p},H_{c})$. Then, the feedback system $(H_{p},H_{c})$ is
locally asymptotically stable.

Proof. First, according to Assumption 6, the candidate Lyapunov function (2.2)
is positive definite in the domain $\mathcal{D}$. Second, since both the
static controller and the dynamic NI controller are NI, the time derivative of
the candidate Lyapunov function (2.2) is analyzed as follows:

$\displaystyle\dot{W}\overset{Eqs.~{}\eqref{eq:general_dissipation_inequality},\eqref{eq:strict_dissipation_inequality}}{\leq}$
$\displaystyle(u_{p}-h_{c}(x_{c})-g_{c}(h_{p}(x_{p})))^{\top}\dot{h}_{p}(x_{p})$
$\displaystyle-\epsilon\|\dot{h}_{p}(x_{p})\|^{2}+(u_{c}-h_{p}(x_{p}))^{\top}\dot{h}_{c}(x_{c})$
$\displaystyle\overset{Eq.~{}\eqref{eq:inputs_outputs}}{\leq}$
$\displaystyle-\epsilon_{p}\|\dot{h}_{p}(x_{p})\|^{2}\leq 0.$ (10)

Thus, in both the cases of a static controller and a dynamic controller, the
feedback system $(H_{p},H_{c})$ is at least locally stable in the sense of
Lyapunov.

Equation (2.2) implies that $\dot{W}$ can remain zero only if
$\dot{h}_{p}(x_{p})$ remains zero. This implies that the OSNI plant $H_{p}$
reaches steady state such that $\dot{h}_{p}(x_{p})\equiv
0\overset{\text{A}2}{\implies}\overline{y}_{p}\overset{\text{A}2}{\implies}\overline{x}_{p}\overset{\text{A}3}{\implies}\overline{u}_{p}\text{
and }\overline{u}_{p}^{\top}\overline{y}_{p}\overset{\text{A}4}{\geq}0.$ Then,
according to Eq. (7) and Eq. (6b), the nonlinear controller $H_{c}$ also
reaches a steady state such that
$\overline{u}_{c}^{\top}\overline{y}_{c}\overset{\text{A}5}{\leq}-\gamma_{c}\|\overline{u}_{c}\|^{2}.$
Thus, $\overline{u}_{c}$ can only be zero, which yields
$\overline{y}_{p}\overset{Eq.~{}\eqref{eq:y_p_u_c}}{\equiv}\overline{u}_{c}\equiv
0.$ Then, for the system $H_{p}$, we obtain that $\overline{y}_{p}\equiv
0\overset{\text{A}2}{\implies}\overline{x}_{p}\equiv 0.$

Next, consider the cases of a nonlinear static controller and a nonlinear
dynamic controller. In the case of a nonlinear static controller, there are no
dynamics. Thus, we can directly conclude that $\dot{W}$ cannot remain zero
unless $x_{p}=0$. According to LaSalle’s invariance principle, $W(x_{p})$ will
keep decreasing until $x_{p}=0$. In the case of a nonlinear dynamic NI
controller, where $\overline{x}_{p}\equiv
0\overset{\text{A}2,\text{A}3}{\implies}\overline{y}_{p}\equiv 0\text{ and
}\overline{u}_{p}\equiv
0\overset{Eq.~{}\eqref{eq:inputs_outputs}}{\implies}\overline{u}_{c}\equiv
0\text{ and }\overline{y}_{c}\equiv
0\overset{Eq.~{}\eqref{eq:h2_output}}{\implies}\dot{h}_{c}(x_{c})\equiv
0\overset{\text{A}2}{\implies}x_{c}\equiv 0$, we observe that $\dot{W}$ cannot
remain zero unless $x_{p}=0$ and $x_{c}=0$. According to LaSalle’s invariance
principle, $W(x_{p},x_{c})$ decreases until $x_{p}=0$ and $x_{c}=0$.
Therefore, in both cases, the feedback system $(H_{p},H_{c})$ is locally
asymptotically stable. The proof is now completed. $\square$

###### Theorem 2

Consider a nonlinear NI plant $H_{p}$ satisfying Assumptions 2, 3, and 4.
Also, consider a nonlinear OSNI controller $H_{c}$ satisfying Assumptions 1,
2, 3, and 5. Further, consider the feedback system $(H_{p},H_{c})$ as depicted
in Fig 1. Suppose Assumption 6 holds for the feedback system $(H_{p},H_{c})$.
Then, the feedback system $(H_{p},H_{c})$ is locally asymptotically stable.

Proof. The proof is similar to that for Theorem 1. $\square$

##  3 Output Consensus of Networked NI Systems

In this section, a network setting is considered, and output consensus results
are presented for the feedback interconnection of two networked NI systems.

###  3.1 Settings for Networked NI Systems

Network Setting. In what follows, we consider a connected and undirected
network $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where
$\mathcal{V}=\\{1,2,\dots,N\\}$ describes the set of $N$ nodes, and
$\mathcal{E}=\\{e_{1},e_{2},\dots,e_{L}\\}\subseteq\mathcal{V}\times\mathcal{V}$
represents the set of $L$ edges connecting the nodes. The index set for edges
is denoted by $\mathcal{L}=\\{1,2,\dots,L\\}.$ Each node is associated with an
independent nonlinear plant, while each edge is deployed with a nonlinear
controller. Each edge takes the outputs of two end nodes as its input, and
each node takes the outputs of its connected edges as its input.

If there exists an edge $(i,j)\in\mathcal{E}$ connecting node
$i\in\mathcal{V}$ and node $j\in\mathcal{V}$, then nodes $i$ and $j$ are
considered neighbors. The neighbors of node $i\in\mathcal{V}$ are indexed in
the set $\mathcal{N}(i)$. The edges that contain node $i$ are indexed in the
set $\mathcal{E}(i)$. The incidence matrix $\mathbf{Q}\in\mathbb{R}^{N\times
L}$ of the network is defined as follows:

$\mathbf{Q}_{ie}=\begin{cases}1,&\text{if }i\text{ is the initial node of edge
}e,\\\ -1,&\text{if }i\text{ is the terminal node of edge }e,\\\ 0,&\text{if
}i\text{ is not connected in edge }e.\end{cases}$

It is noted that a fixed representation of edges is chosen, where each $(i,j)$
or $(j,i)$ can only be chosen once, and “initial/terminal node” does not refer
to a specific orientation.

Node Plants. Each node $i\in\mathcal{V}$ is associated with an independent
nonlinear plant $H_{pi}$ described by:

$\displaystyle H_{pi}:\quad\dot{x}_{pi}$
$\displaystyle=f_{pi}(x_{pi},u_{pi}),$ (11a) $\displaystyle y_{pi}$
$\displaystyle=h_{pi}(x_{pi}),$ (11b)

where $x_{pi}\in\mathbb{R}^{n_{pi}}$ is the state, $u_{pi}\in\mathbb{R}^{m}$
is the input, $y_{pi}\in\mathbb{R}^{m}$ is the output,
$f_{pi}:\mathbb{R}^{n_{pi}}\times\mathbb{R}^{m}\to\mathbb{R}^{n_{pi}}$ is a
Lipschitz continuous function, and
$h_{pi}:\mathbb{R}^{n_{pi}}\to\mathbb{R}^{m}$ is a class C1 function. Also, we
assume $f_{pi}(0,0)=0$ and $h_{pi}(0)=0$. For a compact expression, we collect
the states, inputs and outputs of all nodes — as represented by the aggregated
state vector
$X_{p}=[x_{p1}^{\top},\dots,x_{pN}^{\top}]^{\top}\in\mathbb{R}^{n_{p}}$ with
$n_{p}=\sum_{i=1}^{N}n_{pi}$, the aggregated input vector
$U_{p}=[u_{p1}^{\top},\dots,u_{pN}^{\top}]^{\top}\in\mathbb{R}^{mN}$, and the
aggregated output vector
$Y_{p}=[y_{p1}^{\top},\dots,y_{pN}^{\top}]^{\top}\in\mathbb{R}^{mN}$. We
denote the aggregated node plants by $\mathcal{H}_{p}$, which is described by

$\displaystyle\mathcal{H}_{p}:\dot{X}_{p}=\begin{bmatrix}f_{p1}(x_{p1},u_{p1})\\\
\vdots\\\
f_{pN}(x_{pN},u_{pN})\end{bmatrix},Y_{p}=\begin{bmatrix}h_{p1}(x_{p1})\\\
\vdots\\\ h_{pN}(x_{pN})\end{bmatrix}.$

Edge Controllers. Each edge $e_{l}\in\mathcal{E}$ with $l\in\mathcal{L}$ is
deployed with a nonlinear controller described by

$\displaystyle H_{cl}:\quad\dot{x}_{cl}$
$\displaystyle=f_{cl}(x_{cl},u_{cl}),$ (12a) $\displaystyle y_{cl}$
$\displaystyle=h_{cl}(x_{cl})+g_{cl}(u_{cl}),$ (12b)

where $x_{cl}\in\mathbb{R}^{n_{cl}}$ is the state, $u_{cl}\in\mathbb{R}^{m}$
is the input, $y_{cl}\in\mathbb{R}^{m}$ is the output,
$f_{cl}:\mathbb{R}^{n_{cl}}\times\mathbb{R}^{m}\to\mathbb{R}^{n_{cl}}$ is a
Lipschitz continuous function, and
$h_{cl}:\mathbb{R}^{n_{cl}}\to\mathbb{R}^{m}$ is a class C1 function. Also, we
assume $f_{cl}(0,0)=0$ and $h_{cl}(0)=0$. Assumption 1 is assumed for the
input functions $g_{cl}(u_{cl}),l\in\mathcal{L}$. We allow for the case of a
static system $y_{cl}=g_{cl}(x_{cl}),l\in\mathcal{L}$ as a special case of the
system (12). For a compact expression, we collect the states, the inputs and
the outputs of all edges into the aggregated state vector
$X_{c}=[x_{c1}^{\top},\dots,x_{cL}^{\top}]^{\top}\in\mathbb{R}^{n_{c}}$ with
$n_{c}=\sum_{l\in\mathcal{L}}n_{cl}$, the aggregated input vector
$U_{c}=[u_{c1}^{\top},\dots,u_{cL}^{\top}]^{\top}\in\mathbb{R}^{mL}$, and the
aggregated output vector

$Y_{c}=[y_{c1}^{\top},\dots,y_{cL}^{\top}]^{\top}=\Pi_{cx}(X_{c})+\Pi_{cu}(U_{c})\in\mathbb{R}^{mL},$
(13)

where

$\displaystyle\Pi_{cx}(X_{c})=[h_{c1}(x_{c1})^{\top},\dots,h_{cL}(x_{cL})^{\top}]^{\top}\in\mathbb{R}^{mL},$
(14a)
$\displaystyle\Pi_{cu}(U_{c})=[g_{c1}(u_{c1})^{\top},\dots,g_{cL}(u_{cL})^{\top}]^{\top}\in\mathbb{R}^{mL}.$
(14b)

We denote the aggregated nonlinear controllers by $\mathcal{H}_{c}$, which are
described by

$\displaystyle\mathcal{H}_{c}:\dot{X}_{c}=\begin{bmatrix}f_{c1}(x_{c1},u_{c1})\\\
\vdots\\\
f_{cL}(x_{cL},u_{cL})\end{bmatrix},Y_{c}=\begin{bmatrix}h_{c1}(x_{c1})+g_{c1}(u_{c1})\\\
\vdots\\\ h_{cL}(x_{cL})+g_{cL}(u_{cL})\end{bmatrix}.$

Output Feedback Control Framework. The objective of our control problem is to
achieve output consensus for each node in the network. We now define local
output consensus.

######  Definition 3 (Local Output Consensus)

A distributed output feedback control law achieves local output feedback
consensus for a networked system if there exists an open domain
$\mathcal{D}_{c}\subset\mathbb{R}^{n_{p}\times n_{c}}$ containing the origin
such that $\lim_{t\to\infty}\|y_{pi}(t)-y_{pj}(t)\|=0,$ for all
$i,j\in\mathcal{V},$ for all initial conditions
$(X_{p}(0),X_{c}(0))\in\mathcal{D}_{c}.$

As depicted in Fig. 2, a distributed output feedback control framework
naturally arises based on the incidence matrix $\mathbf{Q}$. We denote the
networked node plants by $\widehat{\mathcal{H}}_{p}=(\mathbf{Q}^{\top}\otimes
I_{m})\mathcal{H}_{p}(\mathbf{Q}\otimes I_{m})$. We further denote the
feedback interconnection of the networked node plants
$\widehat{\mathcal{H}}_{p}$ and the aggregated edge controllers
$\mathcal{H}_{c}$ by $(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$. The
relationship between the inputs and the outputs of the feedback system
$(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$ in Fig. 2 are described by

$\displaystyle\widehat{U}_{p}\equiv Y_{c}\equiv\Pi_{cx}+\Pi_{cu},$ (15a)
$\displaystyle\widehat{Y}_{p}\equiv U_{c},$ (15b) $\displaystyle
U_{p}=(\mathbf{Q}\otimes I_{m})\widehat{U}_{p},$ (15c)
$\displaystyle\widehat{Y}_{p}\equiv(\mathbf{Q}^{\top}\otimes I_{m})Y_{p}.$
(15d)

In a distributed manner, each edge controller $l\in\mathcal{L}$ takes the
difference between the outputs of the neighbouring nodes $i$ and $j$ as its
input, $u_{cl}=\sum_{k=1}^{N}q_{kl}y_{pk}=y_{pi}-y_{pj},$ where $q_{kl}$
represents the $k$th element in the $l$th column of the incidence matrix
$\mathbf{Q}$, and the node $i$ and the node $j$ are the initial node and the
terminal node of the edge $e_{l}$, respectively. Each node plant
$i\in\mathcal{V}$ takes the sum of the outputs from all its connected edge
controllers as its input, $u_{pi}=\sum_{l=1}^{L}q_{il}y_{cl},$ where $q_{il}$
is the $l$th element in the $i$th row of the incidence matrix $\mathbf{Q}.$

Figure 2: The feedback interconnection of nonlinear plants $\mathcal{H}_{p}$
and nonlinear edge controllers $\mathcal{H}_{c}$ based on the incidence matrix
$\mathbf{Q}$.

###  3.2 Main Results

NI/OSNI Property Preservation. Let the storage function for each node plant
$H_{pi},i\in\mathcal{V}$ be denoted by $V_{pi}$ and let the storage function
for each edge controller $H_{cl},l\in\mathcal{L}$ be denoted by $V_{cl}$,
respectively. The storage functions for the aggregated node plants
$\mathcal{H}_{p}$ and the aggregated edge controller $\mathcal{H}_{c}$ are
chosen as $V_{p}=\sum_{i\in\mathcal{V}}V_{pi}$ and
$V_{c}=\sum_{l\in\mathcal{L}}V_{cl}$, respectively. For the networked node
plants $\widehat{\mathcal{H}}_{p}$, the storage function is chosen as the same
for the aggregated node plants $\mathcal{H}_{p}$; i.e.,
$\widehat{V}_{p}=V_{p}=\sum_{i\in\mathcal{V}}V_{pi}.$

In the following two lemmas, we show that aggregation preserves the NI and
OSNI properties.

###### Lemma 1

If each edge controller $H_{cl},l\in\mathcal{L}$ is a nonlinear NI (OSNI)
system, then the aggregated edge controllers $\mathcal{H}_{c}$ is also a
nonlinear NI (OSNI) system.

Proof. (1) NI Property Preservation. According to Definition 1, each edge
controller $H_{cl},l\in\mathcal{L}$ has a positive semidefinite storage
function $V_{cl}(x_{cl})$ such that for all $t\geq 0$,
$\dot{V}_{cl}(x_{cl})\leq u_{cl}^{\top}\dot{h}_{cl}(x_{cl}).$ Next, we analyze
the time derivative of $V_{c}$:

$\dot{V}_{c}=\sum_{l\in\mathcal{L}}\dot{V}_{cl}(x_{cl})\leq
U_{c}^{\top}\dot{\Pi}_{cx}.$ (16)

Therefore, the aggregated edge controllers $\mathcal{H}_{c}$ also satisfies
Definition 1.

(2) OSNI Property Preservation: According to Definition 2, each edge
controller $H_{cl},l\in\mathcal{L}$ has a positive semidefinite storage
function $V_{cl}(x_{cl})$ and a scalar $\epsilon_{l}$ such that for all $t\geq
0$, $\dot{V}_{cl}(x_{cl})\leq
u_{cl}^{\top}\dot{h}_{cl}(x_{cl})-\epsilon_{cl}\|\dot{h}_{cl}(x_{cl})\|^{2}.$
Then, we analyze the time derivative of $V_{c}$:

$\dot{V}_{c}=\sum_{l\in\mathcal{L}}\dot{V}_{cl}(x_{cl})\leq
U_{c}^{\top}\dot{\Pi}_{cx}-\epsilon_{c\min}\|\dot{\Pi}_{cx}\|^{2},$ (17)

where $\epsilon_{c\min}=\min\\{\epsilon_{c1},\dots,\epsilon_{cL}\\}>0.$
Therefore, the aggregated edge controllers $\mathcal{H}_{c}$ also satisfies
Definition 2. The proof is now completed. $\square$

###### Lemma 2

If each node plant $H_{pi},i\in\mathcal{V}$ is a nonlinear NI (OSNI) system,
then the aggregated node plants $\mathcal{H}_{p}$ are also a nonlinear NI
(OSNI) system.

Proof. The form of the node plants is a special case of the form of the edge
controller where the output is only determined by the state. Therefore, the
proof of Lemma 1 applies for Lemma 2. The proof is now completed. $\square$

In the following lemma, we present a preliminary result that is needed for the
proof of output consensus.

###### Lemma 3

If each node plant $H_{pi},i\in\mathcal{V}$ is a nonlinear NI system, then the
storage function for the networked node plants $\widehat{\mathcal{H}}_{p}$ is
such that

$\dot{\widehat{V}}_{p}\leq\widehat{U}_{p}^{\top}\dot{\widehat{Y}}_{p}.$ (18)

Also, if each node plant $H_{pi},i\in\mathcal{V}$ is a nonlinear OSNI system,
then the storage function for the networked node plants
$\widehat{\mathcal{H}}_{p}$ are such that

$\dot{\widehat{V}}_{p}\leq\widehat{U}_{p}^{\top}\dot{\widehat{Y}}_{p}-\epsilon_{p\min}\|\dot{Y}_{p}\|^{2},$
(19)

where $\epsilon_{p\min}=\min\\{\epsilon_{p1},\dots,\epsilon_{pN}\\}$.

Proof.  (1) NI case. If each node plant $H_{pi},i\in\mathcal{V}$ is NI, then
it has a positive semidefinite storage function $V_{pi}(x_{pi})$ such that for
all $t\geq 0$, $\dot{V}_{pi}(x_{pi})\leq u_{pi}^{\top}\dot{h}_{pi}(x_{pi}).$
In compact form, we have
$\dot{\widehat{V}}_{p}=\sum_{i\in\mathcal{V}}\dot{V}_{pi}(x_{pi})\leq
U_{p}^{\top}\dot{Y}_{p}.$ According to the input-output relation (15), we have
$U_{p}^{\top}\dot{Y}_{p}=\widehat{U}_{p}^{\top}(\mathbf{Q}^{\top}\otimes
I_{m})\dot{Y}_{p}=\widehat{U}_{p}^{\top}\dot{\widehat{Y}}_{p}.$ Therefore, we
obtain $\dot{\widehat{V}}_{p}\leq\widehat{U}_{p}^{\top}\dot{\widehat{Y}}_{p}$.

(2) OSNI case. If each node plant $H_{pi},i\in\mathcal{V}$ is OSNI, then it
has a positive semidefinite storage function $V_{pi}(x_{pi})$ such that for
all $t\geq 0$, $\dot{V}_{pi}(x_{pi})\leq
u_{pi}^{\top}\dot{h}_{pi}(x_{pi})-\epsilon_{pi}\|\dot{h}_{pi}(x_{pi})\|^{2}.$
Similar to the proof in (1), we obtain the compact form
$\dot{\widehat{V}}_{p}=\sum_{i\in\mathcal{V}}\dot{V}_{pi}(x_{pi})\leq\widehat{U}_{p}^{\top}\dot{\widehat{Y}}_{p}-\epsilon_{p\min}\|\dot{Y}_{p}\|^{2},$
where $\epsilon_{p\min}=\min\\{\epsilon_{p1},\dots,\epsilon_{pN}\\}$. The
proof is now completed. $\square$

Output Consensus. For the feedback system
$(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$, a candidate Lyapunov function
is selected as

$\displaystyle\widehat{W}=$
$\displaystyle\sum_{i\in\mathcal{V}}V_{pi}(x_{pi})+\sum_{l\in\mathcal{L}}V_{cl}(x_{cl})-\widehat{Y}_{p}^{\top}\Pi_{cx}$
$\displaystyle-\sum_{k=1}^{mL}\int_{0}^{\widehat{Y}_{p}^{k}}\Pi_{cu}^{k}(\xi^{k})d\xi^{k}.$
(20)

###### Assumption 7

There exists an open domain
$\mathcal{D}\subset\mathbb{R}^{n_{p}}\times\mathbb{R}^{n_{c}}$ such that the
candidate Lyapunov function (3.2) is positive definite.

In light of the stability results presented in Section 2, where Assumption 4
is imposed on the plant, we extend a comparable assumption to the system
$\widehat{\mathcal{H}}_{p}$. This parallels the conditions for the plant,
giving a consistent framework for our output consensus results for networked
systems.

###### Assumption 8

For the system $\widehat{\mathcal{H}}_{p}$ with a constant input
$\overline{\widehat{U}}_{p}$ which results in a constant output
$\overline{\widehat{Y}}_{p}$, then
$\overline{\widehat{U}}_{p}^{\top}\overline{\widehat{Y}}_{p}\geq 0.$

The following theorems establish output consensus results for the feedback
system $(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$.

###### Theorem 3

Consider a nonlinear OSNI node plants $H_{pi},i\in\mathcal{V}$. Suppose
Assumptions 2 and 3 hold for each nonlinear OSNI node plant, and Assumption 8
holds for the networked node plants $\widehat{\mathcal{H}}_{p}$. Also consider
nonlinear edge controllers $H_{cl},l\in\mathcal{L}$, each of which is either a
static controller or a dynamic NI controller. Suppose Assumptions 1 and 5 hold
for the nonlinear static controllers, and Assumptions 1, 2, and 5 hold for the
nonlinear dynamic NI controllers. Further, consider the feedback system
$(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$. Suppose Assumption 7 holds for
the feedback system. Then, local output consensus is achieved.

Proof. First, according to Assumption 7, the candidate Lyapunov function (3.2)
of the feedback system $(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$ is
positive definite on an open domain $\mathcal{D}$. Second, we analyze the time
derivative of the candidate Lyapunov function (3.2):

$\displaystyle\dot{\widehat{W}}\overset{Eqs.~{}\eqref{eq:V_hat_OSNI},\eqref{eq:NI_dot_V_c}}{\leq}$
$\displaystyle\
(\widehat{U}_{p}-\Pi_{cx}-\Pi_{cu})^{\top}\dot{\widehat{Y}}_{p}-\epsilon_{p\min}\|\dot{Y}_{p}\|^{2}$
$\displaystyle+(U_{c}-Y_{p})^{\top}\dot{\Pi}_{cx}$
$\displaystyle\overset{Eq.~{}\eqref{eq:network_input_output}}{\leq}$
$\displaystyle-\epsilon_{p\min}\|\dot{Y}_{p}\|^{2}\leq 0,$ (21)

where $\epsilon_{p\min}=\min\\{\epsilon_{p1},\dots,\epsilon_{pN}\\}>0.$
Therefore, the feedback system $(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$
is at least locally stable in the sense of Lyapunov.

Equation (21) implies that $\dot{\widehat{W}}$ can remain zero only if
$\dot{Y}_{p}$ remains zero. This implies the aggregated node plants
$\mathcal{H}_{p}$ reach a steady state such that $\dot{Y}_{p}\equiv
0\overset{\text{A}2}{\implies}\overline{Y}_{p}\overset{\text{A}2}{\implies}\overline{X}_{p}\overset{\text{A}3}{\implies}\overline{U}_{p}.$
The aggregated edge controllers $\mathcal{H}_{c}$ consists of static
controllers and dynamic NI controllers. The static controllers reach a steady
state such that
$\overline{U}_{c}^{[s]}\overset{Eq.~{}\eqref{eq:Y_hat_p_U_c}}{\equiv}\overline{\widehat{Y}}_{p}^{[s]}\overset{Eq.~{}\eqref{eq:Y_hat_p_Y_p}}{\equiv}(\mathbf{Q}^{\top}Y_{p})^{[s]}\implies\overline{Y}_{c}^{[s]}\equiv\Pi_{cu}^{[s]}(\overline{U}_{c}^{[s]})$,
where the supscript $[s]$ represents those static controllers. The dynamic
controllers reach a steady state such that
$\overline{U}_{p}\equiv\overline{\mathbf{Q}Y_{c}}\text{ and
}\overline{U}_{c}\equiv\overline{\widehat{Y}}_{p}\overset{Eq.~{}\eqref{eq:output_vector}}{\implies}\overline{Q\Pi_{cx}}\overset{\text{A}2}{\implies}\overline{X}_{c}\overset{\text{A}2,\text{A}3}{\implies}\overline{U}_{c}^{[d]}\text{
and }\overline{Y}_{c}^{[d]}$. Thus, we conclude that the networked node plants
$\widehat{\mathcal{H}}_{p}$ are subject to the constant input
$\overline{\widehat{U}}_{p}$ and have constant output
$\overline{\widehat{Y}}_{p}$, where the aggregated edge controllers are
subject to constant input $\overline{U}_{c}$ and constant output
$\overline{Y}_{c}$. Then,
$\overline{\widehat{U}}_{p}^{\top}\overline{\widehat{Y}}_{p}\equiv\overline{U}_{c}^{\top}\overline{Y}_{c},\
\overline{\widehat{U}}_{p}^{\top}\overline{\widehat{Y}}_{p}\overset{\text{A}8}{\geq}0,\text{
and
}\overline{U}_{c}^{\top}\overline{Y}_{c}\overset{\text{A}5}{\leq}-\gamma_{c\min}\|\overline{U}_{c}\|^{2},$
where $\gamma_{c\min}=\min\\{\gamma_{c1},\dots,\gamma_{cL}\\}.$ This is only
possible when $\overline{U}_{c}\equiv\overline{\widehat{Y}}_{p}\equiv 0$;
i.e., achieving local output consensus. Therefore, we can conclude that
$\dot{\widehat{W}}$ cannot remain zero unless $\overline{\widehat{Y}}_{p}=0$.
The proof is now completed. $\square$

###### Theorem 4

Consider nonlinear NI node plants $H_{pi},i\in\mathcal{V}$. Suppose
Assumptions 2 and 3 hold for each nonlinear NI node plant, and Assumption 8
holds for the networked node plants $\widehat{\mathcal{H}}_{p}$. Also consider
nonlinear OSNI edge controllers $H_{cl},l\in\mathcal{L}$. Suppose Assumptions
1, 2, 3, and 5 hold for nonlinear dynamic OSNI controllers. Further, consider
the feedback system $(\widehat{\mathcal{H}}_{p},\mathcal{H}_{c})$. Suppose
Assumption 7 holds for the feedback system. Then, local output consensus is
achieved.

Proof. The proof is similar to that for Theorem 3. $\square$

##  4 Application to Power Transmission Systems

In this section, we apply the proposed theoretical results to the practical
problem of frequency synchronization and angle consensus in power transmission
systems.

###  4.1 Transmission Network Model

Consider a transmission network comprised of $N$ nodes representing
synchronous generator buses and $L$ edges representing transmission lines. The
topology of the transmission network is represented by a connected and
undirected graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$. A generator bus has
several components, including an AC generator, and fixed inflexible load. The
AC generator converts mechanical power into electric power through a rotating
prime mover. The fixed inflexible load consumes a known amount of power.

For the transmission network, we denote the nominal frequency by $\omega^{0}$.
For each generator bus $i\in\mathcal{V}$, we denote the rotor speed by
$\omega_{i}$, and the rotor angle with respect to a rotating reference frame
at the speed of $\omega^{0}$ by $\delta_{i}$. We adopt the following
assumptions that are well-justified for power transmission systems (Kundur and
Malik, 2022).

  1. (i)

Each transmission line $(i,j)\in\mathcal{E}$ is lossless and is thereby
characterized by its reactance $X_{ij}$.

  2. (ii)

The internal voltage magnitude $E_{i}^{0}$ of each generator bus
$i\in\mathcal{V}$ is constant.

  3. (iii)

Reactive power injections on buses are controlled to maintain an internal
voltage magnitude $E_{i}^{0}$ for each generator bus $i\in\mathcal{V}$, where
such controllers are decoupled from frequency and angle regulation in
transmission grids.

It is important to note that in this paper we do not conduct a small-signal
stability analysis as our results are valid for both small and large rotor
angle differences between generator buses.

Interconnected Swing Equations. Under assumptions (i)-(iii), the dynamics of
each generator bus $i\in\mathcal{V}$ can be described by the following
nonlinear swing equation (Dorfler and Bullo, 2012):

$\displaystyle\dot{\delta}_{i}$ $\displaystyle=\omega_{i}-\omega^{0},$ (22a)
$\displaystyle\dot{\omega}_{i}$
$\displaystyle=\frac{1}{M_{i}}\Big{[}P^{M}_{i}-D_{i}(\omega_{i}-\omega^{0})-P^{L}_{i}-P^{E}_{i}\Big{]}.$
(22b)

Here, $M_{i}>0$ represents the inertia coefficient of generator bus $i$, and
$D_{i}>0$ represents the damping coefficient of generator bus $i$. The
variables are defined as follows: $P^{M}_{i}$ represents the mechanical power
injection to generator bus $i$, $P^{L}_{i}$ represents the fixed power
consumption of the load at generator bus $i$, and $P^{E}_{i}$ represents the
electric power output of generator bus $i$ to the transmission network. The
electric power output of generator bus $i$ equals the net branch power flow
from generator bus $i$ to other neighboring buses, represented by
$P^{E}_{i}=\sum_{j\in\mathcal{N}(i)}P_{ij}.$ The branch power flow $P_{ij}$
from generator bus $i$ to a neighboring bus $j\in\mathcal{N}(i)$ is
characterized by

$P_{ij}=\frac{E_{i}^{0}E_{j}^{0}}{X_{ij}}\sin(\delta_{i}-\delta_{j})=P_{ij}^{\max}\sin(\delta_{i}-\delta_{j}),$
(23)

where $X_{ij}$ is the reactance of the transmission line $(i,j)$, and
$P_{ij}^{\max}=\frac{E_{i}^{0}E_{j}^{0}}{X_{ij}}$ is the maximum power
transfer on the transmission line $(i,j)$.

Combining Eqs. (22) and (23), the interconnected swing equations for the
transmission network in terms of rotor angles $\delta_{i}$, for all
$i\in\mathcal{V}$, are therefore given by

$M_{i}\ddot{\delta_{i}}+D_{i}\dot{\delta_{i}}=P^{M}_{i}-P^{L}_{i}-\sum_{j\in\mathcal{N}(i)}P_{ij}^{\max}\sin(\delta_{i}-\delta_{j}).$
(24)

In practice, faults arise in power systems that can cause a sudden change in
the energy drawn through transmission lines (e.g., a power line falls to the
ground resulting in a change in network impedance). Prior to a fault
occurrence, the power transmission system (24) is assumed to be at a stable
equilibrium. Denote the steady-state rotor angle difference between two
neighboring buses $i$ and $j$ by
$\overline{\psi}_{ij}=\overline{\delta}_{i}-\overline{\delta}_{j}.$ Denote the
stable equilibrium by

$(\overline{P}_{i}^{M},\overline{P}_{i}^{L},\overline{P}_{ij}^{\max},\overline{\psi}_{ij}),\forall
i\in\mathcal{V},\forall(i,j)\in\mathcal{E}.$ (25)

The relation between steady-state values is described by

$\overline{P}^{M}_{i}-\overline{P}^{L}_{i}-\sum_{j\in\mathcal{N}(i)}\overline{P}_{ij}^{\max}\sin\overline{\psi}_{ij}=0.$
(26)

In the aftermath of a fault, the bus frequencies
$\omega_{i},\forall\in\mathcal{V}$ deviate from their nominal frequency
$\omega^{0}$ and the angle differences
$\delta_{i}-\delta_{j},\forall(i,j)\in\mathcal{E}$ deviate from their steady-
state values $\overline{\psi}_{ij},\forall(i,j)\in\mathcal{E}$.

###  4.2 Resilience in Power Transmission Systems

In what follows, we investigate transient conditions in the aftermath of
fault, during which the mechanical power of synchronous generators, the load
power consumption, and the maximum power transfer throughout the bulk grid,
all return to pre-fault conditions. We also define the resilience of existing
power transmission systems in the aftermath of a fault. Specifically, we
consider the initial power angle deviation during transient conditions and
asertain if it is in a suitable domain to enable the power transmission system
to synchronize bus frequencies to the nominal and recover angle differences to
their pre-fault values.

For each generator bus $i\in\mathcal{V}$, we denote the angle deviation from
the pre-fault angle by
$\widetilde{\delta}_{i}=\delta_{i}-\overline{\delta}_{i}$. For each
transmission line $(i,j)\in\mathcal{E}$, we denote the angle deviation
difference between the two connected generator buses by
$\widetilde{\psi}_{ij}=\widetilde{\delta}_{i}-\widetilde{\delta}_{j}$. Using
Eq. (24) and Eq. (26), the interconnected swing equations can be specifically
rewritten in terms of rotor angle deviations
$\widetilde{\delta}_{i},i\in\mathcal{V}$:

$\displaystyle
M_{i}\ddot{\widetilde{\delta}}_{i}+D_{i}\dot{\widetilde{\delta}}_{i}=\sum_{j\in\mathcal{N}(i)}\overline{P}_{ij}^{\max}\big{(}\sin\overline{\psi}_{ij}-\sin(\widetilde{\psi}_{ij}+\overline{\psi}_{ij})\big{)},$
(27)

with initial angle deviations $\widetilde{\delta}_{i}(0),\forall
i\in\mathcal{V}.$

By reformulating the system (27) into the feedback interconnection of OSNI
node plants and static edge controllers based on the underlying transmission
network, we can utilize the result of output consensus as established in
Section 3 to prove existing power transmission systems can synchronize bus
frequencies and recover angle differences when the initial deviation is in a
suitable domain.

Define
$x_{pi}=[\dot{\widetilde{\delta}}_{pi},\widetilde{\delta}_{pi}]^{\top}\in\mathbb{R}^{2}$,
$u_{pi}\in\mathbb{R}$, $x_{cl}\equiv 0$, and
$u_{cl}=\widetilde{\psi}_{ij}\in\mathbb{R}$ with $j\in\mathcal{N}(i)$. The
system (27) can be represented by the interconnection of the node systems and
the edge systems based on the transmission network:

$\displaystyle H_{pi}\ :\quad\dot{x}_{pi}$
$\displaystyle=A_{pi}x_{pi}+B_{pi}u_{pi},$ (28a) $\displaystyle y_{pi}$
$\displaystyle=C_{pi}x_{pi},$ (28b) $\displaystyle H_{cl}\ :\quad y_{cl}$
$\displaystyle=\overline{P}^{\max}_{l}\big{(}\sin\overline{\psi}_{l}-\sin(u_{cl}+\overline{\psi}_{l})\big{)},$
(28c)

for all $i\in\mathcal{V}$ and all $l\in\mathcal{L}$. The system $H_{pi}$ has
system matrices

$A_{pi}=\begin{bmatrix}\frac{-D_{i}}{M_{i}}&0\\\ 1&0\end{bmatrix},\
B_{pi}=\begin{bmatrix}\frac{1}{M_{i}}\\\ 0\end{bmatrix},\text{ and }\
C_{pi}=\begin{bmatrix}0&1\end{bmatrix}.$

The system $H_{cl}$ has system parameters

$\overline{P}^{\max}_{l}=\overline{P}^{\max}_{ij}\text{ and
}\sin\overline{\psi}_{l}=\sin\overline{\psi}_{ij},$

when the node $i$ and the node $j$ are the initial node and the terminal node,
respectively. The input-output relation induced from the feedback
interconnection based on the transmission network is such that

$\displaystyle u_{pi}=\sum_{l\in\mathcal{L}}q_{il}y_{cl}\ \text{ and }\
u_{cl}=\sum_{k\in\mathcal{V}}q_{kl}y_{pk}.$ (29)

###### Remark 1

The system (27) does not inherently incorporate controllers. For the sake of
applying our results established in Section 3 more directly, we designate the
left side of Eq. (27) as “node plants” corresponding to actual dynamics in
generator buses and the right side as “edge controllers” corresponding to
power flows on transmission lines. Furthermore, this choice provides insights
into how we can leverage batteries to create virtual transmission lines,
thereby enhancing the robustness margin of the power transmission systems in
Section 4.3.

In the following theorem, we prove that the power transmission system can
achieve local output consensus, thereby regulating bus frequencies and
restoring power angle differences. Denote a local domain by
$\mathcal{D}=\mathcal{D}_{1}\cap\mathcal{D}_{2},$ where

$\displaystyle\mathcal{D}_{1}=\\{\widetilde{\delta}_{i},$
$\displaystyle\forall i\in\mathcal{V}\ |\
\widetilde{\psi}_{l}\in(-\pi-2\overline{\psi}_{l},\pi-2\overline{\psi}_{l}),\forall
l\in\mathcal{L}\\},$
$\displaystyle\mathcal{D}_{2}=\bigl{\\{}\widetilde{\delta}_{i},$
$\displaystyle\forall i\in\mathcal{V}\ |\
\sum_{l\in\mathcal{L}}\overline{P}_{l}^{\max}\big{(}\cos\overline{\psi}_{l}-\widetilde{\psi}_{l}\sin\overline{\psi}_{l}$
$\displaystyle-\cos(\widetilde{\psi}_{l}+\overline{\psi}_{l})\big{)}>0\bigr{\\}}\cup\\{\widetilde{\delta}_{i}=0,\forall
i\in\mathcal{V}\\}.$

###### Theorem 5

Consider node plants $\mathcal{H}_{p}$ described by (28a)-(28b) and edge
controllers $\mathcal{H}_{c}$ described by (28c). Consider the feedback
interconnection of node plants and edge controllers based on the underlying
transmission network as depicted in Fig. 2. Then, the feedback system achieves
local output consensus.

Proof. First, since each node plant is a linear system, Assumptions 2 and 3
are satisfied. We are to show that each node plant (28a)-(28b) is OSNI. We
choose the storage function of each node plant as

$\displaystyle V_{pi}=x_{pi}^{\top}P_{pi}x_{pi}=[\dot{\widetilde{\delta}}_{i}\
\widetilde{\delta}_{i}]\begin{bmatrix}\frac{M_{i}}{2}&0\\\
0&0\end{bmatrix}[\dot{\widetilde{\delta}}_{i}\
\widetilde{\delta}_{i}]^{\top}=\frac{M_{i}}{2}\dot{\widetilde{\delta}}_{i}^{2}.$

We analyze the time derivative of the storage function of each node plant:

$\displaystyle\dot{V}_{pi}=\dot{\widetilde{\delta}}_{i}u_{pi}-D_{i}\dot{\widetilde{\delta}}_{i}^{2}\leq
u_{pi}^{\top}\dot{h}_{pi}(x_{pi})-\epsilon_{pi}\|\dot{h}_{pi}(x_{pi})\|^{2}.$

Thus, with $0<\epsilon_{pi}\leq D_{i}$, each node plant satisfies the
definition of OSNI systems. Furthermore, according to Lemma 3, the networked
node plants $\widehat{\mathcal{H}}_{p}$ is at least linear NI. Every linear NI
system satisfies Assumption 8 (Shi et al., 2023). Second, for static edge
controllers, we can verify that Assumption 1 is satisfied; i.e.,
$y_{cl}(0)=0$, and also validate that Assumption 5 is satisfied in the domain
$\mathcal{D}_{1}.$ Third, for the feedback system
$(\mathcal{H}_{p},\mathcal{H}_{c})$, we can validate Assumption 6, where the
candidate Lyapunov function (3.2) is positive definite in the domain
$\mathcal{D}_{2}$. Therefore, Theorem 3 can be applied and the feedback system
achieves local consensus. The proof is completed. $\square$

###  4.3 Battery-based Angle Feedback Controllers

In what follows, we equip generator buses with large-scale batteries and we
design angle based feedback controllers for the battery-based actuators. We
design the angle based feedback controllers to enhance the transient stability
of the power transmission system.

We consider equipping batteries at generator buses. Hence, modifying Eq. (24),
the swing equation is revised as

$M_{i}\ddot{\delta_{i}}+D_{i}\dot{\delta_{i}}=P^{M}_{i}+P^{ST}_{i}-P^{L}_{i}-P^{E}_{i},$
(30)

where $P^{ST}_{i}$ represents the power output of the battery at each
generator bus $i\in\mathcal{V}$.

Prior to a fault occurrence, the system (30) is at a stable equilibrium
$(\overline{P}_{i}^{M},\overline{P}_{i}^{ST},\overline{P}_{i}^{L},\overline{P}_{ij}^{\max},\overline{\psi}_{ij}),\forall
i\in\mathcal{V},\forall(i,j)\in\mathcal{E},$ where
$\overline{P}^{M}_{i}+\overline{P}_{i}^{ST}-\overline{P}^{L}_{i}-\sum_{j\in\mathcal{N}(i)}\overline{P}_{ij}^{\max}\sin\overline{\psi}_{ij}=0.$
In the aftermath of a fault, the generator bus frequency deviates from its
nominal value and the angle differences deviate from pre-fault conditions. In
what follows, we investigate the post-fault transients, during which the
mechanical power of synchronous generators and the maximum power transfer of
the transmission network return back to pre-fault values.

We define the change in storage power output,
$\widetilde{P}^{ST}_{i}=P^{ST}_{i}-\overline{P}^{ST}_{i}$, as the difference
from the storage power output before a fault occurs. The swing equation (30)
can be rewritten in terms of the angle deviation
$\tilde{\delta}_{i},i\in\mathcal{V}$ specifically:

$M_{i}\ddot{\widetilde{\delta}}_{i}+D_{i}\dot{\widetilde{\delta}}_{i}=\widetilde{P}^{ST}_{i}+\sum_{j\in\mathcal{N}(i)}\overline{P}_{ij}^{\max}\big{(}\sin\overline{\psi}_{ij}-\sin(\widetilde{\psi}_{ij}+\overline{\psi}_{ij})\big{)}.$

Without battery based controllers, existing power transmission systems can
synchronize bus frequencies and recover angle differences when the initial
deviation is in a suitable domain. By employing battery-based controllers as
NI controllers, the transient stability of power transmission systems can also
be guaranteed.

In what follows, we investigate a way in which the power grid can be gradually
transitioned into an NI controlled system, one transmission line at a time. We
consider co-locating batteries with generators, positioned at both ends of the
$k$th transmission line. While the edge controllers $l\neq k$ are described by
Eq. (28c), the edge controller $k$ is designed as

$\displaystyle H_{ck}:\quad\dot{x}_{ck}$
$\displaystyle=-\frac{1}{\tau}x_{ck}+\frac{K_{1}}{\tau}u_{ck},$ (31a)
$\displaystyle y_{ck}$ $\displaystyle=x_{ck}-K_{2}u_{ck},$ (31b)

where $\tau>0$ and $K_{2}>K_{1}>0$.

###### Theorem 6

Consider node plants $\mathcal{H}_{p}$ described by (28a)-(28b), edge
controllers $l\neq k$ described by the system (28c), and edge controller $k$
described by the system (31). Consider the feedback interconnection of node
plants and edge controllers based on the underlying transmission network as
depicted in Fig. 2. Then, the feedback system achieves local output consensus.

Proof. In the proof for Theorem 5, we have demonstrated that each node plant
is OSNI and fulfills Assumptions 2 and 3; furthermore, the networked node
plants satisfy Assumption 8. For edge controllers $l\neq k$, we choose the
storage function as $V_{cl}\equiv 0,l\neq k$. For the edge controller $k$, we
choose the storage function as $V_{ck}(x_{ck})=\frac{x_{ck}^{2}}{2K_{1}}$. We
can verify that
$\dot{V}_{ck}(x_{ck})=\frac{1}{K_{1}}x_{ck}\dot{x}_{ck}=\frac{1}{K_{1}}(-\tau\dot{x}_{ck}+K_{1}u_{ck})\dot{x}_{ck}\leq
u_{ck}^{\top}\dot{h}_{ck}(x_{ck}).$ Hence, the edge controller $k$ is NI.
Furthermore, a constant input $\overline{u}_{ck}$ results in a constant state
$\overline{x}_{ck}=K_{1}\overline{u}_{ck}$ and a constant output
$\overline{y}_{ck}=(K_{1}-K_{2})\overline{u}_{ck}.$ We can also verify
Assumption 5 that
$\overline{u}_{ck}^{\top}\overline{y}_{ck}=(K_{1}-K_{2})\overline{u}_{ck}^{2}\leq-\gamma_{ck}\overline{u}_{ck}^{2}$,
where $\gamma_{ck}\leq K_{2}-K_{1}$. For the feedback system, we look at the
candidate Lyapunov function (3.2) is positive definite in the domain

$\displaystyle\mathcal{D}=\bigl{\\{}\widetilde{\delta}_{i},$
$\displaystyle\forall i\in\mathcal{V}\ |\ \sum_{l\neq
k}\overline{P}_{l}^{\max}\big{(}\cos\overline{\psi}_{l}-\widetilde{\psi}_{l}\sin\overline{\psi}_{l}$
$\displaystyle-\cos(\widetilde{\psi}_{l}+\overline{\psi}_{l})\big{)}>0\bigr{\\}}\cup\\{\widetilde{\delta}_{i}=0,\forall
i\in\mathcal{V}\\}.$

Therefore, Theorem 3 can be applied and the feedback system achieves local
consensus. $\square$

###### Remark 2

The edge controller $k$ can equivalently be viewed as two end node controllers
operating in a distributed manner. The control action actuated by the battery
at the end generator bus $i$ is described by

$\widetilde{P}^{ST}_{i}=q_{ik}\big{(}x_{ck}-K_{2}u_{ck}-\overline{P}^{\max}_{l}(\sin\overline{\psi}_{k}-\sin(u_{ck}+\overline{\psi}_{k}))\big{)},$

where $q_{ik}$ is the $k$th element in the $i$th row of the incidence matrix
$\mathbf{Q}$.

##  5 Conclusion

In this paper, we proposed a networked control framework for power
transmission systems. A novel Lur’e-Postnikov-like Lyapunov function was
formulated, and stability proofs were constructed for the feedback
interconnection of two single nonlinear NI systems. Additionally, output
feedback consensus results were established for the feedback interconnection
of two networked nonlinear NI systems based on the network topology. Our
theoretical framework provided support for the design of battery-based control
in power transmission systems. We demonstrated a way in which the electric
power grid could be gradually transitioned into the proposed NI controlled
system, one transmission line at a time. A possible future research direction
will include saturation of battery actuators for networked power transmission
systems.

## References

  * Australian Energy Market Operator (2023) Australian Energy Market Operator (2023).  Transmission Expansion Options Report.  URL https://www.aemo.com.au.
  * Bergen and Hill (1981) Bergen, A.R. and Hill, D.J. (1981).  A structure preserving model for power system stability analysis.  _IEEE transactions on power apparatus and systems_ , (1), 25–35.
  * Borenstein (2022) Borenstein, S. (2022).  It’s time for rooftop solar to compete with other renewables.  _Nature Energy_ , 7(4), 298–298.
  * Chen et al. (2023a) Chen, Y., Petersen, I.R., and Ratnam, E.L. (2023a).  Design and stability of angle based feedback control in power systems: A negative-imaginary approach.  _arXiv preprint arXiv:2310.01781_.
  * Chen et al. (2023b) Chen, Y., Shi, K., Petersen, I.R., and Ratnam, E.L. (2023b).  A nonlinear negative imaginary systems framework with actuator saturation for control of electrical power systems.  _arXiv preprint arXiv:2311.06820_.
  * Dorfler and Bullo (2012) Dorfler, F. and Bullo, F. (2012).  Synchronization and transient stability in power networks and nonuniform Kuramoto oscillators.  _SIAM Journal on Control and Optimization_ , 50(3), 1616–1642.
  * Haddad and Chellaboina (2008) Haddad, W.M. and Chellaboina, V. (2008).  _Nonlinear dynamical systems and control: a Lyapunov-based approach_.  Princeton university press.
  * Hill and Bergen (1982) Hill, D. and Bergen, A. (1982).  Stability analysis of multimachine power networks with linear frequency dependent loads.  _IEEE Transactions on Circuits and Systems_ , 29(12), 840–848.
  * Hill and Chong (1989) Hill, D.J. and Chong, C.N. (1989).  Lyapunov functions of lur’e-postnikov form for structure preserving models of power systems.  _Automatica_ , 25(3), 453–460.
  * Kundur and Malik (2022) Kundur, P.S. and Malik, O.P. (2022).  _Power system stability and control_.  McGraw-Hill Education.
  * Lanzon and Petersen (2008) Lanzon, A. and Petersen, I.R. (2008).  Stability robustness of a feedback interconnection of systems with negative imaginary frequency response.  _IEEE Transactions on Automatic Control_ , 53(4), 1042–1046.
  * Petersen and Lanzon (2010) Petersen, I.R. and Lanzon, A. (2010).  Feedback control of negative-imaginary systems.  _IEEE Control Systems Magazine_ , 30(5), 54–72.
  * Shi et al. (2023) Shi, K., Petersen, I.R., and Vladimirov, I.G. (2023).  Output feedback consensus for networked heterogeneous nonlinear negative-imaginary systems with free-body motion.  _IEEE Transactions on Automatic Control_ , 68(9), 5536–5543.
  * Tran et al. (2019) Tran, V.T., Islam, M.R., Muttaqi, K.M., and Sutanto, D. (2019).  An efficient energy management approach for a solar-powered EV battery charging facility to support distribution grids.  _IEEE Transactions on Industry Applications_ , 55(6), 6517–6526.
  * Višić et al. (2020) Višić, I., Strnad, I., and Marušić, A. (2020).  Synchronous generator out of step detection using real time load angle data.  _Energies_ , 13(13), 3336.


# GelSight EndoFlex: A Soft Endoskeleton Hand with Continuous High-Resolution
Tactile Sensing

Sandra Q. Liu∗, Leonardo Zamora Yañez∗, Edward H. Adelson  ∗Authors with equal
contribution  Massachusetts Institute of Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We describe a novel three-finger robot hand that has high resolution tactile
sensing along the entire length of each finger. The fingers are compliant,
constructed with a soft shell supported with a flexible endoskeleton. Each
finger contains two cameras, allowing tactile data to be gathered along the
front and side surfaces of the fingers. The gripper can perform an enveloping
grasp of an object and extract a large amount of rich tactile data in a single
grasp. By capturing data from many parts of the grasped object at once, we can
do object recognition with a single grasp rather than requiring multiple
touches. We describe our novel design and construction techniques which allow
us to simultaneously satisfy the requirements of compliance and strength, and
high resolution tactile sensing over large areas.

##  I INTRODUCTION

The human hand has provided inspiration for many robot hands. Human fingers
contain an interior articulated skeleton, which is covered with soft skin,
providing the fingers with a combination of strength and compliance. The
fingers are rounded, with tactile sensing present throughout the skin, and
with the best tactile acuity on the front surfaces. When a person holds an
object with an enveloping grasp, the object touches the hand at a great many
points, allowing the person to recognize the object by its shape, size, and
other properties. Our goal is to create a robotic hand that emulates many of
these properties.

Figure 1: Top A CAD model of our GelSight EndoFlex gripper with some of the
parts labeled. Bottom The GelSight EndoFlex is securely grasping a Rubik’s
cube and the corresponding processed difference images of four of the six
sensing regions are displayed. Of note is that the bottom two sensor images
are showing continuous sensing along the side and corner of the cube, while
the top two sensor images are showing one image each from the other two
fingers.

The ability to identify an object using a single grasp is important and
requires “complete” sensing along the grasping surfaces of a finger. Even
though many current finger-inspired sensors can perform object recognition
well with high-resolution finger tip sensors or with low-resolution larger
tactile sensors, they either require that the object be in full contact with
the finger tips or multiple regrasps to classify the object within the hand
[1, 2]. Furthermore, they do not have the compliance afforded by soft
robotics, which can greatly improve secure grasping abilities of the gripper
or make them safer for interaction with the world around them.

In other words, soft robotic manipulators could greatly benefit from having
structural compliance and rigidity, along with high-resolution sensing of
tactile sensors. To this end, we present the following contributions:

  * •

A novel design of a continuous high-resolution tactile sensor along a curved
surface;

  * •

An endoskeleton finger design for a human-inspired gripper that incorporates
tactile sensing (Fig. 1);

  * •

A neural net that can utilize only the tactile images from a single grasp to
classify objects.

##  II RELATED WORK

###  II-A Hand Grippers

Human hand inspired grippers have been previously designed with varying
degrees of sensing, rigidness and anthropomorphism [3, 4, 5]. Although robotic
systems were historically composed of rigid materials, interest in soft
systems has quickly risen [6]. Rigid hands traditionally focused on control
systems and force transmission while neglecting contact rich sensing and
compliant gripping that more closely characterizes human hands [7]. Soft
robotics offers the advantages of compliance, robustness, and can be
compatible with high-resolution geometry sensing with camera-based sensors
[8].

Rigid robots have often enjoyed well defined kinematic models and high
strength, making them ideal manipulators for repeatable and complex motions
[9]. However, gripping often introduces a degree of uncertainty that may
require a softer touch to avoid high energy collisions [10]. Soft robotic
grippers benefit from their natural robustness and compliance which have
proven to be critical when grasping [11]. Due to their compliant nature, soft
robots are considered to have an infinite degrees of freedom leading to
challenges when developing a robust control system. However, recent advances
in simulation and robotics have led to the BCL-26, a soft gripper with 26
controllable degrees of freedom that is capable of dexterous motion with a
high degree of anthropomorphism. [12]. Other modern designs such as the RBO
Hand 3, show great promise with their dexterous manipulation and potential to
incorporate sensors due to its larger size [13].

Many attempts have been made at marrying soft and rigid robotics to achieve
flexible yet strong robots [14, 15, 16]. One approach to strengthening and
increasing precision of soft grippers has been embedding skeletons within
their structure [17]. Although the addition of an endoskeleton brings various
benefits, it also comes with some drawbacks including increased manufacturing
and modeling complexity. To combat the increase in complexity, simulation has
become a popular tool to supplement control design [18]. The properties of
soft-rigid robotics appear to be a significant step towards high fidelity
biomimetic hand grippers.

Despite the various advances in robotics to achieve a soft human-like hand,
there are still critical elements missing from current designs. Most notably,
there is an absence of rich geometric-based sensing in rigid and soft hands
alike [9, 12]. Therefore, there is still progress to be made in developing a
soft anthropomorphic hand with geometry sensing capabilities.

###  II-B Sensing and Soft Grippers

Most previous tactile sensing work in robotic grippers has been force-based
using capacitive or strain sensors [19, 20]. These sensors provide a low cost
option with high response time, but these types of sensors are better for
sensing stiff and flat surfaces [21]. Vision-based sensors can provide
additional sensing data and be highly compatible with soft robots.

Existing vision-based systems rely on cameras to capture the deformation of
some elastomer and process the footage to obtain tactile data [8]. One such
sensor is the TacTip which uses a camera to measure the deformation of a
silicone membrane and superresolution to achieve precise force localization
[22]. The soft nature and highly accurate sensing of TacTip has great
potential, but the sensor size and lack of geometry sensing limit its
application to anthropomorphic hands. The GelSight sensor family offers an
alternative with its high resolution tactile sensing and application to curved
surfaces [23]. GelSight sensors operate with a camera that views a painted
aluminum-silicone skin that can capture finely detailed tactile imprints on
its surface. This surface is then illuminated by different LEDs.

Previous GelSight sensing area has been limited by uni-camera sensing, wide
angle lenses with some distortion, and their large size [24, 25, 26, 27].
GelSight applications have seen limited integration of the sensing surface
with the gripper body [28]. Therefore, there is still space to explore soft
human-like grippers with structural integration of tactile sensors. One
potential design for extending sensing surface area is to expand on the work
of She _et al._ [25] by using two or multiple cameras to create a continuous
sensing surface. To our knowledge, no other GelSight sensor has used multiple
cameras to create one continuous and compact sensing surface. Our novel design
provides wide range GelSight sensing in a compact and soft anthropomorphic
package.

##  III METHODS

###  III-A Hardware

The EndoFlex sensor is composed of an endoskeleton encased in silicone with
two embedded cameras for continuous sensing (Fig. 2). Each endoskeleton was
designed to be one continuous piece with a pair of rigid segments and flexures
to form joints. This design minimizes the number of parts required to
fabricate one finger when compared to traditional rigid fingers. The flexure
design was chosen for its high compliance and low deformation of individual
elements to reduce silicone delamination. We 3D printed the endoskeleton using
an Onyx One printer with Markforged Onyx plastic for its combination of high
strength and relatively low tensile modulus when compared to other extruded
plastics. This combination of properties allowed minimal force loss during
actuation.

Figure 2: A close-up view of an EndoFlex finger with an exploded view. Each
finger operates independently with one degree of freedom and can be quickly
replaced if damaged.

A camera was mounted into each endoskeleton segment to prevent any shifting
during actuation. Three sets of red, green, and blue LEDs were mounted with
cyanoacrylate adhesive onto the rigid segment of the endoskeleton. They were
spaced 90 degrees apart to create a colored light gradient for the GelSight
algorithm. Finally, the endoskeleton was threaded with Piscifun Onyx Braided
Fishing Line soaked in Smooth-On Universal Mold Release to reduce friction
when cast in silicone. We chose to use cable-driven actuation to reduce
potential camera-view obstructions and also so that we could more easily
integrate the camera into the finger skeleton.

A rigid three finger palm was designed with temporary fasteners to allow for
fast replacement of damaged fingers or for future iterations. Fingers were
positioned in a ‘Y’ pattern with two fingers and an opposing thumb. The pair
of fingers was spaced thirty degrees apart to distribute grasping force
without creating collisions. The palm was designed to have a rounded feature
with a polyurethane foam layer to add grasping ability. A separate rigid plate
was designed to be fastened onto the Panda robotic arm. Three Dynamixel AX-12A
servos were mounted between the plate and the palm and served as the actuation
method for the fingers through double axle spools. The double axle design
allowed for actuated contraction and extension of each finger. The palm, plate
and spools were all printed with Markforged Onyx plastic using a Markforged
Onyx printer.

As part of our finger manufacturing process, which is fully shown in Fig. 3, a
two part mold is designed for casting silicone to create the optically clear
medium for the GelSight sensor. The mold was designed to hold the endoskeleton
during the casting process which removed the need for fasteners or adhesives
to hold the silicone layer. The mold had high curvature to create a rounded
finger much like a human finger. One major benefit of the curved surface was
the high reflection of lights within the silicone which aided in sensing by
removing shadows of pressed objects. The mold design removed any air gap
between the camera lens and cast silicone to minimize the refraction of light.
The mold was produced using a Formlabs 2 SLA printer for its high resolution.
To achieve the optical clarity required to use GelSight, the mold was
incrementally sanded with sandpaper reaching 2000 grit.

Figure 3: The manufacturing process for the EndoFlex sensor including assembly
of electronics and casting of silicone.

To allow the silicone to compress when the tendons pulled the endoskeleton
finger to a closed grasp position, we chose to synthesize a softer silicone
for the finger. As a result, we used a ratio of 1 to 15 to 5 parts of XP-565
parts A and B, and plasticizer (Phenyl Trimethicone, Lotioncrafter).
Decreasing the ratio of part A to B for the XP-565 is equivalent to adding
less catalyst, which increases the softness of the silicone, while the
addition of the plasticizer also causes the resulting cured silicone to have a
softer texture.

Before pouring the silicone mixture into the mold, we used a paint brush to
paint a thin layer of Inhibit-X (Smooth-On Inc). After waiting a few minutes
for it to dry, we sprayed a layer of Ease Release 200 (Smooth-On Inc) on the
mold. To create the sensing surface, we combined 2.5 parts 4 $\mu$m Aluminum
cornflakes (Schlenck) with a mixture of 11 parts silicone ink catalyst and
gray silicone ink base (Raw Materials Inc.) and 30 parts NOVOCS Gloss (Smooth-
On Inc), and mixed it for a minute using an ultrasonicator. This mixture was
then sprayed into the inside of the top mold with an airbrush and left to dry
for at least 10 minutes before we fit the threaded endoskeleton inside of the
mold and screwed the mold halves together. Remaining holes and the lips of the
mold were covered in a thin layer of brushed-on silicone adhesive (Devcon),
which created a seal for the mold and prevented any silicone leakage outside
of the mold that could be caused by mold warping or other printing
imperfections.

Once the main body silicone mixture had been degassed, we slowly poured the
mixture into the prepared mold. The entire mold assembly was placed on top of
a vibrating plate for 10 minutes to get rid of any bubbles in the camera-
viewable areas. These bubbles may have been induced by the silicone pouring
over the flexures, electronics, and other 3D printed parts inside of the mold.
Some of the bubbles were retained along the side of the sensor surface, which
is not viewable by the camera and did not negatively affect the sensor
integrity.

Finally, the mold was placed inside of a oven at 125∘F (52∘C) for 12 to 15
hours. This temperature was chosen to prevent any of the electronics or inner
structures from reaching their glass temperatures and causing delamination of
the parts from the silicone. Once the finger was removed from the mold, the
gray sensing membrane surface was no longer smooth and instead had a
reticulated wrinkled texture (Fig. 4). This phenomena only occurred when we
sprayed the paint on the mold first and did not occur if we chose to cure the
finger first without the paint in the mold and spray the paint on the finger
surface afterwards.

Figure 4: A close up image of the reticulated wrinkle surface of the GelSight
EndoFlex sensor. The width of one of the wrinkles is approximately 0.4 mm wide
and was only created when we first sprayed the paint on the mold surface
before casting silicone inside.

The modular fingers were then placed on our palm plate to create our completed
gripper. We also note that this configuration can be changed to enable
different types of grasps, although we chose an enveloping grasp to maximize
the amount of sensing the gripper could obtain from grasping an object in its
palm.

###  III-B Software

Each finger was equipped with two Raspberry Pi Zero spy cameras with a 160∘
field of view, for a total of six cameras. All of the cameras were able to
view a curved segment of the finger, which was illuminated by tri-directional
LEDs. The finger segment images were individually streamed using the mjpg-
streamer package and can be processed using OpenCV and fast poisson solver
[29, 30] to get difference images and uncalibrated reconstruction images, as
shown in Fig. 5.

Figure 5: From left to right, we have raw sensor images of a 3.75 mm ball
bearing array and a M2 screw, followed by their difference images from a
reference image (no tactile contact), and the corresponding uncalibrated depth
image.

##  IV EXPERIMENT

To show the usefulness of having continuous sensing, we collected single
grasps of various objects and performed a classification task based on the
entire finger sensing region. Previous works show that object classification
using finger tip sensing or low-resolution palm sensing is accurate, but only
when the objects were in contact with the fingertips or multiple touches have
been performed [1, 2].

Our grasping object set included three distinct objects from the YCB dataset:
the Rubik’s cube, one of the toy stacking cups, and a plastic orange [31].
These three objects are shown in Fig. 6. For each object, we collected
approximately 500 different grasps using all six of the cameras inside the
fingers to obtain a holistic, “full-hand” tactile view of the entire object.
To capture many different grasps, we had assistants manually reorient each
object randomly such that it could still be feasibly grasped with the gripper,
which allowed different parts of the sensor images to capture different
features of the object that was being grabbed. We also attempted grasps
utilizing a couple of the fingers instead of all of the fingers in the cases
that the third finger did not have a solid contact with the object in its
hand.

For each set of six images we captured, we stitched them together into a 2 by
3 array and used them as inputs for a Resnet-50 neural net architecture with
the three outputs as the objects we used for our grasping data set [32]. We
chose to use stochastic gradient descent as our optimizer, with a learning
rate of 1e-3 and a learning rate scheduler with a step size of 7 and a gamma
set to 0.1. We also implemented data augmentation on the entire set of images
to deal with potential inconsistent lighting or random noise output of the
images, and to account for eventual wear and tear in the silicone over time.
We split our data into training and validation sets in a 80% to 20% ratio. The
complete neural net architecture is shown in Fig. 6.

Figure 6: Neural net architecture for our single grasp classification. Once
the object has been grasped, the six images are stitched together in a 2x3
array, thrown into our Resnet architecture and classified into a toy cup, an
orange, or a Rubik’s cube.

###  IV-A Results

Grasping The GelSight EndoFlex was able to easily and very securely grasp all
of the objects in our object set. In particular, the polyethylene foam layer
on the palm provided a compliant, deformable surface that the grasped objects
could be pressed against. The hand was also able to grasp empty water bottles
without crushing them, as well as heavier objects, like a drill with a
battery, without dropping them. As expected, the compliance of the soft gel
allowed us to grasp more fragile objects, while the rigid endoskeleton allowed
the fingers to withstand the force and weight of a heavier object.

Each finger was also able to bend to around 60∘ at each flexure point using
the Dynamixel motors. Because the silicone was quite soft and because we added
human finger-inspired grooves along the flexures, when the fingers bent, the
silicone was able to more easily compress around the sides. However, the
silicone still obstructed some of the bending angle, and as a result, the
endoskeleton finger was unable to bend to its full 90∘ range that it would
have been able to otherwise. Furthermore, deepening the grooves to facilitate
bending would have limited the sensing area and ultimately interfered with the
continuous sensing. Nonetheless, this limitation in motion did not severely
limit the hand’s ability to grasp objects because the deformable silicone
surface over the endoskeleton finger helped to accommodate any loss of motion
with its compliance and softness.

Casting the mold while the finger was in a slightly bent position helped to
prevent creases in the surface of the silicone when the finger bent. Doing so
also prevented silicone creasing when the finger was straightened out since
the sensing surface was pulled in tension. Unfortunately, over time, pulling
the silicone finger in tension caused parts of the silicone in the base of the
finger to slightly tear. We believe that this problem could potentially be
mitigated by using a softer silicone with higher elongation.

Finally, given a different arrangement of fingers or with a finger that could
behave more thumb-like with an added degree of freedom, we believe that these
fingers have the potential to grasp an even larger variety of objects.

Tactile Sensing As designed, the finger was able to continuously sense along
the entire length of the finger when it was in a “closed” position. The
fingers were also able to sense along the sides as well, although some sensing
was slightly lost at the very tips of the fingers.

Overall, the finger was able to provide extremely high resolution sensing and
the raw sensor images were able to capture details that previous GelSight
sensors could sense, but with additional sensing coverage due to its rounded
shape and the wider camera field of view. However, the wider camera field of
view and the curved shape caused some distortion in the sensing image, which
is most apparent on the sides of the image frame.

Additionally, some of the sensing surfaces appeared to have distinct rings of
lights around the different color channels instead of the blending we would
have expected from using a Lambertian paint on the surface of the silicone
gel. We believe that this phenomena could have been caused by slight
delamination of the silicone from the LEDs. The addition of the air interface
will cause the light to refract from the air to the silicone face and
potentially cause these rings of light to form and prevent even blending of
the light within the silicone. In particular, we noticed that when objects
were pressed against these sensing surfaces, the light circles began to
dissipate. Nonetheless, this did not affect the sensor resolution and the
distinct features of the objects were still distinguishable as the tactile
sensor had extremely high-resolution.

Finally, we noticed that the wrinkles, which were manufactured on some of the
finger sensing surfaces, were helpful in preventing tears in the silicone
membrane. Unlike the smoother sensing surfaces, it seems like the wrinkles
helped to mitigate the high stress points caused by sharp corners poking into
the sensor surface. The surfaces with wrinkles also felt like they had less
friction than the smoother surfaces. Although the wrinkled surface made
surface reconstruction difficult because the wrinkled texture appeared in
difference images, they did not seem to negatively affect our object
classification. The effect could also have been mitigated since we noticed
that if enough pressure was put on the sensor surface, the wrinkles would
smooth out slightly, which would not affect object classification results.

Object Classification Our object classification model was able to obtain 94.1%
accuracy on our validation set. In live testing, which consisted of our
robotic hand grasping the 3 objects ten times each, we were able to correctly
classify 80% of the objects. The orange was able to be recognized 9 out of 10
times, while the classifier slightly struggled with distinguishing between the
Rubik’s cube and the toy cup (80% and 70% accuracy, respectively). We believe
that the discrepancy in the validation set results and the live testing
results could be due to slight tears that developed over the course of the
data collection and testing. Regardless, the hand was able to only use a
single grasp to recognize the identity of an object.

As we expected, the orange, which had the most distinguishable tactile
features, was the easiest for our model to recognize. Not only was the orange
covered in an unique bumpy skin texture, it also had a distinctive stem
portion. On the other hand, unless the fingers directly pressed against a
corner of the Rubik’s cube or along multiple smaller cubes, it was hard to
visually distinguish some of its edges from the edges at the bottom and top of
the toy cup.

We believe that this confusion between the Rubik’s cube and the toy cup could
be mitigated by adding a palm, which could also provide additional sensing.
The added sensing from a larger area on the palm could have helped capture
more tactile details that may have been missed by the fingers. Regardless, the
object classification using continuous sensing along the multi-fingered hand
was fairly robust and able to perform well on our object set. Specifically, it
could be useful for grabbing objects in the dark or in an occluded environment
where external vision would not be useful or could not be used.

##  V CONCLUSION AND DISCUSSION

In this paper, we present the novel design of a continuous high-resolution
tactile sensor incorporated into a finger, which was then integrated into a
human-like hand. The hand was then able to use these large sensing ranges to
be able to somewhat accurately classify objects using a single grasp, which,
to the authors’ knowledge, has not been done before. The ability to identify
an object with a single grasp is akin to the way we as humans are able to grab
an object with some priors and without external vision and determine almost
immediately what we are holding.

Although recent research has focused a lot on large range low-resolution
tactile sensors or high-resolution fingertip sensors for dexterous
manipulation, not much research has been done on high-resolution sensing
across the majority of a finger’s surface. Having this added sensing allows us
to perform many useful classification tasks, and doing so in a soft, compliant
gripper allows us to also safely and securely interact with objects and the
surrounding environment. Sensors similar to the GelSight EndoFlex have the
ability to be used for home-care robots or for human-robot interaction, where
compliance and sensing are key to success.

Future work on this gripper involves adding a thumb-like joint, as well as
full fingertip sensing, which can greatly improve the usability of the gripper
for sensing and dexterous manipulation tasks. We can also continue to draw
inspiration from GelSight sensors and add markers which could help track slip
and shear or torsional forces along the surfaces of the finger. Overall, our
novel endoskeleton finger design begins to solve the problem of designing
human-inspired soft-rigid robotic hands with high-resolution sensing that are
capable of performing more and more complicated tasks.

##  VI ACKNOWLEDGEMENTS

This work was supported by funds from the Toyota Research Institute, the
Office of Naval Research, and the SINTEF BIFROST (RCN313870) project. The
authors would also like to thank James M. Bern and Megha H. Tippur for their
helpful advice and design tips.

## References

  * [1] W. Bottcher, P. Machado, N. Lama, and T. M. McGinnity, “Object recognition for robotics from tactile time series data utilising different neural network architectures,” 2021. [Online]. Available: https://arxiv.org/abs/2109.04573
  * [2] S. Pohtongkam and J. Srinonchat, “Tactile object recognition for humanoid robots using new designed piezoresistive tactile sensor and dcnn,” _Sensors_ , vol. 21, no. 18, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/18/6024
  * [3] M. Tavakoli and A. T. de Almeida, “Adaptive under-actuated anthropomorphic hand: Isr-softhand,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2014, pp. 1629–1634.
  * [4] C.-H. Xiong, W.-R. Chen, B.-Y. Sun, M.-J. Liu, S.-G. Yue, and W.-B. Chen, “Design and implementation of an anthropomorphic hand for replicating human grasping functions,” _IEEE Transactions on Robotics_ , vol. 32, no. 3, pp. 652–671, 2016.
  * [5] B. Choi, S. Lee, H. R. Choi, and S. Kang, “Development of anthropomorphic robot hand with tactile sensor : Skku hand ii,” in _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2006, pp. 3779–3784.
  * [6] F. Iida and C. Laschi, “Soft robotics: Challenges and perspectives,” _Procedia Computer Science_ , vol. 7, pp. 1877–0509, 2011.
  * [7] F. Lotti and G. Vassura, “A novel approach to mechanical design of articulated fingers for robotic hands,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , vol. 2, Sep. 2002, pp. 1687–1692 vol.2.
  * [8] W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” _Sensors_ , vol. 17, no. 12, 2017\. [Online]. Available: https://www.mdpi.com/1424-8220/17/12/2762
  * [9] C. Borst, M. Fischer, S. Haidacher, H. Liu, and G. Hirzinger, “Dlr hand ii: experiments and experience with an anthropomorphic hand,” in _2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422)_ , vol. 1, 2003, pp. 702–707 vol.1.
  * [10] D. Rus and M. Tolley, “Design, fabrication and control of soft robots,” _Nature_ , vol. 521, pp. 467–475, 2015.
  * [11] J. Shintake, V. Cacucciolo, D. Floreano, and H. Shea, “Soft robotic grippers,” _Advanced Materials_ , vol. 30, no. 29, p. 1707035, 2018.
  * [12] J. Zhou, X. Chen, U. Chang, J.-T. Lu, C. C. Y. Leung, Y. Chen, Y. Hu, and Z. Wang, “A soft-robotic approach to anthropomorphic robotic hand dexterity,” _IEEE Access_ , vol. 7, pp. 101 483–101 495, 2019.
  * [13] S. Puhlmann, J. Harris, and O. Brock, “RBO hand 3 - A platform for soft dexterous manipulation,” _CoRR_ , vol. abs/2201.10883, 2022. [Online]. Available: https://arxiv.org/abs/2201.10883
  * [14] H. D. Yang and A. T. Asbeck, “A new manufacturing process for soft robots and soft/rigid hybrid robots,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 8039–8046.
  * [15] N. K. Uppalapati, B. Walt, A. J. Havens, A. Mahdian, G. Chowdhary, and G. Krishnan, “A berry picking robot with a hybrid soft-rigid arm: Design and task space control.” in _Robotics: Science and Systems_ , 2020.
  * [16] J. M. Bern, L. Z. Yañez, E. Sologuren, and D. Rus, “Contact-rich soft-rigid robots inspired by push puppets,” in _2022 IEEE 5th International Conference on Soft Robotics (RoboSoft)_ , 2022, pp. 607–613.
  * [17] J. M. Bern, F. Zargarbashi, A. Zhang, J. Hughes, and D. Rus, “Simulation and fabrication of soft robots with embedded skeletons,” in _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 5205–5211.
  * [18] E. Coevoet, Y. Adagolodjo, M. Lin, C. Duriez, and F. Ficuciello, “Planning of soft-rigid hybrid arms in contact with compliant environment: Application to the transrectal biopsy of the prostate,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4853–4860, 2022.
  * [19] A. Schmitz, P. Maiolino, M. Maggiali, L. Natale, G. Cannata, and G. Metta, “Methods and technologies for the implementation of large-scale robot tactile sensors,” _IEEE Transactions on Robotics_ , vol. 27, no. 3, pp. 389–400, 2011.
  * [20] R. A. S. I. Subad, L. B. Cross, and K. Park, “Soft robotic hands and tactile sensors for underwater robotics,” _Applied Mechanics_ , vol. 2, no. 2, pp. 356–382, 2021. [Online]. Available: https://www.mdpi.com/2673-3161/2/2/21
  * [21] I. Khodasevych, S. Parmar, and O. Troynikov, “Flexible sensors for pressure therapy: Effect of substrate curvature and stiffness on sensor performance,” _Sensors_ , vol. 17, no. 10, 2017. [Online]. Available: https://www.mdpi.com/1424-8220/17/10/2399
  * [22] N. F. Lepora and B. Ward-Cherrier, “Superresolution with an optical tactile sensor,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2015, pp. 2686–2691.
  * [23] B. Romero, F. Veiga, and E. Adelson, “Soft, round, high resolution tactile fingertip sensors for dexterous robotic manipulation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 4796–4802.
  * [24] S. Wang, Y. She, B. Romero, and E. Adelson, “GelSight Wedge: Measuring high-resolution 3d contact geometry with a compact robot finger,” 06 2021.
  * [25] Y. She, S. Q. Liu, P. Yu, and E. Adelson, “Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 10 075–10 081.
  * [26] A. Wilson, S. Wang, B. Romero, and E. Adelson, “Design of a fully actuated robotic hand with multiple gelSight tactile sensors,” 2020. [Online]. Available: https://arxiv.org/abs/2002.02474
  * [27] S. Q. Liu and E. H. Adelson, “GelSight Fin Ray: Incorporating tactile sensing into a soft compliant robotic gripper,” in _2022 IEEE 5th International Conference on Soft Robotics (RoboSoft)_ , 2022, pp. 925–931.
  * [28] Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. Adelson, “Cable manipulation with a tactile-reactive gripper,” _The International Journal of Robotics Research_ , vol. 40, no. 12-14, pp. 1385–1401, 2021. [Online]. Available: https://doi.org/10.1177/02783649211027233
  * [29] G. Bradski, “The OpenCV Library,” _Dr. Dobb’s Journal of Software Tools_ , 2000.
  * [30] J. Doerner, “Fast poisson reconstruction in python,” https://gist.github.com/jackdoerner/b9b5e62a4c3893c76e4c, 2014.
  * [31] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” in _2015 International Conference on Advanced Robotics (ICAR)_ , 2015, pp. 510–517.
  * [32] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015. [Online]. Available: https://arxiv.org/abs/1512.03385